Question: }) } } <ins> func TestGenerateAPIPodStatusPodIPs(t *testing.T) { testcases := []struct { name      string nodeIP    string criPodIPs []string podIPs    []v1.PodIP }{ { name:      "Simple", nodeIP:    "", criPodIPs: []string{"10.0.0.1"}, podIPs: []v1.PodIP{ {IP: "10.0.0.1"}, }, }, { name:      "Dual-stack", nodeIP:    "", criPodIPs: []string{"10.0.0.1", "fd01::1234"}, podIPs: []v1.PodIP{ {IP: "10.0.0.1"}, {IP: "fd01::1234"}, }, }, { name:      "Dual-stack with explicit node IP", nodeIP:    "192.168.1.1", criPodIPs: []string{"10.0.0.1", "fd01::1234"}, podIPs: []v1.PodIP{ {IP: "10.0.0.1"}, {IP: "fd01::1234"}, }, }, { name:      "Dual-stack with CRI returning wrong family first", nodeIP:    "", criPodIPs: []string{"fd01::1234", "10.0.0.1"}, podIPs: []v1.PodIP{ {IP: "10.0.0.1"}, {IP: "fd01::1234"}, }, }, { name:      "Dual-stack with explicit node IP with CRI returning wrong family first", nodeIP:    "192.168.1.1", criPodIPs: []string{"fd01::1234", "10.0.0.1"}, podIPs: []v1.PodIP{ {IP: "10.0.0.1"}, {IP: "fd01::1234"}, }, }, { name:      "Dual-stack with IPv6 node IP", nodeIP:    "fd00::5678", criPodIPs: []string{"10.0.0.1", "fd01::1234"}, podIPs: []v1.PodIP{ {IP: "fd01::1234"}, {IP: "10.0.0.1"}, }, }, { name:      "Dual-stack with IPv6 node IP, other CRI order", nodeIP:    "fd00::5678", criPodIPs: []string{"fd01::1234", "10.0.0.1"}, podIPs: []v1.PodIP{ {IP: "fd01::1234"}, {IP: "10.0.0.1"}, }, }, { name:      "No Pod IP matching Node IP", nodeIP:    "fd00::5678", criPodIPs: []string{"10.0.0.1"}, podIPs: []v1.PodIP{ {IP: "10.0.0.1"}, }, }, { name:      "No Pod IP matching (unspecified) Node IP", nodeIP:    "", criPodIPs: []string{"fd01::1234"}, podIPs: []v1.PodIP{ {IP: "fd01::1234"}, }, }, { name:      "Multiple IPv4 IPs", nodeIP:    "", criPodIPs: []string{"10.0.0.1", "10.0.0.2", "10.0.0.3"}, podIPs: []v1.PodIP{ {IP: "10.0.0.1"}, }, }, { name:      "Multiple Dual-Stack IPs", nodeIP:    "", criPodIPs: []string{"10.0.0.1", "10.0.0.2", "fd01::1234", "10.0.0.3", "fd01::5678"}, podIPs: []v1.PodIP{ {IP: "10.0.0.1"}, {IP: "fd01::1234"}, }, }, } for _, tc := range testcases { t.Run(tc.name, func(t *testing.T) { testKubelet := newTestKubelet(t, false /* controllerAttachDetachEnabled */) defer testKubelet.Cleanup() kl := testKubelet.kubelet if tc.nodeIP != "" { kl.nodeIPs = []net.IP{net.ParseIP(tc.nodeIP)} } pod := podWithUIDNameNs("12345", "test-pod", "test-namespace") criStatus := &kubecontainer.PodStatus{ ID:        pod.UID, Name:      pod.Name, Namespace: pod.Namespace, IPs:       tc.criPodIPs, } status := kl.generateAPIPodStatus(pod, criStatus) if !reflect.DeepEqual(status.PodIPs, tc.podIPs) { t.Fatalf("Expected PodIPs %#v, got %#v", tc.podIPs, status.PodIPs) } if status.PodIP != status.PodIPs[0].IP { t.Fatalf("Expected PodIP %q to equal PodIPs[0].IP %q", status.PodIP, status.PodIPs[0].IP) } }) } } </ins>

What happened: The is taken from without regard for the main family of the cluster. No warning or error is given if gets the wrong family. What you expected to happen: That is taken from the "main" ip family, or at the very least the POD is not started and an error is given if it's not. How to reproduce it (as minimally and precisely as possible): Use the CNI-plugin and ipam. Use a single node cluster to avoid that assign the same address to PODs in different nodes. This may be CRI-plugin dependent, I use . The PODs are assigned dual addresses which has been supported since v1.9. Note the order; IPv6 first. In a dual-stack or a single-stack cluster with main family IPv4 will get an IPv6 address even if the main family is IPv4; Anything else we need to know?: First, note that this is not a dual-stack problem. may get the wrong family in a single-stack cluster if the PODs has dual addresses. The bug may however have been introduced with the dual-stack support, before K8s may have selected the correct family, I have not checked that. This is reported and discussed in There it is considered as a CNI-plugin problem, the CNI-plugin must present addresses in the order K8s wants them. Problem is that CNI-plugins doesn't know this. The addresses takes the path; The problem can be addressed in either of these places. The current situation is that the full responsibility lies on the , it must send the addresses in the "correct order". This is not the best place. The CNI-plugins are not a K8s-only thing and to require that a "prefered-family" or something similar must be supported by CNI-plugins to be "Kubernetes compliant" should be avoided. The CRI-plugin is a K8s-only thing and a "prefered-family" can be introduced so the CRI-plugin can sort the address array before sending it to . This is however undesirable since it adds a configuration complexity, it put a responsibility on the user or installation tool to configure the "prefered-family" to all CNI-plugins now and in the future. The best is to handle this in K8s itself where the "main" family is known. Is this really necessary?

All communication works even if is not of the main family. EndpointSlices seem to get this right regardless of order. Endpoint on the other hand seem to take the (old) podIP. So perhaps; is sufficient? If really must be , then K8s should sort the array so belongs to the main family. The problem exist on all supported K8s versions and on "master". If installed in the default way the situation is; Calico always send IPv4 first Cilum always send IPv6 first I don't think you can control the order, but I have not asked. Environment: Kubernetes versions: v1.17.12, v1.18.9, v1.19.2, v1.20.0-alpha.1, master v1.20.0-alpha.1.257+ Cloud provider or hardware configuration: None OS (e.g: ): xcluster Kernel: linux-5.8.1 Install tools: None Network plugin and version: bridge, host-local v0.8.7 Others: CRI-plugin: cri-o 1.18.3
/sig network
For the record; all communication via services works perfectly fine even if the order in is "faulty" with not belonging to the main family in k8s v1.18.8.
I think this is a duplicate of CNI plugins must report multiple IPs in the order expected by the cluster. The podIPs[0] == podIP requirement is for backwards compatibility with clients that do not read podIPs. Server-side reordering causes problems with clients that continually write/reconcile until the server list exactly matches their local list.
The problem exist in single-stack K8s clusters when PODs has dual addresses (supported since K8s v1.9, i.e way before K8s dual-stack which came in v1.16) And this issue is not a duplicate of . That issue is about a problem when the CNI-plugin uses the "correct" order. This issue is about the fact that K8s puts undocumented requirements on the CNI-plugins. There is no requirement on order in the . And since this requirement is undocumented it is not supported for instance by cilium and calico. The itself should be the requirement is for backwards compatibility with clients that do not read podIPs.

Since they doesn't read podIPs the value of should not matter to them, or? I think this should be handled in K8s, but if the responsibility is pushed downhill to the CNI-plugins then two things must be done in K8s; extra requirement on "Kubernetes compliant" CNI-plugins (outside the CNI spec) must be formally documented and communicated to the CNI-plugin providers. a "faulty" CNI-plugin accidently put's an IP-address of the "wrong" family in the first item then K8s must report an error and not start the POD. Please note again that this is not a K8s dual-stack problem. In a single-stack IPv6 K8s cluster the CNI-plugins must provide an IPv6 address first in the list.
The reason podIP must match podIPs[0] is related to updates to an existing object by an old client. See for a related problem when this was not done. An old client updated an existing object, setting only the podIP field (which is the only one it was aware of). The API server needed to detect that and populate the podIPs field correctly for new clients to read. The detection is based on the mismatch between podIP and podIPs[0], and the server responds by populating . That means that new clients (aware of both fields) must set podIP and podIPs[0] to match.
So this is weird, but... dual-stack clusters do not actually have a "main family", because decisions about IP families are made by multiple components, and kubernetes clusters do not have a single source of configuration truth that applies to all components. So it is explicitly allowed to have kube-apiserver default to IPv4-primary IPv6-secondary for services, and have your network plugin default to IPv6-primary IPv4-secondary for pod IPs, and to have some nodes have a single IPv4 address and some have a single IPv6 address and some have dual-stack addresses. It's not super clear why you'd want to do that, but we decided we had to allow it, simply because there's no easy way to prevent it. Of course in scenarios where the entire cluster is being installed/deployed by a single tool, one would expect that tool to configure all of the components in a consistent way. There is no "order K8s wants them".

Kubernetes doesn't want the addresses in any order. You want them in a particular order. Therefore you must configure the CNI plugin to return them in that order. Eg, in the example above, you should reverse the two in the config. Kubernetes's interpretation of this is that if Cilium sends the IPv6 address first, then it is telling Kubernetes that Kubernetes ought to prefer the IPv6 one. It appears that the CNI spec doesn't actually say that though... It doesn't seem to say anything about the ordering of multiple IPs... So maybe that should be clarified. If the CNI authors feel that the ordering of returned addresses is not meaningful then perhaps kubelet's behavior could be changed.
Thoughts on this?
CNI doesn't make any statement about ordering of IPs. It definitely doesn't have any notion of a "primary IP". It seems that containerd itself does have some
I mean that's literally what I just said :stuckouttongue:. The question is more if that is intentional and/or changeable. It seems strange that the decision about whether to allocate IPv4 and/or IPv6 IPs is left up to the network plugin, with no input from the runtime, but the decision about what to do with IPv4 and/or IPv6 IPs is then left up to the runtime, with no input from the network plugin.
That is very good an as it should be but... The quote is from the initial commant. Please note that this is a single-stack cluster! And that has a main family even though the PODs have dual addresses.
Right; don't comment on issues just before going to bed. Ooops. I don't think we want to introduce the notion of IP ordering / or preference in to CNI for a few reasons -- It's a bit of a breaking change, depending on how you read the spec It would be awkward to push the "preferred family" from k8s all the way down through to CNI It doesn't match how Linux works - interfaces don't have primary IPs (legacy aside). So fixing this in the apisever is probably the right place? BTW, what, exactly, defines a cluster's address family? ClusterCIDR, but that's optional (as is PodCIDR). ServiceCIDR, certainly. Is there an explicit config flag?

no flag, you configure the cluster with IPv4 or IPv6 addresses, of course if you have pod with IPv4 addresses and service with IPv6 that will not work kubelet set the status.PodIPs In a single-stack cluster you don't have a main family, you only have ONE family If we don't have PodIPs and only PodIP in a single-stack cluster this will be clearly a configuration problem in the CNI,no? I personally, think that this is a configuration problem and that we created it because we pluralised PodIP without a feature gate
Yes, and this is how it looks in an IPv4 single-stack cluster ( NOT set) if the CNI-plugin has been configured for dual addresses on PODs but returns the assigned addresses in the "wrong" order (ipv6 first);
Why is requesting an IPv6 address for pods in an IPv4 cluster not a configuration error? To me it seems a bit like running an ARM image on an AMD64 machine; Kubernetes will go through some motions but will not make it work or tell you what you did wrong.
Dual addresses in PODs has been supported from K8s v1.9, i.e. long before K8s dual-stack so that is not a configuration error. In the CNI specification there is nothing about required order for addresses or "prefered family" so it is not a configuration error for the CNI-plugin either. For K8s functions the order in does not matter. But design of upgrade requires that see Problem is that K8s (kubelet) sets "podIP = podIPs[0]" without checking the family. Here some error should be in place, like "CNI-plugin not compatible with K8s" and the POD should not be started, or better, sort the addresses or remove the requirement. This is a K8s problem. CNI-plugins should not have to care about order of reported addresses at least not without a CNI specification update.
This doesn't appear to answer my question. What is the meaning of an IPv6 pod address in an IPv4 cluster?
A cluster does not have an address family. Usually it seems like it does but that's just because the administrator has configured all of the individual components in a consistent way. But most components' configuration only affects what that component does (though in some cases that will have knock-on effects).

kube-controller-manager sets based on its (assuming is set) The network plugin (indirectly) sets (which may be affected by kcm's choice of ) based on its own configuration Either kubelet or an external cloud provider sets : based on , if passed, in the non-cloud-provider case based on cloud-provider-specific configuration (or hardcoded defaults) in the cloud case kubelet sets from kube-apiserver sets based on its kube-controller-manager creates endpoints, mostly based on the IP families chosen by kube-apiserver, except in the case of headless selectorless services, where it lets the user set endpoints of whatever family it wants, because it doesn't usually know whether the cluster is single or dual stack. kube-proxy determines whether rules are actually created for IPv4 and/or IPv6 services, based on the passed in if the dual stack feature gate is disabled, and unconditionally dual stack if the feature gate is enabled the cloud provider determines whether services get IPv4 and/or IPv6 IPs, based on cloud-provider-specific config or hardcoded defaults. (EDIT: well, and probably ) This... is kind of a mess, but there's no real way to fix it centrally given the way that kubernetes works. But we could add a config option to kubelet to unambiguously specify whether it should act like it's in a SSIPv4/SSIPv6/DSIPv4Primary/DSIPv6Primary cluster, and then make assignment make use of that.
Oof. Well. Maybe it makes sense for kubelet to stop setting PodIP, and leave that decision to the apiserver? It could make that decision based on ServiceCIDR, and order the families accordingly?
No, the dual stack KEP even explicitly calls out the fact that that it doesn't work that way: There is no single source of truth for kubernetes cluster config, and thus there is no single truth about whether the cluster is IPv4, IPv6, dual-stack-IPv4-primary, or dual-stack-IPv6-primary. If we want kubelet to impose some policy on PodIPs, then we need to tell it what policy to impose, because it has no entirely-reliable way of determining what policies any other component is imposing.
That is still much better than to require this function from the CNI-plugin which is not even K8s and follows a specification.

Please note that the networking e.g services, external access, works fine whatever order are in and whatever value has. I have not found any communication problem in K8s = v1.19 caused by this. The value of seem more "cosmetic". But it looks wrong to see ipv6 addresses of PODs in an ipv4-only cluster (). The problem seem to be an unfortunate upgrade design that imposed the constraint If that can some way be relaxed this becomes a no-problem as I see it.
/triage accepted /assign
What is the material impact of this behavior?
we can not relax because that will break existing clients. a side note. Using dualstack configured CNI on a none dual stack cluster was pre dualstack hack, if it must remain in use then CNI should only report one IP not two.
what's your reasoning to change CNI rather than kubelet?
We can't just put undocumented requirements on the CNI-plugins, they are not part of K8s, they have a formal specification and they have other users beside K8s. At least K8s must formally document the extra requirements on CNI-plugins that K8s has for the CNI-plugin to be K8s compliant.
As highlighted on the discussion there is no single truth of what clusterIP family is. Also kubelet will have to depend on which is not always used by CNIs. You can have a perfectly functional cluster that does not use PodCIDRs at all (I believe Cilium for example does not use it). Also I am not saying change CNI. A well configured CNI (single family for single stack cluster) will not have this problem.
Is the latter statement a retraction of the former? If not, how can they be reconciled?
He's saying that an administrator deploying a CNI plugin in a single-stack kubernetes cluster ought to configure that CNI plugin to return a single IP address, of the family being used by the kubernetes cluster. A CNI plugin that does not allow you configure what address families it uses may not be compatible with all kubernetes clusters. Eg, a CNI plugin that only supports IPv4 cannot be used as the network plugin in a single-stack IPv6 kubernetes cluster.
So, to sum up: Kubelet will always use the first IP returned by the CNI plugin as , regardless of any other cluster configuration.

Some CNI plugins, when configured to do dual stack, do not give you a way to configure which IP comes first, meaning kubelet may pick a that is not the IP family the admin considers the cluster's primary IP family In dual-stack clusters, this does not appear to cause any actual problems, other than that the "default" pod IPs shown in "" may not be the ones the administrator wants Nothing in the current kubelet configuration is supposed to indicate whether kubelet should prefer IPv4 or IPv6 pod IPs. It would be possible to add a new config option indicating this, or we could just declare that should also affect the sorting of pod IPs, because really why wouldn't you want that? In single-stack clusters, it has been suggested that the right answer is "don't do that then"; ie, the administrator should configure the CNI plugin to only return a single IP, and if they can't do that, then that's the CNI plugin's problem, not Kubernetes's
oh, and: if the CNI plugin returns multiple IPs which are not a dual-stack pair (eg, if it returns 2 IPv4 IPs, or 1 IPv4 IP and 2 IPv6 IPs), then kubelet will fail to update the pod at all, because it assumes it can just copy the IPs returned from CNI into , but the apiserver will only accept if it is either a single IP or a dual-stack pair.
That would be excellent! Then it would not be necessary to stop supporting dual-stack pods in a single stack K8s. And Cilium users wanting to upgrade their ipv4-only cluster to dual stack would not suddenly see ipv6 addresses on a .
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Just for the record, this is an actual problem when trying to deploy rook-ceph. Rook makes strong assumption about family being the same as family and just fails to create Ceph cluster when they do not match. are configured to listen only on but the advertised addresses are the corresponding service .

This result in listening only on pod IPv6 address but being advertised using the service IPv4 address.


Question: Pod:         st.MakePod().Name("ignore").UID("ignore").PVC("unknownPVC").Obj(), NumAllNodes: 2, Diagnosis: framework.Diagnosis{ <del> NodeToStatusMap:      framework.NodeToStatusMap{}, </del> <ins> NodeToStatusMap: framework.NodeToStatusMap{ "node1": framework.NewStatus(framework.UnschedulableAndUnresolvable, `persistentvolumeclaim "unknownPVC" not found`).WithFailedPlugin("VolumeBinding"), "node2": framework.NewStatus(framework.UnschedulableAndUnresolvable, `persistentvolumeclaim "unknownPVC" not found`).WithFailedPlugin("VolumeBinding"), }, </ins> PreFilterMsg:         `persistentvolumeclaim "unknownPVC" not found`, UnschedulablePlugins: sets.New(volumebinding.Name), },

When PreFilter returns UnschedulableAndUnresolvable, we don't need to run the preemption, but currently we run. This unexpected preemption could cause errors from the preemption by trying to read non-exist PreFilter data. As pointed out (), actually we didn't run the preemption in such cases in the past versions, but introduced this bug. /kind bug /kind regression /sig scheduling /triage accepted /priority important-soon
/assign
will fix this.


Question: } } <ins> func TestPodUpdateAllContainersClearsNodeStatus(t *testing.T) { node := makeHealthyNode("machine", "1.2.3.5") pod1 := makePod(api.NamespaceDefault, "foo", "machine", "bar") pod2 := makePod(api.NamespaceDefault, "baz", "machine", "qux") config := podCacheTestConfig{ kubeletContainerInfo: api.PodStatus{ Info: api.PodInfo{"bar": api.ContainerStatus{}}}, nodes: []api.Node{*node}, pods:  []api.Pod{*pod1, *pod2}, } cache := config.Construct() if len(cache.currentNodes) != 0 { t.Errorf("unexpected node cache: %v", cache.currentNodes) } key := objKey{"", "machine"} cache.currentNodes[key] = makeUnhealthyNode("machine").Status cache.UpdateAllContainers() if len(cache.currentNodes) != 1 { t.Errorf("unexpected empty node cache: %v", cache.currentNodes) } if !reflect.DeepEqual(cache.currentNodes[key], node.Status) { t.Errorf("unexpected status:n%#vnexpected:n%#vn", cache.currentNodes[key], node.Status) } } </ins> func TestPodUpdateAllContainers(t *testing.T) { pod1 := makePod(api.NamespaceDefault, "foo", "machine", "bar") pod2 := makePod(api.NamespaceDefault, "baz", "machine", "qux")

I have 3 identical nodes, 1 replication controller with 2 replicas of a pod. I simulate failure of one of the nodes to where pods have been scheduled by simply unplugging the node. Kubernetes realizes that the node has failed, but the pod gets not re-scheduled to the another available node. The documentation at says the following: and So is that a mistake in the documentation or a missing or a bug? I use v0.10.
That's definitely a bug, and something that used to work correctly. I will investigate today, and follow up on this bug. Thanks for the report. Brendan On Feb 5, 2015 8:11 AM, "peter-braun" wrote:
Actually, it's really a missing feature -- . I believe the last of the mechanisms we needed were ~last week. said he would start on it.
Is there a rudimentary solution that we could get in quickly, and then improve next week?
Just sent a simple PR with the fix.
Note that this issue is about a replication-controller set of pods. This isn't which is a broader "fix pods on failed nodes in general" issue.
Ah, a cache bug. Will take a look.
Re-opening this, because I reverted the relevant code in the replication controller with
I'll comment on the other thread later. Sorry I was out yesterday. I'm a bit free this weekend and should be able to work on the problem for at least one day.
Fixed by . Re-open if not fixed.
was merged a few days ago. It referenced , but not this issue.


Question: "--config", es.kubeletStaticPodDir, "--file-check-frequency", "10s", // Check file frequently so tests won't wait too long "--v", LOG_VERBOSITY_LEVEL, "--logtostderr", <ins> "--network-plugin=kubenet", "--pod-cidr=10.180.0.0/24", // Assign a fixed CIDR to the node because there is no node controller. "--hairpin-mode=hairpin-veth", </ins> ) cmd := exec.Command("sudo", cmdArgs...) hcc := newHealthCheckCommand(

Doesn't seem to haven been caused by my change:
/cc
The test only failed on coreos, and the is full of error messages. Plus the log is almost unreadable because of the extremely high verbosity. I filed for that. shows that the node has been rebooted a couple of times. And the kernel log shows This looks like . The hairpin mode is set to hairpin-veth. /cc
Yeah don't run with hairpin-mode I guess. is coreos using kubenet?
Node problem detector should detect this issue and report KernelDeadLock condition.
Yeah, but this is node e2e test, we don't have node problem detector running on node e2e cluster.
Not sure how to assign coreos issues... Assigning to people to reassign based on thread activity. Please reassign correctly!
ping!
/cc
This happens in node e2e test right? I believe node e2e test is NOT using kubenet.
Running node e2e with kubenet+hairpin-veth could prevent this.
Sorry, this is lost in github notifications.. Feel free to ping me on slack if I am not responding in a day next time :)
To run kubenet on a node, the framework needs to download and unpack the cni plugin in kubelet needs to run with and a fixed pod CIDR (since there is no node controller) I did those in , but it failed on the trusty images because is not installed. We need to either install this in the node e2e framework, or bake it in to the image. what's the best practice for installing packages on the images?
I believe updating is the best way to bake it into the image, and I also think that should be preferred over installing it each time. You could do the same with CNI. Thanks for working on this!
I updated the PR to install nsenter on the trusty images. As for cni, I still keep it in the node e2e framework to install it every time because 1) it only downloads from GCS and untar a file of ~14MB, 2) the tarball is maintained and updated by It'd be an extra burden to update the images just for cni updates.
OK with the plan.

Seeing the error message again after switching to kubenet+hairpin-veth do you want to take a look at the logs?
let me take a look
urrrh... we hit it again. Switching back to
There's a difference in the above kernel log; the device is not a interface but rather an interface (presumably the one within a container's net ns since there is no eth0 on the host). I think all the prior instances of this were veths on the host.
True. This one is slightly different. I think deadlock on interface also showed up before.
Haven't seen any new occurrence since landed. We'll know in a day or so whether this problem has been fixed.
AFAIK, there are no new failures after landed. It should be safe to let this issue stay closed.
unfortunately I hit the bug with hairpinmode sets to promiscuous-bridge kubelet v1.3.0, linux kernel 4.6.3 cbr0 has already enter the promiscuous mode, all veth hairpinmode = 1 start with parameter
can you ? what distro are you running?
also are you using a network-plugin, overlay (like flannel), which cloudprovider?
Are the SELinux warnings normal ?
Maybe a little out of topic. Are you running K8s on GCE? Do you have the node problem detector running in your cluster?
all veths hairpin_mode = 1 my distro is coreos 1068.6.0 I don't use any network-plugins, and no cloudprovider support here, controller-manager start with kubelet setup after address allocated by controller-manager, all pods connected each other with static routes which controlled by ospfd running inside docker. I can confirm the selinux warnings not related to the bug.
not running on GCE. no detector running inside.
Alright, just want to verify whether the NPD works as expected.
here is the logs of kubelet startup.
That's the problem, the reason we ran into this to begin with is because of hairpin mode. Apparently it is still getting set, or isn't getting erased in your environment. I don't think we erase existing values.
that's wired.

I checked all my nodes, each veth hairpin_mode is 1. it seems it was set to 1 regardless of --configure-cbr0.
there is no existing veth before kubelet create bridge , docker could not start without cbr0 which created by kubelet.
hairpin_mode is always enabled if the device is a bridge interface,
Trust me it's not on my (and our other e2e test) clusters, or we'd know because tests would fail, and teh default setup is docker running on cbr0 and using veths. In 1.3 we started using kubenet network plugin by default, but hairpin flag wasn't set since 1.2. I can trace the code path down when I have some time, but I suspect it's something to do with the way coreos is setup.
I trust you, I promise. I did some tests. None of results told me the flag is set by kubelet. I could not figure out what was happend to these veths. finally, I duplicate to , the only difference is hairepinEnable default to "0", and invoke the hairpinunset.SetUpContainerPid when false. It works! But, if I create veth by hand, the default hairpin_mode is 0 , oh my god. It will not happen in my test environment, only in production env
will enable hairpin_mode That's my fault, I should not set it to false because kubernetes do not use docker's network model.


Question: bestHint = extendedHint } <ins> // the best hint might violate the NUMA allocation rule on which // NUMA node cannot have both single and cross NUMA node allocations // https://kubernetes.io/blog/2021/08/11/kubernetes-1-22-feature-memory-manager-moves-to-beta/#single-vs-cross-numa-node-allocation if isAffinityViolatingNUMAAllocations(machineState, bestHint.NUMANodeAffinity) { return fmt.Errorf("[memorymanager] preferred hint violates NUMA node allocation") } </ins> var containerBlocks []state.Block maskBits := bestHint.NUMANodeAffinity.GetBits() for resourceName, requestedSize := range requestedResources {

Dual socket server with 96 threads total (2242), ~192G of RAM, cpu & memory manager static policy, topologyManagerPolicy best-effort, 10Gi of RAM reserved on NUMA node 0, 1 core (2 threads) reserved on NUMA node 0 kubeadm config: If I try to allocate 2 Guaranteed pod with 85Gi of RAM each, one pod is admitted, and the second one fails with even if it would fit using memory of both NUMA nodes. As I'm using deployments a new pod is recreated right away and you end up with tons of failed pod in state. Error in the logs is: Either the pod becomes pending, or memory from both NUMA nodes is used Have some memory reserved on NUMA 0, launch 2 identical pods with memory limits really close to the max so 1 pod fit on NUMA 1 but the second doesn't fit on NUMA 0 (the sum fits on the server) Here the If I launch 1 pod with 46 CPUs / 170Gi RAM, I get CPUs from 1 NUMA nodes, and memory from both NUMA nodes <details</detailsIf I launch a pod with 46 CPUs / 85Gi RAM, it's properly placed on NUMA 1 <details</detailsIf I try to launch 2 pods with 46 CPUs / 85Gi RAM, one fails with If I try to launch 2 pods with 46 CPUs / 80Gi RAM, everything works 26.8 NONE / bare-metal Alma 8.8 base + rpm-ostree kubeadm containerd 1.7.5 No response
/sig node
/cc Hi! Just to clarify, the memory manager can't keep the pod pending, this is a scheduler feature and builtin kube scheduler (= any of its plugin) is not NUMA-aware atm. So in this case memory manager should extend the pod to both NUMA zones, and failing to do so is likely the issue
Hello Is it possible that you are hitting the exact scenario describe at the KEP: ? Please check the first two steps.
I was going to say I'm hitting the expected unexpected error, but I'm not sure the KEP doc 100% match the current memoryManager behavior (at least when using ).

test 1: the "reject pod2 from the KEP" actually works: start pod 60 cpus / 120Gi RAM wait start pod 20 cpus / 20Gi RAM <details</detailstest 2 (start a 1 NUMA node container then one than spans 2 NUMA node) also works: start pod 20 cpus / 20Gi RAM wait start pod 60 cpus / 120Gi RAM <details</detailsWhat doesn't work is when you get too close to the limits it seems
Interesting those pods are guaranteed I presume? And what about the total node memory capacity? When you check the node status do you have enough memory left to fulfill the second pod request?
click on details you will have the content of yes they are guaranteed else they would not show up in memorymanagerstate it's less then the total node capacity else pod would be pending
/assign
assigned to me for bookkeeping reason but is doing the real work here
Hello so far I didn't manage to reproduce the issue you're describing. In my case I manage to ran both pods, or end up with the second one . However, I managed to find something similar (different affinity error) but with topology manager policy and not as you described which I'm going to open a separate issue for. Could you please share the journal logs while you're reproducing the issue again? Basically on the node where the pod is about to land, share the output
One more thing please share output of the node before and after you reproduce the issue
/triage accepted /priority important-longterm
off until Tuesday, I'll try to reproduce with 1.28.2 then and share the logs next week
NP. I won't be here next week either, so no rush
Hello did you manage to reproduce the issue?
yes, reproduced on my dev machine in 1.28.2, tell me what else you need / if you want me to rebuild kubelet with some debug that's also possible <details</details
Hi I did mange to reproduce the issue, but following the code it seems exactly like I mentioned in my previous I'll try to elaborate: When the first pod gets admitted it placed on NUMA 0 (in my case).

When the second pod gets admitted, there is not enough memory on each NUMA individually (NUMA 0 or NUMA1) to satisfy the memory request, so memory manager tries to span the memory allocation through NUMA 0 and NUMA 1. But it failed short in this line: because NUMA node 0 is now used for single-NUMA memory allocation which align with the KEP . In such case, the pod cannot move to because is done at the scheduler level, while from the scheduler's POV there is enough memory on the node to satisfy the request. Now, about your comment here: wait start pod 20 cpus / 20Gi RAM Make sense because both NUMA 0 and NUMA 1 are already serves mulit-NUMA allocation, since the first pod spans across the two NUMAs, the second pod can squeezed into one of them with no problems. wait start pod 60 cpus / 120Gi RAM This case is more interesting and I didn't manage to reproduce it - I always got which is align with the KEP. Could you please increase Kubelet logs verbosity, reproduce test 2 and attached the following information: logs file before 1 manifest file after pod1 2 manifest file after pod2 Thanks
Hi this is a different server with a bit less CPUs, but test 2 still works for me for verbosity I've used v=99 instead of v=2 Here part of my KubeletConfiguration The logs: And the commands: <details</details
I think I pinpointed a bit more the issue For my test2 pod the cpu manager generated a hint with both NUMA nodes. For test2 pod, if CPU request span both NUMA nodes, then the memory allocation will work whether it requires to use 1 or 2 NUMA nodes If CPU request only requires 1 NUMA node, then it'll fail if memory allocation need to extend to 2 NUMA node.
Hello , After carefully going through the code and the KEP I think that the behavior you're expecting: "Either the pod becomes pending, or memory from both NUMA nodes is used" is not aligned with the architecture.

Let's break it down: "The pod becomes pending" - From the default scheduler POV, the node have enough resources (because the scheduler is not NUMA aware) so the fact that the pod scheduled on this node is align with the expected behavior. Sidenote: if you want to enable NUMA aware scheduling you can take a look at "memory from both NUMA nodes is used" - Because NUMA 0 is already serves as single NUMA memory allocation, it cannot serves as multi NUMA memory allocation as well, otherwise the memory pinning is not guaranteed since linux kernel doesn't support multi-NUMA allocation (pinning). What it can do is grouping NUMA 0 with NUMA 1 into a single group but again this is against the original design: Please check and look for "No spare capacity, reject Pod 7" (last example) And you were right, when the hint generated by cpu manager, spans for both NUMAs the pod gets admitted, and that's exactly where actual bug is - the pod should never get admitted. IOW you were suppose get the error: So the fix here should be preventing the pod admission even when the CPU manager hint spans across both NUMAs. Now, in your case, if you still want and need both pods to get admitted you can change the from to because as I mentioned, the memory pinning in your scenario is not guaranteed anyway.
Hello I still think the bug is that memoryManager doesn't respect topologyManagerPolicy: best-effort. For now I'm exploring using a simple NRI plugin to set the to the nodes of
, The best-effort means that even if the affinity is not preferred, the pod would get admit anyway. But here we're talking about respecting the NUMA node grouping done by memory manager. This is an architectural design we should refer to the original creator memory manager or at least to topology maintainers folks. do you know who can answer on such questions?
some additional info about the limitation i mentioned above


Question: } } } <ins> // If ExternalTrafficPolicy is not set in federated service, use the ExternalTrafficPolicy // defaulted to in federated cluster. if desiredService.Spec.ExternalTrafficPolicy == v1.ServiceExternalTrafficPolicyType("") { desiredService.Spec.ExternalTrafficPolicy = clusterService.Spec.ExternalTrafficPolicy } </ins> // Update existing service, if needed. if !Equivalent(desiredService, clusterService) {

Test pull-kubernetes-federation-e2e-gce for PR is failing. pull-kubernetes-federation-e2e-gce https://k8s- pull-kubernetes-federation-e2e-gce test this
/sig federation /assign
cc
Details on the types of failures and frequency: In the last two days in particular, the following tests have been timing out: [] Federated Services [Feature:Federation] with clusters Federated Service should create and update matching services in underlying clusters [] Federated Services [Feature:Federation] with clusters DNS should be able to discover a federated service [] Federated Services [Feature:Federation] with clusters Federated Service should recreate service shard in underlying clusters when service shard is deleted cc:
Thanks. And yes we are aware of it. is working on a fix - PR .
Hi I'm trying to find out how to follow up on this. Will there be some sort of notification for the PRs affected this?
I just re-queue that test on your PR. note that the test isn't blocking, so your PR can be reviewed/merged without this test passing.
good to know. Thanks!
you can't re-open an issue unless you authored it or you are assigned to it. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. I understand the commands that are listed . </details
/reopen
Reopening to have an issue to track this. This was thought to be resolved, and was made blocking in There's still something up, and it was reverted in I believe sig-federation is going to look at this more next week in order to figure out a more permanent solution to this.
Closing this in favor of issue which has linked runs which more accurately describe the problem.


Question: framework.KubeDescribe("Kubectl logs", func() { var nsFlag string <del> var rc string containerName := "redis-master" </del> <ins> podName := "logs-generator" containerName := "logs-generator" </ins> ginkgo.BeforeEach(func() { <del> ginkgo.By("creating an rc") rc = commonutils.SubstituteImageName(string(readTestFileOrDie(redisControllerFilename))) </del> <ins> ginkgo.By("creating an pod") </ins> nsFlag = fmt.Sprintf("--namespace=%v", ns) <del> framework.RunKubectlOrDieInput(rc, "create", "-f", "-", nsFlag) </del> <ins> // Agnhost image generates logs for a total of 100 lines over 20s. framework.RunKubectlOrDie("run", podName, "--generator=run-pod/v1", "--image="+agnhostImage, nsFlag, "--", "logs-generator", "--log-lines-total", "100", "--run-duration", "20s") </ins> }) ginkgo.AfterEach(func() { <del> cleanupKubectlInputs(rc, ns, simplePodSelector) </del> <ins> framework.RunKubectlOrDie("delete", "pod", podName, nsFlag) </ins> }) /* Release : v1.9 Testname: Kubectl, logs Description: When a Pod is running then it MUST generate logs. <del> Starting a Pod should have a log line indicating the server is running and ready to accept connections. Also log command options MUST work as expected and described below. </del> <ins> Starting a Pod should have a expected log line. Also log command options MUST work as expected and described below. </ins> ‘kubectl log -tail=1’ should generate a output of one line, the last line in the log. ‘kubectl --limit-bytes=1’ should generate a single byte output. ‘kubectl --tail=1 --timestamp should generate one line with timestamp in RFC3339 format

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: Which test(s) are failing: Kubectl client [] Kubectl logs should be able to retrieve and filter logs [Conformance] Since when has it been failing: Testgrid link: Reason for failure: . . switched to using the docker official redis image and the test string was changed accordingly. On Windows, we still use the old image with the old string, and this caused the test to fail. /cc
Hmm, there doesn't seem to be a redis port for windows 3.2.100 : I guess we can modify the test to find "ready to accept connections" rather that "Ready ... ".
/cc


Question: fs.StringVar(&s.Kubeconfig, "kubeconfig", s.Kubeconfig, "Path to kubeconfig file with authorization information (the master location is set by the master flag).") fs.Var(componentconfig.PortRangeVar{Val: &s.PortRange}, "proxy-port-range", "Range of host ports (beginPort-endPort, inclusive) that may be consumed in order to proxy service traffic. If unspecified (0-0) then ports will be randomly chosen.") fs.StringVar(&s.HostnameOverride, "hostname-override", s.HostnameOverride, "If non-empty, will use this string as identification instead of the actual hostname.") <del> fs.Var(&s.Mode, "proxy-mode", "Which proxy mode to use: 'userspace' (older) or 'iptables' (faster). If blank, look at the Node object on the Kubernetes API and respect the '"+ExperimentalProxyModeAnnotation+"' annotation if provided.  Otherwise use the best-available proxy (currently iptables).  If the iptables proxy is selected, regardless of how, but the system's kernel or iptables versions are insufficient, this always falls back to the userspace proxy.") </del> <ins> fs.Var(&s.Mode, "proxy-mode", "Which proxy mode to use: 'userspace' (older) or 'iptables' (faster). If blank, use the best-available proxy (currently iptables).  If the iptables proxy is selected, regardless of how, but the system's kernel or iptables versions are insufficient, this always falls back to the userspace proxy.") </ins> fs.Int32Var(s.IPTablesMasqueradeBit, "iptables-masquerade-bit", util.Int32PtrDerefOr(s.IPTablesMasqueradeBit, 14), "If using the pure iptables proxy, the bit of the fwmark space to mark packets requiring SNAT with.  Must be within the range [0, 31].") fs.DurationVar(&s.IPTablesSyncPeriod.Duration, "iptables-sync-period", s.IPTablesSyncPeriod.Duration, "The maximum interval of how often iptables rules are refreshed (e.g. '5s', '1m', '2h22m').  Must be greater than 0.") fs.DurationVar(&s.IPTablesMinSyncPeriod.Duration, "iptables-min-sync-period", s.IPTablesMinSyncPeriod.Duration, "The minimum interval of how often the iptables rules can be refreshed as endpoints and services change (e.g. '5s', '1m', '2h22m').")

This annotation dates back almost a year. This is a relatively easy project for someone to contribute.
I'll work on getting a pull request in tomorrow for this. Ive been looking at contributing and think this would be a great place to start
nevermind. Looks like a pull request already exists. Overlooked the mention due to being on mobile. Cheers
We'll file more :)


Question: }) ginkgo.By("Giving the Kubelet time to start up and produce metrics") <del> gomega.Eventually(ctx, getKubeletMetrics, 1*time.Minute, 15*time.Second).Should(matchResourceMetrics) </del> <ins> gomega.Eventually(ctx, getKubeletMetrics, 2*time.Minute, 10*time.Second).Should(matchResourceMetrics) </ins> ginkgo.By("Ensuring the metrics match the expectations a few more times") <del> gomega.Consistently(ctx, getKubeletMetrics, 1*time.Minute, 15*time.Second).Should(matchResourceMetrics) </del> <ins> gomega.Consistently(ctx, getKubeletMetrics, 2*time.Minute, 10*time.Second).Should(matchResourceMetrics) </ins> }) ginkgo.It("should report admission failures when the topology manager alignment is known to fail", func(ctx context.Context) {

/kind failing-test <!-- If this is a flake, please add: /kind flake --/sig node started to fail since 09-13.
/cc
/assign
/cc
/triage accepted /priority important-soon it's indeed failing too often
/reopen The test is still failing: Now timing out at 120s i.e. 2 minutes as we increased the default timeout . We need to understand the root cause and get to the bottom of this.
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Snippet from the build log () corresponding from this failure We see from the above logs that: node has two numa nodes, 8 cpus per NUMA so the root cause of the failure is not because the tests are running on a node that doesn't have multi-numa support. is started and the health check fails a couple of times but eventually succeeds. pod is created with 9 cpus (cpusperNUMA +1 i.e. 8+1) which is expected to fail with Topology Affinity Error as we cannot allocate resources for such a pod from a single NUMAnode. We fail to get matching metrics. It is not clear if the check fails because we don't get kubelet metrics at all or the expected metrics but based on point 6 below it seems like it is because of latter. We can add additional logs for more clarity here. each is executed and Kubelet is restarted and health check is done. are collected from the namespace corresponding to the test pod which shows that the pod started even though it was expected to fail with Topology Affinity Error. We need to understand why the pod got successfully created even though it requested CPUs that couldn't be allocated from a single NUMAnode.


Question: return } <del> metrics.BindingLatency.Observe(metrics.SinceInSeconds(start)) </del> <ins> metrics.DeprecatedBindingLatency.Observe(metrics.SinceInSeconds(start)) </ins> prof.Recorder.Eventf(assumed, nil, v1.EventTypeNormal, "Scheduled", "Binding", "Successfully assigned %v/%v to %v", assumed.Namespace, assumed.Name, targetNode) }

Mark the following metrics as deprecated and . Both of those metrics are now covered as part of , the former as a the latter and a plugin. This can be done by setting in each of the two metrics opts. Changing the name of the metric variable to be prefixed with "Deprecated" is useful as well. /help /sig scheduling
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : Mark the following metrics as deprecated and . Both of those metrics are now covered as part of , the former as a the latter and a plugin. /sig scheduling Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/kind deprecation


Question: }, }, { <del> description: "pod specified tolerations for both `notReady:NoExecute` and `unreachable:NoExecute`, expect no change", </del> <ins> description: "pod specified tolerations for both `not-ready:NoExecute` and `unreachable:NoExecute`, expect no change", </ins> requestedPod: api.Pod{ Spec: api.PodSpec{ Tolerations: []api.Toleration{

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: When discussing issue in , I noticed: although is using key , it has actually been changed to in for quite a while. I am thinking we may need to fix this inconsistency in 1.9, by moving out of alpha. Anything else we need to know?: Task tracker: [x] update [x] update kubernetes doc
/sig scheduling


Question: rc.clientSet.AutoscalingV2().HorizontalPodAutoscalers(rc.nsName).Delete(context.TODO(), autoscalerName, metav1.DeleteOptions{}) } <del> func CreateCPUHorizontalPodAutoscalerWithBehavior(rc *ResourceConsumer, cpu int32, minReplicas int32, maxRepl int32, behavior *autoscalingv2.HorizontalPodAutoscalerBehavior) *autoscalingv2.HorizontalPodAutoscaler { </del> <ins> func CreateCPUHorizontalPodAutoscalerWithBehavior(rc *ResourceConsumer, cpu, minReplicas, maxRepl, downscaleStabilizationSeconds int32) *autoscalingv2.HorizontalPodAutoscaler { </ins> hpa := &autoscalingv2.HorizontalPodAutoscaler{ ObjectMeta: metav1.ObjectMeta{ Name:      rc.name,

gce-cos-autoscaling-hpa-cpu Kubernetes e2e suite.[sig-autoscaling] [Feature:HPA] [Serial] [Slow] Horizontal pod autoscaling (non-default behavior) with long upscale stabilization window should scale up only after the stabilization period Since it was in (first run at 2022-06-14 13:13 CEST failed, all subsequent runs failed too). https://k8s- Times out when waiting for scale down (), logs from (): I'm not sure why exactly, I have 2 ideas: It looks like the consumer briefly consumes 440 mCPU, which could result in a higher recommendation being generated HPA stabilises initial size of the consumer Increasing timeout should fix the issue No response_ /sig autoscaling
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
FYI
This was already reverted (and later the test got fixed and merged). /close
Closing this issue. <detailsIn response to : This was already reverted (and later the test got fixed and merged). Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: assert.Equal(imageFsInfo.Inodes, stats.Inodes) assert.Equal(*imageFsInfo.Inodes-*imageFsInfo.InodesFree, *stats.InodesUsed) } <ins> func TestCadvisorListPodStatsWhenContainerLogFound(t *testing.T) { const ( namespace0 = "test0" ) const ( seedRoot           = 0 seedRuntime        = 100 seedKubelet        = 200 seedMisc           = 300 seedPod0Infra      = 1000 seedPod0Container0 = 0 seedPod0Container1 = 0 ) const ( pName0 = "pod0" ) const ( cName00 = "c0" cName01 = "c1" ) const ( rootfsCapacity    = uint64(10000000) rootfsAvailable   = uint64(5000000) rootfsInodesFree  = uint64(1000) rootfsInodes      = uint64(2000) imagefsCapacity   = uint64(20000000) imagefsAvailable  = uint64(8000000) imagefsInodesFree = uint64(2000) imagefsInodes     = uint64(4000) ) prf0 := statsapi.PodReference{Name: pName0, Namespace: namespace0, UID: "UID" + pName0} infos := map[string]cadvisorapiv2.ContainerInfo{ "/":              getTestContainerInfo(seedRoot, "", "", ""), "/docker-daemon": getTestContainerInfo(seedRuntime, "", "", ""), "/kubelet":       getTestContainerInfo(seedKubelet, "", "", ""), "/system":        getTestContainerInfo(seedMisc, "", "", ""), // Pod0 - Namespace0 "/pod0-i":  getTestContainerInfo(seedPod0Infra, pName0, namespace0, leaky.PodInfraContainerName), "/pod0-c0": getTestContainerInfo(seedPod0Container0, pName0, namespace0, cName00), "/pod0-c1": getTestContainerInfo(seedPod0Container1, pName0, namespace0, cName01), } containerLogStats0 := makeFakeLogStats(0) containerLogStats1 := makeFakeLogStats(0) fakeStats := map[string]*volume.Metrics{ kuberuntime.BuildContainerLogsDirectory(prf0.Namespace, prf0.Name, types.UID(prf0.UID), cName00): containerLogStats0, kuberuntime.BuildContainerLogsDirectory(prf0.Namespace, prf0.Name, types.UID(prf0.UID), cName01): containerLogStats1, } fakeStatsSlice := []*volume.Metrics{containerLogStats0, containerLogStats1} fakeOS := &kubecontainertest.FakeOS{} freeRootfsInodes := rootfsInodesFree totalRootfsInodes := rootfsInodes rootfs := cadvisorapiv2.FsInfo{ Capacity:   rootfsCapacity, Available:  rootfsAvailable, InodesFree: &freeRootfsInodes, Inodes:     &totalRootfsInodes, } freeImagefsInodes := imagefsInodesFree totalImagefsInodes := imagefsInodes imagefs := cadvisorapiv2.FsInfo{ Capacity:   imagefsCapacity, Available:  imagefsAvailable, InodesFree: &freeImagefsInodes, Inodes:     &totalImagefsInodes, } options := cadvisorapiv2.RequestOptions{ IdType:    cadvisorapiv2.TypeName, Count:     2, Recursive: true, } mockCtrl := gomock.NewController(t) defer mockCtrl.Finish() mockCadvisor := cadvisortest.NewMockInterface(mockCtrl) mockCadvisor.EXPECT().ContainerInfoV2("/", options).Return(infos, nil) mockCadvisor.EXPECT().RootFsInfo().Return(rootfs, nil) mockCadvisor.EXPECT().ImagesFsInfo().Return(imagefs, nil) mockRuntime := containertest.NewMockRuntime(mockCtrl) mockRuntime.EXPECT().ImageStats().Return(&kubecontainer.ImageStats{TotalStorageBytes: 123}, nil).AnyTimes() volumeStats := serverstats.PodVolumeStats{} p0Time := metav1.Now() mockStatus := statustest.NewMockPodStatusProvider(mockCtrl) mockStatus.EXPECT().GetPodStatus(types.UID("UID"+pName0)).Return(v1.PodStatus{StartTime: &p0Time}, true) resourceAnalyzer := &fakeResourceAnalyzer{podVolumeStats: volumeStats} p := NewCadvisorStatsProvider(mockCadvisor, resourceAnalyzer, nil, nil, mockRuntime, mockStatus, NewFakeHostStatsProviderWithData(fakeStats, fakeOS)) pods, err := p.ListPodStats() assert.NoError(t, err) assert.Equal(t, 1, len(pods)) // Validate Pod0 Results checkEphemeralStats(t, "Pod0", []int{seedPod0Container0, seedPod0Container1}, nil, fakeStatsSlice, pods[0].EphemeralStorage) } </ins>

Rotated container logs space usage doesnt seem to be counted towards the epehemral storage limit. Always only current log file stats are taken into account. rotated log files usage also should be counted towards ephemeral-storage usage No response <detailsTested on Kubernetes Version : 1.21.7, 1.22 Container Runtime : Cri-o </details<details</details<details</details<details</details<details</details<details</details
/sig node
/cc
/assign we can see if there is a issue
Were the rotated log files in /var/log/pods generated by yourself(not by k8s)? The cadvisor can only collect log files on container's filesystem.
No, I have not created any files. they are generated by k8s itself.
Okay, I see. As the cri-o runtime uses cadvisor stats provider to get container stats, it caculates container logUsage by formular . but the default root dir or log dir of cri-o is not /var/log/pods/{podName}/, so theses log files are not on container's filesystem, then cadvisor may not caculate the data accurately. /cc the containerd uses cri stats provider and gets logstats by HostStatsProvider, it can get right data.
In the kubelet code, , the log path is in the format , but the directory of your environment is , I did not see , Did you set this path yourself?
sorry I truncated the name for better readability, the actual path name is . So I suppose its taking in the above format only.
log files gets generated in the path /var/log/pods/defaultephem15334a37d-df02-435c-b385-/con2 for cri-o in my setup
/area cadvisor /triage accepted /priority important-longterm
/retitle Rotated container log files size not counted towards ephemeral-storage's limit on CRI-O
hi, In the past two days, I have studied the collection of file information by cAdvisor. docker and cri-o runtime basically have the same mechanism in cAdvisor. I have verified that when the runtime is cri-o, the log information statistics are also correct. So I need you to provide more detailed information. I found that the command you show is: , please help to confirm whether the "logs" information belongs to the container con2.

or use the command to try
I have verified and info provided is for con2 only . I have the sample yaml used for testing , please let me know if I'm doing it right or anything missing. I have set the kubelet log configuration as containerLogMaxSize: "5Ki", containerLogMaxFiles: 5 After applying the above yaml stats summary shows con1 logs usedbytes always 0 , con2 logs usedbytes as 4096
attached the stats summary of the command And I'm testing on kubernetes 1.22 release
/assign
what version of CRI-O/kubernetes are you using? using latest main of both kubernetes and cri-o with your reproducer causes the container to be evicted because of it exceeding ephemeral storage
oh I see now: my mistake
another mistake: I was using CRI stats provider, instead of cAdvisor stats provider (as is default)
Is CRI stats provider used by default in 1.23 by default? And also problem is seen only if containerLogMaxSize is less that the ephemeral storage limit set , if its more than the ephemeral storage limit , its the right behavior , pod gets evicted in my setup.
at long last, I believe I've found the issue!


Question: } quotaLock.Lock() defer quotaLock.Unlock() <del> // Current policy is to set individual quotas on each volumes. </del> <ins> // Current policy is to set individual quotas on each volume, // for each new volume we generate a random UUID and we use that as // the internal pod uid. // From fsquota point of view each volume is attached to a // single unique pod. </ins> // If we decide later that we want to assign one quota for all <del> // volumes in a pod, we can simply remove this line of code. </del> <ins> // volumes in a pod, we can simply use poduid parameter directly </ins> // If and when we decide permanently that we're going to adopt // one quota per volume, we can rip all of the pod code out. <del> volumeuid := types.UID(uuid.NewUUID()) if pod, ok := dirPodMap[path]; ok && pod != volumeuid { return fmt.Errorf("requesting quota on existing directory %s but different pod %s %s", path, pod, volumeuid) </del> <ins> externalPodUid := poduid internalPodUid, ok := dirPodMap[path] if ok { if podUidMap[internalPodUid] != externalPodUid { return fmt.Errorf("requesting quota on existing directory %s but different pod %s %s", path, podUidMap[internalPodUid], externalPodUid) } } else { internalPodUid = types.UID(uuid.NewUUID()) </ins> } <del> oid, ok := podQuotaMap[volumeuid] </del> <ins> oid, ok := podQuotaMap[internalPodUid] </ins> if ok { if quotaSizeMap[oid] != ibytes { return fmt.Errorf("requesting quota of different size: old %v new %v", quotaSizeMap[oid], bytes) } <ins> if _, ok := dirPodMap[path]; ok { return nil } </ins> } else { oid = common.BadQuotaID }

I have kubernetes 1.24.7. I tried to use on formated volume. It works until first reboot. After reboot, I get messages: I expect that project quota can be assigned even after reboot. Use xfs for /var/lib/containerd with discard,quota,prjquota No response <details</details<detailsbaremetal </details<details</details<details</details<detailsrke2 </details<details</details
/sig node Can this be reproduced? Can this pod be recreated correctly if you force delete the pod?
/triage accepted
It can be reproduced. Pod is created if I remove . I can test force delete.
/assign we are looking into gate to understand if we need to keep it or deprecate it.
If I force delete pod, it does not start again:
/assign Let me check on the latest code with the fix


Question: // generator pick up manually written conversion funcs from external packages. ExtraPeerDirs []string <ins> // Additional dirs to parse and load, but not consider for peers.  This is // useful when packages depend on other packages and want to call // conversions across them. ExtraDirs []string </ins> // SkipUnsafe indicates whether to generate unsafe conversions to improve the efficiency // of these operations. The unsafe operation is a direct pointer assignment via unsafe // (within the allowed uses of unsafe) and is equivalent to a proposed Golang change to

In the codegenerator we have: So basically we process only packages that are explicitly requested by the user and ignore ones that are imported. So in order to have the conversion func imported by the generator, the package where it lives has to be explicitly imported. So we should basically always pass all input packages to conversion generator. This is problematic, when regenerating conversions, we simply pass to input directories only those that were changed in the meantime: We should fix that - probably by passing all CONVERSIONDIRS as inputs to conversion-gen /kind-bug
I think this caused Not sure if was related yet.
It's the other way around - this is the main bug and root cause and the other that you linked is a consequence.
Also - FYI
I actually already started looking into it - so let me assign this.
is out for review
/triage accepted


Question: "type": "integer" } }, <del> "required": [ "names" ], </del> "type": "object" }, "io.k8s.api.core.v1.ContainerPort": {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --is defined as For , both and are valid and they will be serialized to and in Json respectively. However in openapi-, is marked as required so is not allowed. This cause a mismatch when doing deserialization based on openapi spec. should not be marked as required because is a valid value. Kubernetes version (use ): v1.19 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig node api-machinery
/triage accepted /assign
Hello, is there any update for this issue?
Possibly related to and This bug can prevent a controller we're working on from enumerating nodes in a cluster, as we hit deserialization errors if any node in the cluster exposes an image with . It seems like proposed to address this by using the image ID instead of its name. I imagine that making this field optional would require a KEP, as it's an API change; but perhaps this could be fixed by ensuring that the field is always initialized (i.e. to an empty list) to prevent deserialization errors? This would not require an API change and would make the implementation conform with the API spec.
Looking at the sources, it looks like the image list is constructed here: It's not clear to me how it's even possible for to be uninitialized? I'll try to figure out how we got into this state...
I was just able to trigger this with though I imagine this can be replicated in other environments (at least in ) with a similar workflow: To get into this state, I first built my app with and then imported it into the local k3d cluster with . Then, I made some changes to my application and built it again with the same tag, and imported into the cluster with the same command. It's not clear whether the instance represents the first version of the application image and the named instance refers to the second build on the same tag, or vice versa.
Sorry for the late reply. That's interesting. I agree with you it seems impossible for to be nil.

<- doesn't seem to be the case. Looking
I can reproduce it by changing to . Either I missed something basic or there is a bug in the (kubelet uses v1 types directly) serialization pipeline.
I think I understand what happened: If an image has neither nor , kubelet sets to . However, when the object gets persisted to etcd in protobuf, the data is stored as -- protobuff doesn't distinguish and : Although we ask JSON API clients to tolerate values for such fields, openapi-generated clients (e.g. the ) treat as an invalid value for a required field during deserialization. It's hard to tell what the right fix is: it's a data lost in kube-apiserver serialization/deserialization IMO, but it may be hard to fix, since was the rollback of an attempt to fix the field as optional is an option, but it needs to be approved by someone from SIG Node (and for other APIs that may have similar issue, it would depend on the implementation details of each API). On the plus side, there seems to be no server-side validation that requires to be not . kubelet is the only writer of the field, and the fact is sometimes the field is written as the JSON API client tolerate value (as suggested by ) is an option, but it has to be done in a client-by-client basis. For , I think we can maintain a patch (namely treating as ) in the Python client to fix this. I'm not sure what client you're using though Please note that this could also be fixed if the node API takes a different approach when an image has neither nor , e.g. using the image ID as the name (could be a breaking change to existing clients that tolerates ) assigning a dummy name like <none(which could be confusing if there are multiple such images / an image that is literally called <none)
I'm not familiar enough with the internals of k8s' etcd protobuf serialization to know whether this is feasible in this context, but in other applications I've used wrapper messages to avoid this ambiguity. For example: This lets the value-ness of the names field be encoded distinctly from the list itself.

Another instance of this issue discussed here
Kubernetes python client fix is tracked in contains some API change suggestions for SIG node to consider, which could benefit JSON API clients: /unassign /remove-sig api-machinery /remove-triage accepted let's see if someone from SIG node can pick this up
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
thanks! But isn't that a breaking change if an old client talks to a new server? ah awesome! Probably we should just send a PR to SIG node to mark the field as optional /help
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : thanks! But isn't that a breaking change if an old client talks to a new server? Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign actually let me open one real quick
As I said, I'm not really clear on where etcd protobuf clients are implemented or how public that API is. Generally, though, it could be done gracefully via a deprecation cycle. A new field is introduced and the prior field is considered deprecated; Writers write both the deprecated field and the new type field; The API server continues to read the deprecated field for one major version. Once a major version has passed, change the API server to read the new typed field and stop writing the deprecated version... But this is all way more complicated than marking the field as optional. If it's acceptable to make it optional, that sounds fine to me! Thanks for your help looking into this
that makes sense. Our proto definitions and clients are generated from the files, so supporting the deprecation logic in the clients does sound like more work
Thanks for the fix I am using .

it turns out that already has a fix on master ( -- not yet on ) and I've put up a PR to make it usable from kube-rs ()


Question: t.Fatalf("Expecting to get one conflict when a different applier updates existing list item, got: %v", status.Status().Details.Causes) } } <ins> func TestNoOpApplyWithDefaultsSameResourceVersionCRD(t *testing.T) { // https://github.com/kubernetes/kubernetes/issues/124605 server, err := apiservertesting.StartTestServer(t, apiservertesting.NewDefaultTestServerOptions(), nil, framework.SharedEtcd()) if err != nil { t.Fatal(err) } defer server.TearDownFn() config := server.ClientConfig apiExtensionClient, err := clientset.NewForConfig(config) if err != nil { t.Fatal(err) } dynamicClient, err := dynamic.NewForConfig(config) if err != nil { t.Fatal(err) } noxuDefinition := fixtures.NewNoxuV1CustomResourceDefinition(apiextensionsv1.ClusterScoped) err = json.Unmarshal([]byte(`{ "openAPIV3Schema": { "type": "object", "properties": { "spec": { "type": "object", "properties": { "infrastructureRef": { "type": "object", "properties": { "name": { "type": "string" }, "namespace": { "type": "string", "default": "default-namespace" } }, "x-kubernetes-map-type": "atomic" } } } } } }`), &noxuDefinition.Spec.Versions[0].Schema) if err != nil { t.Fatal(err) } noxuDefinition, err = fixtures.CreateNewV1CustomResourceDefinition(noxuDefinition, apiExtensionClient, dynamicClient) if err != nil { t.Fatal(err) } group := noxuDefinition.Spec.Group kind := noxuDefinition.Spec.Names.Kind resource := noxuDefinition.Spec.Names.Plural version := noxuDefinition.Spec.Versions[0].Name apiVersion := noxuDefinition.Spec.Group + "/" + version gvr := schema.GroupVersionResource{Group: group, Version: version, Resource: resource} name := "mytest" // fieldWithDefault will be defaulted applyConfiguration := &unstructured.Unstructured{ Object: map[string]interface{}{ "apiVersion": apiVersion, "kind":       kind, "metadata": map[string]interface{}{ "name": name, }, "spec": map[string]interface{}{"infrastructureRef": map[string]interface{}{"name": "infrastructure-machine-1n"}}, }, } created, err := dynamicClient.Resource(gvr).Apply(context.TODO(), name, applyConfiguration, metav1.ApplyOptions{FieldManager: "apply_test"}) if err != nil { t.Fatalf("failed to create custom resource with apply: %v:n%v", err, created) } createdAccessor, err := meta.Accessor(created) if err != nil { t.Fatalf("Failed to get meta accessor for updated object: %v", err) } // Sleep for one second to make sure that the times of each update operation is different. time.Sleep(1 * time.Second) updated, err := dynamicClient.Resource(gvr).Apply(context.TODO(), name, applyConfiguration, metav1.ApplyOptions{FieldManager: "apply_test"}) if err != nil { t.Fatalf("failed to update custom resource with apply: %v:n%v", err, updated) } updatedAccessor, err := meta.Accessor(updated) if err != nil { t.Fatalf("Failed to get meta accessor for updated object: %v", err) } if createdAccessor.GetResourceVersion() != updatedAccessor.GetResourceVersion() { t.Fatalf("Expected same resource version to be %v but got: %vnold object:n%vnnew object:n%v", createdAccessor.GetResourceVersion(), updatedAccessor.GetResourceVersion(), created, updated, ) } } </ins>

When continuously re-applying the same object via a controller or kubectl with SSA the resourceVersion increases every time. I expected the resourceVersion to stay the same if the client always sends the same object via SSA, especially if the resulting object doesn't change (apart from resourceVersion & managedFields, but these are managed by the apiserver). Apply this CRD: <details</summary</detailsApply this CR: <details</summary</detailsGet state of the CR after first apply Apply the CR again Get state of the CR after second apply Diff =The resourceVersion and the "time" field of the managedField entry of the "test-manager" increases. The use case we have is a lot more complex than this simple scenario. The tl;dr is that we want to implement caching for our SSA calls, so that we don't have SSA calls on every reconcile of our controller if nothing actually changes. This is relatively hard to do if the resourceVersion increases after every apply. I took a closer look at the APIserver and ~ the following happens: "incoming" SSA patch is applied over the live object from etcd this leads to a change because has which means the field will be dropped Because of that change the "time" field in the managedField entry gets set to "time.Now()" later during admission the namespace field gets defaulted again (I also tried this with a mutating webhook, same result) On a high-level: client always applies the same object defaulting always results in the same object resourceVersion and managedFields[].time continuously go up I think there are many similar cases that would lead to this behavior. Basically whenever the admission chain mutates a field previously set via SSA. Notes: This only happens if there is 1s between each apply, otherwise the managedField[].time is identical and then the resourceVersion stays the same (because we then don't write to etcd) <detailsI also tried this with client-go v0.30 and kube-apiserver v1.30. Same result. </details<detailsRunning this natively on Apple Silicon M2. But can't imagine this makes a difference. Also got the same results in Docker Desktop on Mac. </details
(cc might be interesting for you)
/sig api-machinery
It looks less a bug but more of a intended behavior to me.

/cc /triage accepted
Bypassing the writes triggered by noop applies would break storage version migration (), so we're committed to supporting this behavior as a feature.
I am confused. Are you saying this is intended behavior for a NOP SSA apply to increment resourceVersion? This seems to not actually happen in all cases, only for certain objects/edge cases (see ). For example, try this: No resourceVersion is incremented above). Surprisingly I couldn't find any clear statement that SSA doesn't increment the resource version, but does say "a no-op apply is not very much more work for the API server than an extra GET" which seems impossible if resourceVersion is incremented
is probably only updating it because either the fieldManager changes or in the patch body the apiVersion is different (or both, perhaps)?
We need this in storage version migration to make sure we correctly handles the CRD schema changes. See Joe's comments here - If it helps to understand this better, we have tests to validate this.
I may have misspoke. When storage versions changes, a NOP SSA apply will trigger a resourceVersion bump. If the storage version is unchanged and an update would be a NOP for etcd, I believe we short-circuit the write (?). This might explain the configMap case.
For the reported issue: We've seen cases like this before where the time gets bumped causing a NOP to become a write. This could be improved. Note that this isn't technically a bug because the API server is allowed to bump the resourceVersion for any write (the discussion about storage version changes causing this is a good example of where this can happen in a way that the caller wouldn't expect). But I'd like to see the etcd write optimized away for cases like this.
Sounds similar to perhaps which had some investigation. FWIW as a controller author my expectation is that repeatedly applying does not repeatedly trigger resourceVersion increments. It's fine if it happens rarely (storage version change, etc) but if it happens consistently that means our own writes re-trigger reconciliation and we have an infinite loop of writes. AFAIK all the marketing around writing SSA controllers has been that you just apply what you want and don't try to do a get+diff+apply. Even if we did want to do that, it doesn't really work well currently ().

If we aren't able to rely on NOP writes (usually) being NOPs, I don't see how we are supposed to use SSA to write a controller at all?
This is a really good point. I agree.
looks like it was intended fix this sort of problem. I'm looking into why it doesn't work for this case.
doesn't work for CRs since the managed fields are inside of a unstructured object and are not typed as (for the slow path equality check in IgnoreManagedFieldsTimestampsTransformer).
is a draft proposal for a fix. Feedback welcome on how to make this fix cleaner. I'll test carefully before moving it out of draft.


Question: <ins> /* Copyright 2023 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ /* E2E Node test for DRA (Dynamic Resource Allocation) This test covers node-specific aspects of DRA The test can be run locally on Linux this way: make test-e2e-node FOCUS='[NodeFeature:DynamicResourceAllocation]' SKIP='[Flaky]' PARALLELISM=1  TEST_ARGS='--feature-gates="DynamicResourceAllocation=true" --service-feature-gates="DynamicResourceAllocation=true" --runtime-config=api/all=true' */ package e2enode import ( "context" "os" "path" "path/filepath" "time" "github.com/onsi/ginkgo/v2" "github.com/onsi/gomega" v1 "k8s.io/api/core/v1" resourcev1alpha2 "k8s.io/api/resource/v1alpha2" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/client-go/kubernetes" "k8s.io/klog/v2" admissionapi "k8s.io/pod-security-admission/api" "k8s.io/kubernetes/test/e2e/framework" e2epod "k8s.io/kubernetes/test/e2e/framework/pod" "k8s.io/dynamic-resource-allocation/kubeletplugin" testdriver "k8s.io/kubernetes/test/e2e/dra/test-driver/app" ) const ( driverName                = "test-driver.cdi.k8s.io" cdiDir                    = "/var/run/cdi" endpoint                  = "/var/lib/kubelet/plugins/test-driver/dra.sock" pluginRegistrationPath    = "/var/lib/kubelet/plugins_registry" draAddress                = "/var/lib/kubelet/plugins/test-driver/dra.sock" pluginRegistrationTimeout = time.Second * 60 // how long to wait for a node plugin to be registered ) var _ = ginkgo.Describe("[sig-node] DRA [Feature:DynamicResourceAllocation][NodeFeature:DynamicResourceAllocation]", func() { f := framework.NewDefaultFramework("dra-node") f.NamespacePodSecurityEnforceLevel = admissionapi.LevelBaseline var kubeletPlugin *testdriver.ExamplePlugin ginkgo.Context("Resource Kubelet Plugin [Serial]", func() { ginkgo.BeforeEach(func(ctx context.Context) { kubeletPlugin = newKubeletPlugin(getNodeName(ctx, f)) }) ginkgo.It("must register after Kubelet restart", func(ctx context.Context) { oldCalls := kubeletPlugin.GetGRPCCalls() getNewCalls := func() []testdriver.GRPCCall { calls := kubeletPlugin.GetGRPCCalls() return calls[len(oldCalls):] } ginkgo.By("restarting Kubelet") restartKubelet(true) ginkgo.By("wait for Kubelet plugin re-registration") gomega.Eventually(getNewCalls).WithTimeout(pluginRegistrationTimeout).Should(testdriver.BeRegistered) }) ginkgo.It("must register after plugin restart", func(ctx context.Context) { ginkgo.By("restart Kubelet Plugin") kubeletPlugin.Stop() kubeletPlugin = newKubeletPlugin(getNodeName(ctx, f)) ginkgo.By("wait for Kubelet plugin re-registration") gomega.Eventually(kubeletPlugin.GetGRPCCalls).WithTimeout(pluginRegistrationTimeout).Should(testdriver.BeRegistered) }) ginkgo.It("must process pod created when kubelet is not running", func(ctx context.Context) { // Stop Kubelet startKubelet := stopKubelet() pod := createTestObjects(ctx, f.ClientSet, getNodeName(ctx, f), f.Namespace.Name, "draclass", "external-claim", "drapod") // Pod must be in pending state err := e2epod.WaitForPodCondition(ctx, f.ClientSet, f.Namespace.Name, pod.Name, "Pending", framework.PodStartShortTimeout, func(pod *v1.Pod) (bool, error) { return pod.Status.Phase == v1.PodPending, nil }) framework.ExpectNoError(err) // Start Kubelet startKubelet() // Pod should succeed err = e2epod.WaitForPodSuccessInNamespaceTimeout(ctx, f.ClientSet, pod.Name, f.Namespace.Name, framework.PodStartShortTimeout) framework.ExpectNoError(err) }) }) }) // Run Kubelet plugin and wait until it's registered func newKubeletPlugin(nodeName string) *testdriver.ExamplePlugin { ginkgo.By("start Kubelet plugin") logger := klog.LoggerWithValues(klog.LoggerWithName(klog.Background(), "kubelet plugin"), "node", nodeName) // Ensure that directories exist, creating them if necessary. We want // to know early if there is a setup problem that would prevent // creating those directories. err := os.MkdirAll(cdiDir, os.FileMode(0750)) framework.ExpectNoError(err, "create CDI directory") err = os.MkdirAll(filepath.Dir(endpoint), 0750) framework.ExpectNoError(err, "create socket directory") plugin, err := testdriver.StartPlugin( logger, cdiDir, driverName, "", testdriver.FileOperations{}, kubeletplugin.PluginSocketPath(endpoint), kubeletplugin.RegistrarSocketPath(path.Join(pluginRegistrationPath, driverName+"-reg.sock")), kubeletplugin.KubeletPluginSocketPath(draAddress), ) framework.ExpectNoError(err) gomega.Eventually(plugin.GetGRPCCalls).WithTimeout(pluginRegistrationTimeout).Should(testdriver.BeRegistered) ginkgo.DeferCleanup(plugin.Stop) return plugin } // createTestObjects creates objects required by the test // NOTE: as scheduler and controller manager are not running by the Node e2e, // the objects must contain all required data to be processed correctly by the API server // and placed on the node without involving the scheduler and the DRA controller func createTestObjects(ctx context.Context, clientSet kubernetes.Interface, nodename, namespace, className, claimName, podName string) *v1.Pod { // ResourceClass class := &resourcev1alpha2.ResourceClass{ ObjectMeta: metav1.ObjectMeta{ Name: className, }, DriverName: driverName, } _, err := clientSet.ResourceV1alpha2().ResourceClasses().Create(ctx, class, metav1.CreateOptions{}) framework.ExpectNoError(err) ginkgo.DeferCleanup(clientSet.ResourceV1alpha2().ResourceClasses().Delete, className, metav1.DeleteOptions{}) // ResourceClaim podClaimName := "resource-claim" claim := &resourcev1alpha2.ResourceClaim{ ObjectMeta: metav1.ObjectMeta{ Name: claimName, }, Spec: resourcev1alpha2.ResourceClaimSpec{ ResourceClassName: className, }, } createdClaim, err := clientSet.ResourceV1alpha2().ResourceClaims(namespace).Create(ctx, claim, metav1.CreateOptions{}) framework.ExpectNoError(err) ginkgo.DeferCleanup(clientSet.ResourceV1alpha2().ResourceClaims(namespace).Delete, claimName, metav1.DeleteOptions{}) // Pod containerName := "testcontainer" pod := &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name: podName, }, Spec: v1.PodSpec{ NodeName: nodename, // Assign the node as the scheduler is not running ResourceClaims: []v1.PodResourceClaim{ { Name: podClaimName, Source: v1.ClaimSource{ ResourceClaimName: &claimName, }, }, }, Containers: []v1.Container{ { Name:  containerName, Image: e2epod.GetDefaultTestImage(), Resources: v1.ResourceRequirements{ Claims: []v1.ResourceClaim{{Name: podClaimName}}, }, Command: []string{"/bin/sh", "-c", "env | grep DRA_PARAM1=PARAM1_VALUE"}, }, }, RestartPolicy: v1.RestartPolicyNever, }, } createdPod, err := clientSet.CoreV1().Pods(namespace).Create(ctx, pod, metav1.CreateOptions{}) framework.ExpectNoError(err) ginkgo.DeferCleanup(clientSet.CoreV1().Pods(namespace).Delete, podName, metav1.DeleteOptions{}) // Update claim status: set ReservedFor and AllocationResult // NOTE: This is usually done by the DRA controller createdClaim.Status = resourcev1alpha2.ResourceClaimStatus{ DriverName: driverName, ReservedFor: []resourcev1alpha2.ResourceClaimConsumerReference{ {Resource: "pods", Name: podName, UID: createdPod.UID}, }, Allocation: &resourcev1alpha2.AllocationResult{ ResourceHandles: []resourcev1alpha2.ResourceHandle{ { DriverName: driverName, Data:       "{"EnvVars":{"DRA_PARAM1":"PARAM1_VALUE"},"NodeName":""}", }, }, }, } _, err = clientSet.ResourceV1alpha2().ResourceClaims(namespace).UpdateStatus(ctx, createdClaim, metav1.UpdateOptions{}) framework.ExpectNoError(err) return pod } </ins>

DRA e2e tests should cover handling Kubelet restarts and similar functionality related to Node operations.
/sig node
/assign
/triage accepted
/cc


Question: var spec *volume.Spec // Create spec <del> pv := makeTestPV(tc.pvName, 10, testDriver, "testvol") spec = volume.NewSpecFromPersistentVolume(pv, pv.Spec.PersistentVolumeSource.CSI.ReadOnly) </del> <ins> pv := makeTestPV(tc.pvName, 10, testDriver, tc.volumeId) if tc.removeVolumeHandle { pv.Spec.PersistentVolumeSource.CSI.VolumeHandle = "" } if tc.addVolSource { spec = volume.NewSpecFromVolume(makeTestVol(tc.pvName, testDriver)) } else { spec = volume.NewSpecFromPersistentVolume(pv, pv.Spec.PersistentVolumeSource.CSI.ReadOnly) if tc.skipPVCSISource { spec.PersistentVolume.Spec.CSI = nil } } </ins> // Run mountPath, err := csiAttacher.GetDeviceMountPath(spec)

When two CSI PVs point to the same volume in the storage backend, kubelet can't start pods that use these two PVs on the same node. Both PVs have the same VolumeHandle and kubelet is smart enough to recognize that they're the same volume (because they have the same unique volume ID). It calls only once, however, the staging path contains name of one of the PVs, say PV1: One of the CSI volume calls then fails, because it uses PV2 to compute the staging path and that path does not exist. Both pods should just start. Using CSI HostPath volume, PV1 (dynamically provisioned for simplicity): PV2 (basically edited PV1): PVC1, PVC2, Pod1 and Pod2 are trivial, nothing special there. <detailsToday's master, </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage /priority backlog
I suggest that if two volumes have the same unique volume ID (computed from CSI ), they must have also the same global mount path. Kubelet already has code to call NodeStage once and bind-mount it to different pods, regardless how many different PVs they use. These are several issues changing the global mount path: uses PV parsed from the global path to get the PV and to get enough info to call NodeUnstage: However, this parsing is used only when the volume plugin is not able to find/parse its json it has saved during MountDevice: Can we safely assume that the json is always there? the global path in a running cluster is tricky. We can require nodes are drained when kubelet with new global mount path is installed. This way, no CSI volume is mounted to a global path when the new kubelet starts. Minor cluster upgrades (1.23 -1.24) are , also documented in . Or we could fix kubelet to use the old global path when it already exists. This looks fragile to me.

staging dir needs to be unique enough not to conflict with other CSI volumes. It should be based on the driver name + volume handle, but that could be quite long. Can we compute a hash and hope it won't conflict?
I think in-tree driver can handle this case correctly because global mountpath is using unique volume id instead of pv name. With csi-migration, does it break backward compatibility?
We require users to drain nodes before enabling migration or when updating to a Kubernetes version that have migration enabled by default. All volumes should be therefore unmounted.
probably yes this breaks backward compatibility, but since only migration for PD is being done maybe it's more rare? We saw a customer case for this situation with NFS shares using the csi filestore driver. For PD I think it would only come up when trying to ROX a PD volume across namespaces, I'm not sure if that's common.
Capturing some thoughts here in line to understand your we use a staging path like , the kubelet can locate the both the driver name and the volumeId, needed during UnmountDevice call. e.g for pd csi driver which is the sha256{projects/mattcary-saikatroyc-test/zones/us-central1-c/disks/pvc--f97d-43cd-a5f1-}. This means the volume Id needs to persisted (we already do that in today ), so that the volume Id can be retrieved for UnmountDevice call. we decide to make the change at a minor boundary, then a node drain mandate would unstage any volumes before the kubelet is upgraded. this ensures any staged path for csi volume is using the new format 2 If we do not want to enforce a node drain, kubelet can first lookup json data (the reasoning is if this file does not exist, we will not be able to retrieve the volume handle from the new handle anyway), else fallback to try the old format of staging path - . TBD: Need to check mount reference logic. sha(volumeId) is expected to be unique in the cluster, if I understand. (based on the assumption any storage resource provisioned by a given driver is expected to be uniquely identifiable)
/assign
yes, your summary is correct. We persist a json, please make sure it contains anything that NodeUnstage may need.

It seems to me that people prefer option 2.2. We already do to uniquely identify VolumeAttachment. So far nobody complained about any conflict.


Question: }, "generateName": { SchemaProps: spec.SchemaProps{ <del> Description: "GenerateName is an optional prefix, used by the server, to generate a unique name ONLY IF the Name field has not been provided. If this field is used, the name returned to the client will be different than the name passed. This value will also be combined with a unique suffix. The provided value has the same validation rules as the Name field, and may be truncated by the length of the suffix required to make the value unique on the server.nnIf this field is specified and the generated name exists, the server will NOT return a 409 - instead, it will either return 201 Created or 500 with Reason ServerTimeout indicating a unique name could not be found in the time allotted, and the client should retry (optionally after the time indicated in the Retry-After header).nnApplied only if Name is not specified. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#idempotency", </del> <ins> Description: "GenerateName is an optional prefix, used by the server, to generate a unique name ONLY IF the Name field has not been provided. If this field is used, the name returned to the client will be different than the name passed. This value will also be combined with a unique suffix. The provided value has the same validation rules as the Name field, and may be truncated by the length of the suffix required to make the value unique on the server.nnIf this field is specified and the generated name exists, the server will return a 409.nnApplied only if Name is not specified. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#idempotency", </ins> Type:        []string{"string"}, Format:      "", },

The error code of is 409. Is this an expected behavior? The doc seems to say no: NOT return a 409 - instead, it will either return 201 Created or 500 with Reason ServerTimeout indicating a unique name could not be found in the time allotted, and the client should retry (optionally after the time indicated in the Retry-After header). seems to create which error code is 409. The error code of isn't 409. Or, the doc doesn't say "If this field is specified and the generated name exists, the server will NOT return a 409". Create a large number of Pods with generateName, and cause name conflicts. Just question: if the API is correct in returning 409 for a GenerateName-conflict, how do we determine that error? We want to retry when GenerateName conflict returns in most cases because the retry may solve this conflict. latest master (maybe after ) <details</details<details</details<details</details<details</details<details</details
Sorry if not an appropriate SIG. /sig api-machinery
/assign
It seems related to . If it's expected behavior, the document isn't correct..?
is a mistake. /unassign
Hi do you think this is a good first issue to tackle? If so, I would like to try contributing here.
/assign /triage accepted
Forgive me to ping other people who might know. /cc /cc /cc
I suspect we just failed to update the documentation.
Thanks Okay. I'll create the PR for that. /kind documentation /assign


Question: if objectMeta.GetGeneration() > 0 { objectMeta.SetGeneration(objectMeta.GetGeneration() + 1) } return true, false, nil }

Use apiserver extension(APIService) interface, impl its method to gracefully control resource deletion, when the field is nil, will lead to panic!! To first check if is nil, then get its value. Use apiserver extension(APIService) strategy, impl this function below will lead to panic: As the field is a int64 pointer, need to be careful when init the struct. Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig api-machinery
/triage accepted


Question: // we have a spurious APIService that we're managing, delete it (5A,6A) case desired == nil: <del> return c.apiServiceClient.APIServices().Delete(curr.Name, nil) </del> <ins> opts := &metav1.DeleteOptions{Preconditions: metav1.NewUIDPreconditions(string(curr.UID))} err := c.apiServiceClient.APIServices().Delete(curr.Name, opts) if apierrors.IsNotFound(err) || apierrors.IsConflict(err) { // deleted or changed in the meantime, we'll get called again return nil } return err </ins> // if the specs already match, nothing for us to do case reflect.DeepEqual(curr.Spec, desired.Spec):

k8s version: v1.9.3 When I create customresourcedefinition, There is error like Operation cannot be fulfilled on "": the object has been modified; please apply your changes to the latest version and try again . apiVersion: kind: CustomResourceDefinition metadata: name: annotations: "": crd-install labels: app: istio-pilot spec: group: names: kind: VirtualService listKind: VirtualServiceList plural: virtualservices singular: virtualservice categories: istio-io networking-istio-io scope: Namespaced version: v1alpha3 So what is the reason of it?
a few questions: where are you seeing the error, in the apiserver log or in the output of the kubectl command? what kubectl command are you using to create the object? are you running HA apiservers?
The log is shown by apiserver. I run k8s in HA apiservers. The command I used is "kubectl apply -f "+file.
API servers attempt to update CRD status to indicate whether the names were available. In an HA apiserver setup, the API servers race to do so, and all but the first lose the race with a resource conflict error. That is not unexpected. Opened to avoid logging normal API responses in that scenario.
Thanks ! But I want to know whether the CRD has been set successfully in HA apiservers ? I use the command " kubectl get crd -o yaml" and get the following results. ! I see its status is true. But when i create a virtualservices, it returns error. Error from server (Timeout): error when creating "cms-api-": Timeout: request did not complete within allowed duration ! So i want to know how to solve this error.
/assign
Excuse me. Do you have some greate ideas to solve this issue??
There is a PR for this sent by
I see your request is timed-out not failed. In HA, I expect the request to fail in the first seconds of the CRD creation, but after that it should go through. I need more logs during this timeout or a consistent way to reproduce it.