Question: // annotation. requiresUpdate := kl.reconcileCMADAnnotationWithExistingNode(node, existingNode) requiresUpdate = kl.updateDefaultLabels(node, existingNode) || requiresUpdate <ins> requiresUpdate = kl.reconcileExtendedResource(node, existingNode) || requiresUpdate </ins> if requiresUpdate { if _, _, err := nodeutil.PatchNodeStatus(kl.kubeClient.CoreV1(), types.NodeName(kl.nodeName), originalNode, existingNode); err != nil { glog.Errorf("Unable to reconcile node %q with API server: error updating node: %v", kl.nodeName, err)

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: In 1.11, removed externalID from Node.Spec which used to be used by kubelet to detect node recreation in certain environments during node upgrade. This changed node upgrade behavior in those environments because in tryRegisterWithAPIServer(node *v1.Node), without externalID checking, we now consider the recreated node after upgrade as a previously registered node and thus will get old node state from API server instead of regenerating them based on node local state. For extended resource reported by a device plugin, this causes problem because after node upgrade/recreation, the device plugin DaemonSet pod needs to be restarted by Kubelet, finishes certain setup, and then report to kubelet that its resource is available at the node. However, the node status capacity and allocatable from the API server still contains the old state before the node upgrade. After kubelet syncs up with the API server, the node status field gets overwritten and previously reported extended resources will appear on the node status capacity/allocatable even though the node is not ready for pods to consume such resources. As the result, Kubelet may start a pod requesting such resource without proper container runtime setup it needs to learn from device plugin. Note to cope with the externalID removal change, has been updated () to explicitly uncordon the node after checking it is restarted and becomes ready. This however doesn't help device plugin case because node readiness doesn't depend on individual extended resource readiness. This issue is to explore how we can fix this problem on head and 1.11. In particular, I wonder whether we may switch to the model that we consider Kubelet as the only source to update extended resource capacity/allocatable in node status. I.e., don't support manually updating node status capacity/allocatable fields as documented in This would allow us to re-generate node status capacity/allocatable every time in node status update, which I think is much simpler and more robust. As I heard, some folks have been using this mechanism to do simple resource exporting/accounting through a central controller. I wonder whether people are open to switch to the device plugin model in those cases.

Even though those extended resources may not require special container setup, it seems a more secure model to have Kubelet totally own its node status. FYI, here is the related OSS issue: where we first introduced extended resource concept. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Running the gpu upgrade test from with --upgrade-target set to any 1.11+ version would hit the issue. Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig node
/cc
Having extended resources be a concept encapsulating both device plugin resources and user set resources (opaque int resources) has been a hassle to deal with (as it introduced unexpected behavior in the device plugin lifecycle). We should also consider a few alternatives : separating them from a code perspective (i.e: different fields in node status) . separating them from a name (/dns) perspective.
Agree with If we do need to support the use case of exporting a resource in node status capacity/allocatable not through Kubelet, seems better to introduce special naming convention for such resources.
We had a discussion in another issue about using the check point to keep the node in not ready until all expected device plugins have re-registered. Is that option still on the table? I’m hesitant to suggest changing the semantics of extended resources again, at least until we are sure it’s a valid long term solution.
We did implement the mechanism to use check point data to determine previously registered device plugin resource and keep the resource not ready till the expected device plugin re-register. It helps normal node reboot case but not for node upgrade/recreation case, because boot disk is recreated in the later case and the check point data is also gone.
Kubelet's device manager should recover gracefully from a deleted checkpoint. It should be able to identify which resources in the node object are extended resources and it should remove them if the checkpoint disappears. Checkpoints can disappear for any number of reasons and kubelet needs to react appropriately. cc since this is fallout from . Device manager was relying on node object deletion/recreation to function after GCE upgrades.

+1 Node API object deletion/recreation was only for the case where you have a completely different instance (and deletion was only done because the external ID was made immutable). Deletion/recreation in response to an upgrade or machine restart is a form of data loss (admin- taints, labels, etc, are unrecoverable). Additionally, depending on Node object deletion on upgrade as previously seen in GCE is highly environment-dependent and will not be consistent between cloud-providers, environments, etc.
how about during node registration reconciliation, we reconcile extended resources by setting their capacity to zero? That means any external controller that exports extended resources by directly patching node status capacity/allocatable needs to re-do that after any kubelet restart, which I guess is already what they are doing to handle node upgrade. The required change should be small:
what are the implications of doing that on every kubelet process restart? does available resource flutter impact already-running workloads on a node (does the kubelet evaluation of PodFitsResources evict running workloads? does any controller notice workloads running on a node are using resources the kubelet says it doesn't have, and evict them?)
For extended resources exported through device plugin, kubelet device manager will handle their restarts, currently based on checkpoint data and in the future, we will explore whether we may store this information in some form of API. If the checkpoint data is gone, pods will fail kubelet admission handling and the pod's controller should recreate a new pod. For extended resources exported by an external controller through direct node status capacity patching, the controller should watch for kubelet readiness change, and repatch the field after kubelet becomes ready again.
Having an external controller watching the nodes status update to repatch the nodes sounds like a bad "user" experience. Maybe looking at it from a user's perspective and then figuring out what we'll do is a better way to look at this?

The behavior I'd expect to see is the following: For non device plugin extended resources A user patches the node status with the extended resource If the node restarts I don't expect the Allocatable / Capacity fields to change If the node is upgraded I don't expect the Allocatable / Capacity fields to change If the node is recreated I don't expect the Allocatable / Capacity fields to change For device plugin extended resources Kubelet advertises the extended resources when the device plugin registers itself against kubelet If the node restarts I expect the Allocatable field to be 0 until the resource is ready but still advertise its last known number of devices If the node is upgraded I expect the Allocatable field to be 0 until the resource is ready but still advertise its last known number of devices If the node is recreated I expect the Allocatable field to be 0 until the resource is ready but still advertise its last known number of devices As for the case where the checkpoint data is gone, should we expect the pod's controller to recreate a new pod? That seems like a surprising behavior are we doing this somewhere else? Does the general behavior outlined here seem sane to the people in this thread?
I think in 1.12 time frame, we should start to scope the work on improving extended resource life cycle. Right now, as we are approaching 1.11 code freeze timeline, I am trying to figure out a solution that is least surprise to any existing workloads. Till 1.11, if there is indeed a real use case where an external component directly patches node status capacity to export a resource, the solution already needs to handle the capacity loss after node upgrade by repatching the field. That is the least surprising solution I can think of now. If folks have concerns on the breaking behavior this may introduce, we should seriously consider reverting part of in 1.11.
+1 for considering reverting for v1.11 timeframe. What would be the side-effect of that ? Ownership of fields in the Node object has been a long standing issue. For example node labels suffered from the same issue. I do agree that kubelet has to be resilient and has to recover from control plane data loss with minimal impact on running applications. Having to expose arbitrary extended resources as device plugin resources is also not ideal.

For instance, if one wants to expose inodes or network bandwidth as a resource, device plugins would not be an ideal interface (we can consider fixing this). k8s node upgrades should ideally involve cordoning (or tainting) the node during the duration of the upgrade. During this upgrade phase the node is primarily expected to keep existing pods running (if the upgrades are in place) and the system (scheduler) is not expected to place additional workloads. Since the node object is a shared resource, every component that relies on the node object must ensure eventual consistency. Arbitrary users are not expected to update/patch the node object since it is core to k8s functionality. does node feature discovery continually update the state on the node object to reflect the current state on the node? Until we have a better resource representation format, kubelet will not be able to tag ownership for individual extended resources attached to the node object. I think this limitation is acceptable and the users who introduce additional extended compute resources are expected to be sophisticated enough that they can run long running processes to ensure their additional compute resources are accurately reflected in the node object. As part of Resource Classes, and v2 compute resource APIs, we should consider incorporating a owner reference into concrete compute resource objects which can help with the ownership issues. Node upgrades also need non-trivial automation today to ensure that all node dependencies (runtime, kernel, OS, network, storage plugins, etc.) are logically ready prior to adding a node back to the scheduling pool. Assuming patches to node compute resources are continually updated and nodes are cordoned during the upgrade phase, node upgrade automation can ensure that all the expected extended compute resources on certain nodes are updated to a non-zero capacity prior to adding a node back to the scheduling pool (uncordon) after an upgrade. In addition, additional automation can be employed to auto-taint nodes that do not expose non-zero capacity for expected extended resources. For the near future, kubelet can on-startup can only try to restore state on a best effort basis. If checkpoints are not around, kubelet may override existing patched fields. Device plugins and other extended resource plugins will have to ideally update capacity/allocatable to reflect the state on the local node prior to marking an upgrade as complete.

it would delay removal of the self-deletion permission for yet another release (already pushed to 1.12 because of version-skew), meaning node taints would not be usable for isolating compromised nodes until at least 1.14 (because the associated kubelet could simply delete/re-register their Node object). I also don't think it would actually solve this issue. According to , even were we to revert and restore kubelet self-deletion, changes made in stabilized the computation of external id in GCE such that it no longer changes as part of node upgrades in 1.11+. Undoing that, destabilizing the GCE computed external id, and reintroducing self-deletion cases to solve the stale resource issue in one particular cloud environment doesn't seem like a good approach.
The feature that would be slipping is which is particularly critical for and because it's required to run a cluster with dedicated nodes securely. We would be slipping another release on something that is already painfully slow due to backward compatibility considerations. The PR () that changed the behavior device manager was depending on merged two months ago. It's really unfortunate that this report came in so late. I'm pretty convinced that this is a device manager issue that is only mitigated by clearing the node object. Is there anything I can do to help fix this? I'm happy to argue that changes here should be exempt from the codefreeze deadline.
The node feature discovery project only deals with node labels, and not extended resources. NFD fairly recently the ability to run as a daemon, but also retained the option to publish labels once and exit.
So far, it sounds is perhaps the most realistic solution for 1.11 time frame. Although we want to support good user experience, I do feel we need to balance the complexity we need to add into the system v.s. the use cases we want to optimize for. I feel the ask for the external component to repatch node status capacity after kubelet restart seems reasonable and perhaps is already implemented by any serious use cases.
The change that was backported to 1.10 (and maybe all versions?) for this seems to have had the unexpected consequence of zeroing out all of our extended resources upon kubelet restart and causing a large number of our pods to get evicted in our QA cluster since they all use extended resources.

I filed an issue here: if you have any advice on intentions/workarounds. For now, we reverted that commit in our internal build as a workaround. Thanks!
How is your extended resource exported on a node? Are they being exported by a device plugin or do you directly update your node objects? If it is the later case, do you have a central controller that may watch for node updates and re-export your resources?
When the host is first configured, our configuration management system running on the node starts the kubelet service, waits for kubelet to register itself in the API by polling and then patches the node capacity directly. We do not have a central controller to rexport the resources. It also sounds like watching and rexporting is not sufficient, since the value would still hit zero for some brief period of time before the central controller could react, at which point kubelet will have already begun hard evicting all the pods. The only central solution I think would work is doing it from an admission controller that ignores the kubelet's attempt to set the extended resources in our namespace to zero.
Can the configuration management system re-patch node capacity after each Kubelet registration?
The kubelet service can restart at any time, whether the config management system is currently configuring the system or not. Ex: kubelet could get OOM-killed, crash, fail healthchecks, etc. If every kubelet restart is a kubelet registration, then no, unless we made the kubelet service also launch a script every time it restarts. So lets say that we hypothetically did configure a script to re-patch the node capacity every time kubelet restarts: there is still an unsolvable race condition where the node temporarily has its capacity set to zero. Consider the situation: is up and healthy. The value for . kubelet service restarts reconciliation logic sets . kubelet immediately begins hard-evicting every pod that requested a . script patches the node capacity back to , but it's too late, every pod has already been killed non-gracefully by kubelet due to the resource having been reset back to 0 at step 3.
Sigh, this is the deficiency we documented in Unfortunately, I couldn't think of a clean solution without API change. One thing we may consider is to instead of zeroing out extended resource capacity, we delete them.

Because we are currently taking an open policy on extended resource admission, pods already assigned on the node wouldn't be affected, and your configuration manager can re-patch node later. Another possible workaround you may consider is to implement device plugin API in your configuration manager. Because Kubelet maintains device plugin checkpoint, it would allow pods to pass admission based on the checkpointed state. cc for thoughts on this issue.
Note for the first mentioned workaround, i.e., instead of zeroing out extended resource capacity, we delete them, I think we will rely on to re-populate device plugin resource runtime states during container restart retries. This is not ideal but we already need this to handle the edge cases that node API object may get deleted but pods are still assigned to node, or pods can be admitted before node re-register with API server.
Unfortunately, I couldn't think of a clean solution without API change. The effect of this change as it stands is that kubernetes users run the risk of having 100% of pods in their cluster completely wiped out with no respect for PDB, terminationGracePeriodSeconds, and preStop hooks just by restarting the kubelet processes. This really feels like a critical bug that has much worse consequences than the bugs the change sought to solve. If this is going to continue being the behavior, there should be a really loud disclaimer on which currently makes it appear that extended resources are trivial to manage. I had a team of 5 people restoring 350 stateful database clusters to a clean state for 3+ hours on Monday night. Restarting kubelets was never a destructive action before, so we weren't restarting them as slowly and carefully as if we were rolling out a config change. If this had happened in our production cluster instead, it could have brought the company down for hours. So you're saying that simply removing the entry from the capacity map instead of zeroing it would allow pods using those resources to continue running? When you re-patch the capacity, does that break the accounting at all? (Ex if 3 pods were consuming an extended resource, and the resource is deleted and re-, does the scheduler realize that 3 of that resource are still consumed by pods, or would it reset to thinking zero pods are using it or would it count them correctly?) I'm willing to do that if it's really necessary.

That said, I couldn't find the code that does that during kubelet registration in 1.10, what happens if the device plugin is down during a kubelet restart? Does kubelet then zero out the resource? Does the device plugin need to be running for a container using the resource to successfully start? As a possible suggestion that doesn't require an (http) API change or the cluster admins to run a device plugin daemon, has it been considered to implement similar to ? on kubelet. That way the admin would have a trivial way to explicitly tell kubelet to always register with a static extended resource capacity value, making the reconciliation process much more straightforward.
Paul Furtado <: state. It would see three pods are already assigned to the node and wouldn't assign more. being scheduled on the node till the resource is ready and device plugin re-registers. However, in Kubelet device plugin manager, we use check-pointed state to bypass predicate admission handler so running pods wouldn't be affected. The function that does this is at: Jiaying
Could we add a whitelist configration to skip reconcile specified extended resource? Or skip a specific prefix
That would require changing kubelet flags, which I am hoping to avoid.
but we have no way to distinguish the extended resource is reported by device plugin or patched by users directly. Seems we can't use non device plugin extended resources anymore. b/c we can't guarantee to repatch resources before pods evicted.
It is indeed a question whether we want to support the use case of directly patching node object to export a resource. Capacity/Allocatable are node status fields. By definition, they should be observable from the node. I.e., even when Kubelet is isolated from the API server, it should be able to reconstruct its status fields. I think even without zeroing out extended resource capacity during API registration, there is perhaps already a race between pod admission and API registration. They are performed by separate go routines on Kubelet and there is no guaranteed ordering between these two processes. This wan't a problem in early versions of 1.10 because we changed extended resource admission into open policies. It remains a question whether we want to keep this open admission policies or not.

Adding an explicit Kubelet flag to specify what extended resources (and perhaps also their capacities) it should export may be a better way than directly patching node API object, because it would make that information truly observable. Whether we have enough use cases to justify this flag is perhaps a question we should discuss in sig-node.
Could we have a "generic device plugin" / "extended resource device plugin"? e.g: You deploy this device plugin and start interacting with a CRD rather than patching the node object in the spec?
For this case, shall we take the following ways to avoid pod restarts when kubelet restart? (1) we can use the default namespace prefix ,"" to avoid being zero out (2) remove extended resource before kubelet start, and patch add again when kubelet started Both of the two ways can make pod with extended resource not evicted then kubelet restart


Question: {utiliptables.TableFilter, kubeProxyFirewallChain}, {utiliptables.TableFilter, kubeSourceRangesFirewallChain}, {utiliptables.TableFilter, kubeIPVSFilterChain}, <ins> {utiliptables.TableFilter, kubeIPVSOutFilterChain}, </ins> } var iptablesCleanupChains = []struct {

When using loadBalancerSourceRanges to allow access only from CIDRs outside the cluster segment, access from the K8s nodes (or pods with hostNetwork:true) should be rejected, but for proxy-mode=ipvs it works. /sig network /area kube-proxy /area ipvs Access from K8s nodes should be rejected in the same way as for proxy-mode=iptables Define a service with loadBalancerSourceRanges that allows access from a cidr outside the cluster, e.g. The K8s nodes have addresses on 192.168.1.0/24 and fd00::c0a8:100/120. Make sure a loadBalancerIP is assigned (in this example 10.0.0.0/32) and try to access it from a K8s node: Same for both ipv4 and ipv6. This is related to since a fix requires that the passed to the is correct. Please see v1.27.4 and master None Linux 6.4.4 None N/A N/A
wanted to work on this
Great! But please coordinate with I hope that one get accepted, and then ~the s (both families) passed to the proxiers should be usable.~ No, they are not, please see
Discussed this one in the SIG Network meeting today: we agree this is something we'd like to implement, however we weren't sure exactly how to do it with ipvs during the meeting. Whoever takes this on needs to come up with and communicate a clear plan of how it will be accomplished. /triage accepted
My comment is not feasible. For proxy-mode=ipvs the src and dst is both loadBalancerIP. So the nodeIP will not help :disappointed:
I did a small experimented with kind + metallb. I created a loadbalancer service backed by . loadBalancerIP: 172.18.255.200 podIP: 10.244.1.3 conntrack picks the src and dst both as loadBalancerIP. Tried to create an iptables rules matching source and destination with loadbalancerIP to drop packets but it didn't work. After some exploration I found and with the following rules I was able to prevent access to lbIP from k8s nodes / host network.

The solution looks hacky though, we only need this rule if source ranges are defined for the service and node ip is not covered by any of them. Will try to program this in the ipvs proxier.
Good find! :+1: I had missed the ipvs extension. I started to check nodeIP and the (flawed) option to experiment with setting a nodeIP as "src" for some routes to dodge the lbip-both-as-src-and-dst problem, and an old "vision" I have to not assign clusterIPs to at all, but have one rule to replace them all, like: It can be done, and in a cluster with 1000's of clusterIP's it would simplify


Question: # The business logic for whether a given object should be created # was already enforced by salt, and /etc/kubernetes/addons is the # managed result is of that. Start everything below that directory. <del> KUBECTL=/usr/local/bin/kubectl </del> <ins> KUBECTL=${KUBECTL_BIN:-/usr/local/bin/kubectl} </ins> function create-kubeconfig-secret() { local -r token=$1

has a hardcoded bin path. This is not the path for yum installed Kubernetes and maybe more cases. Should be a variable at least. similar to .
I think has the most kubectl context and can figure out something reasonable to do here?


Question: return nil } <del> func portsFromString(spec string) []api.Port { </del> <ins> func portsFromString(spec string) ([]api.Port, error) { </ins> if spec == "" { <del> return []api.Port{} </del> <ins> return []api.Port{}, nil </ins> } parts := strings.Split(spec, ",") var result []api.Port

Kubernetes will create replication controller and pods in case of port spec error, e.g. will create 'myNginx'. A following correct spec will fail:
/cc


Question: "k8s.io/klog/v2" "k8s.io/mount-utils" <ins> cadvisorapi "github.com/google/cadvisor/info/v1" </ins> v1 "k8s.io/api/core/v1" "k8s.io/apimachinery/pkg/api/resource" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: On a node that has a topology of more than 1 numa node, If some or all of the CPUs of a numa node are turned off (mostly for testing purposes) a valid set can cause the Kubelet service to fail on: This failure has occurred on a node that has 2 numa nodes with hyperthreading enabled. What you expected to happen: When setting a valid set, the Kubelet service should pass validations and function properly even if some CPU cores are turned off but the reserved cpus are still under a valid range. How to reproduce it (as minimally and precisely as possible): minimal cluster should have 1 master and 1 worker node. worker node should have 2 Numa nodes and hyperthreading enabled(BIOS setting) Important: The CPUs divided between the numa nodes cannot be sequential in order to reproduce the issue that are under Numa node 0 only (under ) Kubelet service off all CPUs that belong to numa node 1 () Kubelet service systemctl status kubelet.service and journal logs that will present: Anything else we need to know?: The scenario of split ranges of Numa nodes might occur only with Hyperthreading enabled. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: OS (e.g: ):Kernel (e.g. ): (real time kernel) Install tools:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Here is a real example where the error occurred: For the following CPU layout (after turning off numa node 1 CPUs) using : will result in: CPUs 0-2 and 40-42 are all under numa node 0 and are active.

The validation fails here: In this scenario we have 40 active CPUs (all from numa node 0) but the range check will fail on the last item in reserved CPUs: 42 (last time) = 40 (NumCores)
/cc
/sig node
This is probably not just strictly related to NUMA. The same behaviour could happen when offlining CPUs on any multi-core system, for example: disabling HT on a single numa node may end up with a very similar scenario (depends on the CPU specifics). The kubelet validation is too naive; it needs to be aware (and act accordingly to) the online CPU set. The emerging hidden assumption in kubelet is all the physical cpus are always online.
I wasn't able to recreate with all cpus on numa 1 offline, On-line CPU(s) list: 0-11,24-35 Off-line CPU(s) list: 12-23,36-47 with HT enabled, --reserved-cpus=0-3, kubelet restart fine but as mentioned by this option kubelet validation didn't consider the situation where some cpus are offline.
Ok I was able to reproduce when I used --reserved-cpus=28-35.
Just a FYI, even with the validation update, if you take certain CPUs offline, you need to delete /var/lib/kubelet/cpumanagerstate before restart kubelet. This state check is done by cpumanager, see
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale


Question: var result []api.Pod for _, value := range pods { if api.PodSucceeded != value.Status.Phase && <del> api.PodFailed != value.Status.Phase { </del> <ins> api.PodFailed != value.Status.Phase && api.PodUnknown != value.Status.Phase { </ins> result = append(result, value) } }

I have 3 identical nodes, 1 replication controller with 2 replicas of a pod. I simulate failure of one of the nodes to where pods have been scheduled by simply unplugging the node. Kubernetes realizes that the node has failed, but the pod gets not re-scheduled to the another available node. The documentation at says the following: and So is that a mistake in the documentation or a missing or a bug? I use v0.10.
That's definitely a bug, and something that used to work correctly. I will investigate today, and follow up on this bug. Thanks for the report. Brendan On Feb 5, 2015 8:11 AM, "peter-braun" wrote:
Actually, it's really a missing feature -- . I believe the last of the mechanisms we needed were ~last week. said he would start on it.
Is there a rudimentary solution that we could get in quickly, and then improve next week?
Just sent a simple PR with the fix.
Note that this issue is about a replication-controller set of pods. This isn't which is a broader "fix pods on failed nodes in general" issue.
Ah, a cache bug. Will take a look.
Re-opening this, because I reverted the relevant code in the replication controller with
I'll comment on the other thread later. Sorry I was out yesterday. I'm a bit free this weekend and should be able to work on the problem for at least one day.
Fixed by . Re-open if not fixed.
was merged a few days ago. It referenced , but not this issue.


Question: // Emits a fake PrepareForShutdown dbus message on system dbus. Will cause kubelet to react to an active shutdown event. func emitSignalPrepareForShutdown(b bool) error { <del> cmd := "dbus-send --system /org/freedesktop/login1 org.freedesktop.login1.Manager.PrepareForShutdown boolean:" + strconv.FormatBool(b) _, err := runCommand("sh", "-c", cmd) return err } func lookEmitSignalCommand() error { _, err := exec.LookPath("dbus-send") return err </del> <ins> conn, err := dbus.ConnectSystemBus() if err != nil { return err } defer conn.Close() return conn.Emit("/org/freedesktop/login1", "org.freedesktop.login1.Manager.PrepareForShutdown", b) </ins> } func getNodeReadyStatus(f *framework.Framework) bool {

xref: Graceful node shutdown test uses to invoke dbus - As per some distros, don't have installed so the test must be skipped in those environments. We should consider if we can switch to use or perhaps and fall back to to ensure the test can work across more distros.
/sig node
/sig testing
/remove-kind bug /kind feature I'm putting kind feature to signify that nothing is really broken in k8s, tests needs improvement.
/remove-sig testing This dbus command should be up to SIG node, though tentatively: /triage accepted
/area test
Do we actually have these other distros available to prevent regressions if we change that?
/assign
the change will be to replace skip logic when is missing to use an alternative way to send this. So there will be no regression on distros where it currently runs without the skip. Original bug should have a link to job that was failing
I would like to work on improving the command. Am I understanding correct here We have to change to logic where we skip to use busctl instead. Could you please give me some idea on how can we make it to use , some similar examples in the repo would be really useful.
please have a look at The issue is that the command available is too old and doesn't include .
I see, it looks like we are not just adding a condition for instead we are replacing it with . It looks you are already working on it. Please inform if I can help in anyway. Thanks for the response.
Both are possible... it was easier to get a CI signal if that was used in all cases. I might put the condition later. I am waiting for comments in the PR, but feel free to propose other solutions that might work with older systemd versions.


Question: newScheduling := obj.(*resource.PodSchedulingContext) oldScheduling := old.(*resource.PodSchedulingContext) newScheduling.Spec = oldScheduling.Spec <ins> metav1.ResetObjectMetaForStatus(&newScheduling.ObjectMeta, &oldScheduling.ObjectMeta) </ins> } func (podSchedulingStatusStrategy) ValidateUpdate(ctx context.Context, obj, old runtime.Object) field.ErrorList {

The 's was copied from some other, broken types. A call to is missing and therefore, for example, finalizers can be changed during a status update. That is not supposed to be possible. Object meta changes need to be removed during status updates. n/a No response Since 1.26 till at least 1.29. n/a No response No response No response No response
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
/assign


Question: // it has no more use.  Call this if both (1) plState.queues is // non-nil and reported being idle, and (2) cfgCtlr's lock has not // been released since then. <del> func (cfgCtlr *configController) maybeReapLocked(plName string, plState *priorityLevelState) { </del> <ins> func (cfgCtlr *configController) maybeReapReadLocked(plName string, plState *priorityLevelState) { </ins> if !(plState.quiescing && plState.numPending == 0) { return }

Two flakes recently: They fail with error like: or /assign
/sig scalability
In logs I have found the following slow calls: It's a bug that we use exempt priority level for clusterloader: I created It's quite worrisome that APF handler takes 1s for exempt request.
I checked a APF code and I think the flow for exempt requests is: genericfilters.WithPriorityAndFairness() workEstimate := workEstimator.EstimateWork(r) fcIfc.Handle() cfgCtlr.startRequest() () looks for matching flowschemas in cfgCtrl.flowSchemas returns for exempt for other requests it calculates sha256 for flowDistinguisher and queues request req.Finish updates metrics calls execFn updates headers handler.ServeHTTP which stops the timer for priorityandfairness For me, the most likely reason for slowness is lock contention on We know that the test is super cpu-intensive in that part and other requests are e.g. calculating sha256 keeping that lock. A potential improvement would be to change this lock to RWLock or return the lock before starting calculating sha256 (if we believe that this is the most expensive operation there). /cc
/assign
I think the lock-contention is even worse than what you wrote, as e.g.: is happening under this lock (which is also dispatching requests). I will take a deeper look later this week, but I think that switching to RWLock might do the job without any invasive changes.
may trigger the execution of some requests that were waiting, but they are not executed in the same goroutine. Yes, I think changing that Mutex to a RWMutex would be an easy way to greatly reduce contention.
I agree switching to RWMutex is likely to help a lot, if that's not enough I think with a little work we could e.g. switch to s.
(perhaps because that's what my code originally did? Reviewers thought the retirement dance was too tricky.)
/triage accepted /priority important-soon I will work on the fix.
/sig api-machinery
Wow! Thanks a lot for rapid help here! Let's wait for the next test run to see if it helped. /reopen
Reopened this issue. <detailsIn response to : Wow! Thanks a lot for rapid help here! Let's wait for the next test run to see if it helped.

/reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Yes, can you please report the filter latency regardless of whether it is bad or good? I would like to know how much of a difference that PR made.
Sure, will report the latency change in both cases.
I compared (the one where we saw issue) (first with the lock improvements) I tried to find requests that: use exempt PL (so shouldn't be throttled) has fl_priorityandfairness, key which means that priority and fairness filter took more than 100ms Sample matching request: In the old run, I see ~6k such requests, with the locking improvements there are no such requests. The latency of PATCH jobs seems to improve significantly: ! So overall it looks great. The only worrisome metric I see is that actual P&F throttling for other requests has grown: With locking improvements: ! Old: ! E.g. 'node-high' has now throttling latency of ~1.5s in spikes where in the previous test run it was more like ~800ms. I don't think the lock improvements touched this directly. I expect that this is either: per-test variance (if we rerun the old run again we would see different throttling latencies as well) we throttle exempt requests less (as expected) giving less resources to other requests. Anyway, I don't think this change is meaningful as per-test latency aggregates for PATCH nodes/status and PUT leases (the most common requests in node-high) look ~same as previously: ! ! I think we can close this for now as the issue seems to be resolved. Thanks a lot for your help here! /close
Closing this issue. <detailsIn response to : I compared In the old run, I see ~6k such requests, with the locking improvements there are no such requests. The latency of PATCH jobs seems to improve significantly: ! ! ! I expect that this is either: ! ! /close </details
: thanks for reporting!


Question: if instance.State != nil { state := aws.StringValue(instance.State.Name) // valid state for detaching volumes <del> if state == ec2.InstanceStateNameStopped || state == ec2.InstanceStateNameTerminated { </del> <ins> if state == ec2.InstanceStateNameStopped { </ins> return true, nil } }

/kind bug What happened: After upgrading 1.11.1 -1.12.0 , when a node gets terminated in AWS, the controller-manager doesn't delete the node in the API. This has lead to several side effect for us : pods don't get GC Prometheus alerts firing because 100 of nodes are in the NotReady state for a long period What you expected to happen: kube-controller-manager deletes the node when it stops reporting, after checking with the cloud provider. How to reproduce it (as minimally and precisely as possible): Looking at the code, I believe this is what introduced the behaviour change : - The node gets tainted with because it's in a Terminated state. But it doesn't get removed because it still "exists" in the cloud provider. The behaviour is probably correct for a shutdown node, but in AWS a terminated node can't be restarted so there's no good reason to keep it there ? Environment: Kubernetes version : 1.2.0 Cloud provider or hardware configuration: AWS OS (e.g. from /etc/os-release): CoreOS (sorry for the all the sigs, not sure who it belongs to) /sig aws /sig cloud-provider /sig cluster-lifecycle
as I see it should delete node if instance is in terminated state have you tested that change in that row? I do not see what could be the issue there. Or could it be that the is not in place for all binaries in 1.12.0?
I might be wrong, but I think that part of the code never gets executed once the node is tainted :
good finding. So we should remove terminated state from this row. Then it does not add shutdown taint if node is terminated - instead it will delete node. also this should return false.. because if it returns true node is not deleted
I had the same issue. After I scale down my worker node instance number from 3 to 1. One node is stuck in status and never got deleted. However, one interesting thing is one of the other node is removed successfully. do you see similar behavior too?
- can we make sure this gets cherry picked for 1.12 releases ? I think we already missed the cut for 1.12.2


Question: // Bash's $RANDOM generates pseudorandom int in range 0 - 32767. // Dividing by 16384 gives roughly 50/50 chance of success. job.Spec.Template.Spec.Containers[0].Command = []string{"/bin/sh", "-c", "exit $(( $RANDOM / 16384 ))"} <ins> case "failOnce": // Fail the first the container of the pod is run, and // succeed the second time. Checks for file on emptydir. // If present, succeed.  If not, create but fail. // Note that this cannot be used with RestartNever because // it always fails the first time for a pod. job.Spec.Template.Spec.Containers[0].Command = []string{"/bin/sh", "-c", "if [[ -r /data/foo ]] ; then exit 0 ; else touch /data/foo ; exit 1 ; fi"} </ins> } return job }

Job should run a job to completion when tasks sometimes fail and are locally restarted Expected error: <*errors.errorString | 0xc20807c990: { s: "timed out waiting for the condition", } timed out waiting for the condition not to have occurred
Couple of failures: from
Another failure:
Another failure:
This test was introduced by and in and . It's currently failing at a rate of about 7%. When it passes, it typically does so in 30-90 seconds. Occasionally it times out after 15 minutes. I think that there's a real bug in the system here. The timeout occurs while waiting for the Job to terminate, which it never does.
I'm going to label it as flaky until the bug has been fixed.
Another portion of flakes:
/sub
any progress on this? Is it being actively worked on?
I've been working on this today. Saw 1 e2e failure in 50 runs. Logs suggest the problem is that half of kubelet's brain thinks it started a pod, and the other half of it thinks it didn't. In fact it didn't. So the test times out waiting for the not-started pod to complete.
Update: the kubelet did start all the pods, and they all terminated, and then the job controller somehow didn't notice that they were done until 40+ minutes later.
This is interesting. There were 3 apiserver 500's around the time of the test failure, and no 500's any other time in the 8 hours the cluster was up, and many other test runs occurred. 500s:
The code that is failing is this in at line 271 the gets an error, and then decides the error is a 500.
can you say anything about the above mentioned apiserver internal error which occurs immediately on receiving a watch request?
Look in the relevant kubelet logs to see how they interpreted the error. It's a bit silly that we printed a stack trace but not the error text :( , Eric Tune wrote:
E1022 04:13:24. 3622 ] Failed to watch *api.Pod: Internal error occurred: too old resource version: 171 ()
This is happening at line 269 of Is this an unexpected condition? If not, why is it a 500?
Oh, I thought that was fixed.

It shouldn't be an internal error, you're out of the watch window and need to re-list. , Eric Tune wrote:
OK I reopened . did the kubelets involved re-list soon after getting these 500s? If so, then this is coincidence and not your problem. , Daniel Smith wrote:
In all 3 cases, they relist immediately, but then they don't re-watch for about an hour. That seems wrong, no? Relist at 3:56 and rewatch at 4:53 Relist at 4:07 and rewatch at 5:00 relist at 4:13 and rewatch at 4:55
The time reported is the end of the watch, not the beginning. The last one it seems you actually didn't get the log line for the watch, actually. But the first two look WAI. , Eric Tune wrote:
Oh, duh.
Working on which I suspect may fix this.
Okay, looking at a new test failure I see that there are expected failures, and the test just times out due to backoff by the kubelet. The following things happen: runs and fails (1) runs and fails (2) backs off for 10s runs and fails (3rd failure) backoff 20s runs and fails (4th failure) backoff 40s runs and fails (5th failure) backoff 1m20s runs and fails (6th failure) backoff 2m40s runs and fails (6th failure) kubelet backoff 5m runs and fails (6th failure) backoff 5m timeout now that at least 15m10s has elapsed in backoffs. Failures have a probability of 50%. Chance of getting 8 fails in a row is 1 in 256. Four pods per test run is 4 in 256 or 1 in 64. I see failures about 1 in 50, so that seems about right. So, I need to rewrite the test so that it writes to the emptydir after the first failure, and succeeds after that.
Ugh. My fix for local breaks non-local restarts.
Happened again.


Question: endpoint:             "/var/log/kubelet/plugins/myplugin/csi.sock", versions:             []string{"v1.2.3", "v0.3.0"}, foundInDeprecatedDir: true, <del> shouldFail:           true, </del> <ins> shouldFail:           false, </ins> }, { pluginName:           "test.plugin",

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: CSI allows 0.x drivers only in the deprecated kubelet plugin dir. CSI allows 1.x drivers only in the new kubelet plugin dir. If a driver implements both 0.x and 1.x (which is unlikely since the side car containers do not support this), kubelet requires the driver live in new kubelet plugin dir. So CSI drivers that implement both 0.x and 1.x if they want a seamless transition between k8s 1.12 to 1.13 would need to deploy in both old and new dir. What you expected to happen: argues that this is adding churn for CSI authors. He recommends allowing any driver that implements 0.x (even if it implements 1.x) to use either new or old (not just new). /milestone v1.13 If release team approves, PR is small and I can get prepare it pretty quickly. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
discussed in the burndown meeting, please go ahead and prepare the PR
/sig storage /priority critical-urgent /milestone v1.13
we need to wait for a fix for this before lifting code freeze


Question: Log_File      /var/log/fluentbit.log Daemon        off Parsers_File  parsers.conf <del> HTTP_Server   off </del> <ins> HTTP_Server   off </ins> HTTP_Listen   0.0.0.0 HTTP_PORT     2020 plugins_file plugins.conf

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: There's a race condition between starting Windows service and when the VMs network is actually available to communicate with the VM's metadata server. If the service starts first and cannot communicate with the metadata server or start talking to the stackdriver logging service it'll crash and not resume because the service is not configured to restart on failure. Typically a is used for a service like this but Windows does not support privileged containers so a standard Windows Service is used. What you expected to happen: If the service fails to start it should be automatically be restarted. How to reproduce it (as minimally and precisely as possible): This is a flaky issue but a way to repro this is to effectively by RDP'ing into the node and running: Anything else we need to know?: Environment: Kubernetes version (use ): v1.16 and onward Cloud provider or hardware configuration: GCE OS (e.g: ): Windows Kernel (e.g. ): Windows (any version) Install tools: cluster/gce/windows/README-GCE-Windows-kube- Network plugin and version (if this is a network-related bug): Others:
/sig windows
/assign jeremyedwards
GitHub didn't allow me to assign the following users: jeremyedwards. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign jeremyedwards Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: // These fields are not copied "SessionTicketsDisabled", "SessionTicketKey", <del> "DynamicRecordSizingDisabled", "Renegotiation", </del> // These fields are unexported "serverInitOnce",

When using go 1.7, is failing with the following error:
Emm... I'll need this fix
Wouldn't a full struct copy work if we zero out the fields we don't want retained afterwards? It'll be less fragile as well.
You prefer to re-open my pull request or include that commit in ?
I'm also trouble in the issue when I execute command on release-1.4 branch of Kubernetes. Have someone fixed it? Fails (all fails in ):
, we can not just re-open : in , it did not include & which means build failed in go1.6 if re-open .
BTW, do we still support go1.6 in master? If so, we need upmerge to master.
Is this still a problem? I don't see any test failures on go-1.7.3

Hey Guys, When running command on release-1.4 branch, I got the below errors, my GO version is , how to fix it or something else have I missed?
dup of . fixed by . we can close this. /sig api-machinery
Sure, close it.


Question: github.com/containerd/containerd/platforms # github.com/containerd/ttrpc v1.0.2 => github.com/containerd/ttrpc v1.0.2 github.com/containerd/ttrpc <del> # github.com/containernetworking/cni v0.8.0 => github.com/containernetworking/cni v0.8.0 </del> <ins> # github.com/containernetworking/cni v0.8.1 => github.com/containernetworking/cni v0.8.1 </ins> ## explicit github.com/containernetworking/cni/libcni github.com/containernetworking/cni/pkg/invoke

is identified with with severity rating as high. This is found using snyk scan while running against k/k master We're not using directly however, it gets tagged for vulnerability scans as we use . The fix is already present in and we can bump the dep. Bump dep to which has the mentioned CVE fix. and check Kubernetes version (use ): n/a Cloud provider or hardware configuration: any OS (e.g: ): n/a Kernel (e.g. ): n/a Install tools: n/a Network plugin and version (if this is a network-related bug): n/a Others: n/a /sig architecture /area code-organization for the dependency
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: } type listener struct { <del> addr        Addr connections chan *connection ctx         context.Context cancel      func() </del> <ins> addr   Addr ctx    context.Context cancel func() mutex       sync.Mutex connections [maxConcurrentConnections]*connection </ins> } var _ net.Listener = &listener{}

ci-kubernetes-csi-canary-test-on-kubernetes-master, [It] [sig-storage] CSI mock volume CSI Volume Snapshots [Feature:VolumeSnapshotDataSource] volumesnapshotcontent and pvc in Bound state with deletion timestamp set should not get deleted while snapshot finalizer exists [It] [sig-storage] CSI mock volume CSI Volume Snapshots secrets [Feature:VolumeSnapshotDataSource] volume snapshot create/delete with secrets 25-08-2022? No response Looking at "volume snapshot create/delete with secrets" test, it finishes OK, but the "Cleaning up resources" freezes for ~30 minutes: After these 30 minutes, the test timeouts and kills API server and etcd a and everything fails miserably. Stack trace from "Cleaning up resources", it's stuck at shutting down gRPC server: /sig storage
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I reproduced this issue in somewhat hacked from external-provisioner (or any other kubernetes-csi repo). The script compiles Kubernetes master from a fresh checkout, runs it in KIND, installs hostpath CSI driver with canary images and runs e2e tests for the CSI driver like this: "hostpath CSI driver with canary images" is the interesting part - when I run the test with the latest tagged images, it succeeds, when using canaries it freezes at the cleanup. I found out that using the latest tagged images for everything except is enough to freeze the cleanup. And further I was able to check that node-driver-registrar at commit works, while master does not. There are just 2 commits in between: new release-tools and update to v1.25 libraries. I suspect it's the new libraries. I still don't understand how node-driver-registrar and its libraries could affect gRPC between the hostpath CSI driver and binary.
does this ring a bell? I though node-driver-registrar does not do anything with proxied "mock" CSI calls between and the CSI driver.

In addition, why only these two tests fail reliably, while the rest works?
For the record, this is diff in k/k that adds canary node-driver-registrar to This diff is enough to make the tests fail:
This does not ring a bell at all. You are right that csi-node-driver-registrar should not be involved in the gRPC communication between and the driver. Does it communicate with the driver after the initial registration? If yes, then that will lead into connections that need to be proxied, which might prevent the shutdown. Just a wild guess. But even then it's not clear how grpc.Server.Stop can get stuck. According to its documentation, "It immediately closes all open connections and listeners.". Hmm, could the proxy code get stuck when its listener is closed?
There is in that gets stuck, I did not check what is in that group.
Well, of course it fails :sweat_smile: There is no such canary image. Hmm, but then why do other tests pass? They all use .
There is some magic in your script that replaces with , the registrar runs and registers the driver in the end.
Yep:
One difference is that the driver-registrary canary no longer detects the loss of connection to the mock proxy. Previously it did: It could be that the proxy code relies on the remote side to close their side of the connection before considering everything shut down. That is necessary because the remote side might still send some data that needs to be received. If the remote side keeps the connection open, everything freezes. One can argue that it shouldn't block forever, but the bigger question is why "Lost connection" is not detected anymore with the more recent gRPC.
The "exit on connection loss" depends on our dialer being called again by gRPC:
/assign No idea yet, but I'll keep looking...
I can reproduce the same failure to shut down also in other tests. Must be a timing issue that it only occurred for the same two tests in the CI - not a very satisfying explanation, unfortunately. I have something that works for me locally: closing connections when shutting down the listener -see


Question: o.Prefix = true return o }, <del> expectedOutSubstrings: []string{"[pod/test-sts-0/test-container] test log content for pod test-sts-0n[pod/test-sts-1/test-container] test log content for pod test-sts-1n"}, </del> <ins> expectedOutSubstrings: []string{ "[pod/test-sts-0/test-container] test log content for pod test-sts-0n", "[pod/test-sts-1/test-container] test log content for pod test-sts-1n", }, </ins> }, { name: "pod logs with prefix: init container",

in This flaking test case was newly as part of the above PR Flake was reproducible with stress tool: [root kubernetes]# go version go version go1.22.2 linux/ppc64le [root kubernetes]# stress - ^TestLog /sig testing
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
Looks like in this test case is a concatenation of logs from Pod0 and Pod1, hence it is failing in some cases. I will add a PR to split it.


Question: expectNoError(err) isReady, err := podRunningReady(p) <del> Expect(isReady).NotTo(BeTrue()) </del> <ins> Expect(isReady).NotTo(BeTrue(), "pod should be not ready") </ins> <del> Expect(getRestartCount(p) == 0).To(BeTrue()) </del> <ins> restartCount := getRestartCount(p) Expect(restartCount == 0).To(BeTrue(), "pod should have a restart count of 0 but got %v", restartCount) </ins> }) })

If the container_probe test runs on a node without the nginx image, it sometimes takes 90s to get the pod in running. Most of this time goes in pulling the image, so I'm assuming it's registry related hiccup. Using might make it more reliable. Example failure: Note the following event in the logs:
a) we should not be pulling from docker in our tests - too hit-or-miss b) we should replace nginx (132 MB) with (4.5 MB) whenever possible. , Prashanth B wrote:


Question: return nil, err } <del> disks := *vm.StorageProfile.DataDisks </del> <ins> disks := make([]compute.DataDisk, len(*vm.StorageProfile.DataDisks)) copy(disks, *vm.StorageProfile.DataDisks) </ins> bFoundDisk := false for i, disk := range disks { if disk.Lun != nil && (disk.Name != nil && diskName != "" && *disk.Name == diskName) ||

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: In Azure disk attach/detach function, it uses slice() to store the disk list: And after changed, the original would also be changed, so original azure data disk list is corrupted after attach/detach disk example code as below: What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /sig azure /assign


Question: "k8s.io/apiserver/pkg/endpoints/request" "k8s.io/apiserver/pkg/features" "k8s.io/apiserver/pkg/registry/rest" <ins> genericfilters "k8s.io/apiserver/pkg/server/filters" </ins> utilfeature "k8s.io/apiserver/pkg/util/feature" "k8s.io/component-base/tracing" "k8s.io/klog/v2"

Optimize the memory usage of goroutine stack for serving watch requests. According to the of handling a watch request, the stack memory is 0x3768, which is very close to 16 KiB and Golang runtime will double it to 32 KiB once exceeded. In fact, it does exceed if we print the traceback deeper and take the fact that extra stack memory reserved by Golang itself into consideration. A naive optimization is to run as much as preparation handlers in a separate goroutine and then move the actual serving to another goroutine, so the former routine can free the stack memory earlier and make it more memory efficient. Given the extensive use of watch requests, 32 KiB per watch could add up to a very high amount of memory overhead for . A typical watch request commonly lasts for 30s or longer, which means this memory will be active for quite long. I see a great improvement in memory efficiency to do this optimization.
/cc
/sig api-machinery
/cc if you would like to take a look when you have time :) Thank you /triage accepted


Question: } v, err := conversion.EnforcePtr(listPtr) if err != nil || v.Kind() != reflect.Slice { <del> panic("need ptr to slice") </del> <ins> return fmt.Errorf("need ptr to slice: %v", err) </ins> } if s.pathPrefix != "" {

Remove panic in , please return error instead following .
/sig api-machinery /good-first-issue
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /sig api-machinery /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
Please also add some log if returns error
/remove-kind bug /kind cleanup


Question: return "", err } if !existsLb { <del> return "", fmt.Errorf("Expected to find an IP address for service %s but did not", service.Name) </del> <ins> glog.V(2).Infof("Expected to find an IP address for service %s but did not. Assuming it has been removed", service.Name) return "", nil </ins> } if len(lbStatus.Ingress) < 1 { <del> return "", fmt.Errorf("Expected to find an IP address for service %s but it had no ingresses", service.Name) </del> <ins> glog.V(2).Infof("Expected to find an IP address for service %s but it had no ingresses. Assuming it has been removed", service.Name) return "", nil </ins> } return lbStatus.Ingress[0].IP, nil

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: When creating many LoadBalancer services, some services may exceed Azure basic LB's FrontendIPConfiguations quota (default is 10). Public IPs are created for all services, but it is not removed after deleting the kubernetes services. What you expected to happen: Remove the public IP. How to reproduce it (as minimally and precisely as possible): Create more than 10 LoadBalancer services, wait a while for 10 services success, and then remove all services. Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig azure
/area cloudprovider


Question: "UsageBytes":      bounded(10*e2evolume.Kb, memoryLimit), "WorkingSetBytes": bounded(10*e2evolume.Kb, memoryLimit), "RSSBytes":        bounded(1*e2evolume.Kb, memoryLimit), <del> "PageFaults":      bounded(0, 1000000), </del> <ins> "PageFaults":      bounded(0, expectedPageFaultsUpperBound), </ins> "MajorPageFaults": bounded(0, expectedMajorPageFaultsUpperBound), }) runtimeContExpectations := sysContExpectations().(*gstruct.FieldsMatcher)

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --ci-cos-cgroupv2-containerd-node-e2e <!-- You can identify the SIG from the "prowjobconfigurl" on the testgrid dashboard for a test --<!-- Please see the deflaking doc () for more guidance! --/sig node
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : #### Which jobs are flaking: links to appreciated links to specific failures in spyglass appreciated </details
On cgroupv2, it looks like like the number of for cgroup is sometimes higher than is currently expected in the test. We currently expect this however, in a few runs of the test we've seen We may need to adjust this value similar as was done in for
/cc
yes, good idea. I think we need to adjust the PageFaults test as well. I've opened a PR:
/assign /triage accepted /area test /priority important-soon


Question: // startupProbe fails, but is disabled m.prober.exec = fakeExecProber{probe.Failure, nil} msg = "Started, probe failure, result success" <del> expectStop(t, w, w.doProbe(), msg) </del> <ins> expectContinue(t, w, w.doProbe(), msg) </ins> expectResult(t, w, results.Success, msg) }

We detected behavior change in v1.21.0 Example: POD has and for some containers Behavior in v1.20.5: starts applied startupProbe success, then start checks for livenessProbe working failed restarted startupProbe applied startupProbe success, then start checks for livenessProbe working Behavior in v1.21.0: starts applied startupProbe success, then start checks for livenessProbe working failed restarted applied, startupProbe skipped not ready yet, killed by liveness check in restart loop So, in v1.21 is applied to "fresh" starts only and not used when container restarted. There is not mentions in changelog that behavior of the is changed. should run on container restarts also. POD with and fail that doesn't apply after container restart Kubernetes version (use ):Cloud provider or hardware configuration: baremetal
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
/assign
Thanks for the quick fix
it was fixed in master (1.22) but not for 1.21, does it need a backport? /reopen
Reopened this issue. <detailsIn response to : it was fixed in master (1.22) but not for 1.21, does it need a backport? /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/close never mind i just saw the backports.
Closing this issue. <detailsIn response to : /close </details
can you summarize the impacted patch releases (looks like the regression was backported to 1.18-1.20 as well)
These 4 patch releases are impacted, 1.18.18, 1.19.10, 1.20.6, 1.21.0


Question: ContainersToKill:  map[kubecontainer.ContainerID]containerToKillInfo{}, }, }, <ins> "Verify we do not create a pod sandbox if no ready sandbox for pod with RestartPolicy=OnFailure and all containers succeeded": { mutatePodFn: func(pod *v1.Pod) { pod.Spec.RestartPolicy = v1.RestartPolicyOnFailure }, mutateStatusFn: func(status *kubecontainer.PodStatus) { // no ready sandbox status.SandboxStatuses[0].State = runtimeapi.PodSandboxState_SANDBOX_NOTREADY status.SandboxStatuses[0].Metadata.Attempt = uint32(1) // all containers succeeded for i := range status.ContainerStatuses { status.ContainerStatuses[i].State = kubecontainer.ContainerStateExited status.ContainerStatuses[i].ExitCode = 0 } }, actions: podActions{ SandboxID:         baseStatus.SandboxStatuses[0].Id, Attempt:           uint32(2), CreateSandbox:     false, KillPod:           true, ContainersToStart: []int{}, ContainersToKill:  map[kubecontainer.ContainerID]containerToKillInfo{}, }, }, </ins> "Verify we create a pod sandbox if no ready sandbox for pod with RestartPolicy=Never and no containers have ever been created": { mutatePodFn: func(pod *v1.Pod) { pod.Spec.RestartPolicy = v1.RestartPolicyNever

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: For a Pod with , kubelet would attempt to create a new sandbox for it even after all container succeeded. Logs as below: This issue is similar to which fixed the case of pods whose RestartPolicy are Never. It caused unnecessary CRI and CNI calls, confusing logs and conflicts between the routine that creates the new sandbox and the routine that kills the Pod. It's likely also the root cause of because sometimes the routine that creates the second sandbox and the routine that kills unwanted pod run concurrently, then the netns would be gone when calling CNI. What you expected to happen: kubelet shouldn't create a new sandbox for pod with RestartPolicyOnFailure if all containers succeeded How to reproduce it (as minimally and precisely as possible): Create a pod like this: Anything else we need to know?: Environment: Kubernetes version (use ): v1.18.2 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig node
Created which may fix this.
/assign


Question: if observedReadyCondition.Status != v1.ConditionUnknown { currentReadyCondition.Status = v1.ConditionUnknown currentReadyCondition.Reason = "NodeStatusUnknown" <del> currentReadyCondition.Message = fmt.Sprintf("Kubelet stopped posting node status.") </del> <ins> currentReadyCondition.Message = "Kubelet stopped posting node status." </ins> // LastProbeTime is the last time we heard from kubelet. currentReadyCondition.LastHeartbeatTime = observedReadyCondition.LastHeartbeatTime currentReadyCondition.LastTransitionTime = nc.now() } } <del> // Like NodeReady condition, NodeOutOfDisk was last set longer ago than gracePeriod, so update // it to Unknown (regardless of its current value) in the master. // TODO(madhusudancs): Refactor this with readyCondition to remove duplicated code. _, oodCondition := v1.GetNodeCondition(&node.Status, v1.NodeOutOfDisk) if oodCondition == nil { glog.V(2).Infof("Out of disk condition of node %v is never updated by kubelet", node.Name) node.Status.Conditions = append(node.Status.Conditions, v1.NodeCondition{ Type:               v1.NodeOutOfDisk, Status:             v1.ConditionUnknown, Reason:             "NodeStatusNeverUpdated", Message:            fmt.Sprintf("Kubelet never posted node status."), LastHeartbeatTime:  node.CreationTimestamp, LastTransitionTime: nc.now(), }) } else { glog.V(4).Infof("node %v hasn't been updated for %+v. Last out of disk condition is: %+v", node.Name, nc.now().Time.Sub(savedNodeStatus.probeTimestamp.Time), oodCondition) if oodCondition.Status != v1.ConditionUnknown { oodCondition.Status = v1.ConditionUnknown oodCondition.Reason = "NodeStatusUnknown" oodCondition.Message = fmt.Sprintf("Kubelet stopped posting node status.") oodCondition.LastTransitionTime = nc.now() </del> <ins> // remaining node conditions should also be set to Unknown remainingNodeConditionTypes := []v1.NodeConditionType{v1.NodeOutOfDisk, v1.NodeMemoryPressure, v1.NodeDiskPressure} nowTimestamp := nc.now() for _, nodeConditionType := range remainingNodeConditionTypes { _, currentCondition := v1.GetNodeCondition(&node.Status, nodeConditionType) if currentCondition == nil { glog.V(2).Infof("Condition %v of node %v was never updated by kubelet", nodeConditionType, node.Name) node.Status.Conditions = append(node.Status.Conditions, v1.NodeCondition{ Type:               nodeConditionType, Status:             v1.ConditionUnknown, Reason:             "NodeStatusNeverUpdated", Message:            "Kubelet never posted node status.", LastHeartbeatTime:  node.CreationTimestamp, LastTransitionTime: nowTimestamp, }) } else { glog.V(4).Infof("node %v hasn't been updated for %+v. Last %v is: %+v", node.Name, nc.now().Time.Sub(savedNodeStatus.probeTimestamp.Time), nodeConditionType, currentCondition) if currentCondition.Status != v1.ConditionUnknown { currentCondition.Status = v1.ConditionUnknown currentCondition.Reason = "NodeStatusUnknown" currentCondition.Message = "Kubelet stopped posting node status." currentCondition.LastTransitionTime = nowTimestamp } </ins> } }

I just found a cluster like this cc/
do you know if anyone else is currently working on a fix for this? I've already started looking into a fix, would you be willing to review my PR soon?
I don't know of anyone who is working on it.
(I see your PR, and I'll review it.)


Question: if err != nil { return nil, false, err } <ins> // TODO: expose TTL </ins> creating := false out := e.NewFunc() err = e.Helper.AtomicUpdate(key, out, true, func(existing runtime.Object) (runtime.Object, error) {

The use of AtomicUpdate by genericetcd doesn't handle conflicting writes by rejecting that to a user. It should. Eventually, we may want to hide write conflicts to /pods when status has changed, and to spec when /pods/status changes, but that requires us to have two version counters and that's not worth the effort. Fix incoming.
Also we need a unit/integration test for this.


Question: if err != nil { return nil, err } <ins> if s.tlsConfig != nil && len(s.tlsConfig.ServerName) > 0 { host = s.tlsConfig.ServerName } </ins> err = conn.VerifyHostname(host) if err != nil { return nil, err

an APIService that routes to an external API server (e.g. service catalog) via a service in namespace a caBundle in the APIService the external API server with a certificate signed for Result: Normal API requests (get/put/post) work as expected Normal watch requests work as expected Upgrade requests (e.g. websocket watch requests) return a 500 error with text like cc


Question: } type cacheKey struct { <ins> clusterAddress string </ins> // Canonical issuer URL string of the provider. issuerURL string clientID  string } <del> func (c *clientCache) getClient(issuer, clientID string) (*oidcAuthProvider, bool) { </del> <ins> func (c *clientCache) getClient(clusterAddress, issuer, clientID string) (*oidcAuthProvider, bool) { </ins> c.mu.RLock() defer c.mu.RUnlock() <del> client, ok := c.cache[cacheKey{issuer, clientID}] </del> <ins> client, ok := c.cache[cacheKey{clusterAddress: clusterAddress, issuerURL: issuer, clientID: clientID}] </ins> return client, ok } // setClient attempts to put the client in the cache but may return any clients // with the same keys set before. This is so there's only ever one client for a provider. <del> func (c *clientCache) setClient(issuer, clientID string, client *oidcAuthProvider) *oidcAuthProvider { </del> <ins> func (c *clientCache) setClient(clusterAddress, issuer, clientID string, client *oidcAuthProvider) *oidcAuthProvider { </ins> c.mu.Lock() defer c.mu.Unlock() <del> key := cacheKey{issuer, clientID} </del> <ins> key := cacheKey{clusterAddress: clusterAddress, issuerURL: issuer, clientID: clientID} </ins> // If another client has already initialized a client for the given provider we want // to use that client instead of the one we're trying to set. This is so all transports

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: We using go-code kubectl cli package. We are a long running app serving to many different Kubernetes clusters. Each cluster has a different user credentials and bearer token to access it. But they all use the same issuer and clientid. The oidc cache is keyed off of these two only. So what happens is when we switch to another configuration it pulls in the oidc provider for the other cluster and that bearer token does not have authority to this other cluster. The cluster config has the correct info in it but the oidc provider ignores that and used the stale data. What you expected to happen: oidc provider would always use the appropriate current config context to access the cluster How to reproduce it (as minimally and precisely as possible): In same app instance create two cluster configs to two different clusters with same oidc issuer and clientid by DIFFERENT credentials. Where the credentials on one are not authorized for the other cluster. Try to get namespaces on each, one will work and the other won't. Anything else we need to know?: Environment: Kubernetes version (use ): latest go-code from as a package called from go application.Cloud provider or hardware configuration: IBM Kubernetes OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten


Question: // // An empty slice *is* equal to a nil slice for our purposes; same for maps. // <del> // Unexported field members are not compared. </del> <ins> // Unexported field members cannot be compared and will cause an imformative panic; you must add an Equality // function for these types. </ins> func (e Equalities) DeepEqual(a1, a2 interface{}) bool { if a1 == nil || a2 == nil { return a1 == a2

This tree has a commit that demonstrates two obviously different objects that should not pass the semantic DeepEqual but do.
What shall we do with this issue for triage? Labels? Priority?
P2 for when he returns , Satnam Singh wrote:
OK I am now sufficiently worried about this. Fixes incoming.


Question: requestSizeInMegabytes   int } <del> func NewDynamicResourceConsumer(name string, replicas, initCPU, initMemory int, cpuLimit, memLimit int64, framework *Framework) *ResourceConsumer { return newResourceConsumer(name, replicas, initCPU, initMemory, dynamicConsumptionTimeInSeconds, dynamicRequestSizeInMillicores, dynamicRequestSizeInMegabytes, cpuLimit, memLimit, framework) </del> <ins> func NewDynamicResourceConsumer(name string, replicas, initCPUTotal, initMemoryTotal int, cpuLimit, memLimit int64, framework *Framework) *ResourceConsumer { return newResourceConsumer(name, replicas, initCPUTotal, initMemoryTotal, dynamicConsumptionTimeInSeconds, dynamicRequestSizeInMillicores, dynamicRequestSizeInMegabytes, cpuLimit, memLimit, framework) </ins> } <del> func NewStaticResourceConsumer(name string, replicas, initCPU, initMemory int, cpuLimit, memLimit int64, framework *Framework) *ResourceConsumer { return newResourceConsumer(name, replicas, initCPU, initMemory, staticConsumptionTimeInSeconds, initCPU/replicas, initMemory/replicas, cpuLimit, memLimit, framework) </del> <ins> func NewStaticResourceConsumer(name string, replicas, initCPUTotal, initMemoryTotal int, cpuLimit, memLimit int64, framework *Framework) *ResourceConsumer { return newResourceConsumer(name, replicas, initCPUTotal, initMemoryTotal, staticConsumptionTimeInSeconds, initCPUTotal/replicas, initMemoryTotal/replicas, cpuLimit, memLimit, framework) </ins> } /* NewResourceConsumer creates new ResourceConsumer <del> initCPU argument is in millicores initMemory argument is in megabytes </del> <ins> initCPUTotal argument is in millicores initMemoryTotal argument is in megabytes </ins> memLimit argument is in megabytes, memLimit is a maximum amount of memory that can be consumed by a single pod cpuLimit argument is in millicores, cpuLimit is a maximum amount of cpu that can be consumed by a single pod */ <del> func newResourceConsumer(name string, replicas, initCPU, initMemory, consumptionTimeInSeconds, requestSizeInMillicores, requestSizeInMegabytes int, cpuLimit, memLimit int64, framework *Framework) *ResourceConsumer { </del> <ins> func newResourceConsumer(name string, replicas, initCPUTotal, initMemoryTotal, consumptionTimeInSeconds, requestSizeInMillicores, requestSizeInMegabytes int, cpuLimit, memLimit int64, framework *Framework) *ResourceConsumer { </ins> runServiceAndRCForResourceConsumer(framework.Client, framework.Namespace.Name, name, replicas, cpuLimit, memLimit) rc := &ResourceConsumer{ name:                     name,

Is this fixed via the merged PR?
There are two issues: one is fixed by and another is fixed by , however it requires also new release of Heapster which hasn't been done yet so that I haven't closed this issue.
related PRs ,
It looks like you're on top of this, but it's been failing pretty frequently this morning as well, all on "Horizontal pod autoscaling [Skipped][Autoscaling Suite] should scale from 5 pods to 3 pods and from 3 to 1 (scale resource: Memory)"
The issue is about cluster size autoscaling tests, which haven't failed since was merged. I'll close this issue tomorrow once there will be no more failures. Also there is another issue with Horizontal Pod Autoscaling test - tracking issue
The test is stable now so closing the issue.
This problem is occurring again:
blocking merges; raising to P0
looks like was merged to hopefully address this. Lowering to p1.
Yep, disabled the test, tracking issue and there is fix in flight
Closing as is resolved.


Question: } } <del> func NewTimeoutError(key, msg string) *StorageError { return &StorageError{ Code:               ErrCodeTimeout, Key:                key, AdditionalErrorMsg: msg, } } </del> func NewInvalidObjError(key, msg string) *StorageError { return &StorageError{ Code:               ErrCodeInvalidObj,

various Oct 30 2023 6pm pst No response Looks like different test are failing with connection timeout errors: One of the tests triggered a core dump: No response /sig
are both suffers from pkg/registry and is from compoent-base, which likely same reason from `
/retitle failing-test: pkg/kubelet/eviction TestManagerWithLocalStorageCapacityIsolationOpen
/sig node
TestManagerWithLocalStorageCapacityIsolationOpen only failed in one of the PRs, so I don't think that specific test is the problem.
ok, rename as see fit. just put owning sigs please for triage.
TestStoreDeleteCollectionWithContextCancellation is the one that seems to be failing across all the failing PRs, so let's start there first. /retitle failing-test: TestStoreDeleteCollectionWithContextCancellation /sig api-machinery
triage link at shows this just starting (happening in CI jobs as well)
image used in the unit test job didn't change around the timeframe this was seen ()
looking for jumps in package unit test runtimes when this failure started happening to see if something is taking longer or stressing the system: ! - cc ! - cc ! - cc Could any of those have introduced changes that could be stressing or exhausting listener connections in unit tests?
the PR in question introduced a poll function around an API call that is unit tested with a fake client i.e. the poll is unit tested, but the fake client has reactors that return error or success right away, so in theory the poll should not be stretched to 1 retries with 1 connection attempts per fake client. while testing locally i did not notice any significant time difference after or the prior i can do some more checks to see what's going on.
merged right at the time the flakes were seen DeleteCollection invokes List, which was modified in that PR to cancel more aggressively cc
I can reproduce about a 10-20% flake rate at HEAD with: failures look like: or while testing that package with , I see ~1300-1500 open connections with this:
Let's revert it then. I looked into it and I don't know from a quick glance why it's failing and I don't have time to debug it now.

I opened
Okay, it looks like we need more debugging to open this PR
one optimization to strip some ~40 fake client connections and 2 seconds from a test:
sounds good, pointed at that commit as well
/triage accepted


Question: Release: v1.24 Testname: Jobs, apply changes to status Description: Attempt to create a running Job which MUST succeed. <del> Attempt to patch the Job status to include a new start time which MUST succeed. An annotation for the job that was patched MUST be found. Attempt to replace the job status with a new start time which MUST succeed. Attempt to read its status sub-resource which MUST succeed </del> <ins> Attempt to patch the Job status which MUST succeed. An annotation for the job that was patched MUST be found. Attempt to replace the job status with update which MUST succeed. Attempt to read its status sub-resource which MUST succeed </ins> */ framework.ConformanceIt("should apply changes to a job status", func(ctx context.Context) {

! /kind failing-test <!-- If this is a flake, please add: /kind flake --/sig apps
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign suspecting as cause
Ack, indeed, because this is the suite for alpha features enabled. We have this fix ready for the test itself: /cc
/cc It flakes in
/retitle [Flaking Test] [Conformance] [sig-apps] Job should apply changes to a job status
It used to fail with errors like After it fails with : [FAILED] patched object should have the applied condition with LastTransitionTime time.Date(2024, time.March, 7, 17, 58, 34, 0, time.UTC), got time.Date(2024, time.March, 7, 17, 58, 33, 0, time.Local) instead In [It] at: @ 03/07/24 17:58:34.109 Here I open the follow up fix: PTAL.


Question: } } <ins> func TestPluginCanDeviceMount(t *testing.T) { defer utilfeaturetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, features.CSIInlineVolume, true)() tests := []struct { name           string driverName     string spec           *volume.Spec canDeviceMount bool shouldFail     bool }{ { name:           "non device mountable inline", driverName:     "inline-driver", spec:           volume.NewSpecFromVolume(makeTestVol("test-vol", "inline-driver")), canDeviceMount: false, }, { name:           "device mountable PV", driverName:     "device-mountable-pv", spec:           volume.NewSpecFromPersistentVolume(makeTestPV("test-vol", 20, "device-mountable-pv", testVol), true), canDeviceMount: true, }, { name:           "incomplete spec", driverName:     "device-unmountable", spec:           &volume.Spec{ReadOnly: true}, canDeviceMount: false, shouldFail:     true, }, { name:           "missing spec", driverName:     "device-unmountable", canDeviceMount: false, shouldFail:     true, }, } for _, test := range tests { t.Run(test.name, func(t *testing.T) { plug, tmpDir := newTestPlugin(t, nil) defer os.RemoveAll(tmpDir) pluginCanDeviceMount, err := plug.CanDeviceMount(test.spec) if err != nil && !test.shouldFail { t.Fatalf("unexpected error in plug.CanDeviceMount: %s", err) } if pluginCanDeviceMount != test.canDeviceMount { t.Fatalf("expecting plugin.CanAttach %t got %t", test.canDeviceMount, pluginCanDeviceMount) } }) } } func TestPluginFindDeviceMountablePluginBySpec(t *testing.T) { defer utilfeaturetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, features.CSIInlineVolume, true)() tests := []struct { name           string driverName     string spec           *volume.Spec canDeviceMount bool shouldFail     bool }{ { name:           "non device mountable inline", driverName:     "inline-driver", spec:           volume.NewSpecFromVolume(makeTestVol("test-vol", "inline-driver")), canDeviceMount: false, }, { name:           "device mountable PV", driverName:     "device-mountable-pv", spec:           volume.NewSpecFromPersistentVolume(makeTestPV("test-vol", 20, "device-mountable-pv", testVol), true), canDeviceMount: true, }, { name:           "incomplete spec", driverName:     "device-unmountable", spec:           &volume.Spec{ReadOnly: true}, canDeviceMount: false, shouldFail:     true, }, { name:           "missing spec", driverName:     "device-unmountable", canDeviceMount: false, shouldFail:     true, }, } for _, test := range tests { t.Run(test.name, func(t *testing.T) { tmpDir, err := utiltesting.MkTmpdir("csi-test") if err != nil { t.Fatalf("can't create temp dir: %v", err) } defer os.RemoveAll(tmpDir) client := fakeclient.NewSimpleClientset() host := volumetest.NewFakeVolumeHost(tmpDir, client, nil) plugMgr := &volume.VolumePluginMgr{} plugMgr.InitPlugins(ProbeVolumePlugins(), nil /* prober */, host) plug, err := plugMgr.FindDeviceMountablePluginBySpec(test.spec) if err != nil && !test.shouldFail { t.Fatalf("unexpected error in plugMgr.FindDeviceMountablePluginBySpec: %s", err) } if (plug != nil) != test.canDeviceMount { t.Fatalf("expecting deviceMountablePlugin, but got nil") } }) } } </ins> func TestPluginNewBlockMapper(t *testing.T) { defer utilfeaturetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, features.CSIBlockVolume, true)()

Inline CSI volumes that are embedded in Pod specs still trigger all volume operation stages (attach, device mount, mount). Similar to plugin.CanAttach, a new plugin-wide method should be able to turn device mount off when the source is ephemeral volume. References See volume mount operation : This issue will use code from
/sig storage /assign
This will use work from


Question: } } <ins> // TrackDecodeLatency is used to track latency incurred inside the function // that takes an object returned from the underlying storage layer // (etcd) and performs decoding of the response object. // When called multiple times, the latency incurred inside to // decode func each time will be summed up. func TrackDecodeLatency(ctx context.Context, d time.Duration) { if tracker, ok := LatencyTrackersFrom(ctx); ok { tracker.DecodeTracker.TrackDuration(d) } } </ins> // AuditAnnotationsFromLatencyTrackers will inspect each latency tracker // associated with the request context and return a set of audit // annotations that can be added to the API audit entry.

It is possible to issue a list request which exceeds the duration set by the request-timeout apiserver flag. The request-timeout flag does not appear to be enforced when the apiserver is decoding the response from etcd. Specifically, listing a large number of CRDs from etcd can result in high request latency from using json encoding compared to protobuf encoding. While overall etcd latency is low, the overall listing from storage latency may be over the 1 minute timeout, particularly if the apiserver needs to fetch multiple pages from etcd. We expected the list requests to be subject to the duration specified by the request-timeout flag when this latency is coming from the listing from storage phase of the request. Create 4000 CRs and list all the objects from etcd in 2 pages of 2000 objects. If the time to list 1 page + decode takes around 50s, listing 2 pages will result in the listing from storage phase of the request taking around 1 minute and 40s before the request fails and the client is returned a 500 response code. Logs for apiserver Trace message: Audit log entry: Related issue for supporting protobuf for CRDs: <details</details<detailsAWS </details<details</details<details</details<details</details<details</details
/sig api-machinery CC:
Interesting.. - do you know where exactly we stuck? In the call to storage: we pass the context that has timeout set, so that shouldn't hang. OTOH, for the processing part: (a) I don't know if transformer honors the context - it probably depends on the implementation (b) but the decoding (appendListitem) certainly isn't honoring the context So (b) might definitely be a problem, even independently from from the transformer So I guess what I'm saying is that if listing from etcd finishes fast but the decoding is what takes most of time, then what you described can certainly be a problem. But is that the thing you're observing?
Yes, that is our understanding. Specifically, that the decoding which occurs in appendListItem doesn't respect the context: As a separate follow-up, it would be useful to include decoding time, or at least the overall listing from storage time, as a separate latency annotation, given the audit log entry is only capturing this latency as part of
+1.

We need more granular latency tracker especially on the latency spent on decoding. Right now, it only captures etcd round trip and transformer latency.
I would be supportive for both and would be happy to review it if someone opens up a PR for it.
/triage accepted
/assign
/reopen I think we need to reopen it to document how we resolved the unit test failures brought on by adding it
Reopened this issue. <detailsIn response to : /reopen I think we need to reopen it to document how we resolved the unit test failures brought on by adding it Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: if strings.HasPrefix(mountOption, vers) { versFlag = true } <ins> if strings.HasPrefix(mountOption, gid) { gidFlag = true } </ins> } allMountOptions := mountOptions

If I look at the code of azureFile plugin, it has fsGroup as an argument, but don't see its usage anywhere I also don't see anywhere explicitely stated that azureFile volume supports fsGroup.
/sig storage
For those landing here, a separate path to get the mount point to have the right permissions is to add this annotation to the PV metadata: This makes the plugin mount the volume in the Pod under those UID,GID. Obviously, this implies that one knows what those are for their workload in the container ...
If i create a PV like this, still mounted with the default root used into POD. StorageClass PV
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale cc Let's check whether works for azure file.
/sig azure
is another way to set groupid(gid) While according to this , we also need to support Here is the code about setting fsgroup from securityContext. I will enable this feature and do a test, it should be 3 lines of code. /assign
BTW, this is all for Linux(UserID, GroupID), is there an identical setting for Windows container?
No, setting user for volumes doesn't work on Windows. Only the rw/ro options are implemented.
According to my investigation, though I put the following code in azure file , it does not take effect. The reason is azure file(cifs) mount does not support gid setting after file system mount, by default would be 0(root) So in summary, set fsGroup code in azure file is useless, user could set gid by mountOptions in storage class, here is an Let me know if you hav any concern about this issue Update: Root cause is azure file(cifs) mount does not support gid setting after file system mount, while azure disk supports.
is this true, that cifs does not support guid setting?
Of course cifs support gid setting, see example: I mean cifs does not support second time gid setting, kind of differnet than azure disk.

Anyway, I am working on a PR to support if no gid setting in azure file storage class, while if both sets, only gid from storage class would be used.
while the code is quite different than just add below 3 lines of code, actually below does not work: I will show you the PR tomorrow.