Question: k8s-app: elasticsearch-logging addonmanager.kubernetes.io/mode: Reconcile rules: <del> - apiGroups: - "" resources: - "services" - "namespaces" - "endpoints" verbs: - "get" </del> <ins> - apiGroups: - "" - resources: - "services" - "namespaces" - "endpoints" - verbs: - "get" </ins> --- kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1

What happened: I download fluentd-elasticsearch in addons. The following error occurs then I use "kubectl apply -f es- " Then, i find the elasticsearch-logging pod restart lots of time, and keep status with CrashLoopBackOff. [root ~]# kubectl logs -f elasticsearch-logging-0 -n kube-system export NODENAME=elasticsearch-logging-0 NODENAME=elasticsearch-logging-0 export NODEMASTER=true NODEMASTER=true export NODEDATA=true NODEDATA=true export HTTPPORT=9200 HTTPPORT=9200 export TRANSPORTPORT=9300 TRANSPORTPORT=9300 export MINIMUMMASTERNODES=2 MINIMUMMASTERNODES=2 chown -R elasticsearch:elasticsearch /data ./bin/elasticsearchloggingdiscovery I0902 14:25:06. 9 ] Kubernetes Elasticsearch logging discovery I0902 14:25:06. 9 ] Found [] I0902 14:25:16. 9 ] Found [] I0902 14:25:26. 9 ] Found [] I0902 14:25:36. 9 ] Found [] I0902 14:25:46. 9 ] Found [] I0902 14:25:56. 9 ] Found []
unclear whether the addon is up to date and supported at this point. /sig instrumentation
cc
/assign I will take a look this week.
I have the same problem k8s version:v1.18.8
same
I've pushed a change to the stateful set that should unblock people as there was a 1.18 compatibility problem. however, please make sure you are also creating the service via , the discovery program finds the other elasticsearch nodes via service registration.
please feel free to re-open this issue if the latest version doesn't work for you.
Why was this PR to v1.20?
probably because that is the next release. most people who use this addon do so from master. Do you need this packaged in some other way?


Question: if !ok { if status, ok := event.Object.(*api.Status); ok { if status.Status == api.StatusFailure { <del> glog.Errorf("failed to watch: %v", status) </del> <ins> glog.Errorf("Failed to watch: %v", status) </ins> // Clear resource version here, as above, this won't hurt consistency, but we // should consider introspecting more carefully here. (or make the apiserver smarter) // "why not both?"

I've noticed a issue where if you have multiple replicated slave pods that don't have a HOST assigned and you reduce the number of replicas in the replicationController definition for those pods, sometimes the currently assigned (and running) pods get killed and the unassigned pod will get assigned to run in the killed pods place. The issue can be reproduced using the guestbook example provided in the repo. Steps to reproduce: (Note this is not always deterministic and may need multiple attempts) the master pod the master service the replicated slaves the Slave Controller definition to only have 1 replicas the Slave Controller with new definition The expected behavior should be that the scheduler is able to determine that unassigned pods are more viable candidates when reducing number of replicas. It seems to be more likely that since this issue is not deterministic, the ordering in which the command returns result is not always deterministically in the same order
: please can you look at this or suggested someone else? Thank you.
Unassign myself from this bug since I cannot get working on this soon. Also remove kubelet from the label since it is not kubelet issue. This is obviously a replicaController issue. cc/ Instead of destroying pod in pending state, replicaController unbound running pod from a node. Kubelet running on such node has this new desire state and kill unwanted pod to match it.
Yes, the replication controller should be smarter, taking into account Phase, Readiness, recency, ... This will be a significant problem for auto-scalers, but I expect manual scaling to be infrequent. I doubt we'll get to this soon. A PR would be welcome. Place to change is here:
Hi, I am final year Master's student of International Institute of Information Technology, Hyderabad. I am familiar with docker and containers. I have basic understanding of Go language with some hands on Docker API. This project seems interesting and a solution to this will also be a good addition to Kubernetes. So I would like to pick this as my GSoC project. I have started reading about Replication Controller and Pods and going though the code. I would appreciate if someone has any inputs for me on this. Thanks, Swapnil Patil


Question: // CustomResourceConversion describes how to convert different versions of a CR. type CustomResourceConversion struct { // strategy specifies how custom resources are converted between versions. Allowed values are: <del> // - `None`: The converter only change the apiVersion and would not touch any other field in the custom resource. // - `Webhook`: API Server will call to an external webhook to do the conversion. Additional information </del> <ins> // - `"None"`: The converter only change the apiVersion and would not touch any other field in the custom resource. // - `"Webhook"`: API Server will call to an external webhook to do the conversion. Additional information </ins> //   is needed for this option. This requires spec.preserveUnknownFields to be false, and spec.conversion.webhook to be set. Strategy ConversionStrategyType `json:"strategy" protobuf:"bytes,1,name=strategy"` <del> // webhook describes how to call the conversion webhook. Required when `strategy` is set to `Webhook`. </del> <ins> // webhook describes how to call the conversion webhook. Required when `strategy` is set to `"Webhook"`. </ins> // +optional Webhook *WebhookConversion `json:"webhook,omitempty" protobuf:"bytes,2,opt,name=webhook"` }

Currently we have inconsistency in most of the API where the field name should be used in description so the generated docs match the API fields people actually set. A linter could be for checking To avoid confusing users
/good-first-issue
/triage accepted
/assign
Hey can you provide some more information for this issue
are you still working on this?
Hey feel free to take it. I am unassigning myself
/triage needs-information /remove-good-first-issue
/sig api-machinery


Question: klet.podCache = kubecontainer.NewCache() klet.podManager = kubepod.NewBasicPodManager(kubepod.NewBasicMirrorClient(klet.kubeClient)) <del> klet.volumePluginMgr, err = NewInitializedVolumePluginMgr(klet, volumePlugins) if err != nil { return nil, err } klet.volumeManager, err = volumemanager.NewVolumeManager( enableControllerAttachDetach, hostname, klet.podManager, klet.kubeClient, klet.volumePluginMgr, klet.containerRuntime, mounter) </del> // Initialize the runtime. switch containerRuntime { case "docker":

This block of code: Appears twice in , in . This seems wrong. Most-likely, the duplicate before is the one that should be removed, because that switch statement is where is set.
This is fixed as part of the following pull: ()[]


Question: convert_v1_ServiceSpec_To_api_ServiceSpec, convert_v1_ServiceStatus_To_api_ServiceStatus, convert_v1_Service_To_api_Service, <del> convert_v1_StatusCause_To_api_StatusCause, convert_v1_StatusDetails_To_api_StatusDetails, </del> convert_v1_Status_To_api_Status, convert_v1_TCPSocketAction_To_api_TCPSocketAction, convert_v1_TypeMeta_To_api_TypeMeta,

When accessing a non existing entity (example - namespace), the message specifies the requested field as 'id', but it should be 'name'. Example (see the details section) { "kind": "Status", "apiVersion": "v1beta3", "metadata": {}, "status": "Failure", "message": "namespaces "staging" not found", "reason": "NotFound", "details": { "id": "staging", "kind": "namespaces" }, "code": 404 }
Yes. We should rename "id" to "name" in StatusDetails: Not sure if we can change v1beta3 API now. cc
You can add a new field and fill in both, and MAYBE remove ID in v1 , Nikhil Jindal wrote:
I suggest we fix it at most in v1 only. This is an annoying wart, but we could just fix the description string for now.
If trivial, we should fix v1.


Question: imagePullRetryDelay = time.Second ) <del> // NodeImageWhiteList is a list of images used in node e2e test. These images will be prepulled </del> <ins> // NodePrePullImageList is a list of images used in node e2e test. These images will be prepulled </ins> // before test running so that the image pulling won't fail in actual test. <del> var NodeImageWhiteList = sets.NewString( </del> <ins> var NodePrePullImageList = sets.NewString( </ins> imageutils.GetE2EImage(imageutils.Agnhost), "google/cadvisor:latest", "k8s.gcr.io/stress:v1",

<!-- Please only use this template for submitting enhancement requests --What would you like to be : We should come up with a better name for : Why is this needed: From /sig testing /priority important-soon /cleanup /help /good-first-issue
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : <!-- Please only use this template for submitting enhancement requests --We should come up with a better name for : From /priority important-soon /cleanup /help /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I'd like to help out with this. /assign
mind. It seems like you have a PR for it already. /unassign
And I misread the changes you made in . I'm back on this one. /assign
I'm going to rename this to . Please let me know if there are any objections or if there are things that I should be considering.
please feel free to take it. I'm only changing ImageWhileList withint
ultimately was changed to NodePrePullImageList because that's what it's actually used for, not for any sort of filtering.


Question: #    synthetic-logger-0.25lps-pod_default-synth-lgr-997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b.log #    -> #    /var/lib/docker/containers/997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b/997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b-json.log <del> # The /var/log directory on the host is mapped to the /varlog directory in the container </del> <ins> # The /var/log directory on the host is mapped to the /var/log directory in the container </ins> # running this instance of Fluentd and we end up collecting the file: <del> #   /varlog/containers/synthetic-logger-0.25lps-pod_default-synth-lgr-997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b.log </del> <ins> #   /var/log/containers/synthetic-logger-0.25lps-pod_default-synth-lgr-997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b.log </ins> # This results in the tag: <del> #  varlog.containers.synthetic-logger-0.25lps-pod_default-synth-lgr-997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b.log # The record reformer is used is discard the varlog.containers prefix and </del> <ins> #  var.log.containers.synthetic-logger-0.25lps-pod_default-synth-lgr-997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b.log # The record reformer is used is discard the var.log.containers prefix and </ins> # the Docker container ID suffix and "kubernetes." is pre-pended giving the # final tag which is ingested into Elasticsearch: #   kubernetes.synthetic-logger-0.25lps-pod_default-synth-lgr # This makes it easier for users to search for logs by pod name or by # the name of the Kubernetes container regardless of how many times the # Kubernetes pod has been restarted (resulting in a several Docker container IDs). <del> # TODO: Propagate the labels associated with a container along with its logs # so users can query logs using labels as well as or instead of the pod name # and container name. </del> <source> type tail format json time_key time <del> path /varlog/containers/*.log pos_file /varlog/gcp-containers.log.pos </del> <ins> path /var/log/containers/*.log pos_file /var/log/gcp-containers.log.pos </ins> time_format %Y-%m-%dT%H:%M:%S tag reform.* read_from_head true

When I run Kubernetes cluster on GKE, it generates a lot of system logs coming from docker and kubelet. Even if my cluster runs just a few pods, I can see dozens or hundreds of log entries per second available in Google Cloud Logging in docker and kubelet logs. This makes the actual logs from my application get lost among them, if I do not set appropriate filter. Also, I am afraid that collecting massive amounts of logs imposes unnecessary workload on fluentd pod in my cluster. It would be good if I could just disable collection of kubelet/docker logs (and potentially other Kubernetes system logs) assuming that I do not want to have these logs exported.
/cc
since you tend to have a vested interest in node debug-ability, what's the minimum you'd want to have available here? From GKE's perspective, our users typically have no interest in these logs, so turning them off by default, even without an easy option to re-enable them, makes sense. Any user sophisticated enough to want them shouldn't have problems SSH'ing to the node for them. Is the story all that different in open-source kubernetes? Does anyone even care to have the kubelet and docker logs collected from the nodes? I've never used cloud logging to debug issues with them, it's always been easier to just SSH to the node. I'd lean towards not collecting them on the nodes in general. Also cc in case he has wisdom he'd like to impart
All I can say is that personally I have never accessed kubelet etc. logs "for real" when debugging -- as Alex says -- it is usually more direct just to SSH in. I would support not collecting these logs to improve cluster stability (but perhaps have a switch / technique to make it easy to collect them if the admin does care about this?).
One last call for to complain, because I'm starting on this now
Also
We want to avoid storing logs onto local disk in the future. In-order to achieve that, we need the existing cluster level logging infrastructure to work. Can the GKE problem be solved by moving system logs to a separate namespace? , Alex Robinson wrote:
How far in the future? Unless that's intended to happen soon, collecting the kubelet and docker logs in the meantime offers little value with a nontrivial resource (cpu+memory+egress) cost.

But yes, we can handle the system logs separately if we'd like. I think the most natural way would be to report them as compute engine logs, which could be done by adding an extra config option into the gcp output plugin.
, Alex Robinson wrote:
-- just to confirm, do you think this fix will be sufficient for you? In GKE, logs from containers will be listed under Container Engine, while logs from Docker and the Kubelet will be listed under Compute Engine, with the logs for each effectively being entirely seperated in the log viewer.
Yes, this will be sufficient in terms of not polluting Container Engine logs with system logs. I am still concerned about unnecessary load on fluentd caused by exporting Docker and Kubelet logs so ideally it would be great not to export these system logs if a user is not interested in them. But this is of slightly lower priority.
Hi, What's the status from this request? Is this available in 1.7.x or higher? Thanks.


Question: fRC.mutex.Lock() defer fRC.mutex.Unlock() <ins> fRC.Calls = append(fRC.Calls, "CreateOrUpdate") </ins> if _, ok := fRC.FakeStore[routeTableName]; !ok { fRC.FakeStore[routeTableName] = make(map[string]network.Route) }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Kubernetes shouldn't put route table when it's already been configured. Azure cloud provider should check whether the route has already been created before invoking . What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /sig cloud-provider /priority important-soon
Why this is a bug? A PUT operation should be idempotent and multiple calls should get the same result?
Right. But cloud provider shouldn't make unnecessary API calls.
To check whether it is configured, we need a GET call, so we're going to replace PUT call with GET call in this case?
it's not a replace, we need to LIST routes and check whether it has already been there, then PUT if it's not.
I got your point. But if we directly send a PUT request, seems route RP will also do this comparation? Is there any issue making us have to do the comparation by ourselves instead of sending PUT request to route RP?
It would help to reduce the chances of ARM PUT throttling.
I'd like to work on this
/assign


Question: glog.Errorf("MetaNamespaceKeyFunc failed for pod %q (UID %q) with: %v", dswPodKey, dswPodUID, err) continue } <del> // retrieve the pod object from pod informer with the namespace key informerPodObj, exists, err := dswp.podInformer.GetStore().GetByKey(dswPodKey) if err != nil || informerPodObj == nil { glog.Errorf("podInformer GetByKey failed for pod %q (UID %q) with %v", dswPodKey, dswPodUID, err) </del> <ins> // Retrieve the pod object from pod informer with the namespace key informerPodObj, exists, err := dswp.podInformer.GetStore().GetByKey(dswPodKey) if err != nil { glog.Errorf( "podInformer GetByKey failed for pod %q (UID %q) with %v", dswPodKey, dswPodUID, err) </ins> continue } <ins> if exists && informerPodObj == nil { glog.Info( "podInformer GetByKey found pod, but informerPodObj is nil for pod %q (UID %q)", dswPodKey, dswPodUID) continue } </ins> if exists { informerPod, ok := informerPodObj.(*api.Pod) if !ok {

Problem: An attachable volume referenced by a pod via a PVC object (instead of directly) is not detached if the pod is deleted via namespace deletion (i.e. deleting the pod's namespace instead of the pod object directly). Repro steps: a new volume a new namespace. - PV and PVC objects: - PV/PVC objects are in bound state - pod object that references the volume via PVC - pod gets into running state without issue (i.e. volume is attached) - the namespace - for pod to terminate if volume remains attached after a few minutes Expected: Volume is detached. Actual: Volume remains attached indefinitely. <detailsis: And is: </detailsWorkarounds: the pod object directly before deleting the namespace. In step 7 above do instead If the attachable volume is referenced by a pod using a PVC object, and the pod object is deleted directly (instead of the namespace first), the volume is correctly detached. volume directly in pod (without a PVC object). If the attachable volume is referenced by a pod directly (without a PVC object), namespace deletion results in the volume being correctly detached.
Root Cause: event for pod fails on Attach/Detach controller When a volume referenced via PVC is deleted by namespace, the PV/PVC objects are deleted before the pod object, therefore when the delete event is processed by the attach/detach controller, it fails because it cannot dereference the PVC object. was to ensure volumes are still detached even if a delete event is missed (or fails), however that has a bug in it (the value of is not checked): Proposed Fixes: attach/detach controller to cache PVC/PV so even if it is deleted, it can be recovered on pod deletion events. bug in to check so that it can delete pods even if the delete event is missed (or fails).


Question: } // Drop the packets in INVALID state, which would potentially cause <del> // unexpected connection reset. // https://github.com/kubernetes/kubernetes/issues/74839 proxier.filterRules.Write( "-A", string(kubeForwardChain), "-m", "conntrack", "--ctstate", "INVALID", "-j", "DROP", ) </del> <ins> // unexpected connection reset if nf_conntrack_tcp_be_liberal is not set. // Ref: https://github.com/kubernetes/kubernetes/issues/74839 // Ref: https://github.com/kubernetes/kubernetes/issues/117924 if !proxier.conntrackTCPLiberal { proxier.filterRules.Write( "-A", string(kubeForwardChain), "-m", "conntrack", "--ctstate", "INVALID", "-j", "DROP", ) } </ins> // If the masqueradeMark has been added then we want to forward that same // traffic, this allows NodePort traffic to be forwarded even if the default

When packets with sequence number out-of-window arrived k8s node, conntrack marked them as INVALID. kube-proxy will ignore them, without rewriting DNAT. However, there is no corresponding session link on the host, and the host sends a reset packet, causing the session link to be interrupted connection not reset This problem can be solved by command iptables -t filter -I INPUT -p tcp -m conntrack --ctstate INVALID -j DROP Similar But this issue "drop" is placed on the forward chain, our scenario needs to be placed on the input chain. $ kubectl version Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2021-08-04T18:03:20Z", GoVersion:"go1.16.6", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2021-08-04T17:57:25Z", GoVersion:"go1.16.6", Compiler:"gc", Platform:"linux/amd64"} <details</details<details</details<details</details<details</details<details</details
/sig network is this about pods with hostNetwork: true ? if not, why is this a kube-proxy issue if the problem is in INPUT?
/assign
Yeah, to be clear - filter INPUT applies to packets which are going to be delivered to a process. Is that kube-proxy's job to define?
The "How can we reproduce" sounds a bit like this always happens. Here is a better instruction
IMO is the best description of the problem:
This may be a better solution than let add rules in the INPUT chain
hehe, ironically we should decide to or , one is a sysctl call the other is one (or two) iptables rules. The iptables rule has also the side effect of breaking people (cc ) If we switch to set the option, the change does not seem to be backportable. I tend to agree with Lars, remove the iptables INVALID drop rule and set the sysctl, we are already setting more host sysctls what do you think?

it will be nice to have some iptables hackers opinion too
I tried to recreate this problem on virtual en (kvm/qemu) by requsting a 100MB file with curl and do; to provoke some packet loss. The transfer took longer time, but never failed.
IMO kube-proxy should not set any sysctls that are not literally required for functionality that the user has explicitly opted into (eg for most network plugins). Kube-proxy does not own the host network namespace, and it should not be doing things that will affect other people's host-network traffic, because if we do it's going to break some users. (See also: .) We can document that we think it's a good idea for users/distros to set , and if we think it's almost always a better idea than not, we can warn at startup if it's not set, but (IMO) we shouldn't set it ourselves. If we can get the same effect with an iptables rule that only affects kube-proxy's own traffic, then I think that's better than setting a sysctl that will also affect non-kube-proxy traffic. If I understand the situation here correctly, if kube-proxy a drop rule for the invalid conntrack packets, but then the administrator set , then the result would be that conntrack would not mark some packets as invalid, and so our drop rule would just not get hit, and so our drop rule wouldn't interfere with the "better" sysctl-based solution?
Agree in general. We have done it in the past (e.g. route-localnet) and mostly it seems like a not-great idea. Also agree.
If we see this as a known bug, please consider documenting it in - even if latest Kubernetes includes a fix (we can nevertheless point that out!)
I'll fix this in the next few days.
From further googling, it seems like there have been situations in the past where people were setting that sysctl to work around bugs in conntrack, which were later fixed. I think this may be another such situation; it seems like our problem is that there is some state involving dropped or retransmitted packets which linux would cope with fine in the non-conntrack case, but which it doesn't handle in the conntrack case, unless "be_liberal" is set, and that seems like a bug.

Every time I look away from this set of issues/PRs for longer than 15 minutes I forget the exact scenario that causes the bug, but if someone could summarize it very clearly we could try dragging some kernel developers in. (The regression test in the e2e suite creates a packet with an intentionally-bad sequence number, but it's not clear to me what sort of real-world issue that's supposed to be representing.)
Thanks now that was an excellent article! Well written (not TL;DR that is), and IMO perfect technical level. If there is some collection of recommended reading for persons who want to know about K8s networking, this article should be in it.
Hi thank you for the , it's really helpful! I have some questions: , should we update this sentence accordingly ==we use the above workaround, would there any side effect for the cluster network or cluster nodes?
That says "this problem will occur if conntrack is unable to recognize a packet", but it doesn't explain the real-world scenario that causes conntrack to be unable to recognize a packet. If we go to the kernel developers and say "manually injecting random incorrect packets into a conntrack'ed connection causes it to get confused", then they'll likely respond either "don't do that then" or "use ". But if we can tell them something like "if a connection has a duplicated packet after a dropped packet, then conntrack gets confused", then that sounds a lot more like a bug in conntrack that they ought to fix, such that then everything will just work for everyone in the future without needing any DROP rules or sysctls.
So do we need to revert the drop rules? As mentioned in , it now seems that this makes sense.
There are three potential "fixes" here: this is unambiguously a conntrack bug, then we should figure out the details, and get the kernel devs to fix it. However, getting kernel bug fixes into all k8s clusters in the world takes "a long time", so even if it is a kernel bug, we should still think about the other fixes. cluster admins can use . We should advertise this better, but we feel that it would be dubious to have kube-proxy set this flag itself ().

per we could probably tweak the existing rule so that it didn't interfere with non-k8s packets ~and as suggested by the OP of this issue, we could add a similar rule to the INPUT chain (though this would need a bit of further thinking about to make sure we weren't introducing any new conflicts with non-k8s traffic)~.
We could revert the DROP rules and tell people to use the sysctl instead (and actually, probably that's a better answer for this issue than adding a new DROP rule to the INPUT chain is). But we'd need to warn users in advance if we were going to do that, since it would break existing clusters.
Totally agree, This is a better fix for this issue. If this is a kernel bug and it gets fixed someday, we can easily tell the user that you don't need to set anymore. Also, If we revert the DROP rule, which can solve the routing asymmetry issue mentioned in , which is a big fix for us Yes, We need to document it and tell the user. and In order not to break the existing cluster, Do we need a flag to decide if we need the DROP rule? I'll make it default to , and make it consistent with the current logic.
but the change to only apply the DROP rule to pod traffic should also fix that We should not require administrators to choose between "high-traffic TCP services work" and "asymmetric routing works". The current DROP rule is broken. We should either fix it to work with asymmetric routing, or else remove it and tell people who needed it to use the sysctl instead. In the latter case, we might want to make the transition via a feature gate. Then people who need asymmetric routing to work can enable the feature gate in Alpha (to make kube-proxy stop creating the DROP rule), and people who need high-traffic services to work but who didn't get the memo about using the sysctl can disable the feature gate in Beta until they have a chance to update their node configs to set the right sysctl.
Refer to , The DROP rule to pod doesn't fix our case. In our case, the pods of the external cluster subnet failed to access the pods of the cluster subnet to connect due to asymmetric routing, So the packet still matches The DROP rule to pod. Do we need a KEP for this or only a feature gate?

In general, feature gates imply KEPs (and any change that will take multiple releases to complete, where we want to make sure we don't forget about it halfway through, should get a KEP, because we have processes for tracking those). It doesn't have to be a big KEP, and some of the sections of the KEP template will be irrelevant and can be omitted, because you're not introducing a new feature. (See for example.)
We are going to break users, unfortunatly features gates does not have the effect that we are assuming, and we know that all the users have to set the "ipconntracktcpbeliberal" because eventually some connections will be broken. I do no like to set sysctl, but 'm not in favor of replacing the rule and leaving the problem to users that may not even know what a sysctl is ... agree the DROP rule is broken and breaks some esoteric but legit scenarios, but we are already setting sysctls, so I think that if we don't find any alternative to the DROP rule we should try to set the sysctl for the user by default
If we skip a KEP, make sure that the release team know to track the details (comms, docs, etc) as if it were a KEP. This is an important thing to track.
Should we at least triage this as Accepted?
/triage accepted Some alternatives DROP rule and ask users to set sysctl (I'm not in favor per my previous comment and the least surprise principle) DROP rule and we set sysctl set DROP rule if is not set the DROP rule with a flag/config so users can opt-out if don't want to set it (they can decide to use the sysctl or live with that risk) ...
I can accept KEP, but I think this may take multiple releases to complete. how about this? We remove the DROP rule Control the sysctl with a flag or a feature-gate. If it's a flag, I believe it's relatively simple and we can finish it in 1.29. In the case of feature-gate, this may take multiple releases to complete.
If sets the to true by default, we can safely remove the DROP rule
Yes, but if we were doing it that way, that would be because we considered that to be a feature rather than a bug.

We want to avoid breaking working clusters on upgrade. Maybe just having a release note saying "you may need to pass to kube-proxy" is enough. I think we agreed we don't want to change any sysctls except by explicit admin request (because that affects the behavior of non-kubernetes networking as well).
Agree, but we need to think in the users, this is a problem where the fix that solve a sublte and complex bug that may affect 100% of kubernetes users and most probably they don't know they are affected, impact a esoteric network configuration on the cluster networking, and the solution we are offering is "please read the release notes", that we know only 20% of users do ... I think that advanced users can opt-out to the DROP rule if we don't want to mess with the sysctl,
If setting was a no-brainer good idea, then the kernel would just have that behavior by default. There must be some reason why they don't, and so I worry that if we set that flag by default, then we're just going to start breaking some other subset of users. Also, if it turns out that this is all just a conntrack bug, which eventually gets fixed, then we wouldn't want to be setting the sysctl after it got fixed. But at that point it would be likely that there were some users unknowingly relying on the sysctl for some other side effect.
We could potentially have an iptables rule to count invalid conntrack packets, and expose the count as a metric
I completely and absolutely agree. we should not make that sysctl a default in kube-proxy ... what if we read the sysctl and install the DROP rule only if it is not set? Users that apply the sysctl we no longer have the DROP rule
/assign let me send a PR , I think that approach covers all cases
FWIW, my kernel conntrack informant tells me that there really aren't any bad side effects of setting it ("I don't see anything that could break by enabling this mode"), and in fact OVN sets it unconditionally on hosts it's running on. He also tells me that the bug (at least in ) is fixed in kernel ~6.2~ 6.1 ("").

So I'm not sure if that means "we should just go ahead and set the sysctl because it's harmless" or "we should just keep it like it is now and not set the sysctl since it won't be needed in the future"... (NB: if the latter we still need a docs update to go along with this PR, telling admins about the sysctl.)
If we can determine that the kernel fixes the issue in a certain version, then I think we can do this: Do not install the DROP rule.If the kernel version of the host is greater than the safe kernel version, we do not set sysctl.If less than, we have a flag to set sysctl. I'll make it default to 1. (refer to ), And let the user know that we may remove this sysctl in the future.
Exactly my thought. I made this mistake with route_localnet, and we still pay for it. Let me do some internal outreach too, see if I can get a concurring opinion?
Kernel ppl here concur: "I say : if kernel can not be fixed, set tcpbeliberal to one " "having Kubernetes set net.netfilter.nfconntracktcpbeliberal=1 always SGTM, since you want it to work on any kernel version" argues for a flag, default to true (or a hegative flag, defsult false) which seems a bit paranoid, but probably smart :) Are we all in agreement? Who wants to do the PR?
already has a PR to add to the existing set of configurable conntrack sysctls in kube-proxy () so it's just a matter of flipping the default
If the sysctl defaults to true, I think this still keeps backward compatibility, So we don't need the DROP rule anymore.
I just realized that we need this to be a flag because there are cases that run kube-proxy on environments with readonly /sys subsystem, so trying to set it always will fail will and make kube-proxy unusable for those environments
if people does not set it, they will hit the bug until they use kernel 6.1
I mean this sysctl defaults to true in kube-proxy, I can't imagine why users would need to set it to false unless the kernel version is safe (6.1). Of course, we can also document that the user that it is better not to do this.

There are cases where we CAN'T set sysctls but should not hard-fail kube-proxy. On Wed, Sep 6, 2023, 11:25 PM Cyclinder
after setting the jobs for nftables and hitting the conntrack bug continuously I think we should default the flag to true and set the sysctl nfconntracktcpbeliberal to 1 , We are already setting some sysctl and for rootless or other environment they can always opt out, this is what we do in kind
Sounds OK to me. Risk should be ~zero, right?
Yes, I assumed the risk is when to 1, we do not set it by default, so default to 0(see ). I'd be happy with it, Can I have this one?
Talked with offline, I don't have very strong arguments to make this enable by default , so let's wait for more user feedback, nftables will move to beta this release, we can revisit if we have more feedback from users
we are considering turning on by default in our kube-proxy 1.29+ deployments. Based on the recent discussion here that seems fine and the risk should "be ~zero"(), correct? And you are all even discussing turning this on by default in kube-proxy in some future version? One thing I'm a little confused on: is doing the same thing as turning on ? So by setting you're just turning on behavior that will be on by default in 6.1. Did I get that right?
Not exactly. affects a handful of situations. 6.1 only "normalized" one of them. The general intent of is something like "make conntrack work better with packets from broken TCP implementations". But the netfilter devs realized that one of the places it was being used was actually something that could happen due to totally normal TCP packet drops/retransmissions. And so in 6.1+, that particular situation is just handled correctly automatically, whether is enabled or not. My argument against this is that once we do it, we have to keep doing it forever, because people might start unwittingly depending on the other side effects of that sysctl. And particularly given that people never hit the bug anyway.

(It seems like we got more reports of the "my cluster is broken because of the DROP rule" bug than we ever got of the "my cluster is broken because of conntrack invalid packet handling" bug.) day once everyone has moved to 6.1 the bug will be gone forever ISTM that it's better to still not set it by default.
Thanks for the clarification on the 6.1 bugfix That's true and they probably will. I'm just trying to leverage some of the earlier discussion here to make a decision about the safety of flipping this to true, as it seems like the concern was not too high / basically nil.


Question: } func (o *WardleServerOptions) Complete() error { <ins> // register admission plugins banflunder.Register(o.RecommendedOptions.Admission.Plugins) // add admisison plugins to the RecommendedPluginOrder o.RecommendedOptions.Admission.RecommendedPluginOrder = append(o.RecommendedOptions.Admission.RecommendedPluginOrder, "BanFlunder") </ins> return nil } func (o *WardleServerOptions) Config() (*apiserver.Config, error) { <del> // register admission plugins banflunder.Register(o.RecommendedOptions.Admission.Plugins) </del> // TODO have a "real" external address if err := o.RecommendedOptions.SecureServing.MaybeDefaultWithSelfSignedCerts("localhost", nil, []net.IP{net.ParseIP("127.0.0.1")}); err != nil { return nil, fmt.Errorf("error creating self-signed certificates: %v", err)

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: I made an extension apiserver based on the sample. My extension apiserver adds an admission plugin, as does the sample. But if that admission plugin is enabled from the command line then validation fails. There are two problems. For the first, note that invokes validation before --- which is where the additional admission plugin gets (see how leads to which is what gets to ). I fixed that inversion by promoting some code from to in my extension server. That exposed the other problem, which comes from . The sample apiserver does not add its admission plugin to , so this check trips. What you expected to happen: I expected to be able to use my admission plugin. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
actually, we're facing the same problem in kube-apiserver. but currently we walkaround by explicitly registering all known builtin plugins beforehand.
we're already working on simplifying the launch-flow for servers (including plugin initialization), but it'll take some time. maybe we can a better solution for this issue after that.
You can see how I solved it, in --- at first those lines were in --- and in the additional statement at (with parameter of adjusted accordingly).
no, u don't have to walkaround like that. just registering all the built-in plugins on instantiating the option struct will be fine.
What in sample-apiserver corresponds to ?
the is the existing & possibly correct way to build a server, isn't it? the is just a generic demo, and it doesn't reveal how to extend or make use of advanced features like admission plugin.
sample-apiserver does aim to illustrate how to implement admission plugins. My complaint here is that the illustration is broken. In the kubernetes ecosystem there is a distinction between the main apiserver and extension apiservers. kubeapiserver is the one and only main apiserver; sample-apiserver is meant to be an example of an extension apiserver.

Extension apiservers are intended to support their own admission plugins.
Actually, I do not know why my network apiserver works; it should crash because the field is set in the wrong copy of the . Moving one fewer statement from to fixes that.
cc


Question: } fs.StringSliceVar(&a.EnablePlugins, "enable-admission-plugins", a.EnablePlugins, ""+ <del> "admission plugins that should be enabled in addition to default enabled ones. "+ </del> <ins> "admission plugins that should be enabled in addition to default enabled ones ("+ strings.Join(a.defaultEnabledPluginNames(), ", ")+"). "+ </ins> "Comma-delimited list of admission plugins: "+strings.Join(a.Plugins.Registered(), ", ")+". "+ "The order of plugins in this flag does not matter.") fs.StringSliceVar(&a.DisablePlugins, "disable-admission-plugins", a.DisablePlugins, ""+ <del> "admission plugins that should be disabled although they are in the default enabled plugins list. "+ </del> <ins> "admission plugins that should be disabled although they are in the default enabled plugins list ("+ strings.Join(a.defaultEnabledPluginNames(), ", ")+"). "+ </ins> "Comma-delimited list of admission plugins: "+strings.Join(a.Plugins.Registered(), ", ")+". "+ "The order of plugins in this flag does not matter.") fs.StringVar(&a.ConfigFile, "admission-control-config-file", a.ConfigFile,

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: With the recent change to the way admission control plugins are handled the generated documentation doesn't provide a list of plugins that are enabled: REF: What you expected to happen: I expected to be able to determine what admission plugins are enabled on the apiserver. With the default enabled plugins enumerated I can determine that by evaluating the list against the explicitly and and admission control plugins. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
Reiterating the mentions to trigger a notification: <detailsIn response to : </details
/cc
/assign


Question: cs := context.clientSet podLabel := map[string]string{"service": "securityscan"} <del> // podLabel2 := map[string]string{"security": "S1"} </del> <ins> podLabel2 := map[string]string{"security": "S1"} </ins> tests := []struct { pod       *v1.Pod

A significant number of the of are commented out, possibly was done by mistake in the past. I uncommented the tests to see if they pass, but one of them failed. What we need is to uncomment those tests, fix the one that is not passing and remove the redundant ones if there are any. /sig scheduling /help
/assign I will be moving some of the tests to unit tests and leave a cleaner set of integration tests.
/remove-help
would you like to take this? I can provide guidance.
/help /unassign
/assign
/remove-help


Question: package(default_visibility = ["//visibility:public"]) load("@io_bazel_rules_docker//docker:docker.bzl", "docker_build", "docker_bundle") <ins> load("@io_bazel_rules_docker//container:container.bzl", "container_image") </ins> load("@io_kubernetes_build//defs:build.bzl", "release_filegroup") filegroup(

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: My broke after , after much debugging we discovered that the API server could not reach itself on , further debugging showed that DNS lookups for localhost were occuring. Long story short, while we have entries for localhost, the api server was not respecting them. What you expected to happen: our standard control plane deployments should not lookup localhost via dns. they should respect /etc/hosts How to reproduce it (as minimally and precisely as possible): create a cluster and monitor network traffic. then try again after adding a minimal to the api server pod. Anything else we need to know?: Prior art: This happens because we build with go's build in DNS resolution, which respects glibc's , but we build images based on busybox for some components (like the apiserver) which does not ship with one. Environment: kubernetes using Kubernetes version (use ): /sig release
/assign cc Thanks a million to and for looking into this with me! this was tricky to resolve.
We should be able to fix this by just adding a minimal to the images make go's resolver happy. alternatively we can standardize on eg which already ships one
oh wow. tough one to debug :)
/milestone v1.13 /priority important-soon
I have a PR under way, will file after lunch!
For now adding the , but discussing with and it sounds like switching to debian-base everywhere is the best long term answer. It will actually save a mb or so on each node :sweat_smile: ... and give a more consistent base for component images to build on.
I just got bit by this.
This issue will bite anyone deploying a statically linked go binary, not just the api server. Can we get kubelet to slide in a default that is sane so it wont keep biting people? Similar to
We'd need a feature request to sig-node and probably sig-network. I do like this idea, but, is used for things other than networking so injecting one might be slightly problematic. In this case it was fine because we control the images and the binaries.

People deploying go images need to make sure they have a sane for now, or not deploy staticly linked binaries, unfortunately.
For 1.13, we should just document the problem
would it be difficult to add the file only if it doesn't exist in the image?
workaround for 1.12: echo hosts: files dns myhostname and add to the ClusterConfig:
Yeah that work around for now, I'm not sure how difficult / expensive it would be for the kubelet to conditionally add this, and it might be a surprising behavior change for arbitrary containers / pods.
it may be surprising, but its very surprising now too, that dns takes precident to hosts file unexpectedly. :( I'd rather it be secure by default and surprise them that its safe. I didn't even know glibc had this backwards behavior when was missing. I haven't run up against a glibc based system that didn't have an nsswitch file. All distro's add one.
hmm... etcd is crashing too.... I hope there is an equiv of etcdExtraVolumes....
never mind on the etcd thing that was something else. The above yaml tweaks did solve the issue.
We have just wasted a TON of time on this when a misconfigured Samsung TV on the other side of the world, plus lack of rigor or our DDNS policy broke our Kube 1.12 clusters. I really don't like that these control plane images are hitting DNS for every lookup of localhost. I am very keen to see implemented.
The control plane images in future releases already contain , which could be cherry picked back. Doing this in an automated fashion like that issue would require bringing a feature to sig-network and sig-node for discussion. I suspect supplying a file only if it doesn't exist in the image would be somewhat expensive compared to what we do currently, and it's definitely a breaking change. In the meantime, if you're containerizing go and not using CGO for networking, you need to add this to your own images. On Tue, Nov 20, 2018, 10:00 nickperry wrote:
I would argue, that 1.12 was a broken release. 1.11 and below did not have the issue.

As K8S is maturing, some of us, especially in more enterprise like environments, are not upgrading as fast, so the issues are being found slower. I think backporting the fix for the control plane images is appropriate to resolve the regression until a more universal fix can be put in place.
that sounds like a great WHODUNIT war story :)
generally I'm not upgrading production workloads particularly quickly either :-) We can file cherry picks back to 1.12, but lack of an in the control plane images is not new: EG does not contain an . IMO, setting on control plane pods was certainly broken, which I questioned, but not just for this reason, and that change was eventually reverted IIRC. /cc Regarding -- because Kubernetes wants to be stable there are processes for proposing and vetting features prior to adding them, particularly for user facing changes! If you'd like to see this, please discuss it with sig-network and sig-node. I don't personally have the bandwidth for this right now and I'm not even sure that it is even the right solution.
yes the change was reverted. it also spawned a little bit of a discussion here for a rather complicated workaround: given this affects images that are already deployed at GCR, it's unlikely that it can be cherry picked. the images do support revisions, but this would also mean porting changes to all the bundled tools that use them as hardcoded tags. also this means revisions for all patch release images. overall i see this as something that will not be approved by sig-network and the GCR maintainers.
I guess to prevent a future regression, we probably need a test that inserts a <some other addressrecord somewhere during the test.


Question: deviceMountPath: "path2", stageUnstageSet: false, }, <del> { testName:        "stage_unstage not set no vars should not fail", stageUnstageSet: false, }, </del> } for _, tc := range testCases {

/kind bug /sig storage What happened: When using a CSI driver that does not support NODESTAGE, I can see orphan json files in that are never cleaned up. What you expected to happen: No orphan file /assign


Question: runTaintManager bool, useTaintBasedEvictions bool) (*NodeController, error) { eventBroadcaster := record.NewBroadcaster() <del> recorder := eventBroadcaster.NewRecorder(scheme.Scheme, v1.EventSource{Component: "controllermanager"}) </del> <ins> recorder := eventBroadcaster.NewRecorder(scheme.Scheme, v1.EventSource{Component: "node-controller"}) </ins> eventBroadcaster.StartLogging(glog.Infof) if kubeClient != nil { glog.V(0).Infof("Sending events to api server.")

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: kubect create -f to create statefulset, and describe the statefulset, output the event likes <preEvents: FirstSeen LastSeen Count From SubObjectPath Type Reason Message --------- -------- ----- ---- ------------- -------- ------ ------- 1m 1m 1 statefulset Normal SuccessfulCreate create Claim www-web-0 Pod web-0 in StatefulSet web success 1m 1m 1 statefulset Normal SuccessfulCreate create Pod web-0 in StatefulSet web successful </preWhat you expected to happen: the From of event change to 'statefulset-controller' in output, let know the source of event more obvious, likes the event when create job and other resource(replicaset,replication). <preEvents: FirstSeen LastSeen Count From SubObjectPath Type Reason Message --------- -------- ----- ---- ------------- -------- ------ ------- 3m 3m 1 job-controller Normal SuccessfulCreate Created pod: pi-7gd2m </preHow to reproduce it (as minimally and precisely as possible): create a statefulset and describe it to see the event Anything else we need to know?: Environment: Kubernetes version (use ):kubernetes 1.7 Cloud provider or hardware configuration**: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/assign
/area controller-manager
/sig node


Question: } // bind the pod to its host asynchronously (we can do this b/c of the assumption step above). go func() { <ins> metrics.SchedulerGoroutines.WithLabelValues("binding").Inc() defer metrics.SchedulerGoroutines.WithLabelValues("binding").Dec() </ins> // Bind volumes first before Pod if !allBound { err := sched.bindVolumes(assumedPod)

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Add a metric to track number of scheduler binding goroutines. This will be a generic metric with as the only label for now. metric name Description -- -- scheduler_goroutines Number of running binding goroutines Why is this needed: Helps us track how many binding are running and monitor issues such as goroutine leak. /sig scheduling
/assign


Question: options: NewStorageClass(map[string]string{"zone": "foo"}, generateToplogySelectors(AzureDiskTopologyKey, []string{"foo"})), expErr:  true, }, <ins> { name:       "topology in regions without zones", options:    NewStorageClass(map[string]string{}, generateToplogySelectors(v1.LabelTopologyZone, []string{"0"})), expOptions: NewStorageClass(map[string]string{}, generateToplogySelectors(AzureDiskTopologyKey, []string{""})), }, { name:       "longer topology in regions without zones", options:    NewStorageClass(map[string]string{}, generateToplogySelectors(v1.LabelTopologyZone, []string{"1234"})), expOptions: NewStorageClass(map[string]string{}, generateToplogySelectors(AzureDiskTopologyKey, []string{""})), }, { name:       "topology in regions with zones", options:    NewStorageClass(map[string]string{}, generateToplogySelectors(v1.LabelTopologyZone, []string{"centralus-1"})), expOptions: NewStorageClass(map[string]string{}, generateToplogySelectors(AzureDiskTopologyKey, []string{"centralus-1"})), }, </ins> } for _, tc := range tcs {

This test fails on Kubernetes cluster in "westus" region (without any number) and with CSI migration enabled: Error: failed to provision volume with StorageClass "e2e-topology-84vk7sp": error generating accessibility requirements: topology map[] from selected node "ci-op-9j1nby77--9zlwv-worker-westus-ljw4g" is not in requisite: [map[]] StorageClass generated by the test: PVC created by the test: <details</detailsLabels of the selected node: The external-provisioner does not even call the CSI driver and reports this event: The volume should get provisioned. See above, "westus" region with no availability zones seems to be the key. It looks like the translation from in-tree StorageClass to CSI one translates topology requirements to , but the CSI one expects <details</details<details</details<detailsOpenShift 4.10 </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/priority important-soon cc
/sig storage
per https://kubernetes-, is not a zone, it's actually a fault-domain num. valid zone format is , could you change the storage class with setting, and try again? Thanks.
o, if "westus" region does not support zone, then I think you should test setting in another zone, make sure you have created a zoned node first.
I have as SC that works without CSI migration, with CSI migration it does not work. That's a regression. It should not matter if the SC is somehow odd. The labels are selected by test code, some other production code might use these labels too.
Out of curiosity, how reliable are zone numbers in Azure? I can see .. in a region with zones. If is used only in regions without a zone, then the translation can translate to , which the CSI driver expects.
I would say that's a wrong usage for zone.

following config would create a pv that could only be scheduled to a non-zone node with fault domain , if there is no node supports zone, then should not be set.
But nodes report that they are in , therefore it make sense to accept the same input in StorageClass.
if we translation "0" to "", then the pv created by that sc could be scheduled to any non zone node with any fault domain. That's a corner case that cluster is not in zone, and user still wants to set in sc
I am not sure I understand. If I provision a CSI volume in the westus cluster, its PV gets anyway: I have the cluster only with nodes in "westus" region, so all of them are without any zone. What would the CSI driver do if there were nodes in "westus" and say "ukwest", which does not have zones either? I can't test that setup easily, but I think the driver would report for both of them anyway, hence translating to does not make things worse, I think.
/sig storage /sig cloud-provider /area provider/azure
can you please check my previous comment? In what cases would be translating "0" to "" worse than the translation code we have today?
it's not worse or better, a single num () is actually not a valid zone, such in sc is actually not valid.
I mean we could translate the single digit to to make it work while the usage is wrong.
/sig storage
/sig cloud-provider


Question: func (s *NodeController) isRunningCloudProvider() bool { return s.cloud != nil && len(s.matchRE) > 0 } <ins> // canonicalizeName takes a node list and lowercases all nodes' name. func (s *NodeController) canonicalizeName(nodes *api.NodeList) *api.NodeList { for i := range nodes.Items { nodes.Items[i].Name = strings.ToLower(nodes.Items[i].Name) } return nodes } </ins>

NodeController is unable to register the value of , because it contains mixed case from the OS (in this case OSX). I'm not positive that all Linux systems are guaranteed to lowercase hostname -f, but it's likely that automated processes may end up collecting hostnames from sources that preserve case and passing that to the kubelet or master. I believe we should lowercase names detected from hostname -f and the values the node controller users - it's a minor thing, but internally we can guarantee that we treat node names as insensitive and in their canonical interior form.
cc
I'll take a look tmr.


Question: // Then the service port was just closed so the accept failure is to be expected. break } <ins> if isTooManyFDsError(err) { panic("Accept failed: " + err.Error()) } </ins> glog.Errorf("Accept failed: %v", err) continue }

Restarting might also be acceptable. It's a pretty painful user experience right now if this happens-- it fills the logs with this message.
Yes, it should just panic() , Daniel Smith wrote:
Yeah I guess if it's out of FDs it can't really open a network connection to file an event...
kube-proxy should send an event when it starts up, just like kubelet does with its method. This works around the fact that you can't send an event when you are out of FDs, and it still provides a lot of debugging value.
Agreed with 1) kube-proxy should panic when hitting out of fd 2) restarted kube-proxy should pick up the crash reason like what we do for internal agent 3) crash reason should be propogated to upstream as an event.
Moving to the right sig.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/lifecycle frozen
/remove-lifecycle stale
/area kube-proxy
we're going to close this because, after 3 years or so , decreasing interest in the userspace proxy and nobody other then some openshift folks, are using the linux kube proxy .. can reopen if necessary ping :)
/close
Closing this issue. <detailsIn response to : /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: } return false } <ins> // execSourceIPTest executes curl to access "/clientip" endpoint on target address // from given Pod to check if source ip is preserved. func execSourceIPTest(sourcePod v1.Pod, targetAddr string) (string, string) { var ( err     error stdout  string timeout = 2 * time.Minute ) framework.Logf("Waiting up to %v to get response from %s", timeout, targetAddr) cmd := fmt.Sprintf(`curl -q -s --connect-timeout 30 %s/clientip`, targetAddr) for start := time.Now(); time.Since(start) < timeout; time.Sleep(2 * time.Second) { stdout, err = framework.RunHostCmd(sourcePod.Namespace, sourcePod.Name, cmd) if err != nil { framework.Logf("got err: %v, retry until timeout", err) continue } // Need to check output because it might omit in case of error. if strings.TrimSpace(stdout) == "" { framework.Logf("got empty stdout, retry until timeout") continue } break } framework.ExpectNoError(err) // The stdout return from RunHostCmd is in this format: x.x.x.x:port or [xx:xx:xx::x]:port host, _, err := net.SplitHostPort(stdout) if err != nil { // ginkgo.Fail the test if output format is unexpected. framework.Failf("exec pod returned unexpected stdout: [%v]n", stdout) } return sourcePod.Status.PodIP, host } </ins>

What would you like to be : The test currently only works on Cloud Providers or environments that set the External IP on nodes, because it is using the framework directly to check the IPs of the pods, using a no-snat-test-proxy to check if the Pods are not using SNAT. However, this can be simplified, removing the no-snat-test-proxy Pod and executing the queries directly in a Pod. Why is this needed: The test should work in all environments, and it's failing https://k8s-,%20master&include-filter-by-regex=NoSNAT
/sig network /help wanted /failing test
/remove kind-feature /kind failing-test
/help
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove kind-feature
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
I would like to investigate this problem. /assign
I have filed , how could I verify if my patch fixes the problem?
I run them locally with KIND


Question: <del> package(default_visibility = ["//visibility:public"]) licenses(["notice"]) load( "@io_bazel_rules_go//go:def.bzl", "go_library", "go_test", ) go_library( name = "go_default_library", srcs = ["util.go"], tags = ["automanaged"], ) go_test( name = "go_default_test", srcs = ["token_test.go"], args = ["--kubeadm-path=../kubeadm"], data = ["//cmd/kubeadm"], library = ":go_default_library", tags = [ "automanaged", "integration", "skip", ], ) filegroup( name = "package-srcs", srcs = glob(["**"]), tags = ["automanaged"], visibility = ["//visibility:private"], ) filegroup( name = "all-srcs", srcs = [":package-srcs"], tags = ["automanaged"], ) </del>

go test ./cmd/kubeadm/... fails with: it appears to be following a "RunCmd" pattern used by e2e tests, but unit tests should not require built binaries
can work on this
has a long discussion as to why this is. These tests only get run when gets run, which will produce the needed binary and pass it in as arg. For more information, authored this and should be able to provide more details.
Yup, the target explicitly ignores these tests: and like said, runs this script as part of its target:
then the test should have build tags to exclude it from a regular run of the package
or up for fixing that?
I thought build tags only works on the OS level or if you it would ignore it completely? Wouldnt break the command as well?
I'd also tolerate skipping the test if the built binary doesn't exist
I think that is a great solution. I think in I had initially suggested something like that. If everyone is ok with that, I will go ahead and make that change.
Sounds good, make that change and we'll see how it turns out. I'd really want these tests start working, so +1


Question: } func TestHumanReadableFitError(t *testing.T) { <del> error := &FitError{ </del> <ins> err := &FitError{ </ins> Pod: &v1.Pod{ObjectMeta: metav1.ObjectMeta{Name: "2"}}, FailedPredicates: FailedPredicateMap{ "1": []algorithm.PredicateFailureReason{algorithmpredicates.ErrNodeUnderMemoryPressure},

There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals


Question: fs.Var(&utilflag.IPPortVar{Val: &o.config.MetricsBindAddress}, "metrics-bind-address", "The IP address with port for the metrics server to serve on (set to '0.0.0.0:10249' for all IPv4 interfaces and '[::]:10249' for all IPv6 interfaces). Set empty to disable. This parameter is ignored if a config file is specified by --config.") fs.BoolVar(&o.config.BindAddressHardFail, "bind-address-hard-fail", o.config.BindAddressHardFail, "If true kube-proxy will treat failure to bind to a port as fatal and exit") fs.Var(utilflag.PortRangeVar{Val: &o.config.PortRange}, "proxy-port-range", "Range of host ports (beginPort-endPort, single port or beginPort+offset, inclusive) that may be consumed in order to proxy service traffic. If (unspecified, 0, or 0-0) then ports will be randomly chosen.") <del> fs.Var(&o.config.Mode, "proxy-mode", "Which proxy mode to use: 'userspace' (older) or 'iptables' (faster) or 'ipvs' or 'kernelspace' (windows). If blank, use the best-available proxy (currently iptables). If the iptables proxy is selected, regardless of how, but the system's kernel or iptables versions are insufficient, this always falls back to the userspace proxy."+ </del> <ins> fs.Var(&o.config.Mode, "proxy-mode", "Which proxy mode to use: 'iptables' (Linux-only), 'ipvs' (Linux-only), 'kernelspace' (Windows-only), or 'userspace' (Linux/Windows, deprecated). The default value is 'iptables' on Linux and 'userspace' on Windows."+ </ins> "This parameter is ignored if a config file is specified by --config.") fs.Var(cliflag.NewMapStringBool(&o.config.FeatureGates), "feature-gates", "A set of key=value pairs that describe feature gates for alpha/experimental features. "+ "Options are:n"+strings.Join(utilfeature.DefaultFeatureGate.KnownFeatures(), "n")+

stipulates, that IPVS is better than IPTABLES. This is reiterated at $ kube-proxy --help … There is a contradiction: the best-available proxy currently is iptables, while IPVS has advantages over iptables.
/sig network
I am not sure what is needed by this issue. Are you looking for clarification? /triage needs-information
Is iptables the best available proxy, or is ipvs better than iptables?
iptables had performance problems that the netfilter community have fixed during the last years, also it is the default mode and the most widely used. ipvs addressed those performance issues at the beginning and still has some heavy users, but since the changes in IPVS has to be implemented in the kernel has a slower evolution is not something that you can really compare as one is better than the other, just use the one that fits you best, if no preference use the default:
Since is something subjective, the output of shall not print It shall print instead . What does the above mean? Does IPVS rely on user-space, since its kernel implementation is incomplete. Do the “changes” exist somewhere?
The best available proxy for general use is , what is subjective is the use anybody want to give to the cluster :sweat_smile: I didn't say anything about incomplete, IPVS is implemented in the kernel , the kernel doesn't backport features, if you want feature X that is implemented in 5.6 kernel and you are in kernel 3.11, you have to upgrade your kernel and reboot the node, also the kernel is a huge project, development is slower than if you have your own userspace project
Is my reading correct, that in the 5.14 kernel there is no difference between IPVS and iptables? Do the arguments from still apply:
talks about “IPVS proxier”. Is the IPVS proxier the same as running or is it a separate software?
says: The latter stipulates, that ipvs is better than iptables, while states, that iptables is the best available proxy. As far as there is ordering between iptables and ipvs modes, the documentation contains contradictions about the order.
is better in , there are more dimensions to the comparison.

Agree that the docs and comments are misleading, but I'm trying to figure out if we can help you in something or if you just want the docs to be more accurate. Is this because you are trying to get a full comparison of the proxies or do you need to choose between one or the other for something in particular?
I want the docs to be more accurate. I want to choose between iptables and ipvs . E.g. the documentation states, that iptables is the best, but it does not say how is it better than ipvs. In fact, it mentions how ipvs is better than iptables. If the comparison is subjective, then no general comparison between the the modes shall be made.
Agree, this realy makes misleading. i will try to fix it. /assign
/triage accepted There's a PR to fix the docs linked above, thanks
Agree that the documentation is confusing - "the best" is subjective. I believe to be the more stable and widely-used proxy at this point in time, which is likely where that wording came from. We should make it clear that the choice is highly dependent on the end-user's needs, but it's probably also important to make it clear that for many if not most users, "iptables" is going to be the correct option for them because it performs well enough for normal-scale clusters, and has the longest track-record. did you end up making a docs PR for this? /triage accepted
Oh man, beat me by a matter of seconds! I shouldn't have spent so long typing up that preamble :rofl:
type faster!
I already made a PR for this. PTAL


Question: newClaim := obj.(*resource.ResourceClaim) oldClaim := old.(*resource.ResourceClaim) newClaim.Spec = oldClaim.Spec <ins> metav1.ResetObjectMetaForStatus(&newClaim.ObjectMeta, &oldClaim.ObjectMeta) </ins> } func (resourceclaimStatusStrategy) ValidateUpdate(ctx context.Context, obj, old runtime.Object) field.ErrorList {

The 's was copied from some other, broken types. A call to is missing and therefore, for example, finalizers can be changed during a status update. That is not supposed to be possible. Object meta changes need to be removed during status updates. n/a No response Since 1.26 till at least 1.29. n/a No response No response No response No response
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
/assign


Question: <polygon style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" points="222.862,466.265 218.126,456.137 223.059,458.767 228.122,456.399 "/> </g> <g> <del> <line style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" x1="308.892" y1="503.5" x2="362.289" y2="504.346"/> <polygon style="fill: #000000" points="369.788,504.465 359.71,509.306 362.289,504.346 359.869,499.307 "/> <polygon style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" points="369.788,504.465 359.71,509.306 362.289,504.346 359.869,499.307 "/> </g> <g> </del> <line style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" x1="318.054" y1="544.587" x2="410.664" y2="606.112"/> <polygon style="fill: #000000" points="320.821,540.422 309.725,539.053 315.288,548.752 "/> <polygon style="fill: none; fill-opacity:0; stroke-width: 2; stroke: #000000" points="320.821,540.422 309.725,539.053 315.288,548.752 "/>

CA Advisor runs inside docker containers, but I don't believe it's correct to show the Kubelet and Proxy processes as inside the docker box on the minion. Also, shouldn't the internet-firewall-really load balance across minion proxies instead of be connected to just one SPOF minion, treating that connection more like a logical connection across all minion proxies?
This is the desired state, which is admittedly not the current state. Yeah, this is a good point. k8s currently relies on its environment for load balancing. Maybe or can speak to the desired state here.
We should just review the diagram and make it correctly reflect the state of 1.0.


Question: volumeSpec: volumeSpec, // volume.volumeSpecName is actually InnerVolumeSpecName. It will not be used // for volume cleanup. <del> // in case pod is added back to desired state, outerVolumeSpecName will be updated from dsw information. // See issue #103143 and its fix for details. </del> <ins> // in case reconciler calls mountOrAttachVolumes, outerVolumeSpecName will // be updated from dsw information in ASW.MarkVolumeAsMounted(). </ins> outerVolumeSpecName: volume.volumeSpecName, pod:                 pod, deviceMounter:       deviceMounter,

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --We have even encountered such case that pod continuously reports “Unable to attach or mount volumes for pod;...” when kubelet restarted. The reasons are listed in details as below: , as result of apiserver instantaneous unstablility, dswp get pvc‘s pv spec failed, then the pvc volume does not be to dsw. , syncStates() lists all the volume directories locally, and recontructs the volume from local volume, if the volume isn't in dsw, it will put the recontructed volume to asw, but now the volume.OuterVolumeSpecName isn't correct. chance, at this time, In , the fail to pvc volume is to dsw again normally in reconcile() -mountAttachVolumes(), it compares dsw and asw by volume's innerVolumename, it's all the same. but it could not sense the OuterVolumeSpecName dismatch. so it will not re-mount again to update asw cache , it checks pod's volume attach or mount success by comparing volume.OuterVolumeSpecName and , but this pv'c OuterVolumeSpecName isn't correct in asw. so pod continuously reports “Unable to attach or mount volumes for pod;...” reconcile() could sense the reconstruct volume's dismatch to the dsw, then re-mount again to make volume cache consistency Kubernetes version (use ): 1.19.4 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig node
/triage accepted
/assign
CC
I think this is a point that can be optimized. /cc
I will give a fix for this
Thanks for reporting the issue, this innerSpecName and outerSpecName are confusing and cause issues. Please ping me for review when you have a fix. Thanks!
Please don't triage your own issues immediately, give someone else a chance to review them :( /sig storage
okay, I know


Question: fs.StringVar(&o.config.ResourceContainer, "resource-container", o.config.ResourceContainer, "Absolute name of the resource-only container to create and run the Kube-proxy in (Default: /kube-proxy).") fs.MarkDeprecated("resource-container", "This feature will be removed in a later release.") fs.StringVar(&o.config.ClientConnection.KubeConfigFile, "kubeconfig", o.config.ClientConnection.KubeConfigFile, "Path to kubeconfig file with authorization information (the master location is set by the master flag).") <del> fs.Var(componentconfig.PortRangeVar{Val: &o.config.PortRange}, "proxy-port-range", "Range of host ports (beginPort-endPort, inclusive) that may be consumed in order to proxy service traffic. If unspecified (0-0) then ports will be randomly chosen.") </del> <ins> fs.Var(componentconfig.PortRangeVar{Val: &o.config.PortRange}, "proxy-port-range", "Range of host ports (beginPort-endPort, single port or beginPort+offset, inclusive) that may be consumed in order to proxy service traffic. If (unspecified, 0, or 0-0) then ports will be randomly chosen.") </ins> fs.StringVar(&o.config.HostnameOverride, "hostname-override", o.config.HostnameOverride, "If non-empty, will use this string as identification instead of the actual hostname.") fs.Var(&o.config.Mode, "proxy-mode", "Which proxy mode to use: 'userspace' (older) or 'iptables' (faster) or 'ipvs' (experimental). If blank, use the best-available proxy (currently iptables).  If the iptables proxy is selected, regardless of how, but the system's kernel or iptables versions are insufficient, this always falls back to the userspace proxy.") fs.Int32Var(o.config.IPTables.MasqueradeBit, "iptables-masquerade-bit", utilpointer.Int32PtrDerefOr(o.config.IPTables.MasqueradeBit, 14), "If using the pure iptables proxy, the bit of the fwmark space to mark packets requiring SNAT with.  Must be within the range [0, 31].")

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: Implement this TODO: What you expected to happen: Start kube-proxy with a 'beginPort+offset' format port range configuration. This is needed when I want to allocate specific numbers of port from a host without extra manual calculation. Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig api-machinery
/sig network Does anyone besides kube-proxy use this?
/assign


Question: })) framework.ExpectNoError(err, "Timeout while waiting to confirm PV %q deletion", retrievedPV.Name) }) <ins> ginkgo.It("should apply changes to a pv/pvc status", func(ctx context.Context) { pvClient := c.CoreV1().PersistentVolumes() pvcClient := c.CoreV1().PersistentVolumeClaims(ns) ginkgo.By("Creating initial PV and PVC") pvHostPathConfig := e2epv.PersistentVolumeConfig{ NamePrefix:       ns + "-", Labels:           volLabel, StorageClassName: ns, PVSource: v1.PersistentVolumeSource{ CSI: &v1.CSIPersistentVolumeSource{ Driver:       "e2e-driver-" + string(uuid.NewUUID()), VolumeHandle: "e2e-status-conformance", }, }, } numPVs, numPVCs := 1, 1 pvols, claims, err = e2epv.CreatePVsPVCs(ctx, numPVs, numPVCs, c, f.Timeouts, ns, pvHostPathConfig, pvcConfig) framework.ExpectNoError(err, "Failed to create the requested storage resources") ginkgo.By(fmt.Sprintf("Listing all PVs with the labelSelector: %q", volLabel.AsSelector().String())) pvList, err := pvClient.List(ctx, metav1.ListOptions{LabelSelector: volLabel.AsSelector().String()}) framework.ExpectNoError(err, "Failed to list PVs with the labelSelector: %q", volLabel.AsSelector().String()) gomega.Expect(pvList.Items).To(gomega.HaveLen(1)) initialPV := pvList.Items[0] ginkgo.By(fmt.Sprintf("Listing PVCs in namespace %q", ns)) pvcList, err := pvcClient.List(ctx, metav1.ListOptions{}) framework.ExpectNoError(err, "Failed to list PVCs with the labelSelector: %q", volLabel.AsSelector().String()) gomega.Expect(pvcList.Items).To(gomega.HaveLen(1)) initialPVC := pvcList.Items[0] ginkgo.By(fmt.Sprintf("Reading %q Status", initialPVC.Name)) pvcResource := schema.GroupVersionResource{Group: "", Version: "v1", Resource: "persistentvolumeclaims"} pvcUnstructured, err := f.DynamicClient.Resource(pvcResource).Namespace(ns).Get(ctx, initialPVC.Name, metav1.GetOptions{}, "status") framework.ExpectNoError(err, "Failed to fetch the status of PVC %s in namespace %s", initialPVC.Name, ns) retrievedPVC := &v1.PersistentVolumeClaim{} err = runtime.DefaultUnstructuredConverter.FromUnstructured(pvcUnstructured.UnstructuredContent(), &retrievedPVC) framework.ExpectNoError(err, "Failed to retrieve %q status.", initialPV.Name) gomega.Expect(string(retrievedPVC.Status.Phase)).To(gomega.Equal("Pending"), "Checking that the PVC status has been read") ginkgo.By(fmt.Sprintf("Reading %q Status", initialPV.Name)) pvResource := schema.GroupVersionResource{Group: "", Version: "v1", Resource: "persistentvolumes"} pvUnstructured, err := f.DynamicClient.Resource(pvResource).Get(ctx, initialPV.Name, metav1.GetOptions{}, "status") framework.ExpectNoError(err, "Failed to fetch the status of PV %s in namespace %s", initialPV.Name, ns) retrievedPV := &v1.PersistentVolume{} err = runtime.DefaultUnstructuredConverter.FromUnstructured(pvUnstructured.UnstructuredContent(), &retrievedPV) framework.ExpectNoError(err, "Failed to retrieve %q status.", initialPV.Name) gomega.Expect(string(retrievedPV.Status.Phase)).To(gomega.Or(gomega.Equal("Available"), gomega.Equal("Bound"), gomega.Equal("Pending")), "Checking that the PV status has been read") ginkgo.By(fmt.Sprintf("Patching %q Status", initialPVC.Name)) payload := []byte(`{"status":{"conditions":[{"type":"StatusPatched","status":"True", "reason":"E2E patchedStatus", "message":"Set from e2e test"}]}}`) patchedPVC, err := pvcClient.Patch(ctx, initialPVC.Name, types.MergePatchType, payload, metav1.PatchOptions{}, "status") framework.ExpectNoError(err, "Failed to patch status.") gomega.Expect(patchedPVC.Status.Conditions).To(gstruct.MatchElements(conditionType, gstruct.IgnoreExtras, gstruct.Elements{ "StatusPatched": gstruct.MatchFields(gstruct.IgnoreExtras, gstruct.Fields{ "Message": gomega.ContainSubstring("Set from e2e test"), "Reason":  gomega.ContainSubstring("E2E patchedStatus"), }), }), "Checking that patched status has been applied") ginkgo.By(fmt.Sprintf("Patching %q Status", retrievedPV.Name)) payload = []byte(`{"status":{"message": "StatusPatched", "reason": "E2E patchStatus"}}`) patchedPV, err := pvClient.Patch(ctx, retrievedPV.Name, types.MergePatchType, payload, metav1.PatchOptions{}, "status") framework.ExpectNoError(err, "Failed to patch %q status.", retrievedPV.Name) gomega.Expect(patchedPV.Status.Reason).To(gomega.Equal("E2E patchStatus"), "Checking that patched status has been applied") gomega.Expect(patchedPV.Status.Message).To(gomega.Equal("StatusPatched"), "Checking that patched status has been applied") ginkgo.By(fmt.Sprintf("Updating %q Status", patchedPVC.Name)) var statusToUpdate, updatedPVC *v1.PersistentVolumeClaim err = retry.RetryOnConflict(retry.DefaultRetry, func() error { statusToUpdate, err = pvcClient.Get(ctx, patchedPVC.Name, metav1.GetOptions{}) framework.ExpectNoError(err, "Unable to retrieve pvc %s", patchedPVC.Name) statusToUpdate.Status.Conditions = append(statusToUpdate.Status.Conditions, v1.PersistentVolumeClaimCondition{ Type:    "StatusUpdated", Status:  "True", Reason:  "E2E updateStatus", Message: "Set from e2e test", }) updatedPVC, err = pvcClient.UpdateStatus(ctx, statusToUpdate, metav1.UpdateOptions{}) return err }) framework.ExpectNoError(err, "Failed to update status.") gomega.Expect(updatedPVC.Status.Conditions).To(gstruct.MatchElements(conditionType, gstruct.IgnoreExtras, gstruct.Elements{ "StatusUpdated": gstruct.MatchFields(gstruct.IgnoreExtras, gstruct.Fields{ "Message": gomega.ContainSubstring("Set from e2e test"), "Reason":  gomega.ContainSubstring("E2E updateStatus"), }), }), "Checking that updated status has been applied") ginkgo.By(fmt.Sprintf("Updating %q Status", patchedPV.Name)) var pvToUpdate, updatedPV *v1.PersistentVolume err = retry.RetryOnConflict(retry.DefaultRetry, func() error { pvToUpdate, err = pvClient.Get(ctx, patchedPV.Name, metav1.GetOptions{}) framework.ExpectNoError(err, "Unable to retrieve pv %s", patchedPV.Name) pvToUpdate.Status.Reason = "E2E updateStatus" pvToUpdate.Status.Message = "StatusUpdated" updatedPV, err = pvClient.UpdateStatus(ctx, pvToUpdate, metav1.UpdateOptions{}) return err }) framework.ExpectNoError(err, "Failed to update status.") gomega.Expect(updatedPV.Status.Reason).To(gomega.Equal("E2E updateStatus"), "Checking that updated status has been applied") gomega.Expect(updatedPV.Status.Message).To(gomega.Equal("StatusUpdated"), "Checking that updated status has been applied") }) </ins> }) // testsuites/multivolume tests can now run with windows nodes

[x] test approval issue : [x] test pr : [x] Flake [x] two weeks soak end date : 31 Oct 2023 [X] test promotion pr : [X] remove endpoint from According to following APIsnoop query, there are three PersistentVolume and three PersistentVolumeClaim endpoints that are untested. Using a number of existing e2e test practices a new has been created to provide future Conformance coverage for the 6 endpoints. The e2e logs for this test are listed below. This query shows the following PV and PVC endpoints are hit within a short period of running this e2e test. If a test with these calls gets merged, test coverage will go up by 6 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted /cc
/reopen For tracking the rest of the steps to Promotion
Reopened this issue. <detailsIn response to : /reopen For tracking the rest of the steps to Promotion Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: function upgrade-master() { echo "== Upgrading master to '${SERVER_BINARY_TAR_URL}'. Do not interrupt, deleting master instance. ==" <del> detect-master </del> get-password <del> set-master-htpasswd </del> <ins> get-bearer-token detect-master </ins> # Delete the master instance. Note that the master-pd is created # with auto-delete=no, so it should not be deleted.

See issue as the roll-up for node upgrades. tl;dr: Update MIG template, then call a MIG rollingupdate. More details: When upgrading nodes, an in-place upgrade must worry about leftover state on the boot disk, VM configuration like IP tables, and the idempotency of salt. Getting a fresh VM instead would remove these concerns and provide a clean state for a new node. Furthermore, a versioned node spec () includes the tuple of (kubelet, kube-proxy, docker, kernel/OS). In the general case, changing docker or the kernel/OS means we have to stop the running pods. Once know we have to stop the pods, the simplest upgrade mechanism is to totally down the node and provision a new one. Provisioning a new node is also a great backup to have for node repairs. To be clear, though, this issue is entirely separate from node repairs; once we have the capability to grow the cluster by adding new nodes, this mechanism, and other repair mechanisms, will build on top of that capability. This was blocked by: nodes joining a running cluster (template-only metadata) References: The most relevant discussion regarding the worries of complications with state while doing in-place upgrades is .
FWIW, I think we need this as a fallback to repair completely borked nodes? That is, it may not be necessary for upgrades per se, but could end up being independently useful.
Thanks for the comment, I updated the description to be clear about what this issue is addressing. Having the ability to add new nodes to a cluster will both enable this upgrade path as well as give us a great "hammer" to use for repairs.
Working on this now.


Question: return false } <del> if c.spec.PersistentVolume == nil { klog.V(4).Info(log("mounter.SetupAt Warning: skipping fsGroup permission change, no access mode available. The volume may only be accessible to root users.")) return false } if c.spec.PersistentVolume.Spec.AccessModes == nil { klog.V(4).Info(log("mounter.SetupAt WARNING: skipping fsGroup, access modes not provided")) return false } if !hasReadWriteOnce(c.spec.PersistentVolume.Spec.AccessModes) { klog.V(4).Info(log("mounter.SetupAt WARNING: skipping fsGroup, only support ReadWriteOnce access mode")) return false </del> <ins> if c.spec.PersistentVolume != nil { if c.spec.PersistentVolume.Spec.AccessModes == nil { klog.V(4).Info(log("mounter.SetupAt WARNING: skipping fsGroup, access modes not provided")) return false } if !hasReadWriteOnce(c.spec.PersistentVolume.Spec.AccessModes) { klog.V(4).Info(log("mounter.SetupAt WARNING: skipping fsGroup, only support ReadWriteOnce access mode")) return false } return true } else if c.spec.Volume != nil && c.spec.Volume.CSI != nil { if !utilfeature.DefaultFeatureGate.Enabled(features.CSIInlineVolume) { klog.V(4).Info(log("mounter.SetupAt WARNING: skipping fsGroup, CSIInlineVolume feature required")) return false } // Inline CSI volumes are always mounted with RWO AccessMode by SetUpAt return true </ins> } <del> return true </del> <ins> klog.V(4).Info(log("mounter.SetupAt WARNING: skipping fsGroup, unsupported volume type")) return false </ins> } // isDirMounted returns the !notMounted result from IsLikelyNotMountPoint check

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: If gid is configured in Pod's securityContext.fsGroup, csi inline will panic. log: E0320 16:50:59. ] Operation for """ ("-ae1b-4b5c-be91-")" failed. No retries permitted until 2020-03-20 16:51:31. +0800 CST m=+. (durationBeforeRetry 32s). Error: "recovered from panic "runtime error: invalid memory address or nil pointer dereference".

(err=<nil) Call stack:ngoroutine [running]:(0xc001497e70)n +0xb5npanic(0x3b713e0, 0x7bd2050)n +0x1b5(csiMountMgr).applyFSGroup(0xc0005372c0, 0xc001a2f100, 0x4, 0xc00205ed70, 0x44, 0x0)n +0x64(csiMountMgr).SetUpAt(0xc0005372c0, 0xc000b6cb80, 0x78, 0xc00205ed70, 0x0, 0x0, 0x0)n +0x1180(csiMountMgr).SetUp(0xc0005372c0, 0xc00205ed70, 0x0, 0x0, 0x0)n +0x4d(operationGenerator).GenerateMountVolumeFunc.func1(0x8, 0x453f9f8, 0xc001497e70, 0xc0025ba750)n +0x4a4(GeneratedOperations).Run(0xc0009742a0, 0x452f628, 0xc000d56480, 0xc00135a050, 0x45)n +0xaf(nestedPendingOperations).Run.func1(0xc000d56480, 0xc00135a050, 0x45, 0xc001106d50, 0x24, 0xc0009742a0, 0x0, 0x0, 0x0, 0x0)n +0xcbncreated by (*nestedPendingOperations).Runn +0x378n" code: If using inline volume, c.spec.PersistentVolume will be nil What you expected to happen: fsGroup is successfully applied Environment: Kubernetes version (use ): 1.16.3 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g.

): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig sig/storage
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig sig/storage Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
ran into this bug recently on v1.18, and I don't see anything that would fix this issue on later releases either. /reopen /lifecycle freeze /assign
Reopened this issue. <detailsIn response to : ran into this bug recently on v1.18, and I don't see anything that would fix this issue on later releases either. /reopen /lifecycle freeze /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.

</details
/triage accepted
/lifecycle frozen
I tested this scenario with the hostpath driver on k8s 1.24 (built from master) on a local VM using a slightly modified version of the examples/csi-app- spec ( fsGroup and fsType): And this does not panic kubelet It does not panic because of this check: Which was in However, I do believe we can and should support fsgroup for CSI inline volumes. Instead of checking the PV spec (which is not set for inline volumes) we could pass the directly into , which will be RWO for inline volumes.
Sorry, it was not obvious that the panic was already fixed when I first looked at this. Re-purposing this bug to address the issue that fsgroup still does not get applied for inline volumes. /retitle CSI inline volumes do not apply fsGroup
THX, also link to this
/retitle CSI inline volumes should support fsGroup


Question: # exit on any error set -e <del> SCRIPT_DIR=$(cd $(dirname $0); pwd) </del> <ins> SCRIPT_DIR=$(CDPATH="" cd $(dirname $0); pwd) </ins> source $SCRIPT_DIR/config.sh

What happened: If an in-tree volume is provisioned with in-tree provisioner (without migration enabled) and later migration gets enabled, the PV controller is expected to coordinate with the CSI external provisioner to make sure it does get deleted. For above to happen, this in the PV controller needs to return nil, nil even if is set to a plugin prefixed with . The CSI external provisioner also needs to be enhanced to delete PV's whose is set to a plugin prefixed with . We also need to prevent a race between external provisioner and in-tree PV controller in the delete path when CSI Migration flag is disabled. To achieve this synchronization, PV controller will need to mark the in-tree PVs that need to be deleted by external provisioner with an additional annotation to signal that CSI Migration flag is enabled in-tree and the CSI external provisioner needs to process the deletion. What you expected to happen: CSI driver should process the deletion. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): v1.15 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/assign
/sig storage
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Instead of this would it be possible to just change the annotation on the . That is the signal. Otherwise we run the risk of adding yet another condition to the already complex logic. I wonder if its possible to have the PV controller go in and switch them all on when it detects is on.
cc
This is a high priority to be fixed and should be cherry-picked as well if possible
I'd like to raise the priority for this one as well, especially after the explicit end-user ask for migration to CSI (for AWS EBS and GCE PD, at least) in the official 1.17.0 release notes.

IMO it's not possible to migrate because deletion of PVs created before CSI migration does not work due to this bug. See also:
/open no current released versions of Kubernetes or External-provisioner where this is fixed. Likely to be and
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: utilfeature "k8s.io/apiserver/pkg/util/feature" "k8s.io/client-go/kubernetes" "k8s.io/client-go/transport" <ins> "k8s.io/component-base/tracing" </ins> "k8s.io/component-base/version" v1 "k8s.io/kube-aggregator/pkg/apis/apiregistration/v1" v1helper "k8s.io/kube-aggregator/pkg/apis/apiregistration/v1/helper"

Extension API servers do not currently receive the header when requests are proxied through kube-aggregator, so spans are not linked properly. Traces from the extension API sever should be linked to the parent in kube-aggregator. tracing for kube-apiserver tracing for sample-apiserver and aggregate it to kube-apiserver a request to kube-apiserver for one of the resources from sample-apiserver sample-apiserver traces will not be linked to the kube-apiserver traces No response <detailsThis problem currently exists on . </details<details</details<details</details<details</details<details</details<details</details
/sig api-machinery
/cc /triage accepted