Question: Long:    editLong, Example: fmt.Sprintf(editExample), Run: func(cmd *cobra.Command, args []string) { <ins> args = append([]string{"configmap"}, args...) </ins> err := RunEdit(f, out, errOut, cmd, args, options) cmdutil.CheckErr(err) }, ValidArgs:  validArgs, ArgAliases: argAliases, } <ins> addEditFlags(cmd, options) cmd.AddCommand(NewCmdEditConfigMap(f, out, errOut)) return cmd } func addEditFlags(cmd *cobra.Command, options *resource.FilenameOptions) { </ins> usage := "to use to edit the resource" cmdutil.AddFilenameOptionFlags(cmd, options, usage) cmdutil.AddValidateFlags(cmd)

Kubernetes version : What happened: When I try to display configmap content with yaml output (kubectl get configmap webapp-inifile -o yaml), the output is sometimes pretty, and sometimes not -depending newlines are effectively displayed or not - just n char- I can't find what's the difference between some file and some others. (limit on numer of lines ?) Configmap is created with --from-file option Ex : What you expected to happen: More human readable display, in order to be able to easily change parameter when kubectl edit congifmap How to reproduce it :
Hrm, this seems to me like the right answer is to special case 'kubectl edit' for configmaps, rather than try to hack around with the YAML decoder. I suppose we could use block quotes (e.g. ) if we see newlines but that seems error prone...
Closing this, pretty print is there, kubectl configMap edit support is there.
This still happens on kube 1.7 :-( I'm not 100% sure how to reproduce, but what I did was: have an existing configmap and edit a few lines exit the editor ... now I see the characters instead of actual newlines. even shows the characters
I'm also seeing this issue in kube 1.7. During the past few days I have edited a configmap ~110 times, and seen the formatting get destroyed (spaces messed up, characters visible) twice. I can't determine anything specific that I did in those two edits that would have triggered the formatting messup.
I'm still seeing this issue too. Pushed a lot of identical configmaps, some ones are displaying good, and some other bad. For example, I have a configmap with 3 files in ; they all are identical except one variable ; 2 of them display normally, and the last one displays with n ...
also seeing this issue on 1.7
can we reopen this issue? or open a new one?
I created a brand new configmap using and when I viewed the yaml with it's full of n characters and escape characters. This makes it very difficult to edit.
I am experiencing the same. n character is messing up a configmap I'm creating with --from-file option. Anyone found a reason why this is happening ?

Also failing in 1.8.4
I am also facing the same issue of formatting .. I have a application configuration of 400-500 lines. Tryied many times. For fist 40-50 lines the format is good ans easily readable.. However, when i try to create config for entire configuration file its very difficult to edit the config map..
Same issue here. No idea what makes one configmap show better than another.
This issue corrupted my aws-auth configMap that authenticates my user to access the EKS cluster and while trying to add other users to the configMap the formatting corrupted the configMap file and now I am locked out of my EKS cluster. Version: 1.10.3
I don't know if this will help somebody else but I was facing the same issue with escaped data in a config map edit and get (the config map had a single .conf file), and it seems that I had a unneeded extra space at the end of a single line in my data. When I removed that extra space the pretty printing was maintained in edit and get.
11 Still same
still same ( there is an example if install helm install stable/prometheus , you will find cm/prometheus-server )
I agree with - I can recreate it every time If you were to execute the following (ensuring no trailing spaces), the configmap remains "pretty" However, repeat the above (or edit the existing one with ), and put a trailing space after one of the key/value pairs, and that then messes up the configmap contents
Can we reopen this issue? I need this feature to change the yaml in cm, and I have been struggling with the intent and n for several days ...
Please reopen this ticket, it's incredibly frustrating to have to deal with this even on small configmaps
Got this also. I had to remove tabs from my config file to get the yaml multiline display.
Just created a messed up configmap on kuberntes 1.12.2.
Likewise.
OMG thank you!
Just ran into this in EKS (1.10.3) -- highly annoying. I have one config map that will come out OK if I use create directly, but if I pipe the output of to apply then it gets munged. Does seem like this is something that needs addressing.

Have the same issue and fixed by removing tabs
Have the same issue, don't have any tabs in file. This issue totally should be reopened.
This issue is valid and I am not sure why it was closed
Not only spaces. Trail symbols too. I uses vim for it
"n n" (spaces between end of lines) in one of my files also messed up the pretty display.
Created a new issue when I hit this again and was able to repro using the trailing whitespace
I've removed all the trailing whitespaces and tabs. The problem is gone.
Check any possible special chars. It solve my problem some time ago
I have seen this issue in K8 1.7 -1.13 kubectl create configmap NAMESPACE --from-file=$PATCHFILE --dry-run=true --output=yaml where PATCHFILE is attached file below. I renamed it from .json to .log so this webpage would let me include it.
Solve the issue by delete whitespace at each end of line.
The output of is ugly. Wether using json or yaml, k8s injects a whole bunch of escape characters and messes up the formatting of the original config file. It would be very nice to have a switch that tells it to dump the config out "normally"!
I a new line in a configMap using Tab for identation. After changing to Spaces instead of Tab, I was able to see the configmap as expected...
Same here, white space/s at the end of line ruin the output, everything back to pretty when white space/s removed.
This worked for me.
Since I've still met this on 1.16, I'll point a bullet list of the regular culprits for the configmap messing: spaces before newline:Tabs:Special characters like ãẽõ, áéó, öß will also mess your files. Look up for them in your comments, they might pass through!
An easy one liner to remove trailing whitespaces in a ConfigMap:
Please reopen. This issue still remains. This format makes eks aws-auth configmap very hard to maintain. Once the file is applied with trailing whitespaces the whole thing gets lots of n and . As a temporary measurement is there a way to reformat the configmap in the "readable" way? after the n are created?

if you use way didn't reformat the file for me.
Nurza's method works. But you may have a space before EOF as well: Or you may have CR's in there , in which case you should remove the CRs. Or you may have a TAB, which you should replace with one or more spaces.
No CR/TAB/new lines. I have a "clean" file that after I apply it all works great. (display as expected) After I add one trailing white space to it, it gets those n. I tried to run Nurza's method but it didn't work. Any other Ideas why Nurza's method is not working for me ? This a part of the output I'm getting with
has a space after it, although you don't see it. Because the LF is on the next line, there must be a space before it.
Correct but if I run Nurza's method the n are still there. Am I still missing anything? This is what I run
Nobody said that sed expression was perfect. Once you know this is your doing -- and you know how to fix it -- you're not bothered by this behaviour anymore.
This is scary shit :hankey: ... what is the industry practice to edit the file like these: aws-auth...? I managed to type 20+lines by hand :disappointed: and it worked on 3rd attempt :disappointed_relieved:
Cant' say about the 'like this', but aws-auth you can use eksctl to do the hard work for you:
If the formatting of cm goes wierd a simple hack to get it back to normal is : kubectl get cm configmapname -o yaml cm.yamlNow copy the contents of file and past it on is powerful tool to check the linting of yaml files. This will provide you with the configmap as expected with correct formatting. Paste the output in another yaml file (for e.g - )kubectl apply -f
Anyone knows why the issue is closed? It's not fixed as far as I can tell.

If you just want the raw output as it was read in when created --from-file, you can use jq to get the raw string (without escaped newlines etc) If you created a configmap from a file like this: Get the data:
Edited a cm, came back in, and had n all over. Something about white space in my previous edit? Not great overall :/ On kubectl version Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.5", GitCommit:"", GitTreeState:"clean", BuildDate:"2020-07-04T15:01:15Z", GoVersion:"go1.14.4", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.6", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-11-13T11:11:50Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}
Awesome, very good website~
Thank You very much
I was able to fix this behavior by: 1) Don't use tabs, convert to spaces 2) To remove spaces before a newline character, use this: It will also convert CRLF to LF only, by removing .
Something small that exacerbates this issue is that in python's standard lib module , up to some version (I think python 3.4, see ), pretty printing adds a trailing space after every comma by default, unless you supply a list of separators yourself that omit the spaces (e.g. like ) So if you have old-ish glue (e.g. python 2.7) that parses a k8s configmap into a file for archival purposes, and you made that file readable by pretty-printing it, using that file to regenerate a new configmap results in this gross escape behaviour.
Hello There, One of our colleague has edited configmap with wrong data, now whole cluster inaccessible with any of role, need help on immediate basis and much appreciated. Error: through eksctl - Error: getting auth ConfigMap: Unauthorized through kubectl error: You must be logged in to the server (Unauthorized)
yaml marshallers will write a string to a multiline string e.g.

UNLESS string contains a character string terminates in a space character string contains a substring which matches the regex
I am in the same situation. Were you able to solve this?
Also reproduce in v1.22.3 :(
I occasionally encountered this problem in v1.15.6 or v1.21.2, but eventually I was able to deal with it by using edit and then using vim's own set list to back-propagate the original static configmap code configuration under gitlab to see if there were extra spaces, and then modifying the local configmap file based on the online configmap's set list display under edit. The state of the online configmap under edit, to modify the local configmap file, is ultimately able to deal with this problem, but if there is a better official way to do it would be good.
yes if you are working on linux
i would like to share an observation that helped me sort out the line merging and n. it appears to be related to the last line of the file you are using (--from-) if the last line ends with empty space or tab. the config map is merged up in one line with n as line feed if you remove the empty space from the last line, the ConfigMap will import file as is with +
I got the same error after editing the ConfigMap file. I used an online tool to remove the trailing white space that we cannot see in the text. After copying the content which removed trailing whitespaces I was able to resolve this issue FYI - (online tool which I used)
in v1.24 also i can see same issue. how we can edit cm with many lines.
Finally, I found that the absence of a value defined or the value is empty string in will cause the issue. data: {{ }} is not defined in my yaml just use quote to the value resolved
delete space before n or after n it works for me
did anyone find solution when we edit cm how we can see in proper format as for me really difficult to edit cm as we have many n char and I can still see this issue in k8s version 1.28


Question: in case that it needs that for operations like error injection or checking calls. <del> ## Cluster setup </del> <ins> ## Cluster setup preparation </ins> <del> The container runtime must support CDI. The latest cri-o releases contain support, containerd 1.6.x does not. To bring up a kind cluster with containerd built from their main branch, use: </del> <ins> The container runtime must support CDI. CRI-O supports CDI starting from release 1.23, Containerd supports CDI starting from release 1.7. To bring up a Kind cluster with Containerd, two things are needed: - [build binaries from Kubernetes source code tree](https://github.com/kubernetes/community/blob/master/contributors/devel/development.md#building-kubernetes) - [Kind](https://github.com/kubernetes-sigs/kind) </ins> <del> - [Bash version requirement](https://github.com/kubernetes/community/blob/master/contributors/devel/development.md#bash-version-requirement) </del> <ins> > NB: Kind switched to use worker-node base image with Containerd 1.7 by default starting from release 0.20, build kind from latest main branch sources or use Kind release binary 0.20 or later. </ins> <del> - Build node image </del> <ins> ### Build kind node image After building Kubernetes, in Kubernetes source code tree biuld new node image: </ins> ```bash <del> $ test/e2e/dra/kind-build-image.sh dra/node:latest </del> <ins> $ kind build node-image --image dra/node:latest $(pwd) </ins> ``` <del> - Bring up a kind cluster </del> <ins> ## Bring up a Kind cluster </ins> ```bash $ kind create cluster --config test/e2e/dra/kind.yaml --image dra/node:latest ```

uses containerd for worker nodes by default, and latest release of uses by default base-image with containerd 1.6 for worker node. Unfortunately, CDI was merged into containerd 1.7, and CI at the moment uses binary from unreleased build 0.20-alpha. So running e2e tests locally for is failing unless when building node image a parameter is used with base-image that has 1.7 containerd. The need needs to be mentioned in DRA e2e Dynamic Resource Allocation e2e work locally when running them according to the instruction. Run DRA e2e with kind 0.19. The fix is coming shortly. 27, 1.28-alpha N/A N/A N/A N/A N/A
/sig node /assign
/triage accepted
I recommend testing kubernetes @ HEAD with KIND @ HEAD more generally. Nearly all of Kubernetes CI does this, we can't guarantee that a previously released binary has fixes to handle unreleased kubernetes breaking changes so periodically this is required regardless of this specific issue.
Alright, that recommendation was not noticed by me unfortunately. And thinking that there might be more like me, let's at least document it in the DRA e2e tests README.


Question: "uniqueItems": true }, { <del> "description": "Redirect the standard error stream of the pod for this call. Defaults to true.", </del> <ins> "description": "Redirect the standard error stream of the pod for this call.", </ins> "in": "query", "name": "stderr", "type": "boolean",

As per the PRs kubernetes/kubernetes kubernetes/kubernetes, the defaulting behaviour of the pod exec parameters had been changed however in the code there is a comment where it is mentioned it defaults to True The comment about the default value must be removed and the openapi spec must be fixed Present in the file as comment and also in the OpenApiSpec swagger If this issue seems appropriate can I take it up please <details</details<details</details<details</details<details</details<details</details<details</details
/sig docs
/sig node
/assign
/assign
/help /triage accepted
This request has been marked as needing help from a contributor. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help /triage accepted </details
/kind documentation


Question: verbs:          "get", nonResourceURL: "/logs/,/healthz", expectedClusterRole: &rbac.ClusterRole{ <ins> TypeMeta: v1.TypeMeta{APIVersion: "rbac.authorization.k8s.io/v1", Kind: "ClusterRole"}, </ins> ObjectMeta: v1.ObjectMeta{ Name: clusterRoleName, },


Question: // Querying a non-existent dummy disk on the datastore folder. // It would fail and return an folder ID in the error message. _, err := dc.GetVirtualDiskPage83Data(ctx, dummyDiskVolPath) <ins> canonicalVolumePath, err = getPathFromFileNotFound(err) </ins> if err != nil { <del> re := regexp.MustCompile("File (.*?) was not found") match := re.FindStringSubmatch(err.Error()) canonicalVolumePath = match[1] </del> <ins> return "", fmt.Errorf("failed to get path from dummy request: %v", err) </ins> } } diskPath := vclib.GetPathFromVMDiskPath(canonicalVolumePath)

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: Provsioning a PVC using a storage class with vSphere as the cloud provider failed with error "runtime error: index out of range". What you expected to happen: The PVC should have been created. How to reproduce it (as minimally and precisely as possible): Create a PVC using a storage class with vSphere as the cloud provider and a VMware vCenter Server where the default language is not English. Anything else we need to know?: K8s in conjunction with vSphere PVC relies on English error messages from the VMware API in order to determine the canonical path for a newly created volume (VMDK file). The faulting code (v1.11.3, still the same on master) can be found here (starting at line 312): The line "canonicalVolumePath = match[1]" (340) in the above code is the problem because "File (.*?) was not found" (English) never matches for a different language (e.g. German: "Datei [DATASTORE] kubevols/kube- wurde nicht gefunden") and a “runtime error: index out of range” oocurs in the "kube-controller-manager". This causes the already created volume (VMDK file) not to be reported back to the K8s system which in turn will try to recreate it repeatedly in a loop (fails with "Cannot complete the operation because the file or folder [DATASTORE] /kubevols/kubernetes-dynamic-pvc-[UUID] already exists"). Find below the corresponding log excerpt (kube-controller-manager): Environment: Kubernetes version (use ):Cloud provider or hardware configuration: vSphere OS (e.g. from /etc/os-release): n/a Kernel (e.g. ): n/a Install tools: n/a Others: n/a <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
/assign


Question: if err := os.MkdirAll(kl.getPluginsDir(), 0750); err != nil { return fmt.Errorf("error creating plugins directory: %v", err) } <ins> if err := os.MkdirAll(kl.getPluginsRegistrationDir(), 0750); err != nil { return fmt.Errorf("error creating plugins registry directory: %v", err) } </ins> if err := os.MkdirAll(kl.getPodResourcesDir(), 0750); err != nil { return fmt.Errorf("error creating podresources directory: %v", err) }

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug This is more of an enhancement. What happened: The Plugin Watcher is setup to trigger on file activities when they occur in the subdirectory that it scans. During startup, the Watcher is passed (from within the Kubelet) the Kubelet's root plugin directory as a location to scan for newly plugins - Because the kubelet's plugins subdirectory is also used by existing in-tree drivers (including csi) as a location to store plugins artifacts, the Plugin Watcher is getting triggered by directory/file activities occurring in that subdirectory that has nothing to do with plugin registration as shown in the output below: What you expected to happen: The Plugin Watcher should not be getting triggered by file/directory activities not related to plugin registration/unregistration. One possibility is to have the Watcher scan one directory below plugins like or something similar How to reproduce it (as minimally and precisely as possible): Run plugin watcher with a CSI driver. cc
/sig node
I think you should only place the plugin socket file in the plugin registration directory but not the plugin data.
agree with we can only place the registration file in the specified path instead of placing all data into it. On the other hand, you can place data into the /var/lib/kubelet/plugins/registration even if registration created, that's a no-end loop.
As I mentioned in issue, existing in-tree volume plugins were already using that directory as a location to store storage operation artifacts (not just csi). Here is a shortened list of driver locations that use /var/lib/kubelet/plugins in their code already: And several more locations. The fix would be to have plugin watcher scan a subdirectory of plugins (say ) instead of plugins directly as it is doing today. The watcher functions as intended. Except now it picks up all directory activities whether related or not to plugin registration.
Well given the fact that the pluginwatcher is beta this is a bit of an issue. My understanding of the "beta" guarantees is that we want to be backwards compatible for all plugins published with the beta API. Meaning that even if we move the registration directory to a new location, we'll need to keep scanning the old directory (to pick up v1beta plugins).

From a 10 000 feet point of view it seems easier to me if we move the in-tree volume plugin's data. Though it might be a lot of work. Another less painful option, but a bit grey is the following: Enforce plugins to end with this suffix is recommended for device plugins (though not enforced) and I believe is a convention on linux. This is a bit grey because we would still be breaking v1beta users but since we're moving a convention from "recommended" to "enforced" it seems "less" breaking to me (and I haven't seen device plugins not using that extension). What do you think ?
I like the suffix idea. It can be a recommendation (which csi is already doing).
Hmm, I've just deployed a new k8s cluster on 1.12.0 and noticed this repeating in the logs on a few of our pods. Just for my own sanity, is this something I should be worried about or can I ignore it until a fix is released?
you can ignore these errors. if it's a recommendation then we'll keep spamming the logs with all the data files. Another thing to keep in mind, it seems like CSI plugins puts a lot of data (and creates a lot of directories). It might be worth considering limiting the recursion to one level. I'm concerned we might exhaust the amount of fds by tracking the plugins data.
yeah CSI keeps track of volume operation metadata in the plugins dir. This probably will not change. I understand this is beta. However, besides CSI and Device plugin, are there any other implementers out there for this feature ? Would it be possible to propose a new location (i.e. <pluginsDir/reg/?
let me know if you open a PR for this as discussed.
thanks for open the issue with detail logs, i can see two problems by looking at logs and code. 1) plugin watcher trying to dial non-socket files. 2) too many logs generated (CREATE and DELETE). I a suggestion to fix of this issue we can fix those problems by: 1) check file mode when handle CREATE event, only attempt to dial file when file mode is socket.

2) increase log verbose level for both 'handleDeleteEvent()' and 'handleCreateEvent()' plugin watcher still has activity when any file create/delete under the directory, but they become light weighted (CREATE file will only trigger file mode check, DELETE file event will trigger plugin searching) i am not sure we want to go further to limit directory depth where plugin can live. it has two drawbacks IMO: 1) it is hard to enforce, not sure everyone will read code/readme. 2) it provides less flexibility to plugins: if want to organize many plugin socket files with directory.
These are definitely good suggestions! It seems like there might be some confusion as to why we are thinking of doing this: Most plugins use the plugin directory to store data (actually a lot of files / dirs) Watching all the directories and files created inside the plugin directory puts more burden on the Kubelet in terms of cpu / memory consumption Watching all the directories and files created inside the plugin directory quickly raises the number of fd consumed By limiting the search of the plugins to one directory, we would be severely limiting the ability of the module to unintentionally harm the overall Kubelet. It seems actually pretty easy to enforce, as it's a simple condition. As for not everyone reading code / readme, we can add the documentation to the gRPC interface if this is a concern. In terms of flexibility, is there a real need to create a plugin in a lower directory? Is there a plugin that has so many sockets that a single directory underneath the plugin directory isn't enough? Naively it seems to me that is enough for the different plugins we enable.
ping on the above :)
Naively it seems to me that $ is enough for the different plugins we enable. i can not speak from plugin developer, but both below looks reasonable to me , or as long as we can communicate to plugin developers clearly and effectively (gRPC interface documentation is needed), the directory depth restriction is reasonable to me.
I like . However, why not put plugintype one level up as . For instance
Is the suggestion to hard code the number of levels that the recursion algorithm will search to 3? Or just expect a socket only at a hard coded level (e.g. socket must be nested 3 deep)? Copying my comment from for consistency: Another option I can think of: recursively scanning but before scanning a directory check if it is mounted ().

This will be backwards compatible and will prevent scanning of volume data Thoughts?
i like this idea better if it is always the case that: volume data is mounted, we can modify the watcher code to not watch the mounted directory in the first place.
Skipping mounted dir along with restructuring search path to known (shallow) dir sounds like a good plan. Though with the new suggested paths, sockets will be placed at a location where mounts should not exist.
I think that would be a safe assumption. The plugin directory is only used as a target path for global mounts.
Ephemeral volume types may not need to mount, it could just be a symlink or plain directory. Can we assume any ephemeral volume CSI driver won't implement global mount?
Yes, in-tree ephemeral volumes today don't do a global mount so the mounts directory is which is not the plugin dir. So no issue there. For CSI the will be the global mount path (this plugin dir). Emphermal volumes shouldn't implement call -- but nothing prevents them from doing so. Maybe let's start with just checking for mount and ignoring mount directories. If it is still an issue we can consider adding a configurable black list of paths. Thoughts?
That sounds good. Also, putting the socket files in a specific location that does not clash with mounts will also help (i.e. )
/milestone v1.13 /priority important-soon
I just want to point out that one of the issue we have with recursively watching all directories is that it will significantly increase the amount of fd consumed. Given that plugins actually store data there and in a dynamic way my fear is that it will exhaust the available fds for no real good reasons. Limiting the depth sounds like a reasonable solution to that issue.
btw, because of this verbosity in my cluster writes 5G logs every day and filled up my rootfs (which is small). Is there any way to decrease this verbosity?
We can do both. Don't search mounted dirs, and max depth of search. Who is working on this?
That's pretty bad, can you open another issue to track that.
we have a PR in flight discussing this issue (recursive scanning) here: Thinking more about this, the fact that too much logs is generated is clearly linked to this specific issue and how we process plugins.

In any case I'll be fixing that issues too
how confident are we in getting this fixed by Code freeze tomorrow? 5pm PST
I expect things to wind up today, changes in the above PR aren't very contentious and don't involve significant changes


Question: testSvcLabels := map[string]string{"test": "response"} framework.Logf("Creating pod...") <del> _, err := f.ClientSet.CoreV1().Pods(ns).Create(context.TODO(), &v1.Pod{ </del> <ins> pod := &v1.Pod{ </ins> ObjectMeta: metav1.ObjectMeta{ <del> Name: "agnhost", </del> <ins> Name:      "agnhost", Namespace: ns, </ins> Labels: map[string]string{ "test": "response"}, },

pull-kubernetes-e2e-kind-ipv6 - Not sure. Time out on waiting Pod to be started. No response /sig network
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: }) fakeClient.AddReactor("update", "horizontalpodautoscalers", func(action core.Action) (handled bool, ret runtime.Object, err error) { <del> tc.Lock() defer tc.Unlock() obj := action.(core.UpdateAction).GetObject().(*autoscalingv1.HorizontalPodAutoscaler) assert.Equal(t, namespace, obj.Namespace, "the HPA namespace should be as expected") assert.Equal(t, hpaName, obj.Name, "the HPA name should be as expected") assert.Equal(t, tc.expectedDesiredReplicas, obj.Status.DesiredReplicas, "the desired replica count reported in the object status should be as expected") if tc.verifyCPUCurrent { if assert.NotNil(t, obj.Status.CurrentCPUUtilizationPercentage, "the reported CPU utilization percentage should be non-nil") { assert.Equal(t, tc.CPUCurrent, *obj.Status.CurrentCPUUtilizationPercentage, "the report CPU utilization percentage should be as expected") </del> <ins> handled, obj, err := func() (handled bool, ret *autoscalingv1.HorizontalPodAutoscaler, err error) { tc.Lock() defer tc.Unlock() obj := action.(core.UpdateAction).GetObject().(*autoscalingv1.HorizontalPodAutoscaler) assert.Equal(t, namespace, obj.Namespace, "the HPA namespace should be as expected") assert.Equal(t, hpaName, obj.Name, "the HPA name should be as expected") assert.Equal(t, tc.expectedDesiredReplicas, obj.Status.DesiredReplicas, "the desired replica count reported in the object status should be as expected") if tc.verifyCPUCurrent { if assert.NotNil(t, obj.Status.CurrentCPUUtilizationPercentage, "the reported CPU utilization percentage should be non-nil") { assert.Equal(t, tc.CPUCurrent, *obj.Status.CurrentCPUUtilizationPercentage, "the report CPU utilization percentage should be as expected") } </ins> } <ins> var actualConditions []autoscalingv1.HorizontalPodAutoscalerCondition if err := json.Unmarshal([]byte(obj.ObjectMeta.Annotations[autoscaling.HorizontalPodAutoscalerConditionsAnnotation]), &actualConditions); err != nil { return true, nil, err } // TODO: it's ok not to sort these becaues statusOk // contains all the conditions, so we'll never be appending. // Default to statusOk when missing any specific conditions if tc.expectedConditions == nil { tc.expectedConditions = statusOkWithOverrides() } // clear the message so that we can easily compare for i := range actualConditions { actualConditions[i].Message = "" actualConditions[i].LastTransitionTime = metav1.Time{} } assert.Equal(t, tc.expectedConditions, actualConditions, "the status conditions should have been as expected") tc.statusUpdated = true // Every time we reconcile HPA object we are updating status. return true, obj, nil }() if obj != nil { tc.processed <- obj.Name </ins> } <del> var actualConditions []autoscalingv1.HorizontalPodAutoscalerCondition if err := json.Unmarshal([]byte(obj.ObjectMeta.Annotations[autoscaling.HorizontalPodAutoscalerConditionsAnnotation]), &actualConditions); err != nil { return true, nil, err } // TODO: it's ok not to sort these becaues statusOk // contains all the conditions, so we'll never be appending. // Default to statusOk when missing any specific conditions if tc.expectedConditions == nil { tc.expectedConditions = statusOkWithOverrides() } // clear the message so that we can easily compare for i := range actualConditions { actualConditions[i].Message = "" actualConditions[i].LastTransitionTime = metav1.Time{} } assert.Equal(t, tc.expectedConditions, actualConditions, "the status conditions should have been as expected") tc.statusUpdated = true // Every time we reconcile HPA object we are updating status. tc.processed <- obj.Name return true, obj, nil </del> <ins> return handled, obj, err </ins> }) fakeScaleClient := &scalefake.FakeScaleClient{}

What happened: The unit test for Horizontal Pod Autoscaler frequently deadlocked on the OpenShift CI system. What you expected to happen: The unit test should be robust and succeed without having to win a race condition to avoid deadlock. How to reproduce it (as minimally and precisely as possible): I'm not sure. Anything else we need to know?: Here are some logs of the problem happening: https://openshift-gce- The deadlocks seem to happen when the chan fills up while waiting for events. If that happens, there's a race where immediately after an event is read but before it can acquire the lock to verify the test, 2 event reactors are able to acquire the lock and write to the chan. The second will block forever since the chan is full and there's nobody to read from it. I've opened a PR with the fix. See
/sig autoscaling
/assign


Question: return name, volumeOptions.CapacityGB, labels, fstype, nil } <ins> // getCandidateZones finds possible zones that a volume can be created in func getCandidateZones(cloud *aws.Cloud, selectedNode *v1.Node) (sets.String, error) { if selectedNode != nil { // For topology aware volume provisioning, node is already selected so we use the zone from // selected node directly instead of candidate zones. // We can assume the information is always available as node controller shall maintain it. return sets.NewString(), nil } // For non-topology-aware volumes (those that binds immediately), we fall back to original logic to query // cloud provider for possible zones return cloud.GetCandidateZonesForDynamicVolume() } </ins> // returns volumeOptions for EBS based on storageclass parameters and node configuration func populateVolumeOptions(pluginName, pvcName string, capacityGB resource.Quantity, tags map[string]string, storageParams map[string]string, node *v1.Node, allowedTopologies []v1.TopologySelectorTerm, zonesWithNodes sets.String) (*aws.VolumeOptions, error) { requestGiB, err := volumehelpers.RoundUpToGiBInt(capacityGB)

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: We have pretty large clusters in aws and in production, we found an inefficiency in provisioning dynamic EBS volumes, where aws-ebs volume provisioner tries to look for available zone from aws directly. With scale, this can trigger massive from ec2, and affect cluster stability. This is also related to What you expected to happen: To simply get zones with available instances, we can get information from API server for efficiency. There might be a delay between node gets shutting down and such change is reflected in api server, but in case node is not ready or gets terminated, there will be higher level reconciliation triggered so we are safe. How to reproduce it (as minimally and precisely as possible): We only expect this to be hit when the cluster is huge (or aws account is huge) so not easy to reproduce. /assign /sig aws /sig cloud-provider
/assign


Question: } func TestFindInstanceByNodeNameExcludesTerminatedInstances(t *testing.T) { <ins> awsStates := []struct { id       int64 state    string expected bool }{ {0, ec2.InstanceStateNamePending, true}, {16, ec2.InstanceStateNameRunning, true}, {32, ec2.InstanceStateNameShuttingDown, true}, {48, ec2.InstanceStateNameTerminated, false}, {64, ec2.InstanceStateNameStopping, true}, {80, ec2.InstanceStateNameStopped, true}, } </ins> awsServices := newMockedFakeAWSServices(TestClusterId) nodeName := types.NodeName("my-dns.internal")

If a node is attached to a cloud provider and the machine is in any state other than running/available/ready/whatever cloud provider uses to indicate ready, it will be removed from the kubernetes cluster. Looking at the various cloud providers, some of them filter nodes based on the state whereas others do not. The node controller has logic for handling a node that is not updating status and evicting pods/tainting the node as needed. Is there a reason why the cloud providers only return instances/vms that are "ready" instead of returning any instance/vm that matches the requested name? It seems like it might be more consistent to have the cloud providers only filter on name and allow the caller to determine what to do if the found instance isn't in a desired state.
If that's really the case it's unfortunate - we probably can fix logic to list everything (I guess there's a flag that you can pass to CP to give you everything, not only working machines). I have no idea who deals with that ? OTOH I think the only problem with this would be if the machine would flap, as this would be annoying and cluster autoscaler might get crazy ( If it's really unready/unavailable/whatever then it's fair to assume that nothing's working on it (if StatefulSet can live with that assumption
do you think the NodeController should/should not delete a Node from storage in this situation: Node is configured to use a cloud provider, VM is powered off but not deleted? I would argue for "should not delete".
A trivial answer is that PrivateDNSNames are not unique on AWS. A running instance and a stopped instance could have the same IP, hence same PrivateDNSName. Rob / Marek: Can you clarify with a scenario where something unexpected happens: what that is and what you expect to happen?
can we work around that somehow for ec2?
I don't know what we are working around! The concern is AWS instances that are Stopped but not Terminated? Separately, we've been working since before 1.0 to get us off PrivateDNSName on AWS though...
the issue is that, depending on cloud provider, the NodeController may delete a Node from etcd if the VM is not Running.

If you power off a VM, wait longer than the NodeController's eviction threshold (5 minutes by default), and then power the VM back on later, the NodeController will have deleted the Node from etcd in the interim. Some cloud providers in the kube code base expect an instance to be Running, and if it's not, they don't include that instance in the list of instances, or a "get instance" call returns "not found" (I'm not sure which, has the full details), and so the NodeController doesn't see the powered off VM and deletes the Node.
That is the current behaviour, but you've skipped the bit where something goes wrong :-) Even if we fix this problem though do we close this issue? I would vote to close this issue now on the basis that there's no objective measure for when we can mark this issue resolved, and that we start again with a problem statement like the one you're describing, which feels actionable.
the intent of this issue was what I described. It's possible that it wasn't clear in the initial description.
At least on AWS: The motivation for the filtering is that Terminated instances still have the old PrivateDNSName listed (IIRC), and so the names aren't unique unless you exclude Terminated. But we don't have to exclude Stopped instances (power-off), and the nodes will just get marked NotReady. I don't think that will cause problems, but it may be that the correct place to fix is actually somewhere else - e.g. on kubelet rejoin. What goes wrong when the node tries to rejoin?
At least 1 thing: if you have manually labels to the node post-registration, those labels are gone when it rejoins.
That sounds like it would be correct for some classes of labels.
- small clarification: 5 min threshold has nothing to do here. It only says that if Node is unresponsive for 5 or more minutes we'll start evicting Pods from that Node (modulo rate limiting). Deleting non-existing Nodes is a separate thing - when Node skips few heartbeats (currently 4) NC checks if cloud provider still returns the instance corresponding to that Node, and if not then delete the Node instance. IIUC what wrote the current behavior is valid for AWS, as Node won't come back (it's DNS name might be used by something else).

This isn't the case for only aws. Looking at the code it appears at least openstack, aws, vsphere, ovirt, rackspace cloud providers also filter on available node. I wasn't able to verify whether azure or others were doing the filtering. If they weren't, then at best we have inconsistency with different cloud providers. IMO we shouldn't be deleting a node from kubernetes if it is just stopped or some state other than available. If the instance exists in the cloud provider then imo it should exist in kubernetes, but possibly marked unscheduable if needed.
ouch, that's even worse! Isn't that only 40-50 seconds? and I chatted yesterday and I think there are a couple of possible ways to look at this: 'm used to doing things the "old way" where I treat my servers as pets, and I want to be the one who decides when to retire/delete a server. 'm willing to treat servers as cattle, and if I ever need to take a server down so I can upgrade the kernel/packages/etc, or if a server goes offline for whatever reason, I'll just throw it away and replace it with a newer one. I'd be willing to bet that most IT organizations are still operating under 1). Justin recommended that we need to do a better job at education, and while I agree with that, there is still going to be a transitional period where sysadmins want to be making these kinds of decisions, and they're not yet willing to let the cluster automatically delete nodes. In summary, I think at the very least we need to: this behavior, if it's not already documented an option to the node controller to disable this behavior (and possibly disable it by default) more on evangelizing how we envision server management to happen in a Kubernetes world
Yes, I believe it's 40s. I'm curious why we don't want to just change the implementation of cloud provider, so that we'll use 'full list' (i.e. one including not running Nodes, so only delete ones that are really not present) in NC?
That also works for me, assuming it doesn't cause any issues.
It's strictly better than disabling it altogether, right?

The drawback would be that if you create a VM which will get the same name (I guess it's a DNS name), than the suspended/whatever one, then it'll be treated by k8s as the old one, which may cause a issues when running StatefulSets (especially if old machine didn't release PV or sth, as IIUC k8s will believe that it's attached to the new machine But if you're willing to do that, then I don't see anything that will prevent you from doing so, assuming it will be flag gated.
I think stateful sets are unaffected by this unless both new and old machines are booted at the same time AND either kubelet fails to terminate the pods. I can't think of how that would happen. Not being able to unmount a volume could be caused by other admin actions and is something we can warn about.
For a change of this significance, we'll want an e2e showing the problem, and then a PR that lets the e2e pass.
I'm not sure an e2e is the best way to show this. An e2e would incur costs to run on a cloud provider so likely wouldn't be part of any standard e2e run. I think a unit test could be done though.
The PR I pushed only changes AWS. w/o this change the node is removed from kubernetes shortly after it is no longer running. With this PR the node transitions to unschedulable and details show it hasn't reported its health check.
PTAL and let me know if you see issues with this approach
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/lifecycle frozen
/remove-lifecycle stale


Question: func TestValidate(t *testing.T) { tests := map[string]struct { <del> ranges []extensions.GroupIDRange groups []int64 pass   bool </del> <ins> ranges        []extensions.GroupIDRange groups        []int64 expectedError string </ins> }{ "nil security context": { ranges: []extensions.GroupIDRange{ {Min: 1, Max: 3}, }, <ins> expectedError: "unable to validate empty groups against required ranges", </ins> }, "empty groups": { ranges: []extensions.GroupIDRange{ {Min: 1, Max: 3}, }, <ins> expectedError: "unable to validate empty groups against required ranges", </ins> }, "not in range": { groups: []int64{5},

Is this a BUG REPORT or FEATURE REQUEST?: This is request for improvements. What happened: When PodSecurityPolicy provider validates or fields of a pod, it shows an error like "999 is not an allowed group" if the group doesn't belong to the range(s). What you expected to happen: It should include to the error message also the allowed values -- the ranges against what we were validating this group. This will simplify user experience and debugging. How to reproduce it (as minimally and precisely as possible): run kubernetes cluster with PSP support create a PSP that allows fsGroup/SupplementalGroups in some ranges grant to a user access to this PSP try to create a pod that has fsGroup/SupplementalGroups outside of the allowed ranges Anything else we need to know?: In order to improve the current behavior, we need to modify function here: Reported-by: /kind help
This looks like a good first issue! My suggestion would be something like . I would also suggest to merge overlaying ranges in the log message, like as , to keep the log as concise as possible. I will look into it.
Maybe something like ? Thank you!
We can use the same approach that strategy uses:
Sure, that is probably the most consistent solution.
/remove-help


Question: } return client } <ins> func getIP(hostPort string) net.IP { conn, err := net.Dial("udp", hostPort) if err != nil { log.Fatal(err) } defer conn.Close() localAddr := conn.LocalAddr().(*net.UDPAddr) return localAddr.IP } </ins>

Which jobs are failing: Which test(s) are failing: Since when has it been failing: Testgrid link: Reason for failure: Check Anything else we need to know: /cc /cc /sig cli /sig network /sig api-machinery /sig windows /priority critical-urgent /milestone v1.17
The label(s) cannot be applied. These labels are supported: <detailsIn response to : Which jobs are failing: Since when has it been failing: Testgrid link: Example: /cc /cc /sig network /sig api-machinery] /sig windows /milestone v1.17 </details
The label(s) cannot be applied. These labels are supported: <detailsIn response to : Which jobs are failing: Since when has it been failing: Testgrid link: Check /cc /cc /sig network /sig api-machinery] /sig windows /milestone v1.17 </details
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
The following have been fixed: For the DNS + Services tests, the issue is caused by: Bug report here: Disabling EndpointSlice feature-gate on kube-proxy should be enough to get them green again.
/cc
is back to just the guestbook test being solid red (tracked in ) 17 should green up similarly (was just fast-forwarded this morning)
This seems resolved. can we close this?
/close
Closing this issue. <detailsIn response to : /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: } // calculateScaleUpLimitWithScalingRules returns the maximum number of pods that could be added for the given HPAScalingRules <del> func calculateScaleUpLimitWithScalingRules(currentReplicas int32, scaleEvents []timestampedScaleEvent, scalingRules *autoscalingv2.HPAScalingRules) int32 { </del> <ins> func calculateScaleUpLimitWithScalingRules(currentReplicas int32, scaleUpEvents, scaleDownEvents []timestampedScaleEvent, scalingRules *autoscalingv2.HPAScalingRules) int32 { </ins> var result int32 var proposed int32 var selectPolicyFn func(int32, int32) int32

We created an hpa as following: Then started a deployment with 4 replicas and made the CPU utilization about 10%, which proposed to double the replicas, and it did work: Then we changed the CPU utilization to about 1%, which would trigger scaling down and would kill 2 pods only for scaleDown behavior: Then changed the CPU utilization to about 20%, which would trigger scaling up: It did scale up, and for our expectation, it should be scaled to 620/5=24 but would be limited by scaleUp behavior as 4(1+300%)=16, but in fact, it was just scaled up to 8 It would be scaled up to 16 in last scaling actions Start an hpa with a scaleUp behavior which has little long period and a little great Percent value, then start a deployment with a container whose CPU utilization could be controlled easily. Change the CPU utilization to make it scale up within limit, scale down a little, and scale up again without limit. I think it is caused by , this function seems that it just considered about scaleUp event history when calculate scaleUp limit (so does calculate scaleDown limit). I think it is a bit unreasonable. Does this action match the design? Or it does be a bug? <details</details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig autoscaling
From what I understand, we observed the following behaviour: , there are pods. The HPA scales up new pods, so we have in total. , pods are scaled down, so we have in total. HPA calculates the desired Replicas with . Since all scaling events happened within the policies , the policy value of kicks in and should allow scaling up to . But it only scaled to .

I think that there is a bug in the following code: See the comments in the following snippets for an explanation: With a of 2, the is as follows: I will think of a possible solution to the problem and would like to contribute once I've made some progress on that :)
In fact, I knew this was caused by the calculating method of "getReplicasChangePerPeriod" and mentioned it in the "Anything else we need to know" of this issue. What I wonder now is that if it is an expected action or it is just a bug. If it quite is a bug, I hope I could contribute as well.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale


Question: "reflect" goruntime "runtime" "strconv" <ins> "sync" </ins> "testing" "time"

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: bazel-test-master Which test(s) are flaking: - TestWatchBookmarksWithCorrectResourceVersion Testgrid link: Reason for failure: Anything else we need to know: links to appreciated Somehow nothing seen on triage - wonder why? links to specific failures in spyglass appreciated <!-- Please see the deflaking doc () for more guidance! --/sig testing
is still only reporting up to Sept 21 Other instances of non-deterministic result (NDR)
/sig api-machinery /sig storage Not 100% sure which SIG should triage the data race on which is producing non-deterministic results (NDRs) during CI can you both take a look and let me know? Many thanks, CI Signal Team Lead for 1.20 (release team)
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Looks like the bot didn't pick up the SIG tags - /sig api-machinery /sig storage
/sig scalability /remove-sig storage
I'm sending a fix
t.Fatal was getting called long after the test had exited, because the test was shutting down the framework before the goroutine had exited.


Question: Pod:         st.MakePod().Name("ignore").UID("ignore").Namespace(v1.NamespaceDefault).PVC("existingPVC").Obj(), NumAllNodes: 2, Diagnosis: framework.Diagnosis{ <del> NodeToStatusMap:      framework.NodeToStatusMap{}, </del> <ins> NodeToStatusMap: framework.NodeToStatusMap{ "node1": framework.NewStatus(framework.UnschedulableAndUnresolvable, `persistentvolumeclaim "existingPVC" is being deleted`).WithFailedPlugin("VolumeBinding"), "node2": framework.NewStatus(framework.UnschedulableAndUnresolvable, `persistentvolumeclaim "existingPVC" is being deleted`).WithFailedPlugin("VolumeBinding"), }, </ins> PreFilterMsg:         `persistentvolumeclaim "existingPVC" is being deleted`, UnschedulablePlugins: sets.New(volumebinding.Name), },

When PreFilter returns UnschedulableAndUnresolvable, we don't need to run the preemption, but currently we run. This unexpected preemption could cause errors from the preemption by trying to read non-exist PreFilter data. As pointed out (), actually we didn't run the preemption in such cases in the past versions, but introduced this bug. /kind bug /kind regression /sig scheduling /triage accepted /priority important-soon
/assign
will fix this.


Question: // explicitly as int, to avoid conversions and improve performance. allowedPodNumber int <ins> // Cached tains of the node for faster lookup. taints    []v1.Taint taintsErr error // Cached conditions of node for faster lookup. memoryPressureCondition v1.ConditionStatus diskPressureCondition   v1.ConditionStatus </ins> // Whenever NodeInfo changes, generation is bumped. // This is used to avoid cloning it if the object didn't change. generation int64

Looking into profiles of scheduler, there shows a bunch of things that we are doing unnecessarily. In particular, instead of computing the same information for a given node multiple times, we should cache it as part of NodeInfo: The information that should be cached and reuse later include: [x] node taints: [x] node conditions: disk/memory/inode pressure
Another quick optimization should be optimizing "selectHost" method: In fact it doesn't need to sort the whole list (which is O(nlgn). All we need to do is to choose the elements that have highest score, and sort only those by node name (to leave the logic that we currently have with round-robic logic of choosing from those).
And we should also limit the amount of places where we call "Parallelize" (because those are relatively expensive). The whole MR-like priorities is a step in this direction too.
OK - I've just sent out a PR fixing the first two things.
can we close this, or are there still work items here?
Some of those are not yet done.
- it's not a high priority for us (for now), so if you want it to be done soon, you probably need to find someone to do it.
- assigned.
Why "Parallelize" is considered as relatively expensive ? (comparing to MR style?)
You do MR when there are certain points that need some synchronizations. Or fully parallelized computation will always be faster.
To clarify, I was referring to the Parallelize function: Using this for many very small functions is pretty expensive - you can take a look into cpu-profiles. It happens that 20% or sth of cpu usage is spent on the operations on the synchronization channel.
Is all of the cacheing part completed? I assume "yes". What is remaining here.... Just an audit of parallelization ? Also, I assume "yes". If so I can audit where the parelellization is getting us hung up and then we can close this one out.
It's hard to tell if caching is done - when adding new priority functions/moving existing ones to MR-framework one pretty often discovers that more things need to be cached. There's my PR , that moves basic spreading priority function to MR-framework, which for some reason makes it worse. I didn't have time to look into that though.

This needs to be done, which may require modifying MR-framework slightly (allowing different type as a Map phase result).
Im not sure what the MR Framework stuff is referring to. Can you provide me a pointer ? Should we close this issue since the original patches for caching were done? Sounds like the work we're referring to now is actually more like a second iteration. if not, what work remains to be done before we can close this ticket? maybe we can make a checklist.
"MR-style" parallelization, that was mentioned before. Yeah, I guess we can close this - this is more like an ongoing process thing, when everyone who adds anything to the scheduler needs to keep it in mind.


Question: } return &KubeletFlags{ <del> // TODO(#41161:v1.10.0): Remove the default kubeconfig path and --require-kubeconfig. RequireKubeConfig:                   false, KubeConfig:                          flag.NewStringFlag("/var/lib/kubelet/kubeconfig"), </del> ContainerRuntimeOptions:             *NewContainerRuntimeOptions(), CertDirectory:                       "/var/lib/kubelet/pki", RootDirectory:                       v1alpha1.DefaultRootDir,

This will be the end of the deprecation period for --require-kubeconfig.
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list .
ref
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale


Question: admissiontesting "k8s.io/apiserver/pkg/admission/testing" cloudprovider "k8s.io/cloud-provider" api "k8s.io/kubernetes/pkg/apis/core" <ins> persistentvolume "k8s.io/kubernetes/pkg/controller/volume/persistentvolume/util" </ins> ) type mockVolumes struct {

API server latency for PV creation depends on underlying cloud API latency. We have admission plugin that examines incoming PV and adds topology labels and node selector: When the cluster (and cloud) are under pressure, e.g. lot of PVCs are created and dynamically provisioned at one time and attached to nodes, API server latency increases. We've seen cases when we ran out of cloud API request quota and the admission plugin needed to sleep for couple of seconds (on AWS). There should be an easy fix: check, if the created PV already has all necessary labels and topology information and ask the cloud for them only when they're missing. Dynamically provisioned volumes already have these labels. The admission plugin will then ask the cloud only for manually provisioned PVs, which we believe is just a fraction of all PVs in the cluster. There is one glitch: previously, when user created a PV with wrong topology labels, the admission plugin corrected them. With the fix suggested above, this PV won't be corrected. /sig storage cc
Implementation of the above:


Question: "net" "github.com/spf13/pflag" <ins> "go.opentelemetry.io/otel" </ins> "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc" <ins> "go.opentelemetry.io/otel/metric/noop" </ins> "go.opentelemetry.io/otel/sdk/resource" "go.opentelemetry.io/otel/semconv/v1.12.0" "google.golang.org/grpc"

Since updating from Kubernetes 1.26. to 1.29. we experiences OOM-Kills on control plane nodes. After some investigation we found that adding to the fixed the issue. (Found this via pprof/heap of one apiserver: heap dump could be if needed) Before the update to 1.29.* the control planes were running with 4GB of memory with the same worklow in the cluster without issues. After the workarround using the feature gate the memory consumption of the whole control plane is not at ~2GB. In case there is an issue with other components, memory consumption of the apiserver should not be that excessive. We just had the K8s cluster running and saw a permant increase in memory usage of the control plane nodes. And after some time (can not give a specific duration) the OS of the control plane node started to OOM kill processes. Let us know if you need additional information. <details</detailson-premise; control-planes are running with 4 CPU cores, 8GB of memory and and NVME discs <details</details<details</details<details</detailsNo response
/sig api-machinery
Please provide the pprof, and what is the size of your cluster.
As we were not sure if sharing the pprof is secure, we decided to only share screenshots first: ! ! From this information we found the this which pointed us to the mentioned feature flag. We sometimes see etcd connection timeouts in the apiserver logs, maybe this could be related but should not cause this behaviour. 5 control planes ~100 worker nodes ~1700 Pods
Can you provide the pprof without , so could compare the two?
Sure, the current pprof with set: ! !
cc
I believe this is Sounds like we may need to backport some of the dependency bumps
/assign
/sig instrumentation
/triage accepted (as it's now assigned to
/triage accepted
We need to apply the workaround in It should be simple, but needs to be backported to 1.27
all versions starting 1.27 right ?
Thanks for taking care of this so quickly
yes. All versions starting 1.27.
Actually, this only needs to be backported to v1.29.

At 1.28, we were using v0.35.0 of otelgrpc, which did not include metrics at the time.
Cherry-picks:


Question: corev1 "k8s.io/api/core/v1" apiequality "k8s.io/apimachinery/pkg/api/equality" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" <ins> "k8s.io/apimachinery/pkg/util/sets" </ins> genericapirequest "k8s.io/apiserver/pkg/endpoints/request" utilfeature "k8s.io/apiserver/pkg/util/feature" featuregatetesting "k8s.io/component-base/featuregate/testing"

On the first update of an EndpointSlice using the v1 API, we drop deprecatedTopology with EndpointSlice strategy. Although we only do this if at least 1 endpoint has been modified, there is no guarantee that node name has been populated for all endpoints within the EndpointSlice. This is most noticeable if the first update using the v1 API is only adding endpoint(s). In that case the EndpointSlice controller does not update EndpointSlices and instead just adds the new endpoint(s). That will cause node names to be dropped from the deprecated topology field without having the newer node name field populated. Node name to be retained in deprecated topology until the new field was populated. an EndpointSlice with the v1beta1 API. an endpoint in that EndpointSlice: No response This is most notable after upgrading a 1.20 cluster to 1.21, but can technically occur in any 1.21 or newer cluster. Any Any Any N/A N/A
I'm working on a fix for this, hoping to have it out later today. /triage accepted /sig network /cc
I think this is better characterized as a controller bug, but is the safest mitigation I can think of in the short term, since we don't want to rewrite 100% of endpointslices on startup. Can you open a separate issue to work through the controller reconciliation so that v1 clients can depend on the nodeName field more reliably?


Question: return err } o.PrintObject = printer.PrintObj <ins> o.Context = cmdutil.GetFlagString(cmd, "context") </ins> return nil }

/kind bug What happened: Current context was set to cluster1. Ran and got back information about cluster1 instead. What you expected to happen: I expected the output to be the cluster2 information How to reproduce it (as minimally and precisely as possible): Have two contexts in your kube config, A and B Run to set A as current context Run Anything else we need to know?: I'd like to help contribute, so if this is an easy fix, someone point me in the right direction and I'll try to get it done. Environment: Kubernetes version (use ): 1.10.2 OS (e.g. from /etc/os-release): MacOS 10.11.6 Install tools: kubectl was installed via brew
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign Filed


Question: // An error will be returned if iptables fails to update or acquire the initial lock. // Once a proxier is created, it will keep iptables up to date in the background and // will not terminate if a particular iptables call fails. <del> func NewProxier(ipt utiliptables.Interface, exec utilexec.Interface, syncPeriod time.Duration) (*Proxier, error) { </del> <ins> func NewProxier(ipt utiliptables.Interface, exec utilexec.Interface, syncPeriod time.Duration, MasqueradeAll bool) (*Proxier, error) { </ins> // Set the route_localnet sysctl we need for if err := setSysctl(sysctlRouteLocalnet, 1); err != nil {

Related to . As a safety mesure, it could be very useful to actually include a flag to proxy to MASQUERADE everything (acting very close to a userspace proxy). I think it's not very hard to do and a very good way to diagnose or even fallback in case of problem (I'm thinking about vxlan cases). Something like "--snat-clients" or "--snat-pods" could be appropriate.
/cc
Looking at the current code: It looks like we SNAT almost everything already now? Would adding a set-mark rule (behind the flag of course) at line 548 be sufficient? Deprecated Public IPs, Loadbalancer IPs, and NodePorts all appear to be marked for SNAT already (also endpoint loopback traffic).
I think that is the flag, yeah. , Benjamin Elder wrote:
Ok, I'll push out a PR for that in a moment. I should be able to take care of that and the cleanup flag (PR out but needs some more work) today. On Aug 20, 2015 2:12 PM, "Tim Hockin" wrote:


Question: }, &api.Event{}, 0, <del> controllerFramework.ResourceEventHandlerFuncs{ </del> <ins> controllerframework.ResourceEventHandlerFuncs{ </ins> AddFunc: func(obj interface{}) { events = append(events, obj.(*api.Event)) },

Forked from pointed out that all package names should be lower case, no under_scores or mixedCaps. The same rule applies to renaming imported packages. () generated a list of mixedCaps package names in our codebase: I will upload a PR later to rename kubeletTypes to kubetypes, and the rest is up for grabs.
sorry i have not saw this issue as i open a similar one(), do you mind i participate in this issue? i would happy to do some work for this. best regards.
of course, fixes are welcome. Thanks for picking up on this! We'll try to expedite the review and merge process of such PRs to avoid a lot of churns, but chances are that some rebases are needed :)
ok, i will send some fix sonner, and tag you for review. thanks again.
Not done:


Question: "k8s.io/kubernetes/pkg/util/interrupt" ) <ins> // maxMsgSize use 8MB as the default message size limit. // grpc library default is 4MB const maxMsgSize = 1024 * 1024 * 8 </ins> // DockerServer is the grpc server of dockershim. type DockerServer struct { // endpoint is the endpoint to serve on.

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Nodes in status Pods in status What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: AWS OS (e.g. from /etc/os-release): Debian 8 (jessie) Kernel (e.g. ): Install tools: kops Others: Logs:Possible prerequisites: a lot of jobs (1657) that failed (state: Error) because of internal problems Cluster size: 3 masters, 4 worker nodes Workaround: how to get the node backPossible related to
/sig node
It is limitation on grpc. The default value is (i.e. in error message)
do you know if it possible to configure the value with environment variable or maybe in some config file?
Hitting the same bug here (Kubernetes 1.10.2 with docker 17.12.1). The node has 8309 containers, most of them exited, created by failing jobs.
I'll try to change kubelet's ContainerGC so that, when there are more than say 5000 containers it reclaims the oldest ones. Unless someone thinks it's not the right think to do.
can you please try ?
thanks I'll try to reproduce but it will take time (I had to clean the host).
I don't have free infrastructure to install there K8s from some branch and don't have experience to do this at all. I have used only stable releases.
I can already tell that it probably won't solve the problem (with containerd at least):
given what you are seeing won't help but will help. right? (note that default in grpc is )
yes it would work but it's 1GB so we are in a runaway situation here; have 1GB today and you'll need 2GB tomorrow. 16MB is definitely an improvement and your change to 16MB is LGTM (thanks!), but the root problem still remains. The 2 solutions I can imagine are (1) an API change to include paging (or streaming), or (2) make the kubelet ensure there's less than a total number of containers (running+exited).

Since the issue also impacts I'm participating in the following upstream issue:
Just discovered the way of the stream seems to be the target:
improves the situation but doesn't completely fix this issue ;-) Should I open another one?
k8s-merge-robot is a bot user; I think you need to open a new issue
it was a humoristic remark ;-) I'm waiting for the improvement in containerd. It seems they're even gonna hide the details about List vs Stream in the client, so we may just have to bump de client version at some point here.
Exactly! We just downed a docker swarm mode cluster with mere 80 services with 1 container each.
run into this issue as well, dead container did not remove by gc, this functionality seems deprecated in recent k8s release, no sure why. see
I met the problem for k8s 1.13.2 .Do you have any solutions except clear dead container manually?
Really? There's an absolute universe more information in 1GB that 4MB,, just as it made sense to not make the limit 4096. Although the above is true, can we agree there is huge difference, in human terms, between 4MB and say 1GB. Pretty please change the limit to a round 1GB or something that is configurable at run-time by the service publisher or as a connection option. Maybe an option flag that waves this limit at runtime? This is a scary issue for us b/c now we have to write a segmenter/desegmenter in our client? Fear. Please let consumers of pb tech not have to do this. Everyone will just have to write their own segmenter/desegmenter hurting the pb's rep as the one stop shop for interoperabilty.
streaming is native in protobuf...
Sure... but we're hitting this limit and we're already in a stream call. :
On Jul 21, 2019 17:36, drew-512 <: Sure... but we're hitting this limit and we're already in a stream call. :Sounds weird, you're talking about the limit of the Kubernetes' kubelet gRPC client talking to the CRI runtime? We never got this bug again and it's an old one.

While doing some performance testing with AWS EKS, I ran into a situation where pods would have "CreateContainerError". These were created in a non-default namespace on CentOS 7.4 hosts. Within this cluster - remember it is used for performance testing - there were over 20,000 deployments, all in a failed state. Worker nodes would have entries like: These pods were in a namespace other than default. However, I was able to successfully scale coredns as desired without a problem. So I learned this problem was limited to a namespace. I deleted the namespace and recreated it and I could successfully deploy pods. docker --version Docker version 18.09.8, build kubelet --version Kubernetes v1.12.7 So is there a way of dealing with this error other than deleting/creating the namespace? Some cleanup process? or parameter adjustment?
On Jul 22, 2019 18:31, Dave Augustus <:docker --version Docker version 18.09.8, build kubelet --version Kubernetes v1.12.7 So is there a way of dealing with this error other than deleting/creating the namespace? Some cleanup process? or parameter adjustment?AFAIK using docker does not go through the same interface as containerd. You should probably be able to recover the situation by removing containers (ie: docker system prune).
For anyone operating a k8s cluster, running into this error and not being sure what the actual problem is... If you run this , you'll see that a message describing all containers currently known to docker is bigger than the limit the Kubelet has set for gRPC messages (16MB at the time of writing). This either means you have a LOT of (possibly dead) containers, or there's a LOT of metadata attached to your containers. This metadata probably comes in the form of custom labels attached to the container image by whatever tool you're using to create the container images. Check out for a couple of your containers and the issue should become apparent.
Thanks your comment was really helpful!! However, is there any systematic solution to resolve this? rather than on such node or remove dead containers manually? I mean, is there any chance to increase limit of gRPC message size?
My case was caused by too many tekton dead pod (over 3000 dead containers).

I will try to add to the kubelet to solve the issue
Thanks for sharing! :+1:
My case was caused by too many tekton dead pod (over 5000 dead containers). the kubelet get the message to solve this issue. first , add to kubelet comand line . second . restart kubelet third. recheck and problem is solved. thanks to
I'm glad it helped. In my case, after adding , it still happened once after. So I have to add a docker-prune service for double insurance. Just for reference /etc/systemd/system/docker-prune.service /etc/systemd/system/docker- It will run once a week on Sunday Using systemctl enable and start both. check the status with .
The solution here for me, install nerdctl, then: Once containers are pruned, you need to restart containerd for the node to show ready again.


Question: package storage import ( <ins> "fmt" </ins> "net/http" "reflect" "testing"

What happened: I'm unable to configure the api-server through a to pass admission requests to my for the resource. I'm adding the following rule to the When performing rollbacks, I'm not seeing any calls to the admission webhook. What you expected to happen: To receive validation requests when performing rollbacks. How to reproduce it (as minimally and precisely as possible): Create a ValidatingAdmissionWebhook and perform a rollback request. Anything else we need to know?: Mentioned this on the #sig-api-machinery channel on Slack, and suggested the error was that the -object never is used: Environment: Kubernetes version (use ): 12.3 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Container Linux by CoreOS 1911.5.0 (Rhyolite) 4.14.84-coreos docker://18.6.1 Kernel (e.g. ): Install tools: Others: /kind bug
/sig apps
/sig api-machinery
Good point, should call .


Question: framework.Failf("Expected %q from server, got %q", e, a) } <ins> ginkgo.By("Closing the write half of the client's connection") if err = conn.CloseWrite(); err != nil { framework.Failf("Couldn't close the write half of the client's connection: %v", err) } </ins> ginkgo.By("Waiting for the target pod to stop running") if err := WaitForTerminatedContainer(f, pod, "portforwardtester"); err != nil { framework.Failf("Container did not terminate: %v", err)

Which jobs are flaking: pull-kubernetes-e2e-gce Which test(s) are flaking: Testgrid link: /sig cli
This test seems to stop flake. Can you check if we can close this issue?
still looks flaky to me
Can you take a look?
/milestone v1.19
have you had a chance to look into this?
Possibly related to
Let's see if removing makes it less flaky


Question: // We couldn't parse metadata - fallback to computing it. podRequest = GetResourceRequest(pod) } <del> if podRequest.MilliCPU == 0 && podRequest.Memory == 0 && podRequest.NvidiaGPU == 0 && len(podRequest.OpaqueIntResources) == 0 { </del> <ins> if podRequest.MilliCPU == 0 && podRequest.Memory == 0 && podRequest.NvidiaGPU == 0 && podRequest.StorageOverlay == 0 && podRequest.StorageScratch == 0 && len(podRequest.OpaqueIntResources) == 0 { </ins> return len(predicateFails) == 0, predicateFails, nil }

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): No Note: Please file issues for subcomponents under the appropriate repo What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): local isolation Is this a BUG REPORT or FEATURE REQUEST? (choose one): <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Bug Kubernetes version (use ): kubernetes HEAD Environment: Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: What happened: For local storage isolation, overlay resource predicates for pod with only overlay or scratch storage request doesn't work. What you expected to happen: If a pod requests only overlay or scratch storage, then local storage isolation feature should limit the pods. How to reproduce it (as minimally and precisely as possible): Run local cluster from master branch, and create following pods (against a machine with less overlay fs than 1024Gi). Anything else we need to know:
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
/sig storage
CC
Fix is ready:


Question: f, tf, codec, _ := NewAPIFactory() ns := dynamic.ContentConfig().NegotiatedSerializer tf.Printer = &testPrinter{} <ins> redisMasterDeleted := false frontendDeleted := false </ins> tf.Client = &fake.RESTClient{ NegotiatedSerializer: ns, Client: fake.CreateHTTPClient(func(req *http.Request) (*http.Response, error) { switch p, m := req.URL.Path, req.Method; { <del> case p == "/namespaces/test/replicationcontrollers/redis-master" && (m == http.MethodGet || m == http.MethodPut || m == http.MethodDelete): </del> <ins> case p == "/namespaces/test/replicationcontrollers/redis-master" && m == http.MethodDelete: redisMasterDeleted = true fallthrough case p == "/namespaces/test/replicationcontrollers/redis-master" && m == http.MethodPut: </ins> return &http.Response{StatusCode: http.StatusOK, Header: defaultHeader(), Body: objBody(codec, &rc.Items[0])}, nil <ins> case p == "/namespaces/test/replicationcontrollers/redis-master" && m == http.MethodGet: statusCode := http.StatusOK if redisMasterDeleted { statusCode = http.StatusNotFound } return &http.Response{StatusCode: statusCode, Header: defaultHeader(), Body: objBody(codec, &rc.Items[0])}, nil </ins> case p == "/namespaces/test/replicationcontrollers" && m == http.MethodPost: return &http.Response{StatusCode: http.StatusCreated, Header: defaultHeader(), Body: objBody(codec, &rc.Items[0])}, nil <del> case p == "/namespaces/test/services/frontend" && (m == http.MethodGet || m == http.MethodPut || m == http.MethodDelete): </del> <ins> case p == "/namespaces/test/services/frontend" && m == http.MethodDelete: frontendDeleted = true fallthrough case p == "/namespaces/test/services/frontend" && m == http.MethodPut: </ins> return &http.Response{StatusCode: http.StatusOK, Header: defaultHeader(), Body: objBody(codec, &svc.Items[0])}, nil <ins> case p == "/namespaces/test/services/frontend" && m == http.MethodGet: statusCode := http.StatusOK if frontendDeleted { statusCode = http.StatusNotFound } return &http.Response{StatusCode: statusCode, Header: defaultHeader(), Body: objBody(codec, &svc.Items[0])}, nil </ins> case p == "/namespaces/test/services" && m == http.MethodPost: return &http.Response{StatusCode: http.StatusCreated, Header: defaultHeader(), Body: objBody(codec, &svc.Items[0])}, nil default:

When replacing an existing pod, the command fails as it tries to create a pod with a name that is still used by terminating pod. How does the option works? deletes a pod (e.g. mypod) creates a new pod with the same name (mypod) What happens? As the command tries to recreate the pod, it fails as the is in state. Thus it can not be created. The problem is the does not wait for a pod to be deleted. Reproducer: run mypod --image=yapei/hello-openshift --generator=run-pod/v1 get pod mypod -o yaml replace -f --force The last command ends with:
option introduce by
With set to positive value, what should be the default behaviour of the command? When set to , should the command hang for and then create the pod? Or is there a way to create a callback (inside a scheduler?) that creates the pod once the current one is deleted?
You could watch for the resource to be deleted and create after you see the deletion happen
have you ever tried to set , I think this will do a non graceful deletion. should make sure the old resource is deleted before it creates a new resource, if not, that's a bug.
Though the help says the is to be used only with , it returns the error above.
Then I think there are two issues we need to track: 1. option should be ignored or just returns error if users are not using option. 2. should always make sure the old resource has been deleted before create the new resource.


Question: // badGatewayError renders a simple bad gateway error. func badGatewayError(w http.ResponseWriter, req *http.Request) { <ins> w.Header().Set("Content-Type", "text/plain") w.Header().Set("X-Content-Type-Options", "nosniff") </ins> w.WriteHeader(http.StatusBadGateway) fmt.Fprintf(w, "Bad Gateway: %#v", req.RequestURI) }

The error reporting functions here all repeat a URI without sanitizing it. An XSS attacker could use this to get the server to embed a reference to a malicious script. (As always, please note that kubernetes apiserver is NOT xss hardened and we do NOT recommend using the api directly from a browser.)
The error correctly sets the content type to , so this is only an issue using a browser that does content sniffing. Would it be sufficient to just add the header here?
Yes, that will work for browsers that honor the header. It's probably sufficient-- we could do that globally, we set Content-Type correctly everywhere, I think.


Question: r.lastSyncResourceVersionMutex.Lock() defer r.lastSyncResourceVersionMutex.Unlock() r.lastSyncResourceVersion = v <del> rv, err := strconv.Atoi(v) if err == nil { r.metrics.lastResourceVersion.Set(float64(rv)) } </del> }

What happened: After upgrading to kubernetes 1.12.5 we observe failing nodes, that are caused by kubelet eating all over the memory after some time. ! I use image to run kubelet on 102 clusters and since a week we see some nodes leaking memory, caused by kubelet. I investigated some of these kubelets with strace and pprof. With 3s of running strace I saw = 50 openat() calls to the same file from the same threadid (pid) from kubelet: If I do pprof kubelet it shows clientgo metrics and compress is taking over most of the compute time. Memory profile png: ! The reflector metrics seem to be the problem: What you expected to happen: I expect that kubelet does not need so much memory How to reproduce it (as minimally and precisely as possible): I don't know Anything else we need to know?: One of the affected clusters has only 120 Pods and 3 Pods are in CrashLoopBackOff state, one pod since 6 days on an affected node which was investigated. Environment: Kubernetes version (use ): v1.12.5 Cloud provider or hardware configuration: AWS OS (e.g. from /etc/os-release):Kernel (e.g. ): Linux ip-172-31-10- 4.14.63-coreos SMP Wed Aug 15 22:26:16 UTC 2018 x86_64 Intel(R) Xeon(R) CPU E5-2686 v4 @ 2.30GHz GenuineIntel GNU/Linux Install tools: custom
/sig node
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I think it might be a side effect of , but I couldn't find any numbers in there. The comments mention an offline discussion, was this brought up?
What happens (I only investigated it very briefly) is that kubelet creates a bunch of reflectors for every pod — secret mounts, configmap mounts, token mounts etc. Each reflector registers a number of histograms, gauges, etc. and even when the reflector is stopped and removed the metrics are never removed.

After a few days on a busy cluster you get millions of metrics bloating up kubelet and everything else using the reflectors.
This was broken since v1.12.0, so probably unrelated to .
correct that PR is only in v1.14.0-alpha.2 v1.14.0-alpha.1 and master
For everyone that finds this issue and needs a patch to disable the reflector metrics:
Do you have same metrics after disabling ReflectorMetricsProvider?
no, it drops the reflector metrics
is this indirectly caused by switching to watch-based managers?
Yeah - switching to watch resulted in more extensive using of a reflector. It seems that those metrics are directly coming from reflector (I wasn't aware of those). So it's highly probably that this change cause it. I think that we don't really need those metrics, so if we could switch them off in Kubelets, that should solve the problem.
is sent out for review to fix that. We should probably cherrypick it to previous releases if we agree that's the right way to solve it.
Actually, I realized that I don't fully understand the problem. It should be a problem in default Kubernetes (what we build from this repo), because those metrics are not switched on for kubelet: metrics factory is defaulted to noop the prometheus metrics need to be explicitly linked and initialized (as they are in apiserver e.g.: )
Wouldn't it be better to rewrite the reflector metrics so they're aggregated in a better way rather than relying on people not accidentally enabling them? Or at least put a huge warning on top saying "this will leak memory like crazy" just so the same situation doesn't repeat in other code using client-go?
- that's a separate issue whether the metrics are reasonable or not (and node sig-node related). One thing that I would like to confirm: do you customize k8s yourself? I would like to ensure that this issue you is in your setup and not in upstream k8s.
Our production configuration is this one: Kubelet config: We run with an image called , but it's just a mirror of . So upstream Kubernetes.
I try to figure out an issue why these metrics are even there and I found issues from the past showing leaks already in older versions: Sounds for, that we need a postmortem to make this not happening again.

I don't find any issue in kubernetes nor in client-go, that could reasonable explain why these were introduced in the first place. I guess I just did not found it, and I hope someone can highlight where the decision came from.
I am running Kubernetes v1.12.3 and I don't see reflector metrics being used by kubelet ( shows nothing). FWIW, the cluster is created using kops. Any idea, how are reflector metrics enabled?
and myself will also look into this as well and see what we can find.
Thanks
The issue only exists if hyperkube is used when running kubelet, as hyperkube has imported the package for apiserver and controller-manager. If kubelet binary is used, then there're no reflector metrics (this is same with your findings). Since the metrics is not exposed by kubelet cmd, what do you think of disabling it for kubelet subcommand in hyperkube? I can't reproduce the issue (with hyperkube used) on my own cluster: ! Since both apiserver and controller-manager have also imported the package, have you also observed same issues for them?
+1
Yes, we observe the same thing for apiserver and controller-manager as well.
I would also suggest not using imports for side effects because this usually creates hard to debug and non-obvious problems like this.
So you mean disable the reflector metrics totally?
No, I suggest changing the code so that the end users (kubelet, apiserver, etc) would have to explicitly enable the metrics by calling a function instead of having this happen as a side effect of an import statement.
LGTM. WDYT? We should probably do this for both kubelet and control plane services (apiserver and controller-manager).
I'm fine with that.
+1
+1
Why wouldn't we want to just actually disable this metric wholesale? Enabling this flag would basically be the same thing as saying enable a memory leak, no?
/assign
Han points out that, since the metric has a random suffix each time, it's not very useful for monitoring anyway.
/sig api-machinery


Question: volumeNodeAffinity *v1.VolumeNodeAffinity ) <ins> l.genericVolumeTestResource = genericVolumeTestResource{ driver:  driver, config:  l.config, pattern: pattern, } </ins> // Create volume for pre-provisioned volume tests l.volume = CreateVolume(driver, l.config, pattern.VolType)

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: ci-kubernetes-e2e-gci-gce Which test(s) are failing: I0221 20:48:18.462] [sig-storage] In-tree Volumes I0221 20:48:18.462] [ I0221 20:48:18.462] [Driver: gcepd] I0221 20:48:18.462] [ I0221 20:48:18.462] [Testpattern: Dynamic PV (filesystem volmode)] volumeMode I0221 20:48:18.463] [ I0221 20:48:18.463] should create sc, pod, pv, and pvc, read/write to the pv, and delete all created resources Since when has it been failing: unsure Testgrid link: Reason for failure: From the logs, it looks like it's deleting both PVC and PV objects at the same time, which is not correct. I0221 20:48:18.460] Feb 21 20:48:09.866: INFO: Deleting PersistentVolumeClaim "pvc-cjv2l" I0221 20:48:18.460] Feb 21 20:48:09.923: INFO: Deleting PersistentVolume "pvc--3619-11e9-b923-" should be changed to only delete PVC and wait for PV to be deleted by the driver. Can it just call which does it correctly? Anything else we need to know:
cc
It has been like that already before my recent code refactoring: Therefore I have to defer to - I myself don't really know what this test does or why it was written like that.
/milestone v1.14 We're entering and this looks relevant to the 1.14 release. Please if I am incorrect
/priority critical-urgent


Question: "errors" "fmt" "net" <ins> goruntime "runtime" </ins> // Enable pprof HTTP handlers. _ "net/http/pprof"

A new flag indicating how kube-proxy gets the number of CPUs, either from goruntime or . In kube-proxy, we can use to set node's , the resulting is . We have nodes where pods with none CPU policy and static CPU policy are running together. Pods like kube-proxy are running in none CPU policy by default, and they only gets CPUs left by static CPU policy pods. If static CPU policy pods take away most CPUs cores, kube-proxy can only get one or two cores, then will be set to a very small values. In this case, a lot packet failure can occur. When we set kube-proxy to work in static CPU policy, kube-proxy only gets the number of CPUs selected by the CPU policy, not the number of total CPUs on the node. This may surprise users who are only trying to limit kube-proxy's CPU usages, but resulting in some conntrack fulls faults. Please let me know if this enhancement is reasonable, so I can send a pr later. Thanks!
/sig network
Is it refer to deploy order of kube-proxy? If kube-proxy deploy first in node, will it get just one or two core?
By one or two I mean kube-proxy gets a small number of CPU cores, because most of CPU cores are allocated to static policy pods, sorry for the misunderstanding. The number of conntrack-max depends on the CPU cores when kube-proxy started, even kube-proxy was first deployed, it can get restarted or redeployed by some reasons. Then the conntrack-max will change.
yeah, when kube-proxy deploy firstly, Maybe it can see the whole cpu nums from shared cpu(/var/lib/kubelet/cpumanagestate when kubelet start with static cpu policy). But I think it is a bug for kube-proxy. The cpu info can also get from node resource info. So will it be suitable to add a flag to set cpu num?
Maybe, or we can add a flag for user to choose where kube-proxy should get CPU informations from. There maybe some multi-tenant scenarios where kube-proxy are deployed for per tenant so they would like to limit conntrack-max by CPU cores allocated to each tenant.
I think the simple fix is to replace with something that counts /proc/cpuinfo. I don't know of anyone running multiple proxies, and I don't think it would work well. The intention was "all the CPUs on the node". PRs welcome!
/triage accepted


Question: if volume.RBD != nil && existingVolume.RBD != nil { mon, pool, image := volume.RBD.CephMonitors, volume.RBD.RBDPool, volume.RBD.RBDImage emon, epool, eimage := existingVolume.RBD.CephMonitors, existingVolume.RBD.RBDPool, existingVolume.RBD.RBDImage <del> if haveSame(mon, emon) && pool == epool && image == eimage { </del> <ins> // two RBDs images are the same if they share the same Ceph monitor, are in the same RADOS Pool, and have the same image name // only one read-write mount is permitted for the same RBD image. // same RBD image mounted by multiple Pods conflicts unless all Pods mount the image read-only if haveSame(mon, emon) && pool == epool && image == eimage && !(volume.RBD.ReadOnly && existingVolume.RBD.ReadOnly) { </ins> return true } }

Only a single pod per node is able to use the same Ceph RBD image in read-only mode. Further pods fail to be scheduled with . I would expect multiple pods per node to be able to use the same RBD image in read-only mode, for example by mapping the image once per node and bind-mounting it into each pod. Is there any reason why this isn't possible?
Will wait for confirmation whether this is as per-design or a bug from a storage area engineer. If this is a bug, need a pointer to the log snippets for further debugging. cc
According to , Ceph RBD forbids any two pods from sharing the same monitor, and match pool and image. for details CC
yes, it is a valid use case. let me run some tests first.
This is an issue with the PV scheduler constraints but works in volumes as mentioned here:
opened a fix.
Great, thanks