Question: docs/user-guide/kubectl/kubectl_describe.md docs/user-guide/kubectl/kubectl_drain.md docs/user-guide/kubectl/kubectl_edit.md <ins> docs/user-guide/kubectl/kubectl_edit_configmap.md </ins> docs/user-guide/kubectl/kubectl_exec.md docs/user-guide/kubectl/kubectl_explain.md docs/user-guide/kubectl/kubectl_expose.md

Kubernetes version : What happened: When I try to display configmap content with yaml output (kubectl get configmap webapp-inifile -o yaml), the output is sometimes pretty, and sometimes not -depending newlines are effectively displayed or not - just n char- I can't find what's the difference between some file and some others. (limit on numer of lines ?) Configmap is created with --from-file option Ex : What you expected to happen: More human readable display, in order to be able to easily change parameter when kubectl edit congifmap How to reproduce it :
Hrm, this seems to me like the right answer is to special case 'kubectl edit' for configmaps, rather than try to hack around with the YAML decoder. I suppose we could use block quotes (e.g. ) if we see newlines but that seems error prone...
Closing this, pretty print is there, kubectl configMap edit support is there.
This still happens on kube 1.7 :-( I'm not 100% sure how to reproduce, but what I did was: have an existing configmap and edit a few lines exit the editor ... now I see the characters instead of actual newlines. even shows the characters
I'm also seeing this issue in kube 1.7. During the past few days I have edited a configmap ~110 times, and seen the formatting get destroyed (spaces messed up, characters visible) twice. I can't determine anything specific that I did in those two edits that would have triggered the formatting messup.
I'm still seeing this issue too. Pushed a lot of identical configmaps, some ones are displaying good, and some other bad. For example, I have a configmap with 3 files in ; they all are identical except one variable ; 2 of them display normally, and the last one displays with n ...
also seeing this issue on 1.7
can we reopen this issue? or open a new one?
I created a brand new configmap using and when I viewed the yaml with it's full of n characters and escape characters. This makes it very difficult to edit.
I am experiencing the same. n character is messing up a configmap I'm creating with --from-file option. Anyone found a reason why this is happening ?

Also failing in 1.8.4
I am also facing the same issue of formatting .. I have a application configuration of 400-500 lines. Tryied many times. For fist 40-50 lines the format is good ans easily readable.. However, when i try to create config for entire configuration file its very difficult to edit the config map..
Same issue here. No idea what makes one configmap show better than another.
This issue corrupted my aws-auth configMap that authenticates my user to access the EKS cluster and while trying to add other users to the configMap the formatting corrupted the configMap file and now I am locked out of my EKS cluster. Version: 1.10.3
I don't know if this will help somebody else but I was facing the same issue with escaped data in a config map edit and get (the config map had a single .conf file), and it seems that I had a unneeded extra space at the end of a single line in my data. When I removed that extra space the pretty printing was maintained in edit and get.
11 Still same
still same ( there is an example if install helm install stable/prometheus , you will find cm/prometheus-server )
I agree with - I can recreate it every time If you were to execute the following (ensuring no trailing spaces), the configmap remains "pretty" However, repeat the above (or edit the existing one with ), and put a trailing space after one of the key/value pairs, and that then messes up the configmap contents
Can we reopen this issue? I need this feature to change the yaml in cm, and I have been struggling with the intent and n for several days ...
Please reopen this ticket, it's incredibly frustrating to have to deal with this even on small configmaps
Got this also. I had to remove tabs from my config file to get the yaml multiline display.
Just created a messed up configmap on kuberntes 1.12.2.
Likewise.
OMG thank you!
Just ran into this in EKS (1.10.3) -- highly annoying. I have one config map that will come out OK if I use create directly, but if I pipe the output of to apply then it gets munged. Does seem like this is something that needs addressing.

Have the same issue and fixed by removing tabs
Have the same issue, don't have any tabs in file. This issue totally should be reopened.
This issue is valid and I am not sure why it was closed
Not only spaces. Trail symbols too. I uses vim for it
"n n" (spaces between end of lines) in one of my files also messed up the pretty display.
Created a new issue when I hit this again and was able to repro using the trailing whitespace
I've removed all the trailing whitespaces and tabs. The problem is gone.
Check any possible special chars. It solve my problem some time ago
I have seen this issue in K8 1.7 -1.13 kubectl create configmap NAMESPACE --from-file=$PATCHFILE --dry-run=true --output=yaml where PATCHFILE is attached file below. I renamed it from .json to .log so this webpage would let me include it.
Solve the issue by delete whitespace at each end of line.
The output of is ugly. Wether using json or yaml, k8s injects a whole bunch of escape characters and messes up the formatting of the original config file. It would be very nice to have a switch that tells it to dump the config out "normally"!
I a new line in a configMap using Tab for identation. After changing to Spaces instead of Tab, I was able to see the configmap as expected...
Same here, white space/s at the end of line ruin the output, everything back to pretty when white space/s removed.
This worked for me.
Since I've still met this on 1.16, I'll point a bullet list of the regular culprits for the configmap messing: spaces before newline:Tabs:Special characters like ãẽõ, áéó, öß will also mess your files. Look up for them in your comments, they might pass through!
An easy one liner to remove trailing whitespaces in a ConfigMap:
Please reopen. This issue still remains. This format makes eks aws-auth configmap very hard to maintain. Once the file is applied with trailing whitespaces the whole thing gets lots of n and . As a temporary measurement is there a way to reformat the configmap in the "readable" way? after the n are created?

if you use way didn't reformat the file for me.
Nurza's method works. But you may have a space before EOF as well: Or you may have CR's in there , in which case you should remove the CRs. Or you may have a TAB, which you should replace with one or more spaces.
No CR/TAB/new lines. I have a "clean" file that after I apply it all works great. (display as expected) After I add one trailing white space to it, it gets those n. I tried to run Nurza's method but it didn't work. Any other Ideas why Nurza's method is not working for me ? This a part of the output I'm getting with
has a space after it, although you don't see it. Because the LF is on the next line, there must be a space before it.
Correct but if I run Nurza's method the n are still there. Am I still missing anything? This is what I run
Nobody said that sed expression was perfect. Once you know this is your doing -- and you know how to fix it -- you're not bothered by this behaviour anymore.
This is scary shit :hankey: ... what is the industry practice to edit the file like these: aws-auth...? I managed to type 20+lines by hand :disappointed: and it worked on 3rd attempt :disappointed_relieved:
Cant' say about the 'like this', but aws-auth you can use eksctl to do the hard work for you:
If the formatting of cm goes wierd a simple hack to get it back to normal is : kubectl get cm configmapname -o yaml cm.yamlNow copy the contents of file and past it on is powerful tool to check the linting of yaml files. This will provide you with the configmap as expected with correct formatting. Paste the output in another yaml file (for e.g - )kubectl apply -f
Anyone knows why the issue is closed? It's not fixed as far as I can tell.

If you just want the raw output as it was read in when created --from-file, you can use jq to get the raw string (without escaped newlines etc) If you created a configmap from a file like this: Get the data:
Edited a cm, came back in, and had n all over. Something about white space in my previous edit? Not great overall :/ On kubectl version Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.5", GitCommit:"", GitTreeState:"clean", BuildDate:"2020-07-04T15:01:15Z", GoVersion:"go1.14.4", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.6", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-11-13T11:11:50Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}
Awesome, very good website~
Thank You very much
I was able to fix this behavior by: 1) Don't use tabs, convert to spaces 2) To remove spaces before a newline character, use this: It will also convert CRLF to LF only, by removing .
Something small that exacerbates this issue is that in python's standard lib module , up to some version (I think python 3.4, see ), pretty printing adds a trailing space after every comma by default, unless you supply a list of separators yourself that omit the spaces (e.g. like ) So if you have old-ish glue (e.g. python 2.7) that parses a k8s configmap into a file for archival purposes, and you made that file readable by pretty-printing it, using that file to regenerate a new configmap results in this gross escape behaviour.
Hello There, One of our colleague has edited configmap with wrong data, now whole cluster inaccessible with any of role, need help on immediate basis and much appreciated. Error: through eksctl - Error: getting auth ConfigMap: Unauthorized through kubectl error: You must be logged in to the server (Unauthorized)
yaml marshallers will write a string to a multiline string e.g.

UNLESS string contains a character string terminates in a space character string contains a substring which matches the regex
I am in the same situation. Were you able to solve this?
Also reproduce in v1.22.3 :(
I occasionally encountered this problem in v1.15.6 or v1.21.2, but eventually I was able to deal with it by using edit and then using vim's own set list to back-propagate the original static configmap code configuration under gitlab to see if there were extra spaces, and then modifying the local configmap file based on the online configmap's set list display under edit. The state of the online configmap under edit, to modify the local configmap file, is ultimately able to deal with this problem, but if there is a better official way to do it would be good.
yes if you are working on linux
i would like to share an observation that helped me sort out the line merging and n. it appears to be related to the last line of the file you are using (--from-) if the last line ends with empty space or tab. the config map is merged up in one line with n as line feed if you remove the empty space from the last line, the ConfigMap will import file as is with +
I got the same error after editing the ConfigMap file. I used an online tool to remove the trailing white space that we cannot see in the text. After copying the content which removed trailing whitespaces I was able to resolve this issue FYI - (online tool which I used)
in v1.24 also i can see same issue. how we can edit cm with many lines.
Finally, I found that the absence of a value defined or the value is empty string in will cause the issue. data: {{ }} is not defined in my yaml just use quote to the value resolved
delete space before n or after n it works for me
did anyone find solution when we edit cm how we can see in proper format as for me really difficult to edit cm as we have many n char and I can still see this issue in k8s version 1.28


Question: apiVersion: v1 kind: Pod metadata: <del> name: l7-lb-controller-v0.9.4 </del> <ins> name: l7-lb-controller-v0.9.5 </ins> namespace: kube-system labels: k8s-app: glbc <del> version: v0.9.4 </del> <ins> version: v0.9.5 </ins> kubernetes.io/cluster-service: "true" kubernetes.io/name: "GLBC" spec: terminationGracePeriodSeconds: 600 hostNetwork: true containers: <del> - image: gcr.io/google_containers/glbc:0.9.4 </del> <ins> - image: gcr.io/google_containers/glbc:0.9.5 </ins> livenessProbe: httpGet: path: /healthz

0 Release Blocker:
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list


Question: Subsystem:         KubeletSubsystem, Name:              LastKnownGoodConfigKey, Help:              "The config source the node will fall back to when it encounters certain errors. The count is always 1.", <del> DeprecatedVersion: "1.22", </del> <ins> DeprecatedVersion: "1.22.0", </ins> StabilityLevel:    metrics.ALPHA, }, []string{ConfigSourceLabelKey, ConfigUIDLabelKey, ConfigResourceVersionLabelKey, KubeletConfigKeyLabelKey},

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --kubelet failed to start when specifying ,with following error: Try to start kubelet with : Kubernetes version (use ): master branch Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig node
Submitted PR to fix this issue.
/triage accepted /priority important-soon


Question: fs.DurationVar(&c.ImageMinimumGCAge.Duration, "minimum-image-ttl-duration", c.ImageMinimumGCAge.Duration, "Minimum age for an unused image before it is garbage collected.  Examples: '300ms', '10s' or '2h45m'.") fs.Int32Var(&c.ImageGCHighThresholdPercent, "image-gc-high-threshold", c.ImageGCHighThresholdPercent, "The percent of disk usage after which image garbage collection is always run. Values must be within the range [0, 100], To disable image garbage collection, set to 100. ") fs.Int32Var(&c.ImageGCLowThresholdPercent, "image-gc-low-threshold", c.ImageGCLowThresholdPercent, "The percent of disk usage before which image garbage collection is never run. Lowest disk usage to garbage collect to. Values must be within the range [0, 100] and should not be larger than that of --image-gc-high-threshold.") <del> fs.DurationVar(&c.VolumeStatsAggPeriod.Duration, "volume-stats-agg-period", c.VolumeStatsAggPeriod.Duration, "Specifies interval for kubelet to calculate and cache the volume disk usage for all pods and volumes.  To disable volume calculations, set to 0.") </del> <ins> fs.DurationVar(&c.VolumeStatsAggPeriod.Duration, "volume-stats-agg-period", c.VolumeStatsAggPeriod.Duration, "Specifies interval for kubelet to calculate and cache the volume disk usage for all pods and volumes.  To disable volume calculations, set to a negative number.") </ins> fs.Var(cliflag.NewMapStringBool(&c.FeatureGates), "feature-gates", "A set of key=value pairs that describe feature gates for alpha/experimental features. "+ "Options are:n"+strings.Join(utilfeature.DefaultFeatureGate.KnownFeatures(), "n")) fs.StringVar(&c.KubeletCgroups, "kubelet-cgroups", c.KubeletCgroups, "Optional absolute name of cgroups to create and run the Kubelet in.")

What happened: After set --volume-stats-agg-period=0 in kubelet args, the kubelet logs still shows What you expected to happen: it should be disabled How to reproduce it (as minimally and precisely as possible): add --volume-stats-agg-period=0 to your kubulet config change du to a bin which will sleep more than 4s. Anything else we need to know?: None Environment: Kubernetes version (use ): 1.18 Cloud provider or hardware configuration: vspher OS (e.g: ): centos Kernel (e.g. ): Install tools: DCE daocloud platform Network plugin and version (if this is a network-related bug): parcel( calico and vlan) Others:
/sig node /assign
The second below is not as the comments: I set it to -1s, kubelet log showsVolume stats collection disabled. during startup. I set it to '0', kubelet log doesn't showsVolume stats collection disabled. during startup. (Not disabled) I set it to '1s', kubelet log doesn't showsVolume stats collection disabled. during startup. I don't set it, kubelet log doesn't showsVolume stats collection disabled. during startup.
/triage accepted
/area kubelet is waiting for review.


Question: // 3. Ties are broken by sum of priorities of all victims. // 4. If there are still ties, node with the minimum number of victims is picked. // 5. If there are still ties, the first such node is picked (sort of randomly). <del> //TODO(bsalamat): Try to reuse the "min*Nodes" slices in order to save GC time. </del> <ins> // The 'minNodes1' and 'minNodes2' are being reused here to save the memory // allocation and garbage collection time. </ins> func pickOneNodeForPreemption(nodesToVictims map[*v1.Node]*Victims) *v1.Node { if len(nodesToVictims) == 0 { return nil } minNumPDBViolatingPods := math.MaxInt32 <del> var minPDBViolatingNodes []*v1.Node </del> <ins> var minNodes1 []*v1.Node lenNodes1 := 0 </ins> for node, victims := range nodesToVictims { if len(victims.pods) == 0 { // We found a node that doesn't need any preemption. Return it!

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: Fix the following : The basic logic in of using the is continuing reducing the nodes slices through , , and . It needs 4 slices to do this. This is my thought to reuse the slices: Use 2 nodes slices, such as , The reducing process becomes: -Clear -Clear - please give some suggestion, thanks :smile:
/sig scheduling
The challenge is that clearing a slice and reusing it still needs GC clean up, so it may not be effective. The size of slices shrink as we proceed through the function. So, having two slices and over-writing the elements in them can help reduce GC time, but this approach needs some additional checks to ensure we are not using stale elements.
To avoid clearing the slices, this is my thought: Declare to indicate the actual length of the 2 node slices, instead of . Appending a node to a node slice will add 1 to . Clearing a node slice will set to 0. WDYT
Yes, I can't think of any better solution either, but remember that you should append only to the slice in the first for loop of the function. The second slice should be initialized with the same number of elements as the slice used in the first for loop. Setting elements of the slices should be done by indexing them (not by append) in the second, third, and forth loop.
OK, I will submit a PR to implement it.


Question: if err != nil { return volumes, fmt.Errorf("could not read directory %s: %v", volumePluginPath, err) } <del> for _, volumeDir := range volumeDirs { volumes = append(volumes, filepath.Join(volumePluginPath, volumeDir)) </del> <ins> unescapePluginName := utilstrings.UnescapeQualifiedName(volumePluginName) if unescapePluginName != csi.CSIPluginName { for _, volumeDir := range volumeDirs { volumes = append(volumes, filepath.Join(volumePluginPath, volumeDir)) } } else { // For CSI volumes, the mounted volume path has an extra sub path "/mount", so also add it // to the list if the mounted path exists. for _, volumeDir := range volumeDirs { path := filepath.Join(volumePluginPath, volumeDir) csimountpath := csi.GetCSIMounterPath(path) if pathExists, _ := mount.PathExists(csimountpath); pathExists { volumes = append(volumes, csimountpath) } } </ins> } } return volumes, nil

In , we check mount on first level directory in pod volume directories. It is true for many volume plugins, but not for CSI plugin which . cc
/assign
Pardon my ignorance of kubelet source code, but this looks very much similar to an issue we have been running to many times recently, when developing a CSI plugin. The problem is that when CSI volume publish is called and the mount point is not propagated from the CSI plugin pod (either missing bidirectional flag in yaml or not supported by underlaying k8s cluster implementation), the kubelet nevertheless starts a pod using the volume. The volume is not there, so the app writes to root filesystem in fact, leading to all other kinds of problems - i.e. losing data, filling up root fs. IMHO what kubelet should do is to check that the mounted filesystem is indeed visible after publish call and if not it should fail the volume attach request. If this ticket was fixed, would it solve the issue we are seeing or should I open a new ticket? thanks
I think we can potentially add that check but we need to make sure first it can work with ephemeral CSI drivers. Ephemeral drivers would have to "stage" their directories somewhere, like /var/lib/kubelet/plugins/<csi driver/, and then bind mount it to the publish path. cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Hmm... That is a good question. Does csi require a mount to be made or only that the specified directory be created? Will have to go read through the spec...
/remove-lifecycle stale
The only place in the spec I could find was this comment: // For volumes with an access type of mount, the SP SHALL place the // mounted directory at target_path. So I think having Kubelet enforce that check would be ok with the spec, and a good thing to do.
cc CSI spec says that the target path has to be a mount. Shall we go ahead and enforce it then? Are we going to break drivers that may not be doing this?

Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
The issue mentioned here involves two functions . Now with PR , the issue in CheckVolumeExistenceOperation is solved by checking mount path instead of volume path. The only issue here is for getMountedVolumePathListFromDisk which is used during pod deletion. This function was introduced by PR . Now I am questioning whether we need to check the mount point or just need to check pod dirs exist before deleting the pod. Firstly, for some plugins, there might not be mount point. Secondly, without checking mount point, it means that pod cannot be deleted as long as there are pod volume dirs. In normal case, volume dirs should be cleanedup during volume teardown. For cases that kubelet restarts, we have reconciler try to clean up volume mounts if possible. So we can propose a couple of solutions here mount point check. It solves the problem, the downside is that there might be some rare case that reconciler could not clean up dirs after kubelet restarts. check mount point, and also check the next level "/mount" dir. This might still cause problem for some plugins that do not have mounts. cc
Discussed offline with For proposal 1., pod will not be deleted from api server if there are still volume dirs exist. So to make sure it works well, we need to have reliable way to clean up dirs. Normally when kubelet restarts, reconciler will either add it into actual state (if reconstruction works) or try to clean it up directly once unless volume is still in desired state. So if reconstruction does not work, and it fails to clean up during reconciler sync, there is no other chance to clean up the dirs.

Or if volume was in desired state and removed quickly afterwords, the dir might not get the chance to be cleaned up at all. We are thinking to improve this part, we could use mark volume "uncertain" in actual state feature. During reconstruction, we can remove the check for mounted points (since the mount might be gone during node reboots or driver does not have moutns at all). Reconciler could mark those volumes as uncertain (even if it was in desired state) to make sure those volume got cleaned up or remount whenever needed with retries. With this change, I think it is safe to remove the mount point check for pod deletion too.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: "net/http/httptest" "net/url" "os" <ins> "sort" </ins> "strconv" "strings" "testing"

<!-- Please only use this template for submitting enhancement requests --What would you like to be : I would like that besides User-agent we the Accept header as well with the "acept all" default. Just like curl does. Why is this needed: Because it was my expectation that the probe would work similarly to a curl but is not the case. I think more folks can hit this situation and waste some time to finally sort it out If this is not something you would like to add I would then like the docs to mention this specifically. _ PS: I would love to propose the improvement my self
/sig node network
/assign
Is the goal simply to avoid tripping over servers that look for that header? I don't think this presents any hazards - is that right? If so, it seems like a simple PR if you want to tackle it. /triage accepted /kind feature
cool! will look into it! thanks :)
Hello sorry but be bugging you, but I have opened the PR but I had 0 response since then. I think I have met all the requirements, so I was wondering (as this is my first PR in k8s) if you could help me a bit. Thanks anyway.


Question: } // Save current content of *_proxy and *_PROXY variables. <del> savedEnv := resetProxyEnv() </del> <ins> savedEnv := resetProxyEnv(t) </ins> defer restoreEnv(savedEnv) <del> t.Log("Saved environment: ", savedEnv) os.Setenv("HTTP_PROXY", "http://proxy.example.com:3128") os.Setenv("NO_PROXY", "example.com,10.0.0.0/8,2001:db8::/48") // Check if we can reliably execute tests: // ProxyFromEnvironment caches the *_proxy environment variables and // if ProxyFromEnvironment already executed before our test with empty // HTTP_PROXY it will make these tests return false positive failures req, err := http.NewRequest("GET", "http://host.fake.tld/", nil) if err != nil { t.Fatalf("unexpected err: %v", err) } proxy, err := http.ProxyFromEnvironment(req) if err != nil { t.Fatalf("unexpected err: %v", err) </del> <ins> for _, rt := range tests { warning, _ := rt.check.Check() if (warning != nil) != rt.expectWarnings { t.Errorf( "failed HTTPProxyCIDRCheck:ntexpected: %tnt  actual: %t (CIDR:%s). Warnings: %v", rt.expectWarnings, (warning != nil), rt.check.CIDR, warning, ) } </ins> } <del> if proxy == nil { t.Skip("test skipped as ProxyFromEnvironment already initialized in environment without defined HTTP proxy") </del> <ins> } func TestHTTPProxyCheck(t *testing.T) { var tests = []struct { name           string check          HTTPProxyCheck expectWarnings bool }{ { name: "Loopback address", check: HTTPProxyCheck{ Proto: "https", Host:  "127.0.0.1", }, // Loopback addresses never should produce proxy warnings expectWarnings: false, }, { name: "IPv4 direct access", check: HTTPProxyCheck{ Proto: "https", Host:  "10.96.0.1", }, // Expected to be accessed directly, we set NO_PROXY to 10.0.0.0/8 expectWarnings: false, }, { name: "IPv4 via proxy", check: HTTPProxyCheck{ Proto: "https", Host:  "192.168.0.1", }, // Expected to go via proxy as this range is not listed in NO_PROXY expectWarnings: true, }, { name: "IPv6 direct access", check: HTTPProxyCheck{ Proto: "https", Host:  "[2001:db8::1:15]", }, // Expected to be accessed directly, part of 2001:db8::/48 in NO_PROXY expectWarnings: false, }, { name: "IPv6 via proxy", check: HTTPProxyCheck{ Proto: "https", Host:  "[2001:db8:1::1:15]", }, // Expected to go via proxy, range is not in 2001:db8::/48 expectWarnings: true, }, </ins> } <del> t.Log("http.ProxyFromEnvironment is usable, continue executing test") </del> <ins> // Save current content of *_proxy and *_PROXY variables. savedEnv := resetProxyEnv(t) defer restoreEnv(savedEnv) </ins> for _, rt := range tests { warning, _ := rt.check.Check() if (warning != nil) != rt.expectWarnings { t.Errorf( <del> "failed HTTPProxyCIDRCheck:ntexpected: %tnt  actual: %t (CIDR:%s). Warnings: %v", </del> <ins> "%s failed HTTPProxyCheck:ntexpected: %tnt  actual: %t (Host:%s). Warnings: %v", rt.name, </ins> rt.expectWarnings, (warning != nil), <del> rt.check.CIDR, </del> <ins> rt.check.Host, </ins> warning, ) }

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: When setting an HTTPPROXY with an IPv6 address, the is no HTTPProxy warning. What you expected to happen: There should be a warning that a proxy is being used. How to reproduce it (as minimally and precisely as possible): Use Kubeadm with IPv6 addresses, and IPv6 proxy in the environment. Such as: NOPROXY=2001:420:2c50:2021::/64 Anything else we need to know?: Environment: Kubernetes version (use ): 9 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/assign rpothier
/sig cluster-lifecycle
/area ipv6


Question: } if s.BootstrapKubeconfig != "" { <del> if err := bootstrap.LoadClientCert(s.KubeConfig.Value(), s.BootstrapKubeconfig, s.CertDirectory, nodeName); err != nil { </del> <ins> if err := bootstrap.LoadClientCert(s.KubeConfig, s.BootstrapKubeconfig, s.CertDirectory, nodeName); err != nil { </ins> return err } }

This will be the end of the deprecation period for --require-kubeconfig.
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list .
ref
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale


Question: <ins> .PHONY:	build push TAG = 1.0 build: docker build -t gcr.io/google_containers/fluentd-sidecar-gcp:$(TAG) . push: gcloud preview docker push gcr.io/google_containers/fluentd-sidecar-gcp:$(TAG) </ins>

We should provide a dockerfile for a pre-configured Fluentd image that can be passed file paths to collect logs from. We'll likely want one for exporting to elasticsearch and one for exporting to Google Cloud Logging, as we have today for the per-node stdout/stderr collectors. How to collect log files from within a container is a question that comes up occasionally (including ), and it provides a strong example of the usefulness of sidecar containers. cc
A controller that keeps one instance of a pod alive on each node would be useful for this. Shall I go ahead and write such a controller?
Would it be? This is intended to run as a sidecar container in a pod with the user's actual application. I think a pod-per-node controller is orthogonal (and is being discussed in and ). I'll create the actual Dockerfile, and a quick example of how to use it.
Sorry, my rush to catch up I think I mis-understood this thread.


Question: }) f := framework.NewDefaultFramework("dns") ginkgo.It("should support configurable pod DNS servers", func() { <ins> ginkgo.By("Getting the IP address of the internal Kubernetes service") svc, err := f.ClientSet.CoreV1().Services("kube-system").Get(context.TODO(), "kube-dns", metav1.GetOptions{}) framework.ExpectNoError(err) </ins> ginkgo.By("Preparing a test DNS service with injected DNS names...") <del> testInjectedIP := "1.1.1.1" testSearchPath := "resolv.conf.local" ginkgo.By("Creating a pod with dnsPolicy=None and customized dnsConfig...") testUtilsPod := e2epod.NewAgnhostPod(f.Namespace.Name, "e2e-dns-utils", nil, nil, nil) testUtilsPod.Spec.DNSPolicy = v1.DNSNone testUtilsPod.Spec.DNSConfig = &v1.PodDNSConfig{ Nameservers: []string{testInjectedIP}, </del> <ins> // the default service IP will vary from cluster to cluster, but will always be present and is a good DNS test target testInjectedIP := svc.Spec.ClusterIP testSearchPath := "default.svc.cluster.local" ginkgo.By("Creating a windows pod with dnsPolicy=None and customized dnsConfig...") testPod := e2epod.NewAgnhostPod(f.Namespace.Name, "e2e-dns-utils", nil, nil, nil) testPod.Spec.DNSPolicy = v1.DNSNone testPod.Spec.DNSConfig = &v1.PodDNSConfig{ Nameservers: []string{testInjectedIP, "1.1.1.1"}, </ins> Searches:    []string{testSearchPath}, } <del> testUtilsPod.Spec.NodeSelector = map[string]string{ </del> <ins> testPod.Spec.NodeSelector = map[string]string{ </ins> "kubernetes.io/os": "windows", } <del> testUtilsPod, err := f.ClientSet.CoreV1().Pods(f.Namespace.Name).Create(context.TODO(), testUtilsPod, metav1.CreateOptions{}) </del> <ins> testPod, err = f.ClientSet.CoreV1().Pods(f.Namespace.Name).Create(context.TODO(), testPod, metav1.CreateOptions{}) </ins> framework.ExpectNoError(err) <del> framework.Logf("Created pod %v", testUtilsPod) </del> <ins> ginkgo.By("confirming that the pod has a windows label") framework.ExpectEqual(testPod.Spec.NodeSelector["kubernetes.io/os"], "windows") framework.Logf("Created pod %v", testPod) </ins> defer func() { <del> framework.Logf("Deleting pod %s...", testUtilsPod.Name) if err := f.ClientSet.CoreV1().Pods(f.Namespace.Name).Delete(context.TODO(), testUtilsPod.Name, *metav1.NewDeleteOptions(0)); err != nil { framework.Failf("Failed to delete pod %s: %v", testUtilsPod.Name, err) </del> <ins> framework.Logf("Deleting pod %s...", testPod.Name) if err := f.ClientSet.CoreV1().Pods(f.Namespace.Name).Delete(context.TODO(), testPod.Name, *metav1.NewDeleteOptions(0)); err != nil { framework.Failf("Failed to delete pod %s: %v", testPod.Name, err) </ins> } }() <del> framework.ExpectNoError(e2epod.WaitForPodNameRunningInNamespace(f.ClientSet, testUtilsPod.Name, f.Namespace.Name), "failed to wait for pod %s to be running", testUtilsPod.Name) </del> <ins> framework.ExpectNoError(e2epod.WaitForPodNameRunningInNamespace(f.ClientSet, testPod.Name, f.Namespace.Name), "failed to wait for pod %s to be running", testPod.Name) </ins> <del> ginkgo.By("Verifying customized DNS option is configured on pod...") </del> <ins> // This isn't the best 'test' but it is a great diagnostic, see later test for the 'real' test. ginkgo.By("Calling ipconfig to get debugging info for this pod's DNS and confirm that a dns server 1.1.1.1 can be injected, along with ") </ins> cmd := []string{"ipconfig", "/all"} stdout, _, err := f.ExecWithOptions(framework.ExecOptions{ Command:       cmd, Namespace:     f.Namespace.Name, <del> PodName:       testUtilsPod.Name, </del> <ins> PodName:       testPod.Name, </ins> ContainerName: "agnhost-container", CaptureStdout: true, CaptureStderr: true, }) framework.ExpectNoError(err) framework.Logf("ipconfig /all:n%s", stdout) <del> dnsRegex, err := regexp.Compile(`DNS Servers[s*.]*:(s*[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3})+`) framework.ExpectNoError(err) if dnsRegex.MatchString(stdout) { match := dnsRegex.FindString(stdout) </del> <ins> if !strings.Contains(stdout, "1.1.1.1") { framework.Failf("One of the custom DNS options 1.1.1.1, not found in ipconfig /all") } </ins> <del> if !strings.Contains(match, testInjectedIP) { framework.Failf("customized DNS options not found in ipconfig /all, got: %s", match) } } else { framework.Failf("cannot find DNS server info in ipconfig /all output: n%s", stdout) </del> <ins> // We've now verified that the DNS stuff is injected...  now lets make sure that curl'ing 'wrong' endpoints fails, i.e. // a negative control, to run before we run our final test... ginkgo.By("Verifying that curl queries FAIL for wrong URLs") // the below tests use curl because nslookup doesnt seem to use ndots properly // ideally we'd use the powershell native ResolveDns but, that is not a part of agnhost images (as of k8s 1.20) // TODO @jayunit100 add ResolveHost to agn images cmd = []string{"curl.exe", "-k", "https://kubernetezzzzzzzz:443"} stdout, _, err = f.ExecWithOptions(framework.ExecOptions{ Command:       cmd, Namespace:     f.Namespace.Name, PodName:       testPod.Name, ContainerName: "agnhost-container", CaptureStdout: true, CaptureStderr: true, }) if err == nil { framework.Logf("Warning: Somehow the curl command succeeded... The output was n %v", stdout) framework.Failf("Expected a bogus URL query to fail - something is wrong with this test harness, cannot proceed.") } ginkgo.By("Verifying that injected dns records for 'kubernetes' resolve to the valid ip address") cmd = []string{"curl.exe", "-k", "https://kubernetes:443"} stdout, _, err = f.ExecWithOptions(framework.ExecOptions{ Command:       cmd, Namespace:     f.Namespace.Name, PodName:       testPod.Name, ContainerName: "agnhost-container", CaptureStdout: true, CaptureStderr: true, }) framework.Logf("Result of curling the kubernetes service... (Failure ok, only testing for the sake of DNS resolution) %v ... error = %v", stdout, err) // curl returns an error if the host isnt resolved, otherwise, it will return a passing result. if err != nil { framework.ExpectNoError(err) </ins> } // TODO: Add more test cases for other DNSPolicies. }) })

What would you like to be : a check of output for the dnsPolicy tests Why is this needed: dnsPolicy tests right now are confusing to interpret, bc they grep a regex out of the output of . its great to print the output of but i think its more important to really measure wether DNS is plumbed properly into the container by directly attempting to resolve a hostname, since that is what the definition of implies .
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig windows
/assign
cc


Question: } } <del> func verifyIfKMSTransformersSwapped(t *testing.T, wantPrefix string, test *transformTest) { </del> <ins> func verifyIfKMSTransformersSwapped(t *testing.T, wantPrefix, wantPrefixForEncryptAll string, test *transformTest) { </ins> t.Helper() var swapErr error

transformation expandless | 14m1s Failed; === RUN TestEncryptionConfigHotReload TestEncryptionConfigHotReload When I see this flake, 10-09 is the first one.(But the job history in the testgrid only started from 10-03) some PR that may be related: /sig auth /cc
/cc
/triage accepted
/cc failed two times in recent 3 runs:
/reopen as of this is still .
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: cfg[url] = *cred } } <ins> // add ACR anonymous repo support: use empty username and password for anonymous access cfg["*.azurecr.*"] = credentialprovider.DockerConfigEntry{ Username: "", Password: "", Email:    dummyRegistryEmail, } </ins> return cfg }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Currently we cannot pull image from Azure Container Registry repo which provides anonymous pull access since when do image pull, it will always use service pincipal for .azurecr. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): master Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Others: /sig azure /assign


Question: RestartPolicy: api.RestartPolicyAlways, DNSPolicy:     api.DNSClusterFirst, }, <ins> { // Populate RunAsUser SupplementalGroups FSGroup with minID 0 Containers: []api.Container{{Name: "ctr", Image: "image", ImagePullPolicy: "IfNotPresent"}}, SecurityContext: &api.PodSecurityContext{ SupplementalGroups: []int64{minID}, RunAsUser:          &minID, FSGroup:            &minID, }, RestartPolicy: api.RestartPolicyAlways, DNSPolicy:     api.DNSClusterFirst, }, { // Populate RunAsUser SupplementalGroups FSGroup with maxID 2147483647 Containers: []api.Container{{Name: "ctr", Image: "image", ImagePullPolicy: "IfNotPresent"}}, SecurityContext: &api.PodSecurityContext{ SupplementalGroups: []int64{maxID}, RunAsUser:          &maxID, FSGroup:            &maxID, }, RestartPolicy: api.RestartPolicyAlways, DNSPolicy:     api.DNSClusterFirst, }, </ins> { // Populate HostIPC. SecurityContext: &api.PodSecurityContext{ HostIPC: true,

Help review this, thanks.
Are you asking for better validation on pod creation? I'm not really sure what you mean when you say "tip".
Yes, I mean it's better validate the pod when create. The tip mean the validation info response from server and show by client.


Question: ], embed = [":go_default_library"], deps = [ <ins> "//pkg/cloudprovider/providers/openstack:go_default_library", </ins> "//pkg/util/mount:go_default_library", "//pkg/volume:go_default_library", "//pkg/volume/testing:go_default_library", <ins> "//pkg/volume/util:go_default_library", </ins> "//staging/src/k8s.io/api/core/v1:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/api/resource:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/apis/meta/v1:go_default_library",

We merged to allow configurable volume limits for Cinder. But proposed adding such a limit to as well, among other things this will enable us to support node specific limits more easily - /sig openstack /sig storage /sig scheduling
/assign
cc
/kind feature
This will require updating in-tree Openstack cloudprovider code. are we onboard with that?
What would be the extent of the modification? It must not break current compatibility, and the in-tree provider is scheduled for removal this year. I'm ok with it as long as existing users aren't broken by it.
it should be a change into which would add a new configurable field. It would be backward compatible but would allow admins to configure different limits on a node if they must.
/milestone v1.14 We're entering and this looks relevant to the 1.14 release. Please if I am incorrect


Question: serverCA         *x509.Certificate backdate         time.Duration <ins> userAgentLock   sync.Mutex </ins> expectUserAgent string lock sync.Mutex csr  *certapi.CertificateSigningRequest } <ins> func (s *csrSimulator) SetExpectUserAgent(a string) { s.userAgentLock.Lock() defer s.userAgentLock.Unlock() s.expectUserAgent = a } func (s *csrSimulator) ExpectUserAgent() string { s.userAgentLock.Lock() defer s.userAgentLock.Unlock() return s.expectUserAgent } </ins> func (s *csrSimulator) ServeHTTP(w http.ResponseWriter, req *http.Request) { s.lock.Lock() defer s.lock.Unlock()

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: pull-kubernetes-bazel-test Which test(s) are failing: Test_buildClientCertificateManager Testgrid link: Reason for failure: Anything else we need to know:
/remove-kind failing-test /kind flake /sig scheduling
/sig node


Question: }, Type: corev1.SecretTypeBootstrapToken, Data: map[string][]byte{ <del> bootstrapapi.BootstrapTokenIDKey:               []byte(tokenID), </del> <ins> bootstrapapi.BootstrapTokenIDKey:               []byte(validTokenID), </ins> bootstrapapi.BootstrapTokenSecretKey:           []byte("invalid"), bootstrapapi.BootstrapTokenUsageAuthentication: []byte("true"), }, } <ins> tokenExpiredTime := time.Now().Add(-time.Hour).Format(time.RFC3339) </ins> var expiredBootstrapToken = &corev1.Secret{ ObjectMeta: metav1.ObjectMeta{ Namespace: metav1.NamespaceSystem,

Originally tracked at . After fixing "not include test files in import-boss" in , we have a regression here: is using . We need remove dependencies from integration tests to e2e tests again. /assign /sig testing
/sig auth as per the required change
/kind cleanup /area test
It would be helpful to outline an argument as to why this is the case. I can imagine reasons, but none are mentioned here.
Description updated.
/triage accepted /priority backlog /milestone v1.21 This seems a simple enough fix to land (and thank you for doing this) but marking low enough priority to make it clear we may back out and wait for v1.22 if we don't have bandwidth for this
So while I think we should probably do this anyhow I want to note: This issue lists the reasons: we have no intention to stage e2e (there's been much discussion about this since then) we generally don't want anything non-informational (required jobs) running only sometimes anyhow, it's just not reliable Decoupling these seems like a good idea from a purely code-organization perspective, but the reasons in the previous issue are no longer valid.
/milestone v1.22 The PR didn't land in time. At this point I view this as nice to have. Even if it's low impact, let's wait until next release.


Question: }, expectArgs: []string{ "ginkgobin", "--p", <del> "--focus=", "--skip=[Serial]", </del> <ins> "--focus=", "--skip=[Serial]", </ins> "--noColor=true", "testbin", "--", "--disable-log-dump", "--repo-root=/kubernetes", "--provider=", "--report-dir=", "--kubeconfig=",

What happened: When using sonobuoy to run some community tests, I realized that ginkgo was not invoked properly. More precisely, when invoking sonobuoy like this: I saw the following in the sonobuoy logs: Because I am running the tests in parallel, and sonobuoy does not set , the code defaults to (see ) for , and is set to . This is not the intended regex (should be ) and all the tests are skipped (because all test names have an ). What you expected to happen: should be set to so that only the tests with the "tag" are skipped. How to reproduce it (as minimally and precisely as possible): Use the sonobuoy command provided above. Environment:
I opened a PR for this
/sig testing


Question: import ( "context" <ins> "encoding/json" </ins> "testing" "time" "github.com/google/go-cmp/cmp" <ins> "github.com/google/go-cmp/cmp/cmpopts" </ins> v1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/labels" "k8s.io/apimachinery/pkg/util/sets" <ins> "k8s.io/apimachinery/pkg/util/strategicpatch" </ins> "k8s.io/apimachinery/pkg/util/wait" utilfeature "k8s.io/apiserver/pkg/util/feature" "k8s.io/client-go/informers"

We should enhance the to verify that adding is unit-testable. We originally thought that this would only be possible with finalizers but there looks to be other options. We could follow examples from In podgc, if we discover an orphaned pod (pod on a missing node), we add a DisruptionCondition to the pod. We only test this behavior in the integration tests because we previously rely on finalizers to halt the deletion so we can avoid flaky tests.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/kind cleanup /sig apps
/good-first-issue /help
Will take a look /assign
/assign
/help
newbie question please, I couldn't find the file in the codebase or the PRs. Where can I read it?
Working on it now!
/assign
/assign
Has done very good work with this so I’m not sure we will be able to accept other PRs. We are waiting for an approval on the PR
is this issue still open
Is the issue still open or awaiting for the approval on the pr?
/remove help /remove good-first-issue
The issue is waiting for approval. I removed the tags to reflect that is is almost done.


Question: <-stoppedChan finished := make(chan interface{}) <ins> fakePlugin.Lock() </ins> fakePlugin.UnmountDeviceHook = func(mountPath string) error { // Act: // 3. While a volume is being unmounted, add it back to the desired state of world

Which jobs are flaking: Which test(s) are flaking: periodic-kubernetes-bazel-test-master (xref ) Testgrid link: Reason for failure: Anything else we need to know: to reproduce with : /sig storage
/assign
solved at
still flaking - /reopen
Reopened this issue. <detailsIn response to : still flaking - Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
-- No recent failures in the past 2 weeks, are you okay if we close this?
yeah, looks like resolved it /close
Closing this issue. <detailsIn response to : yeah, looks like resolved it Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
it's back not sure if related to this issue but same test with a data race spotted last night looking into filed


Question: runnningNode: "node-1", }, { <del> name: "nominated node cannot pass all the filters, preemptor should find a different node", nodes: []*nodeConfig{ {name: "node-1", res: defaultNodeRes}, {name: "node-2", res: defaultNodeRes}, }, </del> <ins> name:      "nominated node cannot pass all the filters, preemptor should find a different node", nodeNames: []string{"node-1", "node-2"}, </ins> existingPods: []*v1.Pod{ initPausePod(&pausePodConfig{ <del> Name:      "low-pod1", Namespace: testCtx.NS.Name, </del> <ins> Name:      "low-pod", </ins> Priority:  &lowPriority, Resources: defaultPodRes, NodeName:  "node-1", }), }, pod: initPausePod(&pausePodConfig{ <del> Name:      "preemptor-pod", Namespace: testCtx.NS.Name, Priority:  &highPriority, </del> <ins> Name:     "preemptor-pod1", Priority: &highPriority, </ins> Resources: &v1.ResourceRequirements{Requests: v1.ResourceList{ v1.ResourceCPU:    *resource.NewMilliQuantity(500, resource.DecimalSI), v1.ResourceMemory: *resource.NewQuantity(200, resource.DecimalSI)},

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Integration tests are sometimes flaking locally: PreferNominatedNode Once the node gets deleted, the nodelifecycle controller is racing to update pod status and the pod deletion logic is failing causing tests to flake. Ultimately, we call testCtx.CleanUp() which cleans the nodes and the pods involved the test namespace so we are not going to leak any resources. links to appreciated links to specific failures in spyglass appreciated <!-- You can identify the SIG from the "prowjobconfigurl" on the testgrid dashboard for a test --/sig scheduling <!-- Please see the deflaking doc () for more guidance! --/assign
cc Tried addressing it via
/triage accepted thanks!


Question: "Metric":    gstruct.Ignore(), "Value":     gomega.BeNumerically(">", value), "Timestamp": gstruct.Ignore(), <ins> "Histogram": gstruct.Ignore(), </ins> })) }

There's been a quite a few failures across jobs that query node metrics. The test is Last passing - First failing - appears to point to which is bump in prometheus client library. Possibly related? /kind failing-test <!-- If this is a flake, please add: /kind flake --/sig node
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Verified with presubmit that reverting the prometheus client library results in the test passing. Opened PR for revert in
/cc
/assign
thanks for filing this issue! I think seems like a promising fix (thanks ) and there's a fair chance we won't need the revert
ack, thanks for the fix, it seems we can go with that then and hold the revert.


Question: Weight int32 } <ins> // ExtenderName is a fake plugin name put in UnschedulablePlugins when Extender rejected some Nodes. const ExtenderName = "Extender" </ins> // Diagnosis records the details to diagnose a scheduling failure. type Diagnosis struct { NodeToStatusMap NodeToStatusMap

When Extender filters out some Nodes, we don't set any unschedulable plugins at all. It means Extender is completely ignored during the requeueing process. So, what's happening is: If Extender filters out all Nodes during scheduling, this Pod is soon retried because this Pod doesn't have any plugin name in unschedulable plugins. If Extender filters out some Nodes during scheduling and plugins filter out all other Nodes, this Pod is retried based on plugins' QueueingHint. Even if any cluster events happen and it could change Extender's decision (but not any plugins' decision), this Pod won't be requeued to activeQ/backoffQ. The latter case is serious because it could make Pods being stuck in unschedulable pod pool in 5min. We should have a short-term solution for the latter case. We can requeue Pods, which were rejected by Extender, by any kind of cluster events because we cannot know which events make Pods schedulable. Eventually, this issue makes us wonder how Pods rejected by Extender should be requeued. We cannot implement QHint equivalent in Extenders because it'd be too slow to call Extender every time any cluster events happen. Probably, somehow implementing EventsToRegister equivalent in Extender? Use Extender which does something in Filter. No response master <detailsn/a </details<details</details<details</details<details</details<details</details
/assign
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scheduling
/reopen We'll close it when all cherry-picks are done.
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/close Cherry-picks are done.


Question: volumeMounts: - name: kube-dns-config mountPath: /kube-dns-config <ins> securityContext: allowPrivilegeEscalation: false readOnlyRootFilesystem: true runAsUser: 1001 runAsGroup: 1001 </ins> - name: dnsmasq image: k8s.gcr.io/k8s-dns-dnsmasq-nanny:1.14.13 livenessProbe:

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Apply security best practices for both kube-dns and core-dns. Low hanging fruit: [x] Run as non-root & disallow privilege escalation [x] Run with the default seccomp profile (i.e. ) [x] ReadOnlyRootFilesystem [x] Avoid unnecessary HostPath volumes [x] (n/a) More advanced: [ ] Reduce dependencies in the base image () [ ] Custom seccomp profile [ ] Custom AppArmor profile [x] Drop unneeded capabilities[x] Run as non-root & disallow privilege escalation [x] Run with the default seccomp profile (i.e. ) [x] ReadOnlyRootFilesystem [x] Avoid unnecessary HostPath volumes [x] (n/a) More advanced: [ ] Reduce dependencies in the base image () [ ] Custom seccomp profile [ ] Custom AppArmor profile [x] Drop unneeded capabilities This is based on the long term issue and implemented by PR . Why is this needed: Our system pods should run using security best practices, both to enhance cluster security and serve as examples of best practices to users.
/sig auth /area security


Question: <ins> /* Copyright 2015 Google Inc. All rights reserved. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package e2e import ( "fmt" "path" "github.com/GoogleCloudPlatform/kubernetes/pkg/api" "github.com/GoogleCloudPlatform/kubernetes/pkg/client" "github.com/GoogleCloudPlatform/kubernetes/pkg/util" . "github.com/onsi/ginkgo" ) var _ = Describe("emptyDir", func() { var ( c         *client.Client podClient client.PodInterface ) BeforeEach(func() { var err error c, err = loadClient() expectNoError(err) podClient = c.Pods(api.NamespaceDefault) }) It("volume on tmpfs should have the correct mode", func() { volumePath := "/test-volume" source := &api.EmptyDirVolumeSource{ Medium: api.StorageTypeMemory, } pod := testPodWithVolume(volumePath, source) pod.Spec.Containers[0].Args = []string{ fmt.Sprintf("--fs_type=%v", volumePath), fmt.Sprintf("--file_mode=%v", volumePath), } testContainerOutput("emptydir r/w on tmpfs", c, pod, []string{ "mount type of "/test-volume": tmpfs", "mode of file "/test-volume": dtrwxrwxrwx", // we expect the sticky bit (mode flag t) to be set for the dir }) }) It("should support r/w on tmpfs", func() { volumePath := "/test-volume" filePath := path.Join(volumePath, "test-file") source := &api.EmptyDirVolumeSource{ Medium: api.StorageTypeMemory, } pod := testPodWithVolume(volumePath, source) pod.Spec.Containers[0].Args = []string{ fmt.Sprintf("--fs_type=%v", volumePath), fmt.Sprintf("--rw_new_file=%v", filePath), fmt.Sprintf("--file_mode=%v", filePath), } testContainerOutput("emptydir r/w on tmpfs", c, pod, []string{ "mount type of "/test-volume": tmpfs", "mode of file "/test-volume/test-file": -rw-r--r--", "content of file "/test-volume/test-file": mount-tester new file", }) }) }) const containerName = "test-container" const volumeName = "test-volume" func testPodWithVolume(path string, source *api.EmptyDirVolumeSource) *api.Pod { podName := "pod-" + string(util.NewUUID()) return &api.Pod{ TypeMeta: api.TypeMeta{ Kind:       "Pod", APIVersion: "v1beta1", }, ObjectMeta: api.ObjectMeta{ Name: podName, }, Spec: api.PodSpec{ Containers: []api.Container{ { Name:  containerName, Image: "kubernetes/mounttest:0.1", VolumeMounts: []api.VolumeMount{ { Name:      volumeName, MountPath: path, }, }, }, }, Volumes: []api.Volume{ { Name: volumeName, VolumeSource: api.VolumeSource{ EmptyDir: source, }, }, }, }, } } </ins>

I'd like to commit without e2e support so others can unblock. We need to come back to get e2e on this. I could use help, and it seems like an easy test...
Sorry for my incorrect PR. Now I have amend it.


Question: "strings" "testing" <ins> "github.com/spf13/cobra" </ins> "github.com/stretchr/testify/assert" )

When trying to use kubectl from hyperkube I'm met with "unknown flag" errors. It looks as if the flags aren't being passed through to kubectl but hyperkube is trying to handle them itself (and subsequently giving an error when you give it flags it doesn't understand). You can reproduce simply by running: And you'll see the error:
I have a fix up in PR .


Question: type defaultFunc func(pod *api.Pod) error func tryDecodeSinglePod(data []byte, defaultFn defaultFunc) (parsed bool, pod *api.Pod, err error) { <del> obj, err := api.Scheme.Decode(data) </del> <ins> // JSON is valid YAML, so this should work for everything. json, err := utilyaml.ToJSON(data) if err != nil { return false, nil, err } obj, err := api.Scheme.Decode(json) </ins> if err != nil { return false, pod, err }

Following on from the Kubelet should accept YAML pod specifications. Currnelty it supports only JSON pod specifications.


Question: claim := newClaim(t, ns.GetName(), "") class := newStorageClass(t, ns.GetName(), "") <ins> claim.Spec.StorageClassName = &class.ObjectMeta.Name </ins> testDynamicProvisioning(t, cs, claim, class) }) })

Current CSI E2E test has some issues which prevent it to run successfully the a local cluster. /sig storage /kind bug /assign


Question: "docker_legacy_service.go", "docker_logs.go", "docker_sandbox.go", <del> "docker_sandbox_others.go", "docker_sandbox_windows.go", </del> "docker_service.go", "docker_stats_linux.go", "docker_stats_unsupported.go",

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: is trying to setup DNS for Windows containers, so that Pods on Windows could confirm to dns policies. However, as posted Docker is not managing the networking here for windows. So it doens't really care about any network settings. So passing it to docker shim's hostconfig also doens;t make sense here. DNS for Windows containers will be set by CNI plugins. So PR reverts . This issue tracks the process of pushing DNS configure into CNI and passed that from Kubelet to container runtime. What you expected to happen: Pods on Windows confirm to dns policies. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: /sig windows
Need a linked issue & PR in CNI repo too so that capability arguments can be validated from kubelet
Fixed in by dns capabilities, Azure CNI has already supported this. /close
Closing this issue. <detailsIn response to : Fixed in by dns capabilities, Azure CNI has already supported this. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: import ( "context" <ins> "crypto/ecdsa" "crypto/elliptic" "crypto/rand" </ins> "crypto/rsa" "crypto/x509" <ins> "crypto/x509/pkix" "encoding/pem" </ins> "fmt" <ins> "math" "math/big" </ins> "net" "os" "path/filepath"

found this by running tests on this PR: it's problematic for a couple of reasons: prevents kubeadm from moving out (eventually) there are no API guarantees in the given code/structs/functions in pkiutils the logic in cmd/kube- must be reworked to generate cert/key pair without using kubeadm. originally in pending fix PR
/sig api-machinery cluster-lifecycle architecture /area code-organization
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /sig api-machinery sig-cluster-lifecycle architecture /area code-organization </details
/assign
cc
Please make sure to add an import restriction to prevent a regression in the future.
/triage accepted


Question: string(api.AzureDedicatedBlobDisk), string(api.AzureManagedDisk)) <del> supportedStorageAccountTypes = sets.NewString("Premium_LRS", "Standard_LRS") </del> <ins> supportedStorageAccountTypes = sets.NewString("Premium_LRS", "Standard_LRS", "Standard_GRS", "Standard_RAGRS") </ins> ) func getPath(uid types.UID, volName string, host volume.VolumeHost) string {

Is this a BUG REPORT or FEATURE REQUEST?: What happened: Now for storage class only support skuName with PremiumLRS and StandardLRS Met error if the storage account was created with other sku. But actually it can work well without setting it What you expected to happen: skuName should support others if the storage account support it How to reproduce it (as minimally and precisely as possible): Create a storage account which is using StandardRAGRS sku. Create storage class as below: Create pvc as below PVC would never bind with below error: Failed to provision volume with StorageClass "scazure": azureDisk - StandardRAGRS is not supported sku/storageaccounttype. Supported values are [PremiumLRS StandardLRS] But if I remove the skuName parameter, the pvc could be bound Anything else we need to know?: Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.6+", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-11-12T02:22:57Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.6+", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-11-12T02:22:57Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): RHEL7 Install tools: ansible
/assign andyzhangx
/sig azure
thanks, it's a requirement, we should support StandardLRS, StandardZRS, StandardGRS, StandardRGRS, while for premium, it only support Premium_LRS
/kind feature
StandardZRS, PremiumGRS are not supported for azure disk now


Question: }}, }}) <del> p := CreateProxier(lb, net.ParseIP("0.0.0.0"), &fakeIptables{}, net.ParseIP("127.0.0.1")) </del> <ins> p := createProxier(lb, net.ParseIP("0.0.0.0"), &fakeIptables{}, net.ParseIP("127.0.0.1")) </ins> waitForNumProxyLoops(t, p, 0) svcInfoP, err := p.addServiceOnPort(serviceP, "TCP", 0, time.Second)

If another process is altering iptables rules at the time k8s attempts to add iptables rules it will fail with a locking error. k8s should use -w and retry after a delay. Below, an external process (openshift-sdn-node) modifies iptables at roughly the same time and k8s fails without retrying. Related to
Sorry, I misunderstood the -w flag, with it retry logic wouldn't be necessary as it'd wait. Perhaps simply retrying is a better solution as -w flag may not exist everywhere we'd like.
Looks like -w was to iptables 1.4.20(August 2013).
Yeah, we have to be pretty back-compatible with iptables. We always retry after a few seconds - is there really a bug here? , Scott Dodson wrote:
It definitely seems like there is some sort of a problem. We see this randomly on system startup. In our environment OpenShift is launching kubernetes around the same time other systemd units are modifying the firewall config. If one of these others units has the lock kubernetes will report the error Scott mentions and then will never create the required nat'ing for services. Here's how to reproduce this: At this point if you run you will see the exact same error kubernetes reports. It's probably not obvious why you have to set the breakpoint at 257 but if you set the breakpoint at 260 and then step through the code it appears that the compiler has optimized lines 263 and 257 to be the same. In any case running will verify if you have the lock or not. Once you've verified you have the lock just restart OpenShift (running kubernetes v0.14.1-582-gb12d75d): At this point you can press in gdb to continue the iptables invocation and release the lock. Afterwards I will see output from cadvisor every once in a while, however the will show the firewall is still blank: I initially restarted Kubernetes with a blank firewall at 11:21 and yet at 11:38 I still have nothing:
I probably should have made it more clear, I removed the lock by pressing 'c' in gdb to continue. That's when I started waiting. I'll update my comment to make that more clear.
So something for a holding the lock?

I am out of your office for the rest of this week, but maybe you can figure out - is there a stale process holding a lock? What kind of lock is it? Flock or semaphore or ... ? On Apr 29, 2015 8:44 AM, "Brenton Leanhardt" wrote:
What we're saying is that on startup it's fairly common for multiple systemd units to make firewall additions. If one of them grabs the lock (rarely for more than a few seconds) it will cause kubernetes to fail to initialize iptables and it will never fix itself. The lock only need be grabbed the microsecond kubernetes tries to initialize iptables for this bug to present itself. We know exactly which program in our environment was concurrently making iptables updates. The problem is that kubernetes doesn't appear to be retrying if it fails to initialize iptables.
On RHEL the lock is the xtables unix socket lock in iptables to allow multiple programs to update iptables concurrently. Other platforms may use a flock since that is the default upstream. To solve this kubernetes needs one more more of the following: pass the flag to as scott and the logs mention (possibly only available on some platforms). If kubernetes desides to wait for the lock it will need a timeout. retry the firewall initialization quickly and maybe backoff if it still can't get the lock It's probably worth noting that if iptables on a platform doesn't have iptables lock support concurrency is likely broken already in other ways for that platform.
It does retry all up tables ops. Unless it somehow gets pathologically phase locked with your agent, or there is some deeper bug. We have e2e tests that prove you can delete tables and it will recreate them. Are you saying that it never retries (did you set -v=4?) or that every retry fails? On RHEL the lock is the xtables unix socket lock in iptables to allow multiple programs to update iptables concurrently. Other platforms may use a flock since that is the default upstream. To solve this kubernetes needs one more more of the following: pass the -w flag to iptables as scott and the logs mention (possibly only available on some platforms). If kubernetes desides to wait for the lock it will need a timeout.

retry the firewall initialization quickly and maybe backoff if it still can't get the lock It's probably worth noting that if iptables on a platform doesn't have iptables lock support concurrency is likely broken already in other ways for that platform. — Reply to this email directly or view it on GitHub .
I can grab the lock for just the second that kubernetes needs it and then let it go. The kube-proxy nat'ing will never be rebuilt without intervention (restarting kubernetes). That seems like a bug to me and is what my original comment reproduces. In addition, once the environment is in this state it won't even pick up new service nat rules. Other nodes in my environment will pick them up but this one is dead. I highly suspect it has to do with the fact that iptables initialization is failing. We test all the time that once a system is up and running you can clean iptables and it will come back. This is a slightly different case.
This seems like a v1 blocker - we see this in every realistic environment we put the Kube-proxy in. Disagreement?
I still don't understand this issue. It is demonstrably true that the kube-proxy retries all of the iptables logic periodically. It's possible (given the mention of "won't even pick up new service nat rules") that this locking tickles a bug in kube-proxy, but I am having trouble reproducing it. Can you run your kube-proxy with -v=4, trigger the bug, and then take a look at the logs? If this is real, yeah it's a 1.0 blocker.
The problem is in the initialization process. If you hold the lock while the kube-proxy tries to initialize itself then it will be wedged forever. What you're showing is the kube-proxy can repair the iptables rules after the initialization code has run. The logs I pasted earlier show that after the initialization fails at it will never try again. I tend to run at loglevel 4 by default but I'll try to find time today to run through this again.
I have a fix coming - can you test the PR before commit?
I ran the exact same test as mentioned in my original comment.

While the debug lines have changed the end result is the same: grab the lock (at this point even will fail) start the kube-proxy release the lock (at this point works) wait.. try creating another service wait.. The rules never come back. Here are the logs:
Sure, I'll test the PR. I'm betting it will apply fairly cleanly on to my OpenShift codebase. If not I'll launch Kubernetes directly.
Many of those log lines are not part of upstream. In fact, what upstream has is a glog.Fatalf() which will kill kube-proxy, and when it restarts it will try again. Did you guys break it by not actually exiting?
Looks like we did. We need to put it into a restart loop. ----- Original Message -----
is teh fix, though I assert we should not need it, and if you need it this fix may not help you anyway. I am still OK to commit this.
Although NewProxier sucks because it doesn't return a usable error. I'll take the todo to clean up that part. ----- Original Message -----
hey, it logs its error - nothing wrong with that :) , Clayton Coleman wrote:
:)
Opened and will move the rest back to openshift.
I restarted docker and it worked then


Question: srcs = [ "helpers_test.go", "unstructured_list_test.go", <ins> "unstructured_test.go", </ins> ], embed = [":go_default_library"], deps = [

What happened: In I introduced an optimization - do a deep copy instead of to and from JSON roundtrip to convert anything that implements . I then discovered that the method that is used there in both and may mutate the original object. This is problematic because previously (before ) there was no mutation and because this is unexpected and may lead to data races - it is bad behaviour to mutate original object when you just want a copy of it. This issue was fixed by . However the underlying issue is not fixed here - is brittle and dangerous. Method name does not imply that it mutates data when you call it. And godoc does not mention that either: Something needs to be done about it IMO. This method is misused in these places (mutation side effect is not intended): Also implementation in does not implement the behaviour required by godoc in . What you expected to happen: There should be a method that returns the current unstructured contents without mutation and perhaps not providing the write-through effect. /kind bug /sig api-machinery
/cc
Is this still required and could I work on it? I'm looking for something to work on as a new contributor :)
Yes and yes! :)
Perfect, I'll get started on it!
I've got out. I'd appreciate a review :)
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: } // If volumes have not been unmounted/detached, do not delete directory. // Doing so may result in corruption of data. <ins> // TODO: getMountedVolumePathListFromDisk() call may be redundant with // kl.getPodVolumePathListFromDisk(). Can this be cleaned up? </ins> if podVolumesExist := kl.podVolumesExist(uid); podVolumesExist { klog.V(3).Infof("Orphaned pod %q found, but volumes are not cleaned up", uid) continue

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: Network for all nodes of kubernetes cluster (29 hosts) was turned off for a day. Once the network was brought back, we've scaled down a statefulset that mounts a from local volume which on a root drive and then scaled it back up to previous number of replicas. kubelet on 26 out of 29 hosts proceeded with removing all data from subpaths of the pod. In fact, it was ongoing, such that when we'd create a file on a mounted subpath within the container, the file would be cleaned up within a couple of seconds. The issue would continue until volume-subpath was manually unmounted (with command). Ended up removing 3rd of all of our data (we have 3 drives per host and one was wiped) for the whole cluster. Equivalent definition. Note that host path of local volume has to be on root drive: Kubelet logs (filtered out logs from other pods) during the issue: What you expected to happen: Expected no data loss for peristent local volumes. How to reproduce it (as minimally and precisely as possible): Unfortunately was not able to reproduce. But this bug hit 26 out of 29 nodes simultaneously. Need some way to create a case where a pod becomes orphan and "volumes" folder gets somehow removed (similar to issue ). Then there is a code path that can lead to data loss: We are in orphan pod cleanup and directory is missing (not sure why it's printed twice): No error returned and empty volumes list, so we go into this case which proceeds removing pod directory and along with it even thought they might still be mounted (not sure why): checks if device is the same, but we use root drive for local volume, so the device is the same, so it's not a mount point so ok for removing (this is a bug IMHO). Anything else we need to know?: k8s version upgrades went from 1.9.6 -1.11.1 -1.12.1 -1.12.2 -1.13.0 without restarting the pods.

Environment: Kubernetes version (use ):Cloud provider or hardware configuration: OVH, baremetal servers OS (e.g. from /etc/os-release):Kernel (e.g. ):Install tools: Using Ansible tasks that downloads kubelet from and all other required tools. Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
Thank you for digging into the code. I think there are a number of things that can be done here: why returned no volumes. why base volume was successfully unmounted. Unmounting the volume should have failed because the subpath bind mount would have kept a reference on it. to also look for the volume-subpaths directory to detect bind mounts.
I am working on a fix for 3 and 4
, do you mind to share us the full kubelet log? You can email me jinxu at Thanks!
why podVolumesExist returned no volumes. From the available log, we can see that Unmount volume and Unmount device succeeded. That's why podVolumeExist will return no volume.
/priority critical-urgent
That's unexpected. The subpath bind mount should have prevented the local volume bind mount from successfully unmounting.
Maybe the fact that the local volume mount is itself a bind mount could be the reason
about to why podVolumesExist returned no volumes. I think we do have an issue here podVolumesExist checks pod volume paths and whether they are mounted. Suppose it should stop cleaning up if there are paths still mounted. However, it only checks normal pod volumes /pod-id/volumes, it does not check subPath /pod-id/volume-subpath directories. We should fix this.
I'm adding a check for volume-subpath directories to address 3)
Jing found that the subpath volume directory uses the outer volume spec name, which is wrong. We made a fix for this in 1.11.0, 1.10.0, 1.9.7, 1.8.11, 1.7.16, which changed the on-disk directories to use the inner volume spec name.

had an that in-place node updates were not supported and you would have to drain your pods first. This explains why the volume cleanup proceeded without waiting for the subpath mount cleanup.
We haven't run a version below 1.9.6 on this node.
I've sent the entire kubelet log to your email
to clarify, 1.9.6 uses the old on-disk directory name, so upgrading to anything at least 1.11.0, 1.10.0, 1.9.7, 1.8.11, 1.7.16 requires pod draining first.
Right, I remember stumbling upon when debugging this. However, the release notes say: Shouldn't it specify exact version range where the OuterVolumeSpecName was used (1.9.4-1.9.6)? Unless I'm misreading release notes. I think it would be useful to clarify that.
ah you're right. The release notes should say from at least versions 1.7.14, 1.8.9, and 1.9.4
(will update it)
regardless, I still intend to add more safety checks to this orphaned pod cleanup function
From the log provided by the customer, I think we can confirm the reason of this issue is what mentioned By adding the safety checks during pod cleanup will help avoid this issue.


Question: backoffManager wait.BackoffManager // initConnBackoffManager manages backoff the initial connection with the Watch call of ListAndWatch. initConnBackoffManager wait.BackoffManager <ins> // MaxInternalErrorRetryDuration defines how long we should retry internal errors returned by watch. MaxInternalErrorRetryDuration time.Duration </ins> resyncPeriod time.Duration // ShouldResync is invoked periodically and whenever it returns `true` the Store's Resync operation is invoked

When etcd is unable to elect leader for 3 election cycles, etcd returns "no leader" error through watches. This can be seen in api-server logs as . Also, when apiserver receives this error, it terminates all watchers: . In large clusters, this can destabilize master, as all these watchers need to be recreated almost at the same time, overwhelming api-server quite often. I propose adding new api-server flag that allows to specify timeout, during which we are retrying to reestablish apiserver <-etcd watches, without terminating all api-server watchers. If we are unable to reestablish watches within this specified timeout, we start terminating all watchers. Losing etcd leader for ~2-3s does not destabilize api-server. Trigger prolonged reelection of etcd leader. No response Observed in all kubernetes versions 1.19-1.24 N/A <details</details<details</details<details</details<details</details
/sig scalability /sig api-machinery /priority important-soon
/wg reliability
/assign
/triage accepted
indeed, this is a source of instability. Is this coming from ?
I believe it's coming from this change. As I mentioned, the idea to mitigate it is to add some parameter that will allow us to ignore this error coming from etcd for specific duration without terminating all watchers, for example 10s be default.
With this comment thread in mind: It would probably be cleaner to allow for simply setting the "timeout" as part of WithRequireLeader option (instead of using the constant of 3). But that would require etcd changes, including the API and we want to workaround that in much smaller timeframe. But a comment about how to make that cleaner with some additional context from here would help.


Question: -o -wholename '*/Godeps/*'  -o -wholename '*/codecgen-*-1234.generated.go'  ) -prune  <del> ) -name '*.generated.go')) </del> <ins> ) -name '*.generated.go' | sort -r)) </ins> # Register function to be called on EXIT to remove codecgen # binary and also to touch the files that should be regenerated

See, e.g. This seems to be blocking multiple PRs from even being tested.
Looking.
I think I figured it out. The order of generation matters. The verify script subtly breaks that by renaming and checking the files one at a time. The update script deletes everything and then recreates them.
I have a fix. Will send a PR shortly.


Question: glog.Errorf("unexpected copy error: %v", err) return } <ins> var watchEvent watch.Event </ins> switch { case curObjPasses && !oldObjPasses: <del> c.result <- watch.Event{Type: watch.Added, Object: object} </del> <ins> watchEvent = watch.Event{Type: watch.Added, Object: object} </ins> case curObjPasses && oldObjPasses: <del> c.result <- watch.Event{Type: watch.Modified, Object: object} </del> <ins> watchEvent = watch.Event{Type: watch.Modified, Object: object} </ins> case !curObjPasses && oldObjPasses: <del> c.result <- watch.Event{Type: watch.Deleted, Object: object} </del> <ins> watchEvent = watch.Event{Type: watch.Deleted, Object: object} } select { case c.result <- watchEvent: // don't block on c.result if c.done is closed case <-c.done: </ins> } }

We noticed many goroutines stalled at in a large cluster (~200 nodes, ~1100 pods). It's because if the WatchServer timeout when the channel is full, will be blocked on the write to the even though is , because WatchServer stops consuming the channel after the timeout.
Great finding - thanks for debugging


Question: &api.NodeDaemonEndpoints{ KubeletEndpoint: api.DaemonEndpoint{Port: int(kc.Port)}, }, <ins> kc.OOMAdjuster, </ins> ) if err != nil { return nil, nil, err

The type has a field called to indicate that a container must run as non-root user. The description for this field includes the sentence , which would mean that setting the field to or to (unset) have the same effect: However, the PodSecurityPolicy provider implementation handles the (unset) case differently, and thus does differentiate between unset or values: The API type documentation should be updated to clarify this behaviour and reflect the actual PodSecurityPolicy implementation. /sig node
Thanks for reporting this issue. I think that would be best able to assist with this. /sig auth /remove-sig node
The pod documentation is correct. If the field is not set or is false, the enforcement is not performed by the kubelet. However, podsecuritypolicy with a runasuser strategy of mustrunasnonroot will force a pod that has not specified a runAsNonRoot setting or a runAsUser uid to runAsNonRoot: true The documentation on the MustRunAsNonRoot strategy of podsecuritypolicy should clarify the mutation behavior of the PSP /remove-kind bug /kind documentation
Can I take this issue?
^ this is something that can be changed.
Here's a proposed change to the documentation in : Is this sufficient to solve this issue? If so, should I submit a PR there?
I'd change that to to be a little more complete.
/assign See PR for docs at
GitHub didn't allow me to assign the following users: josiahbjorgaard. Note that only and repo collaborators can be assigned and that issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign See PR for docs at Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
~This would have saved me some battling today! I couldn't work out why my pod wasn't selecting my run-as-root pod security policy and ignoring my restricted pod security policy - I didn't realise I had to explicitly set to get the correct PSP to match (I thought the restrictive PSP would reject the pod and then the run-as-root PSP would match, not that the restrictive PSP would mutate the PodSpec)~ It would have had I read the actual docs better sigh.


Question: AssumeFunc: func(pod *v1.Pod) { gotAssumedPod = pod }, <ins> IsAssumedPodFunc: func(pod *v1.Pod) bool { if pod == nil || gotAssumedPod == nil { return false } return pod.UID == gotAssumedPod.UID }, </ins> } s := &Scheduler{

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Similar problem as , howerver in our case, pod's annotation was updated at scheduling cycle, so it had not been assumed when its annotation was updated. We also need to deal with this case. Detailed steps: a pod . tried to schedule the pod. Then it was prediated by a filter plugin that we write. This plugin modified the pod's annotation to record something. Because pod's annotation changed, scheduuler received a pod update event, and put the pod to the schedule queue again. scheduled the pod successfully for this schedule cycle. tried to schedule the updated pod again(it was put to the schedule queue at step 2). If the pod was scheduled sucessfully, we would see the error log() in scheduler. If scheduler failed to schedue the pod, scheduler would send a failed scheduler event to pod. They are both confusing because the pod had already been successfully. ! Why is this needed: It will be better to handle pod annotation updates during scheduling more gracefully. /kind cleanup /sig scheduling /assign
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : <!-- Please only use this template for submitting enhancement requests --1. Created a pod . Scheduler tried to schedule the pod. Then it was prediated by a filter plugin that we write. This plugin modified the pod's annotation to record something. Because pod's annotation changed, scheduuler received a pod update event, and put the pod to the schedule queue again. Scheduler scheduled the pod successfully for this schedule cycle. Scheduler tried to schedule the updated pod again(it was put to the schedule queue at step 2). If the pod was scheduled sucessfully, we would see the error log() in scheduler. If scheduler failed to schedue the pod, scheduler would send a failed scheduler event to pod. They are both confusing because the pod had already been successfully. /sig scheduler /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: }, { "ImportPath": "gopkg.in/gcfg.v1", <del> "Comment": "v1.0.0", </del> "Rev": "083575c3955c85df16fe9590cceab64d03f5eb6e" }, { "ImportPath": "gopkg.in/gcfg.v1/scanner", <del> "Comment": "v1.0.0", </del> "Rev": "083575c3955c85df16fe9590cceab64d03f5eb6e" }, { "ImportPath": "gopkg.in/gcfg.v1/token", <del> "Comment": "v1.0.0", </del> "Rev": "083575c3955c85df16fe9590cceab64d03f5eb6e" }, { "ImportPath": "gopkg.in/gcfg.v1/types", <del> "Comment": "v1.0.0", </del> "Rev": "083575c3955c85df16fe9590cceab64d03f5eb6e" }, { "ImportPath": "gopkg.in/inf.v0", <del> "Comment": "v0.9.0", </del> "Rev": "3887ee99ecf07df5b447e9b00d9c0b2adaa9f3e4" }, {

Is this a BUG REPORT or FEATURE REQUEST? (choose one): bug report Kubernetes version (use ): Environment: Cloud provider or hardware configuration: vsphere OS (e.g. from /etc/os-release):Kernel (e.g. ): 4.9.24-coreos Install tools: Others: What happened: After bringing a cluster up, there are no container metrics in the kubelet at /metrics What you expected to happen: I expect to see the metrics, but there are none How to reproduce it (as minimally and precisely as possible): Anything else we need to know: If I try and access the metrics in cAdvisor directly on port 4194 instead of on port then I see the following type of error for all of the metrics:
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
/sig node
This appears to be a v1.7 issue. If I run 1.6.6 I get these metrics. Seems like something has happened in 1.7. I'm running the 1.7 beta because I need nodePort for cni, so now I'm stuck.
Can you triage the issue since?
cc/
Can you help verifying is this is a new regression introduced in 1.7 cadvisor through node-e2e? Thanks!
I've tried this on both cos and cvm in 1.7 cluster, and no container metrics show up. However, I created a 1.6.4 GKE cluster, and container metrics are there. This is a regression in 1.7. Will figure out what is wrong here.
Here is a corresponding cadvisor issue
Even the user still can get the container metrics / stats from kubelet /summary api (recommended way), but since this is a regression introduced in 1.7 release, and we aren't dropping prometheus support, let's fix this for 1.7. Thanks!
Fixed in Will close this issue after building new cadvisor release and cherrypick into 1.7
Thanks for the quick find everyone! I'll be happy to give this a go on the next release!

I'm still facing this same issue. My kubernetes cluster is deployed with kubeadm on Centos 7. I run a k8S v1.8.1 cluster. When I try to fetch containers metrics this way: http://localhost:8001/api/v1/proxy/nodes/node1:0/metrics http://localhost:8001/api/v1/proxy/nodes/node1:/metrics http://localhost:8001/api/v1/proxy/nodes/node1:/metrics I find no metrics. kubelet is started with . If I start kubelet with . metrics are available through .


Question: gvr("flowcontrol.apiserver.k8s.io", "v1beta2", "flowschemas"): { Stub:             `{"metadata": {"name": "fs-1"}, "spec": {"priorityLevelConfiguration": {"name": "name1"}}}`, ExpectedEtcdPath: "/registry/flowschemas/fs-1", <ins> ExpectedGVK:      gvkP("flowcontrol.apiserver.k8s.io", "v1beta3", "FlowSchema"), </ins> }, // --

adds new version for in 1.26. For 1.26, we are setting the storage version to because vN-1 understands that level. In 1.27, we should change the storage version from to .
/sig api-machinery
/cc
/cc
/triage accepted
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:


Question: "k8s.io/kubernetes/pkg/util/node" ) <del> const loadBalancerWidth = 16 </del> <ins> const ( loadBalancerWidth = 16 // LabelNodeRoleMaster specifies that a node is a master // It's copied over to here until it's merged in core: https://github.com/kubernetes/kubernetes/pull/39112 LabelNodeRoleMaster = "node-role.kubernetes.io/master" // NodeLabelRole specifies the role of a node NodeLabelRole = "kubernetes.io/role" // NodeLabelKubeadmAlphaRole is a label that kubeadm applies to a Node as a hint that it has a particular purpose. // Use of NodeLabelRole is preferred. NodeLabelKubeadmAlphaRole = "kubeadm.alpha.kubernetes.io/role" ) </ins> // AddHandlers adds print handlers for default Kubernetes types dealing with internal versions. // TODO: handle errors from Handler

/kind bug What happened: doesn't show role Workaround: What you expected to happen: According to and How to reproduce it (as minimally and precisely as possible): I just deployed a multi master kubernetes environment with kops: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: AWS Install tools: kops, terraform
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig cli
I don't think this is a bug. The node role label is not used by kubeadm anymore. Related codes have been changed in by
Hi Thanks for the quick reply I'm actually using the as you can see in the workaround above. I'm not using kubeadm role label.
Your favorite topic came up again. Hi I do think this is a bug. As said, kubectl shouldn't show anything based on the old kubeadm master label, but it should probably on the label, but absolutely on the label. cc as you have opinions on this as well cc