Question: if hasJump(lbRules, nonLocalEpChain, "", 0) { errorf(fmt.Sprintf("Found jump from lb chain %v to non-local ep %v", lbChain, epStrLocal), lbRules, t) } <del> if hasJump(lbRules, svcChain, "", 0) != shouldLBTOSVCRuleExist { prefix := "Did not find " if !shouldLBTOSVCRuleExist { prefix = "Found " } errorf(fmt.Sprintf("%s jump from lb chain %v to svc %v", prefix, lbChain, svcChain), lbRules, t) </del> <ins> if !hasJump(lbRules, svcChain, "", 0) || !hasSrcType(lbRules, "LOCAL") { errorf(fmt.Sprintf("Did not find jump from lb chain %v to svc %v with src-type LOCAL", lbChain, svcChain), lbRules, t) </ins> } if !hasJump(lbRules, localEpChain, "", 0) { errorf(fmt.Sprintf("Didn't find jump from lb chain %v to local ep %v", lbChain, epStrLocal), lbRules, t)

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: This is discussed on When externalTrafficPolicy: Local is used, in GKE, external traffic coming in to the Load Balancer works correctly. But traffic that originates within the GKE cluster is broken, a TCP SYN is not responded to. My use case: I run gitlab, gitlab-runner, docker-registry, wiki inside kubernetes. But I also access the docker-registry outside kubernetes. So my hostname for it is ''. In DNS this resolves to the LB IP. Actually, all 3 services have DNS names that resolve to the same LB IP, which then goes to the nginx ingress. so , , are generally addressed as LB-IP w/ HOSTNAME+SNI. Its 'astonishing' to not be able to access this DNS name when I'm 'inside' but can when I'm 'outside'. If I enable the IP transparency (because in my wiki I want the IP of the user who made the edit), then my gitlab-runner fails(since it cannot pull from the registry). Tim Hockin has this to say: and then: What you expected to happen: My expectation is that it should not matter where my traffic originates, if it is to a globally-routeable public IP it should work. How to reproduce it (as minimally and precisely as possible): Configure a Load Balancer, an ingress, and 2 services. Have service 2 talk to service one using a DNS name that maps to the Load Balancer. Anything else we need to know?

: Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.4", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-06-06T08:13:03Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"10+", GitVersion:"v1.10.2-gke.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-05-10T17:23:18Z", GoVersion:"go1.9.3b4", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: Google GKE OS (e.g. from /etc/os-release): BUILDID=.89.0 NAME="Container-Optimized OS" KERNELCOMMITID= GOOGLECRASHID=Lakitu VERSIONID=66 PRETTYNAME="Container-Optimized OS from Google" VERSION=66 GOOGLEMETRICSPRODUCTID=26 HOMEURL="" ID=cos Kernel (e.g. ): Linux gke-k8s-default-pool--d3ft 4.14.22+ SMP Thu May 10 17:54:42 PDT 2018 x8664 Intel(R) Xeon(R) CPU @ 2.00GHz GenuineIntel GNU/Linux Install tools: Others:
/sig network
Agree that this is weird and feels left-out. The fix is not complicated but I need an owner to carry it through testing.
if nobody is working on it can I take it as my first issue?.
Is anybody working on this issue. Seems like ipvs might have the same problem.
/assign
I believe I'm running into this problem and I think adding as Tim mentioned would fix it. are you working on this? If not I don't mind taking a stab at it :)
Thanks, Sorry I kinda forgot this thing. I was fixed this thing just like you said But encounter some problem when testing in gce environment, which caused by route rules. Sorry I forgot the detail, can you do some test in your environment?
It seems the pull request is stalled on this. Does anyone have any input?
Being relied on by istio
Following this.
Did this fix get lost in the shuffle?

Any chance getting this merged in? This also impacts Digital Ocean's Kubernetes + LB setups out of the box.
As far as I know it hasn't
Merge On Mon, Feb 18, 2019, 7:58 PM Jarred Capellman wrote:
still affecting me on gke. i want to enable the so that I can use the spam-blocking in wordpress (which is by ext ip), but I also want to refer to other services which may be internal to the cluster via their proper name (to get the `tls etc).
Same here.
I have more or less spelled out the fix, but I don't have teh bandwidth to polish it off right now. I am looking for volunteers...
I can take this given I've run into this a while back /assign
I was only able to reproduce this when trying to reach the public IP from the host network. Reaching the public IP from inside the pod network is working (as per this ). It's possible I missed a step in trying to reproduce this but wanted to confirm with you first. Are you seeing this for traffic originating only from host network or does it also apply for pods in the cluster network?
Hi. My situation. I have a DNS name (git.MYDOMAIN). It runs on a self-hosted gitlab via nginx-ingress via LoadBalancer in GKE. Other pods of mine sometimes reference https://git.MYDOMAIN. They may happen to run in the same cluster as git.MYDOMAIN, they may run in a different cluster. When they run in the same cluster on GKE, they fail if is enabled. Specifically, since the DNS resolves to (My external Loadbalancer IP) an IP outside of GKE, it goes out, hits the LB, and then somewhere along the reverse path is lost. I have calico enabled (as per GKE managed). I am not using Istio. I am using a cluster that predates the new private IP ones. I have not tried disabling the LoadBalancer and just using node ip for the ingress. I have not tried to see if there is any difference if the Pod trying to reach my git is on the same node as it or not.

Based on the proxy rules (may be different at this point cause I'm looking at master), I don't think your traffic is even reaching the LB, it's just being dropped by the iptable rules on the host. My original question was, are the pods trying to reach the same-cluster external IP have , if yes, can you reproduce on pods not on host network?
I do not have set.
Are you still on v1.10?
this is the most recent version on gke afaik.
I opened a fix here based on the comments and some testing I did on my GKE cluster. It needs more validation though
Again, I wasn't able to reproduce this for pods that are within the cluster CIDR (but I could on the host network). The rule that allows for pods inside the cluster CIDR to still reach pods behind an LB is this: Are you able to check if a rule like ^ exists on your cluster?
Here's an example, right now (www is hosted on ingress w/ loadbalancer w/ local policy). Note: this example is from the node (not a pod).
Here's the full rules:
^ can you show me the same test from within a pod (not host network)? I have a fix for within a node but I'd like to make sure it's reproducible inside a pod before I dig deeper
Interestingly its working in a Pod on this cluster now. (?!) I guess some previous upgrade of k8s on gke has altered it? on cannot reach on on can reach. hmmm.
ok w/o the node being able to reach this i'm still blocked (since my container registry might run there, and thus kubelet cannot reach, so my gitlab CI pipeline cannot reach). but, i guess, now the pods w/o hostnetwork can, so some progress
Thanks for checking should fix the internal reachability issue from the host. Though it requires SNAT.
Hello Maybe better way is to check if and then skip to KUBE-SERVICES chain? So, request will be sent to loadbalancer and source IP will be preserved. Can you try this?
Solution described above will affect traffic from pods, it will not be . Then maybe we can replace with something to send traffic to loadbalancer?

I think this will break existing implementations, the fwchain is what routes the LB IPs to the correct service chain. Not all LB implementations (think MetalLB) depend on node port for routing
I tried to exclude node IP with from on node without local endpoints and it works! What do you think about this solution? I tested with:
I believe there's a similar rule for that already but it doesn't check source IP If you jump to FW chain, you end up in XLB chain which drops traffic if there's no local endpoints. Does your test also work on nodes where there's no local endpoints? Can you also verify that your traffic actually goes to the LB and is not being masqueraded through the service chain?
Yes, I just source IP check to this rule on node without local endpoints, so local traffic doesn't jump to FW, it doesn't drops in XLB and it goes to LB. Of course, if you do this on node with local endpoints you will not reach that endpoint (at least when using Metallb). This solution doesn't work if nodes haven't network access to LB. Your solution haven't this problem. For me it is more important then saving source IP, so now I think your solution is better. But why you add jump to MARK-MASQ before jump to SVC when each SEP chain already contains jump to MARK-MASQ? I tested your solution with:
Good point, I think might be redundant, will update the PR soon :)


Question: } func (config *NetworkingTestConfig) createNodePortService(selector map[string]string) { <del> config.NodePortService = config.createService(config.createNodePortServiceSpec(nodePortServiceName, selector, false)) </del> <ins> config.NodePortService = config.CreateService(config.createNodePortServiceSpec(nodePortServiceName, selector, false)) </ins> } func (config *NetworkingTestConfig) createSessionAffinityService(selector map[string]string) { <del> config.SessionAffinityService = config.createService(config.createNodePortServiceSpec(sessionAffinityServiceName, selector, true)) </del> <ins> config.SessionAffinityService = config.CreateService(config.createNodePortServiceSpec(sessionAffinityServiceName, selector, true)) </ins> } // DeleteNodePortService deletes NodePort service.

What happened: indicates that we're missing an e2e test for. This specific issue is just a tracker to create a new E2E test that demonstrated .
/sig network
/assign
btw, should this be a part of conformance?
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
/remove-triage unresolved
I can start working on this.
Sure/ are you working full time on it ? If so I'll drop the issue from my work ... Otherwise we can collaborate as I already have some code for this. Let me know. Was hoping to have this done in the next week or so.
I would love to collaborate, especially if you already have some code. But if you want to drop this from your work, I don't mind taking it on full time either. I can pick up from where ever you left off.
ok, if youve got full bandwidth go for it. if make progress on it this wk ill just drop it . I actually dont know where my old branch is for this :), so i guess youll have to start from scratch. One thing to note ~ Might be seeing if any tests can be reused for this or extended in a simple way by just adding a secondary service, rather then writing a full blown test from scratch. Let me know how it goes thanks !
I will let you know how it progresses. Thanks !
thanks for getting this taken care of :) looks like the PR is LGTM.
Thanks for letting me take a stab at it!


Question: fs.StringVar(&f.ExperimentalMounterPath, "experimental-mounter-path", f.ExperimentalMounterPath, "[Experimental] Path of mounter binary. Leave empty to use the default mount.") fs.StringSliceVar(&f.AllowedUnsafeSysctls, "allowed-unsafe-sysctls", f.AllowedUnsafeSysctls, "Comma-separated whitelist of unsafe sysctls or unsafe sysctl patterns (ending in *). Use these at your own risk. Sysctls feature gate is enabled by default.") fs.BoolVar(&f.ExperimentalKernelMemcgNotification, "experimental-kernel-memcg-notification", f.ExperimentalKernelMemcgNotification, "If enabled, the kubelet will integrate with the kernel memcg notification to determine if memory eviction thresholds are crossed rather than polling.") <del> fs.StringVar(&f.RemoteRuntimeEndpoint, "container-runtime-endpoint", f.RemoteRuntimeEndpoint, "[Experimental] The endpoint of remote runtime service. Currently unix socket and tcp endpoints are supported on Linux, while npipe and tcp endpoints are supported on windows.  Examples:'unix:///var/run/dockershim.sock', 'npipe:////./pipe/dockershim'") fs.StringVar(&f.RemoteImageEndpoint, "image-service-endpoint", f.RemoteImageEndpoint, "[Experimental] The endpoint of remote image service. If not specified, it will be the same with container-runtime-endpoint by default. Currently unix socket and tcp endpoints are supported on Linux, while npipe and tcp endpoints are supported on windows.  Examples:'unix:///var/run/dockershim.sock', 'npipe:////./pipe/dockershim'") </del> <ins> fs.StringVar(&f.RemoteRuntimeEndpoint, "container-runtime-endpoint", f.RemoteRuntimeEndpoint, "[Experimental] The endpoint of remote runtime service. Currently unix socket endpoint is supported on Linux, while npipe and tcp endpoints are supported on windows.  Examples:'unix:///var/run/dockershim.sock', 'npipe:////./pipe/dockershim'") fs.StringVar(&f.RemoteImageEndpoint, "image-service-endpoint", f.RemoteImageEndpoint, "[Experimental] The endpoint of remote image service. If not specified, it will be the same with container-runtime-endpoint by default. Currently unix socket endpoint is supported on Linux, while npipe and tcp endpoints are supported on windows.  Examples:'unix:///var/run/dockershim.sock', 'npipe:////./pipe/dockershim'") </ins> fs.BoolVar(&f.ExperimentalCheckNodeCapabilitiesBeforeMount, "experimental-check-node-capabilities-before-mount", f.ExperimentalCheckNodeCapabilitiesBeforeMount, "[Experimental] if set true, the kubelet will check the underlying node for required components (binaries, etc.) before performing the mount") fs.BoolVar(&f.ExperimentalNodeAllocatableIgnoreEvictionThreshold, "experimental-allocatable-ignore-eviction", f.ExperimentalNodeAllocatableIgnoreEvictionThreshold, "When set to 'true', Hard Eviction Thresholds will be ignored while calculating Node Allocatable. See https://kubernetes.io/docs/tasks/administer-cluster/reserve-compute-resources/ for more details. [default=false]") bindableNodeLabels := cliflag.ConfigurationMap(f.NodeLabels)

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: registered custom CRI by starting kubelet with extra options: according to documentation ; kubelet did not start; error message in journalctl was: "failed to create kubelet: only support unix socket endpoint" What you expected to happen: kubelet should have started because says for both and : "currently unix socket and tcp endpoints are supported on Linux"; custom CRI was listening on port 2000 How to reproduce it (as minimally and precisely as possible): e.g. add systemd fragment /etc/systemd/system/kubelet.service.d/20- Anything else we need to know?: current does indeed forbid ; so perhaps text for (in current ) should be changed to "currently unix sockets are supported on Linux" (i.e. does not support tcp endpoints) to fix Environment: Kubernetes version (use ): v1.13.3 OS: Debian 9 Install tools: kubeadm
I've now to the original text.
Reiterating the mentions to trigger a notification: <detailsIn response to : I've now to the original text. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
Thanks of reporting the issue, would file a PR to fix this.


Question: tc.runTest(t) } <ins> func TestReplicaCalcScaleDownIgnoresFailedPods(t *testing.T) { tc := replicaCalcTestCase{ currentReplicas:  5, expectedReplicas: 3, podReadiness:     []v1.ConditionStatus{v1.ConditionTrue, v1.ConditionTrue, v1.ConditionTrue, v1.ConditionTrue, v1.ConditionTrue, v1.ConditionFalse, v1.ConditionFalse}, podPhase:         []v1.PodPhase{v1.PodRunning, v1.PodRunning, v1.PodRunning, v1.PodRunning, v1.PodRunning, v1.PodFailed, v1.PodFailed}, resource: &resourceInfo{ name:     v1.ResourceCPU, requests: []resource.Quantity{resource.MustParse("1.0"), resource.MustParse("1.0"), resource.MustParse("1.0"), resource.MustParse("1.0"), resource.MustParse("1.0"), resource.MustParse("1.0"), resource.MustParse("1.0")}, levels:   []int64{100, 300, 500, 250, 250}, targetUtilization:   50, expectedUtilization: 28, expectedValue:       numContainersPerPod * 280, }, } tc.runTest(t) } </ins> func TestReplicaCalcTolerance(t *testing.T) { tc := replicaCalcTestCase{ currentReplicas:  3,

/kind bug What happened: HPA fails to launch additional pods due to pods in evicted state, even with utilization significantly over target. What you expected to happen: When HPA pods reach the target utilization, new pods should be allocated and provisioned. How to reproduce it (as minimally and precisely as possible): Launch a HPA, let the pods fill the disk on the host which will trigger evictions due to "Pod The node was low on resource: [DiskPressure]", HPA will exceed target and not allocate additional pods. Anything else we need to know?: Seems related to and though those are not specific to HPA. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: GKE OS (e.g. from /etc/os-release): cos-stable-57-9202-64-0 kernel commit id Kernel (e.g. ):Linux gke-prod1-use1b-v2--35s4 4.4.70+ SMP Mon Sep 18 12:43:08 PDT 2017 x86_64 Intel(R) Xeon(R) CPU @ 2.30GHz GenuineIntel GNU/Linux
/sig scheduling
Are there any workarounds while we wait for a fix? We have ran into this on a 1.8.4 GKE cluster.
/sig autoscaling /remove-sig scheduling
I am thinking of fixing this by excluding pods with Phase=Failed from the unreadyPods that are taken into account. It seems the right thing to do as they will not become running again. Do you see any issues?
that seems decent -- anything that will never be running again should not be factored into our calculations.
/status approved-for-milestone Adding to milestone -- if the HPA isn't functioning in the case of evictions, that seems like a decently severe HPA bug.
oops, missing priority was my fault, moving back into milestone: /priority critical-urgent /remove-priority important-soon
[MILESTONENOTIFIER] Milestone Issue Needs Attention Action required: During code freeze, issues in the milestone should be in progress. If this issue is not being actively worked on, please remove it from the milestone. If it is being worked on, please add the label so it can be tracked with other in-flight issues.

Note: This issue is marked as , and must be updated every 1 day during code freeze. Example update: <details<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details


Question: import ( "fmt" <del> "reflect" </del> "runtime/debug" "testing" "time"

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: There are too many pdb update operations compared with other resources. ! That is, there are many "Put" operations for pdb periodically in kube-apiserver log. In my view, the amount of pdb operations is connected with disruption controller's resyncPeriod and pdb's quantity in cluster. For example: disruption-controlle's default resyncPeriod is 30s, and there are 100 pdbs in cluster. We can find that the update operation times per minute is about 200. The data is reasonable, explanation is as follows: According to disruption-controlle's design, there is one resync every 30s. One resync means to call updatePdbStatus function for every pdb, so there are 200 times updatePdbStatus. Function updatePdbStatus judges whether the pdb is changed. If changed, call apiserver to update the pdb' status. When no eviction happens, disruptedPods is empty. In this situation, disruptedPods is an empty map, but pdb.Status.DisruptedPods is nil, empty map is not equal to nil. So reflect.DeepEqual(pdb.Status.DisruptedPods, disruptedPods) returns false. The pdb is thought to has be changed, then apiserver is called to update the pdb' status. In fact, there is no change for the pdb when disruptedPods is an empty map and pdb.Status.DisruptedPods is nil. What you expected to happen: There shouldn't be so many pdb update operations. That is, when pdb has no change, apiserver shouldn't be called to update its status. How to reproduce it (as minimally and precisely as possible): Create some pdbs in cluster, you will see many "Put" operations for pdb periodically in kube-apiserver log. Anything else we need to know?: Environment: kubernetes v1.13.2 Fix suggestions: The bug is caused by the mixed usage of empty and nil for disruptedPods, the type of which is map[string]metav1.Time.

There are two fix suggestions for it: one of empty map and nil for disruptedPods, use it consistently and refuse to mix empty map and nil. For example, when disruptedPods is empty or nil, save it as an empty map in etcd. Then every client gets pdb from etcd, the pdb.Status.DisruptedPods is an empty map. get pdb for etcd, assign an empty map to it if pdb.Status.DisruptedPods is nil. Maybe the mixed usage of empty and nil for map is a common problem in k8s design, not just for disruptedPods. I will propose a work around solution for the problem. If the solution has any problem, please let me know and I can modify the code.
/sig apps
/assign Morten can you PTAL and the open PR. I reviewed as well.
ref


Question: <ins> #!/usr/bin/env bash # If we are completing a flag, use Cobra's builtin completion system. # To know if we are completing a flag we need the last argument starts with a `-` and does not contain an `=` args=("$@") lastArg=${args[((${#args[@]}-1))]} if [[ "$lastArg" == -* ]]; then if [[ "$lastArg" != *=* ]]; then kubectl ns __complete "$@" fi else # TODO Make sure we are not completing the value of a flag. # TODO Only complete a single argument. # Both are pretty hard to do in a shell script.  The better way to do this would be to let # Cobra do all the completions by using `cobra.ValidArgsFunction` in the program. # But the below, although imperfect, is a nice example for plugins that don't use Cobra. # We are probably completing an argument.  This plugin only accepts namespaces, let's fetch them. kubectl get namespaces --output go-template='{{ range .items }}{{ .metadata.name }}{{"n"}}{{ end }}' # Turn off file completion.  See the ShellCompDirective documentation within # https://github.com/spf13/cobra/blob/main/shell_completions.md#completion-of-nouns echo :4 fi </ins>

<!-- Please only use this template for submitting enhancement requests --What would you like to be : I type <kbdTab</kbd<kbdTab</kbdI would like to see kubectl plugins in the list of suggested subcommands. For example, if I have installed , subcommand completions should suggest . the plugin itself also supports command-line completion, <kbdTab</kbd<kbdTab</kbdshould show suggestions from the plugin command completion. Why is this needed: Plugins fill in niches in the kubectl workflow, so that they are used rather seldomly. This means that plugin users need to memorize plugin names and their options over long times without using them. Therefore, plugin usefulness would be greatly enhanced by suggesting plugin sub-commands and their options.
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
In fact, already supports command completion. You can run for more information about it. If you are using bash shelljust run , and then it will work!
Hi , but I'm not talking about command completion in general. I'm talking about command completion for plugins. For example, the new plugin system allows to call a plugin at location as . It would be nice to get suggested/completed, when typing <kbdTab</kbd. Further, if plugin names are completed, so should be options for those plugins -- provided the plugin offers some form of command completion.
I try to implement completion for kubectl plugins!
Were you able to make progress on this? I have a working patch for the completion of the plugin name (part 1 of this issue). I can make a PR for it if it can help.
This is a great first step! I do think however, that the real value is auto-completion for the plugin options. I can roughly remember what plugins I have, but not what options they offer. That's the real value here, I think. This will probably need a contract for how plugins make completion available (e.g. by following the kubectl standard of ). WDYT?
I've been thinking about how to do the plugin options completion.

Based on the patch I have now, I believe what will be needed is to be able to ask the plugin what the possible completions can be (based on the user-specified parameters). So the plugin won't need to generate a bash or zsh completion script (which is difficult for the author of the plugin) but just output possible completions. This also has the benefit of being independent of which shell is used. One way is to do as you mention and have the plugin support as a way to ask the plugin what the completions would be. Although it won't behave in the same fashion as (as it won't generate a completion script, and is shell-independent), I like that option because it simply requires a plugin author to provide a different script called which the bash completion logic can call. Another way would be to ask the plugin itself to support a flag such as which would return the possible completions. I don't like this because if the plugin doesn't support this flag, we'd be calling it and who knows what it would do in that unexpected case. The use of makes it easy to check of presence or absence of completion support. So I suggest that the contract with plugins be that a plugin that supports completion would provide two scripts and that the bash completion script call to get the list of completions. Any thoughts?
Hi guys, I've already implemented completion for plugins (subcommands only, doesn't include options) as However it needs to be refactored more. You can try completion for plugins with the following steps: First, prepare the repository, and build a kubectl. Install kubectl-hello plugin for completion testing. Load the kubectl completion code. You can see completion for some plugins.
Cool! You've used the same technique I did. However, may I suggest re-using the code from the plugin package instead of copying it?
Well, you are absolutely right in saying that we can't just call a plugin with some flag to produce the completion script, because who knows what will happen. Therefore, option completion for plugins has to go the route via an external completion script, like However, I'm pretty sure that those completion files need to be bash/zsh scripts, because they need to process contextual information.

For example, say I have a plugin with sub-commands Then I want completion to work as follows: As you say, this will have to be maintained by the plugin authors. Can you take a typical cobra command completion script as basis and work out how kubectl-plugin-completion and this external script need to work together? You could even try to have kubectl as its own plugin, like and or so :grinning:
kubectl already has some dynamic completion code to allow to complete pod names for example, which actually calls kubectl itself. I think I can use the same technique for plugin parameter completion. So here's an example of what I'm thinking: Then when the user types the kubectl completion script will call . This call will return which will be given to the shell by the kubectl completion script. I haven't actually tested this example, so I could be wrong. I'll try implementing it to see if I'm missing something.
Thank you for supporting me ;) I add you as co-author to commit when I send a PR.
Thanks
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue.

<detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-lifecycle rotten
I've been working on auto-completion for the Helm project and I have a proposal with a PR that fully supports auto-completion for Helm plugins. I believe the fundamental aspects of the solution would apply just as well to kubectl. Some adaptation would be needed to adjust to how kubectl supports plugins, but I think it wouldn't be very hard.
Is there an ETA on this? If I see it correctly, this is currently preventing plugin authors to implement completions as the generated completion suggestions are based off the root command and not the subcommand aka plugin. I've been trying all day to make this work but failed and that's when I stumbled over this issue..
Several major issues I see here are: We don't want files like as proposed in previous comment. We had a requirement prior to kubectl 1.12 and it didn't look great at all. The simplicity of plugins is valuable to us. (Also, unlike Helm plugins, kubectl plugins don't live at a particular directory, or follow a directory structure.)Plugin completions most likely would be integrated into commands, which a lot of people put in their or file. We need to keep this command fast. So what if the command called a special command on each plugin, in parallel, to inquire if it has completion available (e.g.), with a short deadline, and cached the successful results (maybe keyed with on plugin binary’s stat data, or checksum). That way, cmd could also provide data about plugins, while periodically refreshing the cached completion scripts.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with .

Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
any updates on this one ? is it complete/done?
No update, still open..
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale
/lifecycle frozen
Sorry to revive this thread a bit but while working on another cli using the same cobra completion system by (implemented by i believe) I found out that it would not be too hard or too disruptive to have a small patch on the shell completion directly. For example with kubectl if we have a patch like this (using zsh but I am guessing the same technique will apply for bash or others) : the patch will check the first command () and the second command for example "krew" exist as and then use it directly for completion otherwise do the normal completion 'thingy'. the only downside is that the initial plugin would not show i.e: if you do you won't get krew as the plugin name, except if we add all plugin to the Root Command like the pull request from does. I don't think it would be too intensive since it is a zsh/bash builtin and in memory. I have made a small asciinema to demonstrate how it works I guess if we want this we may need to do it directly in cobra, do you think a patch like this could get accepted ?

(I had somehow difficulties to get PR reviewer and merged in cobra last time I tried)
Thanks for the suggestion. The idea is good but doesn't generalize perfectly. The command is available for programs that use Cobra but kubectl plugins can be something else (a script, a Python program, a Go program that doesn't use Cobra, etc). So we couldn't do something like this, unless we required every plugin to provide a command. I'm working on porting Helm's plugin completion solution to kubectl and should have something for the community to try soon. Plugins using Cobra will then be able to use your suggestion and get completion super easily.
That make sense I forgot about non go (and non cobra) plugins, thanks I look forward to see how the helm completion extracted will look like !
I've posted something for people to try out in
It would be nice if some plugin developers try out the solution proposed in and provide feedback. since uses Cobra, the solution is very easy. Here is how to try it out: First build kubectl with the PR. Then create your completion script like this: put this new on $PATH. And now test using the newly built (I build mine at ). For completion of flags (which I realize is the real value for the plugin), kubectl needs to first move to Cobra 1.3.
I would be interested in your opinion on the proposed solution. It will not work out-of-the-box with Krew, but I haven't found a way to do better with respect to Krew. You had mentioned this previously, and although it would be a nice solution, I am concerned it would cause problems to any plugin that ignores all arguments and runs no matter what. With such a plugin, calling a command would trigger the plugin to run and possibly perform undesired actions when completions are requested.
I'm sorry about slow responses. Finally had time to read it and parse these. You're right we shouldn't randomly call a plugin binary without knowing it supports completion. From : kubectl already has a mechanism to find all plugins on $PATH, so I agree that at minimum, kubectl<kbdTab</kbdcan and should show plugins installed on the system (even if the plugins themselves don't support shell completion) I think this is a good model.

Just like how kubectl scans $PATH for , it can scan to see if a plugin supports completion. (It's important to define if that binary takes additional positional args or --options) To make the (2) a reality, I support adding an additional field to the Krew plugin manifest. The shape of that would be up for a discussion (I imagine it might be (a) a separate binary that comes in the plugin archive, or (b) allow linking to the plugin binary itself and the plugin reads argv[0] and prints completion, which can be done with (a) anyway). I think it's doable. I'm not very familiar with how shell completion works in practice, when it's called, how lazy can it be loaded, what are performance implications etc, but if you'd like to drive this effort, I think we can discuss a design on Krew side as well. Please bring this up in a SIG CLI meeting to see what the folks think about your proposal.
brilliant! Thanks for taking the time to think about this. Your suggestion to adapt krew sounds great to me. I will continue with the PR on kubectl's side and if it gets accepted we can look at krew.
The as proposed in accepts the exact same arguments as the standard executable. The new completion executable will examine those arguments and will determine what the shell completion choices following them should be.
do you have an insight into whether a plugin developer would develop and ship a separate executable for the completion, or would they simply link the main plugin executable and see if the argv[0] is the kubectl_complete-*? I assume the answer largely also depends on whether the plugin was authored in Go or bash (basically the only two languages I've ever seen a plugin submission in out of 150+).
I think plugins will choose to ship a separate executable. I feel it keeps the two aspects nicely separate (the plugin functionality and the shell completion functionality). Also, it is very easy for a separate executable to call back the main plugin if the developer wishes to do so, so I find that solution the most flexible. For example, for plugins using the Cobra library, the script would call back the executable like this: BTW, I have looked into the 179 plugins in the krew index and I found that 50% (yes, half!)

of them used Cobra (some of them indirectly by calling back ). This means that shell completion should be very easy to implement for all these plugins.
I'm interested in this support and thanks for adding this feature. One question: in which version this is planned to be released? Looking forward to this.


Question: }, }, } <ins> pvcConfig := e2epv.PersistentVolumeClaimConfig{ StorageClassName: &ns, } </ins> numPVs, numPVCs := 1, 1 pvols, claims, err = e2epv.CreatePVsPVCs(ctx, numPVs, numPVCs, c, f.Timeouts, ns, pvHostPathConfig, pvcConfig)

master-informing: ! Last 2 runs failed consecutively, first failure: No response /sig storage
/cc
I will take a look this week.
/triage accepted
/assign


Question: // Network plugin settings. Shared by both docker and rkt. fs.StringVar(&s.NetworkPluginName, "network-plugin", s.NetworkPluginName, "<Warning: Alpha feature> The name of the network plugin to be invoked for various events in kubelet/pod lifecycle") <ins> //TODO(#46410): Remove the network-plugin-dir flag. </ins> fs.StringVar(&s.NetworkPluginDir, "network-plugin-dir", s.NetworkPluginDir, "<Warning: Alpha feature> The full path of the directory in which to search for network plugins or CNI config") <ins> fs.MarkDeprecated("network-plugin-dir", "Use --cni-bin-dir instead. This flag will be removed in a future version.") </ins> fs.StringVar(&s.CNIConfDir, "cni-conf-dir", s.CNIConfDir, "<Warning: Alpha feature> The full path of the directory in which to search for CNI config files. Default: /etc/cni/net.d") fs.StringVar(&s.CNIBinDir, "cni-bin-dir", s.CNIBinDir, "<Warning: Alpha feature> The full path of the directory in which to search for CNI plugin binaries. Default: /opt/cni/bin") fs.Int32Var(&s.NetworkPluginMTU, "network-plugin-mtu", s.NetworkPluginMTU, "<Warning: Alpha feature> The MTU to be passed to the network plugin, to override the default. Set to 0 to use the default 1460 MTU.")

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Environment: Cloud provider or hardware configuration: bare-metal OS (e.g. from /etc/os-release): Ubuntu 14.04 Kernel (e.g. ): Linux dev-k8s-controller-p1 4.4.0-53-generic ~14.04.1-Ubuntu SMP Fri Dec 2 03:43:31 UTC 2016 x8664 x8664 x86_64 GNU/Linux Install tools: chef cookbook Others: Docker 1.12.6, calico 2.1 CNI binaries: kubelet config: What happened: When trying to run pod error message is generated: So it seems that kubelet is trying to find CNI binaries in paths: /opt/loopback/bin /etc/cni/net.d /opt/calico/bin But it should look at since this is the default valur for parameter. What you expected to happen: Pod starts. How to reproduce it (as minimally and precisely as possible): Run pod: with kubelet 1.6 and CNI binaries in those path (default for calico): Anything else we need to know: After copy to and pod starts fine.
Good, very helpful to me
When set and not set , default path of CNI binaries will not take effect, the path specified by will be used instead. We should use and instead of if use CNI plugin.
I will push a PR to deprecate the flag. It is confusing. Closing this issue.
Was that ? It was closed without merging. I agree should be deprecated.
I think is needed before is removed. The flag is still not deprecated, I will send a PR to deprecate it.


Question: testErrorReason := fmt.Errorf("test-error") emptyContainerID := (&kubecontainer.ContainerID{}).String() testKubelet := newTestKubelet(t, false /* controllerAttachDetachEnabled */) <ins> testKubelet.fakeCadvisor.On("VersionInfo").Return(&cadvisorapi.VersionInfo{}, nil) testKubelet.fakeCadvisor.On("MachineInfo").Return(&cadvisorapi.MachineInfo{}, nil) testKubelet.fakeCadvisor.On("ImagesFsInfo").Return(cadvisorapiv2.FsInfo{}, nil) testKubelet.fakeCadvisor.On("RootFsInfo").Return(cadvisorapiv2.FsInfo{}, nil) </ins> kubelet := testKubelet.kubelet pod := podWithUidNameNs("12345678", "foo", "new") pod.Spec = api.PodSpec{RestartPolicy: api.RestartPolicyOnFailure}

Currently we query the apiserver for the node (indirectly: ) before populating hostip on a pod. I believe this is because the node controller is responsible for calling into the cloud provider and populating the ip address on a node: This is disappointing for 2 reasons: makes using ips to identify nodes harder (). 't separate concerns sufficiently. If the reflector fails to watch nodes reliably pods won't get a hostip.
Also see .
I would argue that if we're unable to watch nodes reliably, we have many other problems. That seems like something we need to fix, rather than work around.
watch is watch, a separation of concerns between the kubelet (which is running on the node that has the hostip) and the apiserver is a different problem imo.
After discussion, we decided to just remove host IP from PodStatus. (It will still be available from "kubectl get nodes" if someone needs it.) does that sound OK with you?
I'm working on making the nodes register themselves rather than having the node controller do it for them (see which should turn into a PR later this week). Will that assuage your concerns here?
I hadn't looked at your PR. We discussed having Kubelet pull ths information from the cloud provider but thought it might not be a good idea due to (1) requiring Kubelet to understand the details of all the cloud providers, and (2) making very hard to rate limit the calls into the cloud provider. I could imagine an approach where we still use NodeController to get the node address, but the node still initiates the registration into the cluster. Not sure whether that's a good idea, though.
Kubelet won't necessarily (and really shouldn't) have the credentials necessary to contact the cloud provider. I don't remember why HostIP was to PodStatus, and I would favor removing it, except that it would be a breaking change. You could ask on google-containers@ whether anyone is relying upon it. HostIP is tricky, as hosts may have multiple addresses, in general, which is why NodeAddresses is now an array.
On GCE / AWS do you even need credentials to determine your external IP? I thought you got it from the metadata server running locally without needing to hit an Auth'd API endpoint.
HostIP wasn't the external IP AFAIK.

It's not well defined what it is, which is one reason it should be eliminated. Yes, one should be able to get the host IP from metadata, but the cloudprovider library doesn't currently do that.
I'm also in favor of removing HostIP from pods. But I don't agree with the reason in the OP (that sometimes it'll be blank when kubelet hasn't heard from apiserver)-- I think that's par for the course.
Should we schedule a meeting to discuss whether it's reasonable for Kubelets to pull their IPs directly from the cloud provider? What is the alternative -- a new API endpoint on the master where unregistered nodes can query their IP address from NodeController? Do nodes really need their IP address when they register with the master?
That sounds like a good idea (I think we can likely clear this up with a quick meeting rather than going back and forth for days in github comments).
We all agreed that we should remove HostIP from pods. Nodes do not need to provide their address(es) when they register. It's not part of the spec. They could provide whatever hostnames and/or addresses they can get through normal Linux mechanisms and/or could contact a local metadata server, but I don't think that's necessary, since the node controller will update the addresses once the node is registered, unless we want to support scenarios with no cloud provider and no external node database. In any case, we shouldn't discuss self-registration in this issue. There are other issues that cover that topic.
OK, so this issue just boils down to removing HostIP from PodStatus, and we'll discuss the rest in ?
Ping on this. e2e "Pods should get a host IP [Conformance]" is currently disabled everywhere. Bumping this to and .
Today Kubelet fills PodStatus.HostIP in generatePodStatus() in Master (e.g. NodeController) is not involved. Currently the test waits two minutes for a pod to get a HostIP Considering that we have proposed to remove this field completely, I think we should just bump up the timeout a bit and see if that fixes the test. Any other thoughts? I have also sent out email to the appropriate mailing lists to see if anyone is using PodStatus.HostIP, since we would like to get rid of it, as discussed upthread.

cc/
BTW the title of this issue says "Kubelet shouldn't require apiserver to populate host ip of pods." But it wasn't clear to me where the dependency on the API server is (unfortunately 's original comment points to a very old revision to so I'm not sure which line he was referring to). The work of finding the HostIP seems to be done by Kubelet.GetHostIP() / Kubelet.GetNode() which assumes Kubelet.nodeInfo is filled in -- is that the dependency? I couldn't find where that gets filled in. would be great if you could shed some light.
I haven't looked at this code recently but nodinfo appears to still be retrieved from a cache populated via apiserver watch:
Ah yes, that looks right, thanks (well, "right" in the sense of "that's where it gets populated," not in the sense of "it makes sense that kubelet is using code defined in the scheduler predicates for this"...) while you're here, do you have a suggestion of the best course of action? Long-term we want to get rid of PodStatus.HostIP, but short-term we need to do something about this test. How can we make populating the cache more reliable? Or should we just increase the timeout on the test? Or something else?
It looks like the kubelet already has the node address: , so assuming no one else is going to change the ip, we could just store it in the kubelet and add it for every pod? Unsure if this is a safe assumption.
I also haven't debugged this new set of test failures, so I'm not sure if it's failing for the same reason
... to make on all clouds
There isn't a new set of failures - Ike just noticed that the test is disabled, from when this issue was originally opened. could one of you comment on whether Prashanth's suggestion sounds reasonable?
I didn't parse through the entire issue since a lot of information is stale. Yes David is right, we moved to the logic of filling hostIP of Pod from apiserver to kubelet long time ago. Recording to the comment of Prashanth: Some users might have node with multiple network interfaces, and want to hostIP with specific one by updating Node object.
Fwiw I think we should turn the test on and handle things as they come up.

I think the major source of flake was handled when we did:
With respect to the long-term future of HostIP: I'm in favor of documenting the field as deprecated and asking users to contact us with their use case if they need it (in the API field documentation). As long as the field exists, I agree we should test that it's set correctly.
in this comment you wrote "Ping on this. e2e "Pods should get a host IP [Conformance]" is currently disabled everywhere. Bumping this to P1 and flake." What made you say it is disabled? I looked at Jenkins and it seems to be running it: (root)/Kubernetes%%20suite/PodsshouldgetahostIPConformance/history/ I looked in and didn't see any special handling of [Conformance]. There is an env var CONFORMANCETESTSKIP_REGEX in hack/gingko- but it doesn't seem to be set anywhere. So I don't see anything that should be causing this test to not be run. (As always there's a high likelihood I'm misunderstanding something.)
Sorry, I'm totally blind, the Jenkins history I linked to clearly shows it is being skipped. But I'm still not sure what is causing it to be skipped.
Sorry I wasn't clear; it's being skipped because it was marked pending via . Let's agree to use and labels in the future for clarity's sake.
Also see for use cases of "I need to know my local node" and possible fixes.
I'm going to remove the flake label, since has been merged and that was the suggestion here We can re-add the label if it flakes again. I'm not closing the issue, because we want to deprecate the field longer-term.


Question: ``` cd kubernetes <del> sudo hack/local-up-cluster.sh </del> <ins> hack/local-up-cluster.sh </ins> ``` This will build and start a lightweight local cluster, consisting of a master

/kind failing-test <!-- If this is a flake, please add: /kind flake --<!-- Please assign a SIG using: /sig SIG-NAME --
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
looks like started a few days ago ! /sig cluster-lifecycle
cc
fix was sent earlier
/sig cluster-lifecycle
thanks :) i saw it yesterday and forgot to raise the issue. i should have checked the PR queue :)


Question: "fmt" "io/ioutil" "os" <ins> "sort" "strings" </ins> "time" "github.com/onsi/ginkgo/config"

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind cleanup What happened: In PR we tightened the --provider parameter checking of the binary such that it only accepted known providers. This turned out to be problematic () because people had started to depend on the previous behavior, which was to accept anything and merely warn about it. For the sake of giving people time to adapt, PR reverted to the older behavior. What you expected to happen: In 1.14 we want to allow only known providers. While at it, the error handling for unknown providers should be enhanced (print proper error message instead of calling Failf, which panics). /assign /sig testing
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
kubeadm was one of the tools which got broken by the tightening of the parameter checking. Does kubeadm still rely on the acceptance of unknown provider parameters or can the current warning be turned into an error now? /remove-lifecycle stale
yes the kubernetes-anywhere deployer is still bound to provider=kubernetes-anywhere (unknown) in test infra jobs. we went back and forth on this topic, but if you wish to replace this support for unknown deployers i can switch those jobs to . LMK.
given one release has passed with the deprecated message on the CLI, it's probably safe to remove this functionality now. :)
"Lubomir I. Ivanov" <: Yes, please do. Once that is complete, I'll turn the warning into an error.
done:


Question: apiVersion: v1 kind: Pod metadata: <del> name: l7-lb-controller-v0.9.5 </del> <ins> name: l7-lb-controller-v0.9.6 </ins> namespace: kube-system annotations: scheduler.alpha.kubernetes.io/critical-pod: '' labels: k8s-app: gcp-lb-controller <del> version: v0.9.5 </del> <ins> version: v0.9.6 </ins> kubernetes.io/name: "GLBC" spec: terminationGracePeriodSeconds: 600 hostNetwork: true containers: <del> - image: gcr.io/google_containers/glbc:0.9.5 </del> <ins> - image: gcr.io/google_containers/glbc:0.9.6 </ins> livenessProbe: httpGet: path: /healthz

GLBC v0.9.5 was built with an old version of the GCE cloudprovider which did not allow unknown key-value pairs in Manifest file must be updated to 0.9.6. /kind bug /sig network /area platform/gce /assign


Question: "foo", }, { <del> "two defaults, error with PVC with class=nil", []*storagev1.StorageClass{defaultClass1, defaultClass2, classWithFalseDefault, classWithNoDefault, classWithEmptyDefault}, claimWithNoClass, true, "", }, { </del> "two defaults, no modification of PVC with class=''", []*storagev1.StorageClass{defaultClass1, defaultClass2, classWithFalseDefault, classWithNoDefault, classWithEmptyDefault}, claimWithEmptyClass,

Presumably this would occur when changing the default from one class to another. E.g. Class X exists and is marked default Class Y is created Class Y is marked as default Uh oh!! 2 defaults Class X is marked as not default At the case in point, when there are 2 defaults, we currently do not assign default values to new PVCs. We should change this semantic - while there are two, pick one randomly or heuristically. It's still correct. Example: Suppose that you have one user changing the default and another user creating a PVC at exactly the same time. What happens? Depending on network, CPU scheduling, memory allocations, (etc - totally non-deterministic) the API server might see either of those operations first. The PVC creator will either win the race and get the old default class or it will lose the race and get the new default class. The difference could literally be whose network has a faster path to the API server. My point is that whichever one you pick, as long as it is one of those two defaults, it is a correct choice! Any other FooClass things we do should probably change, too. I think that's IngressClass and RuntimeClass.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/assign
GitHub didn't allow me to assign the following users: Octopusjust. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
picking either sounds reasonable but what if we allow only one default to be set and disallow/warn users if they try to create another default.

I can think of issues with this too, mainly when switching over to a new default by removing an existing class A and then setting B as a default but managing that orphaned period while the transition happens, maybe we can have the heuristic there? I'm sure I'm missing something here.
/assign
GitHub didn't allow me to assign the following users: Octopusjust. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
There is a problem when there is no default storage class for a while and another user creates a PVC. Such PVC won't get the new SC. We're trying to fix it in Until then, having two default storage classes for a brief moment is IMO slightly better, because at least users get an error message and can retry PVC creation. In addition, (in-tree) admission plugin in HA environment could be racy, as different API servers are admitting updates of different StorageClasses in parallel.
/assign
I wouldn't make 2 an error - just pick one.
Ok, I'd take a shot at this. Thanks for clearing things up.
+1 Still, having two default classes for a longer time is IMO bad, there should be a metric + alert if there are 2 or more default SCs for X minutes.
it's not ideal - a metric seems OK
hello there, im new to open source. i don't know how to contribute. can anyone hep me please?
I see this issue was labeled as "good first issue" and I would like to start working on it!
I see there is a pr related to this issue , are you still working on this?
Is this issue still open?
/assign
Yes, there's already a . It's linked to the issue itself.
The PR is specifically for StorageClass, but we also need to audit other FooClass things - they should probably change, too. I think that's IngressClass and RuntimeClass. Someone can open new PRs for these.
Oh, I see the second PR, too.

Will look at them ASAP
/remove-sig network
Do we emit a response header when someone makes a request that ends up setting two IngressClasses as default?
that seems a nice thing to do, I didn't know you can add warnings on admissions plugins but seems implemented it here
Is this issue still open??
StorageClass implementation is in I am not sure about those. I'd suggest opening separate issues. /close
Closing this issue. <detailsIn response to : whichever one you pick, as long as it is one of those two defaults, it is a correct choice! </details


Question: ValidationErrorTypeForbidden ValidationErrorType = "FieldValueForbidden" ) <del> // ValueOf converts a ValidationErrorType into its corresponding error message. func ValueOf(t ValidationErrorType) string { </del> <ins> // String converts a ValidationErrorType into its corresponding error message. func (t ValidationErrorType) String() string { </ins> switch t { case ValidationErrorTypeNotFound: return "not found"

We now have pretty structured way of validating api object, but the error message to end user is not descriptive, e.g. when creating a replication controller with restart policy 'never', we get: even as simple as label name, we shouldn't assume end-user is familiar with our requirement. Apart from what is wrong, we'd better also give why it is wrong.
The main thing wrong here is not having a useful String() method. and have both been pretty adamant about not attaching code to API objects, but here's a clear UX justification. What if we only attached methods to internal API objects, not versioned ones? Also: this should print %#v, rather than %+v, which would be a BIT more readable.
Orrrr we could switch to spew, which is already a dep.
Actually we're printing a reflect.Value there, which makes no sense at all. I will fix. EDIT: no we aren't, I can't read-- will still fix though


Question: } } <ins> func TestAPIListChunkingWithLabelSelector(t *testing.T) { defer featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, features.APIListChunking, true)() clientSet, _, tearDownFn := setup(t) defer tearDownFn() ns := framework.CreateNamespaceOrDie(clientSet, "list-paging-with-label-selector", t) defer framework.DeleteNamespaceOrDie(clientSet, ns, t) rsClient := clientSet.AppsV1().ReplicaSets(ns.Name) for i := 0; i < 10; i++ { rs := newRS(ns.Name) rs.Name = fmt.Sprintf("test-%d", i) odd := i%2 != 0 rs.Labels = map[string]string{"odd-index": strconv.FormatBool(odd)} if _, err := rsClient.Create(context.TODO(), rs, metav1.CreateOptions{}); err != nil { t.Fatal(err) } } calls := 0 firstRV := "" p := &pager.ListPager{ PageSize: 1, PageFn: pager.SimplePageFunc(func(opts metav1.ListOptions) (runtime.Object, error) { calls++ list, err := rsClient.List(context.TODO(), opts) if err != nil { return nil, err } if calls == 1 { firstRV = list.ResourceVersion } return list, err }), } listObj, _, err := p.List(context.Background(), metav1.ListOptions{LabelSelector: "odd-index=true", Limit: 3}) if err != nil { t.Fatal(err) } if calls != 2 { t.Errorf("unexpected list invocations: %d", calls) } list := listObj.(metav1.ListInterface) if len(list.GetContinue()) != 0 { t.Errorf("unexpected continue: %s", list.GetContinue()) } if list.GetResourceVersion() != firstRV { t.Errorf("unexpected resource version: %s instead of %s", list.GetResourceVersion(), firstRV) } var names []string if err := meta.EachListItem(listObj, func(obj runtime.Object) error { rs := obj.(*apps.ReplicaSet) names = append(names, rs.Name) return nil }); err != nil { t.Fatal(err) } if !reflect.DeepEqual(names, []string{"test-1", "test-3", "test-5", "test-7", "test-9"}) { t.Errorf("unexpected items: %#v", list) } } </ins> func makeSecret(name string) *v1.Secret { return &v1.Secret{ ObjectMeta: metav1.ObjectMeta{

has by default, and since 1.22 has the same. They seem to return wrong results in certain situations combined with / . And / also use , so they also don't work correctly. outputs partial list of pods on the node, doesn't evict some of the pods. To make it work correctly, I had to specify works correctly with any , since it's just a performance optimizing parameter. I can't provide an actual concrete example (no = default ), but instead I show how to reproduce it with a smaller I believe that the same problem occurs with create two deployments. there are 5 pods in each. with --selector=app=another-server and , it shows only 4 pods. I could confirm that the problem happens in both case of and regardless namespace, even with , or <details</details<detailsGCP / GKE Standard </details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig api-machinery
/sig node
I think this is indeed a bug, after some simple testing, it seems to be an bug unique to 1.24, let me fix it. /assign
/assign
I can reproduce this error in a 1.24 cluster, but not in a 1.23.6 cluster. All clusters are created via kind, and I'm guessing gke may have cherry-picked some commits or had some of their own changes to their branch. So I think this fix should at least cherry-pick to 1.23 to help other distributions fix this issue.
/remove-sig api-machinery


Question: options.captureProfile(outputProfile) } <ins> // Logs Enabled Plugins at each extension point, taking default plugins, given config, and multipoint into consideration logger.V(2).Info("the scheduler starts to work with those plugins", "Plugins", *f.ListPlugins()) </ins> f.setInstrumentedPlugins() return f, nil }

Nowadays, when the kube-scheduler is initialized, the provided component config (via flag) is logged. E.g. Unfortunately, the printed config does not mention which plugins are enabled besides those which are explicitly mentioned in the configuration (through enabled and disabled fields). A developer can determine all the enabled plugins based on taking all plugins enabled by default minus those which are explicitly disabled. However, the same can not be told about a "normal" user. Given the enabled plugins are known only to the framework, it is not trivial to "just" print the enabled plugins as part of the component config. Currently, the code only updates the printed component config to display the part (via ). I am reporting the issue in order start a discussion on how to better let a user know which plugins are enabled by default. E.g. by extending the and allowing to print the enabled plugins as part of the component config. I.e. by explicitly setting the enabled plugins.
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hey! Can I work on that?
By all means once there's an agreement on the approach :)
/assign I can try it
hi after reading the code, I think of two feasible solutions: Option 1: extending the FrameworkCapturer (as you said) extend , which marks to enable the plugin Configuration File Contents print: Option 2: Print after initializing the framework. In my case, I prefer option Option 1. If you have other better solutions, welcome to discuss together, look forward to your reply, thx
Poor thoughts, we should not introduce new fields just for documents.

And we do have documents for users to refer to What a means? An ops staff?
Works fine for the kube-scheduler. But not anymore when are included. Anyone who's not so familiar with the documentation and/or the code base to deduce which plugins are enabled.
I'd say just logging info log when initializing each plugin is enough.
Is it still open? I could try that
are you working on this `issue... please let me know. If yes, I would love to learn about this issue and its possible solutions in detail. that is - If you can share about. sincerely,
Hi all, I'd like to give it a try. Please let me know if you far ahead on it /assign
Hi For logging purposes, I thought about using the in  or around to do simple logging of what's enabled/disabled. What are your thoughts on that? Another question, is it expected that out of tree plugins are enabled explicitly in the config to be applied (default would be disabled)? In my tests with custom scheduler below the plugin was omitted from and did not end up in the final profile.Plugins object, I am not sure if it was applied or not to the scheduler. Any guidance appreciated, Cheers.
This will require some digging to get to the bottom of what is and where it is propagated. The last time I checked the was the closest way to propagate the list of enabled/disabled plugins outside for printing. I suggest to write few unit tests and see how the entire initialization is propagated and based on that decide which bits can be refactored/changed so the necessary information is propagated outside.
I opened the draft PR , centralising the logic to discover plugins enabled and disabled inside the block for the captureProfile callback: Any suggestions if that would be better to place the logging inside the captureProfile callback:
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
Hi all, I'd like to give it a try. Please let me know if you far ahead on it /assign
Hello, I would like to work on this issue. I am currently preparing for my CKA exam and I would like to know if and done some work on this issue. I am active learner and i would prove to be beneficial if you guys just explain me this issue in more details. Thank you
/assign
Hey can u please review the changes i made


Question: import ( "errors" <ins> "fmt" </ins> "os" "path" "strings"

During iscsi volume cleanup, kubelet reads from and retrieves a list of device paths where iscsi volume mounts on. The device path was expected to be in the form of . But currently the device path in is in the form of , thus iscsi panicked . I suspect this panic started to happen after switched to use from . seems to always resolve device path (see output below), and thus the expected iscsi configuration in the device path no longer exists.


Question: Effect: v1.TaintEffectNoExecute, } <del> nodeConditionToTaintKeyStatusMap = map[v1.NodeConditionType]struct { taintKey string // noScheduleStatus is the condition under which the node should be tainted as not schedulable for this // NodeConditionType noScheduleStatus v1.ConditionStatus }{ </del> <ins> // map {NodeConditionType: {ConditionStatus: TaintKey}} // represents which NodeConditionType under which ConditionStatus should be // tainted with which TaintKey // for certain NodeConditionType, there are multiple {ConditionStatus,TaintKey} pairs nodeConditionToTaintKeyStatusMap = map[v1.NodeConditionType]map[v1.ConditionStatus]string{ </ins> v1.NodeReady: { <del> taintKey:         algorithm.TaintNodeNotReady, noScheduleStatus: v1.ConditionFalse, </del> <ins> v1.ConditionFalse:   algorithm.TaintNodeNotReady, v1.ConditionUnknown: algorithm.TaintNodeUnreachable, </ins> }, v1.NodeMemoryPressure: { <del> taintKey:         algorithm.TaintNodeMemoryPressure, noScheduleStatus: v1.ConditionTrue, </del> <ins> v1.ConditionTrue: algorithm.TaintNodeMemoryPressure, </ins> }, v1.NodeOutOfDisk: { <del> taintKey:         algorithm.TaintNodeOutOfDisk, noScheduleStatus: v1.ConditionTrue, </del> <ins> v1.ConditionTrue: algorithm.TaintNodeOutOfDisk, </ins> }, v1.NodeDiskPressure: { <del> taintKey:         algorithm.TaintNodeDiskPressure, noScheduleStatus: v1.ConditionTrue, </del> <ins> v1.ConditionTrue: algorithm.TaintNodeDiskPressure, </ins> }, v1.NodeNetworkUnavailable: { <del> taintKey:         algorithm.TaintNodeNetworkUnavailable, noScheduleStatus: v1.ConditionTrue, </del> <ins> v1.ConditionTrue: algorithm.TaintNodeNetworkUnavailable, </ins> }, v1.NodePIDPressure: { <del> taintKey:         algorithm.TaintNodePIDPressure, noScheduleStatus: v1.ConditionTrue, </del> <ins> v1.ConditionTrue: algorithm.TaintNodePIDPressure, </ins> }, } taintKeyToNodeConditionMap = map[string]v1.NodeConditionType{ algorithm.TaintNodeNotReady:           v1.NodeReady, <ins> algorithm.TaintNodeUnreachable:        v1.NodeReady, </ins> algorithm.TaintNodeNetworkUnavailable: v1.NodeNetworkUnavailable, algorithm.TaintNodeMemoryPressure:     v1.NodeMemoryPressure, algorithm.TaintNodeOutOfDisk:          v1.NodeOutOfDisk,

In autoscaling tests, we verify that when a node suddenly disappears from the cluster, Cluster Autoscaler will add another one to replace it. With TaintNodesByCondition (enabled by default in ), this is now failing: https://k8s- Every time a node disappears, a subset of pods become NodeLost and aren't rescheduled on a new node. /cc /sig node
I can work on this.
just got merged. Let's check result of autoscaling testings tomorrow: https://k8s-
latest autoscaling testing is green. I think we can close this issue.
Thanks for fixing this!


Question: Steps: math.MaxInt32, Cap:   maxBackoff, }, <del> backoffTimer:         c.NewTimer(0), </del> <ins> backoffTimer:         nil, </ins> initialBackoff:       initBackoff, lastBackoffStart:     c.Now(), backoffResetDuration: resetDuration,

I noticed that this code: was printing which should not happen. Also occurs in wait.Forever. I suspect that in regressed this as that was the most recent major update and I only started seeing this on a recent apimachinery bump.
/sig api-machinery /assign as the reviewer
GitHub didn't allow me to assign the following users: zhan849. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /sig api-machinery /assign as the reviewer Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
This is a correctness regression because the wait methods expect not to be run run more often than the period and may guard expensive behavior. Suggest a revert followed by a more rigorous test and review on merge.
thanks for reporting this, and I apologize about the regression I made. I did some quick test, and I am able to reproduce it occasionally, which indicate a race somewhere: Test: Failed output: Successful output? In the original code I checked in, I initialized the timer in the backoff manager construction time: However, after I changed to initializing the timer when is called, I'm not able to reproduce the race. The results indicated that the timer initialization is likely to be the place that caused the regression , and seems to be able to get fixed by change the way the timer gets initialized. However I'm unable to identify why it could cause such race the first time. Are you able to provide any insights? /cc
why are we initializing a timer with a zero duration? that looks related to this issue:
I see us resetting a timer we have no guarantee is stopped/expired:
dropped the information needed to accurately stop/reset the timer as documented
Hi thanks for providing the information! seems like the correct place to initialize the timer is in the "Backoff" method, since zero duration is problematic.

When I was putting up the interface, I documented that the timer will not be drained, and the reason is that returns a timer explicitly for user to block on that, so it might be confusing to block in itself Shall we change the impl to always drain the channel in backoff instead or relying on user to use it correctly? with this being said, will be changed to , which means that the will still rely on user's providing information about whether the channel is drained or not.
/cc thoughts? ^^
- sorry for that - we hit the bug with resetting timer not the first time, I should have caught that (I remember at least two other bugs with that). I'm not sure revert is a good option, because there were PRs that were chained on that: - this would have to be a very careful manual revert which I'm not sure it justifies the risk, given that 1.18 is already out. Currently the behavior is unpredictable, because if getNextbackoof() called for the first time is non-zero, it may or may not be ready immediately depending on the race. That is obviously bad. Initializing with infinity may solve this particular problem, but I need to think more about it to ensure that we don't have any problems in the later phase too...
thanks ! and sry again I did not read the timer's doc carefully enough but simply followed the Clock interface :( To fix, we can also initialize the timer to nil and create a fresh one when is first called, it's safe as this component is designed to run in single-threaded environment. But to ensure timer is drained before reset, we need to have user explicitly pass in "if timer is drained" into the to accurately decide if needs to drain. Or, is it ok to document the behavior and assume user will drain on before calling next ?
Very quick (I might have missed something) glance from the code suggests, that the only place we would need to change is actually BackoffUntil method. So passing the "if timer is drained" bit seems less error prone imho.
We should avoid changing the interface in a patch release.

I think initializing the timer in Backoff() and documenting the returned timer must be expired and drained before calling Backoff() again is the most compatible thing we can do:
Thanks and I'm also more inclined to Jordan's proposal, besides avoid interface change, if user are able to accurately pass "if timer is drained" info, it's fair to let user finish draining. do you have a PR already? of if not, I can make one, and add more tests if possible about this failure case.
I do not have a PR open


Question: }) errorCases := map[string][]api.Container{ "zero-length name":     {{Name: "", Image: "image", ImagePullPolicy: "IfNotPresent", TerminationMessagePolicy: "File"}}, <ins> "zero-length-image":    {{Name: "abc", Image: "", ImagePullPolicy: "IfNotPresent", TerminationMessagePolicy: "File"}}, </ins> "name > 63 characters": {{Name: strings.Repeat("a", 64), Image: "image", ImagePullPolicy: "IfNotPresent", TerminationMessagePolicy: "File"}}, "name not a DNS label": {{Name: "a.b.c", Image: "image", ImagePullPolicy: "IfNotPresent", TerminationMessagePolicy: "File"}}, "name not unique": {

Is this a BUG REPORT or FEATURE REQUEST? BUG Kubernetes version (use ): kubernetes 1.5+ Environment: n/a What happened: examining events that happened in my cluster, i see an event like the following: this surprised me, as it implies a container had an imageName=" " What you expected to happen: i expected a validation failure error if a [i].imageName had leading or trailing whitespace. How to reproduce it (as minimally and precisely as possible): kubectl create -f my-pod kubectl edit pod/my-pod -o json change image to " " save Anything else we need to know: nope
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
discovered when looking at odd event messages in production cluster.


Question: fmt.Fprintf(out, "    Ready:t%vn", printBool(status.Ready)) fmt.Fprintf(out, "    Restart Count:t%dn", status.RestartCount) <ins> fmt.Fprintf(out, "    Variables:n") for _, e := range container.Env { if e.ValueFrom != nil && e.ValueFrom.FieldRef != nil { valueFrom := envValueFrom(pod, e) fmt.Fprintf(out, "      %s:t%s (%s:%s)n", e.Name, valueFrom, e.ValueFrom.FieldRef.APIVersion, e.ValueFrom.FieldRef.FieldPath) } else { fmt.Fprintf(out, "      %s:t%sn", e.Name, e.Value) } } </ins> } } <ins> func envValueFrom(pod *api.Pod, e api.EnvVar) string { internalFieldPath, _, err := api.Scheme.ConvertFieldLabel(e.ValueFrom.FieldRef.APIVersion, "Pod", e.ValueFrom.FieldRef.FieldPath, "") if err != nil { return "" // pod validation should catch this on create } valueFrom, err := fieldpath.ExtractFieldPathAsString(pod, internalFieldPath) if err != nil { return "" // pod validation should catch this on create } return valueFrom } </ins> func printBool(value bool) string { if value { return "True"

A "kubectl describe" on a pod ID give a lot of informations about that pod but don't you think it should also print the pod's env var ?


Question: }, }, { <del> name: "custom cloudprovider", </del> <ins> name: "custom cloudprovider for v1.12.0-beta.2", </ins> cfg: &kubeadmapi.ClusterConfiguration{ Networking:        kubeadmapi.Networking{PodSubnet: "10.0.1.15/16"}, CertificatesDir:   testCertsDir, <del> KubernetesVersion: "v1.7.0", </del> <ins> KubernetesVersion: "v1.12.0-beta.2", </ins> }, expected: []string{ "kube-controller-manager",

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: When I use kubeadm to create an old version k8s cluster, the controller manager pod can not work well. The command like this : "kubeadm init --kubernetes-version=v1.11.2" The error of kube controller manager pod like this : What you expected to happen: The controller manager pod can work well. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): v1.12.0 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): CentOS 7 Kernel (e.g. ): Install tools: kubeadm Others:
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-sig contributor-experience /area kubeadm
Please, use when filing kubeadm issues. /sig cluster-lifecycle
OK, thank you for reminding me.


Question: }) }) <ins> var _ = framework.KubeDescribe("Pet set recreate [Slow] [Feature:PetSet]", func() { f := framework.NewDefaultFramework("pet-set-recreate") var c *client.Client var ns string labels := map[string]string{ "foo": "bar", "baz": "blah", } headlessSvcName := "test" podName := "test-pod" petSetName := "web" petPodName := "web-0" BeforeEach(func() { framework.SkipUnlessProviderIs("gce", "vagrant") By("creating service " + headlessSvcName + " in namespace " + f.Namespace.Name) headlessService := createServiceSpec(headlessSvcName, "", true, labels) _, err := f.Client.Services(f.Namespace.Name).Create(headlessService) framework.ExpectNoError(err) c = f.Client ns = f.Namespace.Name }) AfterEach(func() { if CurrentGinkgoTestDescription().Failed { dumpDebugInfo(c, ns) } By("Deleting all petset in ns " + ns) deleteAllPetSets(c, ns) }) It("should recreate evicted petset", func() { By("looking for a node to schedule pet set and pod") nodes := framework.GetReadySchedulableNodesOrDie(f.Client) node := nodes.Items[0] By("creating pod with conflicting port in namespace " + f.Namespace.Name) conflictingPort := api.ContainerPort{HostPort: 21017, ContainerPort: 21017, Name: "conflict"} pod := &api.Pod{ ObjectMeta: api.ObjectMeta{ Name: podName, }, Spec: api.PodSpec{ Containers: []api.Container{ { Name:  "nginx", Image: "gcr.io/google_containers/nginx-slim:0.7", Ports: []api.ContainerPort{conflictingPort}, }, }, NodeName: node.Name, }, } pod, err := f.Client.Pods(f.Namespace.Name).Create(pod) framework.ExpectNoError(err) By("creating petset with conflicting port in namespace " + f.Namespace.Name) ps := newPetSet(petSetName, f.Namespace.Name, headlessSvcName, 1, nil, nil, labels) petContainer := &ps.Spec.Template.Spec.Containers[0] petContainer.Ports = append(petContainer.Ports, conflictingPort) ps.Spec.Template.Spec.NodeName = node.Name _, err = f.Client.Apps().PetSets(f.Namespace.Name).Create(ps) framework.ExpectNoError(err) By("waiting until pod " + podName + " will start running in namespace " + f.Namespace.Name) if err := f.WaitForPodRunning(podName); err != nil { framework.Failf("Pod %v did not start running: %v", podName, err) } var initialPetPodUID types.UID By("waiting until pet pod " + petPodName + " will be recreated and deleted at least once in namespace " + f.Namespace.Name) w, err := f.Client.Pods(f.Namespace.Name).Watch(api.SingleObject(api.ObjectMeta{Name: petPodName})) framework.ExpectNoError(err) // we need to get UID from pod in any state and wait until pet set controller will remove pod atleast once _, err = watch.Until(petPodTimeout, w, func(event watch.Event) (bool, error) { pod := event.Object.(*api.Pod) switch event.Type { case watch.Deleted: framework.Logf("Observed delete event for pet pod %v in namespace %v", pod.Name, pod.Namespace) if initialPetPodUID == "" { return false, nil } return true, nil } framework.Logf("Observed pet pod in namespace: %v, name: %v, uid: %v, status phase: %v. Waiting for petset controller to delete.", pod.Namespace, pod.Name, pod.UID, pod.Status.Phase) initialPetPodUID = pod.UID return false, nil }) if err != nil { framework.Failf("Pod %v expected to be re-created atleast once", petPodName) } By("removing pod with conflicting port in namespace " + f.Namespace.Name) err = f.Client.Pods(f.Namespace.Name).Delete(pod.Name, api.NewDeleteOptions(0)) framework.ExpectNoError(err) By("waiting when pet pod " + petPodName + " will be recreated in namespace " + f.Namespace.Name + " and will be in running state") // we may catch delete event, thats why we are waiting for running phase like this, and not with watch.Until Eventually(func() error { petPod, err := f.Client.Pods(f.Namespace.Name).Get(petPodName) if err != nil { return err } if petPod.Status.Phase != api.PodRunning { return fmt.Errorf("Pod %v is not in running phase: %v", petPod.Name, petPod.Status.Phase) } else if petPod.UID == initialPetPodUID { return fmt.Errorf("Pod %v wasn't recreated: %v == %v", petPod.Name, petPod.UID, initialPetPodUID) } return nil }, petPodTimeout, 2*time.Second).Should(BeNil()) }) }) </ins> func dumpDebugInfo(c *client.Client, ns string) { pl, _ := c.Pods(ns).List(api.ListOptions{LabelSelector: labels.Everything()}) for _, p := range pl.Items {

Description of problem Create a petset, when pod is evicted, the petset can't create a new pod on other node and make sure pod is running. Kubernetes version (use ): v1.4.0-alpha.2.1427+ How to reproduce it (as minimally and precisely as possible): kubele eviction-hard="memory.available<${value}" a peset and wait all pod is running node MemoryPressure=true all pod status What happened: is evicted and petset can't create new pod and make sure pod is running. What you expected to happen: create new pod on other node and make sure pod is running.
Also when node come back with sufficient memory, petset can't make pod running again.
FYI
cc
Was able to reproduce it. Pet set controller just loops with the following logs I0830 12:32:52. 9 ] Syncing PetSet default/cockroachdb with 3 pets I0830 12:32:52. 9 ] PetSet cockroachdb blocked from scaling on pet cockroachdb-0 I0830 12:32:52. 9 ] PetSet cockroachdb waiting on unhealthy pet cockroachdb-0 I0830 12:32:52. 9 ] PetSet cockroachdb waiting on unhealthy pet cockroachdb-1 I0830 12:32:52. 9 ] PetSet cockroachdb waiting on unhealthy pet cockroachdb-2
-- do you have the petset YAML that we can evaluate? What is the quality of service tier for the pods produced by the petset? Can you include the of an evicted petset pod so we can see the final resource requirements spec?
I am able to reproduce it with nginx pet set in e2e test. Here is yaml output of it Also it seems that i have working fix and almost finished debugging e2e test (patchset is linked above). Will be grateful for review.
Ah, this helps clear things up... the linked pr shows the problem in petset controller and not kubelet...
Does it make sense to set milestone 1.4 for this issue? I think the fix is ready for review


Question: // Test access to the volume from pods on different node TestConcurrentAccessToSingleVolume(l.config.Framework, l.cs, l.ns.Name, <del> l.config.ClientNodeSelection, resource.Pvc, numPods, true /* sameNode */) </del> <ins> l.config.ClientNodeSelection, resource.Pvc, numPods, true /* sameNode */, false /* readOnly */) }) // This tests below configuration: // [pod1] [pod2] // [   node1   ] //         /     <- same volume mode (read only) //   [volume1] ginkgo.It("should concurrently access the single read-only volume from pods on the same node", func() { init() defer cleanup() numPods := 2 if !l.driver.GetDriverInfo().Capabilities[CapMultiPODs] { e2eskipper.Skipf("Driver %q does not support multiple concurrent pods - skipping", dInfo.Name) } // Create volume testVolumeSizeRange := t.GetTestSuiteInfo().SupportedSizeRange resource := CreateVolumeResource(l.driver, l.config, pattern, testVolumeSizeRange) l.resources = append(l.resources, resource) // Initialize the volume with a filesystem - it's going to be mounted as read-only below. initializeVolume(l.cs, l.ns.Name, resource.Pvc, l.config.ClientNodeSelection) // Test access to the volume from pods on a single node TestConcurrentAccessToSingleVolume(l.config.Framework, l.cs, l.ns.Name, l.config.ClientNodeSelection, resource.Pvc, numPods, true /* sameNode */, true /* readOnly */) </ins> }) // This tests below configuration:

What happened: Running two pods that use the same iSCSI volume, one of them fails with: What you expected to happen: Both pods should be running Anything else we need to know?: It's caused by iscsi volume plugin persiting iSCSI settings into the mount directory itself (e.g. ): volume plugin creates the global mount directory. volume plugin creates there. volume plugin mounts the volume there. This shadows the file, but it's fine, as long as no other pod uses the same volume on the same node. creates a second pod that uses the volume. volume plugin tries to create in the global volume directory. Since the volume is mounted, it will either writes the file to the volume (!) or fails, if the volume is read-only.
/sig storage /assign /priority impotant-soon
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig storage /assign /priority impotant-soon Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/priority important-soon
Our e2e did not catch this, because they always create only one pod when testing read-only volumes.


Question: return nil } <ins> func (plugin *cniNetworkPlugin) syncNetworkConfig(pluginDir, vendorCNIDirPrefix string) { network, err := getDefaultCNINetwork(pluginDir, vendorCNIDirPrefix) if err != nil { glog.Errorf("error updating cni config: %s", err) return } plugin.setDefaultNetwork(network) } func (plugin *cniNetworkPlugin) getDefaultNetwork() *cniNetwork { plugin.RLock() defer plugin.RUnlock() return plugin.defaultNetwork } func (plugin *cniNetworkPlugin) setDefaultNetwork(n *cniNetwork) { plugin.Lock() defer plugin.Unlock() plugin.defaultNetwork = n } func (plugin *cniNetworkPlugin) checkInitialized() error { if plugin.getDefaultNetwork() == nil { return errors.New("cni config unintialized") } return nil } </ins> func (plugin *cniNetworkPlugin) Name() string { return CNIPluginName } func (plugin *cniNetworkPlugin) SetUpPod(namespace string, name string, id kubecontainer.ContainerID) error { <ins> if err := plugin.checkInitialized(); err != nil { return err } </ins> netnsPath, err := plugin.host.GetRuntime().GetNetNS(id) if err != nil { return fmt.Errorf("CNI failed to retrieve network namespace path: %v", err)

Host network pods should be able to be created before the container network exists. Pod syncs for pods on the container network should fail until the container network is configured. cc
ref:
I think this is a thing we should make working before v1.4, so adding the milestone. Complain if you disagree. Also, does volunteer to make a PR? :)
Going to make a first pass at this in
Great!
/cc


Question: return i.String() < j.String(), nil case reflect.Ptr: return isLess(i.Elem(), j.Elem()) <ins> case reflect.Struct: // special case handling lessFuncList := []structLessFunc{timeLess} if ok, less := structLess(i, j, lessFuncList); ok { return less, nil } // fallback to the fields comparision for idx := 0; idx < i.NumField(); idx++ { less, err := isLess(i.Field(idx), j.Field(idx)) if err != nil || !less { return less, err } } return true, nil case reflect.Array, reflect.Slice: // note: the length of i and j may be different for idx := 0; idx < integer.IntMin(i.Len(), j.Len()); idx++ { less, err := isLess(i.Index(idx), j.Index(idx)) if err != nil || !less { return less, err } } return true, nil </ins> default: return false, fmt.Errorf("unsortable type: %v", i.Kind()) } } <ins> // structLessFunc checks whether i and j could be compared(the first return value), // and if it could, return whether i is less than j(the second return value) type structLessFunc func(i, j reflect.Value) (bool, bool) // structLess returns whether i and j could be compared with the given function list func structLess(i, j reflect.Value, lessFuncList []structLessFunc) (bool, bool) { for _, lessFunc := range lessFuncList { if ok, less := lessFunc(i, j); ok { return ok, less } } return false, false } // compare two unversioned.Time values. func timeLess(i, j reflect.Value) (bool, bool) { if i.Type() != reflect.TypeOf(unversioned.Unix(0, 0)) { return false, false } return true, i.MethodByName("Before").Call([]reflect.Value{j})[0].Bool() } </ins> func (r *RuntimeSort) Less(i, j int) bool { iObj := r.objs[i] jObj := r.objs[j]

I'd like to be able to sort by a timestamp, e.g. but that fails with an exception: I appreciate that the root of the problem is probably outside of kubernetes, yet I hope you'd agree that sorting by a timestamp is very useful functionality.
Agree with sorting by timestamp.. Just make it clear, is of type: while only these types are allowed to be sorted:


Question: return authorizer.DecisionAllow, "always allow", nil } <del> // alwaysEmpty simulates "no authentication" for old tests func alwaysEmpty(req *http.Request) (*authauthenticator.Response, bool, error) { </del> <ins> // unsecuredUser simulates requests to the unsecured endpoint for old tests func unsecuredUser(req *http.Request) (*authauthenticator.Response, bool, error) { auth := req.Header.Get("Authorization") if len(auth) != 0 { return nil, false, nil } </ins> return &authauthenticator.Response{ User: &user.DefaultInfo{ <del> Name: "", </del> <ins> Name:   "system:unsecured", Groups: []string{user.SystemPrivilegedGroup, user.AllAuthenticated}, </ins> }, }, true, nil }

/kind failing-test <!-- If this is a flake, please add: /kind flake --/kind flake <!-- Please assign a SIG using: /sig SIG-NAME --/sig api-machinery
/assign I want to take a try
2 but actual 0 till timeout resource services, expected 2, actual 0 error waiting or ResourceQuota status: timed out waiting for the condition time later, actual state became 2 Seems polls a ResourceQuota status with timeout 10s may be not enough, we should enlarge it into 300s, that is poll for 300s timeout instead of 10s.
/triage accepted


Question: return tc.expectedHints[r][i].LessThan(tc.expectedHints[r][j]) }) if !reflect.DeepEqual(hints[r], tc.expectedHints[r]) { <del> t.Errorf("%v: Expected result to be %v, got %v", tc.description, tc.expectedHints[r], hints[r]) </del> <ins> t.Errorf("%v: Expected result to be %#v, got %#v", tc.description, tc.expectedHints[r], hints[r]) </ins> } } }

A node with: fails a pod that uses device plugin resources (empty but non-nil ) : Our device plugin sends the following registration: So we send an empty which is interpreted as "no possible NUMA affinities": How is this different from value? I can workaround the problem by sending but it's not how this is supposed to work. Can the two if's combined or should we updated the documentation? My environment: a single-node k8s cluster w/ device plugin that does The node needs to run two kinds of pods: those that need/use topology single-numa-node and pods that just use a device plugin resources that does not provide topologyhints. <details</details<detailsN/A (kubeadm single node cluster) </details<details</details<detailskubeadm </details<detailscontainerd </details<detailsintel SGX device plugin v0.24 </details
/cc
It's possible that better documentation is needed, but we could also probably do more to handle this better in the device plugin. The implementation you pointed out above relies on depicting "don't care" and depicting "I tried but couldn't generate any hints that provide alignment". One could argue though that should have generated a set of hints and not a . I would need to look at the code again to know for sure if this makes sense or not though.
This interpretation makes sense but Right, if the hint provider fails to provide hints, I guess the pod should not be blamed (by not making it to run) for that. /sig node
/triage accepted /priority backlog
/cc
I can at least help with the documentation part if we decide that it alone needs changing.
I'm reviewing this area for other reasons, we should indeed calrify the hint API expectations. Up until now my rule of thumb was pretty much "what cpumanager does" which has room for improvement.
This could be isolated to only by returning when : when running is pointless, IMO
Submitted a
The corresponding documentation update: