Question: } func (o *DeleteOptions) deleteResource(info *resource.Info, deleteOptions *metav1.DeleteOptions) error { <del> // TODO: this should be removed as soon as DaemonSet controller properly handles object deletion // see https://github.com/kubernetes/kubernetes/issues/64313 for details mapping := info.ResourceMapping() if mapping.Resource.GroupResource() == (schema.GroupResource{Group: "extensions", Resource: "daemonsets"}) || mapping.Resource.GroupResource() == (schema.GroupResource{Group: "apps", Resource: "daemonsets"}) { if err := updateDaemonSet(info.Namespace, info.Name, o.DynamicClient); err != nil { return err } } </del> if err := resource.NewHelper(info.Client, info.Mapping).DeleteWithOptions(info.Namespace, info.Name, deleteOptions); err != nil { return cmdutil.AddSourceToErr("deleting", info.Source, err) }

/kind bug /priority critical-urgent What happened: In I'm removing all the reapers, unfortunately when invoking foreground deletion on a DaemonSet the delete operation hangs and controller indefinitely tries to manage a removed DS. What you expected to happen: A foreground deletion properly removes DaemonSet and all its dependents. How to reproduce it (as minimally and precisely as possible): Invoke a delete operation against kube API (without using command). Anything else we need to know?: This behavior makes it so that none of the consumers of the API is not able to remove the DS controller due to present in the DS reaper. A working fix that passed the tests with the aforementioned PR is in but I'm not 100% it's correct. This is an issue that needs fixing before 1.11 goes out. /assign /sig apps /cc
fyi
I filed to fix it. As mentioned in the PR, this is because the DaemonSet controller races with garbage collector when it tries to update DaemonSet status of the DaemonSet being deleted. Here's what happens when hangs: /something performs a foreground deletion on a DaemonSet finalizer and DeletionTimestamp are both set history objects (ControllerRevisions) and pods are being deleted by garbage collector; meanwhile, DaemonSet controller tries to update DaemonSet status. Updating DaemonSet status requires constructing DaemonSet history objects, to figure out current revision and which pods do/don't belong to current revision updating DaemonSet status, DaemonSet controller tries to create a DaemonSet history object that matches current DaemonSet spec collector then tries to delete that DaemonSet history object. And repeat. Because we can't make DaemonSet pods be deleted before DaemonSet history objects (DaemonSet history objects don't own DaemonSet pods!), we cannot reliably calculate DaemonSet status without history objects anyways. Therefore, we don't update DaemonSet status for DaemonSet being deleted. Note that the reason why the kubectl delete hack works is because it forces DaemonSet pods to be removed before history objects.
[MILESTONENOTIFIER] Milestone Issue: Up-to-date for process Note: This issue is marked as , and must be updated every 1 day during code freeze. Example update: <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed.

: Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details


Question: "//vendor/k8s.io/apimachinery/pkg/apis/meta/v1:go_default_library", "//vendor/k8s.io/apimachinery/pkg/util/uuid:go_default_library", "//vendor/k8s.io/apimachinery/pkg/util/wait:go_default_library", <ins> "//vendor/k8s.io/client-go/kubernetes:go_default_library", </ins> "//vendor/k8s.io/client-go/kubernetes/typed/certificates/v1beta1:go_default_library", <ins> "//vendor/k8s.io/client-go/rest:go_default_library", </ins> "//vendor/k8s.io/client-go/util/cert:go_default_library", ], )

Is this a BUG REPORT or FEATURE REQUEST? (choose one): TESTING REQUEST Spawned from: It would be nice with some e2e tests for the Node Authorizer so consumers of it (kubeadm, GCE, GKE, etc.) could easily make sure it's enabled and working, just by editing when running e2e's. I wrote a quick bash script of what it could do: The bash script above is in no way comprehensive enough, but on the other hand it took ~5mins to write :) For now (1.7), we might use something like this bash script in our kubeadm e2e CI job to make sure the Node Authorizer/Admission plugin is enabled correctly. cc
was assigned for triaging this... Does such an e2e test already exist somewhere? said that he could take this on if not...
no negative e2e test exists (make sure things the Node authorizer disallows fail) negative integration tests exist at negative e2e tests for this will be tricky, since you'll need node credentials and a specific authz and admission config.
See the above bash script, I was able to do it purely from kubectl by impersonating the group and node user... If we feature-gate this e2e test, we can run it for the deployments that actually have unique node credentials
cc this maybe could be a good task for you... :)
Cool, thanks for point me this.
You basically only have to implement these commands as an e2e test initially (by impersonating a node in the cluster). Then when that's done, we can check if there are more than these four cases that needs to be verified. This should go under a separate and be enabled in deployments (like kubeadm) where nodes have unique credentials.
I didn't find a way to simulate a node credential, do you have any good idea ? after reading the integration testing code, I fell it's enough the guarantee the quality, WDYT?


Question: for _, sample := range samples { count := int64(sample.Value) operation := string(sample.Metric["operation_name"]) <ins> metricPluginName := string(sample.Metric["volume_plugin"]) if len(pluginName) > 0 && pluginName != metricPluginName { continue } </ins> result.latencyMetrics[operation] = count } case "storage_operation_status_count":

Which jobs are flaking: () () Which test(s) are flaking: Reason for failure: Timeout: Anything else we need to know: This test flaking about 10% of the time in this job. /sig storage /kind flake /priority important-soon cc
/remove-kind failing-test
/assign
I noticed one issue with the test is that it's not filtering metrics by plugin name. So it will just count the last plugin's metric. I'm not sure if the metrics collected are in any sorted order or not. If not, then when you have a suite that runs multiple plugins, then you might be pulling from the wrong metric.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-lifecycle rotten
still very prevalent in sig-release dashboards
I have pushed which should hopefully fix this.
Also PR against 1.17 branch -


Question: {UpdatePodFailure, simpleSetFn}, {UpdateSetStatusFailure, simpleSetFn}, {PodRecreateDeleteFailure, simpleSetFn}, <ins> {NewRevisionDeletePodFailure, simpleSetFn}, </ins> } for _, testCase := range testCases {

It was found during a review that stateful set status might not be correctly updated when new revision is being rolled out and API error occurs when deleting a pod. The status should be updated correctly Not easy to reproduce, but should be possible with this unit test No response_ <details</details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig apps


Question: "github.com/Azure/azure-sdk-for-go/services/storage/mgmt/2017-10-01/storage" "github.com/golang/glog" "k8s.io/api/core/v1" <ins> "k8s.io/apimachinery/pkg/api/resource" </ins> "k8s.io/apimachinery/pkg/types" "k8s.io/kubernetes/pkg/cloudprovider/providers/azure" "k8s.io/kubernetes/pkg/util/mount"

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: According to , some volumes(e.g. aws, nfs, ceph) has already supported PV resizing, need to add PV resize support for azure disk What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: PV resize doc is here: Update: For azure disk, there is blocking issue: it does not support disk resize on a Running VM. I will try to work on resizing an azure disk which is not in attached state. May dig into more details later. Environment: Kubernetes version (use ): v1.10.0 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: /sig azure /assign
Hi , is PV size grow feature still in alpha in master now? How do I open this feature, set in kubelet, kube-apiserer and kube-controller-manager config?
I have figured out that should be set in kubelet, kube-apiserer, kube-controller-manager, kube-scheduler config, and also need to enable in apiserver admission control. It works well.
should we go ahead and close this issue?
also file needs to be updated to allow resizing. This plugin provides additional checks for resizing.
For azure disk, there is blocking issue: it does not support disk resize on a Running VM. I would suggest work on first, could you review , it works well in my testing env.
Adding priority important-soon to this issue so that it can stay in 1.10. If that is incorrect, please update. priority important-soon
/priority important-soon
PR has been merged, can this issue be closed?
yup, looks like it's in. Thanks for flagging! /close
you can't close an active issue unless you authored it or you are assigned to it, Can only assign issues to org members and/or repo collaborators.. <detailsIn response to : yup, looks like it's in. Thanks for flagging! /close Instructions for interacting with me using PR comments are available .

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Awww shucks, looks like I don't have magic powers. please close this when you have a chance?
has been merged, while for azure disk work is just in investigation state mainly because there is blocking issue: it does not support disk resize on a Running VM.
could someone move this issue out of milestone v1.10, thanks.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Why was this closed? This is not fixed by that crappy PR. This is the kubernetes repo, and this refers to k8s apis being implemented correctly iyam. The PR that you closed this with does NOT support disk resize on a Running VM. Your words Please reopen this.
pls track this issue for online resize feature:


Question: } }() <del> // Set up 100 loops creating and reading custom resources </del> <ins> // Set up 100 loops creating and reading and watching custom resources </ins> for i := 0; i < 100; i++ { wg.Add(1) go func(i int) {

Reproduced on 1.11.7 as well as master-ish () Deleting a CRD doesn't seem to cause the API server to terminate watches referencing the corresponding CRs, but the watches also don't properly resume if the CRD is created again, meaning that you have to wait for watch timeouts to actually get new watch events. We should probably kill open watches on CRD deletion, so that clients are notified and are forced to re-watch. Either that, or seamlessly resume watches if a CRD is recreated. <details</summary</details<details</summary</detailsthe above CRD: the above CR: (elsewhere) and grab a resource version (elsewhere) the lack of watch events After watches time out, things can re-establish, but that can be a while.
/assign
/cc sttts
/area custom-resources
it's sort a related thread w/ . for now CRD's update will terminate CR's watch connection, the other way round we might fix'em together. one possible fix suggested by months ago is that we make the storage to be a runnable object, e.g. by which we can clearily define the lifecycle of CR's storage
fixed in

What happened: If you establish a WATCH request on a list of CRs, then edit the CRD's object, you will stop getting MODIFIED events on the WATCH request. The WATCH request doesn't fail or report an error, but will not return more MODIFIED events until the request is restarted. This causes problems for custom controllers, which become unresponsive until their requests expire. What you expected to happen: Either the request should end (so it can be restarted safely) or it should continue to return MODIFIED events. How to reproduce it (as minimally and precisely as possible): I've attached . It does the following: Create CRD and an initial CR. Start a proxy, wait for it to come up, then start a watch request. Change the CR. (This change to the CR results in a MODIFIED event) Change the section of the CRD. Make another change to the CR. (This change to the CR doesn't result in an event). I've tested this against GKE masters on 1.9, 1.10 and 1.11, as well as a kubeadm 1.12.2 cluster. The output looks like: Note that you only see MODIFIED once despite the CR being changed twice. Environment: Kubernetes version (use ): 1.11.2-gke.18, 1.12.2 Cloud provider or hardware configuration: GKE, kubeadm on amd64 OS (e.g. from /etc/os-release): Ubuntu 16.04 Kernel (e.g. ): 4.15.0-36-generic ~16.04.1-Ubuntu SMP <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc
/cc
i suppose it's related w/ yes, b/c we switched to another CR storage when you updated CRD spec.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with .

Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
fixed in


Question: if condition == nil { return &pod.CreationTimestamp } <del> return &condition.LastTransitionTime </del> <ins> if condition.LastProbeTime.IsZero() { return &condition.LastTransitionTime } return &condition.LastProbeTime </ins> } // activeQComp is the function used by the activeQ heap algorithm to sort pods.

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: In large clusters, events that trigger rescheduling of the unschedulable pods are frequent. For example, in a cluster with 1000 nodes, node updates, which are often just heartbeats, may happen at an average rate of 100/s. These events trigger rescheduling of unschedulable pods. The scheduler moves unschedulable pods to its active scheduling queue when such events are observed. There are two particularly problematic scenarios: are high-priority unschedulable pods and preemption does not help make them schedulable. For example, they may have node affinity that cannot be satisfied, or they may have pod affinity to other pods that do not exist. Such high priority pods may go to the head of the active scheduling queue and block everything behind them. are many pending pods with the same priority, but a number of them are unschedulable. When these unschedulable pods are moved to the active queue, they are expected to go behind other pods in the queue, but since our queue sorts pods by priority and these pods have the same priority, their order is unspecified. The Pop() algorithm of the heap swaps the head of the queue with the tail, effectively causing the heap to act similar to a LIFO. So, pods which are tried recently, go to the head with a high probability and are retried. This starves pods behind them in the queue. Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug /sig scheduling /assign
For case 1: How about add an field to record how many times scheduler have tried to schedule it? The bigger it is, the latter we should schedule it. Then we could consider to add it as a factor for sorting pods.
Yes, we basically need a . Another approach is to . We will pursue both of these approaches.
Well differentiating optionsounds better
Agree. On the one hand we need to reduce the frequency of unnecessary rescheduling, on the other hand we need to optimize the sorting algorithm.

And the would help to reduce the frequency. But I think "differentiate between node heartbeats vs. actual node updates" is a better approach.
Yes its a better approach indeed but what if we use the recursion there
We should have all three changes to address all known scenarios efficiently. The three changes are: back off mechanism fair sorting of pods in the scheduling queue smarter processing of node updates Node updates are not the only trigger for the scheduler to retry unschedulable pods. The scheduler also retries when running pods terminate, among a couple of other events. So, without having backoff mechanism a high priority unschedulable pod could still block head of the queue in clusters with high churn, for example, in clusters where many pods terminate frequently. Back off mechanism by itself addresses the issue in all scenarios, but the other two changes make the scheduling queue smarter and more efficient.
The three prs above are quite comprehensive : ).
All of the above items are done already.


Question: github.com/xlab/handysort => github.com/xlab/handysort v0.0.0-20150421192137-fb3537ed64a1 github.com/xordataexchange/crypt => github.com/xordataexchange/crypt v0.0.3-0.20170626215501-b2862e3d0a77 go.etcd.io/bbolt => go.etcd.io/bbolt v1.3.3 <del> go.etcd.io/etcd => go.etcd.io/etcd v0.0.0-20191023171146-3cf2f69b5738 </del> <ins> go.etcd.io/etcd => go.etcd.io/etcd v0.0.0-20191023171146-3cf2f69b5738 // 3cf2f69b5738 is the SHA for git tag v3.4.3 </ins> go.mongodb.org/mongo-driver => go.mongodb.org/mongo-driver v1.1.2 go.opencensus.io => go.opencensus.io v0.21.0 go.uber.org/atomic => go.uber.org/atomic v1.3.2

What would you like to be : From looking at our gomod deps we seem to have drifted out of date again and we should update - Why is this needed: Bug fixes, feature adds, validation. /cc
Also cc:
we currently use which is the same as go does not allow referring to a module using a v2+ semver tag unless the module name ends with
opened to add an explanatory comment in our file, which is the best we can do until etcd renames its module to follow semantic import versioning, which requires changing every package import reference
what about v3.3.10+incompatible && the other older etcd deps?
not vendored/linked/used (can verify no code from that dependency is even present in ), just in our graph, so included in replace / directives
uhhh that's weird, we need to dump viper for component config, but no one wanted todo that work.


Question: if realPath != "" { // Only create the symlink when container log path is specified and log file exists. <ins> // Delete possibly existing file first if err = ds.os.Remove(path); err == nil { glog.Warningf("Deleted previously existing symlink file: %q", path) } </ins> if err = ds.os.Symlink(realPath, path); err != nil { return fmt.Errorf("failed to create symbolic link %q to the container log file %q for container %q: %v", path, realPath, containerID, err)

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: I have a transient local disk mounted under /var/lib/docker in order to take advantage of faster IO on Azure VMs. This means that all that data is lost when a reboot occurs, but the rest of the data outside that mountpoint persists. After the reboot, the existing pods are relaunched, and kubelet is unable to create the log file symlinks because they already exist, but they are pointing to an invalid place, that container no longer exists. Here are the relevant log lines: The old symlinks are there: What you expected to happen: The correct symlinks should overwrite the old ones. How to reproduce it (as minimally and precisely as possible): Erase /var/lib/docker between reboots with existing pods. Anything else we need to know?: No. Environment: Kubernetes version (use ): Server Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.0+coreos.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-06-30T17:46:00Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration**: Azure VMs OS (e.g. from /etc/os-release): Container Linux by CoreOS 1409.6.0 (Ladybug) Kernel (e.g. ): Linux node-2-vm 4.11.9-coreos SMP Thu Jul 6 00:54:42 UTC 2017 x86_64 Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40GHz GenuineIntel GNU/Linux Install tools:
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list .

The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
I think this falls under
Reiterating the mentions to trigger a notification: <detailsIn response to : I think this falls under Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: framework.KubeDescribe("DynamicProvisioner", func() { It("should create and delete persistent volumes [Slow]", func() { <del> // added until the GKE startup includes storage.k8s.io/v1beta1 framework.SkipIfProviderIs("gke") </del> framework.SkipUnlessProviderIs("openstack", "gce", "aws", "gke") By("creating a StorageClass")

pull-kubernetes-local-e2e pull-kubernetes-local-e2e.Overall 02-01 and ` No response /sig testing
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
that's not the reason for failure, we don't exit non-zero on this. This job is running and is failing tests.
But what about this one , all tests pass but it fails because a running process is not terminated, i believe is the docker process. I think we are hit by thus the proposed solution to try to send SIG Term wait 30 seconds and if not successfully terminated try to kill the docker process.
/assign
is the PR which I mentioned in the SIG meeting, It introduced the timeout of 30 seconds as a remedy, but unfortunately issue re-appeared.
This job is running , all test passed but failed again for same issue
/assign
wasn't enough to fix this :-( , i checked with , attempts to fix this.
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: framework.KubeDescribe("DynamicProvisioner", func() { It("should create and delete persistent volumes [Slow]", func() { <del> // added until the GKE startup includes storage.k8s.io/v1beta1 framework.SkipIfProviderIs("gke") </del> framework.SkipUnlessProviderIs("openstack", "gce", "aws", "gke") By("creating a StorageClass")

We need to revert when the gke control plane is updated. This should be merged AFTER gke is ready. I think may have been merged too early. I'm doing an issue instead of a pull in case a bot did the merge.
This hasn't been touched in 5d, but is listed as blocking v1.4. Where are we on this?
am I good to re-enable the dynamic provisioning test?
can you help answer for ?
I think we're ok to re-enable this.
Opened to re-enable the test. I don't know of a way for me to trigger the gke tests on the pull before merging
Trying something i spotted for kubemark -


Question: }) } } <ins> func TestCreateServiceWithNamespace(t *testing.T) { svcName := "test-service" ns := "test" tf := cmdtesting.NewTestFactory().WithNamespace(ns) defer tf.Cleanup() tf.ClientConfigVal = &restclient.Config{} ioStreams, _, buf, _ := genericclioptions.NewTestIOStreams() cmd := NewCmdCreateServiceClusterIP(tf, ioStreams) cmd.Flags().Set("dry-run", "client") cmd.Flags().Set("output", "jsonpath={.metadata.namespace}") cmd.Flags().Set("clusterip", "None") cmd.Run(cmd, []string{svcName}) if buf.String() != ns { t.Errorf("expected output: %s, but got: %s", ns, buf.String()) } } </ins>

I'm writing a small bash script which generates some k8s resources in a namespace (other than ). When using to generate a service, the yaml which is generated does not include the defined namespace. I've also tried to move around the argument to the beginning (i.e. , but that doesn't work as well. Generated yaml should include the , when it's defined as an option. Kubernetes version (use ): Cloud provider or hardware configuration: n/a (not really important) OS (e.g: ): Kernel (e.g. ): Install tools: kubectl Network plugin and version (if this is a network-related bug): n/a Others: /sig cli
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cli
/assign


Question: }, Status: corev1.PersistentVolumeClaimStatus{}, }, <del> expectedElements: []string{"nDataSource:n  Name:      srcpvcn  Kind:      PersistentVolumeClaim"}, </del> <ins> expectedElements: []string{"nDataSource:n  Kind:      PersistentVolumeClaimn  Name:      srcpvc"}, </ins> }, { name: "snapshot-datasource",

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Miss APIGroup of DataSource. What you expected to happen: kubectl describe pvc should also describe the apigroup How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/assign
/sig storage


Question: } func (o *Options) Complete() error { <del> if len(o.ConfigFile) == 0 { glog.Warning("WARNING: all flags other than --config are deprecated. Please begin using a config file ASAP.") </del> <ins> if len(o.ConfigFile) == 0 && len(o.WriteConfigTo) == 0 { glog.Warning("WARNING: all flags other than --config, --write-config-to, and --cleanup are deprecated. Please begin using a config file ASAP.") </ins> o.applyDeprecatedHealthzAddressToConfig() o.applyDeprecatedHealthzPortToConfig() o.applyDeprecatedAlgorithmSourceOptionsToConfig() } <ins> if len(o.ConfigFile) > 0 { if c, err := o.loadConfigFromFile(o.ConfigFile); err != nil { return err } else { o.config = c } } // Apply algorithms based on feature gates. // TODO: make configurable? algorithmprovider.ApplyFeatureGates() </ins> return nil }

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: We face But there is no way to generate default config file How to reproduce it (as minimally and precisely as possible): Environment: Kubernetes version 1.9.2
I'll take a look it /sig scheduling
/assign
Hi, same issue with controller-manager
Any updates here? , and don't have this flag. Is it even will be supported?
/cc
if not set, component should be use default value; why we'd like to dump all default config?
Because the documentation on the config file format is either nonexistent or difficult to find. If the program itself can generate a template, that fulfills (most of) the need for documentation of the format.
+1 for this, from my expirence, add an option to write the default config is really helpful.
/assign
also in 1.10
Deprecating all flags, prompting folks to use config files in advance is great, except when there's no documentation nor example anywhere. That means that we can't actually migrate to the new settings, and I can't seem to find clear alpha/beta/stable status for the new config file spec, nor timeline for the removal. It is also pretty confusing that the scheduler expects us to use a config file, but the controller-manager does not?
This issue indeed applies to kube-controller-manager. And there should already had deprecation policy.
According to , the flag will be available in . Since I can't find any reference online to an example config file, perhaps one could be provided or linked to here?
will you be able to pick my patch and generate a config file? I do not get my devbox handy so can only post here until Monday.


Question: cluster/gce/configure-vm.sh:  cloud_config: ${CLOUD_CONFIG} cluster/gce/configure-vm.sh:  env-to-grains "runtime_config" cluster/gce/configure-vm.sh:  kubelet_api_servers: '${KUBELET_APISERVER}' <del> cluster/gce/coreos/node-helper.sh:# cloud_config yaml file should be passed </del> cluster/gce/gci/configure-helper.sh:      reconcile_cidr="false" cluster/gce/gci/configure-helper.sh:  local api_servers="--master=https://${KUBERNETES_MASTER_NAME}" cluster/gce/gci/configure-helper.sh:  local reconcile_cidr="true" cluster/gce/gci/configure-helper.sh:  sed -i -e "s@{{pillar['allow_privileged']}}@true@g" "${src_file}" cluster/gce/trusty/configure-helper.sh:  sed -i -e "s@{{pillar['allow_privileged']}}@true@g" "${src_file}" cluster/gce/util.sh:    local node_ip=$(gcloud compute instances describe --project "${PROJECT}" --zone "${ZONE}"  <del> cluster/juju/layers/kubernetes/reactive/k8s.py:    cluster_name = 'kubernetes' </del> cluster/juju/layers/kubernetes/reactive/k8s.py:    check_call(split(cmd.format(kubeconfig, cluster_name, server, ca))) cluster/juju/layers/kubernetes/reactive/k8s.py:    check_call(split(cmd.format(kubeconfig, context, cluster_name, user))) cluster/juju/layers/kubernetes/reactive/k8s.py:    client_key = '/srv/kubernetes/client.key' <ins> cluster/juju/layers/kubernetes/reactive/k8s.py:    cluster_name = 'kubernetes' </ins> cluster/juju/layers/kubernetes/reactive/k8s.py:    tlslib.client_key(None, client_key, user='ubuntu', group='ubuntu') cluster/lib/logging.sh:      local source_file=${BASH_SOURCE[$frame_no]} cluster/lib/logging.sh:  local source_file=${BASH_SOURCE[$stack_skip]}

seems to have broken the kubernetes service when using : With this patch: Without it: Tested with v 1.3.2.
Looks like we don't like explicitly don't want to allow 127.0.0.1 loopback address here -


Question: }) ginkgo.BeforeEach(func() { <ins> if err := lookEmitSignalCommand(); err != nil { e2eskipper.Skipf("skipping test because: %v", err) return } </ins> ginkgo.By("Wait for the node to be ready") waitForNodeReady() })

Failing on cri-o serial job. Looks like the dbus command isn't available for graceful shutdown tests that aren't running on Ubuntu. /assign /kind failing-test <!-- If this is a flake, please add: /kind flake --/sig node
/triage accepted /priority important-soon
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
hmm, is this still issue? The testgrid job is still failing even after the fix appears to be merged? !
Please take a look when you get a chance, the error still seems to be that dbus-send is unavailable... /cc
oops... I forgot to add this to the other


Question: framework.Logf("Failed to execute %q: %v, stdout: %q, stderr: %q", cmd, err, stdout, stderr) } else { framework.Logf("Tries: %d, in try: %d, stdout: %v, stderr: %v, command run in: %#v", tries, i, stdout, stderr, config.HostTestContainerPod) <del> var output map[string][]string </del> <ins> var output NetexecDialResponse </ins> if err := json.Unmarshal([]byte(stdout), &output); err != nil { framework.Logf("WARNING: Failed to unmarshal curl response. Cmd %v run in %v, output: %s, err: %v", cmd, config.HostTestContainerPod.Name, stdout, err) continue } <del> for _, hostName := range output["responses"] { </del> <ins> for _, hostName := range output.Responses { </ins> trimmed := strings.TrimSpace(hostName) if trimmed != "" { eps.Insert(trimmed)

Example failure: Stacktrace: The problem seem to be related to the fact that we're running the test on private cluster (nodes doesn't have external IP address).
/assign
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
Given that there are more issues with the NetLB tests than just this particular one, I think we should revert my PR that re-enabled them. ! Wojtek just sent a PR to do that -
But let's leave this issue open - we want to fix and re-enable it again.
The revert merged, but we didn't managed to get it in before the yesterday's run started. Today's run should be green.
Yeah - but that doesn't mean we should close it later - we should fix and reenable this test. The issue should be opened until that test is reenabled.
That's clear, I didn't say anywhere that we should close it
Ok, the tests have been disabled and they don't fail the correctness tests anymore. When it comes to fixing this particular test. I took a cursory glance and I think it is fixable. We should change the code to not expect public IPs on the nodes. And then when poking over http we should do it not from the test but proxy via some container. There is already a code written for that, see /help wanted
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
/good-first-issue
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
/remove-triage unresolved
I would like to take this /assign
- thanks!

contains brief description on how to fix that
so are we going to access all the nodes via a test container?
Not sure if I fully understand your question, so if you meant something different let me know: to make the test work in large clusters (5k nodes), we can't access all - we may want to do tricks we do in other tests like access min(cluster size, some constant (e.g. 10 or 100) if you meant "via a test container" - yes, I think it's fine to just migrate the test to always use it
Reopening - since we need to reenable this test.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Does this issue need an owner? I got a little lost wading through all of the issues/PRs attached to this issue. If I wanted to take up the reins, how do I trigger the 5k node test on a PR or is it automatically triggered?
The test is disabled. I think the crucial part is done now - which is: I can't remember what else left to be done. Unfortunately you can't do that from oss. We only have post-submits exposes as these are too expensive. We can try running it internally to check if it now works...
We reenabled this test and it seem to work. So closing this one. Thanks for all the help on it!


Question: "testing" "time" <del> v1 "k8s.io/api/core/v1" </del> <ins> "k8s.io/api/core/v1" </ins> apierrors "k8s.io/apimachinery/pkg/api/errors" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/runtime/schema" "k8s.io/apimachinery/pkg/types" <ins> "k8s.io/apimachinery/pkg/util/intstr" </ins> utilfeature "k8s.io/apiserver/pkg/util/feature" "k8s.io/client-go/informers" "k8s.io/client-go/kubernetes/fake"

Hello , I have saw the commit , can we check the for enqueue to serivce cotnroller work queue? basecause of, In many case for Public Cloud, We always abstract a set for health checks , such as TargetGroup in AWS, instanceGroup in aliyun. We can map to such a collection of instance. If the service controller to check the targetgroup changes, the traffic pass-through pod is easier to implement. In most cases, the instance that in same group has an associated port for external services, and service controller set a target port if which not set in service yaml file. On the other handWe only lose some other performance of service controller. and Implementing targetgroup mapping to targetport depends on the cloud provider. What do you think? /sig cloud-provider /sig network
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.


Question: return nil, fmt.Errorf("attempt to adopt revision owned by %v", owner) } // Use strategic merge patch to add an owner reference indicating a controller ref <del> return rh.client.Apps().ControllerRevisions(parent.GetNamespace()).Patch(revision.GetName(), </del> <ins> return rh.client.AppsV1beta1().ControllerRevisions(parent.GetNamespace()).Patch(revision.GetName(), </ins> types.StrategicMergePatchType, []byte(fmt.Sprintf( `{"metadata":{"ownerReferences":[{"apiVersion":"%s","kind":"%s","name":"%s","uid":"%s","controller":true,"blockOwnerDeletion":true}],"uid":"%s"}}`, parentKind.GroupVersion().String(), parentKind.Kind,

The statefulset controller and controller history implementations should be migrated off of the Apps client (deprecated) and onto the AppsV1BetaClient. Currently there is no user facing issue caused by the use of the deprecated resource, but we should maintain good code hygiene.
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list .
/assign kow3ns
/assign kow3ns
/cc
/assign


Question: // The comparison loops are O(N^2), but we don't expect N to be huge // (there's a hard-limit at 2^16, because they're ports; and even 4 ports would be a lot) for _, oldNodePort := range oldNodePorts { <del> if !contains(newNodePorts, oldNodePort) { </del> <ins> if contains(newNodePorts, oldNodePort) { </ins> continue } nodePortOp.ReleaseDeferred(oldNodePort)

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): K8S 1.5.3, and kubectl 1.5.3 Environment: Cloud provider or hardware configuration: azure OS (e.g. from /etc/os-release): xenial Kernel (e.g. ): latest xenial Install tools: Others: ACS-Engine cluster, 3 masters What happened: Nodeport is assigned to dashboard and to the nginx service that was started within a minute of starting the dashboard. The dashboard is being started as part of the addon manager. What you expected to happen: Each service should receive a separate node port. How to reproduce it (as minimally and precisely as possible): Run 3 master setup, with 3 API servers, 3 addon managers, 3 controllers (but one controller is elected leader). Run dashboard as an add-on. Create service shortly after dashboard is started an assigned nodeport. We run a Jenkin's job, and this repros about 30-50% of the time. Anything else we need to know: We instrumented the build and found the following: is initially created by the addon manager by issuing a POST to/api/v1/namespaces/kube-system/services. This is where port is , and snapshotted to etcd Immediately after the POST, another addon manager issues PATCH /api/v1/namespaces/kube-system/services/kubernetes-dashboard.

The post calls a release to , and this is snapshotted to etcd. API server receives the request for nginx and assigns to nginx service. all api servers start reporting message the node port for service nginx/namespace- was assigned to multiple services. Workaround: delete the nginx service, re-create the service. have you seen this before? This build has your latest changelist around , but from best we can observe is unrelated to the changes.
The attached files are the API Server Logs: Here is a timeline of events: 16:07:02.862 - - Start POST /api/v1/namespaces/kube-system/services 16:07:02.863 - - [Create() 16:07:03.173 - - allocated - in memory 16:07:03.255 - - Create] 16:07:03.107 - - bitmap.Snapshot (177), 16:07:03.256 - - Finish POST /api/v1/namespaces/kube-system/services 16:07:08.474 - - start patch k8s-dashboard 16:07:08.832 - - released from allocator bitmap, and snapshotted to etcd
because type=NodePort ? Why is this releasing the port? On purpose or bug? Ports are allocated randomly from 2767 values, and you're reliably reproducing this collision? Why did they not get the update to the service that released it? Nope. I'll try to repro.
type=NodePort ? Correct, the dashboard addon specifies is this releasing the port? On purpose or bug? Discussed with and we figured it is because the addon manager does not see dynamically created , in the addon yaml (because it is dynamically assigned). This points at a potential bug in the add-on manager that it should ignore the dynamically assigned values, ignoring the need for the patch. are allocated randomly from 2767 values, and you're reliably reproducing this collision? We are running Jenkins, and this repro's 30-50% of the time in the multi-master case. The times it does not repro is when the same API server handles both the PUT of the dashboard and the nginx services, because AllocateNext() chooses the next random value for Nginx. did they not get the update to the service that released it?

It is being released from the allocator bitmap in the registry, but not the service definition. My guess is that it is successfully being removed from the allocator bitmap, but then when is attempted to be removed from the service it fails because the service is of type NodePort. I need to instrument to confirm this. If once confirmed, suggest I look to see if the removal from service can happen first, before removing from the allocator bitmap (nodeport should be gone from service definition, before being removed from allocator registry). That way if removal from service fails, then allocator bitmap still keeps other services from locking it.
for 3) it is the same across multiple machines, I assume it's because the random number generator has the same constant seed. for 4) my guess is that on Service update we're doing the following: M1: Update to service-1 comes in without NodePort (this is an update to an existing service) M1: Allocator releases the port , writes to etcd. M1: Decides to re-allocate to the service-1, but hasn't yet written to etcd M2: New service service-2 is created without NodePort M2: Allocator also decides to allocate to service-2, writes it to etcd. M1: Allocator fails to update etcd At this point, both Service-1 and Service-2 have the same NodePort assigned to them.
And a clarification on the Addon bits: Addon manager has a service that looks like: But when there is a for it, it has in the response: so add-on manager sees a difference, and posts back something missing the which clears and resets and starts this cycle.
OK, I think I have found 1 bug in this path, but I can't convince myself it would cause this failure mode, still working on it.
The addon manager (the one we support, anyway - did you write your own?) uses , so this should be fine. The nodePort is the same as clusterIP in this regard. You can see it by repeatedly running on this and seeing the nodePort not change.
The dashboard service object does not lose port , it is only removed from etcd path /registry/ranges/servicenodeports. Also, I see the same thing using , in that the release is happening, but when I view the dashboard service object, it still shows holding .

Here's the instrumented logs: 21:50:20.7778 ] BEGIN PATCH /api/v1/namespaces/kube-system/services/kubernetes-dashboard 21:50:20.8228 ] [bitmap.Release(177), 21:50:20.8228 ] bitmap.Release(177)] 21:50:20.8381 ] [bitmap.Snapshot, 21:50:20.8381 ] bitmap.Snapshot] 21:50:20.8389 ] END PATCH /api/v1/namespaces/kube-system/services/kubernetes-dashboard
Can you share this patch?
ohh, maybe the same bug is triggering this! God this code is twisty. Are you game to try a patch? in , around line 435, look for change the to - just remove the , then rebuild and re-run? Tim
Yeah, its the standard addon manager. We see it continuously doing a to the API Server. (has moved to the server?)
apply generates a PATCH , Brendan Burns <:
On reflection all afternoon, I am pretty confident this is the bug. I'll send a PR, sans test probably, shortly , Tim Hockin <: We see it continuously doing a PATCH to the API Server. (has apply moved to the server?) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub , or mute the thread
, Tim Hockin <:
Thanks I'm trying the patch and testing now.
Jenkins is now passing 100% on 3 master HA. Also, logs no longer show the removal of the dashboard nodeport from registry. I confirmed this for kubectl apply as well on the dashboard file. Thanks for the fix!
Thanks for the challenging bug. It reminded me how much I hate this code :) , anhowe <:


Question: gvr("flowcontrol.apiserver.k8s.io", "v1beta1", "prioritylevelconfigurations"): { Stub:             `{"metadata": {"name": "conf2"}, "spec": {"type": "Limited", "limited": {"assuredConcurrencyShares":3, "limitResponse": {"type": "Reject"}}}}`, ExpectedEtcdPath: "/registry/prioritylevelconfigurations/conf2", <del> ExpectedGVK:      gvkP("flowcontrol.apiserver.k8s.io", "v1beta2", "PriorityLevelConfiguration"), </del> <ins> ExpectedGVK:      gvkP("flowcontrol.apiserver.k8s.io", "v1beta3", "PriorityLevelConfiguration"), </ins> }, // --

adds new version for in 1.26. For 1.26, we are setting the storage version to because vN-1 understands that level. In 1.27, we should change the storage version from to .
/sig api-machinery
/cc
/cc
/triage accepted
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:


Question: } }) <del> // TODO: Get rid of [DisabledForLargeClusters] tag when issue #56138 is fixed. </del> // [LinuxOnly]: Windows does not support session affinity. <del> ginkgo.It("should have session affinity work for LoadBalancer service with ESIPP on [Slow] [DisabledForLargeClusters] [LinuxOnly]", func() { </del> <ins> ginkgo.It("should have session affinity work for LoadBalancer service with ESIPP on [Slow] [LinuxOnly]", func() { </ins> // L4 load balancer affinity `ClientIP` is not supported on AWS ELB. e2eskipper.SkipIfProviderIs("aws")

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug This is a followup from filed separately now that we're not tracking this as part of 1.9 release. The tests have been disabled with [DisabledForLargeClusters] tag for now, and this fact will be mentioned in the scalability validation report for v1.9. This issue tracks fixing the test so the tag can be removed and the test ran again in the large cluster. /assign /kind e2e-test-failure
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Is this e2e test still broken?
Sorry for the confusion here. Currently this e2e test is disabled on extra large cluster due to some scalability issues with the load balancer related components. At the moment the issues haven't been completely resolved yet.
ok, adding priority so that I know it's not a 1.13 blocker. /priority important-longterm
cc - this is why the relevant tests for are not part of the e2e suite.
This issue is still valid --- some action items on top of my head: Figure out a plan to make service controller properly handle operations in parallel. Check on the status for cloud providers (especially GCP, which the test ran against) on handling large number of nodes as LB backends. /remove-triage unresolved
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or .

/lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten /lifecycle frozen
/reopen Reopening as is being reverted in
Reopened this issue. <detailsIn response to : /reopen Reopening as is being reverted in Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
these tests are flaking in xref
cc since these were enabled by
Yeah - though these failures doesn't seem to be scale related, they seem to be test related to me. This basically means nodes without public ips: It's pretty much the same thing as (just for a different test). should be addressing these kinds of errors. [Not a release blocker for sure]
BTW - has been opened directly for this one.
... so since is closed, Can we close this and re-enable for large clusters ? (i.e. remove the Tag in that disables this test) ? I'm not up to speed on the history but 2 related notes I came across digging into related test today... 's a couple of TODOs in the code referencing this specific issue, but your above direct issue is now closed, so we can close this as well right?... And if so, we can remove the TODO on the test... -enabling the test ... not sure wether the timing is good right now (given that is open right now , which sparked my stumbling upon this issue :))
We tried to enable all of them and that caused us a bunch of issues and those had to be reverted: A couple tests were enabled later in: and those seem fine. The remaining (mostly session-affinity ones) had some issue, which so far we didn't have time to debug and I haven't heard a good explanation why those were failing. It's possible that they were improved with , but I doubt those would really cause the failures we've seen in the past. So until this is debugged (or at least understood) I'm reluctant to enabling them.


Question: return true } } <ins> // Matches the message returned by api server before 1.17.0 if strings.Contains(apierr.Status().Message, "Too large resource version") { return true } </ins> return false }

What happened: Using client-go v1.18.8 to watch resources on API server versions 1.17.x and 1.18.0 - 1.18.5 After the connection to the server is temporarily interrupted the client never recovers from error See the earlier issue for steps to reproduce: This was assumed fixed by But it appears the fix for the client was (probably inadvertently) implemented in a non-backward-compatible fashion and only solves the issue when the server is running v1.18.6 or later as well.
/sig api-machinery
Cc
PR
We generally recommend having apiservers be updated first before all other components. But yeah - it's true that this doesn't work for clients being more fresh - let me take a look at your PR.
/cc
This will be addressed by (and the cherrypicking it back).
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
This has been fixed by and corresponding cherrypicks.

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: I have disconnected a node from network for a few minutes. After reconnecting, I keep receiving such error messages from kubelet on the node, even after 15 minutes in reconnected state: What you expected to happen: I expect that after network recovery kubelet reconnects to the Apiserver as before, and after a recovery such timeouts do not occur. How to reproduce it (as minimally and precisely as possible): Just have a node connected to the cluster. Then, disconnect it from the network for 3-4 minutes, then reconnect. Then observe kubelet's logs. Anything else we need to know?: I have strict tcp keepalive settings in place on master and worker nodes, but this should not be the cause. Restarting kubelet solves the issue, the error messages disappear. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: bare metal OS (e.g: ):Kernel (e.g. ):Install tools: kubeadm Network plugin and version (if this is a network-related bug): kube-router Others:
/sig api-machinery
/sig api-machinery
It seems that after some more time, kubelet recovers. Sometimes it takes one hour, sometimes two. Recovery is successful when I receive similar logs: What is the reason that it takes so long for it to recover?
The root cause is , deduplicating against that issue
Debugging, tracing the code, I am not really sure that this is the same issue as the other mentioned here. Error messages show that the client is requesting higher resource versions that is available at the server: , the former is greater by 989 than the latter. I dont know from where this resource version comes from. I've made a very stupid patch on my kubelet/client-go: I am pretty sure that this is not the best solution, but, it seems that it forces a full list every time ListAndWatch is called.
You're correct, sorry about that... I misread the timeout message. The error it is getting is coming from the server, which means kubelet -apiserver connectivity is fine.

This message comes from the watch cache, which means the watch cache in the API server is behind where the kubelet's informer was, which is pretty surprising. A few questions: Is there a single apiserver or multi-apiserver cluster? Is there a single etcd member or multiple etcd members? Is there an API server running on the node that was detached/reattached?
cc does this ring any bells?
cc for eyes on watch cache
I don't think I've hit this problem specifically. Since this could happen due to an etcd partition or due to a partition between a apiserver and etcd, the answers questions asked in seem like the appropriate next things to figure out.
Yeah - I agree with Joe. It seems like some network partition issue of the apiserver the node was connected to.
I will be able to answer the questions in detail later, but in short: there are 3 masters set up with kubeadm, haproxy load-balances the queries to the apiservers. The only thing to reproduce the issue is to disconnect the node from the network for a few minutes. Masters or etcd members are not touched, and anyway the cluster seems to be healthy.
Today, it occured on a diffrent cluster. Setup is similar: stacked etcd on master nodes, haproxy load-balances traffic between them. Just was playing with haproxy servers and the HA setup, and after a few restarts of haproxy, one of the nodes entered this state. So during a haproxy restart, all kubelets' connections to apiservers are terminated, but rarely, one of them enters this unhealthy state.
Multi apiserver cluster Stacked etcd members are present. I think In my setup it is not relevant, as all the kubelets are talking to the apiserver through the load-balancer, which picks one of the available apiservers. I am doing a rolling restarts of the nodes, and there were at least one occurence, when even one of the master nodes got stucked in this state. How can I help debugging/fixing this?
How can we proceed with this issue?

got same problem in a old cluster, ( up since 3+years) after updating to kube 1.18.3 , one node kubelet start logging those errors ( ~10 errors/minute with various resources) I've now just restarted kubelet and now seems working again.. Update: while testing I've tried to connect the kubelet of that node to each single master nodes to check if there were some kind of network issue / partitioning.. none of the master servers returned errors ( the kubelet worked fine with all master nodes bypassing the nginx proxy that manage masters HA
I think this problem is getting serious. I am not 100% sure, but today, during node restarts (including masters) coredns was migrated multiple times, and unfortunately, kube-proxy pods had not received the updated situation, thus, dns resolution was not working inside the cluster. After detecting this, a simple restart of all kube-proxy pods resolved our problem. There were no network issues during this, so I suspect we've hit the same bug, now in kube-proxy.
I am trying to dig this. Now, I've written a little go prog which uses the same reflector what kubelet uses. It just starts a watch for CSIDrivers, and I've made client-go to print the last resourceVersion. Starting the program multiple times (i.e. connecting to different masters) produces the following: Right now I dont exactly know what url is being fetched and what arguments are passed in, I still have to figure that out. But now it seems, that different masters return different resouceVersions. Etcd does not report any problems/issues.
Any ideas, suggestions where to look further?
Today I have re-initialized 2 of my 3 master nodes to make sure etcd replicas are not corrupted. Just after the nodes have been joined, the same behavior could be observed: connecting to different masters resulted in different resourceVersions.
Now, it can be seen, that querying different apiservers again return different resourceVersions: 192.168.8.60: is a haproxy load-balancing traffic between 3 masters.
please let me know where to go further. My cluster runs on PI boards, perhaps, not the fastest ones.

But if this issue depends on slow hardware, than it can pop up anytime on fast hardware as well.As I wrote, today I reinitialized 2 of my 3 masters to make sure no partitioning occured. Also, please have a look at my comments. Or, if you have a clue that my setup is broken, please let me know!
It seems that it happens to resources where no objects exist at all. Can it be?
Ahh - the fact tht it happens for types with no objects actually made me know what is happening here. Basically what is happening is that when kube-apiserver is starting, watchcache is initialized from "now" (i.e. it will be synced to resource version that was globally last in etcd at the time when it was initialized). However, later it's updated only on changes of resources of those types. So when no objects of that type is being /modified/deleted it will stay the same. So it's kind-of "working-as-implemented" from the kube-apiserver perspective. That can be improved - I'm actually already looking into reusing etcd progress-notify feature to get rid of this behavior ( & ) - I would like to have alpha for that in 1.20 and beta in 1.21 (planning to write down KEP once I'm back from my soon-starting vacation), but that shouldn't be the problem by itself. And it's not what is really biting you. The real problem is that there is a bug in reflector in: It handled the "too old RV" error, but it doesn't handle "too large RV" correctly. What it should be doing instead is to: trigger relist from etcd (to get RV=now") start watching from that RV Watch will work fine because watch will simply wait until events with larger RV appear. I will send out a PR fixing that bug (will try to do it later today).
is out for review
This has been fixed/mitigated in 1.18. Going to backport to 1.18 now. It seems 1.18 is the first version affected (the offending changes were reverted in 1.17 months ago).
This has been mitigated in 1.18 and head. The more proper fix is proposed in Closing this one.

Hello, as i saw, i have the same issue in 1.19.3 version: Our setup is 3 etcd, 2 apiserver bound on those 3 etcd and a haproxy in front of apiservers
If that happens once - that can definitely happen (and that's fine). The bug was that the components were stuck with those errors.
18.0 get same error . restart docker and kubelet return OK. so , should I update the 1.18.0 to 1.18.6+ ?


Question: containersToKeep[containerID] = empty{} } <del> if podChanged { // Also kill associated pod infra container if the pod changed. if podInfraContainer, found, _ := dockerContainers.FindPodContainer(podFullName, uid, dockertools.PodInfraContainerName); found { if err := kl.killContainer(podInfraContainer); err != nil { glog.V(1).Infof("Failed to kill pod infra container %q: %v", podInfraContainer.ID, err) } } podInfraContainerID, err = kl.createPodInfraContainer(pod) if err != nil { glog.Errorf("Failed to recreate pod infra container: %v for pod %q", err, podFullName) } containersToKeep[podInfraContainerID] = empty{} } </del> // Kill any containers in this pod which were not identified above (guards against duplicates). for id, container := range dockerContainers { curPodFullName, curUUID, _, _ := dockertools.ParseDockerName(container.Names[0])

cAdvisor API is not accessible from the node. It is accessible from inside the cadvisor container though. This is the reason for the cadvisor e2e test to fail and the recent heapster errors reported by users. The cadvisor version itself hasn't changed recently though.
cc
There were some changes recently around port forwarding ( that may be the cause.
From the node: From the cadvisor container:
reported the same issue today. Restarting the container seems to fix it. So, its something to do with the initial setup.
Could this be due to Docker 1.5? On Feb 23, 2015 6:12 PM, "Rohit Jnagal" wrote:
test machine is running Docker 1.4.1 It looks like cadvisor container is sharing its networking with the pause container as usual, but the pause container got restarted, so the networking setup isn't working anymore. Restarting cadvisor connects it to the newer instance of the pause container and it starts working again. However docker inspect shows that cadvisor is connected to network of a different pause container: And that container died soon after it started Kubelet killed the pause container. I think this is kubelet's doing. It tried to restart both pause and cadvisor container. It kills and restarts cadvisor container which connects to existing pause container. Then, it goes ahead and kills/restarts the pause container, leaving the new cadvisor container hanging. Maybe this is related to infra pods issue reported by earlier, or maybe just an ordering issue in kubelet.
Thanks for the detailed debugging :) with your report I think I found the issue. In we check if the container has changed and print the message above. After that we the POD container. We should be doing this before the container restart as you pointed out. I'm assigning this to myself and I will fix tomorrow. Will also rename the bug and re-purpose it.
cool, I was just staring at that code. All yours :+1:
Thanks for debugging the issue. This is a regression we introduced lately.
And when the test, we all assumed it was the test failing. Sorry, :)
I fell for that until I realized that the basic networking can also fail :)


Question: * [kubectl](kubectl.md)	 - kubectl controls the Kubernetes cluster manager <del> ###### Auto generated by spf13/cobra at 2015-08-26 09:03:39.977006328 +0000 UTC </del> <ins> ###### Auto generated by spf13/cobra at 2015-08-31 12:51:55.222410248 +0000 UTC </ins> <!-- BEGIN MUNGE: GENERATED_ANALYTICS --> [![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/user-guide/kubectl/kubectl_label.md?pixel)]()

kubectl should support a --dry-run option for every mutation to show what it would do, without actually doing it We might need API changes in order to make this high-fidelity. For instance, if we wanted to know what a pod config would look like after admission control populated defaults, we'd need to execute most of the creation path.
For generator types commands, I think there is a usability argument that the default when a user specifies should be to dry-run (and a user should be allowed to override that with ). I think I'd rather encourage composition of commands (pipe) than every generator command in the system creating objects. We may also want to clarify dry-run against the server vs dry run against the client - there are lots of places where we want to generate locally to a file or an editor or a pipe command before we want to create it. At this point, OpenShift has a few commands that treat -o  as "dry-run" - new-app (complex generator) and export (never expect to send it to the server). We also have a few that are designed to be compositional in a pipe always - process (take a template and turn it into a list of api objects), which never creates the objects for you (you have to pipe to ). I'm somewhat partial to having and support the presence of changing the default of to true, because that's more common and easier than having to set and . The counter argument is that is the preferred action, and is format only - except that by definition --dry-run using the default output format isn't very useful (you created a service named foo and an rc named bar). My gut is that most users once they understand what a particular generator does their default path is always going to be to review the change, and then if they want to create it they can pipe to or just by appending to their existing command. I'm not sure that's a strong enough argument to couple --dry-run and -o, but based on our experience so far it's going to be the "normal" path.
:+1: for this feature. In particular, I would like an ability to validate an object definition separate from actually creating the object. e.g. running at the validation step of my deployment pipeline.
I agree that the validation use case is important. I would distinguish --dry-run from --local.

The former would still entail contacting the server and returning the result of the mutation. The latter would imply that only local information were used.
- I'm planning to add a parameter like 'pretty' or 'labelselector' to API requests. Once the server get this parameter, it will do validation to the resource and check if request can pass admission controls, but return success before write to etcd. I've read the documented about API changing, it lists some principles when changing the api object, but parameters are not mentioned. Is there anything need to be taken special care of when adding a parameter?
We need to be careful about dry-run with admission control... some admission controllers (like quota, specifically), record things to etcd in the process of admitting an object. Also, dry-run wouldn't work with a list of objects intended to be created in order, where later objects reference earlier objects (and those references are validated at create time)
We could potentially carry "dry run" down into the admission controller request path. However, a query parameter may not be appropriate, instead there's probably a HTTP header that already covers this. We try not to mix query parameters and post bodies in many cases. , Jordan Liggitt wrote:
- Thanks for advices! I'm trying to make myself thoroughly familar with request handling process. And during this period, I may need more of your help. I think I've replied this message before but unfortunately it's not showed up here:-( - Can you provide a link to the code about the HTTP header? Thanks!
This feels like a variant of an header with a sub parameter - possibly or possibly content-type . , Victor HU wrote:
+1 for this. My use case is to understand the difference between a stack file and an existing cluster, but this command would be a substitute.
It looks like support for has been . I think this issue can be closed.
We still need to make touch the server, or at least decide that we don't want to.
you mean move server side?
has just released kubeval: From to sig-apps mailing list: I am not sure if it validates the values and other constraints checked on the server-side. Just leaving it here as it might be relevant to the feature discussion. (h/t briangrant for pointing this out).
Issues go stale after 90d of inactivity.

Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
/remove-lifecycle rotten
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
a HTTP header that already covers this. We try not to mix query parameters and post bodies in many cases. Do you remember why you said that? I'm trying to understand ...
should fix this issue, we can close it when the server-side dry-run feature reaches beta. The feature above () will fix this use-case, and even validate against the admission chain. is being built on top of server-side dry-run, and will be a good way to visualize the output of dry-run apply:
This is now beta.


Question: <ins> apiVersion: v1 kind: Service metadata: name: a spec: selector: app: test clusterIP: 10.0.0.12 ports: - port: 80 </ins>

/kind bug What happened: From I expected kubectl apply to support deleting a resource and recreating it, if changes cannot be applied. kubectl replace works: What you expected to happen: How to reproduce it (as minimally and precisely as possible): ` Change clusterIP to any IP address. Anything else we need to know?: Environment: Kubernetes version (use ): client v1.11.0, cluster v1.11.1 Cloud provider or hardware configuration: QEMU/KVM OS (e.g. from /etc/os-release): Cluster CentOS Linux release 7.5.1804 (Core), client macOS Kernel (e.g. ): 3.10.0-862.6.3.el7.x86_64 Install tools: kubeadm Others:
/cc
/assign


Question: import ( "errors" "fmt" <ins> "io/ioutil" "os" "path" </ins> "reflect" "testing" "time"

Kubernetes scheduler enables you to overwrite default Predicates/Priorities as well as configuring extenders in a "scheduler policy". In latest scheduler componentconfig API (v1alpha1), it can be a file or a configmap: However, if you defines a scheduler config file and specify a YAML policy file: It doesn't work. It's because only JSON is supported: It'd nice to enhance it with an UniversalSerilizer so as to support both JSON and YAML. /sig scheduling /help /priority important-longterm
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : Kubernetes scheduler enables you to overwrite default Predicates/Priorities as well as configuring extenders in a "scheduler policy". /help /priority important-longterm </details
/assign


Question: func (p *fakePodStateProvider) ShouldPodContainersBeTerminating(uid kubetypes.UID) bool { _, ok := p.terminating[uid] <ins> // if ShouldPodRuntimeBeRemoved returns true, ShouldPodContainerBeTerminating should also return true if !ok { _, ok = p.removed[uid] } </ins> return ok }

We observed that while a container is in the process of being torn down, but not completed, the volume gets unmounted and then subsequently detached, causing the fs to enter read-only mode. This happened after kubelet restarted. We observed that on the restart, the volume was not to DSW because the Pod was marked for deletion. In the normal case, we only remove the pod from DSW when the containers have finished teardown: However, in the reconstruction case, it looks like if a pod has started termination, but not necessarily completed, then we don't add the pod to DSW: We probably need to be consistent and change the add pod criteria to also add even if it's started terminating. The side effect is that in the normal case we may do unnecessary mounts if a pod is quickly created and deleted. We can also consider a flag to only use the stricter check during reconstruction. What happened in our env was that kubelet was crash looping, and then all the pods on the node were deleted. It looks like there was a good 30 seconds between volume unmount and container teardown. I think this may be easily reproducible if the pod has a long graceful termination period and the container ignores the stop signal. No response <details22 </details<details</details<details</details<details</details<details</details<details</details
/triage accepted
Yeah this looks problematic. What was the exact kube version btw? So it was possible that container is still writing to the volume? Do we know why fs enters read-only mode when this happens? I know the behaviour of kubelet is problematic regardless, but still. I think adding volumes belonging to terminating pod doesn't sound so bad. I wonder if we could change this to .
Does this mean instead of checking ShouldPodContainersBeTerminating to decide if we add the pod or not, we should check IsPodTerminationRequested or not to decide? Or we should use the same check ShouldPodRuntimeBeRemoved with reconstruction(this one is more consistent)? pointed out that there were already scaling challenges with volume manager processing pods it doesn't need to.
For reconstruction, we need to add the volume in DSW if there is a chance that containers are running. So is the best call to determine that.
We observed a similar issue but could only reproduce it with emptydir volumes. Do you think is the same issue?

/assign sunnylovestiramisu
thanks for the report. I think this issue could impact emptydir volumes too.
Is there any recommended way to make a pod having a long graceful termination period? Tried with some simple pods but it just terminated right away.
You can use the sample pod and instructions from here
cc
Here is my repro pod and steps: the pod shell, another window (window 2), ssh to the node another window (window 3), window 2, in window 1, exec back into the pod . The file will be gone. try . It will return When looking at kubelet logs, you'll see the log messages that indicate the volume was reconstructed without being to dsw. and UnmountVolume happen soon after: Pod lifecycle logs:
Tested locally via kind cluster. /k master is showing error of touch: cannot touch '/scratch/test': No such file or directory. cluster with the fix in this pr shows no error. And running shows no result in the log.


Question: } else { clusterEndpoints, localEndpoints, _, _ := proxy.CategorizeEndpoints(endpoints, svcInfo, proxier.nodeLabels) if onlyNodeLocalEndpoints { <del> endpoints = localEndpoints </del> <ins> if len(localEndpoints) > 0 { endpoints = localEndpoints } else { // https://github.com/kubernetes/kubernetes/pull/97081 // Allow access from local PODs even if no local endpoints exist. // Traffic from an external source will be routed but the reply // will have the POD address and will be discarded. endpoints = clusterEndpoints } </ins> } else { endpoints = clusterEndpoints }

<!-- Please only use this template for submitting enhancement requests --What would you like to be : In-cluster traffic for a Service loadbalancer IP with externalTrafficPolicy=Local should be masqueraded and routed to cluster-wide endpoints. A similar patch was to the iptables proxier in v1.15 in Why is this needed: Today, the LB IP traffic for the ipvs proxier is not differentiated from "Local" and "External" traffic, so in the case where a Service is using and traffic is originating in-cluster, the traffic will be black-holed if there are no local endpoints there.
/assign /sig network
/area ipvs
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/remove-triage unresolved
Patching kube-proxy with solves this issue.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
I tried the proposed fix in From within a POD on a node where no server PODs are executing it works fine, with the source (POD) address preserved. From an external machine where incoming packets hit a node without server PODs (where it usually is black-holed) packets goes like this; On the k8s-node without server DNAT happens VIP -POD-address. The problem is that the reply takes a short-cut (much like a DSR) but the source is now the POD address (where it should have been nat'ed back to the VIP). The reply goes all the way back to the sender who have no idea about any connection with the POD-address as source and drops it. But if the external traffic setup is correct then external traffic should never hit any nodes where the server PODs are not executing. So IMHO this fix is OK.

Here is a trace from an external source showing the SYN to the VIP address (10.0.0.90) but the SYN-ACK has the POD address (11.0.4.3) as source:
Traffic from the main netns (e.g hostNetwork:True) from a k8s-node without server fails for a different reason than traffic from an external machine. Traffic from main netns on a k8s-node will get the VIP as source! This because the assign of the address to dev sets up a local route as; SYN is sent OK (e.g 10.0.0.90: -10.0.0.90:80) but the reply to the VIP address fails. This can be "fixed" by changing the route and set the node-ip as src; Then it works.
Is it possible to add the proposed solution in in kube-proxy? It would solve the problem for all users except for PODs with and have no significant draw-backs as far as I can tell.
I will take a look at that PR soon
Would you mind if I create a PR with your update described in ? Or would you prefer to do it yourself?
Hello. To solve this problem, I developed a kubernetes controller called node-network-manager. node-network-manager works on all kubernetes nodes and adds iptables DNAT rules from CLUSTER-IP to Load Balancer IP. Please try this and give me feedback. Thanks.
/area kube-proxy
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale There is a pending PR
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale There is a pending PR
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
hello. Please tell me. Did you manage to reach external-ip from host node ? Without a way to remove or replace the route and with src ip ? I have the same problems. When accessed from POD, it works ok. But when accessing from the host node to external-ip, it does not work. Replacing or removing the route from external-ip on the kube-ipvs0 interface helps. Is there any working solution for this?
No, sorry.

I think that the route must be replaced to make it work, as described in But that is a much more intrusive (i.e dangerous) update than the tiny fix in


Question: return pod } <del> var _ = Describe("Security Context [Skipped]", func() { </del> <ins> var _ = Describe("Security Context [Feature:SecurityContext]", func() { </ins> framework := NewFramework("security-context") It("should support pod.Spec.SecurityContext.SupplementalGroups", func() {

there is a test that you marked as skipped that needs to be reenabled or otherwise taken care of (see ): "Security Context [Skipped]" Can we instead classify these as ? See for context. Happy to answer any questions if anything doesn't make sense.
This is work toward .
I like the reclassification! I would be happy to relabel the test. I'll wait for to merge though
has been merged; if you send a PR, I will review it. Thank you!


Question: "containers": [ { "name": "webserver", <del> "image": "gcr.io/google_containers/nettest:1.7", </del> <ins> "image": "gcr.io/google_containers/nettest:1.8", </ins> "imagePullPolicy": "Always", "args": [ "-service=nettest",

<!-- Please only use this template for submitting enhancement requests --What would you like to be : The documentation suggests that "The default pull policy is IfNotPresent" This is not the case when imagePullPolicy is defined under the spec section in a Pod definition without a specific value. Example: This will result in a Pod having imagePullPolicy as Always. Why is this needed: This is needed for clarification of the default behaviour. It is necessary for the appropriate handling of helm charts, in cases that the imagePullPolicy is empty.
/sig docs
The imagePullPolicy default value is .
In this paragraph you can see that In your reference it points out the default action for the kubelet, not the behaviour when imagePullPolicy is declared. Please do have in mind that the imagePullPolicy option for Never is mentioned only in this paragraph in the document.
If an image tag of "latest" is used, the imagePullPolicy defaults to to "Always": If no tag and no digest are specified on the image, an image tag of "latest" is used: The behavior when no image digest or tag is specified should be documented in the API docs for the and fields:
I am interested on helping this issue, but I am new to kubernete repo, it could take me sometime
I found the following which is well documented in "Best Practices for Configuration" It would be expected to include more information on the options in the main link under
I'd like to work on this one.
Very welcome, but yesterday also mentioned that he is following up on this issue, you can communicate with him before starting

cc
This has failed The two failures are different, and both seemingly totally legitimate. The first one is simply a pod never entering running state within the 5 minute timeout. Expected error: <*errors.errorString | 0xc208475150: { s: "gave up waiting for pod 'nettest-dtwk3' to be 'running' after 5m0s", } gave up waiting for pod 'nettest-dtwk3' to be 'running' after 5m0s not to have occurred The second is not being reachable: Dec 7 01:22:47.275: Unable to connect/talk to the internet: Get dial tcp 74.125.202.147:80: i/o timeout The latter could be a result of a missing MASQ rule on the node, or GCP egress problems. I don't think that this is worth trying to debug further in the context of this test for intra-pod communication. I'll check more generally whether we have a current intermittent problems with respect to pods entering running state, or reaching the internet.
I think this test may be causing builds to time out, too. See comments from me in .
For the future, I actually don't think we need the test at all in the networking test. Maybe even move it do a different test altogether.
This is clearly a bug. I'll send a PR shortly: Regarding the ping to , why did you put in in the first place?
hmmm, not sure what you mean by "in the first place" ? I did not write the test...
sorry, I thought that you'd written that part of the test. I will find out who wrote that, figure out why they put it there in the first place, and then either remove it, or rewrite it to work correctly. Clearly that part of the test needs at least some basic retry to work consistently.
ok great, no problem just wasnt sure what you meant. SGTM. Id be okay w/ deleting the ping entirely regardless of who wrote it, its a very old snippet which doesnt really add much to the overall test suite (and in fact is questionable wether it should even be an assertion at all )
somebody put it there for some reason. Better to understand why before just deleting it.

i looked, its actually rob who i work with :) i will ask him in the morning unless you find him first :) (rrati)
Regarding my alleged "clear bug" above, I was completely wrong. There's nothing wring with that code. And unfortunately, when this test goes into a presumed endless loop (see comment above) it does not produce any output (as the output is cached by Ginkgo until the test completes). So it's very difficult to debug. But in summary I agree with this test is probably never completing, and causing tests to time out. Someone will need to isolate this test in a environment where it produces output and can be debugged. I'll keep trying, but need to head home soon, so if anyone wants to pick it up overnight, go ahead.
Smoking gun:
hmm - I'm just wondering why only this particular test is hanging forever (if we agree that's the case). I don' think that some requests are hanging forever in this test, but not in any other test...
Closing in favor of which should fix this. Or at least some way to set client timeout, which and are still figuring out.
It's happening again.
Talked to offline. I am assigning this one to you for triage. Let me know if you need any help.
AFAICT the last failure was just the pod nettest-7h5jj not coming up
Looks like this should be resolved by your pr: too. :-)
Maybe. Though the webserver only deadlocks in the failure case, so the test is probably still flaky...
Looks like this test is now explicitly flaking by failing, as I expected.
Based on the , it took 15 minutes to pull the image. closing this one as dup of
New Failure:
This looks like a race between populating the endpoints list and the /status, /read methods of the service exposed through the master proxy. It looks like the webserver pod is just trying to confirm that it can reach all peers. We can do this synchronously after having observed all expected endpoints in the test (or re-write the test to use the same pattern we do in services e2e).
The test looks really racy to me. I think the following 15 retries prevents it to fail more often, but not completely.

I will submit a pr to enfore webservers to wait till alll endpoints are up before they start to contact each other.
New failure
I am making this P0
It took more than 4 minutes for a nettest pod to come up. The nettest container is waiting for all endpoints to be ready for 2 minutes plus 15 attempts (5 sec interval) to contact its peers. This happens in less than 4 minutes. Hence the first 2 nettest pods finished contacting peers and declared failure.
The node you want is: I see: that's too much silence ...
From the It looks like kubelet is slow to start the container. Due to the lower logging level in the gke, it's hard to guess what could've slowed down the goroutine...


Question: continue } <ins> if cSvc.Spec.Type == v1.ServiceTypeLoadBalancer { // In federation tests, e2e zone names are used to derive federation member cluster names zone := fedframework.GetZoneFromClusterName(name) serviceLBName := cloudprovider.GetLoadBalancerName(cSvc) framework.Logf("cleaning cloud provider resource for service %q in namespace %q, in cluster %q", service.Name, namespace, name) framework.CleanupServiceResources(serviceLBName, zone) } </ins> err = cleanupServiceShard(c.Clientset, name, namespace, cSvc, fedframework.FederatedDefaultTestTimeout) if err != nil { framework.Logf("Failed to delete service %q in namespace %q, in cluster %q: %v", service.Name, namespace, name, err) } <del> if service.Spec.Type == v1.ServiceTypeLoadBalancer { err = cleanupServiceShardLoadBalancer(name, cSvc, fedframework.FederatedDefaultTestTimeout) if err != nil { framework.Logf("Failed to delete cloud provider resources for service %q in namespace %q, in cluster %q, err: %v", service.Name, namespace, name, err) } } </del> } }

In federation tests, if we observe the build logs. we see cleaning up LoadBalancer from cloud provider seems not working. It has 2 effects: delays the tests by waiting to delete but always returns error. so we waste like 5 minutes. maybe we are leaking the LoadBalancer in the project.
we can see these errors in every build. but just for reference here are some build logs https://k8s-
cc
All the errors in logs are . It doesn't necessarily indicate we are leaking resource, just now we must have clusterID initialized before calling EnsureLoadBalancerDeleted(). Would be better to not call this function directly in e2e test :)
Federation cleanup should use the same implementation as non-federated e2e test:
function looks like only specific to GCE. in future we would run the e2e tests in AWS too. it would be better to use generic functions if any.
using is working for us. here is the test logs https://k8s-
EnsureLoadBalancerResourcesDeleted() actually cleans up nothing, it only lists forwarding rules and probably logging :)


Question: syncNodeStatus := func() {} var shutdownChan chan bool <ins> var shutdownChanMut sync.Mutex </ins> var connChan = make(chan struct{}, 1) systemDbus = func() (dbusInhibiter, error) { defer func() { connChan <- struct{}{} }() <del> shutdownChan = make(chan bool) dbus := &fakeDbus{currentInhibitDelay: systemInhibitDelay, shutdownChan: shutdownChan, overrideSystemInhibitDelay: overrideSystemInhibitDelay} </del> <ins> ch := make(chan bool) shutdownChanMut.Lock() shutdownChan = ch shutdownChanMut.Unlock() dbus := &fakeDbus{currentInhibitDelay: systemInhibitDelay, shutdownChan: ch, overrideSystemInhibitDelay: overrideSystemInhibitDelay} </ins> return dbus, nil }

/kind failing-test <!-- If this is a flake, please add: /kind flake --<!-- Please assign a SIG using: /sig SIG-NAME --
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
/assign


Question: { local template template="{{ range .spec.containers  }}{{ .name }} {{ end }}" <del> __debug "${FUNCNAME} nouns are ${nouns[*]}" </del> <ins> __kubectl_debug "${FUNCNAME} nouns are ${nouns[*]}" </ins> local len="${#nouns[@]}" if [[ ${len} -ne 1 ]]; then

/kind bug I wasn't able to find issue opened for ZSH case, and aware about similar, fixed, case with bash in 1.10.0-b1 bash completion is working fine
/sig cli
can you verify this and label for v1.10 milestone and note if it will be considered a release blocker
I cannot reproduce this on my local machine (zsh 5.3.1 x86_64-apple-darwin16.3.0). Please produce a docker container (with Dockerfile) that shows this issue.
And another important question for the release: can you try with the 1.9 kubectl? Is that working in your setup?
little bit more info: i'm running this on CentOS 7.4 1708 x64 and zsh from official repo same issue happenes with 1.10.0-b3: 3
From sig-cli Slack: Kazuki Suda [01:22] It seems that have to be renamed to due to changes contained (edited)
You can reproduce this with the below:
was renamed to at , but has been used yet in completions generated by . So we have to replace to .
issue still exists with v10.0.0-b4
10.0-beta.4 does not contain the fix, yet.
Because the fix is not cherry-picked to release-1.10 yet.


Question: } // update new iface records for key, val := range params { <del> _, err = b.exec.Run("iscsiadm", "-m", "iface", "-I", newIface, "-o", "update", "-n", key, "-v", val) </del> <ins> _, err = b.exec.Run("iscsiadm", "-m", "iface", "-I", b.Iface, "-o", "update", "-n", key, "-v", val) </ins> if err != nil { <del> b.exec.Run("iscsiadm", "-m", "iface", "-I", newIface, "-o", "delete") lastErr = fmt.Errorf("iscsi: failed to update iface records: %s (%v). iface(%s) will be used", string(out), err, b.Iface) </del> <ins> b.exec.Run("iscsiadm", "-m", "iface", "-I", b.Iface, "-o", "delete") lastErr = fmt.Errorf("iscsi: failed to update iface records: %s (%v). iface(%s) will be used", string(out), err, b.InitIface) </ins> break } }

What happened: When creating a block volume, the iSCSI storage plugin stores volume-related data in different directories: As a result, some data is not cleaned up once the volume is detached: How to reproduce it (as minimally and precisely as possible): These are the objects I used to reproduce this issue: pod- /sig storage CC
/assign