Question: if err := ClearUDPConntrackForIP(&fexec, tc.ip); err != nil { t.Errorf("%s test case:, Unexpected error: %v", tc.name, err) } <del> expectCommand := fmt.Sprintf("conntrack -D --orig-dst %s -p udp", tc.ip) + familyParamStr(isIPv6(tc.ip)) </del> <ins> expectCommand := fmt.Sprintf("conntrack -D --orig-dst %s -p udp", tc.ip) + familyParamStr(IsIPv6String(tc.ip)) </ins> execCommand := strings.Join(fcmd.CombinedOutputLog[svcCount], " ") if expectCommand != execCommand { t.Errorf("%s test case: Expect command: %s, but executed %s", tc.name, expectCommand, execCommand)

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: There is a bug in ipv6, see: What you expected to happen: The judgement logic should be: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: /sig network /area ipv6
/assign cc


Question: UID:        "bar", APIVersion: "v1", }, <del> Reason:  "Killed", Message: "some other verbose message: 1", Source:  v1.EventSource{Component: "eventTest"}, Count:   1, Type:    v1.EventTypeNormal, </del> <ins> Reason:              "Killed", Message:             "some other verbose message: 1", Source:              v1.EventSource{Component: "eventTest"}, ReportingController: "eventTest", Count:               1, Type:                v1.EventTypeNormal, </ins> }, expectLog:    `Event(v1.ObjectReference{Kind:"Pod", Namespace:"baz", Name:"foo", UID:"bar", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'Killed' some other verbose message: 1`, expectUpdate: false,

From the Deprecated API Migration Guide: , since v1.19, the has been using the new reportingController and reportingInstance to replace the decrecated source.component and But when I checked the Kube events in a cluster with version 1.26.0, those new fields are empty and the supposed to be deprecated fields have not been renamed. Seeing the values in the new fields like reportingController and reportingInstance an Kubernetes cluster with version 1.26.0 some Kube events the Kube events details for the field of reportingController, reportingInstance, source.component, No response <detailsServer Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2022-12-11T00:15:47Z", GoVersion:"go1.19.4", Compiler:"gc", Platform:"linux/amd64"} </details<detailsAzure </details<details</details<details</details<details</details<details</details
/sig instrumentation
I have also encountered this issue in my local 1.26.0 Kubernetes cluster, which does not align with the deprecation guide described in I believe I can investigate this further
/assign
Thank you for making the PR to fix this! Also a comment in the PR for the doc update.
/triage accepted


Question: } func portsFromString(spec string) []api.Port { <ins> if spec == "" { return []api.Port{} } </ins> parts := strings.Split(spec, ",") var result []api.Port for _, part := range parts {

$ run thockin/hostnametcp 1 hostname-tcp E1002 14:48:23. ] Container port is not integer: strconv.ParseInt: parsing "": invalid syntax


Question: RepoDigests: []string{"foo-rd31", "foo-rd32"}, Size:        3, }, <ins> { ID:          "4", RepoTags:    []string{"foo-tag41", "foo-tag42"}, RepoDigests: []string{"foo-rd41", "foo-rd42"}, Size:        3, }, </ins> } }

What happened: When two or more images on a given node have the same size, their order inside the field set by the process is not deterministic. This problem has the side effect of continually triggering controllers which assume this list to be always equal on node updates unless images are or removed. What you expected to happen: The order of images should be stable even when two images have the same size, using the image ID as a second sort field would help this. How to reproduce it (as minimally and precisely as possible): Choose a node and run: Run a few times: It should be possible to see and images alternating on the output. Anything else we need to know?: Our particular use case make it really common for multiple images to have the same size on a given node. I came across this problem while debugging slow updates to DaemonSets and finding out that the kube-controller-manager was leaking memory until it eventually dies due to OOM: <img width="800" alt="Screen Shot 2019-06-30 at 20 27 22" src="https://user-"I was able to track this leak to a large number of watched elements in memory and by taking a look at the logs with increased verbosity I figured it was due to too many Node notifications being processed by the DaemonSet controller: <img width="800" alt="Screen Shot 2019-06-30 at 20 36 23" src="https://user-"It's possible that other controllers in the wild that watch nodes and only ignore the conditions timestamps on updates to face the same problem. I'll send a simple PR to fix this shortly. Environment: Kubernetes version (use ): But by taking a look at the code this problem also happens on master.
/sig node
Nice debugging here :)


Question: Force               bool GracePeriodSeconds  int IgnoreAllDaemonSets bool <ins> IgnoreErrors        bool </ins> Timeout             time.Duration DeleteEmptyDirData  bool Selector            string

<!-- Please only use this template for submitting enhancement requests --What would you like to be : on multiple nodes, if one node failed, the command aborting. Without special purpose, maybe it's better to continue processing other nodes. Why is this needed: Follow the pattern of other subcommand, such as kubectl delete.
/sig cli
/triage accepted /assign
done /close
Closing this issue. <detailsIn response to : done /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I would wait the patch get merged before closing :)


Question: BeforeEach(func() { By("deploying csi hostpath driver") <del> clusterRole = externalAttacherClusterRole(cs, config, false) serviceAccount = externalAttacherServiceAccount(cs, config, false) externalAttacherClusterRoleBinding(cs, config, false, serviceAccount, clusterRole) </del> <ins> clusterRole = csiClusterRole(cs, config, false) serviceAccount = csiServiceAccount(cs, config, false) csiClusterRoleBinding(cs, config, false, serviceAccount, clusterRole) </ins> csiHostPathPod(cs, config, false, f, serviceAccount) }) AfterEach(func() { By("uninstalling csi hostpath driver") csiHostPathPod(cs, config, true, f, serviceAccount) <del> externalAttacherClusterRoleBinding(cs, config, true, serviceAccount, clusterRole) serviceAccount = externalAttacherServiceAccount(cs, config, true) clusterRole = externalAttacherClusterRole(cs, config, true) </del> <ins> csiClusterRoleBinding(cs, config, true, serviceAccount, clusterRole) serviceAccount = csiServiceAccount(cs, config, true) clusterRole = csiClusterRole(cs, config, true) </ins> }) It("should provision storage with a hostPath CSI driver", func() {

Current CSI E2E test has some issues which prevent it to run successfully the a local cluster. /sig storage /kind bug /assign


Question: if attachID != "" { t.Errorf("Expected empty attachID, got %q", attachID) } <del> }(spec, test.expectVolumeAttachment) </del> <ins> }(spec) </ins> if test.expectVolumeAttachment { expectedAttachID := getAttachmentName("test-vol", test.driver, "fakeNode") status := storage.VolumeAttachmentStatus{ Attached: true, } <ins> // We want to ensure the watcher, which is created in csiAttacher, // has been started before updating the status of attachment. <-attachmentWatchCreated </ins> markVolumeAttached(t, csiAttacher.k8s, nil, expectedAttachID, status) } wg.Wait()

Which test(s) are flaking: TestAttacherWithCSIDriver Reason for failure: This is flaking rarely enough that it is not caught by our CI jobs, which currently tolerate up to 2 unit test failures per run (!). With that toleration removed in , this flake has been seen (). Reproducible with the following steps: a test binary for the affected package: the affected test using the tool (setting the timeout to a generous timeout relative to the normal test runtime of .02 seconds): failures are seen, look at the logs to see the details about the error: /sig storage cc
/assign can you help investigate this flake?
I have found the root cause. /assign
filed to address this issue.


Question: if e, a := createdNoxuInstance, gottenNoxuInstance2; !reflect.DeepEqual(e, a) { t.Errorf("expected %v, got %v", e, a) } <ins> } func TestDeRegistrationAndReRegistration(t *testing.T) { stopCh, apiExtensionClient, clientPool, err := testserver.StartDefaultServer() if err != nil { t.Fatal(err) } defer close(stopCh) noxuDefinition := testserver.NewNoxuCustomResourceDefinition() ns := "not-the-default" sameInstanceName := "foo" func() { noxuVersionClient, err := testserver.CreateNewCustomResourceDefinition(noxuDefinition, apiExtensionClient, clientPool) if err != nil { t.Fatal(err) } noxuNamespacedResourceClient := NewNamespacedCustomResourceClient(ns, noxuVersionClient, noxuDefinition) if _, err := instantiateCustomResource(t, testserver.NewNoxuInstance(ns, sameInstanceName), noxuNamespacedResourceClient, noxuDefinition); err != nil { t.Fatal(err) } // Remove sameInstanceName since at the moment there's no finalizers. // TODO: as soon finalizers will be implemented Delete can be removed. if err := noxuNamespacedResourceClient.Delete(sameInstanceName, nil); err != nil { t.Fatal(err) } if err := testserver.DeleteCustomResourceDefinition(noxuDefinition, apiExtensionClient); err != nil { t.Fatal(err) } if _, err := testserver.GetCustomResourceDefinition(noxuDefinition, apiExtensionClient); err == nil || !errors.IsNotFound(err) { t.Fatalf("expected a NotFound error, got:%v", err) } if _, err = noxuNamespacedResourceClient.List(metav1.ListOptions{}); err == nil || !errors.IsNotFound(err) { t.Fatalf("expected a NotFound error, got:%v", err) } if _, err = noxuNamespacedResourceClient.Get("foo"); err == nil || !errors.IsNotFound(err) { t.Fatalf("expected a NotFound error, got:%v", err) } }() func() { if _, err := testserver.GetCustomResourceDefinition(noxuDefinition, apiExtensionClient); err == nil || !errors.IsNotFound(err) { t.Fatalf("expected a NotFound error, got:%v", err) } noxuVersionClient, err := testserver.CreateNewCustomResourceDefinition(noxuDefinition, apiExtensionClient, clientPool) if err != nil { t.Fatal(err) } noxuNamespacedResourceClient := NewNamespacedCustomResourceClient(ns, noxuVersionClient, noxuDefinition) initialList, err := noxuNamespacedResourceClient.List(metav1.ListOptions{}) if err != nil { t.Fatal(err) } if _, err = noxuNamespacedResourceClient.Get(sameInstanceName); err == nil || !errors.IsNotFound(err) { t.Fatalf("expected a NotFound error, got:%v", err) } if e, a := 0, len(initialList.(*unstructured.UnstructuredList).Items); e != a { t.Fatalf("expected %v, got %v", e, a) } createdNoxuInstance, err := instantiateCustomResource(t, testserver.NewNoxuInstance(ns, sameInstanceName), noxuNamespacedResourceClient, noxuDefinition) if err != nil { t.Fatal(err) } gottenNoxuInstance, err := noxuNamespacedResourceClient.Get(sameInstanceName) if err != nil { t.Fatal(err) } if e, a := createdNoxuInstance, gottenNoxuInstance; !reflect.DeepEqual(e, a) { t.Fatalf("expected %v, got %v", e, a) } listWithItem, err := noxuNamespacedResourceClient.List(metav1.ListOptions{}) if err != nil { t.Fatal(err) } if e, a := 1, len(listWithItem.(*unstructured.UnstructuredList).Items); e != a { t.Fatalf("expected %v, got %v", e, a) } if e, a := *createdNoxuInstance, listWithItem.(*unstructured.UnstructuredList).Items[0]; !reflect.DeepEqual(e, a) { t.Fatalf("expected %v, got %v", e, a) } if err := noxuNamespacedResourceClient.Delete(sameInstanceName, nil); err != nil { t.Fatal(err) } if _, err = noxuNamespacedResourceClient.Get(sameInstanceName); err == nil || !errors.IsNotFound(err) { t.Fatalf("expected a NotFound error, got:%v", err) } listWithoutItem, err := noxuNamespacedResourceClient.List(metav1.ListOptions{}) if err != nil { t.Fatal(err) } if e, a := 0, len(listWithoutItem.(*unstructured.UnstructuredList).Items); e != a { t.Fatalf("expected %v, got %v", e, a) } }() </ins> }

Kubernetes version (use ): Kubernetes HEAD Environment: Cloud provider or hardware configuration: linux OS (e.g. from /etc/os-release): Kernel (e.g. ):NAME=Fedora25 What happened: apiserver panic What you expected to happen: No panic How to reproduce it (as minimally and precisely as possible): Testcase is going to be sent as a PR
Fixed by


Question: totalPodCapacity += podCapacity.Value() } <ins> WaitForPodsToBeDeleted(cs) </ins> currentlyScheduledPods := WaitForStableCluster(cs, masterNodes) podsNeededForSaturation := int(totalPodCapacity) - currentlyScheduledPods

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: soak-gce-gci-1.13, others Which test(s) are failing: [sig-scheduling] SchedulerPredicates [Serial] validates MaxPods limit number of pods that are allowed to run [Slow] Since when has it been failing: Rougly forever. shows how this test was flaking back during the 1.11 cycle. There's been no sudden increase in frequency, but a test that flakes roughly 1.5 times per hour needs looking into, and probably tweaking. Testgrid link: https://k8s- Reason for failure: Most of the time, these flakes are timeouts. However, sometimes they have other causes; see Triage link: Anything else we need to know: This is part of a general effort to clean up our CI signal by eliminating longstanding, frequent flakes. <!-- DO NOT EDIT BELOW THIS LINE --/kind flake /priority important-longterm /sig scheduling attn:
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale /help-wanted
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
There are several different failures in this test, I was in particular looking at which usually boils down to something like this My hunch is that we are trying to schedule more than we should. This is how we compute how many pods to schedule which just takes max capacity and substracts whatever this returns My guess would be that the computation is not completely correct. My idea would be to add more logging into to be able to figure out more. Does that sound reasonable to you or would you take another approach? cc.

I went over all the pods that were on the node before the test started and then after and diff-ed all the pods that are not started by the test "grep -v maxp-" and did not found any pod that would start in the middle of test and was not present in the beginning :/ but there were 18 pods in the beginning, 330 is the full capacity so we should have started 312 to fully saturate, while we were trying to start 313. (maybe some pod restarted right when we were starting the test?) I am looking at this latest case but there's many like this one
Can we possibly be missing some pods here? pods with no node assigned that are not unschedulable?
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
This just got much worse in the last weeks, I tried looking into PRs along the time and hit into this could someone please verify it that could have made things worse? I'll also try to look tomorrow since I investigated this while ago :)
Anyone can help here? It's constantly red for several days, it's not even flaking anymore in our release dashboards /priority important-soon


Question: p.t.Errorf("invariant violated, expected limit %d and continue %s, got %#v", p.expectPage, expectedContinue, options) return nil, fmt.Errorf("invariant violated") } <ins> if options.Continue != "" && options.ResourceVersion != "" { p.t.Errorf("invariant violated, specifying resource version (%s) is not allowed when using continue (%s).", options.ResourceVersion, options.Continue) return nil, fmt.Errorf("invariant violated") } </ins> var list metainternalversion.List total := options.Limit if total == 0 {

/sig scalability Debugging below done by Failed test attempt: It failed with a number of "WaitingForPods: objects timed out" errors (46 failed objects in total). Sample failing pods: 580 nodes were not ready at 18:37: nodelifecyclecontroller said that gce-scale-cluster-minion-group-2-66d8's status hasn't been updated for 41 seconds. apiserver's logs show that kubelet was doing 'PUT ' every 10 seconds. Something weird happened to watch by 'shared-informers': One watch started at 18:36:52 and finished 9ms later. The next one started at 18:40:34. So it seems that there was a gap in lease watche between 18:36:52 and 18:40:34. Between 18:36:52 and 18:40:34 there is a number of leases LIST requests in the following pattern: a single /leases?limit=500 LIST request that finishes with 200 a single /leases/continue=XXX request that finishes with 400 The kube-controller-manager is retrying every 1 second up to 18:40:34.
/assign
Looking further at this it seems that number of LIST leases by kube-controller-manager increased significantly ~week ago in large scale tests (both gce and kubemark). GCE: ! Kubemark: ! This was caused by which as a side effect enabled pagination of LIST calls when starting/restarting watchers. In some cases we're getting 400 errors from api-server for this calls which results in the symptoms described above, i.e nodes becoming not-ready and pods stuck in running but not ready state for some time. GCE: ! Kubemark: !
There are actually two issues here: that we switched from non-paginated LIST calls to paginated the 10x increase in # calls is expected, but what is worrying is that we already had ~50 LISTs in every test. Such LIST happens only when watcher restarts which shouldn't happen in a healthy cluster This will be address in need to figure out why we're sometimes getting 400 response (BAD REQUEST).

/milestone v1.17
Logs when then 400 errors occurred api-server kube-controller-manager
So what happens here is: in reflector ends with an error different than so the doesn't get reset to next list is issued with set to page is fetched successfully, the next page is requested with the got from the first page and second page fails with due to error the doesn't get cleared anywhere so we go back to 2 and the 400 error keeps repeating until we get 410 (resourceVersionTooOld, on the next etcd compaction) and the is set to takes long enough for the NodeController to mark many nodes as not ready as it doesn't get any updates from watch about node leases This was indeed introduced by but the actual bug is in the which sets both and on the subsequent pages while it should set only . I will send a PR to fix it today.
FYI
Great finding.
Does this mean we have no tests that actually check if a reflector works with paging?
I see, it's the rare situation where "Watch in reflector ends with an error different than 410" that is untested. Can we set up a unit test that exercises this scenario by wrapping etcd3 storage to allow injecting an error in a watch attempt (similar to how we wrapped storage to test stale behavior in )?
I don't think it's the case - see . It clearly seems to be that pager wasn't properly tested. The problem seems to be that we're testing it with artifical stuff, whereas in real life etcd would return "bad request" errors.
Right, I'd like to see if we can trigger the error condition in a way that the restart/paging requests go to a real storage backend.
/milestone v1.18 (v1.17 code freeze)
- this is a regression introduced in 1.17, I actually believe we should fix that in 1.17.
/milestone v1.17 as noted above, this is a release blocking bug


Question: // [Serial]: the tests read global kube-controller-manager metrics, so no other test changes them in parallel. f.Context("SELinuxMount metrics [LinuxOnly]", feature.SELinux, feature.SELinuxMountReadWriteOncePod, f.WithSerial(), func() { <del> // All SELinux metrics. Unless explicitly mentioned in test.expectIncreases, these metrics must not grow during // a test. allMetrics := sets.NewString( "volume_manager_selinux_container_errors_total", "volume_manager_selinux_container_warnings_total", "volume_manager_selinux_pod_context_mismatch_errors_total", "volume_manager_selinux_pod_context_mismatch_warnings_total", "volume_manager_selinux_volume_context_mismatch_errors_total", "volume_manager_selinux_volume_context_mismatch_warnings_total", "volume_manager_selinux_volumes_admitted_total", ) </del> // Make sure all options are set so system specific defaults are not used. seLinuxOpts1 := v1.SELinuxOptions{ User:  "system_u",

[It] [sig-storage] CSI Mock selinux on mount metrics SELinuxMount metrics [LinuxOnly][Feature:SELinux][Feature:SELinuxMountReadWriteOncePod][Serial] warning is not bumped on two Pods with the same context on RWO volume [sig-storage] [It] [sig-storage] CSI Mock selinux on mount metrics SELinuxMount metrics [LinuxOnly][Feature:SELinux][Feature:SELinuxMountReadWriteOncePod][Serial] warning is bumped on two Pods with a different context on RWO volume [sig-storage] Sample failure: October 25, 2023 It fails since The PR a label to some SELinux metric and the tests do not count with that. No response /sig storage
/assign /triage accepted


Question: // BindClusterFlags is a convenience method to bind the specified flags to their associated variables func BindClusterFlags(clusterInfo *clientcmdapi.Cluster, flags *pflag.FlagSet, flagNames ClusterOverrideFlags) { <del> flags.StringVarP(&clusterInfo.Server, flagNames.APIServer, flagNames.APIServerShort, "", "The address of the Kubernetes API server") </del> <ins> flags.StringVarP(&clusterInfo.Server, flagNames.APIServer, flagNames.APIServerShort, "", "The address and port of the Kubernetes API server") </ins> flags.StringVar(&clusterInfo.APIVersion, flagNames.APIVersion, "", "The API version to use when talking to the server") flags.StringVar(&clusterInfo.CertificateAuthority, flagNames.CertificateAuthority, "", "Path to a cert. file for the certificate authority.") flags.BoolVar(&clusterInfo.InsecureSkipTLSVerify, flagNames.InsecureSkipTLSVerify, false, "If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure.")

If the apiserver only listens on one physical IP, then you'd have to use kubectl --server to specify the host. The help message says specify the IP, connection refused: both IP and Port, then ok: So, is it possbile to default the port to 8080 if no port specified at command line or read it from the config file /etc/kubernetes/apiserver, which is 8080 too. If it's impossible to default the port to 8080, it needs an update of the help message at least, like: or add --port flag.
The options are: the port from the port by trying if none is specified a flag the documentation I think (4) is the way to go (I'm not lazy I swear). (1) requires access to the API server's filesystem which is off-limits. (2) seems like it'll work for some configs and is a default that may change. (3) is more verbose and adds yet another flag. Will send a PR.


Question: // Strip value validation. StripValueValidation bool <ins> // Strip nullable. StripNullable bool </ins> // AllowNonStructural indicates swagger should be built for a schema that fits into the structural type but does not meet all structural invariants AllowNonStructural bool }

With in a CRD we get: This is because gnostic's OpenAPI proto types have no nullable support. Luckily CRDs keep working because in the upper error case we don't stop serving, just set the SSA schema to nil.
Thanks for reporting!
I think it should be "This is because gnostic's OpenAPI v2 proto types have no nullable support.". The problem is that we're not supposed to pass-in v3.
Source:
OK, We'll write a test and fix this, thanks Stefan!
/reopen This still happens in slightly different shape: This comes from :
That's a fairly different bug but yes, thanks!
Is this related to the stating that use of "anyOf" wouldn't be allowed after 1.16? We're seeing issue on OpenFaaS after going to 1.17.5 and 1.18. Usually upgrading K8s versions hasn't caused breaking changes and we're trying to figure out where to start with this one.
No. That post was about structural schemas. v1 CRDs are limited to structural schemas. v1beta1 CRDs can continue to create non-structural schemas until v1beta1 is no longer served (in 1.22) Also, is not completely disallowed, specific restrictions inside an clause (like ) are what are not allowed ("logical constraints using oneOf, allOf, anyOf, not are allowed").
extracted the newly found issue to to track separately with isolated reproducer /close
Closing this issue. <detailsIn response to : extracted the newly found issue to to track separately Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: } func (o *PluginListOptions) Run() error { <del> pluginsFound := false isFirstFile := true pluginErrors := []error{} </del> <ins> plugins, pluginErrors := o.ListPlugins() if len(plugins) > 0 { fmt.Fprintf(o.Out, "The following compatible plugins are available:nn") } else { pluginErrors = append(pluginErrors, fmt.Errorf("error: unable to find any kubectl plugins in your PATH")) } </ins> pluginWarnings := 0 <ins> for _, pluginPath := range plugins { if o.NameOnly { fmt.Fprintf(o.Out, "%sn", filepath.Base(pluginPath)) } else { fmt.Fprintf(o.Out, "%sn", pluginPath) } if errs := o.Verifier.Verify(pluginPath); len(errs) != 0 { for _, err := range errs { fmt.Fprintf(o.ErrOut, "  - %sn", err) pluginWarnings++ } } } if pluginWarnings > 0 { if pluginWarnings == 1 { pluginErrors = append(pluginErrors, fmt.Errorf("error: one plugin warning was found")) } else { pluginErrors = append(pluginErrors, fmt.Errorf("error: %v plugin warnings were found", pluginWarnings)) } } if len(pluginErrors) > 0 { errs := bytes.NewBuffer(nil) for _, e := range pluginErrors { fmt.Fprintln(errs, e) } return fmt.Errorf("%s", errs.String()) } return nil } // ListPlugins returns list of plugin paths. func (o *PluginListOptions) ListPlugins() ([]string, []error) { plugins := []string{} errors := []error{} </ins> for _, dir := range uniquePathsList(o.PluginPaths) { if len(strings.TrimSpace(dir)) == 0 {

<!-- Please only use this template for submitting enhancement requests --What would you like to be : I type <kbdTab</kbd<kbdTab</kbdI would like to see kubectl plugins in the list of suggested subcommands. For example, if I have installed , subcommand completions should suggest . the plugin itself also supports command-line completion, <kbdTab</kbd<kbdTab</kbdshould show suggestions from the plugin command completion. Why is this needed: Plugins fill in niches in the kubectl workflow, so that they are used rather seldomly. This means that plugin users need to memorize plugin names and their options over long times without using them. Therefore, plugin usefulness would be greatly enhanced by suggesting plugin sub-commands and their options.
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
In fact, already supports command completion. You can run for more information about it. If you are using bash shelljust run , and then it will work!
Hi , but I'm not talking about command completion in general. I'm talking about command completion for plugins. For example, the new plugin system allows to call a plugin at location as . It would be nice to get suggested/completed, when typing <kbdTab</kbd. Further, if plugin names are completed, so should be options for those plugins -- provided the plugin offers some form of command completion.
I try to implement completion for kubectl plugins!
Were you able to make progress on this? I have a working patch for the completion of the plugin name (part 1 of this issue). I can make a PR for it if it can help.
This is a great first step! I do think however, that the real value is auto-completion for the plugin options. I can roughly remember what plugins I have, but not what options they offer. That's the real value here, I think. This will probably need a contract for how plugins make completion available (e.g. by following the kubectl standard of ). WDYT?
I've been thinking about how to do the plugin options completion.

Based on the patch I have now, I believe what will be needed is to be able to ask the plugin what the possible completions can be (based on the user-specified parameters). So the plugin won't need to generate a bash or zsh completion script (which is difficult for the author of the plugin) but just output possible completions. This also has the benefit of being independent of which shell is used. One way is to do as you mention and have the plugin support as a way to ask the plugin what the completions would be. Although it won't behave in the same fashion as (as it won't generate a completion script, and is shell-independent), I like that option because it simply requires a plugin author to provide a different script called which the bash completion logic can call. Another way would be to ask the plugin itself to support a flag such as which would return the possible completions. I don't like this because if the plugin doesn't support this flag, we'd be calling it and who knows what it would do in that unexpected case. The use of makes it easy to check of presence or absence of completion support. So I suggest that the contract with plugins be that a plugin that supports completion would provide two scripts and that the bash completion script call to get the list of completions. Any thoughts?
Hi guys, I've already implemented completion for plugins (subcommands only, doesn't include options) as However it needs to be refactored more. You can try completion for plugins with the following steps: First, prepare the repository, and build a kubectl. Install kubectl-hello plugin for completion testing. Load the kubectl completion code. You can see completion for some plugins.
Cool! You've used the same technique I did. However, may I suggest re-using the code from the plugin package instead of copying it?
Well, you are absolutely right in saying that we can't just call a plugin with some flag to produce the completion script, because who knows what will happen. Therefore, option completion for plugins has to go the route via an external completion script, like However, I'm pretty sure that those completion files need to be bash/zsh scripts, because they need to process contextual information.

For example, say I have a plugin with sub-commands Then I want completion to work as follows: As you say, this will have to be maintained by the plugin authors. Can you take a typical cobra command completion script as basis and work out how kubectl-plugin-completion and this external script need to work together? You could even try to have kubectl as its own plugin, like and or so :grinning:
kubectl already has some dynamic completion code to allow to complete pod names for example, which actually calls kubectl itself. I think I can use the same technique for plugin parameter completion. So here's an example of what I'm thinking: Then when the user types the kubectl completion script will call . This call will return which will be given to the shell by the kubectl completion script. I haven't actually tested this example, so I could be wrong. I'll try implementing it to see if I'm missing something.
Thank you for supporting me ;) I add you as co-author to commit when I send a PR.
Thanks
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue.

<detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-lifecycle rotten
I've been working on auto-completion for the Helm project and I have a proposal with a PR that fully supports auto-completion for Helm plugins. I believe the fundamental aspects of the solution would apply just as well to kubectl. Some adaptation would be needed to adjust to how kubectl supports plugins, but I think it wouldn't be very hard.
Is there an ETA on this? If I see it correctly, this is currently preventing plugin authors to implement completions as the generated completion suggestions are based off the root command and not the subcommand aka plugin. I've been trying all day to make this work but failed and that's when I stumbled over this issue..
Several major issues I see here are: We don't want files like as proposed in previous comment. We had a requirement prior to kubectl 1.12 and it didn't look great at all. The simplicity of plugins is valuable to us. (Also, unlike Helm plugins, kubectl plugins don't live at a particular directory, or follow a directory structure.)Plugin completions most likely would be integrated into commands, which a lot of people put in their or file. We need to keep this command fast. So what if the command called a special command on each plugin, in parallel, to inquire if it has completion available (e.g.), with a short deadline, and cached the successful results (maybe keyed with on plugin binary’s stat data, or checksum). That way, cmd could also provide data about plugins, while periodically refreshing the cached completion scripts.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with .

Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
any updates on this one ? is it complete/done?
No update, still open..
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale
/lifecycle frozen
Sorry to revive this thread a bit but while working on another cli using the same cobra completion system by (implemented by i believe) I found out that it would not be too hard or too disruptive to have a small patch on the shell completion directly. For example with kubectl if we have a patch like this (using zsh but I am guessing the same technique will apply for bash or others) : the patch will check the first command () and the second command for example "krew" exist as and then use it directly for completion otherwise do the normal completion 'thingy'. the only downside is that the initial plugin would not show i.e: if you do you won't get krew as the plugin name, except if we add all plugin to the Root Command like the pull request from does. I don't think it would be too intensive since it is a zsh/bash builtin and in memory. I have made a small asciinema to demonstrate how it works I guess if we want this we may need to do it directly in cobra, do you think a patch like this could get accepted ?

(I had somehow difficulties to get PR reviewer and merged in cobra last time I tried)
Thanks for the suggestion. The idea is good but doesn't generalize perfectly. The command is available for programs that use Cobra but kubectl plugins can be something else (a script, a Python program, a Go program that doesn't use Cobra, etc). So we couldn't do something like this, unless we required every plugin to provide a command. I'm working on porting Helm's plugin completion solution to kubectl and should have something for the community to try soon. Plugins using Cobra will then be able to use your suggestion and get completion super easily.
That make sense I forgot about non go (and non cobra) plugins, thanks I look forward to see how the helm completion extracted will look like !
I've posted something for people to try out in
It would be nice if some plugin developers try out the solution proposed in and provide feedback. since uses Cobra, the solution is very easy. Here is how to try it out: First build kubectl with the PR. Then create your completion script like this: put this new on $PATH. And now test using the newly built (I build mine at ). For completion of flags (which I realize is the real value for the plugin), kubectl needs to first move to Cobra 1.3.
I would be interested in your opinion on the proposed solution. It will not work out-of-the-box with Krew, but I haven't found a way to do better with respect to Krew. You had mentioned this previously, and although it would be a nice solution, I am concerned it would cause problems to any plugin that ignores all arguments and runs no matter what. With such a plugin, calling a command would trigger the plugin to run and possibly perform undesired actions when completions are requested.
I'm sorry about slow responses. Finally had time to read it and parse these. You're right we shouldn't randomly call a plugin binary without knowing it supports completion. From : kubectl already has a mechanism to find all plugins on $PATH, so I agree that at minimum, kubectl<kbdTab</kbdcan and should show plugins installed on the system (even if the plugins themselves don't support shell completion) I think this is a good model.

Just like how kubectl scans $PATH for , it can scan to see if a plugin supports completion. (It's important to define if that binary takes additional positional args or --options) To make the (2) a reality, I support adding an additional field to the Krew plugin manifest. The shape of that would be up for a discussion (I imagine it might be (a) a separate binary that comes in the plugin archive, or (b) allow linking to the plugin binary itself and the plugin reads argv[0] and prints completion, which can be done with (a) anyway). I think it's doable. I'm not very familiar with how shell completion works in practice, when it's called, how lazy can it be loaded, what are performance implications etc, but if you'd like to drive this effort, I think we can discuss a design on Krew side as well. Please bring this up in a SIG CLI meeting to see what the folks think about your proposal.
brilliant! Thanks for taking the time to think about this. Your suggestion to adapt krew sounds great to me. I will continue with the PR on kubectl's side and if it gets accepted we can look at krew.
The as proposed in accepts the exact same arguments as the standard executable. The new completion executable will examine those arguments and will determine what the shell completion choices following them should be.
do you have an insight into whether a plugin developer would develop and ship a separate executable for the completion, or would they simply link the main plugin executable and see if the argv[0] is the kubectl_complete-*? I assume the answer largely also depends on whether the plugin was authored in Go or bash (basically the only two languages I've ever seen a plugin submission in out of 150+).
I think plugins will choose to ship a separate executable. I feel it keeps the two aspects nicely separate (the plugin functionality and the shell completion functionality). Also, it is very easy for a separate executable to call back the main plugin if the developer wishes to do so, so I find that solution the most flexible. For example, for plugins using the Cobra library, the script would call back the executable like this: BTW, I have looked into the 179 plugins in the krew index and I found that 50% (yes, half!)

of them used Cobra (some of them indirectly by calling back ). This means that shell completion should be very easy to implement for all these plugins.
I'm interested in this support and thanks for adding this feature. One question: in which version this is planned to be released? Looking forward to this.


Question: MASTER_OS_DISTRIBUTION=${KUBE_MASTER_OS_DISTRIBUTION:-gci} NODE_OS_DISTRIBUTION=${KUBE_NODE_OS_DISTRIBUTION:-gci} <del> MASTER_IMAGE=${KUBE_GCE_MASTER_IMAGE:-cos-stable-60-9592-84-0} </del> <ins> MASTER_IMAGE=${KUBE_GCE_MASTER_IMAGE:-cos-stable-60-9592-90-0} </ins> MASTER_IMAGE_PROJECT=${KUBE_GCE_MASTER_PROJECT:-cos-cloud} NETWORK=${KUBE_GCE_NETWORK:-e2e}

The debian-based CVM image is going to be deprecated in September. We should: Change the default OS image for GCE cluster setup script. This means that all the GCE CI builds will use the new default OS image. Add deprecation warning to the cluster setup script if CVM is specified. Maintain the node e2e test coverage for CVM for 1.8. /cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
The CVM tests were turned off a while ago.


Question: plug, _, tmpDir, _ := newTestWatchPlugin(t, fakeClient, true) defer os.RemoveAll(tmpDir) <ins> attachmentWatchCreated := make(chan core.Action) // Make sure this is the first reactor fakeClient.Fake.PrependWatchReactor("volumeattachments", func(action core.Action) (bool, watch.Interface, error) { select { case <-attachmentWatchCreated: // already closed default: // The attacher is already watching the attachment, notify the test goroutine to // update the status of attachment. // TODO: In theory this still has a race condition, because the actual watch is created by // the next reactor in the chain and we unblock the test goroutine before returning here. close(attachmentWatchCreated) } return false, nil, nil }) </ins> attacher, err := plug.NewAttacher() if err != nil { t.Fatalf("failed to create new attacher: %v", err)

Which test(s) are flaking: TestAttacherWithCSIDriver Reason for failure: This is flaking rarely enough that it is not caught by our CI jobs, which currently tolerate up to 2 unit test failures per run (!). With that toleration removed in , this flake has been seen (). Reproducible with the following steps: a test binary for the affected package: the affected test using the tool (setting the timeout to a generous timeout relative to the normal test runtime of .02 seconds): failures are seen, look at the logs to see the details about the error: /sig storage cc
/assign can you help investigate this flake?
I have found the root cause. /assign
filed to address this issue.


Question: // How many seconds to wait for a multipath device if at least two paths are available. multipathDeviceTimeout = 10 <ins> // 'iscsiadm' error code stating that a session is logged in // See https://github.com/open-iscsi/open-iscsi/blob/7d121d12ad6ba7783308c25ffd338a9fa0cc402b/include/iscsi_err.h#L37-L38 iscsiadmErrorSessExists = 15 </ins> ) var (

What happened: In order to communicate with the target, the iscsi plugin creates a new iface. If for some reason an iface with the same name already exists in the node, this command will re-create it and no error will be reported. However, if the iface already exists and is being used by a session, this command will fail and report the error code 15 (). That's because it can't re-create the iface while it's in use by a session. What you expected to happen: The plugin could go ahead and use the existing iface instead of failing. How to reproduce it (as minimally and precisely as possible): pod1 pod2 Here, will re-use the session/iface from the step above. pod1 At this point, k8s will try to delete the session/iface used by pv1. However, it'll fail because the session is currently in use by pv2. pod1 Here the pod's status will be stuck at : Templates used: Anything else we need to know?: problem happens when a custom initiator is used, and it was reported to me that it's necessary in some set-ups in order to "restrict access to LUNs of the storage". to reproduce manually (assuming there's a session using the iface): /sig storage /CC
/assign
is this related to cc
If iface is reused, wouldn't it try to delete it once last volume (from kubelet perspective) is detached?
No


Question: return isErrCode(err, ErrCodeResourceVersionConflicts) } <del> // IsRequestTimeout returns true if and only if err indicates that the request has timed out. func IsRequestTimeout(err error) bool { return isErrCode(err, ErrCodeTimeout) } </del> // IsInvalidObj returns true if and only if err is invalid error func IsInvalidObj(err error) bool { return isErrCode(err, ErrCodeInvalidObj)

various Oct 30 2023 6pm pst No response Looks like different test are failing with connection timeout errors: One of the tests triggered a core dump: No response /sig
are both suffers from pkg/registry and is from compoent-base, which likely same reason from `
/retitle failing-test: pkg/kubelet/eviction TestManagerWithLocalStorageCapacityIsolationOpen
/sig node
TestManagerWithLocalStorageCapacityIsolationOpen only failed in one of the PRs, so I don't think that specific test is the problem.
ok, rename as see fit. just put owning sigs please for triage.
TestStoreDeleteCollectionWithContextCancellation is the one that seems to be failing across all the failing PRs, so let's start there first. /retitle failing-test: TestStoreDeleteCollectionWithContextCancellation /sig api-machinery
triage link at shows this just starting (happening in CI jobs as well)
image used in the unit test job didn't change around the timeframe this was seen ()
looking for jumps in package unit test runtimes when this failure started happening to see if something is taking longer or stressing the system: ! - cc ! - cc ! - cc Could any of those have introduced changes that could be stressing or exhausting listener connections in unit tests?
the PR in question introduced a poll function around an API call that is unit tested with a fake client i.e. the poll is unit tested, but the fake client has reactors that return error or success right away, so in theory the poll should not be stretched to 1 retries with 1 connection attempts per fake client. while testing locally i did not notice any significant time difference after or the prior i can do some more checks to see what's going on.
merged right at the time the flakes were seen DeleteCollection invokes List, which was modified in that PR to cancel more aggressively cc
I can reproduce about a 10-20% flake rate at HEAD with: failures look like: or while testing that package with , I see ~1300-1500 open connections with this:
Let's revert it then. I looked into it and I don't know from a quick glance why it's failing and I don't have time to debug it now.

I opened
Okay, it looks like we need more debugging to open this PR
one optimization to strip some ~40 fake client connections and 2 seconds from a test:
sounds good, pointed at that commit as well
/triage accepted


Question: est.lock.Lock() defer est.lock.Unlock() <del> rrv := est.relatedResourceVersions(endpointSlice) </del> <ins> rrv, ok := est.relatedResourceVersions(endpointSlice) if !ok { rrv = endpointSliceResourceVersions{} est.resourceVersionsByService[getServiceNN(endpointSlice)] = rrv } </ins> rrv[endpointSlice.Name] = endpointSlice.ResourceVersion } <ins> // DeleteService removes the set of resource versions tracked for the Service. func (est *endpointSliceTracker) DeleteService(namespace, name string) { est.lock.Lock() defer est.lock.Unlock() serviceNN := types.NamespacedName{Name: name, Namespace: namespace} delete(est.resourceVersionsByService, serviceNN) } </ins> // Delete removes the resource version in this endpointSliceTracker for the // provided EndpointSlice. func (est *endpointSliceTracker) Delete(endpointSlice *discovery.EndpointSlice) { est.lock.Lock() defer est.lock.Unlock() <del> rrv := est.relatedResourceVersions(endpointSlice) delete(rrv, endpointSlice.Name) </del> <ins> rrv, ok := est.relatedResourceVersions(endpointSlice) if ok { delete(rrv, endpointSlice.Name) } </ins> } // relatedResourceVersions returns the set of resource versions tracked for the <del> // Service corresponding to the provided EndpointSlice. If no resource versions // are currently tracked for this service, an empty set is initialized. func (est *endpointSliceTracker) relatedResourceVersions(endpointSlice *discovery.EndpointSlice) endpointSliceResourceVersions { </del> <ins> // Service corresponding to the provided EndpointSlice, and a bool to indicate // if it exists. func (est *endpointSliceTracker) relatedResourceVersions(endpointSlice *discovery.EndpointSlice) (endpointSliceResourceVersions, bool) { </ins> serviceNN := getServiceNN(endpointSlice) vers, ok := est.resourceVersionsByService[serviceNN] <del> if !ok { vers = endpointSliceResourceVersions{} est.resourceVersionsByService[serviceNN] = vers } return vers </del> <ins> return vers, ok </ins> } // getServiceNN returns a namespaced name for the Service corresponding to the

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: The endpointSliceTracker creates a set of resource versions for each service, the resource versions in the set could be deleted when endpointslices are deleted, but the set and its key in the map is never deleted, leading to memory leak. What you expected to happen: The set of resource versions should be delete from endpointSliceTracker when the service is deleted. Environment: Kubernetes version (use ): 1.18.5
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc


Question: } func RunApply(f cmdutil.Factory, cmd *cobra.Command, out, errOut io.Writer, options *ApplyOptions) error { <del> shortOutput := cmdutil.GetFlagString(cmd, "output") == "name" </del> schema, err := f.Validator(cmdutil.GetFlagBool(cmd, "validate"), cmdutil.GetFlagString(cmd, "schema-cache-dir")) if err != nil { return err

What happened: What you expected to happen: yaml output How to reproduce it (as minimally and precisely as possible): Anything else do we need to know: Related PR


Question: func New(address string, port uint, imageFsInfoProvider ImageFsInfoProvider, rootPath string, usingLegacyStats bool) (Interface, error) { sysFs := sysfs.NewRealSysFs() <del> ignoreMetrics := cadvisormetrics.MetricSet{cadvisormetrics.NetworkTcpUsageMetrics: struct{}{}, cadvisormetrics.NetworkUdpUsageMetrics: struct{}{}} </del> <ins> ignoreMetrics := cadvisormetrics.MetricSet{ cadvisormetrics.NetworkTcpUsageMetrics: struct{}{}, cadvisormetrics.NetworkUdpUsageMetrics: struct{}{}, cadvisormetrics.PerCpuUsageMetrics:     struct{}{}, } </ins> if !usingLegacyStats { ignoreMetrics[cadvisormetrics.DiskUsageMetrics] = struct{}{} }

/kind feature This is the once-per-release cut of a new cAdvisor release, which corresponds to the next kubernetes release. cAdvisor v0.29 corresponds to kubernetes v1.10. /sig node /priority important-soon
self-reminder to add per-cpu stats to ignore metrics when vendoring. reminder to add a release notice that one of the container labels fields is being removed.
/kind bug turns out most of the changes are bug-fixes
please mark this issue as approved-for-milestone, since it certainly seems to be, and it's now code slush. Thanks!


Question: } containerPort, found := params["container-port"] if found && len(containerPort) > 0 { <del> cPort, err := strconv.Atoi(containerPort) if err != nil { service.Spec.ContainerPort = util.NewIntOrStringFromInt(cPort) } else { </del> <ins> if cPort, err := strconv.Atoi(containerPort); err != nil { </ins> service.Spec.ContainerPort = util.NewIntOrStringFromString(containerPort) <ins> } else { service.Spec.ContainerPort = util.NewIntOrStringFromInt(cPort) </ins> } } else { service.Spec.ContainerPort = util.NewIntOrStringFromInt(port)

expose hostnames --port=80 --container-port=9376 --selector="app=hostnames" get se hostnames -o yaml apiVersion: v1beta1 containerPort: "9376" ... obviously, endpoints controller is not happy.


Question: return ds.Spec.Template.Spec.Containers[0].Image, nil } <del> func getContainerImageFromE2ETestDaemonset(dsYamlPath string) (string, error) { data, err := e2etestfiles.Read(dsYamlPath) </del> <ins> func getSampleDevicePluginImage() (string, error) { data, err := e2etestfiles.Read(SampleDevicePluginDSYAML) </ins> if err != nil { <del> return "", fmt.Errorf("failed to read the daemonset yaml: %w", err) </del> <ins> return "", fmt.Errorf("failed to read the sample plugin yaml: %w", err) </ins> } ds, err := e2emanifest.DaemonSetFromData(data) if err != nil { <del> return "", fmt.Errorf("failed to parse daemonset yaml: %w", err) </del> <ins> return "", fmt.Errorf("failed to parse daemon set for sample plugin: %w", err) </ins> } if len(ds.Spec.Template.Spec.Containers) < 1 { <del> return "", fmt.Errorf("failed to parse the container image: cannot extract the container from YAML") </del> <ins> return "", fmt.Errorf("failed to parse the sample plugin image: cannot extract the container from YAML") </ins> } return ds.Spec.Template.Spec.Containers[0].Image, nil }

Failing: ` Failing since . Last passing: First failing:
The diff () shows that went in which looks related
/sig node
/kind regression /kind bug /triage accepted


Question: // Note that discoveryClient should NOT be shared with gc.restMapper, otherwise // the mapper's underlying discovery client will be unnecessarily reset during // the course of detecting new resources. <del> func (gc *GarbageCollector) Sync(discoveryClient discovery.DiscoveryInterface, period time.Duration, stopCh <-chan struct{}) { </del> <ins> func (gc *GarbageCollector) Sync(discoveryClient discovery.ServerResourcesInterface, period time.Duration, stopCh <-chan struct{}) { </ins> oldResources := make(map[schema.GroupVersionResource]struct{}) wait.Until(func() { // Get the current resource list from discovery. newResources := GetDeletableResources(discoveryClient) <ins> // This can occur if there is an internal error in GetDeletableResources. // If the gc attempts to sync with 0 resources it will block forever. // TODO: Implement a more complete solution for the garbage collector hanging. if len(newResources) == 0 { glog.V(5).Infof("no resources reported by discovery, skipping garbage collector sync") return } </ins> // Decide whether discovery has reported a change. if reflect.DeepEqual(oldResources, newResources) { glog.V(5).Infof("no resource updates from discovery, skipping garbage collector sync")

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: I am running etcd-operator in kubernetes, and noticed that when I deleted the etcdcluster (Custom resource definition) the pods & services did not get cleaned up. I looked on another kube environment I had running, and on that env everything worked as expected. I checked logs, and could see on the env that was working I saw the following messages in Controller manager logs: I didn't see these in the env that wasn't working, so I started looking at the garbage collector on there. I noticed that there hadn't been any log message from garbage collecor since 31st Jan, when this was logged: Around this time we experienced some environmental network issues, which caused lots of errors around that time, such as: It looks like these issues may have caused the garbage collector to get into some sort of hung state? When I restarted controller manager everything started working as expected. What you expected to happen: Garbage collector should have kept running, and working as expected. How to reproduce it (as minimally and precisely as possible): Not sure on the exact sequence that caused it, but suspect it was related to a time period when errors were occurring. Anything else we need to know?: We have set, and had previously hit this garbage collector issue - However, we are running with Kube 1.8.6 with the fix for that issue cherry-picked (commit ) Environment: Kubernetes version (use ):Cloud provider or hardware configuration: IBM Cloud OS (e.g. from /etc/os-release): Ubuntu "16.04.3 LTS (Xenial Xerus)" Kernel (e.g. ): 0-78-generic -Ubuntu SMP Thu Apr 27 15:29:09 UTC 2017 x8664 x8664 x86_64 GNU/Linux Install tools: Others: I will attach logs from Controller manager.
/sig api-machinery
/assign /assign
cc
I saw garbage collector stopped working in 1.9.2 too. Most controllers (such as node controllers or deployment controllers) stopped reporting logs in controller- when this happened. Manually restarting controller-manager fixed the problem. I think somehow some controllers got stuck (could it be because of some bugs in watch cache?) and it's not specific to garbage collector.

When the issue happened, still reported healthy for controller-manager.
tracks this
This is happening in production for us too.
did this happen to you to? Or is it that only the garbage collector disappeared from the logs?
For me the other controllers still have logs; it was just the GC that crashed.
does your cluster also use custom resources backed by Aggregated API servers?
Yes, although this happened in one cluster where we don't have any resources yet (the type exists). I'm not sure what that means
I still see messages from and and after the garbage collector stops working - so looks like some controllers are still working. Is there anything specific you were looking for?
I think the cause of this garbage collector hanging here is this: function periodically calls to the comment above GetDeletableResources, , an error in discovery causes the discovery client to no longer discover resources in the cluster, but instead of failing and returning an error, it simply logs the error as and returns an empty list of resources Sync function, upon recieving an empty resource list from discovery, detects that the resources have changed, and calls , which calls with as the argument as shown in the log as , which sets the list of monitors to an empty list because it thinks there are no resources to monitor. the Sync function calls controller.WaitForCacheSync, which calls , which will continually retry the function until it returns true, but it will always return false because is 0. So I think this was caused by which enabled number 2.


Question: SRCS = peer-finder ARCH ?= amd64 TARGET ?= $(CURDIR) <ins> GOARM ?= 7 </ins> GOLANG_VERSION ?= latest SRC_DIR = pets/peer-finder export

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Building the image binaries requires the GOARM variable to be set. Because it is not defined in all Makefiles, cannot the image binaries for the images in question. What you expected to happen: should succeed. How to reproduce it (as minimally and precisely as possible): git clone cd kubernetes/test/images/entrypoint-tester make Anything else we need to know?: Environment: N/A Kubernetes version (use ): N/A Cloud provider or hardware configuration: N/A OS (e.g: ): N/A Kernel (e.g. ): N/A Install tools: N/A Network plugin and version (if this is a network-related bug): N/A Others: N/A
/sig testing
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or .

/lifecycle stale
/lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten


Question: framework.ExpectNoError(err) }) <del> ginkgo.It("should list, patch and delete a LimitRange by collection", func() { </del> <ins> /* Release: v1.26 Testname: LimitRange, list, patch and delete a LimitRange by collection Description: When two limitRanges are created in different namespaces, both MUST succeed. Listing limitRanges across all namespaces with a labelSelector MUST find both limitRanges. When patching the first limitRange it MUST succeed and the fields MUST equal the new values. When deleting the limitRange by collection with a labelSelector it MUST delete only one limitRange. */ framework.ConformanceIt("should list, patch and delete a LimitRange by collection", func() { </ins> ns := f.Namespace.Name lrClient := f.ClientSet.CoreV1().LimitRanges(ns)

4 Oct 2022 [x] Two weeks soak end date: 18 Oct 2022 [x] Test promotion PR: According to this APIsnoop query, there are still three remaining LimitRange endpoints which are untested. listCoreV1LimitRangeForAllNamespacespatchCoreV1NamespacedLimitRangedeleteCoreV1CollectionNamespacedLimitRange Using a number of existing e2e test practices a new has been created for three LimitRange endpoints. The e2e logs for this test are listed below. This query shows which LimitRange endpoints are hit within a short period of running this e2e test example endpoint | useragent --------------------------------------------+
createCoreV1NamespacedLimitRange | should ensure that a limitRange can be listed, patched and deleted by collection deleteCoreV1CollectionNamespacedLimitRange | should ensure that a limitRange can be listed, patched and deleted by collection listCoreV1LimitRangeForAllNamespaces | should ensure that a limitRange can be listed, patched and deleted by collection listCoreV1NamespacedLimitRange | should ensure that a limitRange can be listed, patched and deleted by collection patchCoreV1NamespacedLimitRange | should ensure that a limitRange can be listed, patched and deleted by collection (5 rows) If a test with these calls gets merged, test coverage will go up by 3 point This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: import ( "context" <ins> "encoding/json" </ins> "fmt" "reflect" "strings"

In VMware Tanzu Supervisor Cluster k8s setup, we observed that the AdmissionWebhook tests fail intermittently while updating the namespace with the error: "the object has been modified; please apply your changes to the latest version and try again". This is because the update conflicts with an update being performed by another controller. The test could use Patch while adding labels via types.StrategicMergePatchType to prevent the conflict. /sig testing /sig api-machinery
(sig testing owns e.g. the frameworks but API Machinery own their test cases)
/triage accepted


Question: }, } <ins> actualRoutes, err := az.ListRoutes(ctx, clusterName) if err != nil { klog.V(3).Infof("CreateRoute: creating route: failed(ListRoutes) clusterName= %q instance=%q with error=%v", clusterName, kubeRoute.TargetNode, err) return err } for _, actualRoute := range actualRoutes { if strings.EqualFold(actualRoute.Name, kubeRoute.Name) && strings.EqualFold(string(actualRoute.TargetNode), string(kubeRoute.TargetNode)) && strings.EqualFold(actualRoute.DestinationCIDR, kubeRoute.DestinationCIDR) { klog.V(2).Infof("CreateRoute: route is already existed and matched, no need to re-create or update") return nil } } </ins> klog.V(3).Infof("CreateRoute: creating route: instance=%q cidr=%q", kubeRoute.TargetNode, kubeRoute.DestinationCIDR) err = az.CreateOrUpdateRoute(route) if err != nil {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Kubernetes shouldn't put route table when it's already been configured. Azure cloud provider should check whether the route has already been created before invoking . What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /sig cloud-provider /priority important-soon
Why this is a bug? A PUT operation should be idempotent and multiple calls should get the same result?
Right. But cloud provider shouldn't make unnecessary API calls.
To check whether it is configured, we need a GET call, so we're going to replace PUT call with GET call in this case?
it's not a replace, we need to LIST routes and check whether it has already been there, then PUT if it's not.
I got your point. But if we directly send a PUT request, seems route RP will also do this comparation? Is there any issue making us have to do the comparation by ourselves instead of sending PUT request to route RP?
It would help to reduce the chances of ARM PUT throttling.
I'd like to work on this
/assign


Question: fi # Network plugin if [[ -n "${NETWORK_PROVIDER:-}" || -n "${NETWORK_POLICY_PROVIDER:-}" ]]; then <del> if [[ "${NETWORK_PROVIDER:-}" == "cni" || "${NETWORK_POLICY_PROVIDER:-}" == "calico" ]]; then flags+=" --cni-bin-dir=/home/kubernetes/bin" else flags+=" --network-plugin-dir=/home/kubernetes/bin" fi </del> <ins> flags+=" --cni-bin-dir=/home/kubernetes/bin" </ins> if [[ "${NETWORK_POLICY_PROVIDER:-}" == "calico" ]]; then # Calico uses CNI always. flags+=" --network-plugin=cni"

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Environment: Cloud provider or hardware configuration: bare-metal OS (e.g. from /etc/os-release): Ubuntu 14.04 Kernel (e.g. ): Linux dev-k8s-controller-p1 4.4.0-53-generic ~14.04.1-Ubuntu SMP Fri Dec 2 03:43:31 UTC 2016 x8664 x8664 x86_64 GNU/Linux Install tools: chef cookbook Others: Docker 1.12.6, calico 2.1 CNI binaries: kubelet config: What happened: When trying to run pod error message is generated: So it seems that kubelet is trying to find CNI binaries in paths: /opt/loopback/bin /etc/cni/net.d /opt/calico/bin But it should look at since this is the default valur for parameter. What you expected to happen: Pod starts. How to reproduce it (as minimally and precisely as possible): Run pod: with kubelet 1.6 and CNI binaries in those path (default for calico): Anything else we need to know: After copy to and pod starts fine.
Good, very helpful to me
When set and not set , default path of CNI binaries will not take effect, the path specified by will be used instead. We should use and instead of if use CNI plugin.
I will push a PR to deprecate the flag. It is confusing. Closing this issue.
Was that ? It was closed without merging. I agree should be deprecated.
I think is needed before is removed. The flag is still not deprecated, I will send a PR to deprecate it.


Question: <healthScaleFactor>100.0</healthScaleFactor> </hudson.tasks.junit.JUnitResultArchiver> <ins> # Implements Docker Build and Publish Plugin # https://wiki.jenkins-ci.org/display/JENKINS/CloudBees+Docker+Build+and+Publish+plugin # From the docs: 'This plugin provides the ability to build projects with a Dockerfile, and publish the resultant # tagged image (repo) to the docker registry.' # # repoName: docker registry repo name to publish the image # dockerfilePath: workspace relative path to the 'Dockerfile' to build # tag: Tag to apply to the built image # credentialsId: Jenkins 'Username with password' credentials ID (hex string) to authenticate to docker registry. #                Found under http:<jenkinsurl>/credentials -> "Advanced" - builder: name: docker-build-publish builders: - raw: xml: | <com.cloudbees.dockerpublish.DockerBuilder plugin="docker-build-publish@1.1"> <server plugin="docker-commons@1.2"/> <registry plugin="docker-commons@1.2"> <credentialsId>{credentialsId}</credentialsId> </registry> <repoName>{repoName}</repoName> <noCache>true</noCache> <forcePull>true</forcePull> <dockerfilePath>{dockerfilePath}</dockerfilePath> <skipBuild>false</skipBuild> <skipDecorate>false</skipDecorate> <repoTag>{tag}</repoTag> <skipPush>false</skipPush> <createFingerprint>true</createFingerprint> <skipTagLatest>true</skipTagLatest> </com.cloudbees.dockerpublish.DockerBuilder> - shell: | #!/bin/bash set -e set -x docker rmi -f {repoName}:{tag} </ins> # Default email recipients are set in Jenkins global config - defaults: name: global

We need to commit all the jenkins job configurations into yaml files. Otherwise the yaml updater will soon delete these jobs. Please see for example PRs
Done


Question: } }) <del> It("should provide Internet connection for containers", func() { </del> <ins> It("should provide Internet connection for containers [Feature:Networking-IPv4]", func() { </ins> By("Running container which tries to ping 8.8.8.8") framework.ExpectNoError( <del> framework.CheckConnectivityToHost(f, "", "ping-test", "8.8.8.8", 30)) </del> <ins> framework.CheckConnectivityToHost(f, "", "ping-test", "8.8.8.8", framework.IPv4PingCommand, 30)) }) It("should provide Internet connection for containers [Feature:Networking-IPv6][Experimental]", func() { By("Running container which tries to ping google.com") framework.ExpectNoError( framework.CheckConnectivityToHost(f, "", "ping-test", "google.com", framework.IPv6PingCommand, 30)) </ins> }) // First test because it has no dependencies on variables created later on.

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: e2e tests provide only an (IPv4) ping test for external connectivity. What you expected to happen: Need a way to conditionally run a ping6 external connectivity check, and disable the (IPv4) external connectivity check, for IPv6-only clusters. How to reproduce it (as minimally and precisely as possible): Examine the e2e code ((), or run this test on an IPv6-only cluster. Anything else we need to know?: This test will be needed for IPv6 gating CI tests, at least until Kubernetes dual-stack is fully implemented. /area ipv6 Environment: Kubernetes version (use ): Latest master Cloud provider or hardware configuration**: VirtualBox cluster, Ubuntu 16.04.1 host, CentOS 7 guests OS (e.g. from /etc/os-release): CentOS 7 Kernel (e.g. ): 3.10.0-514.26.2.el7.x8664 SMP Tue Jul 4 15:04:05 UTC 2017 x8664 x8664 x8664 GNU/Linux Install tools: Others:
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig network


Question: return false } <ins> // processHTTPRetryResponse : return true means stop retry, false means continue retry </ins> func (az *Cloud) processHTTPRetryResponse(service *v1.Service, reason string, resp *http.Response, err error) (bool, error) { <del> if resp != nil && isSuccessHTTPResponse(resp) { </del> <ins> if err == nil && resp != nil && isSuccessHTTPResponse(resp) { </ins> // HTTP 2xx suggests a successful response return true, nil }

What happened: In a StatefulSet where each Pod has its own persistent volume provisioned by , it occasionally takes up to 30 min to detach and re-attach those disks to their node during a rollout. The following Pod events repeat: A subset of the controller-manager's logs (full logs on demand): State of the disk few minutes before the attach operation succeeds, returned by the Azure API: In this particular VMSS, each instance has at most 2 disks attached. In busy clusters, a node can have up to 5 disks attached. The operation eventually succeeds, ~20m later in this case: What you expected to happen: Disks take no longer than 3 min to re-attach (the typical time it takes in our customers' environments). How to reproduce it (as minimally and precisely as possible): Here is a copy of the manifest of the affected StatefulSet: <details<summarysolr-</summary</details</brStorage class <details<summarydefault-</summary</details</brAnything else we need to know?: Environment: Kubernetes version: v1.12.7 Cloud provider: Azure (via AKS-Engine) OS (e.g: ): Ubuntu 16.04.6 LTS Kernel: 4.15.0-1042-azure Install tools: Network plugin and version: Others: observed in westeurope and eastus regions paging /sig azure
can you also provide following info on master node: How many disks are on one VM? We are experiencing slow disk attach/detach issue when disk num is large:
I just the info about disks to the original message: between 2 and 5 per VMSS instance. The command you posted returns nothing, but if I remove "vm" from the last grep I get: Feel free to catch up on Slack if you want to have a real-time chat.
/assign paste some info here: the 2 disk attach/detach operations costs s totally, that's reason why it's stuck in so long time. BTW, there is metrics calculation issue here: , but anyway the two operations cost too long time in vmss API call level.
this seems to be an extreme case. In other environments where the issue occurs, API calls cost between 15 and 60 sec on average.

API calls cost between 15 and 60 sec on average and still takes up 30min to re-attach? could it repro easily?
Only in selected clusters unfortunately (about 5%), but when it happens it happens quite consistently. I haven't been able to reproduce outside of clusters belonging to live customers, which makes the issue a bit tricky to troubleshoot at the moment. I'm sitting in front of a cluster where it's currently happening. A Pod has been reporting for 53 min. Yesterday I observed it in the same cluster for 2 hours. We eventually deleted the Pod to unblock the situation (which doesn't always work btw). The following Pod events repeat over and over again: The disk has and just like in my first message. The controller-manager metrics also don't tell us much (14h uptime):
could you provide detailed steps about how to repro this issue? I tried to drain statefulset pods from one node to another on k8s v1.12.7, cannot repro.
The correct way to get disk attach/detach metrics on k8s v1.12.7 is like this The rolling update would cause k8s to detach disk first and then attach disk after, so I will still get following error in my testing env, but finally it could recover:
This one is a patched 1.10, with your retry fix. I have some metrics from an affected 1.12 cluster, but like I said, nothing very useful: We only have one customer on 1.12 so I'm waiting for the next incident to get back to you on a proper 1.12 setup. My understanding of what happens in most (if not all) cases is: Imagine this simplified setup 2 nodes 3 Pods belonging to a Deployment "commerce" 3 Pods belonging to a Deployment "api" 3 Pods with Azure Disks belonging to a StatefulSet "solr" We update both Deployments and the StatefulSet simultaneously (we have an operator that actually does exactly this). Both and terminate before . This time and are both scheduled on . is still Terminating. When is finally down, is already packed, so the scheduler has no other choice but to move it to . And then we are in that mode where the disk is detached, but never re-attached.

It shouldn't be any different from a drain, but at least it can maybe help you understand our typical repro scenario. We recently optimized scheduling to avoid that schenario where stateful Pods must be relocated, but other events can cause relocation like nodes failures, autoscaling, ...
here is the conclusion about what we found now: Periodically k8s cluster failed to attach a disk to a VMSS instance, that disk is actually in “unattached” state, it's caused by vmss instance in unhealthy state, although by azure cli command , the instance shows healthy. workaround: run to update the VMSS instance manually, it could recover in a minute or so We are asking VMSS team to find the root cause of this issue. In brief, if you find k8s could not attach an state disk to a VMSS node, it's probably due to the vmss instance issue, run to update the VMSS instance manually could solve this issue.


Question: } } <ins> func TestAdmitEmptyPodNamespace(t *testing.T) { containerName := "container" pod := &api.Pod{ ObjectMeta: metav1.ObjectMeta{ Name: "mypod", Labels: map[string]string{ "security": "S2", }, }, Spec: api.PodSpec{ Containers: []api.Container{ { Name: containerName, Env:  []api.EnvVar{{Name: "abc", Value: "value2"}, {Name: "ABCD", Value: "value3"}}, }, }, }, } pip := &settings.PodPreset{ ObjectMeta: v1.ObjectMeta{ Name:      "hello", Namespace: "different", // (pod will be submitted to namespace 'namespace') }, Spec: settings.PodPresetSpec{ Selector: v1.LabelSelector{ MatchExpressions: []v1.LabelSelectorRequirement{ { Key:      "security", Operator: v1.LabelSelectorOpIn, Values:   []string{"S2"}, }, }, }, Volumes: []api.Volume{{Name: "vol", VolumeSource: api.VolumeSource{EmptyDir: &api.EmptyDirVolumeSource{}}}}, Env:     []api.EnvVar{{Name: "abcd", Value: "value"}, {Name: "ABC", Value: "value"}}, EnvFrom: []api.EnvFromSource{ { ConfigMapRef: &api.ConfigMapEnvSource{ LocalObjectReference: api.LocalObjectReference{Name: "abc"}, }, }, { Prefix: "pre_", ConfigMapRef: &api.ConfigMapEnvSource{ LocalObjectReference: api.LocalObjectReference{Name: "abc"}, }, }, }, }, } originalPod, err := api.Scheme.Copy(pod) if err != nil { t.Fatal(err) } err = admitPod(pod, pip) if err != nil { t.Fatal(err) } // verify PodSpec has not been mutated if !reflect.DeepEqual(pod, originalPod) { t.Fatalf("Expected pod spec of '%v' to be unchanged", pod.Name) } } </ins> func admitPod(pod *api.Pod, pip *settings.PodPreset) error { informerFactory := informers.NewSharedInformerFactory(nil, controller.NoResyncPeriodFunc()) store := informerFactory.Settings().InternalVersion().PodPresets().Informer().GetStore()

/kind bug What happened: A pod with an empty namespace field was mutated by a pod preset in a different namespace. What you expected to happen: Pod presets should only operate on pods in the same namespace. How to reproduce it (as minimally and precisely as possible): Submit a pod with an empty namespace field and observe pod preset mutation. Anything else we need to know?: This is a fix for an alpha non-default feature. Fix submitted in
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig api-machinery


Question: pflag.CommandLine.AddGoFlagSet(goflag.CommandLine) pflag.Parse() pflag.VisitAll(func(flag *pflag.Flag) { <del> glog.Infof("FLAG: --%s=%q", flag.Name, flag.Value) </del> <ins> glog.V(4).Infof("FLAG: --%s=%q", flag.Name, flag.Value) </ins> }) }

Reported by in Filing a bug for easy tracking. After , now prints all flags and the version. This is pretty unconventional and should probably be fixed. /cc
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
/sig cli
/cc for approval.
cc/ cc/
Is it causing a problem? It does help (a lot) for debugging both in production and e2e test failures.
Oh, wait. I see. The info log is going to stdout or stderr instead of to a log file. Maybe we can detect this in the flag-printing code and only log the flags if the logs are being sent to a file?
Who's going to work on this? Can someone be assigned?
Is this already been fixed in ?


Question: e2eservice.TestReachableHTTP(ctx, ingress, svcPort, loadBalancerLagTimeout) ginkgo.By("checking reachability from pods when LoadBalancerSourceRanges is unset") <del> // We can use timeout 0 here since we know from above that the service is // already running (and we aren't waiting for changes to it to propagate). checkReachabilityFromPod(ctx, true, 0, namespace, acceptPod.Name, ingress) checkReachabilityFromPod(ctx, true, 0, namespace, dropPod.Name, ingress) </del> <ins> // There are different propagation delay for the APIs for different nodes, so it tries // a few times, despite previously it was confirmed that the Service was reachable. checkReachabilityFromPod(ctx, true, e2eservice.KubeProxyEndpointLagTimeout, namespace, acceptPod.Name, ingress) checkReachabilityFromPod(ctx, true, e2eservice.KubeProxyEndpointLagTimeout, namespace, dropPod.Name, ingress) </ins> // Create source ranges that allow acceptPod but not dropPod or // cluster-external sources. We assume that the LBSR rules will either see

master-informing: gce-cos-master-slow Failure cluster No response /sig network
checking why has the context deadline exceeded?
I see, is because of this /assign sending a fix for this
/triage accepted


Question: embed = [":go_default_library"], tags = ["integration"], deps = [ <ins> "//staging/src/k8s.io/api/core/v1:go_default_library", </ins> "//staging/src/k8s.io/apimachinery/pkg/apis/meta/v1:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/runtime/schema:go_default_library", "//staging/src/k8s.io/client-go/kubernetes:go_default_library", "//staging/src/k8s.io/client-go/rest:go_default_library", "//staging/src/k8s.io/component-base/metrics/testutil:go_default_library", "//test/integration/framework:go_default_library", <ins> "//vendor/github.com/prometheus/common/model:go_default_library", </ins> ], )

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: While I was watching grafana for a single run, I noticed that GET requests are reported as LIST requests, e.g. grafana shows ~1K QPS for LIST of pods ! while kube-apiserver's logs show: (there are no pod lists that time). I'm not sure if this is intended change to metrics reporting (kind of unexpected to be honest), but this definitely affects way our scalability tests validate latency metrics: We do not validate GET API call latency at all Validation for LIST requests is also affected, as usually GET API calls are way cheaper than LIST requests, so grouping GET-LISTs and LIST-LISTs together lowers p99 for LISTs. Also perf-dash shows that GET requests simply disappeared at some point: http://perf- ! What you expected to happen: GET requests should be reported with verb=GET in apiserverrequest_total metric. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:

FYI
I managed to narrow the issue by bisecting the runs of which runs load tests using master k/k every 1 hour. This is the last test run that had GETs reported correctly: This is the first test run that has no GETs reported: Looking at the k/k repo commits' hashes inside artifacts in both runs and at what happened between each of them it looks as if might be the culprit.
From the former run's artifacts: This is (among others) what we miss in the latter run - this is understandable from the point of view of this if condition: Why did the reported GETs have as the scope instead of in the first place, however??
It seems that our handing is incorrect: vs Or my understanding of it was incorrect.

It seems that the semantic is: cluster - for cluster-scope requests namespace - for namespace-scoped requests request - for subresources I think the assumption in the monitoring path was: cluster - for cluster-scope requests namespace - for namespace-scope requests requests - for requests touching a single resource (not necessary subresource) I'm not 100% sure which semantic we should use - based on the finding we should fix one or the other place...
cc There should probably be a unit test mapping URL paths to the expected label values.
is opened to fix that.
/triage accepted


Question: req := sts.TokenRequest{ Certificate: &cert, <ins> Delegatable: true, </ins> } signer, err := tokens.Issue(ctx, req)

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: The vSphere cloud provider is unable to fetch the node failure domain labels when using VCP with SAML auth because of an authentication failure. For context, the SAML token auth is described here - SAML auth support for the zones api was introduced in this PR - , which was cherry-picked to 1.14.1 here - I've tested the credentials I'm using with govc to ensure that I'm able to get a token: The credentials also work fine when using VCP without the node labels feature. Relevant kubelet logs: What you expected to happen: VCP should add failure domain labels to the node object. How to reproduce it (as minimally and precisely as possible): Setup SAML auth for VCP with zone labeling enabled. Anything else we need to know?: There was a previous issue that was raised () but that was before SAML auth support for zones was implemented. This could potentially be an issue with govc handling of SAML tokens since even with the Administrator role assigned to the user, I'm still unable to list tags. The role and keys were generated with the following: Environment: Kubernetes version (use ): 14.1 Cloud provider or hardware configuration: vSphere OS (e.g: ): Linux & Windows
/sig vmware
Would you mind taking a look at this? I can provide more info if needed.
govc's command only works with the SOAP endpoint currently. The REST endpoint has its own session manager and requires a separate login. At the moment, govc's tags commands login/logout when each command is run and don't use a persisted session (which are saved to ~/.govmomi/sessions). However, you can use a token returned by to test SAML auth with the tags commands. For example: Can you give that try? There are some TODOs around this btw:
Those commands work for me. So that would imply the solution user, role, and permissions are set up correctly? Would you expect the to also work?

I seem to be getting a 404 when using with the flag - Does seem to work when specifying a tag name: Are there any other govc commands I should check / verify?
yes, the solution user setup looks right. Not sure what the issue is with . Can you share your vCenter build number ()? I'll try to reproduce the issue.
$ govc about -u vcdo Name: VMware vCenter Server Vendor: VMware, Inc. Version: 6.7.0 Build: OS type: linux-x64 API type: VirtualCenter API version: 6.7.2 Product ID: vpx UUID: -55ed-4737-9cc4-
I can reproduce this, but don't know what the issue is yet. The version of Go and vSphere have changed since , but if it was one of those was the cause, I'd expect SAML auth to fail with the govc tags commands too. All I have to go on for now is the vCenter log message below. I'll update when I have more info.
I have it working with the change below, but need to figure out a few things first: Why this is the case when run inside kubelet and not govc (note that the port change to 443 breaks govc) A good way to determine which port to use (depends on the answer to the above) If this is behavior is specific to 6.7U2 or something triggered by a change in K8s I suppose a quick fix would change to try port 443 if 80 fails, but would prefer to avoid that as the failure generates stack traces in the
/assign
This is obvious in retrospect, but the issue is the message signature needs to use the same port as the original request URL. However, in the govc case there was no port set in the request URL. The client defaults to port when connecting (based on the scheme), but does not include any port with the header. In the case, the default is on the server side. This can be reproduced with if includes the port , rather than just as we were using. The VCP tests that use token auth for tags lookup just don't validate the signature, so this wasn't a change related to 6.7U2.

I'll open a K8s PR to pull in the fix:
Updating k/vendor to govmomi master adds almost 6K lines for a 3 line fix. I may just create a govmomi branch off of v0.20 to reduce churn in the K8s release branches for this fix, may have to wait until tomorrow.
Really appreciate the insight into this issue. Can confirm that specifying the port (443) in the GOVC_URL reproduces the issue. Thank you!
Unfortunately, I'm seeing the same issue with v1.14.3: cloud_config Works fine when I use the username and password of a vsphere admin, but using cert and key still has authentication issues.
any related errors in vCenter's ?
Seeing a lot of this exception:
whoa! that stack trace is like punch in the gut (brings back memories!) :)
indeed!! apologies, I must have had the change for PR locally, but failed to include it as I had focused on the signature issue and updating vendor/


Question: } } <del> func TestJobStrategyWithGeneration(t *testing.T) { ctx := genericapirequest.NewDefaultContext() theUID := types.UID("1a2b3c4d5e6f7g8h9i0k") validPodTemplateSpec := api.PodTemplateSpec{ Spec: api.PodSpec{ RestartPolicy: api.RestartPolicyOnFailure, DNSPolicy:     api.DNSClusterFirst, Containers:    []api.Container{{Name: "abc", Image: "image", ImagePullPolicy: "IfNotPresent", TerminationMessagePolicy: api.TerminationMessageReadFile}}, }, } job := &batch.Job{ ObjectMeta: metav1.ObjectMeta{ Name:      "myjob2", Namespace: metav1.NamespaceDefault, UID:       theUID, }, Spec: batch.JobSpec{ Selector: nil, Template: validPodTemplateSpec, }, } Strategy.PrepareForCreate(ctx, job) errs := Strategy.Validate(ctx, job) if len(errs) != 0 { t.Errorf("Unexpected error validating %v", errs) } // Validate the stuff that validation should have validated. if job.Spec.Selector == nil { t.Errorf("Selector not generated") } expectedLabels := make(map[string]string) expectedLabels["controller-uid"] = string(theUID) if !reflect.DeepEqual(job.Spec.Selector.MatchLabels, expectedLabels) { t.Errorf("Expected label selector not generated") } if job.Spec.Template.ObjectMeta.Labels == nil { t.Errorf("Expected template labels not generated") } if v, ok := job.Spec.Template.ObjectMeta.Labels["job-name"]; !ok || v != "myjob2" { t.Errorf("Expected template labels not present") } if v, ok := job.Spec.Template.ObjectMeta.Labels["controller-uid"]; !ok || v != string(theUID) { t.Errorf("Expected template labels not present: ok: %v, v: %v", ok, v) } } func TestJobStatusStrategy(t *testing.T) { </del> <ins> func TestJobStrategy_WarningsOnUpdate(t *testing.T) { </ins> ctx := genericapirequest.NewDefaultContext() <del> if !StatusStrategy.NamespaceScoped() { t.Errorf("Job must be namespace scoped") } if StatusStrategy.AllowCreateOnUpdate() { t.Errorf("Job should not allow create on update") } </del> validSelector := &metav1.LabelSelector{ MatchLabels: map[string]string{"a": "b"}, }

The gets very long testing mulitple functions in such as , and . These checks should be performed in dedicated Test functions using table for different scenarios. Target functions and are being as part of The remaining checks should be migrated to these functions. Also, a test function should be to migrate the remaining tests.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig apps
/assign
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
Any progress on this? I'm working on and I noticed that has relatively low coverage so I was going to work on getting the coverage up.
I created that is somewhat related to this.
no progess. go ahead please
/assign