Question: } } <del> // TODO(#113606): connect this with the incoming context parameter, which comes from the pod worker. // Currently, using that context causes test failures. To remove this todoCtx, any wait.Interrupted // errors need to be filtered from result and bypass the reasonCache - cancelling the context for // SyncPod is a known and deliberate error, not a generic error. todoCtx := context.TODO() </del> <ins> // TODO(#113606): use cancellation from the incoming context parameter, which comes from the pod worker. // Currently, using cancellation from that context causes test failures. To remove this WithoutCancel, // any wait.Interrupted errors need to be filtered from result and bypass the reasonCache - cancelling // the context for SyncPod is a known and deliberate error, not a generic error. // Use WithoutCancel instead of a new context.TODO() to propagate trace context </ins> // Call the container runtime's SyncPod callback <del> result := kl.containerRuntime.SyncPod(todoCtx, pod, podStatus, pullSecrets, kl.backOff) </del> <ins> sctx := context.WithoutCancel(ctx) result := kl.containerRuntime.SyncPod(sctx, pod, podStatus, pullSecrets, kl.backOff) </ins> kl.reasonCache.Update(pod.UID, result) if err := result.Error(); err != nil { // Do not return error if the only failures were pods in backoff

spans are not being parented to other spans - propagation of context seems broken. More context: spans should be parented to other spans enable kubelet tracing at v1.28 and hook up a tracing backend. No response <detailsv1.28 at least </details<detailsNot sure this matters. </details<details</details<details</details<details</details<details</details
/sig instrumentation /assign
/triage accepted
!
repro steps:
/assign Thanks for helping to look into this! Feel free to post the findings here you shared with me on slack
GitHub didn't allow me to assign the following users: fatsheep9146. Note that only with read permissions, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Thanks! I will post this ASAP
The broken issue is due to the pr , which try to propagate context brought by pod worker to syncPod function in order to let the syncPod can be interrupted in time by other asynchronous routines, for example eviction. But in the line we did not pass the context which contain the parent span and trace into function, so the trace is broken. So I want to fix this by passing the into , which is also desired in comment But passing the ctx into syncPod may have something other to do as described in comments, so I tried to fix as comments advised.
cc


Question: Replicas: replicas, Template: api.PodTemplateSpec{ ObjectMeta: api.ObjectMeta{ <del> Labels: labels, Annotations: map[string]string{ "pod.alpha.kubernetes.io/initialized": "false", }, </del> <ins> Labels:      labels, Annotations: map[string]string{}, </ins> }, Spec: api.PodSpec{ Containers: []api.Container{

What happened: What you expected to happen: The statefulset should have scaled down to 0. Instead, even after 10 minutes: How to reproduce it (as minimally and precisely as possible): Run the e2e test on a gce cluster running the latest and greatest kubernetes. cc
if the petset has replicas=0 it's not going to do anything
When I a running petset, and set its scale to 0, that seems to work and delete the pets, and we should have a functioning scaler. Am I misunderstanding something? I'm using the same teardown process for the e2e test I linked in the issue as the other tests (), but it times out sometimes after network partition.
yeah is scale in the petset controller noticing deletion timestamp? because the test code you showed doesn't care
As I mentioned yesterday, it isn't even issuing a delete request even after 10 minutes, because the pods continue to be "running". I'll check the controller manager logs today. I suspect it might be the backoff in the rate limited work-queue.
Hmm, the AGE fields on your pets don't make sense if it isn't even issuing a delete?
In the output I pasted above, at the 7 minute mark, it created the new replacement pet-0, after the network partition was healed. Then tried to scale down the StatefulSet and waited 10 minutes for the pods to disappear.
I see a bunch of different failures modes on the bug: 3 08:58:44.475: Failed waiting for pods to enter running: timed out waiting for the condition Failed to scale statefulset to 0 in 10m0s. Remaining pods: [] 3 08:11:30.728: Unexpected error: <nilBasically 3 translates to We should clarify. 1 and 2 both have events like guessing the node controlelr shouldn't be setting timestamp?
Looking into the case 2: The controller manager log goes: So I guess even deletions are being blocked by the health/readiness.
Both issues appear to be due to the testsuite and the way we use the "initialized" annotation in tests. I'll send a PR to fix these shortly. Thanks


Question: <ins> /* Copyright 2020 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package app import "testing" func TestIsValidPriorityClass(t *testing.T) { testCases := []struct { description           string priorityClassName     string expectedPriorityValue uint32 }{ { description:           "Invalid Priority Class", priorityClassName:     "myPriorityClass", expectedPriorityValue: 0, }, { description:           "Valid Priority Class", priorityClassName:     "IDLE_PRIORITY_CLASS", expectedPriorityValue: uint32(64), }, } for _, test := range testCases { actualPriorityValue := getPriorityValue(test.priorityClassName) if test.expectedPriorityValue != actualPriorityValue { t.Fatalf("unexpected error for %s", test.description) } } } </ins>

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Windows nodes running with high (90%) CPU usage are having calls to /stats/summary time out with "Context Deadline Exceeded". When this happens metrics-server cannot gather accurate CPU/memory usage for the node or pods running on the node which results in HPA not functioning returning <unknowninstead of values What you expected to happen: Windows nodes should be able to run with high CPU utilization without negatively impacting Kubernetes functionality How to reproduce it (as minimally and precisely as possible): a cluster with Windows Server 2019 nodes CPU intensive workloads into cluster ex: workload to put node into high CPU state node metrics with Anything else we need to know?: This issue appears to be easier to reproduce on 1.18.x nodes vs 1.16.x or 1.17.x nodes Ex: ! ! metrics-server logs show E1020 10:58:24. 1 ] unable to fully collect metrics: unable to fully scrape metrics from source kubeletsummary:akswin000000: unable to fetch metrics from Kubelet akswin000000 (10.240.0.115): Get https://10.240.0.115:/stats/summary?onlycpuandmemory=true: context deadline exceeded Kubelet logs show and Environment: Kubernetes version (use ): 1.18.8 Cloud provider or hardware configuration: AKS / Azure OS (e.g: ): WIndows Server 2019 Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: Docker EE version 19.3.11
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.

</details
/sig windows /area kubelet
I was able to reproduce on 1.18.8 during a scale operation: where it fully subscribes all CPU's in Allocatable: Metric server logs: I do not see "context deadline" in kubelet but I did find some logs assocated: Once the pods are scheduled the node remains at 100% cpu and I no longer see errors but is not accurate: on the node (run after the pod deployment stabilized)
Adding to kubelete's flag and I no longer see the errors. This reserves some cpu for kubelet and other processes. We call this out in the but do not do the same for cpu.
Also, without setting system reserved CPU, it looks like does not accurately reflect CPU usage on the nodes when CPU utilization is very high. When I got some nodes into this state I did not see any error messages in logs from metrics-server and also I noticed that that querying the metrics API directly from showed recent timestamps.
I run into this again during scaling, collectiong logs for analysis here: Metric Server logs: Kubelet logs:
have you experienced this at all?
I believe has seen high CPU issues before. Not particularly on this. I've heard there were kernel issues in the past that triggered high CPU cases.
Some investigations on our end are showing that the DiagTrack service running in the infra/pause container can consume non-trivial amounts of CPU. I'll create an issue to address that problem as it might help out some here. That certainly will not eliminate the problem we are seeing tho.
is an issue for DiagTrack
The differences in behavior between 1.17. and 1.18. clusters looks like it is caused by (which was NOT backported to any prior releases) K8s version <= 1.17 behavior Prior to these changes CPU weights for containers were inferred based on limit values set in the pod specs. The observed behavior on Windows system when CPU weights were used was that under high-CPU load pods/containers were not guaranteed their scheduling time. The good news here is that system critical services like kubelet/docker/etc were still able to get enough CPU time for the nodes to function properly. K8s version = 1.18 behavior After these changes CPU limits were enforced for pods.

The observed behavior on 1.18 clusters is the when a Windows node enters a high-CPU state is that containers are guaranteed their scheduling time but host resources start to get starved. Usually this manifests as the node entering the state, metrics not working correctly (because calls to /stats/summary are timing out) or both. Suggestion / possible fixes Set the Windows process priority of the kubelet to  to help ensure it gets CPU time Kubelet can do this itself though calls (GetCurrentProcess() and Get/SetPriorityClass()) Set reserve CPU with with --system-reserves=[cpu=500m] This needs to scaled based on number of CPUs on host based on my experimentations Currently guidance on for Windows only mentions reserving system-memory It is still possible to over-commit CPU and starve kubelet/docker/HCS if users schedule pods with mis-matching requests and limits Update documentation.
Kubelet can do this itself though calls (GetCurrentProcess() and Get/SetPriorityClass()) I can work on this, I am wondering if we should do the same for CRI(like docker) and CNI processes.
Sure, if you want to work on this go ahead. I think we should probably try to make this be toggleable via a kubelet flag. I believe there are other windows specific flags already implemented in the kubelet (like one to run as a windows service IIRC)
docker/containerd should probably have a flag that allows them to run as high priory as well but would likely be different issues in the respective projects
/reopen I don't think this was ever fully addressed
Reopened this issue. <detailsIn response to : /reopen I don't think this was ever fully addressed Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
also related to
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: // No credentials provided, useInstanceMetadata should be enabled for Kubelet. // TODO(feiskyer): print different error message for Kubelet and controller-manager, as they're // requiring different credential settings. <del> if !config.UseInstanceMetadata && az.Config.CloudConfigType == cloudConfigTypeFile { </del> <ins> if !config.UseInstanceMetadata && config.CloudConfigType == cloudConfigTypeFile { </ins> return fmt.Errorf("useInstanceMetadata must be enabled without Azure credentials") }

What happened: While trying to setup a outside of the cluster, I had a panic as follow: What you expected to happen: Produce an error message as my configuration lacks some options. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): 1.18.4 Cloud provider or hardware configuration: Azure Others: --cloud-config= The panics come from because we exit as we're not in this case as the is just defaulted-initialised by the caller to here: and effectively assigned afterwards
/sig cloud-provider
/area provider/azure
cc
thanks for reporting the issue and filing the PR to fix it


Question: assert.EqualValues(t, seed+offsetFsInodesFree, *stats.InodesFree, label+".InodesFree") } <del> func checkEphemeralStats(t *testing.T, label string, containerSeeds []int, volumeSeeds []int, stats *statsapi.FsStats) { </del> <ins> func checkEphemeralStats(t *testing.T, label string, containerSeeds []int, volumeSeeds []int, containerLogStats []*volume.Metrics, stats *statsapi.FsStats) { </ins> var usedBytes, inodeUsage int for _, cseed := range containerSeeds { <del> usedBytes = usedBytes + cseed + offsetFsTotalUsageBytes </del> <ins> usedBytes += cseed + offsetFsBaseUsageBytes </ins> inodeUsage += cseed + offsetFsInodeUsage <ins> // If containerLogStats is nil, then the log stats calculated from cAdvisor // information is used. Since it's Total - Base, and these values are // set to the offset, we can use the calculated difference in the offset // to account for this. if containerLogStats == nil { usedBytes += offsetFsTotalUsageBytes - offsetFsBaseUsageBytes } </ins> } for _, vseed := range volumeSeeds { <del> usedBytes = usedBytes + vseed + offsetFsUsage </del> <ins> usedBytes += vseed + offsetFsUsage </ins> inodeUsage += vseed + offsetFsInodeUsage } <ins> for _, logStats := range containerLogStats { usedBytes += int(logStats.Used.Value()) inodeUsage += int(logStats.InodesUsed.Value()) } </ins> assert.EqualValues(t, usedBytes, int(*stats.UsedBytes), label+".UsedBytes") assert.EqualValues(t, inodeUsage, int(*stats.InodesUsed), label+".InodesUsed") }

Rotated container logs space usage doesnt seem to be counted towards the epehemral storage limit. Always only current log file stats are taken into account. rotated log files usage also should be counted towards ephemeral-storage usage No response <detailsTested on Kubernetes Version : 1.21.7, 1.22 Container Runtime : Cri-o </details<details</details<details</details<details</details<details</details<details</details
/sig node
/cc
/assign we can see if there is a issue
Were the rotated log files in /var/log/pods generated by yourself(not by k8s)? The cadvisor can only collect log files on container's filesystem.
No, I have not created any files. they are generated by k8s itself.
Okay, I see. As the cri-o runtime uses cadvisor stats provider to get container stats, it caculates container logUsage by formular . but the default root dir or log dir of cri-o is not /var/log/pods/{podName}/, so theses log files are not on container's filesystem, then cadvisor may not caculate the data accurately. /cc the containerd uses cri stats provider and gets logstats by HostStatsProvider, it can get right data.
In the kubelet code, , the log path is in the format , but the directory of your environment is , I did not see , Did you set this path yourself?
sorry I truncated the name for better readability, the actual path name is . So I suppose its taking in the above format only.
log files gets generated in the path /var/log/pods/defaultephem15334a37d-df02-435c-b385-/con2 for cri-o in my setup
/area cadvisor /triage accepted /priority important-longterm
/retitle Rotated container log files size not counted towards ephemeral-storage's limit on CRI-O
hi, In the past two days, I have studied the collection of file information by cAdvisor. docker and cri-o runtime basically have the same mechanism in cAdvisor. I have verified that when the runtime is cri-o, the log information statistics are also correct. So I need you to provide more detailed information. I found that the command you show is: , please help to confirm whether the "logs" information belongs to the container con2.

or use the command to try
I have verified and info provided is for con2 only . I have the sample yaml used for testing , please let me know if I'm doing it right or anything missing. I have set the kubelet log configuration as containerLogMaxSize: "5Ki", containerLogMaxFiles: 5 After applying the above yaml stats summary shows con1 logs usedbytes always 0 , con2 logs usedbytes as 4096
attached the stats summary of the command And I'm testing on kubernetes 1.22 release
/assign
what version of CRI-O/kubernetes are you using? using latest main of both kubernetes and cri-o with your reproducer causes the container to be evicted because of it exceeding ephemeral storage
oh I see now: my mistake
another mistake: I was using CRI stats provider, instead of cAdvisor stats provider (as is default)
Is CRI stats provider used by default in 1.23 by default? And also problem is seen only if containerLogMaxSize is less that the ephemeral storage limit set , if its more than the ephemeral storage limit , its the right behavior , pod gets evicted in my setup.
at long last, I believe I've found the issue!


Question: return ret } <del> func externalAttacherClusterRoleBinding( </del> <ins> func csiClusterRoleBinding( </ins> client clientset.Interface, config framework.VolumeTestConfig, teardown bool,

Current CSI E2E test has some issues which prevent it to run successfully the a local cluster. /sig storage /kind bug /assign


Question: ginkgo.Describe("Granular Checks: Pods", func() { <ins> checkNodeConnectivity := func(config *e2enetwork.NetworkingTestConfig, protocol string, port int) { // breadth first poll to quickly estimate failure. failedPodsByHost := map[string][]*v1.Pod{} // First time, we'll quickly try all pods, breadth first. for _, endpointPod := range config.EndpointPods { framework.Logf("Breadth first check of %v on host %v...", endpointPod.Status.PodIP, endpointPod.Status.HostIP) if err := config.DialFromTestContainer(protocol, endpointPod.Status.PodIP, port, 1, 0, sets.NewString(endpointPod.Name)); err != nil { if _, ok := failedPodsByHost[endpointPod.Status.HostIP]; !ok { failedPodsByHost[endpointPod.Status.HostIP] = []*v1.Pod{} } failedPodsByHost[endpointPod.Status.HostIP] = append(failedPodsByHost[endpointPod.Status.HostIP], endpointPod) framework.Logf("...failed...will try again in next pass") } } errors := []error{} // Second time, we pass through pods more carefully... framework.Logf("Going to retry %v out of %v pods....", len(failedPodsByHost), len(config.EndpointPods)) for host, failedPods := range failedPodsByHost { framework.Logf("Doublechecking %v pods in host %v which werent seen the first time.", len(failedPods), host) for _, endpointPod := range failedPods { framework.Logf("Now attempting to probe pod [[[ %v ]]]", endpointPod.Status.PodIP) if err := config.DialFromTestContainer(protocol, endpointPod.Status.PodIP, port, config.MaxTries, 0, sets.NewString(endpointPod.Name)); err != nil { errors = append(errors, err) } else { framework.Logf("Was able to reach %v on %v ", endpointPod.Status.PodIP, endpointPod.Status.HostIP) } framework.Logf("... Done probing pod [[[ %v ]]]", endpointPod.Status.PodIP) } framework.Logf("succeeded at polling %v out of %v connections", len(config.EndpointPods)-len(errors), len(config.EndpointPods)) } if len(errors) > 0 { framework.Logf("pod polling failure summary:") for _, e := range errors { framework.Logf("Collected error: %v", e) } framework.Failf("failed,  %v out of %v connections failed", len(errors), len(config.EndpointPods)) } } </ins> // Try to hit all endpoints through a test container, retry 5 times, // expect exactly one unique hostname. Each of these endpoints reports // its own hostname.

What would you like to be : Right now we poll one node at a time for intra-pod tests. We ultimately fail a test if any one node is unreachable. If we instead polled all nodes, multiple times, and collected the number of times a node was unreachable, this would tell us what % of nodes on the cluster were broken, rather then just telling us the first node that was broken. not break the testing contract (were still verifying intra-pod connectivity, just doing so in a way that gives us holistic cluster info) Why is this needed: I described why this would make triage alot easier, and in general make it easier to see why tests are failing, here ... i'll collect a little more data on this to justify the decision .... as this is a conformance test so the bar for modification is higher (even if the test itself isnt changing) then regular tests.
/sig network
/assign
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/remove-triage unresolved


Question: "-j", "ACCEPT", ) <ins> // Install the kubernetes-specific postrouting rules. We use a whole chain for // this so that it is easier to flush and change, for example if the mark // value should ever change. // NB: THIS MUST MATCH the corresponding code in the kubelet writeLine(proxier.natRules, []string{ "-A", string(kubePostroutingChain), "-m", "mark", "!", "--mark", fmt.Sprintf("%s/%s", proxier.masqueradeMark, proxier.masqueradeMark), "-j", "RETURN", }...) // Clear the mark to avoid re-masquerading if the packet re-traverses the network stack. writeLine(proxier.natRules, []string{ "-A", string(kubePostroutingChain), // XOR proxier.masqueradeMark to unset it "-j", "MARK", "--xor-mark", proxier.masqueradeMark, }...) masqRule := []string{ "-A", string(kubePostroutingChain), "-m", "comment", "--comment", `"kubernetes service traffic requiring SNAT"`, "-j", "MASQUERADE", } if proxier.iptables.HasRandomFully() { masqRule = append(masqRule, "--random-fully") } writeLine(proxier.natRules, masqRule...) // Install the kubernetes-specific masquerade mark rule. We use a whole chain for // this so that it is easier to flush and change, for example if the mark // value should ever change. writeLine(proxier.natRules, []string{ "-A", string(KubeMarkMasqChain), "-j", "MARK", "--or-mark", proxier.masqueradeMark, }...) </ins> // Write the end-of-table markers. writeLine(proxier.filterRules, "COMMIT") writeLine(proxier.natRules, "COMMIT")

Which jobs are failing: pull-kubernetes-e2e-gci-gce-ipvs Which test(s) are failing: Not all of these fail on every run: [sig-network] Networking Granular Checks: Services should function for endpoint-Service: http [sig-network] Services should allow pods to hairpin back to themselves through services [sig-storage] Flexvolumes should be mountable when non-attachable [sig-network] Networking Granular Checks: Services should function for endpoint-Service: udp [sig-network] Services should have session affinity timeout work for service with type clusterIP [LinuxOnly] [Conformance] [sig-network] Services should allow pods to hairpin back to themselves through services Since when has it been failing: Looking through job history, it looks like the last time any runs passed was on May 2. Testgrid link: N/A since testgrid filtered tests based on [sig-network] Anything else we need to know: This was discovered by in , it also appears to be affecting other PRs. /cc /sig network
This seems related to , but if I'm interpreting test results correctly pull-kubernetes-e2e-gci-gce-ipvs never stopped failing.
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/assign I'll try to fix those cases
May I ask why do you think this is related to ? Seems like first two flexvolumes related cases failed, then after some commit, service related cases start failing.
let me handle the services tests, I think I know the problem, will try to submit a fix today. Do you know what happen with the storage ones?
/assign I just submitted a PR to fix the failing tests
Thanks, seems like all cases fixed.


Question: import ( "fmt" "math/rand" <ins> "reflect" </ins> "sort" "strconv" "testing"

What happened: When creating a StatefulSet that uses volumeClaimTemplates, the resulting PersistentVolumeClaims have incorrect labels. Instead of using the labels defined in their metadata, they use the labels from the StatefulSet's selector. What you expected to happen: I expect the created PersistentVolumeClaims to have the labels defined in their metadata. How to reproduce it (as minimally and precisely as possible): Resulting PersistentVolumeClaim: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: GKE Install tools: Others: . /sig apps
+1 just ran into this myself.
/assign
GitHub didn't allow me to assign the following users: cwdsuzhou. Note that only and repo collaborators can be assigned and that issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
As in . Even thought the StatefulSet controller looks for the PVC by name (not by label) we generate labels for the PVC based on the StatefulSet's selector. Removing this would be a backward incompatible change to a v1 API contract (can't do that). Adding additional labels might be acceptable, but, If you want to change the behavior of the controller wrt to this, I'd like to see a (small) KEP for it so it can be discussed a bit more broadly in the SIG.
I very much think this would be a good discussion to have. If you could, please let me know what's required to start a KEP on this topic. As I see it, the current implementation is in violation of the API: the volumeClaimTemplates data type is listed as " array". A PersistentVolumeClaim's metadata is listed as a standard type. The fact that the PersistentVolumeClaim's metadata is being handled differently when defined in a StatefulSet means that the PersistentVolumeClaim API isn't being properly followed in that context. I'd also like to argue that the change wouldn't be backwards incompatible.

The StatefulSet's volumeClaimTemplates description even says that the matching is done on name: "Every claim in this list must have at least one matching (by name) volumeMount in one container in the template." Respecting the PersistentVolumeClaim's metadata wouldn't necessarily cause any incompatibility from a technical perspective (although it may affect some users' implementations that rely on it). In summary, the current implementation violates the principle of least surprise, makes it impossible to label individual volume claims separately, and it directly contradicts the API specification.
The KEP process is described . You can open a KEP via a PR against the SIG Apps repository. You can use me as your approver. As to the compatibility of the current PR, the current label generation is v1 and has worked the same way for several years now. Users that select the PVCs created by the controller have a reasonable expectation that this should continue to function in subsequent releases. If you add additional labels but generate labels that match the selector this would allow for additional labels while maintaining compatibility (imo).


Question: } By("Waiting for the target pod to stop running") <del> waitErr := f.WaitForPodNoLongerRunning(pod.Name) </del> <ins> if err := f.WaitForPodNoLongerRunning(pod.Name); err != nil { framework.Failf("Pod did not stop running: %v", err) } </ins> <del> By("Retrieving logs from the target pod") </del> <ins> By("Verifying logs") </ins> logOutput, err := framework.GetPodLogs(f.Client, f.Namespace.Name, pod.Name, "portforwardtester") if err != nil { <del> framework.Failf("Error retrieving logs: %v", err) </del> <ins> framework.Failf("Error retrieving pod logs: %v", err) </ins> } <del> if waitErr != nil { framework.Logf("Pod log:n%s", logOutput) framework.Failf("Pod did not stop running: %v", waitErr) } By("Verifying logs") </del> verifyLogMessage(logOutput, "^Accepted client connection$") verifyLogMessage(logOutput, "^Received expected client data$") verifyLogMessage(logOutput, "^Done$")

https://k8s- Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
Two hunks (2x 10 "x"s) are not sent back. extends the logging. At least it will tell us whether the sender gets an error or not.
https://k8s- Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
https://k8s- Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
No more insights here. Pod log doesn't show anything special.
https://k8s- Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
https://k8s- Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite} Happened on a presubmit run in .
https://k8s- Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
https://k8s- Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite} Happened on a presubmit run in .
https://k8s- Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite} Happened on a presubmit run in .
I analysed this flake instance: https://k8s- It looks like the pfpod container terminates, the kubelet realizes that and wants to remove the network interface from the net ns using CNI via kubenet.

This fails two times (somehow TeardownNetwork is called multiple times, succeeds twice, but fails after that; compare about multiple TeardownNetwork calls). These failures interrupt the kubelet's tries to kill the pod. It succeeds 30 seconds later during reconciliation. But at that point the e2e test timed out already waiting for the pod to terminate.
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
This is the real problem, not another flake: This happens roughly once per week, pretty stable.
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
Builds: kubernetes-e2e-gke-staging-parallel Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
https://k8s- Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite} Happened on a presubmit run in .
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite} Happened on a presubmit run in .
[FLAKE-PING] This flaky-test issue would love to have more attention.

Builds: kubernetes-e2e-gce-etcd3 kubernetes-e2e-gci-gce-kubenet kubernetes-e2e-gci-gce-release-1.4 kubernetes-e2e-gci-gke-multizone kubernetes-e2e-gci-gke-prod-smoke kubernetes-e2e-gci-gke-staging-parallel kubernetes-e2e-gci-gke kubernetes-e2e-gke-staging-parallel kubernetes-e2e-gke Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
Builds: kubernetes-e2e-gci-gke-staging-parallel kubernetes-e2e-gke-staging-parallel Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
Builds: kubernetes-e2e-aws kubernetes-e2e-gci-gke-prod-smoke kubernetes-e2e-gke-staging-parallel kubernetes-e2e-gke Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite} Happened on a presubmit run in .
https://k8s- Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite} Happened on a presubmit run in .
[FLAKE-PING] This flaky-test issue would love to have more attention.

https://k8s- Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite} Happened on a presubmit run in .
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
Builds: kubernetes-e2e-gci-gke-release-1.4 kubernetes-e2e-gci-gke-staging-parallel kubernetes-e2e-gke Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite} Happened on a presubmit run in .
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
any updates on this issue
No, no further insights. Feel free to invest some time here with some new ideas.

[FLAKE-PING] This flaky-test issue would love to have more attention.
Builds: kubernetes-e2e-gce-multizone kubernetes-e2e-gci-gce-kubenet kubernetes-e2e-gci-gce-proto kubernetes-e2e-gci-gke-staging-parallel Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
Builds: kubernetes-e2e-gci-gce-proto kubernetes-e2e-gci-gke-release-1.4 kubernetes-e2e-gci-gke-staging-parallel kubernetes-e2e-gke-staging-parallel Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
Builds: kubernetes-e2e-aws kubernetes-e2e-gci-gce-kubenet Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.

This needs to be triaged as a release-blocker or not for 1.5
https://k8s- Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
Builds: ci-kubernetes-e2e-aws ci-kubernetes-e2e-gci-gce-kubenet Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
or Can you help us triage this as blocking or non-blocking for 1.5. It was originally marked as non-blocking for the 1.4 release.
The initial triage done by sttts@ is still valid even though we changed something in TeardownNetwork logic in 1.5 release. I don't think this should be release blocker with the same reason provided by sttts@ for 1.4 release. But instead of asking timestclair@ to take further look, I am assigning this to cc/
I marked this as release-blocker just for triage here. Thanks!
Thanks can you take a look and retriage if needed.
We have two flakes here: -- that's what and is about, i.e. Teardown logic. happened 3 times in last 8 days, much less the days before. IMO this is not a blocker. -- this seems to happen multiple times a day when a lot of e2e tests are running. This is the actual network flake. I will try to look into one of those to understand whether there is anything non-network related we can fix.
Here is a debug PR running the flaking test 100 times: None failed until now. Moreover, 200 runs locally do not fail.
Another observation: the number of returned is always a multiple of 10. The test container sleep for 100ms after each 10 . So, this might be a hint that there is some kind of race closing the connection.

Builds: ci-kubernetes-e2e-aws ci-kubernetes-e2e-gci-gke-multizone ci-kubernetes-e2e-gci-gke-prod-parallel ci-kubernetes-e2e-gci-gke-staging-parallel ci-kubernetes-e2e-gci-gke Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
Builds: ci-kubernetes-e2e-aws ci-kubernetes-e2e-gci-gce-release-1.4 ci-kubernetes-e2e-gke-staging-parallel Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
Builds: ci-kubernetes-e2e-aws ci-kubernetes-e2e-gce-release-1.5 ci-kubernetes-e2e-gci-gce-proto ci-kubernetes-e2e-gci-gce Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
Builds: ci-kubernetes-e2e-gce-etcd3 ci-kubernetes-e2e-gci-gce-etcd3 ci-kubernetes-e2e-gci-gce-kubenet Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
https://k8s- Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
Looks like this got a lot worse on 12/3 - https://k8s- Any chance this is related to
yes, it is most probably related to . In fact, was meant to make connection closing much less aggressive on the server side. It looks like our half-open connection support using socat is far from stable. I have in flight which is blocked by broken socat. Maybe that will help eventually to fix this flake.

Builds: ci-kubernetes-e2e-gce-etcd3 ci-kubernetes-e2e-gci-gce ci-kubernetes-soak-gke-test Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
Bumping to P0 - this is happening pretty much all the time now. yes, it is most probably related to . - can we revert that PR then? Or have a fix really soon, this is the biggest trigger of failures now.
we marked both tests as last night: This shouldn't happen anymore after rebasing PRs.
Here is a first attempt for a real fix:
https://k8s- Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
We will re-enable this test via This is also supposed to include a fix. Revert if the issue reappears.
https://k8s- Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
Is it appropriate to move this to the next milestone? (and remove the non-release-blocker tag as well)
Builds: ci-kubernetes-e2e-gci-gce-release-1.4 ci-kubernetes-e2e-gci-gce-release-1.5 Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
Builds: ci-kubernetes-e2e-gci-gke-release-1.4 ci-kubernetes-e2e-gci-gke-staging-parallel Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
Closing this issue as the corresponding test has been removed from cc:


Question: etcd-servers: http://<master-private-ip>:4001 metadata: "role=node" flannel: <del> interface: eth1 </del> etcd_endpoints: http://<master-private-ip>:4001 units: - name: etcd.service

Tried to run through this guide: On VMware, there is no eth1, so the flannel configuration dies. Once you get past that, there is also an issue where etcdctl doesn't properly update the flannel network settings because the etcd server isn't set to the master.


Question: "containers": [{ "name": "php-redis", "image": "brendanburns/php-redis", <ins> "cpu": 100, </ins> "memory": 10000000, "ports": [{"containerPort": 80, "hostPort": 8000}] }]

HPA tests are currently failing in gke-large cluster suite: I did some debugging and the problem seems to be related to pod-nanny. Looking into logs from the last run: was first scheduled on the large node dedicated to it pod-nany updated heapster deployment then heapster was scheduled on the normal node (when it actually shouldn't fit) that point it was never updated by the pod nanny. that point, heapster started crashlooping (OOMing) [which in fact was expected in too small machine] So it's pretty clear that that pod-nanny changed the amount of resources to something smaller, which in fact it shouldn't.


Question: // Handle ExternalTraiffc related updates. if utilfeature.DefaultFeatureGate.Enabled(features.ExternalTrafficLocalOnly) { <del> success, err := rs.healthCheckNodePortUpdate(oldService, service) </del> <ins> success, err := rs.healthCheckNodePortUpdate(oldService, service, nodePortOp) </ins> if !success || err != nil { return nil, false, err }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: healthcheck nodeport allocation doesn't have roll-back, this may lead to nodeport leak. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: /sig network /assign Environment: Kubernetes version (use ): Cloud provider or hardware configuration**: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig network /assign
Could you clarity "roll-back"? It seems that we basically in the same way.
I didn't mean the HealthCheck NodePort release process in . I meant the allocation process in and . Take for example, please see: There is a defer for common NodePorts in : Once fails, the above defer will finally exec. Then the allocated NodePorts will roll-back, i.e. these NodePorts will be released. But the HealthCheck NodePort allocation process doesn't use this roll-back. This will lead to leak. I run a simple test and see the HealthCheck NodePort leak. Will paste the result later.
Testing service manifest, . some simple testing code. See: . Note that I add this error intentionally: kube-apiserver with flag . . We will get below output. This is expected. again. We will get below output. Note the error . That means that when creating service fails at step 3, the NodePort and HealthCheck NodePort should be both released, but the was NOT.
Fortunately, several minute later, the leak was repaired and cleaned up. But anyway, we should release the HealthCheck NodePort when fails. We shouldn't depend on the later repair.
Thanks for tracing down, sent for fixing.
We're having this issue on service deletion. It looks like healthcheck nodeport is not deallocated immediately when deleting the service. This is causing issues for backup and restore of services with heathcheck node port.


Hello , I have saw the commit , can we check the for enqueue to serivce cotnroller work queue? basecause of, In many case for Public Cloud, We always abstract a set for health checks , such as TargetGroup in AWS, instanceGroup in aliyun. We can map to such a collection of instance. If the service controller to check the targetgroup changes, the traffic pass-through pod is easier to implement. In most cases, the instance that in same group has an associated port for external services, and service controller set a target port if which not set in service yaml file. On the other handWe only lose some other performance of service controller. and Implementing targetgroup mapping to targetport depends on the cloud provider. What do you think? /sig cloud-provider /sig network
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.


Question: suffix = ns.Name }) <del> It("should not work when mount is in progress", func() { </del> <ins> It("should not work when mount is in progress [Slow]", func() { </ins> driver := "attachable-with-long-mount" driverInstallAs := driver + "-" + suffix

created a spread sheet of the slowest running tests for the pull-kubernetes-gce job. cc discussion on slack: spreadsheet: pie chart of test durations from SIGs: ! it seems appropriate to evaluate the state of the tests and reduce these runtimes. possibly move tests to periodic by marking them as [Slow]. /assign /kind design /area test (?)
/remove-kind bug
Are pull jobs the right place to be calculating the time? Or should be be counting in CI instead?
pull is the one timing out, so probably?
Looking at some of the top slowness:: subPath should fail for new directories when readOnly specified in the volumeSource: This test case is no longer defined in master HostPathType: This test case has not been checked in yet and is trying to be in a PR PersistentVolumes-local StatefulSet with pod affinity should use volumes spread across nodes when pod has anti-affinity: This test is already marked as slow in master
Created This should address most of the top slow storage tests


Question: } func runPausePod(f *framework.Framework, conf pausePodConfig) *v1.Pod { <ins> return runPausePodWithTimeout(f, conf, framework.PollShortTimeout) } func runPausePodWithTimeout(f *framework.Framework, conf pausePodConfig, timeout time.Duration) *v1.Pod { </ins> pod := createPausePod(f, conf) <del> framework.ExpectNoError(e2epod.WaitTimeoutForPodRunningInNamespace(f.ClientSet, pod.Name, pod.Namespace, framework.PollShortTimeout)) </del> <ins> framework.ExpectNoError(e2epod.WaitTimeoutForPodRunningInNamespace(f.ClientSet, pod.Name, pod.Namespace, timeout)) </ins> pod, err := f.ClientSet.CoreV1().Pods(pod.Namespace).Get(context.TODO(), conf.Name, metav1.GetOptions{}) framework.ExpectNoError(err) return pod

Which jobs are flaking: sig-release-master-blocking Conformance - GCE - master Which test(s) are flaking: E2E: [sig-scheduling] SchedulerPreemption [Serial] validates basic preemption works [Conformance] Testgrid link: Reason for failure: Anything else we need to know: is currently unavailable links to specific failures in spyglass appreciated From TestGrid alert ...
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scheduling
I cannot spot any significant logic flaw in the test. A possible cause is the timeout is set too short (1 minute), which may be not enough for the terminating pod to run. Notice that unlike integration test, the time includes the end to end cycle: (1) preemptor being scheduled and evict low-priority pods, (2) low-priority pods being terminating -reported by kubelet that the victims have been deleted, (3) scheduler gets notifications from API server that the victims have been deleted, (4) preemptor gets scheduled, (5) kubelet reports the preemptor is running. /cc for more thoughts.
Maybe, some logs to / would help. Just making the timeout longer don't make sense.
Is addressing this flake?
I think this flake happens before the failure tries to fix:
/kind flake
That's indeed a lot of steps, and flakiness seems very low. I think increasing the timeout for preemption seems reasonable.
is this flake a must-fix in 1.20?


Question: } } } <del> func BenchmarkScheduling(b *testing.B) { framework.DeleteAllEtcdKeys() var m *master.Master s := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, req *http.Request) { m.Handler.ServeHTTP(w, req) })) defer s.Close() masterConfig := framework.NewIntegrationTestMasterConfig() m = master.New(masterConfig) c := client.NewOrDie(&client.Config{ Host:         s.URL, GroupVersion: testapi.Default.GroupVersion(), QPS:          5000.0, Burst:        5000, }) schedulerConfigFactory := factory.NewConfigFactory(c, nil) schedulerConfig, err := schedulerConfigFactory.Create() if err != nil { b.Fatalf("Couldn't create scheduler config: %v", err) } eventBroadcaster := record.NewBroadcaster() schedulerConfig.Recorder = eventBroadcaster.NewRecorder(api.EventSource{Component: "scheduler"}) eventBroadcaster.StartRecordingToSink(c.Events("")) scheduler.New(schedulerConfig).Run() defer close(schedulerConfig.StopEverything) makeNNodes(c, 1000) N := b.N b.ResetTimer() makeNPods(c, N) for { objs := schedulerConfigFactory.ScheduledPodLister.Store.List() if len(objs) >= N { fmt.Printf("%v pods scheduled.n", len(objs)) /* // To prove that this actually works: for _, o := range objs { fmt.Printf("%sn", o.(*api.Pod).Spec.NodeName) } */ break } time.Sleep(time.Millisecond) } b.StopTimer() } func makeNNodes(c client.Interface, N int) { baseNode := &api.Node{ ObjectMeta: api.ObjectMeta{ GenerateName: "scheduler-test-node-", }, Spec: api.NodeSpec{ ExternalID: "foobar", }, Status: api.NodeStatus{ Capacity: api.ResourceList{ api.ResourcePods:   *resource.NewQuantity(32, resource.DecimalSI), api.ResourceCPU:    resource.MustParse("4"), api.ResourceMemory: resource.MustParse("32Gi"), }, Phase: api.NodeRunning, Conditions: []api.NodeCondition{ {Type: api.NodeReady, Status: api.ConditionTrue}, }, }, } for i := 0; i < N; i++ { if _, err := c.Nodes().Create(baseNode); err != nil { panic("error creating node: " + err.Error()) } } } func makeNPods(c client.Interface, N int) { basePod := &api.Pod{ ObjectMeta: api.ObjectMeta{ GenerateName: "scheduler-test-pod-", }, Spec: api.PodSpec{ Containers: []api.Container{{ Name:  "pause", Image: "gcr.io/google_containers/pause:1.0", Resources: api.ResourceRequirements{ Limits: api.ResourceList{ api.ResourceCPU:    resource.MustParse("100m"), api.ResourceMemory: resource.MustParse("500Mi"), }, Requests: api.ResourceList{ api.ResourceCPU:    resource.MustParse("100m"), api.ResourceMemory: resource.MustParse("500Mi"), }, }, }}, }, } wg := sync.WaitGroup{} threads := 30 wg.Add(threads) remaining := make(chan int, N) go func() { for i := 0; i < N; i++ { remaining <- i } close(remaining) }() for i := 0; i < threads; i++ { go func() { defer wg.Done() for { _, ok := <-remaining if !ok { return } for { _, err := c.Pods("default").Create(basePod) if err == nil { break } } } }() } wg.Wait() } </del>

We have found that on 1k nodes cluster scheduling takes 60-200 ms and becomes the bottleneck. We want to have a standard way to reproduce scheduling latency metrics result and benchmark scheduler as simple and fast as possible. Only with that can we: 1. figure out underlying behaviors and potential bottlenecks quickly; 2. publish results in a standard way and prevent future performance degradation. We have the following goals: Goal 1: Save time on testing. More accurate scheduling latency. We mock up kubelets and only measure scheduling latency and do not care about the latency to start the pod. This is different from Kubemark -- scheduled pods won’t be run by mocked kubelets. We only need to set up the master components. Goal 2: Profiling runtime metrics and find out performance bottleneck. Write scheduler integration test. Take advantage of go test and profiling tools, like cpu-profiling, memory-profiling and block-profiling. This is very fine-grained metrics. Goal 3: Reproduce test result easily. We want to have a known place to do the performance related test for scheduler. Developers should just run one script to collect all the information they need. We have the following plans for writing scheduler tests: density test (by adding a new Go test) schedule 30k pods on 1000 (fake) nodes and 3k pods on 100 (fake) nodes print out scheduling rate every second let you learn the rate changes vs number of scheduled pods benchmark (by modifying the existing benchmark in integration or simply add a new one) make use of and report nanosecond/op. schedule b.N pods when the cluster has N nodes and P scheduled pods. Since it takes relatively long time to finish one round, b.N is small -- level of 10. We are new to k8s codebase. We also want to know where is the best place (directory) to commit in these tests? Any suggestions are welcomed!
/cc
This extends
So density already does this... the one option it looks like you may want is custom control , real or fake is irrelevant as the e2e's just need to point to an api-server. re: benchmarks, seems like we could simply add to the unit-tests.
After chatting with lets add this as an agenda item for the SIG meeting tomorrow.

+1 for adding benchmark (howeveer this should be integration test, not unit-test in my opinion)
cc The next SIG on scheduler meeting is on 12.07, isn't it? If there is one tomorrow, can I have the information please?
retract, +1 to . I was adding this to the list given the context, but it crosses over into both.
Slides from today's meeting:
Here are are the action item notes from the SIG meeting: Everyone agrees this is a good idea and long overdue, but the question is where would this live: New category of tests? (isolated/sandbox’d/component) The current testing structure allows to easily append. create new directory (component) + hack/script to execute Future "we could" gate on known good data, and fail performance regressions
/cc
/cc


Question: volumeManager volumemanager.VolumeManager // Cloud provider interface. <del> cloud                   cloudprovider.Interface </del> <ins> cloud cloudprovider.Interface // DEPRECATED: auto detecting cloud providers goes against the initiative // for out-of-tree cloud providers as we'll now depend on cAdvisor integrations // with cloud providers instead of in the core repo. // More details here: https://github.com/kubernetes/kubernetes/issues/50986 </ins> autoDetectCloudProvider bool // Indicates that the node initialization happens in an external cloud controller externalCloudProvider bool

/kind feature What happened: Continuing the discussion from , we should deprecate (or remove since its alpha) the option to set . See comment and for more details. related issues: related kubernetes-dev thread: !topic/kubernetes-dev/UAxwa2inbTA EDIT: Deprecation plan: [x] v1.8 - deprecate auto-detect cloud provider feature in kubelet [ ] v1.9 - set no cloud provider as the default in kubelet but still allow opt in for auto-detect [ ] v1.10 - completely remove the option to use
/sig cluster-lifecycle
/area cloudprovider
cc
/assign wlan0
cc - Wanted to ask you if you had any thoughts for/against this issue?
/assign andrewsykim As discussed in our meeting I'll be following up with sig-node to see if we can deprecate this as it's going against what we want for external cloud providers
FYI
Would have been nice to keep this issue open to track progress in the next releases
Indeed, should be kept open. Next task is now targeting v1.9
Thanks I edited the issue description to include the deprecation plan going forward. Open to changes on the deprecation plan if we think it's too aggressive.
Great, I think this is a good timeline. We need to make sure we communicate it very well though can you add this to the draft release notes in kubernetes/features please?
Looks like you beat me to it , gave it a review.
hehe, yes, it at the same time as everything else ;)
Can you create a PR for this soon to get this early into v1.9 please?
[MILESTONENOTIFIER] Milestone Issue Current <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Escalate to the issue owners; move out of the milestone after 1 attempt. : Adding tests, refactoring, fixing old bugs. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
i took the liberty to move this along (filed ). Hope that's ok.
Thanks appreciate it. I think there were some discussions in the mailing list thread about v1.9 being too soon.

I think it largely depends on how many users actually take advantage of the auto-detect feature. It's not many from what I've seen but we should follow up and be sure.
Any recent discussion about that? I haven't seen any activity recently but I might just be subject to email overflow. I hope we can get this into v1.9 cc
I didn't follow any discussion related to this topic, but thought this is right move to decouple cloudprovider components from the core system. If I remembered correctly, only cloudproviders for GCE, AWS and Azura depend on the feature, which can be easily override. Any concerns for v1.9?
yep, GCE/AWS/Azure -
Please look at and give your LGTM
Done. Just better document in 1.9 release notes. LGTM
Thank you!
Reopening for the removal of the option in v1.10
please move this to 1.10 milestone


Question: # See the OWNERS docs at https://go.k8s.io/owners approvers: <del> - sig-cluster-lifecycle-maintainers </del> <ins> - luxas - timothysc - fabriziopandini - neolit123 - rosti - ereslibre </ins> reviewers: <del> - sig-cluster-lifecycle </del> <ins> - luxas - timothysc - fabriziopandini - neolit123 - kad - detiber - dixudx - rosti - yagonobre - ereslibre - SataQiu - yastij </ins> labels: - sig/cluster-lifecycle

/assign Cannot find either of these aliases ref
thanks /sig cluster-lifecycle let me know if want me to PR a fix, yet i don't know what action to take. do you wish to keep these aliases?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/lifecycle frozen
/assign /lifecycle active


Question: if job, needsFlush, err = jm.flushUncountedAndRemoveFinalizers(ctx, job, podsToRemoveFinalizer, uidsWithFinalizer, &oldCounters, needsFlush); err != nil { return err } <del> if jm.enactJobFinished(job, finishedCond) { </del> <ins> jobFinished := jm.enactJobFinished(job, finishedCond) if jobFinished { </ins> needsFlush = true } if needsFlush { if _, err := jm.updateStatusHandler(ctx, job); err != nil { return fmt.Errorf("removing uncounted pods from status: %w", err) } <ins> if jobFinished { jm.recordJobFinished(job, finishedCond) } </ins> recordJobPodFinished(job, oldCounters) } return nil

Occasionally job completions are counted twice (or potentially multiple times?) in the metric. This happens for about 20% of jobs, but may depend on the Job specificity, which results in an overestimation of the number of finished jobs. The metric reflects the actual number of completed jobs. a kind cluster with prometheus monitoring stack (for example follow this: ). the number of completed jobs by inspecting the metric and delete a job multiple times in a loop Example job: Example repro script: . Check the number of completed jobs by inspecting the metric and the job Yes, in the kube-controller-logs we find the following log lines which are related and probably indicate repeated updated which results in double counting. <details</details<detailsReproducible with kind </details<details</details<details</details<details</details<details</details
/assign
FYI
/sig apps
/triage accepted


Question: return } <del> go wait.Until(c.sync, c.interval, stopCh) </del> <ins> // derive a context from the stopCh so we can cancel the poll loop ctx := wait.ContextForChannel(stopCh) // wait until first successfully sync // this blocks apiserver startup so poll with a short interval err := wait.PollUntilContextCancel(ctx, 100*time.Millisecond, true, func(ctx context.Context) (bool, error) { syncErr := c.sync() return syncErr == nil, nil }) if err != nil { klog.Infof("error initializing the default ServiceCIDR: %v", err) </ins> <del> select { case <-stopCh: case <-c.readyCh: </del> } <ins> // run the sync loop in the background with the defined interval go wait.Until(func() { err := c.sync() if err != nil { klog.Infof("error trying to sync the default ServiceCIDR: %v", err) } }, c.interval, stopCh) </ins> } <del> func (c *Controller) sync() { </del> <ins> func (c *Controller) sync() error { </ins> // check if the default ServiceCIDR already exist serviceCIDR, err := c.serviceCIDRLister.Get(DefaultServiceCIDRName) // if exists if err == nil { <del> c.setReady() </del> c.syncStatus(serviceCIDR) <del> return </del> <ins> return nil </ins> } // unknown error if !apierrors.IsNotFound(err) { <del> klog.Infof("error trying to obtain the default ServiceCIDR: %v", err) return </del> <ins> return err </ins> } // default ServiceCIDR does not exist

Verify Master - Master Blocking verify.openapi-spec Inconsistently since 2 November. First failure (shown in testgrid) API Server is not becoming ready during Relevant logs /sig-testing
cc for cidr controller
and may be related.
/sig auth network
/remove-sig auth Is almost certainly related
/assign it seems the servicecidr controller takes time to start and it blocks the apiserver readiness
/reopen Test run failed after related PR merger:
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
yeah /assign
/triage accepted
this is sig network can you pelase update the title?
/retitle [Flaking Test][sig-network] ci-kubernetes-verify-master verify-openapi-spec
This still flake in master blocking test grid.
use 60s timeout and here uses 30s timeout by default. It flakes. 10 lines of log cannot show other details. My local test with 15s timeout failed several times. 30s is enough for my local tests. Or we just enlarge the timeout for CI.
having both scripts the same timeout sounds reasonable I'm still worried why it keeps timing out, but for that we should first define a test that measure the apiserver startup and then decide what value is correct and enforce it.
/priority important-soon /milestone v1.30 backport to v1.29 may be needed.
Hello! Release Signal shadow here. This issue has not been updated for a long time, so I'd like to check what's the status. The code freeze is starting (less than a month from now), and while there is still plenty of time, we want to ensure that each PR has a chance to be merged on time. As this issue is tagged for 1.30, is it still planned for this release?
This flakes in Do we have some action items for v1.30?
I need to fix it, is a shame on me, sorry
Hello! Release Signal shadow here. This issue has not been updated for a long time, so we'd like to check the status. The code freeze is starting less than a week, and we want to ensure that each PR has a chance to be merged on time.

As this issue is tagged for 1.30, is it still planned for this release? Please let us know the status if possible.
yeah, I like to get it fixed, there is some race condition that makes some of the boostrap to deadlock , I still didn't find why
didn't forget but didn't have time to figure out the root cause, will try to get back to this end of the week
I got one failure locally of an apiserver that took much time The apiservers need the servicecidr controller to start to be healthy, and the controllers need to sync the informer, but checking one of the requests from the informer I can see this error logging error output: "k8sx00nfnx02v1x12x06Statusx12x96x01nx06nx00x12x00x1ax00x12aFailurex1a2Timeout: Too large resource version: 2, current: 1"aTimeoutCnx00x12x00x1ax00"7nx17ResourceVersionTooLargex12x1aToo large resource versionx1ax00(x012x000xf8x03x1ax00"x00" and this keep going for a while logging error output: "k8sx00nfnx02v1x12x06Statusx12x98x01nx06nx00x12x00x1ax00x12aFailurex1a4Timeout: Too large resource version: 203, current: 1"aTimeoutCnx00x12x00x1ax00"7nx17ResourceVersionTooLargex12x1aToo large resource versionx1ax00(x012x000xf8x03x1ax00"x00" do you know why the resource version is growing?
looks like the server returns that error in a few places, not sure which one is getting exercised: update-openapi-spec starts with AllAlpha=true in order to enable all features / APIs ... are there any alpha features around watch / resourceVersion that could cause this?
Didn't read carefully the issue, so just quickly sharing some thoughts: 1) streaming lists might potentially be related, but it needs to be explicitly requested by the client.

Reflector is the only place that can currently do that, but it requires explicitly set env-var: Is it possible that this value is set? if so, then streaming lists will result is calling etcd for current RV and will wait for watchache to be synced with it - if progress-notify-interval is large and manual progress notify is disabled, we could face things like that ^^ 2) that said, if Alpha features are enabled, then manual-progress notify should also be set: 3) is it possible that we're facing the bug that Marek found 2 days before release that effectively was the main reason we didn't try to promote consistnet read to beta? ^^
Based on the log from I would guess this is the 3002ms implies the 3 second timeout of waiting for revision Empty implies consistent list from cache implies non-recursive list Planned to fix it, but PR was blocked on
thanks a ton for digging into that to deflake this test, I'd set this in the update-openapi- file (since it doesn't impact the generated openapi) with a comment to remove once is resolved


Question: continue } <del> pluginErrors = append(pluginErrors, fmt.Errorf("error: unable to read directory %q in your PATH: %v", dir, err)) </del> <ins> errors = append(errors, fmt.Errorf("error: unable to read directory %q in your PATH: %v", dir, err)) </ins> continue }

<!-- Please only use this template for submitting enhancement requests --What would you like to be : I type <kbdTab</kbd<kbdTab</kbdI would like to see kubectl plugins in the list of suggested subcommands. For example, if I have installed , subcommand completions should suggest . the plugin itself also supports command-line completion, <kbdTab</kbd<kbdTab</kbdshould show suggestions from the plugin command completion. Why is this needed: Plugins fill in niches in the kubectl workflow, so that they are used rather seldomly. This means that plugin users need to memorize plugin names and their options over long times without using them. Therefore, plugin usefulness would be greatly enhanced by suggesting plugin sub-commands and their options.
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
In fact, already supports command completion. You can run for more information about it. If you are using bash shelljust run , and then it will work!
Hi , but I'm not talking about command completion in general. I'm talking about command completion for plugins. For example, the new plugin system allows to call a plugin at location as . It would be nice to get suggested/completed, when typing <kbdTab</kbd. Further, if plugin names are completed, so should be options for those plugins -- provided the plugin offers some form of command completion.
I try to implement completion for kubectl plugins!
Were you able to make progress on this? I have a working patch for the completion of the plugin name (part 1 of this issue). I can make a PR for it if it can help.
This is a great first step! I do think however, that the real value is auto-completion for the plugin options. I can roughly remember what plugins I have, but not what options they offer. That's the real value here, I think. This will probably need a contract for how plugins make completion available (e.g. by following the kubectl standard of ). WDYT?
I've been thinking about how to do the plugin options completion.

Based on the patch I have now, I believe what will be needed is to be able to ask the plugin what the possible completions can be (based on the user-specified parameters). So the plugin won't need to generate a bash or zsh completion script (which is difficult for the author of the plugin) but just output possible completions. This also has the benefit of being independent of which shell is used. One way is to do as you mention and have the plugin support as a way to ask the plugin what the completions would be. Although it won't behave in the same fashion as (as it won't generate a completion script, and is shell-independent), I like that option because it simply requires a plugin author to provide a different script called which the bash completion logic can call. Another way would be to ask the plugin itself to support a flag such as which would return the possible completions. I don't like this because if the plugin doesn't support this flag, we'd be calling it and who knows what it would do in that unexpected case. The use of makes it easy to check of presence or absence of completion support. So I suggest that the contract with plugins be that a plugin that supports completion would provide two scripts and that the bash completion script call to get the list of completions. Any thoughts?
Hi guys, I've already implemented completion for plugins (subcommands only, doesn't include options) as However it needs to be refactored more. You can try completion for plugins with the following steps: First, prepare the repository, and build a kubectl. Install kubectl-hello plugin for completion testing. Load the kubectl completion code. You can see completion for some plugins.
Cool! You've used the same technique I did. However, may I suggest re-using the code from the plugin package instead of copying it?
Well, you are absolutely right in saying that we can't just call a plugin with some flag to produce the completion script, because who knows what will happen. Therefore, option completion for plugins has to go the route via an external completion script, like However, I'm pretty sure that those completion files need to be bash/zsh scripts, because they need to process contextual information.

For example, say I have a plugin with sub-commands Then I want completion to work as follows: As you say, this will have to be maintained by the plugin authors. Can you take a typical cobra command completion script as basis and work out how kubectl-plugin-completion and this external script need to work together? You could even try to have kubectl as its own plugin, like and or so :grinning:
kubectl already has some dynamic completion code to allow to complete pod names for example, which actually calls kubectl itself. I think I can use the same technique for plugin parameter completion. So here's an example of what I'm thinking: Then when the user types the kubectl completion script will call . This call will return which will be given to the shell by the kubectl completion script. I haven't actually tested this example, so I could be wrong. I'll try implementing it to see if I'm missing something.
Thank you for supporting me ;) I add you as co-author to commit when I send a PR.
Thanks
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue.

<detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-lifecycle rotten
I've been working on auto-completion for the Helm project and I have a proposal with a PR that fully supports auto-completion for Helm plugins. I believe the fundamental aspects of the solution would apply just as well to kubectl. Some adaptation would be needed to adjust to how kubectl supports plugins, but I think it wouldn't be very hard.
Is there an ETA on this? If I see it correctly, this is currently preventing plugin authors to implement completions as the generated completion suggestions are based off the root command and not the subcommand aka plugin. I've been trying all day to make this work but failed and that's when I stumbled over this issue..
Several major issues I see here are: We don't want files like as proposed in previous comment. We had a requirement prior to kubectl 1.12 and it didn't look great at all. The simplicity of plugins is valuable to us. (Also, unlike Helm plugins, kubectl plugins don't live at a particular directory, or follow a directory structure.)Plugin completions most likely would be integrated into commands, which a lot of people put in their or file. We need to keep this command fast. So what if the command called a special command on each plugin, in parallel, to inquire if it has completion available (e.g.), with a short deadline, and cached the successful results (maybe keyed with on plugin binary’s stat data, or checksum). That way, cmd could also provide data about plugins, while periodically refreshing the cached completion scripts.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with .

Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
any updates on this one ? is it complete/done?
No update, still open..
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale
/lifecycle frozen
Sorry to revive this thread a bit but while working on another cli using the same cobra completion system by (implemented by i believe) I found out that it would not be too hard or too disruptive to have a small patch on the shell completion directly. For example with kubectl if we have a patch like this (using zsh but I am guessing the same technique will apply for bash or others) : the patch will check the first command () and the second command for example "krew" exist as and then use it directly for completion otherwise do the normal completion 'thingy'. the only downside is that the initial plugin would not show i.e: if you do you won't get krew as the plugin name, except if we add all plugin to the Root Command like the pull request from does. I don't think it would be too intensive since it is a zsh/bash builtin and in memory. I have made a small asciinema to demonstrate how it works I guess if we want this we may need to do it directly in cobra, do you think a patch like this could get accepted ?

(I had somehow difficulties to get PR reviewer and merged in cobra last time I tried)
Thanks for the suggestion. The idea is good but doesn't generalize perfectly. The command is available for programs that use Cobra but kubectl plugins can be something else (a script, a Python program, a Go program that doesn't use Cobra, etc). So we couldn't do something like this, unless we required every plugin to provide a command. I'm working on porting Helm's plugin completion solution to kubectl and should have something for the community to try soon. Plugins using Cobra will then be able to use your suggestion and get completion super easily.
That make sense I forgot about non go (and non cobra) plugins, thanks I look forward to see how the helm completion extracted will look like !
I've posted something for people to try out in
It would be nice if some plugin developers try out the solution proposed in and provide feedback. since uses Cobra, the solution is very easy. Here is how to try it out: First build kubectl with the PR. Then create your completion script like this: put this new on $PATH. And now test using the newly built (I build mine at ). For completion of flags (which I realize is the real value for the plugin), kubectl needs to first move to Cobra 1.3.
I would be interested in your opinion on the proposed solution. It will not work out-of-the-box with Krew, but I haven't found a way to do better with respect to Krew. You had mentioned this previously, and although it would be a nice solution, I am concerned it would cause problems to any plugin that ignores all arguments and runs no matter what. With such a plugin, calling a command would trigger the plugin to run and possibly perform undesired actions when completions are requested.
I'm sorry about slow responses. Finally had time to read it and parse these. You're right we shouldn't randomly call a plugin binary without knowing it supports completion. From : kubectl already has a mechanism to find all plugins on $PATH, so I agree that at minimum, kubectl<kbdTab</kbdcan and should show plugins installed on the system (even if the plugins themselves don't support shell completion) I think this is a good model.

Just like how kubectl scans $PATH for , it can scan to see if a plugin supports completion. (It's important to define if that binary takes additional positional args or --options) To make the (2) a reality, I support adding an additional field to the Krew plugin manifest. The shape of that would be up for a discussion (I imagine it might be (a) a separate binary that comes in the plugin archive, or (b) allow linking to the plugin binary itself and the plugin reads argv[0] and prints completion, which can be done with (a) anyway). I think it's doable. I'm not very familiar with how shell completion works in practice, when it's called, how lazy can it be loaded, what are performance implications etc, but if you'd like to drive this effort, I think we can discuss a design on Krew side as well. Please bring this up in a SIG CLI meeting to see what the folks think about your proposal.
brilliant! Thanks for taking the time to think about this. Your suggestion to adapt krew sounds great to me. I will continue with the PR on kubectl's side and if it gets accepted we can look at krew.
The as proposed in accepts the exact same arguments as the standard executable. The new completion executable will examine those arguments and will determine what the shell completion choices following them should be.
do you have an insight into whether a plugin developer would develop and ship a separate executable for the completion, or would they simply link the main plugin executable and see if the argv[0] is the kubectl_complete-*? I assume the answer largely also depends on whether the plugin was authored in Go or bash (basically the only two languages I've ever seen a plugin submission in out of 150+).
I think plugins will choose to ship a separate executable. I feel it keeps the two aspects nicely separate (the plugin functionality and the shell completion functionality). Also, it is very easy for a separate executable to call back the main plugin if the developer wishes to do so, so I find that solution the most flexible. For example, for plugins using the Cobra library, the script would call back the executable like this: BTW, I have looked into the 179 plugins in the krew index and I found that 50% (yes, half!)

of them used Cobra (some of them indirectly by calling back ). This means that shell completion should be very easy to implement for all these plugins.
I'm interested in this support and thanks for adding this feature. One question: in which version this is planned to be released? Looking forward to this.


Question: return nil } <del> func checkPermissions() error { if uid := os.Getuid(); uid != 0 { return fmt.Errorf("kubelet needs to run as uid `0`. It is being run as %d", uid) } // TODO: Check if kubelet is running in the `initial` user namespace. // http://man7.org/linux/man-pages/man7/user_namespaces.7.html return nil } </del> func setConfigz(cz *configz.Config, kc *kubeletconfiginternal.KubeletConfiguration) error { scheme, _, err := kubeletscheme.NewSchemeAndCodecs() if err != nil {

What happened: follow on to Need to comment and avoid checking UID on windows. requires some research into what we should be checking so that we can add appropriate TODO. What you expected to happen: Lets also check this at the call site, Anything else we need to know?: Environment: Kubernetes version (use ): master OS (e.g: ): windows
/assign
/assign
/good-first-issue
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig windows
Hi are you working on this issue already? If not I'd like to tackle this issue :)
Hi yeah I am already working on this :)
Started branch for this here
thanks
Hey guys is there anything I can do here?
If you want to test perks patch that would be great ?
That sounds great Where do I get started?
Hey ! is there anything I can help with? and how?
I dont think there is anything left to do anything more that you think we need to do to get this merged?
Tested this with containerd as well, I think its good to test
/triage accepted