Question: ChainPrerouting  Chain = "PREROUTING" ChainOutput      Chain = "OUTPUT" ChainInput       Chain = "INPUT" <ins> ChainForward     Chain = "FORWARD" </ins> ) const (

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): 1.5.1 Environment: Cloud provider or hardware configuration: 3x Ubuntu 16.04 nodes running in VMware vSphere OS (e.g. from /etc/os-release): Ubuntu 16.04 Kernel (e.g. ): 4.4.0-57-generic Install tools: kubeadm Others: CNI: Canal (flannel + calico) What happened: I originally opened to make Canal CNI work when the node's FORWARD chain default policy is set to DROP. However, I realized that the DROP policy was the root cause for why my other CNI tests also were failing. When either weave and flannel CNI is used and kube-proxy has --masquerade-all enabled, pod to pod and pod to service communication fails. Specifically, kubernetes-dashboard and tiller-deploy fail to reach the kube-apiserver and enter a crash loop. Also, the kube-dns pod also crashes. <details<summarykubectl get pods --all-namespaces -o wide</summary</details<details<summarykubectl logs --namespace=kube-system kubernetes-dashboard--5vzdq</summary</details<details<summarykubectl describe --namespace=kube-system pod kubernetes-dashboard--5vzdq</summary</details<details<summarykubectl describe --namespace=kube-system pod kube-dns--gz34l</summary</details<details<summarysudo iptables -t filter -v -L FORWARD --line-numbers (197 currently has the dashboard pod scheduled. Note the 2 dropped packets.

)</summary</details</brWhat you expected to happen: Either it should be better documented that the default FORWARD chain policy should be ACCEPT or rules should be to prevent packets from hitting the default policy. As a note, Ubuntu 16.04 defaults the FORWARD chain to DROP. I assume the is why the packets are going through the FORWARD chain in the first place but without , the dashboard still can not access the kube-apiserver. One option that I suggested in is to have kube-proxy add an iptables FORWARD rule to ACCEPT packets that have been marked by KUBE-MARK-MASQ. How to reproduce it (as minimally and precisely as possible): Create a cluster with: Weave or Flannel CNI enabled on all nodes kube-proxy set to kubernetes-dashboard enabled. kubernetes-dashboard will crash often. Anything else do we need to know: is closed because I thought the issue was specific to Canal and instead opened However, this looks to be a larger issue that kubernetes itself should address.
Weave issue:
I think this is going to hit a lot of people as Docker 1.13 is now setting the FORWARD chain to drop by default if ip_forward is set. See
See also
overlap with ?
I think a solution for this issue or (which is closer to what I feel like a proper solution is) should be considered for inclusion for K8s 1.8 since it will officially support docker 1.13. The problem as I see it is that kube-proxy sets up rules to forward traffic for services (specifically I have seen the problem with NodePorts) yet does not ensure or have any requirement (to my knowledge) that the forwarded traffic be allowed when the traffic is forwarded to a host other than the one receiving the traffic. I observed this when using K8s v1.7.5 installed with kubeadm v1.7.5 and Calico v2.5. It can be remedied with the addition of but hopefully there is a better answer than doing that manually and hopefully something more specific to what is needed. I saw the comment on the Docker 1.13 validation issue linking to the configuration script used in the GCE docker images which is why the e2es are not having any issues.
If we can be more precise about which traffic we require to ALLOW, I'm OK with adding rules to open that.

I've been doing some testing with the first two options and chatted with and about these potential options for adding some forwarding rules. They all are a certainly lot more specific than the previous blanket all ALLOW. any traffic sourced or destined for the clusterCIDR. traffic that has the KUBE-MARK-MASQ set and conntrack rules to accept the follow on traffic. a third option of adding a more specific mark that is only set on specific traffic that it is known will need to be forwarded and match on that (this option would probably need the conntrack rules like the above option too) do any of those options appeal to you or do you see any problems with them?
If we add some more rules, we shouldn't put them in the top-level chain since it makes them hard to manage/clean up. Best to add them to a chain that kube-proxy owns that is jumped to from the top-level chain.
doesn't necessarily know the ClusterCIDR. I could see your proposal making sense in conjunction with .
Very good point I should have mentioned that the options I presented are assuming clusterCIDR is set. If clusterCIDR is not defined then the first option is not possible (except for the wide open option that these rules are attempting to avoid). I guess on options 2 & 3 if it was not set then a conntrack rule could be without the clusterCIDR. I would appreciate feedback on the desired behavior if clusterCIDR is not set too.
The PR that was merged solved the issue for pods inside clusterCIDR. But it doesn't handle hostNetwork pods that use the node IP, like when you want to expose the API server as a load-balanced service externally. I'm not sure why the “kubernetes forwarding conntrack pod source rule” has to be limited to the pod CIDR, since --ctstate RELATED,ESTABLISHED should already ensure it does not allow traffic that wasn't already allowed by another rule.
I'd argue that any iptables/firewalling for a pod using isn't the responsibility of Kubernetes, There's no reasonable way for k8s to actually know what interfaces a host-network container binds to, what ports it listens on, or what expectations it has. The cluster operator should probably be responsible for that networking case.
I meant explicitly regarding services where those pods are the endpoints of a service, so Kubernetes does know the address and the port.

Due to limitation of state RELATED,ESTABLISHED in FORWARD rule i.e. current implementation, the unidirectional UDP traffic apps won't work. Because packets (except first) aren't ESTABILISHED from iptables's perspective (no reply from endpoint, no handshake). Shouldn't this be something like and not
To add to comment, bidirectional UDP apps are brittle as well. Dropping or failing to reply to the first packet will prevent the app from ever receiving a second. More discussion on


Question: BASEIMAGE=debian:stretch-slim TEMP_DIR:=$(shell mktemp -d -t conformanceXXXXXX) <ins> #output format for docker buildx build [push, load] OUTPUT=--load </ins> all: build build:

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: produces an error: What you expected to happen: Conformance tests should be launched. Examining image contents indicates executables are of the correct architecture. How to reproduce it (as minimally and precisely as possible): From aarch64 machine: Anything else we need to know?: Works with v1.17.4: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: OS (e.g: ):Kernel (e.g. ):Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig Architecture /sig Testing
/sig release /remove-sig architecture
any thoughts on this issue at all? We use this to test K3d and are blocked on releasing anything in the 1.18 line because of it.
I don't have expertise in the image release pipeline, I'd suggest reaching out to
Hi Did you run this test in an arm64 virtual machine? If so, I have fixed this issue in cadvisor side. This issue is caused by cadvisor. Currently, the version of cadvisor in kubernetes is v0.36.1-0.-. There is a defect in this version, it gets the wrong machine info and reports num_cores=0 on Arm64 guest. I have delivered the PR into cadvisor in 22 days ago. And this patch was merged in 10 days ago. Please see my pr as reference: Currently, I'm waiting a new tag of cadvisor, so that I can use this new tag in kubernetes and fix this issue in k8s side. Please see my issue: Thanks.
Currently we only have one Arm64 conformance test is alive, this conformance test was run on 2 AWS arm64 instances https://k8s- All other VM-based Arm64 conformance tests are down.
/area conformance
Thanks for taking a look just to be sure it is not a VM issue I fired up an a1-metal arm64 aws instance.

Please note I am using sonobuoy for running tests rather than kubetest, I tried to find if kubetest is using the conformance image for testing and from a quick check maybe not. The error is not in the conformance test itself, but launching the conformance test image on arm64. The is . Tried again with with 1.18.2 and got the same error: still works ok for the v1.17 branch: It looks like the build chain for images has changed: So there are now six less layers of the conformance image for the 1.18 branch. Please also note how the arm64 images show amd64 as the architecture for both branches. This has been problematic for trying to air-gap arm64 properly and is a systemic issue for many of the projects on gcr (and elsewhere).
Hi I see. I'll check it. Thanks.
/reopen This issue is still unfixed as of today. here is a reproducer that helps doing the reverse test on a x8664 machine: So this is an image where the base os is actually arm64. starting with v1.18.0 and still as of today (v1.18.5-rc.0 and v1.19.0-beta.2) the uploaded -arm64 image is actually an amd64 image. There must be a typo somewhere. testable this way: This is supposed to generate the 'exec file error', but it doesn't, which indicates that it is actually an x8664 image (amd64).
You can't reopen an issue/PR unless you authored it or you are a collaborator. <detailsIn response to : /reopen x8664 # podman pull # podman run 66652b9ed7f16a9bc848038ddb00159f3e2d22d8c2bb1ba050913309daf101c9 cat /etc/os-release exec user process caused "exec format error" Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
IIRC the fix was reverted due to not working in the release pipeline. I did say that would hurt :upsidedownface:
/reopen
Reopened this issue.

<detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten This is still blocking conformance testing on arm64, so I hope it can be considered open.
Looks like to me this is a very complicated issue which is related with GCB, Moby Buildkit, it's not easy for me to reproduce and some trouble shooting by myself since I am not able to access GCB. Just wanna know is there any follow up to make this PR merge again? , we do have some needs to run the conformance testing on ARM64 .
SIG testing does not use this image (we prefer not to run the test runner as a pod given our clusters may be unreliable since we're testing changes and we want to avoid e.g. the pod being evicted =data loss), however this is currently used by the CNCF conformance program (which is CNCF / SIG arch) and sonobuoy (which is a vmware project). A bunch of our images are shipping with buildkit / buildx currently at HEAD, something like number could probably be brought back if someone wanted to work on this. Currently I'm not going to have time myself.
/cc
cc for arm64 image :(
xref /assign
It looks like that the issue is not in k/release, since we only build/stage the images via . Then we push the images and annotate them to their corresponding architecture.

Looking at the staged artifacts, for example for v1.20.1 arm64: Then we can see that also kube-proxy seems to have this issue: I assume that the fix is somewhere in the build scripts, where I'm now looking into. Beside that, all other architectures (arm, ppc64le, ...) are affected by this bug as well (for the same images only). Edit: Looks like the distroless images are fine, whereas the debian based images (proxy / conformance) are pointing to the wrong architecture. Conformance seems to always use , which might be the issue here. Edit: Done with my experiments. will change the already open PR according to our findings from the slack conversation (), whereas we have to change the used base images for kube-proxy to match the corresponding target architecture (like using for )


Question: return -1, errors.New("compressing error: tried to write data using closed compressor") } c.Header().Set(headerContentEncoding, c.encoding) <ins> defer c.compressor.Flush() </ins> return c.compressor.Write(p) }

/kind bug is not working on Kubernetes 1.8. It gets stuck and only returns when the container exits. I tested it in GKE 1.6.11 and 1.7.8 and it works there. But doesn't work on GKE 1.8.0. Here's a simple reproduction: a 1.8.0 or 1.8.1 cluster. a simple busybox container and . another terminal, try without . It works. now. It gets stuck. more commands in the busybox terminal but there are still no logs in the terminal. another terminal try without again. That still works. Exit the busybox container: would show up on the terminal now. /sig cli cc -
/cc
Please file under kubernetes/kubectl
It seems this is not a kubectl bug, see echo the comment here: /reopen
/unassign /sig node api-machinery
Also note , this only happens in () that turn on all alpha features.
to make sure this isn't something funky with CRI log streaming on the node side do either of you guys know who's responsible for log streaming through API server?
AFAIK, we did not change CRI log streaming in 1.8. Given that this only fails in Alpha clusters, it'd be easier to vet the Alpha features instead.
Adding sig-gcp since only this has been observed only in GKE alpha clusters so far. If anyone from is willing to help out by narrowing down what Alpha features are related, that'd be great.
I turned off in the apiserver in a GKE cluster and the log function is working again. /cc
Ubuntu 16.0.4.5 Kubernetes v1.11.3 Agree with ! If ---"kubectl -f" dose not work
Would you mind sharing how you did this? This now seems to be happening on non-alpha clusters there. The feature gate's alpha status in the Kubernetes docs hasn't changed though.
Reiterating the mentions to trigger a notification: <detailsIn response to : I turned off in the apiserver in a GKE cluster and the log function is working again. /cc Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.

</details
This is still an issue in Alpha Clusters. I encountered this while deploying a GKE 1.13.5-gke.10 alpha cluster. Shouldn't this issue be re-opened? /cc


Question: cache.mu.Lock() defer cache.mu.Unlock() if _, ok := cache.podStates[key]; ok { <del> return fmt.Errorf("pod %v state wasn't initial but get assumed", key) </del> <ins> return fmt.Errorf("pod %v is not in the cache, so can't be assumed", key) </ins> } cache.addPod(pod)

/kind bug There are several error messages in the scheduler cache that are confusing to read, at least one of these surfaced up to end users via Events. I noticed these two in particular but there may be others: Whether or not these should actually be surfacing as events, the messages should be cleaned up.
cc
Seems that these are internal or implement details in scheduler, do our end users can fully understand what the events really mean? I think it is depend on our own implement. idea?
I believe, the problem is with in the message but I am not able to reproduce it, can you please share steps to reproduce this.
No unfortunately, one of our users reported seeing it when there system was in a bad state, not in a reproducible way. The point of the issue is just that the grammar in those messages is bad, which should be fixable even if you can't get them to fire.


Question: visitErr      error ) switch obj := info.Object.(type) { <ins> case *corev1.Node: debugPod, containerName, visitErr = o.visitNode(ctx, obj) </ins> case *corev1.Pod: debugPod, containerName, visitErr = o.visitPod(ctx, obj) default:

This issue tracks implementing node debugging with privileged containers for , as described in the This feature allows administrators to start a privileged container on a node in host namespaces rather than having to SSH to the node. For example: /kind feature /sig cli
/assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: ) klog.V(4).Infof("Detected local IP address as %q", ipnet.IP.String()) } else { <del> klog.Warningf("Failed to patch IP as MAC address %q does not belong to a VMware platform", vmMACAddr) </del> <ins> klog.V(4).Infof("Failed to patch IP for interface %q as MAC address %q does not belong to a VMware platform", i.Name, vmMACAddr) </ins> } } }

On Kubernetes v1.13.4 nodes with the vSphere Cloud Provider I see a lot of log spam (15k messages in one hour, with many duplicate messages logged just microseconds apart) like this one: This MAC address belongs to the interface which is (afaik) created by kube-proxy running in IPVS mode. I see the same message for the MAC address belonging to the interface created by Flannel. I see a few issues here: vSphere Cloud Provider tries to detect IP addresses for kube-proxy and CNI network interfaces; the MAC address does not have a known VMware prefix this gets logged as a warning instead of a debug message; process is apparently retried thousands of times per hour and in quick succession.
/sig vmware
/sig network These SIGs are my best guesses for this issue. Please comment if I am incorrect about one. I am an (alpha) bot run by
/remove-sig network
We are experiencing the same even though we fixed the ip address using flag on the kubelet.
Hello, when updating from v1.12.5 to v1.12.6, the following message appears: This MAC corresponds to the Docker NIC: This error does not appear in version v1.12.5
Hi, Same issue here. Upgrading our cluster to v1.12.6 we have this message. In previous version (v.1.12.5) everything was fine
This is the relevant PR that was backported to v1.11.7, v1.12.6 and v1.13.4:
/assign
/area cloudprovider
Opened a PR to fix this:
I've created cherry pick PRs to fix this in v1.11, v1.12 and v1.13 as well: v1.11: v1.12: v1.13:


Question: docker build -t gcr.io/google_containers/spark-driver driver docker tag gcr.io/google_containers/spark-driver gcr.io/google_containers/spark-driver:$(TAG) <del> zeppelin: </del> <ins> zeppelin-build: docker build -t gcr.io/google_containers/zeppelin-build zeppelin-build docker tag -f gcr.io/google_containers/zeppelin-build gcr.io/google_containers/zeppelin-build:$(ZEPPELIN_TAG) zeppelin: zeppelin-build docker create --name=zeppelin-build-tmp gcr.io/google_containers/zeppelin-build:$(ZEPPELIN_TAG) docker cp zeppelin-build-tmp:/zeppelin.tgz zeppelin docker rm -f zeppelin-build-tmp </ins> docker build -t gcr.io/google_containers/zeppelin zeppelin docker tag -f gcr.io/google_containers/zeppelin gcr.io/google_containers/zeppelin:$(ZEPPELIN_TAG)

The image by is over a gig. Investigate how to trim this down. (Cut out interpretters? Make them more modular somehow, possibly composing them once we have some solution like ?) It's not even clear there's upstream support for making this more modular, though.
cc
remove mvn tooling and all the extra steps, once upstream releases 's like the rest of apache.
I can look at that, but I think I can actually just strip the image of the dev tooling right now. Right now I think the Zeppelin deps themselves are a ridiculous portion of this, though. (I'll poke at it.)
(3) is an install of [bzr git mercurial openssh-client subversion] in the base java image, which seems wasteful to me. you'll have to switch base images to get rid of this. (2) is your install of [curl net-tools build-essential git wget unzip python python-setuptools python-dev python-numpy libfontconfig]. i expect curl is at least redundant, as is git if your using the java image. you should merge this layer w/ (1) and do an uninstall of [python-setuptools python-dev etc] & deps at the end. of course, those are less significant than (1), which is the mvn build. the long term solution here is to use the upstream binaries, when they exist. in the meantime, you should clone zeppelin into tmp, copy out the resulting build artifact and delete the clone. that should at least cut the size of this layer in half, as all the intermediate build artifacts are likely still present and in the final tarball/jar.
I can trim it down some by cutting out the build machinery (and I'll put up a PR shortly), but here's what the looks like: And expanded: (Instead of just deleting, it's actually easy to build, copy out, compose, too.)
Is it me or are all the containers rather large. For example, does spark itself need to be a gig? 1.5.1v3 6 weeks ago 989.7 MB There's some spark binary package that's 73M uncompressed:
Java runtime + baseimage + transitive closer is what is causing most of the bloat. But as we have discussed on other tickets, these images need to move out of the main repository.

I'm afraid I don't understand "transitive closer". For example I see my JDK is around 280M, and the Spark binary is 73M. Then that leaves another 636M for base + whatever. I guess I'll have to look at the imagelayers if I'm really curious.
he means all the dependencies and their dependencies. if you'd like to tackle shrinking the spark image, please open an issue for it. we've done some work on this and you can use the example investigation above to help improve image sizes.


Question: SourceResourceID: &sourceResourceID, }, nil } <ins> func isInstanceNotFoundError(err error) bool { errMsg := strings.ToLower(err.Error()) return strings.Contains(errMsg, errStatusCode400) && strings.Contains(errMsg, errInvalidParameter) && strings.Contains(errMsg, errTargetInstanceIds) } </ins>

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Same symtoms as The detach is not successful until controller-manager is restarted controller-manager logs: What you expected to happen: The detach operation should know that a node is not active or its been deleted and should assume the disk is detached. How to reproduce it (as minimally and precisely as possible): This seems quite hard to reproduce, but we do run into it quite a few times where the pod is stuck in , because the controller-manager is unable to detach disks from deleted nodes. Anything else we need to know?: Environment: Kubernetes version (use ): v1.15.11 Cloud provider or hardware configuration: azure (self-managed) OS (e.g: ): Ubuntu 18.04.3 LTS Kernel (e.g. ): 5.0.0-1027-azure Install tools: kubeadm Network plugin and version (if this is a network-related bug): Others: /sig cloud-provider /area provider/azure paging
/sig azure
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig azure Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cloud-provider
/area provider/azure
hi have you done sth manually on the node? if node is deleted finally, the disk would be in unattached state finally, then disk attach to another node would succeed finally.
as discussed offline, the issue is probably due to vmss instance compute name change(incorrect format: , correct format: ), user should not change vmss instance compute name, otherwise it would break disk attach/detach, since compute name is used in vmss node name passing, code is here:
Hi We don't change the instance compute name - however what i do see in the azure metadata service is in the form - What you're referring to seems to be the computer name, and we do use the defaults as provided by azure. See The attach/detach operations works fairly well most of the time, and I think we're running into some edge case here.

One thing i might have missed out is the fact that there seems to have been an issue with the VM at the time. So the VM was present, but azure platform would return 400 bad request for a detach operation which probably led to this .. But when the VM was eventually deleted by the autoscaler, the re-attach was never attempted by the reconciler it would seem. and the last message about that disk was
should be a valid compute name according to , do you know why we define the format like this?
The function is used to get the name from IMDS, and its format is .
From the above errors, seems the request has been sent to VMSS but failed because it's not active. Do you know the VM state during that time?
I am not sure of the state at the time, but i believe the VM was in some kind of failed state. The VM has since been deleted, but we had errors on the scaleset at the time. The events on the VMSS look like this. The VM was eventually deleted by the autoscaler, but the disk was never re-attached to a different node, and the pod remained in for about 2 days until controller-manager was restarted.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Apparently we hit this issue today as a result of replacing the nodepool. We 2 new nodepools and deleted the old one, and now we have 718 of 1672 pods stuck with this error in one of our clusters. The volumes are failing to be attached to new pods, the VMSS for old nodepool is removed. In AKS portal disks are still reported to be attached to VM instances in the VMSS that was deleted.

hundreds of pods are stuck in Init state won't likely go forward. /remove-lifecycle rotten
/remove-lifecycle rotten
I have seen similar issue when handling one customer's case, it's probably caused by broken vmss cluster, pls file a ticket to VMSS team to fix the issue.
I happened to find this issue in one customer case, this issue is caused by obsolete cache in kube-controller-manager, should skip attach/detach retry when it's 400, below are the details, following two instances were deleted 3min ago before kube-controller-manager trying to attach/detach: workaround restart kube-controller-manager should work
/assign


Question: func LoadPolicyFromBytes(policyDef []byte) (*auditinternal.Policy, error) { policy := &auditinternal.Policy{} <del> decoder := audit.Codecs.UniversalDecoder(apiGroupVersions...) </del> <ins> strictDecoder := serializer.NewCodecFactory(audit.Scheme, serializer.EnableStrict).UniversalDecoder() </ins> <del> _, gvk, err := decoder.Decode(policyDef, nil, policy) </del> <ins> // Try strict decoding first. _, gvk, err := strictDecoder.Decode(policyDef, nil, policy) </ins> if err != nil { <del> return nil, fmt.Errorf("failed decoding: %v", err) </del> <ins> if !runtime.IsStrictDecodingError(err) { return nil, fmt.Errorf("failed decoding: %w", err) } var ( lenientDecoder = audit.Codecs.UniversalDecoder(apiGroupVersions...) lenientErr     error ) _, gvk, lenientErr = lenientDecoder.Decode(policyDef, nil, policy) if lenientErr != nil { return nil, fmt.Errorf("failed lenient decoding: %w", lenientErr) } klog.Warningf("Audit policy contains errors, falling back to lenient decoding: %v", err) </ins> } // Ensure the policy file contained an apiVersion and kind.

Currently the [audit policy decoder][1] silently drops unknown fields. For backwards compatibility, we probably cannot fail on unknown fields, but it would be good to at least log a warning when they're encountered. [1]: /sig api-machinery
/triage accepted
/help
This request has been marked as needing help from a contributor. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I'm not familiar with apiserver, but this seems like a simple one. /assign
How can we find unknown fields in the yaml file? Currently I'm think of converting both the yaml file and the decoded Policy struct to two map[string]interface{}, and compare both map to find out which keys are in the yaml file but not in the decoded struct, is there any better solution?
Courtesy on slack:


Question: } fileName := fmt.Sprintf("%s_udp@%s", fileNamePrefix, name) fileNames = append(fileNames, fileName) <del> probeCmd += fmt.Sprintf(`test -n "$$(dig +notcp +noall +answer +search %s %s)" && echo OK > /results/%s;`, name, lookup, fileName) </del> <ins> probeCmd += fmt.Sprintf(`check="$$(dig +notcp +noall +answer +search %s %s)" && test -n "$$check" && echo OK > /results/%s;`, name, lookup, fileName) </ins> fileName = fmt.Sprintf("%s_tcp@%s", fileNamePrefix, name) fileNames = append(fileNames, fileName) <del> probeCmd += fmt.Sprintf(`test -n "$$(dig +tcp +noall +answer +search %s %s)" && echo OK > /results/%s;`, name, lookup, fileName) </del> <ins> probeCmd += fmt.Sprintf(`check="$$(dig +tcp +noall +answer +search %s %s)" && test -n "$$check" && echo OK > /results/%s;`, name, lookup, fileName) </ins> } for _, name := range hostEntries {

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug /sig testing What happened: Tests checking DNS responses can have false positives, such as "should provide DNS for services". The tests rely on dig-ing certain domain names [1]: test -n "$(dig +notcp +noall +answer +search kubernetes.default A)" && echo OK /results/ The problems could be that, if a host / container doesn't have any entries in their : ubuntu dig +notcp +noall +answer +search kubernetes.default A ;; connection timed out; no servers could be reached , causing the test to pass and to write "OK" into the results file. [1] What you expected to happen: The tests shouldn't be able to pass if dig failed. How to reproduce it (as minimally and precisely as possible): Comment all nameservers or configure a bogus DNS server in, and run: test -n "$(dig +notcp +noall +answer +search kubernetes.default A)" && echo OK Anything else we need to know?: Environment: Kubernetes version (use ): 1.9.6 Cloud provider or hardware configuration: N/A OS (e.g. from /etc/os-release): "Ubuntu 16.04.3 LTS" Kernel (e.g. ): Linux ubuntu 4.4.0-87-generic
/assign
: But one of the purpose of the test is to ensure that the DNS of the cluster is replying. In other words, if it does not reply then we EXPECT this test to fail. If you are not using DNS then isn't it better to SKIP the test ?
I believe means to say "false negatives" ... meaning that in some situations, the test will errantly pass, when it should fail.
oops . Looks like I misread the description, and agree that the dig should work.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale


Question: return p.admitResourceSlice(nodeName, a) case csrResource: <del> if p.kubeletCSRAdmissionValidationDisabled { </del> <ins> if p.allowInsecureKubeletCertificateSigningRequests { </ins> return nil } return p.admitCSR(nodeName, a)

This comes from We've a feature gate, . However, we try to name feature gates after the positive thing they add, even if it's on by default, or even if we're setting if false because the positive thing is being dropped (rare, but it happens). Typically we don't start feature gate names with , as it makes the docs jar: Example quote 1: To temporarily revert to the legacy behavior, enable the feature gate. vs Example quote 2: You can enable the feature gate if you run code that relies on the previous behavior. You should plan to migrate. and because disabling things isn't really a feature. Some examples of gates you can or once could disable by turning them off_ include: - - - So, I recommend a rename of the gate before the v1.31 release. /sig auth architecture
FYI
Example gate names: (off by default) (off by default) (on by default) (on by default) (on by default) from (off by default, deprecated)
/milestone v1.31
You must be a member of the GitHub team to set the milestone. If you believe you should be able to issue the /milestone command, please contact your Milestone Maintainers Team and have them propose you as an additional delegate for this responsibility. <detailsIn response to : /milestone v1.31 Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Another possibility: , deprecated, off by default
Hello there! I am new to kubernetes and I want to contribute in it. Is it something that I can contribute to?
This issue is not a good issue for a first-time contributor to work on.
I'm a fan of , deprecated, off by default
hoping one of you are doing the PR? :) /milestone v1.31


Question: // add a new dummy listener in order to get a port that is free l, _ := net.Listen("tcp", ":0") <del> _, port, _ = net.SplitHostPort(l.Addr().String()) portNum, _ := strconv.Atoi(port) </del> <ins> _, newPort, _ := net.SplitHostPort(l.Addr().String()) portNum, _ := strconv.Atoi(newPort) </ins> l.Close() // Wait for the socket to actually get free. <del> if err := waitForClosedPort(p, port); err != nil { </del> <ins> if err := waitForClosedPort(p, newPort); err != nil { </ins> t.Fatalf(err.Error()) } <ins> if proxyPort == newPort { t.Errorf("expected difference, got %s %s", newPort, proxyPort) } </ins> p.OnUpdate([]api.Service{ {JSONBase: api.JSONBase{ID: "echo"}, Port: portNum}, }) if err := waitForClosedPort(p, proxyPort); err != nil { t.Fatalf(err.Error()) } <del> testEchoConnection(t, "127.0.0.1", port) </del> <ins> testEchoConnection(t, "127.0.0.1", newPort) </ins> }

https://travis-
Looking.
Another instance, or a different thing? https://travis-


Question: fi fi <ins> local address="" if [[ ${GCE_PRIVATE_CLUSTER:-} == "true" ]]; then address="no-address" fi </ins> local network=$(make-gcloud-network-argument  "${NETWORK_PROJECT}"  "${REGION}"  "${NETWORK}"  "${SUBNETWORK:-}"  <del> ""  </del> <ins> "${address}"  </ins> "${ENABLE_IP_ALIASES:-}"  "${IP_ALIAS_SIZE:-}")

Public IPs are expensive resources. Given that we're running scalability tests on 5k-node clusters, we are unnecessarily consuming 5k IPs which is a lot. We should switch our tests so that nodes have only private IP addresses. We are going to do that by configuring NAT so that nodes can talk to outside world (e.g. to download image from dockerhub). But that also requires changing some correctness tests which assume public IPs.
To summarize recent work: setting up private gce clusters with nat, feature is gated via the env test and dashboard with correctness tests on gce private cluster - https://k8s- 27 tests fail on fixing the tests - should fix majority of them
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale /lifecycle frozen
- is there anything missing to be done here?
No, we can close this one. It was done long time ago. Thanks for catching /close
Closing this issue. <detailsIn response to : No, we can close this one. It was done long time ago. Thanks for catching Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: containers: # TODO: Add resources in 1.8 - name: event-exporter <del> image: gcr.io/google-containers/event-exporter:v0.1.0 </del> <ins> image: gcr.io/google-containers/event-exporter:v0.1.0-r2 </ins> command: - '/event-exporter' - name: prometheus-to-sd-exporter <del> image: gcr.io/google-containers/prometheus-to-sd:v0.1.2 </del> <ins> image: gcr.io/google-containers/prometheus-to-sd:v0.1.2-r2 </ins> command: - /monitor - --component=event_exporter

A number of addon images have CVEs that have been fixed in the upstream base images. We should update these images before the final 1.7 release. cc
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
/cc
cc
The new image appears to have removed . I'm not sure if this was expected, but it caused at least fluentd to stop working on COS (). This might also affect kubeproxy, which uses (a derivative of ).
also the metadata-proxy, prometheus-to-sd, the fluentd-event-exporter, and ip-masq-agent.
anything more to do for this issue?
yes, I need to get merged and update those images again. I also need to bump a few other images (dns and ip-masq-agent).
should be the last update needed.


Question: }, } <del> // Columns: Name, Completions, Duration, Age </del> <ins> // Columns: Name, Status, Completions, Duration, Age </ins> expectedRows := []metav1.TableRow{ <del> {Cells: []interface{}{"job1", "1/2", "", "0s"}}, {Cells: []interface{}{"job2", "2/2", "20m", "0s"}}, </del> <ins> {Cells: []interface{}{"job1", "Running", "1/2", "", "0s"}}, {Cells: []interface{}{"job2", "Running", "2/2", "20m", "0s"}}, </ins> } rows, err := printJobList(&jobList, printers.GenerateOptions{})

Suspend is a GA feature and it is big component of Kueue. It would be nice to display if jobs are suspended when doing . A new column that displays if the job is suspended would be helpful. Improve UX around suspend. Its a bit difficult to know why the job has not created pods unless you know about the suspend field.
/sig cli /wg batch /good-first-issue /help-wanted
This request has been marked as suitable for new contributors. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /sig cli /wg batch /good-first-issue /help-wanted Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc Just something I noticed as I was testing some job in JobSet. It would be nice to see which jobs are suspended when doing . Hope its not a controversial change.
cc who worked on
Not controversial at all IMO /triage accepted
/assign
Any suggestions on where to place this column? I'm thinking between completions and durations. Or, should this be displayed only when the output is set to wide?
I would put it in the default view as well. Probably before completions
Addressed in the PR:


Question: &ConfigMapList{}, ) <del> // Register Unversioned types under their own special group scheme.AddUnversionedTypes(Unversioned, &metav1.Status{}, &metav1.APIVersions{}, &metav1.APIGroupList{}, &metav1.APIGroup{}, &metav1.APIResourceList{}, ) </del> return nil }

We have moved to non-global schema in a number of places in 1.6 and 1.7. It is easy to forget registering the unversioned types like , leading to encoding errors in error cases. The registration of the unversioned types should be automatic as for the types, i.e. in .
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
do we need to close out this issue? (since referenced PR has merged)
Fixed by


Question: create-network create-subnetworks detect-subnetworks <ins> create-cloud-nat-router </ins> write-cluster-location write-cluster-name create-autoscaler-config

Public IPs are expensive resources. Given that we're running scalability tests on 5k-node clusters, we are unnecessarily consuming 5k IPs which is a lot. We should switch our tests so that nodes have only private IP addresses. We are going to do that by configuring NAT so that nodes can talk to outside world (e.g. to download image from dockerhub). But that also requires changing some correctness tests which assume public IPs.
To summarize recent work: setting up private gce clusters with nat, feature is gated via the env test and dashboard with correctness tests on gce private cluster - https://k8s- 27 tests fail on fixing the tests - should fix majority of them
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale /lifecycle frozen
- is there anything missing to be done here?
No, we can close this one. It was done long time ago. Thanks for catching /close
Closing this issue. <detailsIn response to : No, we can close this one. It was done long time ago. Thanks for catching Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: // Ingress resource create/read/update/watch verbs ginkgo.By("creating") <del> _, err := ingClient.Create(context.TODO(), ingTemplate, metav1.CreateOptions{}) </del> <ins> _, err := ingClient.Create(context.TODO(), ingress1, metav1.CreateOptions{}) </ins> framework.ExpectNoError(err) <del> _, err = ingClient.Create(context.TODO(), ingTemplate, metav1.CreateOptions{}) </del> <ins> _, err = ingClient.Create(context.TODO(), ingress2, metav1.CreateOptions{}) </ins> framework.ExpectNoError(err) <del> createdIngress, err := ingClient.Create(context.TODO(), ingTemplate, metav1.CreateOptions{}) </del> <ins> createdIngress, err := ingClient.Create(context.TODO(), ingress3, metav1.CreateOptions{}) </ins> framework.ExpectNoError(err) ginkgo.By("getting")

We run sonobuoy tests on production running clusters whenever we upgrade our clusters. We observe that fails when we already have nginx-ingress controller running in the cluster. The newer versions of nginx-ingress controller comes up with a validating admission webhook that validates the ingress definitions. In this specific test, we are creating 3 ingress objects with the same path and the webhook do not permit to create them (the first one is created but the other two would fail) and then the job fails. The tests shouldn't fail and it requires we create ingress objects using different paths instead. Install nginx-ingress controller from Run e2e test. Test fails. No response =1.19 n/a n/a n/a n/a n/a
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig testing /area e2e-test-framework
This is a duplicate, see the comment author of the ingress nginx controller explaining why this is working as expected Edit check again, maybe is not a duplicate
Thanks for pointing this. It's infact the same issue. But the fix above should hopefully help since it would create those ingress objects with different paths and the validating webhook for ingress controller would permit creating objects in that case.
yeah sorry, just checked quickly from the mobile, I will take a look Thanks


Question: DoRaw() } <del> timeout := time.Now().Add(2 * time.Minute) </del> <ins> // nettest containers will wait for all service endpoints to come up for 2 minutes // apply a 3 minutes observation period here to avoid this test to time out before the nettest starts to contact peers timeout := time.Now().Add(3 * time.Minute) </ins> for i := 0; !passed && timeout.After(time.Now()); i++ { time.Sleep(2 * time.Second) Logf("About to make a proxy status call")

<!-- Please only use this template for submitting enhancement requests --What would you like to be : The documentation suggests that "The default pull policy is IfNotPresent" This is not the case when imagePullPolicy is defined under the spec section in a Pod definition without a specific value. Example: This will result in a Pod having imagePullPolicy as Always. Why is this needed: This is needed for clarification of the default behaviour. It is necessary for the appropriate handling of helm charts, in cases that the imagePullPolicy is empty.
/sig docs
The imagePullPolicy default value is .
In this paragraph you can see that In your reference it points out the default action for the kubelet, not the behaviour when imagePullPolicy is declared. Please do have in mind that the imagePullPolicy option for Never is mentioned only in this paragraph in the document.
If an image tag of "latest" is used, the imagePullPolicy defaults to to "Always": If no tag and no digest are specified on the image, an image tag of "latest" is used: The behavior when no image digest or tag is specified should be documented in the API docs for the and fields:
I am interested on helping this issue, but I am new to kubernete repo, it could take me sometime
I found the following which is well documented in "Best Practices for Configuration" It would be expected to include more information on the options in the main link under
I'd like to work on this one.
Very welcome, but yesterday also mentioned that he is following up on this issue, you can communicate with him before starting

cc
This has failed The two failures are different, and both seemingly totally legitimate. The first one is simply a pod never entering running state within the 5 minute timeout. Expected error: <*errors.errorString | 0xc208475150: { s: "gave up waiting for pod 'nettest-dtwk3' to be 'running' after 5m0s", } gave up waiting for pod 'nettest-dtwk3' to be 'running' after 5m0s not to have occurred The second is not being reachable: Dec 7 01:22:47.275: Unable to connect/talk to the internet: Get dial tcp 74.125.202.147:80: i/o timeout The latter could be a result of a missing MASQ rule on the node, or GCP egress problems. I don't think that this is worth trying to debug further in the context of this test for intra-pod communication. I'll check more generally whether we have a current intermittent problems with respect to pods entering running state, or reaching the internet.
I think this test may be causing builds to time out, too. See comments from me in .
For the future, I actually don't think we need the test at all in the networking test. Maybe even move it do a different test altogether.
This is clearly a bug. I'll send a PR shortly: Regarding the ping to , why did you put in in the first place?
hmmm, not sure what you mean by "in the first place" ? I did not write the test...
sorry, I thought that you'd written that part of the test. I will find out who wrote that, figure out why they put it there in the first place, and then either remove it, or rewrite it to work correctly. Clearly that part of the test needs at least some basic retry to work consistently.
ok great, no problem just wasnt sure what you meant. SGTM. Id be okay w/ deleting the ping entirely regardless of who wrote it, its a very old snippet which doesnt really add much to the overall test suite (and in fact is questionable wether it should even be an assertion at all )
somebody put it there for some reason. Better to understand why before just deleting it.

i looked, its actually rob who i work with :) i will ask him in the morning unless you find him first :) (rrati)
Regarding my alleged "clear bug" above, I was completely wrong. There's nothing wring with that code. And unfortunately, when this test goes into a presumed endless loop (see comment above) it does not produce any output (as the output is cached by Ginkgo until the test completes). So it's very difficult to debug. But in summary I agree with this test is probably never completing, and causing tests to time out. Someone will need to isolate this test in a environment where it produces output and can be debugged. I'll keep trying, but need to head home soon, so if anyone wants to pick it up overnight, go ahead.
Smoking gun:
hmm - I'm just wondering why only this particular test is hanging forever (if we agree that's the case). I don' think that some requests are hanging forever in this test, but not in any other test...
Closing in favor of which should fix this. Or at least some way to set client timeout, which and are still figuring out.
It's happening again.
Talked to offline. I am assigning this one to you for triage. Let me know if you need any help.
AFAICT the last failure was just the pod nettest-7h5jj not coming up
Looks like this should be resolved by your pr: too. :-)
Maybe. Though the webserver only deadlocks in the failure case, so the test is probably still flaky...
Looks like this test is now explicitly flaking by failing, as I expected.
Based on the , it took 15 minutes to pull the image. closing this one as dup of
New Failure:
This looks like a race between populating the endpoints list and the /status, /read methods of the service exposed through the master proxy. It looks like the webserver pod is just trying to confirm that it can reach all peers. We can do this synchronously after having observed all expected endpoints in the test (or re-write the test to use the same pattern we do in services e2e).
The test looks really racy to me. I think the following 15 retries prevents it to fail more often, but not completely.

I will submit a pr to enfore webservers to wait till alll endpoints are up before they start to contact each other.
New failure
I am making this P0
It took more than 4 minutes for a nettest pod to come up. The nettest container is waiting for all endpoints to be ready for 2 minutes plus 15 attempts (5 sec interval) to contact its peers. This happens in less than 4 minutes. Hence the first 2 nettest pods finished contacting peers and declared failure.
The node you want is: I see: that's too much silence ...
From the It looks like kubelet is slow to start the container. Due to the lower logging level in the gke, it's hard to guess what could've slowed down the goroutine...


Question: DiskMBpsReadWrite string } <del> func newManagedDiskController(common *controllerCommon) (*ManagedDiskController, error) { return &ManagedDiskController{common: common}, nil } </del> //CreateManagedDisk : create managed disk func (c *ManagedDiskController) CreateManagedDisk(options *ManagedDiskOptions) (string, error) { var err error

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: A storageAccount API call blocks kubelet from becoming Ready when there isn't any outbound IP, 1. eventually (after 10mins), the kubelet registers again... there is a call to the ARM API done by which blocks for 10mins, before letting kubelet complete with its starting logic and making the node as details: What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): v1.11.5 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: /sig azure /assign


Question: assert.True(t, strings.Contains(err.Error(), expectedErr)) assert.Empty(t, diskURI) } <del> func TestDeleteBlobDisk(t *testing.T) { ctrl := gomock.NewController(t) defer ctrl.Finish() b := GetTestBlobDiskController(t) b.common.cloud.BlobDiskController = &b mockSAClient := mockstorageaccountclient.NewMockInterface(ctrl) b.common.cloud.StorageAccountClient = mockSAClient fakeDiskURL := "fake" diskURL := "https://foo.blob./vhds/bar.vhd" err := b.DeleteBlobDisk(fakeDiskURL) expectedErr := fmt.Errorf("failed to parse vhd URI invalid vhd URI for regex https://(.*).blob./vhds/(.*): fake") assert.Equal(t, expectedErr, err) mockSAClient.EXPECT().ListKeys(gomock.Any(), b.common.resourceGroup, "foo").Return(storage.AccountListKeysResult{}, &retryError500) err = b.DeleteBlobDisk(diskURL) expectedErr = fmt.Errorf("no key for storage account foo, err Retriable: false, RetryAfter: 0s, HTTPStatusCode: 500, RawError: <nil>") assert.Equal(t, expectedErr, err) mockSAClient.EXPECT().ListKeys(gomock.Any(), b.common.resourceGroup, "foo").Return(storage.AccountListKeysResult{ Keys: &[]storage.AccountKey{ { KeyName: to.StringPtr("key1"), Value:   to.StringPtr("key1"), }, }, }, nil) b.accounts["foo"] = &storageAccountState{} err = b.DeleteBlobDisk(diskURL) expectedErrStr := "storage: service returned error: StatusCode=403, ErrorCode=AccountIsDisabled, ErrorMessage=The specified account is disabled." assert.Error(t, err) assert.True(t, strings.Contains(err.Error(), expectedErrStr)) } </del>

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: bazel-test-master Which test(s) are flaking: gotest: Testgrid link: Reason for failure: Anything else we need to know: links to appreciated links to specific failures in spyglass appreciated <!-- Please see the deflaking doc () for more guidance! --/sig cloud-provider
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cloud-provider
ping !
Interestingly the expected test string was logged: Test:
Also unable to reproduce this, ran it a few thousand times:
Looks like a true flake. Does this need attention?
/assign
I cannot repro this failure either, while if it's really flacky, I could remove this ut since that's really legacy code, since it's azure blob disk code which is deprecated long time ago, remove ut is quite safe.
here is the PR to remove this flacky ut:
/assign


Question: if err != nil { t.Fatalf("Failed to list pods: %v", err) } <del> if len(pods.Items) != 1 { t.Fatalf("Expect only 1 pod") </del> <ins> if len(pods.Items) > 1 { t.Fatalf("Unexpected pod list: %v", pods.Items) </ins> } // wait for the garbage collector to delete the pod if err := integration.WaitForPodToDisappear(podClient, garbageCollectedPodName, 5*time.Second, 30*time.Second); err != nil {

Had this test fail a few times on unrelated PRs in presubmit. Example failure: https://k8s- Possibly related to which recently merged. /kind flake cc


Question: w := printers.GetNewTabWriter(printer.out) defer w.Flush() <ins> sort.Slice(metrics, func(i, j int) bool { return metrics[i].Name < metrics[j].Name }) </ins> printColumnNames(w, NodeColumns) var usage api.ResourceList for _, m := range metrics {

and node should sort by namespace / name so that results don't jump around. We should consider adding sort, but only after we make a stable sort the default.
next Also, why does it sometimes lose nodes? Annoying
I did not reproduce this. I think this may be caused by the monitoring timeout of heapster. So that the data got missing.


Question: if !IsCRDConditionEquivalent(&tc.expectedcrdCondition[i], &crd.Status.Conditions[i]) { t.Errorf("%v expected %v, got %v", tc.name, tc.expectedcrdCondition, crd.Status.Conditions) } <ins> if crd.Status.Conditions[i].LastTransitionTime.IsZero() { t.Errorf("%q lastTransitionTime should not be null: %v", tc.name, i, crd.Status.Conditions) } </ins> } } }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: the for a CRDs Established condition is always null What you expected to happen: it should have a timestamp How to reproduce it (as minimally and precisely as possible): create a CRD and then examine its yaml/json. For example: Anything else we need to know?: Environment: community master
/assign
/area custom-resource
/sig api-machinery /area custom-resources
/assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
edit: actually, that issue is about kubectl display and clock skew. this issue is about the condition timestamp updating, which still requires correction
Closing this issue. <detailsIn response to : duplicate of Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hello I'm a bug triage shadow for the 1.16 release cycle and considering this issue is tagged for 1.16, I'd like to check its status. Code freeze is starting on August 29th (about 3 days from now), which means that this PR must merged by then. Do we still target this issue to be fixed for 1.16?
yes, will take a look at it tomorrow /assign


Question: func NewMutatingWebhookConfigurationManager(f informers.SharedInformerFactory) generic.Source { informer := f.Admissionregistration().V1().MutatingWebhookConfigurations() manager := &mutatingWebhookConfigurationManager{ <del> configuration: &atomic.Value{}, lister:        informer.Lister(), hasSynced:     informer.Informer().HasSynced, </del> <ins> configuration:              &atomic.Value{}, lister:                     informer.Lister(), hasSynced:                  informer.Informer().HasSynced, initialConfigurationSynced: &atomic.Value{}, </ins> } // Start with an empty list manager.configuration.Store([]webhook.WebhookAccessor{}) <ins> manager.initialConfigurationSynced.Store(false) </ins> // On any change, rebuild the config informer.Informer().AddEventHandler(cache.ResourceEventHandlerFuncs{

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: pull-kubernetes-bazel-test Which test(s) are flaking: Testgrid link: Example pass: Example fail: Reason for failure: Unknown Anything else we need to know: Looks like it's an apiserver test, but I'm not too sure. links to appreciated: <!-- Please see the deflaking doc () for more guidance! --/kind flake
Looks like is flaky too. Example fail:
/sig api-machinery
Thanks for the report. Please report individual failing tests/packages so the responsible sigs can be assigned. The cm/devicemanager issue is tracked in
/assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
I don't see individual issues yet, but I do see open PRs targeting this issue /remove-lifecycle rotten
closing in favor of specific issues: /close
Closing this issue. <detailsIn response to : closing in favor of specific issues: Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: p:             &ExecOptions{}, args:          []string{"foo", "cmd"}, argsLenAtDash: -1, <del> expectedPod:   "foo", </del> <ins> expectedPod:   execPod(), </ins> expectedArgs:  []string{"cmd"}, name:          "cmd, w/o flags", }, { p:             &ExecOptions{}, <ins> args:          []string{"pod/foo", "cmd"}, argsLenAtDash: -1, expectedPod:   execPod(), expectedArgs:  []string{"cmd"}, name:          "pod with 'pod/' prefix, cmd, w/o flags", }, { p:             &ExecOptions{}, </ins> args:          []string{"foo", "cmd"}, argsLenAtDash: 1, <del> expectedPod:   "foo", </del> <ins> expectedPod:   execPod(), </ins> expectedArgs:  []string{"cmd"}, name:          "cmd, cmd is behind dash", },

I feel like I should be able to do: but I get
or we have to do something like this: sounds reasonable to me. needs more comments.
All commands that accept resources should accept resource/NAME syntax. In this case for back compat we must only accept a single argument, is familiar with this.
Exactly. ResourceNames in the resource builder is what is needed here to handle the default resource (pod in this case) and provide the ability to specify more.
Related
If nobody is working on it, I'd like to pick the issue, pls.
I don't think anyone is. , Illia Khudoshyn wrote:
So could you please assign me?
I can't assign to just anyone, unfortunately, but by responding here, anyone else who might work on it will see that you are already in progress. , Illia Khudoshyn wrote:
please go ahead, you did own this now. :)
Has this been reverted again? I cannot use “pod/podname” for v1.11:
^ Seconding the above. I'm getting the same 'may not contain /' error. Naturally it's trivial to use unix commands to filter this, but this is such a basic feature and matches the way the rest of kubectl works, so it's quite a shocking omission.
as a unix novice who's also experiencing this problem, I'd love to know your workaround
If you use selectors, you can do something like this: Otherwise you'll need to manually grep, something like this: is the syntax in bash to run the command and use its output as part of another command. You can also use backticks to achieve the same effect but that's considered bad form by many (myself included). For example: If going the grep route, take care to make sure your regex will only return one result, otherwise you'll have to pipe it to xargs or something similar.
Just ran into this on a brand new default installation. Local Mac Docker for Desktop Enabled k8 in preferences error: invalid resource name "svc/broken-gorilla-postgresql": [may not contain '/']
FWIW you can use combined with the flag, e.g.
Had the same problem. Updating kubectl resolved the issue for me.


Question: if diskManager.volumeOptions.SCSIControllerType == "" { diskManager.volumeOptions.SCSIControllerType = vclib.LSILogicControllerType } <del> // Create virtual disk diskFormat := vclib.DiskFormatValidType[diskManager.volumeOptions.DiskFormat] // Create a virtual disk manager vdm := object.NewVirtualDiskManager(datastore.Client()) </del> <ins> // Check for existing VMDK before attempting create. Because a name collision // is unlikely, "VMDK already exists" is likely from a previous attempt to // create this volume. if dsPath := vclib.GetPathFromVMDiskPath(diskManager.diskPath); datastore.Exists(ctx, dsPath) { klog.V(2).Infof("Create: VirtualDisk already exists, returning success. Name=%q", diskManager.diskPath) return diskManager.diskPath, nil } </ins> // Create specification for new virtual disk <ins> diskFormat := vclib.DiskFormatValidType[diskManager.volumeOptions.DiskFormat] </ins> vmDiskSpec := &types.FileBackedVirtualDiskSpec{ VirtualDiskSpec: types.VirtualDiskSpec{ AdapterType: diskManager.volumeOptions.SCSIControllerType,

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: In some cases, a create volume request can succeed in vSphere but time out in the vSphere storage plugin. This request will be retried, but fails with on subsequent attempts. As a result, the PersistentVolume object never gets created, and the pod requesting it never starts. Recovery requires manual steps. What you expected to happen: Since vSphere successfully created the volume, the storage plugin should reconcile and allow the PersistentVolume to get created. How to reproduce it (as minimally and precisely as possible): This is difficult to reproduce because it is caused by specific timing between vSphere and kubernetes. For testing, I a line of code to [][1] to inject failures after vSphere created the volume. [1]: Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig storage
Fixed by


xref: , The package is failing tests and doesn't seem to work as expected when run under go1.9. It throws some x509 certificate errors. However, the same tests run successfully under go1.8. go1.8 successful output: go1.9 failing output: /priority important-soon
/assign


Question: -mode=fix  -repo_root "${KUBE_ROOT}"  "${KUBE_ROOT}"  <del> 2>&1 | grep -Ev "vendor/github.com/seccomp/libseccomp-golang/seccomp(_internal)?.go: pkg-config not supported: #cgo pkg-config: libseccomp" || (exit "${PIPESTATUS[0]}") </del> <ins> 2>&1 | grep -Ev "$(kube::util::join | "${KNOWN_PKG_CONFIG_ERRORS[@]}")" || (exit "${PIPESTATUS[0]}") </ins> # Run kazel to update the pkg-srcs and all-srcs rules as well as handle # Kubernetes code generators.

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Got the following error while trying to run : How to reproduce it (as minimally and precisely as possible): Ran this on a fresh install of ubuntu 19.04, Go 1.12.6, and bazel 0.26.1. Anything else we need to know?: Seems like the change might have been due to the update in vendored deps from ( ). Although this script is run in CI where it is working so may be some set of recommende pkg versions would help prevent this problem? Environment: Cloud provider or hardware configuration: OS (e.g: ): ubuntu 19.04 Kernel (e.g. ): 5.0.0-15-generic bazel: 0.26.1 Go: 1.12.6
Not sure who owns this... /sig testing
Seems like the errors are "warnings" and can be ignored? It seems that BUILD files are properly updated by update-bazel, And bazel builds do work properly... /cc
/sig release this is not a failure /retitle hack/update-bazel produces warnings there was some discussion on the original PR about filtering them out but nothing conclusive I'll file a PR to do that now.. /assign
aside: this is actually from :-) we're working around the lack of support because we only need it for this package and on all known systems we compile on so far it only needs the one link flag as previously specified in the cgo config.
filed
Glad to see it. The output message is so misleading that I have spent hours on this issue
sorry about that the output does say "error" but it's actually considered a non-fatal warning in the upstream code and doesn't cause the script (or build) to fail.
was merged before this discussion was resolved , so we kinda saw this coming. :/ Hopefully we can get the fix from in merged as quick as possible
/reopen we have a new a new occurrence with libipmctl
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available .

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I have the same question. This output is a bit annoying.
We can update the script to handle this issue and suppress the warning. The previous PR is an example. On Thu, Apr 16, 2020, 12:14 AM <:
/assign


Question: assert.NoError(t, err) assert.Empty(t, containers) } <ins> // TestToKubeContainerStatus tests the converting the CRI container status to // the internal type (i.e., toKubeContainerStatus()) for containers in // different states. func TestToKubeContainerStatus(t *testing.T) { cid := &kubecontainer.ContainerID{Type: "testRuntime", ID: "dummyid"} meta := &runtimeapi.ContainerMetadata{Name: "cname", Attempt: 3} imageSpec := &runtimeapi.ImageSpec{Image: "fimage"} var ( createdAt  int64 = 327 startedAt  int64 = 999 finishedAt int64 = 1278 ) for desc, test := range map[string]struct { input    *runtimeapi.ContainerStatus expected *kubecontainer.ContainerStatus }{ "created container": { input: &runtimeapi.ContainerStatus{ Id:        cid.ID, Metadata:  meta, Image:     imageSpec, State:     runtimeapi.ContainerState_CONTAINER_CREATED, CreatedAt: createdAt, }, expected: &kubecontainer.ContainerStatus{ ID:        *cid, Image:     imageSpec.Image, State:     kubecontainer.ContainerStateCreated, CreatedAt: time.Unix(0, createdAt), }, }, "running container": { input: &runtimeapi.ContainerStatus{ Id:        cid.ID, Metadata:  meta, Image:     imageSpec, State:     runtimeapi.ContainerState_CONTAINER_RUNNING, CreatedAt: createdAt, StartedAt: startedAt, }, expected: &kubecontainer.ContainerStatus{ ID:        *cid, Image:     imageSpec.Image, State:     kubecontainer.ContainerStateRunning, CreatedAt: time.Unix(0, createdAt), StartedAt: time.Unix(0, startedAt), }, }, "exited container": { input: &runtimeapi.ContainerStatus{ Id:         cid.ID, Metadata:   meta, Image:      imageSpec, State:      runtimeapi.ContainerState_CONTAINER_EXITED, CreatedAt:  createdAt, StartedAt:  startedAt, FinishedAt: finishedAt, ExitCode:   int32(121), Reason:     "GotKilled", Message:    "The container was killed", }, expected: &kubecontainer.ContainerStatus{ ID:         *cid, Image:      imageSpec.Image, State:      kubecontainer.ContainerStateExited, CreatedAt:  time.Unix(0, createdAt), StartedAt:  time.Unix(0, startedAt), FinishedAt: time.Unix(0, finishedAt), ExitCode:   121, Reason:     "GotKilled", Message:    "The container was killed", }, }, "unknown container": { input: &runtimeapi.ContainerStatus{ Id:        cid.ID, Metadata:  meta, Image:     imageSpec, State:     runtimeapi.ContainerState_CONTAINER_UNKNOWN, CreatedAt: createdAt, StartedAt: startedAt, }, expected: &kubecontainer.ContainerStatus{ ID:        *cid, Image:     imageSpec.Image, State:     kubecontainer.ContainerStateUnknown, CreatedAt: time.Unix(0, createdAt), StartedAt: time.Unix(0, startedAt), }, }, } { actual := toKubeContainerStatus(test.input, cid.Type) assert.Equal(t, test.expected, actual, desc) } } </ins>

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): Pod status started null, started, 01 Jan 0001 Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-05-10T15:48:59Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-05-10T15:38:08Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"}` Environment: Cloud provider or hardware configuration: - OS (e.g. from /etc/os-release): CentOS 7 Kernel (e.g. ): Install tools: kubeadm 1.6 What happened: When a pod is created and starts running, shows the correct container start time in Containers -containername -State -Started. After the container is terminated, the start time changes to . In addition, shows that is . What you expected to happen: I'm expecting the start time to not change after the pod is terminated, so that afterwards I could check how long a time the container execution took. How to reproduce it (as minimally and precisely as possible): Have a running kubernetes cluster, and create a pod with the following command: kubectl create -f - <<EOF apiVersion: v1 kind: Pod metadata: name: sleep spec: containers: name: sleep image: ubuntu command: ["sleep"] args: ["30"] restartPolicy: Never EOF This created a container that sleeps for 30 seconds and quits.

During those 30 seconds, check the pod status with : ... State: Running Started: Wed, 17 May 2017 10:25:22 +0300 ... After 30 seconds, check the status of the terminated container with again: ... State: Terminated Reason: Completed Exit Code: 0 Started: Mon, 01 Jan 0001 00:00:00 +0000 Finished: Wed, 17 May 2017 10:25:52 +0300 ... With we see that the container's start time is null: ... "state": { "terminated": { "containerID": "docker://ca0dbe0dab3bb91ad0b05ac167e44d5fbf805f736ef98f61b9e106ba4456667f", "exitCode": 0, "finishedAt": "2017-05-17T07:25:52Z", "reason": "Completed", "startedAt": null } } ... Anything else we need to know: I need to measure container run durations and this bug prevents me from doing so.
/cc PR to fix this:


Question: INDIRECT=$$(go list -e                                 -f '{{range .Deps}}{{.}}{{"n"}}{{end}}'           ./cmd/libs/go2idl/deepcopy-gen                     <del> | grep "^$(PRJ_SRC_PATH)"                          </del> <ins> | grep --color=never "^$(PRJ_SRC_PATH)"            </ins> | sed 's|^$(PRJ_SRC_PATH)|./|'                     | xargs go list -e -f '{{.Dir}} {{.Dir}}/*.go');   echo $$DIRECT $$INDIRECT

Using the grep from OSX, fails when GREPOPTIONS/GREPCOLOR is set as shown below (). Workaround : unset GREPOPTIONS/GREPCOLOR Note that using GNU grep using brew install will show that GREPOPTIONS shows immediately that GREP_OPTIONS may be interfering with make ()
oh, here's badly manged conversion-
Will look, thanks. For make-related bugs, please @ me directly.
Will do. thanks
This breaks building kubernetes inside Alpine Linux:
I'm pressing more and more on the idea that the only supported build is the dockerized build. Trying to accomodate every random environment is just too taxing. , Marc Villacorta wrote:
I agree with We can support the stuff in to run on mac to get you dockerized but everything else is only supported in there. I've worked around a few BSD/OSX things in but it isn't fun. We should try to keep that stuff isolated.
I agree, with the usual caveat that we need to be doing all of the build in one container with as little as possible happening outside. OS X tends to have old and broken versions of important programs, which is terrible. We discussed setting up CI for it but it's not going to happen in the near term. Alpine/Busybox purposefully omit features from their versions for good reasons, but it's still more work to support than it's worth.
Ok, I will re-think my workflow.
How did you find yourself building in Alpine, in I may ask? , Marc Villacorta wrote:
nothing is forcing me to use Alpine I just like the distro. I run stuff on CoreOS and my aim is to solve the build-deliver-run pipeline with a single small container. By the way, I have the problem installing the GNU package :-)


Question: ttlSeconds = int(v1.DefaultClientIPServiceAffinitySeconds) //default to 3 hours if not specified.  Should 0 be unlimited instead???? } <del> if _, exists := lb.services[svcPort]; !exists { </del> <ins> if state, exists := lb.services[svcPort]; !exists || state == nil { </ins> lb.services[svcPort] = &balancerState{affinity: *newAffinityPolicy(affinityType, ttlSeconds)} klog.V(4).InfoS("LoadBalancerRR service does not exist, created", "servicePortName", svcPort) } else if affinityType != "" {

As suggested in , some portions of the code contains the map checking for nil values in map, and others don't have. Even though we're never assigning nil values to this map, we should standardize the portions that does not have this check in module. Example: Should be like this: I can take care of this issue.
/sig network
There have been discussions of deprecating the user-space proxier. Maybe transfer it to . A question is if anybody still uses it. Seems like you do, so if possible please explain if you have a use-case where the user-space proxier is wanted/needed.
Hey Actually, I was casually reading the source code and I saw a over there regarding the nil checks. Regarding your question, we should check if anyone actually uses it though, or maybe discuss its usage or deprecation a little further.
Quick question though: I saw in the news that the user-space proxy porting was near completion, maybe it's worth talking to them in order to check upon the status.
I don't object to making it clearer or more consistent, if you want to spend time on it, but it's not a high priority. That said, I think a nil value in that map would be really bad (and should be impossible) and I don't know how we would raise a signal about it.
let us know if you want to try to fix it, otherwise we can just close this.
I already opened a PR with the clearer nil checks for this specific module in , so you can review it. I think whether this would have nil values into the service map we should discuss this apart and try to reproduce this case (if any would happen).
PR submitted, so let's review and discuss further there. /assign dcbw /triage accepted


Question: // the caller in framing the chunk. func NewDocumentDecoder(r io.ReadCloser) io.ReadCloser { scanner := bufio.NewScanner(r) <ins> // the size of initial allocation for buffer 4k buf := make([]byte, 4*1024) // the maximum size used to buffer a token 5M scanner.Buffer(buf, 5*1024*1024) </ins> scanner.Split(splitYAMLDocument) return &YAMLDecoder{ r:       r,

What happened: When calling the token size for the buffer is the default which is NOT changable. Larger documents (which is likely when using a yaml separator) causes a This was uncovered using kudo... the issue is What you expected to happen: limitation should be documented if they exist size should be calling code managed How to reproduce it (as minimally and precisely as possible): Call with a doc larger than token size Anything else we need to know?: The fixing PR provides more detail if needed. Code concern is: duplicate of which I will now close Environment: Kubernetes version (use ): 1.17 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig api-machinery
/cc
OK so the request here is that we call on the scanner to permit larger documents? I think 5MB is a good max size? (Since YAML is more verbose than proto, where the (IIRC) 1.5MB limit is enforced.)
Adding labels-- this is a good one for new contributors.
/assign 0xMH because of open PR
GitHub didn't allow me to assign the following users: 0xMH. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign 0xMH because of open PR Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/remove-help /remove good first issue


Question: Run: runHelp, // Hook before and after Run initialize and write profiles to disk, // respectively. <del> PersistentPreRunE: func(*cobra.Command, []string) error { </del> <ins> PersistentPreRunE: func(cmd *cobra.Command, args []string) error { </ins> rest.SetDefaultWarningHandler(warningHandler) <ins> if cmd.Name() == cobra.ShellCompRequestCmd { // This is the __complete or __completeNoDesc command which // indicates shell completion has been requested. plugin.SetupPluginCompletion(cmd, args) } </ins> return initProfiling() }, PersistentPostRunE: func(*cobra.Command, []string) error {

<!-- Please only use this template for submitting enhancement requests --What would you like to be : I type <kbdTab</kbd<kbdTab</kbdI would like to see kubectl plugins in the list of suggested subcommands. For example, if I have installed , subcommand completions should suggest . the plugin itself also supports command-line completion, <kbdTab</kbd<kbdTab</kbdshould show suggestions from the plugin command completion. Why is this needed: Plugins fill in niches in the kubectl workflow, so that they are used rather seldomly. This means that plugin users need to memorize plugin names and their options over long times without using them. Therefore, plugin usefulness would be greatly enhanced by suggesting plugin sub-commands and their options.
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
In fact, already supports command completion. You can run for more information about it. If you are using bash shelljust run , and then it will work!
Hi , but I'm not talking about command completion in general. I'm talking about command completion for plugins. For example, the new plugin system allows to call a plugin at location as . It would be nice to get suggested/completed, when typing <kbdTab</kbd. Further, if plugin names are completed, so should be options for those plugins -- provided the plugin offers some form of command completion.
I try to implement completion for kubectl plugins!
Were you able to make progress on this? I have a working patch for the completion of the plugin name (part 1 of this issue). I can make a PR for it if it can help.
This is a great first step! I do think however, that the real value is auto-completion for the plugin options. I can roughly remember what plugins I have, but not what options they offer. That's the real value here, I think. This will probably need a contract for how plugins make completion available (e.g. by following the kubectl standard of ). WDYT?
I've been thinking about how to do the plugin options completion.

Based on the patch I have now, I believe what will be needed is to be able to ask the plugin what the possible completions can be (based on the user-specified parameters). So the plugin won't need to generate a bash or zsh completion script (which is difficult for the author of the plugin) but just output possible completions. This also has the benefit of being independent of which shell is used. One way is to do as you mention and have the plugin support as a way to ask the plugin what the completions would be. Although it won't behave in the same fashion as (as it won't generate a completion script, and is shell-independent), I like that option because it simply requires a plugin author to provide a different script called which the bash completion logic can call. Another way would be to ask the plugin itself to support a flag such as which would return the possible completions. I don't like this because if the plugin doesn't support this flag, we'd be calling it and who knows what it would do in that unexpected case. The use of makes it easy to check of presence or absence of completion support. So I suggest that the contract with plugins be that a plugin that supports completion would provide two scripts and that the bash completion script call to get the list of completions. Any thoughts?
Hi guys, I've already implemented completion for plugins (subcommands only, doesn't include options) as However it needs to be refactored more. You can try completion for plugins with the following steps: First, prepare the repository, and build a kubectl. Install kubectl-hello plugin for completion testing. Load the kubectl completion code. You can see completion for some plugins.
Cool! You've used the same technique I did. However, may I suggest re-using the code from the plugin package instead of copying it?
Well, you are absolutely right in saying that we can't just call a plugin with some flag to produce the completion script, because who knows what will happen. Therefore, option completion for plugins has to go the route via an external completion script, like However, I'm pretty sure that those completion files need to be bash/zsh scripts, because they need to process contextual information.

For example, say I have a plugin with sub-commands Then I want completion to work as follows: As you say, this will have to be maintained by the plugin authors. Can you take a typical cobra command completion script as basis and work out how kubectl-plugin-completion and this external script need to work together? You could even try to have kubectl as its own plugin, like and or so :grinning:
kubectl already has some dynamic completion code to allow to complete pod names for example, which actually calls kubectl itself. I think I can use the same technique for plugin parameter completion. So here's an example of what I'm thinking: Then when the user types the kubectl completion script will call . This call will return which will be given to the shell by the kubectl completion script. I haven't actually tested this example, so I could be wrong. I'll try implementing it to see if I'm missing something.
Thank you for supporting me ;) I add you as co-author to commit when I send a PR.
Thanks
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue.

<detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-lifecycle rotten
I've been working on auto-completion for the Helm project and I have a proposal with a PR that fully supports auto-completion for Helm plugins. I believe the fundamental aspects of the solution would apply just as well to kubectl. Some adaptation would be needed to adjust to how kubectl supports plugins, but I think it wouldn't be very hard.
Is there an ETA on this? If I see it correctly, this is currently preventing plugin authors to implement completions as the generated completion suggestions are based off the root command and not the subcommand aka plugin. I've been trying all day to make this work but failed and that's when I stumbled over this issue..
Several major issues I see here are: We don't want files like as proposed in previous comment. We had a requirement prior to kubectl 1.12 and it didn't look great at all. The simplicity of plugins is valuable to us. (Also, unlike Helm plugins, kubectl plugins don't live at a particular directory, or follow a directory structure.)Plugin completions most likely would be integrated into commands, which a lot of people put in their or file. We need to keep this command fast. So what if the command called a special command on each plugin, in parallel, to inquire if it has completion available (e.g.), with a short deadline, and cached the successful results (maybe keyed with on plugin binary’s stat data, or checksum). That way, cmd could also provide data about plugins, while periodically refreshing the cached completion scripts.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with .

Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
any updates on this one ? is it complete/done?
No update, still open..
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale
/lifecycle frozen
Sorry to revive this thread a bit but while working on another cli using the same cobra completion system by (implemented by i believe) I found out that it would not be too hard or too disruptive to have a small patch on the shell completion directly. For example with kubectl if we have a patch like this (using zsh but I am guessing the same technique will apply for bash or others) : the patch will check the first command () and the second command for example "krew" exist as and then use it directly for completion otherwise do the normal completion 'thingy'. the only downside is that the initial plugin would not show i.e: if you do you won't get krew as the plugin name, except if we add all plugin to the Root Command like the pull request from does. I don't think it would be too intensive since it is a zsh/bash builtin and in memory. I have made a small asciinema to demonstrate how it works I guess if we want this we may need to do it directly in cobra, do you think a patch like this could get accepted ?

(I had somehow difficulties to get PR reviewer and merged in cobra last time I tried)
Thanks for the suggestion. The idea is good but doesn't generalize perfectly. The command is available for programs that use Cobra but kubectl plugins can be something else (a script, a Python program, a Go program that doesn't use Cobra, etc). So we couldn't do something like this, unless we required every plugin to provide a command. I'm working on porting Helm's plugin completion solution to kubectl and should have something for the community to try soon. Plugins using Cobra will then be able to use your suggestion and get completion super easily.
That make sense I forgot about non go (and non cobra) plugins, thanks I look forward to see how the helm completion extracted will look like !
I've posted something for people to try out in
It would be nice if some plugin developers try out the solution proposed in and provide feedback. since uses Cobra, the solution is very easy. Here is how to try it out: First build kubectl with the PR. Then create your completion script like this: put this new on $PATH. And now test using the newly built (I build mine at ). For completion of flags (which I realize is the real value for the plugin), kubectl needs to first move to Cobra 1.3.
I would be interested in your opinion on the proposed solution. It will not work out-of-the-box with Krew, but I haven't found a way to do better with respect to Krew. You had mentioned this previously, and although it would be a nice solution, I am concerned it would cause problems to any plugin that ignores all arguments and runs no matter what. With such a plugin, calling a command would trigger the plugin to run and possibly perform undesired actions when completions are requested.
I'm sorry about slow responses. Finally had time to read it and parse these. You're right we shouldn't randomly call a plugin binary without knowing it supports completion. From : kubectl already has a mechanism to find all plugins on $PATH, so I agree that at minimum, kubectl<kbdTab</kbdcan and should show plugins installed on the system (even if the plugins themselves don't support shell completion) I think this is a good model.

Just like how kubectl scans $PATH for , it can scan to see if a plugin supports completion. (It's important to define if that binary takes additional positional args or --options) To make the (2) a reality, I support adding an additional field to the Krew plugin manifest. The shape of that would be up for a discussion (I imagine it might be (a) a separate binary that comes in the plugin archive, or (b) allow linking to the plugin binary itself and the plugin reads argv[0] and prints completion, which can be done with (a) anyway). I think it's doable. I'm not very familiar with how shell completion works in practice, when it's called, how lazy can it be loaded, what are performance implications etc, but if you'd like to drive this effort, I think we can discuss a design on Krew side as well. Please bring this up in a SIG CLI meeting to see what the folks think about your proposal.
brilliant! Thanks for taking the time to think about this. Your suggestion to adapt krew sounds great to me. I will continue with the PR on kubectl's side and if it gets accepted we can look at krew.
The as proposed in accepts the exact same arguments as the standard executable. The new completion executable will examine those arguments and will determine what the shell completion choices following them should be.
do you have an insight into whether a plugin developer would develop and ship a separate executable for the completion, or would they simply link the main plugin executable and see if the argv[0] is the kubectl_complete-*? I assume the answer largely also depends on whether the plugin was authored in Go or bash (basically the only two languages I've ever seen a plugin submission in out of 150+).
I think plugins will choose to ship a separate executable. I feel it keeps the two aspects nicely separate (the plugin functionality and the shell completion functionality). Also, it is very easy for a separate executable to call back the main plugin if the developer wishes to do so, so I find that solution the most flexible. For example, for plugins using the Cobra library, the script would call back the executable like this: BTW, I have looked into the 179 plugins in the krew index and I found that 50% (yes, half!)

of them used Cobra (some of them indirectly by calling back ). This means that shell completion should be very easy to implement for all these plugins.
I'm interested in this support and thanks for adding this feature. One question: in which version this is planned to be released? Looking forward to this.