Question: if err != nil && !strings.Contains(err.Error(), "use of closed network connection") { glog.Errorf("Error proxying data from backend to client: %v", err) } <del> wg.Done() </del> <ins> close(readerComplete) </ins> }() <del> wg.Wait() </del> <ins> // Wait for one half the connection to exit. Once it does the defer will // clean up the other half of the connection. select { case <-writerComplete: case <-readerComplete: } glog.V(6).Infof("Disconnecting from backend proxy %sn  Headers: %v", &location, clone.Header) </ins> return true }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: When opening port-forward requests to a pod, kubelet memory usage increases without releasing it again when the port-forward is stopped. What you expected to happen: Kubelet should cleanup the port-forward when it is closed How to reproduce it (as minimally and precisely as possible): Using the following script I can observe an increase in Kubelet memory usage on the node where the pod runs: 300 connections allocates roughly 30mb by kubelet We first noticed this on the node where Tiller (Helm) runs. The node regularly runs out of memory because of the Kubelet memory usage. see also We don't experience the helm issue on all clusters, but I can reproduce the memory leak with the above script on all clusters. Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: AWS OS (e.g. from /etc/os-release): Debian jessie Kernel (e.g. ): Install tools: Kops 1.8.0 Others:
/sig node In my testing (v1.8.0), I observed create a TCP connection between apiserver and kubelet for each port-forward (and one between apiserver and kubectl). This apiserver<-kubelet TCP connection was not torn down when the corresponding command was killed. I have not tried against a newer apiserver/kubelet, but if someone has one handy, it's easy to just compare output on the apiserver or kubelet host before and after running several commands. Be aware that the CLOSE_WAIT connections can take minutes to expire after the kubectl<-apiserver commands exit, and this is normal and fine - we're interested in the ones that stay in state ESTABLISHED.
I'm not aware of any changes in 1.9 or 1.10 that would have fixed this. Not that we shouldn't fix this issue, but something like would be a good backstop to leaking connections like this
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with .

Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale behaviour is still observable for us on 1.9
On v1.11 I see plenty of and connections that hang around after many minutes.
I some logging to (among many other places) After closing a port-forward from kubectl, I see immediately the reader close on the apiserver, and the writer close on the kubelet. I do not see the other side of the connections close. Maybe 1 on each side when the pod ends. Is there a flag to enable pprof so we can dump some goroutine stacktraces? /sig apimachinery
kubelet stacktrace, the important parts
/sig api-machinery
apiserver stacktrace, the important parts
Lastly, if you restart on one side, the other side does clean up. At least if it's the kubelet. I restarted the kubelet, and apiserver shows Which is the ~600 waiting goroutines.
Based on some conversation with I will look into the approach for this issue. That issue () seems like a similar problem.


Question: flags := &applyFlags{ parent:           parentFlags, imagePullTimeout: 15 * time.Minute, <del> etcdUpgrade:      false, </del> <ins> etcdUpgrade:      true, </ins> } cmd := &cobra.Command{

kubeadm etcd upgrade modifying recovery steps /assign /sig cluster-lifecycle /kind bug


Question: ServiceAccountLookup        bool ServiceAccountTokenGetter   serviceaccount.ServiceAccountTokenGetter KeystoneURL                 string <ins> KeystoneCAFile              string </ins> WebhookTokenAuthnConfigFile string WebhookTokenAuthnCacheTTL   time.Duration

Hi, I tried to integrate the keystone auth plugin (which would be perfect for my needs) into the cluster I'm currently setting up. However, I can't get the connection working since our keystone installation is currently only reachable over http As far as I can see in , there is no option to deactivate this check. I think this should be an optional parameter on startup, similar to defaulting to the check being activated.
I don't think we should enable username/password checking over http.
I agree that we are loosing a security layer here, but you can't guarantee you got the password from client over https, so it might be intercepted at this point anyways. More general, I think the proper way to authenticate to Kubernetes with the Keystone plugin using tokens since they will limited lifetime restricted on keystone only which is a big deal regarding keystone likely being bound to a LDAP and thus the password being valid for other services. So I don't want to check username/password directly. I'm just starting out with the Keystone plugin, so I want to point this out in general, if token auth is supported, the better. Couldn't try it in praxis for now since I can't get the auth URL running.
Assigning to to add a priority label.
Setting the priority low until we have a sense of how many users have this restriction (keystone only reachable over http).
I ran into this same issue with trying to deploy kubernetes with devstack in the same vm and wanted to test integration and do some development. I think there are a significant number of developers in this same boat. http is secure enough since it doesn't leave the developers vm and it isn't intended for production. The use case is the same as:
I'd vastly prefer allowing specifying the CA for the keystone server over allowing http auth calls carrying usernames and passwords
I don't disagree. :) But, I also think Kubernetes shouldn't handle the usernames/passwords at all. In http mode, sending a token is much less risky then a username/password since they expire relatively quickly. See this issue for details:
I agree tokens would be preferable, but I wouldn't expect credentials of any kind sent over http to remote servers
Agreed. No one in their right mind should be passing creds or bearer tokens over http over a network.

I'm just interested in being able to easily deploy a k8s in DevStack with a simple plugin, where that k8s is hooked into the deployed keystone so that we can start developing things in OpenStack that depend on k8s. This gives developers a smooth/easy experience to then focus on doing the development stuff and not have to mess with setting up a CA/k8s, which will prevent some of them from contributing to k8s/OpenStack. The idea is simple for developers, secure for operators. Totally separate use cases.
It should be as simple as telling k8s to trust the keystone cert as a CA (assuming the self-signed cert generated by keystone is correctly marked as a CA, which it should be, since it is signing for itself). I've seen too many production installations running with http or untrusted TLS settings to believe that only developers will use this option if it's made available.
I didn't think Devstack could easily deploy with SSL enabled, but it looks like you can: So http support might not be as important as I thought. It could be relegated to a documentation issue for deploying Devstack in this mode when deploying with Kubernetes. We'd still need a way to specify the root CA though for Keystone.
So, I played around with this a little. For my current non devstack based system, I've deployed kubernetes as a cluster running on my laptop deployed as a set of pods via /etc/kubernetes/manifests. Only kubelet is native and all the rest of the services are containers direct from google. Its working well, but since I have 3 api server containers, and get the containers direct from google, I'd have to jump through a bunch of hoops to add the keystone ssl ca cert to each of the containers. I still think an option to specify the ssl cert for keystone would be useful rather then having to merge it into the system's chain of trust.
That's how I'd want to handle this as well


Question: package network import ( <del> "context" </del> "fmt" "math" "net"

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: n/a Which test(s) are failing: "should resolve connection reset issue [Slow]" Since when has it been failing: it never worked Testgrid link: n/a Reason for failure: The image used for the test is using only IPv4 Anything else we need to know: We should: [x] refactor the code to support IPv6 [ ] ~consolidate the image in agnhost~ [x] configure an IPv6 job to test network https://k8s-,%20IPv6,%20master
/assign /sig-network
GitHub didn't allow me to assign the following users: Billy99. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign /sig-network Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig network
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/remove-triage unresolved
Those labels are not set on the issue: <detailsIn response to : /remove-triage unresolved Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/help wanted
/help wanted /good-first-issue
/remove-good-first-issue the agnhost promotion process is not something easy for new contributors
Hi, I would like to contribute to this.
good, thanks
sorry , I needed to fix it and I took over, thanks for stepping up /assign /remove help-wanted
OK.Yesterday I have refactor the code to support IPv6, but I understand wrong about consolidate the image in agnhost. I'm sorry for not being able to help.
no worries, it was trickier than I thought, thanks again for your help
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: "k8s.io/kubernetes/pkg/apis/scheduling" "k8s.io/kubernetes/test/e2e/framework" <ins> "github.com/godbus/dbus/v5" </ins> v1 "k8s.io/api/core/v1" schedulingv1 "k8s.io/api/scheduling/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/kubernetes/pkg/features" kubeletconfig "k8s.io/kubernetes/pkg/kubelet/apis/config" kubelettypes "k8s.io/kubernetes/pkg/kubelet/types" <del> e2eskipper "k8s.io/kubernetes/test/e2e/framework/skipper" </del> testutils "k8s.io/kubernetes/test/utils" )

xref: Graceful node shutdown test uses to invoke dbus - As per some distros, don't have installed so the test must be skipped in those environments. We should consider if we can switch to use or perhaps and fall back to to ensure the test can work across more distros.
/sig node
/sig testing
/remove-kind bug /kind feature I'm putting kind feature to signify that nothing is really broken in k8s, tests needs improvement.
/remove-sig testing This dbus command should be up to SIG node, though tentatively: /triage accepted
/area test
Do we actually have these other distros available to prevent regressions if we change that?
/assign
the change will be to replace skip logic when is missing to use an alternative way to send this. So there will be no regression on distros where it currently runs without the skip. Original bug should have a link to job that was failing
I would like to work on improving the command. Am I understanding correct here We have to change to logic where we skip to use busctl instead. Could you please give me some idea on how can we make it to use , some similar examples in the repo would be really useful.
please have a look at The issue is that the command available is too old and doesn't include .
I see, it looks like we are not just adding a condition for instead we are replacing it with . It looks you are already working on it. Please inform if I can help in anyway. Thanks for the response.
Both are possible... it was easier to get a CI signal if that was used in all cases. I might put the condition later. I am waiting for comments in the PR, but feel free to propose other solutions that might work with older systemd versions.


Question: return dataSourceRef, cleanupFunc } <ins> // MultiplePVMountSingleNodeCheck checks that multiple PV pointing to the same underlying storage can be mounted simultaneously on a single node. // // Steps: // - Start Pod1 using PVC1, PV1 (which points to a underlying volume v) on node N1. // - Create PVC2, PV2 and prebind them. PV2 points to the same underlying volume v. // - Start Pod2 using PVC2, PV2 (which points to a underlying volume v) on node N1. func MultiplePVMountSingleNodeCheck(client clientset.Interface, timeouts *framework.TimeoutContext, claim *v1.PersistentVolumeClaim, node e2epod.NodeSelection) { pod1Config := e2epod.Config{ NS:            claim.Namespace, NodeSelection: node, PVCs:          []*v1.PersistentVolumeClaim{claim}, } pod1, err := e2epod.CreateSecPodWithNodeSelection(client, &pod1Config, timeouts.PodStart) framework.ExpectNoError(err) defer func() { ginkgo.By(fmt.Sprintf("Deleting Pod %s/%s", pod1.Namespace, pod1.Name)) framework.ExpectNoError(e2epod.DeletePodWithWait(client, pod1)) }() ginkgo.By(fmt.Sprintf("Created Pod %s/%s on node %s", pod1.Namespace, pod1.Name, pod1.Spec.NodeName)) // Create new PV which points to the same underlying storage. Retain policy is used so that deletion of second PVC does not trigger the deletion of its bound PV and underlying storage. e2evolume, err := getBoundPV(client, claim) framework.ExpectNoError(err) pv2Config := e2epv.PersistentVolumeConfig{ NamePrefix:       fmt.Sprintf("%s-", "pv"), StorageClassName: *claim.Spec.StorageClassName, PVSource:         e2evolume.Spec.PersistentVolumeSource, AccessModes:      e2evolume.Spec.AccessModes, VolumeMode:       e2evolume.Spec.VolumeMode, ReclaimPolicy:    v1.PersistentVolumeReclaimRetain, } pvc2Config := e2epv.PersistentVolumeClaimConfig{ NamePrefix:       fmt.Sprintf("%s-", "pvc"), StorageClassName: &claim.Namespace, AccessModes:      e2evolume.Spec.AccessModes, VolumeMode:       e2evolume.Spec.VolumeMode, } pv2, pvc2, err := e2epv.CreatePVCPV(client, timeouts, pv2Config, pvc2Config, claim.Namespace, true) framework.ExpectNoError(err, "PVC, PV creation failed") framework.Logf("Created PVC %s/%s and PV %s in namespace %s", pvc2.Namespace, pvc2.Name, pv2.Name) pod2Config := e2epod.Config{ NS:            pvc2.Namespace, NodeSelection: e2epod.NodeSelection{Name: pod1.Spec.NodeName}, PVCs:          []*v1.PersistentVolumeClaim{pvc2}, } pod2, err := e2epod.CreateSecPodWithNodeSelection(client, &pod2Config, timeouts.PodStart) framework.ExpectNoError(err) ginkgo.By(fmt.Sprintf("Created Pod %s/%s on node %s", pod2.Namespace, pod2.Name, pod2.Spec.NodeName)) ginkgo.By(fmt.Sprintf("Deleting Pod %s/%s", pod2.Namespace, pod2.Name)) framework.ExpectNoError(e2epod.DeletePodWithWait(client, pod2)) err = e2epv.DeletePersistentVolumeClaim(client, pvc2.Name, pvc2.Namespace) framework.ExpectNoError(err, "Failed to delete PVC: %s/%s", pvc2.Namespace, pvc2.Name) err = e2epv.DeletePersistentVolume(client, pv2.Name) framework.ExpectNoError(err, "Failed to delete PV: %s", pv2.Name) } </ins>

When two CSI PVs point to the same volume in the storage backend, kubelet can't start pods that use these two PVs on the same node. Both PVs have the same VolumeHandle and kubelet is smart enough to recognize that they're the same volume (because they have the same unique volume ID). It calls only once, however, the staging path contains name of one of the PVs, say PV1: One of the CSI volume calls then fails, because it uses PV2 to compute the staging path and that path does not exist. Both pods should just start. Using CSI HostPath volume, PV1 (dynamically provisioned for simplicity): PV2 (basically edited PV1): PVC1, PVC2, Pod1 and Pod2 are trivial, nothing special there. <detailsToday's master, </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage /priority backlog
I suggest that if two volumes have the same unique volume ID (computed from CSI ), they must have also the same global mount path. Kubelet already has code to call NodeStage once and bind-mount it to different pods, regardless how many different PVs they use. These are several issues changing the global mount path: uses PV parsed from the global path to get the PV and to get enough info to call NodeUnstage: However, this parsing is used only when the volume plugin is not able to find/parse its json it has saved during MountDevice: Can we safely assume that the json is always there? the global path in a running cluster is tricky. We can require nodes are drained when kubelet with new global mount path is installed. This way, no CSI volume is mounted to a global path when the new kubelet starts. Minor cluster upgrades (1.23 -1.24) are , also documented in . Or we could fix kubelet to use the old global path when it already exists. This looks fragile to me.

staging dir needs to be unique enough not to conflict with other CSI volumes. It should be based on the driver name + volume handle, but that could be quite long. Can we compute a hash and hope it won't conflict?
I think in-tree driver can handle this case correctly because global mountpath is using unique volume id instead of pv name. With csi-migration, does it break backward compatibility?
We require users to drain nodes before enabling migration or when updating to a Kubernetes version that have migration enabled by default. All volumes should be therefore unmounted.
probably yes this breaks backward compatibility, but since only migration for PD is being done maybe it's more rare? We saw a customer case for this situation with NFS shares using the csi filestore driver. For PD I think it would only come up when trying to ROX a PD volume across namespaces, I'm not sure if that's common.
Capturing some thoughts here in line to understand your we use a staging path like , the kubelet can locate the both the driver name and the volumeId, needed during UnmountDevice call. e.g for pd csi driver which is the sha256{projects/mattcary-saikatroyc-test/zones/us-central1-c/disks/pvc--f97d-43cd-a5f1-}. This means the volume Id needs to persisted (we already do that in today ), so that the volume Id can be retrieved for UnmountDevice call. we decide to make the change at a minor boundary, then a node drain mandate would unstage any volumes before the kubelet is upgraded. this ensures any staged path for csi volume is using the new format 2 If we do not want to enforce a node drain, kubelet can first lookup json data (the reasoning is if this file does not exist, we will not be able to retrieve the volume handle from the new handle anyway), else fallback to try the old format of staging path - . TBD: Need to check mount reference logic. sha(volumeId) is expected to be unique in the cluster, if I understand. (based on the assumption any storage resource provisioned by a given driver is expected to be uniquely identifiable)
/assign
yes, your summary is correct. We persist a json, please make sure it contains anything that NodeUnstage may need.

It seems to me that people prefer option 2.2. We already do to uniquely identify VolumeAttachment. So far nobody complained about any conflict.


Question: defer os.RemoveAll(srcFile) tests := map[string]struct { <ins> src         string </ins> dest        string expectedErr bool }{ "copy to directory": { <ins> src:         srcFile, </ins> dest:        "/tmp/", expectedErr: false, }, "copy to root": { <ins> src:         srcFile, </ins> dest:        "/", expectedErr: false, }, "copy to empty file name": { <ins> src:         srcFile, </ins> dest:        "", expectedErr: true, }, <ins> "copy unexisting file": { src:         path.Join(srcFile, "nope"), dest:        "/tmp", expectedErr: true, }, </ins> } for name, test := range tests { opts := NewCopyOptions(ioStreams) src := fileSpec{ <del> File: srcFile, </del> <ins> File: test.src, </ins> } dest := fileSpec{ PodNamespace: "pod-ns",

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Running silently fails if that source file doesn't exist. What you expected to happen: I expected kubectl to exit with a non-zero status code and to print a standard error message like so: How to reproduce it (as minimally and precisely as possible): Find any running pod, try to copy a non-existent file: Anything else we need to know?: works right. For example: I'm sure this has something to do with the fact that uses under the hood. However, itself will report an error if you try to extract a file that doesn't exist, so maybe this is just a matter of not including the to-be-extracted-path(s) on the end of the tar command line. This might be fixed by appending (or rather the basename of it) to the end of the array here: (haven't tried it). Environment: Kubernetes version (use ): and Cloud provider or hardware configuration: docker-for-desktop 2.0.4.1 () and GKE running 1.13.6-gke-0 OS (e.g: ): macOS 10.14.5 Kernel (e.g. ): Install tools: Installed docker from dmg, using the included Network plugin and version (if this is a network-related bug): Others:
/assign /sig cli
any update on this?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Same issue when the file doesn't exist on the pod
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Beep
Rotten issues close after 30d of inactivity.

Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
You can't reopen an issue/PR unless you authored it or you are a collaborator. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen </details
This is the first example where I found the kubectl cli to be sub-par (otherwise the experience has been excellent). Is anyone looking at this?
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen /remove-lifecycle rotten
Reopened this issue. <detailsIn response to : /reopen /remove-lifecycle rotten Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
For me never copies anything and never writes any info to console. Even if I point it to non-existing pod, it just exists with no trace.
I've opened with a fix.
: what version of kubectl are you using? I tried to upload a file to a non-existent pod and it failed with an error message (using the master branch) :
Issues go stale after 90d of inactivity. Mark the issue as fresh with .

Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
Any maintainer here? I had opened with a fix.
Would be great if would return a simple error in the CLI. Our pipeline was failing for a month and none realised because all the kubectl results were green :heavycheckmark:


Question: // handled by pod workers). go wait.Until(kl.podKiller, 1*time.Second, wait.NeverStop) <ins> // Start gorouting responsible for checking limits in resolv.conf if kl.resolverConfig != "" { go wait.Until(func() { kl.checkLimitsForResolvConf() }, 30*time.Second, wait.NeverStop) } </ins> // Start component sync loops. kl.statusManager.Start() kl.probeManager.Start()

Kubernetes version v1.3.0 Kubelet concatenates host and cluster domain entries for my pod, resulting in an invalid config (too many entries with duplicates). The max is only 6 search domains. Any more and DNS resolution breaks when trying to leverage search domains. The example below has only cluster DNS entries, but illustrates the issue: Host search nameserver 10.233.0.2 nameserver 10.90.0.1 options timeout:2 options attempts:2 Container kubectl run -i --tty busybox --image=busybox -- sh (skip waiting text) / # cat search nameserver 10.233.0.2 options attempts:2 options ndots:5 I should be enable cluster DNS config in my host DNS settings and still have functional pod DNS. Kubelet should be intelligent enough to filter out duplicate search domain entries (or drop search domains beyond the 6th entry)
I think some sane checks needs to be too. libc limits search field to 6 records and 255(?) characters. This limitation should be checked before creation.
Hey guys, I'm working on it.
Dup of


Question: Expect(c.Deployments(ns).Delete(deploymentName, nil)).NotTo(HaveOccurred()) }() <del> waitForDeploymentStatus(c, ns, deploymentName, 1, 0, 2) </del> <ins> err = waitForDeploymentStatus(c, ns, deploymentName, 1, 0, 2) Expect(err).NotTo(HaveOccurred()) </ins> // Verify that the pods were scaled up and down as expected. We use events to verify that. deployment, err := c.Deployments(ns).Get(deploymentName) Expect(err).NotTo(HaveOccurred())

cc
Thanks for flagging it! Its not flaky, it is broken as we did not enable deployments for that test suite. Sent to fix that.


Question: return statuses, nil } <ins> func toKubeContainerStatus(status *runtimeapi.ContainerStatus, runtimeName string) *kubecontainer.ContainerStatus { annotatedInfo := getContainerInfoFromAnnotations(status.Annotations) labeledInfo := getContainerInfoFromLabels(status.Labels) cStatus := &kubecontainer.ContainerStatus{ ID: kubecontainer.ContainerID{ Type: runtimeName, ID:   status.Id, }, Name:         labeledInfo.ContainerName, Image:        status.Image.Image, ImageID:      status.ImageRef, Hash:         annotatedInfo.Hash, RestartCount: annotatedInfo.RestartCount, State:        toKubeContainerState(status.State), CreatedAt:    time.Unix(0, status.CreatedAt), } if status.State != runtimeapi.ContainerState_CONTAINER_CREATED { // If container is not in the created state, we have tried and // started the container. Set the StartedAt time. cStatus.StartedAt = time.Unix(0, status.StartedAt) } if status.State == runtimeapi.ContainerState_CONTAINER_EXITED { cStatus.Reason = status.Reason cStatus.Message = status.Message cStatus.ExitCode = int(status.ExitCode) cStatus.FinishedAt = time.Unix(0, status.FinishedAt) } return cStatus } </ins> // generateContainerEvent generates an event for the container. func (m *kubeGenericRuntimeManager) generateContainerEvent(containerID kubecontainer.ContainerID, eventType, reason, message string) { ref, ok := m.containerRefManager.GetRef(containerID)

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): Pod status started null, started, 01 Jan 0001 Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-05-10T15:48:59Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-05-10T15:38:08Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"}` Environment: Cloud provider or hardware configuration: - OS (e.g. from /etc/os-release): CentOS 7 Kernel (e.g. ): Install tools: kubeadm 1.6 What happened: When a pod is created and starts running, shows the correct container start time in Containers -containername -State -Started. After the container is terminated, the start time changes to . In addition, shows that is . What you expected to happen: I'm expecting the start time to not change after the pod is terminated, so that afterwards I could check how long a time the container execution took. How to reproduce it (as minimally and precisely as possible): Have a running kubernetes cluster, and create a pod with the following command: kubectl create -f - <<EOF apiVersion: v1 kind: Pod metadata: name: sleep spec: containers: name: sleep image: ubuntu command: ["sleep"] args: ["30"] restartPolicy: Never EOF This created a container that sleeps for 30 seconds and quits.

During those 30 seconds, check the pod status with : ... State: Running Started: Wed, 17 May 2017 10:25:22 +0300 ... After 30 seconds, check the status of the terminated container with again: ... State: Terminated Reason: Completed Exit Code: 0 Started: Mon, 01 Jan 0001 00:00:00 +0000 Finished: Wed, 17 May 2017 10:25:52 +0300 ... With we see that the container's start time is null: ... "state": { "terminated": { "containerID": "docker://ca0dbe0dab3bb91ad0b05ac167e44d5fbf805f736ef98f61b9e106ba4456667f", "exitCode": 0, "finishedAt": "2017-05-17T07:25:52Z", "reason": "Completed", "startedAt": null } } ... Anything else we need to know: I need to measure container run durations and this bug prevents me from doing so.
/cc PR to fix this:


Question: host HostInterface, resourceAnalyzer stats.ResourceAnalyzer, auth AuthInterface, <del> tp oteltrace.TracerProvider, </del> kubeCfg *kubeletconfiginternal.KubeletConfiguration) Server { server := Server{

Follow-up to Discussed in As part of kubelet tracing beta, we should add tracing support to the kubelet's read-only endpoint, which currently does not have tracing. It should use WithPublicEndpoint to prevent callers from influencing sampling decisions. Part of
/sig node /sig instrumentation
/triage accepted /assign
can i help you with this issue?
I don't think we'll be able to start with this until merges
Ok, I'm waiting to see if I can help you. Thank you
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale is merged now, so this can be worked on.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
Could I help with this issue?
Sure. See We pass a NoOp Tracer provider to disable tracing for the endpoint. Instead, we need to pass the tracerprovider, but make sure that the eventual uses withPublicEndpoint. Looking closer, i'm not sure its currently possible to use WithPublicEndpoint with the otelrestful package... doesn't contain that option (unlike otelhttp: ), so we may need to add that upstream first.

Another thing to keep in mind is that updating otel libraries is currently blocked by (see my attempt in )
I gave the whole thing another try. At least locally, the dependencies seem now to be able to be updated. :raccoon: Could you maybe have a look at it :)
He's on leave for the time being.
oh, can you provide some feedback or point me to the right people?
I'm back now. Feel free to assign me to PRs related to this, and I'll review
Welcome back :) Could you re-run the CI ? Its just the dependencies. The PR itself shouldn't be that much afterwards.
otel is now up-to-date enough. here:
Hi, I would like to ask about one trouble that caused by WithPublicEndpoint. Aggregated APIServer that I customized uses kubernetes's RecommendedConfig. Due to the fact that WithPublicEndpoint is enabled, it can't be tracked by the full trace for its span-link model. And I think child-parent span model is more appropriate. So, could you please give me some advice? Looking forward to your reply.
thanks. We know about this, and are tracking it in


Question: deps = [ "//pkg/apis/core:go_default_library", "//pkg/registry/core/service/allocator:go_default_library", <del> "//vendor/k8s.io/utils/integer:go_default_library", "//vendor/k8s.io/utils/net:go_default_library", </del> ], )

What happened: All the services are asking to be recreated because the service CIDR changed after upgrading from 1.16.3 ro 1.17.0 What you expected to happen: Smooth migration from to , which would not break service CIDR. How to reproduce it (as minimally and precisely as possible): Do an upgrade from version 1.16.3 to 1.17.0 Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: OS (e.g: ):Kernel (e.g. ):Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig node
do you mean that the service cidr changed without any intervention after the upgrade? EDIT 96.0.0/12 is the default service cidr when using kubeadm --service-cidr string Default: "10.96.0.0/12" EDIT if not using kubeadm the default is May it be possible you were using another service cidr before the upgrade and you didn't set it during the upgrade, so it fall back to the default?
Can you provide your kube-apiserver invocation?
xref which warnings if no explicit CIDR flag was passed but should not have changed behavior otherwise
Where do you see that? The default is specified here as 10.0.0.0/24 and did not change between 1.16.0 and 1.17.0:
I was assuming it was using kubeadm based on the log message 96.0.0/12 is the default service CIDR for kubeadm
ah, ok. If they are using kubeadm, that default also did not change between 1.16.0 and 1.17.0:
a few questions for you: how did you install your 1.16.x cluster? did you customize the service cidr in that deployment? how did you upgrade your 1.16.x cluster to 1.17.x? did you apply the same service cidr customization in that deployment?
16.x was a straight and normal install no change on service CIRD, and we have never touched this CIDR configuration. And kubeadm is supposed to be used during the upgrade from 1.16.x to 1.17.x.

We just followed the describe process: The logs system started showing the shared message.
can you reproduce?
can you paste ? That will show you kubeadm config
created a single node cluster with v1.16.4 and a custom service cidr: applied upgrade to v1.17.0: i do not see a change in the custom service cidr.
if you never touched the service CIDR on the initial install it should have been which is the kubeadm default. your config map shows . how did you adapt the IP to work with the kubeadm default service cidr () if you initially deployed without modifying said cidr? to me this suggests that a change was required at some point.
The only change is the update from 1.16.3 to 1.17.0, done yesterday. Please find below a snapshot from our logs: <img width="1635" alt="Screen Shot 2019-12-21 at 22 22 29" src="https://user-"
my point is that if you did not apply a custom service cidr on "kubeadm init", then this cidr was the default out of kubeadm and after upgrade the service cidr was preserved as your config map shows. what was the elasticsearch/logging service IP before upgrade? EDIT: also do you have the YAML that was used to create this service?
101.249.43 belongs to the service rubnet range at minimum the error message is wrong /kind bug
are those errors happening constantly or only happened during the upgrade? are you able to create new services within that range? let's say, with a service IP of 10.101.249.45 Also, can you paste your current service-cluster-cidr configured in the controller-manager and apiserver? i.e.
this is a bug in core: this call in the repair controller returns for an IP that is actually in the CIDR range: also indicates a missing unit test case. latest commits: this might have regressed with the dual-stack changes? cc /remove-triage support needs-information /sig network /remove-sig cluster-lifecycle
/priority important-longterm
I was able to reproduce the same behavior again, with a fresh install k8s release 1.16.4.

Just After installing metrics server, we could notice a shift in the service CIDR, but nothing in the logs.
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/cc
so the problem here is that an IPv4 CIDR such as translates to addresses. but this line here clamps the maximum to 2^16 () so for an address like the logic in the ip allocator fails in the function: as the offset ends up bigger than the maximum. it feels like the maximums for IPv4 and IPv6 should be different and larger. otherwise for CIDRs in the notation : can give incorrect results for IPv4 can give incorrect results for IPv6 cc who touched related logic.
I think that was touched in … was this bug preexisting or did that change behavior?
before PR () i do not see a clamp for IPv4: only one for IPv6 at 16bits.
We should: revert add unit tests for the missed cases get that into 1.17.1 reintroduce the reliance on the function carefully
What would be the safest path to recreate all the services whilst waiting a final solution ? Especially for production K8S.
to mitigate the error in 1.17.0 you need to modify the serivce CIDR to be , instead of . modify the ConfigMap CIDR to have modify the kubeadm static pod manifest for the api server () to have edit problematic Service objects and give them an IP in the range if needed.
Thank you for those instructions, I'm affected by this, does this have anything to do with a possibility of ClusterIP's not properly responding or does it only print out alerts?
hello, can you test if the ClusterIP is not responding? i don't think we got a confirmation if this is only an alert. my guess is that it is just that, but there might be side effects.
i see you logged i will comment there.
How do you "recreate" a service? Literally delete and create again? Or is there a command which would cause it to re-create itself without having to be deleted?


Question: } var recreated bool // Wait until the next log change. <del> found, recreated, err = waitLogs(ctx, containerID, watcher, runtimeService) </del> <ins> found, recreated, err = waitLogs(ctx, containerID, baseName, watcher, runtimeService) </ins> if err != nil { return err }

won't output logs after kubelet rotate log filer of the container. kubectl output logs correctly after the log file is rotated. the config to . So kubelet will rotate the log soon. Build a image with this program a deploy with the previous image. the command No response <details</details<detailsNone </details<details</details<details</details<details</details<details</details
/sig node
/area kubelet
/triage accepted /priority important-longterm
/assign since you sent a PR already
Ran into this issue!


Question: if err != nil { t.Fatal(err) } <del> if matches { t.Errorf("cluster scoped resources (but not a namespace) should be exempted from all webhooks") </del> <ins> if !matches { t.Errorf("cluster scoped resources (but not a namespace) should not be exempted from webhooks") </ins> } }

If I am not mistaken, means that cluster-scoped resources other than namespaces are broken with admission webhooks. There is even a unit test checking that:
To clear any ambiguity around the word "broken", the current code behavior is that most cluster-scoped resources are not subjected to any webhooks. I wrote them like this because I was concerned with the potential bootstrapping deadlocks: a webhook that enacts policies for cluster-scoped resources might cause bootstrapping deadlocks for the cluster or for its own deployment. Thinking out loud, perhaps for cluster-scoped resources, the apiserver should only call the webhooks if the resources' labels match the , so that the critical bootstrapping resources can be excluded from the webhooks, thus breaking the bootstrapping deadlocks. This aligns with what we do for (see ). The downside is that the field name "namespaceSelector" becomes inappropriate.
I think we still need namespaceSelectors. We don't want to label pods. But I agree we actually want another objectLabelSelector. For namespaces these two coincide, i.e. we only have to define how they are applied if both are set or we forbid both at the same time. I tend to the later.
raised up a security issue regarding . A user can change labels to skip webhooks. doesn't have this issue because a user usually doesn't have permission to change the namespace's labels.
Users normally cannot create cluster-scoped objects either. But techincally it's a valid point. On the other hand, my main use-case are CRDs. They are not critical to the cluster health (maybe self-hosted clusters are different with etcd operator), i.e. you normally cannot shoot yourself into the foot with a webhook on them. For CRDs we do not need a selector at all. In other words: IMO we should enable webhooks for cluster-scoped objects, even if we have no in place.
Sounds reasonable. I'll create a PR ()
We would like to have the ability to apply the admission controller on specific namespaces only, and not on cluster level resources. This is useful if the list of possible resource Kinds that we want to apply the controller for is large, and we want to limit when it runs as much as possible to avoid affecting the whole cluster in case there's an issue.

I think this can be done by adding a SkipClusterResources flag (or something similar) in addition to the NamespaceSelector, to the Webhook struct. What do you think?
I think adding a flag is reasonable. It doesn't have the security issue as the idea does (). The expressiveness of is limited compared to , e..g., admins cannot use to run webhooks on some cluster-level resources while skipping others. However it's not a big drawback, because admins can specify more detailed GVK to limit the scope of webhook in that case.
Zeroth thought: s/SkipClusterResources/scope=[clusterboth]/g First thought: can't you just exclude cluster scoped resources, or include only namespace scoped resources in the webhook definition? Second thought: can a namespace selector be crafted that excludes the empty string?
0: Are you sure that's cleaner? The namespace selector picks which namespaces to use, and skipCluster decides whether to include cluster resources or not. Not setting it maintains current behavior (skipCluster=false) and whether cluster level resources are included is very clear. scope=cluster/namespace/both will require us to default to "both" which I find slightly more confusing. Also what to do if I specify cluster AND include namespace selector - is that an invalid spec or do we ignore the selector? 1: Yes. This is the current best thing we can do. If the list of resources is large though (for example, if I am implementing an alternative resource quota controller) then this solution requires us to maintain and update a long list of resources. This may be ok, since as long as our controller can't support certain types of resources it doesn't make sense to watch for them... 2: Doesn't look like it - the current code specifically checks if the attribute is a cluster level resource and skips the entire label matching process.
Re: 0: fair point, but let's at least name it in the positive: matchClusterScopedResources=true|false or something like that.
I proposed it in the negative to make it backwards compatible, so that "false" keeps current behavior. I agree that negatively named booleans are not not bad. I'll treat it as true if unspecified and use matchClusterScopedResources.
In today's semantic, such a namespace selector will exclude namespaced objects whose namespace has no label.


Question: } // disksAreAttached takes map of node and it's volumes and returns map of node, its volumes and attachment state <del> func disksAreAttached(nodeVolumes map[string][]string) (nodeVolumesAttachMap map[string]map[string]bool, err error) { </del> <ins> func disksAreAttached(nodeVolumes map[string][]string) (map[string]map[string]bool, error) { </ins> ctx, cancel := context.WithCancel(context.Background()) defer cancel()

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Following issue was observed while running the e2e test cases for CSI migration testing. <pre•! Panic [50.165 seconds] [sig-storage] vcp-performance [Feature:vsphere] vcp performance tests [It] Test Panicked assignment to entry in nil map Full Stack Trace (0x0, 0x0, 0x0) +0x105 panic(0x4b3a0e0, 0x5c329d0) +0x166 (0xc0032a3140, 0x0, 0x0, 0x0) </preWhat you expected to happen: Tests util methods should not panic How to reproduce it (as minimally and precisely as possible): Execute test suite which can be found here - Anything else we need to know?: The vsphere util method - is used by multiple test suites (Scale tests, performance tests) Environment: Kubernetes version (use ): <preClient Version: version.Info{Major:"1", Minor:"19+", GitVersion:"v1.19.0-alpha.3.1808+-dirty", GitCommit:"", GitTreeState:"dirty", BuildDate:"2020-07-04T03:10:03Z", GoVersion:"go1.14.4", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"19+", GitVersion:"v1.19.0-alpha.3.1808+", GitCommit:"", GitTreeState:"clean", BuildDate:"2020-07-02T20:49:42Z", GoVersion:"go1.14.4", Compiler:"gc", Platform:"linux/amd64"} </pre- Cloud provider or hardware configuration: vSphere Cloud Provider
/sig storage


Question: desc: "service with finalizer with timestamp", svc: &v1.Service{ ObjectMeta: metav1.ObjectMeta{ <ins> Finalizers: []string{servicehelper.LoadBalancerCleanupFinalizer}, </ins> DeletionTimestamp: &metav1.Time{ Time: time.Now(), }, <del> Finalizers: []string{servicehelper.LoadBalancerCleanupFinalizer, "unrelated"}, </del> }, }, expectNeedsCleanup: true,

What happened: Service Load balancer may not be cleaned up on type change if below sequence happen: type changes from to controller restarts/crashes before it gets a chance to process the update from 1. controller starts up again and sees the service doesn't want load balancer, hence skips it. load balancer resource sticks around even if it is no longer needed by the service. Note that this is a different issue than , as load balancer resource will still be cleaned up when the service object is deleted, just not when the type change happens. Anything else we need to know?: This is a corner case that is not covered by the initial service load balancer finalizer implementation (). This can probably be fixed in a trivial way --- make sure the handler takes in service with non-LoadBalancer type but has finalizer attached: I will put more thoughts later. cc
/sig network
Can i reserve this issue in advance? :) I'd like to take this issue as my first issue.
Sure thing, feel free to work on this and glad that you can help :)
Do you mean attach finalizer for service that non-LoadBalancer type in AddFunc? I'm new to kubernetes, hope you can guide me. :)
No worries. I actually meant should include service with non-LoadBalancer type && with finalizer attached. As we only attach finalizer to a service only when we are creating a load balancer for it, a finalizer presents on the service (no matter what type it has) indicates there is a load balancer associated. Hence it makes sense that we should clean up the load balancer if the service no longer wants one, even if the service has not been deleted.
To clarify further, "attach finalizer for service that non-LoadBalancer type in AddFunc" would not be what we want --- we shouldn't attach finalizer to service that we will not be creating load balancer for.
After go through service controller part code, i guess you mean add a condition for as follows: . Am i right? I'm not sure when service type changes if can be involved. Can you clarify this ?
if ok && (wantsLoadBalancer(svc) (svc.Spec.Type != v1.ServiceTypeLoadBalancer && servicehelper.HasLBFinalizer(svc))) . Yea this was what I meant.

On service type change, will be involved instead of . That part is already handled. This issue is explicit for the case when controller fails to handle the type change update before it exiting (e.g. crashing, restart, etc.). Putting such detection to is more off a remedy, it means the next time service controller starts up, it can re-syncs all existing services and find out it still need to cleanup load balancer for this service, even if it doesn't have load balancer type anymore.
Thanks . I have sent a PR assined to you. I think it should be checked in function . It is the case that need to clean up whatever the reason is.


Question: } func getPodStatus(pod *api.Pod, minions client.MinionInterface) (api.PodStatus, error) { <del> if pod.CurrentState.Info == nil || pod.CurrentState.Host == "" { </del> <ins> if pod.CurrentState.Host == "" { </ins> return api.PodWaiting, nil } res, err := minions.ListMinions()

to reproduce: create vagrant kubernetes with 5 minions. try guest book example on it. After provisioning: then kill 10.245.2.6 minion: then we get: Nothing changes even after waiting for significant period of time. Please note, that while redis-master wasn't in replication controller and it's not expected for it to restart, it's still wrongfully marked as Running.


Question: f.ClientSet.RbacV1().RoleBindings(f.Namespace.Name).Create(context.TODO(), binding, metav1.CreateOptions{}) } <ins> func bindClusterRBACRoleToServiceAccount(f *framework.Framework, serviceAccountName, rbacRoleName string) { binding := &rbacv1.ClusterRoleBinding{ ObjectMeta: metav1.ObjectMeta{ Name:      f.Namespace.Name + "-rbac-binding", Namespace: f.Namespace.Name, }, Subjects: []rbacv1.Subject{ { Kind:      "ServiceAccount", Name:      serviceAccountName, Namespace: f.Namespace.Name, }, }, RoleRef: rbacv1.RoleRef{ APIGroup: "rbac.authorization.k8s.io", Kind:     "ClusterRole", Name:     rbacRoleName, }, } f.ClientSet.RbacV1().ClusterRoleBindings().Create(context.TODO(), binding, metav1.CreateOptions{}) } </ins> // createPodWithGmsa creates a pod using the test GMSA cred spec, and returns its name. func createPodWithGmsa(f *framework.Framework, serviceAccountName string) string { podName := "pod-with-gmsa"

Cluster api provides a framework to run the e2e tests which uses the conformance image: The gmsa tests use bash to install the webhook: The test will fail when run in the conformance image because of the bash: we should install the component another way run this e2e conformance image with /sig windows fyi <details</details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Conformance image:
Hi does the "work e2e" test case pass? Deploying GMSA webhook is the same for both "work e2e" and this test case.
no, they both fail due to bash not being present in the conformance image. The gmsa tests themselves pass if I run locally
/sig testing


Question: func alwaysAlice(req *http.Request) (*authenticator.Response, bool, error) { return &authenticator.Response{ User: &user.DefaultInfo{ <del> Name: "alice", </del> <ins> Name:   "alice", UID:    "alice", Groups: []string{user.AllAuthenticated}, </ins> }, }, true, nil }

While moving APF to beta (see comments on ), we discovered that the integration tests commonly present users which are not members of any groups. We had thought that the system enforced membership in either the authenticated or unauthenticated group. If this is supposed to be a system invariant (as we thought), we'd like to ask SIG Auth: Please fix the integration tests Please add e.g. logging so that if real, operational clusters hit this there's at least a log message and people will file bugs. /sig auth
Can I suggest also, if this is a real invariant, that it be documented in some place(s) likely to be read?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale /priority important-longterm /triage accepted
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/lifecycle frozen
fixes the authenticator configured in one of the integration test API server startup helpers which resolved this issue for . Remaining tests that still need fixing that are encountering unexpected users look to be ones set up to specifically test auth paths:
open to fix remaining integration test users


Question: } func (s *Plugin) mountServiceAccountToken(serviceAccount *corev1.ServiceAccount, pod *api.Pod) error { <del> // Find the name of a referenced ServiceAccountToken secret we can mount serviceAccountToken, err := s.getReferencedServiceAccountToken(serviceAccount) if err != nil { return fmt.Errorf("Error looking up service account token for %s/%s: %v", serviceAccount.Namespace, serviceAccount.Name, err) </del> <ins> var ( // serviceAccountToken holds the name of a secret containing a legacy service account token serviceAccountToken string err                 error ) if !s.boundServiceAccountTokenVolume { // Find the name of a referenced ServiceAccountToken secret we can mount serviceAccountToken, err = s.getReferencedServiceAccountToken(serviceAccount) if err != nil { return fmt.Errorf("Error looking up service account token for %s/%s: %v", serviceAccount.Namespace, serviceAccount.Name, err) } </ins> } <del> if len(serviceAccountToken) == 0 { </del> <ins> if len(serviceAccountToken) == 0 && !s.boundServiceAccountTokenVolume { </ins> // We don't have an API token to mount, so return if s.RequireAPIToken { // If a token is required, this is considered an error

What happened: We enabled the feature gate on to start switching from non-expiring to expiring ServiceAccount tokens as . It works well. We then wanted to completely get rid of the "legacy" non-expiring ServiceAccount token Secrets. To do that we configured with (so it wouldn't auto-create the Secrets), removed the existing Secrets and removed the reference from the respective ServiceAccount to the Secret. This works as expected: No non-expiring token Secrets are generated. However, when creating a Pod we now see: This happens because the rejects the Pod because it cannot find a ServiceAccount token Secret (because it doesn't exist anymore as intended). What you expected to happen: When using the feature the legacy ServiceAccount token Secrets aren't used anymore, so a Pod should not be rejected if the Secret doesn't exist. The Pod doesn't even use the Secret at all. How to reproduce it (as minimally and precisely as possible): Enable feature gate Start with Create a new ServiceAccount Deploy a dummy app using that ServiceAccount Observe no Pod being created with Describe the corresponding ReplicaSet to see the above Error event. Anything else we need to know?: Looking at the it seems that at least one legacy Secret must exist in order for the Pod to get admitted. This is enforced even when and we're relying on projected token volumes. Following the code path when is enabled reveals that the token is never going to be used later on as there's no SecretVolume to be mounted in . In other words:  is just fine when . During the transition period it's fine to have the legacy Secrets lying around but ultimately it's desirable to get rid of them because as far as I can tell they are still happily accepted by the API server. Environment: Kubernetes version: tested on but code seems the same in and Cloud provider or hardware configuration: OS (e.g: ): Ubuntu 18.04.3 LTS Kernel (e.g. ): 4.15.0-1054-aws Install tools: /kind bug /sig auth
Fixed in
Thank you!


Question: // ServiceAnnotationLoadBalancerResourceGroup is the annotation used on the service // to specify the resource group of load balancer objects that are not in the same resource group as the cluster. ServiceAnnotationLoadBalancerResourceGroup = "service.beta.kubernetes.io/azure-load-balancer-resource-group" <ins> // ServiceAnnotationAllowedServiceTag is the annotation used on the service // to specify a list of allowed service tags separated by comma ServiceAnnotationAllowedServiceTag = "service.beta.kubernetes.io/azure-allowed-service-tags" ) var ( // supportedServiceTags holds a list of supported service tags on Azure. // Refer https://docs.microsoft.com/en-us/azure/virtual-network/security-overview#service-tags for more information. supportedServiceTags = sets.NewString("VirtualNetwork", "VIRTUAL_NETWORK", "AzureLoadBalancer", "AZURE_LOADBALANCER", "Internet", "INTERNET", "AzureTrafficManager", "Storage", "Sql") </ins> ) // GetLoadBalancer returns whether the specified load balancer exists, and

Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: When using the loadBalancerSourceRanges property of a service in a manifest, we are force to use a CIDR block. This was discovered while debugging the following issue. What you expected to happen: Azure allows us to use Service Tags (e.g. VirtualNetwork, Internet, etc) WE should be able to specify them form a k8s service manifest. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): 1.7.11 Cloud provider or hardware configuration: Azure OS (e.g. from /etc/os-release): centOS 7.3 Kernel (e.g. ): 3.10 Install tools: Others:
/sig azure
Here's the requested Feature.
SourceRanges is part of service spec, while serviceTag is azure specific. I think we don't expect to add cloud-specific features into Kubernetes API. What about adding a new annotation for this? e.g.
I believe that at this stage, (no out of tree cloud provider logic), using an annotation would be the only way to go unfortunately.
/priority important-soon
/status approved-for-milestone
[MILESTONENOTIFIER] Milestone Issue: Up-to-date for process <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. : New functionality. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details


Question: } func (rc *resourceMetricsCollector) collectNodeMemoryMetrics(ch chan<- metrics.Metric, s summary.NodeStats) { <del> if s.Memory == nil { </del> <ins> if s.Memory == nil || s.Memory.WorkingSetBytes == nil { </ins> return }

kubelet panic when accessing metrics/resource endpoint kubelet log kubelet works fine access metrics/resource endpoint of kubelet No response <details</details<details</details<details</details<details</details<detailsINFO[2022-07-14 05:36:21.474173892Z] Starting CRI-O, version: 1.23.0, git: (dirty) Version: 1.23.0 GitCommit: GitTreeState: dirty BuildDate: 1980-01-01T00:00:00Z GoVersion: go1.17.4 Compiler: gc Platform: linux/amd64 Linkmode: static BuildTags: apparmor, excludegraphdriverdevicemapper, seccomp, selinux SeccompEnabled: true AppArmorEnabled: true </details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/sig node


Question: // the user and has network stats. podStats.Network = cadvisorInfoToNetworkStats(&cinfo) } else { <del> podStats.Containers = append(podStats.Containers, *cadvisorInfoToContainerStats(containerName, &cinfo, &rootFsInfo, &imageFsInfo)) </del> <ins> containerStat := cadvisorInfoToContainerStats(containerName, &cinfo, &rootFsInfo, &imageFsInfo) // NOTE: This doesn't support the old pod log path, `/var/log/pods/UID`. For containers // using old log path, they will be populated by cadvisorInfoToContainerStats. podUID := types.UID(podStats.PodRef.UID) logs, err := p.hostStatsProvider.getPodContainerLogStats(podStats.PodRef.Namespace, podStats.PodRef.Name, podUID, containerName, &rootFsInfo) if err != nil { klog.ErrorS(err, "Unable to fetch container log stats", "containerName", containerName) } else { containerStat.Logs = logs } podStats.Containers = append(podStats.Containers, *containerStat) </ins> } }

Rotated container logs space usage doesnt seem to be counted towards the epehemral storage limit. Always only current log file stats are taken into account. rotated log files usage also should be counted towards ephemeral-storage usage No response <detailsTested on Kubernetes Version : 1.21.7, 1.22 Container Runtime : Cri-o </details<details</details<details</details<details</details<details</details<details</details
/sig node
/cc
/assign we can see if there is a issue
Were the rotated log files in /var/log/pods generated by yourself(not by k8s)? The cadvisor can only collect log files on container's filesystem.
No, I have not created any files. they are generated by k8s itself.
Okay, I see. As the cri-o runtime uses cadvisor stats provider to get container stats, it caculates container logUsage by formular . but the default root dir or log dir of cri-o is not /var/log/pods/{podName}/, so theses log files are not on container's filesystem, then cadvisor may not caculate the data accurately. /cc the containerd uses cri stats provider and gets logstats by HostStatsProvider, it can get right data.
In the kubelet code, , the log path is in the format , but the directory of your environment is , I did not see , Did you set this path yourself?
sorry I truncated the name for better readability, the actual path name is . So I suppose its taking in the above format only.
log files gets generated in the path /var/log/pods/defaultephem15334a37d-df02-435c-b385-/con2 for cri-o in my setup
/area cadvisor /triage accepted /priority important-longterm
/retitle Rotated container log files size not counted towards ephemeral-storage's limit on CRI-O
hi, In the past two days, I have studied the collection of file information by cAdvisor. docker and cri-o runtime basically have the same mechanism in cAdvisor. I have verified that when the runtime is cri-o, the log information statistics are also correct. So I need you to provide more detailed information. I found that the command you show is: , please help to confirm whether the "logs" information belongs to the container con2.

or use the command to try
I have verified and info provided is for con2 only . I have the sample yaml used for testing , please let me know if I'm doing it right or anything missing. I have set the kubelet log configuration as containerLogMaxSize: "5Ki", containerLogMaxFiles: 5 After applying the above yaml stats summary shows con1 logs usedbytes always 0 , con2 logs usedbytes as 4096
attached the stats summary of the command And I'm testing on kubernetes 1.22 release
/assign
what version of CRI-O/kubernetes are you using? using latest main of both kubernetes and cri-o with your reproducer causes the container to be evicted because of it exceeding ephemeral storage
oh I see now: my mistake
another mistake: I was using CRI stats provider, instead of cAdvisor stats provider (as is default)
Is CRI stats provider used by default in 1.23 by default? And also problem is seen only if containerLogMaxSize is less that the ephemeral storage limit set , if its more than the ephemeral storage limit , its the right behavior , pod gets evicted in my setup.
at long last, I believe I've found the issue!


Question: return &ValidationError{ValidationErrorTypeNotFound, field, value, ""} } <del> func NewFieldTooLong(field string, value interface{}) *ValidationError { return &ValidationError{ValidationErrorTypeTooLong, field, value, ""} </del> <ins> func NewFieldTooLong(field string, value interface{}, maxLength int) *ValidationError { return &ValidationError{ValidationErrorTypeTooLong, field, value, fmt.Sprintf("must have at most %d characters", maxLength)} </ins> } type ValidationErrorList []error

For example, creating a service with id 'redis-master10012345678999' fails and the returned message contains the following: "message": "service "redis-master10012345678999" is invalid: name: invalid value 'redis-master10012345678999' 'invalid value' is too general, users should understand exactly what is wrong with the input. for example - is it too long? does it contain characters that are not allowed?
Agree - there is now a place to stick the explanations in the ValidationError - we need to link together validation logic and that string. , abonas wrote:
I checked and it seems there are only two errors that do not have detailed explanation. I'll send a PR to fix this.
This is one:
Thanks for the pointer Brian! I'll send second PR to fix this one.
Still second PR to be sent.


Question: importpath = "k8s.io/kube-aggregator/pkg/controllers/autoregister", deps = [ "//staging/src/k8s.io/apimachinery/pkg/api/errors:go_default_library", <ins> "//staging/src/k8s.io/apimachinery/pkg/apis/meta/v1:go_default_library", </ins> "//staging/src/k8s.io/apimachinery/pkg/labels:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/util/runtime:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/util/wait:go_default_library",

k8s version: v1.9.3 When I create customresourcedefinition, There is error like Operation cannot be fulfilled on "": the object has been modified; please apply your changes to the latest version and try again . apiVersion: kind: CustomResourceDefinition metadata: name: annotations: "": crd-install labels: app: istio-pilot spec: group: names: kind: VirtualService listKind: VirtualServiceList plural: virtualservices singular: virtualservice categories: istio-io networking-istio-io scope: Namespaced version: v1alpha3 So what is the reason of it?
a few questions: where are you seeing the error, in the apiserver log or in the output of the kubectl command? what kubectl command are you using to create the object? are you running HA apiservers?
The log is shown by apiserver. I run k8s in HA apiservers. The command I used is "kubectl apply -f "+file.
API servers attempt to update CRD status to indicate whether the names were available. In an HA apiserver setup, the API servers race to do so, and all but the first lose the race with a resource conflict error. That is not unexpected. Opened to avoid logging normal API responses in that scenario.
Thanks ! But I want to know whether the CRD has been set successfully in HA apiservers ? I use the command " kubectl get crd -o yaml" and get the following results. ! I see its status is true. But when i create a virtualservices, it returns error. Error from server (Timeout): error when creating "cms-api-": Timeout: request did not complete within allowed duration ! So i want to know how to solve this error.
/assign
Excuse me. Do you have some greate ideas to solve this issue??
There is a PR for this sent by
I see your request is timed-out not failed. In HA, I expect the request to fail in the first seconds of the CRD creation, but after that it should go through. I need more logs during this timeout or a consistent way to reproduce it.


Question: importpath = "k8s.io/kubernetes/test/e2e/framework/events", visibility = ["//visibility:public"], deps = [ <del> "//staging/src/k8s.io/api/core/v1:go_default_library", </del> "//staging/src/k8s.io/apimachinery/pkg/apis/meta/v1:go_default_library", <del> "//staging/src/k8s.io/apimachinery/pkg/types:go_default_library", </del> "//staging/src/k8s.io/apimachinery/pkg/util/wait:go_default_library", "//staging/src/k8s.io/client-go/kubernetes:go_default_library", <del> "//test/e2e/framework:go_default_library", "//vendor/github.com/onsi/ginkgo:go_default_library", </del> ], )

[x] Original approval issue: [x] Two weeks soak end date: 6 July 2020 This issue is created to allow edit by /wip /hold According to this APIsnoop query, there are still an endpoint which is untested. a set of 3 Events with a static label within a Namespace. that all 3 Events with the label in the Namespaces are created. the set of Namespaced Events with a label via DeleteCollection. that all Events with the label have been deleted. Discover useragents: List endpoints hit by the test: Display endpoint coverage change: Two other endpoints tested here have tests in progress to conformance already. If a test with these calls gets merged, test coverage will go up by 1 points This test is also created with the goal of conformance promotion.
/sig testing /sig architecture /area conformance
During Conformance Meeting: Is there a way to write this test so it is resource agnostic? If we are just changing the resource... can be parameterize the resource-type
/assign
GitHub didn't allow me to assign the following users: heyste. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen PR Merge and soaking
Reopened this issue. <detailsIn response to : /reopen PR Merge and soaking Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: f.lbScheme == other.lbScheme && equalStringSets(f.ports, other.ports) && f.backendService == other.backendService && <del> f.allowGlobalAccess == other.allowGlobalAccess </del> <ins> f.allowGlobalAccess == other.allowGlobalAccess && f.subnetwork == other.subnetwork </ins> } // toForwardingRuleComposite converts a compute beta or GA ForwardingRule into the composite type

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: ILB ip addresses come from the subnet that the K8s cluster uses. There is no ability to specify a different subnet. What you expected to happen: ILB ip addresses should be allocated from a different subnet if specified. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: GCE OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/area platform/gce /assign
/sig network


Question: - monit - nginx - kube-client-tools <ins> {% if grains['cloud'] is defined and grains['cloud'] != 'vagrant' %} </ins> - logrotate <ins> {% endif %} </ins> - kube-addons {% if grains['cloud'] is defined and grains['cloud'] == 'azure' %} - openvpn

It appears that the provision-network script that is run on the kubernetes-master can cause salt provisioning on the minions to fail or block in a forever retry timeout error loop. I think it's because of the service network.restart that happens after openvswitch is installed in provision-network. I only say this because when I disable installing docker on kubernetes-master in vagrant, I do not have an issue. - is there a different way we can structure this on the kubernetes-master? For example, could we run the script in or something?


Question: "k8s.io/apimachinery/pkg/api/resource" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/types" <ins> utilfeature "k8s.io/apiserver/pkg/util/feature" "k8s.io/kubernetes/pkg/features" </ins> "k8s.io/kubernetes/pkg/util/mount" utilstrings "k8s.io/kubernetes/pkg/util/strings" "k8s.io/kubernetes/pkg/volume"

FEATURE REQUEST: vSphere volume plugin does not support volume device yet. Need to add volumeMode in PV spec as other plugin did. release 1.12
/sig storage


Question: "io/ioutil" "log" "math/rand" <ins> "net" </ins> "net/http" <del> "net/url" </del> "os" "sync" "time" <del> "github.com/GoogleCloudPlatform/kubernetes/pkg/api/latest" </del> "github.com/GoogleCloudPlatform/kubernetes/pkg/client" "github.com/GoogleCloudPlatform/kubernetes/pkg/util" )

test panicked: (root)/Kubernetes%%20suite/Monitoringverifymonitoringpodsandallclusternodesareavailableoninfluxdbusingheapster/ cc
Vish please take a look if you have cycles or reassign if there's someone with better context.
I looked at the panic and it was due to the last test (networking) timing out but still running. I think the fix is more in better cleanup between tests.
I just saw this... it's totally due to ginkgo's cleanup when a test times out. I can fix.
I will get this one.
Ugh, the thing ginkgo does when a test times out is totally lame...
Instead of working around this, I've filed I'm going to go back to a manual timeout on this test.


Question: host HostInterface, resourceAnalyzer stats.ResourceAnalyzer, address net.IP, <del> port uint) { </del> <ins> port uint, tp oteltrace.TracerProvider) { </ins> klog.InfoS("Starting to listen read-only", "address", address, "port", port) <del> // TODO: https://github.com/kubernetes/kubernetes/issues/109829 tracer should use WithPublicEndpoint s := NewServer(host, resourceAnalyzer, nil, oteltrace.NewNoopTracerProvider(), nil) </del> <ins> s := NewServer(host, resourceAnalyzer, nil, nil) if utilfeature.DefaultFeatureGate.Enabled(features.KubeletTracing) { s.InstallTracingFilter(tp, otelrestful.WithPublicEndpoint()) } </ins> server := &http.Server{ Addr:           net.JoinHostPort(address.String(), strconv.FormatUint(uint64(port), 10)),

Follow-up to Discussed in As part of kubelet tracing beta, we should add tracing support to the kubelet's read-only endpoint, which currently does not have tracing. It should use WithPublicEndpoint to prevent callers from influencing sampling decisions. Part of
/sig node /sig instrumentation
/triage accepted /assign
can i help you with this issue?
I don't think we'll be able to start with this until merges
Ok, I'm waiting to see if I can help you. Thank you
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale is merged now, so this can be worked on.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
Could I help with this issue?
Sure. See We pass a NoOp Tracer provider to disable tracing for the endpoint. Instead, we need to pass the tracerprovider, but make sure that the eventual uses withPublicEndpoint. Looking closer, i'm not sure its currently possible to use WithPublicEndpoint with the otelrestful package... doesn't contain that option (unlike otelhttp: ), so we may need to add that upstream first.

Another thing to keep in mind is that updating otel libraries is currently blocked by (see my attempt in )
I gave the whole thing another try. At least locally, the dependencies seem now to be able to be updated. :raccoon: Could you maybe have a look at it :)
He's on leave for the time being.
oh, can you provide some feedback or point me to the right people?
I'm back now. Feel free to assign me to PRs related to this, and I'll review
Welcome back :) Could you re-run the CI ? Its just the dependencies. The PR itself shouldn't be that much afterwards.
otel is now up-to-date enough. here:
Hi, I would like to ask about one trouble that caused by WithPublicEndpoint. Aggregated APIServer that I customized uses kubernetes's RecommendedConfig. Due to the fact that WithPublicEndpoint is enabled, it can't be tracked by the full trace for its span-link model. And I think child-parent span model is more appropriate. So, could you please give me some advice? Looking forward to your reply.
thanks. We know about this, and are tracking it in


Question: return err } <ins> if o.ServerSideApply && o.DeleteOptions.ForceDeletion { return fmt.Errorf("--force cannot be used with --server-side") } </ins> if o.DryRunStrategy == cmdutil.DryRunServer && o.DeleteOptions.ForceDeletion { return fmt.Errorf("--dry-run=server cannot be used with --force") }

What happened: A resource is not replaced (deleted and recreated) when passing the flag to and the change cannot be applied. What you expected to happen: I expected the service to be replaced. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ):
/sig api-machinery /wg api-expression
/sig cli
/cc
Good to know, that's certainly worthy of a fix.
/cc
This is slightly related: If we want to support we could try and make the behavior clearer.
Can you weigh in on how important this is? We feel that is super, super dangerous in general. It's also very easy to confuse it with for server side apply, so it's extra dangerous. Do we need to implement this field for this command? Can we give an error, like "use one of: --force-conflicts or --delete-then-create"? Can SIG CLI do something about this flag in general?
/cc
Noted, I'll make sure this is in the agenda for the next sig-cli meeting.
I don't think the two are related. is now error-ing when used with , but it'll still fail with server-side apply.
You're right... my recall was bad. I deleted my comment where I thought was this, but it is not. Sorry about that
Looks like in the SIG-CLI meeting it was decided to not enable the user to use the and flag together. kubectl should fail immediately with a warning that should not be used with server side apply. Do you mind if I start working on this issue? I will reach out to you and other members of the SIG to get more details needed. I am a new contributor to k8s and would like to work on this issue.
Thanks for summarizing the discussion! I'd love to see you submit a PR for that, that's awesome!
/assign /lifecycle-active


Question: if requestInfo.IsResourceRequest && longRunningSubresources.Has(requestInfo.Subresource) { return true } <ins> if !requestInfo.IsResourceRequest && strings.HasPrefix(requestInfo.Path, "/debug/pprof/") { return true } </ins> return false } }

Is this a BUG REPORT or FEATURE REQUEST?: What happened: In the log: What you expected to happen: The timeout should be more relaxed on /debug/pprof endpoints. How to reproduce it (as minimally and precisely as possible):
/assign


Question: outBuiltIn := e2ekubectl.RunKubectlOrDie("", "get", "nodes", node.Name) ginkgo.By(fmt.Sprintf("calling kubectl get nodes %s --subresource=status", node.Name)) outStatusSubresource := e2ekubectl.RunKubectlOrDie("", "get", "nodes", node.Name, "--subresource=status") <del> gomega.Expect(outBuiltIn).To(gomega.Equal(outStatusSubresource)) </del> <ins> // Avoid comparing values of fields that might end up // changing between the two invocations of kubectl. requiredOutput := [][]string{ {"NAME"}, {"STATUS"}, {"ROLES"}, {"AGE"}, {"VERSION"}, {node.Name},                           // check for NAME {""},                                  // avoid comparing STATUS {""},                                  // avoid comparing ROLES {""},                                  // avoid comparing AGE {node.Status.NodeInfo.KubeletVersion}, // check for VERSION } checkOutput(outBuiltIn, requiredOutput) checkOutput(outStatusSubresource, requiredOutput) </ins> }) }) })

Example: this PR seems to be flaking: comparing age of a node from the output of two commands: Kubernetes e2e suite: [It] [sig-cli] Kubectl client kubectl subresource flag GET on status subresource of built-in type (node) returns identical info as GET on the built-in type expandless 5 hours back No response No response No response_ /sig cli
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/priority important-soon
it seems that is comparing subresources and there is something there that changes over time As Sergey correctly points out is the Age The test can not assume the executions will be done in the same second
/assign


Question: _, err := runCommand("sh", "-c", cmd) return err } <ins> func systemctlDaemonReload() error { cmd := "systemctl daemon-reload" _, err := runCommand("sh", "-c", cmd) return err } var ( dbusConfPath = "/etc/systemd/system/dbus.service.d/k8s-graceful-node-shutdown-e2e.conf" dbusConf     = ` [Unit] RefuseManualStart=no RefuseManualStop=no [Service] KillMode=control-group ExecStop= ` ) func overlayDbusConfig() error { err := os.MkdirAll(filepath.Dir(dbusConf), 0755) if err != nil { return err } err = os.WriteFile(dbusConfPath, []byte(dbusConf), 0644) if err != nil { return err } return systemctlDaemonReload() } func restoreDbusConfig() error { err := os.Remove(dbusConf) if err != nil { return err } return systemctlDaemonReload() } </ins>

/kind failing-test <!-- If this is a flake, please add: /kind flake --/sig node /cc
The failure should be caused by
/assign
/triage accepted /assign


Question: } } <ins> func TestRunArgsFollowDashRules(t *testing.T) { _, _, rc := testData() tests := []struct { args          []string argsLenAtDash int expectError   bool name          string }{ { args:          []string{}, argsLenAtDash: -1, expectError:   true, name:          "empty", }, { args:          []string{"foo"}, argsLenAtDash: -1, expectError:   false, name:          "no cmd", }, { args:          []string{"foo", "sleep"}, argsLenAtDash: -1, expectError:   false, name:          "cmd no dash", }, { args:          []string{"foo", "sleep"}, argsLenAtDash: 1, expectError:   false, name:          "cmd has dash", }, { args:          []string{"foo", "sleep"}, argsLenAtDash: 0, expectError:   true, name:          "no name", }, } for _, test := range tests { f, tf, codec := NewAPIFactory() tf.Client = &fake.RESTClient{ Codec: codec, Client: fake.CreateHTTPClient(func(req *http.Request) (*http.Response, error) { return &http.Response{StatusCode: 201, Body: objBody(codec, &rc.Items[0])}, nil }), } tf.Namespace = "test" tf.ClientConfig = &client.Config{} cmd := NewCmdRun(f, os.Stdin, os.Stdout, os.Stderr) cmd.Flags().Set("image", "nginx") err := Run(f, os.Stdin, os.Stdout, os.Stderr, cmd, test.args, test.argsLenAtDash) if test.expectError && err == nil { t.Errorf("unexpected non-error (%s)", test.name) } if !test.expectError && err != nil { t.Errorf("unexpected error: %v (%s)", err, test.name) } } } </ins> func TestGenerateService(t *testing.T) { tests := []struct {

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: ci-kubernetes-e2e-gci-gce-alpha-enabled-default Which test(s) are failing: CSI hostpath-v0 tests Since when has it been failing: 8/8 Testgrid link: https://k8s- Reason for failure: Anything else we need to know: We are removing CSI 0.3 support in 1.18, which is before selflink is going to be removed in 1.20. So we should just disable hostpath-v0 from alpha tests. /assign
/milestone v1.16


Question: // Partially update a node using strategic merge patch kubectl patch node k8s-node-1 -p '{"spec":{"unschedulable":true}}' <ins> // Update a container's image; spec.containers[*].name is required because it's a merge key kubectl patch pod valid-pod -p '{"spec":{"containers":[{"name":"kubernetes-serve-hostname","image":"new image"}]}}' </ins> ``` ### Options

The current documentation is very vague and doesn't explain what can be updated in this way. Can I rename or stop the pod, change number of replicas in RC etc. For example it seems to be impossible to change pod name:
cc
After an in person discussion with we think the docs should point people to the swagger-ui to find if a field is mutable. I'll add this point into the docs.


Question: } }` <ins> crdApplyFinalizerBody = ` { "apiVersion": "%s", "kind": "%s", "metadata": { "name": "%s", "finalizers": %s }, "spec": { "knownField1": "val1", "ports": [{ "name": "portName", "containerPort": 8080, "protocol": "TCP", "hostPort": 8082 }], "embeddedObj": { "apiVersion": "v1", "kind": "ConfigMap", "metadata": { "name": "my-cm", "namespace": "my-ns" } } } }` </ins> patchYAMLBody = ` apiVersion: %s kind: %s metadata: name: %s finalizers: <del> - test-finalizer </del> <ins> - test/finalizer </ins> spec: cronSpec: "* * * * */5" ports:

A client is allowed to add custom finalizers to Custom Resources with a finalizer name that's not fully qualified (e.g. or ). However they're prevented from doing the same on builtin resource types, returning the following error: Invalid value: "": name is neither a standard finalizer name nor is it fully qualified Per , custom finalizer names are expected to be in format "consist of a domain name, a forward slash and the name of the finalizer" (e.g. ). Therefore, I expected the enforcement to be the same for CRs and builtin types. a custom resource on the cluster it to add the edit does not fail the same on a builtin resource (e.g. Namespace or Pod), observe it fails No response Reproducible at least starting from 1.21 and still on latest versions. <details</details<detailsN/A kind cluster </details<detailsN/A - kind cluster accessed via macOS </details<details</details<details</details<details</details
/area api /cc /sig api-machinery
/assign /triage accepted
There is no validation of finalizer in and . Please unassign me if you guys have started to work on it. /assign
it is called from here on custom resources:
lacks validation for finalizer name. I have raised a pr for this. PTAL.


Question: Type:    "DisruptionTarget", Status:  "True", Reason:  "TerminationByKubelet", <del> Message: "The node was low on resource: ephemeral-storage. ", </del> <ins> Message: "The node was low on resource: ephemeral-storage. Threshold quantity: 2Gi, available: 1536Mi. ", </ins> }) }

when the pod is evicted due to low imagefs.available, the prompt is difficult to understand. it showed: But the container was evicted not by exceeding its request of 0 ephemeral-storage, but due to reaching the evictionHard of the node. The correct prompt message points out the reason for the pod eviction. don't print "which exceeds its request of 0.", because the container was evicted not by exceeding its request of 0. % oc label mcp worker custom-kubelet-eviction=small-pods evictionHard by creating a kubeletconfig % oc create -f custom-kubelet- custom-kubelet- several minutes, the mcp worker finish rolling out to the latest mc a pod % oc create -f hello-pod- hello-pod- a large file in container, make imagefs.available < 90Gi (the current available disk size is 111G ) command terminated with exit code 137 the node status the pod status % oc get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES hello-pod-2 0/1 ContainerStatusUnknown 1 7m58s 10.131.0.7 ip-10-0-213- <none<none% oc get pod hello-pod-2 -o yaml No response <details</details<detailsaws </details<details</details<details</details<details</details<details</details
/sig node
/assign
/assign
/triage accepted


Question: StorageConfig:           endpointConfig, Decorator:               generic.UndecoratedStorage, DeleteCollectionWorkers: 0, <del> ResourcePrefix:          c.ExtraConfig.StorageFactory.ResourcePrefix(kapi.Resource(DefaultStorageEndpoint)), </del> <ins> ResourcePrefix:          c.ExtraConfig.StorageFactory.ResourcePrefix(kapi.Resource("endpoints")), </ins> }) endpointRegistry := endpoint.NewRegistry(endpointsStorage) masterLeases := reconcilers.NewLeases(leaseStorage, "/masterleases/", ttl)

/kind bug The lease reconciler creates the kubernetes endpoint with a prefix of kube-apiserver-endpoint, however that is not the prefix that the rest of kube uses. The end result is that the kubernetes service doesn't have an endpoint, and kubectl won't list the kubernetes endpoint. The lease reconciler also creates lease files with a really long lease time instead of the default 15 seconds.
/cc
cc xref
kube-apiserver-endpoint is used for the lease itself, from what I remember. I don't think that's an issue by itself.
Oh but it does look like the endpointRegistry for the lease reconciler is misconfigured to use that
i'll look into it
what is the name that should be used?
services/endpoints afaik -
Yeah, making that change locally fixed it for me
But I'm not seeing the masterleases/apiServerIPInfo data in etcd, or I'm issuing the wrong command to find it
I had that same problem. The lease timeout is set way high, but for some reason that resulted in the lease files being created for only a very short time. I'm not sure why. Playing with the ttl amount got the lease files to stay around....forever. I've got a PR I'm about to push to fix both issues.
FYI has a PR for this now
cc


Question: # This script will build a dev release and bring up a new cluster with that # release. <ins> set -e </ins> # First build the binaries $(dirname $0)/build-go.sh if [ "$?" != "0" ]; then

As I discovered the hard way, if the json GCS api is not turned on for a project, will not surface this error, but just loop 'forever'. API access errors should be easy to catch, so it would be great to surface them on cluster startup and abort with a useful message. (I might take a look at this).
This is described in step 3 of But I agree that it would be better if the tool could catch this case and let the user know (it bit me the first time I tried to run that script as well).


Question: "github.com/GoogleCloudPlatform/kubernetes/pkg/fields" "github.com/GoogleCloudPlatform/kubernetes/pkg/labels" "github.com/GoogleCloudPlatform/kubernetes/pkg/util" <ins> "github.com/GoogleCloudPlatform/kubernetes/pkg/util/wait" </ins> . "github.com/onsi/ginkgo" . "github.com/onsi/gomega"

Still early in diagnosis, but soon after cutting 0.17.0, our early internal testing show this test as flaking. It's pretty bizarre, though, given the test isn't flaking on GCE, or even the continuous integration runs on GKE.
cc
So far, I've tried reproing this on a bare cluster 100 times, no joy. Now I'm trying up/test/down, no immediate joy (but it's a flake). If I get to the point of testing up/e2e/down, I might as well call it "Jenkins" and be done with it, but I guess it's time to start looking through logs instead. (As for the P0, trying to figure out if we need a cherry-pick, or if we let the flake proceed on in the entire 0.17.0 train.)
+1 to ironing out a flake; v0.16.0 had a flake (or maybe even a consistently-failing test) which made us turn off tests that would run against it for weeks. This is especially important now that we're doing soak tests.
(since it's related to events)
The test itself looks racy; there's nothing to guarantee that the events will have made it through the system by the time the pod is running. Generally one would expect that, but there's no guarantee.
Can you paste some sample failures here?
go/k8s-test/job/kubernetes-e2e-gke-test/4534/console
Actually this test is flaky across the board, and has been for some time. If you look at kubernetes-e2e-gce Test Results Analyser in Jenkins, you'll see about 8 failures in the past few hundred runs. I think lavalamp is correct - the test needs to retry/wait for the events to get through. Specifically, this code needs to be wrapped in a util.Poll():
Yeah if the kubelet event shows up less often than the scheduler event, that would make sense, since there'll have been less time.
I can whip up a quick fix now.
If it's a known flake, it's not going to block us. We should get a fix in soon for the next release, though.
Nearly done.