Question: // podsShouldBeOnNode figures out the DaemonSet pods to be created and deleted on the given node: //   - nodesNeedingDaemonPods: the pods need to start on the node //   - podsToDelete: the Pods need to be deleted on the node <del> //   - failedPodsObserved: the number of failed pods on node </del> //   - err: unexpected error func (dsc *DaemonSetsController) podsShouldBeOnNode( node *v1.Node, nodeToDaemonPods map[string][]*v1.Pod, ds *apps.DaemonSet, <del> ) (nodesNeedingDaemonPods, podsToDelete []string, failedPodsObserved int, err error) { </del> <ins> ) (nodesNeedingDaemonPods, podsToDelete []string, err error) { </ins> wantToRun, shouldSchedule, shouldContinueRunning, err := dsc.nodeShouldRunDaemonPod(node, ds) if err != nil {

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: I have a daemonsets with 250 replicas and maxUnavailable is 20%. When I try to update its image, the ds will scale down 50 old replicas and wait for SatisfiedExpectations, after that it will scale up 50 new replicas. But if a pod delete failed (node is notready, and pod stuck at terminating status), ds controller will never reach the expectation and just stuck here. What you expected to happen: When a pod stuck at terminating, ds controller should start new pods like deployment controller, instead of waiting for the delete batch finish and ready for next poll. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"8+", GitVersion:"v1.8.1-8+", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-10-25T13:04:04Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig apps /cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
DaemonSet controller waits for old pods to terminate before creating new ones for best-effort at-most-one pod per node. This is a deliberate design decision.

In most cases, DaemonSets won't tolerate two pods scheduled on the same node at the same time; for example, it's common to communicate with DaemonSet pods via . I'm going to close this ticket as this works as designed. Please feel free to reopen the ticket if you think there are still issues to be resolved.
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
why does the DaemonSet controller wait for all pods 50 to be done, before moving to the NEXT 50 ? I completely understand that we want only at most 1 pod per node. But if a pod deletion fails on the node, move on to the next pod as long as the total unavailable pods are less than maxUnavailable. This will allow us to make progress in the face of bad nodes or failed pod deletions due to failing tear down of bad volumes or other reasons.
/assign
thoughts on this one ?
Any progress on this issue? We also encountered this problem. There are thousands of nodes in our prod cluster and nodes often go down, which cause daemonset rolling update stucked, this problem really bother us.
PTAL
/reopen
You can't reopen an issue/PR unless you authored it or you are a collaborator. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
You can't reopen an issue/PR unless you authored it or you are a collaborator. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: return nil } <del> numaNodeReservation := strings.Split(s, ":") if len(numaNodeReservation) != 2 { return fmt.Errorf("the reserved memory has incorrect format, expected numaNodeID:type=quantity[,type=quantity...], got %s", s) } memoryTypeReservations := strings.Split(numaNodeReservation[1], ",") if len(memoryTypeReservations) < 1 { return fmt.Errorf("the reserved memory has incorrect format, expected numaNodeID:type=quantity[,type=quantity...], got %s", s) } numaNodeID, err := strconv.Atoi(numaNodeReservation[0]) if err != nil { return fmt.Errorf("failed to convert the NUMA node ID, exptected integer, got %s", numaNodeReservation[0]) } memoryReservation := kubeletconfig.MemoryReservation{ NumaNode: int32(numaNodeID), Limits:   map[v1.ResourceName]resource.Quantity{}, } for _, reservation := range memoryTypeReservations { limit := strings.Split(reservation, "=") if len(limit) != 2 { return fmt.Errorf("the reserved limit has incorrect value, expected type=quantatity, got %s", reservation) </del> <ins> numaNodeReservations := strings.Split(s, ";") for _, reservation := range numaNodeReservations { numaNodeReservation := strings.Split(reservation, ":") if len(numaNodeReservation) != 2 { return fmt.Errorf("the reserved memory has incorrect format, expected numaNodeID:type=quantity[,type=quantity...], got %s", reservation) </ins> } <del> resourceName := v1.ResourceName(limit[0]) if resourceName != v1.ResourceMemory && !corev1helper.IsHugePageResourceName(resourceName) { return fmt.Errorf("memory type conversion error, unknown type: %q", resourceName) </del> <ins> memoryTypeReservations := strings.Split(numaNodeReservation[1], ",") if len(memoryTypeReservations) < 1 { return fmt.Errorf("the reserved memory has incorrect format, expected numaNodeID:type=quantity[,type=quantity...], got %s", reservation) </ins> } <del> q, err := resource.ParseQuantity(limit[1]) </del> <ins> numaNodeID, err := strconv.Atoi(numaNodeReservation[0]) </ins> if err != nil { <del> return fmt.Errorf("failed to parse the quantatity, expected quantatity, got %s", limit[1]) </del> <ins> return fmt.Errorf("failed to convert the NUMA node ID, exptected integer, got %s", numaNodeReservation[0]) </ins> } <del> memoryReservation.Limits[v1.ResourceName(limit[0])] = q } </del> <ins> memoryReservation := kubeletconfig.MemoryReservation{ NumaNode: int32(numaNodeID), Limits:   map[v1.ResourceName]resource.Quantity{}, } </ins> <del> *v.Value = append(*v.Value, memoryReservation) </del> <ins> for _, memoryTypeReservation := range memoryTypeReservations { limit := strings.Split(memoryTypeReservation, "=") if len(limit) != 2 { return fmt.Errorf("the reserved limit has incorrect value, expected type=quantatity, got %s", memoryTypeReservation) } resourceName := v1.ResourceName(limit[0]) if resourceName != v1.ResourceMemory && !corev1helper.IsHugePageResourceName(resourceName) { return fmt.Errorf("memory type conversion error, unknown type: %q", resourceName) } q, err := resource.ParseQuantity(limit[1]) if err != nil { return fmt.Errorf("failed to parse the quantatity, expected quantatity, got %s", limit[1]) } memoryReservation.Limits[v1.ResourceName(limit[0])] = q } *v.Value = append(*v.Value, memoryReservation) } </ins> return nil }

I am trying out For that I enabled kubelet memory manager, cpu manager and topology manager. This is tried on a dual socket NUMA server KUBELETKUBEADMARGS="--container-runtime=remote --container-runtime- --cpu-manager-policy=static --feature-gates=RotateKubeletServerCertificate=true,AllAlpha=false,RemoveSelfLink=false,EphemeralContainers=true,BoundServiceAccountTokenVolume=false,KubeletPodResourcesGetAllocatable=true,TopologyManager=true,MemoryManager=true --kube-api-burst=40 --kube-api-qps=20 --kube-reserved=cpu=4,memory=4Gi --memory-manager-policy=static --reserved-cpus=0,1,2,3 --reserved-memory 0:memory=4Gi,1:memory=4196Mi --system-reserved=cpu=4,memory=4Gi --topology-manager-policy=single-numa-node" topologyManagerPolicy: single-numa-node is set on This throws an error 2021-10-20T11:28:49.+00:00 localhost kubelet[]: E1020 11:28:49. ] "Failed to parse kubelet flag" err="invalid argument "0:memory=4Gi,1:memory=4196Mi" for "--reserved-memory" flag: the reserved memory has incorrect format, expected numaNodeID:type=quantity[,type=quantity...], got 0:memory=4Gi,1:memory=4196Mi" This is on kubernetes 1.21.1 Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2021-09-29T11:36:35Z", GoVersion:"go1.16.4", Compiler:"gc", Platform:"linux/amd64"} Any hints on specifying "reserved-memory" for multi-socket NUMA system ???? One way is specifying multiple "reserved-memory" like But problem with this is is map, so it will get over-written. we should be able to specify reserved memory in a multi socket system.

Enable above flags and run kubelet No response Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2021-09-29T11:36:35Z", GoVersion:"go1.16.4", Compiler:"gc", Platform:"linux/amd64"} none SLES 15 sp2 <details</detailscontainerd No response
/sig node
this is the right way to config
Problem with that is KubeletExtraArgs map[string]string is a map, so multiple "reserved-memory" will get over-written.
there is a tracking issue for kubeadm's api to support multiple arguments with the same name: but it is a backwards incompatible change. you should beusing the KubeletConfiguration for this instead:
/remove-kind support /kind bug /assign Looks like a bug in a parsing logic + better error would be beneficial
Is this a parsing bug?, if yes can you just point me where I need to fix this. I will try to fix this in the downstream release(our internal release).
Here the parsing logic - and here it is failing - Give me know if you need some additional help.
Are we planning to add support for multiple numa nodes in single "--reserved-memory" flag, or just displaying a better error msg?
We should support multiple NUMA nodes under the same --reserved-memory flag.
/triage accepted
Can you provide a sample configuration for 2 numa nodes? "." comma seperator is already there for single numa configuration.
Yes, it is the problem, initially we needed to split it by comma, but probably it will not work as expected, so maybe we can split it by and update documentation. Something like:
can you please review the PR and provide your comments


Question: "k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/runtime/schema" "k8s.io/apimachinery/pkg/types" <del> "k8s.io/apimachinery/pkg/util/wait" </del> "k8s.io/apimachinery/pkg/watch" "k8s.io/cli-runtime/pkg/genericclioptions" "k8s.io/cli-runtime/pkg/genericclioptions/printers"

<!-- Please only use this template for submitting enhancement requests --What would you like to be : let's say I run If the deployment is comes up in time it tells me which deployment is up, but if it times out it doesn't tell me on which deployment/resource. Could it be Why is this needed: That would make it easier to use, and the information provided is the same irregardless on whether the condition is met or not
/sig cli
Sounds reasonable for me, i can take care of this one if someone from owners will approve it as a feature.
/assign Zyqsempai
GitHub didn't allow me to assign the following users: Zyqsempai. Note that only and repo collaborators can be assigned and that issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Zyqsempai Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: // request's context's Done channel gets closed by the time // the request is done being processed. doneCh := ctx.Done() <ins> // Retrieve the queueset configuration name while we have the lock // and use it in the goroutine below. configName := qs.qCfg.Name </ins> if doneCh != nil { qs.preCreateOrUnblockGoroutine() go func() {

Which jobs are flaking: ci-kubernetes-cached-make-test ci-kubernetes-generate-make-test-count10 pull-kubernetes-unit Which test(s) are flaking: TestPriorityAndFairnessWithPanicRecoveryAndTimeoutFilter Testgrid link: Reason for failure: Data race: Reproduced in ~2% of runs with : Triage board: /sig api-machinery
cc
The test is timing out after 300 seconds, but I couldn't reproduce this over 1000 runs (with and without )
(In case it's not clear from comments: the actual failure is the timeout, not the panic as stated in the OP.)
Finally reproduced the issue locally after many runs:
thank you for reproducing it! are you planning to try to fix it? Thanks!
I tried to, but I couldn't find the root cause. I'll take another look today. /assign
/accept triage
/triage accepted
/milestone v1.21 /priority critical-urgent Raising priority and adding to the milestone for triage since this looks like a data race in non-test code in a default-enabled filter. This unit test did not exist in 1.20 so it's not clear if this is a regression in 1.21 or not. If this issue existed in 1.20, priority can be lowered to important-soon and it does not block 1.21, but should be fixed and picked to 1.20.x and 1.21.x
I'm afraid I won't have the time to look into this in the short term. Unassigning so that someone else can take a look. /unassign
(looking at it now) /assign
found the data race and put out a quick fix for this:
I ran , and it looks good so far:
cherry-picked it for - Can someone please let me know if I did it correctly?


Question: // Apply cgroupsParent derived from the sandbox config. if lc := sandboxConfig.GetLinux(); lc != nil { // Apply Cgroup options. <del> // TODO: Check if this works with per-pod cgroups. // TODO: we need to pass the cgroup in syntax expected by cgroup driver but shim does not use docker info yet... hc.CgroupParent = lc.GetCgroupParent() </del> <ins> cgroupParent, err := ds.GenerateExpectedCgroupParent(lc.GetCgroupParent()) if err != nil { return "", fmt.Errorf("failed to generate cgroup parent in expected syntax for container %q: %v", config.Metadata.GetName(), err) } hc.CgroupParent = cgroupParent </ins> } // Set devices for container.

Kubernetes version (use ): v1.5.0-alpha.1.2189+-dirty docker: 1.12.3 Environment: fedora24 docker-engine-1.12.3-1.fc24.x86_64 What happened: if docker Cgroup Driver=cgroupfs, and set --cgroup-driver=systemd in kubelet, then enable --experimental-cgroups-per-qos=true; The container cgroup some is not correct. How to reproduce it (as minimally and precisely as possible): Anything else do we need to know: 1) docker Cgroup Driver=cgroupfs and --cgroup-driver=cgroupfs in kubelet works well 2) docker Cgroup Driver=systemd and --cgroup-driver=systemd in kubelet works well
help check, thanks.
In Kubernetes 1.6, we can work to detect this case and return an error. IMHO its a configuration error to have docker running with a cgroupfs driver paired with the kubelet running the systemd cgroup driver. /cc
We can also document the compatibility matrix for now.


Question: }, } podWithHook := getPodWithHook("pod-with-poststart-http-hook", imageutils.GetPauseImageName(), lifecycle) <ins> // make sure we spawn the test pod on the same node as the webserver. nodeSelection := e2epod.NodeSelection{} e2epod.SetAffinity(&nodeSelection, targetNode) e2epod.SetNodeSelection(&podWithHook.Spec, nodeSelection) </ins> testPodWithHook(podWithHook) }) /* Release: v1.9 Testname: Pod Lifecycle, prestop http hook <del> Description: When a pre-stop handler is specified in the container lifecycle using a 'HttpGet' action, then the handler MUST be invoked before the container is terminated. A server pod is created that will serve http requests, create a second pod with a container lifecycle specifying a pre-stop that invokes the server pod to validate that the pre-stop is executed. </del> <ins> Description: When a pre-stop handler is specified in the container lifecycle using a 'HttpGet' action, then the handler MUST be invoked before the container is terminated. A server pod is created that will serve http requests, create a second pod on the same node with a container lifecycle specifying a pre-stop that invokes the server pod to validate that the pre-stop is executed. </ins> */ framework.ConformanceIt("should execute prestop http hook properly [NodeConformance]", func() { lifecycle := &v1.Lifecycle{

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --k8s-e2e-ltsc2019-containerd-flannel-sdnoverlay-master [] Container Lifecycle Hook when create a pod with lifecycle hook should execute prestart http hook properly [NodeConformance] [Conformance] [] Container Lifecycle Hook when create a pod with lifecycle hook should execute poststart http hook properly [NodeConformance] [Conformance] Note: the tests have been excluded until they are fixed. In the case of multinode clusters, the http server pod and the test cluster can spawn on different nodes, which can be problematic for poststart / prestop hooks, as they are executed by the kubelet itself, and the cross-node lifecycle hook might fail (according to the [Kubernetes network model], it is not mandatory for kubelet to be able to access pods on a different node). However, that is not always the case, different Networking setups allow cross-node communication between kubelet and pods, which means that the above tests are passing properly (for example: ), which begs the question if this behaviour should be universal or not, especially since the tests are marked as , which would imply that it should work in all cases. [Kubernetes network model]: links to appreciated links to specific failures in spyglass appreciated Slack: /sig testing /sig networking /sig windows <!-- Please see the deflaking doc () for more guidance! --
/sig network /sig node /area conformance /assign /priority critical-urgent /cc This is important since affects conformance and defines the networking behaviour for the poststart/ prestop hooks
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /sig network /sig node /sig conformance /assign /priority critical-urgent /cc Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
i'd personally let folks have first dibs on what we need to do here. Though from the name of the test itself, it seems like the test is NOT a sig-network test :)
Indeed, it does not have any tag in its name, but the tests rely on networking and on how CNIs work.

As I've stated above, it's not a problem for flannel sdnbridge, but it is for flannel sdnoverlay; other CNIs might have a similar issue too.
because the network is not a thing :) let´s tag the sig owner then /sig node I think that the key here is to decide if kubelet should be able to access pod in other nodes, and based on the link Claudiu is part of the Kubernetes networking model. I saw users that doesn't want kubelet to access even the pods on same host, so I think that is legit to say kubelet MAY reach pods in other nodes but not MUST ... but I'm not familiar with the to fully understand the context. If kubelet "may" reach pods in the other nodes, the test must only test lifecycle hooks for pods in the same node as the kubelet under test, right?
This is correct. The model also says: "For those platforms that support Pods running in the host network (e.g. Linux): pods in the host network of a node can communicate with all pods on all nodes without NAT" I believe that Windows host network is not supported, right?
Right, and there are specific tests that checks that exact scenario you've mentioned: However, the flaky tests I've mentioned in the issue require kubelet to be able to dial-in into a webserver pod for a different pod's poststart / prestop HTTP lifecycle hooks. There are couple of things regarding this: of all, in order to ensure kubelet would be able to run the HTTP lifecycle hooks, we'd have to make sure both pods are on the same node, as that is a supported scenario, according to the Kubernetes Network Model. I have sent a PR for this above. Alternatively, a Cluster IP could probably work as well, but IMO, ensuring that both pods end up on the same node is simpler and easier to clean up. , it's interesting that those HTTP lifecycle hook tests work on different networking setups, without any pods having HostNetwork enabled, even though you've mentioned they should (e.g.: Linux / Windows with flannel l2bridge, and even Linux with flannel overlay). Now, my question is: is this behaviour a feature, or a bug, especially since the tests are marked as `?

oh, I missed that part, so in Linux kubelet must be able to communicate with pods on all nodes (becuase it is running in the hostnetwork) If that is correct, it seems that one solution is to keep the test for Linux and tag it as LinuxOnly, and a new test for WindowsOnly that only spawn the pods under test in the same node EDIT Discussed in sig-net meeting , no clear answer... can be kubelet considered as a hostnetwork pod?
For the purposes of testing, I don't mind forcing the pods onto the same node., but let's document the heck out of that at the code-site?
correct but it will be supported once HostProcess containers are merged in 1.22: Only in Overlay networks (ie Flannel where these tests are flaking) does the pod have to be on the same node. It doesn't require host network mode and these tests are . I would be more inclined to disable this test for Windows Overlay test setups and have a sig-windows test for the overlay functionality. I would also question if the lifecycle hooks only running on the same node is useful and if it should just be documented that it doesn't work in the Windows Overlay setup.
is looking at test PRs from SIG Node. I will let sig-net triage this issue
Next steps on this?
/triage accepted
We still haven't merged this: I also had this, but it has been decided to be blocked:
requires conformance approval, at this point I think that has to be postponed to 1.23 right, since that behavior is not required, we shouldn't test it, because is legit to fail and having a test for something that can fail can create confusion :smile:


Question: return false, nil } <ins> if wantPrefixForEncryptAll != "" { deploymentName := fmt.Sprintf("deployment-%d", idx) _, err := test.createDeployment(deploymentName, "default") if err != nil { t.Fatalf("Failed to create test secret, error: %v", err) } rawEnvelope, err := test.readRawRecordFromETCD(test.getETCDPathForResource(test.storageConfig.Prefix, "", "deployments", deploymentName, "default")) if err != nil { t.Fatalf("failed to read %s from etcd: %v", test.getETCDPathForResource(test.storageConfig.Prefix, "", "deployments", deploymentName, "default"), err) } // check prefix if !bytes.HasPrefix(rawEnvelope.Kvs[0].Value, []byte(wantPrefixForEncryptAll)) { idx++ swapErr = fmt.Errorf("expected deployment to be prefixed with %s, but got %s", wantPrefixForEncryptAll, rawEnvelope.Kvs[0].Value) // return nil error to continue polling till timeout return false, nil } } </ins> return true, nil }) if pollErr == wait.ErrWaitTimeout {

transformation expandless | 14m1s Failed; === RUN TestEncryptionConfigHotReload TestEncryptionConfigHotReload When I see this flake, 10-09 is the first one.(But the job history in the testgrid only started from 10-03) some PR that may be related: /sig auth /cc
/cc
/triage accepted
/cc failed two times in recent 3 runs:
/reopen as of this is still .
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: resourceGroup := matches[1] return strings.Replace(resourceID, resourceGroup, strings.ToLower(resourceGroup), 1), nil } <ins> // isBackendPoolOnSameLB checks whether newBackendPoolID is on the same load balancer as existingBackendPools. // Since both public and internal LBs are supported, lbName and lbName-internal are treated as same. // If not same, the lbName for existingBackendPools would also be returned. func isBackendPoolOnSameLB(newBackendPoolID string, existingBackendPools []string) (bool, string, error) { matches := backendPoolIDRE.FindStringSubmatch(newBackendPoolID) if len(matches) != 2 { return false, "", fmt.Errorf("new backendPoolID %q is in wrong format", newBackendPoolID) } newLBName := matches[1] newLBNameTrimmed := strings.TrimRight(newLBName, InternalLoadBalancerNameSuffix) for _, backendPool := range existingBackendPools { matches := backendPoolIDRE.FindStringSubmatch(backendPool) if len(matches) != 2 { return false, "", fmt.Errorf("existing backendPoolID %q is in wrong format", backendPool) } lbName := matches[1] if !strings.EqualFold(strings.TrimRight(lbName, InternalLoadBalancerNameSuffix), newLBNameTrimmed) { return false, lbName, nil } } return true, "", nil } </ins>

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: SLB backend pools are empty if outbound rules are defined in a separate backend pool. This is because, although we the check for not adding the backend pools from different LBs, the lbName is not checked: The new backend pool's lbName should be also checked in above codes. What you expected to happen: SLB backend pools should be updated correctly when outbound rules are defined in a separate backend pool. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Others: /sig azure /kind bug /assign


Question: // OperationGenerator interface that extracts out the functions from operation_executor to make it dependency injectable type OperationGenerator interface { // Generates the MountVolume function needed to perform the mount of a volume plugin <del> GenerateMountVolumeFunc(waitForAttachTimeout time.Duration, volumeToMount VolumeToMount, actualStateOfWorldMounterUpdater ActualStateOfWorldMounterUpdater) (func() error, error) </del> <ins> GenerateMountVolumeFunc(waitForAttachTimeout time.Duration, volumeToMount VolumeToMount, actualStateOfWorldMounterUpdater ActualStateOfWorldMounterUpdater, isRemount bool) (func() error, error) </ins> // Generates the UnmountVolume function needed to perform the unmount of a volume plugin GenerateUnmountVolumeFunc(volumeToUnmount MountedVolume, actualStateOfWorld ActualStateOfWorldMounterUpdater) (func() error, error)

TL;DR: kubelet keeps trying to remount secrets and configmaps into pods over and over again. My environment: kubernetes 1.3.0 running on Ubuntu 14.04. I used the instructions here to set things up: (i.e. etcd, flanneld and the kube* processes are running on the host OS, not in containers or docker-in-a-vm ala minikube) In trying to track down high disk write rates, I found that the partition I use for my etcd WAL log was getting about 40kb/s write traffic. etcd metrics show that I'm getting 3 qps get and 3qps getrecursive. Write traffic to etcd is essentially nil (pods are all stable). This is a single node system with ~30 user pods, most of which are doing nothing unless I sent traffic to them (which I'm not while doing these measurements). In the upstart kubelet log, I see this over and over again: So it's repeatedly remounting already-mounted secret volumes and configmaps into pods. Analyzing the unique pod IDs, this appears to be affecting all of them: I suspect the 2 missing pods are from old deployments. This goes on ad infinitum every 3-5 seconds, hitting 2-3 pods each time, so presumably some kind of maint loop. The code branch in is this one: and that err is populated from this line: But beyond that I'm not sure why it is consistenly thinking that the mount operation didn't stick. Any ideas where to go from here?
Could it be as simple as that RequiresRemount method in , and are while all the other plugins ?
I can recreate this problem on master with the following sample: Leave it running for 10 mins or so and shows:
Looks like this is by design : The plugin(s) are telling the system to keep remounting to update the contents of the volume. Patching the following to stops the repeated re-mount. Folks are probably using this mechanism to keep the data mounted to be up-to-date. So not sure we should "fix" this behavior. Thoughts please?
Yes, current design keep remounting to update the contents because currently there is no direct way of informing the changes to kubelet.

The remounting normally happens every 10~30 seconds from my observation. In case this behavior cause performance issue, we can consider optimize it.
Could you decrease the verboseness when the volume is already mounted? This is polluting our logs and makes more important failures harder to follow.
Sure. We will work on that. Thanks for posting the issue. Jing 2016721Steffen Hanikel
I'm facing the same issue. Kubernetes 1.3.2 on GKE Container stuck on the ContainerCreating phase since volumes never get mounted. Since default secret keep being remounted all the time, my secret never get mounted:
Hi Taras, I think your problem is different. The log shows that /var/lib/kubelet/pods/ -53de-11e6-b2f2- no such file or directory Looks like it very much related to a name collision bug and there is fix being reviewed Jing , Taras wrote:
Hi Yes, probably it related. Waiting for the fix to get to release to test it on GKE. I needed to downgrade nodes to the 1.2.5 to be able to run pods.
Hello, a parameter to say if the secret need to be updated and remounted periodically would be great :) For most of my pods i just need a secret during the pod first start, and i don't necessarily want the content to stay an be remounted over and over in the pod.
Thanks for your suggestions. I think the log might be a a little bit misleading. If the secret volume is already mounted, it won't mounted ago, But it does rewrite the secret. I think we can avoid rewriting by comparing the contents. Is that good enough? Thanks! Jing , osixia wrote:
yes, how i use secrets not rewriting the content would be great, to avoid rewriting deleted files especially.
do you mean you have some cases that you want to delete some files under secrete directory, and don't want the system to rewrite those files after deletion? It would be great if you could let us know more details about your use case so that we are more clean about what needs to be done. Thanks!

Jing , osixia wrote:
i don't know if it's revelant and may be i just do stupid things just let me know :) a litle example: I have an openldap image, when the container starts for the first time if no database is provided it creates a new one with an admin password. When the database is created the admin password is stored encrypted and i don't want it to stay in the container environment in clear text. (i do the same for mariadb images) So as i don't want the admin password to stay in clear text in the container environment i can't set it via environment variable. I found a solution with secrets : I set the admin password in a configuration file, and put that file in a secret. When a new pod is created the secret was mounted in a volume and the container takes the admin password from the configuration file, create the database and delete the configuration file from the secret's mounted volume. I can start several pods this way with the same secret and admin password and don't have the admin password remaining in the container environment or file system in clear text.
Thank you for posting your use case. We plan to add a feature for addressing it, track it under . Please let me know if you have any suggestions. Thanks! Jing
thank you very mutch :) I have an other thought about the service-account default token, i guess this one will be automatically remounted, but is there any way to not mount this secret to a pod ? I'm concerned about security issues if a pod is compromised, that will allow someone to request the kubernetes api ?
For the problem about excessive disk first raised by I think maybe kubelet should remember which secret UID was last written, and not rewrite if the secret UID has not changed. What do you think For If you want to not have the token mounted, you can do this: It isn't pretty, but it works.
thanks but the trick seems to not work anymore :( returns: if i remove the pod keep crashing with :
What do the pod logs say for the crashing container (e.g. kubectl logs i-have-no-sa)?
The logs are empty
I think your pod is failing because of the combination of image/command.

This gist works: For example, this shows that 's workaround works:
it worked, thanks a lot :)
Could you decrease the verboseness when the volume is already mounted? This is polluting our logs and makes more important failures harder to follow. Sure. We will work on that. Thanks for posting the issue. We're running 1.4.7 now and still seeing a high volume of these messages: With about 125 Pods running in our cluster I'm getting about 6000 log messages per hour similiar to this.
I have similar issue to with fresh kubeadm install on kubernetes 1.5.1
I'm facing the same issue only on one node of my cluster, it fills the number of mounts pretty quick leading to unresponsive OS ( 32k mount )
could you provide some more details about your case? Do you use nfs or gluster in the pods? Could you also give a some of the mounts you see on the node so we can have a better understanding what might cause the issue? Thanks!

Sure sorry, i had lots of tmpfs /home/lib/kubelet/pods/-f25f-11e6-8d1a- tmpfs rw,relatime 0 0 tmpfs /home/lib/kubelet/pods/-f259-11e6-8d1a- tmpfs rw,relatime 0 0 tmpfs /home/lib/kubelet/pods/-f25f-11e6-8d1a- tmpfs rw,relatime 0 0 tmpfs /home/lib/kubelet/pods/-bf6f-11e6-b5ec- tmpfs rw,relatime 0 0 tmpfs /home/lib/kubelet/pods/-f25f-11e6-8d1a- tmpfs rw,relatime 0 0 tmpfs /home/lib/kubelet/pods/-f25f-11e6-8d1a- tmpfs rw,relatime 0 0 tmpfs /home/lib/kubelet/pods/-f25f-11e6-8d1a- tmpfs rw,relatime 0 0 tmpfs /home/lib/kubelet/pods/-ddaa-11e6-8d1a- tmpfs rw,relatime 0 0 tmpfs /home/lib/kubelet/pods/-f25f-11e6-8d1a- tmpfs rw,relatime 0 0 The pod -f25f-11e6-8d1a- was restarting a lot ( a misconfiguration on my part ) i'm using version 1.4.7
which version of kuberentes you are using? I see the exact mounts you posted are repeated many time. In the current mount code path, it should check whether the path is already mounted and if it is yes, it will skip to mount it again. I will check again our code path.
Same here with 1.5.2 on CoreOS - it wasn't the problem with 1.3.7, same setup.
Same issue with 1.5.2_2 on CoreOS latest stable as well.
Same issue with 1.5.4 on Centos 7.2 as well.

because the our log might be a little bit misleading, could you confirm whether you see redundant mounts for a pod volume such as secrete when you run "mount" command on the host? Thanks!
I'm seeing this with v1.5.2 running on AWS, setup via kops: The log has entries from the past ~4 days:


Question: gcp_list_resources="true" # Always pull the script from HEAD, overwriting the local one if it exists. # We do this to pick up fixes if we are running tests from a branch or tag. <del> curl -fsS --retry 3 "https://raw.githubusercontent.com/kubernetes/kubernetes/master/cluster/gce/list-resources.sh" > "${gcp_list_resources_script}" </del> <ins> curl -fsS --retry 3 --keepalive-time 2 "https://raw.githubusercontent.com/kubernetes/kubernetes/master/cluster/gce/list-resources.sh" > "${gcp_list_resources_script}" </ins> else gcp_list_resources="false" fi

http://pr- Seems like a flake? Could probably retry this, but not sure whether it's worth it.
we most definitely want to retry! Will you send a PR?
I just hit this in PR
We should close this, should be fixed by:
that didn't actually solve the problem. Per the docs, only works on "transient" errors, which the manpage defines as We might want to try passing , which some googling suggests might help. The default is 60s, which may be too long.
Please reactive this if you see it after June 15


Question: "context" "fmt" <del> "k8s.io/klog" </del> v1 "k8s.io/api/core/v1" "k8s.io/apimachinery/pkg/labels" "k8s.io/apimachinery/pkg/runtime"

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Logging getting points for ALL priorities set in the default scheduler policy Why is this needed: We have Kubernetes v1.14 with a problem of planning pods for three nodes. The problem is that the scheduler unevenly distributes the pods across the nodes. For example, the CPU on two nodes Node02 and Node03 are 50% loaded, while the CPU on the third Node01 is not loaded at all. While that the pods continue to be planned on Node02 and Node03, but they are not planned on Node01 at all. Scheduler logs (maximum logging level) with uniform planning: and uneven planning: As you can see, the sum of points for the priorities BalancedResourceAllocation, LeastResourceAllocation, SelectorSpreadPriority, TaintTolerationPriority is the same in both cases, but the final result is different. There is no way to understand for what specific policy priorities Node01 did not get 3 points, for to be on a par with the others, because the set of points of some priorities is not logged.
/sig scheduling
I think maybe we missed some logs which can reflect the situation of giving the score. WDYT? If you confirmed, I want to handle this issue.
Thank you for the answer! Here is the full scheduling log (logging level 10): maybe I’m really missing something, but I don’t see information about other priorities here.
Missing scores may be due to some priority functions are using non-map-reduce style, so they didn't hit the printing logic. We can add a logging function there to help you better debugging it. However, the code gets refactored in recent releases, and 1.14 is out of our supporting cycles (only 1.15, ,1.16, 1.17). Can you upgrade to a newer version, or I can provide a code snippet for you to compile on your local 1.14 branch?
Thank you, but I already solved my problem by creating an additional scheduler with a custom scheduling policies for my cronjob. However, it would be cool if you the logging function in the supported versions :)
SG. I created to address this.
I'm running v1.23.8, and I've tried and I'm still not seeing any of the scoring output you in Am I doing something wrong?

Shouldn't this be available in v1.23?
is it possible there is only 1 node candidate after filtering? In that case, the scoring logic will be bypassed as it's meaningless to score for 1 node. If it's not the case, could you check the beginning of the scheduler log, to ensure is configured properly?


Question: "IngressRule.HTTP.Paths.Backend.Service v1": { input: netv1, output: `Name:             bar <ins> Labels:           <none> </ins> Namespace:        foo Address: Default backend:  default-http-backend:80 (<error: endpoints "default-http-backend" not found>)

When command 'kubectl describe ing' Name: test-ing Namespace: practice Address: 10.10.01.10 Default backend: default-http-backend:80 (<error: endpoints "default-http-backend" not found) TLS: test terminates Rules: Host Path Backends ---- ---- -------- / test-svc:80 (<none) Annotations: nginx Events: <noneResult has no information about labels. But Pod, Deployment, Service resources has label information like below. Name: test-room-test-mjsbz Namespace: test Priority: 0 Node: aks-test-test-test/22.22.222. Start Time: Fri, 09 Jul 2021 05:45:26 +0000 Labels: app=test pod-template-hash=test role=mediatest Why ingress has no information about labels? Actually when delete ingresss with labels is working correctly. Kubernetes version (use ): 1.18.14 Cloud provider or hardware configuration: AKS (Azure) OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /sig
/sig
/sig cli
/assign
/assign
/triage accepted /remove-kind bug /kind feature


Question: # This function is for building the go code bin() { <ins> local arch_prefix="" if [[ "${ARCH}" == "arm" ]]; then arch_prefix="GOARM=${GOARM:-7}" fi </ins> for SRC in $@; do docker run --rm -it -v "${TARGET}:${TARGET}:Z" -v "${KUBE_ROOT}":/go/src/k8s.io/kubernetes:Z  golang:"${GOLANG_VERSION}"  /bin/bash -c " cd /go/src/k8s.io/kubernetes/test/images/${SRC_DIR} &&  <del> CGO_ENABLED=0 GOARM=${GOARM} GOARCH=${ARCH} go build -a -installsuffix cgo --ldflags '-w' -o ${TARGET}/${SRC} ./$(dirname "${SRC}")" </del> <ins> CGO_ENABLED=0 ${arch_prefix} GOARCH=${ARCH} go build -a -installsuffix cgo --ldflags '-w' -o ${TARGET}/${SRC} ./$(dirname "${SRC}")" </ins> done }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: login system root, and run below command from "test/images" dir, Will fail with below error message, ./image- line 91: : command not found This is cause by the empty string like "" cannot be understood as a shell command. What you expected to happen: image should be built successfully. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): latest Cloud provider or hardware configuration: OS (e.g: ): all Kernel (e.g. ):all Install tools: Network plugin and version (if this is a network-related bug): Others:
There are no sig labels on this issue. Please add a sig label by either: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. See the . The in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: cd $ $ make ../image- bin webhook ../image- line 145: GOARM: unbound variable Makefile:23: recipe for target 'bin' failed make: [bin] Error 1 What you expected to happen: The binary or image should be built successfully. How to reproduce it (as minimally and precisely as possible): see my desc above, type "make" will see the error message. Anything else we need to know?: Environment*: Kubernetes version (use ): latest Cloud provider or hardware configuration: OS (e.g: ): Ubuntu Kernel (e.g. ): all Install tools: Network plugin and version (if this is a network-related bug): Others:
There are no sig labels on this issue. Please add a sig label by either: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. See the . The in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
I'm pretty sure this is supposed to be built via from which does default this environment variable.
, thanks for the information! I think the harness of the shell scripts could bring us a little more, GOARM should only been set when the ARCH is ARM Make a binary is possible where a Makefile is presented
That seems reasonable, but we should probably avoid computing this in many places, currently the GOARM=${GOARM:-7} is going to always be 7 when building in the currently supported way. We should change that if we're going to compute it elsewhere. cc-ing for test/images fun ...


Question: scheme.AddKnownTypes(SchemeGroupVersion, &PodDisruptionBudget{}, &PodDisruptionBudgetList{}, <ins> &v1.ListOptions{}, &v1.DeleteOptions{}, </ins> ) // Add the watch version that applies versionedwatch.AddToGroupVersion(scheme, SchemeGroupVersion)

It seems to me like all of the garbage collection integration tests produce a ton of log messages of the form:


Question: } } <ins> func TestMetrics(t *testing.T) { mux := http.NewServeMux() InstallHandler(mux) InstallLivezHandler(mux) InstallReadyzHandler(mux) metrics.Register() metrics.Reset() paths := []string{"/healthz", "/livez", "/readyz"} for _, path := range paths { req, err := http.NewRequest("GET", fmt.Sprintf("http://example.com%s", path), nil) if err != nil { t.Errorf("%v", err) } mux.ServeHTTP(httptest.NewRecorder(), req) } expected := strings.NewReader(` # HELP apiserver_request_total [ALPHA] Counter of apiserver requests broken out for each verb, dry run value, group, version, resource, scope, component, client, and HTTP response contentType and code. # TYPE apiserver_request_total counter apiserver_request_total{client="unknown",code="200",component="",contentType="text/plain; charset=utf-8",dry_run="",group="",resource="",scope="",subresource="/healthz",verb="GET",version=""} 1 apiserver_request_total{client="unknown",code="200",component="",contentType="text/plain; charset=utf-8",dry_run="",group="",resource="",scope="",subresource="/livez",verb="GET",version=""} 1 apiserver_request_total{client="unknown",code="200",component="",contentType="text/plain; charset=utf-8",dry_run="",group="",resource="",scope="",subresource="/readyz",verb="GET",version=""} 1 `) if err := testutil.GatherAndCompare(legacyregistry.DefaultGatherer, expected, "apiserver_request_total"); err != nil { t.Error(err) } } </ins> func createGetRequestWithUrl(rawUrlString string) *http.Request { url, _ := url.Parse(rawUrlString) return &http.Request{

What would you like to be : Currently the apiserver has request latencies metrics, which are a histogram of requests to the apiserver. However, these request metrics do not include healthz. While we have recently the ability to get back health check metrics from the healthz endpoint directly, we should also emit this data as a metric, so that we can monitor health check latencies. /assign /sig api-machinery instrumentation
GitHub didn't allow me to assign the following users: jktomer. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : What would you like to be : /sig api-machinery instrumentation Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I noticed this is informally assigned. Has there been any progress? Is this a place I can fit in?
Yes, I'm working on a PR.
/assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: return nil, false, nil } <ins> // Cannot run cloud ipam controller if cloud provider is nil (--cloud-provider not set or set to 'external') if cloud == nil && ccmConfig.ComponentConfig.KubeCloudShared.CIDRAllocatorType == string(ipam.CloudAllocatorType) { return nil, false, errors.New("--cidr-allocator-type is set to 'CloudAllocator' but cloud provider is not configured") } </ins> // failure: bad cidrs in config clusterCIDRs, dualStack, err := processCIDRs(ccmConfig.ComponentConfig.KubeCloudShared.ClusterCIDR) if err != nil {

Cloud-providers started to move out of tree its controller, but they share some functionalities, one of them is the NodeIPAM controller. There is a type that is GCE specific, the type. If this type is used, but no configuration for GCE is passed, the kcm panics. This can be observed in the following job: The question is, should this type be removed or a noop in KCM?
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign /cc
/sig cloud-provider
Is this a case of CCM handling IPAM so KCM needs to not handle it? What I mean is why is this set without a config? What was the intention? The fact that this has GCE-cloud-provider baked in is awful and should probably have been inverted (GCE provider calls a "register" method or something), but that's aside
Context: I started to set up jobs using the CCM by default, the cluster set up uses the script. The script doesn't consider IPAM, so it configures both KCM and CCM, but if set to "external" , it doesn't configure GCEPROVIDER in KCM When , the scripts set ... and CloudAllocator only runs with GCE My concern is that if this happens to me, it is possible that users that have to migrate custom cluster will face the same issue
A panic here seems appropriate, right? They specifically asked for a config that doesn't work and action needed to be taken. Am I misunderstanding?
A panic is appropriate given we never did any handling of this corner case -- I think it would be reasonable going forward to have a check in place to disable nodeipam controller if the allocator is but the cloud provider is not configured.
Opened with the check and a warning when this happens. We can also consider erroring out KCM if we see this happen so it's really clear to users that they misconfigured something


Question: var hns HostNetworkService hns = hnsV1{} supportedFeatures := hcn.GetSupportedFeatures() <del> if supportedFeatures.Api.V2 { </del> <ins> if supportedFeatures.RemoteSubnet { </ins> hns = hnsV2{} }

I have been deploying master-branch kubernetes on Windows using flannel/host-gateway networking, with one master, one linux worker, and three windows worker. Recently, kube-proxy stopped running correctly. It exits with an error about not finding the HNS Network. I have confirmed that the HNS network exists on the host with the correct name (cbr0), and have identified the last commit where kube-proxy works. What happened: Starting with the commit merging , kube-proxy outputs the following error message: This is confusing because does show : (nat network removed from above output) What you expected to happen: kube-proxy to work in host-gateway mode. How to reproduce it (as minimally and precisely as possible): Build kubernetes at or after . Start flanneld, kubelet, and kube-proxy on a windows 2019 vm. See kube-proxy fail despite flanneld having created the network. Anything else we need to know?: Environment: Kubernetes version (use ): kube-proxy breaks on commit and later; works on Cloud provider or hardware configuration: vsphere OS (e.g: ): windows server 2019, with up-to-date patches Kernel (e.g. ): windows nodes are , linux master/worker are Install tools: deployed using /sig windows cc
I'm aware of this issue. The fix will be out within an hour.


Question: // Make the environment variables for a pod in the given namespace. func (kl *Kubelet) makeEnvironmentVariables(pod *v1.Pod, container *v1.Container, podIP string) ([]kubecontainer.EnvVar, error) { var result []kubecontainer.EnvVar <ins> enableServiceLinks := v1.DefaultEnableServiceLinks if pod.Spec.EnableServiceLinks != nil { enableServiceLinks = *pod.Spec.EnableServiceLinks } </ins> // Note:  These are added to the docker Config, but are not included in the checksum computed // by kubecontainer.HashContainer(...).  That way, we can still determine whether an // v1.Container is already running by its hash. (We don't want to restart a container just

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: Kubelet panics after registering node What you expected to happen: Kubelet should not panic How to reproduce it (as minimally and precisely as possible): Use kubelet on an ARM64 machine (i'm on Scaleway provider) Anything else we need to know?: I upgraded from 1.12.3 to 1.13.0 here is the statup log (only the end): Environment: Kubernetes version (use ): 1.13.0 Cloud provider or hardware configuration: Scaleway OS (e.g. from /etc/os-release): Debian GNU/Linux 9 (stretch) Kernel (e.g. ): 4.15.11-mainline-rev1 Install tools: Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
/sig node
a few questions: can you provide full output of , including the server version? did you upgrade kube-apiserver to 1.13.0? are you running static pod manifests on the kubelet?
hello i had rollbacked this node cannot currently no i didn't. I just upgraded on one server kube-proxy and kubelet. Kube-proxy works fine. i don't have static pods, just daemonsets on this node (metrology)
kubelet must not be newer than kube-apiserver be sure to upgrade kube-apiserver to 1.13 before upgrading kubelet
/close
Closing this issue. <detailsIn response to : /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
i understand but why not just have a proper error instead of a ugly stack trace then ? it's more proper than an unhandled error. Here it's a memory error, a memory error is a grave error.
/reopen cc for disposition
Reopened this issue. <detailsIn response to : /reopen cc for disposition Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.

</details
code in question is here: note that this is an unsupported configuration
Refer the discussion here:
Solved using 1.13.0 api server. But i really think we need to handle this more properly than a stack
Should kubelet mark the node NotReady if it sees an older apiserver?
it sounds reasonable, and if we can have a description on the node saying: not compatible can be better


Question: kubectl version`)) ) <del> type VersionOptions struct { </del> <ins> // Options is a struct to support version command type Options struct { </ins> ClientOnly bool Short      bool Output     string

Anyone itching to understand the code and try fixing things in Kubernetes, please help fixing the lint failures Step 1: Remove one or two packages (related packages) from the list in - (note: do not change generated files, the name of defaulting functions, or the name of conversion functions... packages failing linting due to issues in those files will need to be left alone) Step 2: Run and look at the logged errors Step 3: Fix the code to ensure that runs cleanly. Step 4: Run and to make sure we fix any problems with gofmt or bazel. Step 5: File a PR with the change Step 6: Go to step :) Please use the tracking spreadsheet For everyone who is going to contribute, please append the packages you picked at the end. For existing contributor, please make sure this list is update to date
/sig architecture /sig testing
Hey, I would like to resolve this issue. Could you assign it to me ?
there are a lot of packages so there will be many people working on this. Also we cannot assign issue to folks who are not in the github org for kubernetes. please feel free to pick up a few packages like i mentioned and file a PR. Make sure you link back to this issue, so folks can see which packages you filed PR for.
OK. I see. I will firstly pick up the package called to fix.
Hi dims. Should i submit my PR before i remove one or two packages (related packages) from the list in - Or I first remove the package from list, then I fix the issue and submit a PR?
in your PR, you will have both the fixes to packages and some lines removed from /.golint_failures
When I run , I found some errors as below: I found that there is a go file in But there is still a error.
Say your GOPATH looks like this Your kubernetes directory should be () cd to and then run from that directory.
Is it necessary to obey all rules in golint? Rules in golint are set in stone. Maybe using gometalinter to ignore some specific errors is better?
I revise some erros about golint in some packages. Please review it.
which errors would you like to ignore?

if you can open a new issue with the list of errors to ignore (and proposing gometalinter), we can get some opinions there from others in the community.
Let me take the kubectl cmd part...
Can someone please review and let me know if I am doing it right?
I ll work on all the packages related to
we get 800-ish pkgs left to fix. my concern is that if we fix them package by package, then such pulls will be flooding the repo. can't we possibly make it in larger pulls? the larger the better?
+1 to slightly larger pulls with related packages. if allow too large then the reviewers will not be able to do a thorough review. we should spread the load among multiple reviewers. (if we allow a really large pull, we will end up with top level reviewers who will face the burden and the PR(s) will just stagnate)
oh good point. we could put one pkg's change into one commit, and merge all changes under one area into one pull so that it wont spread the load. wdyt? for each pull, we can have hundred-ish lines of change. but it will still be 30~50 pulls i guess? good news is that we can take our time to deal with it, at least before 1.12
sounds great! please feel free to help make it happen :) (by suggesting groups of packages that kinda go together)
I'm looking at fixing pkg/proxy, and I think I see what was talking about ignoring some rules. proxy.ProxyProvider is being suggested because golint is saying that it stutters. In that example, I think it's okay renaming it to proxy.Provider, but I could see an example where you would want to keep the stutter if the package that it's declared in uses it heavily.
sure thing, feel free to propose a PR for switching to gometalinter independently of other patches. We can work through that there.
I sent the pull request to fix pkg/volume/aws_ebs
I've fixed a few of these, but get the following error when I try to run : Are there any instructions on what I need to setup in order to resolve this?

you can find this file in
I have raised a PR to fix golint errors in pkg/api
I have fixed some pkgs under /pkg/kubelet pkg/kubelet/images pkg/kubelet/kuberuntime pkg/kubelet/leaky pkg/kubelet/pleg
I submitted a pull request for golint errors in pkg/security/ edit: I can work on picking up pkg/controller/bootstrap through pkg/controller/namespace/... (114-129)
fix golint errors in package cmd/kubeadm/app, since the codes under cmd/kubeadm/app/apis/kubeadm/v1alpha2 and cmd/kubeadm/app/apis/kubeadm/v1alpha3 are auto-generated, i keep them untouched.
I am working on some packages under pkg/kubelet, anyone interested in this plz leave messages for me :)
fix golint for pkg/util/dbus pkg/util/labels pkg/util/node pkg/util/strings pkg/util/workqueue/prometheus
PR fixes golint errors in pkg/routes,pkg/securitycontext, pkg/serviceaccount
PR fixes golint in pkg/probe
should we have a checklist as part of the issue summary saying what is in progress and what is done? Since this issue is getting long, this will make new contributors to pick up issues easier.
Great idea!, want to throw a google doc with a check-list so it's easy for participants to organize themselves? (please!) If it is the issue summary then one of us with privileges will be a bottle neck.
Created a with all the package that have on going PRs. Maybe you can add this to the summary. For everyone who is going to contribute, please append the packages you picked at the end. For existing contributor, please make sure this list is update to date
Thanks updated the spread sheet
Raised PR to fix golint issues in: pkg/controller/clusterroleaggregation pkg/controller/cronjob pkg/controller/daemon pkg/controller/deployment pkg/controller/deployment/util pkg/controller/disruption pkg/controller/endpoint
This issue is close in accident by . could you reopen it?
/reopen
Reopening this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I found a problem trying to fix golint messages for .

Golint complaints on 's function because of the snake case. When I try to fix it, fails because it is being used in a code generated file. If I clean generated files, tests pass, but the generated file is not invoking it anymore. I failed to find the generator definition using this so I could change the hook to just . Also I'm not sure if this is wanted. I also found other packages in also probably affected by this. I read somewhere about replacing golint with gometalinter, which would allow to define directives in code to omit certain rules. I would consider to have a second exclusion file for golint with these cases, to be updated if move to gometalinter is finally done. This would make this issue completion more feasible. Meanwhile, I'll move onto fixing golint messages for other packages, until this discussion gets cleared.
I also found a problem fixing . A test fails because the directory is marked as readonly (see tests @ Shall we also ignore this directory for this issue?
Is this issue still relevant because hack/verify- is giving me no errors.
the failures are still present in there are details in the first message on how to claim + solve issues :)
Sorry I misunderstood the issue
I'm working on this package:
pkg/apis/admission/v1beta1 pkg/apis/authentication/v1beta1 pkg/apis/authorization/v1beta1 pkg/apis/coordination/v1 pkg/apis/coordination/v1beta1 pkg/apis/imagepolicy/v1alpha1 pkg/apis/scheduling/v1alpha1 pkg/apis/scheduling/v1beta1 pkg/apis/settings/v1alpha1 pkg/apis/storage/v1alpha1
pkg/cloudprovider/providers/vsphere/vclib pkg/kubelet/apis/deviceplugin/v1alpha pkg/kubelet/apis/pluginregistration/v1alpha1 pkg/kubelet/apis/pluginregistration/v1beta1
:
I'm looking at the pkg/registry/*, all those which are left. I expect to give an update on what all is fixed in the next 24-48 hours.
hi there is a google sheet that is a bit outdated, maybe you can update that?
hi I seem to only have view access for the sheet. Can't edit.
I fixed the permission.

Please try it again
pkg/apis/authentication pkg/apis/coordination pkg/apis/events pkg/apis/imagepolicy pkg/apis/settings
hi I think this issue was closed prematurely by (using fixes #issue in commit)
I'll put in a PR shortly for the package
Looking at next
hi thanks! there is a spreadsheet here that tracks the progress on fixing these issues, consider adding the packages you're work on there so we can keep everything organized :+1:
Starting on now
I did indeed see the Google Sheet link (thank you and to take it a little further
test/e2e
test/e2e_node/builder
pkg/registry/admissionregistration/mutatingwebhookconfiguration/storage pkg/registry/admissionregistration/rest pkg/registry/admissionregistration/validatingwebhookconfiguration/storage pkg/registry/apps/daemonset/storage
Hi. I opened a PR - Please review.
Working on Will open a PR soon.
Starting on and should have a PR soon
I keep getting this golint verify error on and was wondering if others saw it: But when I that package to the golint failures in my PR, my reviewer said it isn't necessary? What does that mean? What did I miss? Should I use a different kind of verify? cf:
looks like the disconnect is because that problem does not show up in the version of we use in the CI. please see
Still confused: is that not the golint version I ran? it installs from vendor and uses that I think?
i installed from and i still don't see the problem, can you please check what version of golint you have?
I'm confused why I wouldn't be using the vendored version? how would that get out of sync?
/me looking
Note: I am reserving for a KubeCon Contributor Workshop demonstration. It is noted in the spreadsheet.
I think may not be ideal cause there's a lot of auto generated files there. What do you think about reserving instead for the KubeCON Contributor Workshop?
pkg/apis/scheduling pkg/apis/storage/util pkg/apis/storage/v1beta1/util pkg/apis/storage/v1/util
/priority backlog
Hi I am a new contributor here, I'd like to contribute and make changes to the following file/s. I have marked it as WIP in the Google Sheet. Can someone let me know if I have to do anything else before I start working on this PR?

Umm so there's an issue where Golint tells that underscore shouldn't be allowed. Here's a log of the go-lint warnings:
Thanks The following should not be modified, as they are designed like this for code generation.
I'd like to pick a package and do my first PR, is there a list of packages to be done? Or should I correlate with this issue thread? Thanks in advance!
please see step 1 () please scan other PRs linked to this issue as well just in case.
Will do. Thank you. On Thu, Oct 3, 2019, 7:25 AM Davanum Srinivas <:
Hi there, I'm looking for a good first issue. I'll start taking a look at if that's ok? There doesn't seem to be anyone currently tracked as looking at it in the attached spreadsheet.
I've found a few others I'm actively looking at, you won't get any argument from me. On Sat, Oct 12, 2019, 10:38 AM Pamela Kelly <:
I am working on the iptables pkg. Updated in the google docs
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I have problem running in step 2 $ hack/verify- what should I do?
PTAL for my pr to this issue, thx
you can move kubernetes directory to and re-run this script.
I'd like to take Can we fix linter issues for generated files? Or we should skip packages with zz_generated?
There're 2 lint errors that affects 292/507 packages, meaning it's not possible to cleanup the entire file without solving it: default pattern of Kubectl is the usage of [cmd]Options for initial options structs and it raises: /apis/{custommetrics,externalmetrics} for example raises the underscore error, but It's probably not worth to break backward compatibility renaming these folder for lint error fix: Any advice on these?

/reopen hm, cannot reopen this issue, it was closed by mistake
You can't reopen an issue/PR unless you authored it or you are a collaborator. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I am sure this is all not fixed. Am I missing something? ./hack/verify- installing golint from vendor Congratulations! All Go source files have been linted.
you need to remove package from first.
/remove-sig testing SIG testing doesn't own code failing lints, we own the linter. :upsidedownface:
Hello, is this still a relevent issue ? I would like to work upon
I have fixed
Hi Folks, Some of the golint fixes, require us to change the exposed function's name, in simple words, "change API". I was not sure, if that is welcomed and what is the process.
Heres what Im getting after going over the following steps as mentioned by Say your GOPATH looks like this GOPATH=/Users/dims/go Your kubernetes directory should be ($) cd to and then run hack/verify- from that directory. I get the following output - installing golint Congratulations! All Go source files have been linted. Im not really sure what I am doing wrong. Any help is appreciated!
I think I understand. I misunderstood the steps around .golint_failures. The first step is to remove the pkg that we are targeting from this file and then run the My mistake was to assume that the packages in the file need to be removed after we are done cleaning the golint errors.
Can someone please review and let me know if I've done this right?
After a couple years of seeing changes spawned from this issue, I don't think this is a good use of contributors' or reviewers' time.

golint is very opinionated about: exported type and function names, leading to PRs attempting to change exported things people use downstream missing documentation on interface implementation functions, which leads to not-very-useful copy/paste of interface godoc the govet and staticcheck verify jobs are much more useful since they catch correctness issues, but I would recommend closing this issue and dropping the golint checks from CI golint doc itself discourages from using its output in an automated enforced way: from
i guess one can say gofmt is opinionated too, but something good about Golang is that it comes with an opinion and a set of standard tools. for API documentation at least we found inconsistencies on how most exported API items are documented: golint can help with that, but if there are no "pragmas" to avoid breaking changes and given we have codegen generating non-compliant names enabling golint on API packages is sort of blocked in Kubernetes. for non-API, non-library, non-public code, i think being inconsistent is mostly OK. no preference on my side. shrug
true, but code formatting has the benefits of mostly eliminating meaningless diffs in PRs due to code movement/whitespace/etc if there are specific packages (like API packages) we lack documentation on, adding checks specifically for that could make sense exactly, I don't think golint is the right tool for making sure we have API docs
golint itself doesn't have these and the go project is against this sort of comment AIUI, however golangci-lint does have them. I think there are plenty of other static analysis lints we should use instead, beyond just staticcheck and vet there is e.g.
btw, if we remove golint can we see e.g. all-caps constants or functions with underscores dodging review? perhaps we should add golangci-lint soon after:
Completely agree there are better places to spend review bandwidth. I was curious to see what the effort looked like. There have been 890 commits that touched over the past 3+ years !
/remove-help /remove-good-first-issue
Overall agree on the bandwidth. Also note that it was a good first issue with very low bar to get them in the repo. May be we need to find a similar low bar, but high value good first issue.

UT's come to mind, are there other things we curate for new contributors ?
Do we have an alternative way of enforcing that new exported code (not necessarily API fields) has at least some minimum documentation? As a reviewer, I ask contributors to add meaningful comments. But reality is that I sometimes stumble into old code that doesn't have documentation because we didn't have golint at some point. I agree that fix-golint PRs just add burden to reviewers. But, before removing the lint check, can we wait and see if it's enough to discourage people from sending more PRs like this? Additionally, we can encourage reviewers to just close PRs that only fix golint issues.
+1 for adding burden to reviewers. However, as a new contributor, I think this a very good opportunity for new contributors to study the code base and submit their first PR. In my opinion, I think this issue would be able to attract more new contributors, get familiar with the code base, and get familiar with the git workflow.
thanks everyone for getting us this far. We essentially gave up on the existing structure/system for golint: At some point we can look at adding golangci-lint, there's a WIP in progress in /close
Closing this issue. <detailsIn response to : thanks everyone for getting us this far. We essentially gave up on the existing structure/system for golint: Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: return allErrs } <del> // validateTopologySelectorLabelRequirement tests that the specified TopologySelectorLabelRequirement fields has valid data func validateTopologySelectorLabelRequirement(rq core.TopologySelectorLabelRequirement, fldPath *field.Path) field.ErrorList { </del> <ins> // validateTopologySelectorLabelRequirement tests that the specified TopologySelectorLabelRequirement fields has valid data, // and constructs a set containing all of its Values. func validateTopologySelectorLabelRequirement(rq core.TopologySelectorLabelRequirement, fldPath *field.Path) (sets.String, field.ErrorList) { </ins> allErrs := field.ErrorList{} <ins> valueSet := make(sets.String) valuesPath := fldPath.Child("values") </ins> if len(rq.Values) == 0 { <del> allErrs = append(allErrs, field.Forbidden(fldPath.Child("values"), "must specify as least one value")) </del> <ins> allErrs = append(allErrs, field.Required(valuesPath, "")) </ins> } <ins> // Validate set property of Values field for i, value := range rq.Values { if valueSet.Has(value) { allErrs = append(allErrs, field.Duplicate(valuesPath.Index(i), value)) } valueSet.Insert(value) } </ins> allErrs = append(allErrs, unversionedvalidation.ValidateLabelName(rq.Key, fldPath.Child("key"))...) <del> return allErrs </del> <ins> return valueSet, allErrs </ins> } <del> // ValidateTopologySelectorTerm tests that the specified topology selector term has valid data func ValidateTopologySelectorTerm(term core.TopologySelectorTerm, fldPath *field.Path) field.ErrorList { </del> <ins> // ValidateTopologySelectorTerm tests that the specified topology selector term has valid data, // and constructs a map representing the term in raw form. func ValidateTopologySelectorTerm(term core.TopologySelectorTerm, fldPath *field.Path) (map[string]sets.String, field.ErrorList) { </ins> allErrs := field.ErrorList{} <ins> exprMap := make(map[string]sets.String) exprPath := fldPath.Child("matchLabelExpressions") </ins> if utilfeature.DefaultFeatureGate.Enabled(features.DynamicProvisioningScheduling) { <ins> // Allow empty MatchLabelExpressions, in case this field becomes optional in the future. </ins> for i, req := range term.MatchLabelExpressions { <del> allErrs = append(allErrs, validateTopologySelectorLabelRequirement(req, fldPath.Child("matchLabelExpressions").Index(i))...) </del> <ins> idxPath := exprPath.Index(i) valueSet, exprErrs := validateTopologySelectorLabelRequirement(req, idxPath) allErrs = append(allErrs, exprErrs...) // Validate no duplicate keys exist. if _, exists := exprMap[req.Key]; exists { allErrs = append(allErrs, field.Duplicate(idxPath.Child("key"), req.Key)) } exprMap[req.Key] = valueSet </ins> } } else if len(term.MatchLabelExpressions) != 0 { allErrs = append(allErrs, field.Forbidden(fldPath, "field is disabled by feature-gate DynamicProvisioningScheduling")) } <del> return allErrs </del> <ins> return exprMap, allErrs </ins> } // ValidateAvoidPodsInNodeAnnotations tests that the serialized AvoidPods in Node.Annotations has valid data

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --/kind feature The field essentially has the type , but currently there is no validation for set and map semantics, i.e. unique elements / keys. Having this validation at API level removes logic to validate at the volume plugin level. edit: this check will not be in cases where this makes it difficult to add other fields and make this field optional. /sig api-machinery /sig storage
/assign
Thanks for catching this. Would you be able to implement this for this release? We want to try to go beta if we can, and we won't be able to strengthen validation once that happens.
Yes will do!


Question: taintArgs := []string{} metTaintArg := false for _, s := range args { <del> isTaint := strings.Contains(s, "=") || strings.HasSuffix(s, "-") </del> <ins> isTaint := strings.Contains(s, "=") || strings.Contains(s, ":") || strings.HasSuffix(s, "-") </ins> switch { case !metTaintArg && isTaint: metTaintArg = true

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: says: That doesn't work: It doesn't matter whether there is a node "foo" or not. It doesn't seem to contact the API at all. What you expected to happen: Examples form the help texts should work. Anything else we need to know?: Looking at , it seems to want either a or in the argument. Indeed, does work and does the right thing, i.e. set a taint with key and no value. Environment:
i believe there were updates about this at some point. a PR is welcome /remove-kind bug /kind documentation /sig cli side note: Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.7", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-12-11T12:34:17Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"} always make sure the client MINOR version is equal or +1, or -1 of the server MINOR version. in this case it's +2, which is unsupported.
/assign bharaththiruveedula
Do you mind if I submmit a PR on this because I met this issue before?
I assigned the issue today itself and already started working on it. I will send the PR by end of today. But thanks for showing the interest.
Are we updating the documentation or the code now? Right now fails but works (because of the "-"), which isn't very consistent at all, so I'd suggest we actually fix kubectl itself to do what the help page says.


Question: return err } <ins> if len(e.Subsets) == 0 { // no action is needed to remove the endpoint return nil } </ins> // Remove our IP from the list of addresses new := []corev1.EndpointAddress{} for _, addr := range e.Subsets[0].Addresses {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: In kubernetes 1.15, Panic happends in kube-apiserver. Subsets[0] is nil when endpoints has no subsets The Panic happens in here: And everything returns to normal when we delete the endpoints in etcd What you expected to happen: Maybe kube-apiserver should print error,not panic How to reproduce it (as minimally and precisely as possible): The host's iowait of ETCD is high Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration:NA OS (e.g: ):Kernel (e.g. ):Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig release
/remove-sig release /sig api-machinery
/assign


Question: schedulerapi.DefaultPercentageOfNodesToScore, false) cache.UpdateSnapshot(s.(*genericScheduler).nodeInfoSnapshot) return s.(*genericScheduler) } func TestFindFitAllError(t *testing.T) {

How to reproduce it (as minimally and precisely as possible): pre-requisites: a multi-az cluster, a deployment with 1 pod in each az. a with and . the node(s) in 1 az. the deployment's pod(s) from the cordoned az. What happened: A new pod was created and scheduled in a form that What you expected to happen: I expected that a new pod will be created and be "stuck" in status as it shouldn't have nowhere be scheduled. Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: AWS OS (e.g: ):Kernel (e.g. ): 4.19.84-coreos Others: I created a with how I reproduce the bug
/sig scheduling
cc/
Thanks for raising this issue. I feel like it's a general timing issue - when you delete a Pod, that Pod may still exist for a very short time (in graceful deletion period), and at the same time, the controller backfills a Pod for it. So when scheduling tries to schedule the backfill Pod, it may read the cluster as: [1, 1, 1(deleting)]. To confirm this, can you try deleting the Pod using , and see if the problem goes away?
Yes sure I will try that. It sounds reasonable, thats why in the first demonstration I deleted all the pods and waited for all of them to completely disappear. Nevertheless, this is still something that I believe needs to be fixed. The pod's evaluation should take in consideration pods that are in status and don't count them as pods.
I do believe we exclude the Pods. (if not, it should be a serious regression). What concerns me is that at that moment, scheduler may treats that (deleting) Pod as running in its cache. I will try to debug into this.
Yep, this time the new pod was in status until I uncordoned the relevant nodes.


Question: return gce, nil } <ins> // determineSubnetURL queries for all subnetworks in a region for a given network and returns // the URL of the subnetwork which exists in the auto-subnet range. func determineSubnetURL(service *compute.Service, networkProjectID, networkName, region string) (string, error) { subnets, err := listSubnetworksOfNetwork(service, networkProjectID, networkName, region) if err != nil { return "", err } autoSubnets, err := subnetsInCIDR(subnets, autoSubnetIPRange) if err != nil { return "", err } if len(autoSubnets) == 0 { return "", fmt.Errorf("no subnet exists in auto CIDR") } if len(autoSubnets) > 1 { return "", fmt.Errorf("multiple subnetworks in the same region exist in auto CIDR") } return autoSubnets[0].SelfLink, nil } </ins> func tryConvertToProjectNames(configProject, configNetworkProject string, service *compute.Service) (projID, netProjID string) { projID = configProject if isProjectNumber(projID) {

/kind bug What happened: Continuation of . The fix to that bug only takes the first subnetwork in the correct region. However, if the user manual subnets in the region that the cluster exists, it's possible the cloudprovider could choose the wrong subnet. What you expected to happen: If no subnet is specified for a non-legacy network, always take the subnet which was automatically created. /assign /sig network /area platform/gce


Question: }}, }, }, <ins> { testName:      "endpoint with no subset", serviceName:   "foo", ip:            "5.6.7.8", endpointPorts: []corev1.EndpointPort{{Name: "foo", Port: 8080, Protocol: "TCP"}}, endpointKeys:  []string{"1.2.3.4", "4.3.2.2", "4.3.2.3", "4.3.2.4"}, endpoints: &corev1.EndpointsList{ Items: []corev1.Endpoints{{ ObjectMeta: om("foo"), Subsets:    nil, }}, }, }, </ins> } for _, test := range stopTests { t.Run(test.testName, func(t *testing.T) {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: In kubernetes 1.15, Panic happends in kube-apiserver. Subsets[0] is nil when endpoints has no subsets The Panic happens in here: And everything returns to normal when we delete the endpoints in etcd What you expected to happen: Maybe kube-apiserver should print error,not panic How to reproduce it (as minimally and precisely as possible): The host's iowait of ETCD is high Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration:NA OS (e.g: ):Kernel (e.g. ):Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig release
/remove-sig release /sig api-machinery
/assign


Question: dc.sync(ctx, pdbName) ps.VerifyDisruptionAllowed(t, pdbName, 0) <ins> verifyUnsupportedControllerEventEmitted(t, dc) </ins> } // Verify that disruption controller is not erroring when unmanaged pods are found

When using PDBs with or (percentage), it is not supported to reference pods via a PDB selector, that do not have a supported controller owner reference: Deployment, ReplicationController, ReplicaSet, and StatefulSet controllers or a resource implementing subresource This is also documented Nevertheless, there are still people who create PDBs a pods without controllers that ignore this. For example PDBs that set , want to block eviction, and node drain (it works for them but the PDB status is invalid). Unfortunately there is no feedback for the user to receive when this happens. The KCM logs is the only place that informs the user when unmanaged pods are found. Which I think is insufficient. There should be some way to inform users about this. Since it cannot be know on admission we probably have to do it when reconciling the PDB and unmanaged pods are found. Options: an event when this is detected - probably needs to be throttled in some way a condition in a PDB status that would be set and admins could monitor these to detect misbehaving PDBs create a PDB create unmanaged pod observe incorrect status because this configuration is unsupported No response <detailsNA </details<detailsNA </details<detailsNA </details<details</details<details</details<details</details
/sig apps
/triage accepted This was discussed during today's sig-apps, we've agreed to add a warning event right next to the log statement from the original description
I'll take this on.
/assign
/reopen I did some more testing for the unsupported controllers and found out that these are covered by the event, so I posted a followup to reflect that The new event should cover only the naked pods
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: expectedContainer: "bar", name:              "container in flag", }, <ins> { p:                 &AttachOptions{ContainerName: "initfoo"}, args:              []string{"foo"}, expectedPod:       "foo", expectedContainer: "initfoo", name:              "init container in flag", }, </ins> } for _, test := range tests { f, tf, codec := NewAPIFactory() tf.Client = &fake.RESTClient{

Related issue
is there a simple example that you used to recreate the problem?
I used exactly the example below: So Pod sflask-w-init gets into pending state "Init:0/1" forever. Then you can try to run and you can see that is not working while invoking corresponding command with docker works - it displays current date every second.
thanks for the detailed test, please see the PR :)
Duh! already has a PR


Question: Failf("Failed to get pod: %v", err) } fmt.Printf("%+vn", podWithUid) <ins> var events *api.EventList </ins> // Check for scheduler event about the pod. By("checking for scheduler event about the pod") <del> events, err := c.Events(api.NamespaceDefault).List( labels.Everything(), fields.Set{ "involvedObject.kind":      "Pod", "involvedObject.uid":       string(podWithUid.UID), "involvedObject.namespace": api.NamespaceDefault, "source":                   "scheduler", }.AsSelector(), ) if err != nil { Failf("Error while listing events: %v", err) } Expect(len(events.Items)).ToNot(BeZero(), "scheduler events from running pod") fmt.Println("Saw scheduler event for our pod.") </del> <ins> expectNoError(wait.Poll(time.Second*2, time.Second*60, func() (bool, error) { events, err := c.Events(api.NamespaceDefault).List( labels.Everything(), fields.Set{ "involvedObject.kind":      "Pod", "involvedObject.uid":       string(podWithUid.UID), "involvedObject.namespace": api.NamespaceDefault, "source":                   "scheduler", }.AsSelector(), ) if err != nil { return false, err } if len(events.Items) > 0 { fmt.Println("Saw scheduler event for our pod.") return true, nil } return false, nil })) </ins> // Check for kubelet event about the pod. By("checking for kubelet event about the pod") <del> events, err = c.Events(api.NamespaceDefault).List( labels.Everything(), fields.Set{ "involvedObject.uid":       string(podWithUid.UID), "involvedObject.kind":      "Pod", "involvedObject.namespace": api.NamespaceDefault, "source":                   "kubelet", }.AsSelector(), ) if err != nil { Failf("Error while listing events: %v", err) } Expect(len(events.Items)).ToNot(BeZero(), "kubelet events from running pod") fmt.Println("Saw kubelet event for our pod.") </del> <ins> expectNoError(wait.Poll(time.Second*2, time.Second*60, func() (bool, error) { events, err = c.Events(api.NamespaceDefault).List( labels.Everything(), fields.Set{ "involvedObject.uid":       string(podWithUid.UID), "involvedObject.kind":      "Pod", "involvedObject.namespace": api.NamespaceDefault, "source":                   "kubelet", }.AsSelector(), ) if err != nil { return false, err } if len(events.Items) > 0 { fmt.Println("Saw kubelet event for our pod.") return true, nil } return false, nil })) </ins> }) })

Still early in diagnosis, but soon after cutting 0.17.0, our early internal testing show this test as flaking. It's pretty bizarre, though, given the test isn't flaking on GCE, or even the continuous integration runs on GKE.
cc
So far, I've tried reproing this on a bare cluster 100 times, no joy. Now I'm trying up/test/down, no immediate joy (but it's a flake). If I get to the point of testing up/e2e/down, I might as well call it "Jenkins" and be done with it, but I guess it's time to start looking through logs instead. (As for the P0, trying to figure out if we need a cherry-pick, or if we let the flake proceed on in the entire 0.17.0 train.)
+1 to ironing out a flake; v0.16.0 had a flake (or maybe even a consistently-failing test) which made us turn off tests that would run against it for weeks. This is especially important now that we're doing soak tests.
(since it's related to events)
The test itself looks racy; there's nothing to guarantee that the events will have made it through the system by the time the pod is running. Generally one would expect that, but there's no guarantee.
Can you paste some sample failures here?
go/k8s-test/job/kubernetes-e2e-gke-test/4534/console
Actually this test is flaky across the board, and has been for some time. If you look at kubernetes-e2e-gce Test Results Analyser in Jenkins, you'll see about 8 failures in the past few hundred runs. I think lavalamp is correct - the test needs to retry/wait for the events to get through. Specifically, this code needs to be wrapped in a util.Poll():
Yeah if the kubelet event shows up less often than the scheduler event, that would make sense, since there'll have been less time.
I can whip up a quick fix now.
If it's a known flake, it's not going to block us. We should get a fix in soon for the next release, though.
Nearly done.