Question: }}, }, } volume := &ec2.Volume{ AvailabilityZone: aws.String("us-east-1a"), VolumeId:         aws.String("vol-volumeId0"), <ins> State:            aws.String("available"), </ins> } awsServices.ec2.(*MockedFakeEC2).On("CreateVolume", request).Return(volume, nil) <ins> describeVolumesRequest := &ec2.DescribeVolumesInput{ VolumeIds: []*string{aws.String("vol-volumeId0")}, } awsServices.ec2.(*MockedFakeEC2).On("DescribeVolumes", describeVolumesRequest).Return([]*ec2.Volume{volume}, nil) </ins> volumeID, err := c.CreateDisk(volumeOptions) assert.Nil(t, err, "Error creating disk: %v", err) assert.Equal(t, volumeID, KubernetesVolumeID("aws://us-east-1a/vol-volumeId0"))

What happened: Kubernetes tries to attach EBS volumes while their status is still : I believe this started to happen after , when volumes and their tags started to be created in the same request. Before that, tags would be set through another call (i.e., ), which retries if the volume isn't . What you expected to happen: Kubernetes should try to attach the volume only when it's available. In order to do that, should make sure the volume is when it returns. How to reproduce it (as minimally and precisely as possible): Just create a dynamically provisioned EBS volume. /kind bug /sig storage /sig aws
CC
What is the downside of this? IIRC - even when we tagged separately, we used to have few cases where by the time attachment was attempted, volume was not available. May be tagging while creating the volume makes it worse but someone has to wait for volume to become available - either attach call or create volume call itself.
In worse case this causes first attach to fail - but controller will retry anyway.
I think it re-introduces
No - looking at events attached to the pod, it looks like volume eventually got attached. I think is unrelated.
It is related. If you create volume with invalid keys, AWS returns you a volume and in this case Kubernetes will create a PV for it. But then the volume silently disappears. The only way how to detect that is to wait for the volume to either get Available or disappear.
we are still waiting for volume to become available if volume has kms key -
+1 on this issue. I have seen this quite often on my v1.12 cluster too. So this is an issue even before
we are still waiting for volume to become available if volume has kms key - Oh, sorry for the noise.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
EBS CSI driver is already regardless of whether kms key is used or not. Any objections to do the same for in-tree?
I like it.


Question: "cs":     "componentstatuses", "ev":     "events", "limits": "limitRanges", <del> "mi":     "minions", </del> <ins> "no":     "nodes", </ins> "po":     "pods", "pv":     "persistentVolumes", "pvc":    "persistentVolumeClaims",

kubectl 0.17.0: curl http://localhost:8080/api/v1beta3/minions { "kind": "Status", "apiVersion": "v1beta3", "metadata": {}, "status": "Failure", "message": "the server could not find the requested resource", "reason": "NotFound", "details": {}, "code": 404 } master3 core # /opt/bin/kubectl get mi master3 core # kubectl 0.16.0: uses: http://localhost:8080/api/v1beta1/minions master3 core # ./kubectl get mi NAME LABELS STATUS 192.168.10.70 <noneReady 192.168.10.71 <noneReady in the same time curl http://localhost:8080/api/v1beta3/nodes works well.
Shall we just remove "minion" from kubectl now and throw an error?
remove "minions"?
Yes, that should be part of . cc
Presumably works?
Assigning who's working on
Ref
Could you please take a look at this?

I know that you can get nodes / a node, but the help doesn't say you can or provide examples: To resolve this issue: the help for this command to include nodes (and any other resources we're missing). we generate the help text in a way that ensures this (and probably the many others that are drifting) don't get out of date? Or at least give us a heuristic as to whether this might be true, even if we have to fix it manually?
Can this be closed?
I'll close this one with ; we want to get rid of the minions (mi) resource name, since it doesn't work for v1beta3.


Question: } } <del> if a.jwtAuthenticator.ClaimMappings.Username.Prefix != nil && *a.jwtAuthenticator.ClaimMappings.Username.Prefix != "" { username = *a.jwtAuthenticator.ClaimMappings.Username.Prefix + username </del> <ins> userNamePrefix := a.jwtAuthenticator.ClaimMappings.Username.Prefix if userNamePrefix != nil && *userNamePrefix != "" { username = *userNamePrefix + username </ins> } info := &user.DefaultInfo{Name: username} <del> if a.jwtAuthenticator.ClaimMappings.Groups.Claim != "" { if _, ok := c[a.jwtAuthenticator.ClaimMappings.Groups.Claim]; ok { </del> <ins> groupsClaim := a.jwtAuthenticator.ClaimMappings.Groups.Claim if groupsClaim != "" { if _, ok := c[groupsClaim]; ok { </ins> // Some admins want to use string claims like "role" as the group value. // Allow the group claim to be a single string instead of an array. // // See: https://github.com/kubernetes/kubernetes/issues/33290 var groups stringOrArray <del> if err := c.unmarshalClaim(a.jwtAuthenticator.ClaimMappings.Groups.Claim, &groups); err != nil { return nil, false, fmt.Errorf("oidc: parse groups claim %q: %v", a.jwtAuthenticator.ClaimMappings.Groups.Claim, err) </del> <ins> if err := c.unmarshalClaim(groupsClaim, &groups); err != nil { return nil, false, fmt.Errorf("oidc: parse groups claim %q: %v", groupsClaim, err) </ins> } info.Groups = []string(groups) } } <del> if a.jwtAuthenticator.ClaimMappings.Groups.Prefix != nil && *a.jwtAuthenticator.ClaimMappings.Groups.Prefix != "" { </del> <ins> groupsPrefix := a.jwtAuthenticator.ClaimMappings.Groups.Prefix if groupsPrefix != nil && *groupsPrefix != "" { </ins> for i, group := range info.Groups { <del> info.Groups[i] = *a.jwtAuthenticator.ClaimMappings.Groups.Prefix + group </del> <ins> info.Groups[i] = *groupsPrefix + group </ins> } }

Follow up PR: would be good to use some local vars to make these statements shorter. Originally posted by in
/sig auth /triage accepted /kind cleanup /assign


Question: // TODO(liggitt): drop this once golang json parser limits stack depth (https://github.com/golang/go/issues/31789) if len(p.patchBytes) > 1024*1024 { v := []interface{}{} <del> if err := json.Unmarshal(p.patchBytes, v); err != nil { </del> <ins> if err := json.Unmarshal(p.patchBytes, &v); err != nil { </ins> return nil, errors.NewBadRequest(fmt.Sprintf("error decoding patch: %v", err)) } }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: It returns an error when I create a pod of yaml file over 1M. I checked the codes and found that the coded wrong. It should be “json.Unmarshal(p.patchBytes, &v)”, not “json.Unmarshal(p.patchBytes, v)” in the 344 and 368 line. What you expected to happen: It can be parsed correctly. How to reproduce it (as minimally and precisely as possible): Look at the code and you'll see or you can test it. Anything else we need to know?: no. Environment: Kubernetes version (use ): Community latest code Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/kind bug
/sig api-machinery
/assign


Question: return err } <del> // TODO: This shouldn't be needed, see corresponding TODO in StatefulSetHasDesiredReplicas. // StatefulSet should track generation number. pods := reaper.podClient.Pods(namespace) selector, _ := metav1.LabelSelectorAsSelector(ss.Spec.Selector) options := metav1.ListOptions{LabelSelector: selector.String()} podList, err := pods.List(options) if err != nil { return err } errList := []error{} for i := range podList.Items { pod := &podList.Items[i] controllerRef := controller.GetControllerOf(pod) // Ignore Pod if it's an orphan or owned by someone else. if controllerRef == nil || controllerRef.UID != ss.UID { continue } if err := pods.Delete(pod.Name, gracePeriod); err != nil { if !errors.IsNotFound(err) { errList = append(errList, err) } } } if len(errList) > 0 { return utilerrors.NewAggregate(errList) } </del> // TODO: Cleanup volumes? We don't want to accidentally delete volumes from // stop, so just leave this up to the statefulset. falseVar := false

StatefulSet now as of 1.6. That means the TODO on can be fixed in 1.7. We could not fix it in 1.6 because that code is used by kubectl, which must work against servers that are one minor version ahead of or behind the client.
/cc
I am volunteered to take this job if no one start working on this.
I don't think anyone's working on it, so you are welcome to take it. Feel free to tag me as a reviewer. Thanks!


Question: TEMP_DIR:=$(shell mktemp -d) ifeq ($(ARCH),amd64) <del> BASEIMAGE?=k8s.gcr.io/build-image/debian-base:buster-v1.2.0 </del> <ins> BASEIMAGE?=k8s.gcr.io/build-image/debian-base:buster-v1.3.0 </ins> endif ifeq ($(ARCH),arm) <del> BASEIMAGE?=k8s.gcr.io/build-image/debian-base-arm:buster-v1.2.0 </del> <ins> BASEIMAGE?=k8s.gcr.io/build-image/debian-base-arm:buster-v1.3.0 </ins> endif ifeq ($(ARCH),arm64) <del> BASEIMAGE?=k8s.gcr.io/build-image/debian-base-arm64:buster-v1.2.0 </del> <ins> BASEIMAGE?=k8s.gcr.io/build-image/debian-base-arm64:buster-v1.3.0 </ins> endif ifeq ($(ARCH),ppc64le) <del> BASEIMAGE?=k8s.gcr.io/build-image/debian-base-ppc64le:buster-v1.2.0 </del> <ins> BASEIMAGE?=k8s.gcr.io/build-image/debian-base-ppc64le:buster-v1.3.0 </ins> endif ifeq ($(ARCH),s390x) <del> BASEIMAGE?=k8s.gcr.io/build-image/debian-base-s390x:buster-v1.2.0 </del> <ins> BASEIMAGE?=k8s.gcr.io/build-image/debian-base-s390x:buster-v1.3.0 </ins> endif RUNNERIMAGE?=gcr.io/distroless/static:latest

What happened: The latest release of the debian-base image, v2.1.3, contains a vulnerable version of the apt package. CVE-2020- impacts apt version 1.8.2.1. What you expected to happen: Image should be updated to use at least apt 1.8.2.2 to clear the vulnerability. How to reproduce it (as minimally and precisely as possible):
/sig release
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
kube-proxy is using debain as base image. /assign Let me check other components
checked the base image in and I seems that no new debian or ubuntu official image is built for this CVE ?
That is correct, the purpose of this issue is to get a new image built and released, one which contains a resolution for the CVE.
per comments of @ tianon in Will update it once debain releases it.


Question: t.Error("got incorrect fault domain") } } <ins> func TestSplitProviderID(t *testing.T) { providers := []struct { providerID string name       types.NodeName fail bool }{ { providerID: CloudProviderName + ":///subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/myResourceGroupName/providers/Microsoft.Compute/virtualMachines/k8s-agent-AAAAAAAA-0", name:       "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/myResourceGroupName/providers/Microsoft.Compute/virtualMachines/k8s-agent-AAAAAAAA-0", fail:       false, }, { providerID: CloudProviderName + ":/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/myResourceGroupName/providers/Microsoft.Compute/virtualMachines/k8s-agent-AAAAAAAA-0", name:       "", fail:       true, }, { providerID: CloudProviderName + "://", name:       "", fail:       true, }, { providerID: ":///subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/myResourceGroupName/providers/Microsoft.Compute/virtualMachines/k8s-agent-AAAAAAAA-0", name:       "", fail:       true, }, { providerID: "aws:///subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/myResourceGroupName/providers/Microsoft.Compute/virtualMachines/k8s-agent-AAAAAAAA-0", name:       "", fail:       true, }, } for _, test := range providers { name, err := splitProviderID(test.providerID) if (err != nil) != test.fail { t.Errorf("Expected to failt=%t, with pattern %v", test.fail, test) } if test.fail { continue } if name != test.name { t.Errorf("Expected %v, but got %v", test.name, name) } } } </ins>

This is required to make Cloud Controller Manager operational on the various clouds that Kubernetes runs on. We have already PRs for the following clouds: GCE - AWS - Azure - OpenStack - The following clouds still need to be addresses: CloudStack Mesos OVirt Photon Rackspace vSphere If anyone is interested in contributing to this effort, please ping me directly (on kubernetes slack or through email)
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
/sig cluster-lifecycle
cc:
I will take care of the vSphere and Photon.
I will take care of the Rackspace.
Merging for OpenStack
Reposting I am trying to use my custom out-of-tree cloud provider with Kubernetes. But I noticed that kubelet calls apis like with empty for cloud provider. Digging in the code for 1.7 branch, I see that is null for cloud-providers. As a result returns empty string. As a result, ByID() methods don't work for cloud providers since is not set.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close


Question: finalAttached:        true, trigerWatchEventTime: 5 * time.Millisecond, timeout:              50 * time.Millisecond, <del> sleepTime:            5 * time.Millisecond, </del> shouldFail:           false, }, {

/kind bug /sig storage What happened: CSI unit tests occasionally end up with data race: How to reproduce it (as minimally and precisely as possible): to get CSI unit tests into a binary: the binary 500 times in parallel until it breaks: . Use different argument to really load your machine, it should not be idle! . On my Linux, I got 3 failures in ~1000 runs. puts stderr of failed runs into .
/assign
I make a Fix: PTAL


Question: "github.com/onsi/ginkgo" ) <del> var _ = SIGDescribe("DNS", func() { </del> <ins> var _ = SIGDescribe("[Feature:Windows] DNS", func() { </ins> ginkgo.BeforeEach(func() { e2eskipper.SkipUnlessNodeOSDistroIs("windows")

What happened: Some of our windows tests (in this case ) dont add enough scheduling metadata to behave predictably in environments where, for example, is required to interpret a pods proper location. A "bug" in EKS CNI actually causes EKS, for example, to not properly label such pods with IP addresses (because it uses an adm controller to add metadata before a pod is created, and that md is sent to eks's cni). In any case adding this metadata is a standard in Sig-windows, per conv with yesterday, so we'll want to add these in. As an example of the test failure which results when you dont have this selector.... Looking at the pod Spec: What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig windows


Question: groupVersion := gvr.GroupVersion() result := c.client.clientBase.Patch(pt). AbsPath(c.client.apiPathFor(groupVersion)). <del> Namespace(c.namespace). </del> <ins> NamespaceIfScoped(c.namespace, c.namespace != ""). </ins> Resource(gvr.Resource). Name(name). SubResource("scale").

<!-- Please only use this template for submitting enhancement requests --What would you like to be : The scale client () currently works fine with namespaced resources. However, it doesn't work for non-namespaced resources e.g. cluster-scoped CRDs. The current implementation should be extended in order to support non-namespaced resources. The current interface: /area custom-resources /sig autoscaling
I think scale subresource support for CRDs falls more into sig-apimachinery domain. After DirectXMan12 moved on from sig-autoscaling I don't think there is anyone in our sig-autoscaling with knowledge of scale subresource internals. /sig api-machinery
/assign
The bug report sounds legit. Does the generic scale client need to be changed (I think so), or does solve your issue?
No, the dashboard PR doesn't solve this. I discovered this issue while working on that PR so when gets merged, it will only work for namespaced CRDs. I think the generic scale client needs to be updated, maybe by adding another getter to the ScalesGetter interface: /kind bug
PR:


Question: minimumGCAge            = flag.Duration("minimum_container_ttl_duration", 0, "Minimum age for a finished container before it is garbage collected.  Examples: '300ms', '10s' or '2h45m'") maxContainerCount       = flag.Int("maximum_dead_containers_per_container", 5, "Maximum number of old instances of a container to retain per container.  Each container takes up some disk space.  Default: 5.") authPath                = flag.String("auth_path", "", "Path to .kubernetes_auth file, specifying how to authenticate to API server.") <ins> cAdvisorPort            = flag.Uint("cadvisor_port", 4194, "The port of the localhost cAdvisor endpoint") </ins> apiServerList           util.StringList )

Currently, kubelet uses a for communicating with cAdvisor. , were having an issue because we didn't realize making port available was required for kubernetes.
I prefer not do this since we plan to make 4194 publically reserved port for cAdvisor. Also cAdvisor is the default deamon running on every node of a kubernetes cluster. The issue you encountered sounds like a bad document issue we had for cAdvisor. We should fix the document instead of allow the port being configurable.
can you expound on 'bad document' issue?


Question: numNodes, err := e2enode.TotalRegistered(f.ClientSet) framework.ExpectNoError(err) replicas := int32(numNodes) <del> common.NewRCByName(c, ns, name, replicas, nil) </del> <ins> common.NewRCByName(c, ns, name, replicas, nil, nil) </ins> err = e2epod.VerifyPods(c, ns, name, true, replicas) framework.ExpectNoError(err, "Each pod should start running and responding")

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: ci-kubernetes-e2e-gci-gce-serial Which test(s) are failing: [sig-cluster-lifecycle] Nodes [Disruptive] Resize [Slow] should be able to add nodes [sig-cluster-lifecycle] Nodes [Disruptive] Resize [Slow] should be able to delete nodes Since when has it been failing: These tests have been failing since 6/28 Failures coincide with the following changes: Testgrid link: Reason for failure: The failure messages look as follows: [sig-cluster-lifecycle] Nodes [Disruptive] Resize [Slow] should be able to add nodes [sig-cluster-lifecycle] Nodes [Disruptive] Resize [Slow] should be able to delete nodes /cc /milestone v1.16 /priority critical-urgent /kind failing-test /cc
/sig cluster-lifecycle
For more context, the code for the failing tests is here The error for both comes through  - some Pod is not responding. It is failing to find pods prefixed with "my-hostname-add-node". Within the logs we can see this: So the Pods are stuck in a Pending state. The Pods are supposed to be created by a ReplicationController in , which itself uses this method from the e2e framework . However, after being in a Pending state (after the "failed to wait for pods responding: timed out waiting for the condition" error is registered), they do seem to reach a Ready state
/assign for triage.
I think agnhost needs to be passed , but did not do so in /assign sig-testing
GitHub didn't allow me to assign the following users: sig-testing. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : I think agnhost needs to be passed , but did not do so in Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Reopen to monitor job /reopen
Reopened this issue.

<detailsIn response to : Reopen to monitor job /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Thank you for working on the PR to solve this. indeed solved the issue reported here. /close
Closing this issue. <detailsIn response to : Thank you for working on the PR to solve this. /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: time.Sleep(10 * time.Millisecond) } }(i) <ins> wg.Add(1) go func(i int) { defer wg.Done() for { select { case <-stopChan: return default: w, err := noxuNamespacedResourceClient.Watch(metav1.ListOptions{}) if err != nil { t.Fatalf("unexpected error establishing watch: %v", err) } for event := range w.ResultChan() { switch event.Type { case watch.Added, watch.Modified, watch.Deleted: // all expected default: t.Errorf("unexpected watch event: %#v", event) } } } time.Sleep(10 * time.Millisecond) } }(i) </ins> } // Let all the established get request loops soak

Reproduced on 1.11.7 as well as master-ish () Deleting a CRD doesn't seem to cause the API server to terminate watches referencing the corresponding CRs, but the watches also don't properly resume if the CRD is created again, meaning that you have to wait for watch timeouts to actually get new watch events. We should probably kill open watches on CRD deletion, so that clients are notified and are forced to re-watch. Either that, or seamlessly resume watches if a CRD is recreated. <details</summary</details<details</summary</detailsthe above CRD: the above CR: (elsewhere) and grab a resource version (elsewhere) the lack of watch events After watches time out, things can re-establish, but that can be a while.
/assign
/cc sttts
/area custom-resources
it's sort a related thread w/ . for now CRD's update will terminate CR's watch connection, the other way round we might fix'em together. one possible fix suggested by months ago is that we make the storage to be a runnable object, e.g. by which we can clearily define the lifecycle of CR's storage
fixed in

What happened: If you establish a WATCH request on a list of CRs, then edit the CRD's object, you will stop getting MODIFIED events on the WATCH request. The WATCH request doesn't fail or report an error, but will not return more MODIFIED events until the request is restarted. This causes problems for custom controllers, which become unresponsive until their requests expire. What you expected to happen: Either the request should end (so it can be restarted safely) or it should continue to return MODIFIED events. How to reproduce it (as minimally and precisely as possible): I've attached . It does the following: Create CRD and an initial CR. Start a proxy, wait for it to come up, then start a watch request. Change the CR. (This change to the CR results in a MODIFIED event) Change the section of the CRD. Make another change to the CR. (This change to the CR doesn't result in an event). I've tested this against GKE masters on 1.9, 1.10 and 1.11, as well as a kubeadm 1.12.2 cluster. The output looks like: Note that you only see MODIFIED once despite the CR being changed twice. Environment: Kubernetes version (use ): 1.11.2-gke.18, 1.12.2 Cloud provider or hardware configuration: GKE, kubeadm on amd64 OS (e.g. from /etc/os-release): Ubuntu 16.04 Kernel (e.g. ): 4.15.0-36-generic ~16.04.1-Ubuntu SMP <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc
/cc
i suppose it's related w/ yes, b/c we switched to another CR storage when you updated CRD spec.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with .

Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
fixed in


Question: } func getNodeProblemDetectorImage() string { <del> const defaultImage string = "k8s.gcr.io/node-problem-detector:v0.6.2" </del> <ins> const defaultImage string = "k8s.gcr.io/node-problem-detector/node-problem-detector:v0.8.7" </ins> image := os.Getenv("NODE_PROBLEM_DETECTOR_IMAGE") if image == "" { image = defaultImage

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI -- Impossible to say, the lane is failing because of the global timeout and does not have any attached artifacts. At least for two weeks
/cc
/sig node
It seems to be the impact after this PR /cc
It has been failing for months, if not years... :/ It did consistently do this before kubernetes/test-infra as well, but we no longer see any OOMs now tho. So it might be possible that the tests run fine, and that we only need to increase the timeout to get it properly working. Would be awesome if someone could take a look at it.
/cc
If selecting , you see that the test run sometimes, even though it looks like it times out every time. This is an example on a test run with artifacts , where it looks like the test kills the system/apiserver/kubelet. There is a strange log message around the time where it start failing:
Hmm, yeah, it looks like the apiserver dies because of too many open files, and the e2e suite doesn't restart it... So all in all, kubernetes/test-infra fixed the OOMs, but there are still other issues On my raspberry pi running normal k8s 1.21 i get the following: Since node e2e tests, and therefore the apiserver, is running under sshd, this might be the problem? Anyone who have more knowledge about this?
/triage accepted /priority important-soon
great catch! :) Now I wonder (and I'll check reviewing the tests) that is intentional for the test to kill the apiserver. I wonder how it is supposed to work in the ad-hoc environment the tests run into.
I know a lot of tests are intentionally killing and restarting kubelet, but I don't think that killing the apiserver is wanted. From the tooling where it is started, it looks like there is no tooling around restarting it;
Yep, this is what I was thinking. Restarting/killing kubelet is fine, and supported, we do it in quite some places (at very least cpu, topology manager e2e tests, also memory manager tests if memory serves).

It's killing apiserver and/or etcd, and in general everything-but-kubelet which looks suspicious in the context of the e2enode tests. Maybe these tests are just misplaced? should they be moved in the regular e2e suite, on which they run against a full cluster and not on a mini-, special purpose environment like e2enode tests are running?
Yeah. I think a simple "restart" apiserver if it dies would work. It would however mask potential performance issues, so I think the best option long term is to look at why the apiserver dies; and try to fix that. I think that isn't possible, since the tests in need to be serially executed, and they require full access over kubelet and its lifecycle, together with modifying node stuff. They are also pretty disruptive, especially when testing eviction related things. If the regular e2e tests can allow for this, then I guess that is ok, but I don't think that is the case.
This surely fixes the current issue, and we could do that. But I wonder if it is the right thing to do in the e2enode flow. I mean: should we handle these restarts in the special purpose reduced environment? are these tests already abusing this environment? I'll start doing some archeology in git history as soon as I have more time to spend on this issue. Good point. I agree.
I'll start doing some archeology in git history as soon as I have more time to spend on this issue. Agree! I guess we can do something like this: and see if it mitigates the issue. In case it is actually running with , that isn't a realistic production environment anyways. What do you think about that?
This looks like a fine change and we should most likely fix the open files issue anyway. So thanks for sending the PR!
I also found an instance of , and that doesn't look that good :sweat_smile: Would be nice if anyone could take a look at increasing the timeout in test-infra as well, and see if that makes any difference.
/cc
Still work in progress though.
After some digging, I found out that the reason for the lack of stdout and artifacts from this test suite is the way it is killed upon the timout.

When the global timeout is exceeded, SIGINT is sent to all the processes. If we ignore the first SIGINT in node-e2e, we can print logs to stdout and fetch artifacts. In case the test is still running 15 minutes later, it will be killed via SIGKILL. See . Would be nice to get that merged as soon as possible!
this is a good idea, but is seems racy because it seems there is a hidden assumption we have enough time to collect the log and the artifacts. Which is a big step forward anyway! but let me ask: do we have a simple way to, once the process receives SIGINT, start explicitely the process of waiting for childs to end, collect logs + artifacts and then terminate (instead of just ignoring the signal and wait for things to converge implicitely)? If so maybe worth making it a bit more robust; if not, let's go ahead with this fix anyway/
Thanks for looking The executor will wait 15min before sending SIGKILL. That should be ok with time. If that doesn't work, we can increase that later. Since the runner will send SIGINT to all children processes, the SSH connections, they will be terminated by SIGINT. Printing of logs and collection of artifacts will therefore start imminently after getting the SIGINT. This worked well when running locally, and hopefully it will work the same way here. Yes, at a later stage that might be smart. The big issue here is that we use the same timeout for ginkgo and for the global test runner; resulting in the fact that the global test runner will always kill the test before ginkgo hits its timeout. Does that make sense? Tried to describe it here as well:
The thing that worries me it seems we only rely on time and to a lot of implicit assumptions in general. I fully ack we are in deep red zone and we need to fix the lane. I'm totally onboard with the urgency (and the sorry state of the lane in general). I think we should not rush fixes in anyway which could make our life harder in the future. OTOH it seems that better and more robust handling of timeouts and artifacts collection need some more investigation and a bit more work; your change clearly signals what's going on and why we do what we do, so I'm inclined to go ahead with it.

Yes, it does make sense to me. Your fix seems to hit the sweet spot between unlocking things without making the overall infra more fragile. Still, this is one more hint the flow as whole need some attention and some improvements. Yep, thanks.
Yeah, completely agree! This definitely needs some more love. There are so many layers of things, making it all fuzzy and hard to reason about, and debug... :/ Maybe someone from sig-testing have more insight into what we should do, and how we should do it.
/cc perhaps?
/assign (since you're actively on it...)
Yes, thanks. I'll report my status on today's sig-node CI group meeting
On receiving the timeout signal the process should begin cleanup / teardown, possibly proceeded by log dumping. It should not ignore it. We set a large grace period on to feasibly allow cleanup. When the kill signal is received that is it. If you're hitting the timeout regularly you need to increase the configured timeout or speed up the tests.
Yes. The current behavior is that the test runner get killed by the it gets from prow (or what you would call it), and the logs will therefore not be printed and artifacts are not fetched. With , the test runner will "ignore" the SIGINT from timeout, while all its children processes will be killed. It will therefore imminently start printing the logs and fetching artifacts. The runner will therefore exit cleanly quite quickly after the initial . In a perfect world we could properly terminate the ginkgo suite, but since we get to all procs, but there is currently no way to do that. Is that an ok solution, or do you have any other ideas. Yeah, that is work in progress. The issue now is that we get no logs or artifacts when tests time out, and that is quite impractical :/
FYI, we now have a job from if we want to test fixes on PRs.
cc this scenario is something to be aware of for kubetest -kubetest2
I'm most likely stating the obvious, but if we exclude the eviction tests, the lane seems to complete. Running locally with initial success, checking if it was a lucky run.
from the PRs above, still timed out in CI after 7 hours, I can try to submit a PR to exclude the eviction tests. Can we test those separately?

:tada:
/reopen we are not done yet
Reopened this issue. <detailsIn response to : /reopen we are not done yet Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
/close This issue was fixed in the 1.23 release. The job was replaced by and
Closing this issue. <detailsIn response to : /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: Items: []corev1.Pod{*testPodWithOneContainers()}, }, actions: []testclient.Action{ <del> getLogsAction("test", nil), </del> <ins> getLogsAction("test", &corev1.PodLogOptions{Container: "c1"}), </ins> }, expectedSources: []corev1.ObjectReference{{ Kind:       testPodWithOneContainers().Kind,

/kind bug The logs of pods with label will be output expected without errors. create the following resources: kubectl logs -l Kubernetes version (use ): latest master.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cli
I am able to pull logs from multiple containers in my test setup, this doesn't seem to be an issue with Could you check if all the containers in the specified pod are healthy?
yes, all of these pods' containers are healthy. I the section.
This is working as expected, was not able to find the default container which in both deployments you explicitly said is but only the first deployment has container, the second deployment has . /close
Closing this issue. <detailsIn response to : This is working as expected, was not able to find the default container which in both deployments you explicitly said is but only the first deployment has container, the second deployment has . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: // Output: // clusters: //   cow-cluster: <del> //     LocationOfOrigin: "" </del> //     certificate-authority-data: DATA+OMITTED //     server: http://cow.org:8080 // contexts: //   federal-context: <del> //     LocationOfOrigin: "" </del> //     cluster: cow-cluster //     user: red-user // current-context: federal-context // preferences: {} // users: //   red-user: <del> //     LocationOfOrigin: "" </del> //     client-certificate-data: DATA+OMITTED //     client-key-data: DATA+OMITTED //     token: REDACTED

yaml.Marshal a kubeconfig, as done in the example tests here LocationOfOrigin is not seen in output, as comment says "never serialized." Click "run example" doesn't suffer from this as it does some conversion <details</details<details</details<details</details<details</details<details</details<details</details
/assign
/sig api-machinery
/assign /triage accepted
is what's known as internal type. It is not intended to be serialized by code. The serialization version is in and does not have this field. I could get behind removing all json tags from the internal version, but I wouldn't refine tags that shouldn't be used.


Question: // policy of the containers in which the Pod is running. Parallelism is the Job's parallelism, and completions is the // Job's required number of completions. func NewTestJob(behavior, name string, rPol v1.RestartPolicy, parallelism, completions int32, activeDeadlineSeconds *int64, backoffLimit int32) *batchv1.Job { <ins> anyNode := "" return NewTestJobOnNode(behavior, name, rPol, parallelism, completions, activeDeadlineSeconds, backoffLimit, anyNode) } // NewTestJobOnNode is similar to NewTestJob but supports specifying a Node on which the Job's Pods will run. // Empty nodeName means no node selection constraints. func NewTestJobOnNode(behavior, name string, rPol v1.RestartPolicy, parallelism, completions int32, activeDeadlineSeconds *int64, backoffLimit int32, nodeName string) *batchv1.Job { </ins> manualSelector := false job := &batchv1.Job{ ObjectMeta: metav1.ObjectMeta{

Which jobs are flaking: pull-kubernetes-e2e-kind Which test(s) are flaking: Testgrid link: Reason for failure: Anything else we need to know: /sig apps
Failure at
Hi I looked into the failures and think it was because the test required the job succeeds 3 times with pseudorandom 50% failure chance within 15 minutes, while there is an exponential back-off delay (10s, 20s, 40s …) capped at 6 minutes before recreating failed pods. As 7 consecutive failures (1/128 chance) could exceed the timeout, the test failed intermittently because of "timed out waiting for the condition". For example, in the failure of , the job pods succeeded and failed in the following orders: After the last failure at , the delay has increased to 640s so no pod was created any more before the test failed at . I think it can be fixed by using a deterministic approach to make the job pods succeed and fail alternately, and its executing time can be reduced to tens of seconds stably. Assigning myself. /assign


Question: } } <del> // For a given node checks its conditions and tries to update it. Returns grace period to which given node // is entitled, state of current and last observed Ready Condition, and an error if it occurred. </del> <ins> // tryUpdateNodeStatus checks a given node's conditions and tries to update it. Returns grace period to // which given node is entitled, state of current and last observed Ready Condition, and an error if it occurred. </ins> func (nc *NodeController) tryUpdateNodeStatus(node *v1.Node) (time.Duration, v1.NodeCondition, *v1.NodeCondition, error) { var err error var gracePeriod time.Duration

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: If node is unreachable, NodeController only set , , to unknown; it's better to also set NetworkUnavailable & InodePressure to unknown.
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig node
Isn't a known condition if node is unreachable?
is the status of container, e.g. CNI is not ready. If node is unreachable, it should be Unknown.
Thanks for explanation!
will not set NetworkUnavailable & InodePressure to unknown, refer to .


Question: } } <ins> func TestNominatedNodeCleanUpUponNodeDeletion(t *testing.T) { // Initialize scheduler. testCtx := initTest(t, "preemption") defer testutils.CleanupTest(t, testCtx) cs := testCtx.ClientSet defer cleanupPodsInNamespace(cs, t, testCtx.NS.Name) // Create a node with some resources and a label. nodeRes := &v1.ResourceList{ v1.ResourcePods:   *resource.NewQuantity(32, resource.DecimalSI), v1.ResourceCPU:    *resource.NewMilliQuantity(100, resource.DecimalSI), v1.ResourceMemory: *resource.NewQuantity(100, resource.DecimalSI), } nodeNames := []string{"node1", "node2"} for _, nodeName := range nodeNames { _, err := createNode(testCtx.ClientSet, nodeName, nodeRes) if err != nil { t.Fatalf("Error creating nodes: %v", err) } } // Fill the cluster with one high-priority and one low-priority Pod. highPriPod, err := createPausePod(cs, mkPriorityPodWithGrace(testCtx, "high-pod", highPriority, 60)) if err != nil { t.Fatalf("Error creating high-priority pod: %v", err) } // Make sure the pod is scheduled. if err := testutils.WaitForPodToSchedule(cs, highPriPod); err != nil { t.Fatalf("Pod %v/%v didn't get scheduled: %v", highPriPod.Namespace, highPriPod.Name, err) } lowPriPod, err := createPausePod(cs, mkPriorityPodWithGrace(testCtx, "low-pod", lowPriority, 60)) if err != nil { t.Fatalf("Error creating low-priority pod: %v", err) } // Make sure the pod is scheduled. if err := testutils.WaitForPodToSchedule(cs, lowPriPod); err != nil { t.Fatalf("Pod %v/%v didn't get scheduled: %v", lowPriPod.Namespace, lowPriPod.Name, err) } // Create a medium-priority Pod. medPriPod, err := createPausePod(cs, mkPriorityPodWithGrace(testCtx, "med-pod", mediumPriority, 60)) if err != nil { t.Fatalf("Error creating med-priority pod: %v", err) } // Check its nominatedNodeName field is set properly. if err := waitForNominatedNodeName(cs, medPriPod); err != nil { t.Fatalf("NominatedNodeName was not set for pod %v/%v: %v", medPriPod.Namespace, medPriPod.Name, err) } // Get the live version of low and med pods. lowPriPod, _ = getPod(cs, lowPriPod.Name, lowPriPod.Namespace) medPriPod, _ = getPod(cs, medPriPod.Name, medPriPod.Namespace) want, got := lowPriPod.Spec.NodeName, medPriPod.Status.NominatedNodeName if want != got { t.Fatalf("Expect med-priority's nominatedNodeName to be %v, but got %v.", want, got) } // Delete the node where med-priority pod is nominated to. cs.CoreV1().Nodes().Delete(context.TODO(), got, metav1.DeleteOptions{}) if err := wait.Poll(200*time.Millisecond, wait.ForeverTestTimeout, func() (bool, error) { _, err := cs.CoreV1().Nodes().Get(context.TODO(), got, metav1.GetOptions{}) if apierrors.IsNotFound(err) { return true, nil } return false, err }); err != nil { t.Fatalf("Node %v cannot be deleted: %v.", got, err) } // Finally verify if med-priority pod's nominatedNodeName gets cleared. if err := wait.Poll(200*time.Millisecond, wait.ForeverTestTimeout, func() (bool, error) { pod, err := cs.CoreV1().Pods(medPriPod.Namespace).Get(context.TODO(), medPriPod.Name, metav1.GetOptions{}) if err != nil { t.Errorf("Error getting the medium priority pod info: %v", err) } if len(pod.Status.NominatedNodeName) == 0 { return true, nil } return false, err }); err != nil { t.Errorf("The nominated node name of the medium priority pod was not cleared: %v", err) } } </ins> func mkMinAvailablePDB(name, namespace string, uid types.UID, minAvailable int, matchLabels map[string]string) *policy.PodDisruptionBudget { intMinAvailable := intstr.FromInt(minAvailable) return &policy.PodDisruptionBudget{

What happened: Scheduler decided that it wants to schedule Pod P on Node N after preempting some pods from N. It set the P.nominatedNodeName to N. After a while (before scheduling actually happened) node N was removed from the cluster. Scheduler did not reset P.nominatedNodeName to empty value. What you expected to happen: I would expect scheduler to unset P.nominatedNodeName when the node is no longer valid. The trivial reason for node not being valid any more is it is removed from the cluster. But there are more cases to cover. Out of the top of my head I may think of: extra Pod was scheduled to node and there is not enough resources any more either scheduled by scheduler or directly as static pod/daemonset node became unschedulable How to reproduce it (as minimally and precisely as possible): We did not try to reproduce issue manually. Yet following setup should work. create cluster with nodes A and B. ensure all system pods are running on A. Fill up A so it does not have spare resources. add a low priority Pod L with long termination, with requests matching allocatable of B. It should be scheduled to B. add a high priority Pod H. It should get B as nominatedNodeName kill node B the H.nominatedNodeName will still point to B which is no longer in the cluster Anything else we need to know?: Current behavior does not play well with Cluster Autoscaler which assumes that Pod with nominatedNodeName will be scheduled in a while. Therefore such pods do not trigger scale-up. If we can have P.nominatedNodeName set incorrectly for unbounded period of time CA will never provision node for P to run. Environment: Kubernetes version (use ): We observed the behavior during presubmit tests which run k8s built from HEAD of master on 27.11.2019.

Versions logged in test log: I1125 18:55:11.691] Client Version: version.Info{Major:"1", Minor:"18+", GitVersion:"v1.18.0-alpha.0.1183+", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-11-25T17:04:18Z", GoVersion:"go1.13.4", Compiler:"gc", Platform:"linux/amd64"} I1125 18:55:11.692] Server Version: version.Info{Major:"1", Minor:"18+", GitVersion:"v1.18.0-alpha.0.1183+", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-11-25T17:04:18Z", GoVersion:"go1.13.4", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: GCE /sig scheduling
/assign
/cc
Another interesting case would be creating a new pod with pod antiaffinity blocking the pod with nominated node name from scheduling: up a cluster with pods, including a long-terminating low-priority pod A. high-priority pod B that has the same requests as A. pod C with negligible resource requests and zonal topology pod antiaffinity matching pod B (and no other pods in the cluster). What I'd expect: Pod C keeps pending, nominatedNodeName is cleared. Pod A is restarted on the same node it was running before. I haven't actually tested this scenario, so I'm not sure what will actually happen. If report is correct, this may be a more convenient way of reproducing the issue (no need to add / remove nodes, it's not important which pod will go to which node).
I'm not sure if we need to clear nominatedNodeName on Pod when node is no longer valid immediately. And we have other logic to clear nominatedNodeName. Reuse your example. create cluster with nodes A and B. ensure all system pods are running on A. Fill up A so it does not have spare resources.

add a low priority Pod L with long termination, with requests matching allocatable of B. It should be scheduled to B. add a high priority Pod H. It should get B as nominatedNodeName kill node B the H.nominatedNodeName will still point to B which is no longer in the cluster After this, The H will be scheduled again. Obviously it's going to fail. H will enter the preemption process and the is empty. It will return itself as preemption process will clean nominatedNodeName on the Pod H.
when I have some time I will check if reproduction steps I describe actually work. But we surely observed pods with lingering nominatedNodeName which pointed to inexistent node, during our E2E tests. Quick thought. The method has many return points. If it returned at the place you linked it seems should be cleared. But there are many return points below, which set the list of pods to be cleared to nil. Is it possible that for a pod to consistently exit from with one of those return points?
I think you are right. Actually there are many return points to return nil. I think maybe we need to clear the of the pod after the filter failed again and before the new preemption process to avoid potential risks. WDYT?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or .

/lifecycle stale
/remove-lifecycle stale
I believe it's a bug as in old logic, e.g., v1.18.3, if the Pod (with nominatedNodeName set) cannot be scheduled and also preemption doesn't help ( nil potentialNodes is a corner case, it's more common the pod can't fit on all potentialNodes), it won't hit L413: then in old recordingFailure logic, nominatedNodeName remains as is. In the latest codebase, introduced a fix to combine setting nominatedNodeName as well update the pod's unschedulable condition. However, this doesn't help in this issue either as it ignores the case when the pod's nominatedNodeName is nil:
/reopen as regresses - we need to come up with a complete fix.
Reopened this issue. <detailsIn response to : /reopen as regresses - we need to come up with a complete fix. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle rotten
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle rotten
what are the implications of not clearing the nominated node name?
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: "k8s.io/kubernetes/pkg/kubelet/cri/streaming/remotecommand" "k8s.io/kubernetes/pkg/kubelet/metrics" "k8s.io/kubernetes/pkg/kubelet/prober/results" <ins> "k8s.io/kubernetes/pkg/kubelet/secret" </ins> "k8s.io/kubernetes/pkg/kubelet/status" kubetypes "k8s.io/kubernetes/pkg/kubelet/types" netutils "k8s.io/utils/net"

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --imagePullSecrets can to a pod manifest to reference a secret containing registry credentials. Currently, if you add the name of a secret that does not exist, there is no warning or error. I would expect there to be an error and a failed deployment if the secret does not exist/cannot be found in the namespace. If people are pulling public images, they may still wish to authenticate to a registry, for example dockerhub, before pulling the image, in order to utilise their subscription account rate limits. Currently if they use imagePullSecrets and the referenced secret does not exist, they may assume they are authenticating to their account when they are not, and will still be able to pull public images without any warning that authentication is not taking place. Add to a pod spec, and pull images from
/sig node (based on the field location)
(and/or) /sig auth
correct me if not. Pulling private images with nonexists secret, pod will turn to . However, pulling public image with nonexists secret, no error will display yet.
Similar issue with .
Yes that's correct.
I think so. Working on this later. Will also live up to your expectations, right?
But I check it already have a warnging events if secret not exist and pullImageFail: line:147
Now, only info message when pullsecret not exist
maybe direct return and not enter is better. if pullSecret is not existed
if you pull image from a public repo, the process should not be impacted even no secrets exists. And it's what it is right now.
I think a warning event like could be here /good-first-issue /triage accepted
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /good-first-issue /triage accepted Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.

</details
/assign
GitHub didn't allow me to assign the following users: EricWvi. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
GitHub didn't allow me to assign the following users: EricWvi. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : I would like work on this. /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I would like to work in this. /assign
Thanks for wanting to jump in on this, but it's not clear this is a good first issue… making the proposed change would break deployments/pods that are currently working/running, so it's unlikely this would be acceptable from a compatibility perspective.
Requiring image pull secrets to exist would have to be opt-in, which means an API change. I don't think this is worth the addition, but would like to hear sig-node's perspective. An alternative could be to explore ways to make it more obvious to the user when this situation was occurring without automatically failing the pod
hi , we will not fail the pod, we just want to add some warning to make it obviously, maybe it's the same as you advised. correct me if not.
To be clear here, only adding a warning is marked as good first issue here, not changing any pod behaviour.
gotcha, that seems more reasonable, thanks for clarifying as a note, make sure the warning events are generated at most once in that function, rather than once per missing secret object (if there are multiple missing secrets)
/good-first-issue
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed .

If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Yes, a warning would be good. I feel an error would have be better, but as has already been mentioned above we are too far along now for this change as it could break some users existing deployments, so a warning makes the most sense now.
/assign
based on what you have stated above, I see an INFO log already exists! If I'm understanding this correctly, now we need to raise a warning event in the same code block to fulfill this request. Is that right?
I think has created a above and labeled /lgtm already.
Hey I don't know anything about how to code but i am interested to contribute on kubernetes So help me to contribute
/cc
there is PR already here , just needs approval
it was just FYI =) Reviewed the PR, thank you
Thanks for the review :)
/assign
will like to work on it. /assign
Hey , would love to work on this issue
there is already a PR: do you plan to push the PR forward?
I have been busy with other things recently, if anyone is willing to continue this problem. I will close the PR.
if noone else is working on this task, I would like to take this up.
Feel free to work on this , you can refer to for some advices.
Thanks! Will refer it
/assign
/assign
/assign
Is this issue still open/active?
Hello I was working on it but not able to continue it due to some personal reasons. You can take over if you would like to. You can refer this PR if you want to.
Yes sure!!!
Can I try this if no one is available to work on it ?
Yes go on!!!
/assign
Sorry for keep opening the PR. something went wrong on my fork branch. I eventually needed to re-fork the repo so it caused the PR closed automatically
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
Can I give it a go if no one is working on it?
Thanks for the offer, but it looks like was open and had an update that was waiting for review (which I didn't realize)
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
is this issue still available to pick up?
/remove-lifecycle stale
Hi, is this issue still available?
Hi! It feels that this issue is stalled (although it lacks the label) and the related PRs have been abandoned. I'll work on it if that's ok /assign
As well as an event, we might like to directly report a warning to end users (suggested in ). More complexity though.
should it be part of this issue? it only mentions but as you guys (the contributors) decide. is based on which was labelled as and suggested as a starting point before being closed.


Question: } images := framework.ImageWhiteList.List() klog.V(4).Infof("Pre-pulling images with %s %+v", puller.Name(), images) <del> for _, image := range images { var ( err    error output []byte ) for i := 0; i < maxImagePullRetries; i++ { if i > 0 { time.Sleep(imagePullRetryDelay) } if output, err = puller.Pull(image); err == nil { break </del> <ins> imageCh := make(chan int, len(images)) for i := range images { imageCh <- i } close(imageCh) pullErrs := make([]error, len(images)) ctx, cancel := context.WithCancel(context.Background()) defer cancel() parallelImagePullCount := maxParallelImagePullCount if len(images) < parallelImagePullCount { parallelImagePullCount = len(images) } var wg sync.WaitGroup wg.Add(parallelImagePullCount) for i := 0; i < parallelImagePullCount; i++ { go func() { defer wg.Done() for i := range imageCh { var ( pullErr error output  []byte ) for retryCount := 0; retryCount < maxImagePullRetries; retryCount++ { select { case <-ctx.Done(): return default: } if retryCount > 0 { time.Sleep(imagePullRetryDelay) } if output, pullErr = puller.Pull(images[i]); pullErr == nil { break } klog.Warningf("Failed to pull %s as user %q, retrying in %s (%d of %d): %v", images[i], usr.Username, imagePullRetryDelay.String(), retryCount+1, maxImagePullRetries, pullErr) } if pullErr != nil { klog.Warningf("Could not pre-pull image %s %v output: %s", images[i], pullErr, output) pullErrs[i] = pullErr cancel() return } </ins> } <del> klog.Warningf("Failed to pull %s as user %q, retrying in %s (%d of %d): %v", image, usr.Username, imagePullRetryDelay.String(), i+1, maxImagePullRetries, err) } if err != nil { klog.Warningf("Could not pre-pull image %s %v output: %s", image, err, output) return err } </del> <ins> }() </ins> } <del> return nil </del> <ins> wg.Wait() return utilerrors.NewAggregate(pullErrs) </ins> } // getGPUDevicePluginImage returns the image of GPU device plugin.

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Unless disabled, the tests will prepull all images when running. They do so using the . I think there's the opportunity to improve the runtime of the test suite by prepulling the images in parallel. Why is this needed: Its always nicer to have faster tests :) Specifically for users running locally, it's nice to have the image prepull occur as quick as possible. That being said, we should also be cognizant of any tradeoffs associated with prepulling the images in parallel. If the code is substantially more complex, we would need a large improvement in test speed to justify it. I'm hopeful this discussion could be had once we have a proof of concept diff (which I think should be fairly minimal effort to create).
/sig node
I think this could be a good first issue for someone looking to get involved with contributing to sig-node :) Please post if you plan on working on it, so that there aren't multiple people tackling it.
I'd be interested! I'm looking for a first issue and think this would be a good one
Wonderful! Want to reach out to me on Kubernetes slack? I'm on there too :)
:wave: hiya ! Are you still thinking you could be interested in working on this? Lmk if so, and we can chat about next steps? If not, perhaps I'll open it back up to a newcomer looking for a first commit?
I'm going to open this back up for someone looking for a first commit. Lmk if you're still interest in getting sig-node, and happy to find an issue for you :)
Opening this back up for folks :) I'm happy to mentor whoever picks this up if that'd be helpful :) Ping me on k8s-slack, where I'm also
Can you assign this to meI'd be interested!
/assign
Awesome - feel free to reach out on k8s-slack if you have questions, or if you already have a plan, feel free to tag me in the pull request. Thanks :)
/lifecycle active /priority important-longterm


Question: } func TestEnsureLoadBalancerDeletedDeletesExternalLb(t *testing.T) { <del> t.Parallel() </del> vals := DefaultTestClusterValues() gce, err := fakeGCECloud(vals) require.NoError(t, err)

Since merged on Friday night, my PR () has failed several retests of pull-kubernetes-bazel-test. My PR is unrelated to the load balancer so I suspect that introduced a flake. I see several runs of that test against master that aren't failing, so clearly the test works sometimes (and the original PR passed it several times to before it merged). Perhaps the test harness sometimes runs problem tests in parallel and other times it doesn't. Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: My PR had one good run of pull-kubernetes-bazel-test before merged, but has failed several times since with no code changes to my PR. What you expected to happen: The tests should consistently pass unless related problem code is introduced.
/cc /cc /sig gcp /kind flake
My PR started failing recently too. See https://k8s-
Hey, thanks for identifying the issue. contains tests for - but the race condition was caused when tests with modified a singleton variable in a mock. The short-term fix is removing the calls, but the long-term fix is to not use a singleton. Fix is .


Question: } } <ins> func TestPreconditionalUpdateWithCachedObject(t *testing.T) { podName := "foo" pod := &example.Pod{ ObjectMeta: metav1.ObjectMeta{Name: podName}, Spec:       example.PodSpec{NodeName: "machine"}, } ctx := genericapirequest.WithNamespace(genericapirequest.NewContext(), "test") destroyFunc, registry := newTestGenericStoreRegistry(t, scheme, false) defer destroyFunc() // cached object has old UID oldPod, err := registry.Create(ctx, pod, rest.ValidateAllObjectFunc, &metav1.CreateOptions{}) if err != nil { t.Fatal(err) } registry.Storage.Storage = &staleGuaranteedUpdateStorage{Interface: registry.Storage.Storage, cachedObj: oldPod} // delete and re-create the same object with new UID _, _, err = registry.Delete(ctx, podName, rest.ValidateAllObjectFunc, nil) if err != nil { t.Fatal(err) } obj, err := registry.Create(ctx, pod, rest.ValidateAllObjectFunc, &metav1.CreateOptions{}) if err != nil { t.Fatal(err) } newPod, ok := obj.(*example.Pod) if !ok { t.Fatalf("unexpected object: %#v", obj) } // update the object should not fail precondition newPod.Spec.NodeName = "machine2" res, _, err := registry.Update(ctx, podName, rest.DefaultUpdatedObjectInfo(newPod), rest.ValidateAllObjectFunc, rest.ValidateAllObjectUpdateFunc, false, &metav1.UpdateOptions{}) if err != nil { t.Fatal(err) } // the update should have succeeded r, ok := res.(*example.Pod) if !ok { t.Fatalf("unexpected update result: %#v", res) } if r.Spec.NodeName != "machine2" { t.Fatalf("unexpected, update didn't take effect: %#v", r) } } </ins> // TestRetryDeleteValidation checks if the deleteValidation is called again if // the GuaranteedUpdate in the Delete handler conflicts with a simultaneous // Update.

Which jobs are failing: pull-kubernetes-integration Which test(s) are failing: https://k8s- Not showing up in triage, oddly, but seen in individual failures: not sure if this specific to custom resources, but that's where I'm seeing the flakes /sig api-machinery /priority important-soon /area custom-resources
adding to the milestone until root-caused
/cc
I suspect there is some cache/race that keeps the old CR information around during the update from the CR that fails UID check during an update etcd log: few read (other other version defined by CRD. the port difference indicates a different client) the timestamp, the failure happened after the write a successful CR from another test in that file (TestValidateOnlyStatus, TestGeneration, etc) (different etcd key prefix from the same etcd log) few read (update) few read (the second version. I'm surprised that the port didn't change) few read
leaving in the milestone for visibility as we root cause whether this is a test-only issue or not
I spent a while trying to recreate this and could not (again). I made one of the tests hotloop with QPS 100 doing get/update of the status subresource, then ran that in parallel with the other integration tests in the package. No failures found after running for ~10 minutes. Updated description with testgrid link to locate recent flakes (https://k8s-), found a couple in the last week.
iiuc, the error comes from apiserver the UID against the . The requested obj is up-to-date since it was from etcd, but I didn't see mechanism that guarantees the watch cache to be up-to-date after a "delete-create" or "delete-create-get" sequence (the cacher implementation for Delete, Create and Get(rv="") go through etcd directly). Is this failure an expected behavior, and should we just retry on precondition failure?
the triage page seems to be stale, filed
talked with offline, we get it reproduced by adding some delay to the watch cache: as suggested preconditions should retry on stale watch data instead of surfacing such error, this is not a test issue. I will send a fix


Question: record_command run_kubectl_apply_deployments_tests fi <ins> ################ # Kubectl diff # ################ record_command run_kubectl_diff_tests </ins> ############### # Kubectl get # ###############

/kind bug /sig cli /area kubectl Hello. This is my first interaction with the Kubernetes community, so please forgive me if i have left out any crucial details, or if, perhaps, this is PEBKAC after all! I am keen to have a working feature, as described in and I am running K8s v1.9.3 and realise that this feature is in alpha, but i thought i would ask anyway. We're not on v1.9.4 yet, but the changelog doesn't mention the feature. What you expected to happen: I have some services deployed to Kubernetes with YAML files, that has been working reliably. We always use to deploy things. When a service is already deployed, and i make a minor change in the config file, for example the service port, i expect to see that in the diff. Instead, gives no output and exits with a . How to reproduce it (as minimally and precisely as possible): Consider the file : I make sure that it is correctly running on the cluster: Great, looking good. Now, i change the value to use Docker image tag and save. Now, i run: The stderr/stdout output is empty, and the exit code is . Anything else we need to know?: Environment: Kubernetes version: Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-02-09T21:50:44Z", GoVersion:"go1.9.4", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.3+coreos.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-02-10T01:42:55Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}
I have this same issue - if I change, say, to in my , running returns empty. If I try tweaking the version I'm diffing (which I don't understand) - e.g. - then I see a bunch of differences, but not actually the change I was expecting in the number of replicas.
I'll take this. Sorry for the inconvenience.


Question: }}) } <ins> // MaintainObservations keeps the observers from // metrics.PriorityLevelConcurrencyObserverPairGenerator from falling // too far behind func (cfgCtlr *configController) MaintainObservations(stopCh <-chan struct{}) { wait.Until(cfgCtlr.updateObservations, 10*time.Second, stopCh) } func (cfgCtlr *configController) updateObservations() { cfgCtlr.lock.Lock() defer cfgCtlr.lock.Unlock() for _, plc := range cfgCtlr.priorityLevelStates { if plc.queues != nil { plc.queues.UpdateObservations() } } } </ins> func (cfgCtlr *configController) Run(stopCh <-chan struct{}) error { defer cfgCtlr.configQueue.ShutDown() klog.Info("Starting API Priority and Fairness config controller")

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Not observed yet, but deduced from analyzing the code. If a k/apiserver/pkg/util/flowcontrol/fairqueuing.QueueSet, whose metrics.TimedObserverPair is a pair of k/apiserver/pkg/util/flowcontrol/metrics.sampleAndWaterMarkHistograms, gets zero activity for a long while and then gets some activity, then the loop at will go through a lot of iterations, and do so with a lock held. What you expected to happen: No big hiccups in any scenario. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): branch Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/assign

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: In reviewing I noticed an incorrect fake implementation of Interface. I looked at the comment on the Handle method, and noticed that it does not say when/how the is called. What you expected to happen: I expected the parameter to be as well documented as the others. How to reproduce it (as minimally and precisely as possible): Look at and see how it does not say when/whether/with-what-parameter-values is called. Anything else we need to know?: Environment: Kubernetes version (use ): current Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
Any chance we could write a test for the interface which verifies correct plumbing?
/cc
: that should be do-able


Question: predicate := func(actual []string) bool { return len(actual) == 0 } <del> t.checkDNSRecord(federationDNS, predicate, wait.ForeverTestTimeout) </del> <ins> t.checkDNSRecordFrom(federationDNS, predicate, "cluster-dns", wait.ForeverTestTimeout) </ins> } } else { for label := range federations {

/kind failing-test /priority critical-urgent /sig network /milestone v1.12 /cc
ping
Also failures on
TLS errors connection refused 503 service unavailable especially around metrics and healthz in ways that make me suspect things didn't start correctly. Or perhaps API mismatch? /cc 've looked at some similar failures in the past. Any chance you might give a look over this and ?
in both and here looks like api mismatch?
Test output shows no DNS server can be reached, but logs say coredns pods are running (https://k8s-): cc
The CoreDNS configmap is incorrectly configured and hence the test fails. I'm working on the fix.
/assign
, , Does it make sense to keep that e2e test ?, knowing that this test is specific for "Federation" and that federation has now merged into "multicluster" and removed all e2e tests about "Federation + Serial" (formerly "ci-federation-e2e-gce-serial") : can you say if that use case is still supported in kubernetes v1.12 ?
I can say that Federation v1 has not been exercised since v1.10. I think a good question for the Federation v2 folks is not only whether they still need the tests, but also whether they need the configmap support specific to what Federation v1 used -- I'll defer to
If ya'll decide the use case is no longer valid, please open an issue / remove the test once 1.13 release opens.
tests are failing
/reopen /cc
Reopening this issue. <detailsIn response to : /reopen /cc Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
More failures:
Looks like this is failing since Kube-dns is now enabled.
Since we'd previously discussed this not being a relevant test case, and it seems less appropriate even if it's this coupled to the dns implementation and we've got the kube-dns to coredns transition happening in the next release most likely....Can we just disable/remove the test?
Opened to disable this test.
Thank you!


Question: const ( // SampleDevicePluginDSYAML is the path of the daemonset template of the sample device plugin. // TODO: Parametrize it by making it a feature in TestFramework. <del> SampleDevicePluginDSYAML                    = "test/e2e/testing-manifests/sample-device-plugin.yaml" </del> <ins> SampleDevicePluginDSYAML                    = "test/e2e/testing-manifests/sample-device-plugin/sample-device-plugin.yaml" </ins> SampleDevicePluginControlRegistrationDSYAML = "test/e2e/testing-manifests/sample-device-plugin/sample-device-plugin-control-registration.yaml" // SampleDevicePluginName is the name of the device plugin pod

Serial e2e node jobs, e.g. Since 03-01 No response Seems related to No response /sig node
Failing run -
I think there is a typo there - should fix it
cc
/triage accepted
/cc


Question: Args: []string{ "--logtostderr", fmt.Sprintf("--system-log-monitors=%s", configFile), <del> fmt.Sprintf(" --apiserver-override=%s?inClusterConfig=true", framework.TestContext.Host), </del> <ins> // `ServiceAccount` admission controller is disabled in node e2e tests, so we could not use // inClusterConfig here. fmt.Sprintf("--apiserver-override=%s?inClusterConfig=false&auth=%s", framework.TestContext.Host, kubeConfigFile), </ins> }, Env: []v1.EnvVar{ {

Which jobs are failing: Which test(s) are failing: Since when has it been failing: Last successful run (23 Oct 2020, 10:07 PDT): First unsuccessful run (23 Oct 2020, 12:08 PDT): Testgrid link: Reason for failure: Anything else we need to know: presubmit job in repository also started to fail in the same fashion. Attempted to debug it myself, found out the following stuff: last successful run was accessing the apiserver via HTTP: all unsuccessful runs attempted to access the apiserver via HTTPS but to no avail: perused the history of to find out that the PR is almost certainly the reason of the failures (or rather uncovered some problem with the NPD or its tests): first, it coincides with the start of the failures of , second, we changed the default hostname of the apiserver in node tests in that PR. Judging by the code and the changes in the aforementioned PR I think that a quick mitigation would be to add a flag to the of these jobs. /assign /sig node /cc
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Opened for a mitigation that would make those jobs stop failing.
This won't work, unfortunately. Copying response:
I have filed to address this issue, but it seems I could not run the test in advance to verify my patch, did you know how to manually trigger the presubmit job?
Unfortunately I don't :/. My PR was also a wild guess. could you take a look at the proposed fix? Thanks once again for your help - really appreciated :).
the submission of hasn't fixed the problems, unfortunately :/. Sample failing run: We still have some peculiar problem related to health checking: and now NPD panics, too:
Thanks for your report, I believe the reason is admission controller has been disabled in node e2e tests: We need to figure out a new approach now. /reopen
Reopened this issue.

<detailsIn response to : Thanks for your report, I believe the reason is admission controller has been disabled in node e2e tests: We need to figure out a new approach now. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
filed . I hope we could completely address this issue this time.
/milestone v1.20


Question: ADD https://github.com/coredns/coredns/releases/download/v1.5.0/coredns_1.5.0_windows_amd64.tgz /coredns.tgz ADD https://iperf.fr/download/windows/iperf-2.0.9-win64.zip /iperf.zip <ins> ADD http://download.savannah.nongnu.org/releases/coreutils/windows-64bit-unsupported/coreutils-8.31-28-windows-64bit.zip /coreutils.zip </ins> # we're also creating an empty /uploads folder, which we're copying over to Windows  because # we can't RUN commands on a Windows image. RUN tar -xzvf /coredns.tgz && unzip iperf.zip && mv iperf-2.0.9-win64 iperf && <ins> unzip coreutils.zip && mv coreutils-8.31-28-windows-64bit wincoreutils && </ins> mkdir /uploads FROM --platform=linux/amd64 $REGISTRY/windows-servercore-cache:1.0-linux-amd64-$OS_VERSION as servercore-helper

We'd like to add to the windows nodes Discussed in , some storage e2e tests need the call to flush data to disk, this utility is not available in windows nodes or agnhost /sig storage /sig testing
/sig testing
/sig storage /kind flake
/sig windows
Does sync only need to be in Agnhost or also on the Windows node? I don't think there is any issue with adding to the agnhost for this tests. Has it been tested that it works? /cc in case you have thoughts
In the README of sync it says: Initially I thought it could be to agnhost however told me that Windows containers cannot run in privileged mode as of today. cc
Since we run the Windows Test containers as they should have full access to modify the host files if the host path is mapped in.
I downloaded the zip file and I checked that it has an EULA, do you know which sig would help with the license?
Hm, if the sysinternal is problematic, what about the GNU one? It seems to come from coreutils, which can easily be downloaded and extracted from. Hopefully, they are 64-bit binaries. Anyways, I think adding is a good idea, there are a few other non-conformance tests that are calling sync.
I believe that would be sig-contributor-experience. I thought we were using other sys-internal tools as well but it is a good idea to double check.
/triage accepted /assign
/remove-sig testing we don't own the nodes (?)
I've sent , which would add using that PR. Let us know if that satisfies your requirements, we could merge afterwards.
Thanks I guess we can merge the PR and I can run the storage e2e tests that need .


Question: for portname := range oldPortsToEndpoints { svcPort := proxy.ServicePortName{NamespacedName: types.NamespacedName{Namespace: oldEndpoints.Namespace, Name: oldEndpoints.Name}, Port: portname} if _, exists := registeredEndpoints[svcPort]; !exists { <ins> lb.resetService(svcPort) } } } func (lb *LoadBalancerRR) resetService(svcPort proxy.ServicePortName) { // If the service is still around, reset but don't delete. if state, ok := lb.services[svcPort]; ok { if len(state.endpoints) > 0 { </ins> glog.V(2).Infof("LoadBalancerRR: Removing endpoints for %s", svcPort) <del> // Reset but don't delete. state := lb.services[svcPort] </del> state.endpoints = []string{} <del> state.index = 0 state.affinity.affinityMap = map[string]*affinityState{} </del> } <ins> state.index = 0 state.affinity.affinityMap = map[string]*affinityState{} </ins> } }

/kind bug What happened: We constantly log message like at V(2) that don't tell any useful information.


Question: // ones named //     autoConvert_<pkg1>_<type>_To_<pkg2>_<type> // for each such pair of types --- both with (pkg1,pkg2) = <del> // (internal,external) and (pkg1,pkg2) = (external,internal). // Additionally: if the destination package does not contain one in a // non-generated file then a function named </del> <ins> // (internal,external) and (pkg1,pkg2) = (external,internal).  The // generated conversion functions recurse on the structure of the data // types.  For structs, source and destination fields are matched up // according to name; if a source field has no corresponding // destination or there is a fundamental mismatch in the type of the // field then the generated autoConvert_... function has just a // warning comment about that field.  The generated conversion // functions use standard value assignment wherever possible.  For // compound types, the generated conversion functions call the // `Convert...` functions for the subsidiary types. // // For each pair of types `conversion-gen` will also generate a // function named </ins> //     Convert_<pkg1>_<type>_To_<pkg2>_<type> <del> // is also generated and it simply calls the `autoConvert...` // function.  The generated conversion functions use standard value // assignment wherever possible.  For compound types, the generated // conversion functions call the `Convert...` functions for the // subsidiary types.  Thus developers can override the behavior for // selected types.  For a top-level object type (i.e., the type of an // object that will be input to an apiserver), for such an override to // be used by the apiserver the developer-maintained conversion // functions must also be registered by invoking the // `AddConversionFunc`/`AddGeneratedConversionFunc` method of the // relevant `Scheme` object from k8s.io/apimachinery/pkg/runtime. </del> <ins> // if both of two conditions are met: (1) the destination package does // not contain a function of that name in a non-generated file and (2) // the generation of the corresponding autoConvert_...  function did // not run into trouble with a missing or fundamentally differently // typed field.  A generated Convert_... function simply calls the // corresponding `autoConvert...` function.  `conversion_gen` also // generates a function that updates a given `runtime.Scheme` by // registering all the Convert_... functions found and generated. // Thus developers can override the generated behavior for selected // type pairs by putting the desired Convert_... functions in // non-generated files.  Further, developers are practically required // to override the generated behavior when there are missing or // fundamentally differently typed fields. </ins> // // `conversion-gen` will scan its `--input-dirs`, looking at the // package defined in each of those directories for comment tags that

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: While reviewing work that adds a field to a custom resource defined by code rather than a CRD, I was alerted to a bug in the comment describing what the conversion generator does. states that the Convert... function is generated if it is not present in a non-generated file --- without regard to whether there are any problems in the autoConvert... function. But we see at that there is an additional condition that is relevant. What you expected to happen: I expected the conversion generator to do what it is documented to do. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): This buggy doc was introduced in Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/cc (we took the freedom of assigning the PR to sttts for revision, thanks!)


Question: var _ error = &ValidationError{} func (v *ValidationError) Error() string { <del> s := fmt.Sprintf("%s: %s '%v'", v.Field, ValueOf(v.Type), v.BadValue) </del> <ins> s := spew.Sprintf("%s: %s '%+v'", v.Field, v.Type, v.BadValue) </ins> if v.Detail != "" { s += fmt.Sprintf(": %s", v.Detail) }

We now have pretty structured way of validating api object, but the error message to end user is not descriptive, e.g. when creating a replication controller with restart policy 'never', we get: even as simple as label name, we shouldn't assume end-user is familiar with our requirement. Apart from what is wrong, we'd better also give why it is wrong.
The main thing wrong here is not having a useful String() method. and have both been pretty adamant about not attaching code to API objects, but here's a clear UX justification. What if we only attached methods to internal API objects, not versioned ones? Also: this should print %#v, rather than %+v, which would be a BIT more readable.
Orrrr we could switch to spew, which is already a dep.
Actually we're printing a reflect.Value there, which makes no sense at all. I will fix. EDIT: no we aren't, I can't read-- will still fix though


Question: {% set storage_base='https://storage.googleapis.com/kubernetes-release/docker/' %} <ins> {% set override_deb_url='' %} </ins> {% if grains.get('cloud', '') == 'gce' and grains.get('os_family', '') == 'Debian' and grains.get('oscodename', '') == 'wheezy' -%}

GCE updated docker, but only via a base-image update. AWS should match versions of docker, to slipstream off all the testing that GCE gets. This will be slightly complicated because docker 1.8.3 isn't in the bucket AFAICT. Thanks to for the tip.
This is done by


Question: }) //Now we can proceed with the test. <del> It("should function for intra-pod communication", func(done Done) { defer close(done) </del> <ins> It("should function for intra-pod communication", func() { </ins> if testContext.Provider == "vagrant" { By("Skipping test which is broken for vagrant (See https://github.com/GoogleCloudPlatform/kubernetes/issues/3580)") return

test panicked: (root)/Kubernetes%%20suite/Monitoringverifymonitoringpodsandallclusternodesareavailableoninfluxdbusingheapster/ cc
Vish please take a look if you have cycles or reassign if there's someone with better context.
I looked at the panic and it was due to the last test (networking) timing out but still running. I think the fix is more in better cleanup between tests.
I just saw this... it's totally due to ginkgo's cleanup when a test times out. I can fix.
I will get this one.
Ugh, the thing ginkgo does when a test times out is totally lame...
Instead of working around this, I've filed I'm going to go back to a manual timeout on this test.


Question: klet.containerRuntime = runtime klet.runner = runtime <ins> runtimeCache, err := kubecontainer.NewRuntimeCache(klet.containerRuntime) if err != nil { return nil, err } klet.runtimeCache = runtimeCache </ins> if cadvisor.UsingLegacyCadvisorStats(containerRuntime, remoteRuntimeEndpoint) { klet.StatsProvider = stats.NewCadvisorStatsProvider( klet.cadvisor,

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: In version 1.8.0, accessing the container stats via Kubelet API proxy using the path returns 503. What you expected to happen: In versions prior to 1.8.0, the container stats were returned. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): 1.8.0 OS (e.g. from /etc/os-release): Mac OS X Install tools: Minikube
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
does the summary API also have issues? ?
The summary API works as expected. The affected endpoints are:
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
/remove-lifecycle stale
/remove-lifecycle rotten
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Hmm... this fell of my radar a bit. If you go directly to the node, do you get a response?
no response either. I've created that fixes the issue. I think there has been a refactoring introduced in version 1.8 that broke the sequence of objects initialisation for that endpoint.