Question: eventBroadcaster := record.NewBroadcaster() eventBroadcaster.StartLogging(glog.Infof) eventBroadcaster.StartRecordingToSink(&v1core.EventSinkImpl{Interface: v1core.New(kubeClient.Core().RESTClient()).Events("")}) <del> recorder := eventBroadcaster.NewRecorder(scheme.Scheme, v1.EventSource{Component: "attachdetach"}) </del> <ins> recorder := eventBroadcaster.NewRecorder(scheme.Scheme, v1.EventSource{Component: "attachdetach-controller"}) </ins> adc.desiredStateOfWorld = cache.NewDesiredStateOfWorld(&adc.volumePluginMgr) adc.actualStateOfWorld = cache.NewActualStateOfWorld(&adc.volumePluginMgr)

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: kubect create -f to create statefulset, and describe the statefulset, output the event likes <preEvents: FirstSeen LastSeen Count From SubObjectPath Type Reason Message --------- -------- ----- ---- ------------- -------- ------ ------- 1m 1m 1 statefulset Normal SuccessfulCreate create Claim www-web-0 Pod web-0 in StatefulSet web success 1m 1m 1 statefulset Normal SuccessfulCreate create Pod web-0 in StatefulSet web successful </preWhat you expected to happen: the From of event change to 'statefulset-controller' in output, let know the source of event more obvious, likes the event when create job and other resource(replicaset,replication). <preEvents: FirstSeen LastSeen Count From SubObjectPath Type Reason Message --------- -------- ----- ---- ------------- -------- ------ ------- 3m 3m 1 job-controller Normal SuccessfulCreate Created pod: pi-7gd2m </preHow to reproduce it (as minimally and precisely as possible): create a statefulset and describe it to see the event Anything else we need to know?: Environment: Kubernetes version (use ):kubernetes 1.7 Cloud provider or hardware configuration**: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/assign
/area controller-manager
/sig node


Question: RUN mkdir -p ${K8S_PATCHED_GOROOT}  && curl -sSL https://github.com/golang/go/archive/go${K8S_PATCHED_GOLANG_VERSION}.tar.gz | tar -xz -C ${K8S_PATCHED_GOROOT} --strip-components=1 <ins> # We need a patched go1.7.1 for linux/arm (https://github.com/kubernetes/kubernetes/issues/29904) # We need go1.7.1 for all darwin builds (https://github.com/kubernetes/kubernetes/issues/32999) </ins> COPY golang-patches/CL28857-go1.7.1-luxas.patch ${K8S_PATCHED_GOROOT}/ RUN cd ${K8S_PATCHED_GOROOT}  && patch -p1 < CL28857-go1.7.1-luxas.patch  && cd src  && GOROOT_FINAL=${K8S_PATCHED_GOROOT} GOROOT_BOOTSTRAP=/usr/local/go ./make.bash  <del> && for platform in linux/arm; do GOOS=${platform%/*} GOARCH=${platform##*/} GOROOT=${K8S_PATCHED_GOROOT} go install std; done </del> <ins> && for platform in linux/arm darwin/386 darwin/amd64; do GOOS=${platform%/*} GOARCH=${platform##*/} GOROOT=${K8S_PATCHED_GOROOT} go install std; done </ins>

See
CI has built and uploaded unofficial prerelease darwin binaries for 1.4 using go1.7.1: cc Official binaries will ship as soon as we cut 1.4.1.
We have official v1.4.1 binaries available now, too.
Hooray


Question: RUN mkdir -p ${K8S_PATCHED_GOROOT}  && curl -sSL https://github.com/golang/go/archive/go${K8S_PATCHED_GOLANG_VERSION}.tar.gz | tar -xz -C ${K8S_PATCHED_GOROOT} --strip-components=1 <ins> # We need a patched go1.7.1 for linux/arm (https://github.com/kubernetes/kubernetes/issues/29904) # We need go1.7.1 for all darwin builds (https://github.com/kubernetes/kubernetes/issues/32999) </ins> COPY golang-patches/CL28857-go1.7.1-luxas.patch ${K8S_PATCHED_GOROOT}/ RUN cd ${K8S_PATCHED_GOROOT}  && patch -p1 < CL28857-go1.7.1-luxas.patch  && cd src  && GOROOT_FINAL=${K8S_PATCHED_GOROOT} GOROOT_BOOTSTRAP=/usr/local/go ./make.bash  <del> && for platform in linux/arm; do GOOS=${platform%/*} GOARCH=${platform##*/} GOROOT=${K8S_PATCHED_GOROOT} go install std; done </del> <ins> && for platform in linux/arm darwin/386 darwin/amd64; do GOOS=${platform%/*} GOARCH=${platform##*/} GOROOT=${K8S_PATCHED_GOROOT} go install std; done </ins>

See
CI has built and uploaded unofficial prerelease darwin binaries for 1.4 using go1.7.1: cc Official binaries will ship as soon as we cut 1.4.1.
We have official v1.4.1 binaries available now, too.
Hooray


Question: }, }, { <ins> name: "pod with init containers", opts: &DebugOptions{ CopyTo:     "debugger", Image:      "busybox", PullPolicy: corev1.PullIfNotPresent, }, pod: &corev1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name: "target", }, Spec: corev1.PodSpec{ InitContainers: []corev1.Container{ { Name: "init-container-1", }, { Name: "init-container-2", }, }, Containers: []corev1.Container{ { Name: "debugger-1", }, }, }, }, expected: &corev1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name: "debugger", }, Spec: corev1.PodSpec{ InitContainers: []corev1.Container{ { Name: "init-container-1", }, { Name: "init-container-2", }, }, Containers: []corev1.Container{ { Name: "debugger-1", }, { Name:                     "debugger-2", Image:                    "busybox", ImagePullPolicy:          corev1.PullIfNotPresent, TerminationMessagePolicy: corev1.TerminationMessageReadFile, }, }, }, }, }, { name: "pod with ephemeral containers", opts: &DebugOptions{ CopyTo:     "debugger", Image:      "busybox", PullPolicy: corev1.PullIfNotPresent, }, pod: &corev1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name: "target", }, Spec: corev1.PodSpec{ Containers: []corev1.Container{ { Name: "debugger-1", }, }, EphemeralContainers: []corev1.EphemeralContainer{ { EphemeralContainerCommon: corev1.EphemeralContainerCommon{ Name: "ephemeral-container-1", }, }, { EphemeralContainerCommon: corev1.EphemeralContainerCommon{ Name: "ephemeral-container-2", }, }, }, }, }, expected: &corev1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name: "debugger", }, Spec: corev1.PodSpec{ Containers: []corev1.Container{ { Name: "debugger-1", }, { Name:                     "debugger-2", Image:                    "busybox", ImagePullPolicy:          corev1.PullIfNotPresent, TerminationMessagePolicy: corev1.TerminationMessageReadFile, }, }, }, }, }, { </ins> name: "shared process namespace", opts: &DebugOptions{ CopyTo:                "debugger",

What happened: When using to create a debug container for a pod that has multiple init containers or ephemeral containers, fails with . What you expected to happen: should create a debug container in pod successfully. How to reproduce it (as minimally and precisely as possible): The EphemeralContainers feature gate must be enabled by using flag in your cluster. Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: OS (e.g: ): NAME="CentOS Linux" VERSION="7 (Core)" Kernel (e.g. ): Linux node-1 3.10.0-957.el7.x8664 SMP Thu Nov 8 23:39:32 UTC 2018 x8664 x8664 x8664 GNU/Linux Install tools: Network plugin and version (if this is a network-related bug): Others: /sig cli
I have investigated this issue and it looks like this issue happens in the following cases. When using without option When using with option In those cases, the containerNameToRef function is called and each for-loop in the function refers to the same despite the range is , or .
cc If this issue needs to be fixed, I would like to work on this.
/assign
Thanks for the report and for fixing it!


Question: To run serialization tests just for the current version: ```sh <del> go test ./vendor/k8s.io/api -run //HEAD </del> <ins> go test k8s.io/api -run //HEAD </ins> ``` All three formats of a given group/version/kind are expected to decode successfully to identical objects,

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --I tried rebasing an old open PR, and got merge conflicts on api testdata. I read and tried the command listed there for regenerating the test data. It failed as follows. I expected successful regeneration of the api test data. Checkout and issue the command exhibited above. Kubernetes version (use ): is at commit right nowCloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/cc
ah, latest go versions changed the invocation to
fixed in
/triage accepted


Question: } result, err := p.containerInfo.GetPodStatus(pod.Status.Host, pod.Namespace, pod.Name) <del> newStatus.HostIP = nodeStatus.HostIP </del> if err != nil { <del> glog.Errorf("error getting pod status: %v, setting status to unknown", err) newStatus.Phase = api.PodUnknown newStatus.Conditions = append(newStatus.Conditions, pod.Status.Conditions...) </del> <ins> glog.Infof("error getting pod %s status: %v, retry later", pod.Name, err) </ins> } else { <ins> newStatus.HostIP = nodeStatus.HostIP </ins> newStatus.Info = result.Status.Info newStatus.PodIP = result.Status.PodIP if newStatus.Info == nil {

I have a cluster with ~130 pods. Running back-to-back 'get pods' shows me wildly different understandings of their state, even though I can tell from docker that they have not crashed or anything. thockin kubernetes master /(NF)}' uniq -c Running: ./cluster/../cluster/gce/../../output/dockerized/bin/linux/amd64/kubectl get pods -l app=hostnames 127 Running 1 STATUS 6 Unknown thockin kubernetes master /(NF)}' uniq -c Running: ./cluster/../cluster/gce/../../output/dockerized/bin/linux/amd64/kubectl get pods -l app=hostnames 82 Running 1 STATUS 51 Unknown Nothing changed in between these calls.
cc I blame the pod cache.
has recently changed pod cache? FWIW, I also observed this behavior: 19 pods Running in one call and all Unknown in the next. With 0.10.0.
Yes, I think PR on pod cache should fix the problem reported here. Also it should fix the issue reported by about adds-on services. I have a pending PR to have kubelet deciding on PodStatus too.
Could this have caused ?
I don't see how. I'll comment on that issue.
I don't think this could be the cause of that one.
I don't think PR on pod cache will fix this. That one clears the node status. This issue is with pod going from Running to Unknown even when nodes keep running just fine.
Totally consistent observation based on my personal experience. I just assumed we were shooting for some kind of Hesinberg Uncertainty Principle. S , Tim Hockin wrote:
Agree with clears node status only. One possibility is that we hit transient network problem in between the calls that fails health check, so controller set node condition to not reachable, hence unknown pods. I'll make this more robust anyway. Other than that, it's hard to think of a reason that node status results in the flakiness.
In the absence of node information, the pod status defaults to Unknown. Maybe there's another source of this specific problem, but the pod cache is a source of continuous problems, partly because it's not just a cache, but is implementing or duplicating a significant chunk of system logic in a half-baked manner. My goal is to eliminate it -- see .

Short of that, we need to at least implement , eliminate nodes from the pod cache, and implement a real node cache.
I can look into this. Repro info from Tim, who said he was running on a one-node cluster but has seen the problem with multi-node clusters also: run-container hostnames --image=kubernetes/servehostname --replicas=1 --port=9376 --labels=app=hostnames expose hostnames --port=80 --container-port=9376 --selector="app=hostnames" resize --replicas=150 rc hostnames but I had to hack the node a bit to make it get past 100 on the node: sysctl -w fs.inotify.maxuserinstances=8192 echo "DOCKERNOFILE=" /etc/default/docker; /etc/init.d/docker restart (fails sometimes, dunno why)
I can verify that the situation is rather bad. Lots of fluctuation in the numbers of Pending, Running, and Unknown. serve_hostname containers on the node are exiting; I'm not sure why, or if the pod itself is terminating or just the container. Lots of the following in the log (not sure what this means yet): Sat, 07 Feb 2015 00:06:22 +0000 hostnames-ecd2r Pod failedScheduling {scheduler } Binding rejected: 501: All the given peers are not reachable (Tried to connect to each peer twice and failed) [0]
The pause container is also exiting, so I'm guessing that means the whole pod is exiting, not just serve_hostname...
Ugh, there is an etcd size limit you might be hitting too, I forgot. If you are seeing the fluctuations, maybe you got far enough, though. On Feb 6, 2015 5:13 PM, "David Oppenheimer" wrote:
Just a few more observations: This is reproducible (and a little less chaotic) with one node and 100 pods (but not with 50 pods, so it's definitely load-dependent). You can get to 100 Running pods (0 Pending or Unknown), and then a minute later a dozen or so will become Unknown, and then later go back to Running, etc. As mentioned earlier, the containers continue running on the node all this time (sudo docker ps wc -l), so it is definitely some kind of reporting problem.

As an aside, it looks like there is some kind of race condition with IP address allocation, as during startup (but not at steady state) a number of pods exit with the following in their log (sudo docker logs xxx): listen tcp :9376: bind: address already in use They do restart OK (I'm not sure if this is being restarted locally on the node or making a round-trip back to the scheduler).
Almost certainly the restart is just kubelet retrying after some time. On Feb 8, 2015 2:57 PM, "David Oppenheimer" wrote:
I found that if I comment out the line with the arrow next to it (see below) in , the Pods stop oscillating between Running and Unknown. Commenting out this line changes the logic from "if get an error from Kubelet, change pod status to Unknown" to "if get an error from Kubelet, keep pod status unchanged." I saw in the master logs intermittent errors of this form, which was causing the pods to flip back to Unknown: E0209 04:19:17. 4996 ] Error getting info for pod default/hostnames-v5drx: kubelet "kubernetes-minion-1.c.striped-torus-760.internal" server responded with HTTP error code 500 for pod default/hostnames-v5drx This seems to correspond to lines like this in the kubelet logs: I0209 06:38:11. 4704 ] GET /stats/default/hostnames-gfw5j/-b026-11e4-a070-/hostnames?num_stats=33: (218.857218ms) 500 goroutine 730 [running]: [ ... stack trace, but they're not always the same; I'll dig into this some more, the default stack trace depth wasn't going back past the various functions that just serve up the error ... ] However, pods still flip between Running and Pending, even though no containers are exiting on the minion, so there's another problem here too.
Thanks for investigating this! Pods really shouldn't flip between Running and Pending, as I believe that the only way that they go Pending is if Pod.Status.Host is empty, and this really shouldn't happen post scheduling, unless we are somehow dropping information. Maybe we need to make sure that we don't wipe out status on the error case?

--brendan , David Oppenheimer
I also investigated the same issue last Friday. I observed exact the same as what described above, except the related kubelet log should be: Meanwhile, I observed master is trying to query pod / and failed all the time: Talked to offline, he is going to look into why running pods go back to Pending; meanwhile, I am going to figure out Unknown cases here.
That would be great. It is very hard for me to make sense of any of the scalability experiments that I am doing while the pod status is in hysteresis.
Just to summarize: (1) the Running<-Unknown flapping is caused by the line of code I mentioned earlier; (2) the reason that line of code is being hit on the master is due to two errors on the kubelet (the one I identified earlier was wrong; Dawn found these): I0209 06:42:09. 4704 ] GET /api/v1beta1/podInfo?podID=hostnames-v5drx&podNamespace=default: (8.82543048s) 500 E0209 04:34:13. 4996 ] Error getting info for pod default/hostnames-2y8ny: kubelet "kubernetes-minion-1.c.striped-torus-760.internal" server responded with HTTP error code 500 for pod default/hostnames-2y8ny (3) we don't yet know the cause of the Running<-Pending flapping; I'm going to look into that now
(and Dawn is looking into why we're hitting those two errors on Kubelet)
Are we serving requests concurrently in Kubelet? (Is it possible not to when using go's servemux?) I don't see any locking around the pod map accesses.
This was marked for the fixit but Dawn and I are working on it. Assigned to myself.
I am also investing time writing code to try and report what is going on with unstable cluster status. It would be good to know what you and Dawn are doing in case I am wasting my time. Thanks. , David Oppenheimer
This particular problem is easily reproduced manually using the instructions Tim listed, so I'm not sure it's necessary to write anything sophisticated to drive the cluster into this state.

Anyway, maybe give me and Dawn a couple of days and if we can't figure this out by then we can call in the cavalry.
Am I wrong, or are these instructions also within bounds of an e2e test? Just because our existing e2es only do one or two things doesn't mean we can't do more within O(minutes), assuming you're dealing with the same image.
The "flips back to pending" problem appears to be because updatePodStatus() writes to the cache but not back to the Pod's status. computePodStatus() initializes newStatus based on pod.Status, not based on the cached value of status. This second behavior seems wrong in light of the first behavior. However, I believe this only causes a problem in combination with my fix described earlier, which as a result doesn't actually do what I described (it doesn't "keep the status as-is" but rather copies it out of the pod's status; to "keep the status as-is" it should be copying it out of the cache). I can see if this fixes things. (We also should fix the root cause stuff on the node side, which Dawn is working on.)
Assign the issue to me ... I didn't see the flips between Pending and Running anymore. Don't know why. I consistently see flips between Unknown and Running on those heavily loaded nodes. The root issue is once a while docker inspect API call returns us EAGAIN. Please note that on my test nodes, I already increase NOFILE limit for docker daemon process. The proposed solution is that kubelet deals with all errors internally except EAGAIN one. In this case, computePodStatus logic in pod_cache does nothing when p.containerInfo.GetPodStatus returns an error. Working on a PR to fix this.
Dawn, my suggestion is that computePodStatus() should return the current cached value whenever it gets an error from Kubelet. I can implement this if you want.
, why shouldn't updatePodStatus simply write through to the pod's status? I'd think either we always read the cached value first, or we always write through to the pod and read from there. Only reading the cached value upon an error seems counter-intuitive to me...
why do you computePodStatus returns a cached value here?

The only place calls computePodStatus is updatePodStatus, which triggered by two operations: 1) periodically update all containers 2) the client asks for PodStatus, and there is no cached one, instead calling computePodStatus to retrieve one. I think a cleaner way is 1) Kubelet handles all the errors except this transient one. Of course, we need to make sure docker is auto-restart and there is no too many open files error on node side too. But they are separate issues and should be handled by node side through script or configure etc. 2) Now back to master side, if node status is ok, kubelet should return something; if there is an error from kubelet, it should be transient; and if there is a cached one, simply return cached one upon request; if no cached one, it should be generated by computePodStatus in next round. I have one working PR to avoid the flips between Running and Unknown, but with increasing number of Pending from node side caused by docker failure. Still investigating it...
I didn't understand "if there is an error from kubelet, it should be transient; and if there is a cached one, simply return cached one upon request; if no cached one, it should be generated by computePodStatus in next round." If you're talking about the caching in the podCache on the master, then what you wrote is exactly what I was suggesting. (If you're talking about some kind of caching on the Kubelet, then that's different, because I am not aware of cache for status on the Kubelet.) I think always writing back to the pod's status might be fine but I don't know why the podCache was designed the way it was, so I didn't want to change that without asking Daniel first (I think he's the one who created it). I wasn't suggesting to only read the cached value upon an error; I meant always read the cached value under all circumstances.
and here is my understanding. Correct me if I am wrong. UpdateAllContainers periodically updates all containers, but it gets the initial pods from etcd through this line . each from , UpdateAllContainers calls passes to . computes and returns a new status. Note that the status is computed based on , i.e., the information in etcd. updates the pod cache with the new status.

Note that the new status is not written to etcd. If you repeat the same steps again, you will start with the stale pod status in the etcd because does not check the cached value.
I have tested since last night. The flaps between Running, Pending and Unknown are gone for about 10 hour running 300 pods on 4 nodes except two scenario: 1) etcd issue: cat kube- | grep "status to unknown" I0212 20:18:30. ] node doesn't exist: 501: All the given peers are not reachable (Tried to connect to each peer twice and failed) [0] <nil, setting pod hostnames-yuy8d status to unknown ... This happens a couple of times last 10 hours. We need to figure out why. 2) NodeStatus becomes UNREADY. It happens once last 10 hours, and pods status on that node become Unknown which is expected. I didn't figure out why NodeStatus was UNREADY then, but I heard there is a network earlier this morning.
up to you whether to reopen this issue or file new ones for new issues.
I've also seen the "All the given peers are not reachable" message (see my second entry in this issue). I think that was what was referring to when he said "there is an etcd size limit you might be hitting too" but I may be mis-interpreting.
That error is the etcd client's catch-all for "something went wrong" - it's actually really bad that it obscures errors. , David Oppenheimer
, Tim Hockin wrote:
and Yes, that error message is cache-all error message, which need to investigation. In my particular case, it shouldn't be etcd size limits at all since we have that cluster built and upgraded more than 10 hours before, and all 300 pods were scheduled back then. There is no new events generated either from kubelet or other components since all bound pods are indeed happily running on the nodes. No new writes to etcd except NodeStatus, which is very less frequent and not much data. The problem of etcd here occurs when master tries to read from etcd with an error.


Question: go func() { doneChan <- pf.ForwardPorts() }() <del> select { case <-pf.Ready: case <-time.After(500 * time.Millisecond): t.Fatalf("%d: timed out waiting for listeners", i) } </del> <ins> <-pf.Ready </ins> conn := testCase.Upgrader.conn

this is you
Hmm, I had upped waiting from 100ms to 500ms and that seemed to fix it. Do you have any better ideas?
Don't wait? :)
If the only reason you would timeout would be the code is broken, that's why the test runs have -timeout set.
Then the test fails immediately. It's waiting for the net.Listen calls to finish so it can connect to the specified local ports.
*finish enough so that socket connections work
So I can remove the timeout gating in my tests?
Yes ----- Original Message -----
Just failed again: https://travis-
Seems to be failing frequently now.
If this is test is failing again, my gut says it's not the same root cause. The original failure was a timeout issue. This new one is connection reset by peer. I've got the test running in a loop waiting for a failure but haven't seen one yet. is this still failing frequently?
No. It had failed twice in ~10 builds when I reopened this issue, but hasn't failed since. Reclosing. Will file a new issue if it recurs.


Question: allErrs = append(allErrs, field.Required(fldPath.Child("persistentVolumeName"), "must specify non empty persistentVolumeName")) } case source.InlineVolumeSpec != nil: <del> if utilfeature.DefaultFeatureGate.Enabled(features.CSIMigration) { allErrs = append(allErrs, storagevalidation.ValidatePersistentVolumeSpec(source.InlineVolumeSpec, "", true, fldPath.Child("inlineVolumeSpec"))...) } else { allErrs = append(allErrs, field.Forbidden(fldPath, "may not specify inlineVolumeSpec when CSIMigration feature is disabled")) } </del> <ins> allErrs = append(allErrs, storagevalidation.ValidatePersistentVolumeSpec(source.InlineVolumeSpec, "", true, fldPath.Child("inlineVolumeSpec"))...) </ins> } return allErrs }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: While glancing through storage , I noticed that we were doing feature gate checking in validation for CSI inline migration. At first glance, I think this will cause validation errors when downgrading to a version with the feature disabled. What you expected to happen: We generally don't do feature gate checking in validation in order to support downgrade cases. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/assign cc
In validateVolumeAttachmentSource : If we don't check feature gate, it seems source.InlineVolumeSpec != nil case can be dropped. Otherwise:
These were based on presence of other feature-gate validation like . Now it makes sense to align with their removal/being made unconditional as well. There are a couple of instances of feature-gate checking in validation in non-storage areas as well: and . I assume those will be changed to unconditional too?
/cc for other identified features that may share this issue.
I wonder if this PR from has already corrected the fallback problem.
Both SCTPSupport (in ) and HPAScaleToZero (in the PR where it was introduced) ensure existing objects with feature-gated data are still considered valid even when the feature gate is disabled. This is a particularly fragile/nuanced area of the project.

I'm working on a comprehensive way to tag fields as alpha and exercise the following cases in integration tests to ensure we don't regress: feature enabled creating object with alpha data is allowed and persisted creating object without alpha data is allowed and persisted update existing object without alpha data to one with alpha data is allowed and persisted (for objects that allow mutation) update existing object with alpha data to one without alpha data is allowed and persisted (for objects that allow mutation) feature disabled creating object with alpha data is either rejected or persisted without the alpha data creating object without alpha data is allowed and persisted update existing object without alpha data to one with alpha data is either rejected or persisted without alpha data (for objects that allow mutation) update existing object with alpha data to one without alpha data is allowed and persisted (for objects that allow mutation)


Question: } } }() <del> if e.services == nil { glog.Errorf("can't stop e2e services, because `services` is nil") </del> <ins> if e.services != nil { if err := e.services.kill(); err != nil { glog.Errorf("Failed to stop services: %v", err) } } if e.kubelet != nil { if err := e.kubelet.kill(); err != nil { glog.Errorf("Failed to stop kubelet: %v", err) } </ins> } <del> return e.services.kill() </del> } // RunE2EServices actually start the e2e services. This function is used to

In node e2e, we have the option to keep the services running after test for further debug. However, we set to for the services, which means that once the parent exits, they will receive and exit with the parent if they don't trap the signal. With this, option will never work. If we still need this feature, we should not set if is false. /cc
We still need this feature. It helps with debugging test failures by keeping all the daemons running upon failure. , Lantao Liu wrote:


Question: _, err := cs.CoreV1().PersistentVolumes().Get(context.TODO(), pvc.Spec.VolumeName, metav1.GetOptions{}) framework.ExpectNoError(err) <del> // Get new copy of the snapshot ginkgo.By("checking the snapshot") vs, err = dc.Resource(storageutils.SnapshotGVR).Namespace(vs.GetNamespace()).Get(context.TODO(), vs.GetName(), metav1.GetOptions{}) framework.ExpectNoError(err) // Get the bound snapshotContent snapshotStatus := vs.Object["status"].(map[string]interface{}) snapshotContentName := snapshotStatus["boundVolumeSnapshotContentName"].(string) vscontent, err := dc.Resource(storageutils.SnapshotContentGVR).Get(context.TODO(), snapshotContentName, metav1.GetOptions{}) framework.ExpectNoError(err) snapshotContentSpec := vscontent.Object["spec"].(map[string]interface{}) volumeSnapshotRef := snapshotContentSpec["volumeSnapshotRef"].(map[string]interface{}) </del> <ins> vs := sr.Vs // get the snapshot and check SnapshotContent properties vscontent := checkSnapshot(dc, sr, pattern) </ins> var restoredPVC *v1.PersistentVolumeClaim var restoredPod *v1.Pod <del> // Check SnapshotContent properties ginkgo.By("checking the SnapshotContent") // PreprovisionedCreatedSnapshot do not need to set volume snapshot class name if pattern.SnapshotType != storageframework.PreprovisionedCreatedSnapshot { framework.ExpectEqual(snapshotContentSpec["volumeSnapshotClassName"], vsc.GetName()) } framework.ExpectEqual(volumeSnapshotRef["name"], vs.GetName()) framework.ExpectEqual(volumeSnapshotRef["namespace"], vs.GetNamespace()) </del> ginkgo.By("creating a pvc from the snapshot") restoredPVC = e2epv.MakePersistentVolumeClaim(e2epv.PersistentVolumeClaimConfig{ ClaimSize:        claimSize,

The changes to support snapshot tests for Generic Ephemeral Volumes generated some duplicated code in the file as commented here: , it'd be great to create functions that are reused in both and tests. Things that are needed: A kubernetes dev cluster (I follow to create a GCE cluster) Knowledge about how to /sig storage /area testing /good-first-issue Improve readability of the storage VolumeSnapshot tests
This request has been marked as suitable for new contributors. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : ### What would you like to be ? /area testing /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : ### What would you like to be ? /area testing /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
Hi Could you please give me some more details about what needs to be done in this issue?
I modified the description, it's all about making changes in a single file creating functions that reduce the duplicated code following the thread also linked in the description.
Thanks!

Hi, I'm currently in the process of working on this issue. I'm adding the functions to the file.
is this issue solved because now I am unable to see the duplication which I saw day before yesterday.
Hi When I try to run the snapshottable tests using and I get 8 of the following failures: I have setup and as their documentation suggested, but seem to not be able to run the tests. Do you have an idea why this happens? Thanks!
what was the command that you used? The csi-hostpath driver is embedded into the kubernetes codebase so you don't need to add anything else to your kind? cluster.
The command I used to run the snapshottable tests in a local kind cluster is: . Does that seem OK?
do you have an idea about why this happens? Thanks!
hey sorry for the late response and thanks for the reminder, I always run my tests with and with a GCE cluster (not a kind cluster) for example following . The VolumeSnapshot feature requires a CSI Driver, in our e2e tests we use a CSI Driver that's embedded in the kubernetes codebase called csi-hostpath, from the logs I saw that it used csi-hostpath which I think it's ok, ideally we should test it with a real CSI Driver like PD CSI Driver but that's going to provision storage in GCP. Have you tried with ? I think that there's more info in that page about kubetest than kubetest2
Yes, I tried , but got the same errors as well. Do you think it's maybe a problem with ?
/assign
/assign
hey and you guys have solved this issues or still working on it, because i'm considering to solve this issue if you haven't solved it
/assign
thank you for signing up for this task, I believe should work, we're waiting for additional review


Question: } } <ins> func TestRejectWaitingPod(t *testing.T) { pod := &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name: "pod", UID:  types.UID("pod"), }, } testPermitPlugin := &TestPermitPlugin{} r := make(Registry) r.Register(permitPlugin, func(_ *runtime.Unknown, fh FrameworkHandle) (Plugin, error) { return testPermitPlugin, nil }) plugins := &config.Plugins{ Permit: &config.PluginSet{Enabled: []config.Plugin{{Name: permitPlugin, Weight: 1}}}, } f, err := NewFramework(r, plugins, emptyArgs) if err != nil { t.Fatalf("Failed to create framework for testing: %v", err) } go func() { for { waitingPod := f.GetWaitingPod(pod.UID) if waitingPod != nil { break } } f.RejectWaitingPod(pod.UID) }() permitStatus := f.RunPermitPlugins(context.Background(), nil, pod, "") if permitStatus.message != "pod "pod" rejected while waiting at permit: removed" { t.Fatalf("RejectWaitingPod failed, permitStatus: %v", permitStatus) } } </ins> func buildScoreConfigDefaultWeights(ps ...string) *config.Plugins { return buildScoreConfigWithWeights(defaultWeights, ps...) }

What would you like to be : When the pod is scheduled to the phase and I delete the pod, the pod still remains in the until the plugin allow/reject/timeout(Especially when waiting for a long time). Why is this needed: We need to take the initiative to delete the data in the and terminate the process in waiting state of plugin to ensure data accuracy, if the pod has been deleted /cc /priority important-soon /assign /sig scheduling
I think we can update the event handler for deleting pods to reject the pod. Can you send a PR for that?
OKI will send a pr for it.


Question: "fmt" "reflect" "testing" <del> "time" </del> "github.com/GoogleCloudPlatform/kubernetes/pkg/api" "github.com/GoogleCloudPlatform/kubernetes/pkg/conversion"

https://travis-
Taking a look.
The problem is that TestWatch is using a 10ms timeout, and after parallel test change, we seem to hit the unlucky case more often. This happens when the etcdWatch.translate goroutine never gets a chance to consume the etcdIncoming channel within 10ms. A naive fix would be to remove this 10ms timeout, and have the test rely on the default 30 sec timeout which comes with the test harness.
Ah, I see. Nice catch.


Question: // RemoveCSINode removes overall CSI-related information about node. RemoveCSINode(csiNode *storagev1beta1.CSINode) error <ins> // GetNodeInfo returns the node object with node string. GetNodeInfo(nodeName string) (*v1.Node, error) </ins> // List lists all cached pods (including assumed ones). List(labels.Selector) ([]*v1.Pod, error)

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: I recently saw an issue in which the scheduler hit this log line many times: After some digging, it turns out that PodAffinityChecker has a and . The points at the scheduler's cache instance, while the points at a light wrapper around a . In the scheduler, we use the cache to provide a globally consistent view of the cluster, even if that view is slightly out of date. Data read from the core may not have been processed by the scheduler cache yet. Letting data "skip ahead" like this can cause stale info to get persisted in the scheduler cache, and other subtle bugs. While the original problem is not likely caused by this issue, I think it needs to be cleaned up to avoid potential bugs down the line. What you expected to happen: Predicates should only read state from the scheduler's cache in order to maintain consistency. How to reproduce it (as minimally and precisely as possible): This was observed in a cluster with high rate of pod creation and node creation/deletion. Anything else we need to know?: Environment: Kubernetes version (use ): 1.12.5 Cloud provider or hardware configuration: GKE with preemptible nodes OS (e.g: ): Kernel (e.g. ): Install tools: Others:
/sig scheduling
We do have this kind of issues, and here is another one:
I'm not sure the urgency of this fix, but if its not super urgent, I would love to pick it up, if there's an eng more experienced with the scheduler who could give me some guidance :)
Hi I dug into the source code, and find out is used by the scheduler and kubelet. Maybe we could change the internal struct of , but I do not know why it appears in kubelet which seems irrelevant to the scheduler.
is just a wrapper of corelisters.NodeLister. kubelet can get node info by it.
I think the point is that the kubernetes scheduler was designed to support single instance at the beginning. Even with the addition of multiple scheduler support, the cache between each scheduler is isolated. So at this stage, we can accept this without a shared state solution.

After all, the node information is not actively modified by the kubernetes scheduler.
So is it necessary to ensure the consistency by using node lister instead of cache in the scheduler, or we do not need to fix this issue which seems like to caused by inconsistent cache.
We've recently hit this issue with the scheduler. It appears that our scheduler somehow gets out of sync with the nodes, and start scheduling pods on tainted nodes. The node's kubelet then (rightfully) refuse to admit the pod. Looking at the scheduler logs, they are flooded with the following : Similar scenario, this is a high churn cluster with thousands of pods scheduled per minute - and nodes being autoscaled. Can we try to get a fix ?
I'm glad to pick this up if we decide to fix this issue.
It would be awesome if you could fix this issue for us, we are currently seeing 100's of pods incorrectly being scheduled onto nodes with taints.
The code cited for logging "Node not found, %v" is now in :
/assign
Thanks for the interest on fixing this bug. Sorry I haven't had time to engage on this issue. Feel free to send PRs my way. I'm not 100% certain that this will fix the problem of pods getting scheduled to tainted nodes. How is the taint being applied to the node? Even if the scheduler is internally consistent, there could be a time between node creation and the taint being applied where we send pods to that node.


Question: tolerations: - operator: "Exists" effect: "NoExecute" <ins> - key: "CriticalAddonsOnly" operator: "Exists" </ins>

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): Note: Please file issues for subcomponents under the appropriate repo What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): Is this a BUG REPORT or FEATURE REQUEST? (choose one): <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): master Environment: Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: What happened: Currently, the YAML template for both npd and fluentd service do not have toleration, this may cause the daemonset cannot be started on a node if it is already selected to run , it is better to add toleration for both npd and fluentd so as to make sure those daemonset can run on the nodes even if it is for . All other daemonset in kubernetes also have such tolerations. What you expected to happen: Should add toleration for both npd and fluentd. How to reproduce it (as minimally and precisely as possible): Anything else we need to know:
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
You sent PRs to fix this, right? Can this be closed now?
the PR was not committed yet, can you help lgtm and approve it?


Question: Creates a replication controller to manage the created container(s). ``` <del> kubectl run NAME --image=image [--env="key=value"] [--port=port] [--replicas=replicas] [--dry-run=bool] [--overrides=inline-json] </del> <ins> kubectl run NAME --image=image [--env="key=value"] [--port=port] [--replicas=replicas] [--dry-run=bool] [--overrides=inline-json] [--command] -- [COMMAND] [args...] </ins> ``` ### Examples

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: ci-kubernetes-e2e-gci-gce-alpha-enabled-default Which test(s) are failing: CSI hostpath-v0 tests Since when has it been failing: 8/8 Testgrid link: https://k8s- Reason for failure: Anything else we need to know: We are removing CSI 0.3 support in 1.18, which is before selflink is going to be removed in 1.20. So we should just disable hostpath-v0 from alpha tests. /assign
/milestone v1.16


Question: return false, true, nil } <ins> // `CheckGracefulDelete` will be implemented by specific strategy </ins> if !gracefulStrategy.CheckGracefulDelete(ctx, obj, options) { return false, false, nil } <ins> if options.GracePeriodSeconds == nil { return false, false, errors.NewInternalError(fmt.Errorf("options.GracePeriodSeconds should not be nil")) } </ins> now := metav1.NewTime(metav1.Now().Add(time.Second * time.Duration(*options.GracePeriodSeconds))) objectMeta.SetDeletionTimestamp(&now) objectMeta.SetDeletionGracePeriodSeconds(options.GracePeriodSeconds)

Use apiserver extension(APIService) interface, impl its method to gracefully control resource deletion, when the field is nil, will lead to panic!! To first check if is nil, then get its value. Use apiserver extension(APIService) strategy, impl this function below will lead to panic: As the field is a int64 pointer, need to be careful when init the struct. Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig api-machinery
/triage accepted


Question: validatingWebhooks []testWebhook expectInvocations  []invocation expectError        bool <del> errorContains      string </del> <ins> errorContainsAnyOf []string </ins> }{ { name:           "minimum of request timeout or webhook timeout propagated",

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: Which test(s) are flaking: (regardlessoffailurepolicy)and_fail Testgrid link: Reason for failure: and Anything else we need to know: Spyglass: <!-- Please see the deflaking doc () for more guidance! --
/sig api-machinery /area apiserver
Can confirm this PR:
/assign
/triage accepted
The kube-apiserver processing flow is as follows, (non-critical paths are omitted) ... // other filters parse the http client timeout setting, sets the Ctx with Deadline in the http.Request ...// other filters select the http.Request.Context().Done() Pay attention to the processing logic in . Although the Context of http.Request has been cancelled, the Mutex on baseTimeoutWriter needs to be obtained between entering the function. The inner-handler may have written , ready to write , but the current code flow thinks that it should be Panic quickly, and it is recovered by and sends http2.Reset to the Client. As a result, a appears on the client side.
Currently seeing it in periodic and in presubmit for
Regardless of WithCache or WithoutCache, here is not reasonable.
/priority important-soon /milestone v1.21
PS: There were some recent failures:
/remove-priority important-soon /priority critical-urgent top flake for pull-kubernetes-integration per: triage link: FYI
may be one fix, may also be necessary (thanks for opening these
I believe that can effectively repair the issue, but currently is not the key PR to fix the issue.


Question: // 	}) // }) <del> Describe("ReplicationController", func() { </del> <ins> // These tests take ~20 minutes each. Describe("[Serial] [Slow] ReplicationController", func() { </ins> // CPU tests via replication controllers It(titleUp, func() { scaleUp("rc", kindRC, rc, f)

lightweight test which can be run in gce suite test which verifies that version v1 works


Question: func initializeCSINode(host volume.VolumeHost) error { kvh, ok := host.(volume.KubeletVolumeHost) if !ok { <del> klog.V(4).Info("Cast from VolumeHost to KubeletVolumeHost failed. Skipping CSINodeInfo initialization, not running on kubelet") </del> <ins> klog.V(4).Info("Cast from VolumeHost to KubeletVolumeHost failed. Skipping CSINode initialization, not running on kubelet") </ins> return nil } kubeClient := host.GetKubeClient() if kubeClient == nil { <del> // Kubelet running in standalone mode. Skip CSINodeInfo initialization klog.Warning("Skipping CSINodeInfo initialization, kubelet running in standalone mode") </del> <ins> // Kubelet running in standalone mode. Skip CSINode initialization klog.Warning("Skipping CSINode initialization, kubelet running in standalone mode") </ins> return nil } <del> kvh.SetKubeletError(errors.New("CSINodeInfo is not yet initialized")) </del> <ins> kvh.SetKubeletError(errors.New("CSINode is not yet initialized")) </ins> go func() { defer utilruntime.HandleCrash() <ins> // First wait indefinitely to talk to Kube APIServer nodeName := host.GetNodeName() err := waitForAPIServerForever(kubeClient, nodeName) if err != nil { klog.Fatalf("Failed to initialize CSINode while waiting for API server to report ok: %v", err) } </ins> // Backoff parameters tuned to retry over 140 seconds. Will fail and restart the Kubelet // after max retry steps. initBackoff := wait.Backoff{

What happened: Upgraded from 1.16.6 to 1.17.2 and will no longer start on control-plane hosts. It fails with: What you expected to happen: to start, static pods to launch, to connect to in the static pod How to reproduce it (as minimally and precisely as possible): Have the , , , etc in the static pod manifest folder. Try to start the with the container images not preloaded on the host. Just in case it is needed: is used as the backend is run using Anything else we need to know?: This looks to be two things interacting poorly: 1- It takes a long time for static pods to be created for the first time when the can't talk to yet. 2- The new CSI migration code force terminates causing these log lines to appear in : It looks like introduced this new fatal error. Environment: Kubernetes version (use ): 1.17.2 Cloud provider or hardware configuration: Bare-metal OS (e.g: ): Fedora 31..1 (CoreOS preview)" Kernel (e.g. ): 5.3.12-300.fc31.x86_64 Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig cluster-lifecycle
Actually, probably: /sig node
The referenced PR was in 1.14, so that interaction is not new. What is likely new is the size increase of hyperkube in 1.17 (xref ). When not pre-pulled, this can increase the time it takes to start hyperkube for the first time. Does the image pull make progress as the kubelet restarts/retries?
It might make sense for the initializeCSINode() method to: first wait to successfully talk to the kube apiserver at all, perhaps indefinitely after successfully talking to the kube apiserver, then try to register the CSINode object with backoff, and fail if it cannot do that successfully
/sig storage /remove-sig cluster-lifecycle
It doesn't seem to. Once the connection to containerd is closed, it cancels the context. Otherwise I would expect this to succeed eventually if the pull happened in the background. That sounds reasonable to me assuming it isn't blocking the static pod creation code. Do you happen to know why it takes a very long time for the static pod creation code to be called?

I would expect the pods to be created very soon after the kubelet is started.
Pulling the image and starting the processes for those pods happens immediately (from the containerd logs, you can see the image pull happening... it just doesn't complete in time before the independent kubelet logic to register the CSI node info times out).
Ah, yes. I just upgraded a couple of days ago so I didn't check whether I still saw the 2 minute delay in creating the "pod sandboxes" (this is before the pull even happens). It now is only a 23 second delay: containerd: kubelet:
ack, I like the idea Will work on a fix today and cherry-pick it as necessary


Question: local tries=5 for i in $(seq 1 $tries); do local output <del> output=$(kubectl "${kube_flags[@]:?}" api-resources --api-group mygroup.example.com -oname) </del> <ins> output=$(kubectl "${kube_flags[@]:?}" api-resources --api-group mygroup.example.com -oname || true) </ins> if kube::test::if_has_string "$output" resources.mygroup.example.com; then break fi

pull-kubernetes-integration: ci-kubernetes-integration-master: [Aa]pply&width=5 runkubectlserversideapplytests Unclear, shows up in flakes in triage history as far back as it goes No response_ Representative failure: /sig api-machinery /cc
If I'm not wrong, SSA conflict error is a real test case and expected(). That test looks failing in I could not find any recent change about that but if there is a latency for discovery endpoint being populated, this test should be flaky for a long time.
Another thing is if we get this not found error from discovery, we are not cleaning it properly and that causes run_crd test fails with already exist.
That's a good point, it's hard to clean-up properly though on failures :-/. And yes, I don't think ssa is the problem here, the problem is in getting the api-resources.
Submitted PR to attempt to fix it, it doesn't solve the clean-up issue, but at least it should hide it well :-)
/triage accepted


Question: --cluster-monitoring-mode="${KUBE_ENABLE_CLUSTER_MONITORING:-standalone}"  --prometheus-monitoring="${KUBE_ENABLE_PROMETHEUS_MONITORING:-false}"  --dns-domain="${KUBE_DNS_DOMAIN:-cluster.local}"  <ins> --ginkgo.slowSpecThreshold="${GINKGO_SLOW_SPEC_THRESHOLD:-300}"  </ins> ${KUBE_CONTAINER_RUNTIME:+"--container-runtime=${KUBE_CONTAINER_RUNTIME}"}  ${MASTER_OS_DISTRIBUTION:+"--master-os-distro=${MASTER_OS_DISTRIBUTION}"}  ${NODE_OS_DISTRIBUTION:+"--node-os-distro=${NODE_OS_DISTRIBUTION}"}

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: since we use to run our end to end tests despite nominally being a unit test framework, all tests are "slow test" in the log output due to being greater than the current threshold (default appears to be 5 seconds). What you expected to happen: We should set the threshold to something we consider to be reasonable for an end to end test. What that is might need some thought. /sig testing brought up by
ginkgo does have a we could try setting this in a job and see how it works out before we try to recommend one more generally. WDYT?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale /help
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /remove-lifecycle stale /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This could be as simple as editing:
I can make the change..


Question: defer conn.Close() h := http.Header{} <del> h.Set("type", "input") </del> <ins> h.Set(api.StreamType, api.StreamTypeError) </ins> stream, err := conn.CreateStream(h) if err != nil { t.Fatalf("error creating input stream: %v", err) } defer stream.Reset() <del> select { case <-conn.CloseChan(): close(idleSuccess) case <-time.After(150 * time.Millisecond): t.Fatalf("Timed out waiting for connection closure due to idle timeout") } </del> <ins> <-conn.CloseChan() </ins> } func TestServeExecInContainer(t *testing.T) {

this is you
Hmm, I had upped waiting from 100ms to 500ms and that seemed to fix it. Do you have any better ideas?
Don't wait? :)
If the only reason you would timeout would be the code is broken, that's why the test runs have -timeout set.
Then the test fails immediately. It's waiting for the net.Listen calls to finish so it can connect to the specified local ports.
*finish enough so that socket connections work
So I can remove the timeout gating in my tests?
Yes ----- Original Message -----
Just failed again: https://travis-
Seems to be failing frequently now.
If this is test is failing again, my gut says it's not the same root cause. The original failure was a timeout issue. This new one is connection reset by peer. I've got the test running in a loop waiting for a failure but haven't seen one yet. is this still failing frequently?
No. It had failed twice in ~10 builds when I reopened this issue, but hasn't failed since. Reclosing. Will file a new issue if it recurs.


Question: // differentiate probable errors in connection behavior between normal "this is // disconnected" should use the method. func IsProbableEOF(err error) bool { <ins> if err == nil { return false } </ins> if uerr, ok := err.(*url.Error); ok { err = uerr.Err }

https://k8s- /priority failing-test /kind bug /status approved-for-milestone /sig network cc
Fix was opened as
thanks!
Test is still flaky even after the fix. I will take a further look. cc
/assign
looks like now it start to fail on a lot of L7 LB tests
That seems to be a different failure:
The new failure was introduced by When a nil error is passed into , test will panic: Sent for fixing. cc


Question: # Optional: [Experiment Only] Run kube-proxy as a DaemonSet if set to true, run as static pods otherwise. KUBE_PROXY_DAEMONSET="${KUBE_PROXY_DAEMONSET:-false}" # true, false <ins> # Optional: duration of cluster signed certificates. CLUSTER_SIGNING_DURATION="${CLUSTER_SIGNING_DURATION:-}" </ins> # Optional: enable pod priority ENABLE_POD_PRIORITY="${ENABLE_POD_PRIORITY:-}" if [[ "${ENABLE_POD_PRIORITY}" == "true" ]]; then

/kind e2e-test-failure GCE serial and GCI GCE serial started failing last evening https://k8s- https://k8s-
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/testing: Issue will be escalated to these SIGs if needed. priority/critical-urgent: Never automatically move out of a release milestone; continually escalate to contributor and SIG through all available channels. kind/bug: Fixes a bug discovered during the current release. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details
https://k8s- is still green, I would suspect something ~last Friday afternoon gets merged
Here is what I found, after a while, the node lost their authorization to connect to API server. I believe this has the same root cause as
I am going to revert offending pr () to see if it helps. EDIT: Looks like is in the process of reverting it:

ref Test is failing on GCE slow and GCI GCE slow: https://k8s- https://k8s- Example failure https://k8s-
The test nodes appear to lose their authorization for accessing resources about 26m into the test. One of many "Unauthorized" errors starting at
, does this ring any bells ?
The certificates are all expiring:
I believe this is the regression introduced cc/ Can you guys take a look?
[MILESTONENOTIFIER] Milestone Labels Incomplete Action required: This issue requires label changes. kind: Must specify exactly one of [, , ]. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details
thanks. I'll open a PR to revert that.
Revert here


Question: framework.ImagePrePullList = NodePrePullImageList.Union(commontest.PrePulledImages) // Images from extra envs framework.ImagePrePullList.Insert(getNodeProblemDetectorImage()) <del> framework.ImagePrePullList.Insert(getSRIOVDevicePluginImage()) </del> <ins> if sriovDevicePluginImage, err := getSRIOVDevicePluginImage(); err != nil { klog.Errorln(err) } else { framework.ImagePrePullList.Insert(sriovDevicePluginImage) } if gpuDevicePluginImage, err := getGPUDevicePluginImage(); err != nil { klog.Errorln(err) } else { framework.ImagePrePullList.Insert(gpuDevicePluginImage) } </ins> } func getNodeProblemDetectorImage() string {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig testing
/sig node /triage accepted


Question: if res.StatusCode >= http.StatusOK && res.StatusCode < http.StatusBadRequest { if res.StatusCode >= http.StatusMultipleChoices { // Redirect klog.V(4).Infof("Probe terminated redirects for %s, Response: %v", url.String(), *res) <del> return probe.Warning, body, nil </del> <ins> return probe.Warning, fmt.Sprintf("Probe terminated redirects, Response body: %v", body), nil </ins> } klog.V(4).Infof("Probe succeeded for %s, Response: %v", url.String(), *res) return probe.Success, body, nil

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --Used a HTTP probe with external redirect: Got warning: Kubelet logs show: Warning messages either show additional context or don't show . I would probably have expected or just Not a big deal, just a bit confusing message Apply yaml above Kubernetes version (use ): v1.22.0-beta.1 Cloud provider or hardware configuration: kind OS (e.g: ): linux Kernel (e.g. ): 5.10.40 Install tools: kind /sig node
According to See and
thanks! I am not concerned about the actual implementation behavior, but rather the event: . We should not have an event ending with as it denotes there is more information to follow, but there is not.
Got it. /assign
/triage accepted I send to fix this issue.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
In 1.18 (because of EKS long term support I don't really want to get into), I get this: This is mildly aggravating, as it means I get fourteen lines of warning logs indicating that a service which exists solely to provide redirects is working as expected. Does this bug mean that I'll instead get an even more aggravating warning when I eventually upgrade? :) I found this bug while looking for a way to tell the probe that "getting a 308 or 302 or <insert one or a list of values heremeans success". It'd be ideal in my mind if the httpget probe allowed an "ok return values array" which accepted either single values or ranges, perhaps with a default value like .

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:


Question: "k8s.io/kubernetes/pkg/fields" "k8s.io/kubernetes/pkg/labels" "k8s.io/kubernetes/pkg/runtime" <ins> "k8s.io/kubernetes/pkg/util/errors" </ins> "k8s.io/kubernetes/pkg/watch" )

When a resource violates the limitrange, both for cpu and memory, only one error will be returned at a time. That means when a user is creating such a resource, first it will fail and return something like , them user may edit the cpu usage and create again, then fail again and get . It seem admission controls want to reject the request as fast as possible, if error occurs. But in this case, it may give users a bad experience. Would it be better to aggregate all errors and return them at one time. Also, the error message may cause misunderstanding. , it means the resource requested is 3, it may also cause a confusion that it is limited to 3.
Closing since it's fixed


Question: "k8s.io/client-go/tools/cache" podutil "k8s.io/kubernetes/pkg/api/v1/pod" "k8s.io/kubernetes/pkg/apis/core/helper" <ins> v1helper "k8s.io/kubernetes/pkg/apis/core/v1/helper" </ins> "k8s.io/kubernetes/pkg/features" kubeletapis "k8s.io/kubernetes/pkg/kubelet/apis" "k8s.io/kubernetes/pkg/scheduler"

What happened: When cleanup the existing equivalence class code. I noticed there may be duplication in handlers of Pod, PV, PVC change. For now we will invalidate volume related predicates for all nodes on PV and PVC add. Can we assume that volumes will only be attached to Node during Pod add? So we can just deal with those predicates on Pod add/delete. Don't need to bother PV/PVC at all. Any comments? Anything else we need to know?: Part of Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
Sorry, I'm really busy this week, but will think about this next week. Please ping me if I forget.
Let me think about this one predicate at a time: Volume zone predicate doesn't exist - predicate will fail, not cached a. PV doesn't exist - if you're using the new volume scheduling feature, predicate will pass. I think it is ok to ignore this predicate if you're using that feature, because your volume should be using VolumeBind predicate instead. So nothing to do here. b. PV doesn't exist - normal scenario, predicate will fail, not cached was modified to change its zone/region labels. This should not happen in normal operation, but theoretically it's possible for the user to change it. The predicate should be invalidated in this case (if we want to support it). cannot modify its VolumeName once its set. So we don't have to worry about this. /PV previously existed, but is deleted. We should start failing the predicate again. Summary: Volume zone predicate should be invalidated if: PV update modifies the zone/region labels. PVC, PV is deleted
Volume count predicate: /PV doesn't exist - predicate will succeed, maybe even incorrectly (ie PV is not one of the counted types, like NFS). Ideally, we want the count to become more accurate when the error condition is fixed, so the predicate should be invalidated. not bound - same as above volume type cannot be modified. Don't need to worry about this. /PV previously existed, but is deleted. I think this is ok to not invalidate the predicate in this case, because it would only increase the count (and the count would become less accurate).

is deleted from a node. Predicate should be invalidated for that node if it uses volumes, because it will impact the node volume count. is . Predicate should be invalidated because the node volume count may have increased from the last time we scheduled a pod. Summary: Volume count predicate should be invalidated on: PVC/PV add PVC modify volumeName Pod add if it uses volumes Pod delete for node
Volume bind predicate: PVC doesn't exist: predicate will fail PV add/delete can cause volume selection to choose something different PV modify node affinity can affect volume selection or already bound node selection PVC not bound will trigger PV selection. Once it's bound, then it should be invalidated because the other nodes should fail instead of succeed. PVC/PV deletion can cause predicate to fail when it previously succeeded Summary: Volume bind predicate should be invalidated on: PVC modify volumeName' PVC delete PV add/delete PV modify node affinity
I think in general, we need to have separate invalidation handlers for PVC and PV events because they can occur independently of Pod events. For example, the volume predicates could succeed and get cached, but because of some other unrelated predicate, the Pod won't get scheduled. In the meantime, PVC/PV events could happen that may change the results of the volume predicates.
Thank you very much for such a detailed explanation, it's really appreciated! I am writhing up a patch to fix volume related predicates based on this thread. Will ping you there very soon.


Question: ] }, { <ins> "path": "/apis/extensions/v1beta1/namespaces/{namespace}/replicationcontrollers/{name}/scale", "description": "API at /apis/extensions/v1beta1", "operations": [ { "type": "v1beta1.Scale", "method": "GET", "summary": "read scale of the specified Scale", "nickname": "readNamespacedScaleScale", "parameters": [ { "type": "string", "paramType": "query", "name": "pretty", "description": "If 'true', then the output is pretty printed.", "required": false, "allowMultiple": false }, { "type": "string", "paramType": "path", "name": "namespace", "description": "object name and auth scope, such as for teams and projects", "required": true, "allowMultiple": false }, { "type": "string", "paramType": "path", "name": "name", "description": "name of the Scale", "required": true, "allowMultiple": false } ], "responseMessages": [ { "code": 200, "message": "OK", "responseModel": "v1beta1.Scale" } ], "produces": [ "application/json" ], "consumes": [ "*/*" ] }, { "type": "v1beta1.Scale", "method": "PUT", "summary": "replace scale of the specified Scale", "nickname": "replaceNamespacedScaleScale", "parameters": [ { "type": "string", "paramType": "query", "name": "pretty", "description": "If 'true', then the output is pretty printed.", "required": false, "allowMultiple": false }, { "type": "v1beta1.Scale", "paramType": "body", "name": "body", "description": "", "required": true, "allowMultiple": false }, { "type": "string", "paramType": "path", "name": "namespace", "description": "object name and auth scope, such as for teams and projects", "required": true, "allowMultiple": false }, { "type": "string", "paramType": "path", "name": "name", "description": "name of the Scale", "required": true, "allowMultiple": false } ], "responseMessages": [ { "code": 200, "message": "OK", "responseModel": "v1beta1.Scale" } ], "produces": [ "application/json" ], "consumes": [ "*/*" ] }, { "type": "v1beta1.Scale", "method": "PATCH", "summary": "partially update scale of the specified Scale", "nickname": "patchNamespacedScaleScale", "parameters": [ { "type": "string", "paramType": "query", "name": "pretty", "description": "If 'true', then the output is pretty printed.", "required": false, "allowMultiple": false }, { "type": "unversioned.Patch", "paramType": "body", "name": "body", "description": "", "required": true, "allowMultiple": false }, { "type": "string", "paramType": "path", "name": "namespace", "description": "object name and auth scope, such as for teams and projects", "required": true, "allowMultiple": false }, { "type": "string", "paramType": "path", "name": "name", "description": "name of the Scale", "required": true, "allowMultiple": false } ], "responseMessages": [ { "code": 200, "message": "OK", "responseModel": "v1beta1.Scale" } ], "produces": [ "application/json" ], "consumes": [ "application/json-patch+json", "application/merge-patch+json", "application/strategic-merge-patch+json" ] } ] }, { </ins> "path": "/apis/extensions/v1beta1", "description": "API at /apis/extensions/v1beta1", "operations": [

e.g. scale up if CPU usage exceeds 70% of CPU request. Subtasks: [x] Add API changes [x] Add implementation [x] Add support in kubectl to print status with cpu request utilization [x] Add e2e tests
I've sent main PR for review (). During your day/my night please look into kubectl changes and e2e tests.


Question: { testName: "RunAsUserName's User is too long", windowsOptions: &core.WindowsSecurityContextOptions{ <del> RunAsUserName: toPtr(strings.Repeat("a", maxRunAsUserNameUserLength)), </del> <ins> RunAsUserName: toPtr(strings.Repeat("a", maxRunAsUserNameUserLength+1)), </ins> }, <del> expectedErrorSubstring: "runAsUserName's User length must be under", </del> <ins> expectedErrorSubstring: "runAsUserName's User length must not be longer than", </ins> }, { testName: "RunAsUserName's User cannot contain only spaces or periods",

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Currently, the character limit for the usernames set in the field is 20, which is too low, considering that "ContainerAdministrator" is a valid username and it is longer than 20 characters. A user should be able to run containers as Administrator, if needed. What you expected to happen: Users should be able to spawn pods which run as. How to reproduce it (as minimally and precisely as possible): WindowsRunAsUserName feature gate. () Anything else we need to know?: /sig windows /priority important-soon /milestone 1.16 Environment: Kubernetes version (use ): 1.16 Cloud provider or hardware configuration: N/A OS (e.g: ): Windows Server 2019 Kernel (e.g. ): N/A Install tools: N/A Network plugin and version (if this is a network-related bug): N/A Others:
You must be a member of the GitHub team to set the milestone. If you believe you should be able to issue the /milestone command, please contact your and have them propose you as an additional delegate for this responsibility. <detailsIn response to : <!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! --2. ` /priority important-soon /milestone 1.16 Kubernetes version (use ): 1.16 Cloud provider or hardware configuration: N/A OS (e.g: ): Windows Server 2019 Kernel (e.g. ): N/A Install tools: N/A Network plugin and version (if this is a network-related bug): N/A Others: </details
/milestone v1.16


Question: } } <del> fakeSource.accessToken = "fake token 2" </del> <ins> fakeSource.token = newFakeAzureToken("fake token 2", time.Now().Add(3600*time.Second)) </ins> token, err = tokenSource.Token() if err != nil { t.Errorf("failed to retrieve the cached token: %v", err)

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Currently, when AAD login is configured, refresh token timeout would make kubectl print an error, not able to re-login without manually remove the tokens. Expect kubectl to promote user to login again. Why is this needed: Configure refresh-token timeout from Azure is a way to force re-login after inactivity. kubectl doesn't promote re-login makes users harder to apply timeout policy.
/sig azure
The label(s) cannot be appled. These labels are supported: <detailsIn response to : /sig azure Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The label(s) cannot be appled. These labels are supported: <detailsIn response to : /sig platform/azure Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/area provider/azure
the root cause of this issue is in this first gets the cached token (result of device code flow) at it's trying to refresh an expired token at and failed. An error is returned. Since the expired token is still in the cache, it will always hit this path until user removes the token from kubeconfig. the fix should be: the token from cache a retry to re-initiate the device code flow in
The label(s) cannot be appled. These labels are supported: <detailsIn response to : /sig platform/azure Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cli These SIGs are my best guesses for this issue. Please comment if I am incorrect about one. I am a bot run by vllry.
I'm working on it.
are you working on it? I will grab it if I dont hear it from you :)
have the code ready will try PR this weekend Best, Ping
/assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Closing as merged


Question: } } } <ins> func TestFixDeprecatedTaintKey(t *testing.T) { fakeNow := metav1.Date(2017, 1, 1, 12, 0, 0, 0, time.UTC) evictionTimeout := 10 * time.Minute fakeNodeHandler := &testutil.FakeNodeHandler{ Existing: []*v1.Node{ { ObjectMeta: metav1.ObjectMeta{ Name:              "node0", CreationTimestamp: metav1.Date(2012, 1, 1, 0, 0, 0, 0, time.UTC), Labels: map[string]string{ kubeletapis.LabelZoneRegion:        "region1", kubeletapis.LabelZoneFailureDomain: "zone1", }, }, }, }, Clientset: fake.NewSimpleClientset(&v1.PodList{Items: []v1.Pod{*testutil.NewPod("pod0", "node0")}}), } nodeController, _ := newNodeControllerFromClient(nil, fakeNodeHandler, evictionTimeout, testRateLimiterQPS, testRateLimiterQPS, testLargeClusterThreshold, testUnhealthyThreshold, testNodeMonitorGracePeriod, testNodeStartupGracePeriod, testNodeMonitorPeriod, nil, nil, 0, false, true) nodeController.now = func() metav1.Time { return fakeNow } nodeController.recorder = testutil.NewFakeRecorder() deprecatedTaint := &v1.Taint{ Key:    algorithm.DeprecatedTaintNodeNotReady, Effect: v1.TaintEffectNoExecute, } nodeNotReadyTaint := &v1.Taint{ Key:    algorithm.TaintNodeNotReady, Effect: v1.TaintEffectNoExecute, } tests := []struct { Name           string Node           *v1.Node ExpectedTaints []*v1.Taint }{ { Name: "Node with deprecated taint key", Node: &v1.Node{ ObjectMeta: metav1.ObjectMeta{ Name:              "node0", CreationTimestamp: metav1.Date(2012, 1, 1, 0, 0, 0, 0, time.UTC), Labels: map[string]string{ kubeletapis.LabelZoneRegion:        "region1", kubeletapis.LabelZoneFailureDomain: "zone1", }, }, Spec: v1.NodeSpec{ Taints: []v1.Taint{ *deprecatedTaint, }, }, }, ExpectedTaints: []*v1.Taint{nodeNotReadyTaint}, }, { Name: "Node with not-ready taint key", Node: &v1.Node{ ObjectMeta: metav1.ObjectMeta{ Name:              "node0", CreationTimestamp: metav1.Date(2012, 1, 1, 0, 0, 0, 0, time.UTC), Labels: map[string]string{ kubeletapis.LabelZoneRegion:        "region1", kubeletapis.LabelZoneFailureDomain: "zone1", }, }, Spec: v1.NodeSpec{ Taints: []v1.Taint{ *nodeNotReadyTaint, }, }, }, ExpectedTaints: []*v1.Taint{nodeNotReadyTaint}, }, } for _, test := range tests { fakeNodeHandler.Update(test.Node) if err := syncNodeStore(nodeController, fakeNodeHandler); err != nil { t.Errorf("unexpected error: %v", err) } nodeController.doFixDeprecatedTaintKeyPass(test.Node) if err := syncNodeStore(nodeController, fakeNodeHandler); err != nil { t.Errorf("unexpected error: %v", err) } node0, err := nodeController.nodeLister.Get("node0") if err != nil { t.Errorf("Can't get current node0...") return } if len(node0.Spec.Taints) != len(test.ExpectedTaints) { t.Errorf("%s: Unexpected number of taints: expected %d, got %d", test.Name, len(test.ExpectedTaints), len(node0.Spec.Taints)) } for _, taint := range test.ExpectedTaints { if !taintutils.TaintExists(node0.Spec.Taints, taint) { t.Errorf("%s: Can't find taint %v in %v", test.Name, taint, node0.Spec.Taints) } } } } </ins>

Almost all style strings use dashes, rather than javascript names. vs .
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
Thanks for notification, a quick fix is submitted.
/cc
The PR here isn't merged yet. Can we make that happen please? We're still in alpha, so should be ok
Should I add the v1.8 milestone to this issue?
Hope so. We are pending in , need approve from
I opened with safe changes. "not-ready" change needs to be discussed more, as it's a breaking change. I believe it's best with fixing that to the time we'll be moving to beta, as we'll need to do another breaking change, i.e. remove 'alpha' from the key.
Aren't we doing this at the 11th hour? Is this going to hit 1.8?
Strings like this that are part of the core API should be part of a design doc that is formally reviewed by an API group. We need to find ways to catch this stuff earlier. We should also have an evolving API style guide that lists stuff like this. /cc
Thanks , let me make some summary: quick fix for 1.8 is , will be merged soon. final fix is , we are collaborating to reach a final solution and it's not target 1.8
Agree with Joe, can someone open a PR to that documents that taint names should follow the naming conventions for annotations?


Question: // Do the actual prebinding. Let the PV controller take care of the rest // There is no API rollback if the actual binding fails for _, binding = range bindings { <del> klog.V(5).InfoS("bindAPIUpdate: binding PV to PVC", "pod", klog.KObj(pod), "PV", klog.KObj(binding.pv), "PVC", klog.KObj(binding.pvc)) </del> // TODO: does it hurt if we make an api call and nothing needs to be updated? <del> klog.V(2).InfoS("Claim bound to volume", "PVC", klog.KObj(binding.pvc), "PV", klog.KObj(binding.pv)) </del> <ins> klog.V(5).InfoS("Updating PersistentVolume: binding to claim", "pod", klog.KObj(pod), "PV", klog.KObj(binding.pv), "PVC", klog.KObj(binding.pvc)) </ins> newPV, err := b.kubeClient.CoreV1().PersistentVolumes().Update(ctx, binding.pv, metav1.UpdateOptions{}) if err != nil { <del> klog.V(4).InfoS("Updating PersistentVolume: binding to claim failed", "PV", klog.KObj(binding.pv), "PVC", klog.KObj(binding.pvc), "err", err) </del> <ins> klog.V(4).InfoS("Updating PersistentVolume: binding to claim failed", "pod", klog.KObj(pod), "PV", klog.KObj(binding.pv), "PVC", klog.KObj(binding.pvc), "err", err) </ins> return err } <del> klog.V(4).InfoS("Updating PersistentVolume: bound to claim", "PV", klog.KObj(binding.pv), "PVC", klog.KObj(binding.pvc)) </del> <ins> klog.V(2).InfoS("Updated PersistentVolume with claim. Waiting for binding to complete", "pod", klog.KObj(pod), "PV", klog.KObj(binding.pv), "PVC", klog.KObj(binding.pvc)) </ins> // Save updated object from apiserver for later checking. binding.pv = newPV lastProcessedBinding++

This line reads as if the bound has already happened: But, if I read correctly, the bound happens immediately after: And there is even a wait for confirmation after: The log line to better describe the actual state and another log line at V(2) to confirm that the operation ended. No response all supported versions <details</details<details</details<details</details<details</details<details</details
/sig storage
cc
/triage accepted


Question: const discoveryPath = "/apis" func init() { <del> // Register conversion for apidiscovery apidiscoveryv2.SchemeBuilder.Register(apidiscoveryv2conversion.RegisterConversions) </del> utilruntime.Must(apidiscoveryv2.AddToScheme(scheme)) utilruntime.Must(apidiscoveryv2beta1.AddToScheme(scheme)) <ins> // Register conversion for apidiscovery utilruntime.Must(apidiscoveryv2conversion.RegisterConversions(scheme)) </ins> codecs = runtimeserializer.NewCodecFactory(scheme) }

Seeing data race flakes in unit tests: Appears to have just been in /sig api-machinery /priority critical-urgent /assign /cc
This code is mutating the package-level schemebuilder: I think it should be adding the conversions to the local scheme instead:
Fixed in (found several instances)
/triage accepted


Question: <ins> /* Copyright 2021 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package flowcontrol import ( "fmt" "reflect" "testing" "time" "github.com/google/go-cmp/cmp" flowcontrol "k8s.io/api/flowcontrol/v1beta2" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" ) func Test_configController_generatePatchBytes(t *testing.T) { now := time.Now().UTC() tests := []struct { name      string condition flowcontrol.FlowSchemaCondition want      []byte }{ { name: "check if only condition is parsed", condition: flowcontrol.FlowSchemaCondition{ Type:               flowcontrol.FlowSchemaConditionDangling, Status:             flowcontrol.ConditionTrue, Reason:             "test reason", Message:            "test none", LastTransitionTime: metav1.NewTime(now), }, want: []byte(fmt.Sprintf(`{"status":{"conditions":[{"type":"Dangling","status":"True","lastTransitionTime":"%s","reason":"test reason","message":"test none"}]}}`, now.Format(time.RFC3339))), }, { name: "check when message has double quotes", condition: flowcontrol.FlowSchemaCondition{ Type:               flowcontrol.FlowSchemaConditionDangling, Status:             flowcontrol.ConditionTrue, Reason:             "test reason", Message:            `test ""none`, LastTransitionTime: metav1.NewTime(now), }, want: []byte(fmt.Sprintf(`{"status":{"conditions":[{"type":"Dangling","status":"True","lastTransitionTime":"%s","reason":"test reason","message":"test ""none"}]}}`, now.Format(time.RFC3339))), }, { name: "check when message has a whitespace character that can be escaped", condition: flowcontrol.FlowSchemaCondition{ Type:   flowcontrol.FlowSchemaConditionDangling, Status: flowcontrol.ConditionTrue, Reason: "test reason", Message: `test 		none`, LastTransitionTime: metav1.NewTime(now), }, want: []byte(fmt.Sprintf(`{"status":{"conditions":[{"type":"Dangling","status":"True","lastTransitionTime":"%s","reason":"test reason","message":"test ttnone"}]}}`, now.Format(time.RFC3339))), }, { name: "check when a few fields (message & lastTransitionTime) are missing", condition: flowcontrol.FlowSchemaCondition{ Type:   flowcontrol.FlowSchemaConditionDangling, Status: flowcontrol.ConditionTrue, Reason: "test reason", }, want: []byte(`{"status":{"conditions":[{"type":"Dangling","status":"True","lastTransitionTime":null,"reason":"test reason"}]}}`), }, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { got, _ := makeFlowSchemaConditionPatch(tt.condition) if !reflect.DeepEqual(got, tt.want) { t.Errorf("makeFlowSchemaConditionPatch() got = %s, want %s; diff is %s", got, tt.want, cmp.Diff(tt.want, got)) } }) } } </ins>

A lot of different routines failing just when setting up an apiserver. Example: <details</detailsNo errors The test still passes, but those errors seem very worrisome <detailsmaster </details<details</details<details</details<details</details<details</details<details</details
/sig api-machinery /sig testing cc
yeah, the framework has some long standing tech debt /cc
all the panics I see are:
looks like again, due to
Looks like some integration tests must not be enabling SSA.
yeah, a lot don't, because it requires openapi, which also isn't universally enabled in integration. it is really expensive at API server startup, and when we tried enabling it universally in integration it blew up runtimes and integration timed out super frequently
that's what is still open for
/triage accepted


Question: func TestMasterProcessMetrics(t *testing.T) { _, s := framework.RunAMaster(t) <del> defer s.Close() </del> <ins> // TODO: Uncomment when fix #19254 // defer s.Close() </ins> metrics, err := scrapeMetrics(s) if err != nil {

http://pr- is another failure.
Also seen in http://pr-
Again. Not assigned to anyone, so I am sending to for routing
Again: Is there anyone working on this? I will spend some time looking into this.
I think that the name of this issue is misleading - the issue seems to be unrelated to etcd_watcher Basically, if you look into this part of log: So it seems that, the WaitGroup that is problematic is in httptest.Server. So my feeling is that we are simply talking to a server that is already being close and this is ~roughly duplicate of We should just find where exactly the problem is and try to work-around it.
I think this is exactly what we had before. I will send a PR with the fix today.


Question: } lbName := *lb.Name klog.V(2).Infof("reconcileLoadBalancer for service(%s): lb(%s) wantLb(%t) resolved load balancer name", serviceName, lbName, wantLb) <del> lbFrontendIPConfigName := az.getFrontendIPConfigName(service, subnet(service)) </del> <ins> lbFrontendIPConfigName := az.getFrontendIPConfigName(service) </ins> lbFrontendIPConfigID := az.getFrontendIPConfigID(lbName, lbFrontendIPConfigName) lbBackendPoolName := getBackendPoolName(az.ipv6DualStackEnabled, clusterName, service) lbBackendPoolID := az.getBackendPoolID(lbName, lbBackendPoolName)

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When the specified internal LB uses a subnet with longer name, the LB configuration will fail with following error: What you expected to happen: When constructing the frontend IP configuration name and the load balancing rules name, the subnet name should be truncated if too long. How to reproduce it (as minimally and precisely as possible): Use a subnet that has long name (70 charactors) and create ILB service. Anything else we need to know?: Environment: Kubernetes version (use ): 1.15 Cloud provider or hardware configuration: azure OS (e.g: ): Ubuntu 16.04 Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig cloud-provider /area provider/azure /assign


Question: if len(zonesToSearch) == 0 { // If zone is not provided, get the shared datastore across all node VMs. klog.V(4).Infof("Validating if datastore %s is shared across all node VMs", datastoreName) <del> sharedDsList, err = getSharedDatastoresInK8SCluster(ctx, vs.nodeManager) </del> <ins> sharedDSFinder := &sharedDatastore{ nodeManager:         vs.nodeManager, candidateDatastores: candidateDatastoreInfos, } datastoreInfo, err = sharedDSFinder.getSharedDatastore(ctx) </ins> if err != nil { klog.Errorf("Failed to get shared datastore: %+v", err) return "", err } <del> // Prepare error msg to be used later, if required. err = fmt.Errorf("The specified datastore %s is not a shared datastore across node VMs", datastoreName) </del> <ins> if datastoreInfo == nil { err = fmt.Errorf("The specified datastore %s is not a shared datastore across node VMs", datastoreName) klog.Error(err) return "", err } </ins> } else { // If zone is provided, get the shared datastores in that zone. klog.V(4).Infof("Validating if datastore %s is in zone %s ", datastoreName, zonesToSearch)

Currently whenever a vSphere volume is provisioned, we are looking at all nodes in the cluster and then fetching their hosts and then their datastores and then ensuring that selected datastore in present on all the nodes. This is done every time a new PVC is provisioned. It can be pretty expensive in large clusters and number of involved API calls can be big. /sig storage /area vsphere cc
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : Currently whenever a vSphere volume is provisioned, we are looking at all nodes in the cluster and then fetching their hosts and then their datastores and then ensuring that selected datastore in present on all the nodes. This is done every time a new PVC is provisioned. /area vsphere Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage accepted
/assign


Question: metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/types" "k8s.io/apimachinery/pkg/util/sets" <del> "k8s.io/client-go/kubernetes/fake" </del> cloudvolume "k8s.io/cloud-provider/volume" )

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: We've seen scalability issues with the current aws-ebs volume provisioner impl, which is querying possible zones from cloud provider directly (). The improvement proposed in did make api calls more efficient, but also raises people's concern about whether it will create regression or not, or is there any more reliable ways to improve. To summarize, there are 2 major concerns: zone info in Node label trustworthy? My answer is yes, since currently the zone labels (beta version only) are maintained by node controller, the information is retrieved from cloud provider directly and is always updated during node reconciliation, it can be trusted. Even if kubelet can be misconfigured by wrong zone information, node controller is going to override them. See: Once we have the GA version of zone labels pushed out, we should update node controller to maintain both zone labels. It's fair enough to assume that if user wants kubelet to report zone/region, the information should be correct. the master/minion labels trustworthy? My answer is "semantic wide they are equivalent to checking node labels from cloud provider" - just as people are not required to set master/minion labels in v1.Node, well-known labels in cloud provider is not guaranteed to be set as well. Therefore besides mainstream cluster provisioners, which maintains these sets of labels in both Kubernetes and cloud provider, the function's behavior will remain unchanged: the original impl skips nodes that are explicitly labeled with well-known master labels, same here. Please suggest. /sig aws /sig cloud-provider /kind bug /priority important-soon /cc
for item 2, an alternative is to first call a List() of all v1.Nodes, get all providerID, and make a DescribeInstance call by passing all these IDs, which might still be more efficient than running the filter, and still very heavy-weighted with large scale. Current logic can be applied to the results though. Need input from aws side.
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available .

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
One of the quick optimizations we can do is, to remove those API calls if pod placement has already been decided because of delayed binding. I think, using delayed binding works well in practice and should eliminate calls being made.
Another thing is - since EBS volume plugin is being targeted for CSI migration, we should be super careful approving any changes into in-tree plugins which is not available in EBS CSI plugin. cc
Thanks ! yes I do agree that for topo aware storage class, when we create volume, the node is already bound, so we just get the zone information from the selected node, and this is a good to have short-term improvement. In long term, as cloud providers are going to be out of tree, we have more freedom to introduce informer to the aws library (gce is already doing that), and get zone information from cache. Since the path we are going is to make kubernetes components node role neutral, it's fair enough to say that we don't have to filter out master only zone in volume provisioner, and user should leverage topology aware storage class along with labels, taints, tolerations to achieve that. what do you think?
I believe in long term we want to default all clusters to use policy. That is the direction sig-storage wants to take(IIRC ), so IMO it may not be worth fixing general immediate binding use case for in-tree EBS driver where cloudprovider itself is deprecated (and any proposed change is non-trivial).
if the direction sig-storage is going is to always use , I agree with you that fixing a deprecated path of a deprecated EBS driver would be not worthwhile. Meanwhile, since in-tree EBS driver will still stay for a few more versions, I'm proposing for optimizing code path. Could you please provide inputs? thanks! cc
Related to
Hello! I'm the bug triage lead for the 1.16 release cycle and considering this issue is tagged for 1.16, I'd like to check its status. The code freeze is starting on August 29th EOD PST (about 5 days from now) and once the code freeze starts, only release-blocking changes will be considered to be merged. Is this issue still planned to be fixed for the 1.16 release?

I have addressed this issue via , will close this issue /close
Closing this issue. <detailsIn response to : I have addressed this issue via , will close this issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details