Question: Description: A Pod is created with startup and readiness probes. The Container is started by creating /tmp/startup after 45 seconds, delaying the ready state by this amount of time. This is similar to the "Pod readiness probe, with initial delay" test. */ ginkgo.It("should be ready immediately after startupProbe succeeds", func() { <del> cmd := []string{"/bin/sh", "-c", "echo ok >/tmp/health; sleep 10; echo ok >/tmp/startup; sleep 600"} </del> <ins> // Probe workers sleep at Kubelet start for a random time which is at most PeriodSeconds // this test requires both readiness and startup workers running before updating statuses // to avoid flakes, ensure sleep before startup (22s) > readinessProbe.PeriodSeconds cmd := []string{"/bin/sh", "-c", "echo ok >/tmp/health; sleep 22; echo ok >/tmp/startup; sleep 600"} </ins> readinessProbe := &v1.Probe{ Handler:             execHandler([]string{"/bin/cat", "/tmp/health"}), InitialDelaySeconds: 0, <del> PeriodSeconds:       60, </del> <ins> PeriodSeconds:       20, </ins> } startupProbe := &v1.Probe{ Handler:             execHandler([]string{"/bin/cat", "/tmp/startup"}),

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Seems to be impacting a few KIND jobs on the master branch the most, but also assorted other jobs see the triage link above Reported by Sample failure: /sig node <!-- Please see the deflaking doc () for more guidance! --
xref probing tests
/assign ref /triage accepted
Thanks Elana, still trying today... Does anyone know if we can use the kubelet logs in e2e tests?
Not very portably in cluster e2e tests at least. Maybe in node_e2e instead.
Have you considered that the issue is not with the test but with the underlying code in kubelet like detailed in this issue Like detailed in the issue I can reliably reproduce (on k8s 1.18) the startup probe passing within a few seconds, but the container is only marked as ready after about a minute.
there is already a fix in the 1.21 release for that (part of a bigger PR): And actually this test was to verify that part of the code, but unfortunately sometimes it doesn't respond as quick as it should (but never on my PC, only in ci/cd tests)...
Some recent flakes:
one of the most common flakes in kind jobs currently failures like: Mar 25 17:03:42.508: Pod became ready in 42.007286524s, more than 5s after startupProbe succeeded. It means that the delay readiness probes were not initiated immediately after startup finished.
Maybe we could speed up the review on
Thank you I will take a look. I'm making a pass through flakes tonight trying to identify if anything is on kind's end
I don't think it's kind... since running on my PC with kind it works 100%, I think the issue is either the resources... or contention in the queue as mentioned to in the comment
/milestone v1.22
commented on PR with the suggested fix
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.

</details
Yeah...
Thanks for your great work in keeping things in check
FYI
I have (finally) understood what's happening, and I can even explain why the deflake made things worse... Please have a look at the sleep before the worker starts to process probes: So, you have to remember that we have one worker per pod and per probe, and since we have these probe settings for the test: We have (PeriodSeconds=10) starting sometimes immediately (if the Kubelet has started since more than 10s) or after a quick sleep of rand10s. On the contrary, will likely sleep most of the time for rand60s. This causes a delay in the processing of notifications: If we are lucky, this delay is less than 5s and the test passes: Now you can understand why the deflaked settings were actually worse:
The reason I didn't spot these was that I usually start my cluster once for all and iterate on the tests... so the kubelet had plenty of time to start and I never ended in that sleep.
For the solution, I suggest we either remove the sleep completely, or we find something that would make all 3 probe workers start at the same time to avoid sync issues... For the record, I was able to spot this issue thanks to the following log error (because the channel was overflowing): What do you think ?
I have found another issue in that code: If has already ran (and failed), we go to L245 and don't probe via and therefore wait for the next tick, making the test fail.
This seems consistent with what I found when I was adding e2es for the new probe-level termination grace period and why I did my timing math the way I did.
/reopen I don't think this is fixed, I just hit it again and gubernator is still showing plenty of flakes:
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
how to check the exact commit used for each test?

For instance, I am not sure (from the name) that should include my latest fix...
In your PR the fix wasn't there:
you may send cherry-pick to 1.21 for which is using kubelet 1.21, not 1.22.
is using kubelet. Cherry-pick to 1.20 may be needed as well.
I have found this one that failed even with the fix: corresponding to:
I don't know what could explain this atm... but it seems we still have delay propagating container statuses.
In the job artifacts there are multiple places with this info including the clone records/logs, and In testgrid it's a row.
yeah afaict since the tests ran on my PR after your fix was merged into master, it showed the flake was still present :( Prow will always use the latest master when running CI.
Sorry for doubting what you said... I was probably in deny and hoped this annoyance was fixed once for all :(
Plan: we'll see if Clayton's pod lifecycle refactor and/or the jitter refactor for probes affects this.
Clayton's PR jitter refactor
since isn't (likely) coming to 1.22 we could cut off 90% of flakes by increasing to 15s delay:
it seems it fails even after merging so let's continue our discussion on slack
this is not a regression, our suggestion is not block release on it. feel free to remove the milestone
/milestone v1.23
In Kubelet maybe, but in the flakiness of the e2e tests it was. Unfortunately we can't easily explore the current state :/
In the release blocking jobs the frequency is not horrible now, so unblocking is probably fine, it is not as bad as it was. Outside of the release cycle this is still the flakiest test in some jobs, see e.g. where it's the only flake source we have on record currently.
if we end up needing it, we have a timing/period adjustment PR we can take that should help with the flakiness with


Question: - name: vol containers: - name: pv-recycler <del> image: k8s.gcr.io/busybox:1.27 </del> <ins> image: k8s.gcr.io/debian-base:v2.0.0 </ins> command: - /bin/sh args:

Noticed this issue when looking at logs to figure out which images we use in a e2e test and from where. for example: Looks like it is from here:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign /sig storage cc
see background in and
slack discussion -
Looks like we can switch to see below:
looks like this image supports all the arch(es) we care about as well: !


Question: "github.com/onsi/ginkgo/v2" "github.com/onsi/gomega" <ins> "k8s.io/apimachinery/pkg/util/uuid" </ins> ) type podCondition func(pod *v1.Pod) (bool, error)

New TestGrid alerts at since commit/. Found 2 failing rows in node-kubelet-serial-containerd: 2 new, 0 other 2 new Test Failures in node-kubelet-serial-containerd. Test has failed 3 times since it last passed at commit/. It started failing between commit/ (Fri Mar 24 12:02 PDT) and commit/ (Fri Mar 24 16:03 PDT): . Test has failed 3 times since it last passed at commit/. It started failing between commit/ (Fri Mar 24 12:02 PDT) and commit/ (Fri Mar 24 16:03 PDT): . Not sure suspect No response /sig node release
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/kind flake /remove-kind failing-test See
/remove-sig release
Taking a look at The test failed due to: so it means that the pod was not terminated and deleted properly in the test that ran prior. The is the one creating this pod, but I would expect that as part of the test teardown, the namespace is deleted (and thus the pod should be deleted).
The services log show that the kubelet never sent the request to delete the pod, that is strange...
From the build-logs, the test execution for when the pod was created: Kubelet logs for the pod - Full logs are here -
Here's the termination part of the logs:
It looks like to me the issue here is that the kubelet is being restarted in a bit of an unfortunate time resulting in at least two related issues: The pod status has been updated on the api-server to terminal just prior to the kubelet restarting. The status manager did not yet complete . Then after kubelet restarts and comes back up, it attempts to add this pod , (even though it is in terminal phase). This seems like the first issue: if the pod phase is terminal on the API server, we probably should not add it back.

The pod worker receives the , but the pod worker quickly terminates (). I am assuming this is because of , i.e. based on pod cache lookup the pod is terminal. The side affect of this is that the status manager never receives the signal that the pod is (which is normally done as part of . This is because in this case, we never call which sets to be true. As a result, I think will never return true, so this pod will never be able to be deleted.
I have made a test which will reproduce this behavior in a node e2e, see It can be reproduced by starting a pod and then deleting it and triggering kubelet restart at the right time. A watch is setup to trigger a kubelet restart as soon as the pod enters the succeeded phase. If the kubelet restart happens before the kubelet is able to delete the pod on the API server, but after the phase is updated, then we trigger the issue. The pod will be re- to kubelet after restart, but will never be deleted on the API server despite it having a deletion timestamp set and being in terminal phase. Here are logs from the and running the above repro node e2e test. Here is the kubelet logs portion prior to the restart: and after restart: It also looks like we are hitting another invalid state in based on the log assertion: I tried the same test on 1.26 and it passes. This is because on 1.26, the status manager phase to terminal happens at roughly at the same time when we delete the pod. On 1.27+, after , the status phase to terminal can happen earlier and the status manager can skip deleting the pod because . In this case, after start, will never be true, so the pod never will get deleted.
But in general pods can be in terminal phase and not be deleted (like plain pods and pods from jobs). They would eventually be deleted by the pod GC otherwise. Is this a bug in the device plugin?
That is true, but the issue here is that the pods have a deletion timestamp set as well. If the deletion timestamp is set and the kubelet is up, it is the kubelet's responsibility to terminate and delete those pods.

This doesn't have anything with device plugin, rather we caught the issue due to the device plugin test which asserts that all pods are terminated and deleted prior to starting the test.
I have a WIP fix for this issue in
That makes sense
Should we aim 1.27 with this fix? This seems like an important regression.
I discussed this with yesterday and we agreed that this can wait until 1.27.1 Clients should be able to handle terminated pods. Worst case scenario, the pod GC would take care of these pods eventually.
Cherrypicked to v1.27 (to be released in 1.27.2) in


Question: // It doesn't update, probe or get the pod. type defaultPetHealthChecker struct{} <del> // isHealthy returns true if the pod is running and has the // "pod.alpha.kubernetes.io/initialized" set to "true". </del> <ins> // isHealthy returns true if the pod is ready & running. If the pod has the // "pod.alpha.kubernetes.io/initialized" annotation set to "false", pod state is ignored. </ins> func (d *defaultPetHealthChecker) isHealthy(pod *api.Pod) bool { if pod == nil || pod.Status.Phase != api.PodRunning { return false } <ins> podReady := api.IsPodReady(pod) // User may have specified a pod readiness override through a debug annotation. </ins> initialized, ok := pod.Annotations[StatefulSetInitAnnotation] <del> if !ok { glog.Infof("StatefulSet pod %v in %v, waiting on annotation %v", api.PodRunning, pod.Name, StatefulSetInitAnnotation) return false } b, err := strconv.ParseBool(initialized) if err != nil { return false </del> <ins> if ok { if initAnnotation, err := strconv.ParseBool(initialized); err != nil { glog.Infof("Failed to parse %v annotation on pod %v: %v", StatefulSetInitAnnotation, pod.Name, err) } else if !initAnnotation { glog.Infof("StatefulSet pod %v waiting on annotation %v", pod.Name, StatefulSetInitAnnotation) podReady = initAnnotation } </ins> } <del> return b && api.IsPodReady(pod) </del> <ins> return podReady </ins> } // isDying returns true if the pod has a non-nil deletion timestamp. Since the

PetSets currently support an annotation in their PodTemplateSpec field. From the user documentation: Debugging: If one want to bring up the pets in my petset one after the other, while pausing after each step, confirming that the state is as expected, one may use this functionality. For example, a petset where each pet is a replica of a distributed database; we set the to in the petset template initially. This leads to the following progression: PetSetController creates pet-0 Manually intervene and check that it has initialized as expected. Set annotation to true (equivalent to marking that pet as healthy) PetSetController creates pet-1 Manually intervene and check that it has initialized as expected. ... Blocking the PetSet controller: We may want to ensure that the petset controller is not performing scale operations when a different user-initiated action is in progress (for example, where pets are leaving due to a kubectl drain). This may be used as a mechanism to orphan a pod from a petset controller (for a closer look), while ensuring that a replacement is not created. The use case 1 above can be realized using and is probably easier to do that way rather than use the annotation. I don't see a reason to have that case supported when we can have users scale up a PetSet gradually and manually in an identical manner. It may have been of more importance before v1.4 when the scaler & reaper were not implemented. Due to the second use-case, if it would make sense to continue to have this ability, I propose that we move it from the PodTemplateSpec to the PetSetSpec, where it will eventually reside. This seems to me, like the logical place for such a debug hook to reside. In order to run/pause the scaling operations of the PetSet controller, we could have annotations along the lines of with the annotation defaulting to when absent. /cc
Should we just support Pause on PetSet (for part of these cases)?
Do users have to catch the roll out at the right time in order for this to work? For example if I want to pause between nodes 2 and 3 coming up, do I need to be precise in my timing when editing the resource? How does this look from the users perspective?
Note that Pause allows scaling in Deployments, it just blocks upgrades.

That being said, I think Pause will be useful for PetSets in general. Not sure though if we can use it somehow for this specific issue.
I like the debugging hook for PetSets. If it's optional I don't see any reason to get rid of it.
pauses petset controller if necessary and inspects the state of the system_ With the previous flow, the petset controller was paused after each pet's creation automatically, and here it would be an extra step to pause the controller it if necessary. I don't know how important it is though, but to me it seems easier to understand than a per-pod annotation.
and I had several discussions with the lifecycle of what would be useful. There are a bunch of use cases that overlap: not a primary concern for most users but helpful (prevent changes) - very important if you are triaging failures. Technically scaling is itself a user action, so if you don't want your pet set autoscaled while triaging failures you need to take action on the HPA like deployments. Since PetSets and ReplicaSets are nothing alike w.r.t. scaling I'm not terribly concerned that pause means slightly different things if for end users it Does The Right Thing control by a third party controller - this is the babysitter controller wanting to reuse the petset controller for core stuff but still have the hook angle. Not thought through enough to design around.
Note that Pause does not prevent scaling in Deployments, only upgrades , Clayton Coleman wrote:
Just to summarize: The current behavior with the annotation It is set on a per-pet basis. Setting it to false on any single pod blocks all operations on the petset; scaling, creation of new pets, etc. All pets must contain the annotation, and have its value set to true in order to make progress. By default, it is set to false and makes the petset controller pause after creating one pet. The smallest change we can make for beta: Rename it to (or some better name) Make it optional (with the absence of the annotation meaning that the petset can scale) Surface the reason better when we block on scaling because of the annotation. I still think there is value in moving it to be an annotation on the PetSet resource, rather than having it be in each pod, which is confusing to users. wdyt?
Putting it on the pod template is clever, but surprising.

How many examples are dependent on it? , Anirudh Ramanathan < :
Last I checked, all the examples had set to true. We should just get rid of it from all examples in my opinion. If we make the default (or the absence of any such annotation) to mean that the petset controller can make progress, we can safely do that. was going to do a pass over the examples after the PetSet renaming. This could be an additional step. Does that sound ok?
If it's not found, we assume you want to pause. You're proposing that if it's not found, we default to "forge ahead". SG. It should only be used in debugging/testing today. Basically if you don't know how to write your , you'd create a petset with it set, kubectl exec and write it out, scale the petset by 1, kubectl exec ..., and when you have a working cluster, put what you did in an init container and create the petset without it. This is the absolute min. You're also proposing that we move it to the petset. This makes sense overall, but I initially didn't do that because I didn't know how we would end up doing pause.
We could leave it there as "alpha" and then remove it in 1.6 once we have more time to talk about how upgrades and debugging and pause should work? , Prashanth B wrote:
Yeah the minimum work is to flip the behavior of the controller when the annotation is not found:
Agree on flipping behavior. , Prashanth B wrote:
Flipping SGTM. are you suggesting we leave the annotation-name as it is, and don't change it to ? If we are changing it, we should probably change the name from as well, since it doesn't make as much sense anymore. Probably ?
or healthy rather.
Flipping default behavior (when not present). Or did I misunderstand beeps proposal? On Oct 27, 2016, at 7:11 PM, Anirudh Ramanathan wrote: Flipping SGTM. are you suggesting we leave the annotation-name as it is, and don't change it to ? If we are changing it, we should probably change the name from initialized as well, since it doesn't make as much sense anymore. Probably unhealthy?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub , or mute the thread .
Renaming it to beta is effectively moving it to beta, so I think given that we're still confused about how this ties in to the big picture, that might be premature. If it's either move to beta or drop, I'd probably vote for drop, it's a debug hook after all. Renaming it to something else but keeping in beta makes sense.
sorry, keeping in alpha
whats going on with this tag? it's used in the docs here should i use it going forward?
It's alpha and may be removed at any time. On Nov 2, 2016, at 3:11 PM, Zachary Adam Kaplan wrote: whats going on with this tag? it's used in the docs here should i use it going forward? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub , or mute the thread .


Question: HardPodAffinitySymmetricWeight: pointer.Int32Ptr(1), HealthzBindAddress:             pointer.StringPtr("1.2.3.4:10251"), MetricsBindAddress:             pointer.StringPtr("1.2.3.4:10251"), <ins> DebuggingConfiguration: componentbaseconfig.DebuggingConfiguration{ EnableProfiling:           &enable, EnableContentionProfiling: &enable, }, </ins> LeaderElection: kubeschedulerconfigv1alpha1.KubeSchedulerLeaderElectionConfiguration{ LeaderElectionConfiguration: componentbaseconfig.LeaderElectionConfiguration{ LeaderElect:       pointer.BoolPtr(true),

What would you like to be : Enable profiling by default in the scheduler. We can do this by defaulting the CC flags to true. Why is this needed: It is useful when debugging the performance of a running cluster. API server already enables profiling by default. /sig scheduling /help /milestone v1.17
/assign /remove-help


Question: for i := range uss.clients { fsName := fmt.Sprintf("client%d", i) if atomic.AddInt32(&uss.executions[i], 0) > 0 { <del> uss.expectedExecuting = uss.expectedExecuting + fmt.Sprintf(`				apiserver_flowcontrol_current_executing_requests{flowSchema=%q,priorityLevel=%q} 0%s`, fsName, uss.name, "n") </del> <ins> uss.expectedExecuting = uss.expectedExecuting + fmt.Sprintf(`				apiserver_flowcontrol_current_executing_requests{flow_schema=%q,priority_level=%q} 0%s`, fsName, uss.name, "n") </ins> } if atomic.AddInt32(&uss.rejects[i], 0) > 0 { <del> expectedRejects = expectedRejects + fmt.Sprintf(`				apiserver_flowcontrol_rejected_requests_total{flowSchema=%q,priorityLevel=%q,reason=%q} %d%s`, fsName, uss.name, uss.rejectReason, uss.rejects[i], "n") </del> <ins> expectedRejects = expectedRejects + fmt.Sprintf(`				apiserver_flowcontrol_rejected_requests_total{flow_schema=%q,priority_level=%q,reason=%q} %d%s`, fsName, uss.name, uss.rejectReason, uss.rejects[i], "n") </ins> } } if uss.evalExecutingMetrics && len(uss.expectedExecuting) > 0 {

<!-- Please only use this template for submitting enhancement requests --What would you like to be : API priority and fairness metrics have labels such as "priorityLevel" and "flowSchema". These should be "prioritylevel" and "flowschema" respectively. These metrics have been to an exception list: Why is this needed: snake_case is consistent with other metrics. /sig api-machinery /kind cleanup /cc Do we want to do this before 1.20 beta?
yes, lets do that.
/triage accepted Yes, we do not want to go to beta with known wrongness in the metrics.


Question: testDynamicProvisioning(c, claim, "2Gi") } }) <ins> // NOTE: Slow!  The test will wait up to 5 minutes (framework.ClaimProvisionTimeout) when there is // no regression. It("should not provision a volume in an unmanaged GCE zone. [Slow]", func() { framework.SkipUnlessProviderIs("gce", "gke") var suffix string = "unmananged" By("Discovering an unmanaged zone") allZones := sets.NewString()     // all zones in the project managedZones := sets.NewString() // subset of allZones gceCloud, err := getGCECloud() Expect(err).NotTo(HaveOccurred()) // Get all k8s managed zones managedZones, err = gceCloud.GetAllZones() Expect(err).NotTo(HaveOccurred()) // Get a list of all zones in the project zones, err := gceCloud.GetComputeService().Zones.List(framework.TestContext.CloudConfig.ProjectID).Do() for _, z := range zones.Items { allZones.Insert(z.Name) } // Get the subset of zones not managed by k8s var unmanagedZone string var popped bool unmanagedZones := allZones.Difference(managedZones) // And select one of them at random. if unmanagedZone, popped = unmanagedZones.PopAny(); !popped { framework.Skipf("No unmanaged zones found.") } By("Creating a StorageClass for the unmanaged zone") sc := newStorageClass("kubernetes.io/gce-pd", suffix) // Set an unmanaged zone. sc.Parameters = map[string]string{"zone": unmanagedZone} sc, err = c.Storage().StorageClasses().Create(sc) defer Expect(c.Storage().StorageClasses().Delete(sc.Name, nil)).To(Succeed()) Expect(err).NotTo(HaveOccurred()) By("Creating a claim and expecting it to timeout") pvc := newClaim(ns, suffix, false) pvc, err = c.Core().PersistentVolumeClaims(ns).Create(pvc) defer Expect(c.Core().PersistentVolumeClaims(ns).Delete(pvc.Name, nil)).To(Succeed()) Expect(err).NotTo(HaveOccurred()) // The claim should timeout phase:Pending err = framework.WaitForPersistentVolumeClaimPhase(v1.ClaimBound, c, ns, pvc.Name, 2*time.Second, framework.ClaimProvisionTimeout) Expect(err).To(HaveOccurred()) framework.Logf(err.Error()) }) </ins> }) framework.KubeDescribe("DynamicProvisioner Alpha", func() {

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"4+", GitVersion:"v1.4.0-alpha.3.317+", GitCommit:"", GitTreeState:"clean", BuildDate:"2016-09-01T05:48:36Z", GoVersion:"go1.7", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"4+", GitVersion:"v1.4.0-alpha.3.317+", GitCommit:"", GitTreeState:"clean", BuildDate:"2016-09-01T05:48:36Z", GoVersion:"go1.7", Compiler:"gc", Platform:"linux/amd64"} What happened: Dynamic provisioned volumes in different zones with master failed to deleted What you expected to happen: Since the volume can be dynamic provisioned when the zones differs from master, then it should be deleted after delete the PVC which created them. How to reproduce it (as minimally and precisely as possible): up an all-in-one cluster. The instance is created in zone us-central1-a. several storageClass. One in zone us-central1-a. Another in zone us-central1-b. several PVC. pvc1 using storageClass1(zone us-central1-a), pvc2 using storageClass2(zone us-central1-b). PV/volume were provisioned in proper size/mode/zone. PVCs. with volume in zone us-central1-b failed with error can not find the volume.

Anything else do we need to know: Name: pvc--70d1-11e6-b17e- Labels: failure- failure- Status: Failed Claim: default/pvc5 Reclaim Policy: Delete Access Modes: RWO Capacity: 4Gi Message: Delete of volume "pvc--70d1-11e6-b17e-" failed: GCE persistent disk not found: "kubernetes-dynamic-pvc--70d1-11e6-b17e-" Source: Type: GCEPersistentDisk (a Persistent Disk resource in Google Compute Engine) PDName: kubernetes-dynamic-pvc--70d1-11e6-b17e- FSType: Partition: 0 ReadOnly: false Events: FirstSeen LastSeen Count From SubobjectPath Type Reason Message 11m 11m 1 {persistentvolume-controller } Warning VolumeFailedDelete Delete of volume "pvc--70d1-11e6-b17e-" failed: GCE persistent disk not found: "kubernetes-dynamic-pvc--70d1-11e6-b17e-" creationTimestamp: '2016-09-01T22:49:26.439-07:00' description: '{"":"pvc--70d1-11e6-b17e-","":"pvc5","":"default"}' id: '' kind: compute#disk name: kubernetes-dynamic-pvc--70d1-11e6-b17e- selfLink: sizeGb: '4' status: READY type: zone:
Kubernetes can delete GCE PDs only in zones it manages. I.e. either in the same zone as the master or, when multizone is enabled in GCE config, in all zones in the same region. This should not be allowed in the first place. Kubernetes should not create disks outside zones it manages and report an error instead. So... please use following , it will be allow creation/deletion of disks in all zones in us-central1 region: And I'll fix Kubernetes to report an error when creating a disk in all other zones.


Question: var localStorageVersion = utilversion.MustParseSemantic("v1.8.0-beta.0") <del> // variable set in BeforeEach, never modified afterwards var workerNodes sets.String </del> <ins> // variable populated in BeforeEach, never modified afterwards var workerNodes = sets.String{} </ins> type pausePodConfig struct { Name                              string

Which jobs are failing: Which test(s) are failing: [sig-scheduling] SchedulerPredicates [Serial] validates that NodeSelector is respected if matching [Conformance] [sig-scheduling] SchedulerPredicates [Serial] validates that NodeSelector is respected if not matching [Conformance] [sig-scheduling] SchedulerPredicates [Serial] validates that there exists conflict between pods with same hostPort and protocol but one using 0.0.0.0 hostIP [Conformance] [sig-scheduling] SchedulerPredicates [Serial] validates that there is no conflict between pods with same hostPort but different hostIP and protocol [Conformance] [sig-scheduling] SchedulerPreemption [Serial] PreemptionExecutionPath runs ReplicaSets to verify preemption running path [Conformance] [sig-scheduling] SchedulerPreemption [Serial] validates basic preemption works [Conformance] [sig-scheduling] SchedulerPreemption [Serial] validates lower priority pod preemption by critical pod [Conformance] Since when has it been failing: Testgrid link: Reason for failure: Test Panicked (0x0, 0xc001406da8, 0x1, 0x1, 0x0) +0x78 () +0x4e2 (0xc000e79e60) +0x35b (0xc000e79e60) +0x2b testing.tRunner(0xc000e79e60, 0x4c616e0) +0xdc created by testing.(*T).Run +0x357 Anything else we need to know: /sig scheduling /cc /priority critical-urgent /milestone v1.19
/retitle [Failing test] Conformance - GCE- master (ci-kubernetes-gce-conformance-latest) I am landing a fix for this now /assign
Thanks !


Question: resourceMonitor.Stop() }) <del> Describe("resource usage tracking", func() { density := []int{0, 50} </del> <ins> Describe("regular resource usage tracking", func() { density := []int{0, 35} for i := range density { podsPerNode := density[i] name := fmt.Sprintf( "over %v with %d pods per node.", monitoringTime, podsPerNode) It(name, func() { runResourceTrackingTest(framework, podsPerNode, nodeNames, resourceMonitor) }) } }) Describe("experimental resource usage tracking", func() { density := []int{50} </ins> for i := range density { podsPerNode := density[i] name := fmt.Sprintf( "over %v with %d pods per node.", monitoringTime, podsPerNode) It(name, func() { <del> // Skip this test for GKE. // TODO: Re-activate this for GKE SkipIfProviderIs("gke") totalPods := podsPerNode * numNodes By(fmt.Sprintf("Creating a RC of %d pods and wait until all pods of this RC are running", totalPods)) rcName := fmt.Sprintf("resource%d-%s", totalPods, string(util.NewUUID())) // TODO: Use a more realistic workload Expect(RunRC(RCConfig{ Client:    framework.Client, Name:      rcName, Namespace: framework.Namespace.Name, Image:     "gcr.io/google_containers/pause:go", Replicas:  totalPods, })).NotTo(HaveOccurred()) // Log once and flush the stats. resourceMonitor.LogLatest() resourceMonitor.Reset() By("Start monitoring resource usage") // Periodically dump the cpu summary until the deadline is met. // Note that without calling resourceMonitor.Reset(), the stats // would occupy increasingly more memory. This should be fine // for the current test duration, but we should reclaim the // entries if we plan to monitor longer (e.g., 8 hours). deadline := time.Now().Add(monitoringTime) for time.Now().Before(deadline) { Logf("Still running...%v left", deadline.Sub(time.Now())) time.Sleep(reportingPeriod) logPodsOnNodes(framework.Client, nodeNames.List()) } By("Reporting overall resource usage") logPodsOnNodes(framework.Client, nodeNames.List()) resourceMonitor.LogCPUSummary() resourceMonitor.LogLatest() By("Deleting the RC") DeleteRC(framework.Client, framework.Namespace.Name, rcName) </del> <ins> runResourceTrackingTest(framework, podsPerNode, nodeNames, resourceMonitor) </ins> }) } })

For example: I'm disabling the test in But we need to re-enable it, or roll-back the test.
fyi
Shouldn't this is part slow test, and the fix should be either: 1) Increase Kubernetes-e2-gke-ci timeout value if decide to include slow to that project or 2) disable all slow tests on that project? cc/ Which one is required here?
The test runs 50 pods per node, which is over the default --max-pod limit of kubelet. The test relies on the flag --max-pods=100 being set in cluster/gce/config- I will disable it for all GKE tests in On the other hand, it's not clear to me where the timeout is set for a specific build. /cc
Timeouts for each Jenkins build are in the Jenkins project configurations themselves, which are not currently saved in version control anywhere. :
Discussed with about the test offline. There are several issues: Do we want to run any kind of kubelet resource monitoring test on GKE? Is it useful? We only override max-pods in GCE. Should the test itself figure out what platform it can run based on the #pods in the test (see ) Per-test configuration allows greater flexibility (e.g., we may run 100 pods on GCE to test kubelet's scalability), but it'd make the tests more complicated as they need to be aware of the test environment (which many e2e tests already do); there is also the burden of updating the tests once the max-pod value changes. Does this test belong to SLOW tests or should it belong to a new category? Not all slow tests have the same attributes. Arguably this test could be enabled only on selective builds. The downside would be more flags -- less readable/manageable e2e configs. Not all builds which run SLOW tests have the proper timeout value to run this test. We need to manually change them every time a slow test is . Re-assigning this back to since she has more concerns about the PR.
Had offline discussion with We are going to split current Kubelet resource usage tracking test into two, one is targeted for GCE, another one is for GKE. In this case, we won't mess up the test code itself by introducing knobs. Re-assign this back to


Question: <ins> /* Copyright 2014 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package common import ( "fmt" "os" "path" "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/util/uuid" "k8s.io/kubernetes/test/e2e/framework" imageutils "k8s.io/kubernetes/test/utils/image" . "github.com/onsi/ginkgo" . "github.com/onsi/gomega" ) var _ = Describe("[sig-storage] Projected configMap", func() { f := framework.NewDefaultFramework("projected") /* Release : v1.9 Testname: Projected Volume, ConfigMap, volume mode default Description: A Pod is created with projected volume source ‘ConfigMap’ to store a configMap with default permission mode. Pod MUST be able to read the content of the ConfigMap successfully and the mode on the volume MUST be -rw-r—-r—-. */ framework.ConformanceIt("should be consumable from pods in volume [NodeConformance]", func() { doProjectedConfigMapE2EWithoutMappings(f, 0, 0, nil) }) /* Release : v1.9 Testname: Projected Volume, ConfigMap, volume mode 0400 Description: A Pod is created with projected volume source ‘ConfigMap’ to store a configMap with permission mode set to 0400. Pod MUST be able to read the content of the ConfigMap successfully and the mode on the volume MUST be -r——-——-—-. */ framework.ConformanceIt("should be consumable from pods in volume with defaultMode set [NodeConformance]", func() { defaultMode := int32(0400) doProjectedConfigMapE2EWithoutMappings(f, 0, 0, &defaultMode) }) It("should be consumable from pods in volume as non-root with defaultMode and fsGroup set [NodeFeature:FSGroup]", func() { defaultMode := int32(0440) /* setting fsGroup sets mode to at least 440 */ doProjectedConfigMapE2EWithoutMappings(f, 1000, 1001, &defaultMode) }) /* Release : v1.9 Testname: Projected Volume, ConfigMap, non-root user Description: A Pod is created with projected volume source ‘ConfigMap’ to store a configMap as non-root user with uid 1000. Pod MUST be able to read the content of the ConfigMap successfully and the mode on the volume MUST be -rw—r——r—-. */ framework.ConformanceIt("should be consumable from pods in volume as non-root [NodeConformance]", func() { doProjectedConfigMapE2EWithoutMappings(f, 1000, 0, nil) }) It("should be consumable from pods in volume as non-root with FSGroup [NodeFeature:FSGroup]", func() { doProjectedConfigMapE2EWithoutMappings(f, 1000, 1001, nil) }) /* Release : v1.9 Testname: Projected Volume, ConfigMap, mapped Description: A Pod is created with projected volume source ‘ConfigMap’ to store a configMap with default permission mode. The ConfigMap is also mapped to a custom path. Pod MUST be able to read the content of the ConfigMap from the custom location successfully and the mode on the volume MUST be -rw—r——r—-. */ framework.ConformanceIt("should be consumable from pods in volume with mappings [NodeConformance]", func() { doProjectedConfigMapE2EWithMappings(f, 0, 0, nil) }) /* Release : v1.9 Testname: Projected Volume, ConfigMap, mapped, volume mode 0400 Description: A Pod is created with projected volume source ‘ConfigMap’ to store a configMap with permission mode set to 0400. The ConfigMap is also mapped to a custom path. Pod MUST be able to read the content of the ConfigMap from the custom location successfully and the mode on the volume MUST be -r-—r——r—-. */ framework.ConformanceIt("should be consumable from pods in volume with mappings and Item mode set [NodeConformance]", func() { mode := int32(0400) doProjectedConfigMapE2EWithMappings(f, 0, 0, &mode) }) /* Release : v1.9 Testname: Projected Volume, ConfigMap, mapped, non-root user Description: A Pod is created with projected volume source ‘ConfigMap’ to store a configMap as non-root user with uid 1000. The ConfigMap is also mapped to a custom path. Pod MUST be able to read the content of the ConfigMap from the custom location successfully and the mode on the volume MUST be -r-—r——r—-. */ framework.ConformanceIt("should be consumable from pods in volume with mappings as non-root [NodeConformance]", func() { doProjectedConfigMapE2EWithMappings(f, 1000, 0, nil) }) It("should be consumable from pods in volume with mappings as non-root with FSGroup [NodeFeature:FSGroup]", func() { doProjectedConfigMapE2EWithMappings(f, 1000, 1001, nil) }) /* Release : v1.9 Testname: Projected Volume, ConfigMap, update Description: A Pod is created with projected volume source ‘ConfigMap’ to store a configMap and performs a create and update to new value. Pod MUST be able to create the configMap with value-1. Pod MUST be able to update the value in the confgiMap to value-2. */ framework.ConformanceIt("updates should be reflected in volume [NodeConformance]", func() { podLogTimeout := framework.GetPodSecretUpdateTimeout(f.ClientSet) containerTimeoutArg := fmt.Sprintf("--retry_time=%v", int(podLogTimeout.Seconds())) name := "projected-configmap-test-upd-" + string(uuid.NewUUID()) volumeName := "projected-configmap-volume" volumeMountPath := "/etc/projected-configmap-volume" containerName := "projected-configmap-volume-test" configMap := &v1.ConfigMap{ ObjectMeta: metav1.ObjectMeta{ Namespace: f.Namespace.Name, Name:      name, }, Data: map[string]string{ "data-1": "value-1", }, } By(fmt.Sprintf("Creating projection with configMap that has name %s", configMap.Name)) var err error if configMap, err = f.ClientSet.CoreV1().ConfigMaps(f.Namespace.Name).Create(configMap); err != nil { framework.Failf("unable to create test configMap %s: %v", configMap.Name, err) } pod := &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name: "pod-projected-configmaps-" + string(uuid.NewUUID()), }, Spec: v1.PodSpec{ Volumes: []v1.Volume{ { Name: volumeName, VolumeSource: v1.VolumeSource{ Projected: &v1.ProjectedVolumeSource{ Sources: []v1.VolumeProjection{ { ConfigMap: &v1.ConfigMapProjection{ LocalObjectReference: v1.LocalObjectReference{ Name: name, }, }, }, }, }, }, }, }, Containers: []v1.Container{ { Name:    containerName, Image:   imageutils.GetE2EImage(imageutils.Mounttest), Command: []string{"/mounttest", "--break_on_expected_content=false", containerTimeoutArg, "--file_content_in_loop=/etc/projected-configmap-volume/data-1"}, VolumeMounts: []v1.VolumeMount{ { Name:      volumeName, MountPath: volumeMountPath, ReadOnly:  true, }, }, }, }, RestartPolicy: v1.RestartPolicyNever, }, } By("Creating the pod") f.PodClient().CreateSync(pod) pollLogs := func() (string, error) { return framework.GetPodLogs(f.ClientSet, f.Namespace.Name, pod.Name, containerName) } Eventually(pollLogs, podLogTimeout, framework.Poll).Should(ContainSubstring("value-1")) By(fmt.Sprintf("Updating configmap %v", configMap.Name)) configMap.ResourceVersion = "" // to force update configMap.Data["data-1"] = "value-2" _, err = f.ClientSet.CoreV1().ConfigMaps(f.Namespace.Name).Update(configMap) Expect(err).NotTo(HaveOccurred(), "Failed to update configmap %q in namespace %q", configMap.Name, f.Namespace.Name) By("waiting to observe update in volume") Eventually(pollLogs, podLogTimeout, framework.Poll).Should(ContainSubstring("value-2")) }) /* Release : v1.9 Testname: Projected Volume, ConfigMap, create, update and delete Description: Create a Pod with three containers with ConfigMaps namely a create, update and delete container. Create Container when started MUST not have configMap, update and delete containers MUST be created with a ConfigMap value as ‘value-1’. Create a configMap in the create container, the Pod MUST be able to read the configMap from the create container. Update the configMap in the update container, Pod MUST be able to read the updated configMap value. Delete the configMap in the delete container. Pod MUST fail to read the configMap from the delete container. */ framework.ConformanceIt("optional updates should be reflected in volume [NodeConformance]", func() { podLogTimeout := framework.GetPodSecretUpdateTimeout(f.ClientSet) containerTimeoutArg := fmt.Sprintf("--retry_time=%v", int(podLogTimeout.Seconds())) trueVal := true volumeMountPath := "/etc/projected-configmap-volumes" deleteName := "cm-test-opt-del-" + string(uuid.NewUUID()) deleteContainerName := "delcm-volume-test" deleteVolumeName := "deletecm-volume" deleteConfigMap := &v1.ConfigMap{ ObjectMeta: metav1.ObjectMeta{ Namespace: f.Namespace.Name, Name:      deleteName, }, Data: map[string]string{ "data-1": "value-1", }, } updateName := "cm-test-opt-upd-" + string(uuid.NewUUID()) updateContainerName := "updcm-volume-test" updateVolumeName := "updatecm-volume" updateConfigMap := &v1.ConfigMap{ ObjectMeta: metav1.ObjectMeta{ Namespace: f.Namespace.Name, Name:      updateName, }, Data: map[string]string{ "data-1": "value-1", }, } createName := "cm-test-opt-create-" + string(uuid.NewUUID()) createContainerName := "createcm-volume-test" createVolumeName := "createcm-volume" createConfigMap := &v1.ConfigMap{ ObjectMeta: metav1.ObjectMeta{ Namespace: f.Namespace.Name, Name:      createName, }, Data: map[string]string{ "data-1": "value-1", }, } By(fmt.Sprintf("Creating configMap with name %s", deleteConfigMap.Name)) var err error if deleteConfigMap, err = f.ClientSet.CoreV1().ConfigMaps(f.Namespace.Name).Create(deleteConfigMap); err != nil { framework.Failf("unable to create test configMap %s: %v", deleteConfigMap.Name, err) } By(fmt.Sprintf("Creating configMap with name %s", updateConfigMap.Name)) if updateConfigMap, err = f.ClientSet.CoreV1().ConfigMaps(f.Namespace.Name).Create(updateConfigMap); err != nil { framework.Failf("unable to create test configMap %s: %v", updateConfigMap.Name, err) } pod := &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name: "pod-projected-configmaps-" + string(uuid.NewUUID()), }, Spec: v1.PodSpec{ Volumes: []v1.Volume{ { Name: deleteVolumeName, VolumeSource: v1.VolumeSource{ Projected: &v1.ProjectedVolumeSource{ Sources: []v1.VolumeProjection{ { ConfigMap: &v1.ConfigMapProjection{ LocalObjectReference: v1.LocalObjectReference{ Name: deleteName, }, Optional: &trueVal, }, }, }, }, }, }, { Name: updateVolumeName, VolumeSource: v1.VolumeSource{ Projected: &v1.ProjectedVolumeSource{ Sources: []v1.VolumeProjection{ { ConfigMap: &v1.ConfigMapProjection{ LocalObjectReference: v1.LocalObjectReference{ Name: updateName, }, Optional: &trueVal, }, }, }, }, }, }, { Name: createVolumeName, VolumeSource: v1.VolumeSource{ Projected: &v1.ProjectedVolumeSource{ Sources: []v1.VolumeProjection{ { ConfigMap: &v1.ConfigMapProjection{ LocalObjectReference: v1.LocalObjectReference{ Name: createName, }, Optional: &trueVal, }, }, }, }, }, }, }, Containers: []v1.Container{ { Name:    deleteContainerName, Image:   imageutils.GetE2EImage(imageutils.Mounttest), Command: []string{"/mounttest", "--break_on_expected_content=false", containerTimeoutArg, "--file_content_in_loop=/etc/projected-configmap-volumes/delete/data-1"}, VolumeMounts: []v1.VolumeMount{ { Name:      deleteVolumeName, MountPath: path.Join(volumeMountPath, "delete"), ReadOnly:  true, }, }, }, { Name:    updateContainerName, Image:   imageutils.GetE2EImage(imageutils.Mounttest), Command: []string{"/mounttest", "--break_on_expected_content=false", containerTimeoutArg, "--file_content_in_loop=/etc/projected-configmap-volumes/update/data-3"}, VolumeMounts: []v1.VolumeMount{ { Name:      updateVolumeName, MountPath: path.Join(volumeMountPath, "update"), ReadOnly:  true, }, }, }, { Name:    createContainerName, Image:   imageutils.GetE2EImage(imageutils.Mounttest), Command: []string{"/mounttest", "--break_on_expected_content=false", containerTimeoutArg, "--file_content_in_loop=/etc/projected-configmap-volumes/create/data-1"}, VolumeMounts: []v1.VolumeMount{ { Name:      createVolumeName, MountPath: path.Join(volumeMountPath, "create"), ReadOnly:  true, }, }, }, }, RestartPolicy: v1.RestartPolicyNever, }, } By("Creating the pod") f.PodClient().CreateSync(pod) pollCreateLogs := func() (string, error) { return framework.GetPodLogs(f.ClientSet, f.Namespace.Name, pod.Name, createContainerName) } Eventually(pollCreateLogs, podLogTimeout, framework.Poll).Should(ContainSubstring("Error reading file /etc/projected-configmap-volumes/create/data-1")) pollUpdateLogs := func() (string, error) { return framework.GetPodLogs(f.ClientSet, f.Namespace.Name, pod.Name, updateContainerName) } Eventually(pollUpdateLogs, podLogTimeout, framework.Poll).Should(ContainSubstring("Error reading file /etc/projected-configmap-volumes/update/data-3")) pollDeleteLogs := func() (string, error) { return framework.GetPodLogs(f.ClientSet, f.Namespace.Name, pod.Name, deleteContainerName) } Eventually(pollDeleteLogs, podLogTimeout, framework.Poll).Should(ContainSubstring("value-1")) By(fmt.Sprintf("Deleting configmap %v", deleteConfigMap.Name)) err = f.ClientSet.CoreV1().ConfigMaps(f.Namespace.Name).Delete(deleteConfigMap.Name, &metav1.DeleteOptions{}) Expect(err).NotTo(HaveOccurred(), "Failed to delete configmap %q in namespace %q", deleteConfigMap.Name, f.Namespace.Name) By(fmt.Sprintf("Updating configmap %v", updateConfigMap.Name)) updateConfigMap.ResourceVersion = "" // to force update delete(updateConfigMap.Data, "data-1") updateConfigMap.Data["data-3"] = "value-3" _, err = f.ClientSet.CoreV1().ConfigMaps(f.Namespace.Name).Update(updateConfigMap) Expect(err).NotTo(HaveOccurred(), "Failed to update configmap %q in namespace %q", updateConfigMap.Name, f.Namespace.Name) By(fmt.Sprintf("Creating configMap with name %s", createConfigMap.Name)) if createConfigMap, err = f.ClientSet.CoreV1().ConfigMaps(f.Namespace.Name).Create(createConfigMap); err != nil { framework.Failf("unable to create test configMap %s: %v", createConfigMap.Name, err) } By("waiting to observe update in volume") Eventually(pollCreateLogs, podLogTimeout, framework.Poll).Should(ContainSubstring("value-1")) Eventually(pollUpdateLogs, podLogTimeout, framework.Poll).Should(ContainSubstring("value-3")) Eventually(pollDeleteLogs, podLogTimeout, framework.Poll).Should(ContainSubstring("Error reading file /etc/projected-configmap-volumes/delete/data-1")) }) /* Release : v1.9 Testname: Projected Volume, ConfigMap, multiple volume paths Description: A Pod is created with a projected volume source ‘ConfigMap’ to store a configMap. The configMap is mapped to two different volume mounts. Pod MUST be able to read the content of the configMap successfully from the two volume mounts. */ framework.ConformanceIt("should be consumable in multiple volumes in the same pod [NodeConformance]", func() { var ( name             = "projected-configmap-test-volume-" + string(uuid.NewUUID()) volumeName       = "projected-configmap-volume" volumeMountPath  = "/etc/projected-configmap-volume" volumeName2      = "projected-configmap-volume-2" volumeMountPath2 = "/etc/projected-configmap-volume-2" configMap        = newConfigMap(f, name) ) By(fmt.Sprintf("Creating configMap with name %s", configMap.Name)) var err error if configMap, err = f.ClientSet.CoreV1().ConfigMaps(f.Namespace.Name).Create(configMap); err != nil { framework.Failf("unable to create test configMap %s: %v", configMap.Name, err) } pod := &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name: "pod-projected-configmaps-" + string(uuid.NewUUID()), }, Spec: v1.PodSpec{ Volumes: []v1.Volume{ { Name: volumeName, VolumeSource: v1.VolumeSource{ Projected: &v1.ProjectedVolumeSource{ Sources: []v1.VolumeProjection{ { ConfigMap: &v1.ConfigMapProjection{ LocalObjectReference: v1.LocalObjectReference{ Name: name, }, }, }, }, }, }, }, { Name: volumeName2, VolumeSource: v1.VolumeSource{ Projected: &v1.ProjectedVolumeSource{ Sources: []v1.VolumeProjection{ { ConfigMap: &v1.ConfigMapProjection{ LocalObjectReference: v1.LocalObjectReference{ Name: name, }, }, }, }, }, }, }, }, Containers: []v1.Container{ { Name:  "projected-configmap-volume-test", Image: imageutils.GetE2EImage(imageutils.Mounttest), Args:  []string{"--file_content=/etc/projected-configmap-volume/data-1"}, VolumeMounts: []v1.VolumeMount{ { Name:      volumeName, MountPath: volumeMountPath, ReadOnly:  true, }, { Name:      volumeName2, MountPath: volumeMountPath2, ReadOnly:  true, }, }, }, }, RestartPolicy: v1.RestartPolicyNever, }, } f.TestContainerOutput("consume configMaps", pod, 0, []string{ "content of file "/etc/projected-configmap-volume/data-1": value-1", }) }) }) func doProjectedConfigMapE2EWithoutMappings(f *framework.Framework, uid, fsGroup int64, defaultMode *int32) { userID := int64(uid) groupID := int64(fsGroup) var ( name            = "projected-configmap-test-volume-" + string(uuid.NewUUID()) volumeName      = "projected-configmap-volume" volumeMountPath = "/etc/projected-configmap-volume" configMap       = newConfigMap(f, name) ) By(fmt.Sprintf("Creating configMap with name %s", configMap.Name)) var err error if configMap, err = f.ClientSet.CoreV1().ConfigMaps(f.Namespace.Name).Create(configMap); err != nil { framework.Failf("unable to create test configMap %s: %v", configMap.Name, err) } pod := &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name: "pod-projected-configmaps-" + string(uuid.NewUUID()), }, Spec: v1.PodSpec{ SecurityContext: &v1.PodSecurityContext{}, Volumes: []v1.Volume{ { Name: volumeName, VolumeSource: v1.VolumeSource{ Projected: &v1.ProjectedVolumeSource{ Sources: []v1.VolumeProjection{ { ConfigMap: &v1.ConfigMapProjection{ LocalObjectReference: v1.LocalObjectReference{ Name: name, }, }, }, }, }, }, }, }, Containers: []v1.Container{ { Name:  "projected-configmap-volume-test", Image: imageutils.GetE2EImage(imageutils.Mounttest), Args: []string{ "--file_content=/etc/projected-configmap-volume/data-1", "--file_mode=/etc/projected-configmap-volume/data-1"}, VolumeMounts: []v1.VolumeMount{ { Name:      volumeName, MountPath: volumeMountPath, }, }, }, }, RestartPolicy: v1.RestartPolicyNever, }, } if userID != 0 { pod.Spec.SecurityContext.RunAsUser = &userID } if groupID != 0 { pod.Spec.SecurityContext.FSGroup = &groupID } if defaultMode != nil { //pod.Spec.Volumes[0].VolumeSource.Projected.Sources[0].ConfigMap.DefaultMode = defaultMode pod.Spec.Volumes[0].VolumeSource.Projected.DefaultMode = defaultMode } else { mode := int32(0644) defaultMode = &mode } modeString := fmt.Sprintf("%v", os.FileMode(*defaultMode)) output := []string{ "content of file "/etc/projected-configmap-volume/data-1": value-1", "mode of file "/etc/projected-configmap-volume/data-1": " + modeString, } f.TestContainerOutput("consume configMaps", pod, 0, output) } func doProjectedConfigMapE2EWithMappings(f *framework.Framework, uid, fsGroup int64, itemMode *int32) { userID := int64(uid) groupID := int64(fsGroup) var ( name            = "projected-configmap-test-volume-map-" + string(uuid.NewUUID()) volumeName      = "projected-configmap-volume" volumeMountPath = "/etc/projected-configmap-volume" configMap       = newConfigMap(f, name) ) By(fmt.Sprintf("Creating configMap with name %s", configMap.Name)) var err error if configMap, err = f.ClientSet.CoreV1().ConfigMaps(f.Namespace.Name).Create(configMap); err != nil { framework.Failf("unable to create test configMap %s: %v", configMap.Name, err) } pod := &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name: "pod-projected-configmaps-" + string(uuid.NewUUID()), }, Spec: v1.PodSpec{ SecurityContext: &v1.PodSecurityContext{}, Volumes: []v1.Volume{ { Name: volumeName, VolumeSource: v1.VolumeSource{ Projected: &v1.ProjectedVolumeSource{ Sources: []v1.VolumeProjection{ { ConfigMap: &v1.ConfigMapProjection{ LocalObjectReference: v1.LocalObjectReference{ Name: name, }, Items: []v1.KeyToPath{ { Key:  "data-2", Path: "path/to/data-2", }, }, }, }, }, }, }, }, }, Containers: []v1.Container{ { Name:  "projected-configmap-volume-test", Image: imageutils.GetE2EImage(imageutils.Mounttest), Args: []string{"--file_content=/etc/projected-configmap-volume/path/to/data-2", "--file_mode=/etc/projected-configmap-volume/path/to/data-2"}, VolumeMounts: []v1.VolumeMount{ { Name:      volumeName, MountPath: volumeMountPath, ReadOnly:  true, }, }, }, }, RestartPolicy: v1.RestartPolicyNever, }, } if userID != 0 { pod.Spec.SecurityContext.RunAsUser = &userID } if groupID != 0 { pod.Spec.SecurityContext.FSGroup = &groupID } if itemMode != nil { //pod.Spec.Volumes[0].VolumeSource.ConfigMap.Items[0].Mode = itemMode pod.Spec.Volumes[0].VolumeSource.Projected.DefaultMode = itemMode } else { mode := int32(0644) itemMode = &mode } // Just check file mode if fsGroup is not set. If fsGroup is set, the // final mode is adjusted and we are not testing that case. output := []string{ "content of file "/etc/projected-configmap-volume/path/to/data-2": value-2", } if fsGroup == 0 { modeString := fmt.Sprintf("%v", os.FileMode(*itemMode)) output = append(output, "mode of file "/etc/projected-configmap-volume/path/to/data-2": "+modeString) } f.TestContainerOutput("consume configMaps", pod, 0, output) } </ins>

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: Running the non- conformance tests with produces unpredictable results, sometimes tests show up again with a on the end. This is due to the following tests having multiple tests with identical names: - - What you expected to happen: Tests should have unique names. How to reproduce it (as minimally and precisely as possible): run the conformance tests with a skip on and Anything else we need to know?: example here: https://k8s- /sig node /sig testing cc
er this is probably better /sig storage cc
/assign /assign
/area conformance
/milestone v1.13


Question: } } <del> // Set the arp_ignore sysctl we need for if val, _ := sysctl.GetSysctl(sysctlArpIgnore); val != 1 { if err := sysctl.SetSysctl(sysctlArpIgnore, 1); err != nil { return nil, fmt.Errorf("can't set sysctl %s: %v", sysctlArpIgnore, err) </del> <ins> if strictARP { // Set the arp_ignore sysctl we need for if val, _ := sysctl.GetSysctl(sysctlArpIgnore); val != 1 { if err := sysctl.SetSysctl(sysctlArpIgnore, 1); err != nil { return nil, fmt.Errorf("can't set sysctl %s: %v", sysctlArpIgnore, err) } </ins> } <del> } // Set the arp_announce sysctl we need for if val, _ := sysctl.GetSysctl(sysctlArpAnnounce); val != 2 { if err := sysctl.SetSysctl(sysctlArpAnnounce, 2); err != nil { return nil, fmt.Errorf("can't set sysctl %s: %v", sysctlArpAnnounce, err) </del> <ins> // Set the arp_announce sysctl we need for if val, _ := sysctl.GetSysctl(sysctlArpAnnounce); val != 2 { if err := sysctl.SetSysctl(sysctlArpAnnounce, 2); err != nil { return nil, fmt.Errorf("can't set sysctl %s: %v", sysctlArpAnnounce, err) } </ins> } }

/kind bug Kubernetes version: 1.13.1. kube-proxy mode: IPVS. SDN: flannel. We are used services like this: where IP 192.168.10.201 is from our internal /24 network (in particular, our Kube nodes have IPs 192.168.10.1, 192.168.10.2 etc). Note that there is no conflicts between real IPs and virtual external ones. Before updating from 1.12.3 to 1.13.1, it worked correctly. In partucular these IPs was accessible from other non-Kubernetes hosts from 192.168.10.0/24 network. After update kube-proxy to 1.13.1, this feature stops working: non-Kubernetes hosts cannot resolve service IPs via ARP: On Kubernetes hosts it still working, because not requires ARP resolving. If we reset sysctl parameter arpignore on Kube hosts, all works normally even from non-Kube hosts. Quck debugging shows that problem is in the commit . Please make this behaviour opt-out. (Now we resets arpignore to 0 via cron every 1 min.)
/sig network
Having the same problem on Kubernetes 1.13.2 btw Thanks for a workaround!
Same problem, fix this please!!!
This is similar to We've had a discussion there on how to best implement a solution but haven't decided yet I'm in favor of flag which defaults to the previous behavior to avoid breaking existing setups (alternatives are cleaner but involve creating an additional arptable rule per service which I think we should avoid) what do you think?
OK

What happened: After PR: , we noticed communications issue in our pods. We use the Lyft CNI plugin () which relies on two interfaces in pod: an IPVLAN interface connecting to the AWS VPC which has a VPC IP an Unnumbered point-to-point interface (veth) connecting to the host and that does not have an IP (it "borrows" the one from the IPVLAN interface) On the host side, there is a veth interface which also doesn't have an IP (it borrows one from the main interface). With the latest versions of kube-proxy communication between host and pods break What you expected to happen: Communication still works How to reproduce it (as minimally and precisely as possible): Use the Lyft CNI plugin Deploy kube-proxy Try to ping the host from a pod (or a pod from the host) Anything else we need to know?: The new sysctls create 2 issues: containers started after kube-proxy inherit so they will not answer ARP queries on the veth interface the veth on the host side also does not have an IP so it will no answer ARP queries from the pods I know that our setup is pretty uncommon but assuming that the veth on the host side will have an IP seems dangerous. Some discussions on the PTP plugin for instance suggest not using IP for the host interface: containernetworking/plugins Possible fixes: create a flag to enable ARP filtering (I think to avoid breaking existing setups, we should disable it by default but I'm fine with enabling it by default as long as we document the solution) use arptables to filter ARP queries for clusterIPs (): a bit more work because we need to do it for all clusterIPs (there may be a more efficient way to do it but I could not make it work by filtering on the kube-ipvs interface) and this requires the arptables binary. We could test for the existence of the binary and log a warning if it is missing (similar to behavior with conntrack) Environment: Kubernetes version (use ): kube-proxy OS (e.g. from /etc/os-release): Kernel (e.g. ): /area ipvs /sig network /kind bug /assign ping
I can say only that solution with arptables working fine with MetalLB.
Oh, really? Glad you confirm that.

But I would like to make sure that we won't create too many arptables rules. Can ipset works with arptables?
It doesn't look like arptables works with ipset
Unfortunately doesn't support ipsets, but new do. (not fully implemented yet) Eg we can create ipset for arp: But current implementation is not allowing to use it for arp rules for now: Will return: I guess because there is still no type specified for (destination ip) and (source ip) BTW: nft does not allow to create set of integer components: But there is a way for specify multiple IPs in one rule, eg: Where is IPv4 addresses in decimal format More info here:
Another option would be to use the and filter all queries but this flag is not in kube-proxy today
Bad idea, it will handle but not - which can be really differ
can you explain more detail why Lyft CNI can not assign IPs to the right interfaces for avoid this kind of issues?
Yes you are right. bad idea It could definitely add IPs to these interfaces but it's not the choice that was made. More details here: ("Unnumbered point-to-point interface"). I could probably PR a fix there. I'm a bit worried that some other CNI plugins could break here though
We are about to deploy some 1.13 clusters and can't use kube-proxy 1.13 as is We could start with a custom build but I assume others will be impacted The simplest way forward for now is probably to use a flag to control this behavior. I think the safest is probably to disable this by default to avoid the issue for people upgrading but if you think it is really important we can enable it by default. I know we'd like to avoid adding more flags but other solutions seem a bit more complicated and will most likely result in one iptables rule per service. What do you think ?
Just FYI, we have another CNI plugin that is affected by this.
We addressed the issue by assigning a link-local address 169.254.1.1/32 to the host-side veth in . So this issue is not critical at least for our case.
Good news everyone, new nft supports setting ip address for arp filters : that's makes possible for using sets


Question: v1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime" <ins> utilruntime "k8s.io/apimachinery/pkg/util/runtime" </ins> "k8s.io/apimachinery/pkg/util/wait" "k8s.io/client-go/informers" coreinformers "k8s.io/client-go/informers/core/v1"

Currently the scheduler may add an assumed pod back to the queue if the update didn't include specific fields (see ). There doesn't seem to be a reason why we can't simply ignore all updates of assumed pods. Assumed pods are waiting to be bound, if binding succeeded then the pod is scheduled. If binding fails, the pod's latest version will be put back into the unschedulable queue anyways. Performance: we save on unnecessary scheduling attempts. This is a followup on the discussion we had in /sig scheduling /triage accepted /help
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : #### What would you like to be : /triage accepted /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
What we basically need to do is: 1) remove function 2) in the places where we call , we should simply call
/assign


Question: leaveStdinOpen := o.LeaveStdinOpen waitForExitCode := !leaveStdinOpen && restartPolicy == corev1.RestartPolicyNever if waitForExitCode { <del> pod, err = waitForPod(clientset.CoreV1(), attachablePod.Namespace, attachablePod.Name, podCompleted) </del> <ins> pod, err = waitForPod(clientset.CoreV1(), attachablePod.Namespace, attachablePod.Name, opts.GetPodTimeout, podCompleted) </ins> if err != nil { return err }

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: gce-master-scale-correctness Which test(s) are failing: [sig-cli] Kubectl client Simple pod should return command exit codes Since when has it been failing: 02/05 Testgrid link: https://k8s- Reason for failure: Anything else we need to know: /sig cli
/priority critical-urgent /assign
The test seems to be flaking quite a lot recently ! Friendly ping, sig-cli, could you take a look?
FTR, these are really blocking tests and it's getting more urgent given the forthcoming 1.18 release
Full log of the failure: It looks like the finishes with error. Strange, I thought is supposed to create a pod. Would really appreciate if someone from sig-cli could take a look
4 out of last 8 runs failed: https://k8s- /sig scalability
Another flaky, related test:
SIG CLI Leads -- any update on this? This is affecting scalability signal. /assign
-- Please keep tabs on this one.
yep, monitoring
It looks like there's a race between when the pod gets created and when we try to connect to it, thus we're seeing but further down in the logs the pod is , I'll be looking more into ti.
Thanks, Any update on this?
/cc
Sorry I got sidetracked yesterday and I'm back investigating this just now...
Probable fix in
Will continue to monitor here and reopen if needed :+1:
Unfortunately - it seems that the problem still exists - the test failed today (an hour ago): :( ^^
I'll check it, thx Wojtek for pointing that out.
- I'm not familiar with kubectl code, but I took a quick look and what I found is that this waitForPod() thing is happening only for pods with RestartPolicyNever: I must be missing something here, but for the case that failed it's not really the case, which means we don't wait for it?
This particular one is using , see the actual invocation that failed: That's why I started with this approach.
Sorry - the one for which the bug was originally opened.

But the today`s failure is actually slightly different (the same test, different case - sorry): The restart policy for this one is
Gotcha, so it looks like we progressed, I'll check the possible paths for this one, something tells me we're missing another timeout somewhere :wink:
OK - I think I know what is going on here. Here is what is happening (for the test case from two comments above): the test is setting "-i", so interactive mode is switched on that option triggers that Attach option defaults to true: over the Run() command, we get to the handleAttachPod: handleAttachPod is calling waitForPod And waitForPod is broken. The problem is the preconditionFunc: which is below passed to the UntilWithSync function UntilWithSync underneath is creating an informer and calling that function against its state: But the informer is initialized from apiserver watchcache, so it may be lagging. In large clusters we are observing lags in the order of small hundreds of milliseconds and that seem to be enough. What is happening is that: informer is initialized (but the pod creation is not yet reflected in it) the preconditionFunc is called the state doesn't contain that pod (yet) the whole thing fails with that error I actually believe, that just removing the preconditionFunc should actually solve the problem. I opened for that.
^^ - please take a look at the PR that I assigned to you (it's literally one-liner as of now - if you want to add tests or anything like that, feel free to pick it up).
Actually I went ahead, and fixed couple other commands too - so it's no longer one-liner. But still short one :)


Question: defer cancel() // Invalidate the cache right after updating <del> if err = ss.deleteCacheForNode(vmName); err != nil { return err } </del> <ins> defer ss.deleteCacheForNode(vmName) </ins> klog.V(2).Infof("azureDisk - update(%s): vm(%s) - attach disk(%s, %s) with DiskEncryptionSetID(%s)", nodeResourceGroup, nodeName, diskName, diskURI, diskEncryptionSetID) _, err = ss.VirtualMachineScaleSetVMsClient.Update(ctx, nodeResourceGroup, ssName, instanceID, newVM, "attach_disk")

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: vmss dirty cache issue, caused by What you expected to happen: clean vmss cache should happen after disk attach/detach operation, now it's before those operations, which would lead to dirty cache. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /kind bug /assign /priority important-soon /sig cloud-provider /area provider/azure


Question: "//pkg/scheduler/framework/plugins/nodepreferavoidpods:all-srcs", "//pkg/scheduler/framework/plugins/noderesources:all-srcs", "//pkg/scheduler/framework/plugins/nodevolumelimits:all-srcs", <ins> "//pkg/scheduler/framework/plugins/podtopologyspread:all-srcs", </ins> "//pkg/scheduler/framework/plugins/tainttoleration:all-srcs", "//pkg/scheduler/framework/plugins/volumebinding:all-srcs", "//pkg/scheduler/framework/plugins/volumerestrictions:all-srcs",

What would you like to be : Add a filter plugin that calls into EvenPodsSpread predicate, example Why is this needed: Part of /sig scheduling /help /priority important-soon
/assign
/remove-help
/retitle: [migration phase 1] EvenPodsSpread as filter plugin
/retitle [migration phase 1] EvenPodsSpread as filter plugin
/assign /unassign
May I take over this?
is going to take this over after the offline discussion, and he is familiar with EvenPodsSpread
IC, thanks a lot.


Question: } versionPath := pkgPath(group, version) <del> generator := pkg_runtime.NewConversionGenerator(api.Scheme.Raw(), versionPath) </del> <ins> generator := kruntime.NewConversionGenerator(api.Scheme.Raw(), versionPath) </ins> apiShort := generator.AddImport(path.Join(pkgBase, "api")) generator.AddImport(path.Join(pkgBase, "api/resource")) // TODO(wojtek-t): Change the overwrites to a flag.

Forked from pointed out that all package names should be lower case, no under_scores or mixedCaps. The same rule applies to renaming imported packages. () generated a list of mixedCaps package names in our codebase: I will upload a PR later to rename kubeletTypes to kubetypes, and the rest is up for grabs.
sorry i have not saw this issue as i open a similar one(), do you mind i participate in this issue? i would happy to do some work for this. best regards.
of course, fixes are welcome. Thanks for picking up on this! We'll try to expedite the review and merge process of such PRs to avoid a lot of churns, but chances are that some rebases are needed :)
ok, i will send some fix sonner, and tag you for review. thanks again.
Not done:


Question: // In general, please try to avoid adding flags or configuration fields, // we already have a confusingly large amount of them. type KubeletFlags struct { <del> KubeConfig          flag.StringFlag </del> <ins> KubeConfig          string </ins> BootstrapKubeconfig string RotateCertificates  bool

This will be the end of the deprecation period for --require-kubeconfig.
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list .
ref
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale


Question: resizeOptions.DeviceStagePath = deviceMountPath } <ins> if volumeDeviceMounter != nil && resizeOptions.DeviceStagePath == "" { deviceStagePath, err := volumeDeviceMounter.GetDeviceMountPath(volumeToMount.VolumeSpec) if err != nil { // On failure, return error. Caller will log and retry. eventErr, detailedErr := volumeToMount.GenerateError("MountVolume.GetDeviceMountPath failed for expansion", err) return volumetypes.NewOperationContext(eventErr, detailedErr, migrated) } resizeOptions.DeviceStagePath = deviceStagePath } </ins> // No mapping is needed for hostUID/hostGID if userns is not used. // Therefore, just assign the container users to host UID/GID. hostUID := util.FsUserFrom(volumeToMount.Pod)

When a pod is quickly deleted and re-created on a node, it may not be necessary to unstage the volume and in which case only node-publish is called for the volume. When this happens - stagepath won't be set for RPC call because volume was not staged even though, volume does have capability. We should try and fix this behaviour so that - nodestagepath is always set in RPC call. We should try and fix this behaviour so that - nodestagepath is always set in RPC call. Quickly delete and recreate a pod No response master Any No response_ <details</details<details</details<details</details
/sig storage /triage accepted


Question: return "unknown" } <ins> // lazyAPIGroup implements String() string and it will // lazily get Group from request info. type lazyAPIGroup struct { req *http.Request } func (lazy *lazyAPIGroup) String() string { if lazy.req != nil { ctx := lazy.req.Context() requestInfo, ok := apirequest.RequestInfoFrom(ctx) if ok { return requestInfo.APIGroup } } return "unknown" } // lazyAPIVersion implements String() string and it will // lazily get Group from request info. type lazyAPIVersion struct { req *http.Request } func (lazy *lazyAPIVersion) String() string { if lazy.req != nil { ctx := lazy.req.Context() requestInfo, ok := apirequest.RequestInfoFrom(ctx) if ok { return requestInfo.APIVersion } } return "unknown" } // lazyName implements String() string and it will // lazily get Group from request info. type lazyName struct { req *http.Request } func (lazy *lazyName) String() string { if lazy.req != nil { ctx := lazy.req.Context() requestInfo, ok := apirequest.RequestInfoFrom(ctx) if ok { return requestInfo.Name } } return "unknown" } // lazySubresource implements String() string and it will // lazily get Group from request info. type lazySubresource struct { req *http.Request } func (lazy *lazySubresource) String() string { if lazy.req != nil { ctx := lazy.req.Context() requestInfo, ok := apirequest.RequestInfoFrom(ctx) if ok { return requestInfo.Subresource } } return "unknown" } // lazyNamespace implements String() string and it will // lazily get Group from request info. type lazyNamespace struct { req *http.Request } func (lazy *lazyNamespace) String() string { if lazy.req != nil { ctx := lazy.req.Context() requestInfo, ok := apirequest.RequestInfoFrom(ctx) if ok { return requestInfo.Namespace } } return "unknown" } </ins> // lazyAuditID implements Stringer interface to lazily retrieve // the audit ID associated with the request. type lazyAuditID struct {

Feedback originally provided on slack: It would be helpful to have additional fields on APIServer spans. This is the current set of attributes we provide: It would be helpful to add: group, version, namespace, name, and subresource, if possible. Part of
/sig instrumentation /sig apimachinery /priority important-longterm cc /triage accepted
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /sig instrumentation /sig apimachinery /priority important-longterm cc /triage accepted Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig api-machinery
/assign


Question: # then after merge and successful postsubmit image push / promotion, bump this - name: "agnhost: dependents" <del> version: "2.29" </del> <ins> version: "2.30" </ins> refPaths: - path: test/utils/image/manifest.go match: configs[Agnhost] = Config{promoterE2eRegistry, "agnhost", "d+.d+"}

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --aks-engine-azure-windows-master-staging ServiceAccounts ServiceAccountIssuerDiscovery should support OIDC discovery of service account issuer [Conformance] /sig windows /cc
cc
/assign
/assign Let me know if you want to pair on this
We took a look at this for docker case and we believe it is happening due to Windows DNS networking limitations. DNS can be available sometime after the container starts due to the way networking is set up with docker. In containerd the setup steps are different so the network will be available when the container starts. In this case the test image does a lookup before the DNS is finished coming online and fails.
it fails in linux as well
From the link posted, it looks like the test cluster issuer might be misconfigured (--service-account-issuer is missing https, I'm not sure why this even passed kube-apiserver validation, double-checking that also): Try setting on kube-apiserver.
reminded me we couldn't actually add that validation, because we weren't previously validating schemes and it would've broken existing clusters with bad issuers set... So as far as fixing the tests make sure issuers have schemes, at least.
I guess the config in this case is managed behind AKS service, and AKS has been setting without https scheme?
Thanks! The failures on the Linux tests are a separate issue from the Windows failures on docker. I opened kubernetes-sigs/cloud-provider-azure to resolve the Linux failures (we have already done this for Windows: )
/reopen This is still an issue for windows docker. fixes it
Reopened this issue. <detailsIn response to : /reopen This is still an issue for windows docker. fixes it Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I'm glad this was fixed for open-source test configs, but I assume since the test said "aks" in the name this could be an issue for managed Azure clusters?

Clients should reject issuers without https instead of attempting to validate them, so we'd want all clusters to be using https scheme in the issuer URL, but at the same time, changing the issuer URL on an existing cluster invalidates previously-issued tokens, because kube-apiserver would no longer recognize the old issuer. I just want to confirm that what was happening with tests isn't foreshadowing a bigger issue for existing Azure clusters. can you/someone at Microsoft confirm? FYI
We use aks-engine (self-managed cluster instead of managed AKS cluster) to run the tests. For aks-engine, I opened a PR () to fix the service account issuer. For AKS, I can confirm that the service account issuer does not any scheme in the URL at all. Tagging who works on aks-engine. Tagging who works on AKS.
We confirmed that is being fixed in AKS.
/traige accepted /milestone v1.21 /priority important-soon
/triage accepted
Our ipv6 test is still failing: The service issuer has been set to .
that seems to be a different issue than Windows. could you open a new issue and tag and me?


Question: if [[ "${KUBE_PROMPT_FOR_UPDATE}" == "y" ]]; then echo "Can't find gcloud in PATH.  Do you wish to install the Google Cloud SDK? [Y/n]" read resp <del> else resp="y" </del> fi if [[ "${resp}" != "n" && "${resp}" != "N" ]]; then curl https://sdk.cloud.google.com | bash

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): Is this a BUG REPORT or FEATURE REQUEST? (choose one): FEATURE REQUEST <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): all :smile: Environment: Cloud provider or hardware configuration: GCE OS (e.g. from /etc/os-release): all Kernel (e.g. ): all Install tools: Others: What happened: uses the cli a lot and does a (in the commonly used function ) before doing things like . updates can introduce breaking changes (for example in the recent 132.0.0 release it removed a bunch of flags). This has caused cross-the-board test failures like . But this is not limited to just tests. Any kubernetes GCE users who rely on the and commands could have been broken. What you expected to happen: Stop updating gcloud automatically. We could either warn user to have the required dependencies installed, or pin to specific versions of dependencies like gcloud. How to reproduce it (as minimally and precisely as possible): download 1.4.4 release tarball run and Anything else do we need to know:
cc
/cc
cc I think it's reasonable to remove gcloud update calls from scripts, as I believe gcloud is updated out of band by k8s continuous testing. The scripts will still need to work with latest, so maybe they should check for a minimum version.
Good idea.
Yeah, we definitely want to continue using latest in CI so that we know if things break, but I think it's reasonable to remove the update itself from the kubernetes cluster util scripts.


Question: ginkgo.It("should verify ResourceQuota's priority class scope (quota set to pod count: 1) against 2 pods with same priority class.", func() { _, err := f.ClientSet.SchedulingV1().PriorityClasses().Create(context.TODO(), &schedulingv1.PriorityClass{ObjectMeta: metav1.ObjectMeta{Name: "pclass2"}, Value: int32(1000)}, metav1.CreateOptions{}) <del> framework.ExpectEqual(err == nil || apierrors.IsAlreadyExists(err), true) </del> <ins> if err != nil && !apierrors.IsAlreadyExists(err) { framework.Failf("unexpected error while creating priority class: %v", err) } </ins> hard := v1.ResourceList{} hard[v1.ResourcePods] = resource.MustParse("1")

The tests are full of assertions which compare a boolean against true or false, often without any additional explanation. When those assertions fail, the error message is useless for understanding what went wrong, often just saying: Including an explanation is better, but then we still have the distracting noise about the comparison: All of those assertions should be replace with . Such locations can be found by grepping: UPDATE (2024-08-26): in cases where such an if check is not possible (typically in combination with or ), should be used instead of . They emit only the explanation given to them, without including the noise above. The Ginkgo linter (enabled in golangci-lint) points that out. Some progress has been made in the meantime: v1.23.0: 250 v1.24.0-beta.0: 177 However, we were down to 175 at some point, so the number is going back up. Example: Some other places compare values. Those should be replaced with a comparison assertion. Example: It makes sense to split this work into individual PRs, one per OWNERS file under test/e2e. Here's a list of directories: [ ] test/e2e/apimachinery [x] test/e2e/apps: [x] test/e2e/auth [x] test/e2e/autoscaling: [x] test/e2e/cloud/gcp [x] test/e2e/cloud [x] test/e2e/common/network [x] test/e2e/common/node: [x] test/e2e/common/storage: [x] test/e2e/framework: [x] test/e2e/framework/providers/vsphere [x] test/e2e/framework/volume [x] test/e2e/instrumentation/logging [x] test/e2e/instrumentation: [x] test/e2e/kubectl [x] test/e2e/lifecycle [x] test/e2e/node: [x] test/e2e/scheduling: [x] test/e2e/storage: (prior PR: ) [x] test/e2e/testing-manifests/storage-csi [x] test/e2e/upgrades/apps: [x] test/e2e/windows: , When creating a PR, use as title. Use one commit per PR, squash when addressing feedback.

Unless noted otherwise, will work on PRs. Help may be welcome. /sig testing /help Better output for developers when tests fail.
This request has been marked as needing help from a contributor. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : ### What would you like to be ? Example: Some other places compare values. Those should be replaced with a comparison assertion. Example: /sig testing /help </details
/good-first-issue
This request has been marked as suitable for new contributors. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /good-first-issue </details
/triage accepted
I would like to take this up as my first issue
/assign
I would like to take up this issue as my first issue
Hello I am already working on this issue , can you please take some other issue ?
Okay sure ,how do I un assign myself from this issue?
I don't think you are assigned to this issue.
Oh okay
Hey as discussed in pr I am assuming that you are only working for tests under area.

also finished the work on , see
and only one PR is to be made for the correction under test/e2e/storage/, so how can we distribute the task and also make only single PR. Or can we make multiple PR's
I have a TODO list to the issue description. I'll try to keep that up-to-date with what has been done and who is working on it.
you can continue with in
what is to be changed in this case !
can you inform on which directory you are working.
Currently working on apimachinery
Nothing, just leave it as-is.
But that isn't under - before starting somewhere else, please coordinate with
test/e2e/instrumentation/logging don't have any cases like "Expect.Equal([^,], *(true|false)", so I think it should be removed from tasks to do.
I am thinking that the output should be if !ok { framework.Failf(uexec.ExitError) } in this case, am I thinking reasonable here.
isn't valid Go. The first parameter must be a format string. The code checks whether err is a : Therefore the change should be:
Why don't we just add extra explanation into calls like for easy investigation? I might miss some point.
You have to consider the complete output when that case fails. Gomega will write print the actual value and the expected value, then in addition you get the same information in the custom message. Printing just the custom failure message is shorter and more readable, in particular for a comparison of booleans. For a comparison of integers I see two solutions: ExpectEqual with a fixed message that just explains what is being compared. An if check with Failf where the message includes the actual value. The later seems more useful when the expected value is zero because that value doesn't need to be printed when its obvious.
I am getting difficulty in figuring out what should be output here, what should the appropriate output here. Also what steps you took in knowing so, it would help to improve my understanding.
The "ok" value is the result of checking for a specific key in a map. What a developer therefore wants to know when that check fails is: what was the key what's in the map Both can be included in:
can I work for the section :- test/e2e/scheduling ?

And there was only one changes needed for test/e2e/node which I did but my PR was closed as it only contains trivial edits. That was my first PR, so can you please guide me further?
/assign I'll try to coordinate the work, but won't create PRs myself. I ticked of line items which had no remaining work. feel free to start working on .
I replied there and also suggested an improvement. For those following this issue: try to include more information than before in the error message when the assertion fails. Then it is more obvious to the reviewer that the change is useful without having to read this issue. Examples:
Another tip: don't just reference this issue as explanation for the PR. Reviewers are busy, they won't necessarily invest the time and follow the link to understand what the PR is about. Instead, take the first sentence from this issue and adapt it as needed to the specific changes in the code.
can I start working on test/e2e/common/storage ?
Okay.
for test/e2e/network/netpol , there are already message associated with all of the cases, for example :- So should I just change all of them to ? And same for all the cases in test/e2e/network except these two cases : framework.ExpectEqual(netutils.I sIPv4String(internalIPs[0]) != netutils.IsIPv4String(internalIPs[1]), true) framework.ExpectEqual(netutils.IsIPv4String(p.Status.PodIPs[0].IP) != netutils.IsIPv4String(p.Status.PodIPs[1].IP), true )
Yes.
Okay then I will be working on and
It might make sense to combine them in a single PR, depending on how many changes there are - just use common sense.
IMO : this will never fail because whether err will be nil or it will exist. So, we don't need to change this code. What I had done in this case was :
Note that that is redundant (= not wrong, just not needed). I later clarified that in
/reopen
Reopened this issue. <detailsIn response to : /reopen </details
Hi! I've created a PR to address the tests in e2e/apps It is my first PR so hopefully, everything is ok. Thanks a lot!
Hi can I take test/e2e/common/node?
Yes, please do.

Have we considered / helpers that can skip the comparison output? - is noisy and makes it harder to skim tests, especially given that it's a pretty commonly required pattern.
No, we haven't. Do you find it noisy because it's two lines instead of one? I don't have a problem with that. It seems to be a common approach in unit tests and I find it easy to read, partly because it matches how production code would be written.
It becomes three, plus extra indention - but more importantly stands out in a series of expectations. This draws extra scrutiny to those lines. many unit tests using testify would use to express this because of that.
Isn't that a good thing? is bad without any or no useful extra explanation. The if check makes it obvious that a good failure message is needed. Does print less useless boilerplate code than Gomega? We could use that with Ginkgo (), but then we still need to ensure that the optional explanation is useful.
xref
Hi can I take test/e2e/auth?
for test/e2e/auth, for example : Should I just change it to ?
feel free to work on test/e2e/auth. This one particular example is a bit different: here we are comparing string values, and for those it makes sense to print the actual value because the failure message then adds information. I would leave that one as it is.
OK. test/e2e/auth doesn't need replacing. I : <img width="731" alt="Screen Shot 2022-05-30 at 9 12 59 PM" src="https://user-"There are a lot of positions that need to be modified in . I'll take test/e2e/apps.
That looks like a good candidate. Please go for it. When you need to make up new failure messages, try to think about what information may be useful to have and add that. Then the resulting PR is clearly better than the status quo.
Hi can you please assign me a directory. I want to give it a try. Thanks,
Hi I already posted a PR for e2e/apps some days ago in this issue: If it is fine I'd love to proceed with that. Just to avoid us repeating work.
sorry for not noticing the overlap sooner. Let's go with the older PR.

you can take over test/e2e/storage from There is an incomplete PR for it in You can check out that branch, fix it and submit a new PR, or start from scratch.
it looks like is not active anymore. I've remove him from the todo list and you instead for test/e2e/common/storage.
OK. Thank you. Is there an incomplete PR for .
I in and it returns 0. What should I do?
needs modifying. <img width="1440" alt="Screen Shot 2022-05-31 at 2 40 29 PM" src="https://user-"Can I work on it?
Indeed, did test/e2e/common/storage in You can work on test/e2e/windows.
OK. I'm working on it currently.
Sure, I will look into it. Thanks,
: do you want to continue with test/e2e/apimachinery?
Yes. There's an older PR:
Hi is there anything I could take over?
Hey I am not sure how updated the list is, but I am available to help. Is there any part I could start or continue?
sorry for the late reply. I had (and still have) some big changes for the E2E tests going and had put this on the backseat for a while. If you are interested, how about doing test/e2e/apimachinery and test/e2e/autoscaling? Some work was done on the latter, but there are still several calls (again?).
Sure, I can take that!
Hey I'd also like to contribute on this issue. Is there anything I could take over?
feel free to take my part:
Perfect, then I will take part
Hi :smile: , I would like to contribute to this issue, there is anything that I could do?
I was reluctant to start new work on this because I had one big PR pending that touched a lot of test code (PR ), but that has been merged. you are taking test/e2e/autoscaling, right? test/e2e/apimachinery? you can do test/e2e/apps. That leave test/e2e_node in case someone wants to continue.
yes e2e/autoscaling. I had some trouble to get the tests going initially, but seems fine now!
Hi, Can I do something to contribute to this issue?

you could work on test/e2e_node.
Ok, I will start work on test/e2e_node thanks!
Hi , I would like to contribute. Is there anything I could take over?
thanks for your interest, but right now everything seems to be covered.
Hi Thanks. I am searching for "good-first-issue", feel free to assign one to me. I am exploring api-machinery, but happy to contribute in other sig as well.
maybe you can take a look at if you have some time :)
Hi , There is anything that I could do to contribute with this issue?
Hi , I was looking and found the test/e2e/apimachinery without doing it and only then did I see that it was already assigned if this is a problem just refuse my pr and if you have something else to do let me know.
The remaining work will probably finished in , so there isn't anything else to do. If someone wants to investigate writing a linter for this, then this would be a cool science project - not easy, though. I would start with looking at for this. It has the advantage that it is supported indirectly in golangci-lint ().
Hello I would like to try to do a liter for it. It will be a cool project whit a lot of opportunities to learn. There will be a separate issue for the linter or we continue the discussion here?
Let's continue here.
I ended up creating a separate issue for linter ideas: let's continue there.
Ok!
New tests have been again with this anti-pattern since the last time that I checked. It's clear that without a linter, we'll not get rid of it. So let's focus on that first.
is this issue still open
Yes, this is still open. Current status as of v1.28.0-rc.0: Two calls are in , the rest are in . We have such a linter now, with ginkgolinter in golangci-lint and (soon) the ability to prevent usage of , so it makes sense to continue with removing those calls. do you want to prepare one PR for all remaining calls? I think one PR is enough, no need to split it up.
also has it, which was intentional. I'm undecided whether that should be replaced.

Several of those calls are being touched in , let's wait for that PR to be merged before doing any further work.
no longer finds anything. We might be done... :tada: But not quite. What we have left now is checking a boolean via or , which has the same problem. In some cases, there is an extra message (which is still noisy, see ), in other cases there isn't even that (which makes the failure message fairly useless). We can and should replace some of those calls with the pattern.
is almost resolved. My work-in-progress branch forbids the usage of in favor of the new or plain Go. In some cases, moving the Go code that returns true/false into a matcher might be the best solution because that allows dumping the full actual value that was being checked. Here are the remaining issues: <details</detailsDoes anybody want to work on those?
I would love to!
: please go ahead. Assign PRs to me when you create them. Let's start with a few of those calls initially, do a round of reviews, then continue. /assign
You will have to start with my branch, current master doesn't have gomega with thew new API yet.
I'm starting new to kubernetes. Is there anything easy that I could do ?
Hey Is there anything left I can help you with in this issue ?
Thanks for your interest. There's not much to be done at the moment, though.
Should this be closed?
Oh wow! Looks like we are indeed done. Thanks everyone. /close
Closing this issue. <detailsIn response to : Oh wow! Looks like we are indeed done. Thanks everyone. </details