Question: "experimentalsresourcesusagestracking" # Expect --max-pods=100 ) <ins> # Tests which kills or restarts components and/or nodes. DISRUPTIVE_TESTS=( "DaemonRestart" "Etcdsfailure" "NodessResize" "Reboot" "Services.*restarting" ) </ins> # The following tests are known to be flaky, and are thus run only in their own # -flaky- build variants. GCE_FLAKY_TESTS=(

We should be testing the reliability of a long-running cluster. These tests kill/restart components or nodes in the cluster, defeating the purpose of a soak cluster.
Example tests: NodessResize Services.restarting DaemonRestart NodessResize Reboot Etcd failure
Agreed. Some of them already seem to be skipped: GCESOAKCONTINUOUSSKIPTESTS=( "Density.30spods" "Elasticsearch" "Etcd.SIGKILL" "externalsloadsbalancer" "identicallysnamedsservices" "networkspartition" "Services.Typesgoessfrom" ) Q
FYI
It'd be hard to prevent regression as more tests are . It might be better if we explicitly define a set of DISRUPTIVETESTS, and include them wherever needed, e.g., GCEPARALLELSKIPTESTS and GCESOAKCONTINUOUSSKIPTESTS. That way, people who are adding tests will be aware of the category. WDYT?
Agreed, that seems like a good idea. Want to send in a PR? Or at least open an issue?


Question: err = wait.PollImmediate(podTemplateRetryPeriod, podTemplateRetryTimeout, checkPodTemplateListQuantity(f, "podtemplate-set=true", 0)) framework.ExpectNoError(err, "failed to count required pod templates") <ins> }) ginkgo.It("should replace a pod template", func() { ptClient := f.ClientSet.CoreV1().PodTemplates(f.Namespace.Name) ptName := "podtemplate-" + utilrand.String(5) ginkgo.By("Create a pod template") ptResource, err := ptClient.Create(context.TODO(), &v1.PodTemplate{ ObjectMeta: metav1.ObjectMeta{ Name: ptName, }, Template: v1.PodTemplateSpec{ Spec: v1.PodSpec{ Containers: []v1.Container{ {Name: "e2e-test", Image: imageutils.GetE2EImage(imageutils.Agnhost)}, }, }, }, }, metav1.CreateOptions{}) framework.ExpectNoError(err, "failed to create pod template") ginkgo.By("Replace a pod template") var updatedPT *v1.PodTemplate err = retry.RetryOnConflict(retry.DefaultRetry, func() error { ptResource, err = ptClient.Get(context.TODO(), ptName, metav1.GetOptions{}) framework.ExpectNoError(err, "Unable to get pod template %s", ptName) ptResource.Annotations = map[string]string{ "updated": "true", } updatedPT, err = ptClient.Update(context.TODO(), ptResource, metav1.UpdateOptions{}) return err }) framework.ExpectNoError(err) framework.ExpectEqual(updatedPT.Annotations["updated"], "true", "updated object should have the applied annotation") framework.Logf("Found updated podtemplate annotation: %#vn", updatedPT.Annotations["updated"]) </ins> }) })

[x] test approval issue : [x] test pr : [x] two weeks soak end date : 14 March 2022 [X] test promotion pr : According to this APIsnoop query, there is still one remaining PodTemplate endpoint which is untested. Using a number of existing e2e test practices a new has been created for pod templates. The e2e logs for this test are listed below. This query shows the endpoints hit within a short period of running the e2e test If a test with these calls gets merged, test coverage will go up by 1 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: } volume := Volume{ <del> ID:     volumeV1.ID, Name:   volumeV1.Name, Status: volumeV1.Status, Size:   volumeV1.Size, </del> <ins> AvailabilityZone: volumeV1.AvailabilityZone, ID:               volumeV1.ID, Name:             volumeV1.Name, Status:           volumeV1.Status, Size:             volumeV1.Size, </ins> } if len(volumeV1.Attachments) > 0 && volumeV1.Attachments[0]["server_id"] != nil {

Resizing a persistent volume is partially broken on openstack (cinder) with k8s v1.11.1. Partially in the sense that static resizing (requiring a pod restart) works, but the newly introduced online resizing (not requiring a pod restart) fails. Upon resizing a pvc it remains in the Resizing condition forever (see the warning event) until I delete the pod that has this pvc mounted: After pod deletion it notifies me (correctly) that it's now waiting for the pod to come up to finish resizing: So far the warning event does not break the ExpandPersistentVolumes feature, as this one anyhow requires a pod restart. However it breaks the newly introduced ExpandInUsePersistentVolumes feature (alpha with k8s v1.11; doesn't require a pod restart for resizing to finish), as the pvc will remain in "Resizing" condition forever (or until the pod is restarted but this defeats the purpose of ExpandInUsePersistentVolumes). I'm not sure whether this is fully related but upon digging around I found and a consecutive fix with which was merged on Jan 23. However a few days later this fix seems to be (probably accidentally) overwritten with this commit: /kind bug /sig storage /sig openstack
/assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
It is already fixed. So I will close it.
/close
Closing this issue. <detailsIn response to : /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I am trying cinder online pvc resizing on kubernetes1.12.2. I noticed this kubernetes version has include your fix, but I still failed to resize pvc online.

The error log has no change: Warning VolumeResizeFailed 45s (x2 over 66s) volumeexpand Error expanding volume "default/myclaim-expend" of plugin : volume status is not available The error log is reported below code line "volume.Status != volumeAvailableStatus" // ExpandVolume expands the size of specific cinder volume (in GiB) func (os *OpenStack) ExpandVolume(volumeID string, oldSize resource.Quantity, newSize resource.Quantity) (resource.Quantity, error) { volume, err := os.getVolume(volumeID) if err != nil { return oldSize, err } if volume.Status != volumeAvailableStatus { // cinder volume can not be expanded if its status is not available return oldSize, fmt.Errorf("volume status is not available") } One point confused me, once cinder volume is used by pod, it's status is "in-use", how the online pvc resizing pass the code check "volume.Status != volumeAvailableStatus"? There's where the online pvc resizing fails.
It looks like Cinder can not support online volume expansion - which is what is happening here. Even if online expansion is enabled, I think to actually perform resizing of Cinder volumes - you must bring the pod down.


Question: DOCKER_OPTS="${DOCKER_OPTS} {{grains.docker_opts}}" {% endif %} DOCKER_OPTS="${DOCKER_OPTS} --bridge cbr0 --iptables=false --ip-masq=false -r=false" <ins> DOCKER_NOFILE=1000000 </ins>

I have a cluster with ~130 pods. Running back-to-back 'get pods' shows me wildly different understandings of their state, even though I can tell from docker that they have not crashed or anything. thockin kubernetes master /(NF)}' uniq -c Running: ./cluster/../cluster/gce/../../output/dockerized/bin/linux/amd64/kubectl get pods -l app=hostnames 127 Running 1 STATUS 6 Unknown thockin kubernetes master /(NF)}' uniq -c Running: ./cluster/../cluster/gce/../../output/dockerized/bin/linux/amd64/kubectl get pods -l app=hostnames 82 Running 1 STATUS 51 Unknown Nothing changed in between these calls.
cc I blame the pod cache.
has recently changed pod cache? FWIW, I also observed this behavior: 19 pods Running in one call and all Unknown in the next. With 0.10.0.
Yes, I think PR on pod cache should fix the problem reported here. Also it should fix the issue reported by about adds-on services. I have a pending PR to have kubelet deciding on PodStatus too.
Could this have caused ?
I don't see how. I'll comment on that issue.
I don't think this could be the cause of that one.
I don't think PR on pod cache will fix this. That one clears the node status. This issue is with pod going from Running to Unknown even when nodes keep running just fine.
Totally consistent observation based on my personal experience. I just assumed we were shooting for some kind of Hesinberg Uncertainty Principle. S , Tim Hockin wrote:
Agree with clears node status only. One possibility is that we hit transient network problem in between the calls that fails health check, so controller set node condition to not reachable, hence unknown pods. I'll make this more robust anyway. Other than that, it's hard to think of a reason that node status results in the flakiness.
In the absence of node information, the pod status defaults to Unknown. Maybe there's another source of this specific problem, but the pod cache is a source of continuous problems, partly because it's not just a cache, but is implementing or duplicating a significant chunk of system logic in a half-baked manner. My goal is to eliminate it -- see .

Short of that, we need to at least implement , eliminate nodes from the pod cache, and implement a real node cache.
I can look into this. Repro info from Tim, who said he was running on a one-node cluster but has seen the problem with multi-node clusters also: run-container hostnames --image=kubernetes/servehostname --replicas=1 --port=9376 --labels=app=hostnames expose hostnames --port=80 --container-port=9376 --selector="app=hostnames" resize --replicas=150 rc hostnames but I had to hack the node a bit to make it get past 100 on the node: sysctl -w fs.inotify.maxuserinstances=8192 echo "DOCKERNOFILE=" /etc/default/docker; /etc/init.d/docker restart (fails sometimes, dunno why)
I can verify that the situation is rather bad. Lots of fluctuation in the numbers of Pending, Running, and Unknown. serve_hostname containers on the node are exiting; I'm not sure why, or if the pod itself is terminating or just the container. Lots of the following in the log (not sure what this means yet): Sat, 07 Feb 2015 00:06:22 +0000 hostnames-ecd2r Pod failedScheduling {scheduler } Binding rejected: 501: All the given peers are not reachable (Tried to connect to each peer twice and failed) [0]
The pause container is also exiting, so I'm guessing that means the whole pod is exiting, not just serve_hostname...
Ugh, there is an etcd size limit you might be hitting too, I forgot. If you are seeing the fluctuations, maybe you got far enough, though. On Feb 6, 2015 5:13 PM, "David Oppenheimer" wrote:
Just a few more observations: This is reproducible (and a little less chaotic) with one node and 100 pods (but not with 50 pods, so it's definitely load-dependent). You can get to 100 Running pods (0 Pending or Unknown), and then a minute later a dozen or so will become Unknown, and then later go back to Running, etc. As mentioned earlier, the containers continue running on the node all this time (sudo docker ps wc -l), so it is definitely some kind of reporting problem.

As an aside, it looks like there is some kind of race condition with IP address allocation, as during startup (but not at steady state) a number of pods exit with the following in their log (sudo docker logs xxx): listen tcp :9376: bind: address already in use They do restart OK (I'm not sure if this is being restarted locally on the node or making a round-trip back to the scheduler).
Almost certainly the restart is just kubelet retrying after some time. On Feb 8, 2015 2:57 PM, "David Oppenheimer" wrote:
I found that if I comment out the line with the arrow next to it (see below) in , the Pods stop oscillating between Running and Unknown. Commenting out this line changes the logic from "if get an error from Kubelet, change pod status to Unknown" to "if get an error from Kubelet, keep pod status unchanged." I saw in the master logs intermittent errors of this form, which was causing the pods to flip back to Unknown: E0209 04:19:17. 4996 ] Error getting info for pod default/hostnames-v5drx: kubelet "kubernetes-minion-1.c.striped-torus-760.internal" server responded with HTTP error code 500 for pod default/hostnames-v5drx This seems to correspond to lines like this in the kubelet logs: I0209 06:38:11. 4704 ] GET /stats/default/hostnames-gfw5j/-b026-11e4-a070-/hostnames?num_stats=33: (218.857218ms) 500 goroutine 730 [running]: [ ... stack trace, but they're not always the same; I'll dig into this some more, the default stack trace depth wasn't going back past the various functions that just serve up the error ... ] However, pods still flip between Running and Pending, even though no containers are exiting on the minion, so there's another problem here too.
Thanks for investigating this! Pods really shouldn't flip between Running and Pending, as I believe that the only way that they go Pending is if Pod.Status.Host is empty, and this really shouldn't happen post scheduling, unless we are somehow dropping information. Maybe we need to make sure that we don't wipe out status on the error case?

--brendan , David Oppenheimer
I also investigated the same issue last Friday. I observed exact the same as what described above, except the related kubelet log should be: Meanwhile, I observed master is trying to query pod / and failed all the time: Talked to offline, he is going to look into why running pods go back to Pending; meanwhile, I am going to figure out Unknown cases here.
That would be great. It is very hard for me to make sense of any of the scalability experiments that I am doing while the pod status is in hysteresis.
Just to summarize: (1) the Running<-Unknown flapping is caused by the line of code I mentioned earlier; (2) the reason that line of code is being hit on the master is due to two errors on the kubelet (the one I identified earlier was wrong; Dawn found these): I0209 06:42:09. 4704 ] GET /api/v1beta1/podInfo?podID=hostnames-v5drx&podNamespace=default: (8.82543048s) 500 E0209 04:34:13. 4996 ] Error getting info for pod default/hostnames-2y8ny: kubelet "kubernetes-minion-1.c.striped-torus-760.internal" server responded with HTTP error code 500 for pod default/hostnames-2y8ny (3) we don't yet know the cause of the Running<-Pending flapping; I'm going to look into that now
(and Dawn is looking into why we're hitting those two errors on Kubelet)
Are we serving requests concurrently in Kubelet? (Is it possible not to when using go's servemux?) I don't see any locking around the pod map accesses.
This was marked for the fixit but Dawn and I are working on it. Assigned to myself.
I am also investing time writing code to try and report what is going on with unstable cluster status. It would be good to know what you and Dawn are doing in case I am wasting my time. Thanks. , David Oppenheimer
This particular problem is easily reproduced manually using the instructions Tim listed, so I'm not sure it's necessary to write anything sophisticated to drive the cluster into this state.

Anyway, maybe give me and Dawn a couple of days and if we can't figure this out by then we can call in the cavalry.
Am I wrong, or are these instructions also within bounds of an e2e test? Just because our existing e2es only do one or two things doesn't mean we can't do more within O(minutes), assuming you're dealing with the same image.
The "flips back to pending" problem appears to be because updatePodStatus() writes to the cache but not back to the Pod's status. computePodStatus() initializes newStatus based on pod.Status, not based on the cached value of status. This second behavior seems wrong in light of the first behavior. However, I believe this only causes a problem in combination with my fix described earlier, which as a result doesn't actually do what I described (it doesn't "keep the status as-is" but rather copies it out of the pod's status; to "keep the status as-is" it should be copying it out of the cache). I can see if this fixes things. (We also should fix the root cause stuff on the node side, which Dawn is working on.)
Assign the issue to me ... I didn't see the flips between Pending and Running anymore. Don't know why. I consistently see flips between Unknown and Running on those heavily loaded nodes. The root issue is once a while docker inspect API call returns us EAGAIN. Please note that on my test nodes, I already increase NOFILE limit for docker daemon process. The proposed solution is that kubelet deals with all errors internally except EAGAIN one. In this case, computePodStatus logic in pod_cache does nothing when p.containerInfo.GetPodStatus returns an error. Working on a PR to fix this.
Dawn, my suggestion is that computePodStatus() should return the current cached value whenever it gets an error from Kubelet. I can implement this if you want.
, why shouldn't updatePodStatus simply write through to the pod's status? I'd think either we always read the cached value first, or we always write through to the pod and read from there. Only reading the cached value upon an error seems counter-intuitive to me...
why do you computePodStatus returns a cached value here?

The only place calls computePodStatus is updatePodStatus, which triggered by two operations: 1) periodically update all containers 2) the client asks for PodStatus, and there is no cached one, instead calling computePodStatus to retrieve one. I think a cleaner way is 1) Kubelet handles all the errors except this transient one. Of course, we need to make sure docker is auto-restart and there is no too many open files error on node side too. But they are separate issues and should be handled by node side through script or configure etc. 2) Now back to master side, if node status is ok, kubelet should return something; if there is an error from kubelet, it should be transient; and if there is a cached one, simply return cached one upon request; if no cached one, it should be generated by computePodStatus in next round. I have one working PR to avoid the flips between Running and Unknown, but with increasing number of Pending from node side caused by docker failure. Still investigating it...
I didn't understand "if there is an error from kubelet, it should be transient; and if there is a cached one, simply return cached one upon request; if no cached one, it should be generated by computePodStatus in next round." If you're talking about the caching in the podCache on the master, then what you wrote is exactly what I was suggesting. (If you're talking about some kind of caching on the Kubelet, then that's different, because I am not aware of cache for status on the Kubelet.) I think always writing back to the pod's status might be fine but I don't know why the podCache was designed the way it was, so I didn't want to change that without asking Daniel first (I think he's the one who created it). I wasn't suggesting to only read the cached value upon an error; I meant always read the cached value under all circumstances.
and here is my understanding. Correct me if I am wrong. UpdateAllContainers periodically updates all containers, but it gets the initial pods from etcd through this line . each from , UpdateAllContainers calls passes to . computes and returns a new status. Note that the status is computed based on , i.e., the information in etcd. updates the pod cache with the new status.

Note that the new status is not written to etcd. If you repeat the same steps again, you will start with the stale pod status in the etcd because does not check the cached value.
I have tested since last night. The flaps between Running, Pending and Unknown are gone for about 10 hour running 300 pods on 4 nodes except two scenario: 1) etcd issue: cat kube- | grep "status to unknown" I0212 20:18:30. ] node doesn't exist: 501: All the given peers are not reachable (Tried to connect to each peer twice and failed) [0] <nil, setting pod hostnames-yuy8d status to unknown ... This happens a couple of times last 10 hours. We need to figure out why. 2) NodeStatus becomes UNREADY. It happens once last 10 hours, and pods status on that node become Unknown which is expected. I didn't figure out why NodeStatus was UNREADY then, but I heard there is a network earlier this morning.
up to you whether to reopen this issue or file new ones for new issues.
I've also seen the "All the given peers are not reachable" message (see my second entry in this issue). I think that was what was referring to when he said "there is an etcd size limit you might be hitting too" but I may be mis-interpreting.
That error is the etcd client's catch-all for "something went wrong" - it's actually really bad that it obscures errors. , David Oppenheimer
, Tim Hockin wrote:
and Yes, that error message is cache-all error message, which need to investigation. In my particular case, it shouldn't be etcd size limits at all since we have that cluster built and upgraded more than 10 hours before, and all 300 pods were scheduled back then. There is no new events generated either from kubelet or other components since all bound pods are indeed happily running on the nodes. No new writes to etcd except NodeStatus, which is very less frequent and not much data. The problem of etcd here occurs when master tries to read from etcd with an error.


Question: framework.ExpectNoError(err) }) <ins> ginkgo.It("should list, patch and delete a LimitRange by collection", func() { ns := f.Namespace.Name lrClient := f.ClientSet.CoreV1().LimitRanges(ns) lrName := "e2e-limitrange-" + utilrand.String(5) e2eLabelSelector := "e2e-test=" + lrName patchedLabelSelector := lrName + "=patched" min := getResourceList("50m", "100Mi", "100Gi") max := getResourceList("500m", "500Mi", "500Gi") defaultLimit := getResourceList("500m", "500Mi", "500Gi") defaultRequest := getResourceList("100m", "200Mi", "200Gi") maxLimitRequestRatio := v1.ResourceList{} limitRange := &v1.LimitRange{ ObjectMeta: metav1.ObjectMeta{ Name: lrName, Labels: map[string]string{ "e2e-test": lrName, lrName:     "created", }, }, Spec: v1.LimitRangeSpec{ Limits: []v1.LimitRangeItem{ { Type:                 v1.LimitTypeContainer, Min:                  min, Max:                  max, Default:              defaultLimit, DefaultRequest:       defaultRequest, MaxLimitRequestRatio: maxLimitRequestRatio, }, }, }, } // Create a copy to be used in a second namespace limitRange2 := &v1.LimitRange{} *limitRange2 = *limitRange ctx, cancelCtx := context.WithTimeout(context.Background(), wait.ForeverTestTimeout) defer cancelCtx() ginkgo.By(fmt.Sprintf("Creating LimitRange %q in namespace %q", lrName, f.Namespace.Name)) limitRange, err := lrClient.Create(ctx, limitRange, metav1.CreateOptions{}) framework.ExpectNoError(err, "Failed to create limitRange %q", lrName) ginkgo.By("Creating another limitRange in another namespace") lrNamespace, err := f.CreateNamespace(lrName, nil) framework.ExpectNoError(err, "failed creating Namespace") framework.Logf("Namespace %q created", lrNamespace.ObjectMeta.Name) framework.Logf(fmt.Sprintf("Creating LimitRange %q in namespace %q", lrName, lrNamespace.Name)) _, err = f.ClientSet.CoreV1().LimitRanges(lrNamespace.ObjectMeta.Name).Create(ctx, limitRange2, metav1.CreateOptions{}) framework.ExpectNoError(err, "Failed to create limitRange %q in %q namespace", lrName, lrNamespace.ObjectMeta.Name) // Listing across all namespaces to verify api endpoint: listCoreV1LimitRangeForAllNamespaces ginkgo.By(fmt.Sprintf("Listing all LimitRanges with label %q", e2eLabelSelector)) limitRangeList, err := f.ClientSet.CoreV1().LimitRanges("").List(ctx, metav1.ListOptions{LabelSelector: e2eLabelSelector}) framework.ExpectNoError(err, "Failed to list any limitRanges: %v", err) framework.ExpectEqual(len(limitRangeList.Items), 2, "Failed to find the correct limitRange count") framework.Logf("Found %d limitRanges", len(limitRangeList.Items)) ginkgo.By(fmt.Sprintf("Patching LimitRange %q in %q namespace", lrName, ns)) newMin := getResourceList("9m", "49Mi", "49Gi") limitRange.Spec.Limits[0].Min = newMin limitRangePayload, err := json.Marshal(v1.LimitRange{ ObjectMeta: metav1.ObjectMeta{ CreationTimestamp: limitRange.CreationTimestamp, Labels: map[string]string{ lrName: "patched", }, }, Spec: v1.LimitRangeSpec{ Limits: limitRange.Spec.Limits, }, }) framework.ExpectNoError(err, "Failed to marshal limitRange JSON") patchedLimitRange, err := lrClient.Patch(ctx, lrName, types.StrategicMergePatchType, []byte(limitRangePayload), metav1.PatchOptions{}) framework.ExpectNoError(err, "Failed to patch limitRange %q", lrName) framework.ExpectEqual(patchedLimitRange.Labels[lrName], "patched", "%q label didn't have value 'patched' for this limitRange. Current labels: %v", lrName, patchedLimitRange.Labels) checkMinLimitRange := apiequality.Semantic.DeepEqual(patchedLimitRange.Spec.Limits[0].Min, newMin) framework.ExpectEqual(checkMinLimitRange, true, "LimitRange does not have the correct min limitRange. Currently is %#v ", patchedLimitRange.Spec.Limits[0].Min) framework.Logf("LimitRange %q has been patched", lrName) ginkgo.By(fmt.Sprintf("Delete LimitRange %q by Collection with labelSelector: %q", lrName, patchedLabelSelector)) err = lrClient.DeleteCollection(ctx, metav1.DeleteOptions{}, metav1.ListOptions{LabelSelector: patchedLabelSelector}) framework.ExpectNoError(err, "failed to delete the LimitRange by Collection") ginkgo.By(fmt.Sprintf("Confirm that the limitRange %q has been deleted", lrName)) err = wait.PollImmediate(1*time.Second, 10*time.Second, checkLimitRangeListQuantity(f, patchedLabelSelector, 0)) framework.ExpectNoError(err, "failed to count the required limitRanges") framework.Logf("LimitRange %q has been deleted.", lrName) ginkgo.By(fmt.Sprintf("Confirm that a single LimitRange still exists with label %q", e2eLabelSelector)) limitRangeList, err = f.ClientSet.CoreV1().LimitRanges("").List(ctx, metav1.ListOptions{LabelSelector: e2eLabelSelector}) framework.ExpectNoError(err, "Failed to list any limitRanges: %v", err) framework.ExpectEqual(len(limitRangeList.Items), 1, "Failed to find the correct limitRange count") framework.Logf("Found %d limitRange", len(limitRangeList.Items)) }) </ins> }) func equalResourceRequirement(expected v1.ResourceRequirements, actual v1.ResourceRequirements) error {

4 Oct 2022 [x] Two weeks soak end date: 18 Oct 2022 [x] Test promotion PR: According to this APIsnoop query, there are still three remaining LimitRange endpoints which are untested. listCoreV1LimitRangeForAllNamespacespatchCoreV1NamespacedLimitRangedeleteCoreV1CollectionNamespacedLimitRange Using a number of existing e2e test practices a new has been created for three LimitRange endpoints. The e2e logs for this test are listed below. This query shows which LimitRange endpoints are hit within a short period of running this e2e test example endpoint | useragent --------------------------------------------+
createCoreV1NamespacedLimitRange | should ensure that a limitRange can be listed, patched and deleted by collection deleteCoreV1CollectionNamespacedLimitRange | should ensure that a limitRange can be listed, patched and deleted by collection listCoreV1LimitRangeForAllNamespaces | should ensure that a limitRange can be listed, patched and deleted by collection listCoreV1NamespacedLimitRange | should ensure that a limitRange can be listed, patched and deleted by collection patchCoreV1NamespacedLimitRange | should ensure that a limitRange can be listed, patched and deleted by collection (5 rows) If a test with these calls gets merged, test coverage will go up by 3 point This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: }) } } <ins> func TestIsErrorTypesByReasonAndCode(t *testing.T) { testCases := []struct { name                  string knownReason           metav1.StatusReason otherReason           metav1.StatusReason otherReasonConsidered bool code                  int32 fn                    func(error) bool }{ { name:                  "IsRequestEntityTooLarge", knownReason:           metav1.StatusReasonRequestEntityTooLarge, otherReason:           metav1.StatusReasonForbidden, otherReasonConsidered: false, code:                  http.StatusRequestEntityTooLarge, fn:                    IsRequestEntityTooLargeError, }, { name:                  "TooManyRequests", knownReason:           metav1.StatusReasonTooManyRequests, otherReason:           metav1.StatusReasonForbidden, otherReasonConsidered: false, code:                  http.StatusTooManyRequests, fn:                    IsTooManyRequests, }, { name:                  "Forbidden", knownReason:           metav1.StatusReasonForbidden, otherReason:           metav1.StatusReasonNotFound, otherReasonConsidered: true, code:                  http.StatusForbidden, fn:                    IsForbidden, }, { name:                  "NotFound", knownReason:           metav1.StatusReasonNotFound, otherReason:           metav1.StatusReasonForbidden, otherReasonConsidered: true, code:                  http.StatusNotFound, fn:                    IsNotFound, }, } for _, tc := range testCases { t.Run(tc.name, func(t *testing.T) { t.Run("by known reason", func(t *testing.T) { err := &StatusError{ metav1.Status{ Reason: tc.knownReason, }, } got := tc.fn(err) if !got { t.Errorf("expected reason %s to match", tc.knownReason) } }) t.Run("by code and unknown reason", func(t *testing.T) { err := &StatusError{ metav1.Status{ Reason: metav1.StatusReasonUnknown, // this could be _any_ reason that isn't in knownReasons. Code:   tc.code, }, } got := tc.fn(err) if !got { t.Errorf("expected code %d with reason %s to match", tc.code, tc.otherReason) } }) if !tc.otherReasonConsidered { return } t.Run("by code and other known reason", func(t *testing.T) { err := &StatusError{ metav1.Status{ Reason: tc.otherReason, Code:   tc.code, }, } got := tc.fn(err) if got { t.Errorf("expected code %d with reason %s to not match", tc.code, tc.otherReason) } }) }) } } </ins>

Checked a method error using when the error was caused by a validating webhook that returns a 403 response. In this particular case, the error was generated from a validating webhook implemented with controller-runtime that returns controller-runtime's response. to return true. returned false Many (but not all) of the other family of functions in suffer the same kind of problem. I have changes ready to submit in a PR if we can get agreement that this is a problem and that it should be fixed in the functions that ought to check the of an erorr value that is or wraps an Kubernetes version (use ): 19.3 for the server
/sig api-machinery
are you going to send a PR for this? The issue description looks reasonable. /triage accepted
Thanks, PR is submitted.
/assign


Question: framework.ExpectNoError(err, "failed to ensure number of pods associated with job %s is equal to parallelism count in namespace: %s", job.Name, f.Namespace.Name) ginkgo.By("Updating the job with suspend=true") <del> job, err = e2ejob.GetJob(f.ClientSet, f.Namespace.Name, job.Name) framework.ExpectNoError(err, "failed to retrieve latest job object") job.Spec.Suspend = pointer.BoolPtr(true) job, err = e2ejob.UpdateJob(f.ClientSet, f.Namespace.Name, job) </del> <ins> err = wait.PollImmediate(framework.Poll, framework.SingleCallTimeout, func() (bool, error) { job, err = e2ejob.GetJob(f.ClientSet, f.Namespace.Name, job.Name) if err != nil { return false, err } job.Spec.Suspend = pointer.Bool(true) updatedJob, err := e2ejob.UpdateJob(f.ClientSet, f.Namespace.Name, job) if err == nil { job = updatedJob return true, nil } if apierrors.IsConflict(err) { return false, nil } return false, err }) </ins> framework.ExpectNoError(err, "failed to update job in namespace: %s", f.Namespace.Name) ginkgo.By("Ensuring pods are deleted")

sig-release-master-blocking gce-cos-master-default Kubernetes e2e suite.[sig-apps] Job should delete pods when suspended not clear No response /sig apps cc
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
cc
cc
/assign
Candidate (I'll run the suite a few times)


Question: "k8s.io/kubernetes/pkg/controller/disruption" "k8s.io/kubernetes/test/integration/etcd" "k8s.io/kubernetes/test/integration/framework" <ins> "k8s.io/utils/pointer" </ins> ) func setup(t *testing.T) (*kubeapiservertesting.TestServer, *disruption.DisruptionController, informers.SharedInformerFactory, clientset.Interface, *apiextensionsclientset.Clientset, dynamic.Interface) {

PDB was upgraded to v1 in 1.21. Despite the changelog saying that the behavior of the v1beta1 implementation should have stayed the same, it appears that the for the field was set to . I believe this is the change in question: When running with a starting config that had a with both a key and a key, and a patch that had a with just a key, the behavior between 1.20 and 1.21 differs. What we get is the key gets deleted, rather than merged with the patch's key to make one unified selector. In 1.20, the behavior we'd get is thus, which is desirable for us because it lets us inject a base assumption about our PDBs into everything we do: This is my repro case: : Compare by using the command from earlier in this report: I also brought this up in Slack: I don't think we want to switch to using just for PDBs, because that wouldn't solve issues where our users want to use without having to re-specify the from our base. Putting aside the fact that changing the behavior of the v1beta1 implementation was probably not intentional, it also seems undesirable to change how selectors work in general. I don't know how widely scoped this change will be across various resource types, too, so we're starting down quite a bit of nasty discovery time. <detailsThis is the version I used to demonstrate my repro case, but I confirmed that this occurs even in kubectl v1.21.0. For my "happy" repro case, this is what the version output looks like: </details<detailsAWS (the hard way) </detailsNo response No response No response No response
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment.

<detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
as requested, I am tagging you and on this one
We should narrowly revert the strategic merge patch selector behavior for v1beta1 to match pre-1.21 behavior for backwards compatibility. More broadly, it is unexpected for multiple actors to be writing to a single selector. The pre-1.21 strategic merge patch behavior for v1beta1 PDB was already inconsistent w.r.t. merging portions of the selector and replacing others (it would merge and replace ) For that reason, server-side apply treats selector writes as owning the entire selector, and the v1 PodDisruptionBudget API will do the same for strategic merge patch. edit: reworded the inconsistencies with the < 1.21 selector handling
More broadly, it is unexpected for multiple actors to be writing to a single selector. The pre-1.21 strategic merge patch behavior for v1beta1 PDB has problems with data getting left in the selector incorrectly if the same user submits a patch with two keys, and then submits a patch with only a single key. Would that be considered a bug in the implementation, or was there some inherent, design-caused race condition in patch application that I don't understand? Cuz if I were to sequentially apply patches with the old strategic merge, I would think the selector would just accumulate keys, which seems fine to me–but I can see how a race condition would turn it into chaos
edit: I was wrong about the previous behavior... the issue was that it already inconsistently merged the selector... it would merge and replace , which is still pretty incoherent If you really want to tweak field-level things, jsonpatch can still do surgical insertion/appending/removal of specific fields.
Should we expect to use jsonpatch for anything that can have a selector, going forward? If it changed for PDBs then it should change for everything else, I would assume edit: we discussed more in slack. The answer is no, the fields for other resource types' label selectors are not changing, so we only need to treat PDBs as a special case.


Question: var cache *schedulerCache var snapshot *Snapshot <del> type operation = func() </del> <ins> type operation = func(t *testing.T) </ins> addNode := func(i int) operation { <del> return func() { </del> <ins> return func(t *testing.T) { </ins> if err := cache.AddNode(nodes[i]); err != nil { t.Error(err) } } } removeNode := func(i int) operation { <del> return func() { </del> <ins> return func(t *testing.T) { </ins> if err := cache.RemoveNode(nodes[i]); err != nil { t.Error(err) } } } updateNode := func(i int) operation { <del> return func() { </del> <ins> return func(t *testing.T) { </ins> if err := cache.UpdateNode(nodes[i], updatedNodes[i]); err != nil { t.Error(err) } } } addPod := func(i int) operation { <del> return func() { </del> <ins> return func(t *testing.T) { </ins> if err := cache.AddPod(pods[i]); err != nil { t.Error(err) } } } addPodWithAffinity := func(i int) operation { <del> return func() { </del> <ins> return func(t *testing.T) { </ins> if err := cache.AddPod(podsWithAffinity[i]); err != nil { t.Error(err) } } } removePod := func(i int) operation { <del> return func() { </del> <ins> return func(t *testing.T) { </ins> if err := cache.RemovePod(pods[i]); err != nil { t.Error(err) } } } removePodWithAffinity := func(i int) operation { <del> return func() { </del> <ins> return func(t *testing.T) { </ins> if err := cache.RemovePod(podsWithAffinity[i]); err != nil { t.Error(err) } } } updatePod := func(i int) operation { <del> return func() { </del> <ins> return func(t *testing.T) { </ins> if err := cache.UpdatePod(pods[i], updatedPods[i]); err != nil { t.Error(err) } } } updateSnapshot := func() operation { <del> return func() { </del> <ins> return func(t *testing.T) { </ins> cache.UpdateSnapshot(snapshot) if err := compareCacheWithNodeInfoSnapshot(t, cache, snapshot); err != nil { t.Error(err)

/sig scheduling What happened: reverted an old behavior to ensure Pod information is not lost when Nodes are recreated without its Pods getting recreated. A ghost Pod is kep in The problem is that we rely in this map for determining deleted Nodes when updating the snapshot in . The good thing is that the nodeinfo's are cloned into the snapshot, so we shouldn't have nil pointer accesses. But we might lose a few cycles trying to schedule on nodes that are deleted How to reproduce it (as minimally and precisely as possible): a node with a few pods. node without deleting pods. Result: The snapshot still has the node. Environment: Kubernetes version (use ): 1.18+
/assign
It actually produces the error: which would cause failed scheduling cycles (the map is not recreated, only the lists).


Question: "io/ioutil" "os" "path/filepath" <ins> "syscall" </ins> "testing" utiltesting "k8s.io/client-go/util/testing"

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): TestMetricsDuGetCapacity Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): master branch (tip) Environment: Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): SLES 12 Kernel (e.g. ): Linux pts00449-vm23 3.12.28-4-default SMP Thu Sep 25 17:02:34 UTC 2014 () ppc64le ppc64le ppc64le GNU/Linux Install tools: Others: What happened: make test WHAT="./pkg/volume" Running tests for APIVersion: v1,apps/v1beta1,,,,,autoscaling/v1,autoscaling/v2alpha1,batch/v1,batch/v2alpha1,,extensions/v1beta1,,policy/v1beta1,,,,,,federation/v1beta1 +++ [0404 11:22:46] Running tests without code coverage --- FAIL: TestMetricsDuGetCapacity (0.01s) Unexpected Used for directory with file. Expected 4096, got .

FAIL FAIL 0.137s Makefile:164: recipe for target 'test' failed make: [test] Error 1 Test case failed What you expected to happen: Test case should pass How to reproduce it (as minimally and precisely as possible): make test WHAT="./pkg/volume" Anything else we need to know*: Found this similar issue: However the values returned are different, see this string "Expected 4096, got "
Hi, I am able to run that tests: I am running RHEL ppc64le, latest k8s with go1.8, probably something specific to do with SLES?
I still get the error on SLES, updated the code to latest and using go1.8, maybe it is SLES (or my machine?) specific talks about /tmp space, not sure if its related..
Hi , Some investigation into this i found that the issue is due to the hardcoding on line 32 in (similar from comments here: ) I changed it to remove the hardcoding (patch attached) and the test case passes on SLES as well, i am not sure if the patch is appropriate so requesting you to review it as appropriate
Hi Vaibhav, Have you raised any PR on it? If raised, reviewing would be easier. There is one thing I noticed: In the above code, you are comparing actual used value with total of empty directory usage and expected block size. When you create a new file like "f1 created here with only "Hello World", it consumes 4K in Ubuntu/RHEL. This size is same as for an empty directory. What is this size in SLES? So, either you hardcode this value(conditional for SLES) or call getExpectedBlockSize() on f1 file path instead of tmpDir.
Hi I have created the PR for the patch as suggested.. I have also changed the original patch slightly to use getExpectedBlockSize() on f1 file path instead of tmpDir as you suggested as that I think would be the right way to do it.. Thanks! Vaibhav
/sig storage


Question: }, }, expectedDevicePath:      "/dev/rbd1", <del> expectedDeviceMountPath: fmt.Sprintf("%s/plugins/kubernetes.io/rbd/rbd/pool1-image-image1", tmpDir), </del> <ins> expectedDeviceMountPath: fmt.Sprintf("%s/plugins/kubernetes.io/rbd/mounts/pool1-image-image1", tmpDir), </ins> expectedPodMountPath:    fmt.Sprintf("%s/pods/%s/volumes/kubernetes.io~rbd/vol1", tmpDir, podUID), }) cases = append(cases, &testcase{

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: The method for ceph rbd volume plugin generates the path with code as follows: So the device mount path's format will be: , this implementation has two problems: with other volume plugins, as all the other plugins return , see as an example. method of ceph rbd plugin is not working properly, because rbd plugin uses method to find the device mount path(just like all the other plugins do), and method assumes the device mount path's format is (see ). What you expected to happen: The device mount path of rbd plugin should be . Anything else we need to know?: Do we have any reason for this design? If not, I will submit a PR to fix this, the adjusted method maybe look like this: /sig storage
/assign
nice catch There are two ways to fix this: Change mount path. But this will introduce compatibility issues during upgrade. Use , instead of to replace PR welcome
cc
Hi , there is a difference between and : returns the device path, such as , and returns the device mounted path, such as . What we really want is the second one, as we need to find the pool and image name. So we can't just simply replace with . Maybe we can find the mounted path specially for rbd plugin, use , the code looks like this:
thanks let make a forward moving plan: fix and append to the global path keep in . If no path is found, fall back to old path scheme using your code
Nice idea , I've submit a PR for this, PTAL.
I've checked and the reason is that I tested with the old release branch, the has been changed with in mount path, and causes the problem. thanks for your PR
i read again. There is another return path . So in rbd, the missing dir will not make into , but the second return should still give us the dir we need
You are right , indeed has two return path, but the second return still can't meet our needs. Because the parameter of is the mount path for a specific pod, such as , its format is , so will return the volume name. This volume name is returned by (see ), usually is the PV's name or the name specified in , both of these are not .


Question: It("should surface a failure condition on a common issue like exceeded quota", func() { rcConditionCheck(f) }) <ins> It("should adopt matching pods on creation", func() { testRCAdoptMatchingOrphans(f) }) It("should release no longer matching pods", func() { testRCReleaseControlledNotMatching(f) }) </ins> }) func newRC(rsName string, replicas int32, rcPodLabels map[string]string, imageName string, image string) *v1.ReplicationController {

The merge queue is green with RBAC enabled, and the replicaset controller was missing permission to patch pods until fixed in Pod adoption and release depend on patching pods, which means that there are no merge-queue-blocking tests that test these features. Given that they are complex and rarely tested manually, we need tests that run as part of the merge queue to ensure they function correctly.
I'd like to work on this, where should the tests be ? I see that there are some patch-related tests for the replicaset controller in , and, according to http://submit-, the integration tests are run as part of the merge queue, is that correct? Thanks!
I think we'd want an e2e test that drives behavior externally on a normal cluster (by creating a replicaset, waiting to observe expected pods, modifies replicaset, waits to observe expected results, etc) . many of the integration tests tend to start up fake servers that don't share much in common with real deployments.
Ok thanks, on it.
cc:


Question: }, }, }, <del> "spec.jobTemplate.spec.template.spec.restartPolicy: Unsupported value": { </del> <ins> "spec.jobTemplate.spec.template.spec.restartPolicy: Required value": { </ins> ObjectMeta: metav1.ObjectMeta{ Name:      "mycronjob", Namespace: metav1.NamespaceDefault,

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Deployed an cronJob with an empty and it defaulted to which is an invalid option: Leaving the key out completely has the same result What you expected to happen: The empty key to default to something valid like or . Also document what this default is, my proposal would be to have as default since some tasks can't be reran easily. But it might be nice to force as default to enforce the stateless mindset even for crons :) How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: I guess not but I am open for questions :) Environment: Kubernetes version (use ): 1.19.2
My guess would be /sig api-machinery If it is better practice to wait for the triage instead of guessing the sig, let me know :+1:
/sig api-machinery
/assign
/remove-sig api-machinery /sig apps
/assign
/triage accepted


Question: klog.V(2).Infof("%s: watch of %v returned 429 - backing off", r.name, r.expectedTypeName) <-r.initConnBackoffManager.Backoff().C() continue <ins> case apierrors.IsInternalError(err) && retry.ShouldRetry(): klog.V(2).Infof("%s: retrying watch of %v internal error: %v", r.name, r.expectedTypeName, err) continue </ins> default: klog.Warningf("%s: watch of %v ended with: %v", r.name, r.expectedTypeName, err) }

When etcd is unable to elect leader for 3 election cycles, etcd returns "no leader" error through watches. This can be seen in api-server logs as . Also, when apiserver receives this error, it terminates all watchers: . In large clusters, this can destabilize master, as all these watchers need to be recreated almost at the same time, overwhelming api-server quite often. I propose adding new api-server flag that allows to specify timeout, during which we are retrying to reestablish apiserver <-etcd watches, without terminating all api-server watchers. If we are unable to reestablish watches within this specified timeout, we start terminating all watchers. Losing etcd leader for ~2-3s does not destabilize api-server. Trigger prolonged reelection of etcd leader. No response Observed in all kubernetes versions 1.19-1.24 N/A <details</details<details</details<details</details<details</details
/sig scalability /sig api-machinery /priority important-soon
/wg reliability
/assign
/triage accepted
indeed, this is a source of instability. Is this coming from ?
I believe it's coming from this change. As I mentioned, the idea to mitigate it is to add some parameter that will allow us to ignore this error coming from etcd for specific duration without terminating all watchers, for example 10s be default.
With this comment thread in mind: It would probably be cleaner to allow for simply setting the "timeout" as part of WithRequireLeader option (instead of using the constant of 3). But that would require etcd changes, including the API and we want to workaround that in much smaller timeframe. But a comment about how to make that cleaner with some additional context from here would help.


Question: set = obj.(*apps.StatefulSet) } if set.Status.Replicas != 3 { <del> t.Error("Falied to scale statefulset to 3 replicas") </del> <ins> t.Errorf("set.Status.Replicas = %v; want 3", set.Status.Replicas) </ins> } pods, err := spc.addTerminatingPod(set, 3) if err != nil {

Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: Unit test error message match the conventional Go style: For example: if got, want := set.Status.Replicas, 3; got != want { t.Errorf("set.Status.Replicas = %v; want %v", got, want) }
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig testing
/assign
/reopen


Question: err := decorated.Do(func() { defer close(executed) if !delegated { <del> t.Fatalf("expected delegated execution") </del> <ins> t.Errorf("expected delegated execution") </ins> } }) if err == nil {

Can we get this fixed ASAP? Thanks!
yep, thanks for reporting , Brendan Burns wrote:
how reproduceable is this? I have some changes that I think will resolve it but I couldn't reproduce locally even when all my cpu's were pretty tied up and using a high GOMAXPROCS.
flakey label process?
the labelling is due to me failing @ github's label interface. Bulk editing labels removes them if any of the selected things have the label sigh Thanks for the quick fix.


Question: lbStatus, err := az.getServiceLoadBalancerStatus(service, lb) if err != nil { klog.Errorf("getServiceLoadBalancerStatus(%s) failed: %v", serviceName, err) <del> return nil, err </del> <ins> if err != cloudprovider.InstanceNotFound { return nil, err } </ins> } var serviceIP *string

When upgrade cluster to kubernetes 1.20.15 on Azure platform, I still hit "instance not found" error when ensureHostInPool, and this leads to one of the backend pool members missing since then, users have to add the member back manually. Below is the error: The Detected change in list of current cluster nodes should be in size of 90, but actually it only has 89 nodes here. The symptom matches issue and the But the fix may be incomplete. I expected the "instance not found" error will not affect backend pool members management, and it's ignored somewhere in either or (1.20.15) By upgrading the cluster and trigger the VM recreation, and this may not be easy to reproduce, but it indeed happens several times in our environment <details20.15 </details<detailslegacy Azure cloud provider </details
Hi as you previously worked on the similar issue, could you please kindly help take a look? Thank you in advance!
/sig cloud-provider /area provider/azure /priority critical-urgent
20.x is not a supported version. Could you upgrade to latest supported version?
Hi thank you for reply! Do you mean that this could be already fixed in latest version? I'm not sure if my understanding is correct, but if the issue exists in 1.20.x, even latest version does not ignore this error in the specific place.
yes, please upgrade the version to 1.22 or above with out-of-tree cloud provider
Hi sorry, I don't mean AKS by Azure platform, in fact this is a VMware product built upon Azure, is it possible to port the fix to in-tree cloud provider? That would be great if it's doable, if not, or could you please help point out the fix, maybe we could rase a PR for the in-tree cloud provider. Thank you!
We're still on in-tree cloud provider even if it's upgraded to 1.22.
The error we saw is thrown in this code it appears in out-of-tree code, this still could lead to the issue.
/remove-priority critical-urgent please don't set priorities, those are meant to be used by the community members that triage the issue
Sorry, I copied from previous issue.
are you using VMSS, VMAS or standalone VM?
this is our config, sorry, how do I know which kind of it is?

primaryAvailabilitySetName means the cluster is using VMAS. Per the codes , the issue probably still exists. could you help for the fix? /assign
Thank you for confirming! That's what I'm worried about.
/triage accepted


Question: obj:  &F{}, }, { <ins> data: "{"fj":""}", obj:  &F{}, }, { </ins> // Invalid string data data: "{"fa":123}", obj:  &F{},

I have created the following secret in my cluster. When I add labels to my secret and try to apply it in the cluster I get the following error This also happens when I use or using the go client. Not sure why, but works. I expect that the secret can be updated with the new labels since the is not actually changed. Steps explained above. No response <details</details<detailsKind </details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cli
/assign
/remove-sig cli /sig apps


Question: # If you update this list, please also update build/BUILD. readonly KUBE_CLIENT_TARGETS=( cmd/kubectl <ins> cmd/kubectl-convert </ins> ) readonly KUBE_CLIENT_BINARIES=("${KUBE_CLIENT_TARGETS[@]##*/}") readonly KUBE_CLIENT_BINARIES_WIN=("${KUBE_CLIENT_BINARIES[@]/%/.exe}")

was removed in 1.20 in favor of a plugin. However, this is not included in release artifacts. The binary is included in release artifacts, and ideally easily installable via kubectl plugin managers like . /sig cli /assign
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This question came up in a docs review of a page referencing () Once this is resolved, please ensure the website is updated to document installation/use of this tool.
/sig release


Question: fs.StringVar(&o.ConfigFile, "config", o.ConfigFile, "The path to the configuration file.") fs.StringVar(&o.WriteConfigTo, "write-config-to", o.WriteConfigTo, "If set, write the default configuration values to this file and exit.") <del> fs.StringVar(&o.config.ClientConnection.Kubeconfig, "kubeconfig", o.config.ClientConnection.Kubeconfig, "Path to kubeconfig file with authorization information (the master location is set by the master flag).") </del> <ins> fs.StringVar(&o.config.ClientConnection.Kubeconfig, "kubeconfig", o.config.ClientConnection.Kubeconfig, "Path to kubeconfig file with authorization information (the master location can be overridden by the master flag).") </ins> fs.StringVar(&o.config.ClusterCIDR, "cluster-cidr", o.config.ClusterCIDR, "The CIDR range of pods in the cluster. When configured, traffic sent to a Service cluster IP from outside this range will be masqueraded and traffic sent from pods to an external LoadBalancer IP will be directed to the respective cluster IP instead") fs.StringVar(&o.config.ClientConnection.ContentType, "kube-api-content-type", o.config.ClientConnection.ContentType, "Content type of requests sent to apiserver.") fs.StringVar(&o.master, "master", o.master, "The address of the Kubernetes API server (overrides any value in kubeconfig)")

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Run , we could see two related command-line options. The explanation for is ambiguous, it says the master location is ought to be set by the master flag. This statement is inconsistent with the actual behavior. What you expected to happen: In fact, the flag is optional, it only overrides the control plane address in . That means, if we are using a single-context kubeconfig(or never changes the context), it's OK to just set and leave unset. Kube-proxy will parse the kubeconfig and use the credentials and master address defined in current context to contact cluster. Truth from code shows that kubeconfig parser do honors context setting: How to reproduce it (as minimally and precisely as possible): Environment: Kubernetes version (use ): v1.19.0 /sig network /area kube-proxy
Proposed a fix here
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/assign
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.


Question: dataChangedMsg = "patched" } <ins> // After computing whether we changed data, refresh the resource info with the resulting object if err := info.Refresh(patchedObj, true); err != nil { return err } </ins> if len(options.OutputFormat) > 0 && options.OutputFormat != "name" { return cmdutil.PrintResourceInfoForCommand(cmd, info, f, out) }

The bug is introduced in The oldData, newData in seem always to be the same. So it always showing . After digging a bit, I found that: the place where we save the may be wrong. sometimes oldData and newData are different because of different s, even though the patch is . cc:
cc
info.Object is updated by the info.Refresh call. We should compute old data prior to refreshing the info bug was introduced in
I think the second bug is because using read from stream has less infomation than when you fetch from server, so but didn't


Question: validatePodReadiness(false) }) <ins> ginkgo.It("should delete a collection of pods", func() { podTestNames := []string{"test-pod-1", "test-pod-2", "test-pod-3"} ginkgo.By("Create set of pods") // create a set of pods in test namespace for _, podTestName := range podTestNames { _, err := f.ClientSet.CoreV1().Pods(f.Namespace.Name).Create(context.TODO(), &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name: podTestName, Labels: map[string]string{ "type": "Testing"}, }, Spec: v1.PodSpec{ Containers: []v1.Container{{ Image: imageutils.GetE2EImage(imageutils.Agnhost), Name:  "token-test", }}, RestartPolicy: v1.RestartPolicyNever, }}, metav1.CreateOptions{}) framework.ExpectNoError(err, "failed to create pod") framework.Logf("created %v", podTestName) } // wait as required for all 3 pods to be found ginkgo.By("waiting for all 3 pods to be located") err := wait.PollImmediate(podRetryPeriod, podRetryTimeout, checkPodListQuantity(f, "type=Testing", 3)) framework.ExpectNoError(err, "3 pods not found") // delete Collection of pods with a label in the current namespace err = f.ClientSet.CoreV1().Pods(f.Namespace.Name).DeleteCollection(context.TODO(), metav1.DeleteOptions{}, metav1.ListOptions{ LabelSelector: "type=Testing"}) framework.ExpectNoError(err, "failed to delete collection of pods") // wait for all pods to be deleted ginkgo.By("waiting for all pods to be deleted") err = wait.PollImmediate(podRetryPeriod, podRetryTimeout, checkPodListQuantity(f, "type=Testing", 0)) framework.ExpectNoError(err, "found a pod(s)") }) </ins> }) <ins> func checkPodListQuantity(f *framework.Framework, label string, quantity int) func() (bool, error) { return func() (bool, error) { var err error list, err := f.ClientSet.CoreV1().Pods(f.Namespace.Name).List(context.TODO(), metav1.ListOptions{ LabelSelector: label}) if err != nil { return false, err } if len(list.Items) != quantity { framework.Logf("Pod quantity %d is different from expected quantity %d", len(list.Items), quantity) return false, err } return true, nil } } </ins>

[x] Original approval issue: 09 July 2020 [x] Two weeks soak end date: 24 July 2020 This issue is created to allow edit by /wip /hold According to this APIsnoop query, there are still an endpoint which is untested. a set of 3 Pods with a static label within a Namespace. that all 3 Pods with the label in the Namespace are created. the set of Namespaced Pods with a label via DeleteCollection. that all Pods with the label have been deleted. Discover useragents: List endpoints hit by the test: Display endpoint coverage change: Two other endpoints tested here have tests in progress to conformance already. If a test with these calls gets merged, test coverage will go up by 1 points This test is also created with the goal of conformance promotion.
/sig testing /sig architecture /area conformance
During Conformance Meeting: Is there a way to write this test so it is resource agnostic? If we are just changing the resource... can be parameterize the resource-type
/assign
GitHub didn't allow me to assign the following users: heyste. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Test to be created using "Polling" and not "Watch tooling"
/reopen Merged Issue open, waiting for Promotion PR
Reopened this issue. <detailsIn response to : /reopen Merged Issue open, waiting for Promotion PR Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
The test appears to be flaking, need to reach out to #sig-node for some feedback.
Can you point us to the link of the flaky tests? Thanks!
The Promotion The testgrid Thank you!


Question: fi } <del> # Get default service account credentials of the VM. GCE_METADATA_INTERNAL="http://metadata.google.internal/computeMetadata/v1/instance" function get-credentials { curl "${GCE_METADATA_INTERNAL}/service-accounts/default/token" -H "Metadata-Flavor: Google" -s | python -c  'import sys; import json; print(json.loads(sys.stdin.read())["access_token"])' } function valid-storage-scope { curl "${GCE_METADATA_INTERNAL}/service-accounts/default/scopes" -H "Metadata-Flavor: Google" -s | grep -E "auth/devstorage|auth/cloud-platform" } </del> function download_tarball() { local -r download_path="$1" local -r file="$2"

Many jobs are failing because of Some of the job logs: /priority critical-urgent /cc /cc /milestone v1.18 /sig release
Also, all the jobs report in their log the following: Do we have to take any action to change that?
After some investigation, it seems that file contains instead of
Switching from to podutils should be done by the job owners - this is out of scope for releas team (strictly speaking). And the message is just a warning, should not affect the job itself.
is this related to the workload identity switch yesterday?
Maybe? get- does seem to pass auth headers, possibly this part isn't working correctly? These jobs explicitly mount a service- though so I wouldn't expect workload identity to change things but I could be wrong...
Yes, this will not work until we attach a kubernetes service account to the job that uses workload identity:
Ideally that command should become (and let it decide how to best authenticate) rather than assuming the script runs on a VM
/assign
sent
/reopen doesn't seem to solve the issue. The jobs are still failing because of this reason. Do we need any more actions after ????
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I don't agree, the definitely helped in case of e2e scalability jobs, our presubmits and continuous jobs started passing again, e.g. https://k8s-
Looks green?
Only the 1.18 jobs are green. We need to cherry-pick the fix to the other branches as well. I will keep this issue open.
For some reason I thought kubetest always used the copy from HEAD.
in the case that it didn't we'll need this cherry picked to all active branches :|
I thought the same, but indeed the fixed only 1.18 and master jobs. Older release scalability jobs are still broken ! It's a bit confusing TBH. I thought that get- is always downloaded by kubetest from which IIUC points to the master branch.

Alternatively, get- could've been embedded in the kubekins image, but if that was the case then couldn't have fixed our 1.18/master tests without kubekins version update (which didn't happen). Another options is that kubetest downloads get- from proper release branches, but I couldn't find such logic in the kubetest code. would you mind explaining how it works?
There are a lot of jobs that are failing because of this. - master - 1.18 7 jobs in 1.17 branch 5 jobs in 1.16 branch 5 jobs in 1.15 branch Can you please take a look?
/sig testing
Ping, any updates?
filed a number of cherry picks for the script
Closing... The issue has been resolved. Thanks, /close
Closing this issue. <detailsIn response to : Closing... The issue has been resolved. Thanks, /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: // needs to be updated again by the node status updater. // If the specifed node does not exist in the nodesToUpdateStatusFor list, log the error and return // This is an internal function and caller should acquire and release the lock <del> func (asw *actualStateOfWorld) updateNodeStatusUpdateNeeded(nodeName types.NodeName, needed bool) { </del> <ins> func (asw *actualStateOfWorld) updateNodeStatusUpdateNeeded(nodeName types.NodeName, needed bool) error { </ins> nodeToUpdate, nodeToUpdateExists := asw.nodesToUpdateStatusFor[nodeName] if !nodeToUpdateExists { // should not happen <del> glog.Errorf( "Failed to set statusUpdateNeeded to needed %t because nodeName=%q  does not exist", needed, nodeName) return </del> <ins> errMsg := fmt.Sprintf("Failed to set statusUpdateNeeded to needed %t because nodeName=%q  does not exist", needed, nodeName) glog.Errorf(errMsg) return fmt.Errorf(errMsg) </ins> } nodeToUpdate.statusUpdateNeeded = needed asw.nodesToUpdateStatusFor[nodeName] = nodeToUpdate <ins> return nil </ins> } func (asw *actualStateOfWorld) SetNodeStatusUpdateNeeded(nodeName types.NodeName) { asw.Lock() defer asw.Unlock() <del> asw.updateNodeStatusUpdateNeeded(nodeName, true) </del> <ins> if err := asw.updateNodeStatusUpdateNeeded(nodeName, true); err != nil { glog.Errorf("Failed to update statusUpdateNeeded field in actual state of world: %v", err) } </ins> } func (asw *actualStateOfWorld) DeleteVolumeNode(

was the fix in sufficient, or do you want to bubble the error past SetNodeStatusUpdateNeeded?
do you want this in v1.7?
can you retitle this to be meaningful or close it?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
I checked the sources, now returns as requested here. /close


Question: } } <ins> func validateDaemonSetPodsActive( podClient corev1client.PodInterface, podInformer cache.SharedIndexInformer, numberPods int, t *testing.T, ) { if err := wait.Poll(time.Second, 60*time.Second, func() (bool, error) { objects := podInformer.GetIndexer().List() if len(objects) < numberPods { return false, nil } podsActiveCount := 0 for _, object := range objects { pod := object.(*v1.Pod) ownerReferences := pod.ObjectMeta.OwnerReferences if len(ownerReferences) != 1 { return false, fmt.Errorf("Pod %s has %d OwnerReferences, expected only 1", pod.Name, len(ownerReferences)) } controllerRef := ownerReferences[0] if got, want := controllerRef.Kind, "DaemonSet"; got != want { t.Errorf("controllerRef.Kind = %q, want %q", got, want) } if controllerRef.Controller == nil || *controllerRef.Controller != true { t.Errorf("controllerRef.Controller is not set to true") } if pod.Status.Phase == v1.PodRunning || pod.Status.Phase == v1.PodPending { podsActiveCount += 1 } } return podsActiveCount == numberPods, nil }); err != nil { t.Fatal(err) } } </ins> // podUnschedulable returns a condition function that returns true if the given pod // gets unschedulable status. func podUnschedulable(c clientset.Interface, podNamespace, podName string) wait.ConditionFunc {

I can reproduce this reliably with Calico v3.25.0 CNI, installed straight from the upstream . It seems to be a regression in Kubernetes 1.27.0-rc.0 compared to 1.27.0-beta.0, at least I can't reproduce it with beta.0. Cluster is deployed, and everything is ready & healthy. After a series of node reboots some pods are stuck in state and never recover: This of course disrupts the CNI and affects the scheduled workloads. The daemonset is "happy" with 2 pods being out: If I delete the pod manually, the pod recovers without any issues. The pods are restarted. Bring up a cluster with graceful node shutdown, reboot nodes a few times (including multiple node at anoce). Additional detailed output: <details</details<details</details<detailsnone </details<detailsTalos Linux 1.4.0-alpha.3 </details<details</details<details</details<detailsCalico v3.25.0, also reproducible with 3.24.1 </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
I wonder if that is related to the change
cc this might a regression in 1.27.0-rc.0 with this PR
Even if it is, it sounds like a bug in daemonset instead. Why wouldn't it replace pods that are terminated? It can probably be reproduced in integration.
I think there might be something with the fact that Calico terminates with exit code '0', so it feels like a "successful" run. Still such pod should be restarted, so I'm not sure why it doesn't get restarted after a reboot.
let me ask to get more details about reproducing this locally. Are you running this on Kind? How do you trigger the the restarts? Can you share mode details on the node configuration, such as the graceful shutdown period?

I initially hit this problem under AWS, but then I narrowed it down to Calico CNI. I'm using Talos Linux, this is the full kubelet config, using 30s/10s graceful shutdown period: <details</detailsLet me try to build a reproduce instruction in the next comment.
I also have complete output of under the details in the issue itself.
The daemonset controller should still recreate the pod in that scenario. /sig apps
The reproducer is to follow the , but it should be same under any Talos platforms except Docker (which doesn't have a graceful node shutdown, and any kind of shutdown). I verified it with the assets from the latest . The smallest I can reproduce it with is 1+1 cluster, but it looks same way e.g. in 3+2 cluster: Once the cluster is up, all the pods are running & healthy (). If I issue a reboot of the worker node with , with shows that pod on the worker node quickly enters state, and never recovers.
At the same time I don't quite understand what is exactly wrong, as if I just do a daemonset with command, the pod also enters state, but it gets quickly restarted by the via transition.
I can also send kubelet logs, and anything else which might be helpful to debug the issue, I'm also open to do a debug session with screen sharing.
This is because of . The pod doesn't reach the phase in this case. The kubelet just recreates the containers. However, when a pod is actually marked as "Succeeded" then kubelet would stop looking at it. However, it should be the responsibility of the daemonset controller to recreate a pod.
Following the sig-apps advise opened the PR () with an integration test for DaemonSet. It passes for Failed pods, but fails for Succeeded. In other words, the succeeded pods are not restarted. I'm yet to investigate (probably tomorrow) why this is so, but it seems like a bug. In the meanwhile some expertise about DaemonSets would be helpful.
Thanks for reporting this. I suppose this might be related to the fact that pods (created by daemonset) can now be marked as in v1.27.

When graceful node shutdown happens, we terminate the pod with : which will trigger kubelet to terminate the pod in pod worker () and eventually I assume set a terminal phase on the pod. We should verify if that is what is happening here. In either case, +1 on comment that I would expect that daemonset controller should recreate a pod if it is marked completed.
I will note that the intent for graceful shutdown was always that pods could be marked Succeeded (vs just Failed), since the pod is indeed terminal. It's likely that this was previously racy (no guarantee succeeded made it to the apiserver) and fixed that. However, outside of graceful shutdown this should never occur - do we have any evidence a RestartAlways pod went succeeded in that case (wasn't clear from the thread)? That would be a serious bug.
I think Succeeded would also be possible in other scenarios, like deletion (deletionTimestamp!=nil). This I've tested for Job pods, but I think DS pods would behave the same. Except for deletion I think the behavior is limited to graceful node shutdown, which invokes the kill function, but respects the Succeeded phase: Other uses of the kill function unconditionally set Failed, such as the eviction manager: EDIT: oh, actually for Jobs we don't use restartPolicy=Always, but still deleted pods with restartPolicy=Always may still terminate with Succeeded phase as shown in this test:
Deletion with RestartAlways is ok, because the pod goes away reasonably fast (but we are adding latency in 1.27 to that path) as you note.


Question: ) } <del> # Get default service account credentials of the VM. GCE_METADATA_INTERNAL="http://metadata.google.internal/computeMetadata/v1/instance" function get-credentials { curl "${GCE_METADATA_INTERNAL}/service-accounts/default/token" -H "Metadata-Flavor: Google" -s | python -c  'import sys; import json; print(json.loads(sys.stdin.read())["access_token"])' } </del> function valid-storage-scope { curl "${GCE_METADATA_INTERNAL}/service-accounts/default/scopes" -H "Metadata-Flavor: Google" -s | grep -E "auth/devstorage|auth/cloud-platform" }

Many jobs are failing because of Some of the job logs: /priority critical-urgent /cc /cc /milestone v1.18 /sig release
Also, all the jobs report in their log the following: Do we have to take any action to change that?
After some investigation, it seems that file contains instead of
Switching from to podutils should be done by the job owners - this is out of scope for releas team (strictly speaking). And the message is just a warning, should not affect the job itself.
is this related to the workload identity switch yesterday?
Maybe? get- does seem to pass auth headers, possibly this part isn't working correctly? These jobs explicitly mount a service- though so I wouldn't expect workload identity to change things but I could be wrong...
Yes, this will not work until we attach a kubernetes service account to the job that uses workload identity:
Ideally that command should become (and let it decide how to best authenticate) rather than assuming the script runs on a VM
/assign
sent
/reopen doesn't seem to solve the issue. The jobs are still failing because of this reason. Do we need any more actions after ????
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I don't agree, the definitely helped in case of e2e scalability jobs, our presubmits and continuous jobs started passing again, e.g. https://k8s-
Looks green?
Only the 1.18 jobs are green. We need to cherry-pick the fix to the other branches as well. I will keep this issue open.
For some reason I thought kubetest always used the copy from HEAD.
in the case that it didn't we'll need this cherry picked to all active branches :|
I thought the same, but indeed the fixed only 1.18 and master jobs. Older release scalability jobs are still broken ! It's a bit confusing TBH. I thought that get- is always downloaded by kubetest from which IIUC points to the master branch.

Alternatively, get- could've been embedded in the kubekins image, but if that was the case then couldn't have fixed our 1.18/master tests without kubekins version update (which didn't happen). Another options is that kubetest downloads get- from proper release branches, but I couldn't find such logic in the kubetest code. would you mind explaining how it works?
There are a lot of jobs that are failing because of this. - master - 1.18 7 jobs in 1.17 branch 5 jobs in 1.16 branch 5 jobs in 1.15 branch Can you please take a look?
/sig testing
Ping, any updates?
filed a number of cherry picks for the script
Closing... The issue has been resolved. Thanks, /close
Closing this issue. <detailsIn response to : Closing... The issue has been resolved. Thanks, /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: # Make sure it's gone kube::test::get_object_assert foos "{{range.items}}{{$id_field}}:{{end}}" '' kube::test::get_object_assert bars "{{range.items}}{{$id_field}}:{{end}}" '' <ins> # Test 'kubectl create' with namespace, and namespace cleanup. kubectl "${kube_flags[@]}" create namespace non-native-resources kubectl "${kube_flags[@]}" create -f hack/testdata/TPR/bar.yaml --namespace=non-native-resources kube::test::get_object_assert bars '{{len .items}}' '1' --namespace=non-native-resources kubectl "${kube_flags[@]}" delete namespace non-native-resources # Make sure objects go away. kube::test::wait_object_assert bars '{{len .items}}' '0' --namespace=non-native-resources # Make sure namespace goes away. local tries=0 while kubectl "${kube_flags[@]}" get namespace non-native-resources && [ ${tries} -lt 10 ]; do tries=$((tries+1)) sleep ${tries} done </ins> } run_recursive_resources_tests() {

BUG REPORT Kubernetes version (use ): v1.6.2 Environment: Cloud provider or hardware configuration: GKE What happened: Create TPR "MyResource" Create namespace "soak-test" Create a "MyResource" "a" under "soak-test":Delete namespace "soak-test"wait namespace to be deleted MyResource "a" still exists after ns is deleted.
/area third-party-resource
/sig cli
I'd like to take a look at the problem if nobody else is working on this
Root Cause of this is the fact that Namespace on ThirdPartyResources is getting ignored (both passed as command --namespace, or the one present in yaml) ThirdPartyResources always end up in default namespace (which is causing confusion reported in bug)
How long has that bug existed? might related to your other findings. We know that bug doesn't exist in CRDs (we have tests specifically checking namespace handling), so I'd suggest converting to those as soon as possible.
Isn't CRD available in 1.7+ ?
Yes. If the bug has existed for multiple releases (not a regression), I don't think this is a blocker for 1.7. It could still be patched, just not a blocker for the current release.
Migration to CRD still is not automated and will still take some time. Have you considered that?
Yes, but if the behavior is not a regression, then this does not block the release. The fact that we have a clear path forward to a working resource is gravy.
This bug existed in 1.4 (), got fixed in 1.5, but appears again in 1.6 .
Ok. Not a blocker for 1.7, but if the patch is reasonable as a fix we can merge it into the service stream. For 1.7, I'd still recommend moving to CRD as quickly as possible.
I was able to reproduce this with CRD on master HEAD. What level of tests are there? Anything e2e?
most of them and specifically a storage one. You're saying we have the from above?
No, I don't see any evidence that they always end up in the default namespace (neither for TPR nor CRD). I'm not sure what that's referring to.

What I was able to reproduce is the bug described at the top of this issue: Create item in namespace, delete namespace, item remains. I observed this for TPR on and , and for CRD on .
The problem seems to be that is misspelled as in this line: If I fix that, namespace cleanup works for TPR. However, for CRD, now namespace cleanup gets stuck and controller manager logs this: Edit: To clarify, I "fixed" the check in the linked code by removing the entire highlighted block. It seems like the singular is not the only problem. It was also using the internal version of when was expected.
Regarding namespace cleanup getting stuck for CRD after the "fix", the error on the apiserver side is: According to the stack trace, the error comes from here:
yes, not only the is misspelled, but also the mismatch of the version. this is why in my original patch, I tried to ignore the version of .
I'd like to keep this open until we have a regression test since this is at least the third time we've fixed this. I'll add a test as part of making sure this fix works for CRD.


Question: return err } <del> // Invalidate the cache since we would update it. if err = ss.deleteCacheForNode(nodeName); err != nil { return err } </del> <ins> // Invalidate the cache since right after update defer ss.deleteCacheForNode(nodeName) </ins> // Update vmssVM with backoff. ctx, cancel := getContextWithCancel()

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: vmss dirty cache issue, caused by What you expected to happen: clean vmss cache should happen after disk attach/detach operation, now it's before those operations, which would lead to dirty cache. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /kind bug /assign /priority important-soon /sig cloud-provider /area provider/azure


Question: return nil, false, nil } if ctx.ComponentConfig.CSRSigningController.ClusterSigningCertFile == "" || ctx.ComponentConfig.CSRSigningController.ClusterSigningKeyFile == "" { <ins> klog.V(2).Info("skipping CSR signer controller because no csr cert/key was specified") </ins> return nil, false, nil }

There are multiple cert controllers that use the same base struct, so when they log, you cannot differentiate which is which.
/sig auth


Question: } } <ins> const defaultCobraMessage = "default message from cobra command" const defaultCobraSubMessage = "default sub-message from cobra command" const cobraMessageDesc = "message to print" const cobraSubMessageDesc = "sub-message to print" func testCobraCommand(n string) *Server { var cobraServer *Server var msg string cmd := &cobra.Command{ Use:   n, Long:  n, Short: n, Run: func(cmd *cobra.Command, args []string) { cobraServer.hk.Printf("msg: %sn", msg) }, } cmd.PersistentFlags().StringVar(&msg, "msg", defaultCobraMessage, cobraMessageDesc) var subMsg string subCmdName := "subcommand" subCmd := &cobra.Command{ Use:   subCmdName, Long:  subCmdName, Short: subCmdName, Run: func(cmd *cobra.Command, args []string) { cobraServer.hk.Printf("submsg: %s", subMsg) }, } subCmd.PersistentFlags().StringVar(&subMsg, "submsg", defaultCobraSubMessage, cobraSubMessageDesc) cmd.AddCommand(subCmd) localFlags := cmd.LocalFlags() localFlags.SetInterspersed(false) s := &Server{ SimpleUsage: n, Long:        fmt.Sprintf("A server named %s which uses a cobra command", n), Run: func(s *Server, args []string) error { cobraServer = s cmd.SetOutput(s.hk.Out()) cmd.SetArgs(args) return cmd.Execute() }, flags: localFlags, } return s } </ins> func runFull(t *testing.T, args string) *result { buf := new(bytes.Buffer) hk := HyperKube{

When trying to use kubectl from hyperkube I'm met with "unknown flag" errors. It looks as if the flags aren't being passed through to kubectl but hyperkube is trying to handle them itself (and subsequently giving an error when you give it flags it doesn't understand). You can reproduce simply by running: And you'll see the error:
I have a fix up in PR .


Question: -s "http://127.0.0.1:${API_PORT}" --match-server-version ) <del> [ "$(kubectl get minions -t '{{ .apiVersion }}' "${kube_flags[@]}")" == "v1beta3" ] </del> <ins> [ "$(kubectl get nodes -t '{{ .apiVersion }}' "${kube_flags[@]}")" == "v1beta3" ] </ins> else kube_flags=( -s "http://127.0.0.1:${API_PORT}" --match-server-version --api-version="${version}" ) <del> [ "$(kubectl get minions -t '{{ .apiVersion }}' "${kube_flags[@]}")" == "${version}" ] </del> <ins> [ "$(kubectl get nodes -t '{{ .apiVersion }}' "${kube_flags[@]}")" == "${version}" ] </ins> fi id_field=".metadata.name" labels_field=".metadata.labels"

kubectl 0.17.0: curl http://localhost:8080/api/v1beta3/minions { "kind": "Status", "apiVersion": "v1beta3", "metadata": {}, "status": "Failure", "message": "the server could not find the requested resource", "reason": "NotFound", "details": {}, "code": 404 } master3 core # /opt/bin/kubectl get mi master3 core # kubectl 0.16.0: uses: http://localhost:8080/api/v1beta1/minions master3 core # ./kubectl get mi NAME LABELS STATUS 192.168.10.70 <noneReady 192.168.10.71 <noneReady in the same time curl http://localhost:8080/api/v1beta3/nodes works well.
Shall we just remove "minion" from kubectl now and throw an error?
remove "minions"?
Yes, that should be part of . cc
Presumably works?
Assigning who's working on
Ref
Could you please take a look at this?

I know that you can get nodes / a node, but the help doesn't say you can or provide examples: To resolve this issue: the help for this command to include nodes (and any other resources we're missing). we generate the help text in a way that ensures this (and probably the many others that are drifting) don't get out of date? Or at least give us a heuristic as to whether this might be true, even if we have to fix it manually?
Can this be closed?
I'll close this one with ; we want to get rid of the minions (mi) resource name, since it doesn't work for v1beta3.


Question: <ins> The MIT License (MIT) Copyright (c) 2018 QRI, Inc. Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. </ins>

we have one in k/k and one in k/utils, both need to be in LICENSES /assign
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/area code-organization
I think we should include anything matching the LICENSE/COPYRIGHT/COPYING patthers under or subdirs in k/k or dependencies... a quick sweep showed these:
/sig release ?


Question: } } <ins> func TestCalcNodeCidrSize(t *testing.T) { tests := []struct { name           string podSubnet      string expectedPrefix string }{ { name:           "Malformed pod subnet", podSubnet:      "10.10.10/160", expectedPrefix: "24", }, { name:           "V4: Always uses 24", podSubnet:      "10.10.10.10/16", expectedPrefix: "24", }, { name:           "V6: Use pod subnet size, when not enough space", podSubnet:      "2001:db8::/128", expectedPrefix: "128", }, { name:           "V6: Use pod subnet size, when not enough space", podSubnet:      "2001:db8::/113", expectedPrefix: "113", }, { name:           "V6: Special case with 256 nodes", podSubnet:      "2001:db8::/112", expectedPrefix: "120", }, { name:           "V6: Using /120 for node CIDR", podSubnet:      "2001:db8::/104", expectedPrefix: "120", }, { name:           "V6: Using /112 for node CIDR", podSubnet:      "2001:db8::/103", expectedPrefix: "112", }, { name:           "V6: Using /112 for node CIDR", podSubnet:      "2001:db8::/96", expectedPrefix: "112", }, { name:           "V6: Using /104 for node CIDR", podSubnet:      "2001:db8::/95", expectedPrefix: "104", }, { name:           "V6: Largest subnet currently supported", podSubnet:      "2001:db8::/66", expectedPrefix: "80", }, { name:           "V6: For /64 pod net, use /80", podSubnet:      "2001:db8::/64", expectedPrefix: "80", }, } for _, test := range tests { actualPrefix := calcNodeCidrSize(test.podSubnet) if actualPrefix != test.expectedPrefix { t.Errorf("Case [%s]nCalc of node CIDR size for pod subnet %q failed: Expected %q, saw %q", test.name, test.podSubnet, test.expectedPrefix, actualPrefix) } } } </ins> func TestGetControllerManagerCommandExternalCA(t *testing.T) { tests := []struct {

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: For IPv6, the node CIDR size is hard coded to /64. This causes an error bringing up the kube-controller, as the size must be for a smaller subnet (larger value) that what is used for the pod subnet, which is specified for some CNI plugins (e.g. Calico). In addition, this is a catch-22, as the pod subnet prefix cannot be for a larger network (smaller value) than /66, so we cannot set it to /63, for example. What you expected to happen: The node CIDR size should be computed, based on the pod subnet specified, so that it can meet the requirements. How to reproduce it (as minimally and precisely as possible): Bring up a cluster using KubeAdm, specifying IPv6 and choose any pod subnet prefix desired. Anything else we need to know?: Note, there is a second requirement, that the difference between pod subnet size and node CIDR size be 16 or less (issue ). I'll be submitting a proposed PR to address this. Environment: Kubernetes version (use ): latest on master, using 1.9.2 in my case Cloud provider or hardware configuration: bare metal, single node OS (e.g. from /etc/os-release): Ubuntu 16.04 Kernel (e.g. ): Linux bxb-c2-77 4.4.0-109-generic -Ubuntu SMP Tue Jan 9 19:52:39 UTC 2018 x8664 x8664 x86_64 GNU/Linux Install tools: kubeadm Others: /area ipv6 /sig network
/assign pmichali