Question: shouldContinue = false } } <del> case <-time.After(2 * time.Second): </del> <ins> case <-time.After(wait.ForeverTestTimeout): </ins> shouldContinue = false w2.Stop() }

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: bazel-test-master (periodic-kubernetes-bazel-test-master ) Which test(s) are flaking: gotest: Testgrid link: Reason for failure: Anything else we need to know: links to appreciated links to specific failures in spyglass appreciated <!-- Please see the deflaking doc () for more guidance! --cc
/sig api-machinery
This is continuing to be flaky with more recent failure:
/assign Would you mind taking a look here? thanks! /triage accepted
- can we somehow uncover when this started happening? This test is very old and it seems it started happening very recently, so it would be helpful to see when this started happening...
Triage shows flakes starting ~Sept 30: Before Sept 30 it seems the job was flaking much less frequently.
Might be related to I will take a look.
It seems it's not related to that PR (neither to the previous: ) As there were already failures like that on Sep 24th (before this one was merged). I will take a deeper look.
OK - it's not related to any changes mentioned before. I can reproduce this particular failure by simply adding somewhere in this block: So it's probably connected to cpu-starvation at the test level and the solution is to switch to less aggressive timeouts. Will send out a PR.
Actually - it appeared to be slightly more subtle then that. But I already fully understand the problem. It's purely test issue, but require a bit more changes (and I won't get to it today/tomorrow). I will send a fix for that early next week.
- if that changes anything from your tracking perspective ^^
No board changes yet - the next stage is "Observing" once this issue gets closed :) If you have time - I'd love to pick your brain/understand the error more thoroughly though! Hoping you drop a good explanation in your PR next week
Thanks for sharing those insights, on your discovery of the solution to this bug.
It seems the PR didn't address that fully. Looking.
out for review


Question: } }) }) <ins> ginkgo.Describe("kubectl wait", func() { ginkgo.It("should ignore not found error with --for=delete", func() { ginkgo.By("calling kubectl wait --for=delete") framework.RunKubectlOrDie(ns, "wait", "--for=delete", "pod/doesnotexist") framework.RunKubectlOrDie(ns, "wait", "--for=delete", "pod", "--selector=app.kubernetes.io/name=noexist") }) }) </ins> }) // Checks whether the output split by line contains the required elements.

What happened: given an empty namespace (no pods), if running: , kubectl errors with the message . What you expected to happen: kubectl should exit with status 0, since the condition to wait for is for all pods to be deleted, and that condition is already satisfied. How to reproduce it (as minimally and precisely as possible): run the example kubectl command Anything else we need to know?: Environment: Kubernetes version (use ): Taking the liberty of skipping other version information, since this is (I assume) purely a client-side bug
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cli
Seems like didn't fix it correctly: Test in this PR always passed because the always return nil error and visitCount == 1, and the type conversion always fail here:
thanks for the quick fix!
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale - still an issue, I believe the PR is waiting for re-review
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle rotten
/remove-lifecycle rotten

One use-case of is to wait for a lot of resources that you have just deleted. Unfortunately, this workflow is racy because will fail when encountering a resource that doesn't exist (yes, it's probably already gone). I suspect we should just consider 404 resources to be already gone and integrate that condition as part of the normal flow. /sig cli /kind bug
/assign
Duplicate of ?
yes, I tried to add in in But that PR seems not complete.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
The PR merged, I think we're good now!

Hi. A similar issue was reported here: but the bot had closed it due to inactivity. So, creating a new one with details again. I am facing the same problem on my end. If a resource does not exist or is deleted, but you are using wait --for condition delete, it fails. There is a flag in delete to avoid such errors. A similar condition of flag must be in as well so that it does not fail with errors like below. Thanks.
/sig cli
/assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Sorry I forget to do this, I'll try to submit a PR this week.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Yes, this would be nice to have.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at .

/lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-contributor-experience at . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Any chance this can be revisited? Thanks!


Question: kube::etcd::cleanup() { kill "${ETCD_PID-}" >/dev/null 2>&1 || : <ins> wait "${ETCD_PID-}" >/dev/null 2>&1 || : </ins> rm -rf "${ETCD_DIR-}" }

Integration tests run the same test for v1beta1 and v1beta3. Sometimes second run fails complaining: This causes some flakes in travis/shippable. We should wait for the clean shutdown of the first run and only then start the second one.


Question: ResourceVersion string `json:"resourceVersion,omitempty"` // Optional. If referring to a piece of an object instead of an entire object, this string <del> // should contain a valid field access statement. For example, // if the object reference is to a container within a pod, this would take on a value like: // "spec.containers[2]". Such statements are valid language constructs in // both go and JavaScript. This is syntax is chosen only to have some well-defined way of </del> <ins> // should contain information to identify the sub-object. For example, if the object // reference is to a container within a pod, this would take on a value like: // "spec.containers{name}" (where "name" refers to the name of the container that triggered // the event) or if no container name is specified "spec.containers[2]" (container with // index 2 in this pod). This syntax is chosen only to have some well-defined way of </ins> // referencing a part of an object. // TODO: this design is not final and this field is subject to change in the future. FieldPath string `json:"fieldPath,omitempty"`

Forked from ... The event generated from kubelet for BoundPod and displayed by kubectl should have Container(s) column ; otherwise, the user might think the entire pod is at the running state, but actually only one or some containers are running. When there are missing event(s) for failed containers, the thing gets worse. For example, in case, only net container is running. $ get events | grep testpd Running: cluster/../cluster/gce/../../_output/dockerized/bin/linux/amd64/kubectl get events Mon, 29 Dec 2014 20:58:38 +0000 testpd BoundPod waiting created Created with docker id eb3f53865a8825a3db3b64f6f319b64bfecbcbabc7115a73c2358c8c73793004 Mon, 29 Dec 2014 20:58:38 +0000 testpd Pod Pending scheduled Successfully assigned testpd to kubernetes-minion- Mon, 29 Dec 2014 20:58:38 +0000 testpd BoundPod waiting pulled Successfully pulled image kubernetes/pause:latest Mon, 29 Dec 2014 20:58:39 +0000 testpd BoundPod running started Started with docker id eb3f53865a8825a3db3b64f6f319b64bfecbcbabc7115a73c2358c8c73793004


Question: invalidSeccompDefault.Annotations = map[string]string{ seccomp.DefaultProfileAnnotationKey: "not-good", } <ins> invalidSeccompAllowAnyDefault := validPSP() invalidSeccompAllowAnyDefault.Annotations = map[string]string{ seccomp.DefaultProfileAnnotationKey: "*", } </ins> invalidSeccompAllowed := validPSP() invalidSeccompAllowed.Annotations = map[string]string{ seccomp.AllowedProfilesAnnotationKey: "docker/default,not-good",

It should be possible to create a PodSecurityPolicy with zero restrictions. This was the intent with the (a.k.a. "AllowAny") allowed seccomp profile, but it is rejected during the validation phase. /kind bug
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/auth: Issue will be escalated to these SIGs if needed. priority/important-soon: Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. kind/bug: Fixes a bug discovered during the current release. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details


Question: framework.ExpectNoError(err) newCfg := oldCfg.DeepCopy() updateFunction(newCfg) <ins> if reflect.DeepEqual(*newCfg, *oldCfg) { return } </ins> framework.ExpectNoError(setKubeletConfiguration(f, newCfg)) }) AfterEach(func() {

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Run e2e node test with , and wait for new configuration take effect. But it will return new configuration has taken effect. What you expected to happen: will return with new configuration has taken effect. How to reproduce it (as minimally and precisely as possible): Run any e2e node test with . Anything else we need to know?: This is because we use period = 10 * time.Second and jitterFactor = 0.2 in . In extreme scenarios, it will take at least 12 seconds for Kubelet to catch this new config. We use this to check whether kubelet is restarted: This will return OK immediately as has been set to etcd, but kubelet don't restart with this . This function will return before new configuration has taken effect. I will send a PR to fix this. Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: /area kubelet /sig node /assign /cc PTAL, Thanks!
The comment that will immediately contain the new (presumably changed) config prior to a Kubelet restart is incorrect. scrapes the values from the Kubelet's /configz endpoint, which won't have updated to the new configuration until after the Kubelet has restarted.
Hi, thanks for your explanation! I add some timestamps to test this, but the result is not the some as you described. For example, returned at , but Kubelet restarted at . And after reading the code, just call this API: , and i don't find any code about waiting for Kubelet has restarted. Could you point me these code please? Thanks very much!
Are you actually changing the configuration object's values? will return the old values until the Kubelet restarts (because /configz will return the old values until the Kubelet restarts). In any case where you actually change the config values, the assertion will fail until the Kubelet restarts, and thus you'll wait for the restart before continuing past the .

Thanks, I think this is because will read on starting, the configuration is always equal to the last time changed. So getCurrentKubeletConfig() will return immediately. But I have another question here: should we double check the is not equal to the before we call in ?
returns as soon as it gets a 200 response from /configz. Nothing to do with which config the Kubelet read. If the Eventually is exiting early, it's probably because no values were changed. The Kubelet may not have even restarted yet. That's probably a useful optimization.
Yes, I agree with you. Hi, , PTAL, Thanks.


Question: queue internalqueue.SchedulingQueue, pdbs []*policy.PodDisruptionBudget, ) (map[*v1.Node]*schedulerapi.Victims, error) { nodeToVictims := map[*v1.Node]*schedulerapi.Victims{} var resultLock sync.Mutex

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: kube-scheduler crashes (panics) in rare circumstances. Stack trace: What you expected to happen: flawless operation ;) How to reproduce it (as minimally and precisely as possible): Unclear. The error above looks like it was associated with some cloning of node metadata. Perhaps due to a race during node scale-up/scale-down and cache interactions. Anything else we need to know?: Environment: Kubernetes version (use ): <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
/sig scheduling
are you using hostPort for your workload?
is there a specific workload you had in mind? The does, as well as daemonsets for our service mesh, which are in host networking mode.
There was a PR () in v1.12/v1.11.3 fixing issue when workloads uses hostNetwork, and the solution is to correct the NodeInfo cloning logic. So it's quite likely related with the symptom you described. Could you give a try on v1.11.3?
BTW: the issue that PR fixed is only reproducible on some circumstances.
Thanks - I'll give this a whirl. I think I can optimistically close this issue - we'll reopen if it persists with a more recent autoscaler release. Cheers!
Sounds good! Thanks for the feedback.
we just had a similar issue this morning and our cluster is running kube 1.11.4: and these are the error logs : we do have hostNetwork work load running inside our cluster but not sure if that's caused by other issues since you mentioned that hostNetwork issue has been fixed since 1.11.3.
could you re-open this issue ? or I can raise a new issue for it.
/reopen I will take a look.
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
thanks
are the worker nodes auto scaled out/in? e.g. by an auto-scaler component?

yes, we do have an autoscaler to auto scale our worker nodes and we are using since most of our workloads are job based.
I see. It's likely an internal cache inconsistency. (not related with the hostPort issue I fixed earlier) /assign
One thing I'm curious is from the log it points to L386 of Then it means is nil, but should't give a panic trace at the first call inside ? Say: I looked through the code, but can't find a place that could be nil. So are you using raw v1.11.4-beta.0 branch, or did some modification? As in my opinion, it makes sense the panic comes from L385:
No, in Go if n.requestedResource is nil, you will get a stack trace similar to what is given in this issue.
I read it differently. I wrote an example , it shows deepest trace at L12, rather than L17.
I was actually wondering if it's possible that the stack trace isn't 100% line accurate.. Either that it systematically hits the next line in this case, or that the order in which field copies are executed isn't a simple 1:1 (linear) relationship to the code.
it's interesting, maybe it's some trick (or bug) in apimachinery's runtime.HandleCrash()? And I wrote a test to reproduce the issue, even the same line on the deepest stack trace: (although I think it should be line above)
/reopen I will raise cherrypick PRs to 1.12 and 1.11, and close this issue by then.
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hi i have scaled my cluster to 100 node from 40 nodes as there were 400 pending pods and for some of the nodes all pods get schedule properly while around 28 nodes have only dnsmasq and kube-proxy was running. after around 40 min those node had weave and other pods started running. i coundn't able to find any other issue in my cluster apart from kube-scheduler got restarted because same error mentino in the issue. though it restarted in few second.

i was just wondering that if after restart would it takes 30-40 min to schedule pods? cause after 40 min everything seems to be working properly. I am running kubernetes 1.12.1 and i have installed it using kops 1.10.0
fix of this issue is cherrypicked (PR ) into v1.12.4. Could you give a try on v1.12.4?
Close this as fix has been applied to 1.11+ k8s versions. /close
Closing this issue. <detailsIn response to : Close this as fix has been applied to 1.11+ k8s versions. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: }, }, } <del> return f.ClientSet.Extensions().DaemonSets(f.Namespace.Name).Create(loader) </del> <ins> _, err := f.ClientSet.Core().ReplicationControllers(f.Namespace.Name).Create(loader) framework.ExpectNoError(err, "Failed to create apparmor-loader ReplicationController") // Wait for loader to be ready. getRunningLoaderPod(f) } func getRunningLoaderPod(f *framework.Framework) *api.Pod { label := labels.SelectorFromSet(labels.Set(map[string]string{loaderLabelKey: loaderLabelValue})) pods, err := framework.WaitForPodsWithLabelScheduled(f.ClientSet, f.Namespace.Name, label) framework.ExpectNoError(err, "Failed to schedule apparmor-loader Pod") pod := &pods.Items[0] framework.ExpectNoError(framework.WaitForPodRunningInNamespace(f.ClientSet, pod), "Failed to run apparmor-loader Pod") return pod </ins> }

The "AppArmor load AppArmor profiles should enforce an AppArmor profile" e2e test in failing on our 5k-node gce clusters - https://k8s- (run no. 8) due to timeout: Seems like the test is trying to load the profiles across all the nodes using a daemonset and then running a test pod to check that. It's passing on our 2k-node clusters (https://k8s-) but failing for 5k - suggesting that it's likely due to size of the cluster. I think it's the ds creation taking longer on large cluster. cc
Seems like you're the most relevant here. Could one of you take a look into it? And does increasing the timeout for the test-pod based on no. of nodes sound reasonable?
Assigning it you since is OOO. Feel free to reassign as apt.
This is part of the correctness suite and large cluster tests are release-blocking.
FYI
Reassigning to who is back on Tuesday. He wrote the AppArmor stuff and should be able to debug this quickly. /assign
Ref
Could you manage to take a look into this?
Sorry, I meant to reply to this! I have a fix, just need to confirm it works. I think you're right about the DaemonSet timing out. There's no reason this needs to load the profiles on every node, so the fix just runs a single loader, and uses pod affinity to co-schedule the test pod.
Smooth!
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/node sig/scalability: Issue will be escalated to these SIGs if needed. priority/important-soon: Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. kind/bug: Fixes a bug discovered during the current release. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details


Question: for name, item := range table { fakeClient, registry := NewTestGenericEtcdRegistry(t) fakeClient.Data[path] = item.existing <del> err := registry.CreateWithName(api.NewContext(), key, item.toCreate) </del> <ins> err := registry.CreateWithName(api.NewDefaultContext(), key, item.toCreate) </ins> if !item.errOK(err) { t.Errorf("%v: unexpected error: %v", name, err) } <del> if e, a := item.expect, fakeClient.Data[path]; !api.Semantic.DeepDerivative(e, a) { t.Errorf("%v:n%s", name, util.ObjectDiff(e, a)) </del> <ins> actual := fakeClient.Data[path] if item.objOK != nil { obj, err := api.Scheme.Decode([]byte(actual.R.Node.Value)) if err != nil { t.Errorf("unable to decode stored value for %#v", actual) continue } if !item.objOK(obj) { t.Errorf("%v: unexpected response: %v", name, actual) } } else { if e, a := item.expect, actual; !api.Semantic.DeepDerivative(e, a) { t.Errorf("%v:n%s", name, util.ObjectDiff(e, a)) } </ins> } } } func TestEtcdUpdate(t *testing.T) { podA := &api.Pod{ <ins> ObjectMeta: api.ObjectMeta{Name: "foo", Namespace: api.NamespaceDefault}, Status:     api.PodStatus{Host: "machine"}, } podB := &api.Pod{ ObjectMeta: api.ObjectMeta{Name: "foo", Namespace: api.NamespaceDefault, ResourceVersion: "1"}, Status:     api.PodStatus{Host: "machine2"}, } nodeWithPodA := tools.EtcdResponseWithError{ R: &etcd.Response{ Node: &etcd.Node{ Value:         runtime.EncodeOrDie(testapi.Codec(), podA), ModifiedIndex: 1, CreatedIndex:  1, }, }, E: nil, } newerNodeWithPodA := tools.EtcdResponseWithError{ R: &etcd.Response{ Node: &etcd.Node{ Value:         runtime.EncodeOrDie(testapi.Codec(), podA), ModifiedIndex: 2, CreatedIndex:  1, }, }, E: nil, } nodeWithPodB := tools.EtcdResponseWithError{ R: &etcd.Response{ Node: &etcd.Node{ Value:         runtime.EncodeOrDie(testapi.Codec(), podB), ModifiedIndex: 1, CreatedIndex:  1, }, }, E: nil, } emptyNode := tools.EtcdResponseWithError{ R: &etcd.Response{}, E: tools.EtcdErrorNotFound, } path := "/registry/pods/foo" table := map[string]struct { existing    tools.EtcdResponseWithError expect      tools.EtcdResponseWithError toUpdate    runtime.Object allowCreate bool objOK       func(obj runtime.Object) bool errOK       func(error) bool }{ "normal": { existing: nodeWithPodA, expect:   nodeWithPodB, toUpdate: podB, errOK:    func(err error) bool { return err == nil }, }, "notExisting": { existing: emptyNode, expect:   emptyNode, toUpdate: podA, errOK:    func(err error) bool { return errors.IsNotFound(err) }, }, "createIfNotFound": { existing:    emptyNode, toUpdate:    podA, allowCreate: true, objOK:       hasCreated(t, podA), errOK:       func(err error) bool { return err == nil }, }, "outOfDate": { existing: newerNodeWithPodA, expect:   newerNodeWithPodA, toUpdate: podB, errOK:    func(err error) bool { return errors.IsConflict(err) }, }, } for name, item := range table { fakeClient, registry := NewTestGenericEtcdRegistry(t) registry.UpdateStrategy.(*testRESTStrategy).allowCreateOnUpdate = item.allowCreate fakeClient.Data[path] = item.existing obj, _, err := registry.Update(api.NewDefaultContext(), item.toUpdate) if !item.errOK(err) { t.Errorf("%v: unexpected error: %v", name, err) } actual := fakeClient.Data[path] if item.objOK != nil { if !item.objOK(obj) { t.Errorf("%v: unexpected returned: %#v", name, obj) } actualObj, err := api.Scheme.Decode([]byte(actual.R.Node.Value)) if err != nil { t.Errorf("unable to decode stored value for %#v", actual) continue } if !item.objOK(actualObj) { t.Errorf("%v: unexpected response: %#v", name, actual) } } else { if e, a := item.expect, actual; !api.Semantic.DeepDerivative(e, a) { t.Errorf("%v:n%s", name, util.ObjectDiff(e, a)) } } } } // DEPRECATED func TestEtcdUpdateWithName(t *testing.T) { podA := &api.Pod{ </ins> ObjectMeta: api.ObjectMeta{Name: "foo"}, Status:     api.PodStatus{Host: "machine"}, }

The use of AtomicUpdate by genericetcd doesn't handle conflicting writes by rejecting that to a user. It should. Eventually, we may want to hide write conflicts to /pods when status has changed, and to spec when /pods/status changes, but that requires us to have two version counters and that's not worth the effort. Fix incoming.
Also we need a unit/integration test for this.


Question: v1.FCVolumeSource{TargetWWNs: []string{wwnLun[0]}, Lun: &lun32}) glog.V(5).Infof("ConstructBlockVolumeSpec: TargetWWNs: %v, Lun: %v", fcPV.Spec.PersistentVolumeSource.FC.TargetWWNs, <del> fcPV.Spec.PersistentVolumeSource.FC.Lun) </del> <ins> *fcPV.Spec.PersistentVolumeSource.FC.Lun) </ins> } else { fcPV = createPersistentVolumeFromFCVolumeSource(volumeName, v1.FCVolumeSource{WWIDs: []string{volumeInfo}})

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Currently, FC plugin returns volume name and empty FCVolumeSource during ConstrutVolumeSpec during . However, in order to recreate newMounter from volumeSpec, volume information are lacking. As a result, filesystem volume's reconstruction always fails. What you expected to happen: Filesystem volume's reconstruction works properly, then a volume is unmounted and detached from kubelet node. How to reproduce it (as minimally and precisely as possible): (1) Create filesystem volume using FC plugin (2) Restart kubelet service which a pod with FC volume is running Anything else we need to know?: cc Environment: Kubernetes version (use ):Install tools: local-cluster-
/assign
/sig storage


Question: managedDiskPathRE   = regexp.MustCompile(`.*/subscriptions/(?:.*)/resourceGroups/(?:.*)/providers/Microsoft.Compute/disks/(.+)`) unmanagedDiskPathRE = regexp.MustCompile(`http(?:.*)://(?:.*)/vhds/(.+)`) managed             = string(v1.AzureManagedDisk) <ins> unzonedCSIRegionRE  = regexp.MustCompile(`^[0-9]+$`) </ins> ) var _ InTreePlugin = &azureDiskCSITranslator{}

This test fails on Kubernetes cluster in "westus" region (without any number) and with CSI migration enabled: Error: failed to provision volume with StorageClass "e2e-topology-84vk7sp": error generating accessibility requirements: topology map[] from selected node "ci-op-9j1nby77--9zlwv-worker-westus-ljw4g" is not in requisite: [map[]] StorageClass generated by the test: PVC created by the test: <details</detailsLabels of the selected node: The external-provisioner does not even call the CSI driver and reports this event: The volume should get provisioned. See above, "westus" region with no availability zones seems to be the key. It looks like the translation from in-tree StorageClass to CSI one translates topology requirements to , but the CSI one expects <details</details<details</details<detailsOpenShift 4.10 </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/priority important-soon cc
/sig storage
per https://kubernetes-, is not a zone, it's actually a fault-domain num. valid zone format is , could you change the storage class with setting, and try again? Thanks.
o, if "westus" region does not support zone, then I think you should test setting in another zone, make sure you have created a zoned node first.
I have as SC that works without CSI migration, with CSI migration it does not work. That's a regression. It should not matter if the SC is somehow odd. The labels are selected by test code, some other production code might use these labels too.
Out of curiosity, how reliable are zone numbers in Azure? I can see .. in a region with zones. If is used only in regions without a zone, then the translation can translate to , which the CSI driver expects.
I would say that's a wrong usage for zone.

following config would create a pv that could only be scheduled to a non-zone node with fault domain , if there is no node supports zone, then should not be set.
But nodes report that they are in , therefore it make sense to accept the same input in StorageClass.
if we translation "0" to "", then the pv created by that sc could be scheduled to any non zone node with any fault domain. That's a corner case that cluster is not in zone, and user still wants to set in sc
I am not sure I understand. If I provision a CSI volume in the westus cluster, its PV gets anyway: I have the cluster only with nodes in "westus" region, so all of them are without any zone. What would the CSI driver do if there were nodes in "westus" and say "ukwest", which does not have zones either? I can't test that setup easily, but I think the driver would report for both of them anyway, hence translating to does not make things worse, I think.
/sig storage /sig cloud-provider /area provider/azure
can you please check my previous comment? In what cases would be translating "0" to "" worse than the translation code we have today?
it's not worse or better, a single num () is actually not a valid zone, such in sc is actually not valid.
I mean we could translate the single digit to to make it work while the usage is wrong.
/sig storage
/sig cloud-provider


Question: run(ctx, startSATokenController, initializersFunc) }, OnStoppedLeading: func() { <del> klog.Fatalf("leaderelection lost") </del> <ins> klog.ErrorS(nil, "leaderelection lost") klog.FlushAndExit(klog.ExitFlushTimeout, 1) </ins> }, })

kube-controller-manager yields whole stack trace when it loses leaderelection. kube-controller-manager to only log and to exit (without logging 17K lines of stack trace that is not useful for anyone). kube-controller-manager yields around 17K lines of stack trace (omitted in the above example). No response <detailsv1.20.12 </details<details</details<details</details<details</details<details</details<details</details
/sig api-machinery
/assign /triage accepted
has Fatalf always triggered a stack dump of all goroutines?
/cc
Seems like does a dump of all goroutines Could we and then maybe? /cc in case you have thoughts about this
If I read the right, it only printed the current goroutine to stderr. Dumping everything seems to have been later. make sense to me.
Dumping of all goroutines was in
I have similar doubts. I want to fix it. /assign
kube-scheduler seems to have a similar issue (but a shorter stack trace)
When replacing (useful because it doesn't do structured logging, not just because of the stack trace problem), remember to use + the new . Updated migration instructions are pending in There's also - reverting the klog change might be a faster way to address this.


Question: // - a pod was created with a PV that's supposed to have data // // However there's a caching issue that @jinxu97 explained and it's related with the pod & volume <del> // lifecycle in windows, to understand it we first analyze what the volumemanager does: </del> <ins> // lifecycle, to understand it we first analyze what the volumemanager does: </ins> // - when a pod is delete the volumemanager will try to cleanup the volume mounts //   - NodeUnpublishVolume: unbinds the bind mount from the container <del> //     - Linux: the data is flushed to disk </del> <ins> //     - Linux: the bind mount is removed, which does not flush any cache </ins> //     - Windows: we delete a symlink, data's not flushed yet to disk //   - NodeUnstageVolume: unmount the global mount <del> //     - Linux: disk is detached </del> <ins> //     - Linux: disk is unmounted and all caches flushed. </ins> //     - Windows: data is flushed to disk and the disk is detached // // Pod deletion might not guarantee a data flush to disk, however NodeUnstageVolume adds the logic <del> // to flush the data to disk (see #81690 for details). </del> <ins> // to flush the data to disk (see #81690 for details). We need to wait for NodeUnstageVolume, as // NodeUnpublishVolume only removes the bind mount, which doesn't force the caches to flush. // It's possible to create empty snapshots if we don't wait (see #101279 for details). </ins> // // In the following code by checking if the PV is not in the node.Status.VolumesInUse field we // ensure that the volume is not used by the node anymore (an indicator that NodeUnstageVolume has // already finished) <del> if framework.NodeOSDistroIs("windows") { nodeName := pod.Spec.NodeName gomega.Expect(nodeName).NotTo(gomega.BeEmpty(), "pod.Spec.NodeName must not be empty") </del> <ins> nodeName := pod.Spec.NodeName gomega.Expect(nodeName).NotTo(gomega.BeEmpty(), "pod.Spec.NodeName must not be empty") </ins> <del> ginkgo.By(fmt.Sprintf("[init] waiting until the node=%s is not using the volume=%s", nodeName, pv.Name)) success := storageutils.WaitUntil(framework.Poll, f.Timeouts.PVDelete, func() bool { node, err := cs.CoreV1().Nodes().Get(context.TODO(), nodeName, metav1.GetOptions{}) framework.ExpectNoError(err) volumesInUse := node.Status.VolumesInUse framework.Logf("current volumes in use: %+v", volumesInUse) for i := 0; i < len(volumesInUse); i++ { if strings.HasSuffix(string(volumesInUse[i]), pv.Name) { return false } </del> <ins> ginkgo.By(fmt.Sprintf("[init] waiting until the node=%s is not using the volume=%s", nodeName, pv.Name)) success := storageutils.WaitUntil(framework.Poll, f.Timeouts.PVDelete, func() bool { node, err := cs.CoreV1().Nodes().Get(context.TODO(), nodeName, metav1.GetOptions{}) framework.ExpectNoError(err) volumesInUse := node.Status.VolumesInUse framework.Logf("current volumes in use: %+v", volumesInUse) for i := 0; i < len(volumesInUse); i++ { if strings.HasSuffix(string(volumesInUse[i]), pv.Name) { return false </ins> } <del> return true }) framework.ExpectEqual(success, true) } </del> <ins> } return true }) framework.ExpectEqual(success, true) </ins> } cleanup := func() {

The volume snapshot controller tests can result in an empty VolumeSnapshot being taken if NodeUnstageVolume occurs after the VolumeSnapshot is taken. Previously there was a change made in Windows to ensure that NodeUnstageVolume finishes before continuing with the test; however, this needs to also to be included in other distros as well. NodeUnpublish, which removes the bind mount, does not force the caches to flush. Tests - The CI jobs should pass when running with a non-hostpath driver. I'm uncertain on why we've seen an increase in these failures, but running the following tests with a non-hostpath driver appear to flake regularly: [Testpattern: Pre-provisioned Snapshot (retain policy)] snapshottable[Feature:VolumeSnapshotDataSource] volume snapshot controller should check snapshot fields, check restore correctly works after modifying source data, check deletion [Testpattern: Pre-provisioned Snapshot (delete policy)] snapshottable[Feature:VolumeSnapshotDataSource] volume snapshot controller should check snapshot fields, check restore correctly works after modifying source data, check deletion [Testpattern: Dynamic Snapshot (retain policy)] snapshottable[Feature:VolumeSnapshotDataSource] volume snapshot controller should check snapshot fields, check restore correctly works after modifying source data, check deletion [Testpattern: Dynamic Snapshot (delete policy)] snapshottable[Feature:VolumeSnapshotDataSource] volume snapshot controller should check snapshot fields, check restore correctly works after modifying source data, check deletion In these tests we've seen the following with the AWS EBS CSI Driver. Driver logs on the node: In the driver logs (in the controller pod) I can see:
/sig storage
/triage accepted /priority important-soon all snapshot tests with real CSI drivers are quite flaky, I saw it at least in AWS, Azure and GCE in a downstream CI job


Question: cadvisorapi "github.com/google/cadvisor/info/v1" v1 "k8s.io/api/core/v1" <ins> apierrors "k8s.io/apimachinery/pkg/api/errors" </ins> metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/fields" "k8s.io/apimachinery/pkg/labels"

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: I recently saw an issue in which the scheduler hit this log line many times: After some digging, it turns out that PodAffinityChecker has a and . The points at the scheduler's cache instance, while the points at a light wrapper around a . In the scheduler, we use the cache to provide a globally consistent view of the cluster, even if that view is slightly out of date. Data read from the core may not have been processed by the scheduler cache yet. Letting data "skip ahead" like this can cause stale info to get persisted in the scheduler cache, and other subtle bugs. While the original problem is not likely caused by this issue, I think it needs to be cleaned up to avoid potential bugs down the line. What you expected to happen: Predicates should only read state from the scheduler's cache in order to maintain consistency. How to reproduce it (as minimally and precisely as possible): This was observed in a cluster with high rate of pod creation and node creation/deletion. Anything else we need to know?: Environment: Kubernetes version (use ): 1.12.5 Cloud provider or hardware configuration: GKE with preemptible nodes OS (e.g: ): Kernel (e.g. ): Install tools: Others:
/sig scheduling
We do have this kind of issues, and here is another one:
I'm not sure the urgency of this fix, but if its not super urgent, I would love to pick it up, if there's an eng more experienced with the scheduler who could give me some guidance :)
Hi I dug into the source code, and find out is used by the scheduler and kubelet. Maybe we could change the internal struct of , but I do not know why it appears in kubelet which seems irrelevant to the scheduler.
is just a wrapper of corelisters.NodeLister. kubelet can get node info by it.
I think the point is that the kubernetes scheduler was designed to support single instance at the beginning. Even with the addition of multiple scheduler support, the cache between each scheduler is isolated. So at this stage, we can accept this without a shared state solution.

After all, the node information is not actively modified by the kubernetes scheduler.
So is it necessary to ensure the consistency by using node lister instead of cache in the scheduler, or we do not need to fix this issue which seems like to caused by inconsistent cache.
We've recently hit this issue with the scheduler. It appears that our scheduler somehow gets out of sync with the nodes, and start scheduling pods on tainted nodes. The node's kubelet then (rightfully) refuse to admit the pod. Looking at the scheduler logs, they are flooded with the following : Similar scenario, this is a high churn cluster with thousands of pods scheduled per minute - and nodes being autoscaled. Can we try to get a fix ?
I'm glad to pick this up if we decide to fix this issue.
It would be awesome if you could fix this issue for us, we are currently seeing 100's of pods incorrectly being scheduled onto nodes with taints.
The code cited for logging "Node not found, %v" is now in :
/assign
Thanks for the interest on fixing this bug. Sorry I haven't had time to engage on this issue. Feel free to send PRs my way. I'm not 100% certain that this will fix the problem of pods getting scheduled to tainted nodes. How is the taint being applied to the node? Even if the scheduler is internally consistent, there could be a time between node creation and the taint being applied where we send pods to that node.


Question: LOG_FILE="${LOG_FILE:-${TMP_DIR}/update-vendor.log}" kube::log::status "logfile at ${LOG_FILE}" <ins> function finish { ret=$? if [[ ${ret} != 0 ]]; then echo "An error has occurred. Please see more details in ${LOG_FILE}" fi exit ${ret} } trap finish EXIT </ins> if [ -z "${BASH_XTRACEFD:-}" ]; then exec 19> "${LOG_FILE}" export BASH_XTRACEFD="19"

Seen while running on If hack/update- fails, the output should make it clear Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /area code-organization /sig architecture /cc /assign
Maybe we can add a to check the exit code.
/cc
something like that seems like a good idea
usually when we want to do stuff like that we instead record the exit code for specific steps with something like so we can check after and inform the user of the specific failure (and suggest what to do), then
/assign


Question: }, "time": { "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.Time", <del> "description": "Time is timestamp of when these fields were set. It should always be empty if Operation is 'Apply'" </del> <ins> "description": "Time is the timestamp of when the ManagedFields entry was added. The timestamp will also be updated if a field is added, the manager changes any of the owned fields value or removes a field. The timestamp does not update when a field is removed from the entry because another manager took it over." </ins> } }, "type": "object"

Godoc says that Time should be empty on mangedFields entry created by "Apply" operation, but kubernetes populates time on apply. Godoc consistent with kubernetes behaviour Godoc: Kubernetes: No response <details</details<details</details<details</details<details</details<details</details<details</details
/wg api-expression /sig api-machinery
/cc
/cc /triage accepted
Yeah, good catch. The comment matches the behavior from alpha, which we changed later on. Thanks for letting us now :-) I think we could do a better job describing how this value is changed now.


Question: } type GeneratableKeyCert struct { <ins> // CertKey allows setting an explicit cert/key file to use. </ins> CertKey CertKey <del> // CertDirectory is a directory that will contain the certificates.  If the cert and key aren't specifically set // this will be used to derive a match with the "pair-name" </del> <ins> // CertDirectory specifies a directory to write generated certificates to if CertFile/KeyFile aren't explicitly set. // PairName is used to determine the filenames within CertDirectory. // If CertDirectory and PairName are not set, an in-memory certificate will be generated. </ins> CertDirectory string <ins> // PairName is the name which will be used with CertDirectory to make a cert and key filenames. // It becomes CertDirectory/PairName.crt and CertDirectory/PairName.key PairName string // GeneratedCert holds an in-memory generated certificate if CertFile/KeyFile aren't explicitly set, and CertDirectory/PairName are not set. GeneratedCert *tls.Certificate </ins> // FixtureDirectory is a directory that contains test fixture used to avoid regeneration of certs during tests. // The format is: // <host>_<ip>-<ip>_<alternateDNS>-<alternateDNS>.crt // <host>_<ip>-<ip>_<alternateDNS>-<alternateDNS>.key FixtureDirectory string <del> // PairName is the name which will be used with CertDirectory to make a cert and key names // It becomes CertDirector/PairName.crt and CertDirector/PairName.key PairName string </del> } func NewSecureServingOptions() *SecureServingOptions {

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --BUG REPORT: /kind bug What happened: Kubernetes distro testing v1.12.0-rc.2 shows in setups with self-hosted control planes (i.e. apiserver, controller-manager, scheduler run as pods) that previously worked on v1.11.3 and prior, the kube-controller-manager now crash loops. Notably, the kube-controller-manager is run with: which has been fine through to v1.11.3. Full . What you expected to happen: Controller manager may not assume it has root. Or should clarify if an additional mount is now required. May have been introduced in cc How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): v1.12.0-rc.2 Cloud provider or hardware configuration: GCE, DigitalOcean, AWS OS (e.g. from /etc/os-release): Container Linux 1855.4.0 Kernel (e.g. ): 4.14.67-coreos Install tools: Typhoon
Seems like distros like Typhoon, Tectonic, and Tectonic/Openshift may work around this explicitly passing some other path to kube-controller-manager's or creating an emptyDir mount for . Need to understand the expectations around these certs (multiple pods, restarts, etc.) A release note seems needed to cover the new requirements. Appears this was also noticed by juju in
/sig release
cc
WDYT? Can we get folks to write a note to cover this problem in known issues?
I've an mount to address that crashloop, which seems to work preliminary (). was able to proceed with self-signed cert creation and hit a new bug I've filed separately EDIT: Perhaps a suitable release note for this change is:
the cert should not need to be persisted to disk.

I don't think adding a writeable directory requirement just for a self-signed cert is a very good idea /assign
Possible fixes enable by default if no cert is provided, rather than self-signing the self-sign cert in memory only, no disk persistence needed 't enable by default at all 2 would be ideal but is likely too big a change at this point. 3 would be unfortunate. 1 could be a good compromise if the code change is contained.
It's up to the SIG if this is a 1.12.0 blocker, or a Known Issue and 1.12.1 material. I'm inclined to think the latter.
thanks for the clarifications. I would prefer to avoid my current workaround of giving it a writeable location so those all sound fine to me. is there a full story around these controller-manager changes somewhere? I could mint actual certs, mount from a secret, update clients. If there's merit. Like, how big of a deal is it that this is now always enabled vs before? I'll probably read through the various diffs after vacation.
If this is going to get some text in release notes Known Issues (seems like it should), somebody needs to draft the text and make a suggestion of it in ASAP.
/assign
Do we want to fix this for 1.12.1 (on friday)?
Self signing in memory means you get a different cert each time it starts. It's probably better than nothing but it's not how things are supposed to work (it's too inconvenient to manually verify the cert out of band in places users should trust it). OTOH if you want to do that, maybe you'll provide a writable directory in the first place, or even a real cert. So I guess in-memory is probably OK.
Second thought: is a terrible default for the cert path. It should have the word "certs" in the path somewhere. If folks (inadvisably) run apiserver and controller manager on the same system, they should at least not both try to use certs from the same place, or worse, write them there. I suggest as the default.
What components are now verifying these certs or supposed to be? I'd expect something to be failing since random certs are generated each time the pod restarts, but don't see any impact (implying this isn't yet improving security).

Seems like controller-manager just an HTTPS endpoint, but the HTTP one is still what's used right?
I guess that aligns with the comment
The autogenerated certs only have SANs for localhost and 127.0.0.1.
The auto generated certs are self signed, right? If so, and callers won't be able to validate the signer, which SANs they have don't really matter.
Yeah, they're self-signed. So looks like the 1.12 controller-manager serves on HTTP/ (always on) and HTTPS/ (unless --secure-port=0). I guess the point of the original change was so some client could now choose to use the HTTPS endpoint with insecureSkipVerify. Main reason I was interested in the SAN is to get some clue about how a client would be expected to access these controller-manager pods (for contrast, apiserver SAN has DNS:kubernetes and an internal record that always resolves to a kube-apiserve pod).
is this updated for the openstack cloud provider too?
This issue wasn't related to any particular cloud provider. The cloud provider / hardware config field is just a typical part of filing an issue in case it had been related.
I have a writeable (I've tried other writeable directories too.) I'm encountering this: Should I post elsewhere in a new thread, or continue here?


Question: cmd := &cobra.Command{ Use:     "use-context CONTEXT_NAME", Short:   i18n.T("Sets the current-context in a kubeconfig file"), <ins> Aliases: []string{"use"}, </ins> Long:    `Sets the current-context in a kubeconfig file`, Example: use_context_example, Run: func(cmd *cobra.Command, args []string) {

The '-context' does not seem to add any value
add priority and team please.
Blocked on
/sig cli


Question: if node != nil { zone := nodetopology.GetZoneKey(node) evictionsNumber.WithLabelValues(zone).Inc() <ins> evictionsTotal.WithLabelValues(zone).Inc() </ins> } return true, 0

I would like to propose the graduation of the metric from alpha to stable. The metric has been stable for 2 years now.
/sig api-machinery /sig instrumentation
I am willing to do all StabilityLevels from alpha to stable? Is that right?
/assign
- The request is to have the stability level for the metric graduate to stable. If there are other metrics you want to do at the same time that's a bonus, but not part of the request in this issue.
/sig node /remove-sig api-machinery /cc
To be promoted to stable, the metrics would need to be renamed from to in order to follow Prometheus best practices as well as sig-instrumentation naming guidelines.
/cc
/assign
/triage accepted


Question: ":(glob)${root}"/'**/zz_generated.defaults.go'  | xargs -0 rm -f <del> local inputs=() </del> <ins> local input_args=() </ins> for arg in "${input_pkgs[@]}"; do <del> inputs+=("--input-dirs" "$arg") </del> <ins> input_args+=("--input-dirs" "$arg") </ins> done "${gobin}/defaulter-gen"  -v "${v}"  -O zz_generated.defaults  --go-header-file "${boilerplate}"  --output-base "${out_base}"  <del> "${inputs[@]}" </del> <ins> "${input_args[@]}" </ins> fi # Conversions

Earlier it was possible to pass flag to while calling script. Now with the new script, is called under which doesnot support any other flags apart from , and . As it can be seen here there is no option to pass flag to conversion-gen Earlier in the script, it was considering all the inputs passed flag is supported by under the hood, but the function in script is not allowing the flag.
/cc
/sig api-machinery
/assign for code generation /triage accepted
/assign
Can you explain why you need this flag? What I found in k/k was that we don't actually need it at all.
You know what, my memory was faulty. We do use it (until I fix that, which is in a development branch, not merged). But we don't use it in any of the "submodule" configs.
A lot of the "magic" behind the conversion logic is very brittle. Can you help me understand where / how this impacts you?
I whipped up a quick patch but have not tested much:
We are using the flag several places in our script On removal of the flag, the autoConvert functions in. files has the error : Althoug I can see the function is already present at . But there is also a comment like Hence, I think there is a need for the in order to use this pkg.
I see. The component-base is something I didn't account for. If you apply the above patch, does the new argument to work for you?
Above patch throws an error for the cases when flag is not passed to function. Instead I followed this approach :
I'd rather not accumulate into a string - can you tell me what error it throws? Note, mine wants Also - is the argument here a dir (e.g. /home/me/src/foo/bar) or a pkg (e.g. ) ? I think it's actually a pkg, right?
You implementation throws error for the cases when we don't specify the option while calling , as shown below: For such cases I am getting following error
It seems to be an old-bash problem. Try this one?
Yes this is working fine.
OK. I'll prep a PR soon.
Thank you


Question: if err := addPriorityClasses(ctrl, test.classesBefore); err != nil { t.Errorf("Test %q: unable to add object to informer: %v", test.name, err) } <del> defaultPriority, err := ctrl.getDefaultPriority() </del> <ins> pcName, defaultPriority, err := ctrl.getDefaultPriority() </ins> if err != nil { t.Errorf("Test %q: unexpected error while getting default priority: %v", test.name, err) } <del> if err == nil && defaultPriority != test.expectedDefaultBefore { t.Errorf("Test %q: expected default priority %d, but got %d", test.name, test.expectedDefaultBefore, defaultPriority) </del> <ins> if err == nil && (defaultPriority != test.expectedDefaultBefore || pcName != test.expectedDefaultNameBefore) { t.Errorf("Test %q: expected default priority %s(%d), but got %s(%d)", test.name, test.expectedDefaultNameBefore, test.expectedDefaultBefore, pcName, defaultPriority) </ins> } if test.attributes != nil { err := ctrl.Validate(test.attributes)

Is this a BUG REPORT or FEATURE REQUEST?: /kind cleanup /sig scheduling Description: If Pod did not set , admission controller will set to "default value". If the default value are from default priority class, it's better to also set of pod.
can I take this? /assign
, thanks :) I created a PR for functionality, but that'll be great if you can help add some e2e test for it. I can help to shepherd :)
thanks let me have a try :)


Question: By("Registering the sample API server.") cleanTest(f, true) client := f.ClientSet <ins> restClient := client.Discovery().RESTClient() </ins> iclient := f.InternalClientset aggrclient := f.AggregatorClient

1 tests failed, 11 jobs failed, 241 builds failed. Failure stats cover 1 day time range '17 Aug 2017 22:57 UTC' to '18 Aug 2017 22:57 UTC'.
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
Seeing this failure on the e2e release-master-blocking https://k8s- Example failure https://k8s- Looks like an RBAC issue cc
this is currently blocking the alpha.3 release
looks like this new test was by in which was merged about 6 days ago and is failing consistently
This has started failing again on our GKE test suite https://k8s- https://k8s- cc
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/api-machinery: Issue will be escalated to these SIGs if needed. priority/critical-urgent: Never automatically move out of a release milestone; continually escalate to contributor and SIG through all available channels. kind/bug: Fixes a bug discovered during the current release. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details
/assign
Seems to be flaking now with the error: https://k8s-
- we need to determine (today if possible) if this is truly release blocking. If so, please add the release-blocker label. And, if not, how do we best continue work on this for 1.8.x/1.9.0?
this test has two passes and four failures on the same commit I'm seeing gke-specific authz grants in that test that are incorrect: That commit removed the wait for the authz grant to become effective (which can lead to flakes), and granted superuser permissions to all users, which is incorrect and invalidates any other authz-related tests run in parallel with this test
cc
I can add the wait back.

edit: and fix the test given admin to all authenticated users.
Actually after staring at this test for about half an hour I can't figure out what different users exist or what permissions they're being granted. ClientSet, InternalClientset, and AggregatorClient are all initialized from the same config so I don't see how one would be able to create an RBAC binding but another would fail later. any thoughts here?
I honestly think the gke specific BindClusterRole is a red-herring. It is needed so the client has permission to perform one of the setup steps. (I think it was either to create the wardler cluster role or to bind that role to the anonymous user) Once that setup step is complete we no longer need the that cluster role bound and so I don't think its related.
The gke authorizer allows the “bind” verb, so the client can create a binding to the cluster-admin. It cannot create a role directly unless it has permissions via RBAC. Since we don’t have a way to determine the username associated with , binding to all authenticated users is what was done as a workaround.
I agree that the point at which the tests are failing indicate that the previous authorization issues are not the cause.
/open
/reopen
So a lot more information to work with now but the error is still occurring. I am still looking into this.
any update on the investigation?
Friendly v1.8 release team ping. This failure still seems to be happening, is this actively being worked? Does this need to be in the v1.8 milestone?


Question: if forgetErr := sched.Cache().ForgetPod(assumedPod); forgetErr != nil { klog.Errorf("scheduler cache ForgetPod failed: %v", forgetErr) } <del> sched.recordSchedulingFailure(assumedPod, permitStatus.AsError(), reason, permitStatus.Message()) </del> // trigger un-reserve plugins to clean up state associated with the reserved Pod fwk.RunUnreservePlugins(pluginContext, assumedPod, scheduleResult.SuggestedHost) <ins> sched.recordSchedulingFailure(assumedPod, permitStatus.AsError(), reason, permitStatus.Message()) </ins> return }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: TestUnreservePlugin is flaky: What you expected to happen: Test to always pass. How to reproduce it (as minimally and precisely as possible): This is a flaky test, but the issue is that the unreserve plugin didn't get the change to execute before the test checked the unreserve call counter in the test. Assuming thread1 runs the scheduler, thread2 runs the tests, here is the sequence that caused this flake: thread2 creates a pod and blocks on a . thread1 tries to schedule the pod, executing the pre-bind plugin which denies the pod and . thread2 gets unblocked before thread1 gets to . Thread2 checks the counters, throwing an error since unreserve plugin call counter didn't get to be incremented. What we need is for the test to block until unreserve is called, not until we get a scheduling error. /sig scheduling
/help
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help </details
cc/


Question: // make mountpoint rw/ro work as expected //FIXME revisit pkg/util/mount and ensure rw/ro is implemented as expected mode := "rw" <del> if rbd.readOnly { </del> <ins> if rbd.ReadOnly { </ins> mode = "ro" } rbd.plugin.execCommand("mount", []string{"-o", "remount," + mode, globalPDPath, dir})

etcd cluster backing apiserver died spontaneously, after restart apiserver pulled up the pods on different machines. Node storage was backed by ext4 over rbd, which became mounted on several nodes at once. That could have possibly led to data corruption. I guess that the proper way to mount rbd volume is to take a lock on it first, which will at least fail if it's locked somewhere else, requiring manual intervention, but not destroying data. The lock should be optional, in case a proper cluster fs is mounted over the rbd. In this case there was clearly no need for accessing the data from several pods so ext4 is deemed a proper fs.
, Vladimir Pouzanov < :
want to get some detail. How is your pod-to-rbd mapping like? Since rbd image is a block device, it can be used by one pod at a time. I am not sure if locking is a proper way to go - if the pod holds a lock but dies in the middle of mounting, the rbd image (or any other volume type) is then excluded from access. In such case, a high level coordination at apiserver is a better place to resolve this problem.
Here's my rc: I guess it's better to have a lock to be cleaned up manually rather than to have non-cluster fs to be mounted in several locations at once?
thanks, this pod looks good to me. what would cause multiple nodes run the same pod?
I don't know the exact reason of why kubelet didn't unmount the old fs, but after apiserver and scheduler were restarted pod got allocated on different node.
It is a scary thing to see etcd down, it is meant to be reliable...
Unfortunately I don't have long-term logging in test cluster so I have no idea why it actually died. but using kubedns as primary node resolver was clearly a bad idea, once it died, nodes stuck with no resolver at all.
your thought on HA?
Wrong Tim St. Clair?
- you HAVE to be ready to handle network partitions - the old node might still be alive and even using the volume, but the apiserver can't see the node, so it thinks it deletes the pod and starts it elsewhere. If RBD corrupts itslef in this case, it is not useful and we should strip it from the codebase.

, Huamin Chen wrote:
that makes sense. rbd doesn't corrupt itself. This is inconsistency happens when you have two pods mounting the same rbd image. How does GCE PD handle/prevent this case?
It doesn't allow more than one read-write attachment. Multiple read-only attachments is OK. Better to fail than to allow corruption. , Huamin Chen wrote:
That's neat. I have a lock fencing for rbd, testing now.


Question: // skipPodUpdate checks whether the specified pod update should be ignored. // This function will return true if //   - The pod has already been assumed, AND <del> //   - The pod has only its ResourceVersion, Spec.NodeName, Annotations, ManagedFields and/or Finalizers //     updated. </del> <ins> //   - The pod has only its ResourceVersion, Spec.NodeName, Annotations, //     ManagedFields, Finalizers and/or Conditions updated. </ins> func (sched *Scheduler) skipPodUpdate(pod *v1.Pod) bool { // Non-assumed pods should never be skipped. isAssumed, err := sched.SchedulerCache.IsAssumedPod(pod)

What happened: An extra scheduling attempt is observed in some scenarios when kube-scheduler already has already assumed a pod and this has entered the binding routine. If, during this time, a pod update is received, the pod is back to the scheduling queue, even if binding succeeds. This is the correct behavior, as scheduling conditions might change. However this updates could just be unrelated status updates, such as updates to pod conditions. We observe an extra scheduling attempt, which is visible in the log line: Which indicates that an attempt was made when the pod was already scheduled. What you expected to happen: No extra scheduling attempts How to reproduce it (as minimally and precisely as possible): This depends on other controllers running on the cluster that might report pod conditions. One such controller is
/sig scheduling


Question: testName                 string subsets                  []corev1.EndpointSubset epLabels                 map[string]string <ins> epAnnotations            map[string]string </ins> endpointsDeletionPending bool maxEndpointsPerSubset    int32 existingEndpointSlices   []*discovery.EndpointSlice

If I modify a custom endpoint, the kubectl annotation is mirrored to the generated endpoint slice Since the kubect last-applied-configuration tracks the status of the object, it shouldn't be mirrored since it doesn't apply to the generated object a custom endpoint an update on any of the fields the generated endpoint slice annotations /sig network
ping
if we agree this is a bug, the fix is just and adding some tests of course :)
Thanks for catching this! Agree this is a bug that we should fix.
/triage accepted
/good-first-issue
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hey, I'm on it!
Hi While I have made the update mentioned in the above comment, I feel a bit lost on where to add/update tests. I understand that I probably shouldn't raise a PR without trying to add a few tests for this issue. So could you possibly guide me to the correct files(s) I should be looking into where the annotations for updated EndpointSlices are being tested?
Hey I think the easiest place to add tests might be here: You could add another test case where the Endpoints object had this label and then ensure that label was not copied over to an EndpointSlice.
Alright I'll check out these tests. Thank you!
/assign


Question: // The CSIDriverRegistry feature gate is needed for this test in Kubernetes 1.12. Context("CSI attach test using mock driver [Feature:CSIDriverRegistry]", func() { tests := []struct { <del> name             string driverAttachable bool deployDriverCRD  bool </del> <ins> name            string disableAttach   bool deployDriverCRD bool </ins> }{ { <del> name:             "should not require VolumeAttach for drivers without attachment", driverAttachable: false, deployDriverCRD:  true, </del> <ins> name:            "should not require VolumeAttach for drivers without attachment", disableAttach:   true, deployDriverCRD: true, </ins> }, { <del> name:             "should require VolumeAttach for drivers with attachment", driverAttachable: true, deployDriverCRD:  true, </del> <ins> name:            "should require VolumeAttach for drivers with attachment", deployDriverCRD: true, </ins> }, { <del> name:             "should preserve attachment policy when no CSIDriver present", driverAttachable: true, deployDriverCRD:  false, </del> <ins> name:            "should preserve attachment policy when no CSIDriver present", deployDriverCRD: false, </ins> }, } for _, t := range tests { test := t It(t.name, func() { var err error <del> init(testParameters{registerDriver: test.deployDriverCRD, attachable: test.driverAttachable}) </del> <ins> init(testParameters{registerDriver: test.deployDriverCRD, disableAttach: test.disableAttach}) </ins> defer cleanup() _, claim, pod := createPod()

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: in Which test(s) are failing: Since when has it been failing: 2019-02-27 () -- since it was Testgrid link: Reason for failure: Timeout: Anything else we need to know: /kind failing-test /sig storage /priority important-soon /milestone v1.14 cc
cc Could you help with this please?
/assign
Neither of the failing tests mentioned in this issue are present in testgrid anymore. is all clear


Question: Calls    []string IP       net.IP Machines []string <del> Zone     string </del> <ins> Zone </ins> } func (f *FakeCloud) addCall(desc string) {

Comments that the PR bot adds solely to trigger email notification of a label change, should be immediately deleted by the bot after it adds them. I have no idea what team to assign this to... cc/
, David Oppenheimer wrote:
(Motivation, of course, is to avoid having PR threads that are 90% bot comments...)
should really be an issue over in that's where (some of) the bots live. Lets make a list of which comments we want deleted immediately. [x] Label size. [x] Needs rebase. [x] test this stale CI [x] test this pending CI [x] PR changed after LGTM, removing LGTM. [x] Comment about blocking based on the paths touched (old docs) [x] unapproved cherrypick comment [x] cherrypicks must set a milestone [x] ok to test from bot I don't want to make it all comments. There are way too many times people have asked me "Why did the bot add 'do-not-merge'?" and I can just link them the comment 2 lines above which tells exactly why....
"Test this" with an issue is also useful, isn't it? On Apr 7, 2016 6:30 PM, "Eric Paris" wrote:
OK to test Add to whitelist On Apr 7, 2016 6:30 PM, "Eric Paris" wrote:
I was envisioning deleting the by the bot. (every 48 hours after LGTM, when retesting right before merge) not those by users (with an issue) After LGTM the bot will add , that one I can remove. The request to comes from the jenkins bot. And in that I am clueless...
any interest in a review (I'm trying to not just self merge all my own bot changes). This deleted hundreds and hundreds of comments.
Looks like reviewed it since your last message:

The contains instructions for downloading kubectl. The doesn't contain these instructions but it should if it's now the recommended way of running Kubernetes locally.
+1 Just ran into this.
cc
By the way, we encourage anyone to propose improvements/fixes to docs. You can do so pretty easily by clicking the pencil icon in the top right corner of every page of
kubectl download instructions to the guide in


Question: } } <ins> // truncateMessage truncates a message if it hits the NoteLengthLimit. func truncateMessage(message string) string { max := validation.NoteLengthLimit if len(message) <= max { return message } suffix := " ..." return message[:max-len(suffix)] + suffix } </ins> func updatePod(client clientset.Interface, pod *v1.Pod, condition *v1.PodCondition, nominatedNode string) error { klog.V(3).InfoS("Updating pod condition", "pod", klog.KObj(pod), "conditionType", condition.Type, "conditionStatus", condition.Status, "conditionReason", condition.Reason) podCopy := pod.DeepCopy()

What happened: When a pod can't be scheduled and the cardinality of taints on nodes is high, the Note field in the event can become bigger than the 1024 allowed and the event isn't sent. The pod stays in a state of without en event which makes the User Experience very bad What you expected to happen: I would expect the Kube-scheduler to be able to send the event no matter how many different taints are on nodes How to reproduce it (as minimally and precisely as possible): add many different taints to nodes and then trying to schedule something which can't be successfully scheduled (higher CPU/memory than any node has) Anything else we need to know?: In this case the length of the message was 1334 characters. Environment: Kubernetes version (use ): 1.18.9 Cloud provider or hardware configuration: bare metal Possible solutions: make event size configurable, it's now hardcoded to 1024 characters make scheduler note aware of that limit and work around I, if length 1024 then truncate the string (so at least something is shown) produce N events splitting the text (preferred one): change event Note to be to something like: (the whole complete information would still be present in the pod status field. for debugging purposes)
/sig scheduling
/triage accepted /cc
I could provide the fix in case we define which solution is best
This is a consequence of I think we can just drop some reasons when we are building the status message if the message is too large. This comes from here /help
/remove-help /assign
how many different taints did you have? I think in all cases we should truncate the message as well:
Another remidiation is to supress outoutping nil fields (such as GenerateName:"", SelfLink:"", UID:"", ResourceVersion:"", etc.) in the EventRecorder side.
The size limit we're hitting is specifically on the field
Completely agreed
good issue~
/assign
/unassign


Question: return !skipAttach, nil } <del> // TODO (#75352) add proper logic to determine device moutability by inspecting the spec. </del> <ins> // CanDeviceMount returns true if the spec supports device mount </ins> func (p *csiPlugin) CanDeviceMount(spec *volume.Spec) (bool, error) { <ins> driverMode, err := p.getDriverMode(spec) if err != nil { return false, err } if driverMode == ephemeralDriverMode { klog.V(5).Info(log("plugin.CanDeviceMount skipped ephemeral mode detected for spec %v", spec.Name())) return false, nil } </ins> return true, nil }

Inline CSI volumes that are embedded in Pod specs still trigger all volume operation stages (attach, device mount, mount). Similar to plugin.CanAttach, a new plugin-wide method should be able to turn device mount off when the source is ephemeral volume. References See volume mount operation : This issue will use code from
/sig storage /assign
This will use work from


Question: fi CTLRMGR_LOG=${LOG_DIR}/kube-controller-manager.log <del> ${CONTROLPLANE_SUDO} "${GO_OUT}/hyperkube" controller-manager  </del> <ins> ${CONTROLPLANE_SUDO} "${GO_OUT}/hyperkube" kube-controller-manager  </ins> --v="${LOG_LEVEL}"  --vmodule="${LOG_SPEC}"  --service-account-private-key-file="${SERVICE_ACCOUNT_KEY}"

Hyperkube supports a number of short aliases for commands, such as , , , , etc, rather than the longer , , , etc. There are comments indicating this is "unfortunate" and we still currently create these aliases in the hyperkube docker image: It'd be nice to simplify this, assuming we can find all uses of the short aliases and remove them. /kind cleanup Not sure which sig should own this, so tagging a few. /sig architecture cluster-lifecycle cli cc
interestingly, the command doesn't create the short aliases, though we also don't really use anywhere either.
I think we can just go ahead and start the deprecation cycle for this. WDYT? cc
cc as well
+1 to nuke it
/assign :wink: ok?
SGTM
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
/help
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I would like to work on it!
It seems like this issue was already solved in ?
I don't think the hyperkube code was changed in that PR, only the image?
Yes, thanks for your comment, can i try to fix it ?
I made a PR here: . Review/feedback/suggestions appreciated! :)
/assign /unassign
GitHub didn't allow me to assign the following users: Rand01ph.

Note that only and repo collaborators can be assigned and that issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign /unassign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-sig architecture cli /sig release Does anyone even still use hyperkube? Wouldn't they use kubeadm or k3s instead?
AFAIK, SAP (the company) still use hyperkube.
aks-engine deployer ( ) still uses hyperkube
Since this ticket is completed now. Can this todo be removed?
we are removing the whole thing :)


Question: } } <del> func newReplicaSet(name string, replicas int) *apps.ReplicaSet { </del> <ins> func newReplicaSet(name string, replicas int, rsUuid types.UID) *apps.ReplicaSet { </ins> return &apps.ReplicaSet{ TypeMeta: metav1.TypeMeta{APIVersion: "v1"}, ObjectMeta: metav1.ObjectMeta{ <del> UID:             uuid.NewUUID(), </del> <ins> UID:             rsUuid, </ins> Name:            name, Namespace:       metav1.NamespaceDefault, ResourceVersion: "18",

During implementation of KEP 3393 (PodReplacementPolicy for Jobs) we found that the unit tests of do not follow common patterns established in k/k. 1) The tests are not written using table driven design (). 2) The tests use rather than . This probably should be done as two separate PRs but wanted to highlight some areas that could use some cleanup. Not using means that we don't see diffs in the unit tests that makes this difficult. And not using table driven tests makes it difficult to add new test cases as it usually is brought up during review that we should use a table-driven test but the entire file does not obey this format.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig apps /kind cleanup /help /good-first-issue
This request has been marked as suitable for new contributors. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /sig apps /kind cleanup /help /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I'd like to work on it /assign
I'd also like to make some contributions. /assign
Looks like this need two PRs. tests are not written using table driven design (). tests use Differences rather than Cmp.Diff. I think I can work on the 2nd one and leave the 1st one to you.

Sounds good?
By looking through the , I found the main function to comapre the results have already used cmp.Diff. Can you elaborate more on which part you expect to use cmp.Diff? func verifyDeletedAndPatchedPods(t testing.T, client fake.Clientset, wantDeletedPodNames, wantPatchedPodNames sets.String) { t.Helper() deletedPodNames := getDeletedPodNames(client) if diff := cmp.Diff(wantDeletedPodNames, deletedPodNames); diff != "" { t.Errorf("Deleted pod names (-want,+got):n%s", diff) } patchedPodNames := getPatchedPodNames(client) if diff := cmp.Diff(wantPatchedPodNames, patchedPodNames); diff != "" { t.Errorf("Patched pod names (-want,+got):n%s", diff) } }
I am really sorry about this. Too much multitasking!! I meant not
Hey ! Sorry, I didn't see this before. No problem at all, but consider that one change depends on the other, and since my task implies a big refactor, there might be conflicts when merging your PR... That's why I see two possible scenarios happening: Your PR gets merged before I raise mine: Then fine, I'll rebase your changes and adapt my code Mine is ready before yours is merged: In this case, I propose you commit your changes to my PR. Both of us will appear as coauthors of the change Getting a PR merged in the Kubernetes project takes quite some time (trust me, I talk by experience and can confirm ) WDYT?
Let's see how it goes. We can always rebase the code change to adapt new.
btw, did you already begin to work on task 1? If not, I may just go ahead and work on it right away since I have already dig into it.
yes, I already started And If you ask me, better to keep it this way so we both have an opportunity to have a contribution
cool. Sounds good.
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hi if this issue needs to be worked on any further, please assign me for the same.
has a PR up that will solve the second part of this issue.


Question: check := ImagePullCheck{ runtime:         containerRuntime, imageList:       []string{"img1", "img2", "img3"}, <del> imagePullPolicy: v1.PullIfNotPresent, </del> <ins> imagePullPolicy: "", // should be defaulted to v1.PullIfNotPresent </ins> } warnings, errors := check.Check() if len(warnings) != 0 {

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --https://k8s- kubeadm-init 06-22 https://k8s- all kubeadm jobs are failing against k/k master. is this failing after we merged?
cc /sig cluster-lifecycle /area kubeadm kinder cluster preparation constructs images locally by downloading tarballs and tagging them with: then running it would find the image locally and not pull it from GCR. does not contain pre-release images, so trying to pull these images would fail. i think the problem here is that we broke the default behavior with
ok, the problem is indeed here: empty string "" should be the same as PullIfNotPresent.
should be fixed by
/triage accepted


Question: base_image="" if [[ -f BASEIMAGE ]]; then <del> base_image=$(getBaseImage "${os_arch}" | ${SED} "s|REGISTRY|${REGISTRY}|g") ${SED} -i "s|BASEARCH|${arch}|g" $dockerfile_name </del> <ins> base_image=$(getBaseImage "${os_arch}" | "${SED}" "s|REGISTRY|${REGISTRY}|g") "${SED}" -i "s|BASEARCH|${arch}|g" $dockerfile_name </ins> fi <del> # copy the qemu-*-static binary to docker image to build the multi architecture image on x86 platform if grep -q "CROSS_BUILD_" Dockerfile; then if [[ "${arch}" == "amd64" ]]; then ${SED} -i "/CROSS_BUILD_/d" Dockerfile else ${SED} -i "s|QEMUARCH|${QEMUARCHS[$arch]}|g" Dockerfile # Register qemu-*-static for all supported processors except the current one echo "Registering qemu-*-static binaries in the kernel" local sudo="" if [[ $(id -u) != 0 ]]; then sudo=sudo </del> <ins> # Only the cross-build on x86 is guaranteed by far, other arches like aarch64 doesn't support cross-build # thus, there is no need to tackle a disability feature on those platforms, and also help to prevent from # ending up a wrong image tag on non-amd64 platforms. build_arch=$(uname -m) if [[ ${build_arch} = 'x86_64' ]]; then # copy the qemu-*-static binary to docker image to build the multi architecture image on x86 platform if grep -q 'CROSS_BUILD_' Dockerfile; then if [[ "${arch}" = 'amd64' ]]; then "${SED}" -i '/CROSS_BUILD_/d' Dockerfile else "${SED}" -i "s|QEMUARCH|${QEMUARCHS[$arch]}|g" Dockerfile # Register qemu-*-static for all supported processors except the current one echo 'Registering qemu-*-static binaries in the kernel' local sudo if [[ $(id -u) -ne 0 ]]; then sudo=sudo fi ${sudo} "${KUBE_ROOT}/third_party/multiarch/qemu-user-static/register/register.sh" --reset -p yes curl -sSL https://github.com/multiarch/qemu-user-static/releases/download/"${QEMUVERSION}"/x86_64_qemu-"${QEMUARCHS[$arch]}"-static.tar.gz | tar -xz -C "${temp_dir}" # Ensure we don't get surprised by umask settings chmod 0755 "${temp_dir}/qemu-${QEMUARCHS[$arch]}-static" "${SED}" -i 's/CROSS_BUILD_//g' Dockerfile fi </ins> fi <del> ${sudo} "${KUBE_ROOT}/third_party/multiarch/qemu-user-static/register/register.sh" --reset -p yes curl -sSL https://github.com/multiarch/qemu-user-static/releases/download/"${QEMUVERSION}"/x86_64_qemu-"${QEMUARCHS[$arch]}"-static.tar.gz | tar -xz -C "${temp_dir}" # Ensure we don't get surprised by umask settings chmod 0755 "${temp_dir}/qemu-${QEMUARCHS[$arch]}-static" ${SED} -i "s/CROSS_BUILD_//g" Dockerfile fi </del> <ins> elif [[ "${QEMUARCHS[$arch]}" != "${build_arch}" ]]; then echo "skip cross-build $arch on non-supported platform ${build_arch}." popd continue else "${SED}" -i '/CROSS_BUILD_/d' Dockerfile </ins> fi docker buildx build --progress=plain --no-cache --pull --output=type="${output_type}" --platform "${os_name}/${arch}"

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: when building images on non-amd platform, arm64 for example, and the target image is arm64 too, the code still reset . What you expected to happen: There is no need to reset anything, the shell code is purely for amd64 and haven't considered the non-amd platforms. How to reproduce it (as minimally and precisely as possible): on any platform other than amd, cd test/images build all WHAT=echoserver (or any image under the dir) when coming to image which is exactly the same as the build machine, following is seen from console Anything else we need to know?: Environment: Kubernetes version (use ): latest Cloud provider or hardware configuration: OS (e.g: ): all Kernel (e.g. ): all Install tools: Network plugin and version (if this is a network-related bug): Others:
/assign
/sig testing
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten this issue still stands.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/lifecycle frozen
/remove-lifecycle rotten
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
triage/accepted
/triage accepted

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: some of base images support multi-arch, if no platform specified, the image pulled will actually reflect where the operation is performed, current implementation always assume the default platform is amd64 which make the tags is not correctly defined. for example, on arm64 platform, here is what I got after build the image of ipc-utils, What you expected to happen: tag should be defined correctly How to reproduce it (as minimally and precisely as possible): on arm64 platform, Anything else we need to know?: Environment: Kubernetes version (use ): latest Cloud provider or hardware configuration: all arch except x86 OS (e.g: ): all Kernel (e.g. ): all Install tools: Network plugin and version (if this is a network-related bug): Others:
/assign
/sig testing release
This is the side effect of command, you can fix it by specifying tag while building the image.
The tag is still experimental and introduced in so I really doubt about using this anywhere, I feel it is just a cosmetic issue and only applies for the images with tag. As we are pushing fat manifest images with proper archs so we don't have to worry much about it.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with .

Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/lifecycle frozen
/remove-lifecycle rotten
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available .

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage accepted


Question: if ip := net.ParseIP(host); ip == nil || !ip.IsLoopback() || ip.To4() != nil { t.Fatalf("expected IPv6 host to be loopback, got %q", host) } if port != "443" { t.Fatalf("expected 443 as port, got %q", port) } } <ins> func isIPv6LoopbackSupported() (ipv6 bool, ipv6only bool, err error) { addrs, err := net.InterfaceAddrs() if err != nil { return false, false, err } ipv4 := false for _, address := range addrs { ipnet, ok := address.(*net.IPNet) if !ok || !ipnet.IP.IsLoopback() { continue } if ipnet.IP.To4() == nil { ipv6 = true continue } ipv4 = true } ipv6only = ipv6 && !ipv4 return ipv6, ipv6only, nil } </ins>

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: LoopbackHostPort code is calling the which depends on the machine on which this test is running. Especially in this particular testase it looks for the ipv6 ip address which does not exist because the machine is not configured for ipv6. Because of the LoopbackHostPort function bahaves based on the interfaces configured, test will fail if machine is not really configured with ipv6. Need to rewrite the testcase to handle such situation. What you expected to happen: Testcase should pass
/sig network
/cc
/triage unresolved I am a bot run by
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
/remove-triage unresolved


Question: return } <del> nodeAddresses, err := getNodeAddressesByProviderIDOrName(ctx, instances, node) </del> <ins> nodeAddresses, err := getNodeAddressesByProviderIDOrName(ctx, instances, node.Spec.ProviderID, node.Name) </ins> if err != nil { klog.Errorf("Error getting node addresses for node %q: %v", node.Name, err) return

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: introduces a regression that func is invoked on a Node obj without , hence it can fail fetching the node addresses when the is not set. What you expected to happen: I would expect cloud-controller-manager to be able to initialize new Node. How to reproduce it (as minimally and precisely as possible): the gcp legacy-cloud-controller v1.18.1 () that the node-controller cannot initialize a new Node. Logs of the cloud-controller-manager repeated multiple times: Ensure that the Node is with the taint. Ensure that the of the Node is not set. The cloud-controller-manager is not running on the host machine and that's why it cannot call the gce matadata. But it was never a hard requirement for cloud-controller-manager to run on the host machine. Anything else we need to know?: The legacy cloud-controller-manager v1.17.x is working perfectly fine. Environment: Kubernetes version (use ):v1.18.1 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig cloud-provider /kind regression
/cc
can we open a new issue re: