Question: hash := HashControllerRevision(revision, collisionCount) // Update the revisions name and labels clone.Name = ControllerRevisionName(parent.GetName(), hash) <del> created, err := rh.client.AppsV1().ControllerRevisions(parent.GetNamespace()).Create(clone) </del> <ins> ns := parent.GetNamespace() created, err := rh.client.AppsV1().ControllerRevisions(ns).Create(clone) </ins> if errors.IsAlreadyExists(err) { <ins> exists, err := rh.client.AppsV1().ControllerRevisions(ns).Get(clone.Name, metav1.GetOptions{}) if err != nil { return nil, err } if bytes.Equal(exists.Data.Raw, clone.Data.Raw) { return exists, nil } </ins> *collisionCount++ continue }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT: What happened: When a statefulset object is created, a lot of controllerrevisions are created at the same time. the controllerrevisions have differenct hash value but the same Data.Raw. How to fix it: two controllerrevisions are seen as the same , when [ControllerRevisionHashLabel] , Controllerrevision.Data.Raw , Controllerrevision.Data.Object are strictly the same // this line shoud be to fix the bug like below: Anything else we need to know?: Environment: Kubernetes version (use ): 1.7.8 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
I would help fix it
/kind bug /sig apps
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
StatefulSet uses the following code to construct its history object (controllerrevision): However, collisionCount shouldn't be blindly increased when there's name collision. Because controller uses a cache store, the name collision could be due to a controllerrevision created but not yet observed by the controller. The right way to handle it is to first check if the controllerrevision with the same name has the same content as the one we're going to create, like how DaemonSet controller handles it:


Question: <ins> # How to become a contributor and submit your own code ## Contributor License Agreements We'd love to accept your patches! Before we can take them, we have to jump a couple of legal hurdles. Please fill out either the individual or corporate Contributor License Agreement (CLA). * If you are an individual writing original source code and you're sure you own the intellectual property, then you'll need to sign an [individual CLA](http://code.google.com/legal/individual-cla-v1.0.html). * If you work for a company that wants to allow you to contribute your work, then you'll need to sign a [corporate CLA](http://code.google.com/legal/corporate-cla-v1.0.html). Follow either of the two links above to access the appropriate CLA and instructions for how to sign and return it. Once we receive it, we'll be able to accept your pull requests. ## Contributing A Patch 1. Submit an issue describing your proposed change to the repo in question. 1. The repo owner will respond to your issue promptly. 1. If your proposed change is accepted, and you haven't already done so, sign a Contributor License Agreement (see details above). 1. Fork the desired repo, develop and test your code changes. 1. Submit a pull request. </ins>

When one of the aggregated API servers returns a different definition of an object than what the other servers return, the merged openapi schema may contain multiple identical copies of the object definition (they differ only in the name). Is this a BUG REPORT or FEATURE REQUEST?:BUG /kind bug What happened: After changing the definition of the RawExtension type in one of the aggregated API servers, the aggregator returned the following openapi schema (note that v2, v3 and _v4 are identical; don't mind the actual new definition of RawExtension - it's incorrect): What you expected to happen: The merged schema should only contain two different RawExtension object definitions instead of 4. Environment: Kubernetes version (use ):
/sig api-machinery
/assign


Question: framework.ExpectNoError(err) eps2 := epsTemplate.DeepCopy() eps2.Endpoints = []discoveryv1.Endpoint{ <del> {Addresses: []string{pod2.Status.PodIP}}, </del> <ins> { Addresses:  []string{pod2.Status.PodIP}, Conditions: discoveryv1.EndpointConditions{Ready: &readyCondTrue}, }, </ins> } eps2.Ports = []discoveryv1.EndpointPort{{ Name:     pointer.String("port81"),

The test create custome EndpointSlices to validate some behaviors sending traffic, so it is clear that it expects the EndpointSlices to be ready, The API doc is vague about this situation and leaves room to interpret the "unknown behavior", quoting Since the test expects the EndpointSlices to be Ready we should set them, if we decide to change the API behavior, then we should add an additional e2e test to validate this behavior, but not use the same test to validate two different things /kind bug /kind failing-testing /sig network Slack thread
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : The test create custome EndpointSlices to validate some behaviors sending traffic, so it is clear that it expects the EndpointSlices to be ready, /kind failing-testing /sig network Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
GitHub didn't allow me to assign the following users: aauren. Note that only with read permissions, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Thanks I'll take this one and draft up a PR this weekend. (this comment should hopefully allow the k8s ci robot to assign this to me as well)
if you are planning to contribute more often you should apply as org member


Question: // Invalidate the cache right after updating defer as.cloud.vmCache.Delete(vmName) <del> return as.VirtualMachinesClient.CreateOrUpdate(ctx, nodeResourceGroup, vmName, newVM, "detach_disk") </del> <ins> return as.VirtualMachinesClient.Update(ctx, nodeResourceGroup, vmName, newVM, "detach_disk") </ins> } // GetDataDisks gets a list of data disks attached to the node.

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Instead of using the VM CreateOrUpdate () function of the Azure Go SDK, use the VM Update function () in the azureclient () Why is this needed: Currently, the Azure Cloud Provider is using the VM CreateOrUpdate call when e.g. detach/attaching disks to a VM. In some scenarios, this will re-create a VM that was deleted by an external application. This can lead to orphaned VMs or blocked resources. The Update function will only update an existing VM and will fail if the VM does not exist.
/sig azure
/assign
/sig cloud-provider
Could you describe more about the scenarios? Have you noticed such issues? I wonder adding etags to the API call could help to prevent this?
If you have an application that maintains your K8s clusters, e.g. takes care of OS update/patching. If the application wants to patch the OS of a worker node, it would drain the node first. If draining times out, it will go ahead and delete the VM anyway. But K8s might still be detaching disks. These detach operations, because they use CreateOrUpdate, will recreate the VM. Is there a reason why you would want to create a VM i.e. why CreateOrUpdate is required and Update would not work? AFAIK, Azure Compute RP does not support etags.
that sounds reasonable, thanks change to use function would be safer since it's only detach/attach disk operation. I will do that.
thanks for the information. Reasonable enough. /assign
/milestone v1.16
BTW, the reason why we only use func in the beginning is that Azure Compute go sdk did not provide a func at that time...
Great. Thanks!


Question: cacher.stopWg.Add(1) go func() { defer cacher.stopWg.Done() <ins> defer cacher.terminateAllWatchers() </ins> wait.Until( func() { if !cacher.isStopped() {

Reproduced on 1.11.7 as well as master-ish () Deleting a CRD doesn't seem to cause the API server to terminate watches referencing the corresponding CRs, but the watches also don't properly resume if the CRD is created again, meaning that you have to wait for watch timeouts to actually get new watch events. We should probably kill open watches on CRD deletion, so that clients are notified and are forced to re-watch. Either that, or seamlessly resume watches if a CRD is recreated. <details</summary</details<details</summary</detailsthe above CRD: the above CR: (elsewhere) and grab a resource version (elsewhere) the lack of watch events After watches time out, things can re-establish, but that can be a while.
/assign
/cc sttts
/area custom-resources
it's sort a related thread w/ . for now CRD's update will terminate CR's watch connection, the other way round we might fix'em together. one possible fix suggested by months ago is that we make the storage to be a runnable object, e.g. by which we can clearily define the lifecycle of CR's storage
fixed in

What happened: If you establish a WATCH request on a list of CRs, then edit the CRD's object, you will stop getting MODIFIED events on the WATCH request. The WATCH request doesn't fail or report an error, but will not return more MODIFIED events until the request is restarted. This causes problems for custom controllers, which become unresponsive until their requests expire. What you expected to happen: Either the request should end (so it can be restarted safely) or it should continue to return MODIFIED events. How to reproduce it (as minimally and precisely as possible): I've attached . It does the following: Create CRD and an initial CR. Start a proxy, wait for it to come up, then start a watch request. Change the CR. (This change to the CR results in a MODIFIED event) Change the section of the CRD. Make another change to the CR. (This change to the CR doesn't result in an event). I've tested this against GKE masters on 1.9, 1.10 and 1.11, as well as a kubeadm 1.12.2 cluster. The output looks like: Note that you only see MODIFIED once despite the CR being changed twice. Environment: Kubernetes version (use ): 1.11.2-gke.18, 1.12.2 Cloud provider or hardware configuration: GKE, kubeadm on amd64 OS (e.g. from /etc/os-release): Ubuntu 16.04 Kernel (e.g. ): 4.15.0-36-generic ~16.04.1-Ubuntu SMP <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc
/cc
i suppose it's related w/ yes, b/c we switched to another CR storage when you updated CRD spec.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with .

Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
fixed in


Question: Logf("Cluster validation succeeded") return nil } <ins> // migRollingUpdate starts a MIG rolling update, upgrading the nodes to a new // instance template named tmpl, and waits up to nt times the nubmer of nodes // for it to complete. func migRollingUpdate(tmpl string, nt time.Duration) error { By(fmt.Sprintf("starting the MIG rolling update to %s", tmpl)) id, err := migRollingUpdateStart(tmpl, nt) if err != nil { return fmt.Errorf("couldn't start the MIG rolling update: %v", err) } By(fmt.Sprintf("polling the MIG rolling update (%s) until it completes", id)) if err := migRollingUpdatePoll(id, nt); err != nil { return fmt.Errorf("err waiting until update completed: %v", err) } return nil } // migTemlate (GCE/GKE-only) returns the name of the MIG template that the // nodes of the cluster use. func migTemplate() (string, error) { var errLast error var templ string key := "instanceTemplate" // TODO(mbforbes): Refactor this to use cluster_upgrade.go:retryCmd(...) if wait.Poll(poll, singleCallTimeout, func() (bool, error) { // TODO(mbforbes): make this hit the compute API directly instead of // shelling out to gcloud. o, err := exec.Command("gcloud", "preview", "managed-instance-groups", fmt.Sprintf("--project=%s", testContext.CloudConfig.ProjectID), fmt.Sprintf("--zone=%s", testContext.CloudConfig.Zone), "describe", testContext.CloudConfig.NodeInstanceGroup).CombinedOutput() if err != nil { errLast = fmt.Errorf("gcloud preview managed-instance-groups describe call failed with err: %v", err) return false, nil } output := string(o) // The 'describe' call probably succeeded; parse the output and try to // find the line that looks like "instanceTemplate: url/to/<templ>" and // return <templ>. if val := parseKVLines(output, key); len(val) > 0 { url := strings.Split(val, "/") templ = url[len(url)-1] Logf("MIG group %s using template: %s", testContext.CloudConfig.NodeInstanceGroup, templ) return true, nil } errLast = fmt.Errorf("couldn't find %s in output to get MIG template. Output: %s", key, output) return false, nil }) != nil { return "", fmt.Errorf("migTemplate() failed with last error: %v", errLast) } return templ, nil } // migRollingUpdateStart (GCE/GKE-only) starts a MIG rolling update using templ // as the new template, waiting up to nt per node, and returns the ID of that // update. func migRollingUpdateStart(templ string, nt time.Duration) (string, error) { var errLast error var id string prefix, suffix := "Started [", "]." // TODO(mbforbes): Refactor this to use cluster_upgrade.go:retryCmd(...) if err := wait.Poll(poll, singleCallTimeout, func() (bool, error) { // TODO(mbforbes): make this hit the compute API directly instead of //                 shelling out to gcloud. // NOTE(mbforbes): If you are changing this gcloud command, update //                 cluster/gce/upgrade.sh to match this EXACTLY. o, err := exec.Command("gcloud", "preview", "rolling-updates", fmt.Sprintf("--project=%s", testContext.CloudConfig.ProjectID), fmt.Sprintf("--zone=%s", testContext.CloudConfig.Zone), "start", // Required args. fmt.Sprintf("--group=%s", testContext.CloudConfig.NodeInstanceGroup), fmt.Sprintf("--template=%s", templ), // Optional args to fine-tune behavior. fmt.Sprintf("--instance-startup-timeout=%ds", int(nt.Seconds())), // NOTE: We can speed up this process by increasing //       --max-num-concurrent-instances. fmt.Sprintf("--max-num-concurrent-instances=%d", 1), fmt.Sprintf("--max-num-failed-instances=%d", 0), fmt.Sprintf("--min-instance-update-time=%ds", 0)).CombinedOutput() if err != nil { errLast = fmt.Errorf("gcloud preview rolling-updates call failed with err: %v", err) return false, nil } output := string(o) // The 'start' call probably succeeded; parse the output and try to find // the line that looks like "Started [url/to/<id>]." and return <id>. for _, line := range strings.Split(output, "n") { // As a sanity check, ensure the line starts with prefix and ends // with suffix. if strings.Index(line, prefix) != 0 || strings.Index(line, suffix) != len(line)-len(suffix) { continue } url := strings.Split(strings.TrimSuffix(strings.TrimPrefix(line, prefix), suffix), "/") id = url[len(url)-1] Logf("Started MIG rolling update; ID: %s", id) return true, nil } errLast = fmt.Errorf("couldn't find line like '%s ... %s' in output to MIG rolling-update start. Output: %s", prefix, suffix, output) return false, nil }); err != nil { return "", fmt.Errorf("migRollingUpdateStart() failed with last error: %v", errLast) } return id, nil } // migRollingUpdatePoll (CKE/GKE-only) polls the progress of the MIG rolling // update with ID id until it is complete. It returns an error if this takes // longer than nt times the number of nodes. func migRollingUpdatePoll(id string, nt time.Duration) error { // Two keys and a val. status, progress, done := "status", "statusMessage", "ROLLED_OUT" start, timeout := time.Now(), nt*time.Duration(testContext.CloudConfig.NumNodes) var errLast error Logf("Waiting up to %v for MIG rolling update to complete.", timeout) // TODO(mbforbes): Refactor this to use cluster_upgrade.go:retryCmd(...) if wait.Poll(restartPoll, timeout, func() (bool, error) { o, err := exec.Command("gcloud", "preview", "rolling-updates", fmt.Sprintf("--project=%s", testContext.CloudConfig.ProjectID), fmt.Sprintf("--zone=%s", testContext.CloudConfig.Zone), "describe", id).CombinedOutput() if err != nil { errLast = fmt.Errorf("Error calling rolling-updates describe %s: %v", id, err) Logf("%v", errLast) return false, nil } output := string(o) // The 'describe' call probably succeeded; parse the output and try to // find the line that looks like "status: <status>" and see whether it's // done. Logf("Waiting for MIG rolling update: %s (%v elapsed)", parseKVLines(output, progress), time.Since(start)) if st := parseKVLines(output, status); st == done { return true, nil } return false, nil }) != nil { return fmt.Errorf("timeout waiting %v for MIG rolling update to complete. Last error: %v", timeout, errLast) } Logf("MIG rolling update complete after %v", time.Since(start)) return nil } </ins>

, func restartNodes, assumes that MIG rolling update on GCE will cause node restart. However, there is a feature of MIG rolling update (currently broken) that skips update if the new instance template is the same as the old one. The feature is supposed to work quite soon, so the test will not test anything (reboot will not be performed). A workaround may be to create an identical instance template with a different name. By the way, the current name is misleading. It should be . cc
Hi Marek, Was this changed recently? When we tried this originally it worked. Now, I just tried doing a rolling update to the same template, and it immediately fails with status "PAUSED" and status message "Errors: unknown". The name was chosen to attempt to be future-provider-agnostic, though I agree is better.
Actually, I'm trying it now, and it definitely sometimes works (recreates the instances). If this functionality isn't expected to work (or isn't supported) we can switch this to the equivalent MIG recreate-nodes call. I'll do this.
I've heard that some two months ago MIG update would not recreate new nodes if the template was the same. Then they changed it and instances were recreated (this is when you wrote the test). I think it is considered a bug. And they are going to fix this bug quite soon. So in a week or two (or maybe more - I don't know their release cycle), instances will not be recreated. (sorry for incomprehensible description of this issue - I'm correcting it now).
, I think your error is not related to MIG update - currently instances should be recreated (i.e. the error in MIG rolling update is not fixed). The error may be fixed around Monday (but maybe later - even 2 weeks).
Ack—thanks for the extra detail! I'll just switch this test to and we can stop worrying about the timing. We'll switch this in GKE too. Thanks again, I really appreciate you filing this!
(And by switch I mean just turn off self-upgrades.)
is this v1?
The referenced PR was a refactoring and Github got a bit confused. This issue isn't closed.

I don't think we need a test to recreate the MIG before v1.0 (although if it gets done before the freeze I don't see why we wouldn't merge it).
I'm leaving the project, so reassigning to... Have fun!
/sig testing
Fixed forever ago. /close


Question: ) const ( <del> dockerProcessName     = "dockerd" </del> <ins> dockerProcessName = "dockerd" // dockerd option --pidfile can specify path to use for daemon PID file, pid file path is default "/var/run/docker.pid" </ins> dockerPidFile         = "/var/run/docker.pid" <del> containerdProcessName = "docker-containerd" containerdPidFile     = "/run/docker/libcontainerd/docker-containerd.pid" </del> <ins> containerdProcessName = "containerd" </ins> maxPidFileLength      = 1 << 10 // 1KB )

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: docker-containerd have renamed to containerd, but EnsureDockerInContainer check not update,so it not work What you expected to happen: Is this check still needed How to reproduce it (as minimally and precisely as possible): no need Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
cc
feel free to send an PR update with new name for the process and the location of the pid file.


Question: // First time this key has been used, try creating new value. if index == 0 { <del> _, err = h.Client.Create(key, string(data), 0) </del> <ins> response, err := h.Client.Create(key, string(data), 0) </ins> if IsEtcdNodeExist(err) { continue } <ins> _, _, err = h.extractObj(response, err, ptrToType, false, false) </ins> return err }

The use of AtomicUpdate by genericetcd doesn't handle conflicting writes by rejecting that to a user. It should. Eventually, we may want to hide write conflicts to /pods when status has changed, and to spec when /pods/status changes, but that requires us to have two version counters and that's not worth the effort. Fix incoming.
Also we need a unit/integration test for this.


Question: return os.bsOpts.TrustDevicePath } <ins> // NodeVolumeAttachLimit specifies number of cinder volumes that can be attached to this node. func (os *OpenStack) NodeVolumeAttachLimit() int { return os.bsOpts.NodeVolumeAttachLimit } </ins> // GetLabelsForVolume implements PVLabeler.GetLabelsForVolume func (os *OpenStack) GetLabelsForVolume(ctx context.Context, pv *v1.PersistentVolume) (map[string]string, error) { // Ignore if not Cinder.

We merged to allow configurable volume limits for Cinder. But proposed adding such a limit to as well, among other things this will enable us to support node specific limits more easily - /sig openstack /sig storage /sig scheduling
/assign
cc
/kind feature
This will require updating in-tree Openstack cloudprovider code. are we onboard with that?
What would be the extent of the modification? It must not break current compatibility, and the in-tree provider is scheduled for removal this year. I'm ok with it as long as existing users aren't broken by it.
it should be a change into which would add a new configurable field. It would be backward compatible but would allow admins to configure different limits on a node if they must.
/milestone v1.14 We're entering and this looks relevant to the 1.14 release. Please if I am incorrect


Question: rules: # Discourage import of k8s.io/kubernetes/test/e2e - selectorRegexp: k8s[.]io/kubernetes/test/e2e <del> # TODO: import-boss --include-test-files is catching these; drive to zero allowedPrefixes: # test/integration/auth/bootstraptoken_test.go is using this - k8s.io/kubernetes/test/e2e/lifecycle/bootstrap </del> <ins> forbiddenPrefixes: - "" </ins>

Originally tracked at . After fixing "not include test files in import-boss" in , we have a regression here: is using . We need remove dependencies from integration tests to e2e tests again. /assign /sig testing
/sig auth as per the required change
/kind cleanup /area test
It would be helpful to outline an argument as to why this is the case. I can imagine reasons, but none are mentioned here.
Description updated.
/triage accepted /priority backlog /milestone v1.21 This seems a simple enough fix to land (and thank you for doing this) but marking low enough priority to make it clear we may back out and wait for v1.22 if we don't have bandwidth for this
So while I think we should probably do this anyhow I want to note: This issue lists the reasons: we have no intention to stage e2e (there's been much discussion about this since then) we generally don't want anything non-informational (required jobs) running only sometimes anyhow, it's just not reliable Decoupling these seems like a good idea from a purely code-organization perspective, but the reasons in the previous issue are no longer valid.
/milestone v1.22 The PR didn't land in time. At this point I view this as nice to have. Even if it's low impact, let's wait until next release.


Question: } } <del> return nodesNeedingDaemonPods, podsToDelete, failedPodsObserved, nil </del> <ins> return nodesNeedingDaemonPods, podsToDelete, nil </ins> } // manage manages the scheduling and running of Pods of ds on nodes.

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: I have a daemonsets with 250 replicas and maxUnavailable is 20%. When I try to update its image, the ds will scale down 50 old replicas and wait for SatisfiedExpectations, after that it will scale up 50 new replicas. But if a pod delete failed (node is notready, and pod stuck at terminating status), ds controller will never reach the expectation and just stuck here. What you expected to happen: When a pod stuck at terminating, ds controller should start new pods like deployment controller, instead of waiting for the delete batch finish and ready for next poll. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"8+", GitVersion:"v1.8.1-8+", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-10-25T13:04:04Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig apps /cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
DaemonSet controller waits for old pods to terminate before creating new ones for best-effort at-most-one pod per node. This is a deliberate design decision.

In most cases, DaemonSets won't tolerate two pods scheduled on the same node at the same time; for example, it's common to communicate with DaemonSet pods via . I'm going to close this ticket as this works as designed. Please feel free to reopen the ticket if you think there are still issues to be resolved.
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
why does the DaemonSet controller wait for all pods 50 to be done, before moving to the NEXT 50 ? I completely understand that we want only at most 1 pod per node. But if a pod deletion fails on the node, move on to the next pod as long as the total unavailable pods are less than maxUnavailable. This will allow us to make progress in the face of bad nodes or failed pod deletions due to failing tear down of bad volumes or other reasons.
/assign
thoughts on this one ?
Any progress on this issue? We also encountered this problem. There are thousands of nodes in our prod cluster and nodes often go down, which cause daemonset rolling update stucked, this problem really bother us.
PTAL
/reopen
You can't reopen an issue/PR unless you authored it or you are a collaborator. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
You can't reopen an issue/PR unless you authored it or you are a collaborator. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: "time" "github.com/onsi/ginkgo/v2" <ins> "sigs.k8s.io/yaml" </ins> openapiutil "k8s.io/kube-openapi/pkg/util" "k8s.io/utils/pointer" <del> "sigs.k8s.io/yaml" </del> "k8s.io/apiextensions-apiserver/pkg/apis/apiextensions" apiextensionsv1 "k8s.io/apiextensions-apiserver/pkg/apis/apiextensions/v1"

<img width="1480" alt="image" src="" /kind failing-test <!-- If this is a flake, please add: /kind flake --/sig api-machinery
cc in case you know of something that just merged
Last good run : First bad run : k8s version stamp: GOOD: "v1.30.0-alpha.3.56+" BAD : "v1.30.0-alpha.3.62+" Here's the comparison of what went in: We have a winner :) - Will leave this to !
/assign
Thanks I've opened and kicked off the alpha tests on that PR to ensure the fix sticks.
These tests appear to be passing in latest runs: (although NPD is failing)
yea, for the NPD failure i had a cherry pick to containerd 1.7 here -
/triage accepted


Question: case len(kind) > 0: message = fmt.Sprintf("%s (%s %s)", message, strings.ToLower(verb), kind) } <ins> var causes []api.StatusCause if isUnexpectedResponse { causes = []api.StatusCause{ { Type:    api.CauseTypeUnexpectedServerResponse, Message: serverMessage, }, } } else { causes = nil } </ins> return &StatusError{api.Status{ Status: api.StatusFailure, Code:   code,

If go-restful detects a particular method is not allowed, it will return this error: That is preventing useful methods that resthandler would generate from making it to the client. Expected response:
triggered with
/cc
/cc
Is the right fix to update the client to generate the message versus changing the server to allow the method and report the error? My bias is to improve the client and not change the server as it makes reading our swagger doc easier to not see a bunch of verb unsupported entries. Sent from my iPhone
My bias is also towards not changing the server to keep swagger spec correct. By changing the client, do you mean changing kubectl? can you please clarify how is the json useful to client? It does not contain any useful extra information. On Feb 11, 2015 7:28 PM, "Derek Carr" wrote:
Returning an API response would allow clients that expect API status to get nice messages for free, instead of spitting this out: We can certainly put the burden of interpreting 405 errors on all clients, just seemed nice not to. Also, since the resthandler code expected to be returning API status responses when the method wasn't supported, I wanted to make sure having go-restful intercept and return 405's instead wasn't left as an unintended and unconsidered side-effect.
This was an unintended side effect, but I think the old behavior was an unexpected luxury? I agree from the end-user perspective, the old message was nice, and it's great that the client was politely told that a given verb was not supported on the resource, but I don't agree that a server should appear to accept a method to only report that its not allowed. ----- Original Message ----- From: "Jordan Liggitt" To: "GoogleCloudPlatform/kubernetes" Cc: "Derek Carr" Sent: Thursday, February 12, 2015 1:04:41 PM Subject: Re: [kubernetes] go-restful is emitting less helpful errors than resthandler () Returning an API response would allow clients that expect API status to get nice messages for free, instead of spitting this out: We can certainly put the burden of interpreting 405 errors on all clients, just seemed nice not to.

Also, since the resthandler code expected to be returning API status responses when the method wasn't supported, I wanted to make sure having go-restful intercept and return 405's instead wasn't left as an unintended and unconsidered side-effect. Reply to this email directly or view it on GitHub:
If we want to leave go-restful returning that as an intended and considered side-effect, that's fine. Is the consensus that this issue should change to update kubectl to show a nicer message for that error?
I think so, this is only going to get worse when we support specific types of verbs on specific resources. In practice, it seems fine for kubectl to give a better error specific to the method not allowed response. ----- Original Message ----- From: "Jordan Liggitt" To: "GoogleCloudPlatform/kubernetes" Cc: "Derek Carr" Sent: Thursday, February 12, 2015 1:26:08 PM Subject: Re: [kubernetes] go-restful is emitting less helpful errors than resthandler () If we want to leave go-restful returning that as an intended and considered side-effect, that's fine. Is the consensus that this issue should change to update kubectl to show a nicer message for that error? Reply to this email directly or view it on GitHub:
We could consider submitting a PR to go-restful to allow us to provide an error-generation function or interface implementation.
Yeah, that seems reasonable.
Opened
Copying text from : Some errors return as json and some as plain text. This causes overhead on client side when parsing the errors since need to recheck content type for every error in order not to parse plain text as json (some libraries fail on that). The errors message format should be standardized. Examples: The text/plain errors are from go-restful. cc
+1, thanks
The status objects are always of the default API version, however, rather than the version requested, because the request object isn't available to the error handler.
Rather than: we need:
Can we define an error handler on a web service?
At the moment, it's at the container level:
David and I talked about keeping the API lookup mechanism so we could use that in the error handler (apirequestinfo). We still need it for some use cases (proxies) so it could be used here to identify the version and proposed resource if matched.


Question: return nil } <ins> func (*fakeFramework) RejectWaitingPod(uid types.UID) { } </ins> func (*fakeFramework) ClientSet() clientset.Interface { return nil }

What would you like to be : When the pod is scheduled to the phase and I delete the pod, the pod still remains in the until the plugin allow/reject/timeout(Especially when waiting for a long time). Why is this needed: We need to take the initiative to delete the data in the and terminate the process in waiting state of plugin to ensure data accuracy, if the pod has been deleted /cc /priority important-soon /assign /sig scheduling
I think we can update the event handler for deleting pods to reject the pod. Can you send a PR for that?
OKI will send a pr for it.


Question: "doc.go", "longrunning.go", "maxinflight.go", <del> "panics.go", </del> "timeout.go", <ins> "wrap.go", </ins> ], tags = ["automanaged"], deps = [

This is simple cleanup. It is unnecessarily disconcerting to read through apiserver logs and find every log line is from the file "". We should adjust the code printing these lines, either by changing the file name, or (ideally) by making it more intelligent about logging the source of the call (e.g., callers--).
, i'd like to try this one :).
No activity on this one, will try to fix it.
Has it been resolved? Can I try?
check out for discussion about it.


Question: }, numErrs: 1, }, <ins> { name: "LoadBalancer type cannot have None ClusterIP", tweakSvc: func(oldSvc, newSvc *api.Service) { newSvc.Spec.ClusterIP = "None" newSvc.Spec.Type = api.ServiceTypeLoadBalancer }, numErrs: 1, }, </ins> } for _, tc := range testCases {

Create the following svc: It should fail.
I will try to take a shot at it.
Awesome! Make sure to handle updates in all directions. Historically it is the thing we forget. , Maciej Kwiek wrote:
do you have any examples of this behaviour breaking something? It would help with justifying my PR in code review process :). I know it makes sense to fix this issue, but an example would be nice.
I had a user report thatthey didn't know what was happening - the API should have rejected it with a message that you can't have headless services with load balancers. , Maciej Kwiek wrote:
please what are really use cases where we can use k8s headless service, DB, Proxies .. ?


Question: "aggregator.go", "certs.go", "chunking.go", <ins> "crd_watch.go", </ins> "custom_resource_definition.go", "etcd_failure.go", "framework.go",

I am suspicious that CRD watch is not working, but was unable to find e2e test coverage to persuade myself that it is working. cf .
(in particular, I'm looking for the update tests.)
I think this was due to enabling protobuf, though it also failed in a curl for me, so something strange is happening. (Maybe we have to explicitly ask for json?)
We test CRD watch in and in , but not in . I agree we should add e2e tests as a requirement before GA, and also for the sake of defining conformance.
/area custom-resources
Are we talking about CRD or CR watch and protobuf? The later with protobuf obviously does not work, as commented.
I'm not sure "obviously" is correct - list works, for example. IIRC watch with protobuf just silently fails to do anything - it doesn't return an error. And yes I mean CR watch :-)
it should obviously not work with protobuf, but in the sense that the negotiating serializer switches to JSON if you offer it json and protobuf as alternatives. This seems to be broken.
/cc
I'd like to take this up. I'm familiarising myself with the apiextension code base, please give me some time to come up with a PR :)
We already have integration tests for this. Is an e2e test really needed?
I think we do need an e2e test for CRD watch (among other things) as a requirement for CRD going GA. The most pressing need is to define conformance (which only e2e tests can do). Currently we only test that you can create and delete a CRD, but we don't test that you can actually do anything with it: We should expand the conformance (e2e) test for CRD to include checking that the custom resource gets installed and is usable, including watches. If not for conformance, I think an e2e test for CRD watch would still be appropriate since it's an important end-user flow (a thing we expect users to do on purpose; not an edge case). For example, the apiextensions-apiserver integration test might miss problems that come from embedding into kube-apiserver. Similarly, the kubectl tests are meant to test kubectl client functionality, not to protect against regressions on the server side.

Someone might reasonably change those tests in a way that suits kubectl, without realizing that doing so could leave some server code untested at the e2e level. We also need the e2e test to make sure we honor the guarantees of the raw API, to ensure clients other than kubectl will work.
For our developer purpose I would say that e2e tests are not needed when integration tests are in-place. We can even test with a real kube-apiserver including embdded into kube-apiserver if we want that. With conformance tests might have a point. Though I wonder where the line is between e2e and features we test in integration or unit tests only. We don't test all behaviour in e2e today, by far.
Since this has the help-wanted label, adding some useful info for new contributors to get started: About CRDs: If you are new to the Kubernetes API in general, I'd recommend reading the first. CustomResourceDefinition (CRD) is a way to extend the Kubernetes API. You can read more about it here: If you'd like to do some additional reading about this, I had written something up during the time I did GSoC (shamless plug): About testing: There are lots of integration tests for CRDs and you can get a general idea about testing for CRDs and apiextensions-apiserver in general. There is for CRDs. You can use this and other e2e tests to understand how to write them. contains useful information about how you can run tests. gives general info about e2e tests and mentions tests about writing good e2e tests. Also, you can ping me on Slack (I am there) if you have further questions or need help! :)


Question: ARCH?=amd64 HYPERKUBE_BIN?=_output/dockerized/bin/linux/$(ARCH)/hyperkube <del> BASEIMAGE=gcr.io/google-containers/debian-hyperkube-base-$(ARCH):0.3 </del> <ins> BASEIMAGE=gcr.io/google-containers/debian-hyperkube-base-$(ARCH):0.4 </ins> TEMP_DIR:=$(shell mktemp -d -t hyperkubeXXXXXX) all: build

Is this a BUG REPORT or FEATURE REQUEST?: What happened: When I update the kubelet on a node from 1.7.x to 1.8.0, pods on that node stop being able to connect to services whose pods happen to be on the same node. I use the Kubenet network plugin. What you expected to happen: Pods should be able to connect to a service even if the services's pods are on the same node. How to reproduce it (as minimally and precisely as possible): a cluster the DNS addon another pod that runs on the same node as one of the DNS pods resolution starts to fail when the service redirects the requests to the pod that is on the same node. Anything else we need to know?: I have some packet captures that I can show. Kubelet: 1.7.5 DNS pod: 10.2.4.76 Origin pod: 10.2.4.64 Kubelet: 1.8.0 DNS pod: 10.2.1.254 Origin pod: 10.2.1.234 We can see that in the requests that fail the DNS IP is correctly DNATed to the DNS POD destination address but the response is not masqueraded, it still carries the DNS POD source (it should have an origin of 10.3.0.10). Environment: Kubernetes version (use ):Cloud provider or hardware configuration: Azure StandardA4 OS (e.g. from /etc/os-release): "Container Linux by CoreOS 1465.8.0 (Ladybug)" Kernel (e.g. ): Linux node-1-vm 4.12.14-coreos SMP Wed Sep 20 22:20:05 UTC 2017 x8664 Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40GHz GenuineIntel GNU/Linux Install tools:
/sig node
The problem appears when transitioning from kubelet v1.8.0-alpha.2 to v1.8.0-alpha.3.
Well, I've found the problem. Starting from v1.8.0-alpha.3 the "modprobe" binary is not included in the image and so the br-netfilter kernel module is not loaded. I'm using the image, although through "kubelet-wrapper" from CoreOS. Where should this binary reside?

cc/ 1.8 release team.
There is a PR from so it doesn't get lost.
[MILESTONENOTIFIER] Milestone Issue Current <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details


Question: for _, ip := range sourceRanges { sourceAddressPrefixes = append(sourceAddressPrefixes, ip.String()) } <ins> for _, serviceTag := range serviceTags { sourceAddressPrefixes = append(sourceAddressPrefixes, serviceTag) } </ins> } expectedSecurityRules := []network.SecurityRule{}

Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: When using the loadBalancerSourceRanges property of a service in a manifest, we are force to use a CIDR block. This was discovered while debugging the following issue. What you expected to happen: Azure allows us to use Service Tags (e.g. VirtualNetwork, Internet, etc) WE should be able to specify them form a k8s service manifest. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): 1.7.11 Cloud provider or hardware configuration: Azure OS (e.g. from /etc/os-release): centOS 7.3 Kernel (e.g. ): 3.10 Install tools: Others:
/sig azure
Here's the requested Feature.
SourceRanges is part of service spec, while serviceTag is azure specific. I think we don't expect to add cloud-specific features into Kubernetes API. What about adding a new annotation for this? e.g.
I believe that at this stage, (no out of tree cloud provider logic), using an annotation would be the only way to go unfortunately.
/priority important-soon
/status approved-for-milestone
[MILESTONENOTIFIER] Milestone Issue: Up-to-date for process <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. : New functionality. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details


Question: if err != nil { return fmt.Errorf("error parsing GCE/GKE region from zone %q: %v", zone, err) } <del> if err := gceCloud.DeleteFirewall(loadBalancerName); err != nil && </del> <ins> if err := gceCloud.DeleteFirewall(gcecloud.MakeFirewallName(loadBalancerName)); err != nil && </ins> !IsGoogleAPIHTTPErrorCode(err, http.StatusNotFound) { retErr = err }

A couple of gce-scale-performance runs failed due to failing to delete the network, which in turn was due to a couple of leaked firewall-rules (I manually deleted the network later to fix it): Ref: https://k8s- cc (assigning to you, feel free to reassign as apt)
I digged up a bit and seems like the firewall is created by the service-controller as part of , which should've been deleted in but wasn't. Feel free to close the bug if you think it's just a one-off failure.
Took a quick look, both and runs have these firewall rules initially but doesn't, which indicates something happen in the run. But oddly there is no build log, logs in artifacts are empty as well. Likely that run is badly broken so cluster got terminated forcibly in the middle, and resource are leaked. Will observe this suite a bit longer. /assign /unassign
I think it wasn't caused by this job. We're running this one () and on alternating days in the project. And they both create clusters (network, etc) with the same name. And I think of gce-scale-correctness leaked the firewall (and hence the network) as it runs some correctness tests around load-balancer. That likely caused this job to fail.
Ref The resource leak was likely caused by test timeout --- GCE operations took too long to finish and the cluster was terminated ahead. Because heathcheck and firewall cleanup in gce controller are coupled with targetpool deletion, deleting cluster while targetpool deletion is pending will leak the resources.
Fyi - this affected gce-scale-performance again (ref: run 35 of https://k8s-): Is there a way to make sure they get deleted even in case of a timeout? I have a dirty hack in my mind - of deleting those as part of kube-down. Maybe you have some better suggestions?
Consider the frequency this happens, I think it makes sense to implement some cleanup logic within the e2e test. Will give it a try.
Turned out we already have such cleanup logic, it is just we use an incorrect firewall name and forget to cleanup node health check resources as well. Sent to fix.
Thanks for fixing!
Sure thing, thanks for reporting!
Reopening as it seems like the issue is not fixed yet.

For e.g run-16 of https://k8s- failed due to this: This is also affecting our job, both of them being release-blocking jobs. Could you PTAL?
By any chance we could make and use different names so that it will be easier to target the leaking point? Also, in that way one leaking job will not affect the other. Or maybe we could also make different runs use different network names? The scalability job is expensive so it might be great if newer runs wouldn't be affected by the leaked resources from previous run. I haven't figured out how exactly and when the resource are leaked, but given that the operation cycle of service controller and e2e test process proceed in an asynchronous manner (plus always fail ESIPP tests ), there might be chances we will leak resource even after we put cleanup logic in e2e test. Will take a deeper look later. Also what you suggested before --- deleting those as part of kube-down --- may also be a viable solution.
I thought about this option, but it's better for them to share the same name for network and other resources (instance-prefix, etc) because that way we ensure that if one of them leaks some resources, the other will delete them during it's startup. Else we'll run out of quota in the scale-testing project (for e.g wrt routes or VMs) by trying to start another 5k-node cluster while the previous one still exists. I went to pantheon and deleted the network and that went through successfully (though fails). Wonder if the delete command in kube-down can be modified to delete all sub-resources of the network (like firewalls, etc) before deleting it?
Sounds reasonable to me. We may need yet another env to trigger it? As deleting everything correlated to that network sounds a bit dangerous.
Well, took another look, seems like the actual thing that prevented us from deleting the network was mostly firewall rules, and deleting those sounds safe enough if we are going to delete the network anyway. Will try to make a PR for this.
Thanks for fixing the issue with firewalls. However seems like the latest run of gce-scale-correctness still leaked the network (but doesn't seem due to firewall this time): ref: https://k8s-
I thought we're deleting the subnets in this case too when we're using custom subnets mode.

Yep, we are supposed to. I think the scale CI hit the same issue as described. I've sent a PR for fixing it
For the time-being I manually deleted the subnets and then the network to unblock the next scale job run. But it would be good to have this fixed soon. cc
Closing this issue as we're now deleting all remaining firewall rules in the network before deleting the network itself in kube-down. For other networking resources in general, we have issue .


Question: } func (l *latencyAdapter) Observe(ctx context.Context, verb string, u url.URL, latency time.Duration) { <del> l.m.WithContext(ctx).WithLabelValues(verb, u.String()).Observe(latency.Seconds()) </del> <ins> l.m.WithContext(ctx).WithLabelValues(verb, u.Host).Observe(latency.Seconds()) </ins> } type resultAdapter struct {

Issue originating from: The label seems to be unused, but it can cause issues with high cardinality. On k3s this label has unbound cardinality due to the inclusion of url param like timeout. Examples from my small k3s cluster: ! query returns: Metrics at fault here seems to be: I didn't expect these metrics to generate this many series. Execute the following promQL query or its equivalent for your monitoring platform scraping kubelet: We could mitigate this issue by pruning the query parameter from the label. I am no kubelet expert so I don't know if it is normal that the timeout query parameter is variable, but if not that might be an issue as well. v1.22.3 <detailsNone </details<details</details<details</details<details</details<details</details
/sig instrumentation /sig node /cc
/triage accepted What timeseries is this coming from?
Updated the issue. It should be:
Since these metrics are coming from component-base, I guess we could remove sig-node, wdyt /assign
These metrics appear to have existed for 2 years, suggesting that this bug has been present for many, many releases... is this k3s specific? Edit: rate_limiter metrics weren't registered before 1.22:
/assign FYI
I think these metrics are broken for all components, not just kubelet, so I'd prefer that we try to fix the cardinality issue rather than just deregister from kubelet. I'd suggest we fix them by paring down the URL to something that's actually aggregatable, e.g. becomes That also looks a little like an URL quoting issue, I'd expect the particular namespace is substituted in... Not a release blocker since this has been present since 1.22. /priority important-soon
/remove-sig node
Maybe I misunderstood your point but IIUC the actual namespace value is replaced by the generic value to avoid excessive cardinality.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
were you able to find a work around. We just had the same issue occurred for us on multiple clusters. It brought down Prometheus :(


Question: SubPathExpr: "$(POD_NAME)", }, { <del> Name:      "workdir2", </del> <ins> Name:      "workdir1", </ins> MountPath: "/volume_mount", }, },

/kind cleanup /sig storage /sig node /sig testing Description: Change e2e tests to use EmptyDir instead of HostPath as pre-requisite to promoting VolumeSubpathEnvExpansion to Conformance. Related Issues: Conformance Tests:
/cc
/assign


Question: value: -q volumeMounts: - name: varlog <del> mountPath: /varlog - name: containers </del> <ins> mountPath: /var/log - name: varlibdockercontainers </ins> mountPath: /var/lib/docker/containers terminationGracePeriodSeconds: 30 volumes: - name: varlog hostPath: path: /var/log <del> - name: containers </del> <ins> - name: varlibdockercontainers </ins> hostPath: path: /var/lib/docker/containers

When I run Kubernetes cluster on GKE, it generates a lot of system logs coming from docker and kubelet. Even if my cluster runs just a few pods, I can see dozens or hundreds of log entries per second available in Google Cloud Logging in docker and kubelet logs. This makes the actual logs from my application get lost among them, if I do not set appropriate filter. Also, I am afraid that collecting massive amounts of logs imposes unnecessary workload on fluentd pod in my cluster. It would be good if I could just disable collection of kubelet/docker logs (and potentially other Kubernetes system logs) assuming that I do not want to have these logs exported.
/cc
since you tend to have a vested interest in node debug-ability, what's the minimum you'd want to have available here? From GKE's perspective, our users typically have no interest in these logs, so turning them off by default, even without an easy option to re-enable them, makes sense. Any user sophisticated enough to want them shouldn't have problems SSH'ing to the node for them. Is the story all that different in open-source kubernetes? Does anyone even care to have the kubelet and docker logs collected from the nodes? I've never used cloud logging to debug issues with them, it's always been easier to just SSH to the node. I'd lean towards not collecting them on the nodes in general. Also cc in case he has wisdom he'd like to impart
All I can say is that personally I have never accessed kubelet etc. logs "for real" when debugging -- as Alex says -- it is usually more direct just to SSH in. I would support not collecting these logs to improve cluster stability (but perhaps have a switch / technique to make it easy to collect them if the admin does care about this?).
One last call for to complain, because I'm starting on this now
Also
We want to avoid storing logs onto local disk in the future. In-order to achieve that, we need the existing cluster level logging infrastructure to work. Can the GKE problem be solved by moving system logs to a separate namespace? , Alex Robinson wrote:
How far in the future? Unless that's intended to happen soon, collecting the kubelet and docker logs in the meantime offers little value with a nontrivial resource (cpu+memory+egress) cost.

But yes, we can handle the system logs separately if we'd like. I think the most natural way would be to report them as compute engine logs, which could be done by adding an extra config option into the gcp output plugin.
, Alex Robinson wrote:
-- just to confirm, do you think this fix will be sufficient for you? In GKE, logs from containers will be listed under Container Engine, while logs from Docker and the Kubelet will be listed under Compute Engine, with the logs for each effectively being entirely seperated in the log viewer.
Yes, this will be sufficient in terms of not polluting Container Engine logs with system logs. I am still concerned about unnecessary load on fluentd caused by exporting Docker and Kubelet logs so ideally it would be great not to export these system logs if a user is not interested in them. But this is of slightly lower priority.
Hi, What's the status from this request? Is this available in 1.7.x or higher? Thanks.


Question: } func (rc *resourceMetricsCollector) collectContainerMemoryMetrics(ch chan<- metrics.Metric, pod summary.PodStats, s summary.ContainerStats) { <del> if s.Memory == nil { </del> <ins> if s.Memory == nil || s.Memory.WorkingSetBytes == nil { </ins> return }

kubelet panic when accessing metrics/resource endpoint kubelet log kubelet works fine access metrics/resource endpoint of kubelet No response <details</details<details</details<details</details<details</details<detailsINFO[2022-07-14 05:36:21.474173892Z] Starting CRI-O, version: 1.23.0, git: (dirty) Version: 1.23.0 GitCommit: GitTreeState: dirty BuildDate: 1980-01-01T00:00:00Z GoVersion: go1.17.4 Compiler: gc Platform: linux/amd64 Linkmode: static BuildTags: apparmor, excludegraphdriverdevicemapper, seccomp, selinux SeccompEnabled: true AppArmorEnabled: true </details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/sig node


Question: // After adding, call UpdateNodeAddress to set the CloudProvider provided IPAddresses // So that users do not see any significant delay in IP addresses being filled into the node cnc.updateNodeAddress(curNode, instances) <ins> klog.Infof("Successfully initialized node %s with cloud provider", node.Name) </ins> return nil }) if err != nil { utilruntime.HandleError(err) return } <del> klog.Infof("Successfully initialized node %s with cloud provider", node.Name) </del> } func getCloudTaint(taints []v1.Taint) *v1.Taint {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When creating a cluster with a fairly large number of nodes (~50), some nodes take up to 30 minutes after being to the kubernetes node list to be fully initialized. I am using the openstack external cloud provider. Each call of the AddCloudNode method (in ) takes around 2-4 seconds, though this occasionally spikes up to ~10 seconds. I also see that in the openstack-cloud-controller-manager logs, nodes are being initialized multiple times. This is particularly prominent for the nodes that enter the node list the latest. ! The cloud provider receives an Update event from each node every 10 seconds, but they are supposed to be ignored after the node has been initialized. The problem is that when the event rate (50 per 10 seconds ~ 5 per second) is higher than the processing rate (0.5 per second at best), there must be a queue of events that gets blocked up. The first nodes to be registered in kubernetes start filling up this queue with Update events, which is why it takes 30 minutes for the nodes which are last to get their first Add event processed. The reason that the Update events are not ignored is that the Update event checks the taints on the node object that is provided by the event, which I assume is a snapshot of the node state from the time when the event was triggered. Therefore any Update events that are triggered before the node has its initial Add event processed are treated again as Add events, taking up even more processing time. I tried modifying UpdateCloudNode to get and check the current state of the node, and when running the openstack-cloud-controller-manager with that change all nodes are successfully initialized within 5-20 seconds of first appearing in the node list. What you expected to happen: Each node should only be initialized once, and should be initialized very quickly after appearing in the kubernetes node list.

How to reproduce it (as minimally and precisely as possible): Use a cloud where the AddCloudNode method takes at least a few seconds and create a cluster with a large amount of nodes, or just add a time.Sleep for a few seconds to AddCloudNode. Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: Openstack OS (e.g: ): Kernel (e.g. ): Install tools: Others:
/sig cloud-provider /area controller-manager
/sig scalability Doing so might overload the apiserver, as for each update event we request a node object. cc
thanks for reporting this issue! Can you share what version of the OpenStack external cloud controller manager you are using? cc
/assign
/area cloudprovider
I think in our production kubernetes deployments it's version 0.2, but for my own testing I was replacing with openstack-cloud-controller-manager built from the master branch (and modifying the vendored , which was 1.13). Both were affected. if getting the current state of the node is not feasible at scale, is there any way to determine the age of the Update event when it gets around to being processed? Either by some property of the node that is captured by the event, or by setting up some internal tracking? We know that the Update event snapshot of the node object must be out of date if it was taken before the time when the Add event was processed (assuming that the Add successfully modifies the node), so it would just be a case of discarding the Update event if we can determine that it was from before the Add. I wasn't able to track down exactly where the events originate from, could someone point me in the right direction?
This isn't ideal, but it's not completely unreasonable given how many calls to the cloud provider is done as part of node registration. Yeah, this is a bit unfortunate and I would be happy to discuss ways to improve this. The reason this is hard to work around is that node update events happen frequently due to updates that are triggered from node heartbeats so it's very likely that we receive an update event while the node's original add event is being processed. This may be better starting in v1.14 as we use for node heartbeats but I need to do some testing to validate this.

One solution I think this of is to have a map that stores the current list of nodes under registration (protected by a mutex lock) and only try registering it if it's not in the list. Thoughts?
I opened a tracking issue for this here , this will probably be p0 or p1 (high priority) for v1.15. Thanks for reporting it! Let me know if you'd like to work on this!
before I made the change checking the current state of the node, I did try a quick change which was to store the node names that have already been processed in a map, and check that to skip nodes in the Update event (which also worked). I thought that might cause issues later on, though. Is it definitely the case that a node should never be reprocessed by an Update event -AddCloudNode method once it has been successfully initialized? There's no reason that a node would need to be initialized again, even if the VM is shut down and restarted with a different IP or something like that? I'll start working on this myself, trying the map idea again.
This is enough to fix the problem. The machineIDs are tracked as well since a resize down and then back up can bring back nodes with the same names as old ones. Those nodes would skip the check in Update and go to the Add event first which would replace the name-ID in the map, so it shouldn't really matter, but since mentioned wanting to merge the Add/Update methods this would already support that. If this looks good to you I can open a pull request for it.
To be clear, I think the map should only contain nodes that are undergoing registration, not nodes that are already registered. That way if a node wants to register again we can do so. If a node is shut down & its addresses change, we have a controller to catch that which is separate from node registration. See
Add & Update functions are intentionally different due to logging (we want to log on Add events, but not on Update events)
I agree, the map should contain contain only nodes going through registration. As for deduplication, we can log when initializing the nodes, this might help us refactor this. But this is not a blocker for this issue.
At what point should a node be removed from the map? When the Update method no longer has the uninitialized taint on it's node object?

Yes, I think here would be fine Would be even better if you do it in a line after you add the node to the map.
The problem isn't that the Add and Update events are being processed at the same time, not at all. For example in these logs the initializations of a single node are spread far apart. If my understanding is correct, these would have been the first Add and first 8 Update events triggered after the node appeared in kubernetes. It's just that with the huge volume of Updates from all the other nodes to get through, it never got around to processing these until many minutes after they should have been processed. By which time node has been initialized by the first Add event but the Update events don't know that. So the node gets initialized again for each Update event, blocking the Add events of other nodes, and contributing to why it took 30 minutes for the slowest node to be initialized. The map needs to retain the name/ID of the node it initialized after the initialization is finished because there could be any number of outdated Update events left to process. My suggestion is that once an Update event is observed where it sees the node as initialized, the name/ID can be removed and the following Updates can be rejected the same as they are now.
Seems to me like some API calls to the provider is taking a really long time there which leads to multiple update events piling up before the node was successfully registered. Especially if the open stack provider has backoff delay for API calls. The map needs to retain the name/ID of the node it initialized after the initialization is finished because there could be any number of outdated Update events left to process. My suggestion is that once an Update event is observed where it sees the node as initialized, the name/ID can be removed and the following Updates can be rejected the same as they are now. If a node exists in the map (i.e it's currently being initialized) then just drop the update event. I think it achieves the same behavior but it's easier to understand when you read the controller. You also wouldn't check the map anyways if the node doesn't have the taint so you wouldn't be missing anything from subsequent update events.

Either way, I think both approaches would work, we can discuss further in the PR :)
Sorry, I should clarify, in the logs in my previous comment I was grepping for that one particular node - there were other nodes being initialized in between those lines. The rate of initializations was fairly constant, looking to be around 2 seconds each. After 30 minutes all nodes are properly initialized and there is a big queue of Update events from just the last couple of nodes left over - I'm not sure why these are handled quicker than the ones before, perhaps since it's only for a few nodes the results of the queries to openstack are able to be cached somewhere. ! Taking the logs as an example, the node actually gets initialized at exactly as it says on the first line. The event on the last line is processed finishing at , but the event was triggered before the first event was processed, so the apiv1.Node that the handler handles represents the state of the node before it was initialized. Since it sees the node as uninitialized, it does not discard the event and does the initialization again.
Sorry if I'm missing anything here. For any node with a cloud taint, if we record (via a map) that the node is being initialized (i.e. before we call ), then all following update events where the node still has a taint can be ignored (we can check that by looking at the map). When returns without error, we can remove the node from the map. That should solve ^ this case I think. What do you think?
My thought was that the node should be to the map at the (successful) end of the Add event initializeNode. If the Add fails then the Update should be able to try again, yes? Which call of initializeNode? The initial one from the Add event? Because that should be the only time initializeNode actually get called as long as there are no errors. If you add to the map at the start of the Add initialize and remove from the map at the end of the Add initialize, no Update event processed after that will know that the initialize even happened.
For this particular case, we have to treat Add & Update events in the same way because Add events can be missed (for various reasons). The only thing that matters when either event is being processed is whether or not the taint exists.

So we would add a node into the map on the first event (Add or Update) where we see that it needs to be initialized. This is fine, the update event should be a no-op if the node is initialized anyways.
Yes but the problem here is that an update event created before the initialization but only processed after the initialization is cared about, and it shouldn't be.
I think this is true for most controllers, but this controller is slightly different IMO. We don't care about subsequent updates (unless it's an update from the cloud-controller-manager itself) since any relevant information pertaining to the cloud provider will get overwritten anyways. The provider is also very unlikely to change any information about the node if you asked it again. Even after accounting for that, the controller always gets the before starting the initialization logic to avoid this exact scenario.
In other words, the update event handler is only there to trigger the initialization process but the initialization logic will run independently of the node object passed into the event. Whether this is the correct behavior or not is likely for a different issue :)
Then at the very least we should add another one of these checks after the latest version of the node object is retrieved. Otherwise the whole initialization is being done again on nodes that do not need it. The decision of whether or not to do the initialization at all on receiving an Update event is being done based on the properties of the node object passed in to the event. The latest version of the node object is not retrieved until after the decision has already been made.
If we record what nodes are currently being initialized, we can prevent this IMO. Yeah, the taint determines whether or not to trigger the initialization, but the logic is independent of that & is also idempotent.
Storing the name in the map of nodes being initialized "currently" won't work. In the logs in my previous comment the node was initialized both at and . There's nothing current about that, that's 11 minutes apart. The record in the map needs to persist after the initialization is complete. As long as the events are processed in order of creation, it would be possible to add some lines to my change like this, in UpdateCloudNode on line 207 That would ensure that the node is only removed from the map once it is guaranteed that the next Update events will be discarded based on the nil cloud taint.
^ hard to gauge what this really means.

Both initialization loops could be starting seconds away but finishing 11 minutes apart for various reasons (most likely a call to the provider is hanging?). Can you also log the timestamp at the start of ? As I mentioned in a previous comment, I think both approaches would work but semantically it would be cleaner to remove the node in the map as part of . ^ doing this seems a bit awkward IMO because it's not exactly clear why the map is being updated here and why we are removing a node from a map called if it was just initialized. We're also trying to delete an item in the map on the most common code path. You can achieve similar with: Yes, there can be a small window of time where we receive an update event and the cloud taint still exists but given is already idempotent & fetches the latest node object I think the tradeoff is worth it. Let's continue this discussion in the PR though, I think we can articulate these discussions better when we see the code :)
There are no initializeNode calls that take more than a few seconds. I the time taken from the start of initializeNode to the end of initializeNode to its log line to check this. The raw logs (grepped for this line) are here The red line in the second plot shows the time at which the last new name was observed. So the cluster resize was fully complete with all nodes initialized after 12 minutes, but there were still events being processed almost at 1 hour after starting. I split the distribution of durations for initializeNode into before and after that time when all nodes are properly initialized, but there doesn't seem to be any difference. Most calls take ~1 second, but there are up to 5 of them per second that need to be processed. ! I'll make the PR with what I have now and we can discuss further there.


Question: klog.Warningf("secret %q not found in namespace %q", secretCredentialManager.SecretName, secretCredentialManager.SecretNamespace) } <ins> // Converting server FQIN to lowercase to consolidate with config parsing approach server = strings.ToLower(server) </ins> credential, found := secretCredentialManager.Cache.GetCredential(server) if !found { klog.Errorf("credentials not found for server %q", server)

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: kube-controller-manager is unable to locate the credentials of the vCenter host, even though they are defined and cased identically to the hostname in the cloud-provider-config. What you expected to happen: vCenter hostname should not be case-sensitive, or should be documented that it is. It should successfully locate credentials from the manager How to reproduce it (as minimally and precisely as possible): to an upper-case name kube-controller-manager daemon to update the secret to provision storage with the "thin" storage class will stay suck in pending Anything else we need to know?: Log Messages: ` This appears to be induced by the way the vCenter server name is calculated when the secret containing the credentials is parsed[]. The credentialKey is converted to lower case and is then parsed to lookup the server name. Environment: Kubernetes version (use ): v1.14 Cloud provider or hardware configuration: vSphere OS (e.g: ): Unknown Kernel (e.g. ): Unknown Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig cloud-provider /area/provider/vmware
/area provider/vmware


Question: return podNames, nil } <del> // restartNodes uses provider to do a restart of all nodes in the cluster, // allowing up to nt per node. func restartNodes(provider string, nt time.Duration) error { switch provider { case "gce", "gke": return migRollingUpdateSelf(nt) default: return fmt.Errorf("restartNodes(...) not implemented for %s", provider) </del> <ins> func restartNodes(f *framework.Framework, nodeNames []string) error { // List old boot IDs. oldBootIDs := make(map[string]string) for _, name := range nodeNames { node, err := f.Client.Nodes().Get(name) if err != nil { return fmt.Errorf("error getting node info before reboot: %s", err) } oldBootIDs[name] = node.Status.NodeInfo.BootID </ins> } <del> } // TODO(marekbiskup): Switch this to MIG recreate-instances. This can be done // with the following bash, but needs to be written in Go: // //   # Step 1: Get instance names. //   list=$(gcloud compute instance-groups --project=${PROJECT} --zone=${ZONE} instances --group=${GROUP} list) //   i="" //   for l in $list; do // 	  i="${l##*/},${i}" //   done // //   # Step 2: Start the recreate. //   output=$(gcloud compute instance-groups managed --project=${PROJECT} --zone=${ZONE} recreate-instances ${GROUP} --instance="${i}") //   op=${output##*:} // //   # Step 3: Wait until it's complete. //   status="" //   while [[ "${status}" != "DONE" ]]; do // 	  output=$(gcloud compute instance-groups managed --zone="${ZONE}" get-operation ${op} | grep status) // 	  status=${output##*:} //   done func migRollingUpdateSelf(nt time.Duration) error { By("getting the name of the template for the managed instance group") tmpl, err := framework.MigTemplate() </del> <ins> // Reboot the nodes. args := []string{ "compute", fmt.Sprintf("--project=%s", framework.TestContext.CloudConfig.ProjectID), "instances", "reset", } args = append(args, nodeNames...) args = append(args, fmt.Sprintf("--zone=%s", framework.TestContext.CloudConfig.Zone)) stdout, stderr, err := framework.RunCmd("gcloud", args...) </ins> if err != nil { <del> return fmt.Errorf("couldn't get MIG template name: %v", err) </del> <ins> return fmt.Errorf("error restarting nodes: %snstdout: %snstderr: %s", err, stdout, stderr) } // Wait for their boot IDs to change. for _, name := range nodeNames { if err := wait.Poll(30*time.Second, 5*time.Minute, func() (bool, error) { node, err := f.Client.Nodes().Get(name) if err != nil { return false, fmt.Errorf("error getting node info after reboot: %s", err) } return node.Status.NodeInfo.BootID != oldBootIDs[name], nil }); err != nil { return fmt.Errorf("error waiting for node %s boot ID to change: %s", name, err) } </ins> } <del> return framework.MigRollingUpdate(tmpl, nt) </del> <ins> return nil </ins> }

This test wants to restart all nodes and make sure that kubernetes survives the process. That's a neat test, but it does the actual restart in a poor way. It does a MIG rolling-update to its own image. If a rolling-update already exists in some sort of error state then all future rolling-updates in that project on an instance group with the same name will fail until someone manually cancels the borked update. We can restart in other less-confusing ways. I suggest one of or . Both seem reasonable to me. cc
ref:
ref/


Question: "STANDARD_DS4_V2_PROMO":  32, "STANDARD_DS5_V2":        64, "STANDARD_DS5_V2_PROMO":  64, <ins> "STANDARD_E16-4DS_V4":    32, </ins> "STANDARD_E16-4S_V3":     32, <ins> "STANDARD_E16-8DS_V4":    32, </ins> "STANDARD_E16-8S_V3":     32, "STANDARD_E16AS_V4":      32, "STANDARD_E16A_V4":       32, <ins> "STANDARD_E16DS_V4":      32, "STANDARD_E16D_V4":       32, </ins> "STANDARD_E16S_V3":       32, "STANDARD_E16_V3":        32, "STANDARD_E20AS_V4":      32, "STANDARD_E20A_V4":       32, <ins> "STANDARD_E20DS_V4":      32, "STANDARD_E20D_V4":       32, </ins> "STANDARD_E20S_V3":       32, "STANDARD_E20_V3":        32, "STANDARD_E2AS_V4":       4, "STANDARD_E2A_V4":        4, <ins> "STANDARD_E2DS_V4":       4, "STANDARD_E2D_V4":        4, </ins> "STANDARD_E2S_V3":        4, "STANDARD_E2_V3":         4, <ins> "STANDARD_E32-16DS_V4":   32, </ins> "STANDARD_E32-16S_V3":    32, <ins> "STANDARD_E32-8DS_V4":    32, </ins> "STANDARD_E32-8S_V3":     32, "STANDARD_E32AS_V4":      32, "STANDARD_E32A_V4":       32, <ins> "STANDARD_E32DS_V4":      32, "STANDARD_E32D_V4":       32, </ins> "STANDARD_E32S_V3":       32, "STANDARD_E32_V3":        32, <ins> "STANDARD_E4-2DS_V4":     8, </ins> "STANDARD_E4-2S_V3":      8, "STANDARD_E48AS_V4":      32, "STANDARD_E48A_V4":       32, <ins> "STANDARD_E48DS_V4":      32, "STANDARD_E48D_V4":       32, </ins> "STANDARD_E48S_V3":       32, "STANDARD_E48_V3":        32, "STANDARD_E4AS_V4":       8, "STANDARD_E4A_V4":        8, <ins> "STANDARD_E4DS_V4":       8, "STANDARD_E4D_V4":        8, </ins> "STANDARD_E4S_V3":        8, "STANDARD_E4_V3":         8, <ins> "STANDARD_E64-16DS_V4":   32, </ins> "STANDARD_E64-16S_V3":    32, "STANDARD_E64-32S_V3":    32, "STANDARD_E64AS_V4":      32, "STANDARD_E64A_V4":       32, <ins> "STANDARD_E64DS_V4":      32, "STANDARD_E64D_V4":       32, </ins> "STANDARD_E64IS_V3":      32, "STANDARD_E64I_V3":       32, "STANDARD_E64S_V3":       32, "STANDARD_E64_V3":        32, <ins> "STANDARD_E8-2DS_V4":     16, </ins> "STANDARD_E8-2S_V3":      16, <ins> "STANDARD_E8-4DS_V4":     16, </ins> "STANDARD_E8-4S_V3":      16, "STANDARD_E8AS_V4":       16, "STANDARD_E8A_V4":        16, <ins> "STANDARD_E8DS_V4":       16, "STANDARD_E8D_V4":        16, </ins> "STANDARD_E8S_V3":        16, "STANDARD_E8_V3":         16, "STANDARD_E96AS_V4":      32,

<!-- Please only use this template for submitting enhancement requests -- What would you like to be : related to Why is this needed: /assign /sig azure
here is the way about how to get MaxDataDiskCount: and then build kubelet, copy kubelet to agent node and then get MaxDataDiskCount data: There are 207 VM sizes totally.
Update: this command could get VM size list easily:


Question: }(); err != nil || ok { return err } <del> } else { </del> <ins> } if sliding { </ins> timeCh = t.C() <ins> } for { // Wait for either the context to be cancelled or the next invocation be called </ins> select { case <-doneCh: return ctx.Err() case <-timeCh: } <del> } for { // checking ctx.Err() is slightly faster than checking a select </del> <ins> // IMPORTANT: Because there is no channel priority selection in golang // it is possible for very short timers to "win" the race in the previous select // repeatedly even when the context has been canceled.  We therefore must // explicitly check for context cancellation on every loop and exit if true to // guarantee that we don't invoke condition more than once after context has // been cancelled. </ins> if err := ctx.Err(); err != nil { return err }

While trying to update to use latest version of package and to migrate away from deprecated , function, our tests fail as it seems new function immediately executes given condition twice, which is not mentioned anywhere. Result of executed test specified below: Since has been deprecated and it recommends using , I expected it to have the equivalent semantics to allow easy migration from deprecated functions, as no changelog or documentation mentions new semantics. Playground: Local: file with the following content: as . No response_ <details</details<details</details<details</details<details</details<details</details<details</details
/sig api-machinery
CC since you were involved in
/assign /triage accepted
I'd like to work on this one if it's allowed I think this is a simple fix with few lines removed: i.e.: instead of only wait a loop if immediate is false, we wait a time loop after special execution also if the immediate is true.
/assign
/assign


Question: return printer.PrintObj(info.Object, o.Out) } <del> if !o.DryRun { metadata, err := meta.Accessor(info.Object) if err != nil { return err } </del> <ins> metadata, err := meta.Accessor(info.Object) if err != nil { return err } visitedUids.Insert(string(metadata.GetUID())) if !o.DryRun { </ins> annotationMap := metadata.GetAnnotations() if _, ok := annotationMap[corev1.LastAppliedConfigAnnotation]; !ok { fmt.Fprintf(o.ErrOut, warningNoLastAppliedConfigAnnotation, o.cmdBaseName)

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: The dry run output listed all the resources that would be configured: deployments, HPAs, services, and an ingress: This was expected. However, it then listed all resources again (except for the HPAs) as being pruned: The config was then applied with: As expected, no resources were actually pruned. What you expected to happen: should actually display which resources are going to be pruned. How to reproduce it (as minimally and precisely as possible): a YAML file to with it: run + prune: should be listed as both configured and pruned it again with pruning: that the deployment is configured but not pruned, even though the dry run said it would be. Anything else we need to know?: Was trying to write a devops script that gave a warning before any resources would be pruned. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: GKE OS (e.g. from /etc/os-release): (Lubuntu 18.04) Kernel (e.g. ):Install tools: Others:
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I can reproduce it, and i will try to fix it.


Question: Containers: []v1.Container{ { Name:    "pv-recycler", <del> Image:   "busybox:latest", </del> <ins> Image:   "busybox:1.27", </ins> Command: []string{"/bin/sh"}, Args:    []string{"-c", "test -e /scrub && rm -rf /scrub/..?* /scrub/.[!.]* /scrub/*  && test -z "$(ls -A /scrub)" || exit 1"}, VolumeMounts: []v1.VolumeMount{

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: Use the above two files create one PV and PVC, then delete the PVC, the PV will become failed but not released. The busybox recycler never succeed for power. /sig storage What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration**: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/kind bug The solution is like image, we should enable end user can customize the recycle image. It will not only fail on power but also s390x, so we should make sure the recycle image can be customized. /cc any comments for the solution?
Dup of . This issue can be closed, since we've already got flags and for this.
Just search and , but found that they are only defined as flags and no one is using them, can you please help double check, am I missing anything?
Also the claiming that it does not support multi node cluster, this is not acceptable.
: please see Please try them out, if they do not work then we can see how to get those fixed (and avoid adding yet another config option)
Yes, since we may only got only one (maybe setup several for LB, but they can be considered identical), it is necessary for running on non-amd64 platorm to have the right busybox image, which is exactly trying to fix. I've reopened . , so the simplest way is just replacing to . And docker will help us choose the right os-arch busybox image. What's remaining that mentioned is whether we should adding a new flag to add registry url for such recycler pod image exclusively.
Thanks and I also found an example here We do not need the registry url as the pod template was customized by the end user and they can define where got the recycler images. The problem is that the customized recycler will not work in mixed environments as only one recycle pod can be specified in controller manager, so +1 to , we should make support multi arch.

not quite catch your idea of how to support, can you please provide some dummy yaml templates and some configuration files to explain? Thanks!
Actually all the , including , has been updated along with . This means will help us make the decision to choose the right busybox image. You can have a try to on different architectures, like and . A can help you and multi-arch docker images. With this support, on different os/arch can pull the right image with a same busybox image. As long as we have used such multi-arch images, those flags can work in a multi-node cluster.


Question: err := framework.CheckTestingNSDeletedExcept(cs, ns) framework.ExpectNoError(err) err = framework.WaitForPodsRunningReady(cs, metav1.NamespaceSystem, int32(systemPodsNo), 0, framework.PodReadyBeforeTimeout, ignoreLabels) Expect(err).NotTo(HaveOccurred()) })

Chronically failing test. Perhaps covers deprecated behavior? /sig scalability /assign
shows up in
/sig scheduling is one of scheduling e2e test.
/assign I'll check this one.
According to the log, the balanced pods were not created by scheduler, the cpu/mem usage was unchanged.
This test failing every other run. Fix or disable? Blocking 1.7?
similar to , move to 1.8, but I'll still working on that.


Question: github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg= github.com/google/martian v2.1.0+incompatible/go.mod h1:9I4somxYTbIHy5NJKHRl3wXiIaQGbYVAs8BPL6v8lEs= github.com/google/pprof v0.0.0-20181206194817-3ea8567a2e57/go.mod h1:zfwlbNMJ+OItoe0UupaVj+oy1omPYYDuagoSzA8v9mc= <ins> github.com/google/uuid v1.1.1 h1:Gkbcsh/GbpXz7lPftLA3P6TYMwjCLYm83jiFQZF/3gY= </ins> github.com/google/uuid v1.1.1/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= github.com/googleapis/gax-go/v2 v2.0.4/go.mod h1:0Wqv26UfaUD9n4G6kQubkQ+KchISgw+vpHVxEJEs9eg= github.com/googleapis/gnostic v0.0.0-20170729233727-0c5108395e2d/go.mod h1:sJBsCZ4ayReDTBIg8b9dl28c5xFWyhBTVRp3pOg5EKY=

Minimal Repro: Create 2 different disks Create 2 Pods each using one of the disks Start both pods on the same node You will notice one of the disks gets ed to a path: , then both disks are ed from to a target path: which is similar but at least uses the pod . Both of these volumes are being treated as the same volume in the Kubelet even though they are definitely not the same volume. Note: it may not be "treated" as the same volume - it may just be that since the paths are the same the following volumes see that the path is already a mountpoint and skip the step. This is the more likely scenario based on observations. The value is the "name" of the inline-volume. These can obviously be the same across different pods - we should actually be placing it in a more unique directory (based off the translated maybe?). /cc
/sig storage
The inline volume example I'm using (ignore the fact that is a complete misnomer): I've confirmed that changing the "name" makes inline volumes work well since there is no longer a collision on the paths
/assign
Alright, problem is sort of twofold: The referred to above for inline-volumes is set here: As we can see from the comment: it is set to the because the in-tree CSI Plugin makes a lot of naming assumptions based on the which for PVC/PVs is the PV name (unique in a cluster). Primarily, during unmount the CSI Plugin tries to from a "well known ": which happens to be some path based on the which for the inline volume is just the ( above). So we run into the problem that if we use the directory is not sufficiently unique for different disks. And if we use something else like then the unmount pathway is unable to find the . I think using is a no-go. So we need to use something sufficiently unique (like ) for the ( above) and fix up some of the other codepaths to either use different "well-known paths" to write to, or to fix up path generation code so they give us the correct locations. I'm also going to check to see whether 's refactor fixes this issue (because it might since it translates things earlier)
OK. Really good news.

your refactor in its current state actually fixes this issue if we set the to something unique here: Looks like the early translation is also making the pathway from the "correct" dir for inline migration. So I will create a PR that fixes this issue that should merge after


Question: expected    bool }{ { <del> description: "one of the weights is MaxInt64", configs:     []priorities.PriorityConfig{{Weight: math.MaxInt64}, {Weight: 5}}, </del> <ins> description: "one of the weights is MaxTotalPriority(MaxInt64)", configs:     []priorities.PriorityConfig{{Weight: api.MaxTotalPriority}, {Weight: 5}}, </ins> expected:    true, }, { description: "after multiplication with MaxPriority the weight is larger than MaxWeight", <del> configs:     []priorities.PriorityConfig{{Weight: math.MaxInt64/api.MaxPriority + api.MaxPriority}, {Weight: 5}}, </del> <ins> configs:     []priorities.PriorityConfig{{Weight: api.MaxWeight + api.MaxPriority}, {Weight: 5}}, </ins> expected:    true, }, {

MaxInt const is exported in scheduler api although it is not needed to be exported. It is only used by . discussed that the common practice is to have API constants in /sig scheduling
cc
The problem is that it is part of a v1 API :(
Ok, it turns out they are not actually considered part of the API, and so we can remove them:
Should I open a PR to address removing those const values from ?
Yes, we can remove those in a followup PR to the one handling sized values.
Yes, this was the plan. /assign
remember to remove as well :)