Question: w.proberResultsMetricLabels = prometheus.Labels{ "probe_type":     w.probeType.String(), "container_name": w.container.Name, <ins> "container":      w.container.Name, </ins> "pod_name":       w.pod.Name, <ins> "pod":            w.pod.Name, </ins> "namespace":      w.pod.Namespace, "pod_uid":        string(w.pod.UID), }

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What's happening: Currently, the metrics use the label "podname" to refer to the name of the pod in a timeseries and "containername" to refer to container names within that pod. kube-state-metrics, however, uses "pod" to refer to the names of pods and "container" to refer to names of containers. This difference makes it difficult to do joins on KSM metrics and cadvisor metrics, leading me to write queries like this one (which calculates the percentage of real utilization of quota per container -- namespace and podname templated in Grafana): With normalized labels I could just write Hence, I propose renaming the "podname" labels to "pod" and "containername" labels to "container" to normalize between the two data sources. I can attempt to write a patch for this. Why propose this here rather than telling KSM to match the cadvisor metrics? I reported this against KSM first: The indicate the usage of "container" (vs. containername), "pod" (vs. podname), etc.: Fixing this would ensure the cadvisor metrics conform with the community guidelines. Environment: Kubernetes version (use ): 1.8.7-21 with internal patches Kube-state-metrics image version: 1.2.0
/sig instrumentation
I just this to be discussed in the sig-instrumentation meeting, personally I'm all for it and we've discussed this a number of times. we should figure out the timeline for node-metrics being reworked, if that is going to take a while then I think we should make these changes.
considering that now basically has a dependent change that has to go through first, I think that's ok
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with .


Question: <del> 1.23rc2 </del>  No newline at end of file <ins> 1.22.5 </ins>  No newline at end of file

sgtm, add an issue labeled as release-blocking and in the milestone to either promote to 1.23.0 or revert this to 1.22.x before our 1.31 release Originally posted by in Kubernetes is being built and tested with go1.23rc2 at HEAD () to let it bake in the CI and help iron out issues, if any, with this version of Go. However, for the 1.31 release, the version of Go should be a non-rc, released version. The release timeline of go1.23.0 is ~same as the release timeline of K8s 1.31.0 (see ). If go1.23.0 is not out by then, we should revert to go1.22.x before cutting K8s 1.31.0. /sig release architecture /area code-organization /cc
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage accepted /cc
/milestone v1.31
cc
xposting relevant slack thread:
Also unlike in the past, missing shipping 1.23.0 in 1.31.0 is not a big deal, we have much more room to switch to 1.23.0 in a patch release after all the compat changes with the go team. So we should not fear reverting if the timelines don't line up.
(though go1.23 adds the thing that makes it workable to default to old go version compat settings, so we'll have to eke out another 5 months on 1.21/1.22 for until we can adopt 1.23 on release branches in ~Jan if we miss 1.31, but that's how it looks like it will have to happen)


Question: return err } <del> return gce.waitForZoneOp(createOp, zone) </del> <ins> err = gce.waitForZoneOp(createOp, zone) if isGCEError(err, "alreadyExists") { glog.Warningf("GCE PD %q already exists, reusing", name) return nil } return err </ins> } func (gce *GCECloud) doDeleteDisk(diskToDelete string) error {

From : Full log: https://k8s-
Summary of my investigation: In the log I can see that for a claim started: must have called GCE volume provisioner because GCE PD was created Right after call there must be either or in the log, see [1]. Neither of these is actually in the log. This indicates that call did not finish at all. 1: On the other hand, is kept inside a . E.g. on this line we can see it is actually there and second run of the same operation was refused: The operation must have been deleted from the map because later we can see that a new operation with the same name is admitted: Conclusion: disappeared from GoRoutineMap without finishing and logging ether success or failure. Initially, I thought that call could have panicked, but there would be panic in the log and there is none. Root cause is still unknown. It looks it happened only once.
And the reason is that controller-manager was restarted after provisioning and before saving the PV to Kubernetes (at to be precise). Naturally, newly started controller-manager had empty GoRoutineMap and tries to provision a PV again. Solution: must be interpreted not as failure but as success, Kubernetes should adopt volumes it created and forgot about them. Patch will follow shortly.


Question: } // WarningsOnCreate returns warnings for the creation of the given object. <del> func (customResourceStrategy) WarningsOnCreate(ctx context.Context, obj runtime.Object) []string { return nil </del> <ins> func (a customResourceStrategy) WarningsOnCreate(ctx context.Context, obj runtime.Object) []string { return generateWarningsFromObj(obj, nil) } func generateWarningsFromObj(obj, old runtime.Object) []string { var allWarnings []string fldPath := field.NewPath("metadata", "finalizers") newObjAccessor, err := meta.Accessor(obj) if err != nil { return allWarnings } newAdded := sets.NewString(newObjAccessor.GetFinalizers()...) if old != nil { oldObjAccessor, err := meta.Accessor(old) if err != nil { return allWarnings } newAdded = newAdded.Difference(sets.NewString(oldObjAccessor.GetFinalizers()...)) } for _, finalizer := range newAdded.List() { allWarnings = append(allWarnings, validateKubeFinalizerName(finalizer, fldPath)...) } return allWarnings </ins> } // Canonicalize normalizes the object after validation.

A client is allowed to add custom finalizers to Custom Resources with a finalizer name that's not fully qualified (e.g. or ). However they're prevented from doing the same on builtin resource types, returning the following error: Invalid value: "": name is neither a standard finalizer name nor is it fully qualified Per , custom finalizer names are expected to be in format "consist of a domain name, a forward slash and the name of the finalizer" (e.g. ). Therefore, I expected the enforcement to be the same for CRs and builtin types. a custom resource on the cluster it to add the edit does not fail the same on a builtin resource (e.g. Namespace or Pod), observe it fails No response Reproducible at least starting from 1.21 and still on latest versions. <details</details<detailsN/A kind cluster </details<detailsN/A - kind cluster accessed via macOS </details<details</details<details</details<details</details
/area api /cc /sig api-machinery
/assign /triage accepted
There is no validation of finalizer in and . Please unassign me if you guys have started to work on it. /assign
it is called from here on custom resources:
lacks validation for finalizer name. I have raised a pr for this. PTAL.


Question: "k8s.io/api/core/v1" extensions "k8s.io/api/extensions/v1beta1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" <ins> "k8s.io/client-go/kubernetes/fake" </ins> "k8s.io/client-go/util/workqueue" "k8s.io/kubernetes/pkg/controller/deployment/util" )

Please see There are no unit tests currently for that method. This would be a good started PR for someone interested in helping with Kubernetes.
/sig apps
/cc
/for-new-contributors
/assign
I want this done by the end of this week so we can cherry-pick both the unit test and back to 1.7 and potentially 1.6.
if nobody else is working on it, i'll look into this.
Hey Please do so and cc me on your PR, I haven't had any time to start on it.
/priority important-soon


Question: "strings" ) <del> func GetVMUUID() (string, error) { result, err := exec.Command("wmic", "csproduct", "get", "UUID").Output() </del> <ins> func getRawUUID() (string, error) { result, err := exec.Command("wmic", "bios", "get", "serialnumber").Output() </ins> if err != nil { <del> return "", fmt.Errorf("error retrieving vm uuid: %s", err) </del> <ins> return "", err </ins> } <del> fields := strings.Fields(string(result)) if len(fields) != 2 { </del> <ins> lines := strings.FieldsFunc(string(result), func(r rune) bool { switch r { case 'n', 'r': return true default: return false } }) if len(lines) != 2 { </ins> return "", fmt.Errorf("received unexpected value retrieving vm uuid: %q", string(result)) } <del> return fields[1], nil </del> <ins> return lines[1], nil </ins> }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Running the vSphere cloud provider on a windows node throws the following error: This is because the vm identifier is retrieved with the command - , which assumes that the uuid and serial are the same. On some vmware vm versions, the uuid and serial are not the same. vmware (comment), vmware (comment) What you expected to happen: Kubelet is able to find vm in vCenter from the identifier looked up on the node. or should be used instead to find the vm identifier. How to reproduce it (as minimally and precisely as possible): Issue only occurs on some vm versions, such as 13. Anything else we need to know?: On the linux side, this issue was resolved by switching from looking up the id from to . See Some discussion about this issue here - Environment: Kubernetes version (use ): 14.0-beta.1 Cloud provider or hardware configuration: Vsphere OS (e.g: ): Windows 2019 Kernel (e.g. ): Install tools: Others:
/sig windows /sig vmware
/milestone v1.14
You must be a member of the GitHub team to set the milestone. If you believe you should be able to issue the /milestone command, please contact your and have them propose you as an additional delegate for this responsibility. <detailsIn response to : /milestone v1.14 Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign


Question: eventRecorder:        options.eventRecorder, informerFactory:      options.informerFactory, metricsRecorder:      options.metricsRecorder, <del> runAllFilters:        options.runAllFilters, </del> extenders:            options.extenders, PodNominator:         options.podNominator, parallelizer:         options.parallelizer,

Since scheduler Policy API was total removed, we don't support any more. Remove unsupported configuration.
/sig scheduling /assign
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scheduling /cc
/sig scheduling (trying to tell the bots to remove )
The label(s) cannot be applied. These labels are supported: <detailsIn response to : /remove-needs sig /remove-label needs-sig Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
It's true scheduler doesn't quite use this field anymore, but I'm wondering if some other project like ClusterAutoscaler or simulator still consumes this via .
There's no place we can config this field now by configure file(before we can apply the change by policy api, but now it's removed). So it seems like unless we support this feature with CC, or this field is useless. To other projects, it's the same. When they are using legacy policy api, they can configure this field, but once they synced with the latest kubernetes, with the policy api removed, they can not configure this anymore. In this situation, we should add support in CC. I don't quite understand the relationship between k/k with k/autoscaling or others, I think they sync with k/k periodic.
As I mentioned above, it's still accessible via a framework runtime programmatically, and thus may be still useful. For example, upon a scheduling failure, a ClusterAutoscaler's implementation may choose to dryrun through all filter plugins to ensure a new provisioned node can fit the pod, instead of resolving the failure while failing at another latter filter plugin.
Oh, I haven't thought of configuration via programmatically, that saved my concern. But a little strange.
That's exactly what I meant. Given it's a public option in framework, and there is no alternative if we disabled it, it'd be good to check with potential consumers.

For now I can think of CA ( and k-sigs/scheduler-simulator ( If they can confirm it's not needed in their usecases, I'm fine with removing .
To k/autoscaler, at least these codes are still there together with the legacy policy api. , looking forward to the response.
is not used in sigs/kube-scheduler-simulator.
this isn't used by cluster autoscaler, we can remove it.


Question: "github.com/stretchr/testify/assert" ) <del> func newTestScaleSet() *scaleSet { ss := newScaleSet(getTestCloud()) </del> <ins> func newTestScaleSet(scaleSetName string, vmList []string) *scaleSet { cloud := getTestCloud() setTestVirtualMachineCloud(cloud, scaleSetName, vmList) ss := newScaleSet(cloud) </ins> return ss.(*scaleSet) } <del> func setTestVirtualMachineScaleSets(ss *scaleSet, scaleSetName string, vmList []string) { </del> <ins> func setTestVirtualMachineCloud(ss *Cloud, scaleSetName string, vmList []string) { </ins> virtualMachineScaleSetsClient := newFakeVirtualMachineScaleSetsClient() scaleSets := make(map[string]map[string]compute.VirtualMachineScaleSet) scaleSets["rg"] = map[string]compute.VirtualMachineScaleSet{

/kind bug What happened: https://k8s-
/sig azure
I find the root cause, and can fix it.
/kind bug
/cc
Cool, thanks


Question: reportedLevels:      []uint64{100, 300, 500, 250, 250}, reportedCPURequests: []resource.Quantity{resource.MustParse("1.0"), resource.MustParse("1.0"), resource.MustParse("1.0"), resource.MustParse("1.0"), resource.MustParse("1.0")}, useMetricsAPI:       true, <ins> recommendations:     []timestampedRecommendation{}, </ins> expectedConditions: []autoscalingv1.HorizontalPodAutoscalerCondition{ <del> {Type: autoscalingv1.AbleToScale, Status: v1.ConditionTrue, Reason: "ScaleDownStabilized"}, {Type: autoscalingv1.ScalingActive, Status: v1.ConditionTrue, Reason: "ValidMetricFound"}, {Type: autoscalingv1.ScalingLimited, Status: v1.ConditionFalse, Reason: "DesiredWithinRange"}, </del> <ins> {Type: autoscalingv1.AbleToScale, Status: v1.ConditionTrue, Reason: "SucceededGetScale"}, {Type: autoscalingv1.ScalingActive, Status: v1.ConditionFalse, Reason: "FailedGetExternalMetric"}, </ins> }, } _, _, _, testEMClient, _ := tc.prepareTestClient(t)

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --If you have 2 or more metrics defined on an HPA and one metric is suggesting to scale down and the other metric is unavailable, the HPA will scale down despite missing metrics. Example: Imagine the following Horizontal Pod Autoscaler resource: And the deployment is currently scaled to replicas. If the metric is suggesting to scale to 1 AND the metric is unavailable e.g. HPA can't fetch the metrics. Then it will happily scale down to despite not knowing what the currently unavailable metric would suggest. It's clearly stated in the docs the scale down should not happen if one or more metrics are unavailable: Until Kubernetes v1.15.x the behavior was as expected. However with the . The logic was removing the special handling of stopping scale down in case of unavailable metrics. Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /bug /sig autoscaling
/triage accepted


Question: recorder:                       kubeDeps.Recorder, cadvisor:                       kubeDeps.CAdvisorInterface, cloud:                          kubeDeps.Cloud, <del> autoDetectCloudProvider:   (kubeletconfigv1alpha1.AutoDetectCloudProvider == cloudProvider), </del> externalCloudProvider:     cloudprovider.IsExternal(cloudProvider), providerID:                providerID, nodeRef:                   nodeRef,

/kind feature What happened: Continuing the discussion from , we should deprecate (or remove since its alpha) the option to set . See comment and for more details. related issues: related kubernetes-dev thread: !topic/kubernetes-dev/UAxwa2inbTA EDIT: Deprecation plan: [x] v1.8 - deprecate auto-detect cloud provider feature in kubelet [ ] v1.9 - set no cloud provider as the default in kubelet but still allow opt in for auto-detect [ ] v1.10 - completely remove the option to use
/sig cluster-lifecycle
/area cloudprovider
cc
/assign wlan0
cc - Wanted to ask you if you had any thoughts for/against this issue?
/assign andrewsykim As discussed in our meeting I'll be following up with sig-node to see if we can deprecate this as it's going against what we want for external cloud providers
FYI
Would have been nice to keep this issue open to track progress in the next releases
Indeed, should be kept open. Next task is now targeting v1.9
Thanks I edited the issue description to include the deprecation plan going forward. Open to changes on the deprecation plan if we think it's too aggressive.
Great, I think this is a good timeline. We need to make sure we communicate it very well though can you add this to the draft release notes in kubernetes/features please?
Looks like you beat me to it , gave it a review.
hehe, yes, it at the same time as everything else ;)
Can you create a PR for this soon to get this early into v1.9 please?
[MILESTONENOTIFIER] Milestone Issue Current <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Escalate to the issue owners; move out of the milestone after 1 attempt. : Adding tests, refactoring, fixing old bugs. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
i took the liberty to move this along (filed ). Hope that's ok.
Thanks appreciate it. I think there were some discussions in the mailing list thread about v1.9 being too soon.

I think it largely depends on how many users actually take advantage of the auto-detect feature. It's not many from what I've seen but we should follow up and be sure.
Any recent discussion about that? I haven't seen any activity recently but I might just be subject to email overflow. I hope we can get this into v1.9 cc
I didn't follow any discussion related to this topic, but thought this is right move to decouple cloudprovider components from the core system. If I remembered correctly, only cloudproviders for GCE, AWS and Azura depend on the feature, which can be easily override. Any concerns for v1.9?
yep, GCE/AWS/Azure -
Please look at and give your LGTM
Done. Just better document in 1.9 release notes. LGTM
Thank you!
Reopening for the removal of the option in v1.10
please move this to 1.10 milestone


Question: "time" kubecontainer "k8s.io/kubernetes/pkg/kubelet/container" <del> containertest "k8s.io/kubernetes/pkg/kubelet/container/testing" </del> ) func testGetContainersToDeleteInPod(t *testing.T) {

Kubelet evicts pods whenever there is memory or disk pressure. It marks the pods as to evict them. To help with reducing disk pressure, kubelet should delete all containers belonging to pods right away. This will result in loss of logs. In the future, we can copy out logs from dead containers before deleting them. Logs can be rotated separately from dead containers. IMO, this feature is required for disk usage based evictions to work reliably in kubelet.


Question: } func (ts *azureTokenSourceDeviceCode) Token() (*azureToken, error) { <del> oauthConfig, err := adal.NewOAuthConfig(ts.environment.ActiveDirectoryEndpoint, ts.tenantID) </del> <ins> oauthConfig, err := adal.NewOAuthConfigWithAPIVersion(ts.environment.ActiveDirectoryEndpoint, ts.tenantID, nil) </ins> if err != nil { return nil, fmt.Errorf("building the OAuth configuration for device code authentication: %v", err) }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: In the to acquire AAD token, because of the inclusion of the API version in the URLs for authorization, the returned token is not compatible with many AAD scenarios like OIDC and On-Behalf-Of flow. There is a long list of issues documenting this problem: The fix is pretty simple, use with set to . This incompatible behavior impacts What you expected to happen: Follow the to acquire AAD token How to reproduce it (as minimally and precisely as possible): use kubectl to get AAD token, paste token into and verify the audience claim Anything else we need to know?: Environment: Kubernetes version (use ): all versions Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/area provider-azure
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /area provider-azure Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/area provider/azure
/sig cli
/sig cloud-provider
/assign


Question: description:                "static policy - broken reservation", cpuPolicyName:              "static", nodeAllocatableReservation: v1.ResourceList{}, <del> panicMsg:                   "unable to determine reserved CPU resources for static policy", </del> <ins> expectedError:              fmt.Errorf("unable to determine reserved CPU resources for static policy"), </ins> skipIfPermissionsError:     true, }, { description:                "static policy - no CPU resources", cpuPolicyName:              "static", nodeAllocatableReservation: v1.ResourceList{v1.ResourceCPU: *resource.NewQuantity(0, resource.DecimalSI)}, <del> panicMsg:                   "the static policy requires systemreserved.cpu + kubereserved.cpu to be greater than zero", </del> <ins> expectedError:              fmt.Errorf("the static policy requires systemreserved.cpu + kubereserved.cpu to be greater than zero"), </ins> skipIfPermissionsError:     true, }, }

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: When enable cpumanager by , but not set and , kubelet will get panic error in log What you expected to happen: Should graceful exit with error not panic. How to reproduce it (as minimally and precisely as possible): set and but not set and Anything else we need to know?:
/sig node
Great that it's not panicing anymore, but I'd also love to get the "requires + to be greater than zero" hint in this particular case. It looks like even with this doesn't seem to be the case (since it errors out with the non-actionable "unable to determine reserved CPU resources for static policy". Should I open another issue for this?


Question: registryBurst           = flag.Int("registry_burst", 10, "Maximum size of a bursty pulls, temporarily allows pulls to burst to this number, while still not exceeding registry_qps.  Only used if --registry_qps > 0") runonce                 = flag.Bool("runonce", false, "If true, exit after spawning pods from local manifests or remote urls. Exclusive with --etcd_servers and --enable-server") enableDebuggingHandlers = flag.Bool("enable_debugging_handlers", true, "Enables server endpoints for log collection and local running of containers and commands") <del> minimumGCAge            = flag.Duration("minimum_container_ttl_duration", 0, "Minimum age for a finished container before it is garbage collected.  Examples: '300ms', '10s' or '2h45m'") </del> <ins> minimumGCAge            = flag.Duration("minimum_container_ttl_duration", 1*time.Minute, "Minimum age for a finished container before it is garbage collected.  Examples: '300ms', '10s' or '2h45m'") </ins> maxContainerCount       = flag.Int("maximum_dead_containers_per_container", 5, "Maximum number of old instances of a container to retain per container.  Each container takes up some disk space.  Default: 5.") authPath                = flag.String("auth_path", "", "Path to .kubernetes_auth file, specifying how to authenticate to API server.") cAdvisorPort            = flag.Uint("cadvisor_port", 4194, "The port of the localhost cAdvisor endpoint")

Things like SyncFrequency should have a lower-bound of acceptable, and that really should be 0ns.
The main point of this is now fixed, though we could be better about input checking all over


Question: cp ~/kube/init_scripts/* /etc/init.d/ groupadd -f -r kube-cert <del> ${PROXY_SETTING} ~/kube/make-ca-cert.sh "${MASTER_IP}" "${EXTRA_SANS}" </del> <ins> ${PROXY_SETTING} DEBUG='${DEBUG}' ~/kube/make-ca-cert.sh "${MASTER_IP}" "${EXTRA_SANS}" </ins> mkdir -p /opt/bin/ cp ~/kube/master/* /opt/bin/ service etcd start

cluster/saltbase/salt/generate-cert/make-ca- sets but does not echo what it is doing, so in the case of an error there is no evidence of which command produced the error.


Question: if len(svc.Spec.Ports) > 0 { expectedFrontendIPCount++ expectedFrontendIP := ExpectedFrontendIPInfo{ <del> Name:   az.getFrontendIPConfigName(&svc, subnet(&svc)), </del> <ins> Name:   az.getFrontendIPConfigName(&svc), </ins> Subnet: subnet(&svc), } expectedFrontendIPs = append(expectedFrontendIPs, expectedFrontendIP) } for _, wantedRule := range svc.Spec.Ports { expectedRuleCount++ <del> wantedRuleName := az.getLoadBalancerRuleName(&svc, wantedRule.Protocol, wantedRule.Port, subnet(&svc)) </del> <ins> wantedRuleName := az.getLoadBalancerRuleName(&svc, wantedRule.Protocol, wantedRule.Port) </ins> foundRule := false for _, actualRule := range *loadBalancer.LoadBalancingRules { if strings.EqualFold(*actualRule.Name, wantedRuleName) &&

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When the specified internal LB uses a subnet with longer name, the LB configuration will fail with following error: What you expected to happen: When constructing the frontend IP configuration name and the load balancing rules name, the subnet name should be truncated if too long. How to reproduce it (as minimally and precisely as possible): Use a subnet that has long name (70 charactors) and create ILB service. Anything else we need to know?: Environment: Kubernetes version (use ): 1.15 Cloud provider or hardware configuration: azure OS (e.g: ): Ubuntu 16.04 Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig cloud-provider /area provider/azure /assign


Question: if c.Type == certificates.CertificateDenied { return false, fmt.Errorf("certificate signing request is not approved, reason: %v, message: %v", c.Reason, c.Message) } <del> if c.Type == certificates.CertificateApproved && csr.Status.Certificate != nil { return true, nil </del> <ins> if c.Type == certificates.CertificateApproved { if csr.Status.Certificate != nil { klog.V(2).Infof("certificate signing request %s is issued", csr.Name) return true, nil } klog.V(2).Infof("certificate signing request %s is approved, waiting to be issued", csr.Name) </ins> } } return false, nil

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Currently, during kubelet bootstrapping, while waiting for the node certificate being signed, there is no logs showing what is happening. I would like to add some logging to for easier troubleshooting. Why is this needed: When troubleshooting kubelet bootstrapping process on a node, if there is some issue with certificate signing mechanism, kubelet will wait for its certificate being signed for without a single line of log. Often, the last lines of kubelet log would look like the following, and kubelet will be in an confusing "infinite" wait. And then, some healthchecking monitor for kubelet may kick in and restart kubelet, resulting in a confusing loop. Adding some logging here will greatly help troubleshooting bootstrapping process on the node side. /sig node
The logging would be in the following func: I think we should control the frequency of logging (by remembering the timestamp of last log).


Question: "k8s.io/apiserver/pkg/authentication/request/headerrequest" unionauth "k8s.io/apiserver/pkg/authentication/request/union" "k8s.io/apiserver/pkg/authentication/request/x509" <del> "k8s.io/apiserver/pkg/authentication/user" </del> webhooktoken "k8s.io/apiserver/plugin/pkg/authenticator/token/webhook" authenticationclient "k8s.io/client-go/kubernetes/typed/authentication/v1beta1" "k8s.io/client-go/util/cert"

When an auth proxy tries to proxy an anonymous request, the user.Info ends up with the group system:authenticated, which is wrong.


Question: <del> This file has moved to [https://github.com/kubernetes/examples/blob/master/staging/volumes/flexvolume/README.md](https://github.com/kubernetes/examples/blob/master/staging/volumes/flexvolume/README.md) </del> <ins> This file has moved to [https://github.com/kubernetes/community/blob/master/contributors/devel/flexvolume.md](https://github.com/kubernetes/community/blob/master/contributors/devel/flexvolume.md) </ins>

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Some url in markdown file is invalid or need to be redirected to dest. url one time or more. What you expected to happen: So cleaning them up is necessary. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration**: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
This can be closed now. It is fixed in
/close


Question: } return &unstructured.Unstructured{Object: u}, nil } <ins> type noCorrespondingTypeErr struct { gvk schema.GroupVersionKind } func newNoCorrespondingTypeError(gvk schema.GroupVersionKind) error { return &noCorrespondingTypeErr{gvk: gvk} } func (k *noCorrespondingTypeErr) Error() string { return fmt.Sprintf("no corresponding type for %v", k.gvk) } func isNoCorrespondingTypeError(err error) bool { if err == nil { return false } _, ok := err.(*noCorrespondingTypeErr) return ok } </ins>

Found by failing TestApplyConvertsManagedFieldsVersion test in () It seems that conversion depends on schema info that is collected from served API versions only. However, managedFields stores data in etcd from REST versions served in previous releases which may no longer be served. That means that we have to be able to deal with managedFields persisted via API calls to versions that are no longer served, but which we still have and understand conversions for. /cc /sig api-machinery /wg apply /priority important-soon
This may be fixed in a more recent version of structured-merge-diff. I'll look into it
/assign
At one point we were using the types registered to the scheme to generate the openAPI definitions used by apply, but we switched to using the installed endpoints instead, because it was less invasive (We needed to expose which types are convertible to which other types, and also a way to skip generating openAPI for internal types). We might want to revisit the first approach, given this use case.


Question: ProxierMinSyncPeriod time.Duration NodeLabels           map[string]string RegisterWithTaints   []core.Taint <ins> MaxPods              int ExtendedResources    map[string]string </ins> } const (

A kubernetes cluster is normally constructed from a variety of machine templates. Different instance groups may have different CPU, memory capacity. They may be equipped with specialized accelerators, network devices. I think we should have an extensible way for users to custom node configuration to simulate real production cluster states. Currently, nodelabel and nodetaints are supported. I am consider to accept a template with more configuration like resources so we won't grow option list in the future. Without this feature, it's hard for users to simulate heterogeneous cluster.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/sig scalability


Question: } if !IsConflict(NewConflict(resource("tests"), "2", errors.New("message"))) { <del> t.Errorf("expected to be conflict") </del> <ins> t.Errorf("expected to be %s", metav1.StatusReasonAlreadyExists) </ins> } if !IsNotFound(NewNotFound(resource("tests"), "3")) { t.Errorf("expected to be %s", metav1.StatusReasonNotFound)

Spawned from is converting an AlreadyExists error into a server timeout error if it has a generateName That means that this returns really confusing errors:
/cc
- a simple if for the objectMeta.Name seems to do the trick... is this enough? or were you looking for better way instead of NewServerTimeoutForKind ?
if no name was provided originally, the alreadyexists error occurs after the name field is populated with a generated name, so returning in that case makes this entire function a no-op in every case I can find. I just think inferring a ServerTimeout error from an AlreadyExists error and the presence of GenerateName is really odd
This needs to be triaged as a release-blocker or not for 1.5
Not a blocker
thanks
Is it appropriate to move this to the next milestone or clear the 1.5 milestone? (and remove the non-release-blocker tag as well)
Issues go stale after 30d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
/remove-lifecycle rotten
/help
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cli
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with .

Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Should we return another instead of to include the generate name information ?
seems like the correct error in other cases involving an object that already exists with that name.
Looks like is not active due to age. If anyone is interested, please feel free to work on this! Like mentioned in , a unit test along with the required changes would be good to add.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Would be really helpful if solved :) /remove-lifecycle rotten
I am new to Kubernetes community and was going through role board recommendations on older threads - If everyone is ok, can I pick this issue and work on it.
It seems already created PR for this issue - It doesn't seem to have merged. - Let me know if you are still working, else I can take a stab (Though most of the work is already done by you) to further drive this.
I'm not working on it. Please take it forward. Thanks
Thanks for the confirmation. I will work on this.
I have taken a look at the complete flow of the issue. From what I understand, there are a couple of choices Option - (1) Validate in function if an object has both name and generateName and fail the request if that is the case. Option - (2) in , set generateName to empty if it also has a name. This will make sure that the error in CheckGeneratedNameError is (when the generated name is conflicting with the already existing name of some other object). This is essentially the approach took in this , which no longer is actively worked on. /cc
Issues go stale after 90d of inactivity.

Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/lifecycle frozen
/needs-triage It's kind of embarrassing that this is still open.
(I think the fix may be to delete the mentioned function entirely-- why would we describe this situation as a timeout error?)
do you know someone who's interested working on this? We will come back on this issue in the next triage meeting.
/help
/triage accepted
The origin of :


Question: // Rkt-specific settings. fs.StringVar(&s.RktPath, "rkt-path", s.RktPath, "Path of rkt binary. Leave empty to use the first rkt in $PATH.  Only used if --container-runtime='rkt'.") <ins> fs.MarkDeprecated("rkt-path", "will be removed in a future version. Rktnetes has been deprecated in favor of rktlet (https://github.com/kubernetes-incubator/rktlet).") </ins> fs.StringVar(&s.RktAPIEndpoint, "rkt-api-endpoint", s.RktAPIEndpoint, "The endpoint of the rkt API service to communicate with. Only used if --container-runtime='rkt'.") <ins> fs.MarkDeprecated("rkt-api-endpoint", "will be removed in a future version. Rktnetes has been deprecated in favor of rktlet (https://github.com/kubernetes-incubator/rktlet).") </ins> fs.StringVar(&s.RktStage1Image, "rkt-stage1-image", s.RktStage1Image, "image to use as stage1. Local paths and http/https URLs are supported. If empty, the 'stage1.aci' in the same directory as '--rkt-path' will be used.") <del> fs.MarkDeprecated("rkt-stage1-image", "Will be removed in a future version. The default stage1 image will be specified by the rkt configurations, see https://github.com/coreos/rkt/blob/master/Documentation/configuration.md for more details.") </del> <ins> fs.MarkDeprecated("rkt-stage1-image", "will be removed in a future version. Rktnetes has been deprecated in favor of rktlet (https://github.com/kubernetes-incubator/rktlet).") </ins> }

rktnetes is the old, in-tree rkt integration. is new integration via CRI. We should deprecate and remove the rktnetes integration once rktlet is ready to take over. Based on the , I think we can announce the deprecation of rktnetes once rktlet is ready, and remove the integration after one more release. /cc who have been working on rktlet who are the owners of rktnetes
cc who once asked me about this confusion and I suggested him to do a clean up. _<
Thanks, very glad to do the clean up part. Is there any plan for deprecation part?
It depends on the progress of rktlet. The folks who are currently working on rktlet and are migrating off rktnetes would be a better source.
It was included in sig-node's 1.9 planning draft, and discussed at sig-node meeting. It was removed for 1.9 because 1) the user (for example: blablacar) is currently depending on it 2) rktlet is under development. We can announce the deprecation now to prevent more users from using it. Once rktlet is done, we can remove it completely.
We're currently working towards an initial rktlet release, which will be tested by Blablacar. We'll give an update next week with the progress.
what's the current progress of the rktlet release?
Hi sorry for the late reply. We plan on releasing rktlet 0.1.0 this week. In that release we implement basic support for the CRI including fetching images, running pods, CNI networking, logging and exec. We pass 129/145 e2e conformance tests. Some features (e.g. attach, port forward) are not implemented so we don't yet consider it ready for deprecating rktnetes. Those features were out of scope for the work we were contracted to do so I’m not sure when it will be completed.
thanks for the update! I don't think rktnetes ever supports attach or port forward, so I don't think they should be the blocker for deprecating rktnetes. Adding to confirm.

(rktlet maintainers), I'd like to aim at officially deprecating rktnetes in k8s v1.10 (and encourage users to move to rktlet). Do you think this is reasonable? We can discuss more in the sig node meeting or on slack if you have concerns.
From the today's sig-node meeting, current rktlet maintainers ( etc) mentioned that they may not have spare cycles to work on rktlet this release, but even as of today, rktlet already passes more tests than rktnetes. Given that, I propose we deprecate rktnetes in 1.10, and remove the code in the next release. The deprecation process includes Send out a deprecation announcement to kubernetes-users kubernetes-dev and kubernetes-sig-node@ with links to rktlet. Add deprecation warning to kubelet when --container-runtime is set to rkt. Can someone from the CoreOS (owner of rktnetes and initiator of rktlet) comment on whether the plan is okay?
/cc
LGTM, do you need any help from our side?
I haven't been involved in active rkt development for a while, but this plan sounds fine to me based on everything I know about the latest rktnetes work.
The plan sounds good to me. The only concern I have is whether Blablacar needs rktnetes for a longer time (cc
I'd love to hear Blablacar's feedback on this. At the same time, I think it might be reasonable for Blablacar not to upgrade kubernetes until they are ready. There hasn't been much active development in rktnetes, so staying on a known, good version, is better than upgrading kubernetes anyway IMO.
I think the rktnetes maintainers are the best candidate to send out the deprecation announcement :-) Adding deprecation warning is trivial. If anyone wants to help, that'd be great.
Sorry for the delay, I was off last week. We (BlaBlaCar) are moving away from rktnetes implementation so it's ok for us if it gets deprecated. Thanks for asking :)
has sent out the announcement. Will close this once the PR gets merged.
Seems like is dead and does not support v1.10.


Question: wantMinor: 5, wantErr:   false, }, <del> /*{ </del> <ins> { </ins> name:    "multiple declarations of same tag should return an error", tagName: "someVersionTag:version", tagComments: []string{

The commands under will benefit of a batter unit test coverage as seen in Originally posted by in
/sig api-machinery /assign
GitHub didn't allow me to assign the following users: damiansawicki. Note that only with read permissions, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /sig api-machinery Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage accepted
Hi I would like to work on this issue. Could you tell me how many test files are needed per PR under this issue?
Hi the issue is already assigned to me. Would you consider contributing to other issues to avoid duplication?
Hi do you mind if I work on this issue along with you? I would only like to write tests on limited number of files, if that's okay with you...
Hi thanks for your engagement! If you also want to work on this particular issue, we should coordinate. Are you working on something other than ?
Hey currently I'm thinking of shifting to Do let me know if you are already working on that particular directory. Also I could assist you in any of your work on this issue as I'm not preoccupied at the moment.


Question: {"#commentnnameserver 1.2.3.4n#commentnsearch fooncomment", []string{"1.2.3.4"}, []string{"foo"}, []string{}, false}, {"options ", []string{}, []string{}, []string{}, false}, {"options ndots:5 attempts:2", []string{}, []string{}, []string{"ndots:5", "attempts:2"}, false}, <ins> {"options ndots:1 ndots:5 attempts:2", []string{}, []string{}, []string{"ndots:5", "attempts:2"}, false}, {"options ndots:1 edns0 attempts:2 single-request-reopen", []string{}, []string{}, []string{"edns0", "attempts:2", "single-request-reopen", "ndots:1"}, false}, </ins> {"options ndots:1noptions ndots:5 attempts:3", []string{}, []string{}, []string{"ndots:5", "attempts:3"}, false}, <ins> {"options ndots:1 timeout:3 timeout:1 attempts:3noptions ndots:5", []string{}, []string{}, []string{"ndots:5", "timeout:1", "attempts:3"}, false}, {"options ndots:1 attempts:3noptions ndots:1 attempts:3 ndots:5", []string{}, []string{}, []string{"ndots:5", "attempts:3"}, false}, </ins> {"nameserver 1.2.3.4nsearch foonnameserver 5.6.7.8nsearch barnoptions ndots:5 attempts:4", []string{"1.2.3.4", "5.6.7.8"}, []string{"bar"}, []string{"ndots:5", "attempts:4"}, false}, } for i, tc := range testCases {

If the content of is only could recognize the last option and is ignored. And user has to use to workaround the issue. But this format is valid actually and able to read correctly in golang and libc. Able to parse multiple options lines in correctly. Use in the host's file and then inside POD's , it would become No response Any version. N.A. Ubuntu 22.04 No response No response No response
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig sig-node
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /sig sig-node </details
/sig node
Seems this behaviour is introduced by you at , is it an expected behaviour or a bug?
are those options parsed and used correctly or one of both is ignored?
Others are ignored and online the last line is parsed by kubelet correctly.
and in a normal , does libc use all the options or ignore some and use only one?
I tried centos/ubuntu, both will append extra options to the options list and overwrite them if it already exists by the later settings. For instance, is the same as . But I still think the workaround looks a perfered format.
Yes, but it looks like a bug since kubelet's behavior is different.
I tried to fix it in But I am still not sure if this is a valid fix. I will wait for pr review comments or further confirmation about whether this is valid.
/sig network
EDITED The feature was by a long time ago by There is a design kubernetes/community. The PR does not follow the design per The KEP can be found in KEP
/remove-sig node
Previous discussion and interesting comment from
Just want to clarify that these two are not related --- the change does not follow the design.
Just have the book on my desktop. !
I think the book just said could write it as , but didn't mention is invalid.

And since glibc/golang DNS resolve is able to recognize it correctly, should we consider it as a de-facto way that we should support?
what about muslc, does it support it? we have to support muslc too, a lot of busybox/alpine users can break
I did some initial test with busybox:1.34.1-musl. It supports multi options. for , it works with multi-options. ndots:1, ping will fail for .
/assign
I'm trying to understand this, how the options are merged if they are duplicated? the man page for is clear ,notation%20as%20per%20RFC%. for only says it seems that apply the same logic to handle conflict if I try with this only the last option is used (same for muslc and libc) if I have two different options in different lines then things start to get more complicated, because I can not test all possible options combinations and musl resolver is not compatible with glibc resolver , see I'm worried we can break musl based containers by assuming the host configuration, that is mostly libc based, will be compatible with the container. I think we should be conservative and document this behavior of the dns options, and ask users to merge the options in the same line, raise a warning of something too EDIT I see is suggesting to parse all options and merge them in one line :eyes:
Add a small warning in log maybe another option. We may document it with some details. This is a little aggressive way.
I still can't find any spec that says how this is SUPPOSED to work, and I share concerns that this is likely to break someone somewhere. Did someone encounter this in the wild? Does it REALLY make sense to copy ALL of the options from the host config? Does it make sense to copy ANY of them? Looking at the man page, they seem very specific and maybe we should just leave them alone. What does Docker do here?
I found that there is a logic to merging DNS options in the pod DNS spec generating process. So I followed the logic. If the pod is using with , the current kubelet will merge the node's last option line in with . I did a simple test in Centos 7.9 with docker 19.03.8 like below.

keeps the options as is if no new DNS options. will overwrite the local DNS options. there are multi , docker will not merge them and keep them in one line.
Absent a real user who is really impacted by this, my inclination is to (again) do nothing. The risk of breakage is non-zero. The reward is zero (nobody is complaining, IIUC).
Hi, actually I am impacted by this (in production services) . And looks like I am not the first user who raise this, as other issues link to this thread.
Thanks for your input, I checked muslc and verified via code, muslc would parse all options if we write multiple line . So I don't think would break any user. Instead, it addresses the similar problem happens to muslc users as well.
OK. I am accepting it. Since libc's seem to do this anyway, I don't see why we should break it.
Many thanks


Question: } // Check runtime support. Currently only Docker is supported. <del> if runtime != "docker" { return fmt.Errorf("AppArmor is only enabled for 'docker' runtime. Found: %q.", runtime) </del> <ins> if runtime != "docker" && runtime != "remote" { return fmt.Errorf("AppArmor is only enabled for 'docker' and 'remote' runtimes. Found: %q.", runtime) </ins> } return nil

AppArmor validator returns error if the runtime is not . For other runtime integrated via CRI, the runtime will be , and will not be able to provide AppArmor support because of this. Currently, we decided to always pass validation for remote runtime, but we need a better solution in the future. This is similar with , and also related to . /cc
I think we should pass validation for all container runtimes in validateHost, since it is host-wide. We could add runtime-specific validation after .
Let me remove this requirement. /assign


Question: return &v1.PersistentVolume{ ObjectMeta: metav1.ObjectMeta{ <del> // A.K.A InnerVolumeSpecName required to match for Unmount Name: volume.Name, </del> <ins> // Must be unique per disk as it is used as the unique part of the // staging path Name: fmt.Sprintf("%s-%s", GCEPDDriverName, pdSource.PDName), </ins> }, Spec: v1.PersistentVolumeSpec{ PersistentVolumeSource: v1.PersistentVolumeSource{

Minimal Repro: Create 2 different disks Create 2 Pods each using one of the disks Start both pods on the same node You will notice one of the disks gets ed to a path: , then both disks are ed from to a target path: which is similar but at least uses the pod . Both of these volumes are being treated as the same volume in the Kubelet even though they are definitely not the same volume. Note: it may not be "treated" as the same volume - it may just be that since the paths are the same the following volumes see that the path is already a mountpoint and skip the step. This is the more likely scenario based on observations. The value is the "name" of the inline-volume. These can obviously be the same across different pods - we should actually be placing it in a more unique directory (based off the translated maybe?). /cc
/sig storage
The inline volume example I'm using (ignore the fact that is a complete misnomer): I've confirmed that changing the "name" makes inline volumes work well since there is no longer a collision on the paths
/assign
Alright, problem is sort of twofold: The referred to above for inline-volumes is set here: As we can see from the comment: it is set to the because the in-tree CSI Plugin makes a lot of naming assumptions based on the which for PVC/PVs is the PV name (unique in a cluster). Primarily, during unmount the CSI Plugin tries to from a "well known ": which happens to be some path based on the which for the inline volume is just the ( above). So we run into the problem that if we use the directory is not sufficiently unique for different disks. And if we use something else like then the unmount pathway is unable to find the . I think using is a no-go. So we need to use something sufficiently unique (like ) for the ( above) and fix up some of the other codepaths to either use different "well-known paths" to write to, or to fix up path generation code so they give us the correct locations. I'm also going to check to see whether 's refactor fixes this issue (because it might since it translates things earlier)
OK. Really good news.

your refactor in its current state actually fixes this issue if we set the to something unique here: Looks like the early translation is also making the pathway from the "correct" dir for inline migration. So I will create a PR that fixes this issue that should merge after


Question: }() if wait.PollImmediate(Poll, timeout, func() (bool, error) { <del> // We get the new list of pods and replication controllers in every // iteration because more pods come online during startup and we want to // ensure they are also checked. </del> <ins> // We get the new list of pods, replication controllers, and // replica sets in every iteration because more pods come // online during startup and we want to ensure they are also // checked. replicas, replicaOk := int32(0), int32(0) </ins> rcList, err := c.Core().ReplicationControllers(ns).List(api.ListOptions{}) if err != nil { Logf("Error getting replication controllers in namespace '%s': %v", ns, err) return false, nil } <del> replicas := int32(0) </del> for _, rc := range rcList.Items { replicas += rc.Spec.Replicas <ins> replicaOk += rc.Status.ReadyReplicas } rsList, err := c.Extensions().ReplicaSets(ns).List(api.ListOptions{}) if err != nil { Logf("Error getting replication sets in namespace %q: %v", ns, err) return false, nil } for _, rs := range rsList.Items { replicas += rs.Spec.Replicas replicaOk += rs.Status.ReadyReplicas </ins> } podList, err := c.Core().Pods(ns).List(api.ListOptions{})

The logic to account for all pods being ready is incorrectly counting some pods as being controlled by a replication controller. This is causing the 1.3-1.5 upgrade tests to fail. See https://k8s- as an example. Since , the pre-flight checks fail and the rest of the tests never run. I think this was exposed by which changed some things from ReplicationControllers to Deployments. There's a pod being created by a deployment that also matches the selector of a replication controller, and is therefore being incorrectly counted by cc


Question: } status := &v1.LoadBalancerStatus{} <del> status.Ingress = []v1.LoadBalancerIngress{{IP: vip.Address}} </del> if floatingPool != "" && !internalAnnotation { floatIPOpts := floatingips.CreateOpts{ FloatingNetworkID: floatingPool,

When creating a service passing the annotation , two IPs are created in the service, a private and a public: Having a private IP is not a desired scenario for service discovery in federated global environments. If we have, for example, a cluster in GKE and join a local OpenStack cluster, this private IP is written in the DNS zone but the GKE cluster won't be able to access it. We could add an annotation to say we don't want the private IP to be registered in service's LoadBalancer Ingress field, like: or And when it's false, only the public IP would be written in the service, therefore in the DNS zone /kind feature
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig openstack /sig federation
'' sounds better.
I don't like adding annotations just to make things work. I counter-propose just not publishing the private IP whenever there is also a floating-ip (without a configuration option to control this behaviour).


Question: "io/ioutil" "path" "strings" <del> "syscall" </del> "github.com/GoogleCloudPlatform/kubernetes/pkg/kubelet/dockertools" "github.com/GoogleCloudPlatform/kubernetes/pkg/kubelet/network"

/priority critical-urgent /priority failing-test /kind bug /status approved-for-milestone This test has been failing since 2017-11-22 in which is on the , and prevents us from cutting [v1.9.0-beta.1] (kubernetes/sig-release). Is there work ongoing to bring this job back to green? https://k8s- first bad: https://k8s- this is also failing on (non-release-blocking) triage cluster suspect commit range:
[MILESTONENOTIFIER] Milestone Issue Needs Attention Action required: During code freeze, issues in the milestone should be in progress. If this issue is not being actively worked on, please remove it from the milestone. If it is being worked on, please add the label so it can be tracked with other in-flight issues. Note: This issue is marked as , and must be updated every 1 day during code freeze. Example update: <details<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
/remove-area platform/gke bad copy-pasta
This is a new test that was for a 1.9 alpha feature but actually exposed a bug with a different 1.8 alpha feature. Some discussion in . Since this seems to be a bug introduced in 1.8, I'm not sure if it needs to block 1.9. cc


Question: framework.ExpectNoError(err, "failed to delete Pod by collection") ginkgo.By("watching for the Pod to be deleted") <del> ctxUntil, cancel = context.WithTimeout(ctx, 1*time.Minute) </del> <ins> ctxUntil, cancel = context.WithTimeout(ctx, f.Timeouts.PodDelete) </ins> defer cancel() _, err = watchtools.Until(ctxUntil, preDeleteResourceVersion, w, func(event watch.Event) (bool, error) { switch event.Type {

pull-kubernetes-conformance-kind-ga-only-parallel [sig-node] Pods should run through the lifecycle of Pods and PodStatus [Conformance] After is merged. It relies on events which are not reliable as discussed . No response /sig node
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The test is not failing that much looking at the testgrid. Is it actually an issue?
Not so often, but it does happen. So, I think that it is worth fixing.
still flaking -
flaking conformance tests are concerning and should take priority over new conformance test additions
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale I still see this test failing , also in periodic jobs
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale seems like it's is still failing
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
still flake /remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/assign Namanl2001
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/remove-lifecycle rotten
please unassign if you are not working on it. Seems to be still flaking
It still flakes but not that much. I checked the history: adjusted the timeout from 30s to 1m. changed it not to fail for watching Delete Event, and just print it like below: This time, we see the above adding warning and then failed. Looking into kubelet/container runtime log. The pod is just deleted after 1 min, maybe 1min10s. I'm afraid that we have to adjust the timeout of the delete pod again to fix it. Is a valid pod deletion timeout?

I think most deletion timeout of pod is 5min, however this is too long after pod with .
but why it takes so long kubelet to delete a pod? what are all of those ""Failed to delete cgroup paths" messages? can this be hiding some problem with runc or the kubelet?
Sep 19 13:14:56 kinder-external-etcd-worker-2 kubelet[359]: E0919 13:14:56. 359 ] "Error syncing pod, skipping" err="failed to delete cgroup paths for [kubepods besteffort pod53d19839-496a-4be8-8964-] : unable to destroy cgroup paths for cgroup [kubepods besteffort pod53d19839-496a-4be8-8964-] : Timed out while waiting for systemd to remove kubepods-besteffort-" pod="pods-900/pod-test" podUID=-496a-4be8-8964- Sep 19 13:14:56 kinder-external-etcd-worker-2 kubelet[359]: I0919 13:14:56. 359 ] "Failed to delete cgroup paths" cgroupName=[kubepods besteffort pod53d19839-496a-4be8-8964-] err="unable to destroy cgroup paths for cgroup [kubepods besteffort pod53d19839-496a-4be8-8964-] : Timed out while waiting for systemd to remove kubepods-besteffort-" Sep 19 13:14:56 kinder-external-etcd-worker-2 containerd[220]: time="2023-09-19T13:14:56.925525892Z" level=info msg="StartContainer for "e3b435116d2640d7c8166332e1c4bf6722f97ac1401035cbe443704c8b442efd" returns successfully" Sep 19 13:14:56 kinder-external-etcd-worker-2 kubelet[359]: I0919 13:14:56.

359 ] "Failed to delete cgroup paths" cgroupName=[kubepods besteffort pod53d19839-496a-4be8-8964-] err="unable to destroy cgroup paths for cgroup [kubepods besteffort pod53d19839-496a-4be8-8964-] : Timed out while waiting for systemd to remove kubepods-besteffort-" Sep 19 13:15:01 kinder-external-etcd-worker-2 kubelet[359]: I0919 13:15:01. 359 ] "Cleaned up orphaned pod volumes dir" podUID=-496a-4be8-8964- path="/var/lib/kubelet/pods/-496a-4be8-8964-/volumes" It means timeout. Another failure is not for the same reason.
/joke
I wish I could clean mirrors for a living. It's just something I can see myself doing. <detailsIn response to : /joke Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This is starting to be worrisome
another failure
was another occurrence of this I think. I'm seeing the timeout error for deleting the pod in the build logs: And then the error: It has the same sanbok not found error in the kind-worker logs: Along with the same cgroup error in the logs: So it's probably an instance of the first case mentioned caused by timeout.
Flake in . Delete timeout is 1 minute here. It still need more investigation why this is not deleted in 1 min.(heavy load or logic problems.)
Flake twice in yersterday.


Question: <ins> #!/bin/bash # Copyright 2015 The Kubernetes Authors All rights reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # #     http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # This script is invoked by crond every minute to check if monit is # up and oom protected. If down it restarts monit; otherwise, it exits # after applying oom_score_adj PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin PROGNAME=monit FULLPATH=/usr/bin/$PROGNAME pids=$(pidof $FULLPATH) if [[ "${pids}" == "" ]]; then service $PROGNAME start sleep 10 fi # Apply oom_score_adj: -901 to processes pids=$(pidof $FULLPATH) for pid in "${pids}"; do echo -901 > /proc/$pid/oom_score_adj done </ins>

The cluster I created on Friday (at head) was running over the weekend, and when I tried to talk to it this morning, I couldn't. SSH'ing into the master, I couldn't hit and Docker isn't running and neither is monit so what happened to monit? found in monit. My questions are: "Why?" and "Should we try to debug this further?" pointed out that a possible preventative measure is to have a cron job running that kicks monit if it's dead. I left the cluster running, so if anyone would like me to run commands or would like to dig in more, please let me know.
I haven't thought much about this, but it may be related to Docker being slow at restarting? As a side note, it seems like pretty poor design that monit dies if it fails to restart something.
Could you please take a look at this? Thanks!
you started the cluster on Friday, could you connect to apiserver using kubectl ? you see any errors in the file ?
cc
, the cluster functioned on Friday; I was running tests against it. (If you look at the monit log I pasted above, it appears it was functioning for quite a while.) don't see any errors there, but I can dump it here if it would help (it's quite long, obviously).
out-of-memory errors seen/logged around the time when monit failed to fork a child process. seems to consume 400m of memory. not sure why it is so high the fork did fail because of insufficient memory, but this is a guess and no logs to substantiate it
Just as an explicit followup, there were at least 8 out-of-memory errors that happened on that instance (a master) from the logs in —is this normal? I can provide several different logs or VM access for interested parties...
Regardless of the cause of the problem (which I agree would be interesting to discover) it seems like we need a cron job or something to make sure that monit is always running. can you look into how we can babysit our babysitter?
I will take a look this one today since there are several ones related to each other: , , ...


Question: getReactor                 func(action clienttesting.Action) (bool, runtime.Object, error) onRepeatedHeartbeatFailure func() expectErr                  bool <ins> client                     *fake.Clientset </ins> }{ { desc: "no errors",

Some external controller deleted node object while kubelet was alive Due to ignoring errors in kubelet's lease logic, it created lease without ownerReference set Kubelet doesn't create lease without ownerReference set in any case, even pathological one Some external controller deletes node object Garbage collector properly deletes associated lease object Kubelet tries to update object and it fails on Conflict (saying that UID doesn't match): We go to branch, call backoffEnsureLease and assign result to base, but backoffEnsureLease may return potentially invalid object if ensureLease returned error (which happens when newLeasePostProcessFunc returns error, e.g. when node doesn't exist). We The next loop calls c.newLease on the incorrect base and ignores error in line 181 (so the second attempt to call newLeasePostProcessFunc which also fails with an error) We do UPDATE with incomplete (without effect of newLeasePostProcessFunc) object Given that leases allow CreateOnUpdate a new object, without ownerReference is created: No response <details21.10 </details<details</details<details</details<details</details<details</details<details</details
/sig node
/assign
/cc
/triage accepted need to understand what is the right behavior here
We are in trouble for this problem too. I think It’s exactly as said. Lease has the special update endpoint which can create the new lease objects not only update them. If it receives a request with a new lease object which has non-nil UID field, returns a conflict error. This situation can be occured in the function when a lease object is deleted. The lease object is deleted when the node resource is deleted in the result of GC. When received a conflict error in updating a lease object and the parent node object doesn't exist, , and continue next loop. In the next loop, the function returns the new lease object without ownerReference and UID. As a result, it is created unexpectively by the special update endpoint.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/remove-lifecycle rotten


Question: // nodeConditionMessageFmt is the message for evictions due to resource pressure. nodeConditionMessageFmt = "The node had condition: %v. " // containerMessageFmt provides additional information for containers exceeding requests <del> containerMessageFmt = "Container %s was using %s, which exceeds its request of %s. " </del> <ins> containerMessageFmt = "Container %s was using %s, request is %s, has larger consumption of %v. " </ins> // containerEphemeralStorageMessageFmt provides additional information for containers which have exceeded their ES limit containerEphemeralStorageMessageFmt = "Container %s exceeded its local ephemeral storage limit %q. " // podEphemeralStorageMessageFmt provides additional information for pods which have exceeded their ES limit

when the pod is evicted due to low imagefs.available, the prompt is difficult to understand. it showed: But the container was evicted not by exceeding its request of 0 ephemeral-storage, but due to reaching the evictionHard of the node. The correct prompt message points out the reason for the pod eviction. don't print "which exceeds its request of 0.", because the container was evicted not by exceeding its request of 0. % oc label mcp worker custom-kubelet-eviction=small-pods evictionHard by creating a kubeletconfig % oc create -f custom-kubelet- custom-kubelet- several minutes, the mcp worker finish rolling out to the latest mc a pod % oc create -f hello-pod- hello-pod- a large file in container, make imagefs.available < 90Gi (the current available disk size is 111G ) command terminated with exit code 137 the node status the pod status % oc get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES hello-pod-2 0/1 ContainerStatusUnknown 1 7m58s 10.131.0.7 ip-10-0-213- <none<none% oc get pod hello-pod-2 -o yaml No response <details</details<detailsaws </details<details</details<details</details<details</details<details</details
/sig node
/assign
/assign
/triage accepted


Question: _, err := podClient.Patch(podName, types.StrategicMergePatchType, []byte(fmt.Sprintf(patchStatusFmt, readinessGate1, "True")), "status") framework.ExpectNoError(err) // Sleep for 10 seconds. <del> time.Sleep(maxReadyStatusUpdateTolerance) </del> <ins> time.Sleep(syncLoopFrequency) // Verify the pod is still not ready </ins> framework.ExpectEqual(podClient.PodIsReady(podName), false, "Expect pod's Ready condition to be false with only one condition in readinessGates equal to True") ginkgo.By(fmt.Sprintf("patching pod status with condition %q to true", readinessGate2))

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: Which test(s) are failing: Since when has it been failing: It has been flaking more frequently since 3/13. Testgrid link: Reason for failure: The issue seems to be a timeout cc /kind flake /sig node /priority important-soon /milestone v1.14
Latest PR to touch that test May be due to a readiness probe failing.
/remove-kind failing-test sending this your way for triage too - could you have a look please? I wonder if it's a node problem or if timeouts are originating elsewhere fwiw, it's failing about 10%-15% of the time in e2e tests:
It looks like updating the status after the patch to the readiness gate takes longer than expected.

Looking at this failure: From the test logs: [It] should support pod readiness gates [NodeFeature:PodReadinessGate] ï¿½[1mSTEPï¿½[0m: submitting the pod to kubernetes ï¿½[1mSTEPï¿½[0m: patching pod status with condition "" to true ï¿½[1mSTEPï¿½[0m: patching pod status with condition "" to true Mar 14 01:52:56.720: INFO: Expect the Ready condition of pod "pod-ready" to be true, but got false Mar 14 01:52:57.731: INFO: Expect the Ready condition of pod "pod-ready" to be true, but got false Mar 14 01:52:58.725: INFO: Expect the Ready condition of pod "pod-ready" to be true, but got false Mar 14 01:52:59.707: INFO: Expect the Ready condition of pod "pod-ready" to be true, but got false Mar 14 01:53:00.721: INFO: Expect the Ready condition of pod "pod-ready" to be true, but got false Mar 14 01:53:01.708: INFO: Expect the Ready condition of pod "pod-ready" to be true, but got false Mar 14 01:53:02.753: INFO: Expect the Ready condition of pod "pod-ready" to be true, but got false Mar 14 01:53:03.707: INFO: Expect the Ready condition of pod "pod-ready" to be true, but got false Mar 14 01:53:04.713: INFO: Expect the Ready condition of pod "pod-ready" to be true, but got false Mar 14 01:53:05.709: INFO: Expect the Ready condition of pod "pod-ready" to be true, but got false Mar 14 01:53:05.767: INFO: Expect the Ready condition of pod "pod-ready" to be true, but got false From the kubelet logs: The pod goes to Ready at 1:53:12, and we stopped checking at 1:53:05 This looks expected, based on the description of the parameter in the test. I don't think this should be release-blocking.

/assign who owns this feature to determine how we can make the test less flaky. Feel free to close if this is already tracked somewhere else /unassign
Thanks, good to know. if it could be worked on, it would really help stabilize CI - which would be super appreciated, especially for release-blocking jobs.
When this tests was (2 or 3 release back), I remember the 10 second waiting timeout is enough for kubelet to react and update the pod ready condition. IIRC, kubelet reconciles pod status every 15s. But if pod readiness gate condition is updated, kubelet should reconcile ASAP skipping the 15s period. There might be a bug here.
Thanks - do you have time to take a look? It would help to know (especially if we need to include something in known issues).
I just did a few manual tests. Kubelet reacted very quickly. Not able to reproduce the update latency more than 10s.
I ran the test in a loop more than 50 times now on my own e2e cluster. Not a single failure. Is it because there are many tests running in parallel causing high load on kubelet, hence longer update latency?
Interesting, could well be the case Is there a way to validate that? And, is there something here we should raise in terms of performance/scalability (e.g. "update latency varies with pod load")?
That is a well-known result of the rate limiting of the kubelet. The kubelet can only send 5 qps of updates to the API Server, so the kubelet is limited in its ability to update statuses when there are lots of updates to send, which occurs when there is high pod churn.
This issue seems to not be a problem anymore. /close
Closing this issue. <detailsIn response to : This issue seems to not be a problem anymore. /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Should this test be marked [Serial] then? (Or maybe flaky?) running into this again
Can you share the test grid link where it is flaky?
I've been testing extensively myself offline to re-evaluate tests to use in presubmit.

this sounds like inherently this test is not safe to run with lots of other tests in parallel, which would make it a candidate for
/reopen This is the flakiest test for PRs Triage link: Since this is a GA feature it should have conformance tests covering it, this specific test was prevented from promotion to conformance due to flakiness issues called out here One of the comments suggested rewriting to use an informer or poll for longer than 10s, and not use an explicit sleep. Tagging with [Serial] seems like it would work from a brute force perspective. Is the 10s number an important or documented SLO somewhere?
Reopened this issue. <detailsIn response to : /reopen This is the flakiest test for PRs Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
another data point of sorts: IIRC I disabled this test in kind because it was too flaky, i wasn't aware that it was also flaky on GCE.
opened to deflake
Still seen after disabling auto flake retry:
top-ten flake according to
Confirming this flake is still top-ten after we merged Thanks for all your work here !
no longer top-ten, but still flaky -
back in the top ten according to
/assign assigning sig-node leads for visibility/routing
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale /remove-priority critical-urgent /priority important-longterm Not in the top ten, but it still flakes occasionally
I'm pretty sure we still blacklist this test in most kind CI because it's too flaky. Aaron Crickenberger < :
/milestone v1.19
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or .

/lifecycle stale
Gonna push this with sig node /cc /remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale It's still failing across a number of jobs but I can't tell that it's as painful a flake today
/milestone clear
/assign /triage accepted
GitHub didn't allow me to assign the following users: fromanirh. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign /triage accepted Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
working to reproduce outside CI - way easier to debug
still working my way towards a reproducer.
has been unable to reproduce locally. We are still seeing occasional timeouts in CI: Since this was the only timeout on that test run, we may want to consider tweaking the timeout thresholds?
I'm happy I have time again to work on this issue. It doesn't reproduce locally on BM, and this seems indeed to point to a too busy worker in CI and/or too short timeout. So let's start with the simplest fix:
Today in , it is the top 1 flake for "pr:pull-kubernetes-e2e-gce-ubuntu-containerd".
ok, time for me to prioritize this work
Do we have a testgrid link? I'm looking at to try to understand better the flakes and getting too little from here (but looks relatively good from here?)

More recent triage link: Appears to be failing in a number of containerd and crio jobs. /unassign /priority important-soon /remove-priority important-longterm
/milestone v1.22
Did we vet that this is fixed after ?


Question: import ( "context" "fmt" <ins> "os" "strings" </ins> "k8s.io/api/core/v1" "k8s.io/kubernetes/pkg/cloudprovider"

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: When useInstanceMetadata is set to true for vmss nodes, their providerID is wrong. What you expected to happen: Their providerID should be in format azure:///subscriptions/<subscription-id/resourceGroups/<rg-name/providers/Microsoft.Compute/virtualMachineScaleSets/<scaleset-name/virtualMachines/<instance-id How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig azure
adding priority important-soon so that this issue can stay in 1.10. Please update if incorrect. /priority important-soon


Question: <ins> /* Copyright 2018 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package util import ( "fmt" "k8s.io/api/core/v1" "os" "path" "path/filepath" "sort" "strings" ) // getNestedMountpoints returns a list of mountpoint directories that should be created // for the volume indicated by name. // note: the returned list is relative to baseDir func getNestedMountpoints(name, baseDir string, pod v1.Pod) ([]string, error) { var retval []string checkContainer := func(container *v1.Container) error { var allMountPoints []string // all mount points in this container var myMountPoints []string  // mount points that match name for _, vol := range container.VolumeMounts { cleaned := filepath.Clean(vol.MountPath) allMountPoints = append(allMountPoints, cleaned) if vol.Name == name { myMountPoints = append(myMountPoints, cleaned) } } sort.Strings(allMountPoints) parentPrefix := ".." + string(os.PathSeparator) // Examine each place where this volume is mounted for _, myMountPoint := range myMountPoints { if strings.HasPrefix(myMountPoint, parentPrefix) { // Don't let a container trick us into creating directories outside of its rootfs return fmt.Errorf("Invalid container mount point %v", myMountPoint) } myMPSlash := myMountPoint + string(os.PathSeparator) // The previously found nested mountpoint (or "" if none found yet) prevNestedMP := "" // examine each mount point to see if it's nested beneath this volume // (but skip any that are double-nested beneath this volume) // For example, if this volume is mounted as /dir and other volumes are mounted //              as /dir/nested and /dir/nested/other, only create /dir/nested. for _, mp := range allMountPoints { if !strings.HasPrefix(mp, myMPSlash) { continue // skip -- not nested beneath myMountPoint } if prevNestedMP != "" && strings.HasPrefix(mp, prevNestedMP) { continue // skip -- double nested beneath myMountPoint } // since this mount point is nested, remember it so that we can check that following ones aren't nested beneath this one prevNestedMP = mp + string(os.PathSeparator) retval = append(retval, mp[len(myMPSlash):]) } } return nil } for _, container := range pod.Spec.InitContainers { if err := checkContainer(&container); err != nil { return nil, err } } for _, container := range pod.Spec.Containers { if err := checkContainer(&container); err != nil { return nil, err } } return retval, nil } // MakeNestedMountpoints creates mount points in baseDir for volumes mounted beneath name func MakeNestedMountpoints(name, baseDir string, pod v1.Pod) error { dirs, err := getNestedMountpoints(name, baseDir, pod) if err != nil { return err } for _, dir := range dirs { err := os.MkdirAll(path.Join(baseDir, dir), 0755) if err != nil { return fmt.Errorf("Unable to create nested volume mountpoints: %v", err) } } return nil } </ins>

This bug affects any pod that has a container with a nested volume (any type) beneath an API data volume, such as a secret, configMap, downwardAPI or projected volume. In the description and reproduction details below I use a secret volume, but any of the four will demonstrate this behavior. Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: When creating a pod with nested secrets volumes (e.g. one volume mounted at and one mounted at ), the container fails to start with an error like: What you expected to happen: The container should start with both and mounted. How to reproduce it (as minimally and precisely as possible): out master, any version since a file with 2 secrets and a pod that makes volumes for the 2 secrets, with one of them mounted within the other: the 2 secrets and the pod. Assuming you named the above file : Anything else we need to know?: This may work for other runtimes if they create mountpoints outside of the sandbox. But it's definitely broken with Docker as the runtime.
This should be fixed by /sig storage /sig node /assign


Question: }, } } func (s *disruptiveTestSuite) getTestSuiteInfo() TestSuiteInfo { return s.tsInfo }

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: Which test(s) are failing: [sig-storage] In-tree Volumes [Driver: local][LocalVolumeType: block] [Testpattern: Pre-provisioned PV (block volmode)] disruptive[Disruptive] Should test that pv used in a pod that is force deleted while the kubelet is down cleans up when the kubelet returns. Since when has it been failing: 10/8/19 Testgrid link: https://k8s- Reason for failure: Introduced by but root cause is probably Anything else we need to know:
/assign
/milestone 1.17 /priority important-soon this is consistently failing in release-master-informing job could you please take a look?
The provided milestone is not valid for this repository. Milestones in this repository: [, , , , , , , , ] Use to clear the milestone. <detailsIn response to : /milestone 1.17 /priority important-soon Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/milestone v1.17


Question: }, } } func (s *disruptiveTestSuite) getTestSuiteInfo() TestSuiteInfo { return s.tsInfo }

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: Which test(s) are failing: [sig-storage] In-tree Volumes [Driver: local][LocalVolumeType: block] [Testpattern: Pre-provisioned PV (block volmode)] disruptive[Disruptive] Should test that pv used in a pod that is force deleted while the kubelet is down cleans up when the kubelet returns. Since when has it been failing: 10/8/19 Testgrid link: https://k8s- Reason for failure: Introduced by but root cause is probably Anything else we need to know:
/assign
/milestone 1.17 /priority important-soon this is consistently failing in release-master-informing job could you please take a look?
The provided milestone is not valid for this repository. Milestones in this repository: [, , , , , , , , ] Use to clear the milestone. <detailsIn response to : /milestone 1.17 /priority important-soon Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/milestone v1.17


Question: var metadataAccessor = meta.NewAccessor() <ins> var attachFunc = attach.DefaultAttachFunc </ins> type RunObject struct { Object  runtime.Object Mapping *meta.RESTMapping

The command provides a flag but this doesn't do anything. most likely the flag shouldn't exist. I have no idea how kustomize could be useful here to start a single pod. This creates a pod, and shows that is ignored: No response <details</details<detailsAzure, K3OS</details<detailsOSX 11.6.1 (Big Sur) </detailsNo response No response No response
/sig cli
This comes from the inclusion of DeleteFlags: brings in FilenameFlags: has Kustomize, resulting in the flag: I suspect there are many flags from DeleteFlags that don't really apply to . : Not sure what this would do : seems to be ignored and : doesn't seem to be used : doesn't seem to be used : also probably doesn't apply There might be others as well...
It seems to delete object which created by command.
I agree, As far as I can see, , and have this deleteFlags exposure.
/assign
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage accepted