Question: scheme  ObjectScheme decoder runtime.Decoder lock    sync.RWMutex <del> objects map[schema.GroupVersionResource][]runtime.Object </del> <ins> objects map[schema.GroupVersionResource]map[types.NamespacedName]runtime.Object </ins> // The value type of watchers is a map of which the key is either a namespace or // all/non namespace aka "" and its value is list of fake watchers. // Manipulations on resources will broadcast the notification events into the

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: I used to fake a clientset to test my code. I found that it became very slow to init the clientset with a big number of objects when I was trying to simulate a large scale cluster, and the time seemed exponential increasing. For example: Initializing a clientset with: 5000 objects took 300ms objects took 2.58s objects took 11.82s objects took 54.64s. It's because the fake clientset used a slice to store each kind of objects, so it's quite slow to init the clientset with massive objects because it checked existence of an object by traversing all objects and comparing with them before adding one, which leads to O(n^2) time complexity. What you expected to happen: The fake clientset is very useful for development, it should have a reasonable time complexity to allow developers to use it for large scale scenario. How to reproduce it (as minimally and precisely as possible): Run this code: You would see the time spent on each operation, like: Anything else we need to know?: Environment: Kubernetes version (use ): 1.15.0-1.18.0 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I had PR that uses map to store objects, which can reduce the init time to a reasonable value (tens of ms for objects). /assign


Question: // TODO: consider optimizing to O(1) lookup return containsString(*runtimeClass, a.Configuration.Exemptions.RuntimeClasses) } <ins> // Filter and prioritize pods based on runtimeclass and uniqueness of the controller respectively for evaluation. // The input slice is modified in place and should not be reused. func (a *Admission) prioritizePods(pods []*corev1.Pod) []*corev1.Pod { // accumulate the list of prioritized pods in-place to avoid double-allocating prioritizedPods := pods[:0] // accumulate any additional replicated pods after the first one encountered for a given controller uid var duplicateReplicatedPods []*corev1.Pod evaluatedControllers := make(map[types.UID]bool) for _, pod := range pods { // short-circuit on exempt runtimeclass if a.exemptRuntimeClass(pod.Spec.RuntimeClassName) { continue } // short-circuit if pod from the same controller is evaluated podOwnerControllerRef := metav1.GetControllerOfNoCopy(pod) if podOwnerControllerRef == nil { prioritizedPods = append(prioritizedPods, pod) continue } if evaluatedControllers[podOwnerControllerRef.UID] { duplicateReplicatedPods = append(duplicateReplicatedPods, pod) continue } prioritizedPods = append(prioritizedPods, pod) evaluatedControllers[podOwnerControllerRef.UID] = true } return append(prioritizedPods, duplicateReplicatedPods...) } </ins> func containsString(needle string, haystack []string) bool { for _, s := range haystack { if s == needle {

When updating policy labels on a namespace, existing pods are evaluated to see if they are compatible with the new policy. If there are too many pods, or if evaluation takes too long, we short circuit and warn that not all pods were checked. For that reason, we want to prioritize checking a variety of pods before checking essentially identical pods, so that unique warnings can be surfaced before reaching the pod count cutoff or evaluation timeout. Pods created by the same controller are most likely to be identical (e.g. replicas from a replicaset or job, copies of pods from a statefulset or daemonset, etc) cc
This is an interesting fix I'd like to take this up Can I take this up? I am fairly new to Kubernetes source code so might need some help
I am guessing this is where the logic you mentioned should be implemented right?
thanks for the interest; this is targeted at the beta version... right now, effort is going toward the alpha-blocking work for 1.22
Should I wait for the code freeze before working on this? Sorry for lame questions
If you want to start looking at it, that's fine, but it's possible the code in the ValidateNamespace function will change before we're ready to start merging 1.23 work, and there isn't really reviewer bandwidth to take a look at work-in-progress stuff for next release yet.
Sure, I'll just for code freeze before I start with implementation By the time I'll just assign this to myself /assign
Does it make sense to take it up now? I think we already have a couple of RC builds after the code freeze
sure, it's fine to start work in a branch in your fork and have a WIP PR open, even before master reopens for 1.23 work
Awesome, I'll start working on it Thank you
Hello! Are you still working on this? If you don't have time, I can take this issue. If not, maybe I can give some help
Since this was not an urgent issue I was working on some other issues last week I have started on this, Will be raising a PR maybe today or tomorrow I Will ask you if I get stuck Thank you


Question: } if allowed := annotations[seccomp.AllowedProfilesAnnotationKey]; allowed != "" { for _, p := range strings.Split(allowed, ",") { <ins> if p == seccomp.AllowAny { continue } </ins> allErrs = append(allErrs, apivalidation.ValidateSeccompProfile(p, fldPath.Key(seccomp.AllowedProfilesAnnotationKey))...) } }

It should be possible to create a PodSecurityPolicy with zero restrictions. This was the intent with the (a.k.a. "AllowAny") allowed seccomp profile, but it is rejected during the validation phase. /kind bug
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/auth: Issue will be escalated to these SIGs if needed. priority/important-soon: Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. kind/bug: Fixes a bug discovered during the current release. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details


Question: } hash := cur.Labels[apps.DefaultDaemonSetUniqueLabelKey] <del> if ds.DeletionTimestamp != nil || !dsc.expectations.SatisfiedExpectations(dsKey) { </del> <ins> if !dsc.expectations.SatisfiedExpectations(dsKey) { </ins> // Only update status. return dsc.updateDaemonSetStatus(ds, hash) }

/kind bug /priority critical-urgent What happened: In I'm removing all the reapers, unfortunately when invoking foreground deletion on a DaemonSet the delete operation hangs and controller indefinitely tries to manage a removed DS. What you expected to happen: A foreground deletion properly removes DaemonSet and all its dependents. How to reproduce it (as minimally and precisely as possible): Invoke a delete operation against kube API (without using command). Anything else we need to know?: This behavior makes it so that none of the consumers of the API is not able to remove the DS controller due to present in the DS reaper. A working fix that passed the tests with the aforementioned PR is in but I'm not 100% it's correct. This is an issue that needs fixing before 1.11 goes out. /assign /sig apps /cc
fyi
I filed to fix it. As mentioned in the PR, this is because the DaemonSet controller races with garbage collector when it tries to update DaemonSet status of the DaemonSet being deleted. Here's what happens when hangs: /something performs a foreground deletion on a DaemonSet finalizer and DeletionTimestamp are both set history objects (ControllerRevisions) and pods are being deleted by garbage collector; meanwhile, DaemonSet controller tries to update DaemonSet status. Updating DaemonSet status requires constructing DaemonSet history objects, to figure out current revision and which pods do/don't belong to current revision updating DaemonSet status, DaemonSet controller tries to create a DaemonSet history object that matches current DaemonSet spec collector then tries to delete that DaemonSet history object. And repeat. Because we can't make DaemonSet pods be deleted before DaemonSet history objects (DaemonSet history objects don't own DaemonSet pods!), we cannot reliably calculate DaemonSet status without history objects anyways. Therefore, we don't update DaemonSet status for DaemonSet being deleted. Note that the reason why the kubectl delete hack works is because it forces DaemonSet pods to be removed before history objects.
[MILESTONENOTIFIER] Milestone Issue: Up-to-date for process Note: This issue is marked as , and must be updated every 1 day during code freeze. Example update: <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed.

: Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details


Question: } return scheduled, nil } <ins> // DataItem is the data point. type DataItem struct { // Data is a map from bucket to real data point (e.g. "Perc90" -> 23.5). Notice // that all data items with the same label combination should have the same buckets. Data map[string]float64 `json:"data"` // Unit is the data unit. Notice that all data items with the same label combination // should have the same unit. Unit string `json:"unit"` // Labels is the labels of the data item. Labels map[string]string `json:"labels,omitempty"` } // DataItems is the data point set. It is the struct that perf dashboard expects. type DataItems struct { Version   string     `json:"version"` DataItems []DataItem `json:"dataItems"` } func dataItems2JSONFile(dataItems DataItems, namePrefix string) error { b, err := json.Marshal(dataItems) if err != nil { return err } destFile := fmt.Sprintf("%v_%v.json", namePrefix, time.Now().Format(dateFormat)) if *dataItemsDir != "" { destFile = path.Join(*dataItemsDir, destFile) } return ioutil.WriteFile(destFile, b, 0644) } // prometheusCollector collects metrics from legacyregistry.DefaultGatherer.Gather() endpoint. // Currently only Histrogram metrics are supported. type prometheusCollector struct { metric string cache  *dto.MetricFamily } func newPrometheusCollector(metric string) *prometheusCollector { return &prometheusCollector{ metric: metric, } } func (pc *prometheusCollector) collect() *DataItem { var metricFamily *dto.MetricFamily m, err := legacyregistry.DefaultGatherer.Gather() if err != nil { klog.Error(err) return nil } for _, mFamily := range m { if mFamily.Name != nil && *mFamily.Name == pc.metric { metricFamily = mFamily break } } if metricFamily == nil { klog.Infof("Metric %q not found", pc.metric) return nil } if metricFamily.GetMetric() == nil { klog.Infof("Metric %q is empty", pc.metric) return nil } if len(metricFamily.GetMetric()) == 0 { klog.Infof("Metric %q is empty", pc.metric) return nil } // Histograms are stored under the first index (based on observation). // Given there's only one histogram registered per each metric name, accessaing // the first index is sufficient. dataItem := pc.promHist2Summary(metricFamily.GetMetric()[0].GetHistogram()) if dataItem.Data == nil { return nil } // clear the metrics so that next test always starts with empty prometheus // metrics (since the metrics are shared among all tests run inside the same binary) clearPromHistogram(metricFamily.GetMetric()[0].GetHistogram()) return dataItem } // Bucket of a histogram type bucket struct { upperBound float64 count      float64 } func bucketQuantile(q float64, buckets []bucket) float64 { if q < 0 { return math.Inf(-1) } if q > 1 { return math.Inf(+1) } if len(buckets) < 2 { return math.NaN() } rank := q * buckets[len(buckets)-1].count b := sort.Search(len(buckets)-1, func(i int) bool { return buckets[i].count >= rank }) if b == 0 { return buckets[0].upperBound * (rank / buckets[0].count) } // linear approximation of b-th bucket brank := rank - buckets[b-1].count bSize := buckets[b].upperBound - buckets[b-1].upperBound bCount := buckets[b].count - buckets[b-1].count return buckets[b-1].upperBound + bSize*(brank/bCount) } func (pc *prometheusCollector) promHist2Summary(hist *dto.Histogram) *DataItem { buckets := []bucket{} if hist.SampleCount == nil || *hist.SampleCount == 0 { return &DataItem{} } if hist.SampleSum == nil || *hist.SampleSum == 0 { return &DataItem{} } for _, bckt := range hist.Bucket { if bckt == nil { return &DataItem{} } if bckt.UpperBound == nil || *bckt.UpperBound < 0 { return &DataItem{} } buckets = append(buckets, bucket{ count:      float64(*bckt.CumulativeCount), upperBound: *bckt.UpperBound, }) } // bucketQuantile expects the upper bound of the last bucket to be +inf buckets[len(buckets)-1].upperBound = math.Inf(+1) q50 := bucketQuantile(0.50, buckets) q90 := bucketQuantile(0.90, buckets) q99 := bucketQuantile(0.95, buckets) msFactor := float64(time.Second) / float64(time.Millisecond) return &DataItem{ Labels: map[string]string{ "Metric": pc.metric, }, Data: map[string]float64{ "Perc50":  q50 * msFactor, "Perc90":  q90 * msFactor, "Perc99":  q99 * msFactor, "Average": (*hist.SampleSum / float64(*hist.SampleCount)) * msFactor, }, Unit: "ms", } } func clearPromHistogram(hist *dto.Histogram) { if hist.SampleCount != nil { *hist.SampleCount = 0 } if hist.SampleSum != nil { *hist.SampleSum = 0 } for _, b := range hist.Bucket { if b.CumulativeCount != nil { *b.CumulativeCount = 0 } if b.UpperBound != nil { *b.UpperBound = 0 } } } type throughputCollector struct { podInformer           coreinformers.PodInformer schedulingThroughputs []float64 } func newThroughputCollector(podInformer coreinformers.PodInformer) *throughputCollector { return &throughputCollector{ podInformer: podInformer, } } func (tc *throughputCollector) run(stopCh chan struct{}) { podsScheduled, err := getScheduledPods(tc.podInformer) if err != nil { klog.Fatalf("%v", err) } lastScheduledCount := len(podsScheduled) for { select { case <-stopCh: return case <-time.After(throughputSampleFrequency): podsScheduled, err := getScheduledPods(tc.podInformer) if err != nil { klog.Fatalf("%v", err) } scheduled := len(podsScheduled) samplingRatioSeconds := float64(throughputSampleFrequency) / float64(time.Second) throughput := float64(scheduled-lastScheduledCount) / samplingRatioSeconds tc.schedulingThroughputs = append(tc.schedulingThroughputs, throughput) lastScheduledCount = scheduled klog.Infof("%d pods scheduled", lastScheduledCount) } } } func (tc *throughputCollector) collect() *DataItem { throughputSummary := &DataItem{} if length := len(tc.schedulingThroughputs); length > 0 { sort.Float64s(tc.schedulingThroughputs) sum := 0.0 for i := range tc.schedulingThroughputs { sum += tc.schedulingThroughputs[i] } throughputSummary.Labels = map[string]string{ "Metric": "SchedulingThroughput", } throughputSummary.Data = map[string]float64{ "Average": sum / float64(length), "Perc50":  tc.schedulingThroughputs[int(math.Ceil(float64(length*50)/100))-1], "Perc90":  tc.schedulingThroughputs[int(math.Ceil(float64(length*90)/100))-1], "Perc99":  tc.schedulingThroughputs[int(math.Ceil(float64(length*99)/100))-1], } throughputSummary.Unit = "pods/s" } return throughputSummary } </ins>

<!-- Please only use this template for submitting enhancement requests --What would you like to be : We should allow collecting both default scheduler metrics (Prometheus) and custom metrics easily. At a high level, these collectors should produce a json formatted result which will then be consumed by perf-dash to present in the dashboard. Prometheus metrics We should provide a metrics collector that collects and summarizes default scheduler metrics defined in . To begin with, we can manually pick a list of interesting metrics. Longer term, we could allow configuring the metric list in test definition. already had a PoC . As a reference, is how clusterloader collects scheduler latency metrics. Custom metrics The most important metric is scheduling throughput. is how clusterloader calculates the throughput. We can start with just calculating the average throughput without the percentiles. We may want other custom metrics that are either common to all tests or specific to certain tests. We should allow these custom collectors to be easily and ideally configurable. Why is this needed: Give more insights into scheduler performance. Part of /sig scheduling /assign
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: "//pkg/util/node:go_default_library", "//pkg/util/parsers:go_default_library", "//staging/src/k8s.io/api/core/v1:go_default_library", <del> "//staging/src/k8s.io/apimachinery/pkg/api/errors:go_default_library", </del> "//staging/src/k8s.io/apimachinery/pkg/api/resource:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/apis/meta/v1:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/labels:go_default_library",

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: This switches NodeLister to scheduler cache for . However, the new implementation in the cache never returns IsNotFound error any more. Rather, it returns a normal error: However, we rely on the error type for logging, for example: What you expected to happen: The cache should return IsNotFound error if applicable. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: /sig scheduling cc
/help
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/good-first-issue
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hi. I'd like to work on this for #Hacktoberfest, and for my very first open-source contribution.
Hello, I would love to contribute too in case. I'm already looking at the code in fact :D
Thanks all for the interest, will assign to since he expressed interest first. /assign
I was browsing around the rest of the implementation of the cache and noticed that none of the other accessors return . Also, I couldn't determine if ever returned anything other than an . I'm new here, so I may be doing it wrong. Are you sure the behavior changed, and are you sure returning is expected? Should the other accessors be changed, too?
Let's look at previous release. The object is initialized in [] where it creates a .

Now let's take a look at implementation: There you can see that it calls NodeLister Get() method, which can return a IsNotFound error.
Hello, is this issue being solved ? If not, i am interested in solving this issue.
Thanks but I am working on this currently.
, if you haven't started your work yet, let me know if I can raise a PR for this issue. I have the change ready.
I started, but haven’t been able to finish creating the PR. Go ahead and submit yours. Thanks. Sent from my iPhone


Question: <ins> This directory forms the base of the main SaltStack configuration. The place to start with any SaltStack configuration is [top.sls](top.sls). However, unless you are particularly keen on reading Jinja templates, the following tables break down what configurations run on what providers. (NB: The [_states](_states/) directory is a special directory included by Salt for `ensure` blocks, and is only used for the [docker](docker/) config.) Key: M = Config applies to master, n = config applies to nodes Config                                              | GCE   | Vagrant | AWS | Azure ----------------------------------------------------|-------|---------|-----|------ [cadvisor](cadvisor/)                               | M n   | M n     | M n | M n [debian-auto-upgrades](debian-auto-upgrades/)       | M n   | M n     | M n | M n [docker](docker/)                                   | M n   | M n     | M n |   n [etcd](etcd/)                                       | M     | M       | M   | M [fluentd-es](fluentd-es/) (pillar conditional)      | M n   | M n     | M n | M n [fluentd-gcp](fluentd-gcp/) (pillar conditional)    | M n   | M n     | M n | M n [generate-cert](generate-cert/)                     | M     | M       | M   | M [kube-addons](kube-addons/)                         | M     | M       | M   | M [kube-apiserver](kube-apiserver/)                   | M     | M       | M   | M [kube-controller-manager](kube-controller-manager/) | M     | M       | M   | M [kube-proxy](kube-proxy/)                           |   n   |   n     |   n |   n [kube-scheduler](kube-scheduler/)                   | M     | M       | M   | M [kubelet](kubelet/)                                 | M n   | M n     | M n |   n [logrotate](logrotate/)                             | M n   |   n     | M n | M n [monit](monit/)                                     | M n   | M n     | M n | M n [nginx](nginx/)                                     | M     | M       | M   | M [openvpn-client](openvpn-client/)                   |       |         |     |   n [openvpn](openvpn/)                                 |       |         |     | M [sdn](sdn/) (Vagrant only)                          |   n   | M n     |   n | [static-routes](static-routes/) (vsphere only)      |       |         |     | [base](base.sls)                                    | M n   | M n     | M n | M n [kube-client-tools](kube-client-tools.sls)          | M     | M       | M   | M </ins>

We need to document all the configuration derived by Salt. Ideally this would be a markdown doc that we kept as live documentation within the project. The main reasons to bother: It's hard right now for anyone approaching the directory to get a big picture view of what's is going on, especially across providers. If we ever want to do something disciplined w.r.t. upgrade between (non-dev) versions, we're probably going to need a place to look for the state of the world. Decreasing the velocity with which people can add to Salt configuration will encourage people to delete more Salt configuration. cc


Question: reportedLevels       []uint64 reportedCPURequests  []resource.Quantity reportedPodReadiness []v1.ConditionStatus <ins> reportedPodPhase     []v1.PodPhase </ins> scaleUpdated         bool statusUpdated        bool eventCreated         bool

/kind bug What happened: HPA fails to launch additional pods due to pods in evicted state, even with utilization significantly over target. What you expected to happen: When HPA pods reach the target utilization, new pods should be allocated and provisioned. How to reproduce it (as minimally and precisely as possible): Launch a HPA, let the pods fill the disk on the host which will trigger evictions due to "Pod The node was low on resource: [DiskPressure]", HPA will exceed target and not allocate additional pods. Anything else we need to know?: Seems related to and though those are not specific to HPA. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: GKE OS (e.g. from /etc/os-release): cos-stable-57-9202-64-0 kernel commit id Kernel (e.g. ):Linux gke-prod1-use1b-v2--35s4 4.4.70+ SMP Mon Sep 18 12:43:08 PDT 2017 x86_64 Intel(R) Xeon(R) CPU @ 2.30GHz GenuineIntel GNU/Linux
/sig scheduling
Are there any workarounds while we wait for a fix? We have ran into this on a 1.8.4 GKE cluster.
/sig autoscaling /remove-sig scheduling
I am thinking of fixing this by excluding pods with Phase=Failed from the unreadyPods that are taken into account. It seems the right thing to do as they will not become running again. Do you see any issues?
that seems decent -- anything that will never be running again should not be factored into our calculations.
/status approved-for-milestone Adding to milestone -- if the HPA isn't functioning in the case of evictions, that seems like a decently severe HPA bug.
oops, missing priority was my fault, moving back into milestone: /priority critical-urgent /remove-priority important-soon
[MILESTONENOTIFIER] Milestone Issue Needs Attention Action required: During code freeze, issues in the milestone should be in progress. If this issue is not being actively worked on, please remove it from the milestone. If it is being worked on, please add the label so it can be tracked with other in-flight issues.

Note: This issue is marked as , and must be updated every 1 day during code freeze. Example update: <details<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details


Question: } function upgrade-master() { <ins> local num_masters num_masters=$(get-master-replicas-count) if [[ "${num_masters}" -gt 1 ]]; then echo "Upgrade of master not supported if more than one master replica present. The current number of master replicas: ${num_masters}" exit 1 fi </ins> echo "== Upgrading master to '${SERVER_BINARY_TAR_URL}'. Do not interrupt, deleting master instance. ==" # Tries to figure out KUBE_USER/KUBE_PASSWORD by first looking under

Script does not support upgrade of clusters with HA master setup.
CC
PR does not allow to run upgrade scripts on cluster with HA masters. The upgrade scenario for HA clusters from 1.5 to 1.6 may be the following: all master replicas but one, master upgrade, master replicas again.
I haven't reviewed the PR yet, but the scenario you are proposing defeats the purpose of HA master! I think it's ok to say we do not support HA master upgrades yet, but we should document how this can be done manually and fix this properly later by adding support in
It's too late to add this in for 1.6.0, but we should consider backporting a fix so that users can update an HA master to 1.6.x in the near future. We should also put in a big warning in the release notes warning users not to try and upgrade if they deployed an HA master using kube-
/sig cluster-lifecycle
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale /lifecycle frozen
/remove-lifecycle frozen /sig cloud-provider /area provider/gcp changing with ownership of ""
/remove-sig cluster-lifecycle
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or .

/close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: // GetSELinuxSupport tests if pathname is on a mount that supports SELinux. // Not implemented for testing func (hu *FakeHostUtil) GetSELinuxSupport(pathname string) (bool, error) { <del> return false, errors.New("GetSELinuxSupport not implemented") </del> <ins> return false, nil </ins> } // GetMode returns permissions of pathname.

On a busy system, kubelet failed to tell the container runtime that a pod's CSI volume supports SELinux and should be relabeled with pod's SELinux label. The pod then can't access the volume. Kubelet correctly detects that the CSI volume supports SELinux and tells the container runtime to relabel it. Unknown. Get a busy system and try hard? CSI volume plugin tried to detect that the volume supports SELinux during SetupAt. This check failed: Current CSI volume plugin code silently consumes the error and assumes that the volume does not support SELinux at all: Therefore, the volume is marked as mounted and given to the container runtime without the right SELinux options and the runtime does not relabel volume. <details22.0 </details<detailsNone </details<details</details<details</details<detailsCRI-O, but I think it's not really relevant. </details<details</details
/sig storage /triage accepted
We've updated kubelet in our clusters to v1.20.13 (which contains the cherry picked fix), but still observe the issue


Question: <ins> /* Copyright 2022 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package testsuites import ( "context" "github.com/onsi/ginkgo/v2" v1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/fields" errors "k8s.io/apimachinery/pkg/util/errors" clientset "k8s.io/client-go/kubernetes" "k8s.io/kubernetes/pkg/kubelet/events" "k8s.io/kubernetes/test/e2e/framework" e2eevents "k8s.io/kubernetes/test/e2e/framework/events" e2epod "k8s.io/kubernetes/test/e2e/framework/pod" e2epv "k8s.io/kubernetes/test/e2e/framework/pv" e2eskipper "k8s.io/kubernetes/test/e2e/framework/skipper" storageframework "k8s.io/kubernetes/test/e2e/storage/framework" storageutils "k8s.io/kubernetes/test/e2e/storage/utils" admissionapi "k8s.io/pod-security-admission/api" ) type readWriteOncePodTestSuite struct { tsInfo storageframework.TestSuiteInfo } var _ storageframework.TestSuite = &readWriteOncePodTestSuite{} type readWriteOncePodTest struct { config        *storageframework.PerTestConfig driverCleanup func() cs     clientset.Interface volume *storageframework.VolumeResource pods   []*v1.Pod migrationCheck *migrationOpCheck } func InitCustomReadWriteOncePodTestSuite(patterns []storageframework.TestPattern) storageframework.TestSuite { return &readWriteOncePodTestSuite{ tsInfo: storageframework.TestSuiteInfo{ Name:         "read-write-once-pod", TestPatterns: patterns, FeatureTag:   "[Feature:ReadWriteOncePod]", }, } } // InitReadWriteOncePodTestSuite returns a test suite for the ReadWriteOncePod PersistentVolume access mode feature. func InitReadWriteOncePodTestSuite() storageframework.TestSuite { // Only covers one test pattern since ReadWriteOncePod enforcement is // handled through Kubernetes and does not differ across volume types. patterns := []storageframework.TestPattern{storageframework.DefaultFsDynamicPV} return InitCustomReadWriteOncePodTestSuite(patterns) } func (t *readWriteOncePodTestSuite) GetTestSuiteInfo() storageframework.TestSuiteInfo { return t.tsInfo } func (t *readWriteOncePodTestSuite) SkipUnsupportedTests(driver storageframework.TestDriver, pattern storageframework.TestPattern) { driverInfo := driver.GetDriverInfo() if !driverInfo.Capabilities[storageframework.CapReadWriteOncePod] { e2eskipper.Skipf("Driver %q doesn't support ReadWriteOncePod - skipping", driverInfo.Name) } } func (t *readWriteOncePodTestSuite) DefineTests(driver storageframework.TestDriver, pattern storageframework.TestPattern) { var ( driverInfo = driver.GetDriverInfo() l          readWriteOncePodTest ) // Beware that it also registers an AfterEach which renders f unusable. Any code using // f must run inside an It or Context callback. f := framework.NewFrameworkWithCustomTimeouts("read-write-once-pod", storageframework.GetDriverTimeouts(driver)) f.NamespacePodSecurityEnforceLevel = admissionapi.LevelPrivileged init := func() { l = readWriteOncePodTest{} l.config, l.driverCleanup = driver.PrepareTest(f) l.cs = f.ClientSet l.pods = []*v1.Pod{} l.migrationCheck = newMigrationOpCheck(f.ClientSet, f.ClientConfig(), driverInfo.InTreePluginName) } cleanup := func() { var errs []error for _, pod := range l.pods { framework.Logf("Deleting pod %v", pod.Name) err := e2epod.DeletePodWithWait(l.cs, pod) errs = append(errs, err) } framework.Logf("Deleting volume %s", l.volume.Pvc.GetName()) err := l.volume.CleanupResource() errs = append(errs, err) errs = append(errs, storageutils.TryFunc(l.driverCleanup)) framework.ExpectNoError(errors.NewAggregate(errs), "while cleaning up resource") l.migrationCheck.validateMigrationVolumeOpCounts() } ginkgo.BeforeEach(func() { init() ginkgo.DeferCleanup(cleanup) }) ginkgo.It("should block a second pod from using an in-use ReadWriteOncePod volume", func() { // Create the ReadWriteOncePod PVC. accessModes := []v1.PersistentVolumeAccessMode{v1.ReadWriteOncePod} l.volume = storageframework.CreateVolumeResourceWithAccessModes(driver, l.config, pattern, t.GetTestSuiteInfo().SupportedSizeRange, accessModes) podConfig := e2epod.Config{ NS:           f.Namespace.Name, PVCs:         []*v1.PersistentVolumeClaim{l.volume.Pvc}, SeLinuxLabel: e2epv.SELinuxLabel, } // Create the first pod, which will take ownership of the ReadWriteOncePod PVC. pod1, err := e2epod.MakeSecPod(&podConfig) framework.ExpectNoError(err, "failed to create spec for pod1") _, err = l.cs.CoreV1().Pods(pod1.Namespace).Create(context.TODO(), pod1, metav1.CreateOptions{}) framework.ExpectNoError(err, "failed to create pod1") err = e2epod.WaitTimeoutForPodRunningInNamespace(l.cs, pod1.Name, pod1.Namespace, f.Timeouts.PodStart) framework.ExpectNoError(err, "failed to wait for pod1 running status") l.pods = append(l.pods, pod1) // Create the second pod, which will fail scheduling because the ReadWriteOncePod PVC is already in use. pod2, err := e2epod.MakeSecPod(&podConfig) framework.ExpectNoError(err, "failed to create spec for pod2") _, err = l.cs.CoreV1().Pods(pod2.Namespace).Create(context.TODO(), pod2, metav1.CreateOptions{}) framework.ExpectNoError(err, "failed to create pod2") err = e2epod.WaitForPodNameUnschedulableInNamespace(l.cs, pod2.Name, pod2.Namespace) framework.ExpectNoError(err, "failed to wait for pod2 unschedulable status") l.pods = append(l.pods, pod2) // Delete the first pod and observe the second pod can now start. err = e2epod.DeletePodWithWait(l.cs, pod1) framework.ExpectNoError(err, "failed to delete pod1") err = e2epod.WaitTimeoutForPodRunningInNamespace(l.cs, pod2.Name, pod2.Namespace, f.Timeouts.PodStart) framework.ExpectNoError(err, "failed to wait for pod2 running status") }) ginkgo.It("should block a second pod from using an in-use ReadWriteOncePod volume on the same node", func() { // Create the ReadWriteOncePod PVC. accessModes := []v1.PersistentVolumeAccessMode{v1.ReadWriteOncePod} l.volume = storageframework.CreateVolumeResourceWithAccessModes(driver, l.config, pattern, t.GetTestSuiteInfo().SupportedSizeRange, accessModes) podConfig := e2epod.Config{ NS:           f.Namespace.Name, PVCs:         []*v1.PersistentVolumeClaim{l.volume.Pvc}, SeLinuxLabel: e2epv.SELinuxLabel, } // Create the first pod, which will take ownership of the ReadWriteOncePod PVC. pod1, err := e2epod.MakeSecPod(&podConfig) framework.ExpectNoError(err, "failed to create spec for pod1") _, err = l.cs.CoreV1().Pods(pod1.Namespace).Create(context.TODO(), pod1, metav1.CreateOptions{}) framework.ExpectNoError(err, "failed to create pod1") err = e2epod.WaitTimeoutForPodRunningInNamespace(l.cs, pod1.Name, pod1.Namespace, f.Timeouts.PodStart) framework.ExpectNoError(err, "failed to wait for pod1 running status") l.pods = append(l.pods, pod1) // Get the node name for the first pod now that it's running. pod1, err = l.cs.CoreV1().Pods(pod1.Namespace).Get(context.TODO(), pod1.Name, metav1.GetOptions{}) framework.ExpectNoError(err, "failed to get pod1") nodeName := pod1.Spec.NodeName // Create the second pod on the same node as the first pod. pod2, err := e2epod.MakeSecPod(&podConfig) framework.ExpectNoError(err, "failed to create spec for pod2") // Set the node name to that of the first pod. // Node name is set to bypass scheduling, which would enforce the access mode otherwise. pod2.Spec.NodeName = nodeName _, err = l.cs.CoreV1().Pods(pod2.Namespace).Create(context.TODO(), pod2, metav1.CreateOptions{}) framework.ExpectNoError(err, "failed to create pod2") l.pods = append(l.pods, pod2) // Wait for the FailedMount event to be generated for the second pod. eventSelector := fields.Set{ "involvedObject.kind":      "Pod", "involvedObject.name":      pod2.Name, "involvedObject.namespace": pod2.Namespace, "reason":                   events.FailedMountVolume, }.AsSelector().String() msg := "volume uses the ReadWriteOncePod access mode and is already in use by another pod" err = e2eevents.WaitTimeoutForEvent(l.cs, pod2.Namespace, eventSelector, msg, f.Timeouts.PodStart) framework.ExpectNoError(err, "failed to wait for FailedMount event for pod2") // Wait for the second pod to fail because it is stuck at container creating. reason := "ContainerCreating" err = e2epod.WaitForPodContainerToFail(l.cs, pod2.Namespace, pod2.Name, 0, reason, f.Timeouts.PodStart) framework.ExpectNoError(err, "failed to wait for pod2 container to fail") // Delete the first pod and observe the second pod can now start. err = e2epod.DeletePodWithWait(l.cs, pod1) framework.ExpectNoError(err, "failed to delete pod1") err = e2epod.WaitTimeoutForPodRunningInNamespace(l.cs, pod2.Name, pod2.Namespace, f.Timeouts.PodStart) framework.ExpectNoError(err, "failed to wait for pod2 running status") }) } </ins>

For the ReadWriteOncePod feature to graduate to beta, we need to have E2E test coverage for its current alpha behavior. See this section of KEP-2485 for more context on what cases we need to cover for alpha: This is a requirement for beta graduation, see:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage


Question: CROSS_BUILD_COPY qemu-QEMUARCH-static /usr/bin/ # Base Packages <del> RUN yum install -y wget strace psmisc procps-ng ceph ceph-fuse && yum clean all </del> <ins> RUN yum install -y wget strace psmisc procps-ng ceph ceph-fuse hostname && yum clean all </ins> # Get ports exposed EXPOSE 6789

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --When I run some e2e storage testcases locally, below testcases as an example, I consistently saw the below error message, ceph should be deployed successfully and testcase should pass. Run the testcases mentioned above. e.g. Kubernetes version (use ): ALL Cloud provider or hardware configuration: ARM64 OS (e.g: ): ubuntu18 Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig testing /sig storage
/assign
is this happening in CI? please provide links to occurences?
Hey, Ben, I update the issue description a little bit, this won't be an issue on X86 which is currently the default arch CI is running, but tons of testcases will fail on ARM64 due to this issue. Eventually, we hope the k8s will have a ARM CI along with the more adoption of arch beside the X86, and thus, those e2e testcases should workable on ARM64 firstly. I will explain why it failed on ARM64 in the PR desc, pls help to review, thanks!
/triage accepted


Question: APIVersion: "v1", FieldPath:  "spec.containers[2]", }, <del> Reason:  "Started", Message: "some verbose message: 1", Source:  v1.EventSource{Component: "eventTest"}, Count:   2, Type:    v1.EventTypeNormal, </del> <ins> Reason:              "Started", Message:             "some verbose message: 1", Source:              v1.EventSource{Component: "eventTest"}, ReportingController: "eventTest", Count:               2, Type:                v1.EventTypeNormal, </ins> }, expectLog:    `Event(v1.ObjectReference{Kind:"Pod", Namespace:"baz", Name:"foo", UID:"bar", APIVersion:"v1", ResourceVersion:"", FieldPath:"spec.containers[2]"}): type: 'Normal' reason: 'Started' some verbose message: 1`, expectUpdate: true,

From the Deprecated API Migration Guide: , since v1.19, the has been using the new reportingController and reportingInstance to replace the decrecated source.component and But when I checked the Kube events in a cluster with version 1.26.0, those new fields are empty and the supposed to be deprecated fields have not been renamed. Seeing the values in the new fields like reportingController and reportingInstance an Kubernetes cluster with version 1.26.0 some Kube events the Kube events details for the field of reportingController, reportingInstance, source.component, No response <detailsServer Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2022-12-11T00:15:47Z", GoVersion:"go1.19.4", Compiler:"gc", Platform:"linux/amd64"} </details<detailsAzure </details<details</details<details</details<details</details<details</details
/sig instrumentation
I have also encountered this issue in my local 1.26.0 Kubernetes cluster, which does not align with the deprecation guide described in I believe I can investigate this further
/assign
Thank you for making the PR to fix this! Also a comment in the PR for the doc update.
/triage accepted


Question: bufferSize int decoder decoder <del> rawData []byte </del> } type JSONSyntaxError struct { <del> Line int Err  error </del> <ins> Offset int64 Err    error </ins> } func (e JSONSyntaxError) Error() string { <del> return fmt.Sprintf("json: line %d: %s", e.Line, e.Err.Error()) </del> <ins> return fmt.Sprintf("json: offset %d: %s", e.Offset, e.Err.Error()) </ins> } type YAMLSyntaxError struct {

This bug was privately disclosed and told to open here. What happened: NewYAMLOrJSONDecoder from has an index out of bounds issue which can cause callers to panic. Download attached reproducer file Run golang test: When run with v0.19.3: Additionally, this can be seen in kubectl directly (below is kubectl 1.20.0): Environment: Kubernetes version (use ): Verified with 1.19, 1.20
/sig api-machinery
This piece attempts to find line+column position of a json syntax error. To do that it assumes that full input is available which is not true: is : is only the first data chunk: The subject happens e.g. when syntax error offset is past buffer size and Test to reproduce the panic: I am not sure it is easily possible to find line+column of json syntax error as full input may not be available. Maybe it should only attempt to find line+column if offset is within the first chunk (i.e. below buffer size).
Line number calculation in and updated in
do you want to submit a patch to with a defensive check to avoid the panic ( compare and )
I have raised a discussion on having line number in the . The consensus there is that it should be possible to count lines with a full input at hand and it does not worth to have it in the core library due to performance impact and maintenance costs. The problem of here is that it attempts to find out line number without having full input at hand. I think the defensive patch is not the right approach because line calculation is incorrect anyway even when there is no panic: begins at a value with a syntax error, i.e. in a stream of two values and syntax error in the second value it would contain only the second value and thus new lines in the first value in the stream would not be counted. I think the following options are there: to stream taken from . This is straightforward but changes interface. I can not find any code that uses field, it is only used in the error message that would change: line number only if offset is within the length of the first data chunk (). I am not sure what would be the unknown line number then, perhaps. line number properly during stream decoding, see This has some performance penalty as line number is calculated in any case even if there is no syntax error, i.e.

in the majority of the cases.
cc /triage accepted
This is probably acceptable.
I have opened a PR
Thank you for support.


Question: klog.Infof("Shutting down kubernetes service endpoint reconciler") c.EndpointReconciler.StopReconciling() if err := c.EndpointReconciler.RemoveEndpoints(kubernetesServiceName, c.PublicIP, endpointPorts); err != nil { <del> klog.Error(err) </del> <ins> klog.Errorf("Unable to remove endpoints from kubernetes service: %v", err) </ins> } }()

the kube-apiserver's log print this
/sig kube-apiserver
/triage support /remove-kind bug /sig cli
Hitting this too with 1.13.5
Your cluster was not clean before you installing kubernetes cluster, you should deployed kubernetes on your env before. Clean you ENV, and re-deploy again could resolve this issue.
I use the , It is always new
I hava seem issue. until now,not fix. kube-apiserver logs at below:
I use kubeadm install k8s on one vps. yesterday everything is ok.
I hava seem issue. until now,not fix. kube-apiserver logs at below: k8s-master-1 kube-apiserver[]: E0617 13:55:33. ] Unable to remove old endpoints from kubernetes service: StorageError: key not found, Code: 1, Key: /registry/masterleases/172.21.230.246, ResourceVersion: 0, AdditionalErrorMsg:
trouble resolved by set current apiserver
I have too, v1.14.2 v1.14.4 v1.15.1 apiserver exit and kubeadm hung.
The error likely came from leaseEndpointReconciler#doReconcile :
1 ] Unable to remove old endpoints from kubernetes service: StorageError: key not found, Code: 1, Key: /registry/masterleases/192.168.56.100, ResourceVersion: 0, AdditionalErrorMsg:
can you explain how to please? It will help me a lot.
on worker server run command
, you mean init with --apiserver-advertise-address on master, don't you?
no on worker node
thanks, i had been creating HA cluster, that should not be your situation.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
I'm running into the same issue.

I'm able to run on an ubuntu box and it works but on Centos, I get the following error on kube-apiserver static pod and it restart: Kubeadm version: Docker version: Kubelet --version: kubeadm command:
Was there some clue from the log prior to 'Unable to remove old endpoints' ? thanks
Apiserver logs:
Considering 'StorageError: key not found', the endpoint might not exist. The subsequent "Create" seemed to succeed. Was there other symptom you noticed after 17:07:01 ? Thanks
No, it just crashes and restarts after that
Was there panic after 0120 17:07:10. ? Please share more log beyond 0120 17:07:10. if you have it. Thanks
Yes, the container panics and stops so no more logs after it. I tried querying etcd and there is no keys under , hence the key not found error but the question is why is it trying to remove a key that does not exist. Maybe, I have some old artifact remaining on my machine. I did to cleanup my old kubeadm setup
I take it back, there are more logs.
The issue in my case was k8s api-server was using ipv6 address ([::1]) which was not working. I disabled ipv6 and it fixed the issue for me
ipv6 has been turned off in my environment
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
I have the same problem
I have the same problem. Have you solved it
Hi!. Had same issue, solved it right now by upgrading docker on master host. Was version 18.09, now 19.03, after upgrade works without any errors. Solution inspired by this question , but without reinstall docker.

Just performed
I don't use kubeadm
I have a same problem, I used binary file Problem: Version:1.18.3
your second erroryou should disable the ipv6
I turned off IPv6But the error still exists I care more about the first question
I emailed you. qq mail
github
та же проблема
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
We see this error while we are having PLEG issues with kubelet. The below document says how to fix the PLEG issue. However, KOPS doesn't support in kubelet. There is already an issue open for this
not sure whats causing this. I had virtualbox VMs (3) on windows 10 amd64. and this issue happened after restarts, spent days to figure things out, including trying to set kubernetes up in different distros of linux. then i switched to vmware player, no issue so far. Again, my scenario is arbitrary, if anyone has the same setup as me, probably give vmware a try.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
same error on wsl2 when restarting the kube-apiserver
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-contributor-experience at . /close
Closing this issue.

<detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen /remove-lifecycle rotten
Reopened this issue. <detailsIn response to : /reopen /remove-lifecycle rotten </details
hit this this morning after the node my kind cluster was on restarted. curious what the resolution could be.
It is incredible that this problem has existed for more than 2 years and has not been solved. I encountered this error when deploying k8s using binary files, I tried v1.18 - v1.21, and I checked all the search results on google. Unfortunately, it took me 3 days to still not be able to solve this problem.
Small probability is caused by ipv6 try this If not works, could try this:
/remove-sig cli
I got the same errors with binary in my cluster
/sig api-machinery /remove-kind support /kind bug
/assign
anybody can share the full logs of the apiserver when this happens?
hmm the error is just informative, it happens for a new installation, it doesn't exist, throws the error and later the lease is this error in the log is a red herring, most of the cases reported are related to deployment issues, wrong IP family, KIND cluster restarts, ... I'm going to close this to avoid converting this issue in a magnet, TL;DR is not a problem, if apiserver doesn't start check further in the logs, the reason has to be there /close
Closing this issue.

<detailsIn response to : hmm the error is just informative, it happens for a new installation, it doesn't exist, throws the error and later the lease is this error in the log is a red herring, most of the cases reported are related to deployment issues, wrong IP family, KIND cluster restarts, ... TL;DR is not a problem, if apiserver doesn't start check further in the logs, the reason has to be there /close </details
Let's see if we can improve the error message
/triage accepted
showed up in kube-apiserver log even if configured because that's how golang worked, reproduceable code in: controllers in kube-apiserver accessed because net.InterfaceAddrs() returned (when ipv6 not disabled) and listening on [::]:6443 make that then following function returned [::1]
Faced the same issue today: And api-server didn't want to start. In my case it turned out to be that due to a network accident all our control-planes changed their ip addresses. I managed to start all api-servers and have them work as usual by changing all entries of old ip addresses to the new ones in . The error still presents, but everything works just fine. Maybe it is because of an old leases record with old ips somewhere in etcd (not sure).
Just ran into the same issue after updating from 1.23.17 to 1.24.15 (via kubeadm). The kube-api is unreachable. When we reboot, the API is visible shortly until the calico pods start (calico v 3.25.1). We tried both and with the same outcome.
following this guide everything just works fine on fresh ubuntu 20 and has the above issue on fresh ubuntu 22. did not get to the reason why this happens.


Question: pushd "staging/src/k8s.io/${repo}" >/dev/null 2>&1 echo "=== propagating to ${repo}" >> "${LOG_FILE}" # copy root go.mod, changing module name <del> sed "s#module k8s.io/kubernetes#module k8s.io/${repo}#" < "${KUBE_ROOT}/go.mod" > "${KUBE_ROOT}/staging/src/k8s.io/${repo}/go.mod" </del> <ins> sed "s#module k8s.io/kubernetes#module k8s.io/${repo}#"  < "${KUBE_ROOT}/go.mod"  > "${KUBE_ROOT}/staging/src/k8s.io/${repo}/go.mod" </ins> # remove `require` directives for staging components (will get re-added as needed by `go list`) <del> kube::util::list_staging_repos | xargs -n 1 -I {} echo "-droprequire k8s.io/{}"   | xargs -L 100 go mod edit </del> <ins> kube::util::list_staging_repos  | while read -r X; do echo "-droprequire k8s.io/${X}"; done  | xargs -L 100 go mod edit </ins> # rewrite `replace` directives for staging components to point to peer directories <del> kube::util::list_staging_repos | xargs -n 1 -I {} echo "-replace k8s.io/{}=../{}" | xargs -L 100 go mod edit </del> <ins> kube::util::list_staging_repos  | while read -r X; do echo "-replace k8s.io/${X}=../${X}"; done  | xargs -L 100 go mod edit </ins> popd >/dev/null 2>&1 done

I ran update- Not to see a ton of errors Kubernetes version (use ): HEAD of master
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig architecture (code organization / dependencies) /sig testing (hack scripts)
/triage accepted


Question: message ContainerImage { // Names by which this image is known. // e.g. ["k8s.gcr.io/hyperkube:v1.0.7", "dockerhub.io/google_containers/hyperkube:v1.0.7"] <ins> // +optional </ins> repeated string names = 1; // The size of the image in bytes.

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --is defined as For , both and are valid and they will be serialized to and in Json respectively. However in openapi-, is marked as required so is not allowed. This cause a mismatch when doing deserialization based on openapi spec. should not be marked as required because is a valid value. Kubernetes version (use ): v1.19 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig node api-machinery
/triage accepted /assign
Hello, is there any update for this issue?
Possibly related to and This bug can prevent a controller we're working on from enumerating nodes in a cluster, as we hit deserialization errors if any node in the cluster exposes an image with . It seems like proposed to address this by using the image ID instead of its name. I imagine that making this field optional would require a KEP, as it's an API change; but perhaps this could be fixed by ensuring that the field is always initialized (i.e. to an empty list) to prevent deserialization errors? This would not require an API change and would make the implementation conform with the API spec.
Looking at the sources, it looks like the image list is constructed here: It's not clear to me how it's even possible for to be uninitialized? I'll try to figure out how we got into this state...
I was just able to trigger this with though I imagine this can be replicated in other environments (at least in ) with a similar workflow: To get into this state, I first built my app with and then imported it into the local k3d cluster with . Then, I made some changes to my application and built it again with the same tag, and imported into the cluster with the same command. It's not clear whether the instance represents the first version of the application image and the named instance refers to the second build on the same tag, or vice versa.
Sorry for the late reply. That's interesting. I agree with you it seems impossible for to be nil.

<- doesn't seem to be the case. Looking
I can reproduce it by changing to . Either I missed something basic or there is a bug in the (kubelet uses v1 types directly) serialization pipeline.
I think I understand what happened: If an image has neither nor , kubelet sets to . However, when the object gets persisted to etcd in protobuf, the data is stored as -- protobuff doesn't distinguish and : Although we ask JSON API clients to tolerate values for such fields, openapi-generated clients (e.g. the ) treat as an invalid value for a required field during deserialization. It's hard to tell what the right fix is: it's a data lost in kube-apiserver serialization/deserialization IMO, but it may be hard to fix, since was the rollback of an attempt to fix the field as optional is an option, but it needs to be approved by someone from SIG Node (and for other APIs that may have similar issue, it would depend on the implementation details of each API). On the plus side, there seems to be no server-side validation that requires to be not . kubelet is the only writer of the field, and the fact is sometimes the field is written as the JSON API client tolerate value (as suggested by ) is an option, but it has to be done in a client-by-client basis. For , I think we can maintain a patch (namely treating as ) in the Python client to fix this. I'm not sure what client you're using though Please note that this could also be fixed if the node API takes a different approach when an image has neither nor , e.g. using the image ID as the name (could be a breaking change to existing clients that tolerates ) assigning a dummy name like <none(which could be confusing if there are multiple such images / an image that is literally called <none)
I'm not familiar enough with the internals of k8s' etcd protobuf serialization to know whether this is feasible in this context, but in other applications I've used wrapper messages to avoid this ambiguity. For example: This lets the value-ness of the names field be encoded distinctly from the list itself.

Another instance of this issue discussed here
Kubernetes python client fix is tracked in contains some API change suggestions for SIG node to consider, which could benefit JSON API clients: /unassign /remove-sig api-machinery /remove-triage accepted let's see if someone from SIG node can pick this up
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
thanks! But isn't that a breaking change if an old client talks to a new server? ah awesome! Probably we should just send a PR to SIG node to mark the field as optional /help
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : thanks! But isn't that a breaking change if an old client talks to a new server? Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign actually let me open one real quick
As I said, I'm not really clear on where etcd protobuf clients are implemented or how public that API is. Generally, though, it could be done gracefully via a deprecation cycle. A new field is introduced and the prior field is considered deprecated; Writers write both the deprecated field and the new type field; The API server continues to read the deprecated field for one major version. Once a major version has passed, change the API server to read the new typed field and stop writing the deprecated version... But this is all way more complicated than marking the field as optional. If it's acceptable to make it optional, that sounds fine to me! Thanks for your help looking into this
that makes sense. Our proto definitions and clients are generated from the files, so supporting the deprecation logic in the clients does sound like more work
Thanks for the fix I am using .

it turns out that already has a fix on master ( -- not yet on ) and I've put up a PR to make it usable from kube-rs ()


Question: CLOUD_CONFIG=${CLOUD_CONFIG:-""} KUBELET_PROVIDER_ID=${KUBELET_PROVIDER_ID:-"$(hostname)"} FEATURE_GATES=${FEATURE_GATES:-"AllAlpha=false"} <ins> EMULATED_VERSION=${EMULATED_VERSION:+kube=$EMULATED_VERSION} </ins> CPUMANAGER_POLICY=${CPUMANAGER_POLICY:-""} CPUMANAGER_RECONCILE_PERIOD=${CPUMANAGER_RECONCILE_PERIOD:-""} CPUMANAGER_POLICY_OPTIONS=${CPUMANAGER_POLICY_OPTIONS:-""}

Currently the local-up- script does not have support for the flag. This flag was recently here as a part of For development, testing, and full plumbing through of the feature the flag should be to local-up- to allow for usage of the feature when setting up a cluster via local-up-
/sig apimachinery
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /sig apimachinery Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig api-machinery
/sig testing /triage accepted /assign


Question: "ext2", "ext3", "ext4", <del> "xfs", </del> <ins> //"xfs", disabled see issue https://github.com/kubernetes/kubernetes/issues/74095 </ins> ), } // max file size

Which jobs are failing: ci-kubernetes-e2e-kops-aws pull-kubernetes-e2e-kops-aws Which test(s) are failing: [sig-storage] In-tree Volumes [Driver: local][LocalVolumeType: block] [Testpattern: Pre-provisioned PV (xfs)] volumes should allow exec of files on the volume [sig-storage] In-tree Volumes [Driver: local][LocalVolumeType: block] [Testpattern: Pre-provisioned PV (xfs)] volumes should be mountable Since when has it been failing: Don't know.. as the job was down for a while due to Testgrid link: Reason for failure: To be found.. Anything else we need to know: This issue is a blocker for turning on the kops-aws presubmit, see - /cc /kind failing-test /sig storage
cc
By looking at the log this line is suspicious: Both testcases share similar error message. And that comes from when doing I guess its some mount client missing from the AMI so that the mount library can't find it.
Looks like the test needs to skip XFS on environments that do not support XFS
This is where the environment is checked to skip xfs:
Thanks Michelle! - Could you comment if we need to skip it in aws tests?
Looks like this was broken by - shall we revert? cc
Actually looking at the GCE tests for kops it was broken before . I'm wondering if we assumed xfs back in
/priority critical-urgent this is blocker for kops currently example of failing build
do you have idea could break this?
Yes, in , I tests for local volume which use . Because , PR is not blocked by gce e2e test. hi, Quick way to fix is to remove from supported types, I can submit a PR for this.
Alternatively, we can skip for environments which do not support XFS . What do you think?
I do not know what is the best solution in long term but in short term we should get kops e2e tests working. So if removing xfs from supported types is fastest solution - i would go with that. After that tests could be fixed how those should be. Could you do that PR and remove from supported types?

Hard-coding the knowledge which clusters or tests support XFS (because that's what it is - a dependency on a kernel feature, right?) in the test source code is just bound to fail eventually. What if the presence of xfs support depends on the "gci" version? This line doesn't cover that: I would prefer to mark tests which depend on XFS kernel support with or something like it (no idea how descriptive feature tags need to be) and then enable those test cases only in jobs where the target cluster is known to have XFS. If you extend that check as a short-term solution, then please file an issue about it and link to it next to the check. /cc
Agreeing with here... Isn't there a way to check if the kernel we test for/ against is supporting it more directly? Building a huge list of whats compatible and what's not sounds not perfect to me.
Long term we can enhance the check to ssh into a random node and see if the binary exists. Short term let's just remove xfs testing for local volumes.
If these two are newly tests, agree to skip (or remove) xfs testing to unblock PRs.
assuming all these in-tree tests will be less important over the course of the year as CSI moves forward... so should we assume that the XFS tests will never come back . ?
The local volume driver plan is to remain in-tree for the medium-term future. I think we should continue to add the environment check for xfs so we can add it back.


Question: Expect(strings.TrimSpace(out)).To(BeEquivalentTo("test-after")) } <ins> func testSubpathReconstruction(f *framework.Framework, pod *v1.Pod, forceDelete bool) { // This is mostly copied from TestVolumeUnmountsFromDeletedPodWithForceOption() // Change to busybox pod.Spec.Containers[0].Image = "busybox" pod.Spec.Containers[0].Command = []string{"/bin/sh", "-ec", "sleep 100000"} pod.Spec.Containers[1].Image = "busybox" pod.Spec.Containers[1].Command = []string{"/bin/sh", "-ec", "sleep 100000"} By(fmt.Sprintf("Creating pod %s", pod.Name)) pod, err := f.ClientSet.CoreV1().Pods(f.Namespace.Name).Create(pod) Expect(err).ToNot(HaveOccurred()) err = framework.WaitTimeoutForPodRunningInNamespace(f.ClientSet, pod.Name, pod.Namespace, time.Minute) Expect(err).ToNot(HaveOccurred()) pod, err = f.ClientSet.CoreV1().Pods(f.Namespace.Name).Get(pod.Name, metav1.GetOptions{}) Expect(err).ToNot(HaveOccurred()) nodeIP, err := framework.GetHostExternalAddress(f.ClientSet, pod) Expect(err).NotTo(HaveOccurred()) nodeIP = nodeIP + ":22" By("Expecting the volume mount to be found.") result, err := framework.SSH(fmt.Sprintf("mount | grep %s | grep -v volume-subpaths", pod.UID), nodeIP, framework.TestContext.Provider) framework.LogSSHResult(result) Expect(err).NotTo(HaveOccurred(), "Encountered SSH error.") Expect(result.Code).To(BeZero(), fmt.Sprintf("Expected grep exit code of 0, got %d", result.Code)) By("Expecting the subpath volume mount to be found.") result, err = framework.SSH(fmt.Sprintf("cat /proc/self/mountinfo | grep volume-subpaths | grep %s", pod.UID), nodeIP, framework.TestContext.Provider) framework.LogSSHResult(result) Expect(err).NotTo(HaveOccurred(), "Encountered SSH error.") Expect(result.Code).To(BeZero(), fmt.Sprintf("Expected grep exit code of 0, got %d", result.Code)) By("Stopping the kubelet.") utils.KubeletCommand(utils.KStop, f.ClientSet, pod) defer func() { if err != nil { utils.KubeletCommand(utils.KStart, f.ClientSet, pod) } }() By(fmt.Sprintf("Deleting Pod %q", pod.Name)) if forceDelete { err = f.ClientSet.CoreV1().Pods(pod.Namespace).Delete(pod.Name, metav1.NewDeleteOptions(0)) } else { err = f.ClientSet.CoreV1().Pods(pod.Namespace).Delete(pod.Name, &metav1.DeleteOptions{}) } Expect(err).NotTo(HaveOccurred()) By("Starting the kubelet and waiting for pod to delete.") utils.KubeletCommand(utils.KStart, f.ClientSet, pod) err = f.WaitForPodTerminated(pod.Name, "") if !apierrs.IsNotFound(err) && err != nil { Expect(err).NotTo(HaveOccurred(), "Expected pod to terminate.") } if forceDelete { // With forceDelete, since pods are immediately deleted from API server, there is no way to be sure when volumes are torn down // so wait some time to finish time.Sleep(30 * time.Second) } By("Expecting the volume mount not to be found.") result, err = framework.SSH(fmt.Sprintf("mount | grep %s | grep -v volume-subpaths", pod.UID), nodeIP, framework.TestContext.Provider) framework.LogSSHResult(result) Expect(err).NotTo(HaveOccurred(), "Encountered SSH error.") Expect(result.Stdout).To(BeEmpty(), "Expected grep stdout to be empty (i.e. no mount found).") framework.Logf("Volume unmounted on node %s", pod.Spec.NodeName) By("Expecting the subpath volume mount not to be found.") result, err = framework.SSH(fmt.Sprintf("cat /proc/self/mountinfo | grep volume-subpaths | grep %s", pod.UID), nodeIP, framework.TestContext.Provider) framework.LogSSHResult(result) Expect(err).NotTo(HaveOccurred(), "Encountered SSH error.") Expect(result.Stdout).To(BeEmpty(), "Expected grep stdout to be empty (i.e. no subpath mount found).") framework.Logf("Subpath volume unmounted on node %s", pod.Spec.NodeName) } </ins> func podContainerExec(pod *v1.Pod, containerIndex int, bashExec string) (string, error) { return framework.RunKubectl("exec", fmt.Sprintf("--namespace=%s", pod.Namespace), pod.Name, "--container", pod.Spec.Containers[containerIndex].Name, "--", "/bin/sh", "-c", bashExec) }

Fix status: [x] fixed in master in [x] fixed in release-1.10 branch in , will be in 1.10.0 [x] fixed in release-1.9 branch in , will be in 1.9.7 [x] fixed in release-1.8 branch in , will be in 1.8.11 [x] fixed in release-1.7 branch in , will be in 1.7.16 In SubPath fixes, PR adds bind mount for subpaths. It uses as the path name. The volumename in this path is the container's mount volume name which is also called OuterVolumeSpecName in . This becomes an issue for volume reconstruction when kubelet restarts. Volume scans the volume directories and recover the InnerVolumeSpecName (which is used to construct pod volume paths). Volume reconstruction has not enough information to recover the OuterVolumeSpecName and figure out which volume it maps to. In case kubelet needs to clean up mounts (it happens when volume spec cannot be reconstructed), it will fail to clean up the subpaths because of lacking the OuterVolumeSpecName. We think it is better to use InnerVolumeSpecName for volume subPath instead of OuterVolumeSpecName so that volume clean up can work. The issue we might have is that some users might already deploy a cluster with subpath might already . If the users perform an in-place upgrade, since new version will use InnerVolumeSpecName which might have conflicts with some other older version's OuterVolumeSpecName among different volumes. This change requires clear documentation and communication with the users.
cc
This impacts reconstruction of PVCs, where the inner and outer names are different.
/assign
While testing the e2es against a cluster without the fix, I noticed that subsequent pods using volumes failed to start. My theory is that the unmount volume failed because the subpath was still mounted, and that blocked the volume manager from processing other pods. But I thought that volume manager is supposed to run the unmount volume operation in a separate thread per volume, so this shouldn't happen. Need to inspect the code more carefully to confirm the theory. could the volume reconstruction refactor have a unmount volume call in the main volume manager thread?
runs in reconciler's goroutine - so if that function is blocked, it will not be able to process any more volumes.

Was the problem that failed to unmount volume but got hung/stuck ?
I haven't confirmed for sure, but my guess is that unmount volume failed because the subpath was still mounted
moving into the milestone for discussion /milestone v1.10
/priority critical-urgent /status in-progress
/status approved-for-milestone per discussion in sig-release meeting today
[MILESTONENOTIFIER] Milestone Issue: Up-to-date for process <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
Re-opening to track fixes in release branches
keeping in the v1.10 milestone until merges
since this is now fixed in 1.10, I'm taking it out of the milestone so that the release team doesn't keep checking it.
Can this issue be closed now?


Question: errs = append(errs, validateScorePluginSetForConflictPlugins( pluginsPath.Child("score"), apiVersion, profile)...) } <ins> seenPluginConfig := make(sets.String) for i := range profile.PluginConfig { pluginConfigPath := path.Child("pluginConfig").Index(i) name := profile.PluginConfig[i].Name args := profile.PluginConfig[i].Args if seenPluginConfig.Has(name) { errs = append(errs, field.Duplicate(pluginConfigPath, name)) } else { seenPluginConfig.Insert(name) } if removed, removedVersion := isPluginRemoved(apiVersion, name); removed { errs = append(errs, field.Invalid(pluginConfigPath, name, fmt.Sprintf("was removed in version %q (KubeSchedulerConfiguration is version %q)", removedVersion, apiVersion))) } else if validateFunc, ok := m[name]; ok { // type mismatch, no need to validate the `args`. if reflect.TypeOf(args) != reflect.ValueOf(validateFunc).Type().In(1) { errs = append(errs, field.Invalid(pluginConfigPath.Child("args"), args, "has to match plugin args")) } else { in := []reflect.Value{reflect.ValueOf(pluginConfigPath.Child("args")), reflect.ValueOf(args)} res := reflect.ValueOf(validateFunc).Call(in) // It's possible that validation function return a Aggregate, just append here and it will be flattened at the end of CC validation. if res[0].Interface() != nil { errs = append(errs, res[0].Interface().(error)) } } } } </ins> return errs }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --Have a legacy scheduler config, upgrade to v1beta2 since we are at v1beta2 now, scheduler failed to start with some unreadable raw data, accurate and readable error message. config scheduler config like what has been said above. Kubernetes version (use ): latest. Cloud provider or hardware configuration: OS (e.g: ): all Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /sig scheduling /assign
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: assert.NoError(t, err) fakePodGetter := m.containerGC.podGetter.(*fakePodGetter) <del> makeGCSandbox := func(pod *v1.Pod, attempt uint32, state runtimeapi.PodSandboxState, withPodGetter bool) sandboxTemplate { </del> <ins> makeGCSandbox := func(pod *v1.Pod, attempt uint32, state runtimeapi.PodSandboxState, withPodGetter bool, createdAt int64) sandboxTemplate { </ins> if withPodGetter { // initialize the pod getter fakePodGetter.pods[pod.UID] = pod } return sandboxTemplate{ <del> pod:     pod, state:   state, attempt: attempt, </del> <ins> pod:       pod, state:     state, attempt:   attempt, createdAt: createdAt, </ins> } }

kubernetes pre-submit unit tests for pass some times and fails some times causing the flakiness. Here is one such failure https://k8s- Passing: Failing: cc
Yep, I'm checking with this
Sorry for introducing a flakiness. Sent out the fix in .
/assign
Dup of . /close

https://k8s- Failed: TestSandboxGC {kuberuntime}
https://k8s- Failed: TestSandboxGC {kuberuntime}
/assign
Should've been fixed by


Question: return new(int64), "", nil } <del> // isInitContainerFailed returns true if container has exited and exitcode is not zero // or is in unknown state. </del> <ins> // isInitContainerFailed returns true under the following conditions: // 1. container has exited and exitcode is not zero. // 2. container is in unknown state. // 3. container gets OOMKilled. </ins> func isInitContainerFailed(status *kubecontainer.Status) bool { <ins> // When oomkilled occurs, init container should be considered as a failure. if status.Reason == "OOMKilled" { return true } </ins> if status.State == kubecontainer.ContainerStateExited && status.ExitCode != 0 { return true }

Init container exited with OOMKilled reason, but init container was marked succesful, and main app container was allowed to be started If non init process gets killed the container should not be marked OOMKilled or If OOMKilled is terminate reason main app contaienr should not be started. According to Container should only be marked OOMKilled if init process is killed. the container the When checking init container status: But main app container is running. Kubernetes version (use ): Dockerfile:
According to previous issue will add sig label: /sig node
/assign
I have a quesion. when the oom killer, why your init container exit code is 0. normal is 137.
the init process is the shell script. the oom killer kills the forked process of stress tester, and it continues the script. next step is to exit 0 in the if process. wth error handling this can be caught, but my problem is, if there is a 0 exit code, and init process isnt oom killed, why is there a reason of oomkilled.
when you use shell scripts, you should add . kubelet will collect the status of init container. when init container is oomkilled, the container status will save the oom kill message.
I understand that. but according to the behaviour that is written, this is not expected outcome. The init container was not killed because of oom, just a single process. There should be 2 expected behaviours: Either there should be no printed, since it should only be when the main init process gets killed Or the whole init container should fail if any process is caught with oom.
I think you are right.
/assign
/triage accepted


Question: pod.UID = types.UID(hex.EncodeToString(hasher.Sum(nil)[0:])) glog.V(5).Infof("Generated UID %q for pod %q from URL %s", pod.UID, pod.Name, url) } <ins> // This is required for backward compatibility, and should be removed once we // completely deprecate ContainerManifest. if len(pod.Name) == 0 { pod.Name = string(pod.UID) glog.V(5).Infof("Generate Name %q from UID %q from URL %s", pod.Name, pod.UID, url) } </ins> if len(pod.Namespace) == 0 { hasher := adler32.New() fmt.Fprint(hasher, url)

I'm trying containervm with kubelet built from head and this YAML file: The YAML file is v1beta2 and doesn't set a name to the Pod. When starting kubelet, it doesn't start the "nc" echo server and complains about this on the logs: Chatting with on IM she mentioned it might be due to a change from that now requires non-empty pod names (replaces random generation with validation for non-emptiness) but that means all existing YAML files (even the ones with an older version) will stop working... Is that expected? Thanks! Filipe
Yes, the latest kubelet call validation on all sources: etcd, file, http, etc. now, and PodName is a required field. One potential fix for this is also invoke conversion code before call validation, like what we have done at apiserver. Thoughts?
Just so I get unblocked... Do you have an example YAML file that will set the pod name and will work? Right now this is what I'm using: Should I bump up version to v1beta3 and include a new field somewhere? Thanks! Filipe
The type of that yaml is not pod, and the kubelet code doesn't know how to read pods from a file. So, I think change needs to be reverted before you can make the next ContainerVM release. The v1beta2 vs v1beta3 should not have anything to do with it, unless I'm misunderstanding.
Or will it work to do this:
Still, not sure if you want to release a containerVM version that doesn't accept the old format?
Thanks I made it work with the YAML above! I still have errors in the logs: But the docker containers are being brought up... It's weird that is the YAML field that is missing, considering the source seems to say it's a deprecated field that will be removed at some point... I agree with your point that this needs to be fixed at head before the next containervm release. I opened to increase test coverage of standalone kubelet since we have been seeing many changes break that model recently. Cheers, Filipe
id in v1beta2 maps to name in v1beta3 Tim's change made name required. The internal representation is like v1beta3.

But v1beta3 is not officially stable so we don't want to use it in the GKE or containerVM docs.
The issue is caused by , and affect both v0.8 and v0.9 release for standalone kubelet to support old format ContainerManifests.
Ugh, I thought ID was really required. I spoke with Dawn a bit, and here's my thinking. 1) update all our docs and examples to include ID 2) in file and HTTP config sources, check for Name = "" and apply a default, maybe log a warning that this is deprecated 3) wait for the ContainerManifest -Pod conversion to really make it required


Question: cmdutil.AddApplyAnnotationFlags(cmd) cmdutil.AddRecordFlag(cmd) cmdutil.AddInclude3rdPartyFlags(cmd) <del> return cmd </del> } func RunEdit(f cmdutil.Factory, out, errOut io.Writer, cmd *cobra.Command, args []string, options *resource.FilenameOptions) error {

Kubernetes version : What happened: When I try to display configmap content with yaml output (kubectl get configmap webapp-inifile -o yaml), the output is sometimes pretty, and sometimes not -depending newlines are effectively displayed or not - just n char- I can't find what's the difference between some file and some others. (limit on numer of lines ?) Configmap is created with --from-file option Ex : What you expected to happen: More human readable display, in order to be able to easily change parameter when kubectl edit congifmap How to reproduce it :
Hrm, this seems to me like the right answer is to special case 'kubectl edit' for configmaps, rather than try to hack around with the YAML decoder. I suppose we could use block quotes (e.g. ) if we see newlines but that seems error prone...
Closing this, pretty print is there, kubectl configMap edit support is there.
This still happens on kube 1.7 :-( I'm not 100% sure how to reproduce, but what I did was: have an existing configmap and edit a few lines exit the editor ... now I see the characters instead of actual newlines. even shows the characters
I'm also seeing this issue in kube 1.7. During the past few days I have edited a configmap ~110 times, and seen the formatting get destroyed (spaces messed up, characters visible) twice. I can't determine anything specific that I did in those two edits that would have triggered the formatting messup.
I'm still seeing this issue too. Pushed a lot of identical configmaps, some ones are displaying good, and some other bad. For example, I have a configmap with 3 files in ; they all are identical except one variable ; 2 of them display normally, and the last one displays with n ...
also seeing this issue on 1.7
can we reopen this issue? or open a new one?
I created a brand new configmap using and when I viewed the yaml with it's full of n characters and escape characters. This makes it very difficult to edit.
I am experiencing the same. n character is messing up a configmap I'm creating with --from-file option. Anyone found a reason why this is happening ?

Also failing in 1.8.4
I am also facing the same issue of formatting .. I have a application configuration of 400-500 lines. Tryied many times. For fist 40-50 lines the format is good ans easily readable.. However, when i try to create config for entire configuration file its very difficult to edit the config map..
Same issue here. No idea what makes one configmap show better than another.
This issue corrupted my aws-auth configMap that authenticates my user to access the EKS cluster and while trying to add other users to the configMap the formatting corrupted the configMap file and now I am locked out of my EKS cluster. Version: 1.10.3
I don't know if this will help somebody else but I was facing the same issue with escaped data in a config map edit and get (the config map had a single .conf file), and it seems that I had a unneeded extra space at the end of a single line in my data. When I removed that extra space the pretty printing was maintained in edit and get.
11 Still same
still same ( there is an example if install helm install stable/prometheus , you will find cm/prometheus-server )
I agree with - I can recreate it every time If you were to execute the following (ensuring no trailing spaces), the configmap remains "pretty" However, repeat the above (or edit the existing one with ), and put a trailing space after one of the key/value pairs, and that then messes up the configmap contents
Can we reopen this issue? I need this feature to change the yaml in cm, and I have been struggling with the intent and n for several days ...
Please reopen this ticket, it's incredibly frustrating to have to deal with this even on small configmaps
Got this also. I had to remove tabs from my config file to get the yaml multiline display.
Just created a messed up configmap on kuberntes 1.12.2.
Likewise.
OMG thank you!
Just ran into this in EKS (1.10.3) -- highly annoying. I have one config map that will come out OK if I use create directly, but if I pipe the output of to apply then it gets munged. Does seem like this is something that needs addressing.

Have the same issue and fixed by removing tabs
Have the same issue, don't have any tabs in file. This issue totally should be reopened.
This issue is valid and I am not sure why it was closed
Not only spaces. Trail symbols too. I uses vim for it
"n n" (spaces between end of lines) in one of my files also messed up the pretty display.
Created a new issue when I hit this again and was able to repro using the trailing whitespace
I've removed all the trailing whitespaces and tabs. The problem is gone.
Check any possible special chars. It solve my problem some time ago
I have seen this issue in K8 1.7 -1.13 kubectl create configmap NAMESPACE --from-file=$PATCHFILE --dry-run=true --output=yaml where PATCHFILE is attached file below. I renamed it from .json to .log so this webpage would let me include it.
Solve the issue by delete whitespace at each end of line.
The output of is ugly. Wether using json or yaml, k8s injects a whole bunch of escape characters and messes up the formatting of the original config file. It would be very nice to have a switch that tells it to dump the config out "normally"!
I a new line in a configMap using Tab for identation. After changing to Spaces instead of Tab, I was able to see the configmap as expected...
Same here, white space/s at the end of line ruin the output, everything back to pretty when white space/s removed.
This worked for me.
Since I've still met this on 1.16, I'll point a bullet list of the regular culprits for the configmap messing: spaces before newline:Tabs:Special characters like ãẽõ, áéó, öß will also mess your files. Look up for them in your comments, they might pass through!
An easy one liner to remove trailing whitespaces in a ConfigMap:
Please reopen. This issue still remains. This format makes eks aws-auth configmap very hard to maintain. Once the file is applied with trailing whitespaces the whole thing gets lots of n and . As a temporary measurement is there a way to reformat the configmap in the "readable" way? after the n are created?

if you use way didn't reformat the file for me.
Nurza's method works. But you may have a space before EOF as well: Or you may have CR's in there , in which case you should remove the CRs. Or you may have a TAB, which you should replace with one or more spaces.
No CR/TAB/new lines. I have a "clean" file that after I apply it all works great. (display as expected) After I add one trailing white space to it, it gets those n. I tried to run Nurza's method but it didn't work. Any other Ideas why Nurza's method is not working for me ? This a part of the output I'm getting with
has a space after it, although you don't see it. Because the LF is on the next line, there must be a space before it.
Correct but if I run Nurza's method the n are still there. Am I still missing anything? This is what I run
Nobody said that sed expression was perfect. Once you know this is your doing -- and you know how to fix it -- you're not bothered by this behaviour anymore.
This is scary shit :hankey: ... what is the industry practice to edit the file like these: aws-auth...? I managed to type 20+lines by hand :disappointed: and it worked on 3rd attempt :disappointed_relieved:
Cant' say about the 'like this', but aws-auth you can use eksctl to do the hard work for you:
If the formatting of cm goes wierd a simple hack to get it back to normal is : kubectl get cm configmapname -o yaml cm.yamlNow copy the contents of file and past it on is powerful tool to check the linting of yaml files. This will provide you with the configmap as expected with correct formatting. Paste the output in another yaml file (for e.g - )kubectl apply -f
Anyone knows why the issue is closed? It's not fixed as far as I can tell.

If you just want the raw output as it was read in when created --from-file, you can use jq to get the raw string (without escaped newlines etc) If you created a configmap from a file like this: Get the data:
Edited a cm, came back in, and had n all over. Something about white space in my previous edit? Not great overall :/ On kubectl version Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.5", GitCommit:"", GitTreeState:"clean", BuildDate:"2020-07-04T15:01:15Z", GoVersion:"go1.14.4", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.6", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-11-13T11:11:50Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}
Awesome, very good website~
Thank You very much
I was able to fix this behavior by: 1) Don't use tabs, convert to spaces 2) To remove spaces before a newline character, use this: It will also convert CRLF to LF only, by removing .
Something small that exacerbates this issue is that in python's standard lib module , up to some version (I think python 3.4, see ), pretty printing adds a trailing space after every comma by default, unless you supply a list of separators yourself that omit the spaces (e.g. like ) So if you have old-ish glue (e.g. python 2.7) that parses a k8s configmap into a file for archival purposes, and you made that file readable by pretty-printing it, using that file to regenerate a new configmap results in this gross escape behaviour.
Hello There, One of our colleague has edited configmap with wrong data, now whole cluster inaccessible with any of role, need help on immediate basis and much appreciated. Error: through eksctl - Error: getting auth ConfigMap: Unauthorized through kubectl error: You must be logged in to the server (Unauthorized)
yaml marshallers will write a string to a multiline string e.g.

UNLESS string contains a character string terminates in a space character string contains a substring which matches the regex
I am in the same situation. Were you able to solve this?
Also reproduce in v1.22.3 :(
I occasionally encountered this problem in v1.15.6 or v1.21.2, but eventually I was able to deal with it by using edit and then using vim's own set list to back-propagate the original static configmap code configuration under gitlab to see if there were extra spaces, and then modifying the local configmap file based on the online configmap's set list display under edit. The state of the online configmap under edit, to modify the local configmap file, is ultimately able to deal with this problem, but if there is a better official way to do it would be good.
yes if you are working on linux
i would like to share an observation that helped me sort out the line merging and n. it appears to be related to the last line of the file you are using (--from-) if the last line ends with empty space or tab. the config map is merged up in one line with n as line feed if you remove the empty space from the last line, the ConfigMap will import file as is with +
I got the same error after editing the ConfigMap file. I used an online tool to remove the trailing white space that we cannot see in the text. After copying the content which removed trailing whitespaces I was able to resolve this issue FYI - (online tool which I used)
in v1.24 also i can see same issue. how we can edit cm with many lines.
Finally, I found that the absence of a value defined or the value is empty string in will cause the issue. data: {{ }} is not defined in my yaml just use quote to the value resolved
delete space before n or after n it works for me
did anyone find solution when we edit cm how we can see in proper format as for me really difficult to edit cm as we have many n char and I can still see this issue in k8s version 1.28


Question: return err } <del> o.Namespace, _, err = f.ToRawKubeConfigLoader().Namespace() </del> <ins> o.Namespace, o.EnforceNamespace, err = f.ToRawKubeConfigLoader().Namespace() </ins> if err != nil { return err }

I'm writing a small bash script which generates some k8s resources in a namespace (other than ). When using to generate a service, the yaml which is generated does not include the defined namespace. I've also tried to move around the argument to the beginning (i.e. , but that doesn't work as well. Generated yaml should include the , when it's defined as an option. Kubernetes version (use ): Cloud provider or hardware configuration: n/a (not really important) OS (e.g: ): Kernel (e.g. ): Install tools: kubectl Network plugin and version (if this is a network-related bug): n/a Others: /sig cli
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cli
/assign