Question: annotations: scheduler.alpha.kubernetes.io/critical-pod: '' spec: <del> affinity: podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - weight: 100 podAffinityTerm: labelSelector: matchExpressions: - key: k8s-app operator: In values: ["kube-dns"] topologyKey: kubernetes.io/hostname </del> tolerations: - key: "CriticalAddonsOnly" operator: "Exists"

Our 5k-node cluster jobs recently started failing even before the e2e tests start. The reason is the step which checks that all kube-system pods are running is timing out: cc
From the scheduler logs I'm seeing that the pods are being scheduled quite slowly, getting worse with time: 2-3 pods/min looks quite bad. cc
Also finding such traces: Seems like most of the time is being spent in priority calculation.
From the pod spec, I'm seeing soft inter-pod anti-affinity at node-level topology: I'm almost sure this is the one causing the mischief. We've been having performance issues with anti-affinity traditionally :p
Ok... PR is the cause. It introduced anti-affinity to kube-dns. I think we should revert this change given known issues with anti-affinity. - Opinions?
FYI - it is taking 80m until the last pod is scheduled when I'm running 5k-node cluster locally.
That's quite unfortunate. It looks like we should revert the change until the anti-affinity change is scalable enough to be used more broadly, unless the scheduler folks have a target fix for this use case.
cc
We are very well aware of the performance issues in our affinity/anti-affinity functions, but I am not aware of a concrete solution at this point. This is partly due to the flexible design of affinity/anti-affinity concept. My humble opinion at this point is to drop some of the flexibility as they are rarely used, in order to streamline the algorithm and improve performance. Anyway, I think the proper solution to this problem needs more time than your requirements to address the issue in hand. So, you may want to change your configuration and drop the anti-affinity rule.
would it make sense to add a note to the anti-affinity user documentation to hint at performance problems at large scale, until this is fixed?
Re-opening this issue as this problem has struck again.

The following runs of our 5k-node scalability jobs failed to bring up the cluster on time: https://k8s- https://k8s- https://k8s- https://k8s- And the reason is kube-dns pods pending to be scheduled for a long time:
These failures started happening about 4 days ago and I'm pretty sure it's this one: Add self anti-affinity to kube-dns pods Sadly, we'll have to revert it again for now :( cc - Seems like the improvements to anti-affinity performance made under weren't enough?
We need to find other improvements for the predicate. If we cannot find major improvement opportunities, we may have to change its design to be more scalable at the cost of the reducing its flexibility.
- my personal opinion is that we need to change the design anyway. Having it work both ways (i.e. existing pods everywhere impacting placing of our pod) will always be painful (especially for things like cluster autoscaler)
I agree. The symetrical application of anti-affinity is problematic. I also think we need to change it.
Great. - do you agree with that too?
, I'm wondering after some affinity/anti-affinity optimization in scheduler, do you think if it's fine to test coreDNS with anti-affinity policy in large cluster with 5000 nodes?
We probably should try to see where we are - but this requires testing before enabling that.
I agree with - Could we first try checking with a benchmark integration test () as a proxy for real cluster test?
I'm wondering how many real nodes are required for testing?
Since the integration tests fake out the nodes, I think it's safe to crank it up to 5000 (that's the current largest integration test benchmark size) I'm also hoping to refactor our benchmarks soon, so let me know if you have any thoughts about them.


Question: <ins> /* Copyright 2017 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package preflight import ( "fmt" "net" "net/url" "time" ) const connectionTimeout = 1 * time.Second type connection interface { serverReachable(address string) bool parseServerList(serverList []string) error CheckEtcdServers() (bool, error) } type EtcdConnection struct { ServerList []string } func (EtcdConnection) serverReachable(address string) bool { if conn, err := net.DialTimeout("tcp", address, connectionTimeout); err == nil { defer conn.Close() return true } return false } func parseServerURI(serverURI string) (string, error) { connUrl, err := url.Parse(serverURI) if err != nil { return "", fmt.Errorf("unable to parse etcd url: %v", err) } return connUrl.Host, nil } // CheckEtcdServers will attempt to reach all etcd servers once. If any // can be reached, return true. func (con EtcdConnection) CheckEtcdServers() (done bool, err error) { // Attempt to reach every Etcd server in order for _, serverUri := range con.ServerList { host, err := parseServerURI(serverUri) if err != nil { return false, err } if con.serverReachable(host) { return true, nil } } return false, nil } </ins>

See: This code should make several attempts before ing. Alternatively, we could consider not listening for connections until this code has succeeded.
Feel free to correct me if I'm wrong on any of this since I've never looked at kubernetes code base before. The code for the and the both appear to make multiple attempts before returning an error which leads to a Fatal.
, I think we had waited for to start at . Do we need to wait more or make it configurable?
I would like to work on this if nobody else is.
No need to ask for permission! Just say here that you're working on it, that should be sufficient to prevent duplicate work. :)
Can I have a go at this?
I have a nearly completed solution for this already. Just working on testing/test cases. WIll hopefully sumbit a PR shortly.
Ok.
I wondered why we care about this. After all, kubelet restarts failing pods. For posterity, after talking with the reason is that the apiserver may fail so fast that the kubelet begins to back off on restarts of a failing pod. That backoff may be too aggressive. (Maybe it causes spurious alerts, or maybe it allows too much downtime.) So, let's fix the apiserver. That's fine, but clearly we're not the only user that might encounter this. You may be running a server you don't want to (or can't) modify, and you might wish you could tweak the kubelet's backoff heuristics for your pod. Alternately, maybe the kubelet could choose not to enter the backoff if apiserver exited with a different code. anything like this already in place?
That's a wonderful explanation that hadn't occurred to me. If there isn't a feature request for this already I'll work on something and reference this issue.
AFAIK, the backoff mechanism is there to protect kubelet from being overwhelmed by failed user containers. Trusting the containers with returning a specific exit code sounds like a big "no" to me. Making this configurable per pod might be ok, but it's still not good in general to allow aggressive restarts.
The referred PR () only fixes the issue for kube-apiserver. We still need a generic fix in genericapiserver (Ref ).


Question: package vclib import ( <ins> "sync" </ins> "time" <ins> "github.com/vmware/govmomi/vim25/types" </ins> "k8s.io/component-base/metrics" "k8s.io/component-base/metrics/legacyregistry" )

<!-- Please only use this template for submitting enhancement requests --What would you like to be : We need reporting of version to be shown in the metrics. Why is this needed: The goal behind this is to add some info about the underlying infrastructure for the kubernetes cluster. Some customers were experiencing issues during the cluster upgrades, but nothing in the upgrades themselves or kubernetes code for provider seemingly could cause those. The issue is storage related, and happening on a variety of releases with a significant delay from any other underlying infrastructure change, to which the kubernetes cluster has no relation and a way to know about. So knowing used hostnames, and versions associated with it over time would be extremely beneficial for debugging in similar circumstances. Some of the use cases we face: 1) A is enabled in a customer environment, and so there occurs a periodic automated vMotion for VMs backing kuberentes . We'd like to know on which host the actual kubernetes was located at any moment, and if any vCenter version transitions happened over time, new hosts /removed/renamed.
/sig cloud-provider /wg provider-vsphere
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig cloud-provider /wg provider-vsphere Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/area provider/vmware


Question: if vol.Path != csiMounter.GetPath() { t.Error("csi server may not have received NodePublishVolume call") } <ins> // ensure data file is created dataDir := filepath.Dir(mounter.GetPath()) dataFile := filepath.Join(dataDir, volDataFileName) if _, err := os.Stat(dataFile); err != nil { if os.IsNotExist(err) { t.Errorf("data file not created %s", dataFile) } else { t.Fatal(err) } } data, err := loadVolumeData(dataDir, volDataFileName) if err != nil { t.Fatal(err) } if data[volDataKey.specVolID] != csiMounter.spec.Name() { t.Error("volume data file unexpected specVolID:", data[volDataKey.specVolID]) } if data[volDataKey.volHandle] != csiMounter.volumeID { t.Error("volume data file unexpected volHandle:", data[volDataKey.volHandle]) } if data[volDataKey.driverName] != string(csiMounter.driverName) { t.Error("volume data file unexpected driverName:", data[volDataKey.driverName]) } if data[volDataKey.nodeName] != string(csiMounter.plugin.host.GetNodeName()) { t.Error("volume data file unexpected nodeName:", data[volDataKey.nodeName]) } if data[volDataKey.volumeLifecycleMode] != string(tc.mode) { t.Error("volume data file unexpected volumeLifecycleMode:", data[volDataKey.volumeLifecycleMode]) } </ins> }) } }

Instantiating the interface causes the creation of pod local volume directory and json data file and not - This causes a problem where is failed (and as a result never gets called) then those pod local volume directories will never be cleaned up. /sig storage cc
Looks like is the PR that introduced this change and moved data file creation calls to .
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle stale
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/repen /remove-lifecycle stale
/reopen
Reopened this issue. <detailsIn response to : /reopen </details
/remove-lifecycle stale
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue.

<detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen /remove-lifecycle rotten
Reopened this issue. <detailsIn response to : /reopen /remove-lifecycle rotten Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-contributor-experience at . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen /lifecycle frozen
Reopened this issue. <detailsIn response to : /reopen /lifecycle frozen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment.

<detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: }, numErrs: 0, }, <ins> { name: "add loadBalancerSourceRanges", tweakSvc: func(oldSvc, newSvc *api.Service) { oldSvc.Spec.Type = api.ServiceTypeLoadBalancer newSvc.Spec.Type = api.ServiceTypeLoadBalancer newSvc.Spec.LoadBalancerSourceRanges = []string{"10.0.0.0/8"} }, numErrs: 1, }, { name: "update loadBalancerSourceRanges", tweakSvc: func(oldSvc, newSvc *api.Service) { oldSvc.Spec.Type = api.ServiceTypeLoadBalancer oldSvc.Spec.LoadBalancerSourceRanges = []string{"10.0.0.0/8"} newSvc.Spec.Type = api.ServiceTypeLoadBalancer newSvc.Spec.LoadBalancerSourceRanges = []string{"10.180.0.0/16"} }, numErrs: 1, }, </ins> } for _, tc := range testCases {

a service, type LoadBalancer. that and add a field with a value that the iptables rules do not appear It does seem that the GCE firewall rule is created. patch?
Is this a regression?
No. This has never been "right" as far as I can tell. , Phillip Wittrock wrote:
can this be closed now?
Nope. The root cause has not been fixed yet.


Question: // sync tries to retrieve existing data and send them to process. // The revision to watch will be set to the revision in response. <ins> // All events sent will have isCreated=true </ins> func (wc *watchChan) sync() error { opts := []clientv3.OpOption{} if wc.recursive {

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): No What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): etcd watcher Is this a BUG REPORT or FEATURE REQUEST? (choose one): Bug report <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): HEAD / commit Environment: Cloud provider or hardware configuration: local VM OS (e.g. from /etc/os-release): Fedora 24 Kernel (e.g. ): Linux localhost.localdomain 4.7.7-200.fc24.x8664 SMP Sat Oct 8 00:21:59 UTC 2016 x8664 x8664 x8664 GNU/Linux Install tools: N/A Others: N/A What happened: If you do a watch from resource version "0" using (the etcd v2 client code), it initially performs a get of the key in question and returns Added or Modified events, based on a comparison of each node's createdIndex and modifiedIndex. This works all the time. If you do a watch from resource version "0" using (the etcd v3 client code), it attempts to get for each kv returned, in an attempt to provide both the previous and current versions of each kv/object. This works as long as has not been compacted. If compaction has occurred, the initial will fail, and it's not possible to watch from "0"' for this particular key.

What you expected to happen: Either: v3 watcher should not change the behavior from the v2 watcher; namely, it should not attempt to get the previous revision of each object and instead just return the current version as a event agree that it's ok to change the behavior, and in the event that the database has been compacted and the previous revision of an object cannot be retrieved, that object is sent as a event instead of . How to reproduce it (as minimally and precisely as possible): I'm working on a unit test and fix. Anything else do we need to know: cc
I will take a look and get back to you tomorrow.
If you have a pr ready, just assign to me or hongchao.
I'll have one tonight hopefully
Although it depends on if we pick option 1 or 2
The behavior of etcd2 is not technically correct. Watch from 0 and will give different results if watch from 0 always return "Added" even if 0 and should be the same. But we do not really have defined what does "0" mean in the interface. Also for etcd3, we cannot guarantee that watch can always return the previous value. Option 1 seems to be easier to achieve. If we do option2, then we will have a "Modified" event without having previous value. I am not sure if that is acceptable or not.
Just give an update here: I talked with on this a bit. We agreed that the behavior should be the same as etcd2 watcher. Of course the API seems non-standard right now. We need to rewrite it in better API once this gets more stable.
Actually, it looks like the event type is a red herring.

The etcd2 client has always set Added/Modified based on the equality/inequality of the node's createdindex/modifiedindex: the watch cache always sets the event type to Added when watching from resourceVersion=0 etcd3 (when watching from 0) is still doing a Get() of the previous revision of every KeyValue: The extra get of the previous version is unnecessary if ModRevision == ModCreation, and can fail if the Get returns an ErrCompacted (it also seems like a potential performance problem to fan out N Get() calls during a single watch establishment
potential fix in
I pretty dislike the fact that: on 0 will first do a sync and then start to watch are forced to return prev value for all modification events watch on 0 should just start watch from the latest revision. recovery the initial state (thus the sync or the initial get) should be a separate operation. But, yea, I guess we need to preserve this at least for now.
- yes, I think there is general agreement for it, though we can't change that in v1 API. There is more-than-1-year old issue for that:
For this case, it won't fail. It returns empty result.


Question: // About to get clients and such, detect standaloneMode standaloneMode := true <del> switch { case s.RequireKubeConfig == true: standaloneMode = false glog.Warningf("--require-kubeconfig is deprecated. Set --kubeconfig without using --require-kubeconfig.") case s.KubeConfig.Provided(): </del> <ins> if len(s.KubeConfig) > 0 { </ins> standaloneMode = false }

This will be the end of the deprecation period for --require-kubeconfig.
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list .
ref
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale


Question: if obj.PodEvictionTimeout == zero { obj.PodEvictionTimeout = metav1.Duration{Duration: 5 * time.Minute} } <ins> // NodeMonitorGracePeriod is set to a default value of 50 seconds. // This value should be greater than the sum of HTTP2_PING_TIMEOUT_SECONDS (30s) // and HTTP2_READ_IDLE_TIMEOUT_SECONDS (15s) from the http2 health check // to ensure that the server has adequate time to handle slow or idle connections // properly before marking a node as unhealthy. </ins> if obj.NodeMonitorGracePeriod == zero { <del> obj.NodeMonitorGracePeriod = metav1.Duration{Duration: 40 * time.Second} </del> <ins> obj.NodeMonitorGracePeriod = metav1.Duration{Duration: 50 * time.Second} </ins> } if obj.NodeStartupGracePeriod == zero { obj.NodeStartupGracePeriod = metav1.Duration{Duration: 60 * time.Second}

The controller-manager node-lifecycle-controller watches node lease and update probeTimestamp , and it check node probeTimestamp is before . The node-monitor-grace-period default value is 40s. When the client connection lost, the lease watcher cannot receive new event. With the http2 health check, HTTP2PINGTIMEOUTSECONDS default is 30s, HTTP2READIDLETIMEOUTSECONDS default is 15s, it will delay 45s to kill http2 connection. During this period, node-lifecycle-controller will mark all nodes NotReady. Here are some logs When the http2 connection lost, the watch should stop as soon as possible. The reflector should relist resource, and node-lifecycle-controller should not mark all node NotReady. HTTP2PINGTIMEOUTSECONDS+HTTP2READIDLETIMEOUTSECONDS should be less than 40s. the controller-manager node-lifecycle-controller watch client srcIP:srcPort. Drop all tcp package send into srcIP:srcPort. for more than 40 seconds, all node will be marked NotReady node will be Ready after some seconds. No response <details</details<details</details<details</details<details</details<details</details<details</details
How about double-check node lease before marking node NotReady? The node-lifecycle-controller can get reference lease from apiserver with
/sig node
/sig api-machinery
/cc We have discussed this timeouts values before and myself, I also think got some metrics with lower timeouts for the HTTP2 connections. In this situation is clear that the defaults of the node watcher and the timeouts are related and having good defaults will be ideal, but the question is which values do we change ?
I think in internal network, the ping response almost return in 1 second, the HTTP2PINGTIMEOUT_SECONDS default value should be 5 seconds. And the total timeout will 35 seconds. WDYT?
/triage accepted any pointers in other places where those timeouts were changed for http2? Any notes from those discussions?
/priority important-longterm since it is not a regression I will keep it longterm priority and not higher. But we can reevaluate if we hear more reports about it.

/assign
I'm investigating problems similar to this, but I'm looking for a way to confirm this is what is happening - and if there are tips for how to recreate such a scenario, that would be very helpful. We get infrequent reports of "worker goes NotReady for a short time" incidents that appear to be triggered by events like failing over a LB sitting front of the kube-apiservers. We've done some tests with a simple client-go app and have recreated a scenario in which the app logs messages after 45 secs of "context exceeded" errors. The failures reported to us from customers don't have http2 errors in the kubelet logs, and our attempts to recreate with kubelet have never shown more more than a single error which works on subsequent retry.
Hi folks, we have a simulated recreate of the problem. This recreate involves injecting a 40 second delay in "connectivity" between the Kubernetes API server and etcd (we a 40 second delay to etcd defrag and defragged all etcd members at the same time). The connectivity between these two components is obviously critical to nodes successfully updating their leases in order to stay ready. We use 40 seconds because it matches . The test will usually cause one or more of the cluster nodes will flap their readiness state. Setting on the Kubernetes controller manager prevents the flap. Given this, I think that one or more of the Kubernetes default configurations needs to change in order to tolerate such "connectivity" problems in order to maintain healthy node state.
What's the proposal then? Do we update --node-monitor-grace-period to 50s? It seems the less risky change as the client-go defaults are more sensitive and impact the whole ecosystem, beyond kubernetes itself are you open to increase this default? It seems worth it to improve the reliability of cluster
I think we should update .
can someone send a PR to see how it will look like?
I have been following this discussion as we had encountered the bug as well. I have created a PR - Please let me know if it works
I'll add another data point from our testing. Namely, the simulated recreate noted depends on the Kubernetes controller manager (KCM) leader election configuration. Using the following leader election configuration for the KCM yields a KCM that has no leader after the delayed etcd defrag.

As a result, nodes won't be marked not ready unless the KCM leadership is restored before the nodes renew their lease. Extending the KCM leader election configuration to the following, makes the problem easier to recreate.
/assign Reassigning the issue to me


Question: } } for i := range status.InitContainerStatuses { <del> if status.InitContainerStatuses[i].State.Terminated != nil || status.InitContainerStatuses[i].State.Waiting != nil { </del> <ins> if status.InitContainerStatuses[i].State.Terminated != nil { </ins> continue } status.InitContainerStatuses[i].State = v1.ContainerState{

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --When a pod is evicted under resource pressure, it goes into an error state. The pod terminates normally. Note that the 130GB file is based on 90% of total disk space - current utilization to hit the default eviction threshold. Full pod spec after going into error state: <details</detailsSee also Eviction tests () are currently broken (), there's a PR up to fix: Kubernetes version (use ): HEAD/1.23.x (kubeletVersion="v1.23.0-alpha.3.47+") Cloud provider or hardware configuration: local-up-cluster OS (e.g: ), Kernel (e.g. ):Install tools: local-up- Network plugin and version (if this is a network-related bug): N/A Others: Also reproducible on OpenShift 4.9 (1.23-based) /sig node /cc
mentions this looks very similar to
Confirmed that this is a regression from 1.21. When running the reproducer against the release-1.21 branch, the pod above correctly stays in Evicted status (since pods are mortal). On 1.22, it tries to restart, entering either an Error or ContainerCreating state. /kind regression
/assign
/triage accepted
From : is the first bad commit This was
Seems likely that wasn't actually fixed. The broken eviction tests made it much harder to verify.
Attaching logs from HEAD from reproducer: v=4 Let me also try to get a repro where it goes into Error
Logs from HEAD for reproducer when pod goes into Error: v=4
Roughly, kubelet syncTerminatingPod sends "Failed" to status manager (which looks like it generates the correct apiserver patch for phase running -failed), then somehow by the time Kubelet syncTerminatedPod is called it sends "Running but with errors" to status manager, and status manager allows that transition (internally) and then sends it to the apiserver. So the pod goes Running -Failed -Running which should never happen (we "lost" that the pod was stopped).

Bug 1: pod patch status probably shouldn't allow terminal -nonterminal phase transitions (didn't we fix that??) Bug 2: pod worker is supposed to be getting most up to date status from status manager before calling syncTerminatedPod, which should include "Failed", but that isn't happening? thus the status manager then sends a "non-terminal state" (shouldn't ever happen) Roughly, figure out why status passed to syncTerminatedPod is older than what was sent by syncTerminatingPod, then figure out why status manager allowed that transition back to running, then figure out why apiserver didn't reject.
Staring at logs, I've figured out where the illegal phase transition is happening: If the container failed to properly terminate and generate a status saying as much, generateAPIPodStatus won't have any hints available to it that the pod died because it was killed. (This is happening in the reproducer case because the container won't exit before the termination grace period, and needs to be force killed.) Thus, it will assume that since the container used to be running, its current status is running. This results in an illegal phase transition.
From what I can tell, the issue is that is being passed a pod that still has running containers. From kubelet logs: which gets set as part of . It looks like that will always run so long as succeeds. Thus, I feel like syncTerminatingPod must have the bug: it returns successfully, but the containers of the pod aren't all actually terminated.
/priority critical-urgent based on priority of and the impact of this regression (Kubelet-initiated evictions frequently result in error state pods)
The containers ARE terminated, we just haven't retrieved their status. This is because of a decision to not refresh the container status cache in syncTerminatingPods. So syncTerminatingPod is not a bug.
So then perhaps the right thing to do here would be to refresh the container status cache before syncing another status to the API server in ? Then we can avoid syncing the outdated cached status
Updating from an internal discussion last night - bug 2 isn't actually a bug.

Lantau convinced me we wouldn't refresh the status cache, so what happens is we call generatedAPIStatus with: API status (mostly irrelevant) status manager status to date container cache (but 2s) after The key thing is that phase from runtime status is irrelevant when 1 or 2 are terminal - because runtime has no idea about whether a pod is terminal. So roughly we should be ignoring phase from 3 except when determining when to go from pending to running. The bug here is that we're thinking that this is an "illegal transition" but still copying the latest reason and message (runtime status doesn't get to weigh in on overal terminal phases). We shouldn't need to refresh status at all.
Okay, so there is definitely a bug in what's being synced to the API server for evicted pod statuses right now, possibly for other terminated pods. In 1.21 and earlier, when a pod is evicted, it looks like this: <details</detailsNote that the status field is very minimal, and there are no container statuses. In 1.22+, we attempt to sync container statuses on termination, and sometimes they are wrong (e.g. changing phase from Failed -Running). Here's an example of what an evicted pod could look like on 1.22: <details</details
This appears to be broken even for runOnce pods, tested against latest HEAD (). With spec: After scheduling the pod... And the final pod status: <details</details
Elana identified the issue on status here - previously we NEVER reported container status after an eviction, so the refactor (which correctly ensured container exit codes showed up) was returning containers with state=waiting,lastState=terminated, which is incorrect. generateAPIStatus needs to know the difference between "current container status, waiting for more to show up" and "any status we have is the best we have". So once killPods succeeds in syncTerminatingPods and in syncTerminatedPods, we're no longer "waiting". So generating pod status should correctly summarize the container status as and not inject .


Question: ], deps = [ "//vendor/github.com/golang/glog:go_default_library", <ins> "//vendor/golang.org/x/net/http2:go_default_library", </ins> "//vendor/k8s.io/api/core/v1:go_default_library", "//vendor/k8s.io/apimachinery/pkg/api/errors:go_default_library", "//vendor/k8s.io/apimachinery/pkg/apis/meta/v1:go_default_library",

While this issue happens without this complicated reproduction, here is a sure-fire way to get it. a long-running kubectl get. In this example, I have a cluster with about 2,000 configmaps each around 100KiB. Once you see these lines Press , then type , and then you'll see... What's happening is that the apiserver is hitting , and then it . As the code says, this should make it obvious to the client (kubectl) that it didn't get the entire body, because no 0-length EOF chunk gets sent. I can confirm from using tcpdump that no such chunk gets sent. However, kubectl doesn't notice this, and instead tries to decode a 0-length Response body, giving the cryptic message. What I expected to happen the very least, a less cryptic error message like "connected closed". , this gets detected by the code and we can internally retry, just as if we got, e.g.,
cc Possibly seen here, too: openshift/origin
/assign mengqiy
Created a PR to detect it. Further improvement may be retrying when detect connection closure
closing this since you filed a PR for it


Question: */ package kuberuntime <del> // milliCPUToShares converts milliCPU to CPU shares func milliCPUToShares(milliCPU int64) int64 { return 0 } </del>

Deployment of a POD onto a machine with 488 cores fails if the POD gets deployed with QoS Guaranteed or Burstable when cpu request is larger than 256. Kubelet Error Trace: RunContainerError: failed to start container "4ad8bf2f988c9c387d8136ac664caf4af80648ac436e92dac6ecd1f9aece9144": Error response from daemon: OCI runtime create failed: starting container process caused " container init caused " setting cgroup config for procHooks process caused "The maximum allowed cpu-shares is """: unknown Normal start of the container with the right number of cpu-shares set yo the cgroup the POD runs in. Install kubernetes on a machine with more than 300 cores and deploy a POD with QoS Guaranteed or Burstable where the cpu-request value is set to more than 256 cpus. Deployment without CPU request and limits works. Deployments with cpu-request <= 256 works as well. I found this change: Unfortunately the changed code does not get used in all cases - is not used in my case. Kubernetes version (use ): Kubernetes v1.19.8 Cloud provider or hardware configuration: AWS / EC2 u- (6 TB memory / 488 vCPUS) OS (e.g: ): NAME="SLES" VERSION="15-SP1" VERSIONID="15.1" PRETTYNAME="SUSE Linux Enterprise Server 15 SP1" ID="sles" IDLIKE="suse" ANSICOLOR="0;32" CPENAME="cpe:/o:suse:sles:15:sp1" VARIANT_ID="chost" Kernel (e.g. ): Linux ip-10-250-0- 4.12.14-197.72-default Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig sig/node
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig sig/node Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
Hi thanks for raising the issue. Can you share the pod spec(maybe the part that includes CPU shares) that caused the issue?

Also which container runtime are you using? And its version?
I think might be related to the error message you're seeing. That repo as you can see is archived so I think you should try upgrading your container runtime. Also in my understanding, CPU share is a relative value that tells kernel scheduler how to schedule the cpu time slice to each task. In my understanding we should expect a large cpu quota here, rather than a large CPU share. Just thinking loud.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
This looks like a kernel limitation to me, not kubelet bug per se. Thanks for linking the bug above, I agree it's related. Kernel code where max limits are set: /triage accepted /remove-lifecycle stale It seems we are only clamping for stuff under , I don't know that this is being applied to all pods. /cc I think there's two things we should do: this limit more clearly the clamping to ensure we don't run into this error /kind documentation /help Help requested with improving the docs.
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : This looks like a kernel limitation to me, not kubelet bug per se. Thanks for linking the bug above, I agree it's related. /remove-lifecycle stale Unfortunately the changed code does not get used in all cases - is not used in my case. Update the clamping to ensure we don't run into this error /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I think the issue is in the Kubelet. The kernel uses the range for , but the kubelet can generate and request values outside of the valid range. When using cgroupfs, the kernel automatically clamps the value.

Instead systemd has an additional check and refuses to use an invalid value (even if the kernel would accept and clamp it). The clamping is just to make sure the systemd driver works in the same way as cgroupfs, but now any value set above the valid range will get assigned the same number of . would you mind opening a PR to add the clamping also for your case?
I would like to take this one by opening a PR for the clamping issue.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
have you had a chance to look at this issue?
I just created the PR but not sure if I it to the right place. Could you take a look?
/remove-lifecycle stale
/assign
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale


Question: controllerErr := <-controllerCompletedCh if controllerErr != nil { <del> t.Errorf("Expected no error from the controller, but got: %#v", controllerErr) </del> <ins> t.Errorf("Expected no error from the controller, but got: %s", fmtError(controllerErr)) </ins> } }) } <ins> func fmtError(err error) string { return fmt.Sprintf("%#+v=%q", err, err.Error()) } </ins> func startAPFController(t *testing.T, stopCh <-chan struct{}, apfConfiguration []runtime.Object, serverConcurrency int, requestWaitLimit time.Duration, plName string, plConcurrency int) (utilflowcontrol.Interface, <-chan error) { clientset := newClientset(t, apfConfiguration...)

pull-kubernetes-unit TestPriorityAndFairnessWithPanicRecoveryAndTimeoutFilter 7/14 No response /sig api-machinery
/priority important-soon
/cc /triage accepted
and can this be related to ?
I see no direct relationship, but hey, anything's possible. We learned that with the misery in integration test failures when we merged not-really-guilty other stuff.
The expanded logging turned up some new evidence. See , where the complaint is now
The problem here is not APF, it is something in the timeout filter or the test server.
is something in APF triggering a timeout?
The flake definitely came in with on the prior commit on master: on the merge commit of
/assign
Note that the flaking test is deliberately suffering server-side timeouts. The symptom in flakes is that the test is checking that the client gets kube's designed response to a timeout while handling, a particular form HTTP response message, but sometimes the client gets something else (an http2 internal error?). is a change in metrics, and should have nothing to do with how timeouts while handling and detected or handled. I will see what I can see...
, regarding your comment above (): note that the output shown (the panic "as designed") is normal; if there are failures involved, the failing part was elided.
Also, the failures seem to be coming from one particular subtest () and that one does not involve designed panics.
On closer examination of the timeout filter, it appears that this "INTERNAL ERROR" is something that can happen normally in the course of handling a timeout. So I will adjust the test to accept this.
I have updated to include the fix for this flake.


Question: var container string if len(args) == 1 { <ins> if len(pod.Spec.Containers) != 1 { return fmt.Errorf("POD %s has more than one container; please specify the container to print logs for", pod.ObjectMeta.Name) } container = pod.Spec.Containers[0].Name </ins> } else { container = args[1] }

On a cluster running 0.13.2, I ran , picked out the name of a pod that only contains a single container, let's call it foo, and then ran . It prints nothing other than the message "Unexpected path for command running". If I add on the container name (), then it prints nothing, presumably because the pod has no logs. The docs for kubectl log say that the container name is optional if the pod only has one container, so I wouldn't expect this response.
+1, we're having to work around this on guides


Question: Dial: e.Tunnel.Dial, // TODO(cjcullen): Plumb real TLS options through. TLSClientConfig: &tls.Config{InsecureSkipVerify: true}, <ins> // We don't reuse the clients, so disable the keep-alive to properly // close the connection. DisableKeepAlives: true, </ins> }) client := &http.Client{Transport: transport} resp, err := client.Get(l.healthCheckURL.String())

Forking from In kubernetes-soak-continuous-e2e-gke, the kubelet memory usage continued going up I checked one of the nodes , and there were over 4000 goroutines (and increasing) in kubelet. The same applies to the kubernetes-soak-continuous-e2e-gke-gci nodes. The version of kubelet on the node: Kubernetes v1.5.0-alpha.1.483+ Relevant goroutines: The cluster was deployed on 10/18. Not sure if this has already been fixed. /cc a couple of people who may have more ideas. /cc
Nope this is new.
Just for clarity, that flake was occurring since Aug 6th which was well before we enabled http2. I wonder if there is some issue inside of the client tooling. /cc
the test was flaking since Aug 6th because of some other reasons (docker leaking memory, a known issue). I didn't trace all the test failures, but I believe this is relatively recent.
Maybe this is the TLS deadlock? :) Joking.
Could you take a look at this one? mentioned that the issue only occur to gke cluster, not gce. I also quickly checked gke 1.4 soaking cluster, and didn't see the same issue. It looks like a new regression. Thanks!
This may be obvious, but: net/http.(http2serverConn).serve calls net/http.(http2serverConn).readFrames. So whatever is leaking goroutines is calling one of (from net/http): func (srv Server) Serve(l net.Listener) error func Serve(l net.Listener, handler Handler) error func (srv Server) ListenAndServe() func ListenAndServe(addr string, handler Handler) error func (srv *Server) ListenAndServeTLS(certFile, keyFile string) error func ListenAndServeTLS(addr, certFile, keyFile string, handler Handler) error
The other stack traces that call one of these functions originate at:
Are connections being left open, or only go routines? It wouldn't surprise me if this was related to the streaming functions (exec/attach/port-forward).
Looks like connections are being left open. We have around 2k goroutines that are in the middle of serve(), and around 2k open connections. They are all on localhost:.

Based on default ports, this is the Kubelet Server Port
Take a look at the API server. If the streaming connections are being left open, there might be a lot of go routines here:
This is in the API server logs:
And I see this one as well. Both errors seem to occur about every hour or so. logging error output: "Error: 'EOF'nTrying to reach: 'http://10.240.0.4:4194/containers/'"
""""" logging error output: "Error: 'EOF'nTrying to reach: ''"
there are a few lines in the api server log regarding kube- 21:35:38. 5 ] Error proxying data from client to backend: read tcp 10.240.0.56:443-104.197.45.156:: read: connection reset by peer
netstat output includes 2k+ lines like these: tcp6 0 0 localhost: localhost: ESTABLISHED tcp6 0 0 localhost: localhost: ESTABLISHED tcp6 0 0 localhost: localhost:xxxxx ESTABLISHED The port is the Kubelet's default port. Looking for any logs related to the other ports shows they are all used by healthz calls: sudo grep -r : 08:06:50. 3719 ] GET /healthz: (50.055µs) 200 [[curl/7.26.0] 127.0.0.1:] 17:01:46. 3719 ] GET /healthz: (53.715µs) 200 [[curl/7.26.0] 127.0.0.1:] sudo grep -r : 01:17:18. 3719 ] GET /healthz: (30.595µs) 200 [[curl/7.26.0] 127.0.0.1:] 19:28:03. 3719 ] GET /healthz: (41.365µs) 200 [[Go-http-client/2.0] 127.0.0.1:]
Snippet from the on a gke node: On a gce node: You can clearly see that there are only healthz checks (from a monitoring script) on GCE, while there are healthz requests from Go-http-client on GKE.

As mentioned in , there are a lot of open ports on the GKE cluster. All these ports are used by the Go-http-client healthz requests. These ports are used by ssh connections (which probably didn't close the http connections?). /cc
Thanks for looking into this. This is what I suspected since I know managed node team is working on node repairing logic here. could you take a look at this?
I think this may be caused by the ssh tunnel health check. Similar to kubelet, the apiserver has over 10k goroutines hanging. /cc


Question: * **Kubectl Command Line Interface** ([kubectl.md](kubectl.md)): The `kubectl` command line reference. <ins> * **Sharing Cluster Access** ([sharing-clusters.md](sharing-clusters.md)): How to share client credentials for a kubernetes cluster. </ins> * **Roadmap** ([roadmap.md](roadmap.md)): The set of supported use cases, features, docs, and patterns that are required before Kubernetes 1.0.

Hi, I can't find any documentation how I add more admin users so my team members can deploy pods.
Is there a functioning concept of different user types yet or is the answer here just passing around kubeconfig files?
For now, the answer is to pass around kubeconfig files. There are better answers coming sometime, but afaik, not yet. --brendan , Alex Robinson wrote:
Yep. AFAIK, the apiserver doesn't yet support multiple user credentials, so if you want to grant others access to a cluster, they'll need to share credentials (stored in kubeconfig file).
Ok, and we do have a doc on , but unless I'm mistaken, it appears to be more targeted at developers rather than users and doesn't talk about how to find the file to copy it between machines, or where to put it when you do so. And how is a new user supposed to find that documentation if they don't know that the file is called kubeconfig?
I'm happy to add a short doc describing how to share/copy kubeconfig files for cluster access. Does sound sufficiently discoverable?
Thanks for all the answers. A doc in is fine for me.


Question: master string // healthzPort is the port to be used by the healthz server. healthzPort int32 <ins> // metricsPort is the port to be used by the metrics server. metricsPort int32 </ins> scheme *runtime.Scheme codecs serializer.CodecFactory

What happened: From We used to allow using or in and . Now the validation on kubeproxyconfig disallow that (it only allows ): This seems to break backward compatibility for those two flags. Please comment if I'm missing something. cc How to reproduce it (as minimally and precisely as possible): Try with kube-proxy. Environment: Kubernetes version (use ):
/sig network /kind bug
Tried , the error is: and , the error is: So maybe we need to loosen and revert ?
Loosen the validation sounds good. was meant for loosen the validation as well (allowing not just ip, but also ip:port), but for some reason it doesn't work as intended. Would be great to fix that instead of revert :)
find the root cause of not working as expected. introduced the regression: I send a PR to fix the two issues we mentioned in this thread.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with .

/close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I believe this hasn't been fixed yet. /remove-lifecycle rotten /reopen
Reopened this issue. <detailsIn response to : I believe this hasn't been fixed yet. /remove-lifecycle rotten /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I agree There's also the issue of IPv6-only: Results in:
Thanks for your report. Do you mind opening a new issue regarding IPv6 since the original issue was fixed? Thanks.
Sure. Thanks
Please see:


Question: tc.runTestWithController(t, controller, informerFactory) } <ins> func TestConvertDesiredReplicasWithRules(t *testing.T) { conversionTestCases := []struct { currentReplicas                  int32 desiredReplicas                  int32 hpaMinReplicas                   int32 hpaMaxReplicas                   int32 expectedConvertedDesiredReplicas int32 expectedCondition                string annotation                       string }{ { currentReplicas:                  5, desiredReplicas:                  7, hpaMinReplicas:                   3, hpaMaxReplicas:                   8, expectedConvertedDesiredReplicas: 7, expectedCondition:                "DesiredWithinRange", annotation:                       "prenormalized desired replicas within range", }, { currentReplicas:                  3, desiredReplicas:                  1, hpaMinReplicas:                   2, hpaMaxReplicas:                   8, expectedConvertedDesiredReplicas: 2, expectedCondition:                "TooFewReplicas", annotation:                       "prenormalized desired replicas < minReplicas", }, { currentReplicas:                  1, desiredReplicas:                  0, hpaMinReplicas:                   0, hpaMaxReplicas:                   10, expectedConvertedDesiredReplicas: 1, expectedCondition:                "TooFewReplicas", annotation:                       "1 is minLimit because hpaMinReplicas < 1", }, { currentReplicas:                  20, desiredReplicas:                  1000, hpaMinReplicas:                   1, hpaMaxReplicas:                   10, expectedConvertedDesiredReplicas: 10, expectedCondition:                "TooManyReplicas", annotation:                       "maxReplicas is the limit because maxReplicas < scaleUpLimit", }, { currentReplicas:                  3, desiredReplicas:                  1000, hpaMinReplicas:                   1, hpaMaxReplicas:                   2000, expectedConvertedDesiredReplicas: calculateScaleUpLimit(3), expectedCondition:                "ScaleUpLimit", annotation:                       "scaleUpLimit is the limit because scaleUpLimit < maxReplicas", }, } for _, ctc := range conversionTestCases { actualConvertedDesiredReplicas, actualCondition, _ := convertDesiredReplicasWithRules( ctc.currentReplicas, ctc.desiredReplicas, ctc.hpaMinReplicas, ctc.hpaMaxReplicas, ) assert.Equal(t, ctc.expectedConvertedDesiredReplicas, actualConvertedDesiredReplicas, ctc.annotation) assert.Equal(t, ctc.expectedCondition, actualCondition, ctc.annotation) } } </ins> // TODO: add more tests

We recently had 2 serious bugs caused by non-trivial logic calculating desiredReplicas in hpa reconcileAutoscaler method (, ). I think we should extract this part of logic (starting around ) to a separate function and make sure it's covered with unittests for different possible edge condition. The logic itself takes a few ints as input (calculated desiredReplicas, min/max replicas, currentReplicas, ...) to produce a single integer value and set some status conditions, making it very easily unittestable if we extract it to a separate function. WDYT
+100. It would be super unfortunate if we had yet another issue in this area of the code.
agreed. you've been going over these pieces of code recently, you want to tackle this?
Yes, I'd love to! Thanks for the ping. I'll tackle this this weekend :)


Question: createRoute := ws.POST(path).To(h). Doc("create a " + kind). <del> Operation("create" + kind). Reads(versionedObject) // from the request </del> <ins> Operation("create" + kind) </ins> addParamIf(createRoute, namespaceParam, namespaceScope) if _, ok := storage.(RESTCreater); ok { ws.Route(createRoute.Reads(versionedObject)) // from the request

The v1beta{1/2}.Minion field appears to be duplicated in the method parameter section where ever it appears.
Repro: Use the steps outlined here to stand up a cluster on GCE: View the swagger docs at https://{kubernetes master}/swaggerapi/api/v1beta2
Thanks Ian for reporting! The issue seems to be due to this line: Reads(versionedObject) is twice and hence minion in beta1 and beta2 and Pod in beta3 is repeated twice in POST body params. Will send a PR to fix.


Question: InternalGroupKind: api.Kind("Service"), InternalOperationResources: map[admission.Operation][]api.ResourceName{ admission.Create: allResources, <ins> admission.Update: allResources, </ins> }, MatchedResourceNames: allResources, MatchesScopeFunc:     generic.MatchesNoScopeFunc,

the ability to get the previous version of an object directly via admission control. As a result, admission code can now use this rather than fetching the object directly. Need to update:
/cc - this is a necessary follow-up, will get to it soon.


Question: "time" "github.com/google/go-cmp/cmp" corev1 "k8s.io/api/core/v1" "k8s.io/apimachinery/pkg/api/resource" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"

I deployed a pod with and on a namesapace with a set with and elements and I get the error: The pod to be deployed Create a new namespace Create (using a manifest file named resource-quota-) and verify a named compute-resources Content of the resource-quota- file Create (using a manifest file named ) a pod named high-priority with the and for both and . Content of the file The pod is deployed correctly if I delete the <details</details<detailsaws </details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig Scheduling
/wg API Expression
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /wg API Expression Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/wg <API Expression
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /wg <API ExpressionInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/wg "API Expression"
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /wg "API Expression" Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/wg APIExpression
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /wg APIExpression Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.

</details
/wg wg-api-expression
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /wg wg-api-expression Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
hey, v1.18.20 is out of support now, you should try with a newer version. Besides, I tried with an old version v1.18.19, but worked as well.
Are u using some mutating webhook which inject some sidecar into the pod without setting resources fields?
It's probably that, but IDK which WebHook is doing that. The feature you suggested would really help.
hi the fix pr has been merged, maybe u could use it to locate which container is causing the problem.
Thanks


Question: func setupInProbeMode(t *testing.T, devs []*pluginapi.Device, callback monitorCallback, socketName string, pluginSocketName string) (Manager, <-chan interface{}, *Stub, pluginmanager.PluginManager) { m, updateChan := setupDeviceManager(t, devs, callback, socketName) <del> pm := setupPluginManager(t, pluginSocketName, m) </del> p := setupDevicePlugin(t, devs, pluginSocketName) <ins> pm := setupPluginManager(t, pluginSocketName, m) </ins> return m, updateChan, p, pm }

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: bazel-test-master Which test(s) are flaking: //pkg/kubelet/cm/devicemanager/godefaulttest:run2of_2 Testgrid link: Reason for failure: Anything else we need to know: links to appreciated Triage shows nothings (seems to not have updated data), but there are multiple failures in the job bazel-test-master. links to specific failures in spyglass appreciated <!-- Please see the deflaking doc () for more guidance! --
/sig release
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig sig-release Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Digging further as the step has another recent test fail ... So on the two runs reported by has 2 test failures has 1 test failure And the most recent failure in this area is a TestAdmit/match test failure but on a different test.
/sig node
/assign /triage accepted I'll take a look at them and fix.


Question: // DefaultDaemonSetUniqueLabelKey is the default label key that is added // to existing DaemonSet pods to distinguish between old and new // DaemonSet pods during DaemonSet template updates. <del> DefaultDaemonSetUniqueLabelKey string = "daemonset-controller-hash" </del> <ins> DefaultDaemonSetUniqueLabelKey = appsv1beta1.ControllerRevisionHashLabelKey </ins> ) // DaemonSetList is a collection of daemon sets.

Here are the hash label keys of all controllers: Deployment: DaemonSet: (new in 1.7) StatefulSet: (new in 1.7) Had a brief discussion with and about this, it should be consistent. We can use for DaemonSet and StatefulSet, and keep Deployment as-is. Deployment hash label key should be different from other two because Deployment pods hash labels is used by selector (unlike the other two controllers). We probably should have used hash annotations instead of hash labels for DaemonSet and StatefulSet, since we don't use hash labels to select their history or pods. Hash labels are chosen for consistency. This is fine since we don't modify hash label values.
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
Actually, controller revisions created by statefulset, daemonset has different label: // controller revision created by statefulset label: "" // controller revision created by daemonset label: controller-revision-hash: "" should be consistent?


Question: apierrors "k8s.io/apimachinery/pkg/api/errors" "k8s.io/apimachinery/pkg/api/meta" "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" <ins> "k8s.io/apimachinery/pkg/runtime/schema" </ins> utilerrors "k8s.io/apimachinery/pkg/util/errors" utilruntime "k8s.io/apimachinery/pkg/util/runtime" "k8s.io/apimachinery/pkg/util/sets"

What happened: Created a CRD: Created another CRD defining an overlapping type with CustomResourceDefinition: Verify both exist: Deleted the one overlapping with the in-tree CRD type: Verify only the overlapping CRD remains: and is stuck deleting: What you expected to happen: Creation and deletion of the overlapping type CRD would have no effect on the in-tree data. Anything else we need to know?: The only types in the kube-apiserver (currently) that store data under a group prefix are: APIService () CustomResourceDefinition () Custom resources () When a CRD is deleted, all instances of the type it defines are first removed. That means that defining and deleting a CustomResourceDefinition that defines an overlapping APIService or CustomResourceDefinition type will remove existing APIService or CustomResourceDefinition data. We can special-case these two resources in the CRD deletion code, but we need a more robust solution before CRD v1. /cc /sig api-machinery /priority important-soon /area custom-resources
/sub
Is anyone working on this? I'll like to take a stab at this.
Not currently, no. Great. I'd suggest discussion a potential solution here before diving into code. The two main aspects are: -time - Should we forbid creating CRDs that overlap with built-in types? -time - How do we detect a CRD overlapping with a built-in type and skip the etcd cleanup for it
I think we should forbid creating CRDs that overlap (option 1).
That can help, but is not sufficient. Imagine: n forbids creating overlapping CRDs on version n creates MyType, which is allowed because that is not a built-in type upgrades to version n+1 which adds MyType as a built-in type deletes CRD What should happen?
For that, we may need delete-time detection of overlapping types as well (option 2), which may make option 1 redundant. Do you have any advice for detecting overlapping CRDs?
for now, just delete-time detection in


Question: // Whether the volume permission is set to read-only or not // This value is passed from volume.spec ReadOnly bool <ins> // Inner volume spec name, which is the PV name if used, otherwise // it is the same as the outer volume spec name. InnerVolumeSpecName string </ins> } type VolumeMap map[string]VolumeInfo

Fix status: [x] fixed in master in [x] fixed in release-1.10 branch in , will be in 1.10.0 [x] fixed in release-1.9 branch in , will be in 1.9.7 [x] fixed in release-1.8 branch in , will be in 1.8.11 [x] fixed in release-1.7 branch in , will be in 1.7.16 In SubPath fixes, PR adds bind mount for subpaths. It uses as the path name. The volumename in this path is the container's mount volume name which is also called OuterVolumeSpecName in . This becomes an issue for volume reconstruction when kubelet restarts. Volume scans the volume directories and recover the InnerVolumeSpecName (which is used to construct pod volume paths). Volume reconstruction has not enough information to recover the OuterVolumeSpecName and figure out which volume it maps to. In case kubelet needs to clean up mounts (it happens when volume spec cannot be reconstructed), it will fail to clean up the subpaths because of lacking the OuterVolumeSpecName. We think it is better to use InnerVolumeSpecName for volume subPath instead of OuterVolumeSpecName so that volume clean up can work. The issue we might have is that some users might already deploy a cluster with subpath might already . If the users perform an in-place upgrade, since new version will use InnerVolumeSpecName which might have conflicts with some other older version's OuterVolumeSpecName among different volumes. This change requires clear documentation and communication with the users.
cc
This impacts reconstruction of PVCs, where the inner and outer names are different.
/assign
While testing the e2es against a cluster without the fix, I noticed that subsequent pods using volumes failed to start. My theory is that the unmount volume failed because the subpath was still mounted, and that blocked the volume manager from processing other pods. But I thought that volume manager is supposed to run the unmount volume operation in a separate thread per volume, so this shouldn't happen. Need to inspect the code more carefully to confirm the theory. could the volume reconstruction refactor have a unmount volume call in the main volume manager thread?
runs in reconciler's goroutine - so if that function is blocked, it will not be able to process any more volumes.

Was the problem that failed to unmount volume but got hung/stuck ?
I haven't confirmed for sure, but my guess is that unmount volume failed because the subpath was still mounted
moving into the milestone for discussion /milestone v1.10
/priority critical-urgent /status in-progress
/status approved-for-milestone per discussion in sig-release meeting today
[MILESTONENOTIFIER] Milestone Issue: Up-to-date for process <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
Re-opening to track fixes in release branches
keeping in the v1.10 milestone until merges
since this is now fixed in 1.10, I'm taking it out of the milestone so that the release team doesn't keep checking it.
Can this issue be closed now?


Question: "//pkg/api/v1:go_default_library", "//pkg/api/v1/helper:go_default_library", "//pkg/apis/apps/v1beta1:go_default_library", <ins> "//pkg/apis/autoscaling/v1:go_default_library", </ins> "//pkg/apis/batch/v1:go_default_library", "//pkg/apis/extensions/v1beta1:go_default_library", "//pkg/controller:go_default_library",

Currently the time it takes for HPA and CA to scale up after load increase is not measured at all, and this can lead to regressions that will be missed. This issue is about adding an e2e test that will prevent such regressions.
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list .
Filip, can you reopen? I cannot do this myself, and this issue is clearly not closed yet.
Thanks Wojtek!


Question: <ins> /* Copyright 2019 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package yaml import ( "fmt" "k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/runtime/schema" "sigs.k8s.io/yaml" ) // DefaultMetaFactory is a default factory for versioning objects in JSON or // YAML. The object in memory and in the default serialization will use the // "kind" and "apiVersion" fields. var DefaultMetaFactory = SimpleMetaFactory{} // SimpleMetaFactory provides default methods for retrieving the type and version of objects // that are identified with an "apiVersion" and "kind" fields in their JSON // serialization. It may be parameterized with the names of the fields in memory, or an // optional list of base structs to search for those fields in memory. type SimpleMetaFactory struct{} // Interpret will return the APIVersion and Kind of the JSON wire-format // encoding of an object, or an error. func (SimpleMetaFactory) Interpret(data []byte) (*schema.GroupVersionKind, error) { gvk := runtime.TypeMeta{} if err := yaml.Unmarshal(data, &gvk); err != nil { return nil, fmt.Errorf("could not interpret GroupVersionKind; unmarshal error: %v", err) } gv, err := schema.ParseGroupVersion(gvk.APIVersion) if err != nil { return nil, err } return &schema.GroupVersionKind{Group: gv.Group, Version: gv.Version, Kind: gvk.Kind}, nil } </ins>

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Add an external MetaFactory that allows for serializing multiple YAML (or JSON) files in kubeadm. The logic of marshalling YAML should be factored out from to . Why is this needed: This will allow processing component configs containing multiple documents per file. Additionally, an out-of-tree library would allow sharing this functionality with other components.
I will try work on this as carry-on from /sig cluster-lifecycle /cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: func (s *SchedulerServer) AddHyperkubeFlags(fs *pflag.FlagSet) { s.addCoreFlags(fs) <del> fs.StringVar(&s.kmPath, "km-path", s.kmPath, "Location of the km executable, may be a URI or an absolute file path.") </del> <ins> fs.StringVar(&s.kmPath, "km-path", s.kmPath, "Location of the km executable, may be a URI or an absolute file path; may be prefixed with 'file://' to specify the path to a pre-installed, agent-local km binary.") </ins> } // returns (downloadURI, basename(path))

I'd like to run local in agent host. The parameters is used to set executor, but it did not set 's arguments. The proposal is to provide an argument to set the local path of in agent host. cc
It seems we have a hidden parameter , I'll try it firstly.
any suggestion?
how did work out? if you ran into problems .. what were they?
Just have a try, it seems not work; I'm on sure whether configuration issue.
, just go through the code; the will only point to the binaries in master and Mesos agent will fetch it. It did not match this requirement. Do you have time on this? I'd like to work on this and let you review it :).
, would you help to review the patch?
Yes, just provided some feedback on your PR , Klaus Ma wrote:


Question: }, timeToPass: 1 * time.Minute, newNodeStatus: v1.NodeStatus{ <del> NodeInfo: v1.NodeSystemInfo{ KubeletVersion: "v1.2.0", }, </del> Conditions: []v1.NodeCondition{ { Type:   v1.NodeReady,

As discussed on slack with we might want to clean the node controller from code that ensures backward compatibility for version 1.4 and earlier ones. /sig node
/assign
- how do we plan to devide this one ? The following order for merge should be fine I think: the tests (we can split also test case deletion if it's too long to review): logic itself + kubeletVersion on the according tests: change how we create nodecontroller: clean consts and BUILD:
Yeah, sounds good.
I'll create the PRs accordingly and assign them to you


Question: Master(clusterName string) (string, error) } <del> func GetLoadBalancerName(clusterName, serviceNamespace, serviceName string) string { return clusterName + "-" + serviceNamespace + "-" + serviceName </del> <ins> // TODO(#6812): Use a shorter name that's less likely to be longer than cloud // providers' name length limits. func GetLoadBalancerName(service *api.Service) string { //GCE requires that the name of a load balancer starts with a lower case letter. ret := "a" + string(service.UID) ret = strings.Replace(ret, "-", "", -1) //AWS requires that the name of a load balancer is shorter than 32 bytes. if len(ret) > 32 { ret = ret[:32] } return ret </ins> } // TCPLoadBalancer is an abstract, pluggable interface for TCP load balancers.

Since this PR got merged, the ELB e2e test, which was consistently succeeding, is now consistently failing: Revert now that is in, and fix the tests that were relying on it. (details) Commit by arob Revert now that is in, and fix the tests that were relying on it. The file was modified The file was modified The file was modified I'm going to roll it back to confirm.


Question: "strings" "testing" <del> apps "k8s.io/api/apps/v1" </del> batchv1 "k8s.io/api/batch/v1" batchv1beta1 "k8s.io/api/batch/v1beta1" corev1 "k8s.io/api/core/v1"

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: of Job created by using command will be . This apiVersion is wrong. What you expected to happen: Set the to How to reproduce it (as minimally and precisely as possible): CronJob manifest: <details</detailsApply CronJob and execute command Anything else we need to know?: Environment: Kubernetes version (use ):
/sig cli
This was introduced by . /assign
GitHub didn't allow me to assign the following users: odino. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : This was introduced by . Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I created fix pull request:


Question: type InsecureSuperuser struct{} func (InsecureSuperuser) AuthenticateRequest(req *http.Request) (*authenticator.Response, bool, error) { <ins> auds, _ := request.AudiencesFrom(req.Context()) </ins> return &authenticator.Response{ User: &user.DefaultInfo{ Name:   "system:unsecured", Groups: []string{user.SystemPrivilegedGroup, user.AllAuthenticated}, }, <ins> Audiences: auds, </ins> }, true, nil }

Semantics are going to vary. We need to cross off these authenticators: [x] if ctx has an audience limitation, verify the token audience and echo in the response [x] take implicit audiences at WebhookTokenAuthenticator construction time if ctx is audience-limited, add into the tokenreview object if the tokenreview returns with an audience set that matches the ctx, copy into the response and return success if the tokenreview returns without an audience set, ensure the ctx audiences intersect with the implicit audiences taken at WebhookTokenAuthenticator construction time, and pick one to add into the response [x] echo audience in response if present in ctx [x] echo audience in response if present in ctx [x] check that the the order of (audience in the context) ∩ ( (implicit api audiences) ∪ (clientids) ) 0 Take implicit audiences at Authenticator construction time, and if ctx is audience-limited, ensure it intersects with the implicit audiences, and echo an audience in the response. [x] [x] [x] [x] [x] [x] Do nothing? [x] [x] [x] [x] [x] [x] [x] note: we already keyed the cache with the audience. [x]
There are no sig labels on this issue. Please add a sig label by either: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. See the . The in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/close
Closing this issue. <detailsIn response to : /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: Then you can create the service by running: <del> ```shell cluster/cloudcfg.sh -c redis-slave-controller.json create /replicationControllers </del> <ins> ```js $ cluster/cloudcfg.sh -c examples/guestbook/redis-slave-controller.json create /replicationControllers Using master: kubernetes-master (external IP: 1.2.3.4) { "id": "redisSlaveController", "desiredState": { "replicas": 2, "replicasInSet": { "name": "redisslave" }, "podTemplate": { "desiredState": { "manifest": { "version": "", "volumes": null, "containers": [ { "image": "brendanburns/redis-slave", "ports": [ { "hostPort": 6380, "containerPort": 6379 } ] } ] } }, "labels": { "name": "redisslave" } } }, "labels": { "name": "redisslave" } } </ins> ``` The redis slave configures itself by looking for the Kubernetes service environment variables in the container environment.  In particular, the redis slave is started with the following command:

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: I am running an admission webhook using If there it has been hours since the last mutating request to the webhook, then the next mutating request will take many seconds to be processed. This may even take long enough to have the request time out. What you expected to happen: A request takes approximately the same amount of time to be processed regardless of how long it has been since the last request. How to reproduce it (as minimally and precisely as possible): Create an API server for handling a new API group. Wait a couple hours. Send a CREATE request to the API server. Observe that the CREATE request takes a few seconds to complete. Anything else we need to know?: This is the offending code. It must loop through once for every second since the last request. Environment: Kubernetes version (use ): 1.19.0
/sig api-machinery
I thought we'd already merged a fix for this from before the release?
I assume that the fix was supposed to be From my understanding, those changes only are in effect if the apiserver is running with priority and fairness enabled. In my situation, I do not have that enabled.
Ack, if we have the problem in both places we need the fix in both places :)
/triage accepted
I think we have fixed this in , do you guys agree?


Question: }) } } <ins> func Test_RoundUpToB(t *testing.T) { testcases := []struct { name        string resource    resource.Quantity roundedVal  int64 expectError bool }{ { name:       "round m to B", resource:   resource.MustParse("987m"), roundedVal: int64(1), }, { name:       "round decimal to B", resource:   resource.MustParse("1.2Gi"), roundedVal: int64(1288490189), }, { name:       "round big value to B", resource:   resource.MustParse("8191Pi"), roundedVal: int64(9222246136947933184), }, { name:        "round quantity to B that would lead to an int64 overflow", resource:    resource.MustParse("8192Pi"), roundedVal:  int64(0), expectError: true, }, } for _, test := range testcases { t.Run(test.name, func(t *testing.T) { val, err := RoundUpToB(test.resource) if !test.expectError && err != nil { t.Errorf("expected no error got: %v", err) } if test.expectError && err == nil { t.Errorf("expected error but got nothing") } if val != test.roundedVal { t.Logf("actual rounded value: %d", val) t.Logf("expected rounded value: %d", test.roundedVal) t.Error("unexpected rounded value") } }) } } </ins>

After upgrading our clusters to Kubernetes 1.19, the provisioning of PersistentVolumes started failing with the following Events: The value matches the actual storage request of 3.2Gi. It doesn't seem special in any way, not too big, not too small, and it always worked until 1.19. We look into the issue with and we believe the PR introduced a change that broke the PersistentVolume provisioning even when it doesn’t overflow int64. Amounts of resource in Kubernetes are represented with the . In the PersistentVolume drivers, there are some functions that manipulate storage request in order to allocate a disk with the right size. Before the mentioned PR, those quantities were retrieved by calling which relies on to return the value as an int64. This worked probably fine in most of the cases, but it could happen that the represented value would overflow an int64, and in this case the result of would be wrong. The function’s description states that it’s the caller’s responsibility to check this: This is likely what the PR wanted to handle. The change replaced all calls with which has a different signature. From where it’s used now, you would think that the second return value signals an overflow: : This is probably where the error we’re seeing is generated. I believe that the signature of was misinterpreted. From my understanding of its description, returning doesn’t mean we’re overflowing . It only means that internally, the quantity is represented as a decimal amount: Internally, depending on their values, quantities are stored/represented in : for DecimalExponent (e.g. ) or DecimalSI (e.g. ) for BinarySI that don’t have a decimal part but a unit (simplified) (e.g. ) for all the rest With the PR, Kubernetes now fails to provision volumes that have a storage request represented as . Many of the storage requests are represented this way and still perfectly valid in our opinion. I would have expected the provisioning to continue to work as those requests seemed valid and were always accepted by our underlying cloud-provider in earlier version of Kubernetes.

Create a StatefulSet with a PersistentVolumeClaim that uses vSphere storage: Kubernetes version (use ): 1.19.7 Cloud provider or hardware configuration: vSphere, but I think it impacts other cloud-provider that were changed in the PR: /sig storage /kind bug /cc Possibly related to
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I think the only reason we do not use the quantity library for scaling and why we tried to roll our own rounding code is because when scaling stuff in AsScaledInt64(3) or ScaledValue(3), the library assumes decimal SI (M, 1000), not binary SI units (Mi, 1024): You are right, our self-rolled rounding code doesn't handle values with millibyte or higher precision like 3.2Gi. Our code uses AsInt64 and then performs its own scaling. If AsInt64 returns false, the int64 returned is not in bytes (it must be in millibytes or higher precision), so our rounding code won't work, so it returns that " is too great, overflows int64". In 1.18, we used Value() to round to bytes but we didn't check it against ScaleValue as the signature suggested. So the PR in question fixed cases where Value() overflowed but inadvertently broke your case. The correct way to round, which the resource library's rounding does, is to drop the useless precision and round up. , but as I said it doesn't seem to handle binary SI quantities correctly. For decimal SI quantities, we should rely on the resource library's AsScaledInt64 and be done with it. (We could also use Value() and ScaledValue as the signature suggests but calling AsScaledInt64 directly seems simpler). Like this: For binary SI quantities, we need to do something fancier involving CanonicalizeBytes I think, I'm not sure yet.
I guess for binary SI we can go back to using Value but actually check the overflow this time.

Hi Thanks for the quick feedback on the problem's description already. Indeed, the Scaled() functions don't handle BinarySI and that's why I quickly discarded the idea of using them. Since this is about storage, I suppose most people are using this format. I just finished wrapping up a PR that I hope address both the issue we are facing, but also what was meant to be fixed in the PR your reviewed back then. It doesn't make any difference between the 3 formats however.
Yeah I commented within seconds of your PR. I glanced at it , will review more soon, but it makes sense to me to just use VAlue and check ofr overflow for both decimal SI and binary SI rather than using Scaled() for one and not the other :+1:
I think it means both. i.e - when incoming value is decimal or if value is so big that type can't store it as it switches to decimal. So while my PR fixed later case, it broke the former. :/


Question: treated as "reconcile class addons" for now. - Resources under `$ADDON_PATH` need to have either one of these two labels. Otherwise it will be omitted. <del> - The above label and namespace rule does not stand for `/opt/namespace.yaml` and resources under `/etc/kubernetes/admission-controls/`. addon-manager will attempt to create them regardless during startup. </del> #### How to release

What happened: Add-on manager uses kubectl apply to create the kube-system namespace at start-up. This is redundant because the kube-apiserver creates the namespace in NewBootstrapController. This is a problem because if someone uses apply to make changes to the kube-system namespace, the add-on manager will wipe out these changes. What you expected to happen: Kube-APIServer has the sole responsibility of creating the kube-system namespace. How to reproduce it (as minimally and precisely as possible): kubectl apply a label to the kube-system namespace. restart the add-on manager. Note the labels have been removed. Anything else we need to know?: /sig api-machinery Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Others:
/sig api-machinery
the contention script wipes out namespace ^^
/cc
i guess it is intentional to have the addon manager to ensure kube-system is not modified, it tries to avoid the case that some cluster admin accidentally delete kube-system that result in a non-functional cluster.
The add-on manager will only perform the check when it restarts. The API Server will also perform this check when it restarts. I think we should only have 1 system responsible for this and I believe that system should be the API server.
trying to have my first contribute here, so is it proper to have the kube-system existence checked before applying the yaml file to create the kube-system namespace once again?
The Kube API Server will check for the existence of the kube-system namespace and create it if absent. It does all of this using client-go requests. The Add-On manager is the only system creating the kube-system by applying (a private) yaml file. I believe the correct fix here is to remove the Add-On managers attempts to create the kube-system namespace entirely. This is especially important if a user writes a CI/CD controller to apply their desired state to the kube-system namespace. I would argue the CI/CD is correct but the add-on manager will randomly wipe out the users changes. The API Server on the other hand will not.
, thanks for your help, so i will remove that line and pull a request later
Trying to get my first PR here. Is the issue still open ?
The issue is still open. You may want to check on though as was trying to fix the issue with that PR.


Question: w.Write(LEVEL_0, "Namespace:t%sn", pvc.Namespace) w.Write(LEVEL_0, "StorageClass:t%sn", storageutil.GetPersistentVolumeClaimClass(pvc)) if pvc.ObjectMeta.DeletionTimestamp != nil { <del> w.Write(LEVEL_0, "Status:tTerminating (lasts %s)n", translateTimestampUntil(*pvc.ObjectMeta.DeletionTimestamp)) </del> <ins> w.Write(LEVEL_0, "Status:tTerminating (lasts %s)n", translateTimestampSince(*pvc.ObjectMeta.DeletionTimestamp)) </ins> } else { w.Write(LEVEL_0, "Status:t%vn", pvc.Status.Phase) }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Kubectl describe pv xxx shows last<invalidif pv is terminating ! I found pvc and pod have the same issue What you expected to happen: Terminating (lasts 10s) How to reproduce it (as minimally and precisely as possible): 1 create a pv, pvc or pod is terminating 2 using kubectk describe xxx Anything else we need to know?: Environment: Kubernetes version (use ): 1.13.3 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Others:
/sig cli
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign


Question: ginkgo.BeforeEach(func(ctx context.Context) { setupStorageClass(ctx, config, &waitMode) <ins> ginkgo.DeferCleanup(cleanupStorageClass, config) </ins> for i, node := range config.nodes { ginkgo.By(fmt.Sprintf("Setting up %d local volumes on node %q", volsPerNode, node.Name)) allLocalVolumes[node.Name] = setupLocalVolumes(ctx, config, volType, &config.nodes[i], volsPerNode)

master-informing gce-cos-master-serial Kubernetes e2e suite.[It] [sig-storage] PersistentVolumes-local Stress with local volumes [Serial] should be able to process many pods and reuse local volumes 2023-02-15 15:12:38 +0000 UTC { failed [FAILED] persistentvolumes "local-pvt2rz2" not found In [AfterEach] at: test/e2e/storage/persistent_volumes- @ 02/21/23 18:15:38.461 } Failed multiple times: /sig storage
I think the AfterEach is racing with the test's AfterEach cleanup: Test's PV recycling go routine: We're supposed to before proceeding with cleanup. do you know if this is still a valid assumption?
/triage accepted
The recent Ginkgo changes didn't change anything regarding Go routines. If a test relies on a Go routine to have stopped, it needs to wait for that explicitly. Note that the created for an node automatically gets cancelled when the callback returns. However, that does not necessarily mean that Go routines terminate immediately.
x-post: This has been analyzed as a test issue and not a product issue, so it shouldn't block tomorrow's v1.27.0-alpha.3 release
Just bumping that this test with this same error as OP is still a common flake as of today. <img width="2768" alt="Screenshot 2023-03-20 at 3 22 23 PM" src="https://user-"
FYI, these are still flaking. Here are some failures over the last few days: !
/help
This request has been marked as needing help from a contributor. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command.

<detailsIn response to : /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
! Just bumping that this is still flaky and failed thrice in a row recently.
The test seems to add a DeferCleanup to stop the goroutine mentioned in From the docs Is guaranteed the order and the DeferCleanup will always run before the existing AfterEach block?
I believe the order is , then (first-in-first-out), (first-in-last-out).
yes is correct. for this test that goroutine certainly adds some complexity that makes things a bit harder to reason about. assuming it's necessary and/or if you'd prefer to avoid making too many changes i'd recommend breaking the up into some s I'd be happy to submit a PR that does just that.
then the order is wrong, because the goroutine should be stopped before AfterEach, that may explain the races please be my guest, tag me and/or patrick for review
Certainly we don't need to fix our tests for us?
I know, I assumed he wanted to do it just for fun or something ... is the problem of text, you can't read between lines :)
lol! i'm happy to do it and contribute.
LOL Thanks a ton for all that you do !
PR is here: - I'm not familiar with all the k8s backslash verbs and nouns but let me know if you need me to do anything on the PR to help move it along. (and/or teach me how this works by telling me what I should do next)


Question: package mount import ( <ins> "os" </ins> "path/filepath" "strings"

What happened: Windows documentation claims: V1.VolumeMount.mountPropagation - only MountPropagationHostToContainer is available. However, the internals suggest otherwise; is always returned on Windows: Either the docs should be corrected, or the Kubelet should be updated, or we should have a comment on why this does in fact enable MountPropagationHostToContainer on Windows. /cc /sig windows
/cc
/cc
I think the doc is not accurate. The doc claims the same thing. For MountPropagationHostToContainer option, we can test it whether it is working or not.
i created two PRs to address the documentation gap. we will also test this and get back to this thread.
For 1.15 we may need to update Kubelet code to allow HostToContainer.