Question: noScheduleTaints      = []v1.Taint{{Key: "dedicated", Value: "user1", Effect: "NoSchedule"}} ) <ins> var ( nodeNotReady = []v1.Taint{{ Key:       metav1.TaintNodeNotReady, Effect:    v1.TaintEffectNoExecute, TimeAdded: metav1.Now(), }} nodeUnreachable = []v1.Taint{{ Key:       metav1.TaintNodeUnreachable, Effect:    v1.TaintEffectNoExecute, TimeAdded: metav1.Now(), }} ) </ins> func getKey(ds *extensions.DaemonSet, t *testing.T) string { if key, err := controller.KeyFunc(ds); err != nil { t.Errorf("Unexpected error getting key for ds %v: %v", ds.Name, err)

When writing DaemonSets we decided that they shouldn't be deleted by NodeController in case of Node problems. Now that we're moving to using taints and taint-based evictions (NC will just taint Nodes and the rest will be done by generic taint-eviction mechanism) we need to add infinite toleration for those taints to DaemonSets to keep this behavior. cc
Off the top of my head, I think the best way to implement this is to modify the , to special-case DaemonSet pods and add infinite tolerationSeconds to them. (See definition of field in the API for how to do this.) do you think you can do this? It should be a simple change. cc/ BTW the feature Marek described will be alpha in 1.6, but nobody will be able to use it (without breaking DaemonSet) unless this request is implemented. So the 1.6 milestone tag is appropriate.
Why wouldn't the daemon set controller special case that?
Yeah, I guess that's probably better - have the DaemonSet controller create pods with the toleration.
I can do this, and yes, in the DaemonSet controller.


Question: return deviceName, diskUUID, nil } <ins> func getNextUnitNumber(devices object.VirtualDeviceList, c types.BaseVirtualController) (int32, error) { // get next available SCSI controller unit number var takenUnitNumbers [SCSIDeviceSlots]bool takenUnitNumbers[SCSIReservedSlot] = true key := c.GetVirtualController().Key for _, device := range devices { d := device.GetVirtualDevice() if d.ControllerKey == key { if d.UnitNumber != nil { takenUnitNumbers[*d.UnitNumber] = true } } } for unitNumber, takenUnitNumber := range takenUnitNumbers { if !takenUnitNumber { return int32(unitNumber), nil } } return -1, fmt.Errorf("SCSI Controller with key=%d does not have any avaiable slots (LUN).", key) } </ins> func getSCSIController(vmDevices object.VirtualDeviceList, scsiType string) *types.VirtualController { // get virtual scsi controller of passed argument type for _, device := range vmDevices {

vSphere cloud provider volume plugin attach disk limits the number of volumes because of two issues: new SCSI controller is only created when one does not exist. A new controller should be when required and limit not reached. disk will fail when attaching disk to slot 7 on controller. Slot 7 on SCSI controller is reserved and therefore should be skipped. This issue has been brought up in Govmomi library .


Question: volumes: - name: registration-dir hostPath: <del> path: /var/lib/kubelet/plugins/ </del> <ins> path: /var/lib/kubelet/plugins_registry/ </ins> type: Directory - name: kubelet-dir hostPath:

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug This is more of an enhancement. What happened: The Plugin Watcher is setup to trigger on file activities when they occur in the subdirectory that it scans. During startup, the Watcher is passed (from within the Kubelet) the Kubelet's root plugin directory as a location to scan for newly plugins - Because the kubelet's plugins subdirectory is also used by existing in-tree drivers (including csi) as a location to store plugins artifacts, the Plugin Watcher is getting triggered by directory/file activities occurring in that subdirectory that has nothing to do with plugin registration as shown in the output below: What you expected to happen: The Plugin Watcher should not be getting triggered by file/directory activities not related to plugin registration/unregistration. One possibility is to have the Watcher scan one directory below plugins like or something similar How to reproduce it (as minimally and precisely as possible): Run plugin watcher with a CSI driver. cc
/sig node
I think you should only place the plugin socket file in the plugin registration directory but not the plugin data.
agree with we can only place the registration file in the specified path instead of placing all data into it. On the other hand, you can place data into the /var/lib/kubelet/plugins/registration even if registration created, that's a no-end loop.
As I mentioned in issue, existing in-tree volume plugins were already using that directory as a location to store storage operation artifacts (not just csi). Here is a shortened list of driver locations that use /var/lib/kubelet/plugins in their code already: And several more locations. The fix would be to have plugin watcher scan a subdirectory of plugins (say ) instead of plugins directly as it is doing today. The watcher functions as intended. Except now it picks up all directory activities whether related or not to plugin registration.
Well given the fact that the pluginwatcher is beta this is a bit of an issue. My understanding of the "beta" guarantees is that we want to be backwards compatible for all plugins published with the beta API. Meaning that even if we move the registration directory to a new location, we'll need to keep scanning the old directory (to pick up v1beta plugins).

From a 10 000 feet point of view it seems easier to me if we move the in-tree volume plugin's data. Though it might be a lot of work. Another less painful option, but a bit grey is the following: Enforce plugins to end with this suffix is recommended for device plugins (though not enforced) and I believe is a convention on linux. This is a bit grey because we would still be breaking v1beta users but since we're moving a convention from "recommended" to "enforced" it seems "less" breaking to me (and I haven't seen device plugins not using that extension). What do you think ?
I like the suffix idea. It can be a recommendation (which csi is already doing).
Hmm, I've just deployed a new k8s cluster on 1.12.0 and noticed this repeating in the logs on a few of our pods. Just for my own sanity, is this something I should be worried about or can I ignore it until a fix is released?
you can ignore these errors. if it's a recommendation then we'll keep spamming the logs with all the data files. Another thing to keep in mind, it seems like CSI plugins puts a lot of data (and creates a lot of directories). It might be worth considering limiting the recursion to one level. I'm concerned we might exhaust the amount of fds by tracking the plugins data.
yeah CSI keeps track of volume operation metadata in the plugins dir. This probably will not change. I understand this is beta. However, besides CSI and Device plugin, are there any other implementers out there for this feature ? Would it be possible to propose a new location (i.e. <pluginsDir/reg/?
let me know if you open a PR for this as discussed.
thanks for open the issue with detail logs, i can see two problems by looking at logs and code. 1) plugin watcher trying to dial non-socket files. 2) too many logs generated (CREATE and DELETE). I a suggestion to fix of this issue we can fix those problems by: 1) check file mode when handle CREATE event, only attempt to dial file when file mode is socket.

2) increase log verbose level for both 'handleDeleteEvent()' and 'handleCreateEvent()' plugin watcher still has activity when any file create/delete under the directory, but they become light weighted (CREATE file will only trigger file mode check, DELETE file event will trigger plugin searching) i am not sure we want to go further to limit directory depth where plugin can live. it has two drawbacks IMO: 1) it is hard to enforce, not sure everyone will read code/readme. 2) it provides less flexibility to plugins: if want to organize many plugin socket files with directory.
These are definitely good suggestions! It seems like there might be some confusion as to why we are thinking of doing this: Most plugins use the plugin directory to store data (actually a lot of files / dirs) Watching all the directories and files created inside the plugin directory puts more burden on the Kubelet in terms of cpu / memory consumption Watching all the directories and files created inside the plugin directory quickly raises the number of fd consumed By limiting the search of the plugins to one directory, we would be severely limiting the ability of the module to unintentionally harm the overall Kubelet. It seems actually pretty easy to enforce, as it's a simple condition. As for not everyone reading code / readme, we can add the documentation to the gRPC interface if this is a concern. In terms of flexibility, is there a real need to create a plugin in a lower directory? Is there a plugin that has so many sockets that a single directory underneath the plugin directory isn't enough? Naively it seems to me that is enough for the different plugins we enable.
ping on the above :)
Naively it seems to me that $ is enough for the different plugins we enable. i can not speak from plugin developer, but both below looks reasonable to me , or as long as we can communicate to plugin developers clearly and effectively (gRPC interface documentation is needed), the directory depth restriction is reasonable to me.
I like . However, why not put plugintype one level up as . For instance
Is the suggestion to hard code the number of levels that the recursion algorithm will search to 3? Or just expect a socket only at a hard coded level (e.g. socket must be nested 3 deep)? Copying my comment from for consistency: Another option I can think of: recursively scanning but before scanning a directory check if it is mounted ().

This will be backwards compatible and will prevent scanning of volume data Thoughts?
i like this idea better if it is always the case that: volume data is mounted, we can modify the watcher code to not watch the mounted directory in the first place.
Skipping mounted dir along with restructuring search path to known (shallow) dir sounds like a good plan. Though with the new suggested paths, sockets will be placed at a location where mounts should not exist.
I think that would be a safe assumption. The plugin directory is only used as a target path for global mounts.
Ephemeral volume types may not need to mount, it could just be a symlink or plain directory. Can we assume any ephemeral volume CSI driver won't implement global mount?
Yes, in-tree ephemeral volumes today don't do a global mount so the mounts directory is which is not the plugin dir. So no issue there. For CSI the will be the global mount path (this plugin dir). Emphermal volumes shouldn't implement call -- but nothing prevents them from doing so. Maybe let's start with just checking for mount and ignoring mount directories. If it is still an issue we can consider adding a configurable black list of paths. Thoughts?
That sounds good. Also, putting the socket files in a specific location that does not clash with mounts will also help (i.e. )
/milestone v1.13 /priority important-soon
I just want to point out that one of the issue we have with recursively watching all directories is that it will significantly increase the amount of fd consumed. Given that plugins actually store data there and in a dynamic way my fear is that it will exhaust the available fds for no real good reasons. Limiting the depth sounds like a reasonable solution to that issue.
btw, because of this verbosity in my cluster writes 5G logs every day and filled up my rootfs (which is small). Is there any way to decrease this verbosity?
We can do both. Don't search mounted dirs, and max depth of search. Who is working on this?
That's pretty bad, can you open another issue to track that.
we have a PR in flight discussing this issue (recursive scanning) here: Thinking more about this, the fact that too much logs is generated is clearly linked to this specific issue and how we process plugins.

In any case I'll be fixing that issues too
how confident are we in getting this fixed by Code freeze tomorrow? 5pm PST
I expect things to wind up today, changes in the above PR aren't very contentious and don't involve significant changes


Question: const ( // reconcilerLoopSleepDuration is the amount of time the reconciler loop // waits between successive executions <del> reconcilerLoopSleepDuration time.Duration = 1 * time.Nanosecond </del> <ins> reconcilerLoopSleepDuration = 1 * time.Nanosecond </ins> // waitForAttachTimeout is the maximum amount of time a // operationexecutor.Mount call will wait for a volume to be attached. <del> waitForAttachTimeout         time.Duration     = 1 * time.Second nodeName                     k8stypes.NodeName = k8stypes.NodeName("mynodename") kubeletPodsDir               string            = "fake-dir" testOperationBackOffDuration time.Duration     = 100 * time.Millisecond reconcilerSyncWaitDuration   time.Duration     = 10 * time.Second </del> <ins> waitForAttachTimeout         = 1 * time.Second nodeName                     = k8stypes.NodeName("mynodename") kubeletPodsDir               = "fake-dir" testOperationBackOffDuration = 100 * time.Millisecond reconcilerSyncWaitDuration   = 10 * time.Second </ins> ) func hasAddedPods() bool { return true }

Which jobs are flaking: Which test(s) are flaking: periodic-kubernetes-bazel-test-master (xref ) Testgrid link: Reason for failure: Anything else we need to know: to reproduce with : /sig storage
/assign
solved at
still flaking - /reopen
Reopened this issue. <detailsIn response to : still flaking - Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
-- No recent failures in the past 2 weeks, are you okay if we close this?
yeah, looks like resolved it /close
Closing this issue. <detailsIn response to : yeah, looks like resolved it Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
it's back not sure if related to this issue but same test with a data race spotted last night looking into filed


Question: zones := sets.NewString() for _, instance := range instances { <del> // Only return fully-ready instances when listing instances // (vs a query by name, where we will return it if we find it) if orEmpty(instance.State.Name) == "pending" { glog.V(2).Infof("Skipping EC2 instance (pending): %s", *instance.InstanceId) </del> <ins> // We skip over master nodes, if the installation tool labels them with one of the well-known master labels // This avoids creating a volume in a zone where only the master is running - e.g. #34583 // This is a short-term workaround until the scheduler takes care of zone selection master := false for _, tag := range instance.Tags { tagKey := aws.StringValue(tag.Key) if awsTagNameMasterRoles.Has(tagKey) { master = true } } if master { glog.V(4).Infof("Ignoring master instance %q in zone discovery", aws.StringValue(instance.InstanceId)) </ins> continue }

Kubernetes version (use ): Environment: Cloud provider or hardware configuration: Install tools: default install What happened: AWS EBS backed PersitantVolumeClaim created a volume in an AZ where there was a master but no nodes. Making it unusable. What you expected to happen: Zones with master and no nodes shouldn't be used to create the EBS Volume How to reproduce it (as minimally and precisely as possible): AWS setup across 3 AZs in one region Master in three AZs Nodes in 2 AZs (so one AZ has a Master but no Nodes) Create a Storage class to use EBS without specifying AZCreate PVC against StorageClassPVC can create a volume in an availability zone with no nodes Anything else do we need to know: Mentioned on Slack channel #sig-aws with who asked me to raise this
, Mengqi Yu wrote:
Running Kubernetes master in a zone that has no running nodes is quite strange and I don't think this is a typical case we should support. AWS cloud provider lists all machines in the cluster and chooses a random zone that has at least one running instance. It seems it does not distinguish master/node machines. IMO, GCE has the same problem, it lists all zones in a region where the master runs and it does not distinguish masters and nodes either. We can either introduce a new AWS instance tag with master/node role that is running on the instance or we can move the zone decision into volume plugin itself, where we have a kubeclient and we can filter nodes relatively easily, however we will depend on an admission controller that must add a zone label to nodes. thoughts? As a third option, we could rely on kubernetes admin to fill all zones into StorageClass in this weird case. Right now we support only one zone in "zone" parameter, we could allow a list:
just to add to this, the reason why it's a deal is I actually don't have full control over where the nodes run, as they're in an autoscale group across all AZs. So you could have master in AZ a and nodes in b and c just because that's what the auto scale group picked.
I just ran into this, however, I only have nodes / masters in 3 zones, but the PV was created in a 4th zone.

So right now I can't even spin up a node in the last AZ since in my infrastructure, I'm not supporting it.
I think it may be more than just where there are or not nodes vs masters in an AZ. As an additional use case, consider pods that must run on a "dedicated" tenancy node in AWS, (which have a baseline cost per-zone) for compliance. Since masters are supposed to be an odd number (i.e. 3 zones) it would make sense to have 3 instead of 2. But for a pod that has to be scheduled on a "dedicated" tenancy node, there may be only 2 zones with nodes that are "dedicated" due to said cost. Even if there are nodes in all zones, a pod may need to be scheduled on a specific node or subset thereof. More information about node selection: More information about why this specific case would be necessary here: If PVs are used by pods, and pods must be able to access them from whatever nodes they are constrained/scheduled to, I think whatever solution is implemented needs to account for that.
do you have any other non-Kubernetes instances in us-east-1b in your AWS account? Did you try to tag your AWS instances that run Kubernetes with tag - all AWS instances in one Kubernetes cluster must share the same tag value, so you can run multiple Kubernetes clusters in one AWS account. This tag is not documented, see .
There is another non-k8s instance running in us-east-1b, but it's not related (created via breanstalk). I labels to all my nodes / controller as you sugessted and still it created the PV in . I may bail on Statefulset for now and manage my pv's by hand since I'm building out an operator and it wont be that painful, but would like to see if this could be addressed. I'm also willing to help out if you can help point into the right places this logic lives. I'll also try and dig in and see if I can find it myself.
It would be ideal to have the StorageClass accept an expression or wildcard zone under the aws provisioner. Ideally I shouldn't have to care where my pods are running and where the EBS are created.
We've a PR that adds "zones:" with list of zones instead of single "zone:". No wildcards though.

I just hit this as well. I think moving the "zone" decision closer to the provisioning decision would be best. Perhaps wait until a PVC is consumed by a pod before you create it in a zone. That way corner cases like this are not a problem.
Just want to check whether PR is good enough to solve the issue raised here?
, is right. Another use case is that if pod consumes 1 PVC, associated PVs might be created in different zones, therefore pod becomes unschedulable, as there is no node which belongs 1 AZ. Zone decision when provisioning PV must be made as late as possible, while currently it seems to be a very first thing to do.
I don't think it can be closed yet, reasons are: those magical tags are not documented anywhere, how people or going to know to use them? can't it continue to create volume in a zone where pod with a given PVC can never be scheduled (lets say it has node affinity configured to a node in a particular AZ)?
Would it make sense to say that the dynamic creation of a volume should happen only once a PVC has been attached to a pod which has been successfully scheduled to a node? Then the av_zone of the volume should match that of the selected node?
Yes I would agree with that statement. The PVC should be created when we know the node and AZ it is a apart of.
Just to add another use case to this - I just spun up a new cluster in AWS. Masters spanning , , and . After creating the ASG for nodes, I learned that AWS is out of instance capacity in for that particular instance type. Now I currently don't have nodes in (until capacity is restored and I can move an instance over). The reason this is different is because, in this case, I don't want to tell k8s to only use a subset of AZs. I want it to deploy across all AZs that are available. It just so happens that at this moment in time not all AZs are available. suggestion above would work, I believe. Schedule the node first, then provision a volume. That might cause other problems, though.
I just ran into this today - PVC created in eu-west-1c, but the only node with memory to to run the pod was in eu-west-1b.

This was a new pod and a new volume claim. Can we have this re-opened at least please?
We had to ultimately get rid of all persistence volume claims on AWS because of this. For really large clusters with lots of reserved capacity in all availability zones that woulnd't be such a big deal, but I'd image most companies are trying to optimize their cost and don't have that many node pre-provisioned just for the sake of it. Still looking for a solution which is easy to configure and AZ-failure friendly
Right now, there isn't really. You have a few options: EFS - it is regional, not zonal. You have to be OK working with NFS as the protocol, and potential performance implications. To be fair, NFS has gotten quote good, I know companies using it to store critical data. I am not an NFS expert, so don't ask my opinions here... Commercial solutions - Portworx and StorageOS both provide local storage, backed by local disk or EBS, but replicated at the app level (their app) between nodes, across zones, so you don't depend on zonal resources (like EBS). DIY - have fun. Ceph, some other solutions. For one use case with which I am involved, we are living with it, and using EFS where necessary. We don't love it, but the resources to implement something else aren't there now.
Would in a StorageClass go some way to fixing the issue? (At least for new claims anyway)? From Edit: I tried this, and the kube-controller-manager logs say: but the pod is FailedScheduling "2 node(s) didn't find available persistent volumes to bind.".
I solved this in the Elastic Operator by creating StorageClasses per zone and spreading work across that way. You need to make sure your app is zone tolerant. For example, the Elastic operator sets up sharding and replication across zones, given one AZ goes down, the other two still have enough of the data to continue functioning.
doesn't each pod then need access to storage from a different class (appropriate for the zone it is in)? How does that work?
Just ran into this as well, running work load specific zones in 2 regions for development to save cost over 3, had an EBS volume provisioned in the zone with no instance.

should address the above scenarios when you use set in the storageclass. The plumbing for went into EBS in 1.12 as a beta feature.
I have a similar issue: 3 masters + 3 nodes in 3 regions. I have a PV (created on regionA) for data backups created by a CronJob. When the cron's job gets scheduled on a node on a different region (regionA's CPU has been allocated almost fully and the master chooses another node in a different region) - it can't start because the PV is not accessible. Is there a way to avoid this issue?
I am also facing the same issue with 1.10 version of Kubernetes. Nodes are running in two regions, PV got created in the same region. still, I am getting
As mentioned above, this feature is now in beta in 1.12:
Are there any workaround for Amazon EKS to address the above issue? I am trying to make a multi-az JupyterHub deployment and changing zones causes the "no available volume zone" error...
Is this still in progress? I just ran in to issue as well
we moved it to GCP with 1.10 version and didn't face the issue
I'm also having this issue... So there's no solution to this?
im also having this issue on AWS EKS Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedScheduling 2m (x841 over 1h) default-scheduler 0/2 nodes are available: 1 Insufficient pods, 1 node(s) had no available volume zone. losing hope over here :(
Warning FailedScheduling 3m2s (x660 over 58m) default-scheduler 0/2 nodes are available: 1 Insufficient pods, 1 node(s) had no available volume zone.
Another similar case here, like 0/2 nodes are available: 1 Insufficient pods, 1 node(s) had no available volume zone. Like the use case by but the reason is not memory is pods allocation. The EBS was created in the AZ where the node is full (1c).. you can guess the rest..
Any update on this ? Since Node affinity is a beta feature causing this issue?

Edit: (Beware: Updating your EKS cluster can potentially screw things up & cause nodes and volumes in different zones)
I'm using EKS to deploy my services integrating it with rancher, I ran into the same issue as mentioned above. My setup is something like this: have few labels for the worked nodes to seperate the nodes per namespace/team as each team requires different types of instances. created the nodes for the pods in that node needs persistant volumes in a single availability zone as aws EBS attachment is per availability zone. I deployed my applications, the persistant volume got created but in different availability zone as I used the default storage class which creates EBS in random zones. The issue is fixed by creating another storage class with only single availbility zone.
I am using EKS , and I designed not only my pods to be Cattle, but also my nodes are Cattle as well : I have a Pod created by statefulset. This statefulset defines a PVC. The PVC triggers the creation of new EBS in the same AZ of the Node (i don't have pb here) Because Nodes are Cattle (destoryable any time), it happen that there is no Node exist in the AZ of EBS (PVC) =Pod of statefulset remains in "Pending" state =Pod events shows this error: " 1 node(s) had no available volume zone" Once a New node joins the cluster in the wanted AZ, the Pod of statefulset runs like a charm because the EBS can be attached at that node (same AZ). Thinking to use EFS ( )but looking for better solutions.
from my experience, EFS performance is not that great. better avoid it.
It depends how you use it. It's not great for lots of little files (don't check out a git repo on it, for instance ). It's been working great for us in production for almost a year so far, mainly on larger PDF files etc. But, like anything, definitely test it for your use case.
So I do have a question. If I just put the PVC into the specific availability. zone. Does that allow the other pods on different nodes to use it?

The nodes running on other availability zones couldn't access this PV until if you use EFS rather than EBS
I ran into a similar issue today investigating a error I have a PVC: and originally it bound fine, with a PV in us-east-1d Then, I ran an upgrade and none of the new nodes were created in us-east-1d And, now I have a disk in a zone without any nodes.
I found the issue. The issue is that when you are using AWS then the PVC is set to the Node. That means that if you try to put that pod or deployment on another node then it will not connect with the PVCs and being a pending state. It's an AWS thing. I resolve this by specifying taints and toleration to the nodes within that specific region.
thank you! can you post an example for more elaboration


Question: }) } <del> // newHealthzAndMetricsHandler creates a healthz server from the config, and will also </del> <ins> // newHealthEndpointsAndMetricsHandler creates an API health server from the config, and will also </ins> // embed the metrics handler. <del> func newHealthzAndMetricsHandler(config *kubeschedulerconfig.KubeSchedulerConfiguration, informers informers.SharedInformerFactory, isLeader func() bool, checks ...healthz.HealthChecker) http.Handler { </del> <ins> // TODO: healthz check is deprecated, please use livez and readyz instead. Will be removed in the future. func newHealthEndpointsAndMetricsHandler(config *kubeschedulerconfig.KubeSchedulerConfiguration, informers informers.SharedInformerFactory, isLeader func() bool, healthzChecks, readyzChecks []healthz.HealthChecker) http.Handler { </ins> pathRecorderMux := mux.NewPathRecorderMux("kube-scheduler") <del> healthz.InstallHandler(pathRecorderMux, checks...) </del> <ins> healthz.InstallHandler(pathRecorderMux, healthzChecks...) healthz.InstallLivezHandler(pathRecorderMux) healthz.InstallReadyzHandler(pathRecorderMux, readyzChecks...) </ins> installMetricHandler(pathRecorderMux, informers, isLeader) slis.SLIMetricsWithReset{}.Install(pathRecorderMux)

Quoted from 's Splitting healthz/ into livez/ and readyz/
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc
/sig scheduling
I read all the content of the PR comment and am willing to work on this issue Do we need to continue some discussion for this issue
Hi , thanks for offering your help. But I was working in progress before knowing you can work on this. However, please feel free to contribute the similar issue
/assign
The Kubernetes project currently lacks enough contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale


Question: bs, err := gce.GetRegionBackendService(bsName, gce.region) assert.NoError(t, err) <del> // Check that the instances are updated newNodes, err := createAndInsertNodes(gce, newNodeNames, vals.ZoneName) newIgLinks, err := gce.ensureInternalInstanceGroups(igName, newNodes) backends := backendsFromGroupLinks(newIgLinks) </del> <ins> // Check that the Backends reflect the new InstanceGroups backends := backendsFromGroupLinks(newIGLinks) </ins> assert.Equal(t, bs.Backends, backends) }) }

Seeing several of the tests in ./pkg/cloudprovider/providers/gce flake 1 in 5-10 runs when run locally (I am logged in to gcloud). Is it possible one of the APIs is not correctly shimmed and is fetching real data?
you were the last folks in here - why would this and other internal_tests in here be flaking?
Happens in both master and 1.11 release branches
/cc cheftako


Question: software-properties-common  lsb-release <ins> # focal repo for docker is not yet available, so we use bonic for now # https://github.com/kubernetes/kubernetes/issues/90709 release=$(lsb_release -cs) if [ "$release" == "focal" ]; then release="bionic"; fi </ins> # Add the Docker apt-repository (as we install containerd from there) curl -fsSL https://download.docker.com/linux/$(. /etc/os-release; echo "$ID")/gpg  | apt-key add - add-apt-repository  "deb [arch=amd64] https://download.docker.com/linux/$(. /etc/os-release; echo "$ID")  <del> $(lsb_release -cs) stable" </del> <ins> $release stable" </ins> # Install containerd from Docker repo apt-get update &&

Which jobs are failing: Which test(s) are failing: Since when has it been failing: Testgrid link: Reason for failure: Anything else we need to know: /cc /milestone v1.19 /priority critical-urgent /sig testing
You must be a member of the GitHub team to set the milestone. If you believe you should be able to issue the /milestone command, please contact your and have them propose you as an additional delegate for this responsibility. <detailsIn response to : Which jobs are failing: Up W0501 09:18:42.774] scp: /var/log/konnectivity- No such file or directory W0501 09:18:42.775] scp: No such file or directory W0501 09:18:42.775] scp: No such file or directory W0501 09:18:42.775] scp: *: No such file or directory W0501 09:18:42.780] ERROR: () [/usr/bin/scp] exited with return code [1]. /milestone v1.19 /priority critical-urgent /sig testing Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/milestone v1.19
I am seeing the same error listed above on successful runs as well so I think the issue might be elsewhere
/assign

Which jobs are failing: Which test(s) are failing: Since when has it been failing: Testgrid link: Reason for failure: Anything else we need to know: /cc /milestone v1.19 /priority critical-urgent /sig testing
You must be a member of the GitHub team to set the milestone. If you believe you should be able to issue the /milestone command, please contact your and have them propose you as an additional delegate for this responsibility. <detailsIn response to : Which jobs are failing: Up W0501 09:18:22.072] scp: /var/log/konnectivity- No such file or directory W0501 09:18:22.072] scp: No such file or directory W0501 09:18:22.072] scp: No such file or directory W0501 09:18:22.073] scp: *: No such file or directory W0501 09:18:22.075] ERROR: () [/usr/bin/scp] exited with return code [1]. /milestone v1.19 /priority critical-urgent /sig testing Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/milestone v1.19
/assign


Question: nodeExpansionRequired   bool disableAttach           bool disableResizingOnDriver bool <ins> simulatedCSIDriverError expansionStatus </ins> expectFailure           bool }{ { <del> name:                  "should expand volume without restarting pod if nodeExpansion=off", nodeExpansionRequired: false, </del> <ins> name:                    "should expand volume without restarting pod if nodeExpansion=off", nodeExpansionRequired:   false, simulatedCSIDriverError: expansionSuccess, }, { name:                    "should expand volume by restarting pod if attach=on, nodeExpansion=on", nodeExpansionRequired:   true, simulatedCSIDriverError: expansionSuccess, </ins> }, { <del> name:                  "should expand volume by restarting pod if attach=on, nodeExpansion=on", nodeExpansionRequired: true, </del> <ins> name:                    "should not have staging_path missing in node expand volume pod if attach=on, nodeExpansion=on", nodeExpansionRequired:   true, simulatedCSIDriverError: expansionFailedMissingStagingPath, </ins> }, { <del> name:                  "should expand volume by restarting pod if attach=off, nodeExpansion=on", disableAttach:         true, nodeExpansionRequired: true, </del> <ins> name:                    "should expand volume by restarting pod if attach=off, nodeExpansion=on", disableAttach:           true, nodeExpansionRequired:   true, simulatedCSIDriverError: expansionSuccess, </ins> }, { name:                    "should not expand volume if resizingOnDriver=off, resizingOnSC=on", disableResizingOnDriver: true, expectFailure:           true, <ins> simulatedCSIDriverError: expansionSuccess, </ins> }, } for _, t := range tests {

When a pod is quickly deleted and re-created on a node, it may not be necessary to unstage the volume and in which case only node-publish is called for the volume. When this happens - stagepath won't be set for RPC call because volume was not staged even though, volume does have capability. We should try and fix this behaviour so that - nodestagepath is always set in RPC call. We should try and fix this behaviour so that - nodestagepath is always set in RPC call. Quickly delete and recreate a pod No response master Any No response_ <details</details<details</details<details</details
/sig storage /triage accepted


Question: return rt.Response, rt.Err } <ins> func BenchmarkDiskCache(b *testing.B) { cacheDir, err := ioutil.TempDir("", "cache-rt") if err != nil { b.Fatal(err) } defer os.RemoveAll(cacheDir) d := diskv.New(diskv.Options{ PathPerm: os.FileMode(0750), FilePerm: os.FileMode(0660), BasePath: cacheDir, TempDir:  filepath.Join(cacheDir, ".diskv-temp"), }) k := "localhost:8080/apis/batch/v1.json" v, err := ioutil.ReadFile("../../testdata/apis/batch/v1.json") if err != nil { b.Fatal(err) } c := sumDiskCache{disk: d} for n := 0; n < b.N; n++ { c.Set(k, v) c.Get(k) c.Delete(k) } } </ins> func TestCacheRoundTripper(t *testing.T) { rt := &testRoundTripper{} cacheDir, err := ioutil.TempDir("", "cache-rt")

We use a lot of CRDs (frequently more than 1,000) in the project. This has historically caused a bunch of problems with client-side API discovery, among other things. At the time of writing the latest releases of kubectl use a discovery client rate limiter with a generous QPS and burst (50 qps, 300 qps burst), so client-side rate limiting doesn't become a problem until the API server is serving more than 300 API groups. Despite rate-limiting no longer coming into effect, we noticed that discovery was very slow - taking 10-20 seconds to complete. After having a few folks at run a little test (see and following comments) we noticed that discovery specifically seems slow on MacOS - it's dramatically faster on Linux. Here's discovery happening on Linux with 1,577 CRDs across 343 API groups in around a second: ` Here's the same test run from MacOS against the exact same EKS cluster, taking 12+ seconds. I expected kubectl discovery on MacOS to be as fast as it was on Linux. an EKS cluster. ~1,500 CRDs across ~350 API groups. the discovery cache in how long the first command you run against the EKS cluster takes to complete Refer to and subsequent comments for more examples of folks reproducing this. <detailsNote that the version of kubectl was the same on Mac and Linux for my tests. </details<detailsAWS - EKS </details<detailsMacOS 12.4, M1 Max Linux as below </details<detailsN/A </details<detailsN/A </details<detailsN/A </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cli
will show you how many milliseconds was the GET request taken. There are 37 GET requests after was removed. Is there any difference?
I'll add more details ASAP, but we're confident the slow-down is not in the network (i.e. the many GET requests).

Initial profiling indicates it's something to do with writing what is fetched to the discovery cache on disk.
Watch out for anything where writes trigger : there was an issue on macOS where the only way for an app to be confident that a write had completed was to trigger a whole-filesystem sync. Libraries that like data integrity opted for the slower-but-effective approach. I've stopped using macOS so this hint is from a while back, but it might relevant.
Thanks for the tip The issue definitely seems to be in the . When I apply the diff below to switch over to a the slowness disappears and MacOS performance matches that of Linux - about a second for to perform discovery:
It seems to be a problem that would occur only once when establishing the . Should we add it to known issues? Some explain about fsync in MacOs in
Hi, I have no strong opinion here.
You're right, and this happens every 6 hours in the latest versions of kubectl when the cache is invalidated. It's still not an ideal user experience though - you can imagine kubectl 'hanging' for 10-20 seconds the first time you used it each day.
Also note that this should fix a lot of that problem:
has anyone been able to profile kubectl to see what it's doing?
Also note that your CPU is mostly doing nothing, so it's most likely stuck on IO
I think is correct - my bet is we're blocking on , which per uses on MacOS. Per : A mutex profile shows a lot of time waiting for inside , which you can see sets the argument to . I believe this means we're doing an for every HTTP response body we encounter while doing discovery per .
I've confirmed the slowdown is coming from the . Using the following diff: I see MacOS is down to around one second, similar to Linux:
How do folks feel about not explicitly calling for each file in the HTTP cache? Given that it is indeed only a cache I lean toward optimizing for write performance rather than data safety. If we did want to stop calling it seems like the best option would be to update the package of that we use today to make whether it calls with configurable. I'm skeptical we could get this change merged upstream though - the latest change to was three years ago. Another option might be to switch the the package.

I just took a brief shot at this, but it seems like it would require some refactoring to the to handle calling on the underlying database.
I don't care a ton if we drop a cache file coherently. I care a lot if a write fails in a way that leaves a partially written file in a way that pollutes the cache and makes the user have to delete the file from the cache manually. I don't know which of those we're opening ourselves up to by writing without sync. Can we find out?
Presuming we do find that not fsyncing can result in leaving the cache in a broken state (as opposed to a partial state) would falling back to invaliding the cache and recreating it be acceptable?
Little more insight at and Notably the second link mentions:
I dug into this a bit - dumping state here before I sign off for the day. First, note that the discovery client uses two layers of cache - a "discovery cache" and an "HTTP cache". The former appears to be backed by the latter. My -fu is not strong enough to figure out why we do this. The best I could find is , which refers to "restoring the old cached client behavior" and appears to (re?)introduce the caching HTTP transport. Perhaps or know why this is? Only the HTTP cache is explicitly calling . The discovery cache just writes the files and calls , so it's already subject to the risks of not using . Per that risk is that despite the write having been flushed from the OS to the disk controller (e.g. by successfully calling ): My understanding of the and code that powers the HTTP cache is: If a cache file read returns an error we just act as if the value isn't in cache. If a cache file read returns garbled data that can't parse as an http response would fall back to its underlying (i.e. make a call to the API server) and will happily cache over the top of an existing (potentially garbled) file. The caches will be invalidated after six hours. So the failure mode that wouldn't be covered if we stopped calling is that if power were cut to the drive before its controller could sync down to persistent storage and_ this resulted in a file that was garbled just right such that it appeared to be a valid HTTP response but contained an invalid body (i.e.

not real discovery data) someone would have to either manually remove their cache, or wait six hours for the cache to be automatically invalidated. It seems like we could probably make the discovery client force an uncached request through the caching HTTP transport if it encountered what appeared to be cached, invalid discovery data, but I do wonder how far we really need to go to ensure that this cache is never corrupted.
For either of these caches, there are techniques (checksums and the like) where we can allow the app to see whether or not it is looking at a coherent cache document or something that might not be correct. Whenever a cache entry doesn't look valid, the code can asset that the date is current and the entry checksum is OK. If the entry turns out to be invalid or stale, remove it and then act as if we got a read error. That should cover any kind of unsynchronised write failure, I think. For locking the cache we also don't need to ensure sync-to-disk, so long as all running processes see a consistent view of locking.
Agreed. My question is whether we need to add such techniques to this particular cache (or caches) before we'll feel comfortable removing the fsync. Removing the fsync from the HTTP cache would presumably bring it back down to the same level of guarantee as the discovery cache. My guess would be that the fact we were ever explicitly fsyncing each cache value in the HTTP cache was a happy accident rather than something that was explicitly a desirable quality for the HTTP cache.
It seemed like writing our own httpcache cache backend was the best way to go here, so I went ahead and one that uses checksums rather than syncs per the above PR.


Question: namespace: kube-system labels: k8s-app: elasticsearch-logging <del> version: v7.4.2 </del> <ins> version: v7.4.3 </ins> addonmanager.kubernetes.io/mode: Reconcile spec: serviceName: elasticsearch-logging

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: es-image as part of fluent-elasticsearch cluster addon does not build What you expected to happen: in the es-image folder should work How to reproduce it (as minimally and precisely as possible): cd cluster/addons/fluentd-elasticsearch/es-image make Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/assign /triage accepted
/sig instrumentation


Question: claimsToBind:     []*v1.PersistentVolumeClaim{}, podVolumesByNode: map[string]*scheduling.PodVolumes{}, }, <del> wantFilterStatus: framework.NewStatus(framework.Error, `could not find v1.PersistentVolume "pv-a"`), </del> <ins> wantFilterStatus: framework.NewStatus(framework.UnschedulableAndUnresolvable, `pvc(s) bound to non-existent pv(s)`), </ins> }, }

What would you like to be : Scheduler VolumeBinding plugin reports error when a PVC doesn't exist for example: We need to take a second look at the VolumeBinding plugin and make sure that errors are returned only when there is a system error, not a user error. Why is this needed: To fix the scheduler's error reporting metrics. /sig scheduling /sig storage /cc
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign I'll take a look.
If the PVC object does not exist, the plugin will return status. I guess you mean the plugin will return an error when a referenced PV does not exist? I've created a PR for this:
Thanks! The errors I am seeing are on 1.18 clusters, can we look at 1.18 and try to patch? This is still throwing the scheduler's error metrics off.
sure
I've submitted a patch for the 1.18 release branch:
I guess on the master branch should be reviewed and merged too? cc
hello! the fix you made only apply for 1.18? how about 1.19 and 1.17? thanks!
after the fix on the master branch () is merged, I'll cherry-pick it into the 1.19 branch (the code are almost the same). I'm not sure if it's necessary to fix the 1.17 branch. It's 3 versions behind 1.20.
hi, This is because of this patch: Do you think this fix needs to be cherry-picked into 1.19 and 1.18 branch too?
can you send a patch for 1.19?
ah, here it is
to sum up: master: 1.19: 1.18: and I think this change should be backported to 1.19/1.18 too: I'll send PRs soon.

summary of the issue that an error will be reported instead of UnschedulableAndUnresolvable when PVC not found: master: already implemented, see 1.19: 1.18: can't backport as it does not implement the PreFilter extension point


Question: select all resources in the namespace of the specified resource types .PP <ins> fB--dry-runfP=false If true, only print the object that would be sent, without sending it. .PP </ins> fB-ffP, fB--filenamefP=[] Filename, directory, or URL to a file identifying the resource to update the labels

kubectl should support a --dry-run option for every mutation to show what it would do, without actually doing it We might need API changes in order to make this high-fidelity. For instance, if we wanted to know what a pod config would look like after admission control populated defaults, we'd need to execute most of the creation path.
For generator types commands, I think there is a usability argument that the default when a user specifies should be to dry-run (and a user should be allowed to override that with ). I think I'd rather encourage composition of commands (pipe) than every generator command in the system creating objects. We may also want to clarify dry-run against the server vs dry run against the client - there are lots of places where we want to generate locally to a file or an editor or a pipe command before we want to create it. At this point, OpenShift has a few commands that treat -o  as "dry-run" - new-app (complex generator) and export (never expect to send it to the server). We also have a few that are designed to be compositional in a pipe always - process (take a template and turn it into a list of api objects), which never creates the objects for you (you have to pipe to ). I'm somewhat partial to having and support the presence of changing the default of to true, because that's more common and easier than having to set and . The counter argument is that is the preferred action, and is format only - except that by definition --dry-run using the default output format isn't very useful (you created a service named foo and an rc named bar). My gut is that most users once they understand what a particular generator does their default path is always going to be to review the change, and then if they want to create it they can pipe to or just by appending to their existing command. I'm not sure that's a strong enough argument to couple --dry-run and -o, but based on our experience so far it's going to be the "normal" path.
:+1: for this feature. In particular, I would like an ability to validate an object definition separate from actually creating the object. e.g. running at the validation step of my deployment pipeline.
I agree that the validation use case is important. I would distinguish --dry-run from --local.

The former would still entail contacting the server and returning the result of the mutation. The latter would imply that only local information were used.
- I'm planning to add a parameter like 'pretty' or 'labelselector' to API requests. Once the server get this parameter, it will do validation to the resource and check if request can pass admission controls, but return success before write to etcd. I've read the documented about API changing, it lists some principles when changing the api object, but parameters are not mentioned. Is there anything need to be taken special care of when adding a parameter?
We need to be careful about dry-run with admission control... some admission controllers (like quota, specifically), record things to etcd in the process of admitting an object. Also, dry-run wouldn't work with a list of objects intended to be created in order, where later objects reference earlier objects (and those references are validated at create time)
We could potentially carry "dry run" down into the admission controller request path. However, a query parameter may not be appropriate, instead there's probably a HTTP header that already covers this. We try not to mix query parameters and post bodies in many cases. , Jordan Liggitt wrote:
- Thanks for advices! I'm trying to make myself thoroughly familar with request handling process. And during this period, I may need more of your help. I think I've replied this message before but unfortunately it's not showed up here:-( - Can you provide a link to the code about the HTTP header? Thanks!
This feels like a variant of an header with a sub parameter - possibly or possibly content-type . , Victor HU wrote:
+1 for this. My use case is to understand the difference between a stack file and an existing cluster, but this command would be a substitute.
It looks like support for has been . I think this issue can be closed.
We still need to make touch the server, or at least decide that we don't want to.
you mean move server side?
has just released kubeval: From to sig-apps mailing list: I am not sure if it validates the values and other constraints checked on the server-side. Just leaving it here as it might be relevant to the feature discussion. (h/t briangrant for pointing this out).
Issues go stale after 90d of inactivity.

Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
/remove-lifecycle rotten
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
a HTTP header that already covers this. We try not to mix query parameters and post bodies in many cases. Do you remember why you said that? I'm trying to understand ...
should fix this issue, we can close it when the server-side dry-run feature reaches beta. The feature above () will fix this use-case, and even validate against the admission chain. is being built on top of server-side dry-run, and will be a good way to visualize the output of dry-run apply:
This is now beta.


Question: wg.Wait() } <del> func TestApfCancelWaitRequest(t *testing.T) { epmetrics.Register() server := newApfServerWithSingleRequest(t, decisionCancelWait) defer server.Close() if err := expectHTTPGet(fmt.Sprintf("%s/api/v1/namespaces/default", server.URL), http.StatusTooManyRequests); err != nil { t.Error(err) } checkForExpectedMetrics(t, []string{ "apiserver_current_inflight_requests", "apiserver_request_terminations_total", "apiserver_dropped_requests_total", }) } </del> type fakeFilterRequestDigest struct { *fakeApfFilter requestDigestGot *utilflowcontrol.RequestDigest

As visible in this graph: ! pod-startup time in 5k-node tests increased around beginning of June (only 99% though). It seems to be connected to "run-to-watch" phase. We should investigate and see what we can do with this. /sig scalability /kind bug
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : As visible in this graph: ! /kind bugs Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage accepted Once PR that merged around that time is P&F support for watch-initialitation: As a first step, we may want to confirm/reject if it's this PR and see if we aren't somehow misconfigured here.
I've briefly checked test-infra/perf-tests to see if we had any relevant changes around that time, but I don't see any. Most likely it's related to k/k changes.
/assign
Good news is that we can also see it regression in kubemark (although not as bad as in regular 5k tests): http://perf- But this can help us with finding root cause faster/cheaper. ! I've picked two runs, one before this change, one after: 26th May: Timeline: 12th June: Timeline: First run vs second run I can see significantly higher spike on waiting time in P&F, but according to timeline it seems like it was during huge service and not pod startup latency: 26th May: ! 12th June: ! I can see that number of apiserver goroutines significantly increased (from ~500k to ~900k). I would expect to have 1 more goroutine per watch, we have around 200k watches). ! vs ! Worst 100 pods with slowest latency runtowatch: I've picked the slowest pod "test-2xppkn-33/latency-deployment-60--p7nwz" to analyse.

This pod was bound to node gce-scale-cluster-minion-group-4-txww: Kubelet log for this pod: CL2 logs for this pod: In apiserver logs we can see: So it looks like updating pod status was delayed by P&F by ~500 ms for this pod. Timeline for this pod: 18:37:46. scheduled 18:37:48. container started 18:37:49. kubelet gets pod (1.3s latency with 610 ms delay from P&F) 18:37:50. patch status of pod (545ms latency with 500ms delay from P&F) 18:37:51. observed by CL2 as running To compare with previous run we have this timeline for pod "test-rcdjgd-26/latency-deployment-69--6s95x 2.724031741s": 18:57:08. scheduled on node gce-scale-cluster-minion-group-3-l21s 18:57:10. container started 18:57:10. kubelet gets pod (364 ms latency with 345 ms delay from P&F) 18:57:11. patched status od pod (360 ms latency with 254 ms delay from P&F) So to compare, for these two pods major difference was "get pod" + "patch status" in kubelet: 1.8s vs 700ms. 1s of 1.8s of latency was due to P&F and 0.6s of 0.7s was due to P&F.
In both runs I don't see significant waiting time (99th percentile) on any of Priority levels during pod startup latency phase: 26th May: ! 12th June: !
1) Do we have data about 100% waiting time in P&F? With 99% it my not be enough to to see that data. 2) If really waiting time is small, then the question is what exactly. (a) it might be related to the number of goroutines (b) it might be locking (c) it might be something else... I have just opened the WIP PR: that should help with first two (at least to some extent). Will you be able to test if that helps?

Max waiting time in P&F for pod startup latency phase: 26th May: ! 12th June: ! Buckets are ..., 0.2, 0.5, ... so I don't think it's significant. will test it.
This is definitely non-negligible regression that we should address in 1.22 timeframe. I'm actually have a promissing results for it. /milestone v1.22 /priority critical-urgent
OK - so we proved that addresses the problem: it reduces back to previous value the number of goroutines ! it brings back pod startup time back to less than 3.4s on 99th percentile it brings back API call latencies I'm going to work with reviewers on merging this to fix the regression. - FYI


Question: // ControlPlaneComponent holds settings common to control plane component of the cluster type ControlPlaneComponent struct { // ExtraArgs is an extra set of flags to pass to the control plane component. <ins> // A key in this map is the flag name as it appears on the // command line except without leading dash(es). </ins> // TODO: This is temporary and ideally we would like to switch all components to // use ComponentConfig + ConfigMaps. ExtraArgs map[string]string

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --I was reading the doc about NodeRegistration, with an interest in specifying some . I noticed that the doc (e.g., ) does not tell me exactly how to identify each arg: exactly as on the command line (including leading dash(es)), as on the command line but without leading dash(es), or camelCase. I expected the doc to enable me to write some with confidence. Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cluster-lifecycle
FWIW the answer is flag names as on the command name but without any leading dashes. This also applies to controllerManager.extraArgs, scheduler.extraArgs, etc. which hopefully should all cover this similarly (if not now, after fixing ...)

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --I was reading about how to use kubeadm and noticed a typo in the API documentation. speaks of the "node subnet" where I think "pod subnet" is intended. Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cluster-lifecycle
/area kubeadm


Question: return scaled, nil } <ins> // waitForInactiveReplicaSets will wait until all passed replica sets are inactive and have been noticed // by the replica set controller. func (dc *DeploymentController) waitForInactiveReplicaSets(oldRSs []*extensions.ReplicaSet) error { for i := range oldRSs { rs := oldRSs[i] condition := rsutil.ReplicaSetIsInactive(dc.client.Extensions(), rs) if err := wait.ExponentialBackoff(unversionedclient.DefaultRetry, condition); err != nil { return err } } return nil } </ins> // scaleUpNewReplicaSetForRecreate scales up new replica set when deployment strategy is "Recreate" func (dc *DeploymentController) scaleUpNewReplicaSetForRecreate(newRS *extensions.ReplicaSet, deployment *extensions.Deployment) (bool, error) { scaled, _, err := dc.scaleReplicaSetAndRecordEvent(newRS, deployment.Spec.Replicas, deployment)

The scenario is a single replica with an attached EBS. The create seems to happen right after the delete not waiting for the pod to be terminated. This results in the two kubelets fighting over the EBS which generally results in AWS becoming unhappy meaning the EBS has to be force detached. From the docs I would have thought deployments would wait until the pod is actually killed. Is this a bug or am i misinterpreting the docs?
As a work around, what happens if you use a rolling update with MaxSurge=0 and MaxUnavailable=1?
Just assigned for triage, not to fix the issue. Can clear your assignment once this is triaged.
Deployments with size of 1 should be the same (both in that it's really a Recreate and it has the same problem even though it only matters for Recreate), larger deployments should be rolling.
It's because Deployment scales old RS down to 0 (=0, and RS controller will observe this change later and then kill its pods) and then scales up the new one. So it's possible there's a race here.
Opened that partially fixes this.
Why it's closed? It's not fixed: Recreate still doesn't wait for termination.
what version are you using?
1.6.2-00 from deb package. I'm changing the Deployment to use different configmap. K8s recreates pod, but the new one can't listen on ports that are probably used by the old one (using ). Both are appearing in at the same time (old one is 'Terminating').
This regressed in 1.6 but was fixed again in I don't think it was cherry-picked back to 1.6 though and the cherry-pick is not clean. are you interested in backporting it to the release-1.6 branch?
the cherry-pick isn't clean mostly because of comments. Don't have time to actually build and test the PR though.
I think this is still an issue:
I cannot reproduce it anymore, can you provide a reproducer?
Not easily at the moment -- we ended up changing our pods to handle multiple running simultaneously gracefully.
Hi All, I'm still seeing this on the following version: I am using recreate and can see my current pod go into a terminating state and my new pod come up.

It is my understanding that the new pod should not come up until the old pod is terminated completely.
Seeing this on 1.14.3 too - AWS hardware went bad and stuff was in terminating (but still running) and our stateful singleton was brought back up alongside
We see the same problem with 1.14.3 where the old pod is Terminating and the new pod is in ContainerCreating even if we set the Got problem with Volume is already used by pod(s)...
The easiest way I've found to reproduce this is by using bash interactively - it swallows SIGTERM giving a bigger window to observe the behavior. One thing to note, the behavior is different with an explicit delete and one caused by a changing annotation. When changing an annotation, the new pod won't be scheduled until the old one is completely gone, but an explicit delete will create the new pod right away. That said, I don't know how it will behave with other events, say a node going away or being drained. Here is a way to reproduce it. Delete the pod () and watch the new one spin up right away.
recreate deployments wait for the termination of an old version only during a deployment upgrade. The deployment api has been designed with availability in mind and cannot ensure consistency during node upgrades (use PDBs), unexpected node shutdowns, or any other reason that could cause a pod to be recreated (apart from deployment upgrades). If your application has a hard requirement on consistency (eg. database), use . can you please provide a reproducer? Thanks.
Same issue here with version v1.17.6: Deployment with "Recreate" strategy: If I try to delete a pod, the old pod is still Terminating, the new pod has already get initialized and Running:
/reopen
You can't reopen an issue/PR unless you authored it or you are a collaborator. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
see the comment right above you "recreate deployments wait for the termination of an old version only during a deployment upgrade.

The deployment api has been designed with availability in mind and cannot ensure consistency during node upgrades (use PDBs), unexpected node shutdowns, or any other reason that could cause a pod to be recreated (apart from deployment upgrades). If your application has a hard requirement on consistency (eg. database), use statefulsets."


Question: func TestVersionFromNetwork(t *testing.T) { type T struct {  Content       string Expected      string ErrorExpected bool   Content              string Expected             string FetcherErrorExpected bool ErrorExpected        bool  }  currentVersion := normalizedBuildVersion(constants.CurrentKubernetesVersion.String())  cases := map[string]T{  "stable":     {"stable-1", "v1.4.6", false}, // recursive pointer to stable-1 "stable-1":   {"v1.4.6", "v1.4.6", false}, "stable-1.3": {"v1.3.10", "v1.3.10", false}, "latest":     {"v1.6.0-alpha.0", "v1.6.0-alpha.0", false}, "latest-1.3": {"v1.3.11-beta.0", "v1.3.11-beta.0", false}, "empty":      {"", "", true}, "garbage":    {"<?xml version='1.0'?><Error>NoSuchKey<Message>The specified key does not exist.</Message></Error>", "", true}, "unknown":    {"The requested URL was not found on this server.", "", true},   "stable":     {"stable-1", "v1.4.6", false, false}, // recursive pointer to stable-1 "stable-1":   {"v1.4.6", "v1.4.6", false, false}, "stable-1.3": {"v1.3.10", "v1.3.10", false, false}, "latest":     {"v1.6.0-alpha.0", "v1.6.0-alpha.0", false, false}, "latest-1.3": {"v1.3.11-beta.0", "v1.3.11-beta.0", false, false}, "empty":      {"", currentVersion, true, false}, "garbage":    {"<?xml version='1.0'?><Error>NoSuchKey<Message>The specified key does not exist.</Message></Error>", currentVersion, true, false}, "unknown":    {"The requested URL was not found on this server.", currentVersion, true, false},  } for k, v := range cases {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: I was using kubeadm to generate cluster ca: The CA generation failed because of 502 bad gateway returned and the response body isn't empty when it's trying to fetch stable version from internet. In this case, the function returns error instead of falling back to local client version: is how kubeadm fetches stable version and is how the error is handled for . Should kubeadm always fall back to client version when there is any internet issue? Otherwise, this sounds defeat the purpose of fallback. /sig cluster-lifecycle What you expected to happen: kubeadm falls back to local client version when there is any internet issue. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): 1.13.7 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
I'd like to take this issue. It would be better if you could give some directions. /assign
recreate the issue in kubernetes/kubeadm: /close
Closing this issue. <detailsIn response to : recreate the issue in kubernetes/kubeadm: Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
the fix will be removing the when failed to and add some unit test if I don't missing anything
Got it. Thanks.


Question: } // Pod should fail <del> testPodFailSubpath(f, pod, "SubPath `..`: must not contain '..'") </del> <ins> testPodFailSubpath(f, pod) </ins> }) /* Testname: var-expansion-subpath-with-absolute-path Description: Make sure a container's subpath can not be set using an <del> expansion of environment variables when absoluete path is supplied. </del> <ins> expansion of environment variables when absolute path is supplied. </ins> */ It("should fail substituting values in a volume subpath with absolute path [Feature:VolumeSubpathEnvExpansion][NodeAlphaFeature:VolumeSubpathEnvExpansion][Slow]", func() {

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: Looks like the newly tests are failing: https://k8s-
/assign Also random question to why is the same test run 4 times?
GitHub didn't allow me to assign the following users: kevtaylor. Note that only and repo collaborators can be assigned. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
cc
can anyone shed some light on why events are not being returned from a pod in the alpha suite? This has a 1 minute pod timeout and everything works fine locally
Reiterating the mentions to trigger a notification: <detailsIn response to : can anyone shed some light on why events are not being returned from a pod in the alpha suite? This has a 1 minute pod timeout and everything works fine locally Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I suggest unless anyone can define why the events do not appear, we disable that part of the test and just check for error. If you are okay with that please reopen the bug and I'll put in a new fix
it looks like node alpha suite is having some other issues right now that are preventing the tests from being run. So it's not clear right now if the fix helped or not.
it failed again. I did a comparison of a locally running one and the alpha suite. In the log file https://k8s- there are these references But in the locally running tests I don't see that step This running And the events running locally
/reopen


Question: value.RecordCacheMiss() } uid := string(uuid.NewUUID()) <ins> klog.V(6).InfoS("Decrypting content using envelope service", "uid", uid, "key", string(dataCtx.AuthenticatedData())) </ins> key, err := t.envelopeService.Decrypt(ctx, uid, &DecryptRequest{ Ciphertext:  encryptedObject.EncryptedDEK, KeyID:       encryptedObject.KeyID,

Add additional logging at level that associates UID with the metadata from the request. xref: /sig auth
/triage accepted
/milestone v1.26
Mind if I take a look at this? /assign
Hello I am wondering how's this coming along. Is there anything I can help with?
Sorry. I got a bit occupied with a different project.. Will have a PR opened for this for review tomorrow.. I am trying to find a way to inject the Verbs into log. But haven't found the right way to include that into the context.. Any suggestion on what can be done?
any suggestion on what is the best way to get the additional metadata for verbs into this log? Move that info across using ?
Hello , bug triage team here, just wanted to touch base with you for updates on this ticket. Thanks!
Thanks! This is being worked on and is still on track for .
There is a Pull request opened on this already as well. But Waiting on some clarification to see if that needs additional changes.
/reopen /assign /unassign
Reopened this issue. <detailsIn response to : /reopen /assign /unassign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/milestone v1.27
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
Hi Bug Triage Team for v1.27 here o/ Just wanted to touch base with you for updates on this ticket. Thanks!
I am not working on this anymore Carlos. This was re-assigned
This is tracked for . I've linked the PR.


Question: Help:      "Number of selected preemption victims", // we think #victims>50 is pretty rare, therefore [50, +Inf) is considered a single bucket. Buckets:        metrics.LinearBuckets(5, 5, 10), <del> StabilityLevel: metrics.ALPHA, </del> <ins> StabilityLevel: metrics.STABLE, </ins> }) PreemptionAttempts = metrics.NewCounter( &metrics.CounterOpts{ Subsystem:      SchedulerSubsystem, Name:           "preemption_attempts_total", Help:           "Total preemption attempts in the cluster till now", <del> StabilityLevel: metrics.ALPHA, </del> <ins> StabilityLevel: metrics.STABLE, </ins> }) pendingPods = metrics.NewGaugeVec( &metrics.GaugeOpts{ Subsystem:      SchedulerSubsystem, Name:           "pending_pods", Help:           "Number of pending pods, by the queue type. 'active' means number of pods in activeQ; 'backoff' means number of pods in backoffQ; 'unschedulable' means number of pods in unschedulableQ.", <del> StabilityLevel: metrics.ALPHA, </del> <ins> StabilityLevel: metrics.STABLE, </ins> }, []string{"queue"}) SchedulerGoroutines = metrics.NewGaugeVec( &metrics.GaugeOpts{

I would like to propose the graduation of the following scheduler metrics from alpha to stable: These metrics have been stable for at least 15 months.
/sig scheduling /sig instrumentation
/cc
IIRC, the last addition to these metrics was the profile name label. I don't foresee need for new labels. I don't see many precedents for graduation in core components. are there any defined criteria for graduation?
As requested, the following is how long it has been since a change has been made to each of the requested metrics: - 2 years; last change was to migrate to the metrics stability framework - 17 months; last change was a renaming of the metric - 15 months; last change was a renaming of the metric - 17 months; last change was to add a label - 17 months; last change was to add a label
It's up to the component owners to commit to graduating these metrics, since the component owners own the metrics and therefore need to agree to support them in accordance to stability requirements. If you guys are okay with promoting these metrics, I don't see a reason, from an instrumentation side, why we would object to promotion.
+1 to graduation then. These metrics are very mature. Although, I'm not sure if all metrics already adhere to naming conventions. and don't have a unit suffix, but they are unit-less. Is that fine
is a gauge so it's fine, is a histogram so it's also fine.
I'm recently working with metrics, I'm glad to provide some help if no one has interest about it.
There is already an open PR for this.
Oh sorry, didn't notice about that. Forget it.
These graduations sound good to me, but I would be inclined to rename . I feel that the part of the metric is unnecessary. Having instead and keeping for the algorithm latency is already descriptive enough to know the difference between both. If that's not the case, then the text of the metric can be used to guide the users.
I agree that is rather confusing, so +1 to removing the prefix. The metric doesn't include retries, for example. We have a separate metric that includes retries: We need to rename that one eventually to e2e, but it can only be done after the existing one has been deprecated and deleted for a few releases. /triage accepted
/assign
/cc
The PR for this is ready to be reviewed:


Question: stopCh: make(chan struct{}), stopWg: sync.WaitGroup{}, } <ins> // See startCaching method for explanation and where this is unlocked. </ins> cacher.usable.Lock() <del> // See startCaching method for why explanation on it. watchCache.SetOnReplace(func() { cacher.usable.Unlock() }) </del> watchCache.SetOnEvent(cacher.processEvent) stopCh := cacher.stopCh

cc
This is still hapening. Assigning Isaac for triage.
Three times in the last three days, BeforeSuite failed (filed ). Otherwise, just Namespaces [Serial] should delete fast enough (90 percent of 100 namespaces in 150 seconds) is just totally borked (filed ). what prevented you from triaging this yourself? It'd be good to know so I'm less in the critical path.
Sorry, I read this wrong. I was looking at the gke-serial suite. Triaging now.
It looks like massive failures are correlated extremely strongly with failure of Etcd failure [Disruptive] should recover from SIGKILL Every time that test passes, everything else (except ) passes; every time it fails, the whole suite goes south. Reassigning to for triage.
sorry for delegating the work to you, but I completely didn't have time to do it and nobody else hasn't volunteered to do it for a while. it seems you are the author of the test, so assigning you.
No problem Just wanted to make sure there wasn't some information missing.
I'm looking at this today
It looks like the token controller is not recovering from etcd failing. The test namespace is also stuck in a terminating phase. I'll continue to investigate
Actually lot's of watch related things are broken. I can create/delete objects in the api but watch is broken. Trying to establish a watch in this state causes: Restarting the apiserver fixes the problem. I can start a watch again: cc
Not particularly useful apiserver logs Also none of the components are trying to reestablish watches so I'm assuming that they still have watches open but are not receiving anything.
startCaching is trying to lock here, but something is holding the lock. I'm looking through a stack trace right now for the offender. This is exactly where I'd expect a problem to be from the description of "bringing etcd down and up repeatedly".
Ah I think what happened is, when etcd stays down, a List never successfully finishes, and therefore it never calls the OnReplace function, and therefore the lock is never unlocked, and therefore startCaching can't exit.

There's lots in apiserver log to support this: Additionally I manually looked through the stack trace and confirmed that all of the stuck calls to Watch are stuck waiting to RLock and not some point afterwards.
This is pretty bad, etcd going down for just a bit could freeze up apiserver until it is rebooted.
I think I can make a fix.
Thanks for looking into this.
Yeah, I repro'd in a UT. Fix coming shortly.
I assigned to but if anyone wants it merged sooner than monday, feel free to review.
Many thanks to for handing me a broken server to look at, it made this super easy to find.


Question: go.opentelemetry.io/contrib/instrumentation/github.com/emicklei/go-restful/otelrestful v0.42.0 go.opentelemetry.io/otel v1.28.0 go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.27.0 <ins> go.opentelemetry.io/otel/metric v1.28.0 </ins> go.opentelemetry.io/otel/sdk v1.28.0 go.opentelemetry.io/otel/trace v1.28.0 go.opentelemetry.io/proto/otlp v1.3.1

Since updating from Kubernetes 1.26. to 1.29. we experiences OOM-Kills on control plane nodes. After some investigation we found that adding to the fixed the issue. (Found this via pprof/heap of one apiserver: heap dump could be if needed) Before the update to 1.29.* the control planes were running with 4GB of memory with the same worklow in the cluster without issues. After the workarround using the feature gate the memory consumption of the whole control plane is not at ~2GB. In case there is an issue with other components, memory consumption of the apiserver should not be that excessive. We just had the K8s cluster running and saw a permant increase in memory usage of the control plane nodes. And after some time (can not give a specific duration) the OS of the control plane node started to OOM kill processes. Let us know if you need additional information. <details</detailson-premise; control-planes are running with 4 CPU cores, 8GB of memory and and NVME discs <details</details<details</details<details</detailsNo response
/sig api-machinery
Please provide the pprof, and what is the size of your cluster.
As we were not sure if sharing the pprof is secure, we decided to only share screenshots first: ! ! From this information we found the this which pointed us to the mentioned feature flag. We sometimes see etcd connection timeouts in the apiserver logs, maybe this could be related but should not cause this behaviour. 5 control planes ~100 worker nodes ~1700 Pods
Can you provide the pprof without , so could compare the two?
Sure, the current pprof with set: ! !
cc
I believe this is Sounds like we may need to backport some of the dependency bumps
/assign
/sig instrumentation
/triage accepted (as it's now assigned to
/triage accepted
We need to apply the workaround in It should be simple, but needs to be backported to 1.27
all versions starting 1.27 right ?
Thanks for taking care of this so quickly
yes. All versions starting 1.27.
Actually, this only needs to be backported to v1.29.

At 1.28, we were using v0.35.0 of otelgrpc, which did not include metrics at the time.
Cherry-picks:


Question: func testRESTClientWithConfig(t testing.TB, srv *httptest.Server, contentConfig ClientContentConfig) *RESTClient { base, _ := url.Parse("http://localhost") <ins> var c *http.Client </ins> if srv != nil { var err error base, err = url.Parse(srv.URL) if err != nil { t.Fatalf("failed to parse test URL: %v", err) } <ins> c = srv.Client() </ins> } versionedAPIPath := defaultResourcePathWithPrefix("", "", "", "") <del> client, err := NewRESTClient(base, versionedAPIPath, contentConfig, nil, nil) </del> <ins> client, err := NewRESTClient(base, versionedAPIPath, contentConfig, nil, c) </ins> if err != nil { t.Fatalf("failed to create a client: %v", err) }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --In , client-go don't share transport when c.Dial is not nil, but kubelet custorm Dial by default, After PR is integrated, the connections between kubelet and kube-apiserver in our cluster with 4000 nodes increases by five times. kubelet share transport by default, one kubelet only keep one connection to kube-apiserver. start kubelet with rotate-certificates=false Kubernetes version (use ): v1.19.4 Cloud provider or hardware configuration: OS (e.g: ):EulerOS 2.9 Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig api-machinery /sig node
/triage accepted /assign
how many connections between kubelet and kube apiserver on one node after PR in your case? it's mentioned it increases by five times, does that mean before the PR, there is one connection, after the PR, there are 5 or 6 connections?
before the PR, there is one connection, after the PR, there 3 connections, there are some customization capabilities in our product, and we'll have another two connections. In our kubelet startup parameters, rotate-certificates=false
Some increase was expected, since multiple REST clients constructed from a config with a custom dialer cannot safely share a transport, and client-go constructs a REST client for each API group/version accessed. Something like would be required to rework how client-go constructs clients to start sharing REST clients above the transport level between API groups/versions. I think this is working as intended until client-go client construction is reworked.
do you know where in the code the 5 connections are created? are they all safe to share the transport? also are the 5 connections all active all the time? if not, IdleConnTimeout should cause the connection to be closed.
I test the code on master branch, if I change rotateCertificates to false, there are 6 connections between kubelet and kube-apiserver ! !

transport will share when config.Transport not nil if rotateCertificates is true, config.Transport will been set in kubeletcertificate.UpdateTransport but if rotateCertificates is false, config.Transport will not been set.
/area kubelet Any update on this issue?
I will work on this issue
Hi if rotate-certificates=false kubelet customizes the dialer to provide a closeAllConns to close the connection when the connection is dead but not been closed. In , there is a discuss about another solution after client-go has HTTP 2.0 health check, so we don't need this change. So we can solve the current problem by reverting . I have submitted to revert , and this change requires your final confirmation. Thanks you.
are we guaranteed kubelets always speak to kube-apiserver over http/2? if they use a load-balancer that is not http/2 capable, isn't the "close all connections" fix still required as a backstop?
I thought http1.1 wasn't affected because it will always try to connect a new connection
I'm not confident in that. The original issue dated back to before client-go defaulted to http/2, and we definitely saw issues with stuck TCP connections then.
yeah, I could reproduce it with http1.1, however, it can be solved using the transport CloseIdleConnections method please take a look , the last commit needs more work, but I think that the 2 first commits exposeing the CloseIdleConnections() are useful
ok, this seems to work , but it will be nice to have more feedback would you be able to test and see if this solve your issue?
Hi aojea, I have test on v1.21.1 before merge , there are 6 connections to kube-apiserver from kubelet ! after merge , there is only 1 connection to kube-apiserver from kubelet ! so, can solve my issue. thank you very much.


Question: #   ROOT #   KUBERNETES_RELEASE_TAR function upload-resources() { <del> swift post kubernetes --read-acl '.r:*,.rlistings' </del> <ins> swift post ${SWIFT_OBJECT_STORE} --read-acl '.r:*,.rlistings' </ins> locations=( "${ROOT}/../../_output/release-tars/${KUBERNETES_RELEASE_TAR}"

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: Currently swift object store has a hard-coded name: . E.g., What you expected to happen: Make swift object store parameterizable. E.g., How to reproduce it (as minimally and precisely as possible): This defaults to the hard-coded swift object store. Anything else we need to know?: Here is the patch file to address this issue: Environment: Kubernetes version (use ): Cloud provider or hardware configuration**: OpenStack OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <bre.g., for API Machinery<br(2) specifying the label manually: <bre.g., /sig scalability for sig/scalability<br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
/sig openstack /sig storage /remove-sig api-machinery
cc
Quick question, if you have the diff already, why not send a PR?
I noticed the contributor documentation is extensive (b/c k8s is such a big project, this is understandable) and I wasn't sure how if I was following the rules. I can send the PR, if you would like. This is no problem. Let me know :-)
I created a PR based on the diff in your issue. Please let me know if you'd like me to abandon it so that you can have credit for the work.
Here you go:
hey can you please reopen this?
I don't think I have enough privileges to reopen. It's probably best to create a new issue and reference this one.
Ah. thanks
/assign
/reopen
Looks like the bot commands worked! ()
/close


Question: ObjectMeta: metav1.ObjectMeta{ Name:            jobName, Annotations:     map[string]string{"cronjob.kubernetes.io/instantiate": "manual"}, <del> OwnerReferences: []metav1.OwnerReference{*metav1.NewControllerRef(cronJob, apps.SchemeGroupVersion.WithKind("CronJob"))}, </del> <ins> OwnerReferences: []metav1.OwnerReference{*metav1.NewControllerRef(cronJob, cronJob.GroupVersionKind())}, </ins> }, Spec: batchv1.JobSpec{ Template: corev1.PodTemplateSpec{

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: of Job created by using command will be . This apiVersion is wrong. What you expected to happen: Set the to How to reproduce it (as minimally and precisely as possible): CronJob manifest: <details</detailsApply CronJob and execute command Anything else we need to know?: Environment: Kubernetes version (use ):
/sig cli
This was introduced by . /assign
GitHub didn't allow me to assign the following users: odino. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : This was introduced by . Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I created fix pull request:


Question: } func TestLoadBalancer(t *testing.T) { <del> ipt := iptablestest.NewFake() ipvs := ipvstest.NewFake() ipset := ipsettest.NewFake(testIPSetVersion) fp := NewFakeProxier(ipt, ipvs, ipset, nil) </del> <ins> ipt, fp := buildFakeProxier() </ins> svcIP := "10.20.30.41" svcPort := 80 svcNodePort := 3001

Is this a BUG REPORT or FEATURE REQUEST? /kind bug What happened: In ipvs proxy mode, most service functions like "nodeport/loadbalance" is realized by the association of iptables/ipset/ipvs, yet most ipvs proxier ut only verified ipvs rules, which can't ensure service is functional. What you expected to happen: Check both ipvs rules and iptables chains/rules, ipset entries in ipvsproxier ut.
/sig network /area ipvs /assign
What's the current UT coverage number?
Current is 50.2%.
So, can we achieve 70%?
Yeah, we can do that, and that's definitely necessary.


Question: klog.V(4).Infof("Found active IP %v ", finalIP) return finalIP, nil } <ins> // In case of network setups where default routes are present, but network // interfaces use only link-local addresses (e.g. as described in RFC5549). // the global IP is assigned to the loopback interface, and we should use it loopbackIP, err := getIPFromLoopbackInterface(family, nw) if err != nil { return nil, err } if loopbackIP != nil { klog.V(4).Infof("Found active IP %v on Loopback interface", loopbackIP) return loopbackIP, nil } </ins> } } klog.V(4).Infof("No active IP found by looking at default routes")

What happened: When attempting to kubeadm join a control plane node or worker on a machine with global IPv4 unicast addresses on a loopback address (router to the host w/ BGP Unnumbered), the join fails with the following output: See this issue for details : Also: What you expected to happen: The correct IPv4 address should be selected from the loopback adapter, or we should be able to specify an IPv4 address to use for egress. The IPv6 addresses on the network adapters themselves are only used to establish upstream peering. IPv4 nexthops are fake 169.254.x.x addresses artificially programmed via RFC5549 into the ARP table. How to reproduce it (as minimally and precisely as possible): On a node with the following conditions: Routing to the host with FRR IPv6 link locals on physical network interfaces No IPv4 addresses on physical network interfaces Unicast IPv4 on the loopback adapter IPv4 default route present via RFC5549 Attempt to kubeadm join a node, and it will fail. Anything else we need to know?: We've been patching kubeadm with , which works as expected. There was a fix applied to allow initing control plane nodes w/ bind address 127.0.0.1 which would then select an IPv4 from the loopback, but it doesn't appear to have fixed the joining issues. Pinging and by request on kubeadm issues. Environment: Kubernetes version (use ): 1.17.12 Cloud provider or hardware configuration: baremetal OS (e.g: ): Centos 7 Kernel (e.g. ): 3.10.0-1127.19.1.el7.x86_64 Install tools: kubeadm Network plugin and version (if this is a network-related bug): Calico Others: FRR for peering with routers on physical network interfaces
/sig api-machinery
is defined but we should note the same function is used for the --bind-address of the kube-apiserver. /sig network
/assign
/triage accepted


Question: # scope and that the Logging API has been enabled for the project # in the Google Developer Console. <del> FROM ubuntu:16.04 </del> <ins> FROM gcr.io/google_containers/ubuntu-slim:0.4 </ins> MAINTAINER Alex Robinson "arob@google.com" # Disable prompts from apt.

It's 400+mb and we need to pull it on every node. This leads to a lot of image pulling flake (), and also gets very expensive in some countries where bandwidth is sold at a premium. We took a stab at halving the size with: , but then realized that we were using an unsupported base image We tried to optimize to ~300Mb by building a smaller ubuntu:15 base, but that's still huge. Every other mandatory system daemon is built on busybox and so takes ~10Mb.
I can try again now that the agent supports
Newer Docker images using the spec from the official repository are based on AlpineLinux and are around ~38MB: