Question: ec2Device = "/dev/sd" + string(mountDevice) } <del> attached := false </del> <ins> // attachEnded is set to true if the attach operation completed // (successfully or not) attachEnded := false </ins> defer func() { <del> if !attached { awsInstance.releaseMountDevice(disk.awsID, mountDevice) </del> <ins> if attachEnded { awsInstance.endAttaching(disk.awsID, mountDevice) </ins> } }()

Environment: CoreOS 815.0.0, k8s 1.1.0 () It happens to me quite often that a volume marked as "available" on AWS remains in a "detached" state according to the kubelet: Deleting the pod and recreating it eventually succeeds.
I have some fixes in progress that should fix this
/cc
I can reliably reproduce similar issue, I am not sure it's the same: create a AWS EBS, let's say it has ID Create a pod that uses this EBS: Create the pod. It succeeds eventually: Delete the pod Create the pod again. It will be forever IMO, it's caused by not removing the detached volume from internal map of attached volumes . Subsequent will find it there and it won't even try to attach it again. Patch is available in referenced PR.
can you please confirm that your bugs is the same as I can reproduce? Or do you see different bug, only with similar symptoms?
It is
Was this fixed by ?
I think you referenced the wrong issue / PR, didn't you? As far as I know this is not fixed yet.
Yes, I meant to link . mentioned there that it might fix this issue.
Oh right! It made it to v1.2.0-alpha.4
did not fix this issue (but fixed the DeviceMapping cache invalidation)
I think, what is happening is that you run into an error or timeout while waiting for the disk operation. That leaves the devicecache yet again in an inconsistent state. I have a fix here that just discards the cache on any error. Additionally, it is totally possible that there is concurrent operations which lead to an error and bork the cache. The fix: Unfortunately, this is in our tests still not 100% rock solid. Especially in timeout cases. I'm missing the capability to properly test this to engage in a bigger refactoring of this whole issue. A workaround to actually fixing it properly would be to add a TTL to the cache, so that it eventually starts working again.
Just for information for people who might use the GitHub search, as of Kubernetes 1.1.7 the errors logged by the look like: Deleting the pod or restarting doesn't help, the instance keeps failing attaching the EBS with the same errors if the pod get rescheduled on the same host.

thanks for the fix!
Oh... the same host. I bet I messed up that case... I'm going to move into 1.2 so I take another look at this!
What's the status on this?
I just don't think this is fixable with the volume cache we have. I proposed a PR to remove the cache, and replace it with a map of active attachment operations in progress. A little less efficient, but a lot more correct (presuming passes testing).
How to check what K8 version has this fix?
Is there any progress on this issue? I'm running K8s version v1.5.2, and I'm having similar issues. awsElasticBlockStore volumes are virtually unusable due to one of three errors I'm getting: My volume claim is ReadWriteOnce, so this next error confuses me: And finally: [SchedulerPredicates failed due to persistentvolumeclaims "mysql" not found, which is unexpected., SchedulerPredicates failed due to persistentvolumeclaims "mysql" not found, which is unexpected., SchedulerPredicates failed due to persistentvolumeclaims "mysql" not found, which is unexpected.] Unable to mount volumes for pod "staging-connect-mysql-<id(<id)": timeout expired waiting for volumes to attach/mount for pod "default"/"<id". list of unattached/unmounted volumes=[mysql] Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod "default"/"<id". list of unattached/unmounted volumes=[mysql]
Also having the same issue as ^ Should we be using PV+VPC instead of claiming the EBS volume directly from the deployment spec?


Question: return &tracker{ scheme:   scheme, decoder:  decoder, <del> objects:  make(map[schema.GroupVersionResource][]runtime.Object), </del> <ins> objects:  make(map[schema.GroupVersionResource]map[types.NamespacedName]runtime.Object), </ins> watchers: make(map[schema.GroupVersionResource]map[string][]*watch.RaceFreeFakeWatcher), } }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: I used to fake a clientset to test my code. I found that it became very slow to init the clientset with a big number of objects when I was trying to simulate a large scale cluster, and the time seemed exponential increasing. For example: Initializing a clientset with: 5000 objects took 300ms objects took 2.58s objects took 11.82s objects took 54.64s. It's because the fake clientset used a slice to store each kind of objects, so it's quite slow to init the clientset with massive objects because it checked existence of an object by traversing all objects and comparing with them before adding one, which leads to O(n^2) time complexity. What you expected to happen: The fake clientset is very useful for development, it should have a reasonable time complexity to allow developers to use it for large scale scenario. How to reproduce it (as minimally and precisely as possible): Run this code: You would see the time spent on each operation, like: Anything else we need to know?: Environment: Kubernetes version (use ): 1.15.0-1.18.0 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I had PR that uses map to store objects, which can reduce the init time to a reasonable value (tens of ms for objects). /assign


Question: t.Errorf("unexpected error: %v", err) } <del> verifyCalls(t, fakeDocker, []string{"list", "stop", "list", "create", "start", "stop", "create", "start", "inspect_container"}) </del> <ins> //verifyCalls(t, fakeDocker, []string{"list", "stop", "list", "create", "start", "stop", "create", "start", "inspect_container"}) verifyCalls(t, fakeDocker, []string{"list", "stop", "stop", "list", "create", "start"}) </ins> // A map interation is used to delete containers, so must not depend on // order here.

cAdvisor API is not accessible from the node. It is accessible from inside the cadvisor container though. This is the reason for the cadvisor e2e test to fail and the recent heapster errors reported by users. The cadvisor version itself hasn't changed recently though.
cc
There were some changes recently around port forwarding ( that may be the cause.
From the node: From the cadvisor container:
reported the same issue today. Restarting the container seems to fix it. So, its something to do with the initial setup.
Could this be due to Docker 1.5? On Feb 23, 2015 6:12 PM, "Rohit Jnagal" wrote:
test machine is running Docker 1.4.1 It looks like cadvisor container is sharing its networking with the pause container as usual, but the pause container got restarted, so the networking setup isn't working anymore. Restarting cadvisor connects it to the newer instance of the pause container and it starts working again. However docker inspect shows that cadvisor is connected to network of a different pause container: And that container died soon after it started Kubelet killed the pause container. I think this is kubelet's doing. It tried to restart both pause and cadvisor container. It kills and restarts cadvisor container which connects to existing pause container. Then, it goes ahead and kills/restarts the pause container, leaving the new cadvisor container hanging. Maybe this is related to infra pods issue reported by earlier, or maybe just an ordering issue in kubelet.
Thanks for the detailed debugging :) with your report I think I found the issue. In we check if the container has changed and print the message above. After that we the POD container. We should be doing this before the container restart as you pointed out. I'm assigning this to myself and I will fix tomorrow. Will also rename the bug and re-purpose it.
cool, I was just staring at that code. All yours :+1:
Thanks for debugging the issue. This is a regression we introduced lately.
And when the test, we all assumed it was the test failing. Sorry, :)
I fell for that until I realized that the basic networking can also fail :)


Question: execScripts: []ExecArgs{ {"fsck", []string{"-a", "/dev/foo"}, "", nil}, {"lsblk", []string{"-nd", "-o", "FSTYPE", "/dev/foo"}, "", nil}, <del> {"mkfs.ext4", []string{"-E", "lazy_itable_init=0,lazy_journal_init=0", "-F", "/dev/foo"}, "", fmt.Errorf("formatting failed")}, </del> <ins> {"mkfs.ext4", []string{"-F", "/dev/foo"}, "", fmt.Errorf("formatting failed")}, </ins> }, expectedError: fmt.Errorf("formatting failed"), },

Here's logs from formatting a 10GB HDD disk in Azure as ext4: It took seven and a half minutes to complete . Here's me doing it manually from the node: And even from a container, somewhat emulating kubelet: Kubernetes version (use ): Environment: Cloud provider or hardware configuration: Azure OS (e.g. from /etc/os-release): Ubuntu (Xenial) 16.10 Kernel (e.g. ): Linux 4.4.0-34-generic -Ubuntu SMP Wed Jul 27 16:06:39 UTC 2016 x8664 x8664 x86_64 GNU/Linux Install tools: kubernetes-anywhere / static pods with kubelet / kubelet in container What happened: It took seven and a half minutes to format, causing the overall pod startup time to take over 10 minutes by time attachment occurred What you expected to happen: It to take a matter of seconds, similar to how it does if I execute the command manully. How to reproduce it (as minimally and precisely as possible): Build with on top. Deploy a cluster. Try to attach an unformatted VHD. Observe along with watching logs and confirm it really does take that long. Anything else do we need to know: I'm running containerized. Like this:
Can you confirm you were not seeing this behavior during your testing? I assume you were not running containerized?
Even if I and execute the same command from there it executes in a matter of a few seconds.
I'm seeing this behavior even when running as a normal systemd unit, and not as a docker container under a unit. Here's : And I'm still seeing the very slow , so it seems not related to running kubelet in container. I have also tested to ensure that this isn't something some sort of host caching. Even on another blank disk being attached for the the first time, executes quickly except when run by .
I confirm this is this case on my setup running local-up- I have a 100GB HDD disk. It took 30 seconds to format on the console but 2 minutes by kubelet. Is there any I/O constraint imposed on kubelet?
to see if she knows why would be slow when called from the kubelet.
cc/ Who is working on node performance benchmark.

Can we quickly run an experiment on GCE to see if we can reproduce the issue. Let's first rule the impact of containerized Kubelet. If the issue is still exist, I suspect the issue in Kubelet mounting logic. We can enable some tracing, and find the bottleneck.
I tried formatting a 10GB HDD in GCE Kubernetes (not containerized). The time of formatting in kubelet log is ~1s. The log is:
On Azure, a 10GB disk took 18s to mkfs by kublet but 4s to mkfs on the console. Do you have a chance to test these two scenarios? Size probably is not a critical factor for gce since gce pd is provisioned at fixed IOPS per GB, while Azure (also AWS) disks are provisioned at fixed IOPS per volume.
There's more variables in play or else I'm seeing a different edge case than you are, On Azure with a cluster, a 10GB disk for me consistently takes 7+ minutes. (, standard storage, HDD type). It was happening with a and with premium storage and SSD type data disk too... I can make a cluster available to anyone who would like to investigate the exact behavior I've been observing. I can provide kubeconfig+ssh creds. Contact me on Slack (colemickens) if interested.
mkfs's kernel stack after started by kubelet is always as the following. Not sure if it related to blkio cgroup
I have actually created a branch of to give me a cluster where is never running in a container and I'm still seeing the exact same issue. Took 7.5 minutes again, based on kubelet logs and watching . I think we can safely eliminate "containerized kubelet" as being the source of the problem.
Though if I remember correctly, kubelet places itself in a cgroup when it starts up. ( can confirm) so it's possible that it is still due to cgroup I/O throttling.
Above test result proves my initial suspect that the root cause in 7-min use case is containerized kubelet. But recent experiment is not running in a container.

could you please provide us more information: 1) cat /proc/<kubelet-pid/cgroups to find kubelet's cgroup 2) check & blkio.weightdevice setting 3) ps ax | grep makfs to find process id, the find which cgroup the process belongs to 4) check 3) cgroup's and blkio.weight_device If both 2) and 4) don't have issues with cgroup setting, please check if there is parent cgroup for them, and their setting since both CFQ and throttling supports hierarchy.
FYI: Kubelet does not perform any block IO throttling as of now for itself or any other entity on the node. , Dawn Chen wrote:
Yes, we know Kubelet doesn't. But the issue here indicates containerized Kubelet's IOs is throttled, and we are trying to figure out why, especially the user is running kubernetes-anywhere on Azure, which we, node-team don't normally test.
Here's how I was running originally: Here's how I was running when testing it not in a container: I will try to collect the data you mention but I'm going to have to learn how to do some of it along the way, I don't know much about cgroups.
this is not limited to azure, saw it on aws as well.
it seems only the hit only happened to ext4, xfs didn't have this problem.
I ran across the same performance issues and tried to investigate. I use Azure Dynamic Disk provisioning to provision a 1TB volume and it takes very long time for mkfs to finish. Doing the "mkfs.ext4 -F -E lazyitableinit=0,lazyjournalinit=0 /dev/sdd" manually is also very slow, but I did not compare the performance to the call done from inside kubelet. While running the mkfs manually, I observed that iotop shows around 62mb/s and that the free memory immediately shrinks from 2000mb to 100mb. It looks like mkfs is using up all free memory for the buffer cache. I even observed oom situations in the kernel log and processes being killed because of this. I've seen this behavior while running mkfs manually and while it was run by kubelet.

Removing the extended flag "lazyitableinit=0" from the manual mkfs call fixes both problems, performance and memory/buffer cache usage. My solution will now be to use xfs (as rootfs reported it to work better). But IMHO the lazyitableinit should be completely removed from the mkfs call.
Can we try to make sure this gets addressed in 1.6?
I did some further investigation here and it turns out there are other factors involved as well, especially on Azure. My initial statement that I got OOM due to mkfs.ext4 seems to be not true, as the OOMs were caused by fluentd-elasticsearch. I tried to use mkfs.ext4 with "-E discard=1" as I assumed that it should speed up things dramatically. This did not make a difference however. While trying to understand the mke2fs code from e2fsprogs, I realized that discard is enabled by default and always performed as first step when the underlying block device supports this. From the mkfs.ext4 logs you can actually see that it is executed at the beginning. It turns out that mke2fs does only skip inode table initialization when the kernel reports that a discard also ensures zeroing of the data. mke2fs checks this by doing an ioctl to BLKDISCARDZEROES. Support can also be checked with On Azure however, the kernel reports that a discard does NOT zero out the discarded blocks. This results in a loop, using normal write calls to write a lot of zeros to the block dev. This completely fills up the page cache with dirty pages, slowing down the whole system extremely. If there is not much free memory available that can be used for write buffering, the write rate goes down to very low values as well. I did not check the kernel code for lazyitableinit, but I would assume that it uses much more efficient code to write the zeros to the block device (e.g. by not relying on dirty pages). I'd also assume that it does these checks for the discard+zeroout behavior/support and then use discard if possible. If my assumption is correct, then it should not affect performance in a bad way when lazy init would be enabled again. The kernel would then do the discards in a lazy fashion, but finish pretty fast.

On Azure, the lazy init would still take some time, but use much less system resources (page cache) as the non-lazy solution. And most importantly, the device can already be mounted after a few seconds. (UPDATE: please check next comment) I tried to figure out if a discard (using the blkdiscard cmd) does really leave you with non-zeroed data and it looks like it actually becomes zero. I can however not see if this happens immediately or with a delay. Also, in the Azure Portal I can see that the "Billable Size" goes from several GB back to 0 after a short time. My assumption is that on other cloud providers, e.g. GCE and AWS, the block devices report that discard zeroes out the data. Can anyone with a GCE/AWS machine check this with the cat call from above? This would explain why this slowness does not occur on non-Azure clouds. Can you check why the kernel reports that discard does not zero-out? Is this a bug in Azure maybe?
~Ok, my assumption about the kernel using discard for lazy init was wrong. Just checked the kernel source and it simply issues multiple write bios which write zero-pages. So it would definitely have a negative impact on other cloud providers when lazyitableinit would be enabled.~ Would it be a viable solution to check support for BLKDISCARDZEROES in kubernetes before doing the mkfs and then decide based on that if lazyitableinit should be enabled or not?
Argh, digging deeper into the kernel turned out that it actually uses discard for zeroing out (check and ). So my initial assumption was correct. Sorry for all the go home now :)
Just checked the discard support on AWS and it is not supported at all, so it always does a normal write to clear/init the inode tables.
here's the output from a GCP instance: We haven't seen any slowness on GCP, even though the above settings are identical to our Azure instance's. We're trying to solve the same slowness issue on Azure, just not with kubernetes :) (we're Cloud Foundry). We're not using containers on this particular VM. Also, the problem manifests with but not .

Related issue here:
We discovered that by using instead of the problem went away — disks which previously took 8+ minutes to mkfs.ext4 took seconds. Alternatively, will also work as a partitioner (i.e. the subsequent will finish within seconds). Really. We're not sure why the tool we use to write one sector (i.e. the MBR) on a disk would make such a difference. HTH,
Did you discover this by manually running / on the nodes, or did you actually modify to use ?
This sounds like alignment issues. As far as I remember, sfdisk does not do proper alignment and thus may result on very slow performance when you are not on classical spinning disks. Azure disk provisioner does not even create partitions on the disks and thus alignment should not be the problem here. Just tested creating a partition with sfdisk and iotop showed only 7-10mb/s throughput when doing mkfs. Not using a partition at all or using fdisk gives around 60mb/s.
: We discovered this by manually running gparted / sfdisk on the instances we deployed on Azure (i.e. we're not using containers). We don't use Kubernetes — we are developers on Cloud Foundry, which is an alternative to Kubernetes. When we ran into this problem, we were pleased to discover we were not the only ones who have been bitten by this bug. And we wanted to make sure our Kubernetes kin were aware of the fix that we found to be effective. some notes on our research (and I cannot thank you enough for your detailed descriptions): we also found throughput of 7-10MB/s when was healthy when things were unhealthy, showed throughput of ~300kB/s during . we found that by passing (use direct IO and bypass buffer cache), we doubled our performance to ~700kB/s, but it was still ridiculously long (3 minutes).
I am going to try to do some additional investigation tonight, but I'm unsure of how to translate your observations to Kubernetes here. As implied, we don't use anywhere here (that I'm aware of). There's three scenarios: -dynamically provisioned. User brings a pre-formatted VHD. Everything goes smoothly as the merely needs to mount the block device and tell to mount it into the container. -dynamically provisioned.

User attaches an empty disk to an Azure VM, detaches it, uses it in a normal AzureDisk volume on a Pod. This is the scenario in the original post, and you can see that it is being mkfs'd without any partitions. disk provisioning. The VHD footer is written into the page blob, and then formats the whole disk, unpartitioned (trusting description of the provisioner here). I'm hesitant to rule out an alignment issue since you're seeing similar symptoms as we are here, but I'm not sure how to reconcile that with my other points in this comment...
What do you think about It would solve this issue. It would not tell us why it is slower in kubelet, but the problem itself would be solved. Should I CC more people to get this discussed faster? I would even like to see this in an upcoming 1.5.x release.
a lead Azure Engineer ( ) found the problem — it is an alignment problem gone horribly wrong: In situation like unaligned partition boundary, a big I/O request (e.g. a 32KB write) can be split into 64 512-byte writes by the kernel block I/O subsystem in the worst case, easily hitting the 500-IOPS throttling of Azure. It was made to simplify Hyper-V storage driver’s code, so Hyper-V’s storage driver picked up this patch in kernel versions: 4.4+, that is why only Azure is impacted. The community member who submitted the original patch already for this issue, waiting for Linux community to merge.
Does that also affect cases where the disk is unpartitioned, ?
If I understand the description correctly, this only applies to cases where unaligned partitions are in use. This is not the case for Azure Dynamic Disk provisioning and also for all the tests performed so far by all the participants here. As noted in , I did some manual tests with partitioning. And having unaligned partitions really degrades performance a lot (reducing mkfs throughput from 60mb/s to 10mb/s). But as said, we don't have unaligned partitions when using Azure Dynamic Disk actually don't have any partitions at all as we create the FS directly on the block device (e.g. /dev/sdc instead of /dev/sdc1). The real issue here is that lazy inode table initialization is disabled for mkfs.ext4.

The reason for disabling it can be found here: The assumption (BLKDISCARD+BLKDISCARDZEROES support) made in this commit is not true for all cloud providers, e.g. Azure and at least for AWS magnetic disks. I did some calculations to figure out how much zeroes must be written by mkfs.ext4 to initialize the inode tables for a 1TB disk. If you check "tune2fs -l /dev/sdX" you'll see that there are inodes with a size of 256 bytes allocated for a 1TB file system. This gives a total of 16G! These 16G of zeroes are written by mkfs.ext4 when lazy init is disabled, in a very very naive way. Writing 16G of zeroes already takes 4:30 minutes when you have a throughput of 60mb/s. Now if there is something that throttles the troughput (and it looks like kubelet does this somehow), it gets even worse. And that's not all, while this is written do disk, the complete system suffers on low memory due to all of it being used for write caching/buffering. If other processes want to allocate memory in this situation, they're blocked until enough from the buffers is flushed to disk. Please check It fixes all this by simply re-enabling lazy inode table initialization. I also tried to describe why this change should not affect cloud providers that previously thought that disabling lazy init would be an improvement.
I don't believe the partition alignment issue would affect cases where the disk is unpartitioned (i.e. using the entire disk, e.g. ). I suspect that any use of an entire raw device would be properly aligned (any filesystem that starts on sector 0 of the hard disk would be properly aligned), but that's a hunch and nothing more. : I think your assessment is on target — the problem we (Cloud Foundry) are seeing is caused by a partition misalignment, but it seems that the problem described in this thread has a different cause. We currently use a lazy inode table initialization, so we're covered on that front:

According to : While the is exactly the opposite. This significantly slows down large ext4 volume creation.
This might be worth bringing up at the storage f2f coming up.I am going to add to the agenda. This is an interesting approach to what I think may be a few larger issues. Questions and thoughts: What is a good SLO for mkfs? If the SLO is going to be violated due to size should k8s do something or should the user pre-format? There are tradeoffs for initting the filesystem lazy or not. Supporting all the options for a user may require an API change. Is this a persistent volume only thing?
I may miss something, but on a random GCE node with 500GiB magnetic(!) disk and Fedora 24 takes around 2 seconds. ext3 is a different story, it takes 74 seconds.
I cannot imagine any reason why anyone would be installing ext2/ext3 on a 500GB drive. If they do they should be expecting things to be slow. In any case knowing what the SLO is is still not a crazy idea.
This is probably not a big problem with GCE, since GCE PD's IOPS is per GB. This is mostly a problem for EBS/Cinder/Azure, since the IOPS per volume is fixed.
If we're going to do this at all, we should really think hard about this problem top-to-bottom. Ext3 vs ext4 vs other FS? Myriad filesystem options, etc.
I got bitten by this when implementing e2e tests for all supported storage class options on AWS. "Cold HDD" volume type has max. 250 IOPS and the smallest one is 500 GB. It takes 3-4 minutes just to format the volume, which slows down the test quite considerably. Was there any conclusion on f2f meeting? was in commit , which claims to be rewritten to Go. Original script not exist any more on the link above. Was there some smart idea in original ?
see some inconclusive investigation at
Please take a look at I encountered not only performance issue due to lazyitableinit=0, but also oom situations and thus killed processes. Due to the amount of data written to the disk, all free memory gets filled with dirty buffer cache.


Question: restoreGeneration := pdb.Generation pdb.Generation = oldPdb.Generation <del> if !reflect.DeepEqual(pdb, oldPdb) { </del> <ins> if !reflect.DeepEqual(pdb.Spec, oldPdb.Spec) { </ins> allErrs = append(allErrs, field.Forbidden(field.NewPath("spec"), "updates to poddisruptionbudget spec are forbidden.")) } allErrs = append(allErrs, ValidatePodDisruptionBudgetStatus(pdb.Status, field.NewPath("status"))...)

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: When trying to forcefully update using command segmentatio voliation happens. What you expected to happen: podDistributionBudget to be replaced How to reproduce it (as minimally and precisely as possible): create : Create : Forcefull update (replace) : Anything else we need to know?: The same error happens on kubectl version 1.6 or 1.7. Manually deleting and recreating works. Environment: Kubernetes version (use ): 1.7 Cloud provider or hardware configuration**: gke OS (e.g. from /etc/os-release): Debian GNU/Linux 7 (wheezy) Kernel (e.g. ): 3.16.0-4-amd64
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <bre.g., for API Machinery<br(2) specifying the label manually: <bre.g., for sig/scalability<br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
/sig cli
This issue not only applies to PBP's but NetworkPolicy (and perhaps others).


Question: IPVSProxyMode = "ipvs" ) <del> // Sets of IPVS required kernel modules. var ipvsModules = []string{ "ip_vs", "ip_vs_rr", "ip_vs_wrr", "ip_vs_sh", "nf_conntrack_ipv4", } </del> <ins> // IPVS required kernel modules. const ( // ModIPVS is the kernel module "ip_vs" ModIPVS string = "ip_vs" // ModIPVSRR is the kernel module "ip_vs_rr" ModIPVSRR string = "ip_vs_rr" // ModIPVSWRR is the kernel module "ip_vs_wrr" ModIPVSWRR string = "ip_vs_wrr" // ModIPVSSH is the kernel module "ip_vs_sh" ModIPVSSH string = "ip_vs_sh" // ModNfConntrackIPV4 is the module "nf_conntrack_ipv4" ModNfConntrackIPV4 string = "nf_conntrack_ipv4" // ModNfConntrack is the kernel module "nf_conntrack" ModNfConntrack string = "nf_conntrack" ) </ins> // Equal check the equality of virtual server. // We don't use struct == since it doesn't work because of slice.

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--kube-proxy cannot use ipvs mode on Linux 4.19 due to a check for the module, which was changed to (). uses to check for : It should instead accept on Linux 4.19 or above. What happened: kube-proxy fell back from ipvs mode to iptables mode. What you expected to happen: kube-proxy using ipvs. How to reproduce it (as minimally and precisely as possible): Launch a cluster on an image using Linux 4.19 or above with kube-proxy using ipvs mode (see ). Environment: Kubernetes version (use ): All Kernel (e.g. ): 4.19 <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
/sig network
/area ipvs /assign /cc
Okay, so should check {nfconntrackipv4 | nf_conntrack} and expect load at least one module?
I think so
Please just go ahead :)


Question: sleep 1 done <del> # Uncordon the node. echo "== Uncordon ${instance}. == " >&2 local uncordon_rc "${KUBE_ROOT}/cluster/kubectl.sh" uncordon "${instance}"  && uncordon_rc=$? || uncordon_rc=$? if [[ "${uncordon_rc}" != 0 ]]; then echo "== FAILED to uncordon ${instance} ==" return ${uncordon_rc} fi </del> # Wait for the node to have Ready=True. echo "== Waiting for ${instance} to become ready. ==" >&2 while true; do

https://k8s- /kind bug /priority failing-test /priority important-soon /sig testing /milestone v1.11 cc /assign for triage
GitHub didn't allow me to assign the following users: for, triage. Note that only and repo collaborators can be assigned. <detailsIn response to : ## Failing Job https://k8s- /priority failing-test /priority important-soon /sig testing /milestone v1.11 Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I don't think this is sig-testing. Looks like , , /unassign /remove-sig testing
/sig cluster-lifecycle /assign can you see if this is a cluster lifecycle issue please?
Chiming in for cluster-lifecycle, this is weird. Somehow the nodes are being put back into a cordoned state after the upgrade. The upgrade checks that after a node is upgraded, it becomes both ready and schedulable. Here's an example from the linked failure: But just a few lines below that, we can see the the cluster validation after the upgrade shows that 2 nodes are still unschedulable. Does someone from the node team know why this might be happening? Is it a race condition? It worked for one node, but not two in this case. In other runs, I see it working for two nodes and only leaving one unschedulable. cc
/cc can either of you please help investigate the node side of this issue to unblock This is currently on of the few tests failing our sig-release upgrade blocking suite. If this is not fixed we will have to delay beta for 1.11 which is slated for 5/15.
/cc
cc
/cc I am also running into this issue when trying to add a new upgrade test. First, I think the reason that the test log said the node was ready is because the code is looking at a wrong field for SchedulingDisabled.

The unschedulable field is a field in not a node status condition, so I think we should instead use: local cordoned={KUBE_ROOT}" get node "${instance}" --output='jsonpath=="{.spec.unschedulable}"' I am also puzzled on why recreating instance just rebooted the node. According to 'gcloud instance-groups managed recreate-instances' doc, that command should delete the instance first and create a new one. For the passed cluster upgrade test, that seems the case, but for the failed ones, I saw node was just rebooted and so still got the old unschedulable state from the API server. While trying to figure out why recreating instance just rebooted node sometimes, would it make sense we explicitly use 'gcloud instance-groups managed delete-instances' and 'gcloud instance-groups managed create-instances' ?
We use recreate-instances to maintain the node identity and name. How are you determining that it just rebooted and not recreated? Did the nodes stop recreating their node objects on reboot and no longer clear out the field? Is this a consequence of ? Since is no longer unique across , the node object doesn't get recreated? Would that mean the upgrade script has to uncordon the node now?
I think theory at is right. After was merged, .spec.externalID is mapping to spec.instanceID which wouldn't be unique after recreate. In this case, Kubelet won't delete old node object and recreate new one. I think this is what we want: we want to stop the node to delete and create itself, we want to stop the node to mess up its taints and labels. Instead, we want the controller(s) to explicitly set the taints and labels to a node. also updated upgrade script to use boot id to detect the new boot or recreation: In this very case, we should update upgrade script to explicitly clear the unschedulable field once new boot id was detected.
Thanks, I'll work on a PR and send it out today.

From my manual run, I ssh-ed into the unschedulable node after the test and saw the kubelet log contains multiple restarts and logged the event that the node was previously registered and then later changed to NodeNotSchedulable based on the spec got from the API server: 1754: May 08 18:03:18 e2e-test-jiayingz-minion-group-ssbv kubelet[1073]: I0508 18:03:18. 1073 ] Node e2e-test-jiayingz-minion-group-ssbv was previously registered 1755: May 08 18:03:18 e2e-test-jiayingz-minion-group-ssbv kubelet[1073]: I0508 18:03:18. 1073 ] Successfully registered node e2e-test-jiayingz-minion-group-ssbv ... 1789: May 08 18:03:18 e2e-test-jiayingz-minion-group-ssbv kubelet[1073]: I0508 18:03:18. 1073 ] Event(v1.ObjectReference{Kind:"Node", Namespace:"", Name:"e2e-test-jiayingz-minion-group-ssbv", UID:"e2e-test-jiayingz-minion-group-ssbv", APIVersion:"", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'NodeNotSchedulable' Node e2e-test-jiayingz-minion-group-ssbv status is now: NodeNotSchedulable
That fits with the current theory. The instance is being recreated, but the kubelet is no longer re-registering itself since the externalID is the same.
/cc
was sufficient to fix this issue or are we also waiting on to b merged? I am still waiting for the latest run to verify if the issue is fixed.
/reopen There's a bug with my fix. It fails if the uncordon command happens in the window where the node object doesn't exist. I will have a fix out shortly.
you can't re-open an issue/PR unless you authored it or you are assigned to it. <detailsIn response to : /reopen There's a bug with my fix. It fails if the uncordon command happens in the window where the node object doesn't exist.

I will have a fix out shortly. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/reopen
Note that this bug is flaky. It doesn't happen 100% of the time. Looks like we got 100% pass in the latest run: https://k8s- Here's an upgrade test that's hitting this flake: https://k8s-
cc Please see my comments at for the change Before the change, when instance is recreated, kubelet will delete the node object and a new one will be created. Now, kubelet will not delete the old one if the node name is the same, I think. This might give some trouble for volumes because the old node will contain the outdated volumesAreAttached information. But we do sync up with the cloud provider and update this information eventually. I am not sure whether this old object might cause other problems or not.
Can you move that to another issue? It is a legitimate bug that the upgrade logic should not rely on kubelet recreating the node object. I agree that no longer doing that may lead to other issues, but that should be a separate discussion.
I see the test failing in the last few runs. Is it a different issue plaguing the test now? https://k8s-
/reopen
[MILESTONENOTIFIER] Milestone Issue Needs Approval Action required: This issue must have the label applied by a SIG maintainer. If the label is not applied within 1 day, the issue will be moved out of the v1.11 milestone. <details<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
Here are the since the last failure.
Spoke to offline and SkewTest is a overall for the second half of the suite. So if the other tests in the job pass, this one should turn the corner. so closing this bug for now.


Question: if len(tc.plugin) != 0 { c.AuthProvider = &clientcmdapi.AuthProviderConfig{Name: tc.plugin} } <del> tConfig, err := c.transportConfig() </del> <ins> tConfig, err := c.TransportConfig() </ins> if err != nil { // Unknown/bad plugins are expected to fail here. if !tc.expectErr {

https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite} Previous issues for this test:
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
We understand the problem here - I'm working on the fix.
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}


Question: return p.Status.Phase == v1.PodSucceeded })) failed += int32(countValidPodsWithFilter(job, pods, uncounted.Failed(), func(p *v1.Pod) bool { <del> // Counting deleted Pods as failures to account for orphan Pods that never // have a chance to reach the Failed phase. return p.Status.Phase == v1.PodFailed || (p.DeletionTimestamp != nil && p.Status.Phase != v1.PodSucceeded) </del> <ins> if p.Status.Phase == v1.PodFailed { return true } // When tracking with finalizers: counting deleted Pods as failures to // account for orphan Pods that never have a chance to reach the Failed // phase. return uncounted != nil && p.DeletionTimestamp != nil && p.Status.Phase != v1.PodSucceeded </ins> })) return succeeded, failed }

Since last Friday scalability test become red/very flaky Affected jobs: Those failure are related to failed jobs. Sample test output: Working hypothesis is that code change from , but this requires confirmation.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scalability
This particular regression is purely Job-related (we only see problems with batchv1.Job objects). /sig apps /priority critical-urgent - FYI /milestone v1.22
/assign
After internal testing it seems that can be a fix. Added big scale presubmits to


Question: visitedNamespaces.Insert(info.Namespace) } <ins> // Add change-cause annotation to resource info if it should be recorded if cmdutil.ShouldRecord(cmd, info) { recordInObj := info.VersionedObject if info.VersionedObject == nil { recordInObj = info.Object } if err := cmdutil.RecordChangeCause(recordInObj, f.Command(cmd, false)); err != nil { glog.V(4).Infof("error recording current command: %v", err) } } </ins> // Get the modified configuration of the object. Embed the result // as an annotation in the modified configuration, so that it will appear // in the patch sent to the server.

What happened: What you expected to happen: DS history revision 2 should have change cause from the second command "kubectl apply -f ds-2.yaml --record", not the first /assign


Question: }) } <ins> func (m *MockedFakeELB) AddTags(input *elb.AddTagsInput) (*elb.AddTagsOutput, error) { args := m.Called(input) return args.Get(0).(*elb.AddTagsOutput), nil } </ins> func TestReadAWSCloudConfig(t *testing.T) { tests := []struct { name string

BUG REPORT: /kind bug /sig aws What happened: Adding this annotation to an existing service: did not trigger an update on AWS ELB (no new tags). What you expected to happen: Expected to see those 2 new tags under the existing ELB: How to reproduce it (as minimally and precisely as possible): a service without annotation the annotation tags on AWS Anything else we need to know?: Creating a new service WITH the annotation will create the expected tags on AWS. Environment: Kubernetes version (use ): 1.7.9 Cloud provider or hardware configuration**: AWS OS (e.g. from /etc/os-release): Debian GNU/Linux 8 (jessie) Kernel (e.g. ): Linux ip-xxxxx 4.4.78-k8s SMP Fri Jul 28 01:28:39 UTC 2017 x86_64 GNU/Linux Install tools: Others: Using Kops
+1
+1
+1
+1
+1
I've done some work on a fix for this, will try and get a PR submitted later today.


Question: tryFunc(cleanup) tryFunc(cancelLogging) ginkgo.By(fmt.Sprintf("deleting the driver namespace: %s", ns2)) <del> tryFunc(deleteNamespaceFunc(f.ClientSet, ns2, framework.DefaultNamespaceDeletionTimeout)) </del> <ins> tryFunc(func() { f.DeleteNamespace(ns2) }) </ins> // cleanup function has already ran and hence we don't need to run it again. // We do this as very last action because in-case defer(or AfterEach) races // with AfterSuite and test routine gets killed then this block still

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: ci-kubernetes-e2e-ubuntu-gce-containerd Which test(s) are flaking: [sig-storage] In-tree Volumes [Driver: gluster] [Testpattern: Pre-provisioned PV (default fs)] subPath should support file as subpath [LinuxOnly] Testgrid link: Reason for failure: Will look at it shortly, why it is flaking. Anything else we need to know: links to appreciatedlinks to specific failures in spyglass appreciated
/sig storage
The namespace terminating during test setup is super strange but seems like a test-only issue, not an issue with the functionality being tested. I'd recommend looking at the test, API server, and controller logs to figure out why the test namespace () was deleted and who deleted it
does this help ?
feels like a test only problem like said. We don't even have gluster in our CI/prow to test against... cc Michelle, Xing, Can you please confirm that this is not a blocker for 1.19.0?
Knowing the timeframe for when the test started and when that delete happened would be interesting to know. Possibilities I can see: something in the test timed out, and the cleanup happened in parallel with something in the test attempting to run multiple tests are reusing the same test context and namespace name at the same time (I can never keep straight how ginkgo parallel tests work)
This does not look like a blocker for 1.19.0.
Thanks /milestone v1.20
looking at the logs it seems to be the latter "same test context and namespace name at the same time", but we can deal with this for 1.19.1
cc This looks like another instance of "the same test namespace is being reused". I believe had a mitigation for that to try to make the namespaces more unique.
(btw, the test brings up a containerized gluster server so we are exercising gluster functionality)
Yeah, looking at the build log found that it is the same issue as The issue is still around the double deletion of namespace, i.e. we delete the test namespace during the test execution, but later on the call we delete the namespace again to ensure it truly get cleaned up.

But unfortunately, this namespace has been reused by another test case between this gap.
Some potential fix I can think of: deletion of the primary namespace twice is inevitable, maybe we should consider generate primary namespace based on a hashing of the test cases name? the 4 digits random namespace to 6, this should not solve the problem in nature but it will extremely decrease the chance it happens. using an in-memory namespace map to keep record of already generated namespaces for the whole test run and refuse to generate the same namespace again in the whole lifecycle of a prow job.
do you know if this gluster test is using your mitigation in ?
hash of the testcase name + random suffix (because tests can be run multiple times) seems like a good way to mitigate the chances of tests reusing the same namespace. I'm unsure if we know the test case name at the time of test namespace creation though. I think increasing the length of the random suffix is a quick and easy mitigation believe ginkgo to run in parallel, so this may not work
Yes it is. But my fix in can only reduce the chance of collision and probably improve readability of logs. But not to eliminate this issue in nature.
AFAICT does not affect intree code at all, because it only fixes namespace of driver. If we are seeing collision of primary namespace for intree code then it is a different problem.
on only deals with the collision between test namespace and driver namespace. And that fix is taking affect. failure is because a test namespace collides with another test namespace. And this is not related to in-tree code at all. Any test could potentially become flaky because their namespace can be deleted during test execution. It's just happens to be this intree test this time. The root cause for this problem is: We cleaned up the test namespace twice which gives it a small gap that another testcase can generate the same namespace and be affected by the second namespace deletion from the original testcase.
/assign I would like to give it a shot.
I've opened to address this issue.
/unassign /assign


Question: import ( "fmt" "strconv" "time" "k8s.io/api/core/v1" apierrs "k8s.io/apimachinery/pkg/api/errors" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" <ins> "k8s.io/apimachinery/pkg/util/rand" </ins> clientset "k8s.io/client-go/kubernetes" imageutils "k8s.io/kubernetes/test/utils/image"

Failing Job: These tests started flaking recently (they were consistently green for a long time): Mounted volume expand[Slow] Should verify mounted devices can be resized Volume expand [Slow] Verify if editing PVC allows resize Subpath [Volume type: gcePDPVC] should support restarting containers using file as subpath [Slow] Subpath [Volume type: gcePDPVC] should fail for new directories when readOnly specified in the volumeSource Subpath [Volume type: gcePDPVC] should fail if subpath with backstepping is outside the volume [Slow] Volumes PD should be mountable with ext4 Affected runs: https://k8s- https://k8s- https://k8s- /kind bug /sig storage /priority important-soon /milestone v1.12 /cc - Could you PTAL or delegate? Thanks.
Volume expansion tests have been fixed by
/assign can you investigate the remaining failures?
[MILESTONENOTIFIER] Milestone Issue Needs Approval Action required: This issue must have the label applied by a SIG maintainer. If the label is not applied within 4 days, the issue will be moved out of the v1.12 milestone. <details<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
Reopening this issue, as a bunch of storage-related e2es started flaking in our large-scale jobs recently: https://k8s- https://k8s- /reopen
I think the latest failures are caused by gce API rate limiting: /assign
It seems to only be failing on scalability test suites. I wonder if it's due to some different cidr configuration
/sig scalability
/kind failing-test
/status approved-for-milestone Friendly ping. This is affecting release-blocking scalability job.
hey what needs to be done here? by whom?
Could you comment on what needs to be done here and by whom?
can you guys take a look at the API quota issue?

This is happening consistently in the scale test environment
as Code Freeze is today, this needs to be of priority . Please triage! cc
It looks like the latest run on 9/1 has passed.
Marking closed as the fixes have been merged and the test runs are now passing /close
Closing this issue. <detailsIn response to : Marking closed as the fixes have been merged and the test runs are now passing /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: Namespace: ns, }, } <ins> // New pods can be created between the evictions by a replica set controller. // This can affect the PDB, and pods could potentially be evicted that shouldn't be. // To prevent this, there should always be a sufficient pod readiness delay (see initialDelaySeconds). </ins> err = cs.CoreV1().Pods(ns).EvictV1(ctx, e) if err == nil { evictedPods.Insert(pod.Name)

master-blocking: gce-ubuntu-master-containerd No response /sig apps
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign


Question: func init() { CmdNetexec.Flags().IntVar(&httpPort, "http-port", 8080, "HTTP Listen Port") <ins> CmdNetexec.Flags().StringVar(&certFile, "tls-cert-file", "", "File containing an x509 certificate for HTTPS. (CA cert, if any, concatenated after server cert)") CmdNetexec.Flags().StringVar(&privKeyFile, "tls-private-key-file", "", "File containing an x509 private key matching --tls-cert-file") </ins> CmdNetexec.Flags().IntVar(&udpPort, "udp-port", 8081, "UDP Listen Port") CmdNetexec.Flags().IntVar(&sctpPort, "sctp-port", -1, "SCTP Listen Port") }

<!-- Please only use this template for submitting enhancement requests --What would you like to be : The image used in E2E tests is used for testing a wide variety of OS-agnostic features. One of these is , which runs an HTTP server with various endpoints that can be used to validate things like request/response. It would be useful to allow for 's functionality to also support HTTPS so all of that logic can be tested for TLS-based traffic. Currently, can be run like: I'm proposing something along the lines of: The default certificate/private key being basically identical to what provides. Why is this needed: One use case is where the intent is to add HTTPS functionality to lifecycle hooks (postStart/preStop). There is currently no easy way to test this logic because the container which receives the HTTPS request (in this case, ) does not support HTTPS traffic.
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig testing


Question: } ipvsModules := utilipvs.GetRequiredIPVSModules(kernelVersion) <del> var bmods []string </del> <ins> var bmods, lmods []string </ins> // Find out loaded kernel modules. If this is a full static kernel it will try to verify if the module is compiled using /boot/config-KERNELVERSION modulesFile, err := os.Open("/proc/modules")

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When creating a new cluster using , panics with SIGSEGV on first run: Note that after the crash the pod restarts, and the second run is successful: and everything seems to be working as expected. What you expected to happen: starts successfully the first time. How to reproduce it (as minimally and precisely as possible): Create a new cluster using , with in mode. Anything else we need to know?: Environment: Kubernetes version (use ): 1.18.0 Cloud provider or hardware configuration: Hetzner VPS OS (e.g: ): Ubuntu 18.04.4 LTS Kernel (e.g. ): 5.3.0-45-generic (Ubuntu's default HWE kernel) Install tools: kubeadm Network plugin and version (if this is a network-related bug): Others: using CRI and mode Please let me know if you need any additional information.
/sig network
/assign
According to the panic output, the crashed code is For some reason, might be nil, we should always check it before using.
Do we understand WHY it is nil?
One more thing I've noticed while fiddling with the cluster is that if I remove the node and re-join it, starts just fine, so I'm guessing it has to do with the initial state of the server / ipvs.
That would be my guess too. And that sort of race is worth fixing :)
do you see anything in the logs in the form: ??
I think somewhere here we're failing to initialize the IPVS interface. We probably shouldn't ignore the error from also :grimacing:
Nope, I posted the (full) logs in the issue description.
Thanks, will look into it deeper
do you happen to have a start up script that loads the necessary IPVS modules or is it preloaded on your nodes? Wondering if there could be a race between when the modules get loaded and when proxy starts up. Either way we should log the error ().
Neither, but if you give me the module list I can try to load them before creating the cluster to see if I still get the panic.

For kernel <4.19: For = 4.19:
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/remove-triage unresolved
I can confirm that, if I explicitly preload the modules, starts successfully the first time.
According to this log , we can confirm second check success, missing modules loaded after first failed check. Because missing modules loaded but not put to


Question: // Note that ValidateOwnerReferences() will reject this patch if another // OwnerReference exists with controller=true. addControllerPatch := fmt.Sprintf( <del> `{ "metadata": { "ownerReferences": [{"apiVersion":"%s","kind":"%s","name":"%s","uid":"%s","controller":true,"blockOwnerDeletion":true}], "uid":"%s", "finalizers": ["foregroundDeletion"] } }`, </del> <ins> `{"metadata":{"ownerReferences":[{"apiVersion":"%s","kind":"%s","name":"%s","uid":"%s","controller":true,"blockOwnerDeletion":true}],"uid":"%s"}}`, </ins> m.controllerKind.GroupVersion(), m.controllerKind.Kind, m.controller.GetName(), m.controller.GetUID(), rs.UID) return m.rsControl.PatchReplicaSet(rs.Namespace, rs.Name, []byte(addControllerPatch))

Seeing this in v1.6.1. a Deployment and a Service the Deployment add an ownerRef that points back to the Service delete the Service The Service and the Deployment are deleted, but the ReplicaSet and pods remain. This is neither Orphan nor Foreground/Background policy. This behavior should never happen. Note: if I make a delete call to the API with propagationPolicy set explicitly then things behave as expected, so this only occurs when the resource's default propagation policy is being used.
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
cc
This is expected. The default garbage collection policy for old controller types (rc, rs, deployment) etc. is orphaning, for backward compatibility. And when doing BackgroundGC, the GC controller doesn't set the deleteOptions.propagationPolicy. We met a similar problem for deleting deployment -RS -pods. Because that is a very common use case, we made a special fix in As you noticed, if you set deleteOptions.propagationPolicy to Orphan or ForegroundGC, it works as you expected.
that seems wrong. if I'm using the garbage collection feature to propagate (whether a user specified it or it was a default of a parent resource), I expect GC to set propagation policy.
rather than a one-off fix in the RS controller, why not address the break in the propagation chain in the GC controller?
In the backgroundGC, the owner is deleted immediately, so unlike orphaning or the foregroundGC, there is no finalizer left for the GC controller to determine the intention of the user. In the absence of the user's intention, the GC controller doesn't set deleteOptions.PropagationPolicy when deleting dependents.
If it's not orphaning, and it's not foregroundGC, the only logical option that remains is backgroundGC
There is another case, user might intentionally choose not to set DeleteOptions.Propagation policy and wants to respect the existing finalizers of the dependents or the default GCstrategy defined by the REST storage of the dependents.

except that it isn't... if I delete resource X and specify no propagation policy, that resource strategy's default propagation policy should take effect just as if the user had specified it.
Add CronJobs to the list of affected resources, since you have CronJob-Job-Pod chain. :+1: to what Jordan said just above.

BUG REPORT: <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version: 1.6.1 Environment: Hardware configuration: 2 identical machines, with i7-2600K CPUs and 4GB RAM. OS: CentOS 7 Kernel: 3.10.0-514.10.2.el7.x86_64 Install tools: KubeADM What happened: Deleting a deployment via the API with the DeleteOptions settings or deletes the Deployment and the replica set but NOT the pods. Deleting it with the DeleteOptions setting works as intended. What you expected to happen: The replica set deletion should propagate to the pods. I've read that the garbage collection should work for Deployments in 1.6. How to reproduce it: Anything else we need to know: I am using the and at the moment it only supports the orphanDependants option, since it's still at api version 1.5.
cc:
We discussed the problem in I'll change the GC to make sure it's cascading.
Sent
Any progress on this?
Still trying to figure out a sane fix:
I've encountered the same issue. My pods are still running even though I deleted the deployment.
lets come up with a fix for 1.7. This should work from the first time we introduced cascading deletion in Deployments.
Yes, we should. Do you have any comments on
replied. Let's move this forward. If the fix is small, I am going to backport it to 1.6.
Offline discussion with we punt this to v1.8.
Discussed with if we can get the PR fixed soon, it can still go in 1.7. It's a real bug fix.
Yes, we need to fix this for 1.7 and also backport to 1.6.

Should this be targeting 1.7?
yes, the fix that went in in is going to cause us data issues we don't want
Are you working on this? What's the timeline to resolution?
Based on above comment, looks like actually introduced a new regression. Am I right?
In , I reverted what's introduced in .
- is working on it
I approved this for 1.7 based on comment at We should revert , but are we ok with since it also introducing a new behavior. I will leave to decide.
replaces with a fix that addresses this issue systemically. While this bug did exist in 1.6.x, more things rely on GC in 1.7 than in 1.6, and without that fix, GC leaves more resources vulnerable to unintentional orphaning. Since the issue is actually with general GC behavior, should weigh in.


Question: if !options.Local { helper := resource.NewHelper(client, mapping) <del> patchedObject, err := helper.Patch(namespace, name, patchType, patchBytes) </del> <ins> _, err := helper.Patch(namespace, name, patchType, patchBytes) </ins> if err != nil { return err } if cmdutil.ShouldRecord(cmd, info) { <del> if err := cmdutil.RecordChangeCause(patchedObject, f.Command()); err == nil { // don't return an error on failure.  The patch itself succeeded, its only the hint for that change that failed // don't bother checking for failures of this replace, because a failure to indicate the hint doesn't fail the command // also, don't force the replacement.  If the replacement fails on a resourceVersion conflict, then it means this // record hint is likely to be invalid anyway, so avoid the bad hint resource.NewHelper(client, mapping).Replace(namespace, name, false, patchedObject) </del> <ins> // don't return an error on failure.  The patch itself succeeded, its only the hint for that change that failed // don't bother checking for failures of this replace, because a failure to indicate the hint doesn't fail the command // also, don't force the replacement.  If the replacement fails on a resourceVersion conflict, then it means this // record hint is likely to be invalid anyway, so avoid the bad hint patch, err := cmdutil.ChangeResourcePatch(info, f.Command()) if err == nil { helper.Patch(info.Namespace, info.Name, api.StrategicMergePatchType, patch) </ins> } } count++

Hi, Deployment patching with doesn't show up in the deployment rollout history: first: after: Is it a bug? Should it show up? Thanks, Peter
cc
Which kubernetes and kubectl releases? Did the patch actually trigger a rollout?
I also run the command from CI with the latest . So the problem affects the v1.2.3 kubectl as well
We should also fix the way we sort revisions - 10 should be after 9, not 1.
Can confirm this. The issue is that the patch isn't updating , so the rollout history will just copy the value from the previous version. If you started with you get that for the patches. In my case I have a different line repeated. Builds 7 and 8 were built with different . I'm not sure why the patch worked once. Command I am calling is:
If no one is working on patch - i can try to make one.
I'll like to work on this issue if you haven't started yet
Actually never mind I can look at something else :)
Hi, i see that you are the author of the following comment. I think that currently every valid change cause is discarded because of version conflict. Maybe you have some information why it happens?
command instead generates a patch to update then applies that.
It was the same for patch, but for some reason was changed.
The intent was two fold. First, prevent the change-cause from ever failing patch the CLI command. change-cause isn't authoritative and the patch took hold, so a non-zero return code is inappropriate. Second, if the resource version change post-patch, before the update, the change-cause is unknown, something else touched it after the fact. A command that issues a instead of an could very easily break the history since patches will attempt to reapply themselves regardless of resource version in the face of conflicts. The object returned from the patch operation is the one that we use to avoid hitting a resource conflict with ourselves.

Truth be told, I don't feel very strongly about getting the history right since it can be changed anyone with write access to the object, includes secrets that should be elided (--token for instance), ignores environment variables which affect outcome, ignores the actual path used to locate the binary you're running, and references paths that are only good from one location on one machine, so if you want to switch it back to a second patch, I don't mind as long as a failure to update the object doesn't cause a non-zero return from the command.
Could there be a race between the call and the deployment updating itself to action the changes? On 19 May 2016 9:47 pm, "David Eads" wrote:


Question: # This tarball is used by GCI, Ubuntu Trusty, and CoreOS. KUBE_MANIFESTS_TAR= <del> if [[ "${KUBE_OS_DISTRIBUTION:-}" == "trusty" || "${KUBE_OS_DISTRIBUTION:-}" == "gci" || "${KUBE_OS_DISTRIBUTION:-}" == "coreos" ]]; then </del> <ins> if [[ "${OS_DISTRIBUTION:-}" == "trusty" || "${OS_DISTRIBUTION:-}" == "gci" || "${OS_DISTRIBUTION:-}" == "coreos" ]]; then </ins> KUBE_MANIFESTS_TAR="${KUBE_ROOT}/server/kubernetes-manifests.tar.gz" if [[ ! -f "${KUBE_MANIFESTS_TAR}" ]]; then KUBE_MANIFESTS_TAR="${KUBE_ROOT}/_output/release-tars/kubernetes-manifests.tar.gz"

We recently switched GKE's master node to GCI image, but not GCE master node. This discrepancy causes a lot of issues for us, especially some prs merged might work on GCE, but totally break GKE. We should get rid of that difference. cc/
/cc (two build cops who have recently suffered the effects of this).
The master image for test is defined here: We want to switch all test jobs? Or just one particular job, e.g., k8s ci testing k8s master head? I am afraid that simply replacing the image in that script will affect all test jobs
I think this is a bad plan unless we continue to test container VM start up somehow. We can't break that while people still use it!
An alternative: Set up another jenkins job dedicated for the purpose and mark it as commit queue blocker.
It may be better to just switch GKE masters back to containerVM for now, then redo the sequence (i.e. switch GCE to GCI, then GKE master to GCI).
if we switch both the test and non-test default from cvm -gci then starting with the 1.3 release people won't be using it (by default), so testing becomes less important.
I don't think we have to. The GKE CI job is already a CQ blocker, but it runs after submit so bad PRs still get merged (though they get identified quickly). What we need is a PR builder that runs tests with GCI, so we can block bad changes in the first place.
Definition of PR builers: cc/
Yes. It could be even simpler, it just has to start up a cluster and tear it down. Just run like 1 fast test.
We've got to do something though, the queue has been blocked for like 3 hours on GKE tests not passing.
I'd still like to run the e2e tests as some tests make assumptions of the OS distro and thus fail on GCI. Can we duplicate (that runs gce-e2e on every PR), replacing ContainerVM with GCI as the master and (or) node image?
It's a bit of a pain to do this, since that job also does a full build and it feels like a major waste to build each PR again. The Jenkins agents probably can't handle that much load, anyway.

at some point we probably need to bite the bullet and set up pipelines so we only build kubernetes once, rather than separately in the unit/integration, e2e, and node e2e jobs. I have no idea how well any of that interacts with the pull request plugin, of course.
Thanks for the information . So looks like the first alternative, running tests with GCI/GKE and block bad changes in the first place, is hard to get done due to some infrastructure challenges (as mentioned in ). It's not impossible though, just not prioritized right now. The second alternative would be, keep running the post-submit tests (the GKE e2e tests), and fix issues as they appear, which is not too bad imo since they run every 30 min and block submit queue on failure. At the end of the day, you cannot prevent 100% of breakages in presubmit tests, and that's why we have to have postsubmit tests to complement.
I think start another jenkins job as one more pr builder, which have the master node on GCI image and rest of nodes on ContainerVM is the simplest thing unblock us. The newly created job should only run the basic test suites. After 1.3, I suggested we do
No, this is not acceptable! we've been blocked for 10 out of the last 24 hours due to this! :)
I agree that we need to catch this before merging instead of after merging. The question is how best to do so.
The problem is that the submit-queue puts them in at odd hours, and we don't expect OSS oncall to be watching at 2 AM.
Well, the flip side is you don't have to run all tests (on all providers) so that all your PRs get merged faster. I am not saying I prefer postsubmit tests, it just something you have to face, no matter its gke tests or gce tests. I would prefer that too.
I think the solution to this is to run more tests when the PR is created, but then when the submit queue asks for a retest before merging only run a smoke test. I think that's a good tradeoff.
That's a good topic to talk and I have some ideas about that too, but I'd stop dragging the focus too far.

:smile:
I don't see any compelling reasons to have a different master image-type for GCE and GKE -- there are strong benefits to them being identical, as we've observed. We want 1.3+ to run GCI for GKE, therefore I propose we switch both GCE and GKE to use GCI for the master on all 1.3+ tests by default, including PR and post-submit. If there is a testing gap in that situation, because we think it's important to support containerVM for 1.3+ masters on GCE (but not GKE), then add coverage for that gap. I don't have enough context to know whether that's pre-submit or post-submit or both.
(To append: I think post-submit coverage is horrible and would support a P0 of any test tiger team and/or code-yellow exit criteria being that we have pre-submit coverage of all the various test matrix points. I realize that'll be expensive in both time and machine resources, but it'll be less expensive than our current approach and I suspect generate significant velocity benefits.)
This is a decent idea.
is out for vacation, could you please pick up this one? I think we agreed to have a separate jenkins job as one more pr builder, which have the master node on GCI image and rest of nodes on ContainerVM. The newly created job should only run the basic test suites. If you need some help, please ask
After talking with and we came to the following agreement: will switch the default base image for master nodes running on GCE from debian container-vm to GCI. will add a jenkins job that runs periodically (post-submit) that will continue to test master nodes running the debian container-vm, initially running against the master branch. will pin this job to the 1.3 release branch once it is cut, and no longer run it against the master branch. will delete the new jenkins job once we have full confidence that we will not roll back the master image from GCI to the debian container-vm, sometime between now and the 1.4 release.
I am going to work on a PR for no. 1. cc/
no. 2 is being in
The build broke again because of this:


Question: # Try to match the "git describe" output to a regex to try to extract # the "major" and "minor" versions and whether this is the exact tagged # version or whether the tree is between two tagged versions. <del> if [[ "${KUBE_GIT_VERSION}" =~ ^v([0-9]+).([0-9]+)([.-].*)?$ ]]; then </del> <ins> if [[ "${KUBE_GIT_VERSION}" =~ ^v([0-9]+).([0-9]+)(.[0-9]+)?([-].*)?$ ]]; then </ins> KUBE_GIT_MAJOR=${BASH_REMATCH[1]} KUBE_GIT_MINOR=${BASH_REMATCH[2]} <del> if [[ -n "${BASH_REMATCH[3]}" ]]; then </del> <ins> if [[ -n "${BASH_REMATCH[4]}" ]]; then </ins> KUBE_GIT_MINOR+="+" fi fi

This says 0.6+, which isn't really right, since 0.6+ indicates commits on top of the latest tag. But that isn't the case here. We are exactly on the latest tag...


Question: const ( KubeletSubsystem             = "kubelet" <ins> NodeNameKey                  = "node_name" NodeLabelKey                 = "node" </ins> PodWorkerLatencyKey          = "pod_worker_latency_microseconds" PodStartLatencyKey           = "pod_start_latency_microseconds" CgroupManagerOperationsKey   = "cgroup_manager_latency_microseconds"

What would you like to be : It would be useful if the kubelet exported a metric that contained its own name as a label, for instance: . This name should be consistent with the output of . Why is this needed: This would allow us to join the kubelet metrics against metrics exported by kube-state-metrics. An example of this can be seen in We've arranged in our cluster for the labels to be set to the node's name, but this is controlled by Prometheus and depends on your scrape & relabelling config. As such, in general we can't rely on this to be the case - for instance, with the prometheus-operator the label is a IP:Port combo. /cc
/sig instrumentation
This metric as it is right now doesn’t conform to the instrumentation guidelines, so yes we should changes this as part of the metrics overhaul targeted for 1.14.
you mean to add a label in kubelet metrics ?
precisely.
/assign


Question: return nil } glog.Infof("Setting Proxy IP to %v", hostIP) <del> return CreateProxier(loadBalancer, listenIP, iptables, hostIP) </del> <ins> return createProxier(loadBalancer, listenIP, iptables, hostIP) </ins> } <del> func CreateProxier(loadBalancer LoadBalancer, listenIP net.IP, iptables iptables.Interface, hostIP net.IP) *Proxier { </del> <ins> func createProxier(loadBalancer LoadBalancer, listenIP net.IP, iptables iptables.Interface, hostIP net.IP) *Proxier { </ins> glog.Infof("Initializing iptables") // Clean up old messes.  Ignore erors. iptablesDeleteOld(iptables)

If another process is altering iptables rules at the time k8s attempts to add iptables rules it will fail with a locking error. k8s should use -w and retry after a delay. Below, an external process (openshift-sdn-node) modifies iptables at roughly the same time and k8s fails without retrying. Related to
Sorry, I misunderstood the -w flag, with it retry logic wouldn't be necessary as it'd wait. Perhaps simply retrying is a better solution as -w flag may not exist everywhere we'd like.
Looks like -w was to iptables 1.4.20(August 2013).
Yeah, we have to be pretty back-compatible with iptables. We always retry after a few seconds - is there really a bug here? , Scott Dodson wrote:
It definitely seems like there is some sort of a problem. We see this randomly on system startup. In our environment OpenShift is launching kubernetes around the same time other systemd units are modifying the firewall config. If one of these others units has the lock kubernetes will report the error Scott mentions and then will never create the required nat'ing for services. Here's how to reproduce this: At this point if you run you will see the exact same error kubernetes reports. It's probably not obvious why you have to set the breakpoint at 257 but if you set the breakpoint at 260 and then step through the code it appears that the compiler has optimized lines 263 and 257 to be the same. In any case running will verify if you have the lock or not. Once you've verified you have the lock just restart OpenShift (running kubernetes v0.14.1-582-gb12d75d): At this point you can press in gdb to continue the iptables invocation and release the lock. Afterwards I will see output from cadvisor every once in a while, however the will show the firewall is still blank: I initially restarted Kubernetes with a blank firewall at 11:21 and yet at 11:38 I still have nothing:
I probably should have made it more clear, I removed the lock by pressing 'c' in gdb to continue. That's when I started waiting. I'll update my comment to make that more clear.
So something for a holding the lock?

I am out of your office for the rest of this week, but maybe you can figure out - is there a stale process holding a lock? What kind of lock is it? Flock or semaphore or ... ? On Apr 29, 2015 8:44 AM, "Brenton Leanhardt" wrote:
What we're saying is that on startup it's fairly common for multiple systemd units to make firewall additions. If one of them grabs the lock (rarely for more than a few seconds) it will cause kubernetes to fail to initialize iptables and it will never fix itself. The lock only need be grabbed the microsecond kubernetes tries to initialize iptables for this bug to present itself. We know exactly which program in our environment was concurrently making iptables updates. The problem is that kubernetes doesn't appear to be retrying if it fails to initialize iptables.
On RHEL the lock is the xtables unix socket lock in iptables to allow multiple programs to update iptables concurrently. Other platforms may use a flock since that is the default upstream. To solve this kubernetes needs one more more of the following: pass the flag to as scott and the logs mention (possibly only available on some platforms). If kubernetes desides to wait for the lock it will need a timeout. retry the firewall initialization quickly and maybe backoff if it still can't get the lock It's probably worth noting that if iptables on a platform doesn't have iptables lock support concurrency is likely broken already in other ways for that platform.
It does retry all up tables ops. Unless it somehow gets pathologically phase locked with your agent, or there is some deeper bug. We have e2e tests that prove you can delete tables and it will recreate them. Are you saying that it never retries (did you set -v=4?) or that every retry fails? On RHEL the lock is the xtables unix socket lock in iptables to allow multiple programs to update iptables concurrently. Other platforms may use a flock since that is the default upstream. To solve this kubernetes needs one more more of the following: pass the -w flag to iptables as scott and the logs mention (possibly only available on some platforms). If kubernetes desides to wait for the lock it will need a timeout.

retry the firewall initialization quickly and maybe backoff if it still can't get the lock It's probably worth noting that if iptables on a platform doesn't have iptables lock support concurrency is likely broken already in other ways for that platform. — Reply to this email directly or view it on GitHub .
I can grab the lock for just the second that kubernetes needs it and then let it go. The kube-proxy nat'ing will never be rebuilt without intervention (restarting kubernetes). That seems like a bug to me and is what my original comment reproduces. In addition, once the environment is in this state it won't even pick up new service nat rules. Other nodes in my environment will pick them up but this one is dead. I highly suspect it has to do with the fact that iptables initialization is failing. We test all the time that once a system is up and running you can clean iptables and it will come back. This is a slightly different case.
This seems like a v1 blocker - we see this in every realistic environment we put the Kube-proxy in. Disagreement?
I still don't understand this issue. It is demonstrably true that the kube-proxy retries all of the iptables logic periodically. It's possible (given the mention of "won't even pick up new service nat rules") that this locking tickles a bug in kube-proxy, but I am having trouble reproducing it. Can you run your kube-proxy with -v=4, trigger the bug, and then take a look at the logs? If this is real, yeah it's a 1.0 blocker.
The problem is in the initialization process. If you hold the lock while the kube-proxy tries to initialize itself then it will be wedged forever. What you're showing is the kube-proxy can repair the iptables rules after the initialization code has run. The logs I pasted earlier show that after the initialization fails at it will never try again. I tend to run at loglevel 4 by default but I'll try to find time today to run through this again.
I have a fix coming - can you test the PR before commit?
I ran the exact same test as mentioned in my original comment.

While the debug lines have changed the end result is the same: grab the lock (at this point even will fail) start the kube-proxy release the lock (at this point works) wait.. try creating another service wait.. The rules never come back. Here are the logs:
Sure, I'll test the PR. I'm betting it will apply fairly cleanly on to my OpenShift codebase. If not I'll launch Kubernetes directly.
Many of those log lines are not part of upstream. In fact, what upstream has is a glog.Fatalf() which will kill kube-proxy, and when it restarts it will try again. Did you guys break it by not actually exiting?
Looks like we did. We need to put it into a restart loop. ----- Original Message -----
is teh fix, though I assert we should not need it, and if you need it this fix may not help you anyway. I am still OK to commit this.
Although NewProxier sucks because it doesn't return a usable error. I'll take the todo to clean up that part. ----- Original Message -----
hey, it logs its error - nothing wrong with that :) , Clayton Coleman wrote:
:)
Opened and will move the rest back to openshift.
I restarted docker and it worked then


Question: jig := e2eservice.NewTestJig(cs, ns, serviceName) ginkgo.By("getting the state of the sctp module on nodes") <del> nodes, err := e2enode.GetReadySchedulableNodes(cs) </del> <ins> nodes, err := e2enode.GetBoundedReadySchedulableNodes(cs, 2) </ins> framework.ExpectNoError(err) sctpLoadedAtStart := CheckSCTPModuleLoadedOnNodes(f, nodes)

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI -- - At least 08/08 sequentially performs operations that take ~20s on average. Example: For a 5000 nodes test this causes a major timeout. excludes tests with tag which SCTP had until was merged. This is why there is a shift in red cells in the testgrid after 08/15. We will add a pattern excluding until this issue is resolved. /assign /cc
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig network
/assign


Question: BlockVolume:                                 {Default: false, PreRelease: utilfeature.Alpha}, PVCProtection:                               {Default: false, PreRelease: utilfeature.Alpha}, ResourceLimitsPriorityFunction:              {Default: false, PreRelease: utilfeature.Alpha}, <ins> SupportIPVSProxyMode:                        {Default: false, PreRelease: utilfeature.Beta}, </ins> // inherited features from generic apiserver, relisted here to get a conflict if it is changed // unintentionally on either side:

There are some work to be done in the follow-up of , that is, help IPVS existing alpha. [x] GCE CI failures()[x] cross-node visit NodePort ()[x] hairpin issue ()[x] Using ipset doing snat and packet filtering and reduce iptables call().[x] automatically load kernel modules. (, )[x] creating e2e test suite for IPVS in upstream CI ().[x] ip cmd -netlink bind/unbind IP address to/from dummy device()[x] rsync to HEAD of iptables(, )[x] enable flush batch API()[x] Fix TODOs in codes ()[x] Bug report (, )[x] Default sync period()[x] IPV6 support(, and more...) /cc
/cc
ref ! It will generate annoy error info If unbind multi-ports on one service .
You're right. I think it's a nice finding. Can you fire a PR to fix it?
Sure, I'll make it soon.
I found it is treated as already bound if returned 2 in , but ref to the and of ip , it just means an error was reported by the kernel, such as "Operation not permitted". I prefer to use instead of command to manage ip address.
Actually, I have considered to use netlink before. I think what you said make sense, but can we discuss it in other thread? I mean, create a new issue and link it here? Thanks!
I opened a issue
on the subject :)
OOPS... I will fix that typo. Thanks
Wish to use ipset to replace KUBE-MARK-MASQ chain.
I find it's a reasonable suggestion and am trying to make it workable...
There are to ways to add IPVS CI job: , but doesn't support any kube-proxy config right now, so I had this pr , and also, need support it too, see this pr: kubernetes/kubernetes-anywhere build cluster. See this pr: kubernetes/test-infra The second way is much easier and faster, but I was told they gonna deprecated the , so in the long term, we still need and 's support. If anybody have better way or have some suggestion about this, please let me know, I'll be really appreciate, Thanks.
This one is done as both PRs are merged:
ACK!

/area ipvs
We are almost reaching the beta, except GCE CI failures, see The failure is the case that, visiting Cluster IP from Pod A, and the backend Pod B is in the same host with the Pod A. Pod A failed to get response from Pod B until timeout. I can reproduce the failures in my GCE cluster with kubent as network plugin, but it works fine in my local-up cluster and people has never reported the issue yet. and I are digging...
IPVS CI is green now.


Question: }, }, } <del> cycle := q.SchedulingCycle() go func() { <-moveReqChan q.AddUnschedulableIfNotPresent(unschedulablePod, cycle) wg.Done() }() } firstPod, _ := q.Pop() if !reflect.DeepEqual(&expectedPods[0], firstPod) { t.Errorf("Unexpected pod. Expected: %v, got: %v", &expectedPods[0], firstPod) </del> <ins> q.AddUnschedulableIfNotPresent(unschedulablePod, oldCycle) </ins> } <del> // close moveReqChan here to make sure q.AddUnschedulableIfNotPresent is called after another pod is popped close(moveReqChan) wg.Wait() // all other pods should be in active queue again </del> <ins> // Since there was a move request at the same cycle as "oldCycle", these pods // should be in the backoff queue. </ins> for i := 1; i < totalNum; i++ { <del> if _, exists, _ := q.activeQ.Get(newPodInfoNoTimestamp(&expectedPods[i])); !exists { t.Errorf("Expected %v to be added to activeQ.", expectedPods[i].Name) </del> <ins> if _, exists, _ := q.podBackoffQ.Get(newPodInfoNoTimestamp(&expectedPods[i])); !exists { t.Errorf("Expected %v to be added to podBackoffQ.", expectedPods[i].Name) </ins> } } }

We've noticed a strange behavior of the scheduler in GCE 5K node scalability tests. There were periods of time, lasting about 6s, when scheduler didn't schedule any new pods. It's ruining our 5s pod-startup-latency SLO. See the example below, pod density-latency-pod-2325-dgqrv created on 09:08:59, scheduler didn't schedule it until 09:09:06. An interesting pattern can be found in the around that time. Before 09:09:00 scheduler schedules ~5 pods / s. Then it goes silent for about 6s and at 09:09:06 it outputs about 30 "bound successfully" statements and continues scheduling pods with ~5 pods / s pace. We checked api-server logs for that silent period of time and it turned out that scheduler was doing something, it was constantly sending GET and PUT request for a metrics-server and heapster pod it was trying to schedule. ! We dug deeper and found that scheduler was actually trying to schedule these two pods for over 12h, below is the plot of number of calls per minute that scheduler issued to api-server for these two pods: ! As we can see there are long periods of time when scheduler issues over 1500 requests per minute. That seems strangely a lot given that the requests are only for the two pods. During the whole run scheduler issued over 400K GETs and over 200K PUTs for these two pods. ! We figured out that scheduler wasn't able to schedule these two pods because there was not enough resources in the cluster. While it's understandable that scheduler wasn't able to schedule the pods, I'd expect scheduler to not spam the system with so many retries, which, as we can see in this example, can potentially block / slow-down scheduling of other pods. I've increased the scheduler log verbosity level in , we should have more logs from the tomorrow's run. Will post them here once available.
/sig scheduling
Reiterating the mentions to trigger a notification: <detailsIn response to : /sig scheduling </details
/assign for triage
What is the version of your cluster? If you are using the HEAD, we have a back-off mechanism to the scheduler to delay rescheduling unschedulable pods.

It starts with 1 second delay and doubles it every time the pod is not schedulable up to 10 seconds.
It's from head, so apparently something doesn't work...
Yes, we're using head. The test run on Jan 21 and the cluster was built from head on the same day (commit to be more precise). As you can see in the logs it doesn't look like scheduler is using any back-off mechanism. Definitely not the one you described, it issues requests to api server much more often than 1 per second per pod. !
An update on more logs - we won't have more logs. We tried increasing scheduler's log level to v4 and it resulted in the log growing by few orders of magnitude and it was clogged with mostly useless stuff, e.g. it logged "Node gce-scale-cluster-minion-group-XXX is a potential node for preemption." over 250 000 000 times. We reverted the log level change in If more scheduler logs are needed to debug this we probably should add some new, more useful, ones.
cc/
/assign I can look into this next Tuesday
Friendly ping, any updates?
Whoops. Thanks for the ping. Log level v4 should not be needed. Although the log line you called out should probably get pushed into V9... First off, that's some really great debugging. Thanks for the detailed report. If the scheduler is getting into tight busy loops, that could in theory starve the scheduling thread, causing the observed 6 second pause in scheduling. The bug is in  A pod only gets marked for backoff if . And can increment very quickly in a high churn cluster. This would result in a pod being retried in a tight loop. One (unrelated) observation that stood out to me was this: This is being printed by the package and shows a trace with no steps logged to it, which means that we are hitting one of these four error cases in . Most likely, the PVC for those heapster & metrics-server pods are in a bad/unexpected state.
I think fixes the root problem here. If we eliminate the API server spam, the main scheduling thread hopefully will not get blocked for 6 second periods. Are you able to test with this patch?
Thanks Jonathan!

I've reserved a scale-testing slot for Thursday (Feb 21) to test the fix. Will update this issue once I have the results.
I reproduced the issue (non-schedulable heapster pod due to not enough resources) on a 5K node cluster. Then I run the load test, first on the version without the fix and the on the version with the fix. Below are the graphs of the number of scheduler requests (for the unschedulable pod) per minute. Without fix: ! With fix: ! As we can see on the graphs, the fix definitely helps, without it there were minutes were scheduler was calling api server up to 2K times per minute (2K QPM). With the fix it didn't exceed 100 QPM. But even with the fix the average QPM is ~60 (per single pod) which is 1 query per second per pod on average. is average 60 QPM (1 QPS) reasonable here or should we expect something less from the rate-limiting mechanism?
By default the scheduler rate limit is 20 qps for all scheduler operations. Default backoff delay for unschedulable pods is 1 second and goes exponentially higher every time a pod is retried, but does not exceed 10 seconds. This 10 second max backoff time also means that if a pod is not retried for 10 seconds, for example because there are many other pods in front of it in the queue, its backoff time is reset to 1 second. So the next time it is retried and is found unschedulable it will be backed off for 1 second. I am basically trying to say that 1 qps is not unreasonable. We also an optimization () that should lower qps for unschedulable pods further down. This last PR got merged only a few hours ago. With that PR merged, I would expect to see even lower qps as the scheduler will not update pod status every time it retries scheduling a pod.
Thanks Bobby! That makes sense. In that case I believe we can close this issue once merges.
/milestone v1.14 We're entering and this looks relevant to the 1.14 release. Please if I am incorrect
Thanks for fixing!


Question: } func (rh *realHistory) DeleteControllerRevision(revision *apps.ControllerRevision) error { <del> return rh.client.Apps().ControllerRevisions(revision.Namespace).Delete(revision.Name, nil) </del> <ins> return rh.client.AppsV1beta1().ControllerRevisions(revision.Namespace).Delete(revision.Name, nil) </ins> } func (rh *realHistory) AdoptControllerRevision(parent metav1.Object, parentKind schema.GroupVersionKind, revision *apps.ControllerRevision) (*apps.ControllerRevision, error) {

The statefulset controller and controller history implementations should be migrated off of the Apps client (deprecated) and onto the AppsV1BetaClient. Currently there is no user facing issue caused by the use of the deprecated resource, but we should maintain good code hygiene.
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list .
/assign kow3ns
/assign kow3ns
/cc
/assign


Question: - get - list - watch <ins> - apiGroups: ["apps"] resources: - deployments resourceNames: - metrics-server-v0.6.3 verbs: - get - patch - nonResourceURLs: - /metrics verbs: - get </ins> --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding

broke rbac permissions for pod nanny and it's now not able to scale up with number of nodes. We see in kube-apiserver logs: I tried to fix naively the problem by but it's incomplete as we still see: which means that we are missing more permissions (e.g. get deployments). Sample test run where we see it: At this stage I gave up. /sig instrumentation /cc podnany has access to required data to scale up metrics-server Create empty cluster, watch kube-apiserver logs for podnanny user agent Also you can test e2e by calling '/test pull-kubernetes-e2e-gce-100-performance' and seeing if metrics-server ever resizes. No response <detailsmaster </details<detailsgce </details<details</details<details</details<details</details<details</details
We incorrectly removed RBAC required by metrics-server-nanny. Fixing this should require adding it back. List of permission that should work I would also recommend 2 followups for metrics server project: Make k/k metrics server configuration based on metrics-server project autoscaling configuration. This would require making the autoscaling part of release, releasing the config for v0.6.3, updating k/k repo with it and documenting it as instructions for updating metrics server config in k/k repo. Add tests no errors in nanny used in metrics server autoscaling configuration.
/assign
cc
ok, let me take a look
yeah, I think We dropped some resources which are needed. Let me try to add them back
/triage accepted /assign


Question: } else if last == current { same++ } else if current < last { <del> Failf("Controller %s: Number of submitted pods dropped from %d to %d", last, current) </del> <ins> Failf("Controller %s: Number of submitted pods dropped from %d to %d", name, last, current) </ins> } if same >= failCount {

Currently the density tests are ok if all the pods come up eventually. We should verify there are no failure/error events while the pods are coming up. This could indicate issues dealing with the number of pods and should be investigated.
has been merged, and there's already an issue filed for followup actions, so I think we can close this.