Question: } iscsiTransport = extractTransportname(string(out)) bkpPortal := b.portals for _, tp := range bkpPortal { <ins> // Rescan sessions to discover newly mapped LUNs. Do not specify the interface when rescanning // to avoid establishing additional sessions to the same target. out, err := b.plugin.execCommand("iscsiadm", []string{"-m", "node", "-p", tp, "-T", b.iqn, "-R"}) if err != nil { glog.Errorf("iscsi: failed to rescan session with error: %s (%v)", string(out), err) } </ins> if iscsiTransport == "" { glog.Errorf("iscsi: could not find transport name in iface %s", b.iface) return errors.New(fmt.Sprintf("Could not parse iface file for %s", b.iface))

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): No What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): iscsi session logout Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Environment: Cloud provider or hardware configuration: Libvirt qemu VM, 4 CPU, 8Gig RAM OS (e.g. from /etc/os-release): Ubuntu 16.04.1 LTS (Xenial Xerus) Kernel (e.g. ): Linux localhost 4.4.0-62-generic -Ubuntu SMP Wed Jan 18 14:10:15 UTC 2017 x8664 x8664 x86_64 GNU/Linux Install tools: kubeadm Others: What happened: If you have multiple pods on the same node using PVCs that are backed by iSCSI LUNs on the same external system over a single iSCSI session (same IQN but different LUN numbers), then deleting one of the pods causes the iSCSI session to terminate for all pods on that node that are using it. For example, here and are nginx pods each connected to a different PVC, which are backed by LUNs residing on the same NetApp storage system. These share a single iSCSI session, differing only by LUN number. Once pod2 is deleted, all the connected LUNs are no longer available.

[6:0:0:1] disk NETAPP LUN C-Mode 8300 /dev/sdc What you expected to happen: I expect that in-use iSCSI sessions are not terminated How to reproduce it (as minimally and precisely as possible): 2 PVs and PVCs that are backed by a single storage system that aggrates iSCSI connections into a single session. 2 pods that consume those PVCs one of the pods Anything else we need to know:
Could someone from please take a look? Thanks!
looking...
It looks like the problem is in line 86 of , in the function. The check evaluates whether a given mount path starts with the prefix (protocol + target portal + target IQN) for the LUN; this will always evaluate to false in most set-ups, since each path will look something like . Changing to appears to result in the correct behavior.
The problem is that the mount device shown in /proc/mounts are canonicalized, and thus when searching device prefix, iscsi volume failed to find a match. If we can use mount option, this the device will show up in /proc/mounts. But this is not a reliable fix since there are probable mpio paths too.
If I'm reading the code correctly, the function doesn't look at the actual device that's backing the mount point; it looks at the mount path (i.e., the second token in /proc/mounts, rather than the first). Thus, the change I proposed (HasPrefix -Contains) appears to work with some very basic testing. That said, I've not done anything with multipathing, so there may be something I'm missing; in addition, I suppose there's a chance that Contains may trigger false positives.
I see your point, yes that sounds right to me.
I think in this case, mpio device should work too (since the mount path constructed in the same way). I see you have yet another iscsi issue open, I am good with rescan. Can you create a pr to fix them?
I should be able create a PR for this issue; the LUN rescan PR will probably get handled by someone else on my team. Is the fix itself sufficient, or does it require tests?
thanks, the rescan fix doesn't have to unit test. The mount path fix needs also a unit test fix here

BUG REPORT Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-01-12T04:57:25Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-01-12T04:52:34Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"} Environment: Cloud provider or hardware configuration: Bare metal OS (e.g. from /etc/os-release): CentOS 7; reproduced in Ubuntu 14.0.4.2. Kernel (e.g. ): Linux centos7 3.10.0-514.6.1.el7.x8664 SMP Wed Jan 18 13:06:36 UTC 2017 x8664 x8664 x8664 GNU/Linux; kernel 3.16.0-77-generic What happened: On a host where an iSCSI LUN from a given target is mounted in a pod, a user who creates a new LUN on the same target and attempts to create a pod that mounts the new LUN will fail to create the pod. Specifically, depending on the operating system, the pod will hang in the ContainerCreating stage with either a "Could not attach disk: Timeout after 10s" error (CentOS 7) or an error about establishing a session when one already exists (Ubuntu 14). What you expected to happen: The pod should successfully mount the LUN. How to reproduce it (as minimally and precisely as possible): Given a working iSCSI target and a single node Kubernetes cluster: a LUN on the iSCSI target a pod that uses the LUN from step 1 (either natively or through a PV/PVC) a new LUN on the iSCSI target (note that the target portal and IQN must be identical). a pod that uses the LUN from step 3. The pod created in step 4 will fail to start.

Anything else we need to know: As far as I can tell, this is due to a bug with the Kubernetes iSCSI implementation. The AttachDisk() function in first checks if the LUN exists in /dev/disk/by-path/; if it doesn't, it attempts to log in to the target and recheck for the LUN's existence (lines 123-138 in the 1.5 branch). In this case, though, the session already exists, as it was created with the pod in step 2. In Ubuntu, attempting to log in a second time triggers an error, which gets reported in the pod description. In CentOS 7, this gets treated as a NOP; however, the new LUN will never become visible, and thus, the pod will fail to start. This can be solved manually by logging into the host and running . It should be possible to fix it in Kubernetes by adding a check in AttachDisk() to see if a session already exists for the target portal and IQN and, if so, doing a rescan of the session.
Could someone from please help with triage? Thanks!


Question: {"Pod Template:"}, {"Image:", redisImage}, {"Events:"}} <del> checkOutput(output, requiredStrings) </del> <ins> checkKubectlOutputWithRetry(requiredStrings, "describe", "rc", "redis-master", nsFlag) </ins> // Service <del> output = framework.RunKubectlOrDie("describe", "service", "redis-master", nsFlag) </del> <ins> output := framework.RunKubectlOrDie("describe", "service", "redis-master", nsFlag) </ins> requiredStrings = [][]string{ {"Name:", "redis-master"}, {"Namespace:", ns},

https://k8s- replica set controller maybe?
The rc seems to be updated almost a second after the failure. Maybe the test needs to retry
And the real log for the failure is
/assign flakes PRs at least once a day The can filter down to focus on a single test:
For when you have some time.
Reassign to you
Output of has been change in 1.6. That's why upgrade tests are failing. But I'm not sure why it is not failing 100% from the triage board. Do you know why? Output of 1.5 or earlier: Output in 1.6 or later:
The timeout is now. Is it reasonable to give it a little more time?
That timeout waits for the pods to run based on the API server. The rc controller is using its own cache so it may lag a bit before it actually updates the rc status. The test needs to retry for some time if the output we get back from describe does not match.
In other word, should also have its own retries.
SGTM. Going to send a fix today.
FYI, this test continues to flake PRs at least once per day:
should it show up in
Hm, this seems to fail in the upgrade tests. this failure seems to be related to an image name change as you already pointed out before. Mind finding what changed the test to use that image and fix the image name?
It is actually the key name has been changed in the key-value pair, not the value. As I pointed out in before expected key: now expected key:
Hm, this is unfortunate. These kubectl tests should really be unit tests.
the 24 hr test history is largely focused on failures right now, and this is highlighting flakes specifically (aka which tests are both passing and failing at the same commit the most often). Please follow the reports I'm sending to kubernetes-dev and related sigs. Also you can generate this info yourself using PS: also try out for finding failures (I'm sending out a separate email highlighting the biggest sources of failure). PPS: we are in the process of creating infrastructure to automate creating these reports, but automation never works as well as a fellow human asking for attention.
We still need to patch the release-1.4 and release-1.5 branch to make the upgrade tests passing.

I will send PRs later this week.
We didn't fix
The change mentioned in was introduced in 1.6. The change covers describer and kubectl e2e test. After discussing with we think we are not going to backport the changes to 1.4 and 1.5, instead we are going to skip the tests in version skew cases. WDYT?
should fix this issue
looks like the PR is resolved, should we close?
Please wait until fix to be cherrypick into 1.4.
Sure! Which PR is this?
Is it possible to cherrypick to 1.4 by just changing the milestone to 1.4?
no you need to create a new PR with the target branch of 1.4
Ready to close?
Yup. /close


Question: execScripts: []ExecArgs{ {"fsck", []string{"-a", "/dev/foo"}, "", nil}, {"lsblk", []string{"-nd", "-o", "FSTYPE", "/dev/foo"}, "", nil}, <del> {"mkfs.ext3", []string{"-E", "lazy_itable_init=0,lazy_journal_init=0", "-F", "/dev/foo"}, "", nil}, </del> <ins> {"mkfs.ext3", []string{"-F", "/dev/foo"}, "", nil}, </ins> }, expectedError: nil, },

Here's logs from formatting a 10GB HDD disk in Azure as ext4: It took seven and a half minutes to complete . Here's me doing it manually from the node: And even from a container, somewhat emulating kubelet: Kubernetes version (use ): Environment: Cloud provider or hardware configuration: Azure OS (e.g. from /etc/os-release): Ubuntu (Xenial) 16.10 Kernel (e.g. ): Linux 4.4.0-34-generic -Ubuntu SMP Wed Jul 27 16:06:39 UTC 2016 x8664 x8664 x86_64 GNU/Linux Install tools: kubernetes-anywhere / static pods with kubelet / kubelet in container What happened: It took seven and a half minutes to format, causing the overall pod startup time to take over 10 minutes by time attachment occurred What you expected to happen: It to take a matter of seconds, similar to how it does if I execute the command manully. How to reproduce it (as minimally and precisely as possible): Build with on top. Deploy a cluster. Try to attach an unformatted VHD. Observe along with watching logs and confirm it really does take that long. Anything else do we need to know: I'm running containerized. Like this:
Can you confirm you were not seeing this behavior during your testing? I assume you were not running containerized?
Even if I and execute the same command from there it executes in a matter of a few seconds.
I'm seeing this behavior even when running as a normal systemd unit, and not as a docker container under a unit. Here's : And I'm still seeing the very slow , so it seems not related to running kubelet in container. I have also tested to ensure that this isn't something some sort of host caching. Even on another blank disk being attached for the the first time, executes quickly except when run by .
I confirm this is this case on my setup running local-up- I have a 100GB HDD disk. It took 30 seconds to format on the console but 2 minutes by kubelet. Is there any I/O constraint imposed on kubelet?
to see if she knows why would be slow when called from the kubelet.
cc/ Who is working on node performance benchmark.

Can we quickly run an experiment on GCE to see if we can reproduce the issue. Let's first rule the impact of containerized Kubelet. If the issue is still exist, I suspect the issue in Kubelet mounting logic. We can enable some tracing, and find the bottleneck.
I tried formatting a 10GB HDD in GCE Kubernetes (not containerized). The time of formatting in kubelet log is ~1s. The log is:
On Azure, a 10GB disk took 18s to mkfs by kublet but 4s to mkfs on the console. Do you have a chance to test these two scenarios? Size probably is not a critical factor for gce since gce pd is provisioned at fixed IOPS per GB, while Azure (also AWS) disks are provisioned at fixed IOPS per volume.
There's more variables in play or else I'm seeing a different edge case than you are, On Azure with a cluster, a 10GB disk for me consistently takes 7+ minutes. (, standard storage, HDD type). It was happening with a and with premium storage and SSD type data disk too... I can make a cluster available to anyone who would like to investigate the exact behavior I've been observing. I can provide kubeconfig+ssh creds. Contact me on Slack (colemickens) if interested.
mkfs's kernel stack after started by kubelet is always as the following. Not sure if it related to blkio cgroup
I have actually created a branch of to give me a cluster where is never running in a container and I'm still seeing the exact same issue. Took 7.5 minutes again, based on kubelet logs and watching . I think we can safely eliminate "containerized kubelet" as being the source of the problem.
Though if I remember correctly, kubelet places itself in a cgroup when it starts up. ( can confirm) so it's possible that it is still due to cgroup I/O throttling.
Above test result proves my initial suspect that the root cause in 7-min use case is containerized kubelet. But recent experiment is not running in a container.

could you please provide us more information: 1) cat /proc/<kubelet-pid/cgroups to find kubelet's cgroup 2) check & blkio.weightdevice setting 3) ps ax | grep makfs to find process id, the find which cgroup the process belongs to 4) check 3) cgroup's and blkio.weight_device If both 2) and 4) don't have issues with cgroup setting, please check if there is parent cgroup for them, and their setting since both CFQ and throttling supports hierarchy.
FYI: Kubelet does not perform any block IO throttling as of now for itself or any other entity on the node. , Dawn Chen wrote:
Yes, we know Kubelet doesn't. But the issue here indicates containerized Kubelet's IOs is throttled, and we are trying to figure out why, especially the user is running kubernetes-anywhere on Azure, which we, node-team don't normally test.
Here's how I was running originally: Here's how I was running when testing it not in a container: I will try to collect the data you mention but I'm going to have to learn how to do some of it along the way, I don't know much about cgroups.
this is not limited to azure, saw it on aws as well.
it seems only the hit only happened to ext4, xfs didn't have this problem.
I ran across the same performance issues and tried to investigate. I use Azure Dynamic Disk provisioning to provision a 1TB volume and it takes very long time for mkfs to finish. Doing the "mkfs.ext4 -F -E lazyitableinit=0,lazyjournalinit=0 /dev/sdd" manually is also very slow, but I did not compare the performance to the call done from inside kubelet. While running the mkfs manually, I observed that iotop shows around 62mb/s and that the free memory immediately shrinks from 2000mb to 100mb. It looks like mkfs is using up all free memory for the buffer cache. I even observed oom situations in the kernel log and processes being killed because of this. I've seen this behavior while running mkfs manually and while it was run by kubelet.

Removing the extended flag "lazyitableinit=0" from the manual mkfs call fixes both problems, performance and memory/buffer cache usage. My solution will now be to use xfs (as rootfs reported it to work better). But IMHO the lazyitableinit should be completely removed from the mkfs call.
Can we try to make sure this gets addressed in 1.6?
I did some further investigation here and it turns out there are other factors involved as well, especially on Azure. My initial statement that I got OOM due to mkfs.ext4 seems to be not true, as the OOMs were caused by fluentd-elasticsearch. I tried to use mkfs.ext4 with "-E discard=1" as I assumed that it should speed up things dramatically. This did not make a difference however. While trying to understand the mke2fs code from e2fsprogs, I realized that discard is enabled by default and always performed as first step when the underlying block device supports this. From the mkfs.ext4 logs you can actually see that it is executed at the beginning. It turns out that mke2fs does only skip inode table initialization when the kernel reports that a discard also ensures zeroing of the data. mke2fs checks this by doing an ioctl to BLKDISCARDZEROES. Support can also be checked with On Azure however, the kernel reports that a discard does NOT zero out the discarded blocks. This results in a loop, using normal write calls to write a lot of zeros to the block dev. This completely fills up the page cache with dirty pages, slowing down the whole system extremely. If there is not much free memory available that can be used for write buffering, the write rate goes down to very low values as well. I did not check the kernel code for lazyitableinit, but I would assume that it uses much more efficient code to write the zeros to the block device (e.g. by not relying on dirty pages). I'd also assume that it does these checks for the discard+zeroout behavior/support and then use discard if possible. If my assumption is correct, then it should not affect performance in a bad way when lazy init would be enabled again. The kernel would then do the discards in a lazy fashion, but finish pretty fast.

On Azure, the lazy init would still take some time, but use much less system resources (page cache) as the non-lazy solution. And most importantly, the device can already be mounted after a few seconds. (UPDATE: please check next comment) I tried to figure out if a discard (using the blkdiscard cmd) does really leave you with non-zeroed data and it looks like it actually becomes zero. I can however not see if this happens immediately or with a delay. Also, in the Azure Portal I can see that the "Billable Size" goes from several GB back to 0 after a short time. My assumption is that on other cloud providers, e.g. GCE and AWS, the block devices report that discard zeroes out the data. Can anyone with a GCE/AWS machine check this with the cat call from above? This would explain why this slowness does not occur on non-Azure clouds. Can you check why the kernel reports that discard does not zero-out? Is this a bug in Azure maybe?
~Ok, my assumption about the kernel using discard for lazy init was wrong. Just checked the kernel source and it simply issues multiple write bios which write zero-pages. So it would definitely have a negative impact on other cloud providers when lazyitableinit would be enabled.~ Would it be a viable solution to check support for BLKDISCARDZEROES in kubernetes before doing the mkfs and then decide based on that if lazyitableinit should be enabled or not?
Argh, digging deeper into the kernel turned out that it actually uses discard for zeroing out (check and ). So my initial assumption was correct. Sorry for all the go home now :)
Just checked the discard support on AWS and it is not supported at all, so it always does a normal write to clear/init the inode tables.
here's the output from a GCP instance: We haven't seen any slowness on GCP, even though the above settings are identical to our Azure instance's. We're trying to solve the same slowness issue on Azure, just not with kubernetes :) (we're Cloud Foundry). We're not using containers on this particular VM. Also, the problem manifests with but not .

Related issue here:
We discovered that by using instead of the problem went away — disks which previously took 8+ minutes to mkfs.ext4 took seconds. Alternatively, will also work as a partitioner (i.e. the subsequent will finish within seconds). Really. We're not sure why the tool we use to write one sector (i.e. the MBR) on a disk would make such a difference. HTH,
Did you discover this by manually running / on the nodes, or did you actually modify to use ?
This sounds like alignment issues. As far as I remember, sfdisk does not do proper alignment and thus may result on very slow performance when you are not on classical spinning disks. Azure disk provisioner does not even create partitions on the disks and thus alignment should not be the problem here. Just tested creating a partition with sfdisk and iotop showed only 7-10mb/s throughput when doing mkfs. Not using a partition at all or using fdisk gives around 60mb/s.
: We discovered this by manually running gparted / sfdisk on the instances we deployed on Azure (i.e. we're not using containers). We don't use Kubernetes — we are developers on Cloud Foundry, which is an alternative to Kubernetes. When we ran into this problem, we were pleased to discover we were not the only ones who have been bitten by this bug. And we wanted to make sure our Kubernetes kin were aware of the fix that we found to be effective. some notes on our research (and I cannot thank you enough for your detailed descriptions): we also found throughput of 7-10MB/s when was healthy when things were unhealthy, showed throughput of ~300kB/s during . we found that by passing (use direct IO and bypass buffer cache), we doubled our performance to ~700kB/s, but it was still ridiculously long (3 minutes).
I am going to try to do some additional investigation tonight, but I'm unsure of how to translate your observations to Kubernetes here. As implied, we don't use anywhere here (that I'm aware of). There's three scenarios: -dynamically provisioned. User brings a pre-formatted VHD. Everything goes smoothly as the merely needs to mount the block device and tell to mount it into the container. -dynamically provisioned.

User attaches an empty disk to an Azure VM, detaches it, uses it in a normal AzureDisk volume on a Pod. This is the scenario in the original post, and you can see that it is being mkfs'd without any partitions. disk provisioning. The VHD footer is written into the page blob, and then formats the whole disk, unpartitioned (trusting description of the provisioner here). I'm hesitant to rule out an alignment issue since you're seeing similar symptoms as we are here, but I'm not sure how to reconcile that with my other points in this comment...
What do you think about It would solve this issue. It would not tell us why it is slower in kubelet, but the problem itself would be solved. Should I CC more people to get this discussed faster? I would even like to see this in an upcoming 1.5.x release.
a lead Azure Engineer ( ) found the problem — it is an alignment problem gone horribly wrong: In situation like unaligned partition boundary, a big I/O request (e.g. a 32KB write) can be split into 64 512-byte writes by the kernel block I/O subsystem in the worst case, easily hitting the 500-IOPS throttling of Azure. It was made to simplify Hyper-V storage driver’s code, so Hyper-V’s storage driver picked up this patch in kernel versions: 4.4+, that is why only Azure is impacted. The community member who submitted the original patch already for this issue, waiting for Linux community to merge.
Does that also affect cases where the disk is unpartitioned, ?
If I understand the description correctly, this only applies to cases where unaligned partitions are in use. This is not the case for Azure Dynamic Disk provisioning and also for all the tests performed so far by all the participants here. As noted in , I did some manual tests with partitioning. And having unaligned partitions really degrades performance a lot (reducing mkfs throughput from 60mb/s to 10mb/s). But as said, we don't have unaligned partitions when using Azure Dynamic Disk actually don't have any partitions at all as we create the FS directly on the block device (e.g. /dev/sdc instead of /dev/sdc1). The real issue here is that lazy inode table initialization is disabled for mkfs.ext4.

The reason for disabling it can be found here: The assumption (BLKDISCARD+BLKDISCARDZEROES support) made in this commit is not true for all cloud providers, e.g. Azure and at least for AWS magnetic disks. I did some calculations to figure out how much zeroes must be written by mkfs.ext4 to initialize the inode tables for a 1TB disk. If you check "tune2fs -l /dev/sdX" you'll see that there are inodes with a size of 256 bytes allocated for a 1TB file system. This gives a total of 16G! These 16G of zeroes are written by mkfs.ext4 when lazy init is disabled, in a very very naive way. Writing 16G of zeroes already takes 4:30 minutes when you have a throughput of 60mb/s. Now if there is something that throttles the troughput (and it looks like kubelet does this somehow), it gets even worse. And that's not all, while this is written do disk, the complete system suffers on low memory due to all of it being used for write caching/buffering. If other processes want to allocate memory in this situation, they're blocked until enough from the buffers is flushed to disk. Please check It fixes all this by simply re-enabling lazy inode table initialization. I also tried to describe why this change should not affect cloud providers that previously thought that disabling lazy init would be an improvement.
I don't believe the partition alignment issue would affect cases where the disk is unpartitioned (i.e. using the entire disk, e.g. ). I suspect that any use of an entire raw device would be properly aligned (any filesystem that starts on sector 0 of the hard disk would be properly aligned), but that's a hunch and nothing more. : I think your assessment is on target — the problem we (Cloud Foundry) are seeing is caused by a partition misalignment, but it seems that the problem described in this thread has a different cause. We currently use a lazy inode table initialization, so we're covered on that front:

According to : While the is exactly the opposite. This significantly slows down large ext4 volume creation.
This might be worth bringing up at the storage f2f coming up.I am going to add to the agenda. This is an interesting approach to what I think may be a few larger issues. Questions and thoughts: What is a good SLO for mkfs? If the SLO is going to be violated due to size should k8s do something or should the user pre-format? There are tradeoffs for initting the filesystem lazy or not. Supporting all the options for a user may require an API change. Is this a persistent volume only thing?
I may miss something, but on a random GCE node with 500GiB magnetic(!) disk and Fedora 24 takes around 2 seconds. ext3 is a different story, it takes 74 seconds.
I cannot imagine any reason why anyone would be installing ext2/ext3 on a 500GB drive. If they do they should be expecting things to be slow. In any case knowing what the SLO is is still not a crazy idea.
This is probably not a big problem with GCE, since GCE PD's IOPS is per GB. This is mostly a problem for EBS/Cinder/Azure, since the IOPS per volume is fixed.
If we're going to do this at all, we should really think hard about this problem top-to-bottom. Ext3 vs ext4 vs other FS? Myriad filesystem options, etc.
I got bitten by this when implementing e2e tests for all supported storage class options on AWS. "Cold HDD" volume type has max. 250 IOPS and the smallest one is 500 GB. It takes 3-4 minutes just to format the volume, which slows down the test quite considerably. Was there any conclusion on f2f meeting? was in commit , which claims to be rewritten to Go. Original script not exist any more on the link above. Was there some smart idea in original ?
see some inconclusive investigation at
Please take a look at I encountered not only performance issue due to lazyitableinit=0, but also oom situations and thus killed processes. Due to the amount of data written to the disk, all free memory gets filled with dirty buffer cache.


Question: <ins> // +build !providerless </ins> /* Copyright 2019 The Kubernetes Authors.

make verify WHAT=typecheck-providerless outputs /cc /milestone v1.19
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
see example successful run:
For reference -
fix in
screaming


Question: encoder := zapcore.NewJSONEncoder(encoderConfig) var core zapcore.Core if errorStream == nil { <del> core = zapcore.NewCore(encoder, zapcore.AddSync(infoStream), zapcore.Level(-127)) </del> <ins> core = zapcore.NewCore(encoder, infoStream, zapcore.Level(-127)) </ins> } else { <del> // Set up writing of error messages to stderr and info messages // to stdout. Info messages get optionally buffered and flushed // - through klog.FlushLogs -> zapr Flush -> zap Sync // - when an error gets logged // // The later is important when both streams get merged into a single // stream by the consumer (same console for a command line tool, pod // log for a container) because without it, messages get reordered. flushError := writeWithFlushing{ WriteSyncer: errorStream, other:       infoStream, } </del> highPriority := zap.LevelEnablerFunc(func(lvl zapcore.Level) bool { return lvl >= zapcore.ErrorLevel })

When working on I have found a regression in JSON logging format that happend in 1.23. In v1.23 JSON logging introduced two new features, log stream splitting and info log buffering. When I started measuring throughput it looked very promising, for example for info logs showed 10x improvement when using JSON split stream and 100Mi info log buffer. However when I introduced mixed logs (ever X info logs write 1 error logs), I noticed a large regression. For 10% of error logs written (apiserver during upgrade can write even 30% errors), I have found that just switching json on, results in log throughput being reduced from 25 MiB/s to 0.7MiB. I have bisected git repo and found that was the root cause. I expect that this regression would be noticed if we run scalability tests with JSON enabled. Loader code: /wg structured-logging /cc /sig scalability JSON should generate similar or greater log throughput when compared to text format. Run loader as described in from No response N/A N/A N/A N/A N/A N/A
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
So to reproduce the regression, I need to do (default for error rate is 10%, 300 for message length, 10s duration) for old and new code, right?
Yes, let me give my results. Before text: 30.7MiB/s, json:39.6MiB/s After, text: 30.6MiB/s, json: 6.5MiB/s
should we try to get this in 1.23.1?
Yes, we just moved JSON and Structured Logging to Beta in v1.23. This bug should cause performance issues when there is non trivial amount of errors (1%) written by control plane. I expect that this has really bad impact on upgrades which I have seen can reach 30% of error logs from apiserver.

It's worth pointing out that this regression only occurs when writing to a disk-based file (see ). That also provides a very likely explanation for the huge performance difference: I bet zap not only flushes its buffered info messages, it also syncs the file. That is fast for tmpfs, but slow for real disks.
do you know how the Kubernetes components are run in those cases where logging performance matters? Do they directly write to a real disk or does it all go through pipes? If it is the latter, then this issue matters a lot less in practice. you mentioned at some point that you had scalability issues with logging that went away when using go-runner as wrapper instead of using klog's internal output redirection. I bet the root cause is the same: fast vs. slow sync of the output fd. That's because go-runner always runs output through one or more pipes. I don't know how realistic the scalability setup was, but it must have written log files to some real disk.


Question: allErrs = append(allErrs, field.Required(idxPath, "must specify `addresses` or `notReadyAddresses`")) } for addr := range ss.Addresses { <del> allErrs = append(allErrs, validateEndpointAddress(&ss.Addresses[addr], idxPath.Child("addresses").Index(addr), ipToNodeName)...) </del> <ins> allErrs = append(allErrs, validateEndpointAddress(&ss.Addresses[addr], idxPath.Child("addresses").Index(addr))...) </ins> } for addr := range ss.NotReadyAddresses { <del> allErrs = append(allErrs, validateEndpointAddress(&ss.NotReadyAddresses[addr], idxPath.Child("notReadyAddresses").Index(addr), ipToNodeName)...) </del> <ins> allErrs = append(allErrs, validateEndpointAddress(&ss.NotReadyAddresses[addr], idxPath.Child("notReadyAddresses").Index(addr))...) </ins> } for port := range ss.Ports { allErrs = append(allErrs, validateEndpointPort(&ss.Ports[port], len(ss.Ports) > 1, idxPath.Child("ports").Index(port))...)

/kind bug What happened: We have noticed that one of the node-exporters is down. It turned out that service has "old" endpoint still. After digging deeper i see following errors in controller manager Turns out controller manager can not update endpoints for node-exporter service, because of this error. In endpoint list is see right IPs for worker nodes, but wrong IP for master node (that's why we detected the problem). Context: two worker nodes were recreated with new hostnames (generated randomly), but they use same ip address one master node were recreated with new hostname and new ip I've tried to manually update hostname and got same error, seems nodeName is not updatable field. It also affects all other endpoints, we were forced to recreated all pods (so they will change IPs) to be able to fix endpoints reconciliation. What you expected to happen: controller manager successfully update endpoints after node recreated with same IP, but different hostname. How to reproduce it (as minimally and precisely as possible): 1 master, 1 worker, service that points to pods with host network. k8s API service (keep it shut down) worker node with same IP and different host name to recreate pod on worker and in controller manager should appear error above. Anything else we need to know?: Environment: Kubernetes version (use ): 1.10.1 Cloud provider or hardware configuration: baremetal OS (e.g. from /etc/os-release): CoreOS 1688.5.3 Kernel (e.g. ): 4.14.32-coreos Install tools: own tool () Others:
/sig node
what is the use case of changing node name?
/sig api-machinery
We have ephemeral immutable nodes in our clusters. So upgrade or any other operation will cause node recreation. Hostname generated automatically, but the IP address depends on the underlay hypervisor node, where node will be scheduled. So it's possible that node stays on the same hypervisor and uses same ip address.
/cc
/assign
/cc do you think it's reasonable to loose endpoints validation?
/sig network this API is owned by sig network


Question: if err != nil { glog.Fatalf("FAILED: unable to get pods to list: %v", err) } <del> if err := wait.Poll(time.Second, time.Second*10, podsOnMinions(c, *pods)); err != nil { </del> <ins> if err := wait.Poll(time.Second, time.Second*30, podsOnMinions(c, *pods)); err != nil { </ins> glog.Fatalf("FAILED: pods never started running %v", err) }

https://travis- Setting as P0 since scheduling pods is pretty basic...
Looks like it's the "podBackoff" thing. IMO exponential backoff makes little sense for this; there's either pods that permanently won't schedule, or there's pods having some temporary problem. The former we should give up on and the latter we should retry at some fixed interval.
Actually it's not that because there's no error messages. Hm...
I think fixed this for the moment. But looking at the Travis logs the problem isn't slowness, it's a race-- the pods show up immediately after the 10 second timeout. Probably something is caching something and not updating for 10 seconds.
Actually, not fixed yet: https://travis-
Just extending timeouts isn't sufficient to fix this, or should have worked.
Could this be the problem "fixed" by
I'm betting it was actually fixed by: That's definitely a race/flake inducing bug. (but I'm not 100% positive) --brendan , bgrant0607 wrote:
Closing this as I think we found the cause, and we haven't seen it in a while.


Question: for vc, zoneToHostsMap := range vcToZoneHostsMap { for zone, hosts := range zoneToHostsMap { commonDatastores := retrieveCommonDatastoresAmongHosts(hosts, vcToHostDatastoresMap[vc]) <del> if vcToZoneDatastoresMap[vc] == nil { vcToZoneDatastoresMap[vc] = make(map[string][]string) </del> <ins> if nm.vcToZoneDatastoresMap[vc] == nil { nm.vcToZoneDatastoresMap[vc] = make(map[string][]string) </ins> } <del> vcToZoneDatastoresMap[vc][zone] = commonDatastores </del> <ins> nm.vcToZoneDatastoresMap[vc][zone] = commonDatastores </ins> } } <del> framework.Logf("Zone to datastores map : %+v", vcToZoneDatastoresMap) </del> <ins> framework.Logf("Zone to datastores map : %+v", nm.vcToZoneDatastoresMap) </ins> return nil }

Command run: vSphere intree volume test suite occasionally failed with the following error: The test suite should always pass. We may have a chance to hit this by running ` No response binary version: 1.27.4 </details<detailsvsphere </details<details</details<details</details<details</details<details</details
/assign
/sig testing storage ?
/triage accepted


Question: } // getZoneByProviderIDorName will attempt to get the zone of node using its providerID <del> // then it's name. If both attempts fail, an error is returned func getZoneByProviderIDOrName(ctx context.Context, zones cloudprovider.Zones, node *v1.Node) (cloudprovider.Zone, error) { zone, err := zones.GetZoneByProviderID(ctx, node.Spec.ProviderID) </del> <ins> // then it's name. If both attempts fail, an error is returned. func getZoneByProviderIDOrName(ctx context.Context, zones cloudprovider.Zones, providerID, nodeName string) (cloudprovider.Zone, error) { zone, err := zones.GetZoneByProviderID(ctx, providerID) </ins> if err != nil { providerIDErr := err <del> zone, err = zones.GetZoneByNodeName(ctx, types.NodeName(node.Name)) </del> <ins> zone, err = zones.GetZoneByNodeName(ctx, types.NodeName(nodeName)) </ins> if err != nil { return cloudprovider.Zone{}, fmt.Errorf("Zone: Error fetching by providerID: %v Error fetching by NodeName: %v", providerIDErr, err) }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: introduces a regression that func is invoked on a Node obj without , hence it can fail fetching the node addresses when the is not set. What you expected to happen: I would expect cloud-controller-manager to be able to initialize new Node. How to reproduce it (as minimally and precisely as possible): the gcp legacy-cloud-controller v1.18.1 () that the node-controller cannot initialize a new Node. Logs of the cloud-controller-manager repeated multiple times: Ensure that the Node is with the taint. Ensure that the of the Node is not set. The cloud-controller-manager is not running on the host machine and that's why it cannot call the gce matadata. But it was never a hard requirement for cloud-controller-manager to run on the host machine. Anything else we need to know?: The legacy cloud-controller-manager v1.17.x is working perfectly fine. Environment: Kubernetes version (use ):v1.18.1 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig cloud-provider /kind regression
/cc
can we open a new issue re:


Question: ObservedGeneration:    actualPDB.Generation, } actualStatus := actualPDB.Status <del> if !reflect.DeepEqual(actualStatus, expectedStatus) { </del> <ins> if !apiequality.Semantic.DeepEqual(actualStatus, expectedStatus) { </ins> debug.PrintStack() t.Fatalf("PDB %q status mismatch.  Expected %+v but got %+v.", key, expectedStatus, actualStatus) }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: There are too many pdb update operations compared with other resources. ! That is, there are many "Put" operations for pdb periodically in kube-apiserver log. In my view, the amount of pdb operations is connected with disruption controller's resyncPeriod and pdb's quantity in cluster. For example: disruption-controlle's default resyncPeriod is 30s, and there are 100 pdbs in cluster. We can find that the update operation times per minute is about 200. The data is reasonable, explanation is as follows: According to disruption-controlle's design, there is one resync every 30s. One resync means to call updatePdbStatus function for every pdb, so there are 200 times updatePdbStatus. Function updatePdbStatus judges whether the pdb is changed. If changed, call apiserver to update the pdb' status. When no eviction happens, disruptedPods is empty. In this situation, disruptedPods is an empty map, but pdb.Status.DisruptedPods is nil, empty map is not equal to nil. So reflect.DeepEqual(pdb.Status.DisruptedPods, disruptedPods) returns false. The pdb is thought to has be changed, then apiserver is called to update the pdb' status. In fact, there is no change for the pdb when disruptedPods is an empty map and pdb.Status.DisruptedPods is nil. What you expected to happen: There shouldn't be so many pdb update operations. That is, when pdb has no change, apiserver shouldn't be called to update its status. How to reproduce it (as minimally and precisely as possible): Create some pdbs in cluster, you will see many "Put" operations for pdb periodically in kube-apiserver log. Anything else we need to know?: Environment: kubernetes v1.13.2 Fix suggestions: The bug is caused by the mixed usage of empty and nil for disruptedPods, the type of which is map[string]metav1.Time.

There are two fix suggestions for it: one of empty map and nil for disruptedPods, use it consistently and refuse to mix empty map and nil. For example, when disruptedPods is empty or nil, save it as an empty map in etcd. Then every client gets pdb from etcd, the pdb.Status.DisruptedPods is an empty map. get pdb for etcd, assign an empty map to it if pdb.Status.DisruptedPods is nil. Maybe the mixed usage of empty and nil for map is a common problem in k8s design, not just for disruptedPods. I will propose a work around solution for the problem. If the solution has any problem, please let me know and I can modify the code.
/sig apps
/assign Morten can you PTAL and the open PR. I reviewed as well.
ref


Question: configs[Agnhost] = Config{e2eRegistry, "agnhost", "2.0"} configs[Alpine] = Config{dockerLibraryRegistry, "alpine", "3.7"} configs[AuthenticatedAlpine] = Config{gcAuthenticatedRegistry, "alpine", "3.7"} <ins> configs[AuthenticatedWindowsNanoServer] = Config{gcAuthenticatedRegistry, "windows-nanoserver", "v1"} </ins> configs[APIServer] = Config{e2eRegistry, "sample-apiserver", "1.10"} configs[AppArmorLoader] = Config{e2eRegistry, "apparmor-loader", "1.0"} configs[AuditProxy] = Config{e2eRegistry, "audit-proxy", "1.0"}

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: All sig-windows jobs (master): Which test(s) are failing: [] Container Runtime blackbox test when running a container with a new image should be able to pull from private registry with secret [NodeConformance] [] Container Runtime blackbox test when running a container with a new image should be able to pull image from [NodeConformance] Since when has it been failing: Since this PR merged: Testgrid link: Reason for failure: The mentioned PR removed hardcoded image names from tests, centralizing them into , but there were a few omissions: entry was written for. registry set for ` is because that registry contains the Windows images needed for all the Kubernetes E2E tests. Anything else we need to know: /sig testing /sig windows /area conformance /priority important-soon


Question: params["name"] = GetFlagString(cmd, "service-name") } if _, found := params["selector"]; !found { <ins> rc, err := client.ReplicationControllers(namespace).Get(args[0]) checkErr(err) </ins> params["selector"] = kubectl.MakeLabels(rc.Spec.Selector) } if GetFlagBool(cmd, "create-external-load-balancer") {

expose hostnames --port=80 --container-port=9376 --selector="app=hostnames" get se hostnames -o yaml apiVersion: v1beta1 containerPort: "9376" ... obviously, endpoints controller is not happy.


Question: "http://metadata.google.internal/computeMetadata/v1/instance/tags?wait_for_change=true&timeout_sec=0", "http://metadata.google.internal/computeMetadata/v1/instance/tags?wait_for_change=true&last_etag=d34db33f", } <del> legacySuccessEndpoints = []string{ // Discovery "http://metadata.google.internal/0.1/meta-data", "http://metadata.google.internal/computeMetadata/v1beta1", // Allowed API versions. "http://metadata.google.internal/0.1/meta-data/", "http://metadata.google.internal/computeMetadata/v1beta1/", // Service account token endpoints. "http://metadata.google.internal/0.1/meta-data/service-accounts/default/acquire", "http://metadata.google.internal/computeMetadata/v1beta1/instance/service-accounts/default/token", // Known query params. "http://metadata.google.internal/0.1/meta-data/service-accounts/default/acquire?scopes", } </del> noKubeEnvEndpoints = []string{ // Check that these don't get a recursive result. "http://metadata.google.internal/computeMetadata/v1/instance/?recursive%3Dtrue",   // urlencoded

Warning: The v0.1 and v1beta1 metadata server endpoints are deprecated and scheduled for shutdown on September 30, 2020. This test (which runs in most e2e jobs that run on GCE): Schedules a container that runs this: Resulting in a number of legacy metadata endpoints getting hit. To see if this test is going to start failing across all release branches on Sep 30th, we could try disabling the endpoints ahead of time via: In fact and support for expecting different results based on whether is set. My concern is the test expects 200 for some root endpoints, and 403 for disabled endpoints. However the e-mail I received as a GCP customer tells me any legacy endpoints may return 404. I'm inclined to suggest we remove the legacy endpoints from the check, bump the image version, and cherry-pick use of the new image version back to all release-branches. WDYT since you were involved with the earlier PRs /sig auth /sig cloud-provider /provider gcp /sig testing /priority important-soon (I'm tempted to critical-urgent this but it's not drop-everything yet) /milestone v1.20
/kind cleanup
Heyho I wanted to check how it is going? The code freeze is starting 12.11.2020 (about 3-4 weeks from now) and while there is still plenty of time, we want to ensure that each relevant issue has a chance to be considered. As the issue is tagged for 1.20, is it still planned for this release?
Bug Triage here, we're pretty late into code freeze and it doesn't look like there has been much activity recently. I am going to clear the milestone on this issue. /milestone clear
any update here?
Try to cleanup the legacy endpoints in
/milestone v1.21
/reopen This needs to stay open until the new image has been verified to work and has been promoted (ref: )
Reopened this issue. <detailsIn response to : /reopen This needs to stay open until the new image has been verified to work and has been promoted (ref: ) Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage accepted /assign


Question: if err != nil { return err } <ins> // Wait for the PVC to get fully processed. This avoids races between PV controller and DeleteClaimEvent // below. err = wait.Poll(10*time.Millisecond, wait.ForeverTestTimeout, func() (bool, error) { obj := ctrl.claims.List()[0] claim := obj.(*v1.PersistentVolumeClaim) return claim.Status.Phase == v1.ClaimLost, nil }) if err != nil { return err } </ins> // trying to remove the claim as well obj := ctrl.claims.List()[0] claim := obj.(*v1.PersistentVolumeClaim)

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --pull-kubernetes-unit TestControllerSync /sig storage
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue.

<detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-lifecycle rotten /reopen I still see the flake in recently.
Reopened this issue. <detailsIn response to : /remove-lifecycle rotten /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Fix:


Question: // Defines a time budget that can be spend on waiting for not-ready watchers // while dispatching event before shutting them down. <del> dispatchTimeoutBudget *timeBudget </del> <ins> dispatchTimeoutBudget timeBudget </ins> // Handling graceful termination. stopLock sync.RWMutex

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: bazel-test-master (periodic-kubernetes-bazel-test-master ) Which test(s) are flaking: gotest: Testgrid link: Reason for failure: Anything else we need to know: links to appreciated links to specific failures in spyglass appreciated <!-- Please see the deflaking doc () for more guidance! --cc
/sig api-machinery
This is continuing to be flaky with more recent failure:
/assign Would you mind taking a look here? thanks! /triage accepted
- can we somehow uncover when this started happening? This test is very old and it seems it started happening very recently, so it would be helpful to see when this started happening...
Triage shows flakes starting ~Sept 30: Before Sept 30 it seems the job was flaking much less frequently.
Might be related to I will take a look.
It seems it's not related to that PR (neither to the previous: ) As there were already failures like that on Sep 24th (before this one was merged). I will take a deeper look.
OK - it's not related to any changes mentioned before. I can reproduce this particular failure by simply adding somewhere in this block: So it's probably connected to cpu-starvation at the test level and the solution is to switch to less aggressive timeouts. Will send out a PR.
Actually - it appeared to be slightly more subtle then that. But I already fully understand the problem. It's purely test issue, but require a bit more changes (and I won't get to it today/tomorrow). I will send a fix for that early next week.
- if that changes anything from your tracking perspective ^^
No board changes yet - the next stage is "Observing" once this issue gets closed :) If you have time - I'd love to pick your brain/understand the error more thoroughly though! Hoping you drop a good explanation in your PR next week
Thanks for sharing those insights, on your discovery of the solution to this bug.
It seems the PR didn't address that fully. Looking.
out for review


Question: // Register registers a collectable metric but uses the global registry func Register(c metrics.Registerable) error { err := defaultRegistry.Register(c) <del> // sideload global prom registry as fallback prometheus.Register(c) </del> return err } // MustRegister registers registerable metrics but uses the global registry. func MustRegister(cs ...metrics.Registerable) { defaultRegistry.MustRegister(cs...) <del> // sideload global prom registry as fallback for _, c := range cs { prometheus.Register(c) } </del> } // RawMustRegister registers prometheus collectors but uses the global registry, this

What happened: CI failed when deprecated metrics. Refer to The reason is that a hidden metrics won't be created thus can't be registered to prometheus's registry. I guess the other PR failed for the same reason. What you expected to happen: How to reproduce it (as minimally and precisely as possible): When starting the cluster, panic happens Anything else we need to know?: In my option, we need to remove backoff from legacyregistry. Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /assign /sig instrumentation
Discoveries after run go test ( by ) after removing backoff and deprecating some metrics: The kubelet's metrics are protected by the unit tests, and they work well . The only block is , that is I mean register to legacy registry and gather from Prometheus.
By backoff do you mean the fallback call to prometheus.Register in the various legacyregistry register functions?
Yes. I opened a PR and see what would happen. (I guess it should be fine now)


Question: list = &metav1.APIGroupList{ Groups: []metav1.APIGroup{ { <ins> Name: "apps", Versions: []metav1.GroupVersionForDiscovery{ {GroupVersion: "apps/v1beta1", Version: "v1beta1"}, {GroupVersion: "apps/v1beta2", Version: "v1beta2"}, {GroupVersion: "apps/v1beta3", Version: "v1beta3"}, {GroupVersion: "apps/v1beta4", Version: "v1beta4"}, {GroupVersion: "apps/v1beta5", Version: "v1beta5"}, {GroupVersion: "apps/v1beta6", Version: "v1beta6"}, {GroupVersion: "apps/v1beta7", Version: "v1beta7"}, {GroupVersion: "apps/v1beta8", Version: "v1beta8"}, {GroupVersion: "apps/v1beta9", Version: "v1beta9"}, {GroupVersion: "apps/v1beta10", Version: "v1beta10"}, }, }, { </ins> Name: "extensions", Versions: []metav1.GroupVersionForDiscovery{ {GroupVersion: "extensions/v1beta1", Version: "v1beta1"}, {GroupVersion: "extensions/v1beta2", Version: "v1beta2"}, <ins> {GroupVersion: "extensions/v1beta3", Version: "v1beta3"}, {GroupVersion: "extensions/v1beta4", Version: "v1beta4"}, {GroupVersion: "extensions/v1beta5", Version: "v1beta5"}, {GroupVersion: "extensions/v1beta6", Version: "v1beta6"}, {GroupVersion: "extensions/v1beta7", Version: "v1beta7"}, {GroupVersion: "extensions/v1beta8", Version: "v1beta8"}, {GroupVersion: "extensions/v1beta9", Version: "v1beta9"}, {GroupVersion: "extensions/v1beta10", Version: "v1beta10"}, </ins> }, }, },

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: With client-go v0.17.0, I observed many "Throttling request" when DiscoveryClient is used to get API group resources from API server. This didn't happen with client-go kubernetes-1.16.0. It looks like originally this kind of discovery requests are not throttled because no rate limiter provided and QPS == 0. But with rate limiter is always set if nil rate limiter is provided. is particularly suspicious because it is always true because DefaultQPS is nonzero. What you expected to happen: Discovery requests should not be throttled as before. How to reproduce it (as minimally and precisely as possible): Create DiscoveryClient and then call with it. Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig api-machinery
The NewTokenBucketRateLimiter was via Always negotiate a decoder using ClientNegotiator /cc
To be clear, that was fixing a bug where the documented defaults for rate limiting were not taking effect. I don't think we should completely disable throttling until more of the server-side priority work is completed, but discovery is expected to be bursty, so I could see defaulting to a different amount in the discovery client constructor, similar to
Yeah, everyone must be rate limited. I think we could give discovery a big burst value.
opened


Question: if [[ "${ENABLE_DNS_HORIZONTAL_AUTOSCALER:-}" == "true" ]]; then setup-addon-manifests "addons" "dns-horizontal-autoscaler" "gce" <ins> local -r dns_autoscaler_file="${dst_dir}/dns-horizontal-autoscaler/dns-horizontal-autoscaler.yaml" sed -i'' -e "s@{{.Target}}@${KUBEDNS_AUTOSCALER}@g" "${dns_autoscaler_file}" </ins> fi }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: e2e test "gci-gce" is running in 2 versions: with kube-dns as DNS Service : https://k8s- with CoreDNS as DNS Service : https://k8s- In CoreDNS use case the test is failing What you expected to happen: This e2e tests should not depend on what DNS service is running as default in K8s. CoreDNS is currently BETA, will move to GA soon (after v1.10 though) See : How to reproduce it (as minimally and precisely as possible): look at logs of dashboard : https://k8s- go run -v --test --test_args='--[sig-autoscaling]sDNSshorizontalsautoscalings[DisabledForLargeClusters]skube-dns-autoscalersshouldsscaleskube-dnsspodssinsbothsnonfaultysandsfaultysscenarios$' Anything else we need to know?: Environment: Kubernetes version (use ): branch master Cloud provider or hardware configuration: GCE Install tools: deployment intra-test Others: see https://k8s-
/sig autoscaling
/unsig cli
Looks like the test is searching specifically for a deployment labeled "kube-dns". There are several references to "kube-dns" scattered around it, actually.
Right now the tool "dns-horizontal-autoscaller" seems dedicated for kube-dns. This tools is deployed as "add-on" when enabled by this ENV variable : I guess we should have the definition of another dns-horizontal-autoscaller, dedicated to coredns (unless we generate the yaml file). and the choice of deployment is based on the ENV variable . see
Currently, the translation of the deployment of "dns-horizontal-autoscaller" for CoreDNS is in progress by It is expected that the corresponding e2e test will work as is (as the artifacts are selected by labels, which are the same for CroeDNS and kube-dns).


Question: ss.LoadBalancerSku = loadBalancerSkuStandard <del> expectedVMSS := buildTestVMSS([]string{testLBBackendpoolID0}) </del> <ins> expectedVMSS := buildTestVMSS(testVMSSName, []string{testLBBackendpoolID0}) </ins> if test.isVMSSDeallocating { expectedVMSS.ProvisioningState = &virtualMachineScaleSetsDeallocating }

Which jobs are flaking: pull-kubernetes-bazel-test Which test(s) are flaking: TestEnsureBackendPoolDeletedConcurrently Testgrid link: Reason for failure: /sig cloud-provider
cc
/assign /area provider/azure


Question: } } <ins> func TestValidateAllowNonLocalProjectedTokenPathOption(t *testing.T) { testCases := []struct { name       string oldPodSpec *api.PodSpec wantOption bool }{ { name:       "Create", wantOption: false, }, { name: "UpdateInvalidProjectedTokenPath", oldPodSpec: &api.PodSpec{ Volumes: []api.Volume{ { Name: "foo", VolumeSource: api.VolumeSource{ Projected: &api.ProjectedVolumeSource{ Sources: []api.VolumeProjection{ { ServiceAccountToken: &api.ServiceAccountTokenProjection{ Path: "../foo", }, }, }, }, }, }, }, }, wantOption: true, }, { name: "UpdateValidProjectedTokenPath", oldPodSpec: &api.PodSpec{ Volumes: []api.Volume{ { Name: "foo", VolumeSource: api.VolumeSource{ Projected: &api.ProjectedVolumeSource{ Sources: []api.VolumeProjection{ { ServiceAccountToken: &api.ServiceAccountTokenProjection{ Path: "foo", }, }, }, }, }, }, }, }, wantOption: false, }, { name: "UpdateEmptyProjectedTokenPath", oldPodSpec: &api.PodSpec{ Volumes: []api.Volume{ { Name: "foo", VolumeSource: api.VolumeSource{ Projected: nil, HostPath:  &api.HostPathVolumeSource{Path: "foo"}, }, }, }, }, wantOption: false, }, } for _, tc := range testCases { t.Run(tc.name, func(t *testing.T) { // Pod meta doesn't impact the outcome. gotOptions := GetValidationOptionsFromPodSpecAndMeta(&api.PodSpec{}, tc.oldPodSpec, nil, nil) if tc.wantOption != gotOptions.AllowNonLocalProjectedTokenPath { t.Errorf("Got AllowNonLocalProjectedTokenPath=%t, want %t", gotOptions.AllowNonLocalProjectedTokenPath, tc.wantOption) } }) } } </ins> func TestDropInPlacePodVerticalScaling(t *testing.T) { podWithInPlaceVerticalScaling := func() *api.Pod { return &api.Pod{

Originally posted by in
Hello, I'm new here. Can I try solving this? it seems like a good first issue
sure, context is the change is needed here: We have to tolerate existing pods / pod specs that fail this validation, so we should only do this additional validation on create or if the existing object passes this validation. To do this: an field to , initialize that field to false, and then conditionally override it to if is non-nil and has a projected token volume path that fails (will have to export the function to check this) , if is false, call the validation on the path:
Hi, I'd like to work on this if there is no one is proceeding on. /assign


Question: } else { devicePath = strings.Join([]string{"/dev/disk/by-path/pci", "*", "ip", tp, "iscsi", b.Iqn, "lun", b.lun}, "-") } <del> exist := waitForPathToExist(devicePath, 1, iscsiTransport) </del> <ins> exist := waitForPathToExist(&devicePath, 1, iscsiTransport) </ins> if exist == false { // build discoverydb and discover iscsi target b.plugin.execCommand("iscsiadm", []string{"-m", "discoverydb", "-t", "sendtargets", "-p", tp, "-I", b.Iface, "-o", "new"})

Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"7+", GitVersion:"v1.7.0-beta.1", GitCommit:"", GitTreeState:"not a git tree", BuildDate:"2017-06-08T13:08:32Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-05-10T15:38:08Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"} Environment: Cloud provider or hardware configuration: Bare-Metal, HP SE316M, 2x X5675, 144GB, iSCSI/RDMA/iSer OS: CentOS 7 Kernel: Linux at-host-18 4.9.31 SMP Wed Jun 7 22:40:40 UTC 2017 x8664 x8664 x8664 GNU/Linux Install tools: Others: What happened: I want to mount a iSCSI volume via iSER (RDMA over Infiniband) in this case the iSCSI iface is "iser" and not tcp, therefore the glob path for device mount is selected (), which then is passed to the fsck and mount commands without globbing the path (e.g. calling filepath.Glob) Therefore the iSCSI volume is not mounted and the pod cannot be started. Logs: What you expected to happen: To successfully mount the volume. How to reproduce it (as minimally and precisely as possible): Setup an iSCSI via RDMA (iSER) environment without multipath and try to mount a volume for a pod. Anything else we need to know: I believe it's just needed to add the globbing at line 259 like so: devicePaths = append(devicePaths, filepath.Glob(devicePath)) As I'm not a go programmer and have no experience with it, probably somebody could create a commit which I would gladly test!
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team.

You can find the team list and label list
/sig storage
I'll take a look.
/sig storage
on a local build that fixed the issue: (e.g. it compiles)
I don't want to add filepath.Glob to that general path because that line is also executed "tcp" case. I'm considering the fix below. Pretty similar.
does that work from a go specific point? is that devicePath passed by reference? (from general c/c++/php/node/c# knowledge I would say that this would have no effect on the devicePath variable .. )
Can we suppose that filepath.Glob(devicePath) always only return one device path if transport is "tcp"?
does that work from a go specific point? is that devicePath passed by reference? (from general c/c++/php/node/c# knowledge I would say that this would have no effect on the devicePath variable .. ) Thank you for above comment. I updated the logic. Could you confirm the fix on the PR?
Pretty sure that tcp transport will always only have one path - and it will exist (there's no glob in the check path)


Question: pod             = testPodWithROVolume(podName, source, volumePath) rwcontainerName = containerName + "-rw" ) <ins> pod.Spec.NodeSelector = map[string]string{ "kubernetes.io/os": "windows", } </ins> rwcontainer := v1.Container{ Name:  containerName + "-rw",

similar to PR en route
/sig windows
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: RegistryPullQPS:             5, HairpinMode:                 kubeletconfig.PromiscuousBridge, NodeLeaseDurationSeconds:    1, <del> CPUCFSQuotaPeriod:           metav1.Duration{Duration: 100 * time.Millisecond}, </del> <ins> CPUCFSQuotaPeriod:           metav1.Duration{Duration: 50 * time.Millisecond}, </ins> ReservedSystemCPUs:          "0-3", <ins> FeatureGates: map[string]bool{ "CustomCPUCFSQuotaPeriod": true, }, </ins> } if allErrors := ValidateKubeletConfiguration(successCase2); allErrors != nil { t.Errorf("expect no errors, got %v", allErrors)

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --See for more context. What happened: When you set and , you'd expect container cfs quota period to be . However, unless you also set in the feature flags, the quota value will be silently ignored. What you expected to happen: At least a warning be logged if the feature is not enabled but is set. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig node /assign karan
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig node Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: set -o pipefail KUBE_ROOT=$(dirname "${BASH_SOURCE}")/.. <del> KUBE_ROOT=$(dirname "${BASH_SOURCE}")/.. </del> source "${KUBE_ROOT}/hack/lib/init.sh" kube::golang::setup_env

As mentioned in: We seem to be setting different flags for kubemark-scale. FWIW - I don't think it's the only thing actually. If seems that certain env-vars that we're setting in the kubemark configs: are actually unused - I don't see KUBEMARKCONTROLLERMANAGERTESTARGS (as an exammple) be propagated anywhere. We should fix that (add *TESTARGS env vars) and unify the setting [In particular for the sake of testing P&F configs.]
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage accepted I will try to look into it later this week
This isn't really true - we have this logic that is appropriately setting KUBEMARK_-prefixed env-vars: So the only thing is to unify the settings.


Question: corev1 "k8s.io/api/core/v1" discovery "k8s.io/api/discovery/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" <ins> "k8s.io/apimachinery/pkg/types" </ins> "k8s.io/client-go/kubernetes/fake" "k8s.io/client-go/kubernetes/scheme" "k8s.io/client-go/tools/record"

create a service without selector, and manual create endpoints for this service endpointslicemirroring controller will create an endpointslice for this service. after endpoint slice created, and kube-controller-manager restart. during the kube-controller-manager restart, the endpoint was deleted and recreated after kube-controller-manager restart, garbagecollector controller delete this endpointslice and endpointslice not recreate until endpoint/service update event or kube-controller-manager restart endpointslice will be recreated create a service without selector, and manual create endpoints for this service endpointslicemirroring controller will create an endpointslice for this service. after endpoint slice created, and kube-controller-manager restart. during the kube-controller-manager restart, the endpoint was deleted and recreated No response <details</details<details</details<details</details<details</details<details</details<details</details
/sig network
/assign
Trying to understand this, mostly just restating what you've said above to make sure I'm getting it: during the kube-controller-manager restart, the endpoint was deleted and recreated after kube-controller-manager restart, garbagecollector controller delete this endpointslice and endpointslice not recreate until endpoint/service update event or kube-controller-manager restart So an Endpoints resource is being deleted while kube-controller-manager is not running. The EndpointSlice mirroring controller to point to the Endpoints they're mirroring. When kube-controller-manager is restarted, the garbage collector will delete the mirrored EndpointSlice because it's owned by an Endpoints resource that was previously deleted. That deletion by garbage collection would then be , but if the EndpointSliceTracker doesn't think it should have that resource, it will ignore the deletion and not reconcile, resulting in the mirrored EndpointSlice not being recreated until something updates the Service or Endpoints resource.
/triage accepted


Question: func (flowSchemaStatusStrategy) PrepareForUpdate(ctx context.Context, obj, old runtime.Object) { newFlowSchema := obj.(*flowcontrol.FlowSchema) oldFlowSchema := old.(*flowcontrol.FlowSchema) <ins> // managedFields must be preserved since it's been modified to // track changed fields in the status update. managedFields := newFlowSchema.ManagedFields </ins> newFlowSchema.ObjectMeta = oldFlowSchema.ObjectMeta <ins> newFlowSchema.ManagedFields = managedFields </ins> newFlowSchema.Spec = oldFlowSchema.Spec }

What happened: I've performed server-side apply on the subresource. e.g. The request body is the following for example: The CRD itself has status commonly defined as: in the resulting openAPIv3 validation the correct object annotations are set for listType & listMapKey (thanks to controler-gen v0.2.5) What you expected to happen: The patch is applied to the status subresource, and the is properly set. However, one of two things happens: if the status subfield conditions don't exist the request passes through and properly sets the field. But the isn't set if the specific status condition type exits the request fails with 409 Conflict. It conflicts with other managers, e.g. since the is empty. if the exist beforehand for the status subresource, they are cleared after server-side apply on the subresource and become empty. Anything else we need to know?: This is similar to Also somewhat related to Environment: Kubernetes version (use ):
/wg apply
/sig api-machinery
thanks for the detailed post, you're absolutely right that this is related to the two issues that you mentioned. I don't see how we would be able to fix this in 1.18 or any version before that, we're planning on fixing it in 1.19. We should probably update the documentation to mention that this doesn't work for sub-resources, and maybe even disable it altogether for sub-resources.
I'd rather have it working...buggily(that is via ) than having it disabled altogether.
We trying to find out what this doesn’t work and see if we can back port the fix. We’ll keep you posted, thanks!
I can't reproduce this bug right now, would you mind sharing the CRD and exact reproduction steps?
Sure. The server-side application I've investigated in This is the CRD definition: my experimentations are in on line I've changed whether I've used force or not. The raw HTTP req/resp should be written to the output due to
Let me know if there's anything more I can do to help.
The strategy overriding the metadata in status updates of CRD and CR objects seems like it would definitely cause this. were you able to follow the steps at


Question: **Fixed Versions**: - kube-apiserver v1.25.4 - kube-apiserver v1.24.8 <del> - kube-apiserver v1.23.13 </del> <ins> - kube-apiserver v1.23.14 </ins> - kube-apiserver v1.22.16 This vulnerability was reported by Richard Turnbull of NCC Group as part of the Kubernetes Audit

The section of the Changelog for 1.23.14 (and sibling bugfix releases) says that 1.23.13 both has the vuln and also has the fix. The Fix version should be 1.23.14 in the above and sibling changelogs. Load the link above. No response N/A N/A N/A N/A N/A N/A
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig docs
cc
For v1.23.14, the title is . This seems to be valid. means
You're looking at the wrong text. Follow the link and see .13 in both lists: kube-apiserver v1.25.0 - v1.25.3 kube-apiserver v1.24.0 - v1.24.7 kube-apiserver v1.23.0 - v1.23.13 kube-apiserver v1.22.0 - v1.22.15 kube-apiserver <= v1.21.? kube-apiserver v1.25.4 kube-apiserver v1.24.8 kube-apiserver v1.23.13 kube-apiserver v1.22.16
Got it. Thanks for pointing out the location.
I opened to fix it.
/reopen Can you please cherry-pick to the release branches (1.25-1.22)?
Reopened this issue. <detailsIn response to : /reopen Can you please cherry-pick to the release branches (1.25-1.22)? Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Of course. Done
Cherry-picks are done, thanks a lot /close
Closing this issue. <detailsIn response to : Cherry-picks are done, thanks a lot /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: natRules := bytes.NewBuffer(nil) writeLine(natChains, "*nat") // Start with chains we know we need to remove. <del> for _, chain := range []utiliptables.Chain{kubeServicesChain, kubeNodePortsChain, kubePostroutingChain, KubeMarkMasqChain} { </del> <ins> for _, chain := range []utiliptables.Chain{kubeServicesChain, kubeNodePortsChain, kubePostroutingChain} { </ins> if _, found := existingNATChains[chain]; found { chainString := string(chain) writeBytesLine(natChains, existingNATChains[chain]) // flush

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /sig network What happened: Kube-proxy () deployed as daemonset with ipvs mode produce during startup following error: With verbosity -v5 the logs looks like this: Looks like proxier is trying to perform cleanup part: but failed. Such error is generated only once during startup. What you expected to happen: Error should not exist :) Environment: Kubernetes version (use ): 1.11.1 Cloud provider or hardware configuration: none OS (e.g. from /etc/os-release): Ubuntu 16.04 LTS Kernel (e.g. ): Install tools: custom script based on docker Others:
I've build kube-proxy with additional information: and the logs looks like this:
same issue
Looks like also fixed this one.
I'm on 1.12.0 and the issue is still there I suppose is included in 1.12?
Updated my kube to 1.12.3 and I got same
I'm still seeing this on v1.12.4 as well
same here on v1.13.2, throws this error when kube-proxy initializes. Running on CoreOS in ipvs mode:
Hello, I also encountered this problem in Kubernetes 1.13.3. The operating system is CentOS 7.6. The kernel version is: Linux linux-node4 3.10.0-957.el7.x86_64
Why is this closed? It looks like an unresolved issue. I have the same errors with 1.12.1.
same as me
same as me,1.12.3
same problem OS: Ubuntu 18 kernel: 4.15.0-46-generic Kubernetes: 1.13.4
same problem OS: CentOS 7.6 kernel: 3.10.0-957.el7.x86_64 Kubernetes: 1.13.4
I met this issue on Centos 7.6 with NetworkManager installed, it seems related to the network.
Same problem OS: CentOS 7.4 Kubernetes: 1.12.5 NetworkManager not installed.

Some of the nodes does not have this issue while others do
I am getting in 1.14.0 as well in some worker nodes.
same problem,how to solve it
same problem 1.13.2 ,how fix it?
i update kube-proxy 1.14.2, the error message is disappear
same problem OS: Debian 8 Kubernetes: v1.15.0 iptables v1.4.21 kube-proxy mode ipvs I write some test code like kube-proxy --cleanup and found line 7 is , and run , the error is The problem is we should flush KUBE-LOAD-BALANCER and KUBE-NODE-PORT before delete KUBE-MARK-MASQ , I can try fix it.


Question: // VolumeDynamicallyCreatedByKey is the key of the annotation on PersistentVolume // object created dynamically VolumeDynamicallyCreatedByKey = "kubernetes.io/createdby" <ins> // kubernetesPluginPathPrefix is the prefix of kubernetes plugin mount paths. kubernetesPluginPathPrefix = "/plugins/kubernetes.io/" </ins> ) // IsReady checks for the existence of a regular file

See This happens on ubuntu images on GKE when the node ephemeral storage is on local SSD. /sig storage /assign
/triage accepted


Question: AfterReadingAllFlags(&TestContext) } <ins> func createKubeConfig(clientCfg *restclient.Config) *clientcmdapi.Config { clusterNick := "cluster" userNick := "user" contextNick := "context" config := clientcmdapi.NewConfig() credentials := clientcmdapi.NewAuthInfo() credentials.Token = clientCfg.BearerToken credentials.ClientCertificate = clientCfg.TLSClientConfig.CertFile if len(credentials.ClientCertificate) == 0 { credentials.ClientCertificateData = clientCfg.TLSClientConfig.CertData } credentials.ClientKey = clientCfg.TLSClientConfig.KeyFile if len(credentials.ClientKey) == 0 { credentials.ClientKeyData = clientCfg.TLSClientConfig.KeyData } config.AuthInfos[userNick] = credentials cluster := clientcmdapi.NewCluster() cluster.Server = clientCfg.Host cluster.CertificateAuthority = clientCfg.CAFile if len(cluster.CertificateAuthority) == 0 { cluster.CertificateAuthorityData = clientCfg.CAData } cluster.InsecureSkipTLSVerify = clientCfg.Insecure config.Clusters[clusterNick] = cluster context := clientcmdapi.NewContext() context.Cluster = clusterNick context.AuthInfo = userNick config.Contexts[contextNick] = context config.CurrentContext = contextNick return config } </ins> // AfterReadingAllFlags makes changes to the context after all flags // have been read. func AfterReadingAllFlags(t *TestContextType) { // Only set a default host if one won't be supplied via kubeconfig if len(t.Host) == 0 && len(t.KubeConfig) == 0 { <del> t.Host = defaultHost </del> <ins> // Check if we can use the in-cluster config if clusterConfig, err := restclient.InClusterConfig(); err == nil { if tempFile, err := ioutil.TempFile(os.TempDir(), "kubeconfig-"); err == nil { kubeConfig := createKubeConfig(clusterConfig) clientcmd.WriteToFile(*kubeConfig, tempFile.Name()) t.KubeConfig = tempFile.Name() glog.Infof("Using a temporary kubeconfig file from in-cluster config : %s", tempFile.Name()) } } if len(t.KubeConfig) == 0 { glog.Warningf("Unable to find in-cluster config, using default host : %s", defaultHost) t.Host = defaultHost } </ins> } // Reset the cluster IP range flag to CLUSTER_IP_RANGE env var, if defined. if clusterIPRange := os.Getenv("CLUSTER_IP_RANGE"); clusterIPRange != "" {

/kind feature I am running self-hosted E2E conformance tests in a pod on every freshly deployed cluster (similar as in ). is started without any host, kubeconfig configuration and picks up the InClusterConfig correctly. All conformance tests pass nicely, except from kubectl executions. kubectl gets executed by as and therefore fails on connection refused. In the container running kubectl without --server, will work as expected, since I have service accounts set up correctly. Looking into code, for some reason defaultHost is set to http://127.0.0.1:8080 ( ). That value is not used within as it seems, but is passed on to kubectl, only. Omitting would let the kubectl run with no problem, why is it set to that value (default proxy port is 8001)? Currently I am mitigating by pre-exectuting before running
/sig Testing
This is known, and there is a means to circumvent by passing in a kubeconfig. I've had a todo to fix this for a while, but I'll prioritize to get this fixed sooner. fyi -
can you please leave some notes on how you wanted to fix this? i can take a shot at it
if you can write the in-cluster config out and set KUBECONFIG prior to the kubectl tests that would help.
: ack. : Can you please send me the list of tests that fails? So i can be sure my changes will work for you?
- reviewing the code paths PR should do the trick, but would like to get that list of failed tests so i can confirm. Thanks, Dims
every conformance test that's shelling out kubectl is failing (20-some of the 150 conformance tests). as elegant as your PR looks, I think it won't work, due to authentication: kubectl will work out of the box on a pod with InClusterConfig, however, as soon as you start specifying one single parameter (such as --server), even if you do not change it compared to InClusterConfig, you will need to specify the rest, too. There might be two ways to do it: write out a kubeconfig file and reference it omit when shelling out to kubectl IMHO, the second option is much more elegant, but would need to test to confirm it works.

e2e AND kubectl work with InClusterConfig by themselves, why not leave it that way and only change anything, iff e2e gets any parameters (-host) specified. Otherwise, InClusterConfig should work as designed.
can you give me the list of the 20 tests please?
Sure, rerunning the tests, might take an hour.
Here's the first one: The rest will follow as a list.
It is a 1.7.2 test running against a 1.7.7 cluster.
See, there are three variants on how to invoke kubectl: Only the third one gets to the API and and gets authorized. If I had a kubeconfig file, that would also work.
Excellent! thanks will take a look at the scenarios. thanks for digging in.
/assign
[MILESTONENOTIFIER] Milestone Issue Current <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details


Question: // TODO: We could do a dry-run here if wanted to avoid the spurious cloud-calls & events when we restart <del> // The load balancer doesn't exist yet, so create it. s.eventRecorder.Event(service, v1.EventTypeNormal, "CreatingLoadBalancer", "Creating load balancer") newState, err = s.createLoadBalancer(service) </del> <ins> s.eventRecorder.Event(service, v1.EventTypeNormal, "EnsuringLoadBalancer", "Ensuring load balancer") newState, err = s.ensureLoadBalancer(service) </ins> if err != nil { <del> return fmt.Errorf("Failed to create load balancer for service %s: %v", key, err), retryable </del> <ins> return fmt.Errorf("Failed to ensure load balancer for service %s: %v", key, err), retryable </ins> } <del> s.eventRecorder.Event(service, v1.EventTypeNormal, "CreatedLoadBalancer", "Created load balancer") </del> <ins> s.eventRecorder.Event(service, v1.EventTypeNormal, "EnsuredLoadBalancer", "Ensured load balancer") </ins> } // Write the state if changed

According to code snippet The handles the ServiceUpdate events, it first check out whether or not current service need a LoadBalancer, and if not, it ensures corresponding LoadBalacner deleted. However, another branch of the statement is a little bit confusing. It says The load balancer doesn't exist yet, so create it_, and then it calls . I think at this time, we're not sure whether or not the load balancer already exists (since we didn't call ), so the proper behavior would be ensure load balancer exists (actually just calls indeed), so the comment (L-283) and the function name (L-285 ) isn't very appropriate, it's confusing right?
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig network
/area cloudprovider
Please see
I just follow and I think these are two different issues because 1) focus on the potential logical error in code (should check existence before calling create..) which I don't agree. I think the code logic is fine. But, 2) This issue focus on the code/comment expression, which means the code logic is okay but the comment and function name is NOT appropriate.
The only comment I think should be changed is L283.
Yes, I mean L283 too, however, I think the function name s. should also be changed.
- it may be called , i agree that if you do not have prior knowledge of the servicecontroller it may be confusing. can you open a PR ?
Takes me a lot of time for the first PR preparation... Now I've opened the PR, please check it out.


Question: if pendingGraceful { return e.finalizeDelete(obj, false) } <ins> var ignoreNotFound bool = false var lastExisting runtime.Object = nil </ins> if graceful { out := e.NewFunc() lastGraceful := int64(0)

cc
PTAL
The root cause is that the node controller and kubelet delete the pods using their names (they should use UID instead): (this is not an exhaustive list) In test-, we create and delete the same pod multiple times, so it's possible that the node controller or the kubelet sends a deletion request to the api server, with the intention of deleting the Nth occurrence of a pod, while it's actually deleting the N+1th occurrence of the pod. To reproduce the flake, I introduced a random delay between 500~1500 ms in the node controller's execution path (), then using a script to create/delete a pod repeatedly, and the flake will occur after several rounds.
Why is it not okay to delete by name?
namespace+name pair is not an unique identifier across time.
the client library and the API server don't support deletion by UID, I think we should support.
Thanks!! My thoughts: does node controller care about these pods? I'm pretty sure the original author wasn't expecting that. a temporary fix, is it hard to just use new names for every incarnation of these pods?
Deleting by UID is probably a bigger can of worms than we want to open right now, although it will eventually be important. Thinking out loud here, but maybe it'd be possible to add a resource version to the delete command. I.e., "delete only if it hasn't changed since rv=xxxxx". But I still think it's probably ideal to turn off the node controller for this test.
I'm also thinking if we can centralize all the pod deletion logic into kubelet, but it seems some of the logic need to stay in the node controller, e.g, , because only the node controller knows if the node exists. deletion probably can be done by the kubelet.
Looks like node controller is getting an update of the pod where DeleteionTimestamp is set, which seems like a bug if --grace-period is set to 0. My understanding is that if grace period is zero, the pod should get immediately deleted. That pod will never make it to a kubelet because it hasn't been scheduled yet. Thus which kubelet would delete it? I'm pretty sure none of those deletions can be handled in the kubelet.

Btw, when running hack/test- (), the node controller generates many messages about the failed deletions: or , these are because kubelet wins the race and has deleted the pods. These don't cause test- to fail.
per offline discussion with Mike, I think he's right. I'll check the api server's code.
That's my expectation, too (that grace period being zero should bypass all graceful deletion code and just immediately delete the pod).
When the grace period is zero, api server update the storage(), and then delete the pod from the storage (). The check of if grace period is 0 is after the update(). It seems it should not do the update if grace periods is zero.
Do you agree that the above is a bug?
Hm, is the problem a missing or incorrect BeforeDelete handler?
I think we are dealing with two problems here: grace-period=0, the registry shouldn't do an update. client library should provide a Delete() method that takes an unique identifier, the "name+namespace" pair is ambiguous across time.
Thinking about this more, I wonder if the update is intentional. If so, we need some coordination so that both apiserver and kublet don't try to delete.
Grace period 0 is intentional, but may not be required for correct behavior. If the user signals delete immediately, we record that intent so it shows up in watch and can be a signal kubelet to terminate immediately (although we don't actually do that today, so we could walk back the behavior). We definitely need a precondition delete on uid.
There will never be a single owner of pod lifecycle because pods are bound to nodes, and so all components have to deal with races to delete.
I chose to err on the side of being explicit about user intent with the update to zero out termination. But technically explicit deletion can be the same signal.
A delete operation will also trigger a watch event, so kubelet will be signaled even if we skip the update right? [edit] Oh I see you answered right before I asked.
It should be, yes. In the interest of fixing this test in the short term, I think the easiest thing to do might be to turn off the node controller for this section of the test. (Is it needed by anything in the test?)

But we should file at least two issues: "kubectl delete can race with graceful deletion"-- Things have to handle races, but there shouldn't be a race where you delete something and your delete fails because your own delete caused another component to issue a delete for the same thing... As far as I can see from the code, this could happen if apiserver context-switched at the wrong times (even if the more common thing of an old delete getting a new object is the issue at hand). controllers should delete via UID (or namespace/name + RV) -- I wouldn't be surprised if there's already an issue for this.
+1 to resourceVersion and uid preconditions on delete. Yes, maybe we shouldn't update the grace period before deletion in the case that the grace period is 0. Why can't we use unique pod names in the test? Turning off node controller seems like a bad idea. Index by uid is .
I don't see any flag option to turn of node controller. Pod's name is specified in the config file, is there any clean way to change the pod's name each time is called? Anyway, I'll try to make a PR to skip update if grace period is 0. That should alleviate this flake.
cc/ in case you are running into this issue.
As for how to change the name: we could use sed to replace the name. Or we could change the namespace.
This should be fixed by . I filed for deletion via UID or namespace/name+resourceVersion.
This happened again:
This error "] pods "redis-proxy" not found" here seems related to the error in .
I think the issue in is the same. The last few lines of logs: The symptom seems to align with my hypothesis in suggests the nodecontroller has successfully deleted a pod (probably "redis-master"), so kubectl fails with . I'll add more logs in nodecontroller to make sure this is the problem. In fact, we knew this race condition when investigating , but we discovered other race condition that are more likely to happen so we didn't solve this one.
Sent to show what pod nodecontroller has deleted.
Anywhere where we delete lists of pods (and other resources), including those filtered by label selector, we should ignore not-found errors. There are such loops in the controller reapers for cascading deletion. cc
makes safely ignore the NotFound error.

forces to issue a "GET" before submitting "DELETE". The "GET" will effectively check if the user is trying to delete a resource that is never created, so if later the API server sends a NotFound error in response to the "DELETE" request, kubectl can safely ignore it.


Question: podutil "k8s.io/kubernetes/pkg/api/v1/pod" "k8s.io/kubernetes/pkg/client/clientset_generated/clientset" coreinformers "k8s.io/kubernetes/pkg/client/informers/informers_generated/externalversions/core/v1" <ins> "k8s.io/kubernetes/pkg/client/leaderelection/resourcelock" </ins> corelisters "k8s.io/kubernetes/pkg/client/listers/core/v1" "k8s.io/kubernetes/pkg/controller" "k8s.io/kubernetes/pkg/util/metrics"

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG OR Do i miss something important? Kubernetes version (use ): k8s 1.3.5. still exists in 1.6.2 What happened: HA of kube-controller-manager and kube-scheduler is achieved by create endpoint and update it periodically. However, endpoint controller deletes endpoints without corresponding services which is implemented in checkLeftoverEndpoints func: That will result in leader transition. What you expected to happen: checkLeftoverEndpoints func shouldn't delete endpoints used by LeaderElection. How to reproduce it (as minimally and precisely as possible): turn on LeaderElection for kube-controller-manager and kube-scheduler. After five minutes, endpoint controller will delete orphan endpoints. Anything else we need to know: we can add some logic in checkLeftoverEndpoints, for example, not add endpoint in kube-system namespace to queue.
Fixed by
Thanks