Question: execScripts: []ExecArgs{ {"fsck", []string{"-a", "/dev/foo"}, "", nil}, {"lsblk", []string{"-nd", "-o", "FSTYPE", "/dev/foo"}, "", nil}, <del> {"mkfs.ext4", []string{"-E", "lazy_itable_init=0,lazy_journal_init=0", "-F", "/dev/foo"}, "", nil}, </del> <ins> {"mkfs.ext4", []string{"-F", "/dev/foo"}, "", nil}, </ins> }, expectedError: nil, },

Here's logs from formatting a 10GB HDD disk in Azure as ext4: It took seven and a half minutes to complete . Here's me doing it manually from the node: And even from a container, somewhat emulating kubelet: Kubernetes version (use ): Environment: Cloud provider or hardware configuration: Azure OS (e.g. from /etc/os-release): Ubuntu (Xenial) 16.10 Kernel (e.g. ): Linux 4.4.0-34-generic -Ubuntu SMP Wed Jul 27 16:06:39 UTC 2016 x8664 x8664 x86_64 GNU/Linux Install tools: kubernetes-anywhere / static pods with kubelet / kubelet in container What happened: It took seven and a half minutes to format, causing the overall pod startup time to take over 10 minutes by time attachment occurred What you expected to happen: It to take a matter of seconds, similar to how it does if I execute the command manully. How to reproduce it (as minimally and precisely as possible): Build with on top. Deploy a cluster. Try to attach an unformatted VHD. Observe along with watching logs and confirm it really does take that long. Anything else do we need to know: I'm running containerized. Like this:
Can you confirm you were not seeing this behavior during your testing? I assume you were not running containerized?
Even if I and execute the same command from there it executes in a matter of a few seconds.
I'm seeing this behavior even when running as a normal systemd unit, and not as a docker container under a unit. Here's : And I'm still seeing the very slow , so it seems not related to running kubelet in container. I have also tested to ensure that this isn't something some sort of host caching. Even on another blank disk being attached for the the first time, executes quickly except when run by .
I confirm this is this case on my setup running local-up- I have a 100GB HDD disk. It took 30 seconds to format on the console but 2 minutes by kubelet. Is there any I/O constraint imposed on kubelet?
to see if she knows why would be slow when called from the kubelet.
cc/ Who is working on node performance benchmark.

Can we quickly run an experiment on GCE to see if we can reproduce the issue. Let's first rule the impact of containerized Kubelet. If the issue is still exist, I suspect the issue in Kubelet mounting logic. We can enable some tracing, and find the bottleneck.
I tried formatting a 10GB HDD in GCE Kubernetes (not containerized). The time of formatting in kubelet log is ~1s. The log is:
On Azure, a 10GB disk took 18s to mkfs by kublet but 4s to mkfs on the console. Do you have a chance to test these two scenarios? Size probably is not a critical factor for gce since gce pd is provisioned at fixed IOPS per GB, while Azure (also AWS) disks are provisioned at fixed IOPS per volume.
There's more variables in play or else I'm seeing a different edge case than you are, On Azure with a cluster, a 10GB disk for me consistently takes 7+ minutes. (, standard storage, HDD type). It was happening with a and with premium storage and SSD type data disk too... I can make a cluster available to anyone who would like to investigate the exact behavior I've been observing. I can provide kubeconfig+ssh creds. Contact me on Slack (colemickens) if interested.
mkfs's kernel stack after started by kubelet is always as the following. Not sure if it related to blkio cgroup
I have actually created a branch of to give me a cluster where is never running in a container and I'm still seeing the exact same issue. Took 7.5 minutes again, based on kubelet logs and watching . I think we can safely eliminate "containerized kubelet" as being the source of the problem.
Though if I remember correctly, kubelet places itself in a cgroup when it starts up. ( can confirm) so it's possible that it is still due to cgroup I/O throttling.
Above test result proves my initial suspect that the root cause in 7-min use case is containerized kubelet. But recent experiment is not running in a container.

could you please provide us more information: 1) cat /proc/<kubelet-pid/cgroups to find kubelet's cgroup 2) check & blkio.weightdevice setting 3) ps ax | grep makfs to find process id, the find which cgroup the process belongs to 4) check 3) cgroup's and blkio.weight_device If both 2) and 4) don't have issues with cgroup setting, please check if there is parent cgroup for them, and their setting since both CFQ and throttling supports hierarchy.
FYI: Kubelet does not perform any block IO throttling as of now for itself or any other entity on the node. , Dawn Chen wrote:
Yes, we know Kubelet doesn't. But the issue here indicates containerized Kubelet's IOs is throttled, and we are trying to figure out why, especially the user is running kubernetes-anywhere on Azure, which we, node-team don't normally test.
Here's how I was running originally: Here's how I was running when testing it not in a container: I will try to collect the data you mention but I'm going to have to learn how to do some of it along the way, I don't know much about cgroups.
this is not limited to azure, saw it on aws as well.
it seems only the hit only happened to ext4, xfs didn't have this problem.
I ran across the same performance issues and tried to investigate. I use Azure Dynamic Disk provisioning to provision a 1TB volume and it takes very long time for mkfs to finish. Doing the "mkfs.ext4 -F -E lazyitableinit=0,lazyjournalinit=0 /dev/sdd" manually is also very slow, but I did not compare the performance to the call done from inside kubelet. While running the mkfs manually, I observed that iotop shows around 62mb/s and that the free memory immediately shrinks from 2000mb to 100mb. It looks like mkfs is using up all free memory for the buffer cache. I even observed oom situations in the kernel log and processes being killed because of this. I've seen this behavior while running mkfs manually and while it was run by kubelet.

Removing the extended flag "lazyitableinit=0" from the manual mkfs call fixes both problems, performance and memory/buffer cache usage. My solution will now be to use xfs (as rootfs reported it to work better). But IMHO the lazyitableinit should be completely removed from the mkfs call.
Can we try to make sure this gets addressed in 1.6?
I did some further investigation here and it turns out there are other factors involved as well, especially on Azure. My initial statement that I got OOM due to mkfs.ext4 seems to be not true, as the OOMs were caused by fluentd-elasticsearch. I tried to use mkfs.ext4 with "-E discard=1" as I assumed that it should speed up things dramatically. This did not make a difference however. While trying to understand the mke2fs code from e2fsprogs, I realized that discard is enabled by default and always performed as first step when the underlying block device supports this. From the mkfs.ext4 logs you can actually see that it is executed at the beginning. It turns out that mke2fs does only skip inode table initialization when the kernel reports that a discard also ensures zeroing of the data. mke2fs checks this by doing an ioctl to BLKDISCARDZEROES. Support can also be checked with On Azure however, the kernel reports that a discard does NOT zero out the discarded blocks. This results in a loop, using normal write calls to write a lot of zeros to the block dev. This completely fills up the page cache with dirty pages, slowing down the whole system extremely. If there is not much free memory available that can be used for write buffering, the write rate goes down to very low values as well. I did not check the kernel code for lazyitableinit, but I would assume that it uses much more efficient code to write the zeros to the block device (e.g. by not relying on dirty pages). I'd also assume that it does these checks for the discard+zeroout behavior/support and then use discard if possible. If my assumption is correct, then it should not affect performance in a bad way when lazy init would be enabled again. The kernel would then do the discards in a lazy fashion, but finish pretty fast.

On Azure, the lazy init would still take some time, but use much less system resources (page cache) as the non-lazy solution. And most importantly, the device can already be mounted after a few seconds. (UPDATE: please check next comment) I tried to figure out if a discard (using the blkdiscard cmd) does really leave you with non-zeroed data and it looks like it actually becomes zero. I can however not see if this happens immediately or with a delay. Also, in the Azure Portal I can see that the "Billable Size" goes from several GB back to 0 after a short time. My assumption is that on other cloud providers, e.g. GCE and AWS, the block devices report that discard zeroes out the data. Can anyone with a GCE/AWS machine check this with the cat call from above? This would explain why this slowness does not occur on non-Azure clouds. Can you check why the kernel reports that discard does not zero-out? Is this a bug in Azure maybe?
~Ok, my assumption about the kernel using discard for lazy init was wrong. Just checked the kernel source and it simply issues multiple write bios which write zero-pages. So it would definitely have a negative impact on other cloud providers when lazyitableinit would be enabled.~ Would it be a viable solution to check support for BLKDISCARDZEROES in kubernetes before doing the mkfs and then decide based on that if lazyitableinit should be enabled or not?
Argh, digging deeper into the kernel turned out that it actually uses discard for zeroing out (check and ). So my initial assumption was correct. Sorry for all the go home now :)
Just checked the discard support on AWS and it is not supported at all, so it always does a normal write to clear/init the inode tables.
here's the output from a GCP instance: We haven't seen any slowness on GCP, even though the above settings are identical to our Azure instance's. We're trying to solve the same slowness issue on Azure, just not with kubernetes :) (we're Cloud Foundry). We're not using containers on this particular VM. Also, the problem manifests with but not .

Related issue here:
We discovered that by using instead of the problem went away — disks which previously took 8+ minutes to mkfs.ext4 took seconds. Alternatively, will also work as a partitioner (i.e. the subsequent will finish within seconds). Really. We're not sure why the tool we use to write one sector (i.e. the MBR) on a disk would make such a difference. HTH,
Did you discover this by manually running / on the nodes, or did you actually modify to use ?
This sounds like alignment issues. As far as I remember, sfdisk does not do proper alignment and thus may result on very slow performance when you are not on classical spinning disks. Azure disk provisioner does not even create partitions on the disks and thus alignment should not be the problem here. Just tested creating a partition with sfdisk and iotop showed only 7-10mb/s throughput when doing mkfs. Not using a partition at all or using fdisk gives around 60mb/s.
: We discovered this by manually running gparted / sfdisk on the instances we deployed on Azure (i.e. we're not using containers). We don't use Kubernetes — we are developers on Cloud Foundry, which is an alternative to Kubernetes. When we ran into this problem, we were pleased to discover we were not the only ones who have been bitten by this bug. And we wanted to make sure our Kubernetes kin were aware of the fix that we found to be effective. some notes on our research (and I cannot thank you enough for your detailed descriptions): we also found throughput of 7-10MB/s when was healthy when things were unhealthy, showed throughput of ~300kB/s during . we found that by passing (use direct IO and bypass buffer cache), we doubled our performance to ~700kB/s, but it was still ridiculously long (3 minutes).
I am going to try to do some additional investigation tonight, but I'm unsure of how to translate your observations to Kubernetes here. As implied, we don't use anywhere here (that I'm aware of). There's three scenarios: -dynamically provisioned. User brings a pre-formatted VHD. Everything goes smoothly as the merely needs to mount the block device and tell to mount it into the container. -dynamically provisioned.

User attaches an empty disk to an Azure VM, detaches it, uses it in a normal AzureDisk volume on a Pod. This is the scenario in the original post, and you can see that it is being mkfs'd without any partitions. disk provisioning. The VHD footer is written into the page blob, and then formats the whole disk, unpartitioned (trusting description of the provisioner here). I'm hesitant to rule out an alignment issue since you're seeing similar symptoms as we are here, but I'm not sure how to reconcile that with my other points in this comment...
What do you think about It would solve this issue. It would not tell us why it is slower in kubelet, but the problem itself would be solved. Should I CC more people to get this discussed faster? I would even like to see this in an upcoming 1.5.x release.
a lead Azure Engineer ( ) found the problem — it is an alignment problem gone horribly wrong: In situation like unaligned partition boundary, a big I/O request (e.g. a 32KB write) can be split into 64 512-byte writes by the kernel block I/O subsystem in the worst case, easily hitting the 500-IOPS throttling of Azure. It was made to simplify Hyper-V storage driver’s code, so Hyper-V’s storage driver picked up this patch in kernel versions: 4.4+, that is why only Azure is impacted. The community member who submitted the original patch already for this issue, waiting for Linux community to merge.
Does that also affect cases where the disk is unpartitioned, ?
If I understand the description correctly, this only applies to cases where unaligned partitions are in use. This is not the case for Azure Dynamic Disk provisioning and also for all the tests performed so far by all the participants here. As noted in , I did some manual tests with partitioning. And having unaligned partitions really degrades performance a lot (reducing mkfs throughput from 60mb/s to 10mb/s). But as said, we don't have unaligned partitions when using Azure Dynamic Disk actually don't have any partitions at all as we create the FS directly on the block device (e.g. /dev/sdc instead of /dev/sdc1). The real issue here is that lazy inode table initialization is disabled for mkfs.ext4.

The reason for disabling it can be found here: The assumption (BLKDISCARD+BLKDISCARDZEROES support) made in this commit is not true for all cloud providers, e.g. Azure and at least for AWS magnetic disks. I did some calculations to figure out how much zeroes must be written by mkfs.ext4 to initialize the inode tables for a 1TB disk. If you check "tune2fs -l /dev/sdX" you'll see that there are inodes with a size of 256 bytes allocated for a 1TB file system. This gives a total of 16G! These 16G of zeroes are written by mkfs.ext4 when lazy init is disabled, in a very very naive way. Writing 16G of zeroes already takes 4:30 minutes when you have a throughput of 60mb/s. Now if there is something that throttles the troughput (and it looks like kubelet does this somehow), it gets even worse. And that's not all, while this is written do disk, the complete system suffers on low memory due to all of it being used for write caching/buffering. If other processes want to allocate memory in this situation, they're blocked until enough from the buffers is flushed to disk. Please check It fixes all this by simply re-enabling lazy inode table initialization. I also tried to describe why this change should not affect cloud providers that previously thought that disabling lazy init would be an improvement.
I don't believe the partition alignment issue would affect cases where the disk is unpartitioned (i.e. using the entire disk, e.g. ). I suspect that any use of an entire raw device would be properly aligned (any filesystem that starts on sector 0 of the hard disk would be properly aligned), but that's a hunch and nothing more. : I think your assessment is on target — the problem we (Cloud Foundry) are seeing is caused by a partition misalignment, but it seems that the problem described in this thread has a different cause. We currently use a lazy inode table initialization, so we're covered on that front:

According to : While the is exactly the opposite. This significantly slows down large ext4 volume creation.
This might be worth bringing up at the storage f2f coming up.I am going to add to the agenda. This is an interesting approach to what I think may be a few larger issues. Questions and thoughts: What is a good SLO for mkfs? If the SLO is going to be violated due to size should k8s do something or should the user pre-format? There are tradeoffs for initting the filesystem lazy or not. Supporting all the options for a user may require an API change. Is this a persistent volume only thing?
I may miss something, but on a random GCE node with 500GiB magnetic(!) disk and Fedora 24 takes around 2 seconds. ext3 is a different story, it takes 74 seconds.
I cannot imagine any reason why anyone would be installing ext2/ext3 on a 500GB drive. If they do they should be expecting things to be slow. In any case knowing what the SLO is is still not a crazy idea.
This is probably not a big problem with GCE, since GCE PD's IOPS is per GB. This is mostly a problem for EBS/Cinder/Azure, since the IOPS per volume is fixed.
If we're going to do this at all, we should really think hard about this problem top-to-bottom. Ext3 vs ext4 vs other FS? Myriad filesystem options, etc.
I got bitten by this when implementing e2e tests for all supported storage class options on AWS. "Cold HDD" volume type has max. 250 IOPS and the smallest one is 500 GB. It takes 3-4 minutes just to format the volume, which slows down the test quite considerably. Was there any conclusion on f2f meeting? was in commit , which claims to be rewritten to Go. Original script not exist any more on the link above. Was there some smart idea in original ?
see some inconclusive investigation at
Please take a look at I encountered not only performance issue due to lazyitableinit=0, but also oom situations and thus killed processes. Due to the amount of data written to the disk, all free memory gets filled with dirty buffer cache.


Question: "strings" "k8s.io/apimachinery/pkg/util/json" <del> forkedjson "k8s.io/apimachinery/third_party/forked/golang/json" </del> "k8s.io/apimachinery/pkg/util/mergepatch" <ins> forkedjson "k8s.io/apimachinery/third_party/forked/golang/json" </ins> ) // An alternate implementation of JSON Merge Patch

Extracted from The RawExtension type allows an API object to inline arbitrary JSON. strategicpatch.CreateTwoWayMergePatch returns an error trying to map that inlined JSON to struct fields. It should be able to compute a patch for an object containing a field of that type.
cc
We need to decide what strategic merge patch should do in the following scenarios: unknown fields (can be encountered when the API or a user-defined object has a new field the compiled-in struct doesn't have yet) unmergeable fields (like RawExtension) cc
The current behavior is: unknown fields: trying to merge a unknown field will fail when lookup the metadata in the struct tags unmergeable fields: will be replaced as raw byte array Maybe we can use generic json merge for unmergeable fields?
Did that change since 1.5? In 1.5, we were getting an error during patch computation for objects with a RawExtension field
Sorry. This behavior is in 1.4.7. e.g . It will error in 1.5.
Yeah, the field gets replaced. Even in 1.4.7, an unknown field (like initcontainers in pods) or an object including a RawExtension (anything with a field of type or ) would error
My personal favor would be to instruct the patch to ignore such field, seems the cleanest approach to me. Similarly to how we allow annotating deletion and merging.
wouldn't it be confusing to submit a patch with stuff in that field, have the patch accepted, and have none of your submitted changes take effect? seems like the least surprising thing would be to treat struct fields like or as
How about both, iow. having as a default and as an opt-in. This would allow the resource author to decide how his resource should be treated. Eventually on we could return the ignored fields. Having said that we could start with right away. I think I could work on a patch implementing that if majority agrees (
I think is a good place to start when computing/applying patches for known struct fields like RawExtension. I'll open a separate issue for client-side handling of computing patches for unknown fields
Ignore the field sounds like a really good solution. WDYT of printing a warning to stdeer?
what does ignoring instead of replacing get us?
Sorry I was reading the comment on my phone between other things. Ignore my comment.


Question: }, }, } <ins> } createAndWaitPod := func(pod *v1.Pod) error { podClient.Create(pod) return framework.WaitForPodSuccessInNamespace(f.ClientSet, pod.Name, f.Namespace.Name) } var logPodName string BeforeEach(func() { if framework.TestContext.ContainerRuntime == "docker" { // Container Log Path support requires JSON logging driver. // It does not work when Docker daemon is logging to journald. d, err := getDockerLoggingDriver() framework.ExpectNoError(err) if d != "json-file" { framework.Skipf("Skipping because Docker daemon is using a logging driver other than "json-file": %s", d) } // Even if JSON logging is in use, this test fails if SELinux support // is enabled, since the isolation provided by the SELinux policy // prevents processes running inside Docker containers (under SELinux // type svirt_lxc_net_t) from accessing the log files which are owned // by Docker (and labeled with the container_var_lib_t type.) // // Therefore, let's also skip this test when running with SELinux // support enabled. e, err := isDockerSELinuxSupportEnabled() framework.ExpectNoError(err) if e { framework.Skipf("Skipping because Docker daemon is running with SELinux support enabled") } } podClient = f.PodClient() logPodName = "log-pod-" + string(uuid.NewUUID()) err := createAndWaitPod(makeLogPod(logPodName, logString)) framework.ExpectNoError(err, "Failed waiting for pod: %s to enter success state", logPodName) }) It("should print log to correct log path", func() { logDir := kubelet.ContainerLogsDir // get containerID from created Pod createdLogPod, err := podClient.Get(logPodName, metav1.GetOptions{}) logContainerID := kubecontainer.ParseContainerID(createdLogPod.Status.ContainerStatuses[0].ContainerID) framework.ExpectNoError(err, "Failed to get pod: %s", logPodName) // build log file path expectedlogFile := logDir + "/" + logPodName + "_" + f.Namespace.Name + "_" + logContainerName + "-" + logContainerID.ID + ".log" logCheckPodName := "log-check-" + string(uuid.NewUUID()) err = createAndWaitPod(makeLogCheckPod(logCheckPodName, logString, expectedlogFile)) framework.ExpectNoError(err, "Failed waiting for pod: %s to enter success state", logCheckPodName) }) It("should print log to correct cri log path", func() { logCRIDir := "/var/log/pods" // get podID from created Pod createdLogPod, err := podClient.Get(logPodName, metav1.GetOptions{}) podID := string(createdLogPod.UID) // build log cri file path expectedCRILogFile := logCRIDir + "/" + podID + "/" + logContainerName + "/0.log" </ins> <del> podClient.Create(checkPod) err = framework.WaitForPodSuccessInNamespace(f.ClientSet, checkPodName, ns) framework.ExpectNoError(err, "Failed waiting for pod: %s to enter success state", checkPodName) </del> <ins> logCRICheckPodName := "log-cri-check-" + string(uuid.NewUUID()) err = createAndWaitPod(makeLogCheckPod(logCRICheckPodName, logString, expectedCRILogFile)) framework.ExpectNoError(err, "Failed waiting for pod: %s to enter success state", logCRICheckPodName) </ins> }) }) })

The should be updated to check CRI log paths.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
May I try to help with this?
GitHub didn't allow me to assign the following users: fatsheep9146. Note that only and repo collaborators can be assigned. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: status := v1.PodStatus{ Phase:   v1.PodFailed, <del> Message: nodeShutdownMessage, </del> Reason:  nodeShutdownReason, <ins> Message: nodeShutdownMessage, </ins> } err := m.killPod(pod, status, &gracePeriodOverride)

When using Graceful Node Shutdown feature, status and reason of "shutdown" pod are confusing for users. See where I attempt to address this confusion by documenting the behavior. However, it would be great to improve messages. Perhaps the following renaming: Reason "Shutdown" is confusing because it may appear as a process of shutting down. Better message will also be useful. Reason and Message changed for pods terminated in response to Node shutdown. Shutdown the node and observer the pod status. KEP: kubernetes/enhancements /cc /sig node /help /good-first-issue Kubernetes version (use ): 1.19+ (with feature enabled)
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : #### What happened: /cc /sig node /help /good-first-issue Kubernetes version (use ): 1.19+ (with feature enabled) </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc
/assign
/assign


Question: err := framework.RunRC(config) framework.ExpectNoError(err, "Replica must be created") } <ins> // CreateTCPUDPServicePods creates a replication controller with the label same as service. Service listens to TCP and UDP. func (j *TestJig) CreateTCPUDPServicePods(c clientset.Interface, ns string, replica int) { config := testutils.RCConfig{ Client:       c, Name:         j.Name, Image:        framework.ServeHostnameImage, Command:      []string{"/agnhost", "serve-hostname", "--http=false", "--tcp", "--udp"}, Namespace:    ns, Labels:       j.Labels, PollInterval: 3 * time.Second, Timeout:      framework.PodReadyBeforeTimeout, Replicas:     replica, } err := framework.RunRC(config) framework.ExpectNoError(err, "Replica must be created") } </ins>

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Following are two e2e testing service functioning over same port but different protocol, but do not test service reachability. be able to update NodePorts with two same port numbers but different protocols use same NodePort with same port but different protocols Note: Both the e2e are verifying the same behavior of "service using same port but different protocols" differs in and . Primary goal: Test service reachability for these E2Es. Secondary goal: As both the above e2e are verifying same E2E behavior, it's good to have single e2e in spite of 2 differing in "updating" and "create only" nodePort service. For "create only", suite has another specific e2e . Why is this needed: E2E Improvement and make e2e eligible for Conformance. Service related e2e are getting promoted in /area conformance /sig testing /assign


Question: stickyMaxAgeSeconds = int(*service.Spec.SessionAffinityConfig.ClientIP.TimeoutSeconds) } info := &serviceInfo{ <del> clusterIP:  net.ParseIP(service.Spec.ClusterIP), port:       int(port.Port), protocol:   port.Protocol, nodePort:   int(port.NodePort), </del> <ins> clusterIP: net.ParseIP(service.Spec.ClusterIP), port:      int(port.Port), protocol:  port.Protocol, nodePort:  int(port.NodePort), // targetPort is zero if it is specified as a name in port.TargetPort. // Its real value would be got later from endpoints. </ins> targetPort: port.TargetPort.IntValue(), // Deep-copy in case the service instance changes loadBalancerStatus:       *service.Status.LoadBalancer.DeepCopy(),

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: Tried to create a service using targetPort as a string referencing the name of the pod port. Traffic was not routed to pod. What you expected to happen: Service to route traffic to pod How to reproduce it (as minimally and precisely as possible): See line targetPort Anything else we need to know?: Link that describes how this feature is supposed to work Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-04-27T09:22:21Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"windows/amd64"} Server Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-09-27T16:55:41Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): WindowsBuildLabEx : .1000.amd64fre.rs_prerelease.-1413 Kernel (e.g. ): Install tools: Others:
/sig windows
/assign


Question: // Creates a new StorageOS volume and makes it available as a device within // /var/lib/storageos/volumes. func (u *storageosUtil) CreateVolume(p *storageosProvisioner) (*storageosVolume, error) { <ins> klog.V(4).Infof("creating StorageOS volume %q with namespace %q", p.volName, p.volNamespace) </ins> if err := u.NewAPI(p.apiCfg); err != nil { return nil, err }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When a PVC is created and the PV mounted into a Pod using the StorageOS in-tree volume driver, the StorageOS CLI & UI do not report that the volume is mounted. The PV is created and mounted into the Pod successfully, and all k8s functionality works as expected. However, since the correct mount status is not visible to administrators using the StorageOS UI or CLI, they may make changes to or delete a mounted volume without realising, and the safeguards (must use to delete a mounted volume) do not trigger. This is a regression caused by , released in 1.14.0. What you expected to happen: The StorageOS CLI should report volume mount information (node name, mount timestamp). How to reproduce it (as minimally and precisely as possible): StorageOS node containers with CSI disabled: application Pod with a PVC using the StorageOS StorageClass: StorageOS to list volumes: Anything else we need to know?: Environment: Kubernetes version (use ): 1.14.0 and later Cloud provider or hardware configuration: any OS (e.g: ): any linux (incl. Ubuntu 18.04) Kernel (e.g. ): any linux (incl. 4.15.0-33-generic) Install tools: any (incl. minikube) Network plugin and version (if this is a network-related bug): N/A Others: Requires StorageOS deployment /sig storage /assign


Question: // test sources _ "k8s.io/kubernetes/test/e2e/apimachinery" _ "k8s.io/kubernetes/test/e2e/apps" <ins> _ "k8s.io/kubernetes/test/e2e/architecture" </ins> _ "k8s.io/kubernetes/test/e2e/auth" _ "k8s.io/kubernetes/test/e2e/autoscaling" _ "k8s.io/kubernetes/test/e2e/cloud"

I can create a cluster with one node and pass conformance, ~there are some tests that require at least 2 nodes to run~ ~~ ~This is because the tests Skip if they are not able to run~ ~~ ~As consequence, the list of tests for conformance depends on the number of nodes, a single cluster node (that is not a cluster by definition of cluster :)) will run less tests than a multi cluster node and still be Conformant~ EDIT I have to correct myself, it seems that the conformance tests doesn't skip, the problem is: there are no conformance tests that exercise anything that requires more than one node, hence, single node cluster can pass conformance. I expect a cluster (a group of similar things or people positioned or occurring closely together) and a "distributed system" to not be able to be single-node. In addition, lots of most important Kubernetes features can not be exercised on one node, lowering the quality bar. local- pass conformance Related discussion <details</details<details</details<details</details<details</details<details</details<details</details
/assign Maybe this has changed, but last time I've checked the minimum for conformance was 1 control-plane and 2 workers, if that didn't change I will submit a PR to enforce it in the suite Regarding sig-network, with one node you can not verify the networking model, where pod to pod in different nodes should be able to communicate without NAT, that is a strong requirement ... I have to review closely what tests do we have in Conformance
/area conformance /sig network
that seems pretty intentional... changing the definition of conformance to require multiple nodes probably deserves more feedback than an issue/PR
I'm not changing the definition, just asserting it
if we are testing features like replicaset, daemonset, services, ..., in one node only we are missing a lot of testing on those features ... a podman/docker single node can be a kubernetes conformance cluster ... do I need a scheduler or network for anything, I'm just deploying always in the same node?
No I don't think so. Where are your results and configuration? You need two schedulable nodes to pass the conformance suite. That's not correct, we test rolling pod behavior with replicasets. This test requires multiple nodes. There is at least one.

That job is not passing conformance, from the configuration: (see: ) Conformance requires not setting --, you must run all tests without skipping. We don't, at the very least replicaset is being tested multi-node, just not in this job that is not running the full conformance suite. When conformance results are submitted they are now validated to ensure that all tests were run.


Question: var _ = Describe("Probing container", func() { framework := Framework{BaseName: "container-probe"} var podClient client.PodInterface <del> probe := nginxProbeBuilder{} </del> <ins> probe := webserverProbeBuilder{} </ins> BeforeEach(func() { framework.beforeEach()

If the container_probe test runs on a node without the nginx image, it sometimes takes 90s to get the pod in running. Most of this time goes in pulling the image, so I'm assuming it's registry related hiccup. Using might make it more reliable. Example failure: Note the following event in the logs:
a) we should not be pulling from docker in our tests - too hit-or-miss b) we should replace nginx (132 MB) with (4.5 MB) whenever possible. , Prashanth B wrote:


Question: #   make verify #   make verify BRANCH=branch_x .PHONY: verify <del> verify: </del> <ins> verify: verify_generated_files </ins> KUBE_VERIFY_GIT_BRANCH=$(BRANCH) hack/make-rules/verify.sh -v hack/make-rules/vet.sh

Right now on head () if you run it will end up having a modified file. This happened because some type has been changed and this file should have been generated before commit, and verification should have failed but they didn't.
This is not only a problem with openapi-gen. deepcopy-gen also has the same problem. I've a field to a type (autoscaling/v1) and verify-all passed for the commit (without adding generated code to it). those files will be generated on verify-codegen and it will not fail and the commit can be merged without generated code.
Yeah. So when the Makefile stuff originally went in those files were not checked in. Then we un-did that part, but I didn't re-add them to verify...


Question: return true }) sortObjects(inFlightClaims) <del> stripObjects(inFlightClaims) </del> return inFlightClaims }

pull-kubernetes-unit dynamicresources unknown multiple failures in tests as seen in n/a /sig scheduling
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
probably cc
/assign
DeepCopy missing here:


Question: func (m *Master) InstallSwaggerAPI() { // Enable swagger UI and discovery API swaggerConfig := swagger.Config{ <del> WebServices: m.handlerContainer.RegisteredWebServices(), </del> <ins> WebServicesUrl: m.readWriteServer, WebServices:    m.handlerContainer.RegisteredWebServices(), </ins> // TODO: Parameterize the path? ApiPath:         "/swaggerapi/", SwaggerPath:     "/swaggerui/",

It's 127.0.0.1 regardless of the public address. Looks like we need to set WebServicesUrl: where we create the endpoint: using PublicAddress: and ReadWritePort:
/cc
In particular: , bgrant0607 wrote:


Question: }, ImagePullPolicy: "IfNotPresent", }, <ins> { Name:  "same-host-port-different-protocol", Image: "image", Ports: []api.ContainerPort{ {ContainerPort: 80, HostPort: 80, Protocol: "TCP"}, {ContainerPort: 80, HostPort: 80, Protocol: "UDP"}, }, ImagePullPolicy: "IfNotPresent", }, </ins> {Name: "abc-1234", Image: "image", ImagePullPolicy: "IfNotPresent", SecurityContext: fakeValidSecurityContext(true)}, } if errs := validateContainers(successCase, volumes); len(errs) != 0 {

Pods which use to bind both TCP and UDP sockets to the same port number are rejected with the error: An example pod follows.
Looks like we don't check protocol
The issue was discussed before in , which was closed due to it being low priority.
Hmm, that's 2 pods and the scheduler, this is 2 containers in the same pod and validation at create time. Maybe we should just be more clear about where we stand with hostPort.
Posting here what I posted in the other bug: This can't be worked around when running NFS from a pod. Linux does not allow the NFS port (2049) to be changed, and the Portmap port (111) also cannot be changed. Additionally, you can't separate the UDP from the TCP ports.
Why does NFS need host ports? , Russell Harmon wrote:
Fix is in , Tim Hockin wrote:


Question: "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" "k8s.io/apimachinery/pkg/runtime" <ins> "k8s.io/apimachinery/pkg/runtime/schema" </ins> "k8s.io/kube-openapi/pkg/util/proto" "sigs.k8s.io/structured-merge-diff/typed" "sigs.k8s.io/structured-merge-diff/value"

Found by failing TestApplyConvertsManagedFieldsVersion test in () It seems that conversion depends on schema info that is collected from served API versions only. However, managedFields stores data in etcd from REST versions served in previous releases which may no longer be served. That means that we have to be able to deal with managedFields persisted via API calls to versions that are no longer served, but which we still have and understand conversions for. /cc /sig api-machinery /wg apply /priority important-soon
This may be fixed in a more recent version of structured-merge-diff. I'll look into it
/assign
At one point we were using the types registered to the scheme to generate the openAPI definitions used by apply, but we switched to using the installed endpoints instead, because it was less invasive (We needed to expose which types are convertible to which other types, and also a way to skip generating openAPI for internal types). We might want to revisit the first approach, given this use case.


Question: if size < 0 { return nil, fmt.Errorf("watch cache size cannot be negative: %s", c) } watchCacheSizes[schema.ParseGroupResource(tokens[0])] = size } return watchCacheSizes, nil

Currently we signal undecorated storage by simply affecting the undecoratedStorage to the decorator. This is a blocker for kubernetes/enhancements as we need to enable watch cache for events (which currently hardcoded and not configurable) cc /assign /sig api-machinery


Question: klog.V(5).Infof("deletePod %s/%s, no node", pod.Namespace, pod.Name) return } klog.V(4).Infof("deletePod %s/%s for node %s", pod.Namespace, pod.Name, pod.Spec.NodeName) <ins> startTime := time.Now() </ins> g.graph.DeletePod(pod.Name, pod.Namespace) <ins> klog.V(5).Infof("deletePod %s/%s for node %s completed in %v", pod.Namespace, pod.Name, pod.Spec.NodeName, time.Since(startTime)) </ins> } func (g *graphPopulator) addPV(obj interface{}) {

Log statements and metrics indicating when the node authorizer graph has be updated for a pod. There is already a log statement for when the graph starts an update, but this is before the graph lock is obtained and doesn't tell when the graph update is completed. We are seeing node authorizer failures which prevent pods from starting for extended periods of time. It is believed that this is caused by a race between: the node authorizer graph being updated the node requesting the secret It is important to have logging that indicates when the graph finished updating in order to reflect lock contention on the graph.
/sig auth
/triage accepted Seems like a reasonable feature request.
Hi, folks I would like to work on this. /assign
Hi Fixed this issue here PTAL!


Question: timer.Reset(i.timeout) } case <-expired: <del> for _, stream := range i.conn.streams { stream.Reset() } i.conn.Close() break Loop </del> <ins> i.conn.streamCond.L.Lock() streams := i.conn.streams i.conn.streams = make(map[spdy.StreamId]*Stream) i.conn.streamCond.Broadcast() i.conn.streamCond.L.Unlock() go func() { for _, stream := range streams { stream.resetStream() } i.conn.Close() }() </ins> case <-i.conn.closeChan: if timer != nil { timer.Stop()

Hit this while :
cc/ Andy can you take a look at this failure?
looking now
Working with on a fix
Proposed fix:


Question: } return a.Amount.Cmp(b.Amount) == 0 }, <ins> func(a, b util.Time) bool { return a.Unix() == b.Unix() }, </ins> ) var standardResources = util.NewStringSet(

This tree has a commit that demonstrates two obviously different objects that should not pass the semantic DeepEqual but do.
What shall we do with this issue for triage? Labels? Priority?
P2 for when he returns , Satnam Singh wrote:
OK I am now sufficiently worried about this. Fixes incoming.


Question: return false, fmt.Errorf("multiple instances found for instance: %s", instanceID) } <ins> state := instances[0].State.Name if *state == ec2.InstanceStateNameTerminated { glog.Warningf("the instance %s is terminated", instanceID) return false, nil } </ins> return true, nil }

If a node is attached to a cloud provider and the machine is in any state other than running/available/ready/whatever cloud provider uses to indicate ready, it will be removed from the kubernetes cluster. Looking at the various cloud providers, some of them filter nodes based on the state whereas others do not. The node controller has logic for handling a node that is not updating status and evicting pods/tainting the node as needed. Is there a reason why the cloud providers only return instances/vms that are "ready" instead of returning any instance/vm that matches the requested name? It seems like it might be more consistent to have the cloud providers only filter on name and allow the caller to determine what to do if the found instance isn't in a desired state.
If that's really the case it's unfortunate - we probably can fix logic to list everything (I guess there's a flag that you can pass to CP to give you everything, not only working machines). I have no idea who deals with that ? OTOH I think the only problem with this would be if the machine would flap, as this would be annoying and cluster autoscaler might get crazy ( If it's really unready/unavailable/whatever then it's fair to assume that nothing's working on it (if StatefulSet can live with that assumption
do you think the NodeController should/should not delete a Node from storage in this situation: Node is configured to use a cloud provider, VM is powered off but not deleted? I would argue for "should not delete".
A trivial answer is that PrivateDNSNames are not unique on AWS. A running instance and a stopped instance could have the same IP, hence same PrivateDNSName. Rob / Marek: Can you clarify with a scenario where something unexpected happens: what that is and what you expect to happen?
can we work around that somehow for ec2?
I don't know what we are working around! The concern is AWS instances that are Stopped but not Terminated? Separately, we've been working since before 1.0 to get us off PrivateDNSName on AWS though...
the issue is that, depending on cloud provider, the NodeController may delete a Node from etcd if the VM is not Running.

If you power off a VM, wait longer than the NodeController's eviction threshold (5 minutes by default), and then power the VM back on later, the NodeController will have deleted the Node from etcd in the interim. Some cloud providers in the kube code base expect an instance to be Running, and if it's not, they don't include that instance in the list of instances, or a "get instance" call returns "not found" (I'm not sure which, has the full details), and so the NodeController doesn't see the powered off VM and deletes the Node.
That is the current behaviour, but you've skipped the bit where something goes wrong :-) Even if we fix this problem though do we close this issue? I would vote to close this issue now on the basis that there's no objective measure for when we can mark this issue resolved, and that we start again with a problem statement like the one you're describing, which feels actionable.
the intent of this issue was what I described. It's possible that it wasn't clear in the initial description.
At least on AWS: The motivation for the filtering is that Terminated instances still have the old PrivateDNSName listed (IIRC), and so the names aren't unique unless you exclude Terminated. But we don't have to exclude Stopped instances (power-off), and the nodes will just get marked NotReady. I don't think that will cause problems, but it may be that the correct place to fix is actually somewhere else - e.g. on kubelet rejoin. What goes wrong when the node tries to rejoin?
At least 1 thing: if you have manually labels to the node post-registration, those labels are gone when it rejoins.
That sounds like it would be correct for some classes of labels.
- small clarification: 5 min threshold has nothing to do here. It only says that if Node is unresponsive for 5 or more minutes we'll start evicting Pods from that Node (modulo rate limiting). Deleting non-existing Nodes is a separate thing - when Node skips few heartbeats (currently 4) NC checks if cloud provider still returns the instance corresponding to that Node, and if not then delete the Node instance. IIUC what wrote the current behavior is valid for AWS, as Node won't come back (it's DNS name might be used by something else).

This isn't the case for only aws. Looking at the code it appears at least openstack, aws, vsphere, ovirt, rackspace cloud providers also filter on available node. I wasn't able to verify whether azure or others were doing the filtering. If they weren't, then at best we have inconsistency with different cloud providers. IMO we shouldn't be deleting a node from kubernetes if it is just stopped or some state other than available. If the instance exists in the cloud provider then imo it should exist in kubernetes, but possibly marked unscheduable if needed.
ouch, that's even worse! Isn't that only 40-50 seconds? and I chatted yesterday and I think there are a couple of possible ways to look at this: 'm used to doing things the "old way" where I treat my servers as pets, and I want to be the one who decides when to retire/delete a server. 'm willing to treat servers as cattle, and if I ever need to take a server down so I can upgrade the kernel/packages/etc, or if a server goes offline for whatever reason, I'll just throw it away and replace it with a newer one. I'd be willing to bet that most IT organizations are still operating under 1). Justin recommended that we need to do a better job at education, and while I agree with that, there is still going to be a transitional period where sysadmins want to be making these kinds of decisions, and they're not yet willing to let the cluster automatically delete nodes. In summary, I think at the very least we need to: this behavior, if it's not already documented an option to the node controller to disable this behavior (and possibly disable it by default) more on evangelizing how we envision server management to happen in a Kubernetes world
Yes, I believe it's 40s. I'm curious why we don't want to just change the implementation of cloud provider, so that we'll use 'full list' (i.e. one including not running Nodes, so only delete ones that are really not present) in NC?
That also works for me, assuming it doesn't cause any issues.
It's strictly better than disabling it altogether, right?

The drawback would be that if you create a VM which will get the same name (I guess it's a DNS name), than the suspended/whatever one, then it'll be treated by k8s as the old one, which may cause a issues when running StatefulSets (especially if old machine didn't release PV or sth, as IIUC k8s will believe that it's attached to the new machine But if you're willing to do that, then I don't see anything that will prevent you from doing so, assuming it will be flag gated.
I think stateful sets are unaffected by this unless both new and old machines are booted at the same time AND either kubelet fails to terminate the pods. I can't think of how that would happen. Not being able to unmount a volume could be caused by other admin actions and is something we can warn about.
For a change of this significance, we'll want an e2e showing the problem, and then a PR that lets the e2e pass.
I'm not sure an e2e is the best way to show this. An e2e would incur costs to run on a cloud provider so likely wouldn't be part of any standard e2e run. I think a unit test could be done though.
The PR I pushed only changes AWS. w/o this change the node is removed from kubernetes shortly after it is no longer running. With this PR the node transitions to unschedulable and details show it hasn't reported its health check.
PTAL and let me know if you see issues with this approach
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/lifecycle frozen
/remove-lifecycle stale


Question: } return false } <del> t.checkDNSRecord(federationDNS, predicate, wait.ForeverTestTimeout) </del> <ins> t.checkDNSRecordFrom(federationDNS, predicate, "cluster-dns", wait.ForeverTestTimeout) </ins> } } } func (t *dnsFederationsConfigMapTest) setConfigMap(cm *v1.ConfigMap, fedMap map[string]string, isValid bool) { <ins> t.fedMap = nil </ins> if isValid { t.fedMap = fedMap }

/kind failing-test /priority critical-urgent /sig network /milestone v1.12 /cc
ping
Also failures on
TLS errors connection refused 503 service unavailable especially around metrics and healthz in ways that make me suspect things didn't start correctly. Or perhaps API mismatch? /cc 've looked at some similar failures in the past. Any chance you might give a look over this and ?
in both and here looks like api mismatch?
Test output shows no DNS server can be reached, but logs say coredns pods are running (https://k8s-): cc
The CoreDNS configmap is incorrectly configured and hence the test fails. I'm working on the fix.
/assign
, , Does it make sense to keep that e2e test ?, knowing that this test is specific for "Federation" and that federation has now merged into "multicluster" and removed all e2e tests about "Federation + Serial" (formerly "ci-federation-e2e-gce-serial") : can you say if that use case is still supported in kubernetes v1.12 ?
I can say that Federation v1 has not been exercised since v1.10. I think a good question for the Federation v2 folks is not only whether they still need the tests, but also whether they need the configmap support specific to what Federation v1 used -- I'll defer to
If ya'll decide the use case is no longer valid, please open an issue / remove the test once 1.13 release opens.
tests are failing
/reopen /cc
Reopening this issue. <detailsIn response to : /reopen /cc Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
More failures:
Looks like this is failing since Kube-dns is now enabled.
Since we'd previously discussed this not being a relevant test case, and it seems less appropriate even if it's this coupled to the dns implementation and we've got the kube-dns to coredns transition happening in the next release most likely....Can we just disable/remove the test?
Opened to disable this test.
Thank you!


Question: operator: Exists - key: {{ .MasterTaintKey }} effect: NoSchedule <del> # TODO: Remove this affinity field as soon as we are using manifest lists affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: beta.kubernetes.io/arch operator: In values: - {{ .Arch }} </del> <ins> nodeSelector: beta.kubernetes.io/arch: {{ .Arch }} </ins> ` // KubeDNSService is the kube-dns Service manifest

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: After joining a new ppc64le node to the k8s cluster, the kube-proxy DaemonSet fails on the new node. What you expected to happen: It was expected that the kube-proxy DS would "just work" as it does when adding an amd64 node to the cluster. How to reproduce it (as minimally and precisely as possible): Set up an amd64 based k8s cluster, then join a ppc64le node (or any non-amd64 arch) to the cluster. See the pod fail via 'kubectl -n kube-system get pods' because it's pulling the amd64 kube-proxy docker image instead of the ppc64le image. Anything else we need to know?: The yaml for the DaemonSet specifies the image: <versionIt would be preferable to have a manifest so that the image in the DS could be specified as <versionand could "just work" on any platform. See: Environment: Kubernetes version (use ): v1.10.3 Cloud provider or hardware configuration: self - cluster installed via kubeadm OS (e.g. from /etc/os-release): NAME="Ubuntu" VERSION="16.04.3 LTS (Xenial Xerus)" ID=ubuntu IDLIKE=debian PRETTYNAME="Ubuntu 16.04.3 LTS" VERSIONID="16.04" HOMEURL="" SUPPORTURL="" BUGREPORTURL="" VERSIONCODENAME=xenial UBUNTU_CODENAME=xenial Kernel (e.g. ): N/A Install tools: N/A Others: N/A
if you are running multi-architecture nodes, your daemonsets should specify a node selector to limit their arch-specific image to run on matching nodes
/close
I am well aware that a node selector can be used. That is what we are doing now. But it is a hack. The default should "just work". Adding a new non-amd64 node should not result in kube-proxy immediately failing on it because the default daemon set is amd64 arch specific. If kube-proxy is meant to "just work" clusterwide, a node selector should not be needed.

Please use the manifest-tool so that the default "just works". Please see the weave-net plugin. They are using the manifest-tool and it "just works" on amd64 and ppc64le. Their daemon set does not require/use a nodeSelector.
The current kube-proxy daemon set implementation is not properly architected, IMHO. Please re-open this
Reiterating the mentions to trigger a notification: <detailsIn response to : The current kube-proxy daemon set implementation is not properly architected, IMHO. Please re-open this </details
ah, this was a comment about the kube-proxy image, not the daemonset functionality /reopen
the kube-proxy addon daemonset is probably the multi-arch kube-proxy image to support that daemonset might be
Looks like it's coming from here: /assign /sig cluster-lifecycle Lucas, Do we want to avoid the references?
A dirty workaround for this is to pull and rename to . The kube-proxy is running as daemonset, created during . That means the kube-proxy image arch keeps consistent with your master node. Currently we don't have a multi-arch kube-proxy image yet. We have to explicitly specify the right name, like .
Hi .. Yes, we just created a separate daemonset with a ppc64le node slector that refers to the ppc64le image directly. But it's a bit of a hack. That is why I opened this ticket looking for a multi-arch image. Thanks!
I don't know that mixed-ISA clusters are supported. cc
The requirement is: only use "" in DaemonSet yaml file, and the k8s will download the correct image according to the ARCH info. It's more about image registry's feature; and it seems some registries had supported that: .
Yeah, has already supported multi-arch. But actually all the images now are architecture specific, like and . We've not built any multi-arch images right now.
for the test images we have the ball rolling here - - so if you can do the same (or similar) for other images, that would be great!
I have pending PR in releases project for creating manifest for those -
The manifest images created in kubernetes/release should work for future released versions, but will they be created during CI testing?

if you see one of the jobs that pushes images say you can grep for that's basically from the same files that got changed in so i fully expect the images pushed to CI will reflect the changes made in 516.
please confirm that i am reading this right ^ :)
I was more concerned with ci jobs that do not use cross builds, such as https://k8s-
please open an issue about pushing manifests in CI as well


Question: if [[ -n "${VOLUME_PLUGIN_DIR:-}" ]]; then params+=" --flex-volume-plugin-dir=${VOLUME_PLUGIN_DIR}" fi <ins> if [[ -n "${CLUSTER_SIGNING_DURATION:-}" ]]; then params+=" --experimental-cluster-signing-duration=$CLUSTER_SIGNING_DURATION" fi </ins> local -r kube_rc_docker_tag=$(cat /home/kubernetes/kube-docker-files/kube-controller-manager.docker_tag) local container_env="" if [[ -n "${ENABLE_CACHE_MUTATION_DETECTOR:-}" ]]; then

/kind e2e-test-failure GCE serial and GCI GCE serial started failing last evening https://k8s- https://k8s-
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/testing: Issue will be escalated to these SIGs if needed. priority/critical-urgent: Never automatically move out of a release milestone; continually escalate to contributor and SIG through all available channels. kind/bug: Fixes a bug discovered during the current release. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details
https://k8s- is still green, I would suspect something ~last Friday afternoon gets merged
Here is what I found, after a while, the node lost their authorization to connect to API server. I believe this has the same root cause as
I am going to revert offending pr () to see if it helps. EDIT: Looks like is in the process of reverting it:

ref Test is failing on GCE slow and GCI GCE slow: https://k8s- https://k8s- Example failure https://k8s-
The test nodes appear to lose their authorization for accessing resources about 26m into the test. One of many "Unauthorized" errors starting at
, does this ring any bells ?
The certificates are all expiring:
I believe this is the regression introduced cc/ Can you guys take a look?
[MILESTONENOTIFIER] Milestone Labels Incomplete Action required: This issue requires label changes. kind: Must specify exactly one of [, , ]. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details
thanks. I'll open a PR to revert that.
Revert here


Question: <del> apiVersion: apps/v1beta2 </del> <ins> apiVersion: apps/v1 </ins> kind: StatefulSet metadata: name: web

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: The following issues are observed while running this e2e test case for vSphere CSI Migration feature validation. Issue-1 - test failed with Unexpected error: This is happening because YAML is using version. Issue-2 - After upgrading from . Test executed successfully but reported failure in the cleanup stage. when is called in DeleteAllStatefulSets, it does not find statefulset. moving call to in the defer block helps avoid printing this message. There may be a race between namespace deletion and after each execution. I am not familiar with recent e2e framework changes. What you expected to happen: Test should pass. How to reproduce it (as minimally and precisely as possible): execute using Kubernetes master branch builds. Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: vSphere Cloud Provider
/sig testing /sig storage
/remove-sig testing The framework and CI are SIG testing. SIG storage owns their tests.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten


Question: } func runCompletionZsh(out io.Writer, boilerPlate string, kubectl *cobra.Command) error { <del> zshHead := "#compdef kubectlncompdef _kubectl kubectln" </del> <ins> zshHead := fmt.Sprintf("#compdef %[1]sncompdef _%[1]s %[1]sn", kubectl.Name()) </ins> out.Write([]byte(zshHead)) if len(boilerPlate) == 0 {

The command is generating a couple of hardcoded which seemingly causes any commands wrapping this package downstream to break. In this example, the OpenShift client: This should generate the compdef based on the rather than forcing kubectl, for example with : It could be even cleaner with the compdef function removed and the extraneous tag repeated below the boilerplate: compdef command command boilerplate stuff body Such as: generates the same first two lines as There is also a related downstream issue: Ideally, this might get resolved here at the source as it might help anyone else wrapping these commands downstream. I also have a possible solution which involves changing the hardcoded values from to use the which fixes the downstream without changing anything upstream. <details</details<detailsDownstream client-side issue unrelated to a specific cloud provider. </details<details</details<details- Latest </br- OpenShift client = 4.9 </detailsNo response <details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cli


Question: volumetest "k8s.io/kubernetes/pkg/volume/testing" ) <del> const expectedBlockSize = 4096 </del> <ins> func getExpectedBlockSize(path string) int64 { statfs := &syscall.Statfs_t{} err := syscall.Statfs(path, statfs) if err != nil { return 0 } return int64(statfs.Bsize) } </ins> // TestMetricsDuGetCapacity tests that MetricsDu can read disk usage // for path

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): TestMetricsDuGetCapacity Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): master branch (tip) Environment: Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): SLES 12 Kernel (e.g. ): Linux pts00449-vm23 3.12.28-4-default SMP Thu Sep 25 17:02:34 UTC 2014 () ppc64le ppc64le ppc64le GNU/Linux Install tools: Others: What happened: make test WHAT="./pkg/volume" Running tests for APIVersion: v1,apps/v1beta1,,,,,autoscaling/v1,autoscaling/v2alpha1,batch/v1,batch/v2alpha1,,extensions/v1beta1,,policy/v1beta1,,,,,,federation/v1beta1 +++ [0404 11:22:46] Running tests without code coverage --- FAIL: TestMetricsDuGetCapacity (0.01s) Unexpected Used for directory with file. Expected 4096, got .

FAIL FAIL 0.137s Makefile:164: recipe for target 'test' failed make: [test] Error 1 Test case failed What you expected to happen: Test case should pass How to reproduce it (as minimally and precisely as possible): make test WHAT="./pkg/volume" Anything else we need to know*: Found this similar issue: However the values returned are different, see this string "Expected 4096, got "
Hi, I am able to run that tests: I am running RHEL ppc64le, latest k8s with go1.8, probably something specific to do with SLES?
I still get the error on SLES, updated the code to latest and using go1.8, maybe it is SLES (or my machine?) specific talks about /tmp space, not sure if its related..
Hi , Some investigation into this i found that the issue is due to the hardcoding on line 32 in (similar from comments here: ) I changed it to remove the hardcoding (patch attached) and the test case passes on SLES as well, i am not sure if the patch is appropriate so requesting you to review it as appropriate
Hi Vaibhav, Have you raised any PR on it? If raised, reviewing would be easier. There is one thing I noticed: In the above code, you are comparing actual used value with total of empty directory usage and expected block size. When you create a new file like "f1 created here with only "Hello World", it consumes 4K in Ubuntu/RHEL. This size is same as for an empty directory. What is this size in SLES? So, either you hardcode this value(conditional for SLES) or call getExpectedBlockSize() on f1 file path instead of tmpDir.
Hi I have created the PR for the patch as suggested.. I have also changed the original patch slightly to use getExpectedBlockSize() on f1 file path instead of tmpDir as you suggested as that I think would be the right way to do it.. Thanks! Vaibhav
/sig storage


Question: cmd.Flags().BoolVar(&options.DeleteAll, "all", options.DeleteAll, "Delete all resources, including uninitialized ones, in the namespace of the specified resource types.") cmd.Flags().BoolVar(&options.IgnoreNotFound, "ignore-not-found", options.IgnoreNotFound, "Treat "resource not found" as a successful delete. Defaults to "true" when --all is specified.") cmd.Flags().BoolVar(&options.Cascade, "cascade", options.Cascade, "If true, cascade the deletion of the resources managed by this resource (e.g. Pods created by a ReplicationController).  Default true.") <del> cmd.Flags().IntVar(&options.GracePeriod, "grace-period", options.GracePeriod, "Period of time in seconds given to the resource to terminate gracefully. Ignored if negative.") </del> <ins> cmd.Flags().IntVar(&options.GracePeriod, "grace-period", options.GracePeriod, "Period of time in seconds given to the resource to terminate gracefully. Ignored if negative. Set to 1 for immediate shutdown. Can only be set to 0 when --force is true (force deletion).") </ins> cmd.Flags().BoolVar(&options.DeleteNow, "now", options.DeleteNow, "If true, resources are signaled for immediate shutdown (same as --grace-period=1).") <del> cmd.Flags().BoolVar(&options.ForceDeletion, "force", options.ForceDeletion, "Immediate deletion of some resources may result in inconsistency or data loss and requires confirmation.") </del> <ins> cmd.Flags().BoolVar(&options.ForceDeletion, "force", options.ForceDeletion, "Only used when grace-period=0. If true, immediately remove resources from API and bypass graceful deletion. Note that immediate deletion of some resources may result in inconsistency or data loss and requires confirmation.") </ins> cmd.Flags().DurationVar(&options.Timeout, "timeout", options.Timeout, "The length of time to wait before giving up on a delete, zero means determine a timeout from the size of the object") cmdutil.AddOutputVarFlagsForMutation(cmd, &options.Output) cmdutil.AddInclude3rdPartyVarFlags(cmd, &options.Include3rdParty)

IMO it's surprising that does not force delete a resource. only takes effect when . The description of reads this: We should make it more clear what does and even print warnings when is specified without .
Changing the meaning of force to actually break cluster consistency be default could lead to data loss, which is why we haven’t changed it so far. agree we should print more info and clarfiy


Question: container: .container-$(ARCH) .container-$(ARCH): bin/$(BIN)-$(ARCH) <del> docker build --pull -t $(IMAGE_WITH_ARCH):$(TAG) --build-arg ARCH=$(ARCH) . </del> <ins> DOCKER_CLI_EXPERIMENTAL=enabled docker buildx build --pull --platform linux/$(ARCH) -t $(IMAGE_WITH_ARCH):$(TAG) --build-arg ARCH=$(ARCH) . </ins> touch $@ push: .push-$(ARCH)

What happened: Non-amd64 images of claim to be for amd64. Obviously, those images are executing just fine but the image configs claim to be for the amd64 architecture. We just started noticing the issue as the library has some more rigorous checks now. What you expected to happen: Each image config should contain the correct architecture. How to reproduce it (as minimally and precisely as possible): A more detailed analysis can be found here: Anything else we need to know?: I suggest to update the 3.1 images. Although most runtimes/container engines are happily eating the image, the configs still claim to be for a different architecture. We haven't noticed the issue since the os/arch validity checks are fairly new in the containers/image library. Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
cc
/sig release
cc
I think this is happening because we cross-build these docker images on x86 architecture and inspect shows amd64 for that particular image it is trying to pull:
Will have to figure out a way to tag the appropriate architecture while building docker image, I see an experimental flag flag been introduced which can be used to fix this issue but unfortunately it is not working properly when we build it image
Hey :wave:, I'm currently working on setting the arch/OS for a container image during image build (see ). We could think about building our images with buildah/podman if the feature gets accepted and fits our needs. WDYT?
Most of the build scripts are now using docker client but I'm open for using other tools if they are serving the purpose! wdys
as long as someone signs up to do the work and handle follow ups (AND cover all the scenarios that we do currently - like windows!) cc
ah! windows will be challenging I feel at the moment with buildah, will find a workaround for that!
Is buildah available on Mac or runnable in a docker container?
Ye, will look into it. But I think someone was already doing work on this, will have to check how it looks. And we can probably add the Image Promoter to this, including the Windows bits in a similar way we're doing with the test images.
It runs inside docker containers.

Can we create a list of requirements that I could prepare a demo container? What is the whole set of platforms which we have to support?
So, there used to be a PR that was adding support for Windows for this image: But a few things have changed since then, so things should be a bit different: we'll need multiple Windows OS version support: for 1809, 1903, 1909. we'll have to include wincat: - it is needed for a few kube-proxy usecases, if I remember correctly. As for building, we could probably do it using the Image Promoter? I currently have a few PRs that are adding the Windows bits to the k/k e2e test images, including the part in which the Image Promoter is using a few Windows Server build nodes to build the Windows container images (), so I could add the same bits to the pause image, if people want it. If so, I'll have to know first: is the Image Promoter in place for this image, or do I have to do it from scratch? :smile:
The problem is not just with windows images, most of the time the problem will be with cross-building images for non-amd64 archs like arm64, ppc64le, s390x platform using qemu.
Sorry, this is a bit unclear for me. the problem is that we're using qemu to cross-build the images for non-amd64 archs and we shouldn't?
I just upgraded to latest crio and it refuses to download/use the pause image for me since then. I assume this is from a change in container/libpod implementing more strict checks. In my point of view this is impossible to fix for all images, but at least the base cluster images (pause, static pods for kubernetes like apiserver, etcd) should be fixed. For reference here is the according crio issue I opened:
when we cross-build the images on the amd64, the image will contain the target architecture bits but the image will be signed with host arch(amd64) and this is what is creating issue while pulling the image by crio where they have a strict check of the image architecture and expects to match the host architecture(arm64 or ppc64le).

The solution here is to build all these images natively on the target architecture or feed the proper platform while cross-building(the above buildah PR is to achieve this)
works as expected with . I will prepare a trivial PR to fix this, but then we will need to decide how to roll out the new image...
/assign
sounds good to me,
Should be fixed in
Thanks a lot to all involved!
(whoops :|) building the images now, then we'll start updating references to it.
-the image is now pushed to staging after sorting out the manifest .. will file a promotion to prod now
actually updates to the new image, which should now be live at


Question: } return imageFs, imageFs, nil } <del> containerFsInfo, err := p.cadvisor.ContainerFsInfo() if err != nil { return nil, nil, fmt.Errorf("failed to get container fs info: %v", err) } </del> imageStats, err := p.imageService.ImageFsInfo(ctx) if err != nil || imageStats == nil { return nil, nil, fmt.Errorf("failed to get image stats: %v", err)

Test is failing on crio conformance (cgroup v1 / cgroup v2) Test doesn't fail. You can reproduce it on a PR with /test pull-kubernetes-node-crio-e2e. I reverted go bump and that was not the case. master CI <details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/priority critical
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /priority critical </details
failed to get imageFS data
Ah. Yea this confirms it.
looks like the KEP-4191 bump to beta is causing the failure.
/priority critical-urgent /sig node
/assign kwilczynski
CRI-O version and runtime configuration from the test run: Version string (from the log)Configuration
CRI-O is not configured to enable the split image file system per: (an excerpt from the runtime configuration)
The code that is returning the error:
Other logs I see:
Are we planning to get this fixed for v1.31 or revert the promotion of KEP 4191 to beta with Since code freeze is in effect now, we'd need approval from the release team either way. If we choose to revert the graduation to beta, the KEP won't be part of v1.31.
I think we have a bug where we have the feature enabled but the container runtime is not configured.
I am looking at fixing this as soon as possible...
That PR was mostly to find out the issue. I'd like to fix it and we will file a release exception as we are past the freeze.
Since you're planning to file an exception request, I'm removing the KEP from the milestone so that the release team can better track KEPs. If the issue is not fixed and the exception request is not raised, we would still need to revert the KEP back into alpha ().

We should either fix the issue and merge any additional code changes with an exception request or revert the feature gate to alpha as done in the above PR.
/cc
this is actually going to be quite a common use case, as users would most likely not have their runtimes configured initially or even at all (if someone does not care about things like a split or separate filesystem). I wonder how we can reliably detect whether a runtime has support enabled or not. At the moment, the error originates from cAdvisor, where we ask for stats under a specific label, which does not exist. Since the feature is enabled in Kubernetes, we assume that the runtime also has it enabled and send the label that pertains to ContainreFs. There is a features discovery bit to the CRI API, which we already use to divulge details about support for Recursive Read-only Mounts, Supplemental Groups, etc. I wonder if we should also convey support about split filesystem support that way?
/milestone v1.31 Not only stat summary, we can see other similar failures.
So this only fails in the case of crio with feature on but system is not configured. I used the ImgaeFsInFo to detect if the system is split so I am curious if something went wrong there. Containerd is not failing at all with this feature on so we are detecting containers lack of support without issue.
this is easily reproducible if you have CRI-O running without the option set. So, is this what you mean by "system not configured"?
Yes.
I see , is this going to be reverted or are you working on a fix?
/assign kannon92
We have open to solve this. Waiting on exception approval and this to the milestone. Confirmed that these tests work now.


Question: # function kube::util::ensure-bash-version { # shellcheck disable=SC2004 <del> if ((${BASH_VERSINFO[0]}<5)); then echo "ERROR: This script requires a minimum bash version of 5.0" </del> <ins> if ((${BASH_VERSINFO[0]}<4)) || ( ((${BASH_VERSINFO[0]}==4)) && ((${BASH_VERSINFO[1]}<2)) ); then echo "ERROR: This script requires a minimum bash version of 4.2, but got version of ${BASH_VERSINFO[0]}.${BASH_VERSINFO[1]}" </ins> exit 1 fi }

What happened: On our Ubuntu 18.04 build machines(with bash 4.2), build failed with error: What you expected to happen: build succeeds without error. How to reproduce it (as minimally and precisely as possible): On a Ubuntu 18.04 machine with the latest bash installed from official repo, do make by Anything else we need to know?: This is related to PR
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig release
version 5 was forced here:
The support of test was in bash version 4.2. It would be better to limit the mimimum verison to 4.2+, since some people, including us, are using Ubuntu 18.04 with bash 4.x to do the build. Thanks.
Or better yet use another method like; BTW I removed this flawed test and build happily on Ubuntu 18 without any problem.
i do not disagree with that. looks like the right people are already assigned to your PR.
This is breaking downstream CIs, I´ve found this today in one of thems is not possible to relax this to 4.2? /sig-testing
Better to not use the "test -v" options at all since it is completely unnecessary, see It is not very user-friendly to enforce a global upgrade when there is absolutely no need. Ubuntu 18 has 4.2 but others may not.
:)
Heh , that was fast :)
this is actually not an equivilant check, but it is possible to implement one without using -v I left a comment on the original PR with how to do this.
see: reports if the variable was set_, which includes In this case I do think the intent was to check if the variable was set or not. It's still possible to do without but it's a big more ... arcane.

Kubernetes has long required a reasonably up to date bash FWIW, on macOS you cannot use the system bash, and for a while you couldn't use debian stable's bash to build kubernetes as it had an array bug (4.0 IIRC). I don't think we have a clear support policy here, and I think this is a good reason to reconsider writing large volumes of bash


Question: "github.com/prometheus/client_golang/prometheus" ) <ins> // resettableCollector is the interface implemented by prometheus.MetricVec // that can be used by Prometheus to collect metrics and reset their values. type resettableCollector interface { prometheus.Collector Reset() } </ins> var ( // TODO(a-robinson): Add unit tests for the handling of these metrics once // the upstream library supports it.

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: endpoint of different k8s components expose Prometheus metrics that are not for them. For example, exposes metrics: or : What you expected to happen: Prometheus metrics only for to appear on kubelet's endpoint. How to reproduce it (as minimally and precisely as possible): Access the endpoint of all Kubernetes components Anything else we need to know?: This problem comes from usage of functions in different packages that automatically register Prometheus metrics. Just importing that package or some package that depends on it can cause those metrics to appear in unexpected places. In the list bellow you can find the different packages that have init functions and affected binaries: [ ] () cloud-controller-manager hyperkube kube-apiserver kube-controller-manager kube-proxy kube-scheduler [ ] () cloud-controller-manager hyperkube kube-apiserver kube-controller-manager kube-proxy kube-scheduler [ ] cloud-controller-manager hyperkube kube-apiserver kube-controller-manager [ ] cloud-controller-manager hyperkube kube-apiserver kube-controller-manager kubelet [ ] cloud-controller-manager hyperkube kube-apiserver kube-controller-manager kube-proxy kube-scheduler [ ] cloud-controller-manager hyperkube kube-apiserver kube-controller-manager kube-proxy kube-scheduler kubectl kubelet [ ] cloud-controller-manager hyperkube kube-apiserver kube-controller-manager kube-proxy kube-scheduler kubelet [ ] hyperkube kube-apiserver kube-controller-manager [ ] hyperkube kube-apiserver [ ] cloud-controller-manager hyperkube kube-apiserver kube-controller-manager kubelet [ ] cloud-controller-manager hyperkube kube-apiserver kube-controller-manager kube-proxy kube-scheduler kubelet Environment: Kubernetes version (use ):Cloud provider or hardware configuration: N/A OS (e.g. from /etc/os-release): N/A Kernel (e.g. ): N/A Install tools: N/A Others: N/A /sig instrumentation
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: loadBalancer LoadBalancer mu           sync.Mutex // protects serviceMap serviceMap   map[string]*serviceInfo <ins> address      string </ins> } <del> // NewProxier returns a new Proxier given a LoadBalancer. func NewProxier(loadBalancer LoadBalancer) *Proxier { </del> <ins> // NewProxier returns a new Proxier given a LoadBalancer and an // address on which to listen func NewProxier(loadBalancer LoadBalancer, address string) *Proxier { </ins> return &Proxier{ loadBalancer: loadBalancer, serviceMap:   make(map[string]*serviceInfo), <ins> address:      address, </ins> } }

kube-proxy listens on all ports, but should be configurable. create a flag to specify which interface to listen on. Thanks for finding this bug.
For context, some providers apparently do external IPs by having a whole interface that gets 1:1 NAT without any firewalling. We do not want service ports on those interfaces. , Brian Ketelsen wrote:


Question: case svc.Interrogate: s <- c.CurrentStatus case svc.Stop, svc.Shutdown: <del> s <- svc.Status{State: svc.Stopped} // TODO: Stop the kubelet gracefully instead of killing the process os.Exit(0) </del> <ins> klog.Infof("Service stopping") // We need to translate this request into a signal that can be handled by the the signal handler // handling shutdowns normally (currently apiserver/pkg/server/signal.go). // If we do not do this, our main threads won't be notified of the upcoming shutdown. // Since Windows services do not use any console, we cannot simply generate a CTRL_BREAK_EVENT // but need a dedicated notification mechanism. graceful := server.RequestShutdown() // Free up the control handler and let us terminate as gracefully as possible. // If that takes too long, the service controller will kill the remaining threads. // As per https://docs.microsoft.com/en-us/windows/desktop/services/service-control-handler-function s <- svc.Status{State: svc.StopPending} // If we cannot exit gracefully, we really only can exit our process, so atleast the // service manager will think that we gracefully exited. At the time of writing this comment this is // needed for applications that do not use signals (e.g. kube-proxy) if !graceful { go func() { // Ensure the SCM was notified (The operation above (send to s) was received and communicated to the // service control manager - so it doesn't look like the service crashes) time.Sleep(1 * time.Second) os.Exit(0) }() } break Loop </ins> } } }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: I configured Kubelet to run as a service on Windows, with a setting to restart on failure (). I noticed that it restarts even when I explicitly stop the service with , even though the documentation states the following (I left it at , the default): What you expected to happen: I expected the Kubelet to only restart when killed unexpectedly, per above docs. Anything else we need to know?: I think the problem might be due to this part of our service control integration, but I'm not really sure: It looks like we send and then , but the official golang example sends and just returns without exiting. (). Is this difference meaningful? Kubernetes version (use ): 1.13.2 Cloud provider or hardware configuration: GCE OS (e.g. from /etc/os-release): Windows /cc /sig node /sig windows
The same should apply to kube-proxy as well.
i believe you implemented this functionality. can you please take a look? cc:


Question: eventsRule(), rbacv1helpers.NewRule("create").Groups(legacyGroup).Resources("endpoints", "secrets", "serviceaccounts").RuleOrDie(), rbacv1helpers.NewRule("delete").Groups(legacyGroup).Resources("secrets").RuleOrDie(), <del> rbacv1helpers.NewRule("get").Groups(legacyGroup).Resources("endpoints", "namespaces", "secrets", "serviceaccounts").RuleOrDie(), </del> <ins> rbacv1helpers.NewRule("get").Groups(legacyGroup).Resources("endpoints", "namespaces", "secrets", "serviceaccounts", "configmaps").RuleOrDie(), </ins> rbacv1helpers.NewRule("update").Groups(legacyGroup).Resources("endpoints", "secrets", "serviceaccounts").RuleOrDie(), // Needed to check API access.  These creates are non-mutating rbacv1helpers.NewRule("create").Groups(authenticationGroup).Resources("tokenreviews").RuleOrDie(),

BUG REPORT: /kind bug /sig auth What happened: Kubernetes distro testing v1.12.0-rc.2 shows in setups with self-hosted control planes (i.e. apiserver, controller-manager, scheduler run as pods) that previously worked on v1.11.3 and prior, the kube-controller-manager now crash loops with: The kube-controller-manager is run with a with a to the default system ClusterRole , which seems to be defined at Editing the ClusterRole manually to add for was sufficient to fix the crash loop. What you expected to happen: I suspect a change between v1.11.3 and v1.12.0 release candidates altered kube-controller-manager to begin looking for configmap , but didn't update the ClusterRoles. How to reproduce it (as minimally and precisely as possible): Release candidate testing is happening in Anything else we need to know?: Environment: Kubernetes version (use kubectl version): v1.12.0-rc.2 Cloud provider or hardware configuration: GCE, DigitalOcean, AWS OS (e.g. from /etc/os-release): Container Linux 1855.4.0 Kernel (e.g. uname -a): 4.14.67-coreos Install tools: Typhoon
/sig cluster-lifecycle /area kubeadm /remove-sig auth edit: sorry, got this mixed up with another 1.12/1.11 issue. is this related to the secured controller manager work you did? /remove-sig cluster-lifecycle /sig auth /sig api-machinery
Those labels are not set on the issue: <detailsIn response to : /sig cluster-lifecycle /area kubeadm /remove-sig auth /sig auth /sig api-machinery Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
can you please try if works to skip that lookup?
we can make this non-fatal, compare
adding that flag to the controller-manager does also work to skip the lookup, without me manually editing the ClusterRole. Although if controller-manager defaults to looking up the configmap, I'd imagine the clusterrole should default to allowing configmap get's, no?

i was just checking if there's a work around we could suggest for 1.12 :) longer term, we can lean towards whatever and decide (adding to bootstrap policy vs making things non-fatal)
This seems like it should get a Known Issue comment in the release notes for 1.12. Can somebody please draft test and drop it in the known issues bucket issue ASAP?


Question: } } <ins> // resourceRequest = max(sum(podSpec.Containers), podSpec.InitContainers) + overHead </ins> func calculateResource(pod *v1.Pod) (res Resource, non0CPU int64, non0Mem int64) { resPtr := &res for _, c := range pod.Spec.Containers { resPtr.Add(c.Resources.Requests) non0CPUReq, non0MemReq := schedutil.GetNonzeroRequests(&c.Resources.Requests) non0CPU += non0CPUReq non0Mem += non0MemReq // No non-zero resources for GPUs or opaque resources. } <ins> for _, ic := range pod.Spec.InitContainers { resPtr.SetMaxResource(ic.Resources.Requests) non0CPUReq, non0MemReq := schedutil.GetNonzeroRequests(&ic.Resources.Requests) if non0CPU < non0CPUReq { non0CPU = non0CPUReq } if non0Mem < non0MemReq { non0Mem = non0MemReq } } </ins> // If Overhead is being utilized, add to the total requests for the pod if pod.Spec.Overhead != nil && utilfeature.DefaultFeatureGate.Enabled(features.PodOverhead) { resPtr.Add(pod.Spec.Overhead) if _, found := pod.Spec.Overhead[v1.ResourceCPU]; found { non0CPU += pod.Spec.Overhead.Cpu().MilliValue() }

The scheduler does not properly calculate pod requests for a resource as - it ignores init containers completely in . This would allow a workload to get "free" scheduling above the limit of containers (mitigated by quota, which calculates this properly) and so the scheduler would make non-optimal fits. Both kubelet and scheduler enforce the , so you can't actually schedule, but you sometimes get steered to the wrong node if you have more init container resource usage than containers. in should use and the similar helpers instead of writing its own method to calculate the request for resources. /sig scheduling
/priority important-soon
Let me follow this issue. /assign
fyi
After I checked the implementation of function, I think we just need to use it to instead of the function in the scheduler.
/priority critical-urgent
Is the pod phase not considered? When the pod phase is running, the requested resource is actually considered as . This change seems to reduce the resource utilization of the node.
Actually, I don't think we need to consider the pod.Phase. The resource request & limit are mainly used in the scheduling process. The scenario you said is not belong to the responsibility of kube-scheduler
Correct. Setting appropriate requests/limits that maintain good node usage is users responsibility. Considering the max(initContainers, sum(containers)) is the documented behavior.
I understand. Thank you for your comments!


Question: func failedPodsBackoffKey(ds *apps.DaemonSet, nodeName string) string { return fmt.Sprintf("%s/%d/%s", ds.UID, ds.Status.ObservedGeneration, nodeName) } <ins> // getPodsWithoutNode returns list of pods assigned to not existing nodes. func getPodsWithoutNode( runningNodesList []*v1.Node, nodeToDaemonPods map[string][]*v1.Pod) []string { var results []string isNodeRunning := make(map[string]bool) for _, node := range runningNodesList { isNodeRunning[node.Name] = true } for n, pods := range nodeToDaemonPods { if !isNodeRunning[n] { for _, pod := range pods { results = append(results, pod.Name) } } } return results } </ins>

What happened: DaemonSet doesn't delete pods that: weren't scheduled yet, node was deleted so pod can't be scheduled. What you expected to happen: Pod should be deleted when node is removed. How to reproduce it (as minimally and precisely as possible): I encountered this problem when doing tests with cluster with 1000 nodes and pods running. During tests I was deleting and adding nodes using GCE API without draining them. I reproduced problem on much smaller cluster but it requires stopping K8s scheduler. Scenario: cluster with 2 nodes. till fluentd-gcp pods are running on nodes. K8s scheduler. fluentd-gcp pods. till daemonset controller creates pending pods. one node. scheduler. of the fluentd-gcp pods will be pending and scheduler won't be able to schedule. Anything else we need to know?: Environment: GKE Kubernetes version (use ): <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
/sig apps
/assign janetkuo
/assign
We have PodGC to clean up orphaned pods every 20s, is that enough?
I'm removing pods that have only node affinity set and don't have pod.Spec.NodeName set. These pods were never touched by scheduler before node was removed and after node was removed can't be scheduled as node is doesn't exist anymore. Such pods are pending forever and PodGC can't remove them as pod.Spec.NodeName is empty.
Yes, that's right :) let me take few time to review the PR, thanks.
Should I backport fix to 1.13 and 1.12?
+1


Question: "verb", verb, "URI", rl.req.RequestURI, "latency", latency, <del> "userAgent", rl.req.UserAgent(), </del> <ins> // We can't get UserAgent from rl.req.UserAgent() here as it accesses headers map, // which can be modified in another goroutine when apiserver request times out. // For example authentication filter modifies request's headers, // This can cause apiserver to crash with unrecoverable fatal error. // More info about concurrent read and write for maps: https://golang.org/doc/go1.6#runtime "userAgent", rl.userAgent, </ins> "audit-ID", auditID, "srcIP", rl.req.RemoteAddr, } <ins> // Lock for accessing addedKeyValuePairs and addedInfo rl.mutex.Lock() defer rl.mutex.Unlock() </ins> keysAndValues = append(keysAndValues, rl.addedKeyValuePairs...) if rl.hijacked {

Apiserver crashed with stacktrace: Not to crash I was able to reproduce it by "slowing down" deleting headers in Authentication filter (): And making requests (around 5-10 requests to trigger this race condition): Crash was caused by race condition - one goroutine was modifying headers of requests (for example authentication filter). The other goroutine was trying to get UserAgent from request, which is also taken from request's headers map. This causes unrecoverable panic. <detailsAll versions, tested on 1.21 </details<detailsN/A </details<details</details<details</details<details</details<details</details
/sig api-machinery /priority important-soon
/triage accepted
Reopening - since we need to cherrypick it to previous releasses.
- friendly ping on opening cherrypicks
I've opened two cherry-picks:
One more for 1.20:
All merged - thanks!