Question: return detailedErr } <ins> simpleMsg, detailedMsg := volumeToMount.GenerateMsg("MountVolume.SetUp succeeded", "") </ins> verbosity := glog.Level(1) if isRemount { verbosity = glog.Level(7) <ins> } else { og.recorder.Eventf(volumeToMount.Pod, v1.EventTypeNormal, kevents.SuccessfulMountVolume, simpleMsg) </ins> } <del> glog.V(verbosity).Infof(volumeToMount.GenerateMsgDetailed("MountVolume.SetUp succeeded", "")) </del> <ins> glog.V(verbosity).Infof(detailedMsg) </ins> // Update actual state of world markVolMountedErr := actualStateOfWorld.MarkVolumeAsMounted(

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): No What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): FailedMount Is this a BUG REPORT or FEATURE REQUEST? (choose one): Bug <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): 8 Environment: Cloud provider or hardware configuration: GKE, GCE OS (e.g. from /etc/os-release): GCI What happened: The FailedMount event can be misleading to users because it's not clear if the mount eventually succeeded or still failed. You would have to understand the internals of the system to correctly interpret the message, which is not user friendly. In this case, the mount did succeed, but the user still sees this failure msg in the events. What you expected to happen: A success event should be generated when a volume successfully attaches and mounts. Care needs to be taken to make sure we don't flood the event log. Maybe only have an event when the mount is successful, and not the attach. How to reproduce it (as minimally and precisely as possible): Create a deployment that uses a GCE PD volume, and move it to different nodes in order to cause the volume to need to be detached and attached to different nodes. cc
/assign
mount failures are a fairly common reason why some of the volume e2e tests fail. It has been useful to me to see the FailedMount event in the target pod.

In the case of ceph-rdb we get a useful error msg: but for other plugins, not so helpful: If the mount eventually succeeds is it possible to annotate an earlier event record (not likely), or even know there was a previously recorded mount failure?
I'm not sure it's possible to modify an earlier event. So I think in this case, it would just always trigger a successful mount event, regardless of if it had previously failed.
And to clarify, I intend to leave the failed events as is. I just want to add a success event as well.
OK, that sounds good to me. Thanks for the clarification.
/sig storage


Question: <ins> /* Copyright 2022 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package windows import ( "context" "time" "github.com/onsi/ginkgo" v1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/util/uuid" podutil "k8s.io/kubernetes/pkg/api/v1/pod" "k8s.io/kubernetes/test/e2e/framework" e2eskipper "k8s.io/kubernetes/test/e2e/framework/skipper" imageutils "k8s.io/kubernetes/test/utils/image" ) var _ = SIGDescribe("[Feature:Windows] [Excluded:WindowsDocker] [MinimumKubeletVersion:1.22] RebootHost containers [Serial] [Disruptive] [Slow]", func() { ginkgo.BeforeEach(func() { e2eskipper.SkipUnlessNodeOSDistroIs("windows") }) f := framework.NewDefaultFramework("reboot-host-test-windows") ginkgo.It("should run as a reboot process on the host/node", func() { ginkgo.By("selecting a Windows node") targetNode, err := findWindowsNode(f) framework.ExpectNoError(err, "Error finding Windows node") framework.Logf("Using node: %v", targetNode.Name) windowsImage := imageutils.GetE2EImage(imageutils.Agnhost) // Create Windows pod on the selected Windows node Using Agnhost podName := "pod-" + string(uuid.NewUUID()) agnPod := &v1.Pod{ TypeMeta: metav1.TypeMeta{ Kind:       "Pod", APIVersion: "v1", }, ObjectMeta: metav1.ObjectMeta{ Name: podName, }, Spec: v1.PodSpec{ Containers: []v1.Container{ { Name:  "windows-container", Image: windowsImage, Ports: []v1.ContainerPort{{ContainerPort: 80}}, }, }, RestartPolicy: v1.RestartPolicyAlways, NodeName:      targetNode.Name, }, } agnPod.Spec.Containers[0].Args = []string{"test-webserver"} ginkgo.By("creating a windows pod and waiting for it to be running") agnPod = f.PodClient().CreateSync(agnPod) // Create Linux pod to ping the windows pod linuxBusyBoxImage := imageutils.GetE2EImage(imageutils.Nginx) podName = "pod-" + string(uuid.NewUUID()) nginxPod := &v1.Pod{ TypeMeta: metav1.TypeMeta{ Kind:       "Pod", APIVersion: "v1", }, ObjectMeta: metav1.ObjectMeta{ Name: podName, }, Spec: v1.PodSpec{ Containers: []v1.Container{ { Name:  "linux-container", Image: linuxBusyBoxImage, Ports: []v1.ContainerPort{{ContainerPort: 80}}, }, }, NodeSelector: map[string]string{ "kubernetes.io/os": "linux", }, Tolerations: []v1.Toleration{ { Operator: v1.TolerationOpExists, Effect:   v1.TaintEffectNoSchedule, }, }, }, } ginkgo.By("Waiting for the Linux pod to run") nginxPod = f.PodClient().CreateSync(nginxPod) ginkgo.By("checking connectivity to 8.8.8.8 53 (google.com) from Linux") assertConsistentConnectivity(f, nginxPod.ObjectMeta.Name, "linux", linuxCheck("8.8.8.8", 53)) ginkgo.By("checking connectivity to www.google.com from Windows") assertConsistentConnectivity(f, agnPod.ObjectMeta.Name, "windows", windowsCheck("www.google.com")) ginkgo.By("checking connectivity from Linux to Windows for the first time") assertConsistentConnectivity(f, nginxPod.ObjectMeta.Name, "linux", linuxCheck(agnPod.Status.PodIP, 80)) initialRestartCount := podutil.GetExistingContainerStatus(agnPod.Status.ContainerStatuses, "windows-container").RestartCount ginkgo.By("scheduling a pod with a container that verifies reboot selected node works as well") trueVar := true podName = "reboot-host-test-pod" user := "NT AUTHORITYSYSTEM" pod := &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name: podName, }, Spec: v1.PodSpec{ SecurityContext: &v1.PodSecurityContext{ WindowsOptions: &v1.WindowsSecurityContextOptions{ HostProcess:   &trueVar, RunAsUserName: &user, }, }, HostNetwork: true, Containers: []v1.Container{ { Image: windowsImage, Name:  "reboot-computer-test", Command: []string{ "powershell.exe", "-Command", "$os = Get-WmiObject -Class win32_operatingsystem;", "[Environment]::SetEnvironmentVariable("TMP_BOOT_DATE", $os.LastBootUpTime, "Machine");", "[Environment]::SetEnvironmentVariable("TMP_INSTALL_DATE", $os.InstallDate, "Machine");", "shutdown.exe -r -t 30", }, }, }, RestartPolicy: v1.RestartPolicyNever, NodeName:      targetNode.Name, }, } f.PodClient().Create(pod) ginkgo.By("Waiting for pod to run") f.PodClient().WaitForFinish(podName, 3*time.Minute) ginkgo.By("Then ensuring pod finished running successfully") p, err := f.ClientSet.CoreV1().Pods(f.Namespace.Name).Get( context.TODO(), podName, metav1.GetOptions{}) framework.ExpectNoError(err, "Error retrieving pod") framework.ExpectEqual(p.Status.Phase, v1.PodSucceeded) ginkgo.By("Waiting for Windows worker rebooting") restartCount := 0 timeout := time.After(time.Minute * 10) FOR: for { select { case <-timeout: break FOR default: if restartCount > 0 { break FOR } ginkgo.By("Then checking existed agn-test-pod is running on the rebooted host") agnPodOut, err := f.ClientSet.CoreV1().Pods(f.Namespace.Name).Get(context.TODO(), agnPod.Name, metav1.GetOptions{}) if err == nil { lastRestartCount := podutil.GetExistingContainerStatus(agnPodOut.Status.ContainerStatuses, "windows-container").RestartCount restartCount = int(lastRestartCount - initialRestartCount) } time.Sleep(time.Second * 30) } } ginkgo.By("Checking whether agn-test-pod is rebooted") framework.ExpectEqual(restartCount, 1) agnPodOut, err := f.ClientSet.CoreV1().Pods(f.Namespace.Name).Get(context.TODO(), agnPod.Name, metav1.GetOptions{}) framework.ExpectEqual(agnPodOut.Status.Phase, v1.PodRunning) framework.ExpectNoError(err, "getting pod info after reboot") assertConsistentConnectivity(f, nginxPod.ObjectMeta.Name, "linux", linuxCheck(agnPodOut.Status.PodIP, 80)) // create another host process pod to check system boot time checkPod := &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name: "check-reboot-pod", }, Spec: v1.PodSpec{ SecurityContext: &v1.PodSecurityContext{ WindowsOptions: &v1.WindowsSecurityContextOptions{ HostProcess:   &trueVar, RunAsUserName: &user, }, }, HostNetwork: true, Containers: []v1.Container{ { Image: windowsImage, Name:  "reboot-computer-check", Command: []string{ "powershell.exe", "-Command", "$os = Get-WmiObject -Class win32_operatingsystem;", "$lastBootTime = [Environment]::GetEnvironmentVariable("TMP_BOOT_DATE", "Machine");", "$lastInstallTime = [Environment]::GetEnvironmentVariable("TMP_INSTALL_DATE", "Machine");", "$timeInterval = $os.ConvertToDateTime($os.LastBootUpTime) -  $os.ConvertToDateTime($lastBootTime);", "$installInterval = $os.ConvertToDateTime($os.InstallDate) -  $os.ConvertToDateTime($lastInstallTime);", "if ( $timeInterval.TotalSeconds -le 0 ) {exit -1};", "if ( $installInterval.TotalSeconds -ne 0 ) {exit -1};", "[Environment]::SetEnvironmentVariable("TMP_BOOT_DATE", $null, "Machine");", "[Environment]::SetEnvironmentVariable("TMP_INSTALL_DATE", $null, "Machine");", }, }, }, RestartPolicy: v1.RestartPolicyNever, NodeName:      targetNode.Name, }, } f.PodClient().Create(checkPod) ginkgo.By("Waiting for pod to run") f.PodClient().WaitForFinish("check-reboot-pod", 3*time.Minute) ginkgo.By("Then ensuring pod finished running successfully") p, err = f.ClientSet.CoreV1().Pods(f.Namespace.Name).Get( context.TODO(), "check-reboot-pod", metav1.GetOptions{}) framework.ExpectNoError(err, "Error retrieving pod") framework.ExpectEqual(p.Status.Phase, v1.PodSucceeded) }) }) </ins>

As of now, when we reboot windows nodes, OVS goes down which breaks Antrea CNI. I suspect reboots may have other effects on other services/CNIs in windows clusters. Thus, we need a Reboot test case (azure has its own tests for node reboots, but theyre not upstreamed yet). So, lets add a , , reboot test for nodes. This is a good way for us to begin exersizing Priveliged containers. However, if theres another way to get a windows node to reboot (from inside a container) we should explore it.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
<-- we can borrow from this as a template to get started
/sig windows
There are two cases. One enables MHC while the other should disable MHC. host process example:
Hi I think we can use cron-job or as mentioned above the MHC way( ) I also had a few more doubts, like here are some of the ways to reboot windows: shutdown command. This is a command line tool available on most any windows computer which will allow you to remotely reboot any computer joined to an active directory domain. All that’s required are the right credentials. This method invokes Windows’ native ability to run system DLL’s which can perform system functions. And yes, thank you so much for this as it gives a good template to understand what we can do. Please do suggest what method we can use if we have to consider any of the above methods to be executed programmatically. In fact, I would love to pick this issue and contribute, however, I have few initial doubts on how much complexity we can face while doing this inside container, would love to know more on this, thanks a lot!
/assign weiwenli97
thanks everyone !

ok i assigned wenli.... ... By the way this is wenlis first upstream issue, so will be a good leanring experience for her , lets try to give her some support along the way :) (wenli feel free to ask some questions in this thread !!!)
All the best Looking forward for all the amazing contributions you make
Enable or disable MHC, the expected behavior for windows nodes reboot should be only reboot not re-created and rejoined. I am trying host process pod to reboot windows node.
These tests should not be "aware" really of MHC... they need to be generic kubernetes tests, so they won'y have any references to Cluster API.
Design on workload nodes Reboot tests : Overview: host process pod to reboot the windows node(s) 5 minutes, checking whether the node(s) is ready or re-created. The workers should only reboot not re-create. Details: 1 host-process pod to run powershell restart-computer on the Windows node for 5 minutes another 1 host-process pod to run powershell command to check whether the hostname is the same as pre-fectched targetNode.Name.
+1 on testing reboots. They may resurface underlying OS specific problems too.
can you post the code you have in a PR ? would like to start discussing this even if its not ready yet :)
I have made a draft PR for it Could you please help review it? Thank you!
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/remove-lifecycle rotten


Question: podTolerations.SetExternalKubeInformerFactory(externalInformers) context = initTestScheduler(t, context, true, nil) <ins> defer cleanupTest(t, context) </ins> cs := context.clientSet informers := context.informerFactory _, err := cs.CoreV1().Namespaces().Create(context.ns)

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: Which test(s) are failing: TaintBasedEvictions Since when has it been failing: probably since it was in this PR? Testgrid link: Reason for failure: Anything else we need to know: /sig scheduling /kind flake /priority important-soon /milestone v1.17
You must be a member of the GitHub team to set the milestone. If you believe you should be able to issue the /milestone command, please contact your and have them propose you as an additional delegate for this responsibility. <detailsIn response to : <!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs -- TaintBasedEvictions probably since it was in this PR? Anything else we need to know: /sig scheduling /kind flake /priority important-soon /milestone v1.17 </details
cc.
Looking into this. We moved it to integration because it was flaking in e2e too
/assign damemi
Not sure yet, but I do see this in the logs of the failing tests: which, if I understand this, would remove all the taints and mark nodes as reachable (which would explain why the test is failing waiting for node taints: ). But I need to figure out why all the nodes are becoming unavailable
is that a full fix for this flake or just partial? I still saw it failed yesterday so just checking... :)
I think I also run into this on 13th Oct: Full logs:
Now gets merged, which might fix the flake. Let's keep an eye on this.
reported it still flaked: Can you continue looking into this?
I'll keep trying, from my previous debugging it seems that in some cases the test cluster is entering full disruption mode and removing all of the taints from the nodes. I know we specifically made changes to the test (using multiple nodes) to ensure it wouldn't become fully disrupted, do you have any idea what could still be causing that?
There is some flag controlling how the dispution thing is managed internally, maybe you can check how a node lifecycle manager is constructed.

The test still flakes after we merge , we still need to investigate the cause ,
/assign
Another flake:
These flakes are consistently (always) timeouts waiting on node taints:
I'm working on this. Will send out a PR soon.
/reopen We see another flake since got merged: However, don't panic :) The original "Entering master disruption mode" cause has been fixed - you won't see that message in the above log. This time it's a smaller issue. I will explain it in another PR.
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen The job is still flaky
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/close But for another test :)
Closing this issue. <detailsIn response to : /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: return proxier, nil } <ins> // Create a knftables.Interface and check if we can use the nftables proxy mode on this host. func getNFTablesInterface(ipFamily v1.IPFamily) (knftables.Interface, error) { var nftablesFamily knftables.Family if ipFamily == v1.IPv4Protocol { nftablesFamily = knftables.IPv4Family } else { nftablesFamily = knftables.IPv6Family } // We require (or rather, knftables.New does) that the nft binary be version 1.0.1 // or later, because versions before that would always attempt to parse the entire // nft ruleset at startup, even if you were only operating on a single table. // That's bad, because in some cases, new versions of nft have added new rule // types in ways that triggered bugs in older versions of nft, causing them to // crash. Thus, if kube-proxy used nft < 1.0.1, it could potentially get locked // out of its rules because of something some other component had done in a // completely different table. nft, err := knftables.New(nftablesFamily, kubeProxyTable) if err != nil { return nil, err } // Likewise, we want to ensure that the host filesystem has nft >= 1.0.1, so that // it's not possible that *our* rules break *the system's* nft. (In particular, we // know that if kube-proxy uses nft >= 1.0.3 and the system has nft <= 0.9.8, that // the system nft will become completely unusable.) Unfortunately, we can't easily // figure out the version of nft installed on the host filesystem, so instead, we // check the kernel version, under the assumption that the distro will have an nft // binary that supports the same features as its kernel does, and so kernel 5.13 // or later implies nft 1.0.1 or later. https://issues.k8s.io/122743 // // However, we allow the user to bypass this check by setting // `KUBE_PROXY_NFTABLES_SKIP_KERNEL_VERSION_CHECK` to anything non-empty. if os.Getenv("KUBE_PROXY_NFTABLES_SKIP_KERNEL_VERSION_CHECK") != "" { kernelVersion, err := utilkernel.GetVersion() if err != nil { return nil, fmt.Errorf("could not check kernel version: %w", err) } if kernelVersion.LessThan(version.MustParseGeneric(utilkernel.NFTablesKubeProxyKernelVersion)) { return nil, fmt.Errorf("kube-proxy in nftables mode requires kernel %s or later", utilkernel.NFTablesKubeProxyKernelVersion) } } return nft, nil } </ins> // internal struct for string service information type servicePortInfo struct { *proxy.BaseServicePortInfo

continuing discussion from and
/sig network /triage accepted
from I think we'll officially declare that we support kernel 5.4+ and nftables command-line 0.9.2+, and we can use a expression to check that. (Which means really we'll support kernel 5.3, but we won't advertise it that way.) FTR (and maybe worth documenting in a comment wherever we put the check): If we decide we need concatenations-with-ranges support again later, that would imply 5.6/0.9.4. If we want to use in sets/maps, that would require at least 0.9.4 (and maybe later depending on exactly how we're using it; we'd have to re-check the release notes). Support for comments on tables/chains/sets/maps/etc (anything but rules and elements) didn't come until 5.10/0.9.7, but knftables itself already does a for that at construct time, and just drops those comments if they're not supported.
There was a and about nft segfaults. It appears that specifically, nft v0.9.4 to v0.9.8 (inclusive) will segfault if asked to do anything when any chain in any table contains a certain kind of set/map as created by nft v1.0.3 or later. Specifically, rules like these trigger it (I think specifically it's caused by using and together, which the anonymous map as-generated-by-1.0.3+ uses. Using a named map rather than an anonymous one does not help.) Note that with nft < 1.0.1, doing any operation requires parsing the entire current ruleset. So that means when kube-proxy is using nft 1.0.3 or newer, nothing else in the host network namespace can use nft 0.9.4 - 0.9.8 for anything_. if the system firewall or something else in the host network namespace is using nft 1.0.3 or newer, then it might create rules that will break kube-proxy if kube-proxy is using nft 0.9.4 - 0.9.8.

If kube-proxy is using nft 1.0.1 or later, then it should not be possible for any other well-behaved component to break it, even if there are future bugs like the current one, because kube-proxy never looks at any tables but its own, so it wouldn't see things that other components did that it couldn't understand. OK, so what does this mean for us? We could just ship the official kube-proxy images with hand-built nft 0.9.2 rather than whatever's current in Debian. Then we guarantee we don't install rules other people can't deal with. Except that then we potentially run into problems if the system uses a much newer nft with some new bug that breaks 0.9.2. (We could do something -esque here, but ... ugh.) I think we actually want to go the opposite way, and have kube-proxy print a warning at startup if your nft is older then 1.0.1, because older versions run the risk of being broken by something someone else is doing in another table. But then there's the problem of us breaking other people. Or, even if we don't break other software, we still make it that admins on some systems would be unable to debug nftables-based problems... I guess we could avoid using the known-problematic syntax... the simplest way to do this would be to revert back to doing endpoints the way that iptables does: of course, that's less efficient, but...
Sounds to me that 1.0.2 is the golden version. It doesn't break 0.9.8, and it doesn't parse the entire current ruleset.
golden relative to the bugs we know about right now anyway... but maybe building our official images with 1.0.2 for a while would be a good plan... (Unfortunately we would have to actually build it ourselves. We can't just pull a 1.0.2 build from a random older Debian release because Debian never shipped that version; bullseye had 0.9.8 and bookworm had 1.0.6.)
thoughts on "use less efficient rules" vs "build images with a very specific nft version"?

build images with nft 1.0.1 seems the best option to me , because it seems that nft < 1.0.1 is not prod ready based on the different behaviors we are observing. If this happens between minors in 1.x.y then I think is a different discussion
The question wasn't just about what version we should use in our images, it was how we should deal with the bad interactions with older versions of nftables. It's one thing to say "nftables kube-proxy will not work when using nft < 1.0.1", but it's another thing to say "you are not allowed to run nftables kube-proxy on hosts where anybody is using nft < 1.0.1 (because otherwise kube-proxy will break other users of nft)". There is no reliable way for us to figure out the version of nft on the host from within the kube-proxy container. The best we could do would be to assume that the nft and kernel versions are roughly contemporary, and use the kernel version as a proxy. nft 1.0.1 was contemporary with kernel 5.16, and nft 1.0.0 was contemporary with 5.13. If we require kernel 5.13 / nft 1.0.1, we can support distros newer than mid 2021: RHEL 9 (kernel 5.14 + backports / nft 1.0.4) Debian 12 / bookworm / newstable (kernel 6.1 / nft 1.0.6) Ubuntu 22.04LTS (kernel 5.15 or newer / nft 1.0.2) (Not sure about SLES?)

Recent Fedora and other non-LTS distros Whereas our current plan of requiring kernel 5.4 / nft 0.9.2 means "newer than late 2019" and adds: RHEL 8 (kernel 4.18 + many backports / originally nft 0.9.3, later 1.0.4) Debian 11 / bullseye / oldstable (kernel 5.10 / nft 0.9.8) Ubuntu 20.04LTS (kernel 5.4 or newer / nft 0.9.3) (Not sure about SLES?) OK, so first decision: do we feel like we need to support running nftables kube-proxy on RHEL 8 / Debian 11 / Ubuntu 20.04LTS, or are we OK with only supporting the newer distros? For comparison, seems to be leaning toward 4.18/4.19 (from late 2018) as the minimum supported kernel for kubernetes as a whole, so requiring kernel 5.13 for nftables would add several more years before we could really even consider making it the default backend for kube-proxy. (We could also have kube-proxy require the newer versions by default, but have a config option to allow the older versions, and document exactly what can go wrong with that.) If we do want to still support the older distros (either unconditionally or via config option), there are two more questions: we decouple our kernel and nft requirements, and say that kube-proxy requires nft = 1.0.1 regardless of what kernel you're using? (I think "yes"?) we want to do some sort of workaround for the segfault bug (either changing the endpoint rules, or shipping the kube-proxy image with nft == 1.0.2 rather than whatever-version-is-currently-in-debian)? (If we had a config option to allow older kernels, we could make that option also control whether we used vmap-style or iptables-style endpoint rules?)

OK, leaning toward: Docs: We document that nftables kube-proxy requires nftables = 1.0.1, and recommends a kernel = 5.13 We also document that, currently, it can be used on systems with kernels as old as 5.4 if you set the option in the kube-proxy config, but (a) this will likely cause problems with other nftables users on the system (with details), and (b) this option is intended to go away at some point as the LTS distros with older kernels become unsupported. Code: We add to We check the nft client version (via a new knftables API), and if it's < 1.0.1, we error out. We check the kernel version (via ) and if it's < 5.13: if is (the default) we error out if is : if an with a expression passes, we continue, after warning that we might break other users on the system if the fails, we error out because the kernel is too oldWe don't change our rules or change the version of nft in our images, because the problems will only affect users who have explicitly opted in.
oh, and CI apparently has a 5.15 kernel, so we won't need to update the kind job config
I envision we'll hit more of these things in the future, so it will be nice you implement it, in a modular way, kind of the iptables runner that has all these different behaviors hidden from the users
The idea was that for things that affect knftables's own functioning, or its public API, it would check those things itself and hide the details from the caller. But the caller would be expected to use to figure out things like "can I use in a rule", since knftables can't just abstract around that (in general). (Right now the only thing knftables abstracts around is whether table/chain/set/map comments are supported; you're always allowed to specify them in the API, but .) I was originally thinking that the "require nft 1.0.1" restriction would be at the kube-proxy level rather than the knftables level, but I'm thinking now it would make more sense to require that for all knftables users.


Question: return &sg, nil } <ins> func (az *Cloud) shouldUpdateLoadBalancer(clusterName string, service *v1.Service) bool { _, _, existsLb, _ := az.getServiceLoadBalancer(service, clusterName, nil, false) return existsLb && service.ObjectMeta.DeletionTimestamp == nil } </ins> func logSafe(s *string) string { if s == nil { return "(nil)"

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: The issue has been observed on Azure but should also apply to other providers as well. On Azure we observed that there are orphaned Public IPs that don't belong to any Kubernetes LoadBalancer service but from looking at the tags and the name it is clear that Kubernetes created them. In the Azure Activity Log, you can see that the Public IP was deleted when the K8s service was deleted but recreated a few minutes later. That re-creation is also visible in the K8s controller manager logs as a reconcile of the service. In our case, we add additional worker nodes and delete services at the same time. I think this leads to the re-creation here Because the service list is retrieved at the beginning of the reconcile loop, services that are deleted while the loop is reconciling a recreated by that loop and afterwards orphaned. What you expected to happen: No orphaned public IPs How to reproduce it (as minimally and precisely as possible): Create a few Kubernetes Load Balancer services (we have around 20). Then scale your worker nodes and delete one service at the same time. Anything else we need to know?: Environment: Kubernetes version (use ): 1.15.0 Cloud provider or hardware configuration: Azure OS (e.g: ): Core OS Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig azure
/sig cloud-provider
Thanks for reporting the issue. I think this should be fixed during v1.16 release cycle. /milestone v1.16
In the nodeSyncLoop -updateLoadBalancerHosts -lockedUpdateLoadBalancerHosts call chain, I don't see where the mutex lock is obtained.
Hello! I'm the bug triage lead for the 1.16 release cycle and considering this issue is tagged for 1.16, I'd like to check its status. The code freeze is starting on August 29th EOD PST (about 4 days from now) and once the code freeze starts, only release-blocking changes will be considered to be merged.

Is this issue still planned to be fixed for the 1.16 release?
The issue would probably not able to be fixed in v1.16. AFAIK, no one is working on it now.
Thank you for the update! I'll go ahead and remove the issue from the milestone. If anything changes in the meanwhile, please let us know. /milestone clear
/assign
/assign
GitHub didn't allow me to assign the following users: chewong. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Doesn't LB Finalizer support fix this? We enabled this by default starting v1.16
Initial PR: Bump to beta in v1.16:


Question: // we don't perform the clean up just yet at that could destroy information // needed for the container to start (e.g. Windows credentials stored in // registry keys); instead, we'll clean up when the container gets removed <del> ds.containerCleanupInfos[containerID] = cleanupInfo </del> <ins> ds.setContainerCleanupInfo(containerID, cleanupInfo) </ins> } return &runtimeapi.CreateContainerResponse{ContainerId: containerID}, nil }

What happened: Windows kubelet panic occasionally on nodes which have high volume of pods run and delete. Full logs: Preliminary investigate result: How to reproduce it (as minimally and precisely as possible): Run high volume of pods and delete them when they completed in Windows nodes. Environment: Kubernetes version (use ): 1.17.7 Cloud provider or hardware configuration: Azure AKS OS (e.g: ): Windows Server 2019 Kernel (e.g. ): 10.0..1282
/assign I would like to give this a shot.
/remove-sig windows I think this is not a windows specific problem. /sig node /area kubelet
Hi! Thanks for your report, I have filed to address this issue.
This issue is a bit urgent for our situation, it occurs every few days in production clusters. Seems the PR is pending a few days, is it possible to increase the priority to merge it trunk and 1.17/1.18 branch? Thanks
I understand how urgent this issue is, but I am not privileged to do anything further :(
Do you know anyone who can help?
looks like a simple fix, add a lock for , I could help.
There is already a fix by , the problem is the PR is pending, not sure if you or someone can increase the priority to merge it trunk and 1.17/1.18 branch asap. Our AKS clusters are affecting by this frequently.
Thanks guys.


Question: log.Fatalf("Error: %#v", err) } return <ins> case "resize": args := flag.Args() if len(args) < 3 { log.Fatal("usage: cloudcfg resize <name> <replicas>") } name := args[1] replicas, err := strconv.Atoi(args[2]) if err != nil { log.Fatalf("Error parsing replicas: %#v", err) } err = cloudcfg.ResizeController(name, replicas, kube_client.Client{Host: *httpServer, Auth: auth}) if err != nil { log.Fatalf("Error: %#v", err) } return </ins> case "rm": err = cloudcfg.DeleteController(flag.Arg(1), kube_client.Client{Host: *httpServer, Auth: auth}) if err != nil {

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT Kubernetes version (use ): HEAD (latest) Environment: dind What happened: Enpoints object is constantly getting updated every 2 seconds, even though nothing has changed in the object. It will produce lot of traffic on any informer watching on endpoints. An update event is produced every 2 second per endpoint. What you expected to happen: Endpoints(any object) should be updated only when there is change in either spec or status. How to reproduce it (as minimally and precisely as possible): Just watch on endpoints Also see the logs from federation controller manager, which is watching on endpoints updates from federated cluster Anything else we need to know: Looking at the endpoints controller code . it looks like reflect.DeepEqual is not working for endpoints labels and Subsets (may be need sorting, just a guess). I also see another issue raised by some time back , related but slightly different. I think this is a bug and if fixed will reduce lot of traffic to etcd.
Those two endpoints specifically are used as shared lock objects by controllers. Agree that's a bad object to use because so many things watch them for updating iptables, etc
code in question:
dupe of


Question: s.Spec.Ports[0].Port++ }) framework.ExpectNoError(err) <del> if int(udpService.Spec.Ports[0].Port) == svcPort { framework.Failf("UDP Spec.Ports[0].Port (%d) did not change", udpService.Spec.Ports[0].Port) </del> <ins> svcPortOld := svcPort svcPort = int(udpService.Spec.Ports[0].Port) if svcPort == svcPortOld { framework.Failf("UDP Spec.Ports[0].Port (%d) did not change", svcPort) </ins> } if int(udpService.Spec.Ports[0].NodePort) != udpNodePort { framework.Failf("UDP Spec.Ports[0].NodePort (%d) changed", udpService.Spec.Ports[0].NodePort)

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: () Which test(s) are failing: Since when has it been failing: Jan 25 6:25AM PST was the first unsuccessful run Testgrid link: https://k8s- Reason for failure: . These are the logs from after the pod was . Anything else we need to know: Related to and (possibly?) Latest failing run as of writing: /sig network /cc
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: "//vendor/k8s.io/apimachinery/pkg/util/wait:go_default_library", "//vendor/k8s.io/apiserver/pkg/server/healthz:go_default_library", "//vendor/k8s.io/apiserver/pkg/util/feature:go_default_library", <del> "//vendor/k8s.io/apiserver/pkg/util/flag:go_default_library", </del> "//vendor/k8s.io/client-go/informers:go_default_library", "//vendor/k8s.io/client-go/informers/core/v1:go_default_library", "//vendor/k8s.io/client-go/informers/storage/v1:go_default_library",

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: I pull the latest code, and run with below cmd. But the cluster is not up. And I notice the kube-, What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): GitCommit:"" Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig cluster-lifecycle
/assign
Can you please try the fix above?
/assign
/unassign
your pr does not fix it yet? If not, I can help.
The fix from works for me! o/
ok, thanks
reopen , make less modifications.
sorry, i miss hyper...


Question: return nil, err } <del> if string(line) == separator || err == io.EOF { </del> <ins> sep := len([]byte(separator)) if i := bytes.Index(line, []byte(separator)); i == 0 { // We have a potential document terminator i += sep after := line[i:] if len(strings.TrimRightFunc(string(after), unicode.IsSpace)) == 0 { if buffer.Len() != 0 { return buffer.Bytes(), nil } if err == io.EOF { return nil, err } } } if err == io.EOF { </ins> if buffer.Len() != 0 { <ins> // If we're at EOF, we have a final, non-terminated line. Return it. </ins> return buffer.Bytes(), nil } <del> if err == io.EOF { return nil, err } } else { buffer.Write(line) </del> <ins> return nil, err </ins> } <ins> buffer.Write(line) </ins> } }

Kubernetes version (use ): Environment: Cloud provider or hardware configuration: Any OS (e.g. from /etc/os-release): Any Kernel (e.g. ): Any Install tools: Any What happened: If I use a template with the seperator, and if the line with the separator has trailing whitespace characters, then kubectl considers that to be the EOF instead of reading further. e.g. template What you expected to happen: It should not stop reading if there are trailing whitespace characters, it should just trim the separator line and ignore the whitespaces after the . This is how it worked until v1.3.0 kubectl version. How to reproduce it (as minimally and precisely as possible): Just use the template from above and you'll notice that it won't create the Service, but it'll create the Replication controller. Anything else do we need to know: This is on kubectl version 1.4.0
kubectl regression. cc
This bug is general (can be found on get, apply, delete, ...). May be somewhere in resource builder.
Might be , Janet Kuo wrote:
Confirmed - is the cause. would you take a look?
We may need to cherrypick the fix
Yaml rules are separator must be at beginning of line, three dashes, and new line. Did we change how strict we were in that PR? On Sep 27, 2016, at 7:46 PM, Janet Kuo wrote: We may need to cherrypick the fix — You are receiving this because you are on a team that was mentioned. Reply to this email directly, view it on GitHub , or mute the thread .
Yes, and the test was changed too (originally tests allow with trailing spaces)
According to spec (if my spec-alese is correct) newline + three dashes is the only requirement to terminate a document. Whitespace or prefixes are allowed afterwards, but for it to be valid YAML those must be valid at the beginning of a doc as well (as far as I can tell). So we should split and remove on any line starting with a newline and three dashes - but not consume the remaining whitespace (I believe) , Janet Kuo wrote:
We should probably exit non-0 and have an error message if we see something in the file that we don't expect. E.g.

is neither a separator nor parseable yaml.
We should add a test that uses a YAML document prefix rule and verify that it correctly passes / fails when YAML goes to parse. From reading the spec it appears that anything that is invalid when n--- is present is also invalid in a single document context, and vice versa. , Phillip Wittrock wrote:
I have rechecked the code, and I have located the root cause, since I am outing for a training these days, I will fix this right now after the training on Friday.
I am quite sorry about the regression. :(
should we allow trailing whitespace characters after , if so we can trim the whitespaces when we read line by line?
is it possible to include this fix to one of the 1.4 releases? We'd really like our users to have of k8s alpha features (Scheduled jobs is one of them) that can be configured only using 1.4 client; w/o them potentially hitting this regression.
I will double check this regression, and open a cherry-pick PR.
I am thinking create a cherry-pick PR for this to include it in 1.4, wdyt?
I would like to see this in 1.4 since as you note it may break scripts. On Oct 19, 2016, at 8:19 PM, TonyAdo wrote: I am thinking create a cherry-pick PR for this to include it in 1.4, wdyt? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub , or mute the thread .
Do we have a test case now? :) On Oct 19, 2016, at 8:19 PM, TonyAdo wrote: I am thinking create a cherry-pick PR for this to include it in 1.4, wdyt? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub , or mute the thread .
we already test this, you can see


Question: } func (rc *resourceMetricsCollector) collectContainerCPUMetrics(ch chan<- metrics.Metric, pod summary.PodStats, s summary.ContainerStats) { <del> if s.CPU == nil { </del> <ins> if s.CPU == nil || s.CPU.UsageCoreNanoSeconds == nil { </ins> return }

kubelet panic when accessing metrics/resource endpoint kubelet log kubelet works fine access metrics/resource endpoint of kubelet No response <details</details<details</details<details</details<details</details<detailsINFO[2022-07-14 05:36:21.474173892Z] Starting CRI-O, version: 1.23.0, git: (dirty) Version: 1.23.0 GitCommit: GitTreeState: dirty BuildDate: 1980-01-01T00:00:00Z GoVersion: go1.17.4 Compiler: gc Platform: linux/amd64 Linkmode: static BuildTags: apparmor, excludegraphdriverdevicemapper, seccomp, selinux SeccompEnabled: true AppArmorEnabled: true </details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/sig node


Question: "density.go", "framework.go", "hybrid_network.go", <ins> "memory_limits.go", </ins> "networking.go", "volumes.go", ], importpath = "k8s.io/kubernetes/test/e2e/windows", visibility = ["//visibility:public"], deps = [ <ins> "//pkg/kubelet/apis/config:go_default_library", </ins> "//staging/src/k8s.io/api/core/v1:go_default_library", <ins> "//staging/src/k8s.io/apimachinery/pkg/api/resource:go_default_library", </ins> "//staging/src/k8s.io/apimachinery/pkg/apis/meta/v1:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/labels:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/runtime:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/util/sets:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/util/uuid:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/watch:go_default_library", <ins> "//staging/src/k8s.io/client-go/kubernetes/scheme:go_default_library", </ins> "//staging/src/k8s.io/client-go/tools/cache:go_default_library", <ins> "//staging/src/k8s.io/kubelet/config/v1beta1:go_default_library", </ins> "//test/e2e/framework:go_default_library", "//test/utils/image:go_default_library", "//vendor/github.com/onsi/ginkgo:go_default_library",

Kubelet and System-reserve are implemented on Windows along with pod/container reserves which can help prevent overprovisioning, but we still need to test OOM behavior. We need a test case that will: kubelet & system reserve pods until node is full something that will take up system ram so that OOM is triggered /sig windows /milestone v1.14
I have started to look into this test case and have a few questions: What is the expected behavior we are testing for? What does "run something that will take up system ram" mean? Does that mean run a pod that uses up memory or run a system service that uses memory? After setting the reserves and as mentioned in we should be able to schedule pods until full and then once we trigger OOM (either by pod or system service) kubelet should continue running becuase of the reserves? Do the two settings need to be tested independently or they should be tested together? I found that seems to be doing something similar but it is in node-e2e. Is sig-windows running node-e2e tests currently?
/assign
GitHub didn't allow me to assign the following users: jsturtevant. Note that only and repo collaborators can be assigned and that issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
After researching this further and talking with I found that the does test the allocatable memory but uses CGroups to set reserves to test the reported vs actual. As windows does not have CGroups this test will not port as is. There is and the is not implemented so will not work either. The proposed testing solution would be two parts: kubelet & system reserve and verify allocatable is reporting properly. Schedule pods with known memory consumption values and verify the allocatable is correclty updated.Schedule until full and verify scheduling is not allowed. kubelet & system reserve and verify allocatable is reporting properly. Schedule Schedule until full and verify scheduling is not into machine and run process that consumes memory and verify kubelet and pods remain running. There is an open question on the second test as to whether kubelet and will be have properly.

If it does not we will submit test 1 and open an issue to resolve 2.
I would suggest adding a third case that would test the behavior when excessive memory pressure is exerted on the node from the pods. This is perhaps a more common use case than 2 since the pod workloads are more likely to exhibit greater resource variability than node system processes. It would also carry over as a test for when/if node allocatable enforcement get implemented for windows nodes. Additionally, to complement this work would it be suitable to test windows pod memory limit enforcement? Or is this tested elsewhere?
Agreed this is a more common case. I am not sure that it is. Is this tested on linux somewhere?
I am seeing strange numbers reported when performing a test case. I scheduled a few pods to test memory limits and then schedule a pod with no memory limit and a process that will consume all of the available memory 500mb at a time: After the pod consumes all available memory, When I compare Node Allocatable/Node usage/Pod Usage I am not seeing the numbers add up properly: My node is staying online and kubelet is reporting no memory pressure. The total node memory consumption is over the Allocatable memory. What is the expected behavior and what should the test look like?
I believe node allocatable will always just be capacity minus reserves (system and kubelet) - So in this scenario it looks like you have and/or set? The memory % on the node is relative to the allocatable amount, and still seems to be under the total memory capacity - are you sure that all available memory has been consumed by your oom pod? The node should be tainted with memory-pressure when a hard or soft eviction threshold is met (). What do you have set for and ?
Thanks for the clarification. I currently have these settings: And on the machine I am seeing most memory being used: If I understand correctly, I start to see memory-pressure once I get into the setting above. In that case I should start to see pods get evicted, so for a valid test I could adjust the values to be easier to hit then test to see that pods get evicted properly which would be similar the .

I am going to adjust a few of the settings and manually test that this is working then see what it will take to port the PriorityMemoryEviction, which currently uses cgroups which won't work on windows. This combined with testing that Memory Allocatable is being properly set should give us a good test coverage for windows on this scenario.
I performed manual testing and found that even when memory was used up on the machine the "Memory Pressure" notification was not fired. After looking deeper I found is not implemented for windows. The , which uses cgroups, provides the . This means even though is being it isn't used. On Windows, Node allocatable when used with limits and requests does prevent the node from over scheduling. I believe the the test and  (this only shows cpu and I could not find a test that demonstrated memory) demonstrate this works. But this doesn't stop a pod from consuming all the memory on the node and since eviction (hard/soft/node allocatable) is not implemented on windows this means a node can end up resource starved. I think the only valid test to be written for this scenario, as it stands now, is validating that Node Allocatable is being reported properly. Beyond that there is no functionality that can be validated around OOM behavior for the windows node.
Yeah, that sounds to be in agreement with what I've observed on the windows nodes that I run. To prevent resource starvation (as a workaround), would setting mem/cpu requests equal to mem/cpu limits for all windows pods work? I think the request / limit values get passed down into the docker daemon when it starts a container, so is it possible that docker could enforce the memory limit?
Yes, setting limits==requests would help the scheduler not over provision the nodes. I think the best way to approach this particular issue for 1.14 is to make sure docs call out what works/doesn't work and give guidance to on how to prevent resource starvation on nodes. Is there a place in the docs that would make sense for this information?

The call outs would be: , , and flags do update Node Allocatable is not implemented and are not implemented Condition is not implemented in Windows There are no OOM actions taken by kubelet As a work around to previous, and should always be equal to prevent memory resource starvation as this allows the scheduler to to use limits and Kubelet running on the windows node does not have memory restrictions. and system-reserve does not set limits on kubelet or processes running the host. This means kubelet or a process on the host could cause memory resource starvation outside the and scheduler
thank you for the thorough investigation


Question: } } <ins> type fakeClosable struct { closed bool } func (c *fakeClosable) Close() error { c.closed = true return nil } func TestRevertPorts(t *testing.T) { testCases := []struct { replacementPorts []localPort existingPorts    []localPort expectToBeClose  []bool }{ { replacementPorts: []localPort{ {port: 5001}, {port: 5002}, {port: 5003}, }, existingPorts:   []localPort{}, expectToBeClose: []bool{true, true, true}, }, { replacementPorts: []localPort{}, existingPorts: []localPort{ {port: 5001}, {port: 5002}, {port: 5003}, }, expectToBeClose: []bool{}, }, { replacementPorts: []localPort{ {port: 5001}, {port: 5002}, {port: 5003}, }, existingPorts: []localPort{ {port: 5001}, {port: 5002}, {port: 5003}, }, expectToBeClose: []bool{false, false, false}, }, { replacementPorts: []localPort{ {port: 5001}, {port: 5002}, {port: 5003}, }, existingPorts: []localPort{ {port: 5001}, {port: 5003}, }, expectToBeClose: []bool{false, true, false}, }, { replacementPorts: []localPort{ {port: 5001}, {port: 5002}, {port: 5003}, }, existingPorts: []localPort{ {port: 5001}, {port: 5002}, {port: 5003}, {port: 5004}, }, expectToBeClose: []bool{false, false, false}, }, } for i, tc := range testCases { replacementPortsMap := make(map[localPort]closeable) for _, lp := range tc.replacementPorts { replacementPortsMap[lp] = &fakeClosable{} } existingPortsMap := make(map[localPort]closeable) for _, lp := range tc.existingPorts { existingPortsMap[lp] = &fakeClosable{} } revertPorts(replacementPortsMap, existingPortsMap) for j, expectation := range tc.expectToBeClose { if replacementPortsMap[tc.replacementPorts[j]].(*fakeClosable).closed != expectation { t.Errorf("Expect replacement localport %v to be %v in test case %v", tc.replacementPorts[j], expectation, i) } } for _, lp := range tc.existingPorts { if existingPortsMap[lp].(*fakeClosable).closed == true { t.Errorf("Expect existing localport %v to be false in test case %v", lp, i) } } } } </ins> // TODO(thockin): add a test for syncProxyRules() or break it down further and test the pieces.

The symptom is the nodeport service is accessible thru hostip:hostport. But the port on host is not occupied. This is exposed by e2e on kubenet. I noticed in the kube- Kube-proxy did open the port. But later, kube-proxy closed the ports upon iptables-restore failure. It never reopen the ports. Hence causing the e2e failure. The code section in question: Not sure why failed. Error messge is not helpful since line 139 is ref:
Also of interest: did it exit with 1? This should be clear from the iptables commands it executed before the commit, which are hopefully in the logs (can you post them). If it's something like a non-exist chain, we should fix that. we retry on exit 1? Handwaving without looking at code: I'd expect something to get requeued, and to reinvoke syncProxyRules once again, in a time frame << test timeout.
Yeah, what was line 139 in the restore output? , Prashanth B wrote:
is No retry though. It is going to catch up in the next sync. So I think no retry for is needed.
FYI Jenkins job:
That iptables restore is 139 lines long, so it's unhappy about the last line? can you hunt ths? , Minhan Xia wrote:
The next sync is whenver the next service/endpoint changes right? Errors during the sync should cause an immediate retry of sync, preferably with backoff. If a service or endpoint never changes we should still recover. Yes we should hunt down the bad line, but it's not the last one right? COMMIT will fail if eg: Even though it's obviously not commit but the tcp/upd thing.
iptables resyncs every 30s, not ideal but in this case I don't expect that to fix the error I meant that we need to root cause it :) , Prashanth B wrote:
But in this case, iptables-restore is going to complain . Not line 2
Hmm
I manually restored the 139 lines above it worked fine - no errors. Even more curious. , Prashanth B wrote:
I also tried to execute the same sequence of iptables-restore. It worked. No errors.

Was this a one-time thing? Were there any other changes in your git repo? , Minhan Xia wrote:
This happened in kubenet suite. Only difference is . I looked at a few runs where e2e failed. This happened 3-4 times over the whole test, but not consecutively. In all-passed runs I checked, this never happened (Maybe I am lucky).
Of , Minhan Xia wrote:


Question: // GPU drivers might have already been installed. if !areGPUsAvailableOnAllSchedulableNodes(f) { // Install Nvidia Drivers. <del> ds := dsFromManifest(cosNvidiaDriverInstallerPath) </del> <ins> ds := dsFromManifest(cosNvidiaDriverInstallerUrl) </ins> ds.Namespace = f.Namespace.Name _, err := f.ClientSet.Extensions().DaemonSets(f.Namespace.Name).Create(ds) framework.ExpectNoError(err, "failed to create daemonset")

xref: /sig node /area test
I am working on this.
Can you update this with the types of tests that you are adding? Is this similar to ?
No, this is for running the existing GPU e2e tests on GKE which are currently running on only GCE.
The referenced PRs appear to have merged, closing.


Question: framework.Failf("Error retrieving logs: %v", err) } <ins> if waitErr != nil { framework.Logf("Pod log:n%s", logOutput) framework.Failf("Pod did not stop running: %v", waitErr) } </ins> By("Verifying logs") verifyLogMessage(logOutput, "Accepted client connection") verifyLogMessage(logOutput, "Done")

https://k8s- Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends no data, and disconnects [Conformance] {Kubernetes e2e suite}
So far I haven't found anything in the logs. The pfpod container starts, the e2e test receives all the data from it. At that time, the container should exit and the pod should be stopped. For some reason, either the container doesn't stop, or the kubelet doesn't see it stop. It's hard to tell which because the logs don't have enough information to make that determination.
https://k8s- Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends no data, and disconnects [Conformance] {Kubernetes e2e suite}
Might be the same issue as
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
Builds: kubernetes-e2e-gce-etcd3 kubernetes-e2e-gci-gce Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends no data, and disconnects [Conformance] {Kubernetes e2e suite}
https://k8s- Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends no data, and disconnects [Conformance] {Kubernetes e2e suite} Happened on a presubmit run in .
https://k8s- Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends no data, and disconnects [Conformance] {Kubernetes e2e suite}
Adding to assignee list b/c of ssh tunnel flakes (i.e.

this issue has multiple, different root causes)
Builds: ci-kubernetes-e2e-gce-etcd3 ci-kubernetes-e2e-gci-gce-etcd3 ci-kubernetes-e2e-gci-gce-proto kubernetes-e2e-gce-etcd3 kubernetes-e2e-gci-gce-kubenet kubernetes-e2e-gci-gce-proto kubernetes-e2e-gci-gce-release-1.4 kubernetes-e2e-gci-gke-prod-smoke Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends no data, and disconnects [Conformance] {Kubernetes e2e suite}
https://k8s- Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends no data, and disconnects [Conformance] {Kubernetes e2e suite}
Closing this issue as the corresponding test has been removed from cc:


Question: err = ipt.Restore(utiliptables.TableNAT, natLines, utiliptables.NoFlushTables, utiliptables.RestoreCounters) if err != nil { klog.Errorf("Failed to execute iptables-restore for %s: %v", utiliptables.TableNAT, err) <ins> metrics.IptablesRestoreFailuresTotal.Inc() </ins> encounteredError = true } }

What would you like to be : kube-proxy should expose better metrics around tracking success and failures. Administrators should be able to alert on failure counts. Why is this needed: There are several motivating cases: iptables lock contention causes to fail at an increasing rate as services + endpoints grows or are failing to execute Ultimately, there is essentially no feedback from kube-proxy if it fails to program the service network. This makes it annoyingly difficult to debug when things go wrong. As many reasonable hints we can provide should be . /sig network
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
is anyone working one this..
what cases do you think are poorly covered? /assign
thanks for triaging this :-) The specific cases are when fails - that function logs an error but doesn't otherwise surface any information. We should have a top-level metric for failing, along with any other lower-level metrics that might make sense for administrators, e.g. fails, fails.
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
/help-wanted
/assign


Question: if notifier, ok := s.ClientCA.(dynamiccertificates.Notifier); ok { notifier.AddListener(dynamicCertificateController) } <ins> if notifier, ok := s.Cert.(dynamiccertificates.Notifier); ok { notifier.AddListener(dynamicCertificateController) } </ins> // start controllers if possible if controller, ok := s.ClientCA.(dynamiccertificates.ControllerRunner); ok { // runonce to be sure that we have a value.

When we want to rotate certificates for aggregated API servers, we have to restart (kill) pods in order for API server to pick the new certificates. It would be very useful to implement some sort of self-awareness for the aggregated API servers, so they can pick-up new certificates when they're updated, without user interaction. If we have certificate files in pods, that are provided by some volumeSource, that can be persistent storage or a Secret, when file gets changed on the volume, the kubelet updates file in the pod usually in matters of few seconds. However, aggregated API servers are not self-aware of such change and users have to manually restart the pods. This brings additional complexity, especially if you want to automate the process. The first problem is that you need to know what pods you need to restart. But the potentially biggest problem is that you need additional permissions to do so. That would include RBAC permissions for listing pods, listing deployments and for deleting pods. That can make a security issue in some setups. By making aggregated API servers sort of self-aware, this could be avoided. Create a health check somewhere in , that utilizes a file watcher to watch for certificates changes. When certificate is changed, fail a health check. Failing health check will restart the pod without user interaction. This could be the most easiest and simplest solution to the problem. Such feature should/could be optional and guarded by a flag, let's name it . Some delay could be required between restarting pods. We probably don't want to kill all pods in the same time. This is up to be discussed. I would like to work on this issue, so feel free to assign it to me. /kind feature /sig api-machinery /area apiserver /cc
this could also be done by a purpose-built container that watched the same input files. as you point out, you likely would not want to restart all apiservers at once, which makes it unclear whether you'd want to build in the restart logic to the individual servers
That coordination would certainly work and be a valuable tool for someone to build to help manage rolling generic processes if we never get a pod restart. However, being able to have a "usually won't all go down together" option built-in is easy to understand and will resolve the problem until then.

It isn't a stretch to say that if your process isn't fulfilling the current intent of the pod, it isn't healthy. This gives us a thing between dynamic config reloading and nothing. A container that can coordinate based on a service would be really cool, but I think that'd be a separate thing.
/cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: conn.Close() By("Waiting for the target pod to stop running") <del> if err := f.WaitForPodNoLongerRunning(pod.Name); err != nil { framework.Failf("Pod did not stop running: %v", err) } </del> <ins> waitErr := f.WaitForPodNoLongerRunning(pod.Name) </ins> By("Retrieving logs from the target pod") logOutput, err := framework.GetPodLogs(f.Client, f.Namespace.Name, pod.Name, "portforwardtester")

https://k8s- Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends no data, and disconnects [Conformance] {Kubernetes e2e suite}
So far I haven't found anything in the logs. The pfpod container starts, the e2e test receives all the data from it. At that time, the container should exit and the pod should be stopped. For some reason, either the container doesn't stop, or the kubelet doesn't see it stop. It's hard to tell which because the logs don't have enough information to make that determination.
https://k8s- Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends no data, and disconnects [Conformance] {Kubernetes e2e suite}
Might be the same issue as
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
Builds: kubernetes-e2e-gce-etcd3 kubernetes-e2e-gci-gce Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends no data, and disconnects [Conformance] {Kubernetes e2e suite}
https://k8s- Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends no data, and disconnects [Conformance] {Kubernetes e2e suite} Happened on a presubmit run in .
https://k8s- Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends no data, and disconnects [Conformance] {Kubernetes e2e suite}
Adding to assignee list b/c of ssh tunnel flakes (i.e.

this issue has multiple, different root causes)
Builds: ci-kubernetes-e2e-gce-etcd3 ci-kubernetes-e2e-gci-gce-etcd3 ci-kubernetes-e2e-gci-gce-proto kubernetes-e2e-gce-etcd3 kubernetes-e2e-gci-gce-kubenet kubernetes-e2e-gci-gce-proto kubernetes-e2e-gci-gce-release-1.4 kubernetes-e2e-gci-gke-prod-smoke Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends no data, and disconnects [Conformance] {Kubernetes e2e suite}
https://k8s- Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends no data, and disconnects [Conformance] {Kubernetes e2e suite}
Closing this issue as the corresponding test has been removed from cc:


Question: } func updateDataChangeMsg(oldObj []byte, newObj []byte) string { <del> msg := "not labeled" </del> <ins> msg := MsgNotLabeled </ins> if !reflect.DeepEqual(oldObj, newObj) { <del> msg = "labeled" </del> <ins> msg = MsgLabeled if len(newObj) < len(oldObj) { msg = MsgUnLabeled } </ins> } return msg }

Currently, the cmd output msg is not distinguished between label and unlabel (using suffix ), both of them response . To distinguish between label and unlabel msg: , just like .
/sig cli
/triage accepted


Question: "testing" "github.com/stretchr/testify/assert" <del> "github.com/stretchr/testify/require" </del> v1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" <ins> "k8s.io/kubernetes/pkg/apis/scheduling" </ins> ) <ins> var ( systemPriority      = int32(scheduling.SystemCriticalPriority) systemPriorityUpper = systemPriority + 1000 ) // getTestPod generates a new instance of an empty test Pod func getTestPod(annotations map[string]string, podPriority *int32) *v1.Pod { pod := v1.Pod{ TypeMeta: metav1.TypeMeta{ Kind:       "Pod", APIVersion: "v1", }, ObjectMeta: metav1.ObjectMeta{ Name:      "foo", Namespace: "default", }, } // Set pod Priority in Spec if exists if podPriority != nil { pod.Spec = v1.PodSpec{ Priority: podPriority, } } // Set annotations if exists if annotations != nil { pod.Annotations = annotations } return &pod } func configSourceAnnotation(source string) map[string]string { return map[string]string{ConfigSourceAnnotationKey: source} } func configMirrorAnnotation() map[string]string { return map[string]string{ConfigMirrorAnnotationKey: "true"} } </ins> func TestGetValidatedSources(t *testing.T) { <del> // Empty. sources, err := GetValidatedSources([]string{""}) require.NoError(t, err) require.Len(t, sources, 0) // Success. sources, err = GetValidatedSources([]string{FileSource, ApiserverSource}) require.NoError(t, err) require.Len(t, sources, 2) // All. sources, err = GetValidatedSources([]string{AllSource}) require.NoError(t, err) require.Len(t, sources, 3) // Unknown source. _, err = GetValidatedSources([]string{"taco"}) require.Error(t, err) </del> <ins> tests := []struct { name        string sources     []string errExpected bool sourcesLen  int }{ { name:        "empty source", sources:     []string{""}, errExpected: false, sourcesLen:  0, }, { name:        "file and apiserver source", sources:     []string{FileSource, ApiserverSource}, errExpected: false, sourcesLen:  2, }, { name:        "all source", sources:     []string{AllSource}, errExpected: false, sourcesLen:  3, }, { name:        "unknown source", sources:     []string{"unknown"}, errExpected: true, sourcesLen:  0, }, } for _, test := range tests { t.Run(test.name, func(t *testing.T) { sources, err := GetValidatedSources(test.sources) if test.errExpected { assert.Error(t, err) } else { assert.NoError(t, err) } assert.Len(t, sources, test.sourcesLen) }) } </ins> } func TestGetPodSource(t *testing.T) { <del> cases := []struct { pod         v1.Pod </del> <ins> tests := []struct { name        string pod         *v1.Pod </ins> expected    string errExpected bool }{ { <del> pod:         v1.Pod{}, </del> <ins> name:        "cannot get pod source", pod:         getTestPod(nil, nil), </ins> expected:    "", errExpected: true, }, { <del> pod: v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Annotations: map[string]string{ "kubernetes.io/config.source": "host-ipc-sources", }, }, }, </del> <ins> name:        "valid annotation returns the source", pod:         getTestPod(configSourceAnnotation("host-ipc-sources"), nil), </ins> expected:    "host-ipc-sources", errExpected: false, }, } <del> for i, data := range cases { source, err := GetPodSource(&data.pod) if data.errExpected { assert.Error(t, err) } else { assert.NoError(t, err) } assert.Equal(t, data.expected, source, "test[%d]", i) t.Logf("Test case [%d]", i) </del> <ins> for _, test := range tests { t.Run(test.name, func(t *testing.T) { source, err := GetPodSource(test.pod) if test.errExpected { assert.Error(t, err) } else { assert.NoError(t, err) } assert.Equal(t, test.expected, source) }) </ins> } } func TestString(t *testing.T) { <del> cases := []struct { </del> <ins> tests := []struct { </ins> sp       SyncPodType expected string }{

What would you like to be : Currently, in the following public methods do not have unit test coverage: - - These methods are great candidates for adding unit test coverage. Why is this needed: Adding unit test coverage will both ensure they are working correctly and (more importantly), encode their expected behavior in the test suite. Additionally, if we want to change their behavior in the future, it'll be helpful to already have the test suites. Extra note I think this could be a great candidate for someone who is new to contributing to sig-node/k8s. I'm happy to mentor any first time contributors :) Please respond saying you'd like to work on this issue and I'll assign it to you. Please reach out to me on k8s slack (where I'm also if I can be of any assistance.
/sig node
/good-first-issue
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hi, I would like to work in one of these.
/assign Awesome!
Hi, I am a first time contributor and would be happy to contribute to one of these.
Hi, I want to help if still need it.


Question: type Cluster struct { // LocationOfOrigin indicates where this object came from.  It is used for round tripping config post-merge, but never serialized. // +k8s:conversion-gen=false <del> LocationOfOrigin string </del> <ins> LocationOfOrigin string `json:"-"` </ins> // Server is the address of the kubernetes cluster (https://hostname:port). Server string `json:"server"` // TLSServerName is used to check server certificate. If TLSServerName is empty, the hostname used to contact the server is used.

yaml.Marshal a kubeconfig, as done in the example tests here LocationOfOrigin is not seen in output, as comment says "never serialized." Click "run example" doesn't suffer from this as it does some conversion <details</details<details</details<details</details<details</details<details</details<details</details
/assign
/sig api-machinery
/assign /triage accepted
is what's known as internal type. It is not intended to be serialized by code. The serialization version is in and does not have this field. I could get behind removing all json tags from the internal version, but I wouldn't refine tags that shouldn't be used.


Question: <ins> /usr/local/bin/kubecfg: file.managed: - source: salt://kube-bins/kubecfg - user: root - group: root - mode: 755 /usr/local/bin/kubectl: file.managed: - source: salt://kube-bins/kubectl - user: root - group: root - mode: 755 </ins>

Currently the master VM specified by the Vagrantfile does not have the built kubecfg binary in the vagrant user's path. This request is to have this binary present on the host system and in the vagrant user's path so I can Vagrant ssh into the machine and do 'kubecfg list pods'.
I'm going to generalize this a little so that kubecfg is available in our general cluster install scripts. This should be easier as we move to binary installs.
Should be kubectl, not kubecfg, but otherwise is a good idea. etcd tools would be useful, too:
I'll take this on if no one else wants to do it. May take me a day or two to get to it.
Thinking out loud here... do we really want to encourage people to do admin and/or job runny stuff on their master?
I see it as a debugging tool.


Question: disks := []compute.DataDisk{} if vm.StorageProfile != nil && vm.StorageProfile.DataDisks != nil { <del> disks = *vm.StorageProfile.DataDisks </del> <ins> disks = make([]compute.DataDisk, len(*vm.StorageProfile.DataDisks)) copy(disks, *vm.StorageProfile.DataDisks) </ins> } bFoundDisk := false for i, disk := range disks {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: In Azure disk attach/detach function, it uses slice() to store the disk list: And after changed, the original would also be changed, so original azure data disk list is corrupted after attach/detach disk example code as below: What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /sig azure /assign


Question: func detachAndDeletePDs(diskName string, hosts []string) { for _, host := range hosts { <ins> framework.Logf("Detaching GCE PD %q from node %q.", diskName, host) </ins> detachPD(host, diskName) By(fmt.Sprintf("Waiting for PD %q to detach from %q", diskName, host)) waitForPDDetach(diskName, host)

PD E2E tests have very aggressive clean up routine that does not wait for volumes to safely detached. After the test completes, the test cleanup routine unilaterally detaches the PD (instead of waiting for k8s to do it cleanly). Although this does not cause the test to fail, it results in errors in the kernel logs: We should modify these tests to wait for k8s to safely detach the disk.


Question: } func (cm *containerManagerStub) GetDevicePluginResourceCapacity() (v1.ResourceList, v1.ResourceList, []string) { <del> return nil, nil, []string{} </del> <ins> return cm.extendedPluginResources, cm.extendedPluginResources, []string{} </ins> } func (cm *containerManagerStub) NewPodContainerManager() PodContainerManager {

A kubernetes cluster is normally constructed from a variety of machine templates. Different instance groups may have different CPU, memory capacity. They may be equipped with specialized accelerators, network devices. I think we should have an extensible way for users to custom node configuration to simulate real production cluster states. Currently, nodelabel and nodetaints are supported. I am consider to accept a template with more configuration like resources so we won't grow option list in the future. Without this feature, it's hard for users to simulate heterogeneous cluster.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/sig scalability


Question: onFailure := newCaller() <del> addr, err := getAddressByName(r.compute, route.TargetNode) </del> <ins> ip, _, _ := net.ParseCIDR(route.DestinationCIDR) isCIDRv6 := ip.To4() == nil addr, err := getAddressByName(r.compute, route.TargetNode, isCIDRv6) </ins> if err != nil { return err }

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: The Kubernetes master created routes in the neutron router that assign an IPv6 nexthop for a IPv4 prefix. This is not possible and causes a reachability problem between the pods. Example route: What you expected to happen: When describing a node the master has the following information It should detect the address type and choose the correct InternalIP How to reproduce it (as minimally and precisely as possible): Deploy on a Openstack cloud with IPv6 enabled using the following playbook Anything else we need to know?: The wrong next hop is coming from calling the function and nowhere is a check if the address is IPv4 or IPv6 Environment: Kubernetes version (use ):Cloud provider or hardware configuration: Openstack OS (e.g. from /etc/os-release): Ubuntu Xenial Kernel (e.g. ):Install tools: Others: /sig openstack
- Thank you for this detailed analysis. Seems to me that the best way to address this would be to add an IP family argument to getAddressByName() to let it know what V4/V6 family to look for for a next-hop address. In the calls to getAddressByName(), this argument could be set based on the family of route.DestinationCIDR.
Issue is probably related.


Question: up{8}, down{oldReady: 4, newReady: 8, to: 1}, up{10}, <del> down{oldReady: 10, newReady: 1, to: 0}, </del> <ins> down{oldReady: 1, newReady: 10, to: 0}, </ins> }, output: `Created foo-v2 Scaling up foo-v2 from 0 to 10, scaling down foo-v1 from 10 to 0 (keep 9 pods available, don't exceed 12 pods)

I intentionally cause the rolling update to fail by specifying a non-existent image in the spec, so that it times out. rollback then fails with a timeout as well, despite the original replication controller having 1 healthy replica: This is on 1.1.2. I expect the rollback to succeed as the original rc (copper) has 1 healthy replica still.
With 2 replicas the rollback seems to work as expected.
I found the bug inside of the function of . It erroneously assumes reducing replicas on scale down will always drop the minimum available, when this isn't the case if you are rolling back from a failed deployment (where the scale down will be on the NotReady/failed pods). It needs to consider not ready replicas as well in each RC when deciding if it can scale down. This test case illustrates the problem:


Question: "github.com/GoogleCloudPlatform/kubernetes/pkg/kubelet" "github.com/GoogleCloudPlatform/kubernetes/pkg/types" "github.com/GoogleCloudPlatform/kubernetes/pkg/util" <ins> utilyaml "github.com/GoogleCloudPlatform/kubernetes/pkg/util/yaml" </ins> "github.com/ghodss/yaml" "github.com/golang/glog"

Following on from the Kubelet should accept YAML pod specifications. Currnelty it supports only JSON pod specifications.


Question: if err != nil { return fmt.Errorf("error when parsing kube-proxy configmap template: %v", err) } <del> proxyDaemonSetBytes, err = kubeadmutil.ParseTemplate(KubeProxyDaemonSet19, struct{ ImageRepository, Arch, Version, ImageOverride string }{ </del> <ins> proxyDaemonSetBytes, err = kubeadmutil.ParseTemplate(KubeProxyDaemonSet19, struct{ ImageRepository, Arch, Version, ImageOverride, MasterTaintKey string }{ </ins> ImageRepository: cfg.GetControlPlaneImageRepository(), Arch:            runtime.GOARCH, Version:         kubeadmutil.KubernetesVersionToImageTag(cfg.KubernetesVersion), ImageOverride:   cfg.UnifiedControlPlaneImage, <ins> MasterTaintKey:  kubeadmconstants.LabelNodeRoleMaster, </ins> }) if err != nil { return fmt.Errorf("error when parsing kube-proxy daemonset template: %v", err)

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: After joining a new ppc64le node to the k8s cluster, the kube-proxy DaemonSet fails on the new node. What you expected to happen: It was expected that the kube-proxy DS would "just work" as it does when adding an amd64 node to the cluster. How to reproduce it (as minimally and precisely as possible): Set up an amd64 based k8s cluster, then join a ppc64le node (or any non-amd64 arch) to the cluster. See the pod fail via 'kubectl -n kube-system get pods' because it's pulling the amd64 kube-proxy docker image instead of the ppc64le image. Anything else we need to know?: The yaml for the DaemonSet specifies the image: <versionIt would be preferable to have a manifest so that the image in the DS could be specified as <versionand could "just work" on any platform. See: Environment: Kubernetes version (use ): v1.10.3 Cloud provider or hardware configuration: self - cluster installed via kubeadm OS (e.g. from /etc/os-release): NAME="Ubuntu" VERSION="16.04.3 LTS (Xenial Xerus)" ID=ubuntu IDLIKE=debian PRETTYNAME="Ubuntu 16.04.3 LTS" VERSIONID="16.04" HOMEURL="" SUPPORTURL="" BUGREPORTURL="" VERSIONCODENAME=xenial UBUNTU_CODENAME=xenial Kernel (e.g. ): N/A Install tools: N/A Others: N/A
if you are running multi-architecture nodes, your daemonsets should specify a node selector to limit their arch-specific image to run on matching nodes
/close
I am well aware that a node selector can be used. That is what we are doing now. But it is a hack. The default should "just work". Adding a new non-amd64 node should not result in kube-proxy immediately failing on it because the default daemon set is amd64 arch specific. If kube-proxy is meant to "just work" clusterwide, a node selector should not be needed.

Please use the manifest-tool so that the default "just works". Please see the weave-net plugin. They are using the manifest-tool and it "just works" on amd64 and ppc64le. Their daemon set does not require/use a nodeSelector.
The current kube-proxy daemon set implementation is not properly architected, IMHO. Please re-open this
Reiterating the mentions to trigger a notification: <detailsIn response to : The current kube-proxy daemon set implementation is not properly architected, IMHO. Please re-open this </details
ah, this was a comment about the kube-proxy image, not the daemonset functionality /reopen
the kube-proxy addon daemonset is probably the multi-arch kube-proxy image to support that daemonset might be
Looks like it's coming from here: /assign /sig cluster-lifecycle Lucas, Do we want to avoid the references?
A dirty workaround for this is to pull and rename to . The kube-proxy is running as daemonset, created during . That means the kube-proxy image arch keeps consistent with your master node. Currently we don't have a multi-arch kube-proxy image yet. We have to explicitly specify the right name, like .
Hi .. Yes, we just created a separate daemonset with a ppc64le node slector that refers to the ppc64le image directly. But it's a bit of a hack. That is why I opened this ticket looking for a multi-arch image. Thanks!
I don't know that mixed-ISA clusters are supported. cc
The requirement is: only use "" in DaemonSet yaml file, and the k8s will download the correct image according to the ARCH info. It's more about image registry's feature; and it seems some registries had supported that: .
Yeah, has already supported multi-arch. But actually all the images now are architecture specific, like and . We've not built any multi-arch images right now.
for the test images we have the ball rolling here - - so if you can do the same (or similar) for other images, that would be great!
I have pending PR in releases project for creating manifest for those -
The manifest images created in kubernetes/release should work for future released versions, but will they be created during CI testing?

if you see one of the jobs that pushes images say you can grep for that's basically from the same files that got changed in so i fully expect the images pushed to CI will reflect the changes made in 516.
please confirm that i am reading this right ^ :)
I was more concerned with ci jobs that do not use cross builds, such as https://k8s-
please open an issue about pushing manifests in CI as well


Question: e2eoutput.TestContainerOutput(ctx, f, "check pod SecurityContext username", pod, 1, []string{"ContainerAdministrator"}) }) <del> ginkgo.It("should ignore SELinux Specific SecurityContext if set", func(ctx context.Context) { </del> <ins> ginkgo.It("should ignore Linux Specific SecurityContext if set", func(ctx context.Context) { </ins> ginkgo.By("Creating a pod with SELinux options") // It is sufficient to show that the pod comes up here. Since we're stripping the SELinux and other linux // security contexts in apiserver and not updating the pod object in the apiserver, we cannot validate the

After merged we are seeing the new test in fail since it was turned on be default now (it passed in the PR linked because it didn't have the alpha feature enabled). <br class="Apple-interchange-newline"<!--EndFragment--</body /sig windows /sig node /cc
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
ptal


Question: - Example: [TestNamespaceAuthorization](../../test/integration/auth_test.go) * Integration tests must run in parallel - Each test should create its own master, httpserver and config. <del> - Example: [TestPodUpdateActiveDeadlineSeconds](../../test/integration/pods.go) </del> <ins> - Example: [TestPodUpdateActiveDeadlineSeconds](../../test/integration/pods_test.go) </ins> * See [coding conventions](coding-conventions.md). ### Install etcd dependency

hack/test- passes even with an explicit "t.Fatalf()" in the (see ). Appreciate any insight.
The script essentially runs . Check the build tags at the top of the file. It's not even compiled in.
The script is using the 'no-docker' tag, not 'no-etcd', so I expect the tests to be compiled.
run with those build tags and see if that file in in there, just to be sure.
doesn't have "_test" in the filename.. I'll send a fix.
Thanks the command is handy.


Question: f() return err } <ins> func generateDriverCleanupFunc( f *framework.Framework, driverName, testns, driverns string, driverCleanup, cancelLogging func()) func() { cleanupHandle := new(framework.CleanupActionHandle) // Cleanup CSI driver and namespaces. This function needs to be idempotent and can be // concurrently called from defer (or AfterEach) and AfterSuite action hooks. cleanupFunc := func() { ginkgo.By(fmt.Sprintf("deleting the test namespace: %s", testns)) // Delete the primary namespace but it's okay to fail here because this namespace will // also be deleted by framework.Aftereach hook tryFunc(func() { f.DeleteNamespace(testns) }) ginkgo.By(fmt.Sprintf("uninstalling csi %s driver", driverName)) tryFunc(driverCleanup) tryFunc(cancelLogging) ginkgo.By(fmt.Sprintf("deleting the driver namespace: %s", driverns)) tryFunc(func() { f.DeleteNamespace(driverns) }) // cleanup function has already ran and hence we don't need to run it again. // We do this as very last action because in-case defer(or AfterEach) races // with AfterSuite and test routine gets killed then this block still // runs in AfterSuite framework.RemoveCleanupAction(*cleanupHandle) } *cleanupHandle = framework.AddCleanupAction(cleanupFunc) return cleanupFunc } </ins>

Tracking bug for /sig storage /important-longterm /assign
/sig storage /triage accepted
/priority important-longterm


Question: V=3 kube::log::status "Starting rsyncd container" kube::build::run_build_command_ex  "${KUBE_RSYNC_CONTAINER_NAME}" -p 127.0.0.1:"${KUBE_RSYNC_PORT}":"${KUBE_CONTAINER_RSYNC_PORT}" -d  <del> -e ALLOW_HOST="$(${IPTOOL} | grep -Eo 'inet (addr:)?([0-9]*.){3}[0-9]*' | grep -Eo '([0-9]*.){3}[0-9]*' | grep -v '127.0.0.1')"  </del> <ins> -e ALLOW_HOST="$(${IPTOOL} | grep -Eo 'inet (addr:)?([0-9]*.){3}[0-9]*' | grep -Eo '([0-9]*.){3}[0-9]*' | grep -v '127.0.0.1' | tr 'n' ' ')"  </ins> -- /rsyncd.sh >/dev/null local mapped_port

I'm attempting to follow on Fedora 32 with (a package which aliases to ) and it currently fails like this:
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Building Kubernetes requires docker docker, not podman. We use buildkit.
Though also building kubectl doesn't require or docker, only the images, or kubelet if not building on Linux (need cgo / Linux toolchain which we have in docker).
please open a separate issue if you think the docs can be improved. /close
Closing this issue. <detailsIn response to : please open a separate issue if you think the docs can be improved. /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: s := NewYAMLToJSONDecoder(bytes.NewReader([]byte(`--- stuff: 1 <del> --- </del> <ins> --- </ins> `))) obj := generic{} if err := s.Decode(&obj); err != nil {

Kubernetes version (use ): Environment: Cloud provider or hardware configuration: Any OS (e.g. from /etc/os-release): Any Kernel (e.g. ): Any Install tools: Any What happened: If I use a template with the seperator, and if the line with the separator has trailing whitespace characters, then kubectl considers that to be the EOF instead of reading further. e.g. template What you expected to happen: It should not stop reading if there are trailing whitespace characters, it should just trim the separator line and ignore the whitespaces after the . This is how it worked until v1.3.0 kubectl version. How to reproduce it (as minimally and precisely as possible): Just use the template from above and you'll notice that it won't create the Service, but it'll create the Replication controller. Anything else do we need to know: This is on kubectl version 1.4.0
kubectl regression. cc
This bug is general (can be found on get, apply, delete, ...). May be somewhere in resource builder.
Might be , Janet Kuo wrote:
Confirmed - is the cause. would you take a look?
We may need to cherrypick the fix
Yaml rules are separator must be at beginning of line, three dashes, and new line. Did we change how strict we were in that PR? On Sep 27, 2016, at 7:46 PM, Janet Kuo wrote: We may need to cherrypick the fix — You are receiving this because you are on a team that was mentioned. Reply to this email directly, view it on GitHub , or mute the thread .
Yes, and the test was changed too (originally tests allow with trailing spaces)
According to spec (if my spec-alese is correct) newline + three dashes is the only requirement to terminate a document. Whitespace or prefixes are allowed afterwards, but for it to be valid YAML those must be valid at the beginning of a doc as well (as far as I can tell). So we should split and remove on any line starting with a newline and three dashes - but not consume the remaining whitespace (I believe) , Janet Kuo wrote:
We should probably exit non-0 and have an error message if we see something in the file that we don't expect. E.g.

is neither a separator nor parseable yaml.
We should add a test that uses a YAML document prefix rule and verify that it correctly passes / fails when YAML goes to parse. From reading the spec it appears that anything that is invalid when n--- is present is also invalid in a single document context, and vice versa. , Phillip Wittrock wrote:
I have rechecked the code, and I have located the root cause, since I am outing for a training these days, I will fix this right now after the training on Friday.
I am quite sorry about the regression. :(
should we allow trailing whitespace characters after , if so we can trim the whitespaces when we read line by line?
is it possible to include this fix to one of the 1.4 releases? We'd really like our users to have of k8s alpha features (Scheduled jobs is one of them) that can be configured only using 1.4 client; w/o them potentially hitting this regression.
I will double check this regression, and open a cherry-pick PR.
I am thinking create a cherry-pick PR for this to include it in 1.4, wdyt?
I would like to see this in 1.4 since as you note it may break scripts. On Oct 19, 2016, at 8:19 PM, TonyAdo wrote: I am thinking create a cherry-pick PR for this to include it in 1.4, wdyt? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub , or mute the thread .
Do we have a test case now? :) On Oct 19, 2016, at 8:19 PM, TonyAdo wrote: I am thinking create a cherry-pick PR for this to include it in 1.4, wdyt? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub , or mute the thread .
we already test this, you can see