Question: t.Error("expected empty provisionings, got nil") } else { for i := 0; i < aLen; i++ { <del> if !reflect.DeepEqual(expectedProvisionings[i], provisionedClaims[i]) { t.Errorf("provisioned claims doesn't match [A-expected, B-got]: %s", diff.ObjectDiff(expectedProvisionings[i], provisionedClaims[i])) </del> <ins> if diff := cmp.Diff(expectedProvisionings[i], provisionedClaims[i]); diff != "" { t.Errorf("provisioned claims doesn't match (-want, +got):n%s", diff) </ins> } } }

When scheduling pod, we encountered the following event , then the scheduler will backoff to reschedule again, but this add more waiting time. From the event message, we can see it is the PVC update conflict leads to the , so need to add retry logic on PVC updating.### Need to add retry logic on PVC updating, to avoid the backoff, to improve the scheduling efficiency. a PVC and then create Pod; scheduing; the PVC annotation or label before the Pod scheduled success; will encounter this issue. No response <details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage note that this is not a supported client/server skew: Server Version: version.Info{Major:"1", Minor:"18+", GitVersion:"v1.18.4-tke.23", GitCommit:"", GitTreeState:"clean", BuildDate:"2022-03-15T09:25:31Z", GoVersion:"go1.15.10", Compiler:"gc", Platform:"linux/amd64"}
/sig scheduling
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue with Mark this issue as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close not-planned [1]:
Closing this issue, marking it as "Not Planned". <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue with Mark this issue as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: // Fix liveness probe By("Rewriting the file") <del> out, err = framework.RunKubectl("exec", fmt.Sprintf("--namespace=%s", pod.Namespace), pod.Name, "--container", pod.Spec.Containers[0].Name, "--", "/bin/sh", "-c", fmt.Sprintf("echo test-after > %v", volumeMount)) framework.Logf("Pod exec output: %v", out) </del> <ins> stdout, _, err = f.ExecShellInPodWithFullOutput(pod.Name, fmt.Sprintf("echo test-after > %v", volumeMount)) framework.Logf("Pod exec output: %v", stdout) </ins> Expect(err).ToNot(HaveOccurred(), "while rewriting the probe file") // Wait for container restarts to stabilize

Which jobs are failing: Tests which exec into containers and alter lifecycles Which test(s) are failing: All new tests introduced for api change which exec into containers New alpha subpathExpr tests [Feature:VolumeSubpathEnvExpansion][NodeAlphaFeature:VolumeSubpathEnvExpansion] Since when has it been failing: 27/February/2019 Testgrid link: https://k8s- Reason for failure: is only set in e2e tests and the binary is not available in node e2e tests Need to restructure and call the API rather than through the framework convenience function Anything else we need to know:
/sig node /sig storage /assign /cc


Question: if e, a := (expectedEmptyDirUsage.Value() + getExpectedBlockSize(filepath.Join(tmpDir, "f1"))), actual.Used.Value(); e != a { t.Errorf("Unexpected Used for directory with file.  Expected %v, got %d.", e, a) } <ins> // create a hardlink and expect inodes count to stay the same previousInodes := actual.InodesUsed.Value() err = os.Link(filepath.Join(tmpDir, "f1"), filepath.Join(tmpDir, "f2")) if err != nil { t.Errorf("Unexpected error when creating hard link %v", err) } actual, err = metrics.GetMetrics() if err != nil { t.Errorf("Unexpected error when calling GetMetrics %v", err) } if e, a := previousInodes, actual.InodesUsed.Value(); e != a { t.Errorf("Unexpected Used for directory with file.  Expected %v, got %d.", e, a) } </ins> } // TestMetricsDuRequireInit tests that if MetricsDu is not initialized with a path, GetMetrics

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: in calculates number of inodes wrong when there are hard links. What you expected to happen: inodes with hardlinks should only be counted once. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: OS (e.g: ):Kernel (e.g. ):Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle rotten
Issue is still not solved. There is an open PR waiting for review. /remove-lifecycle rotten
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale
still waiting for to be merged

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: kubelet fails to run pods on Alpine Linux with busybox find. What you expected to happen: I expected the call to exit with success, not failure. How to reproduce it (as minimally and precisely as possible): Install and run kublet on Alpine Linux without installing GNU findutils, so busybox implementation of is used. Anything else we need to know?: failed because the argument is not implemented in busybox . The option is not specified in so it can not be expected to be supported in all implementations. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: QEMU/KVM virtual machine OS (e.g: ):Kernel (e.g. ):Install tools: custom Network plugin and version (if this is a network-related bug): not network-related bug Others: I think it would be better to use instead of Related to
/sig Storage
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
This is solved by which is waiting for a review. /remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle rotten
Still waiting for the PR to be merged. /remove-lifecycle rotten
/remove-lifecycle rotten

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Running test suite on a cluster with workers on Alpine Linux, which uses busybox instead of GNU coreutils fails. The log shows that it fails to execute the command. What you expected to happen: I expected the call to exit with success. How to reproduce it (as minimally and precisely as possible): Install and run kublet on Alpine Linux without installing . Anything else we need to know?: It failed due to the option which is not implemented in busybox. The option is not specified in so I don't think it is reasonable to expect it to implemented in busybox. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: QEMU/KVM virtual machine OS (e.g: ):Kernel (e.g. ):Install tools: custom Network plugin and version (if this is a network-related bug): not network-related Others:
/sig Storage
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale fixes this
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle rotten
/remove-lifecycle rotten
Still waiting for PR to be merged


Question: for _, existingPod := range existingPods { existingPodNode, err := c.info.GetNodeInfo(existingPod.Spec.NodeName) if err != nil { <del> if apierrors.IsNotFound(err) { klog.Errorf("Pod %s has NodeName %q but node is not found", podName(existingPod), existingPod.Spec.NodeName) continue } return nil, err </del> <ins> klog.Errorf("Pod %s has NodeName %q but node is not found", podName(existingPod), existingPod.Spec.NodeName) continue </ins> } existingPodTopologyMaps, err := getMatchingAntiAffinityTopologyPairsOfPod(pod, existingPod, existingPodNode) if err != nil {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: This switches NodeLister to scheduler cache for . However, the new implementation in the cache never returns IsNotFound error any more. Rather, it returns a normal error: However, we rely on the error type for logging, for example: What you expected to happen: The cache should return IsNotFound error if applicable. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: /sig scheduling cc
/help
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/good-first-issue
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hi. I'd like to work on this for #Hacktoberfest, and for my very first open-source contribution.
Hello, I would love to contribute too in case. I'm already looking at the code in fact :D
Thanks all for the interest, will assign to since he expressed interest first. /assign
I was browsing around the rest of the implementation of the cache and noticed that none of the other accessors return . Also, I couldn't determine if ever returned anything other than an . I'm new here, so I may be doing it wrong. Are you sure the behavior changed, and are you sure returning is expected? Should the other accessors be changed, too?
Let's look at previous release. The object is initialized in [] where it creates a .

Now let's take a look at implementation: There you can see that it calls NodeLister Get() method, which can return a IsNotFound error.
Hello, is this issue being solved ? If not, i am interested in solving this issue.
Thanks but I am working on this currently.
, if you haven't started your work yet, let me know if I can raise a PR for this issue. I have the change ready.
I started, but haven’t been able to finish creating the PR. Go ahead and submit yours. Thanks. Sent from my iPhone


Question: "cluster-signing-key-file":         filepath.Join(cfg.CertificatesDir, kubeadmconstants.CAKeyName), "use-service-account-credentials":  "true", "controllers":                      "*,bootstrapsigner,tokencleaner", <del> "authentication-kubeconfig":        filepath.Join(kubeadmconstants.KubernetesDir, kubeadmconstants.ControllerManagerKubeConfigFileName), "authorization-kubeconfig":         filepath.Join(kubeadmconstants.KubernetesDir, kubeadmconstants.ControllerManagerKubeConfigFileName), "client-ca-file":                   filepath.Join(cfg.CertificatesDir, kubeadmconstants.CACertName), "requestheader-client-ca-file":     filepath.Join(cfg.CertificatesDir, kubeadmconstants.FrontProxyCACertName), </del> <ins> } //add the extra arguments for v1.12+ if k8sVersion.Major() >= 1 && k8sVersion.Minor() >= 12 { defaultArguments["authentication-kubeconfig"] = filepath.Join(kubeadmconstants.KubernetesDir, kubeadmconstants.ControllerManagerKubeConfigFileName) defaultArguments["authorization-kubeconfig"] = filepath.Join(kubeadmconstants.KubernetesDir, kubeadmconstants.ControllerManagerKubeConfigFileName) defaultArguments["client-ca-file"] = filepath.Join(cfg.CertificatesDir, kubeadmconstants.CACertName) defaultArguments["requestheader-client-ca-file"] = filepath.Join(cfg.CertificatesDir, kubeadmconstants.FrontProxyCACertName) </ins> } // If using external CA, pass empty string to controller manager instead of ca.key/ca.crt path,

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: When I use kubeadm to create an old version k8s cluster, the controller manager pod can not work well. The command like this : "kubeadm init --kubernetes-version=v1.11.2" The error of kube controller manager pod like this : What you expected to happen: The controller manager pod can work well. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): v1.12.0 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): CentOS 7 Kernel (e.g. ): Install tools: kubeadm Others:
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-sig contributor-experience /area kubeadm
Please, use when filing kubeadm issues. /sig cluster-lifecycle
OK, thank you for reminding me.


Question: function usage() { echo "!!! EXPERIMENTAL !!!" <ins> echo "!!! This upgrade script is not meant to be run in production !!!" </ins> echo "" echo "${0} [-M | -N | -P] [-o] (-l | <version number or publication>)" echo "  Upgrades master and nodes by default"

Which jobs are failing: Multiple jobs on . Which test(s) are failing: on the following jobs: Since when has it been failing: Since 5/17. Testgrid links: See above. Reason for failure: All the failures mentioned above seem to have to have the same root cause, namely failure while waiting for CoreDNS to be ready. Example: /sig testing /priority critical-urgent /kind failing test /milestone v1.15 /cc
Some of this (possible all of it) is due to , which is dependent on being merged.
/sig cluster-lifecycle
Not the gce jobs thoght. Only the kinder job. So the gce jobs are failing with a problem outside of kubeadm/kinder but possibly still related to the corefile plugin change
/assign (At kubecon this week)
The gce failures appear to show the same behavior, but with kube-up instead of kubeadm. I.e. it appears that kube-up may also be retaining old configs when upgrading/downgrading. If thats true, then we may need to integrate the migration tools into kube-up as well.
I have opened which aims to fix all the failing tests except , which will be fixed via
Great job so far and Would it be possible to ask for the open PRs to be merged and/or a solution found soon? We have these tests failing for quite some time and code freeze coming up in the next days. Thank you in advance for your efforts!
thanks for the heads up. let me know if you don't have the time for the refactor that i requested and i can try to take over there. the alternative is to rollback the coredns version which i assume is not ideal, due to a variety of fixes in the new version.
is there any more work planned for this issue? I see that some of the kinder jobs are still failing after merged:
merging and will resolve and close this issue.
Thank you for the update !
updates: kinder tests were fixed via
/unassign i was here for: it's a bit flaky (which is due to the tooling), but green.
Since and we're moved to 1.16, and we're waiting for to merge, can we move this issue to 1.16?
? is that the right PR?

Good catch, I meant
The GCE test failures were fixed by and /close
Closing this issue. <detailsIn response to : The GCE test failures were fixed by and /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: UID:       types.UID(fmt.Sprintf("test-puid%v", i)), }, Spec: v1.PodSpec{ <del> NodeName: fmt.Sprintf("test-node%v", i), </del> <ins> NodeName: fmt.Sprintf("test-node%v", i%10), </ins> }, } pods = append(pods, pod)

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: If we delete node first, then remove pods on this node, the scheduler will crash in function What you expected to happen: Not crashed How to reproduce it (as minimally and precisely as possible): some pods on node A delete node A delete pods on node A Anything else we need to know?: Environment: Kubernetes version (use ): 1.16-1.18, even master Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig scheduling
can you show the core dump please of this crash?
/assign
I patched the unit test shown in into the , and branches. None crashed. I'll await for the crash dump.
Here is the fresh dump.
and which version was this?
This is master branch. And I've tested it from 1.16-1.18, it can be reproduced easily.
Thanks a lot. I was able to finally reproduce in unit tests. The scenario is: a node with at least 2 pods. node. pod1. In unit test, this is first an update to set the termination timestamp. No issues up to this point. pod2. Again, first an update to set the termination timestamp. Here it crashes. I'm working on a fix.
It looks like this is not an issue in 1.17 and below, because we preserved a placeholder node where the operations occur. do you have stacktraces for 1.17 and 1.16?
Yes, it seems only be found at 1.18. :smile:


Question: f := framework.NewDefaultFramework("multi-az") var zoneCount int var err error <del> image := framework.ServeHostnameImage </del> ginkgo.BeforeEach(func() { e2eskipper.SkipUnlessProviderIs("gce", "gke", "aws") if zoneCount <= 0 {

It's not being used. Alternatively, pass the right image name to start with. /assign /sig scheduling /sig testing
/kind cleanup


Question: <ins> // +build !linux /* Copyright 2017 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package mount import ( "errors" ) type execMounter struct{} // ExecMounter is a mounter that uses provided Exec interface to mount and // unmount a filesystem. For all other calls it uses a wrapped mounter. func NewExecMounter(exec Exec, wrapped Interface) Interface { return &execMounter{} } func (mounter *execMounter) Mount(source string, target string, fstype string, options []string) error { return nil } func (mounter *execMounter) Unmount(target string) error { return nil } func (mounter *execMounter) List() ([]MountPoint, error) { return []MountPoint{}, nil } func (mounter *execMounter) IsMountPointMatch(mp MountPoint, dir string) bool { return (mp.Path == dir) } func (mounter *execMounter) IsNotMountPoint(dir string) (bool, error) { return IsNotMountPoint(mounter, dir) } func (mounter *execMounter) IsLikelyNotMountPoint(file string) (bool, error) { return true, nil } func (mounter *execMounter) GetDeviceNameFromMount(mountPath, pluginDir string) (string, error) { return "", nil } func (mounter *execMounter) DeviceOpened(pathname string) (bool, error) { return false, nil } func (mounter *execMounter) PathIsDevice(pathname string) (bool, error) { return true, nil } func (mounter *execMounter) MakeRShared(path string) error { return nil } func (mounter *execMounter) GetFileType(pathname string) (FileType, error) { return FileType("fake"), errors.New("not implemented") } func (mounter *execMounter) MakeDir(pathname string) error { return nil } func (mounter *execMounter) MakeFile(pathname string) error { return nil } func (mounter *execMounter) ExistsPath(pathname string) bool { return true } </ins>

/kind bug What happened: pull-kubernetes-cross complains: https://k8s-
/sig storage
For reference, root cause


Question: framework.ExpectNoError(err, "inconsistent device assignment after pod restart") }) <ins> ginkgo.It("will not attempt to admit the succeeded pod after the kubelet restart and device plugin removed", func(ctx context.Context) { podRECMD := fmt.Sprintf("devs=$(ls /tmp/ | egrep '^Dev-[0-9]+$') && echo stub devices: $devs && sleep %s", sleepIntervalToCompletion) podSpec := makeBusyboxPod(SampleDeviceResourceName, podRECMD) podSpec.Spec.RestartPolicy = v1.RestartPolicyNever // Making sure the pod will not be garbage collected and will stay thru the kubelet restart after // it reached the terminated state. Using finalizers makes the test more reliable. podSpec.ObjectMeta.Finalizers = []string{testFinalizer} pod := e2epod.NewPodClient(f).CreateSync(ctx, podSpec) deviceIDRE := "stub devices: (Dev-[0-9]+)" devID1, err := parseLog(ctx, f, pod.Name, pod.Name, deviceIDRE) framework.ExpectNoError(err, "getting logs for pod %q", pod.Name) gomega.Expect(devID1).To(gomega.Not(gomega.Equal("")), "pod requested a device but started successfully without") pod, err = e2epod.NewPodClient(f).Get(ctx, pod.Name, metav1.GetOptions{}) framework.ExpectNoError(err) ginkgo.By("Wait for node to be ready") gomega.Expect(e2enode.WaitForAllNodesSchedulable(ctx, f.ClientSet, 5*time.Minute)).To(gomega.Succeed()) ginkgo.By("Waiting for pod to succeed") gomega.Expect(e2epod.WaitForPodSuccessInNamespace(ctx, f.ClientSet, pod.Name, pod.Namespace)).To(gomega.Succeed()) ginkgo.By("Deleting the device plugin") e2epod.NewPodClient(f).DeleteSync(ctx, devicePluginPod.Name, metav1.DeleteOptions{}, time.Minute) waitForContainerRemoval(ctx, devicePluginPod.Spec.Containers[0].Name, devicePluginPod.Name, devicePluginPod.Namespace) gomega.Eventually(getNodeResourceValues, devicePluginGracefulTimeout, f.Timeouts.Poll).WithContext(ctx).WithArguments(SampleDeviceResourceName).Should(gomega.Equal(ResourceValue{Allocatable: 0, Capacity: int(expectedSampleDevsAmount)})) ginkgo.By("Restarting Kubelet") restartKubelet(true) ginkgo.By("Wait for node to be ready again") gomega.Expect(e2enode.WaitForAllNodesSchedulable(ctx, f.ClientSet, 5*time.Minute)).To(gomega.Succeed()) ginkgo.By("Pod should still be in Succeed state") // This ensures that the pod was admitted successfully. // In the past we had and issue when kubelet will attempt to re-admit the terminated pod and will change it's phase to Failed. // There are no indication that the pod was re-admitted so we just wait for a minute after the node became ready. gomega.Consistently(func() v1.PodPhase { pod, err = f.ClientSet.CoreV1().Pods(f.Namespace.Name).Get(ctx, pod.Name, metav1.GetOptions{}) return pod.Status.Phase }, 1*time.Minute, f.Timeouts.Poll).Should(gomega.Equal(v1.PodSucceeded)) ginkgo.By("Removing the finalizer from the pod so it can be deleted now") e2epod.NewPodClient(f).RemoveFinalizer(context.TODO(), podSpec.Name, testFinalizer) }) </ins> // simulate device plugin re-registration, *but not* container and kubelet restart. // After the device plugin has re-registered, the list healthy devices is repopulated based on the devices discovered. // Once Pod2 is running we determine the device that was allocated it. As long as the device allocation succeeds the

Pods that already succeeded and waiting for finalizer are being readmitted, fail to be admitted and moving from Succeeded Phase to Failed (admission failed) phase. The correct behavior would be to not try to re-admit the succeeded (or failed) Pod. Specific case: Node is exposing Devices using device plugin. Device plugin is implemented as a Pod Job's pod that is using the Device run to completion and waiting for Job Status finalizer kubelet is being restarted DevicePlugin pod is reconnecting to the kubelet. While it is reconnecting, kubelet don't have availability of devices kubelet re-admits the Pod, even though it is terminated already kubelet marks the pod as "failed to be admitted" But the same can happen with the variations of GC speed and other admission handlers. /sig node /priority important-soon /cc There is also a wider issue: , that concentrates on running pods. This issue is about terminated Pods, which is less controversial. <detailsFor sure on master, may be on earlier versions. </details<detailsN/A </details<detailsN/A </details<detailsN/A </details<detailsN/A </details<detailsN/A </details
/triage accepted I think we have at least another issue mentioning the same behavior but observed for jobs, I'll look it up and crosslink later


Question: "admissionregistration.k8s.io/v1/mutatingwebhookconfigurations":    "Sqi0GUgDaX0=", "admissionregistration.k8s.io/v1/validatingwebhookconfigurations":  "B0wHjQmsGNk=", "events.k8s.io/v1/events":                                          "r2yiGXH7wu8=", <del> "flowcontrol.apiserver.k8s.io/v1beta3/flowschemas":                 "G+8IkrqFuJw=", "flowcontrol.apiserver.k8s.io/v1beta3/prioritylevelconfigurations": "wltM4WMeeXs=", "flowcontrol.apiserver.k8s.io/v1beta2/flowschemas":                 "G+8IkrqFuJw=", "flowcontrol.apiserver.k8s.io/v1beta2/prioritylevelconfigurations": "wltM4WMeeXs=", </del> <ins> "flowcontrol.apiserver.k8s.io/v1beta3/flowschemas":                 "9NnFrw7ZEmA=", "flowcontrol.apiserver.k8s.io/v1beta3/prioritylevelconfigurations": "+CwSrEWTDhc=", "flowcontrol.apiserver.k8s.io/v1beta2/flowschemas":                 "9NnFrw7ZEmA=", "flowcontrol.apiserver.k8s.io/v1beta2/prioritylevelconfigurations": "+CwSrEWTDhc=", </ins> }

adds new version for in 1.26. For 1.26, we are setting the storage version to because vN-1 understands that level. In 1.27, we should change the storage version from to .
/sig api-machinery
/cc
/cc
/triage accepted
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:


Question: t.Fatal(err) } <del> if e, a := "/apis/mygroup.example.com/v1beta1/foo", createdCurletInstance.GetSelfLink(); e != a { </del> <ins> if e, a := "/apis/mygroup.example.com/v1beta1/curlets/foo", createdCurletInstance.GetSelfLink(); e != a { </ins> t.Errorf("expected %v, got %v", e, a) } }

Is this a BUG REPORT or FEATURE REQUEST?: What happened: I created a custom resource definition with -this is important - scope "Cluster". Then I created an element of that kind. Then I fetched the element and saw that the "metadata.selfLink" attribute had an incorrect value. The path did not contain the custom resource name. Trying to fetch the element using API results in 404 response. What you expected to happen: The "metadata.selfLink" of an instance of a custom resource should be correct. How to reproduce it (as minimally and precisely as possible): Prepare a custom resource definition with scope: Cluster (important! with scope Namespaced the selfLink is correct) in a file (slightly modified standard example from here: ) Create it: Now prepare a custom object in a file my- with this content: And create it: Now get the custom object list: The result is: As you can see the selfLink is . This is wrong, it should be . Using the selfLink as it is causes 404 when fetching the resource via API. Anything else we need to know?: This only happens when the custom resource definition has scope: Cluster. Environment: Kubernetes version (use ): v1.9.2 Cloud provider or hardware configuration: 2017 Macbook Pro OS (e.g. from /etc/os-release): macOS High Sierra 10.13.3 Kernel (e.g. ): Darwin Kernel Version 17.4.0 Install tools: minikube 0.24.1
There are no sig labels on this issue. Please add a sig label. <detailsA sig label can be by either: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. See the . The in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.

</details
Reiterating the mentions to trigger a notification: <detailsIn response to : </details
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
Created the PR:
/cc


Question: LabelSelectorParam(o.LabelSelector). FilenameParam(o.EnforceNamespace, o.FilenameOptions). ResourceTypeOrNameArgs(true, o.BuilderArgs...). <del> SingleResourceType(). </del> <ins> ContinueOnError(). </ins> Latest(). Do() <del> err := r.Err() if err != nil { return err } infos, err := r.Infos() if err != nil { return err } if len(infos) != 1 { return fmt.Errorf("rollout status is only supported on individual resources and resource collections - %d resources were found", len(infos)) } info := infos[0] mapping := info.ResourceMapping() statusViewer, err := o.StatusViewerFn(mapping) </del> <ins> err := r.Err() </ins> if err != nil { return err } <del> fieldSelector := fields.OneTermEqualSelector("metadata.name", info.Name).String() lw := &cache.ListWatch{ ListFunc: func(options metav1.ListOptions) (runtime.Object, error) { options.FieldSelector = fieldSelector return o.DynamicClient.Resource(info.Mapping.Resource).Namespace(info.Namespace).List(context.TODO(), options) }, WatchFunc: func(options metav1.ListOptions) (watch.Interface, error) { options.FieldSelector = fieldSelector return o.DynamicClient.Resource(info.Mapping.Resource).Namespace(info.Namespace).Watch(context.TODO(), options) }, } // if the rollout isn't done yet, keep watching deployment status ctx, cancel := watchtools.ContextWithOptionalTimeout(context.Background(), o.Timeout) intr := interrupt.New(nil, cancel) return intr.Run(func() error { _, err = watchtools.UntilWithSync(ctx, lw, &unstructured.Unstructured{}, nil, func(e watch.Event) (bool, error) { switch t := e.Type; t { case watch.Added, watch.Modified: status, done, err := statusViewer.Status(e.Object.(runtime.Unstructured), o.Revision) if err != nil { return false, err } fmt.Fprintf(o.Out, "%s", status) // Quit waiting if the rollout is done if done { return true, nil } shouldWatch := o.Watch if !shouldWatch { return true, nil </del> <ins> return r.Visit(func(info *resource.Info, err error) error { mapping := info.ResourceMapping() statusViewer, err := o.StatusViewerFn(mapping) if err != nil { return err } fieldSelector := fields.OneTermEqualSelector("metadata.name", info.Name).String() lw := &cache.ListWatch{ ListFunc: func(options metav1.ListOptions) (runtime.Object, error) { options.FieldSelector = fieldSelector return o.DynamicClient.Resource(info.Mapping.Resource).Namespace(info.Namespace).List(context.TODO(), options) }, WatchFunc: func(options metav1.ListOptions) (watch.Interface, error) { options.FieldSelector = fieldSelector return o.DynamicClient.Resource(info.Mapping.Resource).Namespace(info.Namespace).Watch(context.TODO(), options) }, } // if the rollout isn't done yet, keep watching deployment status ctx, cancel := watchtools.ContextWithOptionalTimeout(context.Background(), o.Timeout) intr := interrupt.New(nil, cancel) return intr.Run(func() error { _, err = watchtools.UntilWithSync(ctx, lw, &unstructured.Unstructured{}, nil, func(e watch.Event) (bool, error) { switch t := e.Type; t { case watch.Added, watch.Modified: status, done, err := statusViewer.Status(e.Object.(runtime.Unstructured), o.Revision) if err != nil { return false, err } fmt.Fprintf(o.Out, "%s", status) // Quit waiting if the rollout is done if done { return true, nil } shouldWatch := o.Watch if !shouldWatch { return true, nil } return false, nil case watch.Deleted: // We need to abort to avoid cases of recreation and not to silently watch the wrong (new) object return true, fmt.Errorf("object has been deleted") default: return true, fmt.Errorf("internal error: unexpected event %#v", e) </ins> } <del> return false, nil case watch.Deleted: // We need to abort to avoid cases of recreation and not to silently watch the wrong (new) object return true, fmt.Errorf("object has been deleted") default: return true, fmt.Errorf("internal error: unexpected event %#v", e) } </del> <ins> }) return err </ins> }) <del> return err </del> }) }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: fails with . What you expected to happen: The suggest that should work for multiple resources in a directory when running . should've been applied to all resources under following was omitted in and I wasn't able to find a discussion regarding this omission in other issues. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): 1.10.7 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
/sig cli
/sig apps
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale /remove-lifecycle rotten
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity.

Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen /remove-lifecycle rotten I agree with the assessment that this is a bug the , , and all work with spec files with multiple resources in them. does not. All of the commands would additionally benefit from only operating on supported resource types for rollout (Deployment, DaemonSet) from the spec file, but that's really a sperate enhancement. Client kubectl 1.16.6 Server EKS 1.14.9 How Status Behaves: How Pause Behaves: How Resume Behaves: How History Behaves:
You can't reopen an issue/PR unless you authored it or you are a collaborator.

<detailsIn response to : /reopen /remove-lifecycle rotten Server EKS 1.14.9 kubectl rollout status -f .ClusterServicecore- -n kube-system error: rollout status is only supported on individual resources and resource collections - 6 resources were found kubectl rollout pause -f .ClusterServicecore- -n kube-system paused error: serviceaccounts "coredns" pausing is not supported error: "system:coredns" pausing is not supported error: "system:coredns" pausing is not supported error: configmaps "coredns" pausing is not supported error: services "kube-dns" pausing is not supported kubectl rollout resume -f .dc-infra-k8s-infrastructureClusterServicecore- -n kube-system resumed error: serviceaccounts "coredns" resuming is not supported error: "system:coredns" resuming is not supported error: "system:coredns" resuming is not supported error: configmaps "coredns" resuming is not supported error: services "kube-dns" resuming is not supported Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Could you reopen this? If not I guess I'll file a new issue.
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
What is needed to move this forward?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or .

/lifecycle stale
/remove-lifecycle stale poke poke
There appears to be a PR that was closed without merging Strangely no changes are recorded in the PR
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale
Any updates on this?
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen /remove-lifecycle rotten
Reopened this issue. <detailsIn response to : /reopen /remove-lifecycle rotten Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: running++ } else if containerStatus.State.Termination != nil { stopped++ <ins> if containerStatus.State.Termination.ExitCode == 0 { succeeded++ } else { failed++ } } else if containerStatus.State.Waiting != nil { waiting++ </ins> } else { unknown++ }

I'm actually not sure if it's by design. But certainly replication controller doesn't behave like it expects this.
If restart policy is to restart always, they should be restarted, but in that case they shouldn't be reported as failed.
This is a known issue, and by design based on upper-level will observe the status reported from kubelet, but not check with RestartPolicy and generate PodCondition / PodStatus. When RestartPolicy is Always, even Kubelet reports all containers are at failed state, apiserver shouldn't mark Pod as failed state. This is why initially I want kubelet report Pod state / status / condiction.
As a quick fix, we could let apiserver populate the right state / status based on containerStatus and RestartPolicy for now. In a long term, I still think we should make kubelet smarter and atomic and self-contained component, not just a wrapper around docker.
Short term fix should be P1 then we can de-prio for a "realer" fix.
Yes, I am working on the short term fix, and should have it included in 0.6 cut.
are you still working on this?
Yes, I am working on this. Should finish it today.


Question: return fmt.Errorf("Expected Node Allocatable Cgroup Does not exist") } // TODO: Update cgroupManager to expose a Status interface to get current Cgroup Settings. <del> nodeList, err := f.ClientSet.CoreV1().Nodes().List(metav1.ListOptions{}) if err != nil { return err } if len(nodeList.Items) != 1 { return fmt.Errorf("Unexpected number of node objects for node e2e. Expects only one node: %+v", nodeList) } node := nodeList.Items[0] capacity := node.Status.Capacity allocatableCPU, allocatableMemory := getAllocatableLimits("200m", "200Mi", capacity) // Total Memory reservation is 200Mi excluding eviction thresholds. // Expect CPU shares on node allocatable cgroup to equal allocatable. if err := expectFileValToEqual(filepath.Join(subsystems.MountPoints["cpu"], "kubepods", "cpu.shares"), int64(cm.MilliCPUToShares(allocatableCPU.MilliValue())), 10); err != nil { return err } // Expect Memory limit on node allocatable cgroup to equal allocatable. if err := expectFileValToEqual(filepath.Join(subsystems.MountPoints["memory"], "kubepods", "memory.limit_in_bytes"), allocatableMemory.Value(), 0); err != nil { return err } </del> <ins> // The node may not have updated capacity and allocatable yet, so check that it happens eventually. Eventually(func() error { nodeList, err := f.ClientSet.CoreV1().Nodes().List(metav1.ListOptions{}) if err != nil { return err } if len(nodeList.Items) != 1 { return fmt.Errorf("Unexpected number of node objects for node e2e. Expects only one node: %+v", nodeList) } node := nodeList.Items[0] capacity := node.Status.Capacity allocatableCPU, allocatableMemory := getAllocatableLimits("200m", "200Mi", capacity) // Total Memory reservation is 200Mi excluding eviction thresholds. // Expect CPU shares on node allocatable cgroup to equal allocatable. if err := expectFileValToEqual(filepath.Join(subsystems.MountPoints["cpu"], "kubepods", "cpu.shares"), int64(cm.MilliCPUToShares(allocatableCPU.MilliValue())), 10); err != nil { return err } // Expect Memory limit on node allocatable cgroup to equal allocatable. if err := expectFileValToEqual(filepath.Join(subsystems.MountPoints["memory"], "kubepods", "memory.limit_in_bytes"), allocatableMemory.Value(), 0); err != nil { return err } </ins> <del> // Check that Allocatable reported to scheduler includes eviction thresholds. schedulerAllocatable := node.Status.Allocatable // Memory allocatable should take into account eviction thresholds. allocatableCPU, allocatableMemory = getAllocatableLimits("200m", "300Mi", capacity) // Expect allocatable to include all resources in capacity. if len(schedulerAllocatable) != len(capacity) { return fmt.Errorf("Expected all resources in capacity to be found in allocatable") } // CPU based evictions are not supported. if allocatableCPU.Cmp(schedulerAllocatable[v1.ResourceCPU]) != 0 { return fmt.Errorf("Unexpected cpu allocatable value exposed by the node. Expected: %v, got: %v, capacity: %v", allocatableCPU, schedulerAllocatable[v1.ResourceCPU], capacity[v1.ResourceCPU]) } if allocatableMemory.Cmp(schedulerAllocatable[v1.ResourceMemory]) != 0 { return fmt.Errorf("Unexpected cpu allocatable value exposed by the node. Expected: %v, got: %v, capacity: %v", allocatableCPU, schedulerAllocatable[v1.ResourceCPU], capacity[v1.ResourceMemory]) } </del> <ins> // Check that Allocatable reported to scheduler includes eviction thresholds. schedulerAllocatable := node.Status.Allocatable // Memory allocatable should take into account eviction thresholds. allocatableCPU, allocatableMemory = getAllocatableLimits("200m", "300Mi", capacity) // Expect allocatable to include all resources in capacity. if len(schedulerAllocatable) != len(capacity) { return fmt.Errorf("Expected all resources in capacity to be found in allocatable") } // CPU based evictions are not supported. if allocatableCPU.Cmp(schedulerAllocatable[v1.ResourceCPU]) != 0 { return fmt.Errorf("Unexpected cpu allocatable value exposed by the node. Expected: %v, got: %v, capacity: %v", allocatableCPU, schedulerAllocatable[v1.ResourceCPU], capacity[v1.ResourceCPU]) } if allocatableMemory.Cmp(schedulerAllocatable[v1.ResourceMemory]) != 0 { return fmt.Errorf("Unexpected memory allocatable value exposed by the node. Expected: %v, got: %v, capacity: %v", allocatableMemory, schedulerAllocatable[v1.ResourceMemory], capacity[v1.ResourceMemory]) } return nil }, time.Minute, 5*time.Second).Should(BeNil()) </ins> if !cgroupManager.Exists(cm.CgroupName(kubeReservedCgroup)) { return fmt.Errorf("Expected kube reserved cgroup Does not exist")

Node Allocatable setup test has been flaking recently. Example flake: https://k8s- /sig node cc
The kubelet logs look fine. After speaking with we are querying the node api object immediately after the kubelet configuration is updated. At this point, the kubelet may not have updated its node status, meaning we see the old node status rather than the new, correct one.


Question: // waitForEvents waits for adds, updates, and deletes to be populated with at least one event. func (is *informerSpy) waitForEvents(t *testing.T, wantEvents bool) { t.Helper() <ins> // wait for create/update/delete 3 events for 30 seconds waitTimeout := time.Second * 30 if !wantEvents { // wait just 15 seconds for no events waitTimeout = time.Second * 15 } </ins> <del> err := wait.PollImmediate(time.Second, time.Second*30, func() (bool, error) { </del> <ins> err := wait.PollImmediate(time.Second, waitTimeout, func() (bool, error) { </ins> is.mu.Lock() defer is.mu.Unlock() return len(is.adds) > 0 && len(is.updates) > 0 && len(is.deletes) > 0, nil

wanted sync true, got sync false wanted sync true, got sync false wanted sync true, got sync false informer never changed resource versions from "": timed out waiting for the condition /kind failing-test /kind flake as it is in /sig auth Open it as it is the top flake of pull-kubernetes-integration:
Commented on PR.
/triage accepted
I've seen this test fail 2 times today on my PR Going to hope that I close my eyes and and never see it if it happens again I'll reopen this...
!
Opened to track flakes once again


Question: "//pkg/api:go_default_library", "//pkg/api/install:go_default_library", "//pkg/capabilities:go_default_library", <ins> "//pkg/cloudprovider/providers/fake:go_default_library", </ins> "//pkg/kubelet/apis:go_default_library", "//pkg/kubelet/apis/kubeletconfig:go_default_library", "//pkg/kubelet/cadvisor/testing:go_default_library",

BUG REPORT: /kind bug What happened: After upgrading the kubernetes cluster from 1.6.4 with the VSphere Cloud Provider enabled to 1.6.5/1.6.6 or 1.6.7, the cluster nodes don't have anymore InternalIPs Several issues occur at this step. For example, it breaks heapster with such an error : What you expected to happen: The node should have an InternalIP. How to reproduce it (as minimally and precisely as possible): Deploy a 1 node kubernetes cluster in a VM, and enable the vsphere cloud provider. Anything else we need to know?: with kubernetes 1.6.4 we get : Downgrading to 1.6.4 fixes the issue. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: vsphere OS (e.g. from /etc/os-release):Kernel (e.g. ):Install tools: Others: It may be related to some changes around PR kubernetes /area platform/vsphere cloudprovider
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <bre.g., for API Machinery<br(2) specifying the label manually: <bre.g., for sig/scalability<br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
/sig cluster-ops /sig network /area platform/vsphere /area cloudprovider
Performing some more tests on a mini instance outside of the production cluster, I think I've identified the issue. The configuration required to start kubelet with the option (otherwise the wrong IP/interface is used to communicate between nodes). The issue is triggered when this option is set. The cloud provider looks OK. But the issue is located here : When the is not , the loop iterates on the node addresses and returns only the first one found. Since PR kubernetes, the list always returns an ExternalIP first. I think the loop in should iterate on all the addresses to rebuild the list, so that it will include the ExternalIP and the InternalIP. Can someone confirm the analysis ?
/sig node
ping
Modifying the code like this solved the issue : which returns : I guess it can be done in a cleaner way, but it was to quickly test the fix.

CC:
Thanks for filing this issue related to vSphere cloud provider. I agree with your analysis. It seems the current function assumes that the IP address specified by user can only be InternalIP. Your fix to is a better solution than work-around in cloud provider.
Hi, what are the next steps to discuss about the PR ? Are you able to make some tests ?


Question: lb.lock.RLock() defer lb.lock.RUnlock() state, exists := lb.services[svcPort] <del> // TODO: while nothing ever assigns nil to the map, *some* of the code using the map // checks for it.  The code should all follow the same convention. return exists && state != nil && len(state.endpoints) > 0 </del> <ins> if !exists || state == nil { return false } return len(state.endpoints) > 0 </ins> } // NextEndpoint returns a service endpoint.

As suggested in , some portions of the code contains the map checking for nil values in map, and others don't have. Even though we're never assigning nil values to this map, we should standardize the portions that does not have this check in module. Example: Should be like this: I can take care of this issue.
/sig network
There have been discussions of deprecating the user-space proxier. Maybe transfer it to . A question is if anybody still uses it. Seems like you do, so if possible please explain if you have a use-case where the user-space proxier is wanted/needed.
Hey Actually, I was casually reading the source code and I saw a over there regarding the nil checks. Regarding your question, we should check if anyone actually uses it though, or maybe discuss its usage or deprecation a little further.
Quick question though: I saw in the news that the user-space proxy porting was near completion, maybe it's worth talking to them in order to check upon the status.
I don't object to making it clearer or more consistent, if you want to spend time on it, but it's not a high priority. That said, I think a nil value in that map would be really bad (and should be impossible) and I don't know how we would raise a signal about it.
let us know if you want to try to fix it, otherwise we can just close this.
I already opened a PR with the clearer nil checks for this specific module in , so you can review it. I think whether this would have nil values into the service map we should discuss this apart and try to reproduce this case (if any would happen).
PR submitted, so let's review and discuss further there. /assign dcbw /triage accepted


Question: "k8s.io/kubernetes/pkg/api" "k8s.io/kubernetes/pkg/apis/apps" "k8s.io/kubernetes/pkg/client/cache" <ins> fake_internal "k8s.io/kubernetes/pkg/client/clientset_generated/internalclientset/fake" "k8s.io/kubernetes/pkg/client/clientset_generated/internalclientset/typed/apps/unversioned" "k8s.io/kubernetes/pkg/client/clientset_generated/internalclientset/typed/apps/unversioned/fake" </ins> "k8s.io/kubernetes/pkg/controller" "k8s.io/kubernetes/pkg/util/errors" )

Is this a request for help? Nope What keywords did you search in Kubernetes issues before filing this one? petset replicas current desired Is this a BUG REPORT or FEATURE REQUEST? BUG Kubernetes version Client Version: version.Info{Major:"", Minor:"", GitVersion:"v0.0.0-master+", GitCommit:"", GitTreeState:"not a git tree", BuildDate:"2016-09-01T10:49:36Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"", Minor:"", GitVersion:"v0.0.0-master+", GitCommit:"", GitTreeState:"not a git tree", BuildDate:"2016-09-01T10:49:36Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"} Environment: Cloud provider or hardware configuration: 24 cores, 64 GB RAM OS Arch Linux Kernel (e.g. ): Linux hostname 4.4.13-1-lts SMP Wed Jun 8 16:44:31 CEST 2016 x86_64 GNU/Linux Install tools: local-cluster- Others: What happened: While creating PetSet based on this tutorial: first PetSet pod was stuck in pending state due to lack of PersistentVolumes in cluster. When getting the petset info via I expected to see "CURRENT" set to 1, while the kubectl output was as follows: While clearly shows that only one pod is created: In json and yaml output, status.replicas value was also 2. What you expected to happen: I expected that CURRENT value shown by kubectl would be due to only one pod being created. How to reproduce it (as minimally and precisely as possible): Create via on empty cluster and run and . Anything else do we need to know: Nope
CURRENT is the number of pods that have been created, not that they are READY. See for example the latest api change for replica sets where we exactly for this reason. All controllers (petsets, daemonsets, deployments, etc.) should have ReadyReplicas cc:
See also
I guess it doesn't work anyway, since only one pod was created for this PetSet, but the CURRENT says that 2 were created. I will change bug description then.

Hmm, the idea is to sync every pet periodically, but only create them one at a time. The logic that counts status.replicas is wrong. This counter shouldn't be incremented in the case where we didn't update/sync or create a pet: Today we do increment it all pet creation is blocked on the first pet becoming unhealthy. Should be an easy fix if anyone wants to jump in ( fyi) otherwise I'll get to it once 1.4 is out.
I can try to fix it if that's OK. Thanks for the pointer to the source code!
Please do, co-assigned


Question: capPath := specPath.Child("capacity") for r, qty := range pv.Spec.Capacity { allErrs = append(allErrs, validateBasicResource(qty, capPath.Key(string(r)))...) <ins> allErrs = append(allErrs, ValidatePositiveQuantityValue(qty, capPath.Key(string(r)))...) </ins> } if len(string(pv.Spec.PersistentVolumeReclaimPolicy)) > 0 { if !supportedReclaimPolicy.Has(string(pv.Spec.PersistentVolumeReclaimPolicy)) {

Is this a BUG REPORT or FEATURE REQUEST?: BUG REPORT /kind bug What happened: If someone requests a zero (0) capacity PVC and an administrator creates a zero (0) capacity PV, Kubernetes will happily bind them together. Unfortunately, any use of this PVC/PV combination by a pod will fail. It fails because the zero capacity PV cannot be formatted or mounted successfully. Note: The same issue happens for dynamically provisioned PVs. What you expected to happen: I expect that Kubernetes would not allow creation of invalid PVC and PV objects, aka zero (0) capacity PVs and PVCs. How to reproduce it (as minimally and precisely as possible): Create a pod that requests a zero capacity PVC, using dynamic PV provisioning is simpler than creating your own zero (0) capacity PV. The pod will fail to reach a running state, since the related PV fails to be formatted and mounted successfully. Here is the error that I received for a test I performed requesting a zero capacity PVC using ceph rbd: Anything else we need to know?: Here is a related item with a specific fix for external ceph rbd provisioner: Environment: Nothing specific to environment
/sig storage
/assign


Question: return e2epod.NewPodClient(f).Create(ctx, pod) } <del> func expectSoftRejection(status v1.PodStatus) { </del> <ins> func expectRejection(status v1.PodStatus) { </ins> args := []interface{}{"PodStatus: %+v", status} <del> gomega.Expect(status.Phase).To(gomega.Equal(v1.PodPending), args...) </del> <ins> gomega.Expect(status.Phase).To(gomega.Equal(v1.PodFailed), args...) </ins> gomega.Expect(status.Reason).To(gomega.Equal("AppArmor"), args...) gomega.Expect(status.Message).To(gomega.ContainSubstring("AppArmor"), args...) <del> gomega.Expect(status.ContainerStatuses[0].State.Waiting.Reason).To(gomega.Equal("Blocked"), args...) </del> } func isAppArmorEnabled() bool {

/kind failing-test <!-- If this is a flake, please add: /kind flake --/sig node
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Failures look like started when landed


Question: } } <ins> func TestReconcileHugePageResource(t *testing.T) { testKubelet := newTestKubelet(t, false /* controllerAttachDetachEnabled */) hugePageResourceName64Ki := v1.ResourceName("hugepages-64Ki") hugePageResourceName2Mi := v1.ResourceName("hugepages-2Mi") hugePageResourceName1Gi := v1.ResourceName("hugepages-1Gi") cases := []struct { name         string testKubelet  *TestKubelet initialNode  *v1.Node existingNode *v1.Node expectedNode *v1.Node needsUpdate  bool }{ { name:        "no update needed when all huge page resources are similar", testKubelet: testKubelet, needsUpdate: false, initialNode: &v1.Node{ Status: v1.NodeStatus{ Capacity: v1.ResourceList{ v1.ResourceCPU:              *resource.NewMilliQuantity(2000, resource.DecimalSI), v1.ResourceMemory:           *resource.NewQuantity(10e9, resource.BinarySI), v1.ResourceEphemeralStorage: *resource.NewQuantity(5000, resource.BinarySI), hugePageResourceName2Mi:     resource.MustParse("100Mi"), hugePageResourceName64Ki:    *resource.NewQuantity(0, resource.BinarySI), }, Allocatable: v1.ResourceList{ v1.ResourceCPU:              *resource.NewMilliQuantity(2000, resource.DecimalSI), v1.ResourceMemory:           *resource.NewQuantity(10e9, resource.BinarySI), v1.ResourceEphemeralStorage: *resource.NewQuantity(5000, resource.BinarySI), hugePageResourceName2Mi:     resource.MustParse("100Mi"), hugePageResourceName64Ki:    *resource.NewQuantity(0, resource.BinarySI), }, }, }, existingNode: &v1.Node{ Status: v1.NodeStatus{ Capacity: v1.ResourceList{ v1.ResourceCPU:              *resource.NewMilliQuantity(2000, resource.DecimalSI), v1.ResourceMemory:           *resource.NewQuantity(10e9, resource.BinarySI), v1.ResourceEphemeralStorage: *resource.NewQuantity(5000, resource.BinarySI), hugePageResourceName2Mi:     resource.MustParse("100Mi"), hugePageResourceName64Ki:    *resource.NewQuantity(0, resource.BinarySI), }, Allocatable: v1.ResourceList{ v1.ResourceCPU:              *resource.NewMilliQuantity(2000, resource.DecimalSI), v1.ResourceMemory:           *resource.NewQuantity(10e9, resource.BinarySI), v1.ResourceEphemeralStorage: *resource.NewQuantity(5000, resource.BinarySI), hugePageResourceName2Mi:     resource.MustParse("100Mi"), hugePageResourceName64Ki:    *resource.NewQuantity(0, resource.BinarySI), }, }, }, expectedNode: &v1.Node{ Status: v1.NodeStatus{ Capacity: v1.ResourceList{ v1.ResourceCPU:              *resource.NewMilliQuantity(2000, resource.DecimalSI), v1.ResourceMemory:           *resource.NewQuantity(10e9, resource.BinarySI), v1.ResourceEphemeralStorage: *resource.NewQuantity(5000, resource.BinarySI), hugePageResourceName2Mi:     resource.MustParse("100Mi"), hugePageResourceName64Ki:    *resource.NewQuantity(0, resource.BinarySI), }, Allocatable: v1.ResourceList{ v1.ResourceCPU:              *resource.NewMilliQuantity(2000, resource.DecimalSI), v1.ResourceMemory:           *resource.NewQuantity(10e9, resource.BinarySI), v1.ResourceEphemeralStorage: *resource.NewQuantity(5000, resource.BinarySI), hugePageResourceName2Mi:     resource.MustParse("100Mi"), hugePageResourceName64Ki:    *resource.NewQuantity(0, resource.BinarySI), }, }, }, }, { name:        "update needed when new huge page resources is supported", testKubelet: testKubelet, needsUpdate: true, initialNode: &v1.Node{ Status: v1.NodeStatus{ Capacity: v1.ResourceList{ v1.ResourceCPU:              *resource.NewMilliQuantity(2000, resource.DecimalSI), v1.ResourceMemory:           *resource.NewQuantity(10e9, resource.BinarySI), v1.ResourceEphemeralStorage: *resource.NewQuantity(5000, resource.BinarySI), hugePageResourceName2Mi:     *resource.NewQuantity(0, resource.BinarySI), hugePageResourceName1Gi:     resource.MustParse("2Gi"), }, Allocatable: v1.ResourceList{ v1.ResourceCPU:              *resource.NewMilliQuantity(2000, resource.DecimalSI), v1.ResourceMemory:           *resource.NewQuantity(10e9, resource.BinarySI), v1.ResourceEphemeralStorage: *resource.NewQuantity(5000, resource.BinarySI), hugePageResourceName2Mi:     *resource.NewQuantity(0, resource.BinarySI), hugePageResourceName1Gi:     resource.MustParse("2Gi"), }, }, }, existingNode: &v1.Node{ Status: v1.NodeStatus{ Capacity: v1.ResourceList{ v1.ResourceCPU:              *resource.NewMilliQuantity(2000, resource.DecimalSI), v1.ResourceMemory:           *resource.NewQuantity(10e9, resource.BinarySI), v1.ResourceEphemeralStorage: *resource.NewQuantity(5000, resource.BinarySI), hugePageResourceName2Mi:     resource.MustParse("100Mi"), }, Allocatable: v1.ResourceList{ v1.ResourceCPU:              *resource.NewMilliQuantity(2000, resource.DecimalSI), v1.ResourceMemory:           *resource.NewQuantity(10e9, resource.BinarySI), v1.ResourceEphemeralStorage: *resource.NewQuantity(5000, resource.BinarySI), hugePageResourceName2Mi:     resource.MustParse("100Mi"), }, }, }, expectedNode: &v1.Node{ Status: v1.NodeStatus{ Capacity: v1.ResourceList{ v1.ResourceCPU:              *resource.NewMilliQuantity(2000, resource.DecimalSI), v1.ResourceMemory:           *resource.NewQuantity(10e9, resource.BinarySI), v1.ResourceEphemeralStorage: *resource.NewQuantity(5000, resource.BinarySI), hugePageResourceName2Mi:     *resource.NewQuantity(0, resource.BinarySI), hugePageResourceName1Gi:     resource.MustParse("2Gi"), }, Allocatable: v1.ResourceList{ v1.ResourceCPU:              *resource.NewMilliQuantity(2000, resource.DecimalSI), v1.ResourceMemory:           *resource.NewQuantity(10e9, resource.BinarySI), v1.ResourceEphemeralStorage: *resource.NewQuantity(5000, resource.BinarySI), hugePageResourceName2Mi:     *resource.NewQuantity(0, resource.BinarySI), hugePageResourceName1Gi:     resource.MustParse("2Gi"), }, }, }, }, { name:        "update needed when huge page resource quantity has changed", testKubelet: testKubelet, needsUpdate: true, initialNode: &v1.Node{ Status: v1.NodeStatus{ Capacity: v1.ResourceList{ v1.ResourceCPU:              *resource.NewMilliQuantity(2000, resource.DecimalSI), v1.ResourceMemory:           *resource.NewQuantity(10e9, resource.BinarySI), v1.ResourceEphemeralStorage: *resource.NewQuantity(5000, resource.BinarySI), hugePageResourceName1Gi:     resource.MustParse("4Gi"), }, Allocatable: v1.ResourceList{ v1.ResourceCPU:              *resource.NewMilliQuantity(2000, resource.DecimalSI), v1.ResourceMemory:           *resource.NewQuantity(10e9, resource.BinarySI), v1.ResourceEphemeralStorage: *resource.NewQuantity(5000, resource.BinarySI), hugePageResourceName1Gi:     resource.MustParse("4Gi"), }, }, }, existingNode: &v1.Node{ Status: v1.NodeStatus{ Capacity: v1.ResourceList{ v1.ResourceCPU:              *resource.NewMilliQuantity(2000, resource.DecimalSI), v1.ResourceMemory:           *resource.NewQuantity(10e9, resource.BinarySI), v1.ResourceEphemeralStorage: *resource.NewQuantity(5000, resource.BinarySI), hugePageResourceName1Gi:     resource.MustParse("2Gi"), }, Allocatable: v1.ResourceList{ v1.ResourceCPU:              *resource.NewMilliQuantity(2000, resource.DecimalSI), v1.ResourceMemory:           *resource.NewQuantity(10e9, resource.BinarySI), v1.ResourceEphemeralStorage: *resource.NewQuantity(5000, resource.BinarySI), hugePageResourceName1Gi:     resource.MustParse("2Gi"), }, }, }, expectedNode: &v1.Node{ Status: v1.NodeStatus{ Capacity: v1.ResourceList{ v1.ResourceCPU:              *resource.NewMilliQuantity(2000, resource.DecimalSI), v1.ResourceMemory:           *resource.NewQuantity(10e9, resource.BinarySI), v1.ResourceEphemeralStorage: *resource.NewQuantity(5000, resource.BinarySI), hugePageResourceName1Gi:     resource.MustParse("4Gi"), }, Allocatable: v1.ResourceList{ v1.ResourceCPU:              *resource.NewMilliQuantity(2000, resource.DecimalSI), v1.ResourceMemory:           *resource.NewQuantity(10e9, resource.BinarySI), v1.ResourceEphemeralStorage: *resource.NewQuantity(5000, resource.BinarySI), hugePageResourceName1Gi:     resource.MustParse("4Gi"), }, }, }, }, { name:        "update needed when a huge page resources is no longer supported", testKubelet: testKubelet, needsUpdate: true, initialNode: &v1.Node{ Status: v1.NodeStatus{ Capacity: v1.ResourceList{ v1.ResourceCPU:              *resource.NewMilliQuantity(2000, resource.DecimalSI), v1.ResourceMemory:           *resource.NewQuantity(10e9, resource.BinarySI), v1.ResourceEphemeralStorage: *resource.NewQuantity(5000, resource.BinarySI), hugePageResourceName1Gi:     resource.MustParse("2Gi"), }, Allocatable: v1.ResourceList{ v1.ResourceCPU:              *resource.NewMilliQuantity(2000, resource.DecimalSI), v1.ResourceMemory:           *resource.NewQuantity(10e9, resource.BinarySI), v1.ResourceEphemeralStorage: *resource.NewQuantity(5000, resource.BinarySI), hugePageResourceName1Gi:     resource.MustParse("2Gi"), }, }, }, existingNode: &v1.Node{ Status: v1.NodeStatus{ Capacity: v1.ResourceList{ v1.ResourceCPU:              *resource.NewMilliQuantity(2000, resource.DecimalSI), v1.ResourceMemory:           *resource.NewQuantity(10e9, resource.BinarySI), v1.ResourceEphemeralStorage: *resource.NewQuantity(5000, resource.BinarySI), hugePageResourceName2Mi:     *resource.NewQuantity(0, resource.BinarySI), hugePageResourceName1Gi:     resource.MustParse("2Gi"), }, Allocatable: v1.ResourceList{ v1.ResourceCPU:              *resource.NewMilliQuantity(2000, resource.DecimalSI), v1.ResourceMemory:           *resource.NewQuantity(10e9, resource.BinarySI), v1.ResourceEphemeralStorage: *resource.NewQuantity(5000, resource.BinarySI), hugePageResourceName2Mi:     *resource.NewQuantity(0, resource.BinarySI), hugePageResourceName1Gi:     resource.MustParse("2Gi"), }, }, }, expectedNode: &v1.Node{ Status: v1.NodeStatus{ Capacity: v1.ResourceList{ v1.ResourceCPU:              *resource.NewMilliQuantity(2000, resource.DecimalSI), v1.ResourceMemory:           *resource.NewQuantity(10e9, resource.BinarySI), v1.ResourceEphemeralStorage: *resource.NewQuantity(5000, resource.BinarySI), hugePageResourceName1Gi:     resource.MustParse("2Gi"), }, Allocatable: v1.ResourceList{ v1.ResourceCPU:              *resource.NewMilliQuantity(2000, resource.DecimalSI), v1.ResourceMemory:           *resource.NewQuantity(10e9, resource.BinarySI), v1.ResourceEphemeralStorage: *resource.NewQuantity(5000, resource.BinarySI), hugePageResourceName1Gi:     resource.MustParse("2Gi"), }, }, }, }, } for _, tc := range cases { t.Run(tc.name, func(T *testing.T) { defer testKubelet.Cleanup() kubelet := testKubelet.kubelet needsUpdate := kubelet.reconcileHugePageResource(tc.initialNode, tc.existingNode) assert.Equal(t, tc.needsUpdate, needsUpdate, tc.name) assert.Equal(t, tc.expectedNode, tc.existingNode, tc.name) }) } } </ins> func TestReconcileExtendedResource(t *testing.T) { testKubelet := newTestKubelet(t, false /* controllerAttachDetachEnabled */) testKubelet.kubelet.kubeClient = nil // ensure only the heartbeat client is used

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When I change default hugepage size 2Mi to 1Gi and reboot, kubelet can't update node status What you expected to happen: kubelet can update node info and node status was ready How to reproduce it (as minimally and precisely as possible): change the node condition validate Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration:OS (e.g: ):Kernel (e.g. ):Install tools:Network plugin and version (if this is a network-related bug):Others:
/sig node
Hi, Ref. the error message, it looks like you have allocated hugepages for multiple sizes, . Can you run: to verify? If more than one of the numbers is , that is the reason. This is currently being discussed in , and the reason for implementing the check is here: Imo. this should be supported, and it would only require removing an if statement in order to implement it. I see no reason to limit the node from allocating huge pages with more than one size. Because of how it is implemented, adding support for allowing a pod to request huge pages with multiple sizes isn't trivial, and will require some potentially breaking changes, but that is not relevant to this problem. /cc
Also, can you also provide the result of , that will list the supported huge page sizes (both when is set, and when it is not.).
Thanks ! Looks like your kernel only enables huge pages with the default size. My kernel 5.0+ enable all (atleast 2MiB and 1GiB on x8664) by default. When the node booted with defaulthugepagesz=1GB it "removed support" the 2MiB one, and becuse kubelet use patch on the node status when connecting to the api server, you ended up with both and , even tho 2Mi wast supported by your machine (the sw, not the hw). Even if the error is removed, this will result in a crash when kubelet tries to write to the 2MiB hugetlb cgroup files.

I think using should work, otherwise the only current solution is to remove the node from the cluster, and rejoin afterwords. Ref. it looks like you have sorted out the problem, but I still think this is a valid issue, so don't close it. The correct behaviour for kubelet is to remove unsupported sizes, and only allow use of the supported ones.
"Imo. this should be supported, and it would only require removing an if statement in order to implement it. I see no reason to limit the node from allocating huge pages with more than one size." Could you fix this one as well? We use 2M pages only for pods but we need 1G pages for system processes. So it was little surprising when k8s doesn't start because two different pages were configured no matter they are really used or not. Thanks!
I am hitting the same, I used tuned to change the default size from 2Mi to 1Gi, i hit the same problem, after the reboot node is NotReady and it cannot rejoin , giving the error: Node "test-worker-0" is invalid: [status.capacity.hugepages-2Mi: Invalid value: resource.Quantity{i:resource.int64Amount{value:, scale:0}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:"20Mi", Format:"BinarySI"}: may not have pre-allocated hugepages for multiple page sizes, status.allocatable.hugepages-2Mi: Invalid value: resource.Quantity{i:resource.int64Amount{value:, scale:0}, My system just reports information for 1Gi, not 2Mi anymore. Removing the node and rebooting fixed it, the node rejoined again reporting the right capacity. But this is not a solution for us, changing sizes should be supported.
Hitting same K8s installed with my ubuntu 18 enabled 2Mi hugepage by default. As I change my master node's hugepage to 1Gi, this happened. I have only one master, so I have to rebuild the k8s cluster.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with .

Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
I'm a little confused. when i used tuned to change the hugepagesz from "defaulthugepagesz=2M hugepagesz=2M hugepages=8192" to " defaulthugepagesz=1G hugepagesz=1G hugepages=16", i hit the same problem, after the reboot node is NotReadybut when i used tuned to change the hugepagesz from " defaulthugepagesz=1G hugepagesz=1G hugepages=16" to "default_hugepagesz=2M hugepagesz=2M hugepages=8192", after the rebootkubelet status can still keep Ready. why?
Hi This will be fixed in , but it is still not merged. Hoping to get it into next release. In order to solve the issue (and use 1GiB huge pages) you have to delete the node in kubernetes via , and join the node again.
Thanks
Thanks
I think this is already fixed by can you confirm?
Hi WIth this would not cause a crash, but is still required in order to fix this properly.
I am facing the same issue. After changing the hugepage setting to . I stopped the kubelet process on the worker node and removed the node from the cluster using command and rebooted it. But after it comes back online and kubelet starts, it struggles to join the cluster . Do I have to run the join command again ?


Question: } refRelated, err := reference.GetReference(recorder.scheme, related) if err != nil { <del> klog.Errorf("Could not construct reference to: '%#v' due to: '%v'.", related, err) </del> <ins> klog.V(9).Infof("Could not construct reference to: '%#v' due to: '%v'.", related, err) </ins> } if !util.ValidateEventType(eventtype) { klog.Errorf("Unsupported event type: '%v'", eventtype)

Per the comments in v1beta1 Event: The field claimed to be optional, but an error is dumped if it's illegal - including the case: This doesn't look right as it means the caller which passes in Related field will always get an err, for example, the scheduler binding event: If appropriate, I'm gonna raise a PR to tolerate the nil case. please advice.
/sig scheduling /sig api-machinery
/assign I'll update the log to info or warning cc
It shouldn't be warning - it's a valid and allowed case. I'm wondering if we even should be logging info, but maybe we can for now.
/assign


Question: // Check that the hostname known by the master matches the hostname // the kubelet knows hostname := s.host.GetHostname() <del> if masterHostname != hostname { </del> <ins> if masterHostname != hostname && masterHostname != "127.0.0.1" && masterHostname != "localhost" { </ins> s.error(w, errors.New("Kubelet hostname ""+hostname+"" does not match the hostname expected by the master ""+masterHostname+""")) return }

We've run into issues like where the Kubelet was up but not healthy. The monit monitoring for Kubelet should ensure it is "healthy" rather than just alive. We need to define what healthy means in this context and monitor it based on that. An alternate option is for the Kubelet to exit when it gets into a situation it can't serve from. This may lead to crashloops so we'd need to be careful going this route. The answer is probably a bit of both.
Courtesy of healthchecking with monit:
Health check is under this umbrella issue:
Will send a PR shortly.


Question: c = f.Client ns = f.Namespace.Name nodeList = &api.NodeList{} <del> nodes, err := c.Nodes().List(api.ListOptions{}) masterNodes = sets.NewString() for _, node := range nodes.Items { if system.IsMasterNode(&node) { masterNodes.Insert(node.Name) } else { nodeList.Items = append(nodeList.Items, node) } } err = framework.CheckTestingNSDeletedExcept(c, ns) </del> <ins> masterNodes, nodeList = framework.GetMasterAndWorkerNodesOrDie(c) err := framework.CheckTestingNSDeletedExcept(c, ns) </ins> framework.ExpectNoError(err) // Every test case in this suite assumes that cluster add-on pods stay stable and

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --Create a pod: the pod is always , and describes as: kubelet log: Deny the pod orchestration with duplicate volumes, OR the pod can be running successfully. Create a pod with the duplicate pvc as above. Root cause The key of persistent volume info in DSW is named after, and the is a : When getting the unmounted volumes, it uses : In our case, volume will corve the volume in DSW, and the volume info is lost in ASW. Solution: 2 choices: Consider it as an invalid request, and deny creating it. Construct uniqueVolumeName with when the volume is a persistent volume.Kubernetes version (use ): v1.20.4 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /sig apps /sig node /sig storage
I don't think it's common sense to write duplicate volumes, and people don't use them that way, but if you want to achieve the above YAML intent, you can do it this way so I think it might be better this way if write duplicate volumes
/assign
I recognize your point, but I think kubernetes should validate that.
Yes, I think verification should be as well I'll try to do that
Then I thought about it again, and I think this YAML might sometimes be used this way So it may not be a good idea to say no outright Can you give me some advice ?
, I think that it is possible in some situations that a pod is running with duplicate persistent volume. For example, a container running web application reads application data from a persistent volume and saves logs to another PV.
This seems possible and Thank you for your description of the situation.
Does this address your use case?

I've tried it and it works But does this duplicate PV need to take certain measures, verification failed or can be successful or other pod Always in the phase
/unassign
This is more a support request rather than a bug... however I am not closing it since it would be good to document this solution: /remove-kind bug /kind support /triage accepted /priority backlog /area documentation /good-first-issue
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : This is more a support request rather than a bug... however I am not closing it since it would be good to document this solution: /kind support /triage accepted /priority backlog /area documentation /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : This is more a support request rather than a bug... however I am not closing it since it would be good to document this solution: /kind support /triage accepted /priority backlog /area documentation /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hey I am new to open source and will like to contribute to this project can you guide me on what needs to done.
Welcome ! Sure, so the goal is to document that example of a double mount of the same pv. I would put it somewhere here: Which corresponds to this repo:
If you have another idea about the best page to document it, please make it as you think.
are you taking this up , or should i take?
yes, I am taking this up.
/remove-kind support /kind documentation
Ah documentation is a kind... Makes sense. Thx Elana!
Hii after reading docs . I think the example of " double mount of the same PV " needs to be either before or at the end of the page. what did you say? And it may look something like this.

Running pods with the duplicate persistent volume, Example: A container running a web application reads application data from a persistent volume and saves logs to another PV. as the example was mentioned by
Thanks for looking at it. I wonder if that example would nicely add to as it's more hands on than the other page. What do you think?
yeah, docs seems the right place to add the " double mount of the same PV" example. and I think It should be after the . What do you think?
Yes, so now sign the CLA, open a PR and link this issue. We'll see what they say during the review...
I open a PR but still it is not reviewed. Do I need to assign it to someone?
I see your PR and prow has requested review from 2 reviewers... Let's give them one or two days and then we can poke them again
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
cc
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.

</details
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:


Question: if err != nil { return mountedVolumes, err } <ins> // Only use IsLikelyNotMountPoint to check might not cover all cases. For CSI volumes that // either: 1) don't mount or 2) bind mount in the rootfs, the mount check will not work as expected. // We plan to remove this mountpoint check as a condition before deleting pods since it is // not reliable and the condition might be different for different types of volumes. But it requires // a reliable way to clean up unused volume dir to avoid problems during pod deletion. See discussion in issue #74650 </ins> for _, volumePath := range volumePaths { isNotMount, err := kl.mounter.IsLikelyNotMountPoint(volumePath) if err != nil { <del> return mountedVolumes, err </del> <ins> return mountedVolumes, fmt.Errorf("fail to check mount point %q: %v", volumePath, err) </ins> } if !isNotMount { mountedVolumes = append(mountedVolumes, volumePath)

In , we check mount on first level directory in pod volume directories. It is true for many volume plugins, but not for CSI plugin which . cc
/assign
Pardon my ignorance of kubelet source code, but this looks very much similar to an issue we have been running to many times recently, when developing a CSI plugin. The problem is that when CSI volume publish is called and the mount point is not propagated from the CSI plugin pod (either missing bidirectional flag in yaml or not supported by underlaying k8s cluster implementation), the kubelet nevertheless starts a pod using the volume. The volume is not there, so the app writes to root filesystem in fact, leading to all other kinds of problems - i.e. losing data, filling up root fs. IMHO what kubelet should do is to check that the mounted filesystem is indeed visible after publish call and if not it should fail the volume attach request. If this ticket was fixed, would it solve the issue we are seeing or should I open a new ticket? thanks
I think we can potentially add that check but we need to make sure first it can work with ephemeral CSI drivers. Ephemeral drivers would have to "stage" their directories somewhere, like /var/lib/kubelet/plugins/<csi driver/, and then bind mount it to the publish path. cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Hmm... That is a good question. Does csi require a mount to be made or only that the specified directory be created? Will have to go read through the spec...
/remove-lifecycle stale
The only place in the spec I could find was this comment: // For volumes with an access type of mount, the SP SHALL place the // mounted directory at target_path. So I think having Kubelet enforce that check would be ok with the spec, and a good thing to do.
cc CSI spec says that the target path has to be a mount. Shall we go ahead and enforce it then? Are we going to break drivers that may not be doing this?

Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
The issue mentioned here involves two functions . Now with PR , the issue in CheckVolumeExistenceOperation is solved by checking mount path instead of volume path. The only issue here is for getMountedVolumePathListFromDisk which is used during pod deletion. This function was introduced by PR . Now I am questioning whether we need to check the mount point or just need to check pod dirs exist before deleting the pod. Firstly, for some plugins, there might not be mount point. Secondly, without checking mount point, it means that pod cannot be deleted as long as there are pod volume dirs. In normal case, volume dirs should be cleanedup during volume teardown. For cases that kubelet restarts, we have reconciler try to clean up volume mounts if possible. So we can propose a couple of solutions here mount point check. It solves the problem, the downside is that there might be some rare case that reconciler could not clean up dirs after kubelet restarts. check mount point, and also check the next level "/mount" dir. This might still cause problem for some plugins that do not have mounts. cc
Discussed offline with For proposal 1., pod will not be deleted from api server if there are still volume dirs exist. So to make sure it works well, we need to have reliable way to clean up dirs. Normally when kubelet restarts, reconciler will either add it into actual state (if reconstruction works) or try to clean it up directly once unless volume is still in desired state. So if reconstruction does not work, and it fails to clean up during reconciler sync, there is no other chance to clean up the dirs.

Or if volume was in desired state and removed quickly afterwords, the dir might not get the chance to be cleaned up at all. We are thinking to improve this part, we could use mark volume "uncertain" in actual state feature. During reconstruction, we can remove the check for mounted points (since the mount might be gone during node reboots or driver does not have moutns at all). Reconciler could mark those volumes as uncertain (even if it was in desired state) to make sure those volume got cleaned up or remount whenever needed with retries. With this change, I think it is safe to remove the mount point check for pod deletion too.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale