Question: ), expectedPrioritizers: sets.NewString(), }, <ins> { policy: `apiVersion: v1 kind: Policy predicates: - name: PredicateOne - name: PredicateTwo priorities: - name: PriorityOne weight: 1 - name: PriorityTwo weight: 5 `, expectedPredicates: sets.NewString( "CheckNodeCondition", // mandatory predicate "PredicateOne", "PredicateTwo", ), expectedPrioritizers: sets.NewString( "PriorityOne", "PriorityTwo", ), }, { policy: `apiVersion: v1 kind: Policy `, expectedPredicates: sets.NewString( "CheckNodeCondition", // mandatory predicate "CheckNodeDiskPressure", "CheckNodeMemoryPressure", "CheckNodePIDPressure", "CheckVolumeBinding", "GeneralPredicates", "MatchInterPodAffinity", "MaxAzureDiskVolumeCount", "MaxCSIVolumeCountPred", "MaxEBSVolumeCount", "MaxGCEPDVolumeCount", "NoDiskConflict", "NoVolumeZoneConflict", "PodToleratesNodeTaints", ), expectedPrioritizers: sets.NewString( "BalancedResourceAllocation", "InterPodAffinityPriority", "LeastRequestedPriority", "NodeAffinityPriority", "NodePreferAvoidPodsPriority", "SelectorSpreadPriority", "TaintTolerationPriority", "ImageLocalityPriority", ), }, { policy: `apiVersion: v1 kind: Policy predicates: [] priorities: [] `, expectedPredicates: sets.NewString( "CheckNodeCondition", // mandatory predicate ), expectedPrioritizers: sets.NewString(), }, </ins> } { // Add a ConfigMap object. configPolicyName := fmt.Sprintf("scheduler-custom-policy-config-%d", i)

Kubernetes scheduler enables you to overwrite default Predicates/Priorities as well as configuring extenders in a "scheduler policy". In latest scheduler componentconfig API (v1alpha1), it can be a file or a configmap: However, if you defines a scheduler config file and specify a YAML policy file: It doesn't work. It's because only JSON is supported: It'd nice to enhance it with an UniversalSerilizer so as to support both JSON and YAML. /sig scheduling /help /priority important-longterm
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : Kubernetes scheduler enables you to overwrite default Predicates/Priorities as well as configuring extenders in a "scheduler policy". /help /priority important-longterm </details
/assign


Question: }, }, } <del> pod = e2epod.NewPodClient(f).Create(ctx, pod) framework.ExpectNoError(e2epod.WaitForPodContainerToFail(ctx, f.ClientSet, pod.Namespace, pod.Name, 0, "CreateContainerConfigError", framework.PodStartShortTimeout)) var err error pod, err = f.ClientSet.CoreV1().Pods(f.Namespace.Name).Get(ctx, pod.Name, metav1.GetOptions{}) framework.ExpectNoError(err) gomega.Expect(pod.Status.ContainerStatuses[0].State.Waiting.Message).To( gomega.ContainSubstring("failed to resolve recursive read-only mode: volume "mnt" requested recursive read-only mode, but it is not read-only")) </del> <ins> _, err := f.ClientSet.CoreV1().Pods(pod.Namespace).Create(ctx, pod, metav1.CreateOptions{}) gomega.Expect(err).To(gomega.MatchError(gomega.ContainSubstring("spec.containers[0].volumeMounts.recursiveReadOnly: Forbidden: may only be specified when readOnly is true"))) </ins> }) // By // See also the unit test [pkg/kubelet.TestResolveRecursiveReadOnly] for more invalid conditions (e.g., incompatible mount propagation) }) // It }) // Context ginkgo.Context("when the runtime does not support recursive read-only mounts", func() { f.It("should accept non-recursive read-only mounts", func(ctx context.Context) { <ins> e2eskipper.SkipUnlessFeatureGateEnabled(features.RecursiveReadOnlyMounts) </ins> ginkgo.By("waiting for the node to be ready", func() { waitForNodeReady(ctx) if supportsRRO(ctx, f) {

<img width="1064" alt="image" src=""### Failure cluster /kind failing-test


Question: }) } } <ins> func TestCompareAuthzModes(t *testing.T) { var tests = []struct { name   string modesA []string modesB []string equal  bool }{ { name:   "modes match", modesA: []string{"a", "b", "c"}, modesB: []string{"a", "b", "c"}, equal:  true, }, { name:   "modes order does not match", modesA: []string{"a", "c", "b"}, modesB: []string{"a", "b", "c"}, }, { name:   "modes do not match; A has less modes", modesA: []string{"a", "b"}, modesB: []string{"a", "b", "c"}, }, { name:   "modes do not match; B has less modes", modesA: []string{"a", "b", "c"}, modesB: []string{"a", "b"}, }, } for _, rt := range tests { t.Run(rt.name, func(t *testing.T) { equal := compareAuthzModes(rt.modesA, rt.modesB) if equal != rt.equal { t.Errorf("failed compareAuthzModes:nexpected:n%vnsaw:n%v", rt.equal, equal) } }) } } </ins>

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: when i installed k8s using kubeadm and my custom config like then output is It is a bug that the default value(Node,RBAC) is entered but a warning message is displayed. A function must be to compare two string slices. What you expected to happen: If the default value and custom value are the same, no warning message should be displayed. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): 17.3 Install tools: kubeadm /area kubeadm /feature bug /sig cluster-lifecycle
/lifecycle active


Question: } activeDeadlineSeconds = int64(0) <ins> minID = int64(-1) maxID = int64(2147483648) </ins> failureCases := map[string]api.PodSpec{ "bad volume": { Volumes:       []api.Volume{{}},

Help review this, thanks.
Are you asking for better validation on pod creation? I'm not really sure what you mean when you say "tip".
Yes, I mean it's better validate the pod when create. The tip mean the validation info response from server and show by client.


Question: go_test( name = "go_default_test", srcs = [ <ins> "bootime_util_linux_test.go", </ins> "util_test.go", "util_unix_test.go", "util_windows_test.go",

What would you like to be : Currently, does not have unit test coverage. Specifically, there's no unit test coverage for . I'd like to see unit test coverage for . See for a sample of how to write platform specific unit tests. Why is this needed: There's a fairly small chance of being modified, and thus breaking, in the future without this test coverage. However, it should be relatively straightforward to add the test coverage, and could make a good first issue for a new contributor. There's also the benefit that it helps keep the test coverage in high :) And we never know, it might help catch a bug some day :) Extra note I think this ticket could be a great candidate for someone who is new to contributing to sig-node/k8s. I'd prefer to please give first dibs for picking this up to any folks who are currently underrepresented in the k8s community. Please reach out to on k8s Slack (or post directly in #sig-node and tag me) if you're interested, and I'll assign you the issue. I'm happy to mentor and be helpful in whatever way I can :)
Assigning this to Ali Farah (" (Ali, for some reason it won't let me assign you - can you assign yourself?) /sig node
Thanks ! /assign


Question: <del> // +build linux /* Copyright 2021 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package e2enode import ( "time" "golang.org/x/sys/unix" "github.com/onsi/ginkgo" "github.com/onsi/gomega" "k8s.io/kubernetes/test/e2e/framework" ) const contentionLockFile = "/var/run/kubelet.lock" var _ = SIGDescribe("Lock contention [Slow] [Disruptive] [NodeFeature:LockContention]", func() { ginkgo.It("Kubelet should stop when the test acquires the lock on lock file and restart once the lock is released", func() { ginkgo.By("perform kubelet health check to check if kubelet is healthy and running.") // Precautionary check that kubelet is healthy before running the test. gomega.Expect(kubeletHealthCheck(kubeletHealthCheckURL)).To(gomega.BeTrue()) ginkgo.By("acquiring the lock on lock file i.e /var/run/kubelet.lock") // Open the file with the intention to acquire the lock, this would imitate the behaviour // of the another kubelet(self-hosted) trying to start. When this lock contention happens // it is expected that the running kubelet must terminate and wait until the lock on the // lock file is released. // Kubelet uses the same approach to acquire the lock on lock file as shown here: // https://github.com/kubernetes/kubernetes/blob/master/cmd/kubelet/app/server.go#L530-#L546 // and the function definition of Acquire is here: // https://github.com/kubernetes/kubernetes/blob/master/pkg/util/flock/flock_unix.go#L25 fd, err := unix.Open(contentionLockFile, unix.O_CREAT|unix.O_RDWR|unix.O_CLOEXEC, 0600) framework.ExpectNoError(err) // Defer the lock release in case test fails and we don't reach the step of the release // lock. This ensures that we release the lock for sure. defer func() { err = unix.Flock(fd, unix.LOCK_UN) framework.ExpectNoError(err) }() // Acquire lock. err = unix.Flock(fd, unix.LOCK_EX) framework.ExpectNoError(err) ginkgo.By("verifying the kubelet is not healthy as there was a lock contention.") // Once the lock is acquired, check if the kubelet is in healthy state or not. // It should not be. gomega.Eventually(func() bool { return kubeletHealthCheck(kubeletHealthCheckURL) }, 10*time.Second, time.Second).Should(gomega.BeFalse()) ginkgo.By("releasing the lock on lock file i.e /var/run/kubelet.lock") // Release the lock. err = unix.Flock(fd, unix.LOCK_UN) framework.ExpectNoError(err) ginkgo.By("verifying the kubelet is healthy again after the lock was released.") // Releasing the lock triggers kubelet to re-acquire the lock and restart. // Hence the kubelet should report healthy state. gomega.Eventually(func() bool { return kubeletHealthCheck(kubeletHealthCheckURL) }, 10*time.Second, time.Second).Should(gomega.BeTrue()) }) }) </del>

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI -- tests 08/11 ~2:30PDT First fail run These tests appear to be newly in They have never passed as far as I can tell. Not sure why this was merged/not caught in CI. /cc /sig node
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hi thanks for pointing out. I was looking at this I thought I saw green runs in the latest tests. Please.correct me if I am wrong.
They're fine on that tab but they're broken in
I'm puzzled why they're even getting pulled into that job since they're marked ...
why does this feature need its own separate job for a single test?
Apologies, I'm not sure of the conventions for the tests. I wasn't given info/notified regarding this. Where else could I have put a test for this ?
since it's just the one test there are a number of jobs we could add this to. Let's discuss at the CI subproject tomorrow? I'll add this to the agenda.
When does the CI subproject happen ? If it fits the timezone, if like to atttend. I'd very much like to re-revert the changes and put this test in proper place with green runs
Agreed! It's tomorrow at 8am Pacific Time:


Question: } } <ins> func TestRemoveOutOfDiskCondition(t *testing.T) { now := time.Now() var cases = []struct { desc       string inputNode  *v1.Node expectNode *v1.Node }{ { desc: "should remove stale OutOfDiskCondition from node status", inputNode: &v1.Node{ Status: v1.NodeStatus{ Conditions: []v1.NodeCondition{ *makeMemoryPressureCondition(false, now, now), { Type:   v1.NodeOutOfDisk, Status: v1.ConditionFalse, }, *makeDiskPressureCondition(false, now, now), }, }, }, expectNode: &v1.Node{ Status: v1.NodeStatus{ Conditions: []v1.NodeCondition{ *makeMemoryPressureCondition(false, now, now), *makeDiskPressureCondition(false, now, now), }, }, }, }, } for _, tc := range cases { t.Run(tc.desc, func(t *testing.T) { // construct setter setter := RemoveOutOfDiskCondition() // call setter on node if err := setter(tc.inputNode); err != nil { t.Fatalf("unexpected error: %v", err) } // check expected node assert.True(t, apiequality.Semantic.DeepEqual(tc.expectNode, tc.inputNode), "Diff: %s", diff.ObjectDiff(tc.expectNode, tc.inputNode)) }) } } </ins> // Test Helpers: // sortableNodeAddress is a type for sorting []v1.NodeAddress

What happened: removed reporting of the OutOfDisk condition in the kubelet. A 1.12 kubelet upgraded to a 1.13 kubelet leaves the previously reported OutOfDisk condition in place, and it eventually gets switched to Unknown status. What you expected to happen: If this condition is no longer reported, it should be removed by the kubelet. How to reproduce it (as minimally and precisely as possible): a 1.12 kubelet, let it report OutOfDisk condition status. to 1.13 the condition is no longer updated a while, the node controller flips the status to Unknown Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug /sig node
The fix for this should be picked to 1.13
/assign happened to have a fix handy, if no one is working on it.
after upgrade kubelet should ignore this condition . or change it to 'DiskPressure'
Better not. What if both condition and exist?
should be picked to 1.13 also, I think.
13.2 doesn't have this patch; would it be possible to make it for 1.13.3?
Unfortunately this was not mentioned in the release notes (or I missed it), got hit by this too. 13.2 is affected too, would be good for me to see this getting fixed in 1.13.3


Question: "least_requested_test.go", "metadata_test.go", "most_requested_test.go", <del> "node_prefer_avoid_pods_test.go", </del> "requested_to_capacity_ratio_test.go", "resource_limits_test.go", "selector_spreading_test.go",

/sig scheduling /priority important-soon See as an example. Part of This shouldn't need a PostFilter Plugin, it is not worth caching the result of metav1.GetControllerOf. /assign


Question: "//staging/src/k8s.io/cri-api/pkg/apis/runtime/v1alpha2:go_default_library", "//third_party/forked/golang/expansion:go_default_library", "//vendor/k8s.io/klog:go_default_library", <ins> "//vendor/k8s.io/utils/net:go_default_library", </ins> ], )

What happened: I tried to create a deployment with duplicate containerPorts with hostIPs of two different address families. Only the first mapping was provided as input into my CNI plugin chain. Also, a log message from kubelet warning about a duplicate containerPort was created. What you expected to happen: Both mappings to be provided as input into my CNI plugins. No log messages should be created. How to reproduce it (as minimally and precisely as possible): Create this deployment: Then grab kubelet logs on the host e.g: Anything else we need to know?: This issue here provides some context as to how I ran into this issue: Environment: Kubernetes version (use ): v1.15.1 Cloud provider or hardware configuration: DigitalOcean, custom setup OS (e.g: ):Kernel (e.g. ):Install tools: None Network plugin and version (if this is a network-related bug): Cilium v1.6.0 and Portmap v0.8.2 Others: N/A I've already identified the problem and will submit a PR which fixes this shortly. /sig network
is this dual-stack only? if so, does the fix for this need to target 1.16?
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
This applies to dual-stack only yes. I don't know if it should target 1.16, personally I'd like to get this out so I don't have to run a forked version of kubelet.
/assign
I would agree that this should hit v1.16 if possible.
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.


Question: logPodStates(badPods) return false, nil }) != nil { <del> return errors.New(errorBadPodsStates(badPods, desiredPods, ns, timeout)) </del> <ins> return errors.New(errorBadPodsStates(badPods, desiredPods, ns, "RUNNING and READY", timeout)) </ins> } wg.Wait() if waitForSuccessError != nil {

https://k8s- Failed: BeforeSuite {Kubernetes e2e suite} Previous issues for this test:


Question: rm -r etcd-v2.2.5-linux-amd64* fi <ins> # Install nsenter for ubuntu images cat /etc/*-release | grep "ID=ubuntu" if [ $? -eq 0 ]; then if ! which nsenter > /dev/null; then echo "Do not find nsenter. Install it." mkdir -p /tmp/nsenter-install cd /tmp/nsenter-install curl https://www.kernel.org/pub/linux/utils/util-linux/v2.24/util-linux-2.24.tar.gz | tar -zxf- sudo apt-get update sudo apt-get --yes install make sudo apt-get --yes install gcc cd util-linux-2.24 ./configure --without-ncurses make nsenter sudo cp nsenter /usr/local/bin rm -rf /tmp/nsenter-install fi fi </ins> # Install docker hash docker 2>/dev/null if [ $? -ne 0 ]; then

Doesn't seem to haven been caused by my change:
/cc
The test only failed on coreos, and the is full of error messages. Plus the log is almost unreadable because of the extremely high verbosity. I filed for that. shows that the node has been rebooted a couple of times. And the kernel log shows This looks like . The hairpin mode is set to hairpin-veth. /cc
Yeah don't run with hairpin-mode I guess. is coreos using kubenet?
Node problem detector should detect this issue and report KernelDeadLock condition.
Yeah, but this is node e2e test, we don't have node problem detector running on node e2e cluster.
Not sure how to assign coreos issues... Assigning to people to reassign based on thread activity. Please reassign correctly!
ping!
/cc
This happens in node e2e test right? I believe node e2e test is NOT using kubenet.
Running node e2e with kubenet+hairpin-veth could prevent this.
Sorry, this is lost in github notifications.. Feel free to ping me on slack if I am not responding in a day next time :)
To run kubenet on a node, the framework needs to download and unpack the cni plugin in kubelet needs to run with and a fixed pod CIDR (since there is no node controller) I did those in , but it failed on the trusty images because is not installed. We need to either install this in the node e2e framework, or bake it in to the image. what's the best practice for installing packages on the images?
I believe updating is the best way to bake it into the image, and I also think that should be preferred over installing it each time. You could do the same with CNI. Thanks for working on this!
I updated the PR to install nsenter on the trusty images. As for cni, I still keep it in the node e2e framework to install it every time because 1) it only downloads from GCS and untar a file of ~14MB, 2) the tarball is maintained and updated by It'd be an extra burden to update the images just for cni updates.
OK with the plan.

Seeing the error message again after switching to kubenet+hairpin-veth do you want to take a look at the logs?
let me take a look
urrrh... we hit it again. Switching back to
There's a difference in the above kernel log; the device is not a interface but rather an interface (presumably the one within a container's net ns since there is no eth0 on the host). I think all the prior instances of this were veths on the host.
True. This one is slightly different. I think deadlock on interface also showed up before.
Haven't seen any new occurrence since landed. We'll know in a day or so whether this problem has been fixed.
AFAIK, there are no new failures after landed. It should be safe to let this issue stay closed.
unfortunately I hit the bug with hairpinmode sets to promiscuous-bridge kubelet v1.3.0, linux kernel 4.6.3 cbr0 has already enter the promiscuous mode, all veth hairpinmode = 1 start with parameter
can you ? what distro are you running?
also are you using a network-plugin, overlay (like flannel), which cloudprovider?
Are the SELinux warnings normal ?
Maybe a little out of topic. Are you running K8s on GCE? Do you have the node problem detector running in your cluster?
all veths hairpin_mode = 1 my distro is coreos 1068.6.0 I don't use any network-plugins, and no cloudprovider support here, controller-manager start with kubelet setup after address allocated by controller-manager, all pods connected each other with static routes which controlled by ospfd running inside docker. I can confirm the selinux warnings not related to the bug.
not running on GCE. no detector running inside.
Alright, just want to verify whether the NPD works as expected.
here is the logs of kubelet startup.
That's the problem, the reason we ran into this to begin with is because of hairpin mode. Apparently it is still getting set, or isn't getting erased in your environment. I don't think we erase existing values.
that's wired.

I checked all my nodes, each veth hairpin_mode is 1. it seems it was set to 1 regardless of --configure-cbr0.
there is no existing veth before kubelet create bridge , docker could not start without cbr0 which created by kubelet.
hairpin_mode is always enabled if the device is a bridge interface,
Trust me it's not on my (and our other e2e test) clusters, or we'd know because tests would fail, and teh default setup is docker running on cbr0 and using veths. In 1.3 we started using kubenet network plugin by default, but hairpin flag wasn't set since 1.2. I can trace the code path down when I have some time, but I suspect it's something to do with the way coreos is setup.
I trust you, I promise. I did some tests. None of results told me the flag is set by kubelet. I could not figure out what was happend to these veths. finally, I duplicate to , the only difference is hairepinEnable default to "0", and invoke the hairpinunset.SetUpContainerPid when false. It works! But, if I create veth by hand, the default hairpin_mode is 0 , oh my god. It will not happen in my test environment, only in production env
will enable hairpin_mode That's my fault, I should not set it to false because kubernetes do not use docker's network model.


Question: ) func TestExtractVmssVMName(t *testing.T) { <del> ss := &scaleSet{} </del> cases := []struct { description        string vmName             string

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: When useInstanceMetadata is set to true for vmss nodes, their providerID is wrong. What you expected to happen: Their providerID should be in format azure:///subscriptions/<subscription-id/resourceGroups/<rg-name/providers/Microsoft.Compute/virtualMachineScaleSets/<scaleset-name/virtualMachines/<instance-id How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig azure
adding priority important-soon so that this issue can stay in 1.10. Please update if incorrect. /priority important-soon


Question: var bmods []string <del> // Find out loaded kernel modules. If this is a full static kernel it will thrown an error </del> <ins> // Find out loaded kernel modules. If this is a full static kernel it will try to verify if the module is compiled using /boot/config-KERNELVERSION </ins> modulesFile, err := os.Open("/proc/modules") <ins> if err == os.ErrNotExist { klog.Warningf("Failed to read file /proc/modules with error %v. Assuming this is a kernel without loadable modules support enabled", err) kernelConfigFile := fmt.Sprintf("/boot/config-%s", kernelVersionStr) kConfig, err := ioutil.ReadFile(kernelConfigFile) if err != nil { return nil, fmt.Errorf("Failed to read Kernel Config file %s with error %v", kernelConfigFile, err) } for _, module := range ipvsModules { if match, _ := regexp.Match("CONFIG_"+strings.ToUpper(module)+"=y", kConfig); match { bmods = append(bmods, module) } } return bmods, nil } </ins> if err != nil { <del> klog.Warningf("Failed to read file /proc/modules with error %v. Kube-proxy requires loadable modules support enabled in the kernel", err) return nil, err </del> <ins> return nil, fmt.Errorf("Failed to read file /proc/modules with error %v", err) </ins> } mods, err := getFirstColumn(modulesFile)

<Reopened this valid and in-demand bug after the auto-bot expired it/kind bug /sig network /area kube-proxy What happened: Building a fully static Linux kernel with kubeproxy's desired IPVS (Virtual Server) modules linked statically: CONFIGIPVS=y CONFIGIPVSRR=y CONFIGIPVSWRR=y CONFIGIPVSSH=y CONFIGNFCONNTRACKIPV4=y Causes kubeproxy to believe that these features don't exist because it can't find modules for them: What you expected to happen: If Kubeproxy's desired kernel features are linked statically in the kernel, kubeproxy should carry on and not attempt and throw error messages about modules not being found. Under the hood, kubeproxy should simply check for the existence of the desired kernel features in the /proc tree (ie: , and others) before attempting to look for and load modules. If, after module loads fails, then throw the error messages and switch to iptables. How to reproduce it (as minimally and precisely as possible): and extract the current stable kernel source version (today it's 4.18.9) from your existing running into the source tree as and answer any questions to switch to a fully static kernel and ensure the following exist, and are not : and deselect (box is empty) your boot loader's config () or () depending on your distribution. and select your new kernel with the kubeproxy modules linked statically. kubeproxy and watch it throw the above errors about not finding the modules. Anything else we need to know?: Fully static kernels will not generate so inspecting that file will fail - as well in the kubeproxy source. The gold-standard is to simply check for the features in the active tree first, and consider those as "built-in". The next best options is checking for statements, which tells you if they're built-in. Note that files are almost universally installed by OS maintainers. And finally, you could try checking , which is a compressed version of the externally held .config text file, in a similar manner noted above, however it's up to the maintainer to build that feature into the kernel (so it might not exist).

For reference, here's the list of modules kubeproxy wants: Here we see kube_proxy's errant module-centric code that needs to be made compatible w/ static kernels: Environment: Kubernetes version (use ): 1.11.3 Cloud provider or hardware configuration: On premise OS (e.g. from /etc/os-release): Leap 15.0 Kernel (e.g. ): 4.18.9 statically linked Install tools: gcc 8.1 Others: <reopened as it appears still unfixed and in-demandSee how far the team got on it before it was auto-closed here:
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
Hi is the issue just the log messages, or do the kube-proxy features actually fail?
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
/unassign
/remove-triage unresolved
To check kernel-modules instead of functions is not limited to in k8s. I build my own kernel which was static from the start (with ipvs support). It was a very tiresome trial-and-error time to find out which functions k8s is checking by it's modules. But those who build the Linux kernel probably can probably accept less "help", so a flag to the apropriate k8s programs may be a solution.
I've checked the code, this is only a warning. Maybe while loading the modules, this function could be reduced to something like "check if this exists exists in the kernel builtin modules and append it into the mod list, otherwise try to modprobe it". I've made a PR to it, let's see if this solves the "problem"
/assign
Those are not options, as compilling the kernel doesn' t imply that one will copy the config file into /boot after recompiling it. As an example, the step ` is optional. So the only way right now I've found to be plausible is to verify the existence of the related but what happens if this path changes in the future?

take a look at to see if this is some way to follow Don't think adding a new flag is a good idea (but I might be wrong either, and TBH I've tried ;) )
rikatz, that looks like a reasonable approach. Although the kernel API isn't standardized, the developers do treat /proc as close to stable as possible, and only change existing files and structures as a last resort (they realize that change here ripple across critial userspace, including stuff like systemd). So I think checking proc for the desired features is the first and most reliable approach and then falling back to those in the order that you've suggested in your write up. The error message can even indicate the name of the /boot/config-* file, to help those building their own kernels who might have forgot to provide a correspondingly-named config.
/reopen The patch was a partial solution, We still need to verify if the module loader will support kernels with all builtin
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale Em ter, 14 de jan de 2020 às 07:32, fejta-bot <:


Question: var shPids []int ginkgo.By("checking besteffort container's oom-score-adj") gomega.Eventually(func() error { <del> shPids, err = getPidsForProcess("serve_hostname", "") </del> <ins> shPids, err = getPidsForProcess("agnhost", "") </ins> if err != nil { return fmt.Errorf("failed to get list of serve hostname process pids: %v", err) } if len(shPids) != 1 { <del> return fmt.Errorf("expected only one serve_hostname process; found %d", len(shPids)) </del> <ins> return fmt.Errorf("expected only one agnhost process; found %d", len(shPids)) </ins> } return validateOOMScoreAdjSetting(shPids[0], 1000) }, 2*time.Minute, time.Second*4).Should(gomega.BeNil())

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: ci-kubernetes-node-kubelet-serial Which test(s) are failing: [] Container Manager Misc [Serial] Validate OOM score adjustments [NodeFeature:OOMScoreAdj] once the node is setup pod infra containers oom-score-adj should be -998 and best effort container's should be 1000 Since when has it been failing: Testgrid link: Reason for failure: Anything else we need to know:
/assign
Let's monitor this one and see if worked /reopen /milestone v1.17 /priority critical-urgent
Reopened this issue. <detailsIn response to : Let's monitor this one and see if worked /reopen /milestone v1.17 /priority critical-urgent </details
It seems work well.
These now have 8 successful runs without a single flake. Looks like worked! :smile: 's%20should%20be% cc
thanks /close
Closing this issue. <detailsIn response to : thanks Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: // ValidateEvent makes sure that the event makes sense. func ValidateEvent(event *api.Event) field.ErrorList { allErrs := field.ErrorList{} <del> // There is no namespace required for node or persistent volume. // However, older client code accidentally sets event.Namespace // to api.NamespaceDefault, so we accept that too, but "" is preferred. if (event.InvolvedObject.Kind == "Node" || event.InvolvedObject.Kind == "PersistentVolume") && event.Namespace != api.NamespaceDefault && event.Namespace != "" { allErrs = append(allErrs, field.Invalid(field.NewPath("involvedObject", "namespace"), event.InvolvedObject.Namespace, "not allowed for node")) </del> <ins> // Make sure event.Namespace and the involvedObject.Namespace agree if len(event.InvolvedObject.Namespace) == 0 { // event.Namespace must also be empty (or "default", for compatibility with old clients) if event.Namespace != api.NamespaceNone && event.Namespace != api.NamespaceDefault { allErrs = append(allErrs, field.Invalid(field.NewPath("involvedObject", "namespace"), event.InvolvedObject.Namespace, "does not match event.namespace")) } } else { // event namespace must match if event.Namespace != event.InvolvedObject.Namespace { allErrs = append(allErrs, field.Invalid(field.NewPath("involvedObject", "namespace"), event.InvolvedObject.Namespace, "does not match event.namespace")) } </ins> } <del> if event.InvolvedObject.Kind != "Node" && event.InvolvedObject.Kind != "PersistentVolume" && event.Namespace != event.InvolvedObject.Namespace { allErrs = append(allErrs, field.Invalid(field.NewPath("involvedObject", "namespace"), event.InvolvedObject.Namespace, "does not match involvedObject")) </del> <ins> // For kinds we recognize, make sure involvedObject.Namespace is set for namespaced kinds if namespaced, err := isNamespacedKind(event.InvolvedObject.Kind, event.InvolvedObject.APIVersion); err == nil { if namespaced && len(event.InvolvedObject.Namespace) == 0 { allErrs = append(allErrs, field.Required(field.NewPath("involvedObject", "namespace"), fmt.Sprintf("required for kind %s", event.InvolvedObject.Kind))) } if !namespaced && len(event.InvolvedObject.Namespace) > 0 { allErrs = append(allErrs, field.Invalid(field.NewPath("involvedObject", "namespace"), event.InvolvedObject.Namespace, fmt.Sprintf("not allowed for kind %s", event.InvolvedObject.Kind))) } </ins> } for _, msg := range validation.IsDNS1123Subdomain(event.Namespace) { allErrs = append(allErrs, field.Invalid(field.NewPath("namespace"), event.Namespace, msg)) } return allErrs } <ins> // Check whether the kind in groupVersion is scoped at the root of the api hierarchy func isNamespacedKind(kind, groupVersion string) (bool, error) { group := apiutil.GetGroup(groupVersion) g, err := registered.Group(group) if err != nil { return false, err } restMapping, err := g.RESTMapper.RESTMapping(unversioned.GroupKind{Group: group, Kind: kind}, apiutil.GetVersion(groupVersion)) if err != nil { return false, err } scopeName := restMapping.Scope.Name() if scopeName == meta.RESTScopeNameNamespace { return true, nil } return false, nil } </ins>

ValidateEvent() in checks for "Node" specifically, but should really check for "any non-namespaced object.
I'd like to fix this.
Harder than it sounds, because events may reference 3rd party objects for which we don't know this piece of information...


Question: verifyEvents(t, expected, actual) } <ins> // TestEventChannelFull test when channel is full, the events will be discard. func TestEventChannelFull(t *testing.T) { testPleg := newTestGenericPLEGWithChannelSize(4) pleg, runtime := testPleg.pleg, testPleg.runtime ch := pleg.Watch() // The first relist should send a PodSync event to each pod. runtime.AllPodList = []*containertest.FakePod{ {Pod: &kubecontainer.Pod{ ID: "1234", Containers: []*kubecontainer.Container{ createTestContainer("c1", kubecontainer.ContainerStateExited), createTestContainer("c2", kubecontainer.ContainerStateRunning), createTestContainer("c3", kubecontainer.ContainerStateUnknown), }, }}, {Pod: &kubecontainer.Pod{ ID: "4567", Containers: []*kubecontainer.Container{ createTestContainer("c1", kubecontainer.ContainerStateExited), }, }}, } pleg.relist() // Report every running/exited container if we see them for the first time. expected := []*PodLifecycleEvent{ {ID: "1234", Type: ContainerStarted, Data: "c2"}, {ID: "4567", Type: ContainerDied, Data: "c1"}, {ID: "1234", Type: ContainerDied, Data: "c1"}, } actual := getEventsFromChannel(ch) verifyEvents(t, expected, actual) runtime.AllPodList = []*containertest.FakePod{ {Pod: &kubecontainer.Pod{ ID: "1234", Containers: []*kubecontainer.Container{ createTestContainer("c2", kubecontainer.ContainerStateExited), createTestContainer("c3", kubecontainer.ContainerStateRunning), }, }}, {Pod: &kubecontainer.Pod{ ID: "4567", Containers: []*kubecontainer.Container{ createTestContainer("c4", kubecontainer.ContainerStateRunning), }, }}, } pleg.relist() // event channel is full, discard events expected = []*PodLifecycleEvent{ {ID: "1234", Type: ContainerRemoved, Data: "c1"}, {ID: "1234", Type: ContainerDied, Data: "c2"}, {ID: "1234", Type: ContainerStarted, Data: "c3"}, {ID: "4567", Type: ContainerRemoved, Data: "c1"}, } actual = getEventsFromChannel(ch) verifyEvents(t, expected, actual) } </ins> func TestDetectingContainerDeaths(t *testing.T) { // Vary the number of relists after the container started and before the // container died to account for the changes in pleg's internal states.

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: In our stress testing, we found the deadlock issue in PLEG relist() for healthy checking and Kubelet syncLoop() The goroutine stack trace is as following We did some investigation, and found the following issue. The main loop for event processing in Kubelet will check if there are some runtimeErrors: if yes, wait until runtime back to normal; if no, it will call syncLoopIteration to read and process events. And the runtime healthChecks depends on the But in the relist() impl, it will send the pod lifecycle event to eventChannel which may introduce deadlock when eventChannel is full. And in the above code, the eventChannel will not be consumed if the runtime is not healthy. In some situation, the eventChannel may full, and the relist() will hang in sending new pod lifecycle event. And the runtime status will not be able to recovered. It is related to issues What you expected to happen: No such issue How to reproduce it (as minimally and precisely as possible): It is not easy to reproduce, it happens after few days stress testing (scaling the deployment repeatedly with high load). Anything else we need to know?: Environment: Kubernetes version (use ): 1.11.5 Cloud provider or hardware configuration: Bare metal OS (e.g. from /etc/os-release): CentOS 7.4 Kernel (e.g. ): Linux bjc-qtt-k8s-node-010 3.10.0-693.2.2.el7.x8664 SMP Tue Sep 12 22:26:13 UTC 2017 x8664 x8664 x8664 GNU/Linux Install tools: kubeadm Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
/sig node
Sure, thank you We had some internal fix to break the deadlock, but I am wondering if there are good fix for such kind of issue.
When pleg do relist, can we invoke to judge runtime is ok, if not, not do relist
I submit a PR
Thanks for finding the issue! The issue description makes sense to me. I think PLEG is a core component, we should try our best to unblock it.

The event channel becoming full seems more like an issue in the event consumer side, which shouldn't block PLEG. One solution is to queue the events in PLEG and send later, but that doesn't fundamentally solve the problem, if we don't set a limit on the event number we queue, it seems like a potential memory leak to me; if we set a limit, we may eventually hit this problem again. Another solution is to add a retry for sending event in PLEG, and set a bit in PLEG in between the retry which tells that PLEG is not blocking, it is trying to send out the event. And in the function of PLEG, it returns healthy if the bit is set. Actually, even if we fail to send the event and discard it, since we have already updated the pod cache, the periodical pod resync should eventually pick it up. This may delay pod sync for up to . But if the channel is full, it means that either 1) the syncloop is blocked for some reason, or 2) the node is under heavy loads. In both cases, even the event is successfully sent, the pod sync will likely be delayed anyhow. We can: 1) Enlarge the event channel size to reduce the chance that the channel becoming full. Currently it is arbitrarily set as . 2) Discard the event in PLEG if the channel is already full. In the 3 options above, I'm fine with either 2 or 3. Option 2 doesn't have the 1min pod sync delay, but a bit ad-hoc; option 3 is more generic, but may delay pod sync. There might be other options. /cc Any thoughts?
If PLEG is already blocked by a full event channel, this doesn't help right?
cc/ This looks like the one production issue you mentioned to me a while back.
of options enumerated, option 3 was preferred for me.
Thanks for reporting the issue! what was the load on the node that triggered this condition? If the node wasn't function at all, and the events slowly built up over time, bumping the size of the channel won't help. This should work, and doesn't complicate the logic in PLEG more, so it's my preferred option too.

We need some unit tests, and ideally, a stress test, as mentioned in the sig-node meeting.
Agree, giving a proper size for event channel is tricky as the scale also grows with time. We will ship fix as option 3 () for 1.14. Thanks for summarizing all these up as sig meeting! /assign
If possible, could you share in how large the stress is? It helps understand the issue and kubelet stress handling more. :)
Sure, our product team ( ) will input the stress test data after removing sensitive information in follow up comments.
Thanks for reply. Following is our testcase to reproduce this issue: The test cluster has 5 node on aliyun ECS(like EC2). Three 4C8G machine as master node, Two 96C384G as worker node. Firstly, We create 2 deployment running on the worker node. One deployment is http server pods. Another is siege pods to produce network stress to http server deployment. Those stress test deployments produce 50 CPU Core load on worker node. Then we create a job frequently create/delete 20 pod every 5 seconds. After about 1 day of stess test, The worker node will become NotReady. Dig the pprof goroutine stacktrace found the channel block in .
Yeah, I can see how frequent container state change can cause this issue. The number is very useful, thanks a lot!
cloud you please help to share how to dump the goroutines stack trace of kubelet like what you pasted in your comment. thanks a lot.
This issue is fixed by
using pprof is a answer to my question thank you anyway.


Question: "--node-cidr-mask-size=80", }, }, <ins> { name: "custom certs dir for v1.11.3", cfg: &kubeadmapi.ClusterConfiguration{ CertificatesDir:   testCertsDir, KubernetesVersion: "v1.11.3", }, expected: []string{ "kube-controller-manager", "--address=127.0.0.1", "--leader-elect=true", "--kubeconfig=" + kubeadmconstants.KubernetesDir + "/controller-manager.conf", "--root-ca-file=" + testCertsDir + "/ca.crt", "--service-account-private-key-file=" + testCertsDir + "/sa.key", "--cluster-signing-cert-file=" + testCertsDir + "/ca.crt", "--cluster-signing-key-file=" + testCertsDir + "/ca.key", "--use-service-account-credentials=true", "--controllers=*,bootstrapsigner,tokencleaner", }, }, { name: "custom cloudprovider for v1.11.3", cfg: &kubeadmapi.ClusterConfiguration{ Networking:        kubeadmapi.Networking{PodSubnet: "10.0.1.15/16"}, CertificatesDir:   testCertsDir, KubernetesVersion: "v1.11.3", }, expected: []string{ "kube-controller-manager", "--address=127.0.0.1", "--leader-elect=true", "--kubeconfig=" + kubeadmconstants.KubernetesDir + "/controller-manager.conf", "--root-ca-file=" + testCertsDir + "/ca.crt", "--service-account-private-key-file=" + testCertsDir + "/sa.key", "--cluster-signing-cert-file=" + testCertsDir + "/ca.crt", "--cluster-signing-key-file=" + testCertsDir + "/ca.key", "--use-service-account-credentials=true", "--controllers=*,bootstrapsigner,tokencleaner", "--allocate-node-cidrs=true", "--cluster-cidr=10.0.1.15/16", "--node-cidr-mask-size=24", }, }, { name: "custom extra-args for v1.11.3", cfg: &kubeadmapi.ClusterConfiguration{ Networking:                 kubeadmapi.Networking{PodSubnet: "10.0.1.15/16"}, ControllerManagerExtraArgs: map[string]string{"node-cidr-mask-size": "20"}, CertificatesDir:            testCertsDir, KubernetesVersion:          "v1.11.3", }, expected: []string{ "kube-controller-manager", "--address=127.0.0.1", "--leader-elect=true", "--kubeconfig=" + kubeadmconstants.KubernetesDir + "/controller-manager.conf", "--root-ca-file=" + testCertsDir + "/ca.crt", "--service-account-private-key-file=" + testCertsDir + "/sa.key", "--cluster-signing-cert-file=" + testCertsDir + "/ca.crt", "--cluster-signing-key-file=" + testCertsDir + "/ca.key", "--use-service-account-credentials=true", "--controllers=*,bootstrapsigner,tokencleaner", "--allocate-node-cidrs=true", "--cluster-cidr=10.0.1.15/16", "--node-cidr-mask-size=20", }, }, { name: "custom IPv6 networking for v1.11.3", cfg: &kubeadmapi.ClusterConfiguration{ Networking:        kubeadmapi.Networking{PodSubnet: "2001:db8::/64"}, CertificatesDir:   testCertsDir, KubernetesVersion: "v1.11.3", }, expected: []string{ "kube-controller-manager", "--address=127.0.0.1", "--leader-elect=true", "--kubeconfig=" + kubeadmconstants.KubernetesDir + "/controller-manager.conf", "--root-ca-file=" + testCertsDir + "/ca.crt", "--service-account-private-key-file=" + testCertsDir + "/sa.key", "--cluster-signing-cert-file=" + testCertsDir + "/ca.crt", "--cluster-signing-key-file=" + testCertsDir + "/ca.key", "--use-service-account-credentials=true", "--controllers=*,bootstrapsigner,tokencleaner", "--allocate-node-cidrs=true", "--cluster-cidr=2001:db8::/64", "--node-cidr-mask-size=80", }, }, </ins> } for _, rt := range tests {

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: When I use kubeadm to create an old version k8s cluster, the controller manager pod can not work well. The command like this : "kubeadm init --kubernetes-version=v1.11.2" The error of kube controller manager pod like this : What you expected to happen: The controller manager pod can work well. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): v1.12.0 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): CentOS 7 Kernel (e.g. ): Install tools: kubeadm Others:
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-sig contributor-experience /area kubeadm
Please, use when filing kubeadm issues. /sig cluster-lifecycle
OK, thank you for reminding me.


Question: if err != nil && !tc.shouldFail { t.Errorf("test should not fail, but error occurred: %v", err) } else if err == nil { <ins> expectedMountPath := filepath.Join(pluginDir, testDriver, generateSha(tc.volumeId), globalMountInGlobalPath) </ins> if tc.shouldFail { t.Errorf("test should fail, but no error occurred") <del> } else if mountPath != tc.expectedMountPath { t.Errorf("mountPath does not equal expectedMountPath. Got: %s. Expected: %s", mountPath, tc.expectedMountPath) </del> <ins> } else if mountPath != expectedMountPath { t.Errorf("mountPath does not equal expectedMountPath. Got: %s. Expected: %s", mountPath, expectedMountPath) </ins> } } }

When two CSI PVs point to the same volume in the storage backend, kubelet can't start pods that use these two PVs on the same node. Both PVs have the same VolumeHandle and kubelet is smart enough to recognize that they're the same volume (because they have the same unique volume ID). It calls only once, however, the staging path contains name of one of the PVs, say PV1: One of the CSI volume calls then fails, because it uses PV2 to compute the staging path and that path does not exist. Both pods should just start. Using CSI HostPath volume, PV1 (dynamically provisioned for simplicity): PV2 (basically edited PV1): PVC1, PVC2, Pod1 and Pod2 are trivial, nothing special there. <detailsToday's master, </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage /priority backlog
I suggest that if two volumes have the same unique volume ID (computed from CSI ), they must have also the same global mount path. Kubelet already has code to call NodeStage once and bind-mount it to different pods, regardless how many different PVs they use. These are several issues changing the global mount path: uses PV parsed from the global path to get the PV and to get enough info to call NodeUnstage: However, this parsing is used only when the volume plugin is not able to find/parse its json it has saved during MountDevice: Can we safely assume that the json is always there? the global path in a running cluster is tricky. We can require nodes are drained when kubelet with new global mount path is installed. This way, no CSI volume is mounted to a global path when the new kubelet starts. Minor cluster upgrades (1.23 -1.24) are , also documented in . Or we could fix kubelet to use the old global path when it already exists. This looks fragile to me.

staging dir needs to be unique enough not to conflict with other CSI volumes. It should be based on the driver name + volume handle, but that could be quite long. Can we compute a hash and hope it won't conflict?
I think in-tree driver can handle this case correctly because global mountpath is using unique volume id instead of pv name. With csi-migration, does it break backward compatibility?
We require users to drain nodes before enabling migration or when updating to a Kubernetes version that have migration enabled by default. All volumes should be therefore unmounted.
probably yes this breaks backward compatibility, but since only migration for PD is being done maybe it's more rare? We saw a customer case for this situation with NFS shares using the csi filestore driver. For PD I think it would only come up when trying to ROX a PD volume across namespaces, I'm not sure if that's common.
Capturing some thoughts here in line to understand your we use a staging path like , the kubelet can locate the both the driver name and the volumeId, needed during UnmountDevice call. e.g for pd csi driver which is the sha256{projects/mattcary-saikatroyc-test/zones/us-central1-c/disks/pvc--f97d-43cd-a5f1-}. This means the volume Id needs to persisted (we already do that in today ), so that the volume Id can be retrieved for UnmountDevice call. we decide to make the change at a minor boundary, then a node drain mandate would unstage any volumes before the kubelet is upgraded. this ensures any staged path for csi volume is using the new format 2 If we do not want to enforce a node drain, kubelet can first lookup json data (the reasoning is if this file does not exist, we will not be able to retrieve the volume handle from the new handle anyway), else fallback to try the old format of staging path - . TBD: Need to check mount reference logic. sha(volumeId) is expected to be unique in the cluster, if I understand. (based on the assumption any storage resource provisioned by a given driver is expected to be uniquely identifiable)
/assign
yes, your summary is correct. We persist a json, please make sure it contains anything that NodeUnstage may need.

It seems to me that people prefer option 2.2. We already do to uniquely identify VolumeAttachment. So far nobody complained about any conflict.


Question: return err } <del> return wait.Poll(time.Second, b.bindTimeout, func() (bool, error) { </del> <ins> err = wait.Poll(time.Second, b.bindTimeout, func() (bool, error) { </ins> b, err := b.checkBindings(assumedPod, bindings, claimsToProvision) return b, err }) <ins> if err != nil { pvcName := "" if len(claimsToProvision) > 0 { pvcName = claimsToProvision[0].Name } return fmt.Errorf("Failed to bind volumes: provisioning failed for PVC %q: %v", pvcName, err) } return nil </ins> } func getPodName(pod *v1.Pod) string {

What happened: Provisioning of a volume with failed and this event were created on a pod: The event does not say anything useful and only confuses users. What you expected to happen: Either no event (event on PVC about failed provisioning could be enough) or something generic about "waiting for provisioning to finish". How to reproduce it (as minimally and precisely as possible): Tested on AWS. It should be the same with GCE and other block-based in-tree volumes. StorageClass with PVC requesting RWX (so provisioning fails): pod that uses the PVC. Pod events: Anything else we need to know?: I've been able to track down what happens: Scheduler adds to the PVC Provisioner tries to create a volume for the PVC and fails. The annotation is removed in  The scheduler expects that the annotation exists here: I am not sure how to fix it though. Ignoring the error might be the easiest thing to do. Environment: Kubernetes version (use ): today's master /sig storage /sig scheduling /assign
yes, the error message is confusing. "FailedScheduling" event will be emitted if the scheduler failed to schedule the pod. How about wrapping the internal error message in function, e.g. ? Users will know the pod failed because pod volumes not bound successfully. Sometimes, users can't just wait for the scheduler to recover from the error state, manual intervention might be required. The internal reason will be useful. In this scenario, .
's suggestion sounds good to me


Question: default: jobDuration = duration.HumanDuration(obj.Status.CompletionTime.Sub(obj.Status.StartTime.Time)) } <ins> var status string if hasJobCondition(obj.Status.Conditions, batch.JobComplete) { status = "Complete" } else if hasJobCondition(obj.Status.Conditions, batch.JobFailed) { status = "Failed" } else if obj.ObjectMeta.DeletionTimestamp != nil { status = "Terminating" } else if hasJobCondition(obj.Status.Conditions, batch.JobSuspended) { status = "Suspended" } else if hasJobCondition(obj.Status.Conditions, batch.JobFailureTarget) { status = "FailureTarget" } else { status = "Running" } </ins> <del> row.Cells = append(row.Cells, obj.Name, completions, jobDuration, translateTimestampSince(obj.CreationTimestamp)) </del> <ins> row.Cells = append(row.Cells, obj.Name, status, completions, jobDuration, translateTimestampSince(obj.CreationTimestamp)) </ins> if options.Wide { names, images := layoutContainerCells(obj.Spec.Template.Spec.Containers) row.Cells = append(row.Cells, names, images, metav1.FormatLabelSelector(obj.Spec.Selector))

Suspend is a GA feature and it is big component of Kueue. It would be nice to display if jobs are suspended when doing . A new column that displays if the job is suspended would be helpful. Improve UX around suspend. Its a bit difficult to know why the job has not created pods unless you know about the suspend field.
/sig cli /wg batch /good-first-issue /help-wanted
This request has been marked as suitable for new contributors. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /sig cli /wg batch /good-first-issue /help-wanted Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc Just something I noticed as I was testing some job in JobSet. It would be nice to see which jobs are suspended when doing . Hope its not a controversial change.
cc who worked on
Not controversial at all IMO /triage accepted
/assign
Any suggestions on where to place this column? I'm thinking between completions and durations. Or, should this be displayed only when the output is set to wide?
I would put it in the default view as well. Probably before completions
Addressed in the PR:


Question: factor = 2 ) duration := base <ins> // Responsible for checking limits in resolv.conf // The limits do not have anything to do with individual pods // Since this is called in syncLoop, we don't need to call it anywhere else if kl.dnsConfigurer != nil && kl.dnsConfigurer.ResolverConfig != "" { kl.dnsConfigurer.CheckLimitsForResolvConf() } </ins> for { if err := kl.runtimeState.runtimeErrors(); err != nil { klog.Errorf("skipping pod synchronization - %v", err)

What happened: Ref , we noticed that there are some unnecessary check around limit in kubelet, specifically at: and is moving this check to outside of the pods loop make it better, but we should be looking at whether this check can exist in an even higher layer, as it doesn't make much sense in either or . cc
/sig network
/assign Pending outcome of the discussion here, I would be happy to present a PR
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/remove-triage unresolved
poke :-)
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
How about lifting the check to the beginning of Kubelet#syncLoop (outside the for loop) ? All the events emitted by CheckLimitsForResolvConf are at EventTypeWarning level (not error). The CheckLimitsForResolvConf call in HandlePodUpdates would be removed as well.


Question: kube::log::status "Testing kubectl(${version}:nodes)" <del> kube::test::get_object_assert nodes '{{range.items}}{{.id}}:{{end}}' '127.0.0.1:' </del> <ins> kube::test::get_object_assert nodes "{{range.items}}{{.$id_field}}:{{end}}" '127.0.0.1:' </ins> ########### # Minions #

This works: Are we using /var/run for testing?
Curiously travis shows a similar error message regarding /var/run/kubernetes, but it isn't failing: I suspect the /var/run/kubernetes issue is not what's causing the failure here.
However, I am able to reproduce the failure on my machine. I note that in your case, the fatal error says For me, it says The ip addresses showing through here (which appear to be in the GCE space?) are worrying.
It seems it still failing. I've run it locally and it failed with the following error: The aforementioned line: I think it is expected that there are some pods started by k8s (e.g. elasticsearch). Does it mean the test has a bug?
Strangely the same code () passes tests on travis (https://travis-)
I think you're reading it backwards. The test is expecting no pods to be started, but finding the k8s add-on pods, which is normal if it were running against a normal master. In Travis, it's a synthetic environment with no add-ons. This should be assigned to whoever the checks (on mobile.. I know I merged the PR but don't remember who :)). We need to exclude the add-ons (perhaps by finally getting around to putting them in a separate namespace).
the assertions, so CC'ing for comment.
I looked into this with and it seems that test- may fail locally depending on the kubeconfig. Some commands are using defaults for kubectl and thus sometimes they contact another cluster. I'll send a PR to fix this.


Question: "type": "string" }, "mountPropagation": { <del> "description": "mountPropagation determines how mounts are propagated from the host to container and the other way around. When not set, MountPropagationHostToContainer is used. This field is alpha in 1.8 and can be reworked or removed in a future release.", </del> <ins> "description": "mountPropagation determines how mounts are propagated from the host to container and the other way around. When not set, MountPropagationHostToContainer is used. This field is beta in 1.10.", </ins> "type": "string" }, "name": {


Question: import ( "fmt" <ins> "net/http" </ins> "os" "os/exec" "reflect"

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): No What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): kubectl "does not allow access" Is this a BUG REPORT or FEATURE REQUEST? (choose one): Bug report <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Environment: Cloud provider or hardware configuration: GCP GKE 1.4.5 OS (e.g. from /etc/os-release): Not sure, using gci Kernel (e.g. ): Not sure, using gci Install tools: GKE web UI Others: N/A What happened: In juggling multiple gcloud accounts with GKE clusters I accidentally used with the wrong application default credentials and it cached bad creds in After pointing gcloud at the right application default credentials and re-running , it continued trying and failing to use the bad creds cached in ; at this point I was stuck and confused for a few hours To repair, I had to manually edit to remove , and then started working again (or else wait ~1h for the access token to expire and refresh itself) What you expected to happen: After pointing gcloud at the right application default credentials should just work, or at least provide an error msg to the user that can help them figure out that they need to revoke the bad access token and how to do it How to reproduce it (as minimally and precisely as possible): Good scenario: just works Bad scenario: user makes one mistake and gets stuck
+1 Removing helped, thanks.
I can look into fixing this if no-one else has already made a start on it?
feel free to take it, thanks!

Ok, I think I have an understanding of what is happening call to is storing credentials in a json file in a known location. a subsequent credential-requiring kubectl command, the GCP plugin is being loaded, and the application default credentials are being exchanged for a token. That token is then cached in the .kube/config file. So where do things go wrong? As far as I can tell, if I switch to a cluster-2 without refreshing the application default credentials from it's separate account, those existing credentials will still be exchanged successfully for an oauth2 token and cached against cluster-2. Those tokens will then fail against a call to cluster-2 - because they were intended for cluster-1. The only sure-fire fix I can think of would be to ensure the application default credentials client ID is stored in users[].user.authprovider. Doing this would at least enable the ability to verify that the default credentials were valid for that particular cluster, and if not, prompt the user with a message to correctly aquire the application default credentials for it. Lets call this option 1. Of course, there is a problem here with determining what the Client ID is in the first place. I think this would require that a call to would store the client ID in the generated config. This would require exposing the client id from oauth2/google. Otherwise, perhaps storing the client ID the first time around would be enough - assuming the user at least manages to authenticate successfully against the cluster in the first instance. Then only oauth2/google would require a change, and not the gcloud command. The other option (option 2) would be, upon failure, check if is true and if-so, prompt the user that their application default credentials may be invalid for the target cluster, and to either prompt or clear the currently cached credentials in .kube/config. This would probably mean extending the auth provider interface such that a function may be provided that can be called upon failure that enables the auth provider plugin to do some checking of the state and to provide a reasonable message as to what might be wrong with the credentials that have been provided. I feel like option 2 is probably the better option in the long term. Different auth plugins are likely to have different failure modes. Providing a way to communicate back to the user about possible edge cases seems a little more flexible. Any thoughts?

edit: On closer inspection this seems possible by wrapping the oauth2.Transport call in WrapTransport. I would then be able to check for an authorization failure and provide a reasonable error message based on the existing configuration.
Just come across this, its particularly strange to have to edit the config file, when there is a folder in the folder.
Is it really fixed? I still can reproduce it on k8s 1.14 with description provided in initial comment.
I also reproduced this, unfortunately. Tried to use kubectl to access a cluster from an unauthorised account. Then, switched to the authorised account but the invalid authentication was cached. To resolve it, I set the to an expired date.
Believe I'm hitting this issue as well
I just ran into this as well, gcloud 311.0.0, kubectl 1.18.6.
I've got the same issue. I need to change the expiry time to 1 year back as pointed out.
Solution that usually works for me So it will use different credentials when switching contexts
You'll need  installed ().


Question: if port != "443" { t.Fatalf("expected 443 as port, got %q", port) } <ins> } func TestLoopbackHostPortIPv6(t *testing.T) { ipv6, _, err := isIPv6LoopbackSupported() if err != nil { t.Fatalf("fail to enumerate network interface, %s", err) } if !ipv6 { t.Fatalf("no ipv6 loopback interface") } </ins> <del> host, port, err = LoopbackHostPort("[ff06:0:0:0:0:0:0:c3]:443") </del> <ins> host, port, err := LoopbackHostPort("[ff06:0:0:0:0:0:0:c3]:443") </ins> if err != nil { t.Fatalf("unexpected error: %v", err) }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: LoopbackHostPort code is calling the which depends on the machine on which this test is running. Especially in this particular testase it looks for the ipv6 ip address which does not exist because the machine is not configured for ipv6. Because of the LoopbackHostPort function bahaves based on the interfaces configured, test will fail if machine is not really configured with ipv6. Need to rewrite the testcase to handle such situation. What you expected to happen: Testcase should pass
/sig network
/cc
/triage unresolved I am a bot run by
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
/remove-triage unresolved


Question: <del> /* Copyright 2016 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package capabilities import ( "reflect" "testing" api "k8s.io/kubernetes/pkg/apis/core" "k8s.io/kubernetes/pkg/apis/extensions" ) func TestGenerateAdds(t *testing.T) { tests := map[string]struct { defaultAddCaps []api.Capability containerCaps  *api.Capabilities expectedCaps   *api.Capabilities }{ "no required, no container requests": {}, "no required, no container requests, non-nil": { containerCaps: &api.Capabilities{}, expectedCaps:  &api.Capabilities{}, }, "required, no container requests": { defaultAddCaps: []api.Capability{"foo"}, expectedCaps: &api.Capabilities{ Add: []api.Capability{"foo"}, }, }, "required, container requests add required": { defaultAddCaps: []api.Capability{"foo"}, containerCaps: &api.Capabilities{ Add: []api.Capability{"foo"}, }, expectedCaps: &api.Capabilities{ Add: []api.Capability{"foo"}, }, }, "multiple required, container requests add required": { defaultAddCaps: []api.Capability{"foo", "bar", "baz"}, containerCaps: &api.Capabilities{ Add: []api.Capability{"foo"}, }, expectedCaps: &api.Capabilities{ Add: []api.Capability{"bar", "baz", "foo"}, }, }, "required, container requests add non-required": { defaultAddCaps: []api.Capability{"foo"}, containerCaps: &api.Capabilities{ Add: []api.Capability{"bar"}, }, expectedCaps: &api.Capabilities{ Add: []api.Capability{"bar", "foo"}, }, }, "generation does not mutate unnecessarily": { defaultAddCaps: []api.Capability{"foo", "bar"}, containerCaps: &api.Capabilities{ Add: []api.Capability{"foo", "foo", "bar", "baz"}, }, expectedCaps: &api.Capabilities{ Add: []api.Capability{"foo", "foo", "bar", "baz"}, }, }, "generation dedupes": { defaultAddCaps: []api.Capability{"foo", "bar"}, containerCaps: &api.Capabilities{ Add: []api.Capability{"foo", "baz"}, }, expectedCaps: &api.Capabilities{ Add: []api.Capability{"bar", "baz", "foo"}, }, }, "generation is case sensitive - will not dedupe": { defaultAddCaps: []api.Capability{"foo"}, containerCaps: &api.Capabilities{ Add: []api.Capability{"FOO"}, }, expectedCaps: &api.Capabilities{ Add: []api.Capability{"FOO", "foo"}, }, }, } for k, v := range tests { container := &api.Container{ SecurityContext: &api.SecurityContext{ Capabilities: v.containerCaps, }, } strategy, err := NewDefaultCapabilities(v.defaultAddCaps, nil, nil) if err != nil { t.Errorf("%s failed: %v", k, err) continue } generatedCaps, err := strategy.Generate(nil, container) if err != nil { t.Errorf("%s failed generating: %v", k, err) continue } if v.expectedCaps == nil && generatedCaps != nil { t.Errorf("%s expected nil caps to be generated but got %v", k, generatedCaps) continue } if !reflect.DeepEqual(v.expectedCaps, generatedCaps) { t.Errorf("%s did not generate correctly.  Expected: %#v, Actual: %#v", k, v.expectedCaps, generatedCaps) } } } func TestGenerateDrops(t *testing.T) { tests := map[string]struct { defaultAddCaps   []api.Capability requiredDropCaps []api.Capability containerCaps    *api.Capabilities expectedCaps     *api.Capabilities }{ "no required, no container requests": { expectedCaps: nil, }, "no required, no container requests, non-nil": { containerCaps: &api.Capabilities{}, expectedCaps:  &api.Capabilities{}, }, "required drops are defaulted": { requiredDropCaps: []api.Capability{"foo"}, expectedCaps: &api.Capabilities{ Drop: []api.Capability{"foo"}, }, }, "required drops are defaulted when making container requests": { requiredDropCaps: []api.Capability{"baz"}, containerCaps: &api.Capabilities{ Drop: []api.Capability{"foo", "bar"}, }, expectedCaps: &api.Capabilities{ Drop: []api.Capability{"bar", "baz", "foo"}, }, }, "required drops do not mutate unnecessarily": { requiredDropCaps: []api.Capability{"baz"}, containerCaps: &api.Capabilities{ Drop: []api.Capability{"foo", "bar", "baz"}, }, expectedCaps: &api.Capabilities{ Drop: []api.Capability{"foo", "bar", "baz"}, }, }, "can drop a required add": { defaultAddCaps: []api.Capability{"foo"}, containerCaps: &api.Capabilities{ Drop: []api.Capability{"foo"}, }, expectedCaps: &api.Capabilities{ Drop: []api.Capability{"foo"}, }, }, "can drop non-required add": { defaultAddCaps: []api.Capability{"foo"}, containerCaps: &api.Capabilities{ Drop: []api.Capability{"bar"}, }, expectedCaps: &api.Capabilities{ Add:  []api.Capability{"foo"}, Drop: []api.Capability{"bar"}, }, }, "defaulting adds and drops, dropping a required add": { defaultAddCaps:   []api.Capability{"foo", "bar", "baz"}, requiredDropCaps: []api.Capability{"abc"}, containerCaps: &api.Capabilities{ Drop: []api.Capability{"foo"}, }, expectedCaps: &api.Capabilities{ Add:  []api.Capability{"bar", "baz"}, Drop: []api.Capability{"abc", "foo"}, }, }, "generation dedupes": { requiredDropCaps: []api.Capability{"baz", "foo"}, containerCaps: &api.Capabilities{ Drop: []api.Capability{"bar", "foo"}, }, expectedCaps: &api.Capabilities{ Drop: []api.Capability{"bar", "baz", "foo"}, }, }, "generation is case sensitive - will not dedupe": { requiredDropCaps: []api.Capability{"bar"}, containerCaps: &api.Capabilities{ Drop: []api.Capability{"BAR"}, }, expectedCaps: &api.Capabilities{ Drop: []api.Capability{"BAR", "bar"}, }, }, } for k, v := range tests { container := &api.Container{ SecurityContext: &api.SecurityContext{ Capabilities: v.containerCaps, }, } strategy, err := NewDefaultCapabilities(v.defaultAddCaps, v.requiredDropCaps, nil) if err != nil { t.Errorf("%s failed: %v", k, err) continue } generatedCaps, err := strategy.Generate(nil, container) if err != nil { t.Errorf("%s failed generating: %v", k, err) continue } if v.expectedCaps == nil && generatedCaps != nil { t.Errorf("%s expected nil caps to be generated but got %#v", k, generatedCaps) continue } if !reflect.DeepEqual(v.expectedCaps, generatedCaps) { t.Errorf("%s did not generate correctly.  Expected: %#v, Actual: %#v", k, v.expectedCaps, generatedCaps) } } } func TestValidateAdds(t *testing.T) { tests := map[string]struct { defaultAddCaps []api.Capability allowedCaps    []api.Capability containerCaps  *api.Capabilities expectedError  string }{ // no container requests "no required, no allowed, no container requests": {}, "no required, allowed, no container requests": { allowedCaps: []api.Capability{"foo"}, }, "required, no allowed, no container requests": { defaultAddCaps: []api.Capability{"foo"}, expectedError:  `capabilities: Invalid value: "null": required capabilities are not set on the securityContext`, }, // container requests match required "required, no allowed, container requests valid": { defaultAddCaps: []api.Capability{"foo"}, containerCaps: &api.Capabilities{ Add: []api.Capability{"foo"}, }, }, "required, no allowed, container requests invalid": { defaultAddCaps: []api.Capability{"foo"}, containerCaps: &api.Capabilities{ Add: []api.Capability{"bar"}, }, expectedError: `capabilities.add: Invalid value: "bar": capability may not be added`, }, // container requests match allowed "no required, allowed, container requests valid": { allowedCaps: []api.Capability{"foo"}, containerCaps: &api.Capabilities{ Add: []api.Capability{"foo"}, }, }, "no required, all allowed, container requests valid": { allowedCaps: []api.Capability{extensions.AllowAllCapabilities}, containerCaps: &api.Capabilities{ Add: []api.Capability{"foo"}, }, }, "no required, allowed, container requests invalid": { allowedCaps: []api.Capability{"foo"}, containerCaps: &api.Capabilities{ Add: []api.Capability{"bar"}, }, expectedError: `capabilities.add: Invalid value: "bar": capability may not be added`, }, // required and allowed "required, allowed, container requests valid required": { defaultAddCaps: []api.Capability{"foo"}, allowedCaps:    []api.Capability{"bar"}, containerCaps: &api.Capabilities{ Add: []api.Capability{"foo"}, }, }, "required, allowed, container requests valid allowed": { defaultAddCaps: []api.Capability{"foo"}, allowedCaps:    []api.Capability{"bar"}, containerCaps: &api.Capabilities{ Add: []api.Capability{"bar"}, }, }, "required, allowed, container requests invalid": { defaultAddCaps: []api.Capability{"foo"}, allowedCaps:    []api.Capability{"bar"}, containerCaps: &api.Capabilities{ Add: []api.Capability{"baz"}, }, expectedError: `capabilities.add: Invalid value: "baz": capability may not be added`, }, "validation is case sensitive": { defaultAddCaps: []api.Capability{"foo"}, containerCaps: &api.Capabilities{ Add: []api.Capability{"FOO"}, }, expectedError: `capabilities.add: Invalid value: "FOO": capability may not be added`, }, } for k, v := range tests { strategy, err := NewDefaultCapabilities(v.defaultAddCaps, nil, v.allowedCaps) if err != nil { t.Errorf("%s failed: %v", k, err) continue } errs := strategy.Validate(nil, nil, v.containerCaps) if v.expectedError == "" && len(errs) > 0 { t.Errorf("%s should have passed but had errors %v", k, errs) continue } if v.expectedError != "" && len(errs) == 0 { t.Errorf("%s should have failed but received no errors", k) continue } if len(errs) == 1 && errs[0].Error() != v.expectedError { t.Errorf("%s should have failed with %v but received %v", k, v.expectedError, errs[0]) continue } if len(errs) > 1 { t.Errorf("%s should have failed with at most one error, but received %v: %v", k, len(errs), errs) } } } func TestValidateDrops(t *testing.T) { tests := map[string]struct { requiredDropCaps []api.Capability containerCaps    *api.Capabilities expectedError    string }{ // no container requests "no required, no container requests": {}, "required, no container requests": { requiredDropCaps: []api.Capability{"foo"}, expectedError:    `capabilities: Invalid value: "null": required capabilities are not set on the securityContext`, }, // container requests match required "required, container requests valid": { requiredDropCaps: []api.Capability{"foo"}, containerCaps: &api.Capabilities{ Drop: []api.Capability{"foo"}, }, }, "required, container requests invalid": { requiredDropCaps: []api.Capability{"foo"}, containerCaps: &api.Capabilities{ Drop: []api.Capability{"bar"}, }, expectedError: `capabilities.drop: Invalid value: []core.Capability{"bar"}: foo is required to be dropped but was not found`, }, "validation is case sensitive": { requiredDropCaps: []api.Capability{"foo"}, containerCaps: &api.Capabilities{ Drop: []api.Capability{"FOO"}, }, expectedError: `capabilities.drop: Invalid value: []core.Capability{"FOO"}: foo is required to be dropped but was not found`, }, } for k, v := range tests { strategy, err := NewDefaultCapabilities(nil, v.requiredDropCaps, nil) if err != nil { t.Errorf("%s failed: %v", k, err) continue } errs := strategy.Validate(nil, nil, v.containerCaps) if v.expectedError == "" && len(errs) > 0 { t.Errorf("%s should have passed but had errors %v", k, errs) continue } if v.expectedError != "" && len(errs) == 0 { t.Errorf("%s should have failed but received no errors", k) continue } if len(errs) == 1 && errs[0].Error() != v.expectedError { t.Errorf("%s should have failed with %v but received %v", k, v.expectedError, errs[0]) continue } if len(errs) > 1 { t.Errorf("%s should have failed with at most one error, but received %v: %v", k, len(errs), errs) } } } </del>

The name of this file is odd: The file context has nothing to do with Is this a BUG REPORT or FEATURE REQUEST?: /assign


Question: expectKnownTerminated: true, }, { <del> name: "a pod that is terminal and has never started is finished immediately if the runtime has a cached terminal state", </del> <ins> name: "a pod that is terminal and has never started advances to finished if the runtime has a cached terminal state", </ins> update: UpdatePodOptions{ UpdateType: kubetypes.SyncPodCreate, Pod:        newPodWithPhase("1", "done-pod", v1.PodSucceeded), }, runtimeStatus: &kubecontainer.PodStatus{ /* we know about this pod */ }, <del> expect: &podSyncStatus{ </del> <ins> expectBeforeWorker: &podSyncStatus{ fullname:      "done-pod_ns", syncedAt:      time.Unix(1, 0), terminatingAt: time.Unix(1, 0), terminatedAt:  time.Unix(1, 0), pendingUpdate: &UpdatePodOptions{ UpdateType: kubetypes.SyncPodCreate, Pod:        newPodWithPhase("1", "done-pod", v1.PodSucceeded), }, finished:           false, // Should be marked as not finished initially (to ensure `SyncTerminatedPod` will run) and status will progress to terminated. startedTerminating: true, working:            true, }, expect: hasContext(&podSyncStatus{ </ins> fullname:           "done-pod_ns", syncedAt:           time.Unix(1, 0), terminatingAt:      time.Unix(1, 0), terminatedAt:       time.Unix(1, 0), <ins> startedAt:          time.Unix(3, 0), </ins> startedTerminating: true, finished:           true, <ins> activeUpdate: &UpdatePodOptions{ UpdateType: kubetypes.SyncPodSync, Pod:        newPodWithPhase("1", "done-pod", v1.PodSucceeded), }, </ins> // if we have never seen the pod before, a restart makes no sense restartRequested: false, <del> }, </del> <ins> }), </ins> expectKnownTerminated: true, }, {

New TestGrid alerts at since commit/. Found 2 failing rows in node-kubelet-serial-containerd: 2 new, 0 other 2 new Test Failures in node-kubelet-serial-containerd. Test has failed 3 times since it last passed at commit/. It started failing between commit/ (Fri Mar 24 12:02 PDT) and commit/ (Fri Mar 24 16:03 PDT): . Test has failed 3 times since it last passed at commit/. It started failing between commit/ (Fri Mar 24 12:02 PDT) and commit/ (Fri Mar 24 16:03 PDT): . Not sure suspect No response /sig node release
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/kind flake /remove-kind failing-test See
/remove-sig release
Taking a look at The test failed due to: so it means that the pod was not terminated and deleted properly in the test that ran prior. The is the one creating this pod, but I would expect that as part of the test teardown, the namespace is deleted (and thus the pod should be deleted).
The services log show that the kubelet never sent the request to delete the pod, that is strange...
From the build-logs, the test execution for when the pod was created: Kubelet logs for the pod - Full logs are here -
Here's the termination part of the logs:
It looks like to me the issue here is that the kubelet is being restarted in a bit of an unfortunate time resulting in at least two related issues: The pod status has been updated on the api-server to terminal just prior to the kubelet restarting. The status manager did not yet complete . Then after kubelet restarts and comes back up, it attempts to add this pod , (even though it is in terminal phase). This seems like the first issue: if the pod phase is terminal on the API server, we probably should not add it back.

The pod worker receives the , but the pod worker quickly terminates (). I am assuming this is because of , i.e. based on pod cache lookup the pod is terminal. The side affect of this is that the status manager never receives the signal that the pod is (which is normally done as part of . This is because in this case, we never call which sets to be true. As a result, I think will never return true, so this pod will never be able to be deleted.
I have made a test which will reproduce this behavior in a node e2e, see It can be reproduced by starting a pod and then deleting it and triggering kubelet restart at the right time. A watch is setup to trigger a kubelet restart as soon as the pod enters the succeeded phase. If the kubelet restart happens before the kubelet is able to delete the pod on the API server, but after the phase is updated, then we trigger the issue. The pod will be re- to kubelet after restart, but will never be deleted on the API server despite it having a deletion timestamp set and being in terminal phase. Here are logs from the and running the above repro node e2e test. Here is the kubelet logs portion prior to the restart: and after restart: It also looks like we are hitting another invalid state in based on the log assertion: I tried the same test on 1.26 and it passes. This is because on 1.26, the status manager phase to terminal happens at roughly at the same time when we delete the pod. On 1.27+, after , the status phase to terminal can happen earlier and the status manager can skip deleting the pod because . In this case, after start, will never be true, so the pod never will get deleted.
But in general pods can be in terminal phase and not be deleted (like plain pods and pods from jobs). They would eventually be deleted by the pod GC otherwise. Is this a bug in the device plugin?
That is true, but the issue here is that the pods have a deletion timestamp set as well. If the deletion timestamp is set and the kubelet is up, it is the kubelet's responsibility to terminate and delete those pods.

This doesn't have anything with device plugin, rather we caught the issue due to the device plugin test which asserts that all pods are terminated and deleted prior to starting the test.
I have a WIP fix for this issue in
That makes sense
Should we aim 1.27 with this fix? This seems like an important regression.
I discussed this with yesterday and we agreed that this can wait until 1.27.1 Clients should be able to handle terminated pods. Worst case scenario, the pod GC would take care of these pods eventually.
Cherrypicked to v1.27 (to be released in 1.27.2) in


Question: sort.Strings(entries) return strings.Join(entries, ", ") } <del> func makeBusyboxDeviceRequiringPod(resourceName, cmd string) *v1.Pod { podName := "device-manager-test-" + string(uuid.NewUUID()) rl := v1.ResourceList{ v1.ResourceName(resourceName): *resource.NewQuantity(2, resource.DecimalSI), } return &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name: podName, }, Spec: v1.PodSpec{ RestartPolicy: v1.RestartPolicyNever, Containers: []v1.Container{{ Image: busyboxImage, Name:  podName, // Runs the specified command in the test pod. Command: []string{"sh", "-c", cmd}, Resources: v1.ResourceRequirements{ Limits:   rl, Requests: rl, }, }}, }, } } // BeReady verifies that a node is ready and devices have registered. func BeReady() types.GomegaMatcher { return gomega.And( // This additional matcher checks for the final error condition. gcustom.MakeMatcher(func(ready bool) (bool, error) { if !ready { return false, fmt.Errorf("Expected node to be ready=%t", ready) } return true, nil }), BeInReadyPhase(true), ) } // BeInReadyPhase matches if node is ready i.e. ready is true. func BeInReadyPhase(isReady bool) types.GomegaMatcher { return gcustom.MakeMatcher(func(ready bool) (bool, error) { return ready == isReady, nil }).WithTemplate("Expected Node Ready {{.To}} be in {{format .Data}}nGot instead:n{{.FormattedActual}}").WithTemplateData(isReady) } func isNodeReadyWithSampleResources(ctx context.Context, f *framework.Framework) (bool, error) { node, ready := getLocalTestNode(ctx, f) if !ready { return false, fmt.Errorf("Expected node to be ready=%t", ready) } if CountSampleDeviceCapacity(node) <= 0 { return false, fmt.Errorf("Expected devices to be advertised") } return true, nil } // HaveAllocatableDevices verifies that a node has allocatable devices. func HaveAllocatableDevices() types.GomegaMatcher { return gomega.And( // This additional matcher checks for the final error condition. gcustom.MakeMatcher(func(hasAllocatable bool) (bool, error) { if !hasAllocatable { return false, fmt.Errorf("Expected node to be have allocatable devices=%t", hasAllocatable) } return true, nil }), hasAllocatable(true), ) } // hasAllocatable matches if node is ready i.e. ready is true. func hasAllocatable(hasAllocatable bool) types.GomegaMatcher { return gcustom.MakeMatcher(func(hasAllocatableDevices bool) (bool, error) { return hasAllocatableDevices == hasAllocatable, nil }).WithTemplate("Expected Node with allocatable {{.To}} be in {{format .Data}}nGot instead:n{{.FormattedActual}}").WithTemplateData(hasAllocatable) } func isNodeReadyWithAllocatableSampleResources(ctx context.Context, f *framework.Framework, devCount int64) (bool, error) { node, ready := getLocalTestNode(ctx, f) if !ready { return false, fmt.Errorf("Expected node to be ready=%t", ready) } if CountSampleDeviceCapacity(node) != devCount { return false, fmt.Errorf("Expected devices capacity to be: %d", devCount) } if CountSampleDeviceAllocatable(node) != devCount { return false, fmt.Errorf("Expected devices allocatable to be: %d", devCount) } return true, nil } func isNodeReadyWithoutSampleResources(ctx context.Context, f *framework.Framework) (bool, error) { node, ready := getLocalTestNode(ctx, f) if !ready { return false, fmt.Errorf("Expected node to be ready=%t", ready) } if CountSampleDeviceCapacity(node) > 0 { return false, fmt.Errorf("Expected devices to be not present") } return true, nil } // HaveFailedWithAdmissionError verifies that a pod fails at admission. func HaveFailedWithAdmissionError() types.GomegaMatcher { return gomega.And( gcustom.MakeMatcher(func(hasFailed bool) (bool, error) { if !hasFailed { return false, fmt.Errorf("Expected pod to have failed=%t", hasFailed) } return true, nil }), hasFailed(true), ) } // hasFailed matches if pod has failed. func hasFailed(hasFailed bool) types.GomegaMatcher { return gcustom.MakeMatcher(func(hasPodFailed bool) (bool, error) { return hasPodFailed == hasFailed, nil }).WithTemplate("Expected Pod failed {{.To}} be in {{format .Data}}nGot instead:n{{.FormattedActual}}").WithTemplateData(hasFailed) } func getPod(ctx context.Context, f *framework.Framework, podName string) (bool, error) { pod, err := e2epod.NewPodClient(f).Get(ctx, podName, metav1.GetOptions{}) if err != nil { return false, fmt.Errorf("Expected node to get pod=%q got err=%q", pod.Name, err) } expectedStatusReason := "UnexpectedAdmissionError" expectedStatusMessage := "Allocate failed due to can't allocate unhealthy devices" // This additional matcher checks for the final error condition. if pod.Status.Phase != v1.PodFailed { return false, fmt.Errorf("Expected pod to reach phase %q, got final phase %q instead.", v1.PodFailed, pod.Status.Phase) } if pod.Status.Reason != expectedStatusReason { return false, fmt.Errorf("Expected pod status reason to be %q, got %q instead.", expectedStatusReason, pod.Status.Reason) } if !strings.Contains(pod.Status.Message, expectedStatusMessage) { return false, fmt.Errorf("Expected pod status reason to contain %q, got %q instead.", expectedStatusMessage, pod.Status.Message) } return true, nil } </del>

Failing: ` Failing since . Last passing: First failing:
The diff () shows that went in which looks related
/sig node
/kind regression /kind bug /triage accepted


Question: psp, err = f.ClientSet.ExtensionsV1beta1().PodSecurityPolicies().Create(psp) ExpectNoError(err, "Failed to create PSP %s", podSecurityPolicyPrivileged) <del> // Create the Role to bind it to the namespace. _, err = f.ClientSet.RbacV1beta1().ClusterRoles().Create(&rbacv1beta1.ClusterRole{ ObjectMeta: metav1.ObjectMeta{Name: podSecurityPolicyPrivileged}, Rules: []rbacv1beta1.PolicyRule{{ APIGroups:     []string{"extensions"}, Resources:     []string{"podsecuritypolicies"}, ResourceNames: []string{podSecurityPolicyPrivileged}, Verbs:         []string{"use"}, }}, }) ExpectNoError(err, "Failed to create PSP role") </del> <ins> if IsRBACEnabled(f) { // Create the Role to bind it to the namespace. _, err = f.ClientSet.RbacV1beta1().ClusterRoles().Create(&rbacv1beta1.ClusterRole{ ObjectMeta: metav1.ObjectMeta{Name: podSecurityPolicyPrivileged}, Rules: []rbacv1beta1.PolicyRule{{ APIGroups:     []string{"extensions"}, Resources:     []string{"podsecuritypolicies"}, ResourceNames: []string{podSecurityPolicyPrivileged}, Verbs:         []string{"use"}, }}, }) ExpectNoError(err, "Failed to create PSP role") } </ins> }) <del> By(fmt.Sprintf("Binding the %s PodSecurityPolicy to the default service account in %s", podSecurityPolicyPrivileged, namespace)) BindClusterRoleInNamespace(f.ClientSet.RbacV1beta1(), podSecurityPolicyPrivileged, namespace, rbacv1beta1.Subject{ Kind:      rbacv1beta1.ServiceAccountKind, Namespace: namespace, Name:      "default", }) ExpectNoError(WaitForNamedAuthorizationUpdate(f.ClientSet.AuthorizationV1beta1(), serviceaccount.MakeUsername(namespace, "default"), namespace, "use", podSecurityPolicyPrivileged, schema.GroupResource{Group: "extensions", Resource: "podsecuritypolicies"}, true)) </del> <ins> if IsRBACEnabled(f) { By(fmt.Sprintf("Binding the %s PodSecurityPolicy to the default service account in %s", podSecurityPolicyPrivileged, namespace)) BindClusterRoleInNamespace(f.ClientSet.RbacV1beta1(), podSecurityPolicyPrivileged, namespace, rbacv1beta1.Subject{ Kind:      rbacv1beta1.ServiceAccountKind, Namespace: namespace, Name:      "default", }) ExpectNoError(WaitForNamedAuthorizationUpdate(f.ClientSet.AuthorizationV1beta1(), serviceaccount.MakeUsername(namespace, "default"), namespace, "use", podSecurityPolicyPrivileged, schema.GroupResource{Group: "extensions", Resource: "podsecuritypolicies"}, true)) } </ins> }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: (This is to have an accompanying issue for the PR ) Creating privileged PSPs during e2e tests depends on RBAC being enabled in the target cluster. However it is possible to use PSPs without having RBAC enabled thus only enable creating PSPs during e2e tests if both PSP and RBAC is enabled. What you expected to happen: RBAC role binding should only be created when RBAC is enabled in the cluster being tested.
There are no sig labels on this issue. Please add a sig label. <detailsA sig label can be by either: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. See the . The in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: // If there are still volume directories, do not delete directory volumePaths, err := kl.getPodVolumePathListFromDisk(uid) if err != nil { <del> glog.Errorf("Orphaned pod %q found, but error %v occurred during reading volume dir from disk", uid, err) </del> <ins> orphanVolumeErrors = append(orphanVolumeErrors, fmt.Errorf("Orphaned pod %q found, but error %v occurred during reading volume dir from disk", uid, err)) </ins> continue } if len(volumePaths) > 0 { <del> glog.Errorf("Orphaned pod %q found, but volume paths are still present on disk.", uid) </del> <ins> orphanVolumeErrors = append(orphanVolumeErrors, fmt.Errorf("Orphaned pod %q found, but volume paths are still present on disk.", uid)) </ins> continue } glog.V(3).Infof("Orphaned pod %q found, removing", uid) if err := removeall.RemoveAllOneFilesystem(kl.mounter, kl.getPodDir(uid)); err != nil { glog.Errorf("Failed to remove orphaned pod %q dir; err: %v", uid, err) <del> errlist = append(errlist, err) </del> <ins> orphanRemovalErrors = append(orphanRemovalErrors, err) } } logSpew := func(errs []error) { if len(errs) > 0 { glog.Errorf("%v : There were a total of %v errors similar to this.  Turn up verbosity to see them.", errs[0], len(errs)) for _, err := range errs { glog.V(5).Infof("Orphan pod: %v", err) } </ins> } } <del> return utilerrors.NewAggregate(errlist) </del> <ins> logSpew(orphanVolumeErrors) logSpew(orphanRemovalErrors) return utilerrors.NewAggregate(orphanRemovalErrors) </ins> }

What we see in if any mismatches for volumes are around: The Problem: Obviously this can get pretty large. Since theres not much variability, it seems like rolling up all the orphanXYZ messages would be more useful and readable. Note this runs in the sync loop, so it happens every few (2) seconds for me. Obviously we see this worse on larger nodes (100 * n) / before but , I've only reproduced this on a local cluster for this PR.