Question: <del> /* Copyright 2022 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package e2enode import ( "context" "fmt" "time" v1 "k8s.io/api/core/v1" "k8s.io/apimachinery/pkg/api/resource" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/kubernetes/test/e2e/framework" e2epod "k8s.io/kubernetes/test/e2e/framework/pod" admissionapi "k8s.io/pod-security-admission/api" "github.com/onsi/ginkgo/v2" ) type testCase struct { podSpec                *v1.Pod oomTargetContainerName string } const PodOOMKilledTimeout = 2 * time.Minute var _ = SIGDescribe("OOMKiller [LinuxOnly] [NodeConformance]", func() { f := framework.NewDefaultFramework("oomkiller-test") f.NamespacePodSecurityEnforceLevel = admissionapi.LevelPrivileged containerName := "oomkill-target-container" oomPodSpec := getOOMTargetPod("oomkill-target-pod", containerName) runOomKillerTest(f, testCase{podSpec: oomPodSpec, oomTargetContainerName: containerName}) }) func runOomKillerTest(f *framework.Framework, testCase testCase) { ginkgo.Context("", func() { ginkgo.BeforeEach(func() { ginkgo.By("setting up the pod to be used in the test") e2epod.NewPodClient(f).Create(context.TODO(), testCase.podSpec) }) ginkgo.It("The containers terminated by OOM killer should have the reason set to OOMKilled", func() { ginkgo.By("Waiting for the pod to be failed") e2epod.WaitForPodTerminatedInNamespace(context.TODO(), f.ClientSet, testCase.podSpec.Name, "", f.Namespace.Name) ginkgo.By("Fetching the latest pod status") pod, err := f.ClientSet.CoreV1().Pods(f.Namespace.Name).Get(context.TODO(), testCase.podSpec.Name, metav1.GetOptions{}) framework.ExpectNoError(err, "Failed to get the recent pod object for name: %q", pod.Name) ginkgo.By("Verifying the OOM target container has the expected reason") verifyReasonForOOMKilledContainer(pod, testCase.oomTargetContainerName) }) ginkgo.AfterEach(func() { ginkgo.By(fmt.Sprintf("deleting pod: %s", testCase.podSpec.Name)) e2epod.NewPodClient(f).DeleteSync(context.TODO(), testCase.podSpec.Name, metav1.DeleteOptions{}, framework.PodDeleteTimeout) }) }) } func verifyReasonForOOMKilledContainer(pod *v1.Pod, oomTargetContainerName string) { container := e2epod.FindContainerStatusInPod(pod, oomTargetContainerName) if container == nil { framework.Failf("OOM target pod %q, container %q does not have the expected state terminated", pod.Name, container.Name) } if container.State.Terminated == nil { framework.Failf("OOM target pod %q, container %q is not in the terminated state", pod.Name, container.Name) } framework.ExpectEqual(container.State.Terminated.ExitCode, int32(137), fmt.Sprintf("pod: %q, container: %q has unexpected exitCode: %q", pod.Name, container.Name, container.State.Terminated.ExitCode)) framework.ExpectEqual(container.State.Terminated.Reason, "OOMKilled", fmt.Sprintf("pod: %q, container: %q has unexpected reason: %q", pod.Name, container.Name, container.State.Terminated.Reason)) } func getOOMTargetPod(podName string, ctnName string) *v1.Pod { return &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name: podName, }, Spec: v1.PodSpec{ RestartPolicy: v1.RestartPolicyNever, Containers: []v1.Container{ getOOMTargetContainer(ctnName), }, }, } } func getOOMTargetContainer(name string) v1.Container { return v1.Container{ Name:  name, Image: busyboxImage, Command: []string{ "sh", "-c", // use the dd tool to attempt to allocate 20M in a block which exceeds the limit "sleep 5 && dd if=/dev/zero of=/dev/null bs=20M", }, Resources: v1.ResourceRequirements{ Requests: v1.ResourceList{ v1.ResourceMemory: resource.MustParse("15Mi"), }, Limits: v1.ResourceList{ v1.ResourceMemory: resource.MustParse("15Mi"), }, }, } } </del>

Follow up on meeting of 28-03-2022 - In the meeting was specified that this most likely won't need a KEP and should be started with a Issue I see that the kernel in cgroups v2 has an option which would make all the pids of a given container to get killed on OOM: Some workloads that fork the process in the container, when the OOM kicks in, it kills only one process of the cgroup, causing a lot of noise due to process failures but health checks being successfull. We think that for our scenarios, killing all the pids in the containter would be better.
/cc
The way I’d summarize this issue with is: A small number of important workloads handle processes and OOM correctly (nginx, mature multi process systems like DBs) - we should not prevent these workloads from benefitting from per process OOM. A much larger and more diverse set of workloads are unlikely to handle OOM of multi process behavior incorrectly (bash child processes, especially in entrypoint scripts) - these users are unlikely to test OOM behavior in advance and thus are much more vulnerable to behaving incorrectly if we don’t default to killing the whole container. That also apples to containers with shared process namespaces. We generally bias towards the later group (low expertise / high diversity / low frequency / unlikely to read documentation) when designing our pod semantics, while allowing advanced users to set optional flags.
/cc
/cc
/sig node
/triage accepted /priority important-longterm


Question: // - volumes bound to another claim // - volumes whose labels don't match the claim's selector, if specified // - volumes in Class that is not requested <ins> // - volumes whose NodeAffinity does not match the node </ins> if volume.Spec.ClaimRef != nil { continue } else if selector != nil && !selector.Matches(labels.Set(volume.Labels)) {

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /assign What happened: Volume got bound to PV on node "kubernetes-minion-group-82c8" When the pod went back through scheduling, the predicate returned success for other nodes even though we should have updated the PV cache to point to the PVC: I think in , instead of continuing if the node affinity doesn't match, it should fail if the PV is prebound.
This causes two issues: On the second pass, the scheduler may choose a different node. Then in Assume/Bind volumes, we'll also prebind a second set of PVs. If the scheduler chose a different node on the second pass, and the PVCs became fully bound in between the predicates and assume volumes, then assume volumes will return success and the pod will get scheduled to the wrong node.


Question: glog.Errorf("ContainerStatus for %s error: %v", c.Id, err) return nil, err } <del> annotatedInfo := getContainerInfoFromAnnotations(c.Annotations) labeledInfo := getContainerInfoFromLabels(c.Labels) cStatus := &kubecontainer.ContainerStatus{ ID: kubecontainer.ContainerID{ Type: m.runtimeName, ID:   c.Id, }, Name:         labeledInfo.ContainerName, Image:        status.Image.Image, ImageID:      status.ImageRef, Hash:         annotatedInfo.Hash, RestartCount: annotatedInfo.RestartCount, State:        toKubeContainerState(c.State), CreatedAt:    time.Unix(0, status.CreatedAt), } if c.State == runtimeapi.ContainerState_CONTAINER_RUNNING { cStatus.StartedAt = time.Unix(0, status.StartedAt) } else { cStatus.Reason = status.Reason cStatus.Message = status.Message cStatus.ExitCode = int(status.ExitCode) cStatus.FinishedAt = time.Unix(0, status.FinishedAt) </del> <ins> cStatus := toKubeContainerStatus(status, m.runtimeName) if status.State == runtimeapi.ContainerState_CONTAINER_EXITED { // Populate the termination message if needed. annotatedInfo := getContainerInfoFromAnnotations(status.Annotations) labeledInfo := getContainerInfoFromLabels(status.Labels) </ins> fallbackToLogs := annotatedInfo.TerminationMessagePolicy == v1.TerminationMessageFallbackToLogsOnError && (cStatus.ExitCode != 0 || cStatus.Reason == "OOMKilled") tMessage, checkLogs := getTerminationMessage(status, annotatedInfo.TerminationMessagePath, fallbackToLogs) if checkLogs {

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): Pod status started null, started, 01 Jan 0001 Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-05-10T15:48:59Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-05-10T15:38:08Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"}` Environment: Cloud provider or hardware configuration: - OS (e.g. from /etc/os-release): CentOS 7 Kernel (e.g. ): Install tools: kubeadm 1.6 What happened: When a pod is created and starts running, shows the correct container start time in Containers -containername -State -Started. After the container is terminated, the start time changes to . In addition, shows that is . What you expected to happen: I'm expecting the start time to not change after the pod is terminated, so that afterwards I could check how long a time the container execution took. How to reproduce it (as minimally and precisely as possible): Have a running kubernetes cluster, and create a pod with the following command: kubectl create -f - <<EOF apiVersion: v1 kind: Pod metadata: name: sleep spec: containers: name: sleep image: ubuntu command: ["sleep"] args: ["30"] restartPolicy: Never EOF This created a container that sleeps for 30 seconds and quits.

During those 30 seconds, check the pod status with : ... State: Running Started: Wed, 17 May 2017 10:25:22 +0300 ... After 30 seconds, check the status of the terminated container with again: ... State: Terminated Reason: Completed Exit Code: 0 Started: Mon, 01 Jan 0001 00:00:00 +0000 Finished: Wed, 17 May 2017 10:25:52 +0300 ... With we see that the container's start time is null: ... "state": { "terminated": { "containerID": "docker://ca0dbe0dab3bb91ad0b05ac167e44d5fbf805f736ef98f61b9e106ba4456667f", "exitCode": 0, "finishedAt": "2017-05-17T07:25:52Z", "reason": "Completed", "startedAt": null } } ... Anything else we need to know: I need to measure container run durations and this bug prevents me from doing so.
/cc PR to fix this:


Question: // NodeMapper contains information to generate nameToNodeInfo and vcToZoneDatastore maps type NodeMapper struct { <ins> nodeInfoRWLock        *sync.RWMutex nameToNodeInfo        map[string]*NodeInfo vcToZoneDatastoresMap map[string](map[string][]string) </ins> } // NodeInfo contains information about vcenter nodes

Command run: vSphere intree volume test suite occasionally failed with the following error: The test suite should always pass. We may have a chance to hit this by running ` No response binary version: 1.27.4 </details<detailsvsphere </details<details</details<details</details<details</details<details</details
/assign
/sig testing storage ?
/triage accepted


Question: return csiDriverName, nil } } <del> return "", fmt.Errorf("could not find CSI Driver name for plugin %v", pluginName) </del> <ins> return "", fmt.Errorf("could not find CSI Driver name for plugin %s", pluginName) </ins> } // GetInTreeNameFromCSIName returns the name of the in-tree plugin superseded by

I noticed that kube-controller-manager is consuming visible amount of CPU in completely idle 5k node cluster (~7 cores/s) while for comparison kube-apiserver was using much less that time. I fetched pprof and noticed that we are burning significant in metrics.(attachDetachStateCollector).CollectWithStability:! Going further, most of the cpu is used for fmt.Errorf for formatting v1.Volume in line 160: which is never actually used by the caller We do not use significant cpu for preparing error messages that are then ignored. pprof cluster with some pods and volumes. No response_ <details26.1 </details<detailsgke, any? </details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
A trivial fix would be actually to change GetInTreePluginNameFromSpec to write only PV/Volume names instead of entire specs in error message.
/assign In scheduler we use this function to check if we can translate the in-tree PV to CSI, I think we also only need the name of the pv. I'll create a pr for this one line change.
/sig storage Instrumentation ?


Question: } klog.V(5).Infof("fc: volumeSpecToMounter volumeMode %s", volumeMode) return &fcDiskMounter{ <del> fcDisk:     fcDisk, fsType:     fc.FSType, volumeMode: volumeMode, readOnly:   readOnly, mounter:    volumeutil.NewSafeFormatAndMountFromHost(fcPluginName, host), deviceUtil: volumeutil.NewDeviceHandler(volumeutil.NewIOHandler()), </del> <ins> fcDisk:       fcDisk, fsType:       fc.FSType, volumeMode:   volumeMode, readOnly:     readOnly, mounter:      volumeutil.NewSafeFormatAndMountFromHost(fcPluginName, host), deviceUtil:   volumeutil.NewDeviceHandler(volumeutil.NewIOHandler()), mountOptions: volumeutil.MountOptionFromSpec(spec), </ins> }, nil } return &fcDiskMounter{ <del> fcDisk:     fcDisk, fsType:     fc.FSType, readOnly:   readOnly, mounter:    volumeutil.NewSafeFormatAndMountFromHost(fcPluginName, host), deviceUtil: volumeutil.NewDeviceHandler(volumeutil.NewIOHandler()), </del> <ins> fcDisk:       fcDisk, fsType:       fc.FSType, readOnly:     readOnly, mounter:      volumeutil.NewSafeFormatAndMountFromHost(fcPluginName, host), deviceUtil:   volumeutil.NewDeviceHandler(volumeutil.NewIOHandler()), mountOptions: volumeutil.MountOptionFromSpec(spec), </ins> }, nil }

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Currently many in-tree storage plugins, including the iSCSI plugin, support mount options. The FC plugin does not. I have poked around the code enough to know where the plugin reports that mount options are not supported, but it isn't clear to me WHY they aren't supported. Why is this needed: Mount options can drastically change the performance and data persistence characteristics of a filesystem. For example, "noatime" can reduce metadata operations on frequently read data or "sync" can ensure all writes go immediately down to a backing storage array. I am working on a deployment across an FC backed cluster for which the the containerized application benefits from specific mount options, and I would guess other users might find similar use cases. It seems to me that if mount options are supportable on iSCSI, they should be equally supportable on FC, but I certainly recognize the possibility that I may be missing something. Otherwise, I'm guessing they simply haven't been a priority?
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/good-first-issue /help
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /good-first-issue /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Yeah, we just miss a volunteer that would implement this feature. It should be relatively easy and straightforward, following iscsi as an example.
I would like to work on it! As this my first issue I would like to get some help.
/assign sathish-thangaraj
GitHub didn't allow me to assign the following users: sathish-thangaraj. Note that only and repo collaborators can be assigned and that issues/PRs can only have 10 assignees at the same time.

For more information please see <detailsIn response to : /assign sathish-thangaraj Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
that's great! I was going to write detailed info how to fix this, however, I found out that mount options should work, at least in Kubernetes 1.13, see this commit: So there is a simple bug somewhere. Looking around the code, this code smells fishy: . Can you check with "BlockVolume" feature turned off in kubelet, i.e. ? Mount options should work there. And then it should be fairly easy to fix it in .
It definitely looks like there's an issue there. I don't think the kubelet ever made it that far in my initial testings, as mounting (with mountOptions listed in the PersistentVolume object) failed with the following message: I assume this is related to the following code: Is there a way to disable a check like this without rebuilding all of Kubernetes?
Good catch! No. IMO it should be enough to rebuild kubelet here and it's fast and easy: and find it somewhere in .
Thanks for the help so far I flipped the SupportsMountOption() bool to "true" and the BlockVolume feature gate to false. I was then able to mount an ext4 volume with the "noatime" and "nodiratime" option (confirmed on the worker node). However, I attempted to use the "nobarrier" option but the worker node mounted the volume without it. Finally, I attempted to use a completely made-up option ("hello"). The volume did not fail to mount (as is suggested in the Kubernetes documentation. Instead, it mounted without the "hello" options. I am attempting to locate useful logs to try to understand why some mount options are ignored or why incorrect mount options aren't causing failures, but the kubelet logs haven't been helpful thus far. Unfortunately I don't currently have any easy access to any iSCSI equipment for comparison.
anyone working on this?
is there anywhere that I could help on the issue?
First time contributor here - I just changed 2 lines of code to make FC similar to other volume types (eg. iSCSI).

I'm not sure about the mount option validation that mentions. Should that be a separate issue?
Apologies for the extreme delay in keeping up with this issue. I believe the issue with ignored mount options may be more generally related to the way the Linux mounter (/pkg/utils/mount/mount-) handles mount options, although I don't have enough experience with other volume plugins to speak authoritatively on the issue. I recommend we try to get the basic changes suggested above into the code, as my limited testing shows them to be effective for at least some mount options. I'd try to submit a pull request myself, but I don't want to take it from as they've already put forth the work to get it started.
To clarify my previous comment, during Attach, the FC volume plugin mounts an FC volume to a global path with no mount options. During Mount, the Linux mounter bind mounts (-o bind) and bind remounts (-o bind,remount) an FC volume to a Pod specific path with mount options. However, the bind remount action is only capable of applying a limited number of mount options (those not on the list are silently ignored). In order to expand our mount option capabilities, we would (I believe) need to use mount options in the Attach phase.
First time contributor, taking this one /assign
you are, of course, welcome to work on this issue. However, I wanted to let you know that I made code changes back in November that (I believe) address my previous comments. I only received approval from my employer in the last couple of days to make the contributions, or I would have submitted them already. I plan to clean things up and get a PR in sometime this afternoon. I am also a first time contributor, so we'll see how it goes. /assign
No problem at all dude. You take this one since you already made the changes. On Thu, 23 Jan 2020 at 18:47 Eric Weber <:


Question: var _ = Describe("Initial Resources", func() { f := NewFramework("initial-resources") <del> It("[Skipped] should set initial resources based on historical data", func() { </del> <ins> It("[Skipped][Autoscaling Suite] should set initial resources based on historical data", func() { // Cleanup data in InfluxDB that left from previous tests. influxdbClient, err := getInfluxdbClient(f.Client) expectNoError(err, "failed to create influxdb client") _, err = influxdbClient.Query("drop series autoscaling.cpu.usage.2m", influxdb.Second) expectNoError(err) _, err = influxdbClient.Query("drop series autoscaling.memory.usage.2m", influxdb.Second) expectNoError(err) </ins> cpu := 100 mem := 200 for i := 0; i < 10; i++ {

Jenkins project: kubernetes-e2e-gce-autoscaling failed last couple of runs: Initial Resources [Skipped][Autoscaling Suite] should set initial resources based on historical data Expected <int64: to be ~ <int:
couple failing instances:
Re-opening. This flaked again:
status?
The test is disabled and there is no plan to enable it within next few weeks, so closing the issue and I'll back to it some time in the future.


Question: v1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/util/sets" <ins> "k8s.io/apimachinery/pkg/version" </ins> clientset "k8s.io/client-go/kubernetes" restclient "k8s.io/client-go/rest" )

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: N/A Which test(s) are failing: [] [sig-node] PreStop should call prestop when killing a pod [Conformance] Since when has it been failing: N/A Testgrid link: N/A Reason for failure: When spawning a Pod, it's container might start before all its networking plumbing has been completed. Thus, if the container tests / needs its network connectivity, it mgiht fail. The ` image tries to connect to the Kubernetes API, but if it cannot, the Pod will end up in an Error state. Retrying the Kubernetes API connection will avoid this issue. Anything else we need to know:
/sig node
/remove-sig node /sig testing


Question: # Dry-run create the CR kubectl "${kube_flags[@]:?}" apply --server-side --dry-run=server -f hack/testdata/CRD/resource.yaml "${kube_flags[@]:?}" # Make sure that the CR doesn't exist <del> ! kubectl "${kube_flags[@]:?}" get resource/myobj || exit 1 </del> <ins> ! kubectl "${kube_flags[@]:?}" get resource/myobj 2>/dev/null || exit 1 </ins> # clean-up kubectl "${kube_flags[@]:?}" delete customresourcedefinition resources.mygroup.example.com

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: Which test(s) are flaking: Testgrid link: Reason for failure: Anything else we need to know: /sig cli /cc /milestone v1.20 /prority important-soon
after creating the CRD, before applying the CR, kubectl should wait until includes :
/assign
/assign is this issue related to ?
no, this is simply kubectl trying to apply a CR before the server has made the API available at all
Got it! and I have filed to address this issue.
Still happening:
still happening: looks like the root cause is not related to CRD: I think the message Error from server (NotFound): "myobj" not found is expected, as the CR was not actually created: /reopen
Reopened this issue. <detailsIn response to : still happening: I think the message Error from server (NotFound): "myobj" not found is expected, as the CR was not actually created: Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
yes, the actual error is later: Line 177 succeeds, and the next line fails:


Question: } badName := "!@#$%^" badValue := "spaces not allowed" <ins> badAPIGroup := "example.com/v1" goodAPIGroup := "example.com" </ins> scenarios := map[string]struct { class        *resource.ResourceClass

Following YAML file is accepted without error. Notice the field, it is an instead of , containing character. This can lead to an issue when resource driver generates structured parameters' in-tree object with being just without version. The DRA scheduler plugin will try to match the parameters object based on the reference here: In latest master the code appears to use an indexer, and the issue might be hidden, if applicable at all. I did not check the reproducibility in master yet. However, main question is whether the should allow value with character in general. An error is expected from the apiserver upon object creation request, reporting that APIGroup containes APIVersion. Maybe. Have any resource driver that supports structured parameters, , for instance, also puts (gpucrd.APIGroup) value into . v1.30.1 <details</details<details</details<details</details<details</details<detailsscheduler's dynamicresources plugin </details
/sig node /triage accepted
/assign


Question: actual := ResourceConfigForPod(testCase.pod, testCase.enforceCPULimits, testCase.quotaPeriod) if !reflect.DeepEqual(actual.CpuPeriod, testCase.expected.CpuPeriod) { <del> t.Errorf("unexpected result, test: %v, cpu period not as expected", testName) </del> <ins> t.Errorf("unexpected result, test: %v, cpu period not as expected. Expected: %v, Actual:%v", testName, *testCase.expected.CpuPeriod, *actual.CpuPeriod) </ins> } if !reflect.DeepEqual(actual.CpuQuota, testCase.expected.CpuQuota) { <del> t.Errorf("unexpected result, test: %v, cpu quota not as expected", testName) </del> <ins> t.Errorf("unexpected result, test: %v, cpu quota not as expected. Expected: %v, Actual:%v", testName, *testCase.expected.CpuQuota, *actual.CpuQuota) </ins> } if !reflect.DeepEqual(actual.CpuShares, testCase.expected.CpuShares) { <del> t.Errorf("unexpected result, test: %v, cpu shares not as expected", testName) </del> <ins> t.Errorf("unexpected result, test: %v, cpu shares not as expected. Expected: %v, Actual:%v", testName, *testCase.expected.CpuShares, &actual.CpuShares) </ins> } if !reflect.DeepEqual(actual.Memory, testCase.expected.Memory) { <del> t.Errorf("unexpected result, test: %v, memory not as expected", testName) </del> <ins> t.Errorf("unexpected result, test: %v, memory not as expected. Expected: %v, Actual:%v", testName, *testCase.expected.Memory, *actual.Memory) </ins> } } }

I noticed ResourceConfigForPod isn't checking here: The methods called by ( and ) do check , but it looks like and can be true for burstable pods even when some of the initContainers do not declare cpu/memory limits. I'm not sure what the implications of setting these values are if an initContainer has no cpu or memory limit: /sig node /assign
After reading the code, I prefer that this is a bug according to . would you confirm? /assign I opened .
/triage accepted


Question: "k8s.io/apimachinery/pkg/util/sets" "k8s.io/apimachinery/pkg/util/wait" <ins> "k8s.io/apiserver/pkg/endpoints/metrics" </ins> "k8s.io/apiserver/pkg/server/httplog" "k8s.io/klog" )

What would you like to be : Currently the apiserver has request latencies metrics, which are a histogram of requests to the apiserver. However, these request metrics do not include healthz. While we have recently the ability to get back health check metrics from the healthz endpoint directly, we should also emit this data as a metric, so that we can monitor health check latencies. /assign /sig api-machinery instrumentation
GitHub didn't allow me to assign the following users: jktomer. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : What would you like to be : /sig api-machinery instrumentation Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I noticed this is informally assigned. Has there been any progress? Is this a place I can fit in?
Yes, I'm working on a PR.
/assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: node: nodeWithTaints("nodeA", []v1.Taint{{Key: "dedicated", Value: "user1", Effect: "NoSchedule"}}), }, { <del> name:       "A pod which can't be scheduled on a dedicated node assigned to user2 with effect NoSchedule", pod:        podWithTolerations("pod1", []v1.Toleration{{Key: "dedicated", Operator: "Equal", Value: "user2", Effect: "NoSchedule"}}), node:       nodeWithTaints("nodeA", []v1.Taint{{Key: "dedicated", Value: "user1", Effect: "NoSchedule"}}), wantStatus: unschedulable, </del> <ins> name: "A pod which can't be scheduled on a dedicated node assigned to user2 with effect NoSchedule", pod:  podWithTolerations("pod1", []v1.Toleration{{Key: "dedicated", Operator: "Equal", Value: "user2", Effect: "NoSchedule"}}), node: nodeWithTaints("nodeA", []v1.Taint{{Key: "dedicated", Value: "user1", Effect: "NoSchedule"}}), wantStatus: framework.NewStatus(framework.UnschedulableAndUnresolvable, "node(s) had taint {dedicated: user1}, that the pod didn't tolerate"), </ins> }, { name: "A pod can be scheduled onto the node, with a toleration uses operator Exists that tolerates the taints on the node",

When trying to see why pods don't start using either kubectl get events or kubectl describe pod , the base message is not detailed enough. Warning FailedScheduling 5s (x4 over 65s) default-scheduler 0/1 nodes are available: 1 node(s) had taints that the pod didn't tolerate. What would you like to be : Details of the taint and details of what it didn't tolerate Why is this needed: To get much much better debug information by default when trying to understand why pods can't launch.
/sig cli
/sig scheduling /priority important-longterm
/cc
/remove-sig cli
Those labels are not set on the issue: <detailsIn response to : /remove-sig cli </details
/good-first-issue
/assign starizard
Can I get an idea of what this would involve? I might be able to put some work on it.
I assume is taking this already. The error message is produced in We probably also require changes in to get the name of the missing toleration.
/remove-help


Question: return fmt.Errorf("failed to get list of pause pids: %v", err) } for _, pid := range pausePids { <ins> if existingPausePIDSet.Has(pid) { // Not created by this test. Ignore it. continue } </ins> if err := validateOOMScoreAdjSetting(pid, -998); err != nil { return err }

The test is failing in . https://k8s- https://k8s- https://k8s- The reason is that the test expects that all pause containers () should have . However, in many tests, we are starting test containers with . Those pause containers have . Ideally, containers started in previous test should be cleaned up, so in serial suite it should not affect later test. However, it is not always true when CRI is enabled. For example, in the oom message, we could see: There are 2 problems here: Why are containers not properly cleaned up when CRI enabled? I can't even find which test leaked pause container from the log, because there are many test using pause image, and some of them start a lot of containers, such as density test, restart test. The oom score adj test should not rely on process name . However, I can't find an alternative for now, because we don't report pause container status in our api, we don't even know the container id to inspect with docker or CRI. /cc
This test is still flaking in cri-serial. Some examples: https://k8s- https://k8s- Need to check the logs to see if the symptoms are the same.
It could be that with CRI, it takes kubelet longer to clean up. Or, even worse, that kubelet is not cleaning up properly. We need more information on this on failure. Dumping pods on the node Dumping containers () on the node. Accessing docker directly is not ideal, but it's okay because the test itself only applies to docker. Alternatively, we can get the current running containers from kubelet's /runningPods endpoint. The downside is that this may lag behind the actual state a little bit (seconds). We should also make the test suite run more frequently. It runs once a day, which makes the debugging round-trip overhead too high.
I can reproduce this process leak by running the "Network should recover from ip leaks" on my own node, without CRI enabled. The image used in the test was , and the docker version 1.11.2. The test creates 100 pods, then restarts the docker service by 6 times, with 20s interval in between.

After the test ends, there could be multiple processes running left on the node. Those processes were reparented to pid 1, and showed no containers running. /cc
On the side note, the test should not be blindly checking all processes. It should, instead, take a snapshot of the system before the test and only check the new processes created by the tests.
I filed a new issue for the docker bug since it's not unique to CRI and this test is catching it by accident: Leaving this issue to fix the test by not blindly checking all processes:
The docker bug is not a regression, and is tracked in . The problem of the test is not necessarily to be fixed in v1.6, at least it's not release blocker. :)


Question: describeSecurityGroupsRequest := &ec2.DescribeSecurityGroupsInput{ GroupIds: []*string{&securityGroupId}, } <ins> // We don't apply our tag filters because we are retrieving by ID </ins> groups, err := s.ec2.DescribeSecurityGroups(describeSecurityGroupsRequest) if err != nil {

If there are two security groups on an instance, we currently make an arbitrary choice. We should choose based on tags, as we normally do. For back-compat, we need to cope if there is one security group and it is not tagged. We should make sure kube-up tags our security groups, though it isn't really required for kube-up, but it is just good behaviour. cf


Question: e2elog.Logf("kube-apiserver version: %s", serverVersion.GitVersion) } <ins> // Obtain the default IP family of the cluster // Some e2e test are designed to work on IPv4 only, this global variable // allows to adapt those tests to work on both IPv4 and IPv6 // TODO(dual-stack): dual stack clusters should pass full e2e testing at least with the primary IP family // the dual stack clusters can be ipv4-ipv6 or ipv6-ipv4, order matters, // and services use the primary IP family by default // If we´ll need to provide additional context for dual-stack, we can detect it // because pods have two addresses (one per family) framework.TestContext.IPFamily = getDefaultClusterIPFamily(c) e2elog.Logf("Cluster IP family: %s", framework.TestContext.IPFamily) </ins> // Reference common test to make the import valid. commontest.CurrentSuite = commontest.E2E

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: https://k8s-(IPv6),%20master%20(dev) Which test(s) are failing: [sig-scheduling] SchedulerPredicates [Serial] validates that there is no conflict between pods with same hostPort but different hostIP and protocol [Conformance] Since when has it been failing: Testgrid link: https://k8s-(IPv6),%20master%20(dev) Reason for failure: The tests checks for 0.0.0.0 that is IPv4 only Anything else we need to know: Probably I should check with SIG-Testing how can we add an ipv6 job and update the KEP accordingly EDIT Probably I should check with SIG-Testing how can we add an ipv6 job as part of the k/k presubmits, and update the KEP accordingly
/assign /cc
/sig scheduling

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Introduce a mechanism that allows specifying the IP protocol version used in the e2e testing. Why is this needed: This is a followup on , my understanding is that in order to discriminate between IPv4 or IPv6 we have to use the test labels and this adds complexity to the testing automation and doesn't allow to send this information to the context of the tests. Having a mechanism to specify the IP protocol will simplify the testing for IPv4 and IPv6 and will allow reusing most of the code.
/sig testing /cc /assign Does this make sense? If affirmative, what should be the best way to add this functionality? Checking the code I think that or can provide this, don't know if checking some of the cluster config parameters, ...
the testing-commons group (sig-testing) maintains the framework, but ideally the decision on how to proceed in terms of adding ipv6 detection should come from sig-network. possibly having this as agenda for a sig-network meeting is a good start, then discussing it in a testing-commons meeting. /sig network the linked issue is a big one, so i don't have all the context here. but IMO existing tests should just work for ipv6 without the requirement for a new "IPV6" tag in test names. dual-stack or protocol specific tests should have their descriptions contain the details properly (i.e. what is this test doing) we might want to add an utility function under: (there is a PR to move it BTW ) then such an utility might be useful only to tests that need to detect the protocol. one of the goals is to refactor the code in testcontext / framework, so adding this logic there is less preferred.
/area e2e-test-framework
I think that we can add an option to test_context to specify the cluster IP family the same way that exists one for the cloud provider this is simpler because when setting up the CI jobs you will know beforehand what IP family you want to test, it also will allow reusing all the e2e code for IPv6 and Dual Stack (cc: and by default will be IPv4, so it´s backward compatible.
Being able to override and specify exactly what you want the test to beleive about your cluster is nice, but ideally it should generally work without specifying more flags.

Surely we can detect this?
I was thinking during some time about this, detecting the IP family is very simple having IPv4 or IPv6 only clusters, but with the introduction of dual stack I´m afraid about the supportability and reliability of the autodetecting mechanisms in the long term. If you see the dual stack PRs there are still a lot of unknowns about the cluster behavior. My point is, let´s have something first stable that works and is simple, and once dual stack is implemented, we can replace the flag with a function in the framework . I guess that migrating from a flag variable to a function is straight forward, is just replacing by , what do you think?
in is as part of the context. do you mean adding as a function which will auto-detecting the ip family. less flags for the framework binary is preferred.
once we have this I will open a new issue to start cleaning up and , we no longer need to discriminate based on tags


Question: e2epod.WaitForPodToDisappear(f.ClientSet, clientPod.Namespace, clientPod.Name, labels.Everything(), framework.Poll, framework.PodDeleteTimeout) }() <del> framework.ExpectNoError(e2epod.WaitForPodRunningInNamespace(f.ClientSet, clientPod)) </del> testVolumeContent(f, clientPod, fsGroup, fsType, tests) }

Which jobs are flaking: gce-cos-master-alpha-features Which test(s) are flaking: [sig-storage] CSI Volumes [Driver: ][Serial] [Testpattern: Dynamic PV (default fs)] provisioning should provision storage with snapshot data source [Feature:VolumeSnapshotDataSource] Testgrid link: Reason for failure: It takes 6 minutes to start a pod with PVC that has been restored from a snapshot, while the test waits only 5 minutes: csi-attacher sidecar says: Perhaps the provisioner should wait until the volume is ready to be used? That may not help with this test timeout, as PVC is provisioned after the Pod is scheduled and the total time could be the same. All this is caused by my PR to test block snapshots, It reworked the tests from using . If snapshots are slow, should all persistent storage tests use / ?
/sig storage
Generally, cutting a snapshot is quick, but uploading can take longer. Does provisioner wait until the snapshot is "ready" before returning success?
cc
(But yes, we should increase the timeout because overall time will still be longer no matter which order we take)
The provisioner does not wait. If snapshot is not readyToUse, creating a volume from it will fail. I'll have to check to see if there's logic to wait in the e2e test. Regarding GCE PD, at restore time, do we need to download the snapshot from object store? If that's the case, it could take a while to download as well.
volume controller should keep trying until snapshot is ready. Looks like attacher tried to attach while disk is not ready. It is strange because I think attach only happen after volume is successfully provisioned.
AFAIK, most cloud providers' snapshot ATM will be uploaded to some remote when taking, and will be downloaded when restoring. GCE PD is one of them.
It's not taking snapshots what slows the test, it's restoring it - returns quickly, but the returned volume is not really usable and attach fails for ~4 minutes until the snapshot is fully restored somewhere in GCE. Now that I am thinking about it, maybe it's better if the driver returns the volume early, so it does not end up orphaned when user deletes corresponding PVC and the provisioner is restarted at the same time.

Anyway, I'll extend the timeout in volume checks. /assign
Which brings me to idea: what about other storage backends? Some of them may take even longer that 10 minutes to restore a snapshot. Or more than 5 minutes to provision an empty volume and run a pod. There should be some configuration of timeouts (timeout multiplier?) for
I opened to investigate from GCE side if this behavior is expected.


Question: case event := <-busChan: if event == nil || len(event.Body) == 0 { klog.Errorf("Failed obtaining shutdown event, PrepareForShutdown event was empty") <ins> continue </ins> } shutdownActive, ok := event.Body[0].(bool) if !ok { klog.Errorf("Failed obtaining shutdown event, PrepareForShutdown event was not bool type as expected") <del> return </del> <ins> continue </ins> } shutdownChan <- shutdownActive }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: kubelet from panic after getting the wrong signal, and restart. ! What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): v1.20.1 Cloud provider or hardware configuration: OS (e.g: ): 7 Kernel (e.g. ): Linux zsm-test-2 3.10.0-1160.11.1.el7.x8664 SMP Fri Dec 18 16:34:56 UTC 2020 x8664 x8664 x86_64 GNU/Linux Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node


Question: // Provide implements DockerConfigProvider func (g *dockerConfigUrlKeyProvider) Provide() credentialprovider.DockerConfig { // Read the contents of the google-dockercfg-url key and load a .dockercfg from there <del> if url, err := credentialprovider.ReadUrl(dockerConfigUrlKey, g.Client, metadataHeader); err == nil { </del> <ins> if url, err := credentialprovider.ReadUrl(dockerConfigUrlKey, g.Client, metadataHeader); err != nil { glog.Errorf("while reading 'google-dockercfg-url' metadata: %v", err) } else { </ins> if strings.HasPrefix(string(url), "http") { <del> if cfg, err := credentialprovider.ReadDockerConfigFileFromUrl(string(url), g.Client, nil); err == nil { </del> <ins> if cfg, err := credentialprovider.ReadDockerConfigFileFromUrl(string(url), g.Client, nil); err != nil { glog.Errorf("while reading 'google-dockercfg-url'-specified url: %s, %v", string(url), err) } else { </ins> return cfg } } else {

It looks like we always try and hit the metadata server even if we aren't on GCE. We then print an error out. This should either be silent (assume after X attempts that we aren't on GCE?) or should only be activated with an option. Email thread: !topic/google-containers/w-ei5Xs6K0Y
cc:
FYI, the way the credentialprovider logic is set up, providers are asked once per keyring creation whether they are enabled. If they are enabled, then they are asked to provide their contribution to .dockercfg each time the lazy keyring is accessed. For efficiency, there is a caching provider, which composes with heavier providers and stores their .dockercfg contribution for a predetermined TTL. What was happening here was that "Enabled()" for our GCE-metadata implementations was logging about errors fetching "http://metadata", where the entire "Enabled()" check is "err == nil".
I believe that this was fixed by


Question: "go_test", ) <ins> go_test( name = "go_default_test", srcs = [ "cluster_test.go", "configmap_test.go", "deployment_test.go", "generate_test.go", "kubectl_test.go", "namespace_test.go", "proxy_server_test.go", "quota_test.go", "rolebinding_test.go", "rolling_updater_test.go", "rollout_status_test.go", "run_test.go", "scale_test.go", "secret_for_docker_registry_test.go", "secret_for_tls_test.go", "secret_test.go", "service_basic_test.go", "service_test.go", "serviceaccount_test.go", "sorting_printer_test.go", "stop_test.go", ], library = ":go_default_library", tags = ["automanaged"], deps = [ "//federation/apis/federation/v1beta1:go_default_library", "//pkg/api:go_default_library", "//pkg/api/testapi:go_default_library", "//pkg/api/testing:go_default_library", "//pkg/api/v1:go_default_library", "//pkg/apis/apps:go_default_library", "//pkg/apis/apps/v1beta1:go_default_library", "//pkg/apis/batch:go_default_library", "//pkg/apis/batch/v1:go_default_library", "//pkg/apis/batch/v2alpha1:go_default_library", "//pkg/apis/extensions:go_default_library", "//pkg/apis/extensions/v1beta1:go_default_library", "//pkg/apis/rbac:go_default_library", "//pkg/client/clientset_generated/internalclientset:go_default_library", "//pkg/client/clientset_generated/internalclientset/fake:go_default_library", "//pkg/client/clientset_generated/internalclientset/typed/batch/internalversion:go_default_library", "//pkg/client/clientset_generated/internalclientset/typed/core/internalversion:go_default_library", "//pkg/client/clientset_generated/internalclientset/typed/extensions/internalversion:go_default_library", "//pkg/kubectl/util:go_default_library", "//vendor/github.com/spf13/cobra:go_default_library", "//vendor/k8s.io/apimachinery/pkg/api/equality:go_default_library", "//vendor/k8s.io/apimachinery/pkg/api/errors:go_default_library", "//vendor/k8s.io/apimachinery/pkg/api/resource:go_default_library", "//vendor/k8s.io/apimachinery/pkg/apis/meta/v1:go_default_library", "//vendor/k8s.io/apimachinery/pkg/apis/meta/v1/unstructured:go_default_library", "//vendor/k8s.io/apimachinery/pkg/runtime:go_default_library", "//vendor/k8s.io/apimachinery/pkg/runtime/schema:go_default_library", "//vendor/k8s.io/apimachinery/pkg/util/intstr:go_default_library", "//vendor/k8s.io/apimachinery/pkg/util/sets:go_default_library", "//vendor/k8s.io/apimachinery/pkg/util/uuid:go_default_library", "//vendor/k8s.io/apimachinery/pkg/watch:go_default_library", "//vendor/k8s.io/client-go/rest:go_default_library", "//vendor/k8s.io/client-go/rest/fake:go_default_library", "//vendor/k8s.io/client-go/testing:go_default_library", "//vendor/k8s.io/client-go/util/testing:go_default_library", ], ) </ins> go_library( name = "go_default_library", srcs = [

StatefulSet now as of 1.6. That means the TODO on can be fixed in 1.7. We could not fix it in 1.6 because that code is used by kubectl, which must work against servers that are one minor version ahead of or behind the client.
/cc
I am volunteered to take this job if no one start working on this.
I don't think anyone's working on it, so you are welcome to take it. Feel free to tag me as a reviewer. Thanks!


<!-- Please only use this template for submitting enhancement requests --What would you like to be : Implement the feature described here: Why is this needed: In a cluster that has a service mesh a lot of the work being done by kube-proxy is redundant and wasted. Specifically, services that are only reached via other services in the mesh will never use the service abstraction implemented by kube-proxy in iptables (or ipvs). By informing the kube-proxy of this, we can lighten the work it is doing and the burden on its proxy backend. <!-- DO NOT EDIT BELOW THIS LINE --/kind feature
/sig network


Question: # install.  See https://github.com/saltstack/salt-bootstrap/issues/270 # # -M installs the master <del> curl -L http://bootstrap.saltstack.org | sh -s -- -M -X </del> <ins> curl -L http://bootstrap.saltstack.com | sh -s -- -M -X </ins> echo $MASTER_HTPASSWD > /srv/salt/nginx/htpasswd

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: After deleting a pod and scaling up the statefulset with different valid envs in the same container simultaneously, then the output events of executing the command is below: What you expected to happen: update successfully. How to reproduce it (as minimally and precisely as possible): a statefulset. apply this statefulset's yaml: After a while, execute , and the pod is running a new stateful's yaml, which a new env key and more replicas. the pod and apply a new stateful's yaml the statefulset Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.4", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-02-28T13:37:52Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.4", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-02-28T13:30:26Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}
/sig apps
Meet same problem while using kube-addon-manager. vs and patchStrategy
+1
In some cases, add in the env which using can solve the problem.
/assign
I think the root cause of this is the strategic patch. As mentioned, we have to use the to fix the problem which tells the strategic patch to remove the field so the spec could pass the validation process of statefulsets. It would lead to API broken if we change the behaviour of strategic merge patch, I'll dig around in the apps validation process and try to find out the solution to this.
Thank you for your reply. According to your advice, I tried to add in my second statefulset's yaml, but I still could not update statefulset correctly. How to fix my yamls to get around the problem?

/unassign Sorry, I do not have enough time for this, anyone interested could take it from here.
I don't think the problem comes from your YAML, the problem comes from the controller that doesn't update the fields properly.
/assign


Question: "github.com/golang/glog" "github.com/spf13/cobra" <del> appsv1 "k8s.io/api/apps/v1" </del> "k8s.io/apimachinery/pkg/api/errors" "k8s.io/apimachinery/pkg/api/meta" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" <del> "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" "k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/runtime/schema" "k8s.io/apimachinery/pkg/util/uuid" "k8s.io/apimachinery/pkg/util/wait" </del> "k8s.io/client-go/dynamic" "k8s.io/kubernetes/pkg/kubectl/cmd/templates" cmdutil "k8s.io/kubernetes/pkg/kubectl/cmd/util"

/kind bug /priority critical-urgent What happened: In I'm removing all the reapers, unfortunately when invoking foreground deletion on a DaemonSet the delete operation hangs and controller indefinitely tries to manage a removed DS. What you expected to happen: A foreground deletion properly removes DaemonSet and all its dependents. How to reproduce it (as minimally and precisely as possible): Invoke a delete operation against kube API (without using command). Anything else we need to know?: This behavior makes it so that none of the consumers of the API is not able to remove the DS controller due to present in the DS reaper. A working fix that passed the tests with the aforementioned PR is in but I'm not 100% it's correct. This is an issue that needs fixing before 1.11 goes out. /assign /sig apps /cc
fyi
I filed to fix it. As mentioned in the PR, this is because the DaemonSet controller races with garbage collector when it tries to update DaemonSet status of the DaemonSet being deleted. Here's what happens when hangs: /something performs a foreground deletion on a DaemonSet finalizer and DeletionTimestamp are both set history objects (ControllerRevisions) and pods are being deleted by garbage collector; meanwhile, DaemonSet controller tries to update DaemonSet status. Updating DaemonSet status requires constructing DaemonSet history objects, to figure out current revision and which pods do/don't belong to current revision updating DaemonSet status, DaemonSet controller tries to create a DaemonSet history object that matches current DaemonSet spec collector then tries to delete that DaemonSet history object. And repeat. Because we can't make DaemonSet pods be deleted before DaemonSet history objects (DaemonSet history objects don't own DaemonSet pods!), we cannot reliably calculate DaemonSet status without history objects anyways. Therefore, we don't update DaemonSet status for DaemonSet being deleted. Note that the reason why the kubectl delete hack works is because it forces DaemonSet pods to be removed before history objects.
[MILESTONENOTIFIER] Milestone Issue: Up-to-date for process Note: This issue is marked as , and must be updated every 1 day during code freeze. Example update: <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed.

: Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details


Question: } } } <ins> func TestParseSystemdToCgroupName(t *testing.T) { testCases := []struct { input    string expected CgroupName }{ { input:    "/test", expected: []string{"test"}, }, { input:    "/test.slice", expected: []string{"test"}, }, } for _, testCase := range testCases { if actual := ParseSystemdToCgroupName(testCase.input); !reflect.DeepEqual(actual, testCase.expected) { t.Errorf("Unexpected result, input: %v, expected: %v, actual: %v", testCase.input, testCase.expected, actual) } } } </ins>

start kubelet with below config: with below config: check the cgroup, get the following output: then, start kubelet.service, run as follows: why? Environment: Kubernetes version (use ): v1.14.1 Cloud provider or hardware configuration: OS (e.g: ): Debian GNU/Linux 9 (stretch) Kernel (e.g. ): SMP Debian 4.9.30-2+deb9u2 (2017-06-26) x86_64 GNU/Linux Install tools: Network plugin and version (if this is a network-related bug): Others:
/area kubelet /sig node
From cgroupManagerImpl#Exists : Can you turn on log V4 so that we can see the above log ? thanks
It seems if we change return value type to error, the caller would be able to incorporate error message from Exists func.
Thanks for sharing :) Hope we can help you debug! Is the information here
thanks for your rely. when I turn on log V4, logs as followed: Cgroup path is not expected: , so when I modify , it‘s ok. so Is this normal?
Ah, it looks like its appending automatically to the cgroup name. At the very least, I think the documentation could be clearer to indicate this will happen. An additional option could be adding the suffix only if doesn't already exist.
thank you
no problem :) If this issue is resolved, would you mind closing it? Thanks!
It might be worth considering adding a little code somewhere that simply strips off the ".slice" from the cgroup name supplied in configuration option and just normalizing the input in the configuration package itself (possibly with a warning that prints out stating cgroup should not end in ".slice")
I think that's a great idea. I will work on that, unless you want to :) /assign
thank you


Question: plugin := plugins[0] fakeMounter := iscsiDiskMounter{ iscsiDisk: &iscsiDisk{ <ins> Iface:  "192.168.1.10:pv0001", </ins> plugin: plugin.(*iscsiPlugin)}, exec: fakeExec, } <del> newIface := "192.168.1.10:pv0001" cloneIface(fakeMounter, newIface) </del> <ins> cloneIface(fakeMounter) </ins> if cmdCount != 5 { t.Errorf("expected 5 CombinedOutput() calls, got %d", cmdCount) }

What happened: When creating a block volume, the iSCSI storage plugin stores volume-related data in different directories: As a result, some data is not cleaned up once the volume is detached: How to reproduce it (as minimally and precisely as possible): These are the objects I used to reproduce this issue: pod- /sig storage CC
/assign


Question: Subsystem:         KubeletSubsystem, Name:              AssignedConfigKey, Help:              "The node's understanding of intended config. The count is always 1.", <del> DeprecatedVersion: "1.22", </del> <ins> DeprecatedVersion: "1.22.0", </ins> StabilityLevel:    metrics.ALPHA, }, []string{ConfigSourceLabelKey, ConfigUIDLabelKey, ConfigResourceVersionLabelKey, KubeletConfigKeyLabelKey},

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --kubelet failed to start when specifying ,with following error: Try to start kubelet with : Kubernetes version (use ): master branch Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig node
Submitted PR to fix this issue.
/triage accepted /priority important-soon


Question: if nodeShuttingDown { return lifecycle.PodAdmitResult{ Admit:   false, <del> Reason:  nodeShutdownReason, Message: nodeShutdownNotAdmitMessage, </del> <ins> Reason:  nodeShutdownNotAdmittedReason, Message: nodeShutdownNotAdmittedMessage, </ins> } } return lifecycle.PodAdmitResult{Admit: true}

When using Graceful Node Shutdown feature, status and reason of "shutdown" pod are confusing for users. See where I attempt to address this confusion by documenting the behavior. However, it would be great to improve messages. Perhaps the following renaming: Reason "Shutdown" is confusing because it may appear as a process of shutting down. Better message will also be useful. Reason and Message changed for pods terminated in response to Node shutdown. Shutdown the node and observer the pod status. KEP: kubernetes/enhancements /cc /sig node /help /good-first-issue Kubernetes version (use ): 1.19+ (with feature enabled)
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : #### What happened: /cc /sig node /help /good-first-issue Kubernetes version (use ): 1.19+ (with feature enabled) </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc
/assign
/assign


Question: "//staging/src/k8s.io/api/apps/v1:go_default_library", "//staging/src/k8s.io/api/core/v1:go_default_library", "//staging/src/k8s.io/api/policy/v1beta1:go_default_library", <ins> "//staging/src/k8s.io/apimachinery/pkg/api/equality:go_default_library", </ins> "//staging/src/k8s.io/apimachinery/pkg/apis/meta/v1:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/util/intstr:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/util/uuid:go_default_library",

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: There are too many pdb update operations compared with other resources. ! That is, there are many "Put" operations for pdb periodically in kube-apiserver log. In my view, the amount of pdb operations is connected with disruption controller's resyncPeriod and pdb's quantity in cluster. For example: disruption-controlle's default resyncPeriod is 30s, and there are 100 pdbs in cluster. We can find that the update operation times per minute is about 200. The data is reasonable, explanation is as follows: According to disruption-controlle's design, there is one resync every 30s. One resync means to call updatePdbStatus function for every pdb, so there are 200 times updatePdbStatus. Function updatePdbStatus judges whether the pdb is changed. If changed, call apiserver to update the pdb' status. When no eviction happens, disruptedPods is empty. In this situation, disruptedPods is an empty map, but pdb.Status.DisruptedPods is nil, empty map is not equal to nil. So reflect.DeepEqual(pdb.Status.DisruptedPods, disruptedPods) returns false. The pdb is thought to has be changed, then apiserver is called to update the pdb' status. In fact, there is no change for the pdb when disruptedPods is an empty map and pdb.Status.DisruptedPods is nil. What you expected to happen: There shouldn't be so many pdb update operations. That is, when pdb has no change, apiserver shouldn't be called to update its status. How to reproduce it (as minimally and precisely as possible): Create some pdbs in cluster, you will see many "Put" operations for pdb periodically in kube-apiserver log. Anything else we need to know?: Environment: kubernetes v1.13.2 Fix suggestions: The bug is caused by the mixed usage of empty and nil for disruptedPods, the type of which is map[string]metav1.Time.

There are two fix suggestions for it: one of empty map and nil for disruptedPods, use it consistently and refuse to mix empty map and nil. For example, when disruptedPods is empty or nil, save it as an empty map in etcd. Then every client gets pdb from etcd, the pdb.Status.DisruptedPods is an empty map. get pdb for etcd, assign an empty map to it if pdb.Status.DisruptedPods is nil. Maybe the mixed usage of empty and nil for map is a common problem in k8s design, not just for disruptedPods. I will propose a work around solution for the problem. If the solution has any problem, please let me know and I can modify the code.
/sig apps
/assign Morten can you PTAL and the open PR. I reviewed as well.
ref


Question: return gracePeriod, observedReadyCondition, currentReadyCondition, err } <del> func (nc *NodeController) checkForNodeAddedDeleted(nodes []*v1.Node) (added, deleted []*v1.Node) { for i := range nodes { if _, has := nc.knownNodeSet[nodes[i].Name]; !has { added = append(added, nodes[i]) </del> <ins> // classifyNodes classifies the allNodes to three categories: //   1. added: the nodes that in 'allNodes', but not in 'knownNodeSet' //   2. deleted: the nodes that in 'knownNodeSet', but not in 'allNodes' //   3. newZoneRepresentatives: the nodes that in both 'knownNodeSet' and 'allNodes', but no zone states func (nc *NodeController) classifyNodes(allNodes []*v1.Node) (added, deleted, newZoneRepresentatives []*v1.Node) { for i := range allNodes { if _, has := nc.knownNodeSet[allNodes[i].Name]; !has { added = append(added, allNodes[i]) } else { // Currently, we only consider new zone as updated. zone := utilnode.GetZoneKey(allNodes[i]) if _, found := nc.zoneStates[zone]; !found { newZoneRepresentatives = append(newZoneRepresentatives, allNodes[i]) } </ins> } } // If there's a difference between lengths of known Nodes and observed nodes // we must have removed some Node. <del> if len(nc.knownNodeSet)+len(added) != len(nodes) { </del> <ins> if len(nc.knownNodeSet)+len(added) != len(allNodes) { </ins> knowSetCopy := map[string]*v1.Node{} for k, v := range nc.knownNodeSet { knowSetCopy[k] = v } <del> for i := range nodes { delete(knowSetCopy, nodes[i].Name) </del> <ins> for i := range allNodes { delete(knowSetCopy, allNodes[i].Name) </ins> } for i := range knowSetCopy { deleted = append(deleted, knowSetCopy[i])

<!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"8+", GitVersion:"v1.8.0-alpha.1.51+-dirty", GitCommit:"", GitTreeState:"dirty", BuildDate:"2017-06-20T10:51:10Z", GoVersion:"go1.8", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"8+", GitVersion:"v1.8.0-alpha.1.51+-dirty", GitCommit:"", GitTreeState:"dirty", BuildDate:"2017-06-20T11:29:14Z", GoVersion:"go1.8", Compiler:"gc", Platform:"linux/amd64"} What happened: After adding label to a node, node controller panics. What you expected to happen: Everything should still work. How to reproduce it (as minimally and precisely as possible): Run and wait for the node controller to panic: Anything else we need to know: I've checked the code and it looks like the node controller expects the label to be present during node registration and it can not be later. Even if the label can not be latter it shouldn't panic.
/sig node
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
/assign let me try this :).


Question: "//vendor/github.com/vmware/govmomi/vapi/tags:go_default_library", "//vendor/github.com/vmware/govmomi/vim25:go_default_library", "//vendor/github.com/vmware/govmomi/vim25/mo:go_default_library", <ins> "//vendor/github.com/vmware/govmomi/vim25/soap:go_default_library", "//vendor/github.com/vmware/govmomi/vim25/types:go_default_library", </ins> "//vendor/gopkg.in/gcfg.v1:go_default_library", "//vendor/k8s.io/klog:go_default_library", ],

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: Provsioning a PVC using a storage class with vSphere as the cloud provider failed with error "runtime error: index out of range". What you expected to happen: The PVC should have been created. How to reproduce it (as minimally and precisely as possible): Create a PVC using a storage class with vSphere as the cloud provider and a VMware vCenter Server where the default language is not English. Anything else we need to know?: K8s in conjunction with vSphere PVC relies on English error messages from the VMware API in order to determine the canonical path for a newly created volume (VMDK file). The faulting code (v1.11.3, still the same on master) can be found here (starting at line 312): The line "canonicalVolumePath = match[1]" (340) in the above code is the problem because "File (.*?) was not found" (English) never matches for a different language (e.g. German: "Datei [DATASTORE] kubevols/kube- wurde nicht gefunden") and a “runtime error: index out of range” oocurs in the "kube-controller-manager". This causes the already created volume (VMDK file) not to be reported back to the K8s system which in turn will try to recreate it repeatedly in a loop (fails with "Cannot complete the operation because the file or folder [DATASTORE] /kubevols/kubernetes-dynamic-pvc-[UUID] already exists"). Find below the corresponding log excerpt (kube-controller-manager): Environment: Kubernetes version (use ):Cloud provider or hardware configuration: vSphere OS (e.g. from /etc/os-release): n/a Kernel (e.g. ): n/a Install tools: n/a Others: n/a <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
/assign


Question: plugin, err := app.StartPlugin(logger, "/cdi", d.Name, nodename, app.FileOperations{ Create: func(name string, content []byte) error { <del> ginkgo.By(fmt.Sprintf("creating CDI file %s on node %s:n%s", name, nodename, string(content))) </del> <ins> klog.Background().Info("creating CDI file", "node", nodename, "filename", name, "content", string(content)) </ins> return d.createFile(&pod, name, content) }, Remove: func(name string) error { <del> ginkgo.By(fmt.Sprintf("deleting CDI file %s on node %s", name, nodename)) </del> <ins> klog.Background().Info("deleting CDI file", "node", nodename, "filename", name) </ins> return d.removeFile(&pod, name) }, },

When a ResourceClaim: uses delayed allocation does not get created for a specific pod (i.e. not from a template) then it currently gets allocated and remains allocated until: scheduling of a pod fails because the node originally chosen for the allocation isn't suitable, in which case the scheduler triggers deallocation via the user deletes the claim What we want instead is that deallocation gets triggered when the last user of such an allocated claim gets removed from . To avoid races, deallocation must be done through . This can be implemented either as a controller or directly in the API server.
/assign
/triage accepted /sig node /priority important-longterm /kind feature


Question: if !ok { if !isDeleteList { // If it's not in the original document, just take the patch value. <ins> if mergeOptions.IgnoreUnmatchedNulls { discardNullValuesFromPatch(patchV) } </ins> original[k] = patchV } continue

Note: I'm using OpenShift. s/oc/kubectl/ below if that's how you roll. I'm using a command similar to the following to idempotently ensure a label is absent on a Namespace object: If the object's contains anything other than , this works fine: the resulting object has no key. However, if is initially absent, this command adds it with an entry with the empty string as a value: Running the command again removes the entry and the . Using this logic in a controller causes a hot loop as we thrash back and forth between the above and no labels. Using a value in a dict in a StrategicMerge patch should idempotently ensure the key is absent, even if the dict is initially absent/empty. I didn't try this on anything other than a Namespace; and I didn't try it on any dict other than . Workaround: Add any other label, as seen above. <details</details<detailsAWS </details<details</details<details</detailsI believe the issue is on the server, but... v0.9.3-0.- </details<details</details
/sig api-machinery
/assign /cc /triage accepted
can you look into it?
It looks that this idempotency problem is fixed in latest releases(after being label as you mentioned). All responses are legit; Kubernetes Version
The bug happens when there are no other labels. Are you saying the "fix" was for kubernetes to always add a label to the object?
That was an incidental change specific to namespaces. I'd expect the issue still exists for other types
you are both right. This is incidental "fix" for namespaces. I'll look at others to find the actual problem.


Question: - -k - --cache-size=1000 - --no-negcache <ins> - --dns-loop-detect </ins> - --log-facility=- - --server=/{{ pillar['dns_domain'] }}/127.0.0.1#10053 - --server=/in-addr.arpa/127.0.0.1#10053

Is this a BUG REPORT or FEATURE REQUEST?: What happened: If hosts points to a local DNS server, or if it contains the cluster's DNS server (to enable looking up services from host), it'll cause DNS loop which can effectively disable kube-dns or make it very slow. What you expected to happen: At least, kube-dns should detect localhost or cluster DNS server entries in and not send queries to them. How to reproduce it (as minimally and precisely as possible): Use a system with local cashing DNS server, or add cluster's DNS server address to hosts Anything else we need to know?: According to documentation, adding should make dnsmasq detect that and disable the upstream DNS server which cause the loop. It should prevent DNS from becoming unresponsive in the first case (a better solution is to extract real upstream servers, which probably cannot be done for every configuration); and for the second case it should skip sending queries to itself and send to real upstream servers.
/sig network
Could you describe the use-case more? If the host's contains cluster DNS server and some other upstream server - like 8.8.8.8, will services lookups work reliably from the host? If the services lookup goes to the 8.8.8.8 server, it will return NXDOMAIN. If it hits the clusterDNS server, only then it would work, correct?
Also ref If the first server returns NXDOMAIN, the second server will not be tried. How do you make both cluster DNS name and non-cluster DNS name resolvable on the host?
If cluster DNS server comes before a real DNS server, it works fine. Also note that what man page says is the glibc (or libc) behavior, not other users (e.g. dnsmasq sends its queries to any one of them, but can be configured to query them in strict order ( flag), or to query all of them (). Anyway, consider when cluster DNS server comes before the real server: cluster DNS queries, the cluster DNS server responds with the IP of the cluster or pod. external names, the query is sent to cluster DNS server. It doesn't find the name, so it tries to send it to upstream DNS servers; which in my example are cluster DNS server itself and an external DNS server.

If dnsmasq sends it to external DNS server, it'll receive the answer and can serve the query. If it sends it to itself, there is a loop and the same process happens again until dnsmasq decides to query the other server (which might not happen!). WIth flag, when dnsmasq sends the query to itself, it'll recognize the loop and will remove its own address from the list of upstream DNS servers, so it'll always send external names to the external DNS server from now on.
Thanks for the explanation. I believe it might work with explicit configuration. Though adding a flag to upstream for fulfilling a not quite common use case seems a bit odd to me. Have you thought of configuring upstream DNS servers via kube-dns config [1] such that it will not send request to itself at all? [1] -
Note that the flag has two use cases, the other one is prevent kube-dns breakage when upstream DNS server is a local one (e.g. If you deploy to Ubuntu, which has local caching DNS server by default (e.g. in 18.04, it is systemd-resolved and contains 127.0.0.53)). Actually I guess there are a number of DNS related issues which might be solved by this flag too. So, while this flag will not make kubernetes magically find real upstream dns servers in such cases, it prevents kube-dns crash; which has caused frustration and some time for me to discover. So, either kubelet should error out and exit if the given contains a local address, or such flag should be used by kube-dns to prevent DNS loops. And, IIRC, the only reason that cluster's DNS server is not to host, at least by kubespray, is mentioned to be host specific in the docs. So, my use case seems to be an ideal one and it would be the default one if it was easy to setup for different platforms. About using kube-dns config, no. We're going to provide automatic kubernetes deployment for many environments, and so we don't know the upstream DNS servers upfront, and we don't want to hard code it. We want it to be the DNS servers used in the target network. So, configuring a specific upstream DNS server is a no-go.


Question: Help:           "Total preemption attempts in the cluster till now", StabilityLevel: metrics.ALPHA, }) pendingPods = metrics.NewGaugeVec( &metrics.GaugeOpts{ Subsystem:      SchedulerSubsystem,

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Add a metric to track number of scheduler binding goroutines. This will be a generic metric with as the only label for now. metric name Description -- -- scheduler_goroutines Number of running binding goroutines Why is this needed: Helps us track how many binding are running and monitor issues such as goroutine leak. /sig scheduling
/assign


Question: rerr := as.VirtualMachinesClient.Update(ctx, nodeResourceGroup, vmName, newVM, "attach_disk") if rerr != nil { <del> klog.Errorf("azureDisk - attach disk(%s, %s) failed, err: %v", diskName, diskURI, rerr) detail := rerr.Error().Error() if strings.Contains(detail, errLeaseFailed) || strings.Contains(detail, errDiskBlobNotFound) { // if lease cannot be acquired or disk not found, immediately detach the disk and return the original error klog.V(2).Infof("azureDisk - err %v, try detach disk(%s, %s)", rerr, diskName, diskURI) as.DetachDisk(diskName, diskURI, nodeName) </del> <ins> klog.Errorf("azureDisk - attach disk(%s, %s) on rg(%s) vm(%s) failed, err: %v", diskName, diskURI, nodeResourceGroup, vmName, rerr) if rerr.HTTPStatusCode == http.StatusNotFound { klog.Errorf("azureDisk - begin to filterNonExistingDisks(%s, %s) on rg(%s) vm(%s)", diskName, diskURI, nodeResourceGroup, vmName) disks := as.filterNonExistingDisks(ctx, *newVM.VirtualMachineProperties.StorageProfile.DataDisks) newVM.VirtualMachineProperties.StorageProfile.DataDisks = &disks if rerr = as.VirtualMachinesClient.Update(ctx, nodeResourceGroup, vmName, newVM, "attach_disk"); rerr != nil { return rerr.Error() } </ins> } <del> return rerr.Error() </del> } <del> klog.V(2).Infof("azureDisk - attach disk(%s, %s) succeeded", diskName, diskURI) </del> <ins> klog.V(2).Infof("azureDisk - update(%s): vm(%s) - attach disk(%s, %s) succeeded", nodeResourceGroup, vmName, diskName, diskURI) </ins> return nil }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: check disk status before disk azure disk There is little possibility that attach disk would cost 10min, and disk could be attached, deleted in same time, that would cause race condition. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /kind bug /assign /priority important-soon /sig cloud-provider /area provider/azure


Question: - controller-manager - scheduler - nginx <ins> - kube-client-tools </ins> - logrotate {% if grains['cloud'] is defined and grains['cloud'] == 'azure' %} - openvpn

Currently the master VM specified by the Vagrantfile does not have the built kubecfg binary in the vagrant user's path. This request is to have this binary present on the host system and in the vagrant user's path so I can Vagrant ssh into the machine and do 'kubecfg list pods'.
I'm going to generalize this a little so that kubecfg is available in our general cluster install scripts. This should be easier as we move to binary installs.
Should be kubectl, not kubecfg, but otherwise is a good idea. etcd tools would be useful, too:
I'll take this on if no one else wants to do it. May take me a day or two to get to it.
Thinking out loud here... do we really want to encourage people to do admin and/or job runny stuff on their master?
I see it as a debugging tool.


Question: // Write it. if err := ipt.Restore(utiliptables.TableFilter, filterLines, utiliptables.NoFlushTables, utiliptables.RestoreCounters); err != nil { klog.Errorf("Failed to execute iptables-restore for %s: %v", utiliptables.TableFilter, err) <ins> metrics.IptablesRestoreFailuresTotal.Inc() </ins> encounteredError = true } }

What would you like to be : kube-proxy should expose better metrics around tracking success and failures. Administrators should be able to alert on failure counts. Why is this needed: There are several motivating cases: iptables lock contention causes to fail at an increasing rate as services + endpoints grows or are failing to execute Ultimately, there is essentially no feedback from kube-proxy if it fails to program the service network. This makes it annoyingly difficult to debug when things go wrong. As many reasonable hints we can provide should be . /sig network
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
is anyone working one this..
what cases do you think are poorly covered? /assign
thanks for triaging this :-) The specific cases are when fails - that function logs an error but doesn't otherwise surface any information. We should have a top-level metric for failing, along with any other lower-level metrics that might make sense for administrators, e.g. fails, fails.
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
/help-wanted
/assign


Question: responses: 200: body: <del> application/json: example: !include examples/service.json </del> <ins> example: !include examples/service.json </ins> put: description: Update a service body: <del> json/application: schema: !include doc/service-schema.json example: !include examples/service.json </del> <ins> schema: !include doc/service-schema.json example: !include examples/service.json </ins> delete: description: Delete a specific service responses: 200: body: <del> application/json: example: | { "success": true } </del> <ins> example: | { "success": true } </ins>

Starting a large data set (32gb allocated, 4gb resident) in the api server causes about 7.5gb allocations of . The tree for that shows: This appears to be due to openapi aggregation. If there is an easy fix in 1.7 it might be worth it. Otherwise this should be addressed for 1.8.
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
This is all API server startup - for small clusters it's about 20-60% of startup time
Most of those are that is the library we used to convert our spec to proto. The key is this line: Do you have any idea how we can get a better space time/complexity here?
I have a PR for that now.
Fixed in
Even without aggregation, starting and serving OpenAPI docs was a huge chunk of startup time. The linked change makes it disappear.
This should not be related to aggregation at all (as far as I can tell from that tree). Thanks for the fix
Thanks, it's in gnostic master now.


Question: return m.clientset } <ins> func (m TestClientBuilder) DiscoveryClient(name string) (discovery.DiscoveryInterface, error) { return m.clientset.Discovery(), nil } func (m TestClientBuilder) DiscoveryClientOrDie(name string) discovery.DiscoveryInterface { ret, err := m.DiscoveryClient(name) if err != nil { panic(err) } return ret } </ins> // FakeDiscoveryWithError inherits DiscoveryInterface(via FakeDiscovery) with some methods accepting testing data. type FakeDiscoveryWithError struct { fakediscovery.FakeDiscovery

Ran kube-controller-manager with and observed this in the log every 30 seconds: This is also happening in 1.20, 1.19, and 1.18 The GC and quota clients look up discovery information every 30 seconds. They have a QPS of 20 requests per second and a burst of 30 requests. Default configuration of kube-controller-manager runs without constant throttling warnings. /sig api-machinery
/assign /triage accepted


Question: r.Status = result.Status if !r.Status.Authenticated { <del> return nil, false, nil </del> <ins> var err error if len(r.Status.Error) != 0 { err = errors.New(r.Status.Error) } return nil, false, err </ins> } var extra map[string][]string

What happened: I configured my API server with a Webhook token authentication server by way of the "authentication-token-webhook-config-file" command-line flag, and confirmed that the API server does pass bearer tokens through, issuing HTTP POST requests with a JSON-encoded body. When my Webhook server indicates authentication failure, it responds with HTTP status code 200 and a JSON-encoded in the body, this time with the object's "Error" field populated. Here's an example: When the API server receives this response, it refuses the original motivating request with an HTTP status code 401 response, but the API server writes the following into its log file: This use of this default error message is due to an omission in file , where , and . What you expected to happen: The API server should instead log a message that includes the value from . Per a brief discussion with , he agreed that the API server log message should include such an error message from the Webhook server, provided it's present and nonempty. How to reproduce it (as minimally and precisely as possible): Configure an API server to delegate to a Webhook token authentication server. Force that Webhook server to respond "successfully" with an HTTP 200 response, but with the "Authenticated" field set to false (or omitted) and with a particular error message in the "Error" field. Make an HTTP request of the API server with an HTTP "Authentication" header with a value starting with the string "Bearer " followed by at least one more character. Inspect the logs from the API server, looking for a message that begins with "Unable to authenticate the request due to an error." Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: AWS EC2 OS (e.g. from /etc/os-release):Kernel (e.g. ):Install tools: kubeadm_
/sig auth
/assign


Question: selfLinkPrefix = "/" + path.Join("apis", crd.Spec.Group, crd.Spec.Version, "namespaces") + "/" } <ins> clusterScoped := crd.Spec.Scope == apiextensions.ClusterScoped </ins> requestScope := handlers.RequestScope{ Namer: handlers.ContextBasedNaming{ GetContext: func(req *http.Request) apirequest.Context {

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: got an error when trying to use a fieldSelector when watching a custom resource What you expected to happen: no error How to reproduce it (as minimally and precisely as possible): a CRD and then a CR for it. to watch with a fieldSelector, such as Anything else we need to know?: The customresource implementation has a defined, and it looks like it should support at least a field selector for and , but I'm guessing we need a bit more in place to make this work for . Environment: Kubernetes version (use ): HEAD (commit ) /sig api-machinery I can try to submit a PR to fix once I finish my current work item, if nobody picks this up sooner.
/area third-party-resource
cc
This is a more general form of . I don't have immediate plans to work on either of them, so you're welcome to take it.
sounds good. Do you want me to close this as a dupe?
I think there's a useful distinction between the two issues. If we support only , that would be enough to but this one tracks the idea of supporting arbitrary field selectors. By the way, if I recall correctly, when I applied the hack mentioned in the other thread to skip the step for CRDs, I think I saw that actually arbitrary field selectors worked too (not just ). That would mean enabling this might be easy on a technical level, although we may need to think through whether this could tie our hands if we ever wanted to add versioning support for CRD in the future. This issue is a better home for that discussion than the other one.
This would help a lot with debugging in prow. Today, the only thing we can do is bubble up information from the spec of a prowjob as labels and use a label selector.
What about just having 's just return the passed in label and value without any "conversion" for now?
I don't think we'd want to do it for all Unstructured, since there's no guarantee all Unstructured objects are CRD objects. However, it should be pretty easy to arrange for ConvertFieldLabel to be a no-op for CRD objects, which makes sense currently because CRD only allows one version for a given resource.

We just need to make sure this will not prevent us from doing more with CRD versioning in the future. My suspicion is that it should be fine, but I haven't investigated properly.
ok, I can put together a PR to make this change. any gut reactions?
PR:
I am sorry to comment on this closed thread. Currently, I am working on a CRD and needs to support arbitrary field selectors. I've tried to do it by learning from what you do for . But it always throws error: . So, I am wondering if I am doing something wrong or it only supports limited field selectors for CRD, which is stated . Here is my code snippet:
CRDs still only support namespace and name for field selectors
Any plan to support arbitrary field selector for CRDs?
Not currently

Follow a TPR an instance a single instance return 400. (watch a collection of TPR works) master and kubectl version: 1.7.1 CRD has the same issue. /assign
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
I was able to reproduce this with CRD on master. The problem is that kubectl implements a watch on a single object as a watch on the collection with a field selector that filters by . However, field selectors don't work on CRD objects because they are stored as , which does not support : This method is used to transform the field selector from versioned to internal field names: As a test, I bypassed that check for Unstructured objects, and watches on single CRD objects started working. Given that CRD doesn't support multiple versions anyway, it seems feasible to support this with a properly-targeted override of specific to CRD objects. cc
If we could support name and namespace generally on all objects containing object meta, I think I'm in favor of the concept.
I think it's feasible to fix this for CRD at some point. However, given that TPR is already gone in master, I don't think it's worth fixing for TPR unless it's somehow a regression in 1.7. Do you have any objections to that?
Having CRD support is good enough for me and service catalog. Will you backport the fix for CRD to 1.7?
At this stage I wouldn't expect that since it isn't a regression (I don't think) for either extension mechanism.
cc
I am also getting the same error when i try to watch CRDs and filter on a specific field. Im a currently using the to watch for CRDs. On a related note, i noticed that the takes a . Is there a way to pass a label selector so that i can watch CRDs using labels?
Also is this issue slated for 1.8?

(For CRDs that is)
The general issue of supporting field selectors for CRD is tracked in . I don't think it's slated for 1.8. The methods on should take a , which can contain an optional .


Question: OpenAPI:         o.OpenAPISchema, Force:           force, ServerSideApply: o.ServerSideApply, <ins> FieldManager:    o.FieldManager, </ins> ForceConflicts:  o.ForceConflicts, IOStreams:       o.Diff.IOStreams, }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: fails. What you expected to happen: I would expect the PatchOptions to not include FieldManager. The following fix will eliminate the problem. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): 1.16 and 1.17 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig cli
cc
/wg apply
cc
I agree, we clearly missed that. Would you mind sending the fix in the form of a PR? Also please include a test, because that shouldn't have happened if I had tested my code properly :-(
Clearly we should also cherry-pick that into 1.17. I'll send the PR on Monday if you prefer! Thanks.
Ok, I will send a PR. However, I can't include a test immediately because I don't understand how the test works.
Thanks, I’ll try to show you how to get started!
Any update we clearly need this promptly since code-freeze is coming! Thanks.
I'll send a PR on Monday if you haven't! Thanks.
Sorry, I haven't started working on this test. Thanks.