Question: // scoped to the set control groups it understands.  this is being discussed // in https://github.com/opencontainers/runc/issues/1440 // once resolved, we can remove this code. <del> whitelistControllers := sets.NewString("cpu", "cpuacct", "cpuset", "memory", "hugetlb", "systemd") </del> <ins> whitelistControllers := sets.NewString("cpu", "cpuacct", "cpuset", "memory", "systemd") </ins> // If even one cgroup path doesn't exist, then the cgroup doesn't exist. for controller, path := range cgroupPaths {

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: In kubernetes 1.6.4, I configure kubelet with parameter ' --system-reserved=cpu=500m,memory=400Mi --system-reserved-cgroup=/system --enforce-node-allocatable=pods,system-reserved'. kubelet work failed, and output group /system not exist. the output kublet log is: ] Failed to start ContainerManager Failed to enforce System Reserved Cgroup Limits on "/system": "/system" cgroup does not exist. What you expected to happen: kublet work well, and setting --system-reserved can isolate the resources reserved for system process. How to reproduce it (as minimally and precisely as possible): run kubernetes 1.6.4 on fedora atomic 25. and setting the startup parameters(--system-reserved=cpu=500m,memory=400Mi --system-reserved-cgroup=/system --enforce-node-allocatable=pods,system-reserved) for kubelet process Anything else we need to know?: from the kubelet code v1.6.4, know it will first check --system-reserved-cgroup cgroup exist. the check way is see is exist in all subsystem. check the fedora os, I can know [fedora ~]$ sudo find /sys/fs/cgroup -name "" /sys/fs/cgroup/cpu, and check the latest kubelet code, I find the code is like this only check the subsystem defined in whitelistControllers. But the default system, not exists in hugetlb subsystem. and more important, in kubelet, this is not relevant to hugetlb subsystem. I think can delete 'hugetlb' from the whitelistControllers. Environment: Kubernetes version (use ):kubernetes 1.6.4 Cloud provider or hardware configuration**:None OS (e.g.

from /etc/os-release): [fedora cgroup]$ cat /etc/os-release NAME=Fedora VERSION="25 (Atomic Host)" ID=fedora VERSIONID=25 PRETTYNAME="Fedora 25 (Atomic Host)" ANSICOLOR="0;34" CPENAME="cpe:/o:fedoraproject:fedora:25" HOMEURL="" BUGREPORTURL="" REDHATBUGZILLAPRODUCT="Fedora" REDHATBUGZILLAPRODUCTVERSION=25 REDHATSUPPORTPRODUCT="Fedora" REDHATSUPPORTPRODUCTVERSION=25 VARIANT="Atomic Host" Kernel (e.g. ): Linux test2.novalocal 4.10.14-200.fc25.x8664 SMP Wed May 3 22:52:30 UTC 2017 x8664 x8664 x8664 GNU/Linux Install tools: Others:
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/area kubelet /sig node /assign
I encountered same error, but I fixed it now. hope can help you.
I am having a similar issue but with cpuset controller instead of hugetlb on Centos 7. cpuset is not a default controller, so even though it is mounted, no processes are using it. This prevents kubelet from finding ANY cgroups that I specify for --kube-reserved-cgroup or --system-reserved-cgroup. If I simply create a directory, for example at , I stop getting those errors. Is there an issue about changing how cpuset is handled?
ref fixed it


Question: clientNodeInfo, serverNodeInfo nodeInfo ) <del> logContainsFn := func(text string) wait.ConditionFunc { </del> <ins> logContainsFn := func(text, podName string) wait.ConditionFunc { </ins> return func() (bool, error) { <del> logs, err := e2epod.GetPodLogs(cs, ns, podClient, podClient) </del> <ins> logs, err := e2epod.GetPodLogs(cs, ns, podName, podName) </ins> if err != nil { // Retry the error next time. return false, nil

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: n/a Which test(s) are failing: "should resolve connection reset issue [Slow]" Since when has it been failing: it never worked Testgrid link: n/a Reason for failure: The image used for the test is using only IPv4 Anything else we need to know: We should: [x] refactor the code to support IPv6 [ ] ~consolidate the image in agnhost~ [x] configure an IPv6 job to test network https://k8s-,%20IPv6,%20master
/assign /sig-network
GitHub didn't allow me to assign the following users: Billy99. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign /sig-network Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig network
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/remove-triage unresolved
Those labels are not set on the issue: <detailsIn response to : /remove-triage unresolved Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/help wanted
/help wanted /good-first-issue
/remove-good-first-issue the agnhost promotion process is not something easy for new contributors
Hi, I would like to contribute to this.
good, thanks
sorry , I needed to fix it and I took over, thanks for stepping up /assign /remove help-wanted
OK.Yesterday I have refactor the code to support IPv6, but I understand wrong about consolidate the image in agnhost. I'm sorry for not being able to help.
no worries, it was trickier than I thought, thanks again for your help
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: }, NodeName: node.Name, }), true, framework.Logf) <ins> if err != nil { return err } </ins> } for _, node := range nodes { By("Compute Cpu, Mem Fraction after create balanced pods.") computeCpuMemFraction(cs, node, requestedResource) } <ins> return nil </ins> } func computeCpuMemFraction(cs clientset.Interface, node v1.Node, resource *v1.ResourceRequirements) (float64, float64) {

Chronically failing test. Perhaps covers deprecated behavior? /sig scalability /assign
shows up in
/sig scheduling is one of scheduling e2e test.
/assign I'll check this one.
According to the log, the balanced pods were not created by scheduler, the cpu/mem usage was unchanged.
This test failing every other run. Fix or disable? Blocking 1.7?
similar to , move to 1.8, but I'll still working on that.


Question: ) const ( <del> nodeShutdownReason          = "Shutdown" nodeShutdownMessage         = "Node is shutting, evicting pods" nodeShutdownNotAdmitMessage = "Node is in progress of shutting down, not admitting any new pods" dbusReconnectPeriod         = 1 * time.Second </del> <ins> nodeShutdownReason             = "Terminated" nodeShutdownMessage            = "Pod was terminated in response to imminent node shutdown." nodeShutdownNotAdmittedReason  = "NodeShutdown" nodeShutdownNotAdmittedMessage = "Pod was rejected as the node is shutting down." dbusReconnectPeriod            = 1 * time.Second </ins> ) var systemDbus = func() (dbusInhibiter, error) {

When using Graceful Node Shutdown feature, status and reason of "shutdown" pod are confusing for users. See where I attempt to address this confusion by documenting the behavior. However, it would be great to improve messages. Perhaps the following renaming: Reason "Shutdown" is confusing because it may appear as a process of shutting down. Better message will also be useful. Reason and Message changed for pods terminated in response to Node shutdown. Shutdown the node and observer the pod status. KEP: kubernetes/enhancements /cc /sig node /help /good-first-issue Kubernetes version (use ): 1.19+ (with feature enabled)
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : #### What happened: /cc /sig node /help /good-first-issue Kubernetes version (use ): 1.19+ (with feature enabled) </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc
/assign
/assign


Question: // TODO(#40229): Remove the docker-exec-handler flag. fs.StringVar(&c.DockerExecHandlerName, "docker-exec-handler", c.DockerExecHandlerName, "Handler to use when executing a command in a container. Valid values are 'native' and 'nsenter'.") fs.MarkDeprecated("docker-exec-handler", "this flag will be removed and only the 'native' handler will be supported in the future.") <del> fs.StringVar(&c.NonMasqueradeCIDR, "non-masquerade-cidr", c.NonMasqueradeCIDR, "Traffic to IPs outside this range will use IP masquerade.") </del> <ins> fs.StringVar(&c.NonMasqueradeCIDR, "non-masquerade-cidr", c.NonMasqueradeCIDR, "Traffic to IPs outside this range will use IP masquerade. Set to '0.0.0.0/0' to never masquerade.") fs.MarkDeprecated("non-masquerade-cidr", "will be removed in a future version") </ins> fs.StringVar(&c.PodCIDR, "pod-cidr", "", "The CIDR to use for pod IP addresses, only used in standalone mode.  In cluster mode, this is obtained from the master.") fs.StringVar(&c.ResolverConfig, "resolv-conf", c.ResolverConfig, "Resolver configuration file used as the basis for the container DNS resolution configuration.") fs.BoolVar(&c.CPUCFSQuota, "cpu-cfs-quota", c.CPUCFSQuota, "Enable CPU CFS quota enforcement for containers that specify CPU limits")

When the ip-masq-agent is activated, kubelet installs a rule that looks like: which is nonsensical.


Question: // must unmounted prior to detach. // If a volume with the name volumeName does not exist in the list of // attached volumes, an error is returned. <del> SetVolumeGloballyMounted(volumeName v1.UniqueVolumeName, globallyMounted bool) error </del> <ins> SetVolumeGloballyMounted(volumeName v1.UniqueVolumeName, globallyMounted bool, devicePath string) error </ins> // DeletePodFromVolume removes the given pod from the given volume in the // cache indicating the volume has been successfully unmounted from the pod.

We fixed invalid device path in bunch of places via - but call still could use invalid device path. That check can sometimes use invalid path and hence can cause UnmountDevice operation to fails even though device has already been unmounted. /sig storage /sig openstack /assign
Actually this does not affect version 1.10 onwards. Not sure if 1.9 is still supported. I think I will close this for now. /close
I just noticed that 1.9 is still supported and hence it is probably worth fixing there for now.
How come the check can "sometimes" use invalid device path?
So on Cinder for example - call would return something like as attach device path. But if you see the linked PR(), will do a and use that device path for global mount. In general it is always a good idea to use device path returned by as single source of truth. This applies to something like EBS with NVMe too where device is discovered via call (but nvme disks aren't directly affected by this bug because returned device path on EBS is bogus).
So on EBS with NVMe for example - check will always fail (but still return success) because it will use something like as device path which does not actually exist on the node.
You said the issue only happens "sometimes". Can you clarify why it only happens occasionally?
doesn't the linked PR make it clear? detection based on the drive serial number (aka It's cinder ID) on the kubelet itself. As for why - Cinder sometimes returns wrong device path on call, I don't know. But it happens and we have worked around it.
Fixed by


Question: ./cmd/libs/go2idl/deepcopy-gen                     | grep "^$(PRJ_SRC_PATH)"                          | xargs go list -e -f '{{.Dir}} {{.Dir}}/*.go');   <del> echo $$DIRECT $$INDIRECT | sed 's/ / nt/g';        </del> <ins> echo $$DIRECT $$INDIRECT | sed 's/ / =,/g' | tr '=,' 'nt';  </ins> ) | sed "s|$$(pwd -P)/||" > $@.tmp;                     cmp -s $@.tmp $@ || cat $@.tmp > $@ && rm -f $@.tmp

I don't know where the at the beginning of the target comes from, but probably from assumptions about sed version that are wrong. I had to update to gnu-sed 4.2.2 to build.
Also ootb on a Mac doesn't work with the rules specified (on el capitan):
does work
I take it back - updating sed did not appear to work. EDIT: I take that back, I had to clean after fixing paste.
most recent OSX version is UNIX certified, a POSIX requires as file argument when reading input from stdin. For , is only guaranteed to work in GNU sed, you need to insert literal newline and tab character there for portability.
Can someone with a mac send a patch? I don't have a mac...
It's update and every should end in instead of having no arguments. , Tim Hockin wrote:
(user should update sed, make needs changes to paste) , Clayton Coleman wrote:
I think we should stick to portable way for , it's too risky for relying on GNUism. If you don't mind, I will make a PR.
Sure, thanks , Cuong Manh Le wrote:
Are you sure the worked? I pull the latest from master, then run the same command above, I got: File is empty.
Make sure you are in your GOPATH in the correct spot for Kube On Jul 14, 2016, at 2:47 AM, Cuong Manh Le wrote: Are you sure the Makefile worked? I pull the latest from master, then run the same command above, I got: F0714 13:36:50. ] Error: Failed making a parser: unable to add directory "": unable to import "": expected 'package', found 'EOF' make: * [Makefile.generatedfiles:201: gendeepcopy] Error 255 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub , or mute the thread .
If you get the "expected 'package', found 'EOF'" message it is usually because something somewhere failed, and left an empty .go file. make cleangenerated , Cuong Manh Le wrote:
From BSD sed man page: Maybe try that?
It's not about ERE ( enables it), it's about expanding escape sequences in RHS of search and replace .


Question: if auth != nil { server.InstallAuthFilter() } <del> if utilfeature.DefaultFeatureGate.Enabled(features.KubeletTracing) { server.InstallTracingFilter(tp) } </del> server.InstallDefaultHandlers() if kubeCfg != nil && kubeCfg.EnableDebuggingHandlers { server.InstallDebuggingHandlers()

Follow-up to Discussed in As part of kubelet tracing beta, we should add tracing support to the kubelet's read-only endpoint, which currently does not have tracing. It should use WithPublicEndpoint to prevent callers from influencing sampling decisions. Part of
/sig node /sig instrumentation
/triage accepted /assign
can i help you with this issue?
I don't think we'll be able to start with this until merges
Ok, I'm waiting to see if I can help you. Thank you
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale is merged now, so this can be worked on.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
Could I help with this issue?
Sure. See We pass a NoOp Tracer provider to disable tracing for the endpoint. Instead, we need to pass the tracerprovider, but make sure that the eventual uses withPublicEndpoint. Looking closer, i'm not sure its currently possible to use WithPublicEndpoint with the otelrestful package... doesn't contain that option (unlike otelhttp: ), so we may need to add that upstream first.

Another thing to keep in mind is that updating otel libraries is currently blocked by (see my attempt in )
I gave the whole thing another try. At least locally, the dependencies seem now to be able to be updated. :raccoon: Could you maybe have a look at it :)
He's on leave for the time being.
oh, can you provide some feedback or point me to the right people?
I'm back now. Feel free to assign me to PRs related to this, and I'll review
Welcome back :) Could you re-run the CI ? Its just the dependencies. The PR itself shouldn't be that much afterwards.
otel is now up-to-date enough. here:
Hi, I would like to ask about one trouble that caused by WithPublicEndpoint. Aggregated APIServer that I customized uses kubernetes's RecommendedConfig. Due to the fact that WithPublicEndpoint is enabled, it can't be tracked by the full trace for its span-link model. And I think child-parent span model is more appropriate. So, could you please give me some advice? Looking forward to your reply.
thanks. We know about this, and are tracking it in


Question: v1 "k8s.io/api/core/v1" storage "k8s.io/api/storage/v1" <ins> metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" </ins> ) func NewStorageClass(params map[string]string, allowedTopologies []v1.TopologySelectorTerm) *storage.StorageClass {

Was just perusing the code and looks like we translate regional disks into the "zonal volume ID fmt", and there doesn't seem to be any tests for regional disks running so we should confirm that this works and fix it if it doesn't /assign /cc
/sig storage
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/lifecycle frozen /assign
/assign


Question: <ins> /* Copyright 2020 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package common import ( "encoding/json" v1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/types" "k8s.io/apimachinery/pkg/util/uuid" "k8s.io/kubernetes/test/e2e/framework" "github.com/onsi/ginkgo" ) var _ = ginkgo.Describe("[sig-architecture] PodTemplates", func() { f := framework.NewDefaultFramework("podtemplate") ginkgo.It("should run the lifecycle of PodTemplates", func() { testNamespaceName := f.Namespace.Name podTemplateName := "nginx-pod-template-" + string(uuid.NewUUID()) // get a list of PodTemplates (in all namespaces to hit endpoint) podTemplateList, err := f.ClientSet.CoreV1().PodTemplates("").List(metav1.ListOptions{ LabelSelector: "podtemplate-static=true", }) framework.ExpectNoError(err, "failed to list all PodTemplates") framework.ExpectEqual(len(podTemplateList.Items), 0, "unable to find templates") // create a PodTemplate _, err = f.ClientSet.CoreV1().PodTemplates(testNamespaceName).Create(&v1.PodTemplate{ ObjectMeta: metav1.ObjectMeta{ Name: podTemplateName, Labels: map[string]string{ "podtemplate-static": "true", }, }, Template: v1.PodTemplateSpec{ Spec: v1.PodSpec{ Containers: []v1.Container{ {Name: "nginx", Image: "nginx"}, }, }, }, }) framework.ExpectNoError(err, "failed to create PodTemplate") // get template podTemplateRead, err := f.ClientSet.CoreV1().PodTemplates(testNamespaceName).Get(podTemplateName, metav1.GetOptions{}) framework.ExpectNoError(err, "failed to get created PodTemplate") framework.ExpectEqual(podTemplateRead.ObjectMeta.Name, podTemplateName) // patch template podTemplatePatch, err := json.Marshal(map[string]interface{}{ "metadata": map[string]interface{}{ "labels": map[string]string{ "podtemplate": "patched", }, }, }) framework.ExpectNoError(err, "failed to marshal patch data") _, err = f.ClientSet.CoreV1().PodTemplates(testNamespaceName).Patch(podTemplateName, types.StrategicMergePatchType, []byte(podTemplatePatch)) framework.ExpectNoError(err, "failed to patch PodTemplate") // get template (ensure label is there) podTemplateRead, err = f.ClientSet.CoreV1().PodTemplates(testNamespaceName).Get(podTemplateName, metav1.GetOptions{}) framework.ExpectNoError(err, "failed to get PodTemplate") framework.ExpectEqual(podTemplateRead.ObjectMeta.Labels["podtemplate"], "patched", "failed to patch template, new label not found") // delete the PodTemplate err = f.ClientSet.CoreV1().PodTemplates(testNamespaceName).Delete(podTemplateName, &metav1.DeleteOptions{}) framework.ExpectNoError(err, "failed to delete PodTemplate") // list the PodTemplates podTemplateList, err = f.ClientSet.CoreV1().PodTemplates("").List(metav1.ListOptions{ LabelSelector: "podtemplate-static=true", }) framework.ExpectNoError(err, "failed to list PodTemplate") framework.ExpectEqual(len(podTemplateList.Items), 0, "PodTemplate list returned items, failed to delete PodTemplate") }) }) </ins>

This issue proposes the writing of a new test for the API endponts , , , and with the intention of promoting it to Conformance. This issue contains a very basic mock test in order to start the conversation about what a good e2e test would look like for this endpoint. The code below is verified to be hitting the intended endpoint, show as per the queries in APIsnoop's live view of the cluster's audit logs. According to these results from APIsnoop, we are lacking in test coverage and conformant tests for PodTemplate. We chose these three following endpoints for this mock test: - - Outline: PodTemplates, Expect no PodTemplates a PodTemplate (1), Expect no error PodTemplate (1), Ensure the same as how it was created PodTemplate (1) with label a=1, Expect no error PodTemplate (1), Expect label in returned data to be present PodTemplates by label a=1, Expect one PodTemplate PodTemplate (1) Expect no error PodTemplates, Expect no PodTemplates Fetch dependencies: This mock test is using client-go's standard functions to talk to the PodTemplate part of the API. If a test with these calls gets merged in, conformance will go up by 3 endpoints. /sig architecture /area conformance
We decided not to deprecate. We can link to that, but otherwise this is OK to write / promote.
ref: for the decision not to deprecate
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: return utilerrors.NewAggregate(validationErrors) } <ins> // validatePriorityRedeclared checks if any custom priorities have been declared multiple times in the policy config // by examining the specified priority arguments func validatePriorityRedeclared(priorities map[string]schedulerapi.PriorityPolicy, priority schedulerapi.PriorityPolicy) error { var priorityType string if priority.Argument != nil { if priority.Argument.LabelPreference != nil { priorityType = "LabelPreference" } else if priority.Argument.RequestedToCapacityRatioArguments != nil { priorityType = "RequestedToCapacityRatioArguments" } else if priority.Argument.ServiceAntiAffinity != nil { priorityType = "ServiceAntiAffinity" } else { return fmt.Errorf("No priority arguments set for priority %s", priority.Name) } if existing, alreadyDeclared := priorities[priorityType]; alreadyDeclared { return fmt.Errorf("Priority '%s' redeclares custom priority '%s', from:'%s'", priority.Name, priorityType, existing.Name) } priorities[priorityType] = priority } return nil } // validatePredicateRedeclared checks if any custom predicates have been declared multiple times in the policy config // by examining the specified predicate arguments func validatePredicateRedeclared(predicates map[string]schedulerapi.PredicatePolicy, predicate schedulerapi.PredicatePolicy) error { var predicateType string if predicate.Argument != nil { if predicate.Argument.LabelsPresence != nil { predicateType = "LabelsPresence" } else if predicate.Argument.ServiceAffinity != nil { predicateType = "ServiceAffinity" } else { return fmt.Errorf("No priority arguments set for priority %s", predicate.Name) } if existing, alreadyDeclared := predicates[predicateType]; alreadyDeclared { return fmt.Errorf("Predicate '%s' redeclares custom predicate '%s', from:'%s'", predicate.Name, predicateType, existing.Name) } predicates[predicateType] = predicate } return nil } </ins> // validateExtendedResourceName checks whether the specified name is a valid // extended resource name. func validateExtendedResourceName(name v1.ResourceName) []error {

Currently custom predicate and priorities can be created through the . The problem is that a custom predicate/priority, like ServiceAffinity, can be multiple times in the Policy with different names, possibly with different arguments. This semantic (i.e., specifying multiple instances of the same custom predicate with different arguments) is only useful if different s were specified because in all cases of custom predicates/priorities the arguments are merge-able (for example, from multiple CheckServiceAffinity custom predicates can be merged into one set). In the world of framework plugins, we don't currently have a way to maintain those semantics. to run. The question here: do we really want to maintain the old semantics of custom predicates and priorities? in other words, do we need to change the ComponentConfig API to somehow allow specifying multiple instances of the same Plugin with different arguments/weights, or should we consider that this is not supported? I always lean towards simplifying things, so my suggestion is to not support that, but I would like to know if there are actual deployments that rely on this. /sig scheduling /cc
/cc
We discussed this in the sig meeting, and put the issue for discussion on the scheduler and kubernetes-dev mailing list, I believe this behavior is not intentional, and so we should add logic to fail the scheduler configuration when a policy includes more than one custom predicate/priority of the same type. /assign
/assign
any progress on this one?
I haven't made any progress on this yet, but it's on my priority now
In migrating custom predicates to filter plugins , I am gonna ignore the custom predicate name and use the default name.


Question: done return 1 } <ins> kube::test::version::object_to_file() { name=$1 flags=${2:-""} file=$3 kubectl version $flags | grep "$name Version:" | sed -e s/"$name Version: version.Info{"/'/' -e s/'}'/'/' -e s/', '/','/g -e s/':'/'=/g' -e s/'"'/""/g | tr , 'n' > "${file}" } kube::test::version::json_object_to_file() { flags=$1 file=$2 kubectl version $flags --output json | sed -e s/'"'/''/g -e s/'}'/''/g -e s/'{'/''/g -e s/'clientVersion:'/'clientVersion:,'/ -e s/'serverVersion:'/'serverVersion:,'/ | tr , 'n' > "${file}" } kube::test::version::json_client_server_object_to_file() { flags=$1 name=$2 file=$3 kubectl version $flags --output json | jq -r ".${name}" | sed -e s/'"'/''/g -e s/'}'/''/g -e s/'{'/''/g -e /^$/d -e s/','/''/g  -e s/':'/'='/g > "${file}" } kube::test::version::yaml_object_to_file() { flags=$1 file=$2 kubectl version $flags --output yaml | sed -e s/' '/''/g -e s/'"'/''/g -e /^$/d > "${file}" } kube::test::version::diff_assert() { local original=$1 local comparator=${2:-"eq"} local latest=$3 local diff_msg=${4:-""} local res="" if [ ! -f $original ]; then echo ${bold}${red} echo "FAIL! ${diff_msg}" echo "the file '${original}' does not exit" echo ${reset}${red} caller echo ${reset} return 1 fi if [ ! -f $latest ]; then echo ${bold}${red} echo "FAIL! ${diff_msg}" echo "the file '${latest}' does not exit" echo ${reset}${red} caller echo ${reset} return 1 fi sort ${original} > "${original}.sorted" sort ${latest} > "${latest}.sorted" if [ "$comparator" == "eq" ]; then if [ "$(diff -iwB ${original}.sorted ${latest}.sorted)" == "" ] ; then echo -n ${green} echo "Successful: ${diff_msg}" echo -n ${reset} return 0 else echo ${bold}${red} echo "FAIL! ${diff_msg}" echo "  Expected: " echo "$(cat ${original})" echo "  Got: " echo "$(cat ${latest})" echo ${reset}${red} caller echo ${reset} return 1 fi else if [ ! -z "$(diff -iwB ${original}.sorted ${latest}.sorted)" ] ; then echo -n ${green} echo "Successful: ${diff_msg}" echo -n ${reset} return 0 else echo ${bold}${red} echo "FAIL! ${diff_msg}" echo "  Expected: " echo "$(cat ${original})" echo "  Got: " echo "$(cat ${latest})" echo ${reset}${red} caller echo ${reset} return 1 fi fi } </ins>

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): no What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): "kubectl version", "kubectl version" json Is this a BUG REPORT or FEATURE REQUEST? (choose one): Feature request Kubernetes version (use ): $ kubectl version --client Client Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.6", GitCommit:"", GitTreeState:"clean", BuildDate:"2016-11-12T05:22:15Z", GoVersion:"go1.7.1", Compiler:"gc", Platform:"darwin/amd64"} Environment: Cloud provider or hardware configuration: GKE OS (e.g. from /etc/os-release): MacOS 10.12.2 I find it difficult to parse the output of programmatically. I have to use a YAML parser to do it. I'd prefer to use JSON, or to be given just the string and asked to parse it with a regex. This is obviously a relatively low-priority request.
I wouldnt mind helping out on this, if there are no objections?
are you working on this? If not, I would like to work on it.
I plan to, but if i'll keep you posted if i get too delayed.
How can I direct message you?
didnt see this till now, felt bad i had not kept up on this so i decided to just push through it. If my email is not available in my profile, please let me know, otherwise AT gmail would be the best way to contact me.
Thanks!
Included below are some examples. Let me know what you think, my only concern is the the original output of included the key and then the object type which is a little strange. Note the output flag is optional so we don't break previous behavior and contract, which is now baked and part of e2e testing. here are the couple of samples:


Question: } if daemon.Generation <= daemon.Status.ObservedGeneration { if daemon.Status.UpdatedNumberScheduled < daemon.Status.DesiredNumberScheduled { <del> return fmt.Sprintf("Waiting for rollout to finish: %d out of %d new pods have been updated...n", daemon.Status.UpdatedNumberScheduled, daemon.Status.DesiredNumberScheduled), false, nil </del> <ins> return fmt.Sprintf("Waiting for daemon set %q rollout to finish: %d out of %d new pods have been updated...n", name, daemon.Status.UpdatedNumberScheduled, daemon.Status.DesiredNumberScheduled), false, nil </ins> } if daemon.Status.NumberAvailable < daemon.Status.DesiredNumberScheduled { <del> return fmt.Sprintf("Waiting for rollout to finish: %d of %d updated pods are available...n", daemon.Status.NumberAvailable, daemon.Status.DesiredNumberScheduled), false, nil </del> <ins> return fmt.Sprintf("Waiting for daemon set %q rollout to finish: %d of %d updated pods are available...n", name, daemon.Status.NumberAvailable, daemon.Status.DesiredNumberScheduled), false, nil </ins> } return fmt.Sprintf("daemon set %q successfully rolled outn", name), true, nil }

Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: In our CI, we run multiple times in parallel when we update multiple deployments. This lets us see the status of all deployments as they run. When a deployment succeeds, the final message tells us the name of the deployment. However, the intermediate "Waiting for rollout to finish" messages do not say the name of the deployment. This means that in order to look at multiple rollouts in parallel, we need to manually pipe through another command to prefix it with the rollout name. What you expected to happen: Each line to mention the rollout name. How to reproduce it (as minimally and precisely as possible): Update any deployment, run , note that intermediate status names don't contain the deployment name. Anything else we need to know?: Arguably another fix would be to allow you to pass multiple rollouts to and have kubectl handle polling multiple deployments in parallel. Environment: This is still true on master by code inspection.
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: # curl-ing the metrics-server service downloads 14 bytes of unprintable binary # data and sets a return code of success (0). if ! $kubectl exec "$linux_command_pod" --  <del> curl -s -m 20 "http://$service_ip:$service_port" &> $output_file; then </del> <ins> curl -s -m 20 --insecure "https://$service_ip:$service_port" &> $output_file; then </ins> cleanup_deployments echo "Failing output: $(cat $output_file)" echo "FAILED: ${FUNCNAME[0]}"

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Add Support for Windows Server version 2004 and 20H2. Why is this needed: Windows Server version 1909 is going end of life.
/sig windows /triage accepted


Question: // Cache may be overridden by API object with higher version, compare but ignore resource version. newCachedPV := cachedPV.DeepCopy() newCachedPV.ResourceVersion = pv.ResourceVersion <del> if !reflect.DeepEqual(newCachedPV, pv) { t.Errorf("cached PV check failed [A-expected, B-got]:n%s", diff.ObjectDiff(pv, cachedPV)) </del> <ins> if diff := cmp.Diff(pv, newCachedPV); diff != "" { t.Errorf("cached PV check failed (-want, +got):n%s", diff) </ins> } }

When scheduling pod, we encountered the following event , then the scheduler will backoff to reschedule again, but this add more waiting time. From the event message, we can see it is the PVC update conflict leads to the , so need to add retry logic on PVC updating.### Need to add retry logic on PVC updating, to avoid the backoff, to improve the scheduling efficiency. a PVC and then create Pod; scheduing; the PVC annotation or label before the Pod scheduled success; will encounter this issue. No response <details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage note that this is not a supported client/server skew: Server Version: version.Info{Major:"1", Minor:"18+", GitVersion:"v1.18.4-tke.23", GitCommit:"", GitTreeState:"clean", BuildDate:"2022-03-15T09:25:31Z", GoVersion:"go1.15.10", Compiler:"gc", Platform:"linux/amd64"}
/sig scheduling
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue with Mark this issue as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close not-planned [1]:
Closing this issue, marking it as "Not Planned". <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue with Mark this issue as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: } t.Run(gvk.String(), func(t *testing.T) { <del> for i := 0; i < 50; i++ { </del> <ins> for i := 0; i < 3; i++ { </ins> item := fuzzObject(t, gvk) builder := applyconfigurations.ForKind(gvk) jsonRoundTripApplyConfiguration(t, item, builder)

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --pull-kubernetes-unit Two new tests were just in that take significant amounts of time in CI: ~These tests also appear to be flaking regularly: /assign
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Related: At some point we should probably adjust the timeout. I'd like to get CI to a point where it uses whatever the default is in make test.
yes.
This loops each type 50 times, I'll drop that down to say, 3 times. We honestly don't need to fuzz these that hard, they typically catch problems quick.
The failures I see in gubernator shouldn't happen now that the PR is merged. Before myPR to add this test was merged, other PRs that fields (NamespaceSelector) would cause myPR's unit tests to fail and I would need to rebase it and run . Now that it is merged, will be run by the PR author that adds a field to a type to keep code in sync.
ah, so these failures on other PRs mean they needed to regen?
Yeah, the first one adds and the fuzzer detected the apply config builder had not been regenerated by . The others have field changes too.
ok, so it's noise for PRs in progress but not flakes. looks like the runtime/timeout issue is the only flake, which you can drop iterations significantly to solve
50-3 seems like it should get us to 10-15s total for both new tests


Question: <ins> /* Copyright 2022 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package e2enode import ( "context" "fmt" v1 "k8s.io/api/core/v1" "k8s.io/apimachinery/pkg/api/resource" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/kubernetes/test/e2e/framework" e2epod "k8s.io/kubernetes/test/e2e/framework/pod" admissionapi "k8s.io/pod-security-admission/api" "github.com/onsi/ginkgo/v2" libcontainercgroups "github.com/opencontainers/runc/libcontainer/cgroups" ) type testCase struct { name                   string podSpec                *v1.Pod oomTargetContainerName string } var _ = SIGDescribe("OOMKiller [LinuxOnly] [NodeConformance]", func() { f := framework.NewDefaultFramework("oomkiller-test") f.NamespacePodSecurityEnforceLevel = admissionapi.LevelPrivileged testCases := []testCase{{ name:                   "single process container", oomTargetContainerName: "oomkill-single-target-container", podSpec: getOOMTargetPod("oomkill-target-pod", "oomkill-single-target-container", getOOMTargetContainer), }} // If using cgroup v2, we set memory.oom.group=1 for the container cgroup so that any process which gets OOM killed // in the process, causes all processes in the container to get OOM killed if libcontainercgroups.IsCgroup2UnifiedMode() { testCases = append(testCases, testCase{ name:                   "multi process container", oomTargetContainerName: "oomkill-multi-target-container", podSpec: getOOMTargetPod("oomkill-target-pod", "oomkill-multi-target-container", getOOMTargetContainerMultiProcess), }) } for _, tc := range testCases { runOomKillerTest(f, tc) } }) func runOomKillerTest(f *framework.Framework, testCase testCase) { ginkgo.Context(testCase.name, func() { ginkgo.BeforeEach(func() { ginkgo.By("setting up the pod to be used in the test") e2epod.NewPodClient(f).Create(context.TODO(), testCase.podSpec) }) ginkgo.It("The containers terminated by OOM killer should have the reason set to OOMKilled", func() { ginkgo.By("Waiting for the pod to be failed") err := e2epod.WaitForPodTerminatedInNamespace(context.TODO(), f.ClientSet, testCase.podSpec.Name, "", f.Namespace.Name) framework.ExpectNoError(err, "Failed waiting for pod to terminate, %s/%s", f.Namespace.Name, testCase.podSpec.Name) ginkgo.By("Fetching the latest pod status") pod, err := f.ClientSet.CoreV1().Pods(f.Namespace.Name).Get(context.TODO(), testCase.podSpec.Name, metav1.GetOptions{}) framework.ExpectNoError(err, "Failed to get the recent pod object for name: %q", pod.Name) ginkgo.By("Verifying the OOM target container has the expected reason") verifyReasonForOOMKilledContainer(pod, testCase.oomTargetContainerName) }) ginkgo.AfterEach(func() { ginkgo.By(fmt.Sprintf("deleting pod: %s", testCase.podSpec.Name)) e2epod.NewPodClient(f).DeleteSync(context.TODO(), testCase.podSpec.Name, metav1.DeleteOptions{}, framework.PodDeleteTimeout) }) }) } func verifyReasonForOOMKilledContainer(pod *v1.Pod, oomTargetContainerName string) { container := e2epod.FindContainerStatusInPod(pod, oomTargetContainerName) if container == nil { framework.Failf("OOM target pod %q, container %q does not have the expected state terminated", pod.Name, container.Name) } if container.State.Terminated == nil { framework.Failf("OOM target pod %q, container %q is not in the terminated state", pod.Name, container.Name) } framework.ExpectEqual(container.State.Terminated.ExitCode, int32(137), fmt.Sprintf("pod: %q, container: %q has unexpected exitCode: %q", pod.Name, container.Name, container.State.Terminated.ExitCode)) framework.ExpectEqual(container.State.Terminated.Reason, "OOMKilled", fmt.Sprintf("pod: %q, container: %q has unexpected reason: %q", pod.Name, container.Name, container.State.Terminated.Reason)) } func getOOMTargetPod(podName string, ctnName string, createContainer func(name string) v1.Container) *v1.Pod { return &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name: podName, }, Spec: v1.PodSpec{ RestartPolicy: v1.RestartPolicyNever, Containers: []v1.Container{ createContainer(ctnName), }, }, } } // getOOMTargetContainer returns a container with a single process, which attempts to allocate more memory than is // allowed by the container memory limit. func getOOMTargetContainer(name string) v1.Container { return v1.Container{ Name:  name, Image: busyboxImage, Command: []string{ "sh", "-c", // use the dd tool to attempt to allocate 20M in a block which exceeds the limit "sleep 5 && dd if=/dev/zero of=/dev/null bs=20M", }, Resources: v1.ResourceRequirements{ Requests: v1.ResourceList{ v1.ResourceMemory: resource.MustParse("15Mi"), }, Limits: v1.ResourceList{ v1.ResourceMemory: resource.MustParse("15Mi"), }, }, } } // getOOMTargetContainerMultiProcess returns a container with two processes, one of which attempts to allocate more // memory than is allowed by the container memory limit, and a second process which just sleeps. func getOOMTargetContainerMultiProcess(name string) v1.Container { return v1.Container{ Name:  name, Image: busyboxImage, Command: []string{ "sh", "-c", // use the dd tool to attempt to allocate 20M in a block which exceeds the limit "dd if=/dev/zero of=/dev/null bs=20M & sleep 86400", }, Resources: v1.ResourceRequirements{ Requests: v1.ResourceList{ v1.ResourceMemory: resource.MustParse("15Mi"), }, Limits: v1.ResourceList{ v1.ResourceMemory: resource.MustParse("15Mi"), }, }, } } </ins>

Follow up on meeting of 28-03-2022 - In the meeting was specified that this most likely won't need a KEP and should be started with a Issue I see that the kernel in cgroups v2 has an option which would make all the pids of a given container to get killed on OOM: Some workloads that fork the process in the container, when the OOM kicks in, it kills only one process of the cgroup, causing a lot of noise due to process failures but health checks being successfull. We think that for our scenarios, killing all the pids in the containter would be better.
/cc
The way I’d summarize this issue with is: A small number of important workloads handle processes and OOM correctly (nginx, mature multi process systems like DBs) - we should not prevent these workloads from benefitting from per process OOM. A much larger and more diverse set of workloads are unlikely to handle OOM of multi process behavior incorrectly (bash child processes, especially in entrypoint scripts) - these users are unlikely to test OOM behavior in advance and thus are much more vulnerable to behaving incorrectly if we don’t default to killing the whole container. That also apples to containers with shared process namespaces. We generally bias towards the later group (low expertise / high diversity / low frequency / unlikely to read documentation) when designing our pod semantics, while allowing advanced users to set optional flags.
/cc
/cc
/sig node
/triage accepted /priority important-longterm


Question: imageutils.GetE2EImage(imageutils.TestWebserver), imageutils.GetE2EImage(imageutils.VolumeNFSServer), imageutils.GetE2EImage(imageutils.VolumeGlusterServer), <ins> imageutils.GetE2EImage(imageutils.NonRoot), </ins> ) type testImagesStruct struct {

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: ci-kubernetes-node-kubelet-orphans Which test(s) are failing: [] Security Context should run with an image specified user ID [It] [] Security Context When creating a container with runAsNonRoot should not run with an explicit root user ID [LinuxOnly] [It] Since when has it been failing: Testgrid link: Reason for failure: "" is not in the white list Anything else we need to know:
/area test /sig node


Question: fi } <ins> # Deletes a security group # usage: delete_security_group <sgid> function delete_security_group { local -r sg_id=${1} echo "Deleting security group: ${sg_id}" # We retry in case there's a dependent resource - typically an ELB n=0 until [ $n -ge 20 ]; do $AWS_CMD delete-security-group --group-id ${sg_id} > $LOG && return n=$[$n+1] sleep 3 done echo "Unable to delete security group: ${sg_id}" exit 1 } </ins> function ssh-key-setup { if [[ ! -f "$AWS_SSH_KEY" ]]; then ssh-keygen -f "$AWS_SSH_KEY" -N ''

See example log below. I'm pretty sure the SG in question was from an ELB, but we should just auto-retry anyway.


Question: // Remove unidentified containers. for _, container := range unidentifiedContainers { glog.Infof("Removing unidentified dead container %q with ID %q", container.name, container.id) <del> err = cgc.dockerClient.RemoveContainer(docker.RemoveContainerOptions{ID: container.id}) </del> <ins> err = cgc.dockerClient.RemoveContainer(docker.RemoveContainerOptions{ID: container.id, RemoveVolumes: true}) </ins> if err != nil { glog.Warningf("Failed to remove unidentified dead container %q: %v", container.name, err) }

I'm conducting PoC about distributed build using Kubernetes. Jenkins master distribute jenkins slave pods through Kubernetes API Server. When build is done, jenkins slave pod is also deleted by kubernetes API Server. But, kubernetes didn't delete container. (Pod is deleted well) After searching, I found kubelet’s containerGC option. So, I adjusted below options to kubelet. --minimum-container-ttl-duration=0s --maximum-dead-containers=0 After adjusting both 2 options, container was deleted, too. But, kubernetes containerGC didn’t delete host volume path. (/var/lib/docker/vfs/dir/..) Due to aufs’s low performance, I used data volume of host instead of container inside. So, data volume was accumulated and host's inodes was fulled, too. Is there any solution to clean the containers wreckage? My environment was.. Docker: v1.6.2 Kubernetes: v0.18.2 (container based multi clustering)
are you using the host volumes you created yourself? If this is the case, kubelet does not clean the volumes since they are external to kubelet. If not, kubelet should clean the volumes before container GC kicks in.
No.. volumes were created by kubenetes API server. LGTM. Will it be adjusted next release?
You said "I used data volume of host instead of container inside" which says you used a hostPath mount - correct? That is explicitly NOT managed by kubernetes - you've gone to the host, you get to manage it. But you also said "Due to aufs’s low performance" - an emptyDir volume is NOT stored on aufs, but it IS managed with the pod. , gghonor wrote:
The issue as i understand it is dangling volumes, that is volumes that are created via a VOLUME directive in Dockerfile with no corresponding kubernetes volume in spec which therefore has no host path & is not cleaned up by the kubelet. These volumes reside in /var/lib/docker/vfs/. This waste of disk space has the potential to cause out of space errors on the node & therefore make the node unstable.
As the volume dir you mentioned is I assume you didn't specify any volume in your Kubernetes pod spec but did use the directive in your Dockerfile?

As mentions I guess this isn't a Kubernetes managed volume as that would be in the Kubernetes volume dir, right? That dir is used by Docker for volumes specified in Dockerfiles or a docker container that is run with with no specified host path. The latter doesn't happen with Kubernetes managed containers AFAIK.
This makes sense. Thanks for the explanation :) I see the point of removing the dangling volumes. Even though kubelet doesn't manage those volumes, it doesn't hurt to remove them.
Do we have a full understanding of their lifetime, though? We don't control them, I don't see how we can know when to delete them. Docker's volume implementation is just too under-defined to actually be useful. , Yu-Ju Hong wrote:


Question: configs[APIServer] = Config{e2eRegistry, "sample-apiserver", "1.17"} configs[AppArmorLoader] = Config{e2eRegistry, "apparmor-loader", "1.0"} configs[BusyBox] = Config{dockerLibraryRegistry, "busybox", "1.29"} <del> configs[CheckMetadataConcealment] = Config{e2eRegistry, "metadata-concealment", "1.2"} </del> <ins> configs[CheckMetadataConcealment] = Config{promoterE2eRegistry, "metadata-concealment", "1.6"} </ins> configs[CudaVectorAdd] = Config{e2eRegistry, "cuda-vector-add", "1.0"} configs[CudaVectorAdd2] = Config{e2eRegistry, "cuda-vector-add", "2.0"} configs[DebianIptables] = Config{buildImageRegistry, "debian-iptables", "buster-v1.3.0"}

Warning: The v0.1 and v1beta1 metadata server endpoints are deprecated and scheduled for shutdown on September 30, 2020. This test (which runs in most e2e jobs that run on GCE): Schedules a container that runs this: Resulting in a number of legacy metadata endpoints getting hit. To see if this test is going to start failing across all release branches on Sep 30th, we could try disabling the endpoints ahead of time via: In fact and support for expecting different results based on whether is set. My concern is the test expects 200 for some root endpoints, and 403 for disabled endpoints. However the e-mail I received as a GCP customer tells me any legacy endpoints may return 404. I'm inclined to suggest we remove the legacy endpoints from the check, bump the image version, and cherry-pick use of the new image version back to all release-branches. WDYT since you were involved with the earlier PRs /sig auth /sig cloud-provider /provider gcp /sig testing /priority important-soon (I'm tempted to critical-urgent this but it's not drop-everything yet) /milestone v1.20
/kind cleanup
Heyho I wanted to check how it is going? The code freeze is starting 12.11.2020 (about 3-4 weeks from now) and while there is still plenty of time, we want to ensure that each relevant issue has a chance to be considered. As the issue is tagged for 1.20, is it still planned for this release?
Bug Triage here, we're pretty late into code freeze and it doesn't look like there has been much activity recently. I am going to clear the milestone on this issue. /milestone clear
any update here?
Try to cleanup the legacy endpoints in
/milestone v1.21
/reopen This needs to stay open until the new image has been verified to work and has been promoted (ref: )
Reopened this issue. <detailsIn response to : /reopen This needs to stay open until the new image has been verified to work and has been promoted (ref: ) Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage accepted /assign


Question: driverNodeID, maxVolumePerNode, accessibleTopology, err := csi.NodeGetInfo(ctx) if err != nil { <del> klog.Error(log("registrationHandler.RegisterPlugin failed at CSI.NodeGetInfo: %v", err)) </del> if unregErr := unregisterDriver(pluginName); unregErr != nil { klog.Error(log("registrationHandler.RegisterPlugin failed to unregister plugin due to previous error: %v", unregErr)) <del> return unregErr </del> } return err } err = nim.InstallCSIDriver(pluginName, driverNodeID, maxVolumePerNode, accessibleTopology) if err != nil { <del> klog.Error(log("registrationHandler.RegisterPlugin failed at AddNodeInfo: %v", err)) </del> if unregErr := unregisterDriver(pluginName); unregErr != nil { klog.Error(log("registrationHandler.RegisterPlugin failed to unregister plugin due to previous error: %v", unregErr)) <del> return unregErr </del> } return err }

What happened: We currently unregister the CSI driver if an error happens while registering it. Then, if another error happens, we return only the unregistration error. The problem is that the user will only see the unregistration error if they check the registrar logs. This gives the impression that it never tried to register the driver. What you expected to happen: The user should see the registration error instead. How to reproduce it (as minimally and precisely as possible): Try to install a CSI driver in a cluster where CSINodeInfo and/or CSIDriver CRDs are not created. Check registrar logs /kind bug /sig storage CC
/assign


Question: newPodPreset := obj.(*settings.PodPreset) oldPodPreset := old.(*settings.PodPreset) <ins> pod.DropDisabledVolumeMountsAlphaFields(oldPodPreset.Spec.VolumeMounts) pod.DropDisabledVolumeMountsAlphaFields(newPodPreset.Spec.VolumeMounts) </ins> // Update is not allowed newPodPreset.Spec = oldPodPreset.Spec }

says "Before persisting the object to storage, clear disabled alpha fields". There is that actually implements it but there are some discussions about it. This issues is a reminder that introduced by needs to be cleared too when the discussion in settles down. /assign
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig api-machinery
kind/cleanup priority/important-soon
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/api-machinery: Issue will be escalated to these SIGs if needed. priority/important-soon: Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. kind/bug: Fixes a bug discovered during the current release. <detailsAdditional instructions available <a href=""here</a</details


Question: # Admission Controllers to invoke prior to persisting objects in cluster ENABLE_ADMISSION_PLUGINS="LimitRanger,ResourceQuota" <del> DISABLE_ADMISSION_PLUGINS="ServiceAccount,PersistentVolumeLabel,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,StorageObjectInUseProtection" </del> <ins> DISABLE_ADMISSION_PLUGINS="ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,StorageObjectInUseProtection" </ins> # Include RBAC (to exercise bootstrapping), and AlwaysAllow to allow all actions AUTHORIZATION_MODE="RBAC,AlwaysAllow"

In 1.30, the PersistentVolumeLabel admission plugin blocks creation of in-tree AzureDisk and vSphere PVs: The reason is that the cloud provider was removed (), but the admission plugin still calls GetCloudProvider there, which fails: The PV should be admitted + created. Enable PersistentVolumeLabel admission plugin. For example, when running local-up- Create AzureDisk PV on any 1.30 cluster, you don't need Azure cloud: /sig storage /priority important-soon <detailsToday's master </details<detailsAzure </detailsNo response <detailslocal-up- </details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I am fixing the worst bug in and I plan to backport it to 1.30. However, we should consider removing the whole PersistentVolumeLabel admission plugin. Most of the cloud providers were removed, only GCE remains and IMO even that one will be removed sooner or later. The admission plugin has no way to get volume zone + region and is therefore useless. The admission said (in 1.26, where we had all in-tree cloud providers): Such webhook never materialized for AWS, Azure nor vSphere and I am not aware of any plans for GCE.
It did materialize in So removing it from k/k is the best way forward.


Question: }) expectNoError(err) defer c.Pods(ns).Delete(labelPodName, api.NewDeleteOptions(0)) <del> expectNoError(waitForPodRunningInNamespace(c, labelPodName, ns)) </del> <ins> // check that pod got scheduled. We intentionally DO NOT check that the // pod is running because this will create a race condition with the // kubelet and the scheduler: the scheduler might have scheduled a pod // already when the kubelet does not know about its new label yet. The // kubelet will then refuse to launch the pod. expectNoError(waitForPodNotPending(c, ns, labelPodName)) </ins> labelPod, err := c.Pods(ns).Get(labelPodName) expectNoError(err) Expect(labelPod.Spec.NodeName).To(Equal(nodeName))

Jenkins project kubernetes-e2e-gke-ci build: One more flakiness blocks entire submit queue.
cc/
Looking into this.
do you have a longer test trace for me? I cannot see much that could be flaky due to the tests. It just does this: start pod without-label remember NodeName delete without-label set label <some-uuid=42 on NodeName launch pod with-label and NodeSelector set to {"": nodeName, <some-uuid":42} The only idea I have is that the patch command does not work reliably (assuming that the NodeSelector support in the scheduler works and is not flaky itself):
here is the most recent failure from our Jenkins build:
thanks for the logs. This clarifies what happens: the kubelet does not know its new labels. Maybe, its node storage didn't update fast enough and there is a race with the scheduler:
This might be fixed with .


Question: detect-project echo "Bringing down cluster" <ins> set +e  # Do not stop on error </ins> <del> gcloud preview managed-instance-groups --zone "${ZONE}" delete  </del> <ins> # The gcloud APIs don't return machine parsable error codes/retry information. Therefore the best we can # do is parse the output and special case particular responses we are interested in. deleteCmdOutput=$(gcloud preview managed-instance-groups --zone "${ZONE}" delete  </ins> --project "${PROJECT}"  --quiet  <del> "${NODE_INSTANCE_PREFIX}-group" || true </del> <ins> "${NODE_INSTANCE_PREFIX}-group") if [[ "$deleteCmdOutput" != ""  ]]; then # Managed instance group deletion is done asyncronously, we must wait for it to complete, or subsequent steps fail deleteCmdOperationId=$(echo $deleteCmdOutput | grep "Operation:" | sed "s/.*Operation:s//" | sed "s/s.*//" | sed "s/ //g") if [[ "$deleteCmdOperationId" != ""  ]]; then deleteCmdStatus="PENDING" while [[ "$deleteCmdStatus" != "DONE" ]] do sleep 5 deleteCmdOperationOutput=$(gcloud preview managed-instance-groups --zone "${ZONE}" get-operation $deleteCmdOperationId) deleteCmdStatus=$(echo $deleteCmdOperationOutput | grep -i "status:" | sed "s/.*status:s//" | sed "s/s.*//" | sed "s/ //g") echo "Waiting for MIG deletion to complete. Current status: " $deleteCmdStatus done fi fi </ins> gcloud compute instance-templates delete  --project "${PROJECT}"  --quiet  <del> "${NODE_INSTANCE_PREFIX}-template" || true </del> <ins> "${NODE_INSTANCE_PREFIX}-template" </ins> # First delete the master (if it exists). gcloud compute instances delete

'kubernetes-soak-weekly-deploy' sometimes fails to deploy because the clean-up cluster step fails to delete the template: Then the subsequent cluster step fails to create the template, because it already exists: It keeps retrying and failing, because the template already exists, until it times out and the deployment fails. Suggested fixes: Cluster down should try harder to delete the template (maybe wait/retry). Cluster up should recognize that the template exists, and try to delete/recreate, or just re-use the existing.
Looks like ran in to this on Monday, May 11, 2015 around 1:00 PM PST. I'll assign this to my self and take a crack at fixing it.
This is causing most of our e2e tests to fail in continuous integration as well as users private e2e test clusters. Bumping to priority/p0 and v1.0 milestone. Saad thinks he's figured it out, and is working on a fix as we speak.
In essence... Saad: "I think what's happening is the previous clean up step to delete the managed instance group happens asynchronously, therefore there is a race condition to delete the template. Working on a fix".


Question: import ( "sync" "time" <ins> "k8s.io/apimachinery/pkg/util/clock" </ins> ) const (

Instead of relying on the clock: we should do something similar to the golang rate-limitter is doing:
/sig api-machinery /priority important-longterm
/help
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hi, I would like to help on this.
/assign Thanks! /triage accepted
can you please provide more context or some guidance on how to proceed?
Sorry for delayed response. - what we should do is to basically: instead of having an additional go-routine that every 1s refreshes the budget change it roughly to: (a) keep the value of budget and the last time when we were refreshing it (b) on takeAvailable method: first recompute takeAvailable, which will be "min(current budget + time elapsed since last refresh * refreshValue, maxAllowedBudget) update last refresh time to now return that value [We also would need to update the existing tests to match the new logic.] - would you be willing to work on it?
-- I can take this up. One question I had was should the method rely on or the current time should be passed on by the caller like in the following: Having the caller pass the time completely removes the dependency on the clock at least for the implementation.
/assign
Let's may be start with: passing on budget creation to it relying on
-- thank you for the thoughts. I drafted a rough implementation in which still depends on . I will make the change as you suggested and rely solely on a instance passed on initialization. I have to rethink the testing strategy a bit, possibly using a .
Yes - FakeClock is the way to test this.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity.

Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-contributor-experience at . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Reopening - this didn't happen.


Question: <ins> # See the OWNERS docs at https://go.k8s.io/owners approvers: - sig-scheduling-maintainers reviewers: - sig-scheduling labels: - sig/scheduling </ins>

/kind cleanup /kind documentation /sig scheduling /sig node /sig apps The ownership of controller/nodelifecycle is unclear. The currently has some individual names and no labels, although it gets a sig/apps label from the parent pkg/controller OWNERS. This controller has the following functionality: Add taints based on health of the node (from conditions) Evict Pods based on NoExecute taints. The candidates for long term ownership are: : The ones who introduced it and overall maintain taints and tolerations. : Because it's called "node lifecycle", although this could be a misnomer. : Due to implications to the lifecycle of applications and because they are the main shepherds of kcm. I think it makes the most sense under scheduling, but wanted to gather all stakeholder's feedback.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
cc
cc as original author
I think both scheduling and node can own it, if node is too overwhelmed, sig scheduling makes sense as a home.
Thanks, It would be nice to sort out the ownership. It sounds reasonable for sig-scheduling to own it. /cc
Outcome from SIG Scheduling meeting: As already identified, this controller actually has 2 functionalities. From a technical standpoint, the following ownership makes sense to SIG Scheduling leads: Setting taints based on Node conditions -SIG Node Evicting pods that don't tolerate NoExecute taints -SIG Scheduling This ownership structure can be set-up even before is implemented, as the taint functionality has its own folder. I'll create a PR based on the above. But and please raise any concerns.
+1 on this proposal.
+1 on the proposal - I believe if we were to assign responsibility to who owns node level invariants that apply to workloads, it would be sig-node (they generally own node + pod). pod-safety, for instance, I considered owned by sig-node even though it touches storage, apps, api-machinery, and scheduling because the node owns the pod at the time safety is imposed.


Question: NominatedNodeName: "node1", }, }) <ins> nonExistentPodInfo = framework.NewPodInfo(&v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name:      "ne", Namespace: "ns1", UID:       "nens1", }, }) scheduledPodInfo = framework.NewPodInfo(&v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name:      "sp", Namespace: "ns1", UID:       "spns1", }, Spec: v1.PodSpec{ NodeName: "foo", }, }) ) </ins> func getUnschedulablePod(p *PriorityQueue, pod *v1.Pod) *v1.Pod { pInfo := p.unschedulablePods.get(pod)

The scheduler cache entered a corrupted state where a Pod, according to the cache dump, was nominated to node A, while the status said it was nominated to node B. This lead to unschedulable low priority pods due to the phantom pod in the nominator cache. The cache to match the Pod status Setup: two nodes with all the resources used. A high priority pod comes in that is able to preempt at least one pod. This might cause these sequence of events X doesn't fit, it gets nominated to and back to scheduling queue. start a second scheduling attempt for Pod X. get a Pod update for the . This puts the pod back into the scheduling queue. second scheduling attempt finishes and schedules the pod successfully in start a third scheduling attempt for Pod X (due to 3) get a Pod update for the . This should have cleared the nominated node in the cache, but there is nothing currently there. the third scheduling attempt, the pod doesn't fit, so the scheduler nominates it to . Nothing clears this nomination afterwards Usually step 2 happens after 3, so everything works as expected. But in a busy cluster, 3 might be delayed. No response <detailsGKE </details<details</details<details</details<details</details<details</details
/sig scheduling cc
This Pod actually got scheduled: The cache dump was obtained much later at : I think the problem is that we got an update for the Pod (probably the nominated node itself), scheduled it again and assigned a second nominated node for it. The second scheduling attempt failed, but we didn't clear the cache.
/triage accepted
/priority important-soon
Sounds like is related?
It could be the same, or it could be another bug. Note that this is a second scheduling attempt for a Pod that was already scheduled.
My theory of events: X doesn't fit, it gets nominated to and back to scheduling queue. start a second scheduling attempt for Pod X. get a Pod update for the . This puts the pod back into the scheduling queue. second scheduling attempt finishes and schedules the pod successfully in start a third scheduling attempt for Pod X (due to 3) get a Pod update for the . This should have cleared the nominated node in the cache, but there is nothing currently there.

the third scheduling attempt, the pod doesn't fit, so the scheduler nominates it to . Not sure why this pod update fails, but nevertheless, we have a corrupted cache at this point.
thanks for raising up this and the fix. Could you update the "How can we reproduce it (as minimally and precisely as possible)?" section a bit? Per our offline conversation, the issue will leave a "phantom" pod holding a number of resources in the internal nominator cache. And due to that unnecessary resource reservation, low-priority pods may not get scheduled while they should have been. This might be the key trigger of this issue. In practice, it can be caused by a (way-too-quick) movePodBackToActiveQ call (due to relevant/irrelevant events, or backoff's completion) after step 1 and before step 3.
Updated the description. I will update the PRs on Monday.


Question: "testing" "github.com/GoogleCloudPlatform/kubernetes/pkg/client" <ins> "github.com/GoogleCloudPlatform/kubernetes/pkg/kubectl/cmd" </ins> ) <ins> func TestExtraArgsFail(t *testing.T) { buf := bytes.NewBuffer([]byte{}) f, _, _ := NewAPIFactory() c := f.NewCmdCreate(buf) if cmd.ValidateArgs(c, []string{"rc"}) == nil { t.Errorf("unexpected non-error") } } </ins> func TestCreateObject(t *testing.T) { _, _, rc := testData() rc.Items[0].Name = "redis-master-controller"

cc
When deleting/stopping using file, you don't specify resource (in this case 'pod'), as it's inferred from the file.
Thanks and sorry for confusion
Ok, but should not work as well.


Question: return successfulDeletes, errorFromChannel(errCh) } <ins> func nonIgnoredFailedPodsCount(jobCtx *syncJobCtx, failedPods []*v1.Pod) int { result := len(failedPods) if feature.DefaultFeatureGate.Enabled(features.JobPodFailurePolicy) && jobCtx.job.Spec.PodFailurePolicy != nil { for _, p := range failedPods { _, countFailed, _ := matchPodFailurePolicy(jobCtx.job.Spec.PodFailurePolicy, p) if !countFailed { result-- } } } return result } </ins> // deleteJobPods deletes the pods, returns the number of successful removals // and any error. func (jm *Controller) deleteJobPods(ctx context.Context, job *batch.Job, jobKey string, pods []*v1.Pod) (int32, error) {

Pod backoff calculation excludes pods that match an rule in the failure policy: This could lead to fast retries, overwhelming the control plane, if most failures match an rule. The semantics of is not to count towards the limit, but the backoff should still be respected. /sig apps Recreations to be throttled. a Job with failure policy that matches every kind of failure. a container that always fails You would observe immediate recreation. No response_ 27+ <detailsN/A </details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
cc


Question: }) } <ins> func (e *eventBroadcasterImpl) Shutdown() { e.Broadcaster.Shutdown() } </ins> func recordToSink(sink EventSink, event *v1.Event, eventCorrelator *EventCorrelator, sleepDuration time.Duration) { // Make a copy before modification, because there could be multiple listeners. // Events are safe to copy like this.

In order to initialize a client-go , you need an . and friends initialize a new : launches a goroutine: This goroutine is stoppable with . However, the interface returned above does not expose anything that calls that shutdown method, AFAICT. This means that that goroutine cannot ever be cleaned up. That's not great. It's fine if you're using in like k/k does and never stopping anything, but if you ever want to stop your controllers and start up new ones, it becomes an issue. Tested/tracked down against master (as of ). /sig apimachinery /kind bug
The label(s) cannot be applied. These labels are supported: <detailsIn response to : In order to initialize a client-go , you need an . and friends initialize a new : /kind bug </details
/sig api-machinery I always forget the dash


Question: return } volumesNeedUpdate := make(map[v1.UniqueVolumeName]*reconstructedVolume) <ins> volumeNeedReport := []v1.UniqueVolumeName{} </ins> for _, volume := range podVolumes { <del> if rc.desiredStateOfWorld.VolumeExistsWithSpecName(volume.podName, volume.volumeSpecName) { glog.V(4).Infof("Volume exists in desired state (volume.SpecName %s, pod.UID %s), skip cleaning up mounts", volume.volumeSpecName, volume.podName) continue } </del> if rc.actualStateOfWorld.VolumeExistsWithSpecName(volume.podName, volume.volumeSpecName) { glog.V(4).Infof("Volume exists in actual state (volume.SpecName %s, pod.UID %s), skip cleaning up mounts", volume.volumeSpecName, volume.podName) <ins> // There is nothing to reconstruct </ins> continue } <ins> volumeInDSW := rc.desiredStateOfWorld.VolumeExistsWithSpecName(volume.podName, volume.volumeSpecName) </ins> reconstructedVolume, err := rc.reconstructVolume(volume) if err != nil { <ins> if volumeInDSW { // Some pod needs the volume, don't clean it up and hope that // reconcile() calls SetUp and reconstructs the volume in ASW. glog.V(4).Infof("Volume exists in desired state (volume.SpecName %s, pod.UID %s), skip cleaning up mounts", volume.volumeSpecName, volume.podName) continue } // No pod needs the volume. </ins> glog.Warningf("Could not construct volume information, cleanup the mounts. (pod.UID %s, volume.SpecName %s): %v", volume.podName, volume.volumeSpecName, err) rc.cleanupMounts(volume) continue } <ins> if volumeInDSW { // Some pod needs the volume. And it exists on disk. Some previous // kubelet must have created the directory, therefore it must have // reported the volume as in use. Mark the volume as in use also in // this new kubelet so reconcile() calls SetUp and re-mounts the // volume if it's necessary. volumeNeedReport = append(volumeNeedReport, reconstructedVolume.volumeName) glog.V(4).Infof("Volume exists in desired state (volume.SpecName %s, pod.UID %s), marking as InUse", volume.volumeSpecName, volume.podName) continue } // There is no pod that uses the volume. </ins> if rc.operationExecutor.IsOperationPending(reconstructedVolume.volumeName, nestedpendingoperations.EmptyUniquePodName) { glog.Warning("Volume is in pending operation, skip cleaning up mounts") }

/kind bug What happened: kubelet logs lot of when it starts and a pod was deleted while it was offline. How to reproduce it (as minimally and precisely as possible): a pod with AWS EBS volume (should work also with GCE or any other volume) kubelet the pod kubelet It's not always reproducible, is there some sort of race? My current theory: desiredStateOfWorldPopulator scans pods and adds this pod to DSW reconciler.syncStates() runs. it finds mounted volume but it does not add it to actual state of world because it's in DSW: When kubelet fully starts, it kills the pod and cleans DSW, but the volume never reached AWS so nothing is unmounted. Environment: Kubernetes version (use ): today's master I have troubles reproducing this on master. /sig storage /sig node
Ok, check at won't add the volume to ASW. But the volume is in DSW and subsequent reconcile() should check that the volume is attached and mounted and add it to ASW. This does not happen, because: Volume is not in because it was overwritten by new kubelet (?)
This problem goes beyond a simple logging error. The problem also is - such volumes never get unmounted correctly and hence volume can not be used anywhere. IMO this is kind of serious. Steps to reproduce: a pod on node X kubelet the pod. The pod will be stuck in "Terminating" state back the kubelet There is a race which is rather easy to reproduce and what happens is - volume gets to DSW at kubelet but it never makes into ASW because volume is indeed never in use on node (because pod was deleted while kubelet was down). This results in volume never getting unmounted and hence any attempt to use the volume on different node fails.
IMO, this happens: is called only once at kubelet startup and at that point the pod still exists (it's in Terminating state). The volume is in DSW and it's not to ASW. is called periodically. It could add the volume to ASW, but it does not, because the volume is not in yet. kills pod containers and confirms it's deleted. The pod is deleted from API server. removes the pod from DSW is called and the volume is not there because it's not in DSW.

We have a mounted volume that's not in DSW nor in ASW. There are several possible solutions: a. kubelet can add the volume to ASW in regardless of DSW. b. kubelet can update node status earlier, just after . Will there still be race when pod is deleted during and thus it won't reach ? c. kubelet could call regularly. I am inclined to a), it looks robust to me and it just reflects reality in ASW. It should not matter if the volume is in DSW or not at this point - will sort it out and do the right thing.
solution d.: in reconciler.syncStates() try to reconstruct volume from DSW, fall back to only if the volume is not there. DSW already has spec and other useful info.
Experimental fix (solution d.):
FYI, this might be another 1.10 (or 1.10.1) candidate. We need thorough review though, volumeManager is quite complicated.
another simpler option is perhaps to mark a volume as "ReportedInUse" if it was found on the disk during reconstruction phase. That will make sure that, volume gets correctly to ASW via reconciler(rather than us doing it on reconstruction) and it will make sure that volume is unmounted when volume does get removed from DSW.
I closed , it would blow up when kubelet was restarted at wrong time when creating a pod. Interesting idea, I opened with it. I am not sure it covers all the corner cases though. Newly started kubelet does not overwrite its own status until runs, I hope it's ensured by something and not just a coincidence.
I think might still hit a race condition issue as below is normally deleted (without force delete) during kubelet restart restarts, pod is received again so it is add to the pod Manager is to desired state, so reconciler skip to reconstruct to the actual state kill containers and then try to delete the Pod from API server. If actual state has not yet updated from reconciler, pod will be deleted from pod manager and desired state too will reconcile actual state and desired state. But if step 4 happened first, it will not be able to update actual state since volume is removed from desired state.

Depending on which happens first between step 4 and 5, volume might or might not have a chance to get into the actual state. I am thinking seems better to solve the issue with some additional protection. The idea is to avoid letting kubelet to delete pod before actual state has updated the volume. We can set a flag in actual state to indicate when reconstruction finishes.
what hook you are planning to use to stop kubelet from deleting pod unless it has been processed into asw?
My thought is to add a flag (e.g., finishReconstruct) in actualstateofworld to indicate whether reconstruction finished or not. volumemanager will add a function (e.g., IsActualStateReconstructed) to check the flag is set or not. When call canBeDeleted() function, the PodResourcesAreReclaimed in will return false if actual state has not yet finished reconstructing.
Talked with offline, come up with a solution which combines both and . The proposal still follows the idea of pod is to desired state (no matter whether it might be removed soon or not), do not reconstruct directly (reconstruction does not has the full information which might fail or cause problem having wrong information since there is not enough checking), instead reconcile should have a chance to reconcile actual state by going through the volume set up process. if the pod was force deleted from API server during kubelet down, reconciler tries to clean up mounts directly if possible. The main problem currently is that pod might be deleted from desired state before reconciler updates the actual state. is trying to prevent of this. makes sure that reconciler can update the actual state as soon as possible.
please add this to 1.10 milestone. This should be a 1.10 blocker.
SIG-leads, please confirm, thanks!
/priority critical-urgent /status in-progress /kind bug
You must be a member of the github team to add status labels.
Adding to 1.10 as blocker per
/status in-progress
[MILESTONENOTIFIER] Milestone Issue: Up-to-date for process Note: This issue is marked as , and must be updated every 1 day during code freeze. Example update: <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed.

: Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
I spoke with offline. The volume reconstruction work (PR ) is non-critical but touches critical mount/unmount code paths. While the PRs that and have ( and ) may resolve the issue identified here, I'm afraid there may be other regressions introduced that we are not aware of. Therefore, I'm asking Jing to revert the volume reconstruction PR. We can reintroduce it along with the patches early in 1.11 along with more extensive testing. Unfortunately, because changes went in on top, an automatic revert is not possible. Jing is working on a manual revert. could you please help test her revert PR when it is ready?
Update on this issue: spent the morning trying to create a revert PR , but it turns out a couple of other big refactors (including ) went in on top of her volume reconstruction PR, making it much more difficult to revert. At this point says she can either spend more time backing out her changes and, in doing so, possibly introduce new bugs or spend time making sure the proposed fixes are rock solid. She says she feels confident that the bugs are understood, and the proposed PRs fixing it should be sufficient, and would like to spend time to double and triple checking it with instead. So and will proceed with this.
i support this approach
I support this as well. Thank you and for working on this.
okay I integrated both the PRs in a branch and tested both on AWS and GCE . Ran e2e tests as well and it looks all good.
I lied about e2e test run. A second run surfaced some failures:
the error message indicates the pods failed to enter running condition. I think my fix should not affect pod creation.
yeah I agree. I noticed that these tests use NFS based PVs which are not ideal source for testing the bug we are trying to reproduce. In this case the NFS server itself may not start correctly and hence the test may fail.

I am looking into refactoring the test suite, so as it can use GCE/EBS sources as well.
I created a follow up issue for creating the disruptive e2e tests so that bugs like these can be more easily caught - . Will follow up with PR tomorrow. Also ran e2e test after removing NFS part and just relying on GCE-PD and it all passed. I am going to let run couple of hours in a loop and see if there is a regression.
thanks for your work here. How did it come out?
we still have some problems that needs to be addressed for . I will sync up with when online.
Perfect, thank you.
/reopen
you can't re-open an issue/PR unless you authored it or you are assigned to it. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
can we keep it open, we still need the second PR
/cc
re-opening per
The second PR got merged late night yesterday as well. So we can close this. /close


Question: resourceName: testCase.Name, resource:     testCase.Resource, } <del> body, err := ioutil.ReadAll(testCase.Res.Body) if err != nil { t.Errorf("failed to read body: %v", err) } _, _, err = r.transformResponse(testCase.Res, testCase.Req, body) </del> <ins> result := r.transformResponse(testCase.Res, testCase.Req) err := result.err </ins> if !testCase.ErrFn(err) { t.Errorf("unexpected error: %v", err) continue

Storing r.req / r.resp internally (for use from DoRaw and Do) is probably not the right factoring. There should be a common method doRaw that returns request and body.


Question: testableResources := etcd.GetEtcdStorageDataForNamespace(f.Namespace.Name) for _, group := range apiGroups { <ins> // This limits the scope of this test to exclude CRDs. This // assumes that CRDs will not have a .k8s.io group and will have // a . in their name. if !strings.Contains(group.GroupVersion, ".k8s.io") && strings.Contains(group.GroupVersion, ".") { continue } </ins> for _, resource := range group.APIResources { if !verbsContain(resource.Verbs, "get") || ignoredResources[resource.Kind] || strings.HasPrefix(resource.Name, "e2e-test") { continue

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: gce-master-scale-correctness Which test(s) are failing: [sig-cli] Kubectl client kubectl get output should contain custom columns for each resource Since when has it been failing: 11/6 Testgrid link: https://k8s- Reason for failure: Anything else we need to know: n/a /sig cli /sig scalability /cc
The label(s) cannot be applied. These labels are supported: <detailsIn response to : <!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --gce-master-scale-correctness [sig-cli] Kubectl client kubectl get output should contain custom columns for each resource 11/6 https://k8s- Anything else we need to know: n/a /sig scalability /cc Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Doesn't seem to be scale-related - failed for me the same way in the presubmit:
- do we run correctness test in smaller scale (so we can detect whether failures are scale related or not)? I can't find them on neither https://k8s- nor https://k8s- dashboards.
This is a new test that seem to be flaky at head: - seems that you this in Can you please take a look?
/assign
/reopen Let's wait for the test to pass /assign
Reopened this issue. <detailsIn response to : /reopen /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
That seems to be creating one of every object from the etcd data test in an e2e test. The failing object looks like a node object, which is likely getting deleted by the Node controller as soon as it is created.
opened to skip nodes for this test
/reopen The job is not green
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.

</details
failed test run started from a commit prior to the fix merging ( started from ) /close
Closing this issue. <detailsIn response to : failed test started from a commit prior to the fix merging ( started from ) Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: } } <ins> func TestRootSelfLink(t *testing.T) { storage := map[string]rest.Storage{} simpleStorage := GetWithOptionsRootRESTStorage{ SimpleTypedStorage: &SimpleTypedStorage{ baseType: &genericapitesting.SimpleRoot{}, // a root scoped type item: &genericapitesting.SimpleRoot{ ObjectMeta: metav1.ObjectMeta{Name: "foo"}, Other:      "foo", }, }, takesPath: "atAPath", } storage["simple"] = &simpleStorage handler := handle(storage) server := httptest.NewServer(handler) defer server.Close() resp, err := http.Get(server.URL + "/" + prefix + "/" + testGroupVersion.Group + "/" + testGroupVersion.Version + "/simple/foo") if err != nil { t.Fatalf("unexpected error: %v", err) } if resp.StatusCode != http.StatusOK { t.Errorf("Unexpected status: %d, Expected: %d, %#v", resp.StatusCode, http.StatusOK, resp) body, err := ioutil.ReadAll(resp.Body) if err != nil { t.Fatalf("unexpected error: %v", err) } t.Logf("Data: %s", string(body)) } var out genericapitesting.SimpleRoot if _, err := extractBody(resp, &out); err != nil { t.Fatalf("unexpected error: %v", err) } if out.SelfLink != "/"+prefix+"/"+testGroupVersion.Group+"/"+testGroupVersion.Version+"/simple/foo" { t.Errorf("unexpected self link: %#v", out) } } </ins> func TestMetadata(t *testing.T) { simpleStorage := &MetadataRESTStorage{&SimpleRESTStorage{}, []string{"text/plain"}} h := handle(map[string]rest.Storage{"simple": simpleStorage})

Bug report Kubernetes version Client Version: version.Info{Major:"1", Minor:"5+", GitVersion:"v1.5.0-beta.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2016-11-24T22:35:03Z", GoVersion:"go1.7.3", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"5+", GitVersion:"v1.5.0-beta.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2016-11-24T22:30:23Z", GoVersion:"go1.7.3", Compiler:"gc", Platform:"linux/amd64"} When you curl the api you get a list of third party resources, however their selflinks are missing a forward slash:
fixed with , you may take a look.
This looks like it affects root scoped resources in general -- both custom ones (e.g. ServiceCatalog brokers, as noted in kubernetes-incubator/service-catalog) and "built-in" ones, like namespaces and nodes:
cc


Question: cmdutil.AddFilenameOptionFlags(cmd, options, usage) cmd.Flags().StringP("selector", "l", "", "Selector (label query) to filter on, supports '=', '==', and '!='.(e.g. -l key1=value1,key2=value2)") cmd.Flags().Bool("all-namespaces", false, "If present, list the requested object(s) across all namespaces. Namespace in current context is ignored even if specified with --namespace.") <del> cmd.Flags().BoolVar(&describerSettings.ShowEvents, "show-events", true, "If true, display events related to the described object.") </del> <ins> cmd.Flags().BoolVar(&describerSettings.ShowEvents, "show-events", describerSettings.ShowEvents, "If true, display events related to the described object.") </ins> cmdutil.AddInclude3rdPartyFlags(cmd) cmdutil.AddIncludeUninitializedFlag(cmd) return cmd


Question: }, expectedErr: fmt.Errorf("failed to find a primary nic for the vm. vmname="vm""), }, <ins> { description:       "GetPrimaryInterfaceID should report an error if there's no network interface on the VMSS VM", existedInterfaces: []compute.NetworkInterfaceReference{}, expectedErr:       fmt.Errorf("failed to find the network interfaces for vm vm"), }, </ins> } for _, test := range testCases {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: kube-controller-manager panic when network interfaces for Azure VMSS are null. This happens when Azure cloud provider tried to get IP address for VMSS VM: What you expected to happen: kube-controller-manager should report an error instead of panic. How to reproduce it (as minimally and precisely as possible): Scale cluster nodes up and down frequently. Anything else we need to know?: /kind bug /area provider/azure Environment: Kubernetes version (use ): 1.15+ Cloud provider or hardware configuration: Azure OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig cloud-provider /assign


Question: if state.SkipFilterPlugins.Has(pl.Name()) { continue } <del> metrics.PluginEvaluationTotal.WithLabelValues(pl.Name(), metrics.Filter, f.profileName).Inc() </del> if status := f.runFilterPlugin(ctx, pl, state, pod, nodeInfo); !status.IsSuccess() { if !status.IsUnschedulable() { // Filter plugins are not supposed to return any status other than

We observed degradation in scheduling throughput, putting it under 100 pods/s. Respective pprof of kube-scheduler points it to ! /sig scheduling /kind regression 100 pods/node n/a No response <details</details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc
/sig scalability
Thanks for noticing it /assign
just to confirm this bug is present only in 1.27 ( introduced with pluginevaluationtotal metric) , isn't it ?
Exactly, 1.27+ (so the current master as well) to be specific.