Question: const ( archiveName  = "e2e_node_test.tar.gz" <del> CNIRelease   = "8a936732094c0941e1543ef5d292a1f4fffa1ac5" </del> <ins> CNIRelease   = "9d5e6e60e79491207834ae8439e80c943db65a69" </ins> CNIDirectory = "cni" )

We should update cni binaries to containernetworking/cni and release binaries for all arches. I do not have push access to the repo, so I guess some googler have to do it.
Thanks. I have this on my plate.


Question: APIVersion: "v1", FieldPath:  "spec.containers[2]", }, <del> Reason:  "Started", Message: "some verbose message: 1", Source:  v1.EventSource{Component: "eventTest"}, Count:   1, Type:    v1.EventTypeNormal, </del> <ins> Reason:              "Started", Message:             "some verbose message: 1", Source:              v1.EventSource{Component: "eventTest"}, ReportingController: "eventTest", Count:               1, Type:                v1.EventTypeNormal, </ins> }, expectLog:    `Event(v1.ObjectReference{Kind:"Pod", Namespace:"baz", Name:"foo", UID:"bar", APIVersion:"v1", ResourceVersion:"", FieldPath:"spec.containers[2]"}): type: 'Normal' reason: 'Started' some verbose message: 1`, expectUpdate: false,

From the Deprecated API Migration Guide: , since v1.19, the has been using the new reportingController and reportingInstance to replace the decrecated source.component and But when I checked the Kube events in a cluster with version 1.26.0, those new fields are empty and the supposed to be deprecated fields have not been renamed. Seeing the values in the new fields like reportingController and reportingInstance an Kubernetes cluster with version 1.26.0 some Kube events the Kube events details for the field of reportingController, reportingInstance, source.component, No response <detailsServer Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2022-12-11T00:15:47Z", GoVersion:"go1.19.4", Compiler:"gc", Platform:"linux/amd64"} </details<detailsAzure </details<details</details<details</details<details</details<details</details
/sig instrumentation
I have also encountered this issue in my local 1.26.0 Kubernetes cluster, which does not align with the deprecation guide described in I believe I can investigate this further
/assign
Thank you for making the PR to fix this! Also a comment in the PR for the doc update.
/triage accepted


Question: restoreStrategy := statefulSet.Spec.UpdateStrategy statefulSet.Spec.UpdateStrategy = oldStatefulSet.Spec.UpdateStrategy <del> if !reflect.DeepEqual(statefulSet.Spec, oldStatefulSet.Spec) { </del> <ins> if !apiequality.Semantic.DeepEqual(statefulSet.Spec, oldStatefulSet.Spec) { </ins> allErrs = append(allErrs, field.Forbidden(field.NewPath("spec"), "updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden.")) } statefulSet.Spec.Replicas = restoreReplicas

/kind bug When deploying a StatefulSet, I included a PVC that specified for the storage capacity. worked perfectly. Running again without making any changes failed with the error: After comparing the server-side yaml to my original, I noticed that it had changed the storage capacity of my PVC to . Changing my original from to allowed me to then successfully run . There is clearly a string representation being used to compare those changes rather than converting to bytes and comparing the numeric representation. How to reproduce it (as minimally and precisely as possible): Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-06-27T22:29:25Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.5", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-06-21T11:34:22Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: Azure AKS
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
cc This is probably an issue for any resource.Quantity type.
/sig api-machinery
the issue is use of in statefulset update validation, instead of : explicitly compares quantity objects semantically:
/sig apps
fix is in with picks open to 1.11, 1.10, 1.9


Question: containers: - name: %s image: %s <del> restartPolicy: %s </del> command: ["sh", "-c", "i=0; while [ $i -lt %d ]; do %s i=$(($i+1)); done; while true; do sleep 5; done"] <ins> restartPolicy: %s </ins> ` file := staticPodPath(dir, name, namespace) <del> podYaml := fmt.Sprintf(template, name, namespace, name, image, string(restart), iterations, command) </del> <ins> podYaml := fmt.Sprintf(template, name, namespace, name, image, iterations, command, string(restart)) </ins> f, err := os.OpenFile(file, os.O_RDWR|os.O_TRUNC|os.O_CREATE, 0666) if err != nil {

! /kind failing-test <!-- If this is a flake, please add: /kind flake --/sig node
Sidecar PR is merged in 07/08. /cc Another potential root cause is according to
The last successful run is on Jul 05, 2023.
/triage accepted
! FYI: ^ this test started failing on 07/08.
Yes, the spec is wrong. I guess it was trying to use pod's not container's (which was ignored before the SidecarContainers PR) I'll propose a fix. /assign


Question: "//pkg/volume/scaleio:go_default_library", "//pkg/volume/secret:go_default_library", "//pkg/volume/storageos:go_default_library", <del> "//pkg/volume/util/nsenter:go_default_library", </del> "//pkg/volume/util/subpath:go_default_library", "//pkg/volume/vsphere_volume:go_default_library", "//staging/src/k8s.io/api/core/v1:go_default_library",

There was a previous attempt : We have open issues that is not being worked on: I think it's time to delete the CI jobs and start the deprecation cycle. /sig node /assign
We discussed this at today's SIG Node meeting. SIG Node agreed to deprecate "containerized kubelet" with the following action items: The project was initiated for self-host cluster by SIG cluster lifecycle. We should communicate with them first to understand the potential dependencies. There were some demand from the community for this feature in the past, but we don't know who are using this in their production. We should announce this deprecation at Kubernetes Community weekly meeting and send email to kubernetes-dev
will do thanks
/sig storage /sig cluster-lifecycle
cc cc
Does this deprecate based in any way? I have a project that depends on this and I think it is a real use case for a container based . Running either a based or a dedicated container is key to our design.
it doesn't. the main thing to go would be this flag, if you are not using it, you are good. [1]
What is the reason behind this decision? We are heavily using containerized in production running on CoreOS. Would that mean that we no longer can run kubelet in a container?
Do you use the flag as shown here? Do you have mounts as shown just below the line pointed above If so, you have about a 3 releases per our deprecation policies to switch over to some other way of running kubelet. RedHat/CoreOS folks who initially championed this approach have given up on it, there are no test suites, no owners. sig-storage is having a hard time trying to keep things working. So please ping the CoreOS folks to research for an alternative for the future. Note that still runs kubelet in a container, it just does not use the flag or volume mounts as shown above.
Is it necessary to remove/deprecate the feature or just disable the flaky CI check? I find that feature and running kubelet in container helpful. Briefly looking at the implementation there is at least unit test suit.

that's the tip of the ice berg, there's a lot more things that can be simplified when we drop this feature, example see Another example of what we had to go through is here - There is no appetite to continue taking on this burden. I'll let and others chime in on the undue cost of keeping this alive.
fwiw I don't think we should deprecate the entire use case b/c it's highly valuable for testing. I'm not certain why this matters for the kubelet, b/c it really shouldn't unless it's a design flaw.
can you please chime in with the ongoing cost of supporting this feature? how much effort is to keep it going?
Yes we use this list of volume: This seems to work pretty nice for us and we don't have any volume issues so far.
Maintenance of containerized kubelet is increasingly painful. Kubelet / its storage code must know about all mounts on the host, resolve symlinks on the host, execute various helpers on the host and similar stuff, which is especially hard when it does not actually run on the host but in a weird container. See and for all the hacks we use. They are brittle and tend to break in very unexpected way. It's also largely undocumented and various ways how to run kubelet in a container are cargo-culted around. See above, , really? No offense, I know why it's there, but whole containerized kubelet is hack on top of hack on top of hack.
Not only is the code complex, but it is also not tested, so we have had to rely on manual testing when making bug fixes, and we don't know about regressions until after the release when users report issues. sig-storage has tried for 6+ months to find an owner that can dedicate resources to support and maintain this environment, but nobody has stepped up.
+1 to what asnd said. Either we should deprecate the feature, or find an owners who can commit to fixing issues and adding much needed tests. We tried to find an owner in SIG storage during Q2 2018 but no one stepped up.
Just because I like causing trouble, another alternative is to run executable from a container (or systemd portable unit, etc), but unify the kubelet's rootfs view with that of the host. Specifically: do the equivalent of , but for .

Then you would run from docker with (or equivalent), and early in chroot() to /rootfs. This loads the executable from the docker image, but all later filesystem operations are done from a pov equivalent to the host root fs. I think this keeps the kubelet lifecycle management benefits of running the kubelet binary from a container and allows the storage folks to just do regular filesystem operations, like there was nothing clever going on.
The function needs ownership from both SIG Node and SIG Storage. There were attempts early in the project to remove containerized kubelet support, but Red Hat offered contributors to maintain this model in both SIGs for ~11 releases of Kubernetes. Kubernetes was much smaller 11 releases ago and our shared project experience was limited. It's one thing for a feature to work initially, and another thing to reason about the impact of alternative deployment models when dealing with CVE responses or just doing normal maintainer review of incoming PRs or feature function. That lived experience is informing our future choices at Red Hat to focus on just deploying kubelet as systemd unit (like many other Kubernetes service providers), but if there are others in the community that want to keep the function alive and maintain it we need owners to volunteer and commit to fix issues and add much needed tests. If not, it seems the responsible thing for the community is to clearly deprecate it.
I like this suggestion to simplify the mechanism while retaining the packaging and runtime benefits of the container image. I'm curious what we're doing in talos if we're not using the flag and how what we're doing solves the use-case of running kubelet inside a container.
I kind of like the idea of mounting to as well. Although, one of the benefits we see in mounting only the needed host paths is that it is less of the host filesystem exposed in case of a security vulnerability. On the other hand we currently mount everything we think we need: and that could change in any release. The above solution would allow us to never worry about that again really. We don't use the flag, and I can't say I have seen any of the mentioned issues in our setup. We pass Sonobuoy conformance tests as well. So I can't say I would know exactly what we are doing to solve these problems since I personally have ran into them. I have had rook run just fine as well.

Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: __kubectl_parse_get "${nouns[${#nouns[@]} -1]}" } <ins> __kubectl_get_resource_pod() { __kubectl_parse_get "pod" } </ins> # $1 is the name of the pod we want to get the list of containers inside __kubectl_get_containers() {

TODOs: Last updated on Sept 2 by Janet [x] kubectl label - suggest a resource type [x] kubectl get and delete also use kubectl.NewHumanReadablePrinter to get the resources back, thus should be fixed as well. [x] kubectl exec - suggest the pod name [x] kubectl rolling-update - suggest the rc name [x] check that is doing the right thing [x] Distribute with kubernetes binaries [x] Put information about kubectl completion in user guide [x] docs/getting-started- Recently some users reported that there is no completion for the kubectl command. But there is. Unfortunately it is not documented how to install it. I found that all needs to be done is to: (or maybe just copy this file into /etc/bash_completion.d/) This information should be put into the user guide (maybe it is there - I just can't find it) I think it should be included for instance here: docs/getting-started- There are also some minor issues: does not suggest the rc name does not suggest the pod name does not suggest a resource type (maybe this is ok because we can give --overwrite here)
It looks like the bash-completion file is not distributed with kubernetes binaries:
:+1: This should definitely be provided with the binaries.
Good suggestions. Cc
, please cofirm that the bash completion file is distributed with k8s binaries: namely contrib/completions/bash/kubectl For me it looks like it isn't.
cc
I confirmed and it isn't.
TODOs: [x] kubectl label - suggest a resource type [x] kubectl get and delete also use kubectl.NewHumanReadablePrinter to get the resources back, thus should be fixed as well. [x] kubectl exec - suggest the pod name [x] kubectl rolling-update - suggest the rc name [x] check that is doing the right thing [x] Distribute with kubernetes binaries [x] Put information about kubectl completion in user guide [x] docs/getting-started- Last updated on Sept 2
Maybe add: check that is doing the right thing....
Thanks for the suggestion. Checklist updated.
Depending on where you want it in the tarball, is a good place to start. You can use and then iterate on until you're comfortable with the changes.

(If I'm making packaging changes, I'll frequently start by dumping the sorted output of that command into a file, then later diff against it to make sure I got exactly the change I wanted.)
Right now both and brings up all containers that has. Since is optional (the without -c way is supported as legacy behavior), this is expected. I'm not sure how to force the bash completion to print instead of container names when since flag isn't set as required in code.


Question: } // ServiceBackendPort is the service port being referenced. <ins> // +structType=atomic </ins> type ServiceBackendPort struct { // name is the name of the port on the Service. // This is a mutually exclusive setting with "Number".

I created an Ingress resource with a port number manually, and then tried to use an Operator with server-side apply to update the resource from a port number to a port name. This failed with the error message "cannot set both port name & port number". I expected the port number to be cleared as the operator would take over the ownership of the ServiceBackendPort. To simulate the behavior in the operator in a most simple example I use : ingress- ingress- This will print: It seems to me that the merge type for the struct ServiceBackendPort should to be set to "atomic" , so that the Operator updating the port should take over the whole struct, and two applications shouldn't update single fields in it. Red Hat OpenShift on AWS OS version of the client: NAME="Fedora Linux" VERSION="40 (Workstation Edition)" Linux 6.9.6-200.fc40.x8664 SMP PREEMPTDYNAMIC Fri Jun 21 15:48:21 UTC 2024 x86_64 GNU/Linux
/sig network
So, on one hand you made a problem when you used two different owners to manage one aspect of the object. On the other hand, I think you're right that it SHOULD work. Probably. is not a super common tag, and ingress way predates it, so no surprise it isn't used. I ACK the issue and reproduced it. I ran update-codegen and built a KinD cluster from HEAD.
this may be something you want to look at for DRA API - I didn't even realize this was a thing until today :)
The API uses for , which is very similar. So using it for LGTM. SSA will auto-migrate existing resources using once the change is released.
/triage accepted
"this" being "add "? I can see where this might be useful, probably on all one-of structs and perhaps on those containing a discriminator because otherwise SSA leaves fields set which should be cleared when some other owner takes over. Is this important enough to put into I don't see a big need for SSA in general and even less for SSA with different owners at the moment - this will change once we do more with status.
No, not urgent. This is the first time it has come up in my awareness ever. But now that I know about it, I could see how it applies to dra.

Thank to everyone involved here to resolve this! This will make things simpler in the Keycloak operator once it is released.
Thanks for filing a good (and easy to fix :) bug. Easy repro means we can jump on it.


Question: create-static-ip "${MASTER_NAME}-ip" "${REGION}" MASTER_RESERVED_IP=$(gcloud compute addresses describe "${MASTER_NAME}-ip"  --project "${PROJECT}" --region "${REGION}" -q --format='value(address)') <del> KUBELET_APISERVER="${MASTER_RESERVED_IP}" </del> <ins> if [[ "${REGISTER_MASTER_KUBELET:-}" == "true" ]]; then KUBELET_APISERVER="${MASTER_RESERVED_IP}" fi </ins> KUBERNETES_MASTER_NAME="${MASTER_RESERVED_IP}" create-certs "${MASTER_RESERVED_IP}"

Kubernetes version (use ): HEAD Environment: GCE What happened: When I set up I can see the following message: What you expected to happen: master should not be registered
cc
Looks like the env var isn't being respected any longer by the master scripts (but the logic for # of expected nodes did adjust correctly).
/cc I think this may have broken in because it which unconditionally sets the field that tells a kubelet to register with the master outside of the guard that checks .
Fix in progress: . There was a problem caused by unconditional setting on the master in .


Question: } } <del> if !qosLimitsFound.HasAll(string(corev1.ResourceMemory), string(corev1.ResourceCPU)) { </del> <ins> if !qosLimitsFound.HasAll(string(core.ResourceMemory), string(core.ResourceCPU)) { </ins> isGuaranteed = false } } if len(requests) == 0 && len(limits) == 0 { <del> return corev1.PodQOSBestEffort </del> <ins> return core.PodQOSBestEffort </ins> } // Check is requests match limits for all resources. if isGuaranteed {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: In kubectl library, logic is not the same as the one in What you expected to happen: logic should be the same. How to reproduce it (as minimally and precisely as possible): Create a pod with a guaranteed QoS class and a best-effort QoS initcontainer. Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig cli
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
It seems to be duplicate functions here.
The intention for duplicates is coming from the fact that kubectl can't rely on packages from kuberentes main repository. Either way they should both be identical except for the fact that the internval version will use internal APIs, whereas kubectl's version will use versioned.

feel free to open a PR updating kubectl version updating it to match both versions.
/triage accepted /assign
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: configuration data, with the running set of containers by starting or stopping Docker containers.`, Run: func(_ *Server, _ []string, stopCh <-chan struct{}) error { <ins> if s.ExperimentalDockershim { return app.RunDockershim(&s.KubeletFlags, &s.KubeletConfiguration) } </ins> return app.Run(s, nil) }, }


Question: attachChanged         bool shareProcessedChanged bool <del> podClient corev1client.PodsGetter </del> <ins> podClient corev1client.CoreV1Interface </ins> genericclioptions.IOStreams }

changed the Kind used with the subresource from to . This change will break old clients. (The ephemeral containers API is still in alpha, so this is acceptable.) Currently, compatibility is broken in both directions: versions prior to 1.22 are incompatible with server version 1.22 version 1.22 is incompatible with server versions prior to 1.22 Remediating the first requires complex server-side backwards compatibility logic which is not worth implementing for an alpha API. Remediating the second for should be much easier. The struct hasn't changed, only the path in the object. We could detect the kind returned by the server and change the path used in the JSON patch (or even just naively try the previous path if the server returns an error). This behavior would be removed after the old server version falls out of the support window. This isn't strictly needed, but it would improve UX for what I expect is a common scenario: using the latest kubectl with older servers. /sig cli /priority important-soon
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
what do you think about adding this backwards compatibility behavior to kubectl?
Are we able to detect this and fail gracefully client side with a warning to users to upgrade their client? Agree that this fix is the important one. I wonder if a commit for backwards compatibility is something we can cherry pick to other release branches? Seeing as earlier branches will have both structs it might even be cleaner to do version conditional in kubectl there? is that something we've done before?
Just to make in my head a matrix of effects/noneffects, please confirm for me if these are right: 1.22 will have the compatibility logic baked in. So it will be able to talk to older servers. -picking that logic to older branches would not alter in any way how pre-1.22 kubectls interact with old (pre 1.22) servers.

cherry-pick would only add the needed logic to allow old kubectls to talk to 1.22 and later servers Are these assumptions correct?
Correct. I haven't prototyped it, but this should be possible. We could trigger the logic on a server error that indicates it received an unexpected Kind. That being said, I don't think it's worth cherry-picking forward compatibility into older versions of kubectl. This is for an alpha feature that must be explicitly enabled with the feature flag. It's not unreasonable to require people experimenting with an alpha feature to use a kubectl version = server version. There's far more risk than benefit. It would be more of a burden to keep around old versions of kubectl for older clusters, so I thought adding backwards compatibility in 1.22 would be nice, even if it's not strictly required. I agree it would be nice if we could give users of pre-1.22 kubectl a better error message, but I at least have kubernetes/website out to indicate a minimum kubectl version requirement of 1.22.
I'd like to take up this issue. /assign
How's it going? I'd like to make sure this makes it into 1.22.
Hi , I am able to reproduce this issue. I am looking for some guidance on how to go about fixing this issue. You have mentioned that "detect the kind returned by the server and change the path used in the JSON patch ", where is the kind being returned by the server and what JSON patch are you referring to?
Hi given that time zones are difficult and the code freeze is near, I think it'll be most efficient to communicate with examples. I'll prepare a PR with what I have in mind.


Question: imagePullTest(image, false, v1.PodPending, true, false) }) <del> ginkgo.It("should not be able to pull non-existing image from gcr.io [NodeConformance]", func() { image := imageutils.GetE2EImage(imageutils.Invalid) imagePullTest(image, false, v1.PodPending, true, false) }) ginkgo.It("should be able to pull image from gcr.io [NodeConformance]", func() { image := imageutils.GetE2EImage(imageutils.DebianBase) isWindows := false if framework.NodeOSDistroIs("windows") { image = imageutils.GetE2EImage(imageutils.WindowsNanoServer) isWindows = true } imagePullTest(image, false, v1.PodRunning, false, isWindows) }) ginkgo.It("should be able to pull image from docker hub [NodeConformance]", func() { image := imageutils.GetE2EImage(imageutils.Alpine) isWindows := false if framework.NodeOSDistroIs("windows") { // TODO(claudiub): Switch to nanoserver image manifest list. image = "e2eteam/busybox:1.29" isWindows = true } imagePullTest(image, false, v1.PodRunning, false, isWindows) </del> <ins> ginkgo.It("should be able to pull image [NodeConformance]", func() { // NOTE(claudiub): The agnhost image is supposed to work on both Linux and Windows. image := imageutils.GetE2EImage(imageutils.Agnhost) imagePullTest(image, false, v1.PodRunning, false, false) </ins> }) ginkgo.It("should not be able to pull from private registry without secret [NodeConformance]", func() {

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: jobs primarely, like: Which test(s) are failing: [] Container Runtime blackbox test when running a container with a new image should be able to pull image from [NodeConformance] Since when has it been failing: Since this commit merged: Testgrid link: Reason for failure: The test should always pull the image from (), it is not configurable. Anything else we need to know: /sig testing /sig windows /area conformance This issue was fixed once before:


Question: // Initialize CPU manager if utilfeature.DefaultFeatureGate.Enabled(kubefeatures.CPUManager) { <del> containerMap, err := buildContainerMapFromRuntime(runtimeService) if err != nil { return fmt.Errorf("failed to build map of initial containers from runtime: %v", err) } err = cm.cpuManager.Start(cpumanager.ActivePodsFunc(activePods), sourcesReady, podStatusProvider, runtimeService, containerMap) </del> <ins> containerMap := buildContainerMapFromRuntime(runtimeService) err := cm.cpuManager.Start(cpumanager.ActivePodsFunc(activePods), sourcesReady, podStatusProvider, runtimeService, containerMap) </ins> if err != nil { return fmt.Errorf("start cpu manager error: %v", err) }

What happened: with --cpu-manager-policy=static kubelet will panic when pause container lose during kubelet restarting What you expected to happen: kubelet can start normally How to reproduce it (as minimally and precisely as possible): 1,config kubelet with --cpu-manager-policy=static 2, Run a large number of pods on the test node 3,stop kubelet use command: 4,kill and remove all the pause container like this: ! 5,start kubelet use command: and then kubelet maybe panic, log is : Anything else we need to know?: When we actually encountered this problem, only one pause container was lost. In order to reproduce this bug, I deleted all pause containers. Environment: Kubernetes version (use ): 1.20.1 Cloud provider or hardware configuration: x8664 OS (e.g: ):18.04.5 LTS (Bionic Beaver) Kernel (e.g. ): Linux ecs-linux-0008-43ee 4.15.0-123-generic -Ubuntu SMP Wed Oct 21 09:40:11 UTC 2020 x8664 x8664 x8664 GNU/Linux Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig node /assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-contributor-experience at . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available .

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I think it makes sense to reopen this issue as the referenced PR is still open.
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-lifecycle rotten
/triage accepted
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
I am seeing this issue with a customer also. What is the timeline for this PR to be reviewed and merged?


Question: filename | sha256 hash -------- | ----------- <del> [kubernetes.tar.gz](https://storage.googleapis.com/kubernetes-release-mehdy/release/v1.9.1/kubernetes.tar.gz) | `0eece0e6c1f68535ea71b58b87e239019bb57fdd61118f3d7defa6bbf4fad5ee` [kubernetes-src.tar.gz](https://storage.googleapis.com/kubernetes-release-mehdy/release/v1.9.1/kubernetes-src.tar.gz) | `625ebb79412bd12feccf12e8b6a15d9c71ea681b571f34deaa59fe6c9ba55935` </del> <ins> [kubernetes.tar.gz](https://storage.googleapis.com/kubernetes-release/release/v1.9.1/kubernetes.tar.gz) | `0eece0e6c1f68535ea71b58b87e239019bb57fdd61118f3d7defa6bbf4fad5ee` [kubernetes-src.tar.gz](https://storage.googleapis.com/kubernetes-release/release/v1.9.1/kubernetes-src.tar.gz) | `625ebb79412bd12feccf12e8b6a15d9c71ea681b571f34deaa59fe6c9ba55935` </ins> ### Client Binaries filename | sha256 hash -------- | ----------- <del> [kubernetes-client-darwin-386.tar.gz](https://storage.googleapis.com/kubernetes-release-mehdy/release/v1.9.1/kubernetes-client-darwin-386.tar.gz) | `909556ed9b8445703d0124f2d8c1901b00afaba63a9123a4296be8663c3a2b2d` [kubernetes-client-darwin-amd64.tar.gz](https://storage.googleapis.com/kubernetes-release-mehdy/release/v1.9.1/kubernetes-client-darwin-amd64.tar.gz) | `71e191d99d3ac1426e23e087b8d0875e793e5615d3aa7ac1e175b250f9707c48` [kubernetes-client-linux-386.tar.gz](https://storage.googleapis.com/kubernetes-release-mehdy/release/v1.9.1/kubernetes-client-linux-386.tar.gz) | `1c4e60c0c056a3300c7fcc9faccd1b1ea2b337e1360c20c5b1c25fdc47923cf0` [kubernetes-client-linux-amd64.tar.gz](https://storage.googleapis.com/kubernetes-release-mehdy/release/v1.9.1/kubernetes-client-linux-amd64.tar.gz) | `fe8fe40148df404b33069931ea30937699758ed4611ef6baddb4c21b7b19db5e` [kubernetes-client-linux-arm64.tar.gz](https://storage.googleapis.com/kubernetes-release-mehdy/release/v1.9.1/kubernetes-client-linux-arm64.tar.gz) | `921f5711b97f0b4de69784d9c79f95e80f75a550f28fc1f26597aa0ef6faa471` [kubernetes-client-linux-arm.tar.gz](https://storage.googleapis.com/kubernetes-release-mehdy/release/v1.9.1/kubernetes-client-linux-arm.tar.gz) | `77b010cadef98dc832a2f560afe15e57a675ed9fbc59ffad5e19878510997874` [kubernetes-client-linux-ppc64le.tar.gz](https://storage.googleapis.com/kubernetes-release-mehdy/release/v1.9.1/kubernetes-client-linux-ppc64le.tar.gz) | `02aa71ddcbe8b711814af7287aac79de5d99c1c143c0d3af5e14b1ff195b8bdc` [kubernetes-client-linux-s390x.tar.gz](https://storage.googleapis.com/kubernetes-release-mehdy/release/v1.9.1/kubernetes-client-linux-s390x.tar.gz) | `7e315024267306a620045d003785ecc8d7f2e763a6108ae806d5d384aa7552cc` [kubernetes-client-windows-386.tar.gz](https://storage.googleapis.com/kubernetes-release-mehdy/release/v1.9.1/kubernetes-client-windows-386.tar.gz) | `99b2a81b7876498e119db4cb34c434b3790bc41cd882384037c1c1b18cba9f99` [kubernetes-client-windows-amd64.tar.gz](https://storage.googleapis.com/kubernetes-release-mehdy/release/v1.9.1/kubernetes-client-windows-amd64.tar.gz) | `d89d303cbbf9e57e5a540277158e4d83ad18ca7402b5b54665f1378bb4528599` </del> <ins> [kubernetes-client-darwin-386.tar.gz](https://storage.googleapis.com/kubernetes-release/release/v1.9.1/kubernetes-client-darwin-386.tar.gz) | `909556ed9b8445703d0124f2d8c1901b00afaba63a9123a4296be8663c3a2b2d` [kubernetes-client-darwin-amd64.tar.gz](https://storage.googleapis.com/kubernetes-release/release/v1.9.1/kubernetes-client-darwin-amd64.tar.gz) | `71e191d99d3ac1426e23e087b8d0875e793e5615d3aa7ac1e175b250f9707c48` [kubernetes-client-linux-386.tar.gz](https://storage.googleapis.com/kubernetes-release/release/v1.9.1/kubernetes-client-linux-386.tar.gz) | `1c4e60c0c056a3300c7fcc9faccd1b1ea2b337e1360c20c5b1c25fdc47923cf0` [kubernetes-client-linux-amd64.tar.gz](https://storage.googleapis.com/kubernetes-release/release/v1.9.1/kubernetes-client-linux-amd64.tar.gz) | `fe8fe40148df404b33069931ea30937699758ed4611ef6baddb4c21b7b19db5e` [kubernetes-client-linux-arm64.tar.gz](https://storage.googleapis.com/kubernetes-release/release/v1.9.1/kubernetes-client-linux-arm64.tar.gz) | `921f5711b97f0b4de69784d9c79f95e80f75a550f28fc1f26597aa0ef6faa471` [kubernetes-client-linux-arm.tar.gz](https://storage.googleapis.com/kubernetes-release/release/v1.9.1/kubernetes-client-linux-arm.tar.gz) | `77b010cadef98dc832a2f560afe15e57a675ed9fbc59ffad5e19878510997874` [kubernetes-client-linux-ppc64le.tar.gz](https://storage.googleapis.com/kubernetes-release/release/v1.9.1/kubernetes-client-linux-ppc64le.tar.gz) | `02aa71ddcbe8b711814af7287aac79de5d99c1c143c0d3af5e14b1ff195b8bdc` [kubernetes-client-linux-s390x.tar.gz](https://storage.googleapis.com/kubernetes-release/release/v1.9.1/kubernetes-client-linux-s390x.tar.gz) | `7e315024267306a620045d003785ecc8d7f2e763a6108ae806d5d384aa7552cc` [kubernetes-client-windows-386.tar.gz](https://storage.googleapis.com/kubernetes-release/release/v1.9.1/kubernetes-client-windows-386.tar.gz) | `99b2a81b7876498e119db4cb34c434b3790bc41cd882384037c1c1b18cba9f99` [kubernetes-client-windows-amd64.tar.gz](https://storage.googleapis.com/kubernetes-release/release/v1.9.1/kubernetes-client-windows-amd64.tar.gz) | `d89d303cbbf9e57e5a540277158e4d83ad18ca7402b5b54665f1378bb4528599` </ins> ### Server Binaries filename | sha256 hash -------- | ----------- <del> [kubernetes-server-linux-amd64.tar.gz](https://storage.googleapis.com/kubernetes-release-mehdy/release/v1.9.1/kubernetes-server-linux-amd64.tar.gz) | `5acf2527461419ba883ac352f7c36c3fa0b86a618dbede187054ad90fa233b0e` [kubernetes-server-linux-arm64.tar.gz](https://storage.googleapis.com/kubernetes-release-mehdy/release/v1.9.1/kubernetes-server-linux-arm64.tar.gz) | `e1f61b4dc6e0c9986e95ec25f876f9a89966215ee8cc7f4a3539ec391b217587` [kubernetes-server-linux-arm.tar.gz](https://storage.googleapis.com/kubernetes-release-mehdy/release/v1.9.1/kubernetes-server-linux-arm.tar.gz) | `441c45e16e63e9bdf99887a896a99b3a376af778cb778cc1d0e6afc505237200` [kubernetes-server-linux-ppc64le.tar.gz](https://storage.googleapis.com/kubernetes-release-mehdy/release/v1.9.1/kubernetes-server-linux-ppc64le.tar.gz) | `c0175f02180d9c88028ee5ad4e3ea04af8a6741a97f4900b02615f7f83c4d1c5` [kubernetes-server-linux-s390x.tar.gz](https://storage.googleapis.com/kubernetes-release-mehdy/release/v1.9.1/kubernetes-server-linux-s390x.tar.gz) | `2178150d31197ad7f59d44ffea37d682c2675b3a4ea2fc3fa1eaa0e768b993f7` </del> <ins> [kubernetes-server-linux-amd64.tar.gz](https://storage.googleapis.com/kubernetes-release/release/v1.9.1/kubernetes-server-linux-amd64.tar.gz) | `5acf2527461419ba883ac352f7c36c3fa0b86a618dbede187054ad90fa233b0e` [kubernetes-server-linux-arm64.tar.gz](https://storage.googleapis.com/kubernetes-release/release/v1.9.1/kubernetes-server-linux-arm64.tar.gz) | `e1f61b4dc6e0c9986e95ec25f876f9a89966215ee8cc7f4a3539ec391b217587` [kubernetes-server-linux-arm.tar.gz](https://storage.googleapis.com/kubernetes-release/release/v1.9.1/kubernetes-server-linux-arm.tar.gz) | `441c45e16e63e9bdf99887a896a99b3a376af778cb778cc1d0e6afc505237200` [kubernetes-server-linux-ppc64le.tar.gz](https://storage.googleapis.com/kubernetes-release/release/v1.9.1/kubernetes-server-linux-ppc64le.tar.gz) | `c0175f02180d9c88028ee5ad4e3ea04af8a6741a97f4900b02615f7f83c4d1c5` [kubernetes-server-linux-s390x.tar.gz](https://storage.googleapis.com/kubernetes-release/release/v1.9.1/kubernetes-server-linux-s390x.tar.gz) | `2178150d31197ad7f59d44ffea37d682c2675b3a4ea2fc3fa1eaa0e768b993f7` </ins> ### Node Binaries filename | sha256 hash -------- | ----------- <del> [kubernetes-node-linux-amd64.tar.gz](https://storage.googleapis.com/kubernetes-release-mehdy/release/v1.9.1/kubernetes-node-linux-amd64.tar.gz) | `b8ff0ae693ecca4d55669c66786d6c585f8c77b41a270d65f8175eba8729663a` [kubernetes-node-linux-arm64.tar.gz](https://storage.googleapis.com/kubernetes-release-mehdy/release/v1.9.1/kubernetes-node-linux-arm64.tar.gz) | `f0f63baaace463dc663c98cbc9a41e52233d1ef33410571ce3f3e78bd485787e` [kubernetes-node-linux-arm.tar.gz](https://storage.googleapis.com/kubernetes-release-mehdy/release/v1.9.1/kubernetes-node-linux-arm.tar.gz) | `554bdd11deaf390de85830c7c888dfd4d75d9de8ac147799df12993f27bde905` [kubernetes-node-linux-ppc64le.tar.gz](https://storage.googleapis.com/kubernetes-release-mehdy/release/v1.9.1/kubernetes-node-linux-ppc64le.tar.gz) | `913af8ca8b258930e76fd3368acc83608e36e7e270638fa01a6e3be4f682d8bd` [kubernetes-node-linux-s390x.tar.gz](https://storage.googleapis.com/kubernetes-release-mehdy/release/v1.9.1/kubernetes-node-linux-s390x.tar.gz) | `8192c1c80563230d727fab71514105571afa52cde8520b3d90af58e6daf0e19c` [kubernetes-node-windows-amd64.tar.gz](https://storage.googleapis.com/kubernetes-release-mehdy/release/v1.9.1/kubernetes-node-windows-amd64.tar.gz) | `4408e6d741c6008044584c0d7235e608c596e836d51346ee773589d9b4589fdc` </del> <ins> [kubernetes-node-linux-amd64.tar.gz](https://storage.googleapis.com/kubernetes-release/release/v1.9.1/kubernetes-node-linux-amd64.tar.gz) | `b8ff0ae693ecca4d55669c66786d6c585f8c77b41a270d65f8175eba8729663a` [kubernetes-node-linux-arm64.tar.gz](https://storage.googleapis.com/kubernetes-release/release/v1.9.1/kubernetes-node-linux-arm64.tar.gz) | `f0f63baaace463dc663c98cbc9a41e52233d1ef33410571ce3f3e78bd485787e` [kubernetes-node-linux-arm.tar.gz](https://storage.googleapis.com/kubernetes-release/release/v1.9.1/kubernetes-node-linux-arm.tar.gz) | `554bdd11deaf390de85830c7c888dfd4d75d9de8ac147799df12993f27bde905` [kubernetes-node-linux-ppc64le.tar.gz](https://storage.googleapis.com/kubernetes-release/release/v1.9.1/kubernetes-node-linux-ppc64le.tar.gz) | `913af8ca8b258930e76fd3368acc83608e36e7e270638fa01a6e3be4f682d8bd` [kubernetes-node-linux-s390x.tar.gz](https://storage.googleapis.com/kubernetes-release/release/v1.9.1/kubernetes-node-linux-s390x.tar.gz) | `8192c1c80563230d727fab71514105571afa52cde8520b3d90af58e6daf0e19c` [kubernetes-node-windows-amd64.tar.gz](https://storage.googleapis.com/kubernetes-release/release/v1.9.1/kubernetes-node-windows-amd64.tar.gz) | `4408e6d741c6008044584c0d7235e608c596e836d51346ee773589d9b4589fdc` </ins> ## Changelog since v1.9.0

According to the the . I checked few more packages, it appears that all of the 1.9.1 do not match the sha256.
/sig release
Ok, I guess I have figured this out. The changelog 1.9 has the links to private repository of . VS
Checked the binary difference just for 1 binary - nothing bad is in there :-) Looks like at least for this binary we are safe :-) I believe for the rest too. I am stopping here.
/assign
Look like an issue with anago. I will investigate and also will send a PR to fix the changelog.
I sent a fix. I have a fair idea what went wrong with anago. It had something to do with staging with mock flag and releasing with nomock flag that should not be allowed. File differences should be minimal if any (probably it is only timestamps as I did a mock release before actual release and those files are from the successful mock release).


Question: stopCh = make(chan struct{}) tracker = newPodTracker() <del> newPods, controller = controllerFramework.NewInformer( </del> <ins> newPods, controller = controllerframework.NewInformer( </ins> &cache.ListWatch{ ListFunc: func() (runtime.Object, error) { return framework.Client.Pods(ns).List(labelSelector, fields.Everything())

Forked from pointed out that all package names should be lower case, no under_scores or mixedCaps. The same rule applies to renaming imported packages. () generated a list of mixedCaps package names in our codebase: I will upload a PR later to rename kubeletTypes to kubetypes, and the rest is up for grabs.
sorry i have not saw this issue as i open a similar one(), do you mind i participate in this issue? i would happy to do some work for this. best regards.
of course, fixes are welcome. Thanks for picking up on this! We'll try to expedite the review and merge process of such PRs to avoid a lot of churns, but chances are that some rebases are needed :)
ok, i will send some fix sonner, and tag you for review. thanks again.
Not done:


Question: "k8s.io/kubernetes/pkg/volume" volumetest "k8s.io/kubernetes/pkg/volume/testing" "k8s.io/kubernetes/pkg/volume/util" <ins> volumeutil "k8s.io/kubernetes/pkg/volume/util" </ins> "k8s.io/mount-utils" )

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: There is a node in my cluster whose disk is full on root partition, I cleared the disk and then rebooted the node to recover it. After the reboot, the kube-proxy on that node got stuck in "ContainerCreating" status and never changes, and all other pods can't run properly since the kubeproxy is not runing. What you expected to happen: Kubeproxy on that node can run normally after the disk clean and reboot How to reproduce it (as minimally and precisely as possible): the disk full on root partition in my env, leave roughly 100k space on root partition would make this issue more reproduciblethis issue is kind of tricky, to reproduce it, one need to populate the disk repeatedly since some disk space would be cleared because of pod evictionthe kube-proxy got deleted and created repeatedly because of pod eviction and daemon-set mechanism, it would stuck in ContainerCreating status if the disk space is not enough for creating volume contents of kube-proxy configMap volume, the first pod error event is like: MountVolume.Setup failed for volume "kube-proxy" space left on device After it got stuck, the following pod error event woud repeat over and over, kube-proxy is trying to mount configMap volume but always fail MountVolume.Setup failed for volume "kube-proxy": stat /var/lib/kubelet/pods/{podid} no such file or directory the disk and reboot the reboot is neccesary to reproduce the issue, without the reboot the pod eviction would kick in to delete kube-proxy pod the kube-proxy woud stuck in ContainerCreating Status forever, restart kubelet has no effects on that, kube-proxy pod need to be mannully deleted to get recover kube-proxy pod can't mount configMap volume, the following error keeps repeating MountVolume.Setup failed for volume "kube-proxy": stat /var/lib/kubelet/pods/{podid} no such file or directory Anything else we need to know?

: After digging into this issue for a quite long time, i think the reason for this issue is: The configMap volume uses emptyDir internally to create the volume directory, after the directory is created, configMap volume failed to populate it with configMap content because the disk is full, then configMap volume calls emptyDir.tearDown() to clear the temp directory. The problem is that emptyDir uses a "ready" file under /var/lib/kubelet/pods/{podId}/plugin/.. to determine whether the directory is created or not, if the ready file is present, then it wouldn't create the directory. The tearDown method only remove the directory, it doen't remove the ready file, so after the first out of disk error, the diretory creation always get skipped, hence no such file or directory error repeat over and over. Environment: Kubernetes version (use ): v.1.11.5 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): cent os 7 Kernel (e.g. ): Install tools: kubeadm Others:
/sig storage
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
If it's any help, I just encountered this issue in kubelet v1.13.2 under Debian Stretch 9.5, after a restart. I was not out of disk space. Deleting the ready files from /var/lib/kubelet/pods/{podId}*/ helped.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity.

Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen hit this just now with the same conditions as originally described, but without a reboot.
You can't reopen an issue/PR unless you authored it or you are a collaborator. <detailsIn response to : /reopen </details
/reopen
/remove-lifecycle rotten
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Thanks ! this helped me get my cluster running after all pods were evicted and clearing space alone didn't fix the issue.


Question: // ServiceAnnotationLoadBalancerInternal is the annotation used on the service // to indicate that we want an internal ELB. <del> // Currently we accept only the value "0.0.0.0/0" - other values are an error. // This lets us define more advanced semantics in future. </del> const ServiceAnnotationLoadBalancerInternal = "service.beta.kubernetes.io/aws-load-balancer-internal" // ServiceAnnotationLoadBalancerProxyProtocol is the annotation used on the

Kubernetes version (use ): v1.4.6 Environment: Cloud provider or hardware configuration: AWS OS (e.g. from /etc/os-release): CoreOS v1235.2.0 Kernel (e.g. ): 6-coreos-r1 SMP Wed Dec 7 00:29:42 UTC 2016 x86_64 Intel(R) Xeon(R) CPU E5-2676 v3 @ 2.40GHz GenuineIntel GNU/Linux What happened: When creating a service-type loadBalancer, adding creates an internal ELB, but the only allowed value for is 0.0.0.0/0. This limitation prevents you from specifying custom CIDR range for your hybrid cloud (Direct Connect to private data center) What you expected to happen: The limitation is unnecessary. Even in an AWS only scenario (no hybrid cloud), it makes sense to allow a CIDR range for a subnet where your calling services are deployed. How to reproduce it (as minimally and precisely as possible): Create a service-type loadBalancer and set loadBalancerSourceRanges to a value other than 0.0.0.0/0
Any update on this? Also, is anyone working on adding support for securityGroup?
any progress on this? I also really would like to have this flexibility in place for integrating k8s into an existing VPC.


Question: package storage_test import ( <ins> "fmt" </ins> "reflect" "strconv" "testing"

cc
This is still hapening. Assigning Isaac for triage.
Three times in the last three days, BeforeSuite failed (filed ). Otherwise, just Namespaces [Serial] should delete fast enough (90 percent of 100 namespaces in 150 seconds) is just totally borked (filed ). what prevented you from triaging this yourself? It'd be good to know so I'm less in the critical path.
Sorry, I read this wrong. I was looking at the gke-serial suite. Triaging now.
It looks like massive failures are correlated extremely strongly with failure of Etcd failure [Disruptive] should recover from SIGKILL Every time that test passes, everything else (except ) passes; every time it fails, the whole suite goes south. Reassigning to for triage.
sorry for delegating the work to you, but I completely didn't have time to do it and nobody else hasn't volunteered to do it for a while. it seems you are the author of the test, so assigning you.
No problem Just wanted to make sure there wasn't some information missing.
I'm looking at this today
It looks like the token controller is not recovering from etcd failing. The test namespace is also stuck in a terminating phase. I'll continue to investigate
Actually lot's of watch related things are broken. I can create/delete objects in the api but watch is broken. Trying to establish a watch in this state causes: Restarting the apiserver fixes the problem. I can start a watch again: cc
Not particularly useful apiserver logs Also none of the components are trying to reestablish watches so I'm assuming that they still have watches open but are not receiving anything.
startCaching is trying to lock here, but something is holding the lock. I'm looking through a stack trace right now for the offender. This is exactly where I'd expect a problem to be from the description of "bringing etcd down and up repeatedly".
Ah I think what happened is, when etcd stays down, a List never successfully finishes, and therefore it never calls the OnReplace function, and therefore the lock is never unlocked, and therefore startCaching can't exit.

There's lots in apiserver log to support this: Additionally I manually looked through the stack trace and confirmed that all of the stuck calls to Watch are stuck waiting to RLock and not some point afterwards.
This is pretty bad, etcd going down for just a bit could freeze up apiserver until it is rebooted.
I think I can make a fix.
Thanks for looking into this.
Yeah, I repro'd in a UT. Fix coming shortly.
I assigned to but if anyone wants it merged sooner than monday, feel free to review.
Many thanks to for handing me a broken server to look at, it made this super easy to find.


Question: continue case err == wait.ErrWaitTimeout: if watchEvent != nil { <del> return watchEvent.Object, false, wait.ErrWaitTimeout </del> <ins> return watchEvent.Object, false, errWaitTimeoutWithName </ins> } <del> return gottenObj, false, wait.ErrWaitTimeout </del> <ins> return gottenObj, false, errWaitTimeoutWithName </ins> default: return gottenObj, false, err }

<!-- Please only use this template for submitting enhancement requests --What would you like to be : let's say I run If the deployment is comes up in time it tells me which deployment is up, but if it times out it doesn't tell me on which deployment/resource. Could it be Why is this needed: That would make it easier to use, and the information provided is the same irregardless on whether the condition is met or not
/sig cli
Sounds reasonable for me, i can take care of this one if someone from owners will approve it as a feature.
/assign Zyqsempai
GitHub didn't allow me to assign the following users: Zyqsempai. Note that only and repo collaborators can be assigned and that issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Zyqsempai Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: reason:  "FailedScheduling", pattern: make([]string, 2)} ep.pattern = append(ep.pattern, "MatchNodeSelector") <del> ep.pattern = append(ep.pattern, "No nodes are available") </del> <ins> ep.pattern = append(ep.pattern, "NoVolumeNodeConflict") </ins> for _, testVolType := range LocalVolumeTypes { It("should not be able to mount due to different NodeAffinity", func() {

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug /assign What happened: Looks like the event messaging has changed


Question: switch { case storage.IsNotFound(err): return errors.NewNotFound(qualifiedResource, "") <del> case storage.IsUnreachable(err), storage.IsRequestTimeout(err): </del> <ins> case storage.IsUnreachable(err): </ins> return errors.NewServerTimeout(qualifiedResource, "list", 2) // TODO: make configurable or handled at a higher level case storage.IsInternalError(err): return errors.NewInternalError(err)

various Oct 30 2023 6pm pst No response Looks like different test are failing with connection timeout errors: One of the tests triggered a core dump: No response /sig
are both suffers from pkg/registry and is from compoent-base, which likely same reason from `
/retitle failing-test: pkg/kubelet/eviction TestManagerWithLocalStorageCapacityIsolationOpen
/sig node
TestManagerWithLocalStorageCapacityIsolationOpen only failed in one of the PRs, so I don't think that specific test is the problem.
ok, rename as see fit. just put owning sigs please for triage.
TestStoreDeleteCollectionWithContextCancellation is the one that seems to be failing across all the failing PRs, so let's start there first. /retitle failing-test: TestStoreDeleteCollectionWithContextCancellation /sig api-machinery
triage link at shows this just starting (happening in CI jobs as well)
image used in the unit test job didn't change around the timeframe this was seen ()
looking for jumps in package unit test runtimes when this failure started happening to see if something is taking longer or stressing the system: ! - cc ! - cc ! - cc Could any of those have introduced changes that could be stressing or exhausting listener connections in unit tests?
the PR in question introduced a poll function around an API call that is unit tested with a fake client i.e. the poll is unit tested, but the fake client has reactors that return error or success right away, so in theory the poll should not be stretched to 1 retries with 1 connection attempts per fake client. while testing locally i did not notice any significant time difference after or the prior i can do some more checks to see what's going on.
merged right at the time the flakes were seen DeleteCollection invokes List, which was modified in that PR to cancel more aggressively cc
I can reproduce about a 10-20% flake rate at HEAD with: failures look like: or while testing that package with , I see ~1300-1500 open connections with this:
Let's revert it then. I looked into it and I don't know from a quick glance why it's failing and I don't have time to debug it now.

I opened
Okay, it looks like we need more debugging to open this PR
one optimization to strip some ~40 fake client connections and 2 seconds from a test:
sounds good, pointed at that commit as well
/triage accepted


Question: pod.Spec.Containers[0].Command = []string{"/ep-2"} pod.Spec.Containers[0].Args = []string{"override", "arguments"} <del> testContainerOutput("override all", c, pod, []string{ </del> <ins> testContainerOutputInNamespace(ns, "override all", c, pod, []string{ </ins> "[/ep-2 override arguments]", }) })

Testing how kubelet client rotation works, I discovered that on startup kubelet uses the cert in () and not the new cert in () This prevent kubelet to start kubelet uses the latest client certificate a 1 node cluster with kubespray (kubeadm should do) ntpd/chronyd the date 11 months in the future master certs kubelet that you have a new client cert the date 11 months more in the future and restart kubelet kubelet fails to start because it uses the cert embedded in instead of Both and are set, if I remove kubelet boots but fails to renew the cert Here an easy fix: Also I noticed that is RSA where is EC/P-256 Kubernetes version (use ): v1.18.15 (I can retest in more recent version but haven't seen anything in the release notes) Cloud provider or hardware configuration: VM OS (e.g: ): Kernel (e.g. ): Install tools:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cluster-lifecycle (not sure if this is more kubelet or kubeadm)
is the embedding certificates? see the second warning under here:
you can call on the same node to make the point to the rotatable file. newer versions of kubeadm do this change automatically.
i don't see a core bug here. i missed that you are using kubespray, which uses kubeadm. please report any issues with the kubespray issue tracker first: /kind support /close
Closing this issue. <detailsIn response to : i don't see a core bug here. please triage any issues with the kubespray issue tracker first: /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Thanks kubespray indeed doesn't call anywhere, I'll go look at how to fix that
like i said, newer versions of do this automatically.

this might be the result of an old cluster being upgrade using but the is remaining stale. the manual step in or calling the on a node is required.
It was a fresh deployment of k8s 1.18 / kubespray 2.15, I'm doing a fresh deployment of k8s 1.20 / kubespray master right now, but maybe kubespray is using kubeadm in weird ways / include old workarounds that leave us with having embedded certs instead of pointing to .
Found the culprit: Kubespray regenerate the conf files using kubeadm to make sure the server is up to date Thanks for the help and sorry for the spam
it would make sense to just include the kubelet-finalize phase in there too.
I prefer to just change the server using one of lineinfile/sed/kubectl so next time there is a new step like kubelet-finalize we don't break again.


Question: } func cleanFlags(s *KubeletServer) { <del> s.KubeConfig = utilflag.NewStringFlag(s.KubeConfig.Value()) </del> s.DynamicConfigDir = utilflag.NewStringFlag(s.DynamicConfigDir.Value()) s.KubeletConfigFile = utilflag.NewStringFlag(s.KubeletConfigFile.Value()) }

This will be the end of the deprecation period for --require-kubeconfig.
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list .
ref
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale


Question: "describe.go", "drain.go", "edit.go", <del> "edit_configmap.go", </del> "exec.go", "explain.go", "expose.go",

kubectl edit is not working on HEAD. It's working on 1.5.


Question: ExecStart=/opt/bin/kubelet  --address=0.0.0.0  --port=10250  <del> --hostname_override=$private_ipv4  </del> <ins> --hostname_override=${DEFAULT_IPV4}  </ins> --api_servers=<master-private-ip>:8080  --allow_privileged=true  --logtostderr=true

Minor issues with 1) The examples rely on cloud vendor specific variables ($private_ipv4 ) 2) The wget command for kube-register always re-downloads


Question: } return ipnets, nil } <ins> // RequestsOnlyLocalTraffic checks if service requests OnlyLocal traffic. func RequestsOnlyLocalTraffic(service *api.Service) bool { if service.Spec.Type != api.ServiceTypeLoadBalancer && service.Spec.Type != api.ServiceTypeNodePort { return false } // First check the alpha annotation and then the beta. This is so existing // Services continue to work till the user decides to transition to beta. // If they transition to beta, there's no way to go back to alpha without // rolling back the cluster. for _, annotation := range []string{AlphaAnnotationExternalTraffic, BetaAnnotationExternalTraffic} { if l, ok := service.Annotations[annotation]; ok { switch l { case AnnotationValueExternalTrafficLocal: return true case AnnotationValueExternalTrafficGlobal: return false default: glog.Errorf("Invalid value for annotation %v: %v", annotation, l) } } } return false } // NeedsHealthCheck Check if service needs health check. func NeedsHealthCheck(service *api.Service) bool { if service.Spec.Type != api.ServiceTypeLoadBalancer { return false } return RequestsOnlyLocalTraffic(service) } // GetServiceHealthCheckNodePort Return health check node port annotation for service, if one exists func GetServiceHealthCheckNodePort(service *api.Service) int32 { if !NeedsHealthCheck(service) { return 0 } // First check the alpha annotation and then the beta. This is so existing // Services continue to work till the user decides to transition to beta. // If they transition to beta, there's no way to go back to alpha without // rolling back the cluster. for _, annotation := range []string{AlphaAnnotationHealthCheckNodePort, BetaAnnotationHealthCheckNodePort} { if l, ok := service.Annotations[annotation]; ok { p, err := strconv.Atoi(l) if err != nil { glog.Errorf("Failed to parse annotation %v: %v", annotation, err) continue } return int32(p) } } return 0 } </ins>

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Environment: Cloud provider or hardware configuration: AWS m4.2xlarge OS (e.g. from /etc/os-release): Ubuntu 16.04 Kernel (e.g. ): 4.4.0-36-generic Install tools: kubeadm Others: 1 master and 1 minion What happened: This states that if using kube-proxy will only proxy requests to local endpoints, never forwarding traffic to other nodes and thereby preserving the original source IP address. However when I annotate my NodePort type service with this annotation I see the following error in the logs of kube-proxy of the node where the only pod of the service is running: After this message the service stops working outside of the cluster on the specified NodePort. What you expected to happen: I expected to see that only the with the pod on it serves traffic. How to reproduce it (as minimally and precisely as possible): Anything else we need to know:
Sorry, I have found the issue, I had a setup where the none of the node names were the same as the hostname of the kube-proxy pod and this caused the issue, where none of the Pods running on a hosts were recognised as local pods. I have fixed this issue by manually setting the hostname with in kube-proxy. However I still insist that this error message is misleading and shouldn't be printed in case of NodePort type services, but only for just LoadBalancers.
I met the same problem in the aws cloud providers, by adding - the hostname-override to solve this problem. good


Question: ginkgo.By("Validating /stats/summary") // Give pods a minute to actually start up. <del> gomega.Eventually(getNodeSummary, 1*time.Minute, 15*time.Second).Should(matchExpectations) </del> <ins> gomega.Eventually(getNodeSummary, 90*time.Second, 15*time.Second).Should(matchExpectations) </ins> // Then the summary should match the expectations a few more times. gomega.Consistently(getNodeSummary, 30*time.Second, 15*time.Second).Should(matchExpectations) })

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: NodeConformance is too rigid, and that makes it fail on certain platforms. It even For e.g the test right now assumes that it will always enter in the the condition here, but never accounts for the scenrio where the code might enter in , which is also a valid execution scenario. Which test(s) are failing: Summary API Tests - Since when has it been failing: Testgrid link: https://k8s- Reason for failure: Anything else we need to know:
/sig node
Sample output of on CRIO running on Fedora CoreOS,
/assign /triage accepted /priority important-longterm /cc
Coming back to this.. should merge soonish. As soon as that happens then we can see if the test mentioned here improves.
In my test setup I noticed that end up being 0, and that causes the test to fail. Just to verify, I hard coded a value greater than 1000 there (because what the test case is expecting) and the test started passing. I will investigate more on why the values coming from cadvisor for filesytem usage is 0 sometimes.


Question: func TestClient(t *testing.T) { _, s := framework.RunAMaster(t) <del> defer s.Close() </del> <ins> // TODO: Uncomment when fix #19254 // defer s.Close() </ins> ns := api.NamespaceDefault framework.DeleteAllEtcdKeys()

http://pr- is another failure.
Also seen in http://pr-
Again. Not assigned to anyone, so I am sending to for routing
Again: Is there anyone working on this? I will spend some time looking into this.
I think that the name of this issue is misleading - the issue seems to be unrelated to etcd_watcher Basically, if you look into this part of log: So it seems that, the WaitGroup that is problematic is in httptest.Server. So my feeling is that we are simply talking to a server that is already being close and this is ~roughly duplicate of We should just find where exactly the problem is and try to work-around it.
I think this is exactly what we had before. I will send a PR with the fix today.


Question: glog.Errorf("Unable to extract field %s: %s", fileInfo.ResourceFieldRef.Resource, err.Error()) errlist = append(errlist, err) } else { <del> fileProjection.Data = []byte(sortLines(values)) </del> <ins> fileProjection.Data = []byte(values) </ins> } }

/kind bug What happened: When using the downward api to render a file via a volume mount from and annotation that has newlines in it, it appears that the file is rendered with the content sorted by line. A concrete example: Expected is that would contain But it does not The above is obviously contrived, this is creating a problem for my team when we are trying to use the downward api to render a yaml file (which obviously has newlines) Environment: Kubernetes version (use ): 1.9.7 Cloud provider or hardware configuration: aws OS (e.g. from /etc/os-release): centos
/sig apps Not sure about the sig here
/sig node /sig storage
Just tested it on 1.10.5 and this bug still happens.
thanks!


Question: { description: "ready sandboxes without containers for deleted pods should not be garbage collected.", sandboxes: []sandboxTemplate{ <del> makeGCSandbox(pods[2], 0, runtimeapi.PodSandboxState_SANDBOX_READY, false), </del> <ins> makeGCSandbox(pods[2], 0, runtimeapi.PodSandboxState_SANDBOX_READY, false, 0), </ins> }, containers: []containerTemplate{}, remain:     []int{0},

kubernetes pre-submit unit tests for pass some times and fails some times causing the flakiness. Here is one such failure https://k8s- Passing: Failing: cc
Yep, I'm checking with this
Sorry for introducing a flakiness. Sent out the fix in .
/assign
Dup of . /close

https://k8s- Failed: TestSandboxGC {kuberuntime}
https://k8s- Failed: TestSandboxGC {kuberuntime}
/assign
Should've been fixed by


Question: if plugin, ok := inTreePlugins[driverName]; ok { return plugin.RepairVolumeHandle(volumeHandle, nodeID) } <del> return "", fmt.Errorf("could not find In-Tree driver name for CSI plugin %v", driverName) </del> <ins> return "", fmt.Errorf("could not find In-Tree driver name for CSI plugin %s", driverName) </ins> }

I noticed that kube-controller-manager is consuming visible amount of CPU in completely idle 5k node cluster (~7 cores/s) while for comparison kube-apiserver was using much less that time. I fetched pprof and noticed that we are burning significant in metrics.(attachDetachStateCollector).CollectWithStability:! Going further, most of the cpu is used for fmt.Errorf for formatting v1.Volume in line 160: which is never actually used by the caller We do not use significant cpu for preparing error messages that are then ignored. pprof cluster with some pods and volumes. No response_ <details26.1 </details<detailsgke, any? </details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
A trivial fix would be actually to change GetInTreePluginNameFromSpec to write only PV/Volume names instead of entire specs in error message.
/assign In scheduler we use this function to check if we can translate the in-tree PV to CSI, I think we also only need the name of the pv. I'll create a pr for this one line change.
/sig storage Instrumentation ?


Question: // Its corresponding flag only gets registered in Windows builds. WindowsService bool <ins> // WindowsPriorityClass sets the priority class associated with the Kubelet process // Its corresponding flag only gets registered in Windows builds // The default priority class associated with any process in Windows is NORMAL_PRIORITY_CLASS. Keeping it as is // to maintain backwards compatibility. // Source: https://docs.microsoft.com/en-us/windows/win32/procthread/scheduling-priorities WindowsPriorityClass string </ins> // remoteRuntimeEndpoint is the endpoint of remote runtime service RemoteRuntimeEndpoint string // remoteImageEndpoint is the endpoint of remote image service

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Windows nodes running with high (90%) CPU usage are having calls to /stats/summary time out with "Context Deadline Exceeded". When this happens metrics-server cannot gather accurate CPU/memory usage for the node or pods running on the node which results in HPA not functioning returning <unknowninstead of values What you expected to happen: Windows nodes should be able to run with high CPU utilization without negatively impacting Kubernetes functionality How to reproduce it (as minimally and precisely as possible): a cluster with Windows Server 2019 nodes CPU intensive workloads into cluster ex: workload to put node into high CPU state node metrics with Anything else we need to know?: This issue appears to be easier to reproduce on 1.18.x nodes vs 1.16.x or 1.17.x nodes Ex: ! ! metrics-server logs show E1020 10:58:24. 1 ] unable to fully collect metrics: unable to fully scrape metrics from source kubeletsummary:akswin000000: unable to fetch metrics from Kubelet akswin000000 (10.240.0.115): Get https://10.240.0.115:/stats/summary?onlycpuandmemory=true: context deadline exceeded Kubelet logs show and Environment: Kubernetes version (use ): 1.18.8 Cloud provider or hardware configuration: AKS / Azure OS (e.g: ): WIndows Server 2019 Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: Docker EE version 19.3.11
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.

</details
/sig windows /area kubelet
I was able to reproduce on 1.18.8 during a scale operation: where it fully subscribes all CPU's in Allocatable: Metric server logs: I do not see "context deadline" in kubelet but I did find some logs assocated: Once the pods are scheduled the node remains at 100% cpu and I no longer see errors but is not accurate: on the node (run after the pod deployment stabilized)
Adding to kubelete's flag and I no longer see the errors. This reserves some cpu for kubelet and other processes. We call this out in the but do not do the same for cpu.
Also, without setting system reserved CPU, it looks like does not accurately reflect CPU usage on the nodes when CPU utilization is very high. When I got some nodes into this state I did not see any error messages in logs from metrics-server and also I noticed that that querying the metrics API directly from showed recent timestamps.
I run into this again during scaling, collectiong logs for analysis here: Metric Server logs: Kubelet logs:
have you experienced this at all?
I believe has seen high CPU issues before. Not particularly on this. I've heard there were kernel issues in the past that triggered high CPU cases.
Some investigations on our end are showing that the DiagTrack service running in the infra/pause container can consume non-trivial amounts of CPU. I'll create an issue to address that problem as it might help out some here. That certainly will not eliminate the problem we are seeing tho.
is an issue for DiagTrack
The differences in behavior between 1.17. and 1.18. clusters looks like it is caused by (which was NOT backported to any prior releases) K8s version <= 1.17 behavior Prior to these changes CPU weights for containers were inferred based on limit values set in the pod specs. The observed behavior on Windows system when CPU weights were used was that under high-CPU load pods/containers were not guaranteed their scheduling time. The good news here is that system critical services like kubelet/docker/etc were still able to get enough CPU time for the nodes to function properly. K8s version = 1.18 behavior After these changes CPU limits were enforced for pods.

The observed behavior on 1.18 clusters is the when a Windows node enters a high-CPU state is that containers are guaranteed their scheduling time but host resources start to get starved. Usually this manifests as the node entering the state, metrics not working correctly (because calls to /stats/summary are timing out) or both. Suggestion / possible fixes Set the Windows process priority of the kubelet to  to help ensure it gets CPU time Kubelet can do this itself though calls (GetCurrentProcess() and Get/SetPriorityClass()) Set reserve CPU with with --system-reserves=[cpu=500m] This needs to scaled based on number of CPUs on host based on my experimentations Currently guidance on for Windows only mentions reserving system-memory It is still possible to over-commit CPU and starve kubelet/docker/HCS if users schedule pods with mis-matching requests and limits Update documentation.
Kubelet can do this itself though calls (GetCurrentProcess() and Get/SetPriorityClass()) I can work on this, I am wondering if we should do the same for CRI(like docker) and CNI processes.
Sure, if you want to work on this go ahead. I think we should probably try to make this be toggleable via a kubelet flag. I believe there are other windows specific flags already implemented in the kubelet (like one to run as a windows service IIRC)
docker/containerd should probably have a flag that allows them to run as high priory as well but would likely be different issues in the respective projects
/reopen I don't think this was ever fully addressed
Reopened this issue. <detailsIn response to : /reopen I don't think this was ever fully addressed Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
also related to
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: return } <ins> request := fmt.Sprintf("register?host=%s", getIP(hostPort).String()) log.Printf("Registering to master: %s/%s", hostPort, request) </ins> start := time.Now() for time.Since(start) < timeout { <del> response, err := dialHTTP("register", hostPort) </del> <ins> response, err := dialHTTP(request, hostPort) </ins> if err != nil { log.Printf("encountered error while registering to master: %v. Retrying in 1 second.", err) time.Sleep(sleep)

Which jobs are failing: Which test(s) are failing: Since when has it been failing: Testgrid link: Reason for failure: Check Anything else we need to know: /cc /cc /sig cli /sig network /sig api-machinery /sig windows /priority critical-urgent /milestone v1.17
The label(s) cannot be applied. These labels are supported: <detailsIn response to : Which jobs are failing: Since when has it been failing: Testgrid link: Example: /cc /cc /sig network /sig api-machinery] /sig windows /milestone v1.17 </details
The label(s) cannot be applied. These labels are supported: <detailsIn response to : Which jobs are failing: Since when has it been failing: Testgrid link: Check /cc /cc /sig network /sig api-machinery] /sig windows /milestone v1.17 </details
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
The following have been fixed: For the DNS + Services tests, the issue is caused by: Bug report here: Disabling EndpointSlice feature-gate on kube-proxy should be enough to get them green again.
/cc
is back to just the guestbook test being solid red (tracked in ) 17 should green up similarly (was just fast-forwarded this morning)
This seems resolved. can we close this?
/close
Closing this issue. <detailsIn response to : /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: if !ok { return errors.NewBadRequest("resource was marked with kind Pod but was unable to be converted") } <del> if _, isMirrorPod := pod.Annotations[api.MirrorPodAnnotationKey]; isMirrorPod { return nil } </del> // Make sure that the client has not set `priority` at the time of pod creation. if operation == admission.Create && pod.Spec.Priority != nil { return admission.NewForbidden(a, fmt.Errorf("the integer value of priority must not be provided in pod spec. Priority admission controller populates the value from the given PriorityClass name"))

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Priority admission controller of mirror pods. This is an error and should be removed. What you expected to happen: Integer value of Priority should be populated by the admission controller. How to reproduce it (as minimally and precisely as possible): Create a static pod with PriorityClassName set. The created mirror pod will not get the integer value of priority. /sig scheduling xref
what is the kubelet expected to do with a low priority static pod if it gets evicted?
also doesn't this mean a user can claim their pod is a mirror pod via an annotation and get it admitted with any priority they want?
knows better about kubelet evictions, but in my opinion a static low priority pod should be evicted similar to other pods. Whether it makes sense for static pods to be low priority or not is a relevant question. I would say it doesn't make sense, but one might find a reason to have static low priority pod. Not really. This check effectively bypasses integer resolution of priority causing the priority of all those mirror pods to be considered zero. This if statement is wrong and should be removed.
It also bypasses the check that prevents the pod creator from setting an integer priority themselves. In any case, agree removing the bypass is good
Static pods that fail for any reason (eviction is just one) are not re-run. To avoid evictions entirely as a static pod, users can use the critical pod annotation.
, it seems we just log error if creating mirror pod failed; refer to . Does it make sense?
I am not sure I understand the question entirely. The mirror pod exists only to show the status of the static pod. If the mirror pod doesn't exist, the static pod should still function correctly. Since that is in , it will be retried periodically.
The scheduler dependent on mirror pods for resource usage :). If creating mirror pod failed (e.g. wrong priority class name), there'll be ping-pong between scheduler and kubelet: scheduler send pod to node, kubelet reject it. But we can-t just return error, kubelet can start in standalone mode.
However critical pod annotation is ignored if pod priorities are in use.

This is not the case. See in your issue.
However critical pod annotation is ignored if pod priorities are in use. the intent of pod priorities was to replace the critical pod annotation with something more principled. is that not how it was implemented?
Priority replaces most of the functionality that the critical pod annotation provides. However, the critical pod annotation is still useful for static pods, as it can prevent kubelet evictions. But for all other use-cases, I agree that priority is a much better way to manage which workloads are run, and which remain pending. I am not sure if the scheduler translates a "critical" pod to a "system-priority" pod for scheduling purposes, but the kubelet still prevents the eviction of static critical pods as an escape hatch for important static pods which cannot tolerate kubelet evictions.
Why would that not be modeled as a system priority? I expected critical pod annotations to not be present once the priority feature was completed.
I agree with this. do you think we should not evict static pods which are system priority, so that even users of static pods can move from using the critical pod annotation to use pod priority?
I think Kubelet should not evict static system pods, but even with the current logic of eviction, if these Pods do not use more resources than requested, kubelet never evicts them as their priority is higher than other pods.
I think as a best practice users that write static pod manifests should apply a system priority to prevent eviction, but I do not think the kubelet should do anything magical about that source.


Question: t.Errorf("Expected %v, but got %v", e, a) } <ins> // Test a simple id (using second ip). location, _, err = redirector.ResourceLocation(ctx, "foo-second-ip") if err != nil { t.Errorf("Unexpected error: %v", err) } if location == nil { t.Errorf("Unexpected nil: %v", location) } if e, a := "//[2001:db7::]:80", location.String(); e != a { t.Errorf("Expected %v, but got %v", e, a) } </ins> // Test a name + port. location, _, err = redirector.ResourceLocation(ctx, "foo:p") if err != nil {

Hi, I have a issue with a dualstack cluster for my Kubernetes Dashboard deployment. I opened a ticket here (), but I am informed that it is not related to Kubernetes Dashboard. What is the problem? Errors in kube-api: The service is created with the correct ipFamily (=IPv6). My endpoint is fully in IPv6. Indeed, my pod has one IP in IPv4 and another in IPv6. Do you have also the issue? Kubernetes version: 1.18 OS: RHEL7
/sig network /sig api-machinery
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
I think the following should fix the error you saw:
/remove-sig api-machinery


Question: }, []string{"result"}) <ins> CacheSize = metrics.NewGaugeVec( &metrics.GaugeOpts{ Subsystem:      SchedulerSubsystem, Name:           "scheduler_cache_size", Help:           "Number of nodes, pods, and assumed (bound) pods in the scheduler cache.", StabilityLevel: metrics.ALPHA, }, []string{"type"}) </ins> metricsList = []metrics.Registerable{ scheduleAttempts, SchedulingLatency,

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Add a metric to track scheduler cache size. metric name Description -- -- schedulercachesize Number of elements in scheduler cache. Why is this needed: Gives better observability of scheduler cache. For, the number of "assumedpods" gives us an idea of how many pods are "assumed" and therefore being "bound". We can compare this with number of to make sure assume-bind works as expected. /sig scheduling
Is anyone already working on these metrics? If not I'd like to help out /assign
Thanks . Feel free to work on this one. Just a reminder, we may need feedback/approval from sig-instrumentation.


Question: <ins> /* Copyright 2014 Google Inc. All rights reserved. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package config import ( "reflect" "testing" "github.com/GoogleCloudPlatform/kubernetes/pkg/api" "github.com/GoogleCloudPlatform/kubernetes/pkg/api/testapi" "github.com/ghodss/yaml" ) func noDefault(*api.Pod) error { return nil } func TestDecodeSinglePod(t *testing.T) { pod := &api.Pod{ TypeMeta: api.TypeMeta{ APIVersion: "", }, ObjectMeta: api.ObjectMeta{ Name:      "test", UID:       "12345", Namespace: "mynamespace", }, Spec: api.PodSpec{ RestartPolicy: api.RestartPolicyAlways, DNSPolicy:     api.DNSClusterFirst, Containers: []api.Container{{ Name:                   "image", Image:                  "test/image", ImagePullPolicy:        "IfNotPresent", TerminationMessagePath: "/dev/termination-log", }}, }, } json, err := testapi.Codec().Encode(pod) if err != nil { t.Errorf("unexpected error: %v", err) } parsed, podOut, err := tryDecodeSinglePod(json, noDefault) if testapi.Version() == "v1beta1" { // v1beta1 conversion leaves empty lists that should be nil podOut.Spec.Containers[0].Resources.Limits = nil podOut.Spec.Containers[0].Resources.Requests = nil } if !parsed { t.Errorf("expected to have parsed file: (%s)", string(json)) } if err != nil { t.Errorf("unexpected error: %v (%s)", err, string(json)) } if !reflect.DeepEqual(pod, podOut) { t.Errorf("expected:n%#vngot:n%#vn%s", pod, podOut, string(json)) } externalPod, err := testapi.Converter().ConvertToVersion(pod, "v1beta3") if err != nil { t.Errorf("unexpected error: %v", err) } yaml, err := yaml.Marshal(externalPod) if err != nil { t.Errorf("unexpected error: %v", err) } parsed, podOut, err = tryDecodeSinglePod(yaml, noDefault) if !parsed { t.Errorf("expected to have parsed file: (%s)", string(yaml)) } if err != nil { t.Errorf("unexpected error: %v (%s)", err, string(yaml)) } if !reflect.DeepEqual(pod, podOut) { t.Errorf("expected:n%#vngot:n%#vn%s", pod, podOut, string(yaml)) } } func TestDecodePodList(t *testing.T) { pod := &api.Pod{ TypeMeta: api.TypeMeta{ APIVersion: "", }, ObjectMeta: api.ObjectMeta{ Name:      "test", UID:       "12345", Namespace: "mynamespace", }, Spec: api.PodSpec{ RestartPolicy: api.RestartPolicyAlways, DNSPolicy:     api.DNSClusterFirst, Containers: []api.Container{{ Name:                   "image", Image:                  "test/image", ImagePullPolicy:        "IfNotPresent", TerminationMessagePath: "/dev/termination-log", }}, }, } podList := &api.PodList{ Items: []api.Pod{*pod}, } json, err := testapi.Codec().Encode(podList) if err != nil { t.Errorf("unexpected error: %v", err) } parsed, podListOut, err := tryDecodePodList(json, noDefault) if testapi.Version() == "v1beta1" { // v1beta1 conversion leaves empty lists that should be nil podListOut.Items[0].Spec.Containers[0].Resources.Limits = nil podListOut.Items[0].Spec.Containers[0].Resources.Requests = nil } if !parsed { t.Errorf("expected to have parsed file: (%s)", string(json)) } if err != nil { t.Errorf("unexpected error: %v (%s)", err, string(json)) } if !reflect.DeepEqual(podList, &podListOut) { t.Errorf("expected:n%#vngot:n%#vn%s", podList, &podListOut, string(json)) } externalPodList, err := testapi.Converter().ConvertToVersion(podList, "v1beta3") if err != nil { t.Errorf("unexpected error: %v", err) } yaml, err := yaml.Marshal(externalPodList) if err != nil { t.Errorf("unexpected error: %v", err) } parsed, podListOut, err = tryDecodePodList(yaml, noDefault) if !parsed { t.Errorf("expected to have parsed file: (%s)", string(yaml)) } if err != nil { t.Errorf("unexpected error: %v (%s)", err, string(yaml)) } if !reflect.DeepEqual(podList, &podListOut) { t.Errorf("expected:n%#vngot:n%#vn%s", pod, &podListOut, string(yaml)) } } </ins>

Following on from the Kubelet should accept YAML pod specifications. Currnelty it supports only JSON pod specifications.


Question: "$ref": "#/definitions/intstr.IntOrString" }, "maxUnavailable": { <del> "description": "The maximum number of pods that can be unavailable during the update. Value can be an absolute number (ex: 5) or a percentage of desired pods (ex: 10%). Absolute number is calculated from percentage by rounding up. This can not be 0 if MaxSurge is 0. By default, a fixed value of 1 is used. Example: when this is set to 30%, the old RC can be scaled down to 70% of desired pods immediately when the rolling update starts. Once new pods are ready, old RC can be scaled down further, followed by scaling up the new RC, ensuring that the total number of pods available at all times during the update is at least 70% of desired pods.", </del> <ins> "description": "The maximum number of pods that can be unavailable during the update. Value can be an absolute number (ex: 5) or a percentage of desired pods (ex: 10%). Absolute number is calculated from percentage by rounding down. This can not be 0 if MaxSurge is 0. By default, a fixed value of 1 is used. Example: when this is set to 30%, the old RC can be scaled down to 70% of desired pods immediately when the rolling update starts. Once new pods are ready, old RC can be scaled down further, followed by scaling up the new RC, ensuring that the total number of pods available at all times during the update is at least 70% of desired pods.", </ins> "$ref": "#/definitions/intstr.IntOrString" } }

API documentation says is rounded up, but it is actually rounded down: The code shows it is rounded down, not up: and
Yes, the documentation is obsolete. We changed maxUnavailable to round down so that we wouldn't accidentally infer 100% unavailability for non-100 percentages. For example, maxSurge=25% and maxUanavailable=25% with a Deployment of 1 replica meant that we could scale down 1 before thus leading in 100% maxUnavailable (implicit).


Question: # KUBE_RACE="-race" make -C "${KUBE_ROOT}" test  WHAT="${WHAT:-$(kube::test::find_integration_test_dirs | paste -sd' ' -)}"  <del> KUBE_GOFLAGS="${KUBE_GOFLAGS:-}"  </del> <ins> GOFLAGS="${GOFLAGS:-}"  </ins> KUBE_TEST_ARGS="${KUBE_TEST_ARGS:-} ${SHORT:--short=true} --vmodule=garbage*collector*=6 --alsologtostderr=true"  KUBE_RACE=""  KUBE_TIMEOUT="${KUBE_TIMEOUT}"

We should just use GOFLAGS
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
/sig contributor-experience
/assign