Question: addCRIPodCPUStats(ps, criSandboxStat) addCRIPodMemoryStats(ps, criSandboxStat) addCRIPodProcessStats(ps, criSandboxStat) <del> p.makePodStorageStats(ps, rootFsInfo) </del> <ins> makePodStorageStats(ps, rootFsInfo, p.resourceAnalyzer, p.hostStatsProvider, true) </ins> summarySandboxStats = append(summarySandboxStats, *ps) } return summarySandboxStats, nil

I have a cronjob that runs a Python script every 5 minutes. After the cronjob finishes running, I start seeing a bunch of log messages in my systemd journal like the following: Those directories in /var/log don't exist as they belong to old pods that have finished running. These messages keep increasing in volume as time passes (almost received a 1000 of those messages at one time, after which I had to reboot the node to fix the issue). It almost looks like the kubelet is not cleaning up the old pods for some reason. I can't see these old pods when I run , so I'm assuming their is some issue with the kubelet's cleanup process. kubelet (or cadvisor) shouldn't try to read logs for pods that don't exist. Setup a short lived cronjob running every 5 minutes and observe the logs (Kubernetes v1.22.4, CRI-O v1.22.1, Ubuntu 20.04) No response <details</detailsBaremetal cluster <details</details<detailskubeadm (baremetal cluster) </details<detailsCRI-O v1.22.1 </details<details</details
/sig node
/assign for triage
I have something similar on a fresh cluster I use for educational purposes using kubernetes 22.5 with cri-o. Here's an excerpt: This is from the master node. Nodes run Debian bulsseye in KVM provisioned with Vagrant. I have not yet tested it out with different versions. If I can assist in testing, I'd be happy to do so!
/triage accepted /priority important-longterm
I've produced the same issue on my clusters. I'm running Debian 11 on bare metal (homelab) and in VM (test). My clusters are installed using kubeadm, and I'm using multus and calico as CNI. The issue is triggered as soon as a pod is removed. I've triggered it by deleting a simple Pod, scaling down a Deployment, or running a CronJob. On the node where the pod was removed, the logs are then flooded by messages, at the rate of 1 line per pod removed every 10 seconds. I've seen it both on aregular worker nodes, and untainted master nodes.

I've tested under Vagrant with various K8S / CRI-O versions. With CRI-O , I never triggered the issue with Kubernetes : , and . However, with CRI-O = , all my tests triggered the issue : Kubernetes , CRI-O Kubernetes , CRI-O Kubernetes , CRI-O Let me know if I can help by running other tests or capturing specific logs.
Is this only issue with CRI-O? What about containerd? The reports above seem to all be on CRI-O? /cc
openshift seems to have hit this case a couple of times and digging into some bugs leads me to think it will be fixed with and kube 1.24. can you try one of the kube alpha releases? cri-o 1.23.1 should do, or you can try the main branch
it's possible it's unrelated, but is why I am making this guess
I'm used to installing a cluster using Debian packages, so it took me a few tries to have a working v1.24 cluster. I followed the "Without a package manager" to install from the binaries. I used Kubernetes and CRI-O . I'm not certain everything is working fine, but I can schedule a Pod / Deployment / Job, so I assume it's good enough to test this issue. And I still reproduce the issue as soon as I remove a Pod.
I'm also hitting the issue in my lab environment(Vagrant). Running Rocky Linux 8.5 Kubernetes 1.22.6 and CRI-O 1.22.2
ah hah! I found a fix (and found out why it's only being hit in cri-o and not cadvisor)
We're also having this issue on a cluster hosting Gitlab CI jobs, this is quite a trouble since many jobs are spawned and removed daily. Kubernetes version: v1.22.7 CRI-O version: 1.22.2 Is there any news about the fix ? I looked into 's PR, it seemed ok to me but I'm not really experienced in Go... Nevertheless I'd be happy to help
up
I can confirm on a bare metal cluster that I'm running on containerd that I am also seeing these log messages.
Observing similar issues on (k8s and crio ) and (k8s and crio ).

The strange thing is that one cluster just some nodes affected while the other, all of them. Below an overview of 30m occurrences of Cluster running version : Cluster running version : No idea where the discrepancy is coming from.
Tested with and on a fresh cluster bootstraped with kubeadm, all machines run on Debian 11. Ran a job 3 times and there's a log line for every execution, so it spams 3 log lines per kubelet loop.
It looks like cgroups are left behind. Removing them fixes the logspam (temporarily). Can use something like this to find all of them, depending on actual error in journal: Then iterate over the and remove them, while checking that running pods are not considered: It's quick and dirty, but does the job. I've deployed a garbage collector daemonset with the above and until this is properly addressed, I'm quite happy with the fix. The problem is, for a medium to large size cluster with pods coming and going, it's very easy to bust your elasticsearch storage and performance with this log spam.
Yesterday we tested with the newly released version of and the problem was still here. After reading answer I digged a little bit because we use systemd to manage cgroups and it seemed the error was on our side. I updated the kernel boot command in the grub file because in order to have cgroupsv2 unified behavior in systemd you have to set both options: and and we only had : Used the following command for the changes to take effect: And then rebooted the node. At first it seemed like this did the trick as I popped a job on the node I changed and the kubelet didn't spam the logfile anymore. As soon as I deleted the pod from the cluster the kubelet started spamming the logs again with: So this did not resolve the issue. This was tested on a node running , , and .
Looks like the cgroup slices not being cleaned up by systemd is an old issue:
would you mind giving a try to see if it helps?
while your fix does make sense, the root cause (which atm is somewhat unknown) is not really addressed. Something's leaking cgroup slices... and have both been closed off.
I am not sure if they're leaking as much as have yet to be cleaned up. Have those pods been deleted by the kubelet yet?

and by deleted, I mean fully removed (no traces if searched by ). The kubelet will clean up the pod scope when GC kicks in, but will keep them around until then to allow for forensics and inspection to be performed. Since your script uses to find the scopes, I am guessing the pod themselves have yet to be GC'd, and should still exist. Which is why I think that is the correct solution: we expect those scopes to be there, but shouldn't unnecessarily log an error if a pod that's stopped but not yet removed is found
to verify my suspicion, I would be interested in seeing for all the pods that are showing up in your script, as well as to cross reference pods to containers, if you wouldn't mind uploading that somewhere.
I'm sure they're not in crio anymore. I'm only using to ensure I'm never going to touch a scope that's reported under a pod still seen by crio. I'll scale down the DaemonSet on my test cluster, run some workload to generate the leaking pods and then provide you some log snippets and requested output from crictl.
I took one of my clusters that's suffering heavily on this issue (hosting gitlab CI jobs), reactivated the logs on it for a second, doesn't take long for the kubelet to log things like this: Trying to inspect the pod with returns: I tested with a few pods and the result is always the same. This seems logic as gives me the following: But the slices are still here tho: So in my opinion crictl return the right things, it's either the kubelet that doesn't update correctly the list of pods to watch or the systemd problem with the slices.
I've attached the logs and requested outputs here (sanitized hostnames). Let me know if you'd need anything else; for now I'll leave them around and not scale up the DS on the test cluster.
I've posted a quick gist with all the manifests I'm using to deploy a garbage collector daemon set. Nothing fancy, just a bit of patchwork to address the issues I was having.
Here's another really interesting bit of information. In my case, this issue seems to be only affecting VMs and not bare-metal nodes.

It's completely puzzling, but I have 12 bare-metal nodes and 1 VM node in one cluster (the VM is just for testing) and it manifests only on the VM node. As well, my test cluster is made up fully of VMs (ie no bare-metal) and there it happens across the board. The bare metal nodes where the VMs are running are the same are same hardware as the bare-metal nodes of my main cluster (just more NVMe drives). Edit: amended to clarify this is observed in my case running OKD.
Sorry but I have the same issue on bare-metal nodes (x86 ArchLinux kernel 5.17.5 with K8S 1.23.6/Crio 1.23.2)
The issue in title happens on bare metal nodes too.
I've got the same issue on bare-metal nodes(k8s v1.23.0/cri-o v1.22.3) and I'm not sure if it's for the same reason, but I have this problem as well.
I figured out the reason for the discrepancy. The VM node was using cgroups v2, while the bare-metal, older nodes, had the kernel parameter . Removing that and rebooting, triggers the problem.
Did some testing and I can confirm that switching from CRI-O to containerd has stopped these log messages from popping up all the time. For now, this seems like a reasonable solution, at least for me. Using Cgroups v2 Kubernetes Version: v1.23.7 OS: Ubuntu 22.04 Containerd Version: v1.6.4 Runc Version: 1.1.1 EDIT: Kubernetes Version was v1.23, not v1.24
Another option is one could switch to the CRI stats provider by setting kubelet's containerruntimeendpoint to instead of
Is this also an option for OKD/OCP? Is there any progress on a fix or should we look at the above workaround instead?
Out of curiosity, why does this work? /cc
This would work for OKD, but wouldn't be supported for OCP OOTB. should fix, but it needs a reviewer/approver... Fun kubernetes fact! The CRI-O team found performance issues with the CRI stats provider, so there's a hardcoded check to see if the socket is .

However, since is symlinked to , and the check is a string comparison, one can use and not trigger that toggle, turning on CRI stats provider. We're working on fixing the perf troubles with CRI stats KEP 2371
Using doesn't solve the problem on my environment, we're still seeing the same error logs. We're running: Flatcar 3033.2.4 Kubelet 1.22.4 CRI-O 1.22.5 cgroup v2 enabled.
I had this error on Kubernetes version 1.24, with CRI-O 1.24. Today I did a fresh install with Kubernetes 1.25 + CRI-O 1.25 (with the same machine types and OS versions) and the error went away. I tried deployment scale-down, pod deletion, but I no longer see the mentioned "Unable to fetch log stats" messages in the log. VirtualBox machines OS: RHEL 9 with kernel 5.14.0-70.22.1.el90.x8664 Kubernetes version: 1.25.0 CRI-O version: 1.25.0 Runc version 1.1.3 Using cgroups v2
We are running kubernetes 1.24.9 with cri-o 1.24.4 and runc 1.1.4 on Ubuntu 22.04 with cgroups v2 and experienced the same issue. For some time I was running a deployment with a dozen of pods and rotated them every 10 seconds to create a lot of garbage in cgroups. This caused rather slow increase in both mem & cpu usage and a lot of error messages like : Kubelet mem & cpu usage: ! ! After I stopped generating those pods the resource usage stopped. After running script from the CPU&mem usage dropped.


Question: blockVolumeMapper: reconstructedVolume.blockVolumeMapper, mounter:           reconstructedVolume.mounter, } <del> if cachedInfo, ok := volumesNeedUpdate[reconstructedVolume.volumeName]; ok { gvl = cachedInfo } gvl.addPodVolume(reconstructedVolume) </del> if volumeInDSW { // Some pod needs the volume. And it exists on disk. Some previous // kubelet must have created the directory, therefore it must have

This issue is very similar to , they have the same symptom: remote volumes of the pods are becoming local mount point after rebooting the Node. And this issue is triggered by The difference is that in , kubelet didn't call , so the global mount point is local, but it will still call for the pod to bind mount the pod volume to the global mount point. Whereas in this issue, kubelet neither call globally nor call for the pod. This is more harmful: After writing some data to the pod volume, pod can't be deleted because kubelet encounter error while unmounting the pod volume: ! Reproduce steps: a cluster with one Node a deployment with two replicas(say and ) using Non-attachable volume (for example Azurefile) the Node with option (Use force deletion to simulate the situation that mount point was left over before rebooting the Node.) back the Node is always in Running state, however, after rebooting, it won't call and the volume it uses is local volume. should call . As mentioned above. No response <details</details<detailsN/A </details<details</details<details</details<details</details<details</details
Clusters with enabled is not impacted.
/sig node storage (for triage)
/remove-sig node sig-storage please triage this first. Thanks
/triage accepted


Question: t.Errorf("Expected endpoint: %+v, got: %+v", testCase.expectedEndpoint, endpoint) } <ins> expectTrackedResourceVersion(t, r.endpointSliceTracker, &slice, "100") </ins> expectMetrics(t, expectedMetrics{desiredSlices: 1, actualSlices: 1, desiredEndpoints: 1, addedPerSync: 1, removedPerSync: 0, numCreated: 1, numUpdated: 0, numDeleted: 0}) }) }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: With EndpointSlice feature enabled, after EndpointSliceController creates or updates an EndpointSlice when reconciling a Service, the Service would always be processed one extra time, which is triggered by the EndpointSlice /updated event made by itself during the first processing. What you expected to happen: EndpointSliceController shouldn't reprocess a Service due to the EndpointSlice update made by itself. How to reproduce it (as minimally and precisely as possible): EndpointSlice feature, improve kube-controller-manager log to 5. a nginx deployment a Service backed by the above deployment. kube-controller-manager log: the service will be processed twice in succession. Anything else we need to know?: It's because EndpointSliceController tracks stale resource version after creating/updating an EndpointSlice. Environment: Kubernetes version (use ): v1.18.0-alpha.3 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/assign
/sig network
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/remove-triage unresolved


Question: if !ok { return false } <ins> providedSlices := map[types.UID]int64{} </ins> for _, endpointSlice := range endpointSlices { <ins> providedSlices[endpointSlice.UID] = endpointSlice.Generation </ins> g, ok := gfs[endpointSlice.UID] if ok && (g == deletionExpected || g > endpointSlice.Generation) { return true } } <ins> for uid, generation := range gfs { if generation == deletionExpected { continue } _, ok := providedSlices[uid] if !ok { return true } } </ins> return false }

The controllers can create a second EndpointSlice after an immediate sync if the first EndpointSlice they created is not in the cache yet. Here's what can happen: gets event that triggers sync for Service syncs and creates EndpointSlice gets event that triggers sync for same Service syncs and creates EndpointSlice (the slice created in 2 has not made it to the cache yet) A single EndpointSlice to be created. Run integration tests hundreds of times that involve 2 consecutive events that will trigger a sync. Example: /assign /sig network
/triage accepted
/cc
When this bug is fixed, we'll need to rollback
just amend it in the PR that fix the bug, so we merge all together


Question: <ins> --- apiVersion: apps/v1 kind: DaemonSet metadata: name: kube-network-policies namespace: kube-system labels: tier: node app: kube-network-policies k8s-app: kube-network-policies addonmanager.kubernetes.io/mode: Reconcile spec: selector: matchLabels: app: kube-network-policies template: metadata: labels: tier: node app: kube-network-policies k8s-app: kube-network-policies spec: hostNetwork: true tolerations: - operator: Exists effect: NoSchedule serviceAccountName: kube-network-policies containers: - name: kube-network-policies image: registry.k8s.io/networking/kube-network-policies:v0.1.0 command: - /bin/sh - -c - /bin/netpol -v 4 1>>/var/log/kube-network-policies.log 2>&1 resources: requests: cpu: "100m" memory: "50Mi" securityContext: privileged: true volumeMounts: - mountPath: /var/log name: varlog readOnly: false - mountPath: /lib/modules name: lib-modules readOnly: true volumes: - name: varlog hostPath: path: /var/log - name: lib-modules hostPath: path: /lib/modules </ins>

,%20google-gce all I can't say , but at least a few weeks ,%20google-gce No response No response /sig network
/assign
/triage accepted


Question: "k8s.io/klog" ) <del> // SharedInformer has a shared data cache and is capable of distributing notifications for changes // to the cache to multiple listeners who registered via AddEventHandler. If you use this, there is // one behavior change compared to a standard Informer.  When you receive a notification, the cache // will be AT LEAST as fresh as the notification, but it MAY be more fresh.  You should NOT depend // on the contents of the cache exactly matching the notification you've received in handler // functions.  If there was a create, followed by a delete, the cache may NOT have your item.  This // has advantages over the broadcaster since it allows us to share a common cache across many // controllers. Extending the broadcaster would have required us keep duplicate caches for each // watch. </del> <ins> // SharedInformer provides eventually consistent linkage of its // clients to the authoritative state of a given collection of // objects.  An object is identified by its API group, kind/resource, // namespace, and name.  One SharedInfomer provides linkage to objects // of a particular API group and kind/resource.  The linked object // collection of a SharedInformer may be further restricted to one // namespace and/or by label selector and/or field selector. // // The authoritative state of an object is what apiservers provide // access to, and an object goes through a strict sequence of states. // A state is either "absent" or present with a ResourceVersion and // other appropriate content. // // A SharedInformer maintains a local cache, exposed by Store(), of // the state of each relevant object.  This cache is eventually // consistent with the authoritative state.  This means that, unless // prevented by persistent communication problems, if ever a // particular object ID X is authoritatively associated with a state S // then for every SharedInformer I whose collection includes (X, S) // eventually either (1) I's cache associates X with S or a later // state of X, (2) I is stopped, or (3) the authoritative state // service for X terminates.  To be formally complete, we say that the // absent state meets any restriction by label selector or field // selector. // // As a simple example, if a collection of objects is henceforeth // unchanging and a SharedInformer is created that links to that // collection then that SharedInformer's cache eventually holds an // exact copy of that collection (unless it is stopped too soon, the // authoritative state service ends, or communication problems between // the two persistently thwart achievement). // // As another simple example, if the local cache ever holds a // non-absent state for some object ID and the object is eventually // removed from the authoritative state then eventually the object is // removed from the local cache (unless the SharedInformer is stopped // too soon, the authoritative state service emnds, or communication // problems persistently thwart the desired result). // // The keys in Store() are of the form namespace/name for namespaced // objects, and are simply the name for non-namespaced objects. // // A client is identified here by a ResourceEventHandler.  For every // update to the SharedInformer's local cache and for every client, // eventually either the SharedInformer is stopped or the client is // notified of the update.  These notifications happen after the // corresponding cache update and, in the case of a // SharedIndexInformer, after the corresponding index updates.  It is // possible that additional cache and index updates happen before such // a prescribed notification.  For a given SharedInformer and client, // all notifications are delivered sequentially.  For a given // SharedInformer, client, and object ID, the notifications are // delivered in order. // // A delete notification exposes the last locally known non-absent // state, except that its ResourceVersion is replaced with a // ResourceVersion in which the object is actually absent. </ins> type SharedInformer interface { // AddEventHandler adds an event handler to the shared informer using the shared informer's resync // period.  Events to a single handler are delivered sequentially, but there is no coordination // between different handlers. AddEventHandler(handler ResourceEventHandler) <del> // AddEventHandlerWithResyncPeriod adds an event handler to the shared informer using the // specified resync period.  Events to a single handler are delivered sequentially, but there is // no coordination between different handlers. </del> <ins> // AddEventHandlerWithResyncPeriod adds an event handler to the // shared informer using the specified resync period.  The resync // operation consists of delivering to the handler a create // notification for every object in the informer's local cache; it // does not add any interactions with the authoritative storage. </ins> AddEventHandlerWithResyncPeriod(handler ResourceEventHandler, resyncPeriod time.Duration) <del> // GetStore returns the Store. </del> <ins> // GetStore returns the informer's local cache as a Store. </ins> GetStore() Store // GetController gives back a synthetic interface that "votes" to start the informer GetController() Controller // Run starts the shared informer, which will be stopped when stopCh is closed. Run(stopCh <-chan struct{}) <del> // HasSynced returns true if the shared informer's store has synced. </del> <ins> // HasSynced returns true if the shared informer's store has been // informed by at least one full LIST of the authoritative state // of the informer's object collection.  This is unrelated to "resync". </ins> HasSynced() bool // LastSyncResourceVersion is the resource version observed when last synced with the underlying // store. The value returned is not synchronized with access to the underlying store and is not

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: As a developer, I had important questions about SharedInformer that I could not answer from the comments on it. Colleagues also have questions. The comments on SharedInformer mainly contain a contrast with "standard Informer" --- but there no longer is such a thing. So most of the important facts about SharedInformer are not documented in the comments. What you expected to happen: I expected to be able to understand the contract between a SharedInformer and its clients by reading the comments on that interface. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:


Question: if left <= len(data) { copy(data, d.remaining) d.remaining = nil <del> return len(d.remaining), nil </del> <ins> return left, nil </ins> } // caller will need to reread

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: In a go program, using the YAML utility package "", the YAMLDecoder Read function always returns n=0 bytes, indicating that nothing was read. What you expected to happen: The number of bytes read should be returned as specified in the How to reproduce it (as minimally and precisely as possible): See Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: minikube & AWS OS (e.g. from /etc/os-release): macOS 10.13.1 (17B1003) Kernel (e.g. ): Darwin 17.2.0 Install tools: Others:
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc


Question: <ins> // +build !darwin,!linux /* Copyright 2021 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package copycerts func setNoUmask() { // no-op on not-unix } </ins>

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Running locally. Today? There is none, but I'm not filing this as a flaking test because it's a dependable failure, not a flake. I suspect this depends on the user's umask given the error: originally /area kubeadm /sig cluster-lifecycle
/assign
there were related changes to TestDownloadCerts last month: we could pin the mode for these newly created files in kubeadm. /triage accepted
but the problem seems odd. kubeadm uses client-go utils for writing a cert file with mode 644: i don't understand why something would override this mode to 0640.
i don't understand why something would override this mode to 0640. From the initial issue filing: -- users may be running in an environment with a less permissive umask, but the test depends on a full permissive umask.
ok, right. a quick fix would be to get the current process umask and expect it to be applied to the file permissions: but note: the test should be made Unix only too.
I think it would be reasonable to just set umask without getting it in the TestMain for the package. IIRC go runs tests as a binary / process per package, so it would only affect that package. We could make setting the umask only happen on unix without making the tests unix only as well, by using a conditionally compiled wrapper. I can send a PR for this, I suspect the system umask is google's workstation customization (~~debian testing), and probably relatively unusual elsewhere. I recall now we've run into this in the past during builds where e.g. does not result in the permissions on other systems so we've chmod to e.g.
the test would fail on Windows because -would always return there. so we should os-branch the checks.
I'm not sure I understand the windows part tbh, I think this particular failure can be addressed without affecting windows though. After meetings I got around to confirming on my personal debian stable instance reports while on my "gLinux" (~debian testing) workstation it reports . AFAICT should be more or less the standard across distros but debian and ubuntu at least provide a mechanism out of the box to configure it for the system.

yeah, the test is already broken on non-Unix, so we could just set the umask (syscall.Umask) to make the test happy. the non-Unix fix seems like something that can be done separately.
filed to set it to 0 in test main so it would be technically not umask sensitive anymore on linux and macOS at least.


Question: software-properties-common  lsb-release <del> # focal repo for docker is not yet available, so we use bonic for now # https://github.com/kubernetes/kubernetes/issues/90709 </del> release=$(lsb_release -cs) <del> if [ "$release" == "focal" ]; then release="bionic"; fi </del> # Add the Docker apt-repository (as we install containerd from there) curl -fsSL https://download.docker.com/linux/$(. /etc/os-release; echo "$ID")/gpg

We have some scripts that install docker/containerd etc and are having trouble installing on focal based images as does not yet support focal fossa. So in we have a patch that uses debs for focal. Additional context, please see ci test failures in and This issue is to track fixing the script back up when focal support is to
/sig release /sig testing
/priority important-soon /area release-eng
/assign


Question: "net/http/httptest" "net/url" "os" <del> // "reflect" </del> <ins> "reflect" </ins> "strings" "testing" "time"

This tree has a commit that demonstrates two obviously different objects that should not pass the semantic DeepEqual but do.
What shall we do with this issue for triage? Labels? Priority?
P2 for when he returns , Satnam Singh wrote:
OK I am now sufficiently worried about this. Fixes incoming.


Question: } } } <ins> if config, ok := c.configs[version]; ok { return config, nil } </ins> // TODO: have a better config copy method config := *c.defaultConfig <del> if len(version) != 0 { config.Version = version </del> <ins> negotiatedVersion, err := client.NegotiateVersion(&config, version) if err != nil { return nil, err </ins> } <ins> config.Version = negotiatedVersion </ins> client.SetKubernetesDefaults(&config) <ins> c.configs[version] = &config </ins> return &config, nil }

This seems like a bug to me No output, and command exits success... wat. Have to explicitly pass to see expected output. If the client version is greater the server version, it should be making requests based on the server's newest accepted apiversion.
cc


Question: kubecontainer "k8s.io/kubernetes/pkg/kubelet/container" "k8s.io/kubernetes/pkg/kubelet/lifecycle" "k8s.io/kubernetes/pkg/kubelet/pluginmanager/cache" <del> "k8s.io/kubernetes/pkg/kubelet/qos" </del> "k8s.io/kubernetes/pkg/kubelet/stats/pidlimit" "k8s.io/kubernetes/pkg/kubelet/status" schedulerframework "k8s.io/kubernetes/pkg/scheduler/framework"

runs in a container. Talos sets OOM score to -450 for the kubelet container and passes in the kubelet config. End result is that OOM score adj is -999 though. should have configured OOM score . Source code analysis. First, OOM score is set correctly from the config (flags): But then, in container manager, OOM score is reset to -999 via the following: This function should skip action if the PID is not running on the host: But the function looks to be confusing: When kubelet runs in a container with PID namespace shared with the host, kubelet's PID namespace will be equal to PID 1 namespace. If kubelet runs in a container with unshared PID namespace, kubelet's PID is 1 and its PID namespace is identical to PID 1 namespace. So how this function can return ever? Moreover, function is always setting OOM score to a hardcoded constant -999 ignoring configuation (). No response <details, reproduced with 1.22. </details<details</details<details</details<details</details<details</details<details</details
/sig node
/triage accepted
/assign
I don't think this method needs to change... I have make a draft PR for the default value to use the flag if present, but I would like more time to add some testing. Maybe you can have a look: ?
Thanks, this PR looks good to me, as it's going to set the OOM score to the configured value. And in our case kubelet runs in a container (not on the host, with proper OOM score via OCI spec), so this code should be even skipped, not sure how this can be addressed.
neither do I...
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale


Question: "github.com/spf13/pflag" ) <del> const ( ExperimentalProxyModeAnnotation = "net.experimental.kubernetes.io/proxy-mode" ) </del> // ProxyServerConfig configures and runs a Kubernetes proxy server type ProxyServerConfig struct { componentconfig.KubeProxyConfiguration

This annotation dates back almost a year. This is a relatively easy project for someone to contribute.
I'll work on getting a pull request in tomorrow for this. Ive been looking at contributing and think this would be a great place to start
nevermind. Looks like a pull request already exists. Overlooked the mention due to being on mobile. Cheers
We'll file more :)


Question: <del> /* Copyright 2014 Google Inc. All rights reserved. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package e2e import ( "fmt" "time" "github.com/GoogleCloudPlatform/kubernetes/pkg/api" "github.com/GoogleCloudPlatform/kubernetes/pkg/client" "github.com/GoogleCloudPlatform/kubernetes/pkg/util" "github.com/golang/glog" . "github.com/onsi/ginkgo" . "github.com/onsi/gomega" ) // TestClusterDNS checks that cluster DNS works. func TestClusterDNS(c *client.Client) bool { if testContext.provider == "vagrant" { glog.Infof("Skipping test which is broken for vagrant (See https://github.com/GoogleCloudPlatform/kubernetes/issues/3580)") return true } podClient := c.Pods(api.NamespaceDefault) //TODO: Wait for skyDNS // All the names we need to be able to resolve. namesToResolve := []string{ "kubernetes-ro", "kubernetes-ro.default", "kubernetes-ro.default.kubernetes.local", "google.com", } probeCmd := "for i in `seq 1 600`; do " for _, name := range namesToResolve { probeCmd += fmt.Sprintf("wget -O /dev/null %s && echo OK > /results/%s;", name, name) } probeCmd += "sleep 1; done" // Run a pod which probes DNS and exposes the results by HTTP. pod := &api.Pod{ TypeMeta: api.TypeMeta{ Kind:       "Pod", APIVersion: "v1beta1", }, ObjectMeta: api.ObjectMeta{ Name: "dns-test-" + string(util.NewUUID()), }, Spec: api.PodSpec{ Volumes: []api.Volume{ { Name: "results", Source: api.VolumeSource{ EmptyDir: &api.EmptyDir{}, }, }, }, Containers: []api.Container{ { Name:  "webserver", Image: "kubernetes/test-webserver", VolumeMounts: []api.VolumeMount{ { Name:      "results", MountPath: "/results", }, }, }, { Name:    "pinger", Image:   "busybox", Command: []string{"sh", "-c", probeCmd}, VolumeMounts: []api.VolumeMount{ { Name:      "results", MountPath: "/results", }, }, }, }, }, } _, err := podClient.Create(pod) if err != nil { glog.Errorf("Failed to create %s pod: %v", pod.Name, err) return false } defer podClient.Delete(pod.Name) waitForPodRunning(c, pod.Name) pod, err = podClient.Get(pod.Name) if err != nil { glog.Errorf("Failed to get pod %s: %v", pod.Name, err) return false } // Try to find results for each expected name. var failed []string for try := 1; try < 100; try++ { failed = []string{} for _, name := range namesToResolve { _, err := c.Get(). Prefix("proxy"). Resource("pods"). Namespace("default"). Name(pod.Name). Suffix("results", name). Do().Raw() if err != nil { failed = append(failed, name) glog.V(4).Infof("Lookup using %s for %s failed: %v", pod.Name, name, err) } } if len(failed) == 0 { break } glog.Infof("lookups using %s failed for: %v", pod.Name, failed) time.Sleep(10 * time.Second) } if len(failed) != 0 { glog.Errorf("DNS using %s failed for: %v", pod.Name, failed) return false } // TODO: probe from the host, too. glog.Infof("DNS probes using %s succeeded", pod.Name) return true } var _ = Describe("TestClusterDNS", func() { It("should pass", func() { // TODO: Instead of OrDie, client should Fail the test if there's a problem. // In general tests should Fail() instead of glog.Fatalf(). Expect(TestClusterDNS(loadClientOrDie())).To(BeTrue()) }) }) </del>

Create a ginkgo suite to test services. Initially this could contain the following tests: TestClusterDNS TestKubernetesROService
Part of the ginko-ization of the e2e tests
Fixed by .


Question: "k8s.io/apimachinery/pkg/fields" "k8s.io/apimachinery/pkg/labels" "k8s.io/apimachinery/pkg/runtime" <ins> "k8s.io/apimachinery/pkg/types" </ins> "k8s.io/apimachinery/pkg/util/diff" genericapirequest "k8s.io/apiserver/pkg/endpoints/request" "k8s.io/apiserver/pkg/registry/generic"

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --The api bind a pod to a node. But the pod is specified by name (not uid), which is not a unique identifier for pod. So the caller of can't make sure the pod is exactly what he want. api can bind a pod with specified uid to a node. If the uid not match, return an error a pending pod by statefulset named busybox, make sure it's pending, the scheduler will not bind it(use non-existed node selector or other method). So we will get a pending pod busybox-0. following code the in the code, delete the pod busybox-0 manually during the sleep or other method. The statefulset busybox will create another pod busybox-0 with different uid , the api in the code above will bind an unexpected pod to node The logic in scheduler is the same. If the pod is recreate with the same name and different resource require or selector between scheduler's and , scheduler will do a wrong bind for the pod. I think the root cause is the api does not check the uid. Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"20+", GitVersion:"v1.20.0-alpha.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2020-10-20T09:01:41Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"16+", GitVersion:"v1.16.9-aliyunedge.1", GitCommit:"", GitTreeState:"", BuildDate:"2021-03-08T09:56:01Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig api-machinery
/sig scheduling for triage (the binding feature of (pod to node) is for scheduling).
For the sake of precedence, are there other APIs that are based on object UID?
uids are used as a precondition is used in many places. if the server started checking that a uid on the binding object (if specified) matched the uid of the pod being modified, that seems sufficient
is there any way for the caller of Bind to confirm that he is binding the right pod? or not uid, use resource version?
that would require: the server to check that bind UID (if specified) matches the pod UID (passing the binding uid from BindingREST#Create → assignPod → setPodHostAndAnnotations and verifying the uid match in setPodHostAndAnnotations) the binding API caller to set in the posted binding object (as illustrated in the description of this issue)
actually, kube-scheduler(the most common binding API caller) has set the in the posted binding object. but the server doesn't check the uid. in my program, i do those to work around(not elegant, but it works...): call the binding API with an uid in annotation(i found that BindingREST#Create → assignPod → setPodHostAndAnnotations will carry annotation). check the matchs , if not match, delete the pod and wait for an new pod ... from your point of view, is it necessary to ensure the pod is the caller want when someone call bind API? what's your advice for this issue
given the scheduler made decisions based on the current shape of the pod spec, I think it makes sense for it to work to make sure it is binding that instance of the pod I would advocate for supporting uid as a precondition in the binding API and starting to use it from the scheduler
i agree with you
/remove-sig api-machinery
I also agree with this and this cannot achieve only on sig/scheduling. Sorry, if it is not suitable, but I'd like to add sig/api-machinery label again.

/sig api-machinery
Either scheduling or node sigs own the pod binding API. Either of those would be a good place to coordinate adding a fix to honor uid as a precondition API machinery owns the generic API plumbing and API server behavior, not implementations of most specific APIs like the pod or pod binding APIs /remove-sig api-machinery
Given you already exercised Jodan's suggestion locally, would you mind raising an offical PR?
how about this PR


Question: topology *topology.CPUTopology nodeAllocatableReservation v1.ResourceList <ins> // sourcesReady provides the readiness of kubelet configuration sources such as apiserver update readiness. // We use it to determine when we can purge inactive pods from checkpointed state. sourcesReady config.SourcesReady </ins> } var _ Manager = &manager{} <ins> type sourcesReadyStub struct{} func (s *sourcesReadyStub) AddSource(source string) {} func (s *sourcesReadyStub) AllReady() bool          { return true } </ins> // NewManager creates new cpu manager based on provided policy func NewManager(cpuPolicyName string, reconcilePeriod time.Duration, machineInfo *cadvisorapi.MachineInfo, numaNodeInfo topology.NUMANodeInfo, nodeAllocatableReservation v1.ResourceList, stateFileDirectory string, affinity topologymanager.Store) (Manager, error) { var topo *topology.CPUTopology

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: PR , it will remove "inactive containers" while reconciling. But the cpumanager started very beginning, it will get empty activePods at it first run. This make state file no sense. What you expected to happen: We should wait a moment or notification from podmanager before starting reconcile. Some log may be before remove "inactive containers". How to reproduce it (as minimally and precisely as possible): restart kubelet Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig node
In reconcileState : You mean the cpu assignments would be wiped for the first reconciliation ?
Yes, you can test it by restarting kubelet.
Do you see any of the following log:
I see this
/assign


Question: ], deps = [ "//cmd/kube-apiserver/app/options:go_default_library", <ins> "//cmd/kube-apiserver/app/testing:go_default_library", </ins> "//staging/src/k8s.io/api/admission/v1beta1:go_default_library", "//staging/src/k8s.io/api/admissionregistration/v1beta1:go_default_library", <ins> "//staging/src/k8s.io/api/apps/v1:go_default_library", </ins> "//staging/src/k8s.io/api/apps/v1beta1:go_default_library", "//staging/src/k8s.io/api/core/v1:go_default_library", "//staging/src/k8s.io/api/extensions/v1beta1:go_default_library",

Please write a test that: Starts kube-apiserver Adds a webhook which blocks everything it is possible to block. Re-starts kube-apiserver Verifies that kube-apiserver comes back up Verifies that it can remove the broken webhook. This would have caught
cc
Hi I'm new to kubernetes but eager to contribute. I think this will be a good first issue for me but I'll need some guide on how to proceed I will need the first two points explained and also which file I'm supposed to be making my changes to.
Here is a good example of how to start a realistic kube-apiserver in integration tests:
"Verifies that it can remove the broken webhook." sounds dangerous. The apiserver should come up and it should be possible to remove a webhook via the API. But removing a potentially broken (or intentionally DoS'ed) webhook asks for security attacks.
I interpreted "Verifies that it can remove the broken webhook" as just ensuring that no webhook can be registered which could potentially deny its own (or another webhook in the same namespace's) deletion. Broken webhooks definitely shouldn't be treated differently in terms of determining who can delete them, but validation should probably prevent webhooks from applying to creates, updates and deletes of webhook objects, wildcard selectors should exclude webhook objects, etc. Also, this behavior would need to be documented. I think the safest way to ensure that a webhook does not get deleted is with RBAC rules, rather than using a validating webhook to prevent its deletion. I can't think of a legitimate use case for webhooks which affect the admission of webhook objects, it seems like any workflow which relies on that could be implemented in another, easier to reason about way. thoughts?
I have the same understanding. Excluding webhooks from webhooks seems reasonable. But it's only one piece in the (restart when webhook broke) puzzle. We reconcile RBAC rules, create namespaces (or keep them when they exist already). So I expect some more exceptions.
Is this still something you are working on?
If you can make a webhook which prevents the apiserver from starting at all, this is a problem, no matter what the webhook is verifying/doing, because you can't recover from that state.

I'm not suggesting that the system should automatically remove a broken webhook, which would be a security problem, rather that it needs to be possible for an appropriately authenticated sysadmin to do so. Especially since making a broken webhook non-broken may require executing API calls that the broken webhook is preventing.
/assign
as you assigned this to you, can you sketch what the plan is for this?
The plan was to use an integration test, (i.e. ) which I implemented in but it turns out that since the integration test apiserver does not follow exactly the same startup chain as the real one, that it is insufficient to ensure the apiserver can actually come up with a broken webhook installed. The integration test doesn't even fail against the code before merged. So currently the plan is to do it as an e2e test, probably with the [Slow] and [Destructive] tags.
it should be 99.9% the same. What are you missing? I think an integration test would be much better than an e2e test. We have much more control there.
I think the bug fixed was something a controller (bootstrap controller?) was doing, not the apiserver
those bootstrap controller are also executed by the testserver. So I would expect that the bug could be hit in testserver integration tests too. Compare with the real start code and for the test server start code. They are the same. The only difference is that we tweak the options beforehand a bit to run with a random port and random etcd prefix.
Hmm, then I think the reason that the test doesn't fail pre- is because the post start hook changed by that PR isn't called unless api-aggregation is being used. I guess the integration test for this will have to do that as well. Although the one in is still somewhat valuable to make sure new startup namespace creations in non aggregated apiservers are done correctly.
- you probably already thought about it, but one more thing to ensure here is that: apiserver can starte responding "OK" to "/healthz" even if all webhooks are broken [that means e.g. finishing all post-startup hooks] Without that, apiserver would be consistently being killed + restarted by Kubelet on which it is running (at least in some setups).

Yeah, the post-start hooks are the thing that inspired this issue :) Wojciech Tyczynski <:
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Reopened this issue. <detailsIn response to : /reopen : — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub , or mute the thread </details
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available .

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Reopened this issue. <detailsIn response to : /reopen /remove-lifecycle rotten Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: FieldManager     string CreateAnnotation bool Namespace        string <ins> EnforceNamespace bool </ins> Client         corev1client.CoreV1Interface DryRunStrategy cmdutil.DryRunStrategy

I'm writing a small bash script which generates some k8s resources in a namespace (other than ). When using to generate a service, the yaml which is generated does not include the defined namespace. I've also tried to move around the argument to the beginning (i.e. , but that doesn't work as well. Generated yaml should include the , when it's defined as an option. Kubernetes version (use ): Cloud provider or hardware configuration: n/a (not really important) OS (e.g: ): Kernel (e.g. ): Install tools: kubectl Network plugin and version (if this is a network-related bug): n/a Others: /sig cli
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cli
/assign


Question: // - replaces old recommendation with the newest recommendation, // - returns {max,min} of recommendations that are not older than constraints.Scale{Up,Down}.DelaySeconds func (a *HorizontalController) stabilizeRecommendationWithBehaviors(args NormalizationArg) (int32, string, string) { <del> recommendation := args.DesiredReplicas </del> <ins> now := time.Now() </ins> foundOldSample := false oldSampleIndex := 0 <del> var scaleDelaySeconds int32 var reason, message string var betterRecommendation func(int32, int32) int32 </del> <ins> upRecommendation := args.DesiredReplicas upDelaySeconds := *args.ScaleUpBehavior.StabilizationWindowSeconds upCutoff := now.Add(-time.Second * time.Duration(upDelaySeconds)) </ins> <del> if args.DesiredReplicas >= args.CurrentReplicas { scaleDelaySeconds = *args.ScaleUpBehavior.StabilizationWindowSeconds betterRecommendation = min reason = "ScaleUpStabilized" message = "recent recommendations were lower than current one, applying the lowest recent recommendation" } else { scaleDelaySeconds = *args.ScaleDownBehavior.StabilizationWindowSeconds betterRecommendation = max reason = "ScaleDownStabilized" message = "recent recommendations were higher than current one, applying the highest recent recommendation" } maxDelaySeconds := max(*args.ScaleUpBehavior.StabilizationWindowSeconds, *args.ScaleDownBehavior.StabilizationWindowSeconds) obsoleteCutoff := time.Now().Add(-time.Second * time.Duration(maxDelaySeconds)) </del> <ins> downRecommendation := args.DesiredReplicas downDelaySeconds := *args.ScaleDownBehavior.StabilizationWindowSeconds downCutoff := now.Add(-time.Second * time.Duration(downDelaySeconds)) </ins> <del> cutoff := time.Now().Add(-time.Second * time.Duration(scaleDelaySeconds)) </del> <ins> // Calculate the upper and lower stabilization limits. </ins> for i, rec := range a.recommendations[args.Key] { <del> if rec.timestamp.After(cutoff) { recommendation = betterRecommendation(rec.recommendation, recommendation) </del> <ins> if rec.timestamp.After(upCutoff) { upRecommendation = min(rec.recommendation, upRecommendation) } if rec.timestamp.After(downCutoff) { downRecommendation = max(rec.recommendation, downRecommendation) </ins> } <del> if rec.timestamp.Before(obsoleteCutoff) { </del> <ins> if rec.timestamp.Before(upCutoff) && rec.timestamp.Before(downCutoff) { </ins> foundOldSample = true oldSampleIndex = i } } <ins> // Bring the recommendation to within the upper and lower limits (stabilize). recommendation := args.CurrentReplicas if recommendation < upRecommendation { recommendation = upRecommendation } if recommendation > downRecommendation { recommendation = downRecommendation } // Record the unstabilized recommendation. </ins> if foundOldSample { a.recommendations[args.Key][oldSampleIndex] = timestampedRecommendation{args.DesiredReplicas, time.Now()} } else { a.recommendations[args.Key] = append(a.recommendations[args.Key], timestampedRecommendation{args.DesiredReplicas, time.Now()}) } <ins> // Determine a human-friendly message. var reason, message string if args.DesiredReplicas >= args.CurrentReplicas { reason = "ScaleUpStabilized" message = "recent recommendations were lower than current one, applying the lowest recent recommendation" } else { reason = "ScaleDownStabilized" message = "recent recommendations were higher than current one, applying the highest recent recommendation" } </ins> return recommendation, reason, message }

When configuring new behaviors section in HPA I two scenarios that I think are incorrect. First one: stabilizationWindowSeconds set up to 5 minutes for both scale up and scale down min replica count 2 Application consumes big amount of resources during warm-up 0 minute: current replica count: 2 1 minute: desired replica count: 3 - no scale up stabilization window worked 2 minute: desired replica count: 1 (!) - scale up to 3 (max number from previous recommendations during stabilization window period) ! Second one Stabilization window for scale up 0 Stabilization window for scale down 5 minutes min replica count 2 behavior set to scale up new pod in 2 minutes scaleUp: policies: type: Pods value: 1 periodSeconds: 120 What happens: 0 minute: 2 replicas 1 minute : performance spike scale up to 3 (desired value 4) 2 minute : no spike no scale up 3 minute: no spike scale up to 4(!) - supposedly desired value 1 and supposedly worked this way because of scale down window set to 5 minutes picking max value from that 5 minutes window that was 4 replicas Expected behaviors 1 Scenario no scale up for 5 minutes 2 Scenario correct measurement and no scale up when there is no need for it Environment: Kubernetes version (use ): client 1.19.3 Server 1.19.0 Cloud provider or hardware configuration: Azure aks
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig autoscaling
/assign josephburnett
The HPA controller keeps a flat history of recommendations for stabilization. ! When downscale stabilizing the maximum recommendation is selected. Purple is pre-stabilized recommendation. Green is final recommendation. Blue is max recommendation within window. ! When upscale stabilization the minimum recommendation is selected. ! However when both up and down scale stabilization is configured, the interpretation of the history changes depending on the direction of movement.

Red is the undesired, actuated recommendation. ! What we want is to keep the stabilized recommendation within the envelope of the minimum and maximum over configured stabilization windows. We should only move when the envelope forces a move. ! We first stabilize up and down, and check if the current value fits within it. If it does (or the range is "inverted"), we do nothing. If it does not, we emit the "nearest possible" value, which will be lower or upper bound, depending on where the current size is in relation to the range. (thanks and for the paragraph above)


Question: source "${KUBE_ROOT}/test/cmd/core.sh" source "${KUBE_ROOT}/test/cmd/crd.sh" source "${KUBE_ROOT}/test/cmd/create.sh" <ins> source "${KUBE_ROOT}/test/cmd/debug.sh" </ins> source "${KUBE_ROOT}/test/cmd/delete.sh" source "${KUBE_ROOT}/test/cmd/diff.sh" source "${KUBE_ROOT}/test/cmd/discovery.sh"

What would you like to be : Add integration tests for Why is this needed: This is part of the kubernetes/enhancements /sig cli /priority important-soon
/assign
/triage accepted


Question: import ( "context" <ins> "fmt" </ins> "time" v1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" <ins> "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" "k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/runtime/schema" clientscheme "k8s.io/client-go/kubernetes/scheme" "k8s.io/client-go/util/retry" </ins> "k8s.io/kubernetes/test/e2e/framework" e2epod "k8s.io/kubernetes/test/e2e/framework/pod" imageutils "k8s.io/kubernetes/test/utils/image"

[x] test approval issue : [x] test pr : 18 May 2023 [x] two weeks soak end date : 01 June 2023 [x] test promotion pr : According to following APIsnoop query, there are still 2 PodEphemeralcontainers endpoints that are not tested for Conformance. Using a number of existing e2e test practices a new has been created to address these 2 endpoints. The e2e logs for this test are listed below. This query shows the following podephemeralcontainers endpoints are hit within a short period of running this e2e test. If a test with these calls gets merged, test coverage will go up by 2 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted cc
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: var devicePluginPod, dptemplate *v1.Pod ginkgo.BeforeEach(func() { <del> e2eskipper.Skipf("Device Plugin tests are currently broken and being investigated") </del> ginkgo.By("Wait for node to be ready") gomega.Eventually(func() bool { nodes, err := e2enode.TotalReady(f.ClientSet)

ci-kubernetes-node-kubelet-serial-containerd: ci-kubernetes-node-kubelet-serial: (This is just an example, most of the tests appear to be failing) /sig node /kind failing-test /milestone v1.23 since test is failing in a release-blocking job /priority critical-urgent /assign /cc I believe this stems from
/triage accepted These failures look weird - I don't understand why dockershim is sometimes failing after restarting the kubelet. I guess we should double check that timings match the old GPU test (I think I might've tightened them a bit, bc they were passing when I ran the device suite on the same nodes). What's weirder is that they passed on my PR and all of a sudden broke after merging
Discussed at today - the easiest solution now that 1.23 has been branched is to punt these tests out of the serial lane so we can get it green again for 1.23, since any changes at this point must be backported and we would prefer that this is not release-blocking.
/milestone clear should address this; we won't get a fresh serial run for another ~6 hours so let's confirm next business day that this is temporarily fixed.
/remove-priority critical-urgent /priority important-longterm
can we keep it in the milestone until we get a verified green run?
CI is green today after skipping DevicePluginProbe tests .
/assign Maybe related to
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
Looks like serial has been stable for a while, closing: /close
Closing this issue. <detailsIn response to : Looks like serial has been stable for a while, closing: Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen it is referred to here: Tests are skipped today
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
did you have a chance to look at this? I remember you were close to understanding on what is going wrong here
oh welp, missed that
I was (past tense) but now I need to go back and look at my notes. If someone else wants to take over, be my guest, I've been quite busy with other tasks lately. Otherwise I'll get back to this!
So! I'm able to resume working in this area. So I'm testing last k/k master branch tip and running with : and the tests seems to be running and passing quite reliably! (5 times in a row seems like a promising start :) ) To move forward I suggest to: a explicit s in . This will make more evident we have a pending issue in our camp, that's the most egregious exception we have and it's confusing the skip in the config: - this will enable us to do focused work in this area a PR to remove the skip and iterate over it running CI until it's good enough I can take care of all the above. If CI is still failing, we need to explore possible missing/wrong configuration, but then other e2enode tests should be failing, not just this one.
I agree, this plan sounds good. Thank you!

xref:
started to fail also locally - we have baremetal lanes in RH - in the project CI they skip the relevant test. Investigating a possible common root cause
after a very long troubleshooting and quite few attempts, is ready for review


Question: //TODO (yastij) : compute average predicate restrictiveness to export it as Prometheus metric if predicate, exist := predicateFuncs[predicateKey]; exist { if eCacheAvailable { <ins> // Lock ecache here to avoid a race condition against cache invalidation invoked // in event handlers. This race has existed despite locks in eCache implementation. ecache.Lock() </ins> // PredicateWithECache will return its cached predicate results. <del> fit, reasons, invalid = ecache.PredicateWithECache(pod.GetName(), info.Node().GetName(), predicateKey, equivCacheInfo.hash) </del> <ins> fit, reasons, invalid = ecache.PredicateWithECache(pod.GetName(), info.Node().GetName(), predicateKey, equivCacheInfo.hash, false) </ins> } if !eCacheAvailable || invalid {

Fix this TODO (raised by : Part of: One thoughts to fix this is, we can redesign the equivalence cache data structure to: bool instead of deleting the entry. certain version tag in the entry, update the version per WRITE op (Update and Invalidate). the version after eCache predicate, and check it before update eCache.
Another possible approach is: eCache before running a predicate. the predicate. the eCache for the predicate. eCache. Basically, in the existing code all predicates are run first and their results are collected and then at the end of the function the eCache is updated. Instead of that, we can move the eCache update for each predicate inside the for loop and next to each predicate execution and synchronize before start of the execution of each predicate. The downside to my suggested approach is that performance may suffer due to frequent locking and unlocking.
Prefer to bsalamat@ 's option which is simpler :). Comparing to performance improvement by eCache, the lock is acceptable to me.
yup same here, going with the locking is easier and is acceptable. - also using a flag for invalidation might just move the problem from eCache to the flag ?
Oh I missed this notification somehow, busy on cleaning scheduler integration test. Yep so if locking is acceptable, it should be cleanest way to fix this, optimistic control would be a more complicated story.
- are you handling this one ?
Already been taken by volunteer days ago :)


Question: <ins> //go:build !windows // +build !windows /* Copyright 2024 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package logs import ( "os" ) func openFileShareDelete(path string) (*os.File, error) { // Noop. Only relevant for Windows. return os.Open(path) } </ins>

The test is failing on Windows [0]: The idea of the test is to check that kubelet still follows and reads the Container Logs after a Log Rotation occured. However, because the followed file is open, it cannot be renamed, and thus, the container logs cannot be rotated. This issue could be prevented if the file would be opened with the [0] [1] Kubelet should not block container log rotation. Run the test ` is failing on Windows. /sig windows /kind bug /kind failing-test latest, master. This unit test has been failing for a while. N/A Windows nodes </detailsN/A N/A N/A
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/kind failing-test /sig windows


Question: // The order is based on the restrictiveness & complexity of predicates. // Design doc: https://github.com/kubernetes/community/blob/master/contributors/design-proposals/scheduling/predicates-ordering.md var ( <del> predicatesOrdering = []string{CheckNodeConditionPred, </del> <ins> predicatesOrdering = []string{CheckNodeConditionPred, CheckNodeUnschedulablePred, </ins> GeneralPred, HostNamePred, PodFitsHostPortsPred, MatchNodeSelectorPred, PodFitsResourcesPred, NoDiskConflictPred, PodToleratesNodeTaintsPred, PodToleratesNodeNoExecuteTaintsPred, CheckNodeLabelPresencePred,

DaemonSet should not respect node's field. It's because cluster operators often want system components to be running and ready before workloads run on a given node. OTOH, DaemonSet does respect NoSchedule taints. This behavior is broken by , because DaemonSet share some predicates code with default scheduler.
/assign
Added this to 1.10 milestone, because it's a regression introduced in 1.10.
can you (or a SIG lead) add status/approved-for-milestone please? I'll add the priority. /priority important-soon
how is this use case satisfied by the daemonset controller creating pods and depending on the scheduler to assign them to nodes? having the scheduler ignore the unschedulable field seems nonsensical
I'd like get this as limitation for alpha feature; when upgrade to beta, I'd like to use taint to replace un-scheduable, so DaemonSet pod can tolerate it on-demand.
Escalating to critical-urgent, since we are now in Code Freeze. Please update daily with status, thanks! /priority critical-urgent /remove-priority important-soon
In short term, scheduler can check of pods and ignore bit of nodes for DS pods. Long term, when TaintNodesByCondition is stable, we should replace bit by taint and have DS controller add toleration for the taint when creating pods.
That doesn't seem proper. Making pods behave differently depending on the type of their creating object is very counterintuitive. That seems more reasonable. I'm less certain about auto- tolerations
- indeed there's some work currently to move from to a taint based behaviour, tracked by


Question: go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.53.0 go.opentelemetry.io/otel v1.28.0 go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.27.0 <ins> go.opentelemetry.io/otel/metric v1.28.0 </ins> go.opentelemetry.io/otel/sdk v1.28.0 go.opentelemetry.io/otel/trace v1.28.0 go.uber.org/zap v1.26.0

Since updating from Kubernetes 1.26. to 1.29. we experiences OOM-Kills on control plane nodes. After some investigation we found that adding to the fixed the issue. (Found this via pprof/heap of one apiserver: heap dump could be if needed) Before the update to 1.29.* the control planes were running with 4GB of memory with the same worklow in the cluster without issues. After the workarround using the feature gate the memory consumption of the whole control plane is not at ~2GB. In case there is an issue with other components, memory consumption of the apiserver should not be that excessive. We just had the K8s cluster running and saw a permant increase in memory usage of the control plane nodes. And after some time (can not give a specific duration) the OS of the control plane node started to OOM kill processes. Let us know if you need additional information. <details</detailson-premise; control-planes are running with 4 CPU cores, 8GB of memory and and NVME discs <details</details<details</details<details</detailsNo response
/sig api-machinery
Please provide the pprof, and what is the size of your cluster.
As we were not sure if sharing the pprof is secure, we decided to only share screenshots first: ! ! From this information we found the this which pointed us to the mentioned feature flag. We sometimes see etcd connection timeouts in the apiserver logs, maybe this could be related but should not cause this behaviour. 5 control planes ~100 worker nodes ~1700 Pods
Can you provide the pprof without , so could compare the two?
Sure, the current pprof with set: ! !
cc
I believe this is Sounds like we may need to backport some of the dependency bumps
/assign
/sig instrumentation
/triage accepted (as it's now assigned to
/triage accepted
We need to apply the workaround in It should be simple, but needs to be backported to 1.27
all versions starting 1.27 right ?
Thanks for taking care of this so quickly
yes. All versions starting 1.27.
Actually, this only needs to be backported to v1.29.

At 1.28, we were using v0.35.0 of otelgrpc, which did not include metrics at the time.
Cherry-picks:


Question: } } } <ins> func Test_completedConfig_NewBootstrapController(t *testing.T) { _, ipv4cidr, err := netutils.ParseCIDRSloppy("192.168.0.0/24") if err != nil { t.Fatalf("Unexpected error %v", err) } _, ipv6cidr, err := netutils.ParseCIDRSloppy("2001:db8::/112") if err != nil { t.Fatalf("Unexpected error %v", err) } ipv4address := netutils.ParseIPSloppy("192.168.1.1") ipv6address := netutils.ParseIPSloppy("2001:db8::1") type args struct { legacyRESTStorage corerest.LegacyRESTStorage serviceClient     corev1client.ServicesGetter nsClient          corev1client.NamespacesGetter eventClient       corev1client.EventsGetter readyzClient      rest.Interface } tests := []struct { name        string config      genericapiserver.Config extraConfig *ExtraConfig args        args wantErr     bool }{ { name: "master endpoint reconciler - IPv4 families", extraConfig: &ExtraConfig{ EndpointReconcilerType: reconcilers.MasterCountReconcilerType, ServiceIPRange:         *ipv4cidr, }, config: genericapiserver.Config{ PublicAddress: ipv4address, SecureServing: &genericapiserver.SecureServingInfo{Listener: fakeLocalhost443Listener{}}, }, wantErr: false, }, { name: "master endpoint reconciler - IPv6 families", extraConfig: &ExtraConfig{ EndpointReconcilerType: reconcilers.MasterCountReconcilerType, ServiceIPRange:         *ipv6cidr, }, config: genericapiserver.Config{ PublicAddress: ipv6address, SecureServing: &genericapiserver.SecureServingInfo{Listener: fakeLocalhost443Listener{}}, }, wantErr: false, }, { name: "master endpoint reconciler - wrong IP families", extraConfig: &ExtraConfig{ EndpointReconcilerType: reconcilers.MasterCountReconcilerType, ServiceIPRange:         *ipv4cidr, }, config: genericapiserver.Config{ PublicAddress: ipv6address, SecureServing: &genericapiserver.SecureServingInfo{Listener: fakeLocalhost443Listener{}}, }, wantErr: true, }, { name: "master endpoint reconciler - wrong IP families", extraConfig: &ExtraConfig{ EndpointReconcilerType: reconcilers.MasterCountReconcilerType, ServiceIPRange:         *ipv6cidr, }, config: genericapiserver.Config{ PublicAddress: ipv4address, SecureServing: &genericapiserver.SecureServingInfo{Listener: fakeLocalhost443Listener{}}, }, wantErr: true, }, { name: "lease endpoint reconciler - IPv4 families", extraConfig: &ExtraConfig{ EndpointReconcilerType: reconcilers.LeaseEndpointReconcilerType, ServiceIPRange:         *ipv4cidr, }, config: genericapiserver.Config{ PublicAddress: ipv4address, SecureServing: &genericapiserver.SecureServingInfo{Listener: fakeLocalhost443Listener{}}, }, wantErr: false, }, { name: "lease endpoint reconciler - IPv6 families", extraConfig: &ExtraConfig{ EndpointReconcilerType: reconcilers.LeaseEndpointReconcilerType, ServiceIPRange:         *ipv6cidr, }, config: genericapiserver.Config{ PublicAddress: ipv6address, SecureServing: &genericapiserver.SecureServingInfo{Listener: fakeLocalhost443Listener{}}, }, wantErr: false, }, { name: "lease endpoint reconciler - wrong IP families", extraConfig: &ExtraConfig{ EndpointReconcilerType: reconcilers.LeaseEndpointReconcilerType, ServiceIPRange:         *ipv4cidr, }, config: genericapiserver.Config{ PublicAddress: ipv6address, SecureServing: &genericapiserver.SecureServingInfo{Listener: fakeLocalhost443Listener{}}, }, wantErr: true, }, { name: "lease endpoint reconciler - wrong IP families", extraConfig: &ExtraConfig{ EndpointReconcilerType: reconcilers.LeaseEndpointReconcilerType, ServiceIPRange:         *ipv6cidr, }, config: genericapiserver.Config{ PublicAddress: ipv4address, SecureServing: &genericapiserver.SecureServingInfo{Listener: fakeLocalhost443Listener{}}, }, wantErr: true, }, { name: "none endpoint reconciler - wrong IP families", extraConfig: &ExtraConfig{ EndpointReconcilerType: reconcilers.NoneEndpointReconcilerType, ServiceIPRange:         *ipv4cidr, }, config: genericapiserver.Config{ PublicAddress: ipv6address, SecureServing: &genericapiserver.SecureServingInfo{Listener: fakeLocalhost443Listener{}}, }, wantErr: false, }, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { c := &completedConfig{ GenericConfig: tt.config.Complete(nil), ExtraConfig:   tt.extraConfig, } _, err := c.NewBootstrapController(tt.args.legacyRESTStorage, tt.args.serviceClient, tt.args.nsClient, tt.args.eventClient, tt.args.readyzClient) if (err != nil) != tt.wantErr { t.Errorf("completedConfig.NewBootstrapController() error = %v, wantErr %v", err, tt.wantErr) return } }) } } </ins>

Clusters created with kubeadm using DualStack are not able to work, because CoreDNS pods never get ready Seeing one failed job logs It seems that the pod is "waiting for the kubernetes service"?? coredns version seems the same as in the working jobs CoreDNS pod should be ready Instal a Dual-stack cluster using master with kubeadm (or kind ) ! It started to fail on Oct12 <details</details<details</details<details</details<details</details<details</details<details</details
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc I didn't notice it was failing for more than 1 month :facewithhead_bandage: I didn't have time to dig into it yet, but this PR landed just that day how is that the pods running has the version 1.8.4 but the PR seems to update kubeadm to 1.8.6?
Looking into this today. The error from CoreDNS should only happen when the kubernetes plugin fails to sync with the Kubernetes API, either because it can’t connect, or it can’t sync the lists of services/endpointslices. There were changes and fixes to the API connection code, so it could be a regression.
Actually no. The changes I was thinking of were already in 1.8.4.
How can we reproduce it : Instal a Dual-stack cluster using master with kubeadm (or kind ) I was unable to reproduce this issue in a kind dual stack cluster.

After upgrading coredns to 1.8.6 (from 1.8.0 installed by kind), and adding list/watch permission for endpointslice, the pods were ready. edit: It also works for coredns 1.8.4 - I'm unclear as to which version I should be checking.
let me check if is a regression in kind :thinking:
working with 1.8.6 and yesterdays kubernetes podIPs
This is weird, I hit a similar issue in openshift, endpoints doesn't match the IP family of the kubernetes.default service apiserver service is single stack, at least until this KEP fixes it What I don't understand is why the order of the IPs in kind is different let me try to fix both things /assign
my fault, sorry for the noise :sweat_smile:


Question: switch { case t.ExitCode == 1: // expected <del> case t.ExitCode == 128 && reBug88766.MatchString(t.Message): </del> <ins> case t.ExitCode == 128 && t.Reason == "ContainerCannotRun" && reBug88766.MatchString(t.Message): </ins> // pod volume teardown races with container start in CRI, which reports a failure framework.Logf("pod %s on node %s failed with the symptoms of https://github.com/kubernetes/kubernetes/issues/88766") default:

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: pull-kubernetes-e2e-gce Which test(s) are flaking: Testgrid link: Reason for failure: Anything else we need to know: Triage board shows flakes here spiked dramatically on 3/4 /sig node /priority critical-urgent /milestone v1.18
are you able to assist if this is related to recent topology work?
sure, I'll take a look
is this the test you just in ?
As mentioned above, this is a new test that was just , and the test sometimes fails. I don't see that any of the recent topology manager changes would cause these failures. I'll be happy to help out and dig more if or others think otherwise.
This is me, the bugs are untriaged containerd / Kubelet failures with low occurrence rates that this test now exposes. Investigating.
still flaking in non-dockershim environments: pod pod-submit-status-0-14 on node kind-worker2 container unexpected exit code 128: start=1970-01-01 00:00:00 +0000 UTC end=2020-03-06 16:16:54 +0000 UTC reason=StartError message=failed to create containerd task: OCI runtime create failed: starting container process caused " container init caused " mounting "/var/lib/kubelet/pods/-819e-4840-8369-" to rootfs "" at "" caused "stat /var/lib/kubelet/pods/-819e-4840-8369- no such file or directory""": unknown `
Different reason this time. Will continue to enumerate failure modes.


Question: "type": "array", "items": { "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.Initializer" <del> } </del> <ins> }, "x-kubernetes-patch-merge-key": "name", "x-kubernetes-patch-strategy": "merge" </ins> }, "result": { "description": "If result is set with the Failure field, the object will be persisted to storage and then deleted, ensuring that other clients can observe the deletion.",

There are two issues: doesn't have merge key The validation code regards it as an error if both Initializers.Result and Initializers.Pending are nil. So if multiple initializer controllers sending patches removing themselves from Initializers.Pending at the same time, the last patch that gets applied will get a validation error.
/sig api-machinery


Question: promoterE2eRegistry     = registry.PromoterE2eRegistry buildImageRegistry      = registry.BuildImageRegistry gcAuthenticatedRegistry = registry.GcAuthenticatedRegistry <ins> gcEtcdRegistry          = registry.GcRegistry </ins> gcRegistry              = registry.GcRegistry sigStorageRegistry      = registry.SigStorageRegistry gcrReleaseRegistry      = registry.GcrReleaseRegistry

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: Which test(s) are failing: Since when has it been failing: 02/19 20:21 PST Testgrid link: Reason for failure: Image pull failure Anything else we need to know:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig windows cc


Question: // APILifecycleDeprecated is an autogenerated function, returning the release in which the API struct was or will be deprecated as int versions of major and minor for comparison. // It is controlled by "k8s:prerelease-lifecycle-gen:deprecated" tags in types.go or  "k8s:prerelease-lifecycle-gen:introduced" plus three minor. func (in *CustomResourceDefinition) APILifecycleDeprecated() (major, minor int) { <del> return 1, 19 </del> <ins> return 1, 16 </ins> } // APILifecycleReplacement is an autogenerated function, returning the group, version, and kind that should be used instead of this deprecated type.

The following code snippet is very confusing: Is v1beta1 deprecated in 1.16 or 1.19? Is it to be removed in 1.19 or 1.22? Please clarify the exact versions so the many users of CRD v1beta1 APIs know what to do.
/sig api-machinery
/kind deprecation
/unkind feature
Deprecated in 1.16. Planned to be removed in 1.22. We'll update the docs to be consistent before 1.19 is released.
/assign


Question: func makeDeviceMountPath(plugin *csiPlugin, spec *volume.Spec) (string, error) { if spec == nil { <del> return "", errors.New(log("makeDeviceMountPath failed, spec is nil")) </del> <ins> return "", errors.New("makeDeviceMountPath failed, spec is nil") </ins> } <del> pvName := spec.PersistentVolume.Name if pvName == "" { return "", errors.New(log("makeDeviceMountPath failed, pv name empty")) </del> <ins> driver, err := GetCSIDriverName(spec) if err != nil { return "", err } if driver == "" { return "", errors.New("makeDeviceMountPath failed, csi source driver name is empty") } csiSource, err := getPVSourceFromSpec(spec) if err != nil { return "", errors.New(log("makeDeviceMountPath failed to get CSIPersistentVolumeSource: %v", err)) } if csiSource.VolumeHandle == "" { return "", errors.New("makeDeviceMountPath failed, CSIPersistentVolumeSource volume handle is empty") </ins> } <del> return filepath.Join(plugin.host.GetPluginDir(plugin.GetPluginName()), persistentVolumeInGlobalPath, pvName, globalMountInGlobalPath), nil </del> <ins> result := sha256.Sum256([]byte(fmt.Sprintf("%s", csiSource.VolumeHandle))) volSha := fmt.Sprintf("%x", result) return filepath.Join(plugin.host.GetPluginDir(plugin.GetPluginName()), driver, volSha, globalMountInGlobalPath), nil </ins> } func getDriverAndVolNameFromDeviceMountPath(k8s kubernetes.Interface, deviceMountPath string) (string, string, error) {

When two CSI PVs point to the same volume in the storage backend, kubelet can't start pods that use these two PVs on the same node. Both PVs have the same VolumeHandle and kubelet is smart enough to recognize that they're the same volume (because they have the same unique volume ID). It calls only once, however, the staging path contains name of one of the PVs, say PV1: One of the CSI volume calls then fails, because it uses PV2 to compute the staging path and that path does not exist. Both pods should just start. Using CSI HostPath volume, PV1 (dynamically provisioned for simplicity): PV2 (basically edited PV1): PVC1, PVC2, Pod1 and Pod2 are trivial, nothing special there. <detailsToday's master, </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage /priority backlog
I suggest that if two volumes have the same unique volume ID (computed from CSI ), they must have also the same global mount path. Kubelet already has code to call NodeStage once and bind-mount it to different pods, regardless how many different PVs they use. These are several issues changing the global mount path: uses PV parsed from the global path to get the PV and to get enough info to call NodeUnstage: However, this parsing is used only when the volume plugin is not able to find/parse its json it has saved during MountDevice: Can we safely assume that the json is always there? the global path in a running cluster is tricky. We can require nodes are drained when kubelet with new global mount path is installed. This way, no CSI volume is mounted to a global path when the new kubelet starts. Minor cluster upgrades (1.23 -1.24) are , also documented in . Or we could fix kubelet to use the old global path when it already exists. This looks fragile to me.

staging dir needs to be unique enough not to conflict with other CSI volumes. It should be based on the driver name + volume handle, but that could be quite long. Can we compute a hash and hope it won't conflict?
I think in-tree driver can handle this case correctly because global mountpath is using unique volume id instead of pv name. With csi-migration, does it break backward compatibility?
We require users to drain nodes before enabling migration or when updating to a Kubernetes version that have migration enabled by default. All volumes should be therefore unmounted.
probably yes this breaks backward compatibility, but since only migration for PD is being done maybe it's more rare? We saw a customer case for this situation with NFS shares using the csi filestore driver. For PD I think it would only come up when trying to ROX a PD volume across namespaces, I'm not sure if that's common.
Capturing some thoughts here in line to understand your we use a staging path like , the kubelet can locate the both the driver name and the volumeId, needed during UnmountDevice call. e.g for pd csi driver which is the sha256{projects/mattcary-saikatroyc-test/zones/us-central1-c/disks/pvc--f97d-43cd-a5f1-}. This means the volume Id needs to persisted (we already do that in today ), so that the volume Id can be retrieved for UnmountDevice call. we decide to make the change at a minor boundary, then a node drain mandate would unstage any volumes before the kubelet is upgraded. this ensures any staged path for csi volume is using the new format 2 If we do not want to enforce a node drain, kubelet can first lookup json data (the reasoning is if this file does not exist, we will not be able to retrieve the volume handle from the new handle anyway), else fallback to try the old format of staging path - . TBD: Need to check mount reference logic. sha(volumeId) is expected to be unique in the cluster, if I understand. (based on the assumption any storage resource provisioned by a given driver is expected to be uniquely identifiable)
/assign
yes, your summary is correct. We persist a json, please make sure it contains anything that NodeUnstage may need.

It seems to me that people prefer option 2.2. We already do to uniquely identify VolumeAttachment. So far nobody complained about any conflict.


Question: t.Errorf("Expected %v, but got %v", e, a) } <ins> // Test a name + port number (service port 9393 -> target port "p" -> endpoint port 93, using second ip) location, _, err = redirector.ResourceLocation(ctx, "foo-second-ip:9393") if err != nil { t.Errorf("Unexpected error: %v", err) } if location == nil { t.Errorf("Unexpected nil: %v", location) } if e, a := "//[2001:db7::]:93", location.String(); e != a { t.Errorf("Expected %v, but got %v", e, a) } </ins> // Test a scheme + name + port. location, _, err = redirector.ResourceLocation(ctx, "https:foo:p") if err != nil {

Hi, I have a issue with a dualstack cluster for my Kubernetes Dashboard deployment. I opened a ticket here (), but I am informed that it is not related to Kubernetes Dashboard. What is the problem? Errors in kube-api: The service is created with the correct ipFamily (=IPv6). My endpoint is fully in IPv6. Indeed, my pod has one IP in IPv4 and another in IPv6. Do you have also the issue? Kubernetes version: 1.18 OS: RHEL7
/sig network /sig api-machinery
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
I think the following should fix the error you saw:
/remove-sig api-machinery


Question: } } <ins> func TestCreateFirewallFails(t *testing.T) { name := "loadbalancer" region := "us-central1" desc := "description" gce := &GCECloud{} if err := gce.createFirewall(name, region, desc, nil, nil, nil); err == nil { t.Errorf("error expected when creating firewall without any tags found") } } </ins> func TestRestrictTargetPool(t *testing.T) { const maxInstances = 5 tests := []struct {

In some custom setup cases the prefix-matching logic for GCE load-balancers can break and end up creating a firewall rule with no tags. We should instead refuse to create the rule. In fact, if we find it, we should probably delete the rule, but that's maybe too aggressive. Looking at the logic, if we return error in the tag-selection, we'll abort the whole load-balancer creation, which is unfortunate, but not TERRIBLE. We could just silently not create the firewall but not return error, which will just retry over and over.
/cc
Are you active on this? may have some starter-project bandwidth to help.
Not active. It would indeed be a good starter project :)