Question: utilfeature "k8s.io/apiserver/pkg/util/feature" featuregatetesting "k8s.io/component-base/featuregate/testing" "k8s.io/kubernetes/pkg/features" <ins> st "k8s.io/kubernetes/pkg/scheduler/testing" </ins> ) func TestNewResource(t *testing.T) {

We have a pod fake function in , but this is not a base util anymore, we have a lot of arguments, we can replace this with here /sig scheduling /kind cleanup /help This is an easy one to get start. Cleaner code for maintenance.
This request has been marked as needing help from a contributor. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : ### What would you like to be ? /kind cleanup /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-kind feature
I want to try this one, the problem we solve here is to replace the *v1.Pod with type PodWrapper {}? /assign
Replace with in
you mean replace ` in I check the code and didn't found out that we have makeBasePod in
I have tried it. But there will be error about cycle import, because package import package, where locate.
yes, I've tried it as well. guess I need to wait for kerthcet's response <img width="805" alt="image" src=""
Sorry, forgot to reply on this, I introduced another PR to help solve this import cycle, but waits for other maintainers' response. So generally, if that pr accepted, we can work on this one then. Thanks for the volunteer.
the import cycle is resolved, you can start your work now, thanks!
thanks for the ping!


Question: metrics.SetShowHidden() } <ins> hostname, err := utilnode.GetHostname(config.HostnameOverride) if err != nil { return nil, err } client, eventClient, err := createClients(config.ClientConnection, master) if err != nil { return nil, err } nodeIP := detectNodeIP(client, hostname, config.BindAddress) protocol := utiliptables.ProtocolIPv4 if utilsnet.IsIPv6(nodeIP) { klog.V(0).Infof("kube-proxy node IP is an IPv6 address (%s), assume IPv6 operation", nodeIP.String()) protocol = utiliptables.ProtocolIPv6 } else { klog.V(0).Infof("kube-proxy node IP is an IPv4 address (%s), assume IPv4 operation", nodeIP.String()) } iptInterface = utiliptables.New(execer, protocol) </ins> // Create event recorder eventBroadcaster := record.NewBroadcaster() recorder := eventBroadcaster.NewRecorder(proxyconfigscheme.Scheme, v1.EventSource{Component: "kube-proxy", Host: hostname})

What happened: Currently kube-proxy cleanup not work, which will left iptables (and ipvs) rules in host and. What you expected to happen: Kube-proxy cleanup can work and not relay on hostName or ClientConnection config. How to reproduce it (as minimally and precisely as possible):
/sig network /assign
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.


Question: v1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" <ins> "k8s.io/apimachinery/pkg/types" </ins> _ "k8s.io/kubernetes/pkg/apis/core/install" <ins> "k8s.io/utils/mount" </ins> ) func validateDirExists(dir string) error {

In , we check mount on first level directory in pod volume directories. It is true for many volume plugins, but not for CSI plugin which . cc
/assign
Pardon my ignorance of kubelet source code, but this looks very much similar to an issue we have been running to many times recently, when developing a CSI plugin. The problem is that when CSI volume publish is called and the mount point is not propagated from the CSI plugin pod (either missing bidirectional flag in yaml or not supported by underlaying k8s cluster implementation), the kubelet nevertheless starts a pod using the volume. The volume is not there, so the app writes to root filesystem in fact, leading to all other kinds of problems - i.e. losing data, filling up root fs. IMHO what kubelet should do is to check that the mounted filesystem is indeed visible after publish call and if not it should fail the volume attach request. If this ticket was fixed, would it solve the issue we are seeing or should I open a new ticket? thanks
I think we can potentially add that check but we need to make sure first it can work with ephemeral CSI drivers. Ephemeral drivers would have to "stage" their directories somewhere, like /var/lib/kubelet/plugins/<csi driver/, and then bind mount it to the publish path. cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Hmm... That is a good question. Does csi require a mount to be made or only that the specified directory be created? Will have to go read through the spec...
/remove-lifecycle stale
The only place in the spec I could find was this comment: // For volumes with an access type of mount, the SP SHALL place the // mounted directory at target_path. So I think having Kubelet enforce that check would be ok with the spec, and a good thing to do.
cc CSI spec says that the target path has to be a mount. Shall we go ahead and enforce it then? Are we going to break drivers that may not be doing this?

Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
The issue mentioned here involves two functions . Now with PR , the issue in CheckVolumeExistenceOperation is solved by checking mount path instead of volume path. The only issue here is for getMountedVolumePathListFromDisk which is used during pod deletion. This function was introduced by PR . Now I am questioning whether we need to check the mount point or just need to check pod dirs exist before deleting the pod. Firstly, for some plugins, there might not be mount point. Secondly, without checking mount point, it means that pod cannot be deleted as long as there are pod volume dirs. In normal case, volume dirs should be cleanedup during volume teardown. For cases that kubelet restarts, we have reconciler try to clean up volume mounts if possible. So we can propose a couple of solutions here mount point check. It solves the problem, the downside is that there might be some rare case that reconciler could not clean up dirs after kubelet restarts. check mount point, and also check the next level "/mount" dir. This might still cause problem for some plugins that do not have mounts. cc
Discussed offline with For proposal 1., pod will not be deleted from api server if there are still volume dirs exist. So to make sure it works well, we need to have reliable way to clean up dirs. Normally when kubelet restarts, reconciler will either add it into actual state (if reconstruction works) or try to clean it up directly once unless volume is still in desired state. So if reconstruction does not work, and it fails to clean up during reconciler sync, there is no other chance to clean up the dirs.

Or if volume was in desired state and removed quickly afterwords, the dir might not get the chance to be cleaned up at all. We are thinking to improve this part, we could use mark volume "uncertain" in actual state feature. During reconstruction, we can remove the check for mounted points (since the mount might be gone during node reboots or driver does not have moutns at all). Reconciler could mark those volumes as uncertain (even if it was in desired state) to make sure those volume got cleaned up or remount whenever needed with retries. With this change, I think it is safe to remove the mount point check for pod deletion too.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: ecache *EquivalenceCache, queue SchedulingQueue, alwaysCheckAllPredicates bool, <ins> equivCacheInfo *equivalenceClassInfo, </ins> ) (bool, []algorithm.PredicateFailureReason, error) { var ( <del> equivalenceHash  uint64 failedPredicates []algorithm.PredicateFailureReason </del> eCacheAvailable  bool <ins> failedPredicates []algorithm.PredicateFailureReason </ins> invalid          bool fit              bool reasons          []algorithm.PredicateFailureReason

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Today, . This seems unnecessary and should be done once for all the nodes. Is this a BUG REPORT or FEATURE REQUEST?: /kind enhancement /sig scheduling cc:
/assign Thanks, will send fix per slack discussion.
/cc


Question: # github.com/google/btree v1.0.0 => github.com/google/btree v1.0.0 github.com/google/btree # github.com/google/btree => github.com/google/btree v1.0.0 <del> # github.com/google/cadvisor v0.38.5 => github.com/google/cadvisor v0.38.5 </del> <ins> # github.com/google/cadvisor v0.38.6 => github.com/google/cadvisor v0.38.6 </ins> ## explicit <del> # github.com/google/cadvisor => github.com/google/cadvisor v0.38.5 </del> <ins> # github.com/google/cadvisor => github.com/google/cadvisor v0.38.6 </ins> github.com/google/cadvisor/accelerators github.com/google/cadvisor/cache/memory github.com/google/cadvisor/client/v2

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When I restart containerd.service, kubelet can not connect to it again and cause 100% cpu usage. there are many logs like this: I can show container info via crictl at the meantime and restart kubelet.service can resolve the problem. What you expected to happen: kubelet can reconnect to containerd after containerd restart. How to reproduce it (as minimally and precisely as possible): cpu usage and syslog. Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: armbian(debian on arm64) OS (e.g: ):Kernel (e.g. ):Install tools: kubeadm Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
/assign
Reproduce: can reproduce the issue by restarting when is the container runtime another setup where was used used as container runtime, didn't experience the issue by restarting , but restarted the underlying , same issue shown up again and had to restart to resolve the issue. Will continue with the investigation, might need to reconnect to after the socket is broken.
To me it seems there are a couple of problems here. 1. kubelet does not reconnect to the containerd socket. 2. The reconnect is agressive and consumes a lot if cpu. I don't know the cause for 1, but for 2 this might be an indication that the retry interval should be relaxed.
Spent a few hours on the investigation, it's likely to be some internal states being messed up in .
TL;DR: It should be related to gRPC.

Hacked the vendored gRPC mod and changed its to something like this: Then CPU only spike for a few seconds after restart then comes back to normal instead of failing the all the time. kubelet passes a customized dialer to the gRPC client. I logged the address during kubelet startup: As you can see, the address of the is at . Before restarts, I can see from log that every time, the gRPC function is using the customized dialer that kubelet passed: However, once got restarted, the address of the dialer is changed: And that function at keeps failing. As I forced to use the original customized dialer, the issue is gone. Without too much knowledge on gRPC, my blind guess is as is passed by reference to gRPC client, some component could modify it during the process lifetime and the change got persisted. Anyway, I'll check with gRPC team on this issue and see if there's already a fix on their side. If so then I can port the fix here.
/cc
Hi, are there any updates on this. I am experiencing what appears to be the same issue. If I restart containerd while the kubelet is running, my kubelet logs get flooded with: Restarting the kubelet appears to be the only way to re-establish the connection with containerd. Version information is:
Hi thanks for the info, I'm still working on the root cause. As I mentioned above, it's something related to gRPC. I collected the object info, this is the object before containerd restart: ! And this is the object after containerd restart: ! There's one thing made me concern is, the connection state is still 2 (i.e. , see ) after the failure. I'm not sure if this is related to the cause but anyway it still should be something related to gRPC. Btw I've followed their to use the latest gRPC client without customized dialer and still see this issue, will still work on the root cause.
We're seeing this now when we update our nodes to 1.19 where our Kata Containers install does a . Our current workaround is to also do , but would be good to get this fixed.
Is this MR relative to the problem?

No I don't think so, I've cherry-picked that change and tried in my environment weeks back, and the issue is still there. I only get around the issue by modifying the gRPC code as I mentioned in the above, I'll continue to check which component modified the function in after thanksgiving.
For the sake of priority. This hit us hard these days too, as security updates for container.d triggered a restart and a ton of nodes flooded the central log management. ;) Thanks for working hard on a fix.
Same here. Two systems were affected by automatic container.d upgrades & restarts. But now I wonder if - generally - "unattended-upgrades" (Ubuntu 18.04) is a wise thing to do? Or should the OS for kubernetes deployments always be updated manually & controlled? I can not find best practices for this in the documentation.
After the recent CVE-2020-15257many users will upgrade and restart containerd. It would be a big problem.
I see, will prioritize on the root cause and fix. At the mean time, you need to restart after you restart .
Had another round of debugging, and made one step towards the root cause. I following snipple to the as I mentioned in previous : When I reproduced the error, I got following information for the dialer function: That leads to cadvisor, which is the only component that uses the in the repo: So I'm convinced that this should be related to and will update the thread once I've made further progress.
EOD updates: issue is indeed caused by cadvisor, reproduced the issue in this . confusing timeout in log should be . root cause might be a bug for using context that already exceeded deadline in cadvisor. To validate that, I need to explore all methods of this : Will work on the fix on cadvisor for the rest of the week.
Fix was made in Copying from to ensure we can capture it on the issue here: I've cut new cAdvisor releases (v0.37.3 and v0.38.6) with this fix cherrypicked. We should get the cAdvisor fix back into kubernetes.

We'll need 3 PRs to k/k with following updates: [x] k/k master to update cAdvisor to v0.38.6 -- [x] k/k release-1.20 to update cAdvisor to v0.38.6 -- [x] k/k release-1.19 to update cAdvisor to v0.37.3 --
Thanks for fixing this one, definitely a tricky one to track down.
Thanks for all the patience and team efforts on getting this issue fixed, also credits to on the guides and discussions, won't be able to get it done without your helps!
The fix was applied on master as well as cherrypicked into and releases.
I remember you told me, but I forgot. Does issue exist in 1.18 and 1.17?
I don't believe 1.18 or 1.17 should be affected. I think this issue was introduced with which commit was included in cAdvisor v0.37 release. cAdvisor v0.37 was vendored into k8s 1.19.


Question: recursive := cmdutil.GetFlagBool(cmd, "recursive") apiV := cmdutil.GetFlagString(cmd, "api-version") <del> swagSchema, err := kubectl.GetSwaggerSchema(apiV, client) </del> <ins> mapper, _ := f.Object() group, inModel, fieldsPath, err := kubectl.SplitAndParseResourceRequest(args[0], mapper) </ins> if err != nil { return err } <del> mapper, _ := f.Object() inModel, fieldsPath, err := kubectl.SplitAndParseResourceRequest(args[0], mapper) </del> <ins> if len(group) == 0 { // TODO: We should deduce the group for a resource by discovering the supported resources at server. group, err = mapper.GroupForResource(inModel) if err != nil { return err } } if len(apiV) == 0 { groupMeta, err := latest.Group(group) if err != nil { return err } apiV = groupMeta.GroupVersion } swagSchema, err := kubectl.GetSwaggerSchema(apiV, client) </ins> if err != nil { return err }

It would be nice if let me explain experimental resources to aid users ability to experiment.
It's possible this works today, but it was non-obvious how to use. Attempts:
cc
cc
No, this isn't supported, since current implementation only use v1 Client but not experimental Client.
do we need to fix this for 1.1?
How difficult do you think it would be to fix?
It has dependency on ServerAPIVersions(), for which we plan to make a clean-up PR. I guess it needs 3-4 days to get two PRs merged.
With , this can be fixed now. What's the expected behavior of ? I assume in case of , it should search hpa in both v1 and extensions/v1bata1, and determines hpa as a resource in the extensions group? BTW, the use of "api-version" flag is problematic It adds extra meaning to the generic "api-version" flag.


Question: pod. Modify the labels of one of the Job's Pods. The Job MUST release the Pod. release: v1.16 file: test/e2e/apps/job.go <ins> - testname: Jobs, apply changes to status codename: '[sig-apps] Job should apply changes to a job status [Conformance]' description: Attempt to create a running Job which MUST succeed. Attempt to patch the Job status to include a new start time which MUST succeed. An annotation for the job that was patched MUST be found. Attempt to replace the job status with a new start time which MUST succeed. Attempt to read its status sub-resource which MUST succeed release: v1.24 file: test/e2e/apps/job.go </ins> - testname: Ensure Pods of an Indexed Job get a unique index. codename: '[sig-apps] Job should create pods for an Indexed job with completion indexes and specified hostname [Conformance]'

[x] Test PR : [x] Two weeks soak end date : 6 April 2022 [X] Test promotion PR : According to this APIsnoop query, there are still some remaining Job status endpoints which are untested. There are conformance tests for CronJob status endpoints already the existing as a template, a new for job test has been created. The e2e logs for this test are listed below. This query shows the endpoints hit within a short period of running the e2e test If a test with these calls gets merged, test coverage will go up by 3 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: users: - name: minikube user: <del> token: minikube-token` + "n"} test.run(t) </del> <ins> token: minikube-token` + "n", }, { description: "Testing for kubectl config view --minify=true --context=my-cluster", config:      conf, flags:       []string{"--minify=true", "--context=my-cluster"}, expected: `apiVersion: v1 clusters: - cluster: server: https://192.168.0.1:3434 name: my-cluster contexts: - context: cluster: my-cluster user: mu-cluster name: my-cluster current-context: my-cluster kind: Config preferences: {} users: - name: mu-cluster user: token: minikube-token` + "n", }, } for _, test := range testCases { cmdTest := viewClusterTest{ description: test.description, config:      test.config, flags:       test.flags, expected:    test.expected, } cmdTest.run(t) } </ins> } func (test viewClusterTest) run(t *testing.T) {

/kind bug What happened: Current context was set to cluster1. Ran and got back information about cluster1 instead. What you expected to happen: I expected the output to be the cluster2 information How to reproduce it (as minimally and precisely as possible): Have two contexts in your kube config, A and B Run to set A as current context Run Anything else we need to know?: I'd like to help contribute, so if this is an easy fix, someone point me in the right direction and I'll try to get it done. Environment: Kubernetes version (use ): 1.10.2 OS (e.g. from /etc/os-release): MacOS 10.11.6 Install tools: kubectl was installed via brew
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign Filed


Question: PROVIDER_VARS="${PROVIDER_VARS:-} ENABLE_IP_ALIASES" fi <ins> # Enable GCE Alpha features. if [[ -n "${GCE_ALPHA_FEATURES:-}" ]]; then PROVIDER_VARS="${PROVIDER_VARS:-} GCE_ALPHA_FEATURES" fi </ins> # If we included ResourceQuota, we should keep it at the end of the list to prevent incrementing quota usage prematurely. ADMISSION_CONTROL="${KUBE_ADMISSION_CONTROL:-Initializers,NamespaceLifecycle,LimitRanger,ServiceAccount,PersistentVolumeLabel,PodPreset,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,Priority,ResourceQuota}"

The environment variable GCEALPHAFEATURES is not set correctly in the test custer.
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/network: Issue will be escalated to these SIGs if needed. priority/critical-urgent: Never automatically move out of a release milestone; continually escalate to contributor and SIG through all available channels. kind/bug: Fixes a bug discovered during the current release. <detailsAdditional instructions available <a href=""here</a</details


Question: f.Namespace.Name), "failed to wait for pod %s to be running", windowsPodWithSELinux.Name) }) <del> ginkgo.It("should ignore ProcMount Specific SecurityContext if set", func(ctx context.Context) { ginkgo.By("Creating a pod with ProcMount options") // It is sufficient to show that the pod comes up here. Since we're stripping the SELinux and other linux // security contexts in apiserver and not updating the pod object in the apiserver, we cannot validate the // pod object to not have those security contexts. However the pod coming to running state is a sufficient // enough condition for us to validate since prior to https://github.com/kubernetes/kubernetes/pull/93475 // the pod would have failed to come up. windowsPodWithSELinux := createTestPod(f, imageutils.GetE2EImage(imageutils.Agnhost), windowsOS) windowsPodWithSELinux.Spec.Containers[0].Args = []string{"test-webserver-with-selinux"} windowsPodWithSELinux.Spec.SecurityContext = &v1.PodSecurityContext{} pmt := v1.UnmaskedProcMount containerUserName := "ContainerAdministrator" windowsPodWithSELinux.Spec.Containers[0].SecurityContext = &v1.SecurityContext{ ProcMount:      &pmt, WindowsOptions: &v1.WindowsSecurityContextOptions{RunAsUserName: &containerUserName}} windowsPodWithSELinux.Spec.Tolerations = []v1.Toleration{{Key: "os", Value: "Windows"}} windowsPodWithSELinux, err := f.ClientSet.CoreV1().Pods(f.Namespace.Name).Create(ctx, windowsPodWithSELinux, metav1.CreateOptions{}) framework.ExpectNoError(err) framework.Logf("Created pod %v", windowsPodWithSELinux) framework.ExpectNoError(e2epod.WaitForPodNameRunningInNamespace(ctx, f.ClientSet, windowsPodWithSELinux.Name, f.Namespace.Name), "failed to wait for pod %s to be running", windowsPodWithSELinux.Name) }) </del> ginkgo.It("should not be able to create pods with containers running as ContainerAdministrator when runAsNonRoot is true", func(ctx context.Context) { ginkgo.By("Creating a pod")

After merged we are seeing the new test in fail since it was turned on be default now (it passed in the PR linked because it didn't have the alpha feature enabled). <br class="Apple-interchange-newline"<!--EndFragment--</body /sig windows /sig node /cc
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
ptal


Question: "sync" "k8s.io/api/core/v1" <ins> "k8s.io/apimachinery/pkg/types" </ins> "k8s.io/apimachinery/pkg/util/sets" cp "k8s.io/kubernetes/pkg/kubelet/checkpoint" "k8s.io/kubernetes/pkg/kubelet/checkpointmanager"

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: of a static pod (really the mirror pod) results in multiple attempts to re-create the mirror pod before it finally succeeds (note that in some runs it goes into Running or Terminating state -- it just depends upon the exact timing of the . Note that in the below trace the pod name is elided from the output. Note that the actual static pod continues to run as normal. What you expected to happen: Mirror pod should start without additional state transitions How to reproduce it (as minimally and precisely as possible): Create a static pod, it, and Anything else we need to know?: Environment: Kubernetes version (use ): 1.14/OpenShift 4.2 (master has the same code) Cloud provider or hardware configuration: AWS (not relevant) OS (e.g: ): Not relevant Kernel (e.g. ): Not relevant Install tools: Not relevant Network plugin and version (if this is a network-related bug): N/A Others:
/sig node
More details: The bug appears to be a race condition between deleting the old mirror pod, the API server deleting its record of it, and creating a replacement that's not related to skew. I suspect the reason I didn't see it bouncing back and forth in my earlier tests was simply that I wasn't running the kubectl get node frequently enough; if I run it continuously, I see it bounce. The sequence is: 1) Somebody kubectl deletes the mirror pod. 2) The static pod code in (around line 1630) notices that the pod has been marked deleted (its deletion timestamp is not nil and it is a mirror pod), deletes it (in DeleteMirrorPod in , via the apiserverClient), and creates a new one (again, via the apiserverClient). 3) The next time around, the static pod code may see either the old pod (that has been deleted) or the new one as the mirror pod.

If the old pod is the one that's seen, it sees that there's a deletion timestamp, so it deletes it again -- which, since pod deletion is by name (UID isn't involved) results in the new pod being deleted and yet another new pod being created. If the new pod is the one that's seen, then we break out with the spin cycle complete. Still TBD is why the mirror pod isn't getting replaced right away. But here are traces of the relevant sequences: Bad sequence: Good sequence: The messages at lines 7 and 18 were printed immediately after the call to apiServerClient...Create(). But the key is at lines 9 and 19. At line 9, note that the UID (...) is the UID of the old mirror pod, which has a deletion timestamp and therefore the mirror code deletes the pod -- but the pod that's actually in existence at the time is the new pod. At line 19, however, note that the mirror pod is now the new one, which is not marked for deletion, and is therefore left alone. The solution (I'll open a PR shortly) is to check both the pod name and the UID to make sure that we're in fact trying to delete the pod we think we want to delete.
Thanks for the detailed issue :) Really awesome!
I'm seeing this issue on an upgrade from v1.14.4 to v1.15.1 following the on a CentOS 7 system. After installing the updated kubelet package and restarting kubelet per the instructions, the cluster went into a loop of continually restarting the etcd pod from one control plane node and the scheduler pod from another control plane node. This continued for approximately 10 minutes before the etcd pod finally stayed up. It was another 5 minutes before the scheduler pod finally ran. I noticed v1.15.2 was released although the change log doesn't show this issue included. Was there a plan to back-port this fix to 1.15 and 1.14 (not to the same extent, but I have seen this behavior on a 1.14.4 cluster)? Since this behavior can happen to the apiserver pod, it has the potential to be very disruptive to a cluster.
The actual etcd pod, or the mirror pod (which is more likely)?

I hadn't specifically planned on backporting it, but it shouldn't be a problem; this code has been stable.
Ah, good point--the actual pod does appear to remain running. That makes it less important than I was initially thinking. However, it is a bit disconcerting for someone who happens to be looking at the state of the cluster (or for monitoring) to see the pod appear to be stuck in a restart loop. If it were just a few iterations, it might not matter too much. However, I'm seeing this happen for minutes at a time. For example, to verify that the real pod remained available I did a on the apiserver pod for one master, and output showed it in the restart loop for 15 minutes before it settled down. IMO that's worth backporting if it's low risk.
I opened a cherrypick against 1.15 for this.
Appreciated. Thanks!


Question: return nil } <ins> // check if it's update and it doesn't change the images referenced by the pod spec func isUpdateWithNoNewImages(attributes admission.Attributes) bool { if attributes.GetOperation() != admission.Update { return false } pod, ok := attributes.GetObject().(*api.Pod) if !ok { klog.Warningf("Resource was marked with kind Pod but pod was unable to be converted.") return false } oldPod, ok := attributes.GetOldObject().(*api.Pod) if !ok { klog.Warningf("Resource was marked with kind Pod but old pod was unable to be converted.") return false } oldImages := sets.NewString() pods.VisitContainersWithPath(&oldPod.Spec, field.NewPath("spec"), func(c *api.Container, _ *field.Path) bool { oldImages.Insert(c.Image) return true }) hasNewImage := false pods.VisitContainersWithPath(&pod.Spec, field.NewPath("spec"), func(c *api.Container, _ *field.Path) bool { if !oldImages.Has(c.Image) { hasNewImage = true } return !hasNewImage }) return !hasNewImage } </ins> func shouldIgnore(attributes admission.Attributes) bool { // Ignore all calls to subresources or resources other than pods. if len(attributes.GetSubresource()) != 0 || attributes.GetResource().GroupResource() != api.Resource("pods") { return true } <ins> if isUpdateWithNoNewImages(attributes) { return true } </ins> return false }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: We enabled AlwaysPullImages admission in an existing cluster, and found annotations/labels/finalizers of existing Pods whose imagePullPolicy configured as could not be updated. AlwaysPullImages admission mutated containerSpec by . And this change later got denied by Pod validation. It also caused terminating Pods with finalizers stuck forever until we removed AlwaysPullImages admission. What you expected to happen: Existing Pods are not affected by AlwaysPullImages admission controller. How to reproduce it (as minimally and precisely as possible): the kube-apiserver with AlwaysPullImages admission controller disabled a Pod with imagePullPolicy configured as in its containers AlwaysPullImages admission controller annotation to Pod Anything else we need to know?: If this is confirmed a bug, I can submit a PR to fix this. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig api-machinery
seems to be a bug. /assign
/remove-sig api-machinery /sig node auth
try to fix it in When AlwaysPullImages is disabled, you cannot update imagePullPolicy of pod. The error message is the same as yours. imagePullPolicy is immutable. When AlwaysPullImages is enabled, old pod(with imagePullPolicy=ifNotPresent) update will change imagePullPolicy as well. Workaround: recreated pod -_-
Thanks for the fixes. I still have regarding the PR. IMHO, it's OK that imagePullPolicy is not updatable. Why don't we just ignore Update in the admission, but choose to update spec instead? Any concerns?

This is another fix option. I'm not sure which is better. The pod will be still ifNotPresent after updating, even if the AlwaysPullImages is enabled. Which is expected behavior?
In my case, I would expect the admission not to touch existing Pods. Just want to make sure if there's any concerns.
I would recommend ignoring pod updates in AlwaysPullImages if the update does not change the set of image names referenced by containers in the pod. Changing images inline in a pod update is unusual, and that change would allow unrelated updates (to annotations, labels, etc) to proceed
Agree.
is updated .
This issue is fixed by


Question: func TestExtractFromHttpMultiple(t *testing.T) { manifests := []api.ContainerManifest{ <del> {Version: "v1beta1", ID: ""}, {Version: "v1beta1", ID: "bar"}, </del> <ins> {Version: "v1beta1", ID: "", Containers: []api.Container{{Name: "1", Image: "foo"}}}, {Version: "v1beta1", ID: "bar", Containers: []api.Container{{Name: "1", Image: "foo"}}}, </ins> } data, err := json.Marshal(manifests) if err != nil {

The hash function generates names with upper-case letters, but the validation only allows lower-case. First, we clearly need a test for this. Second, we need to fix it. Should we allow upper-case characters (which the RFC does allow for DNS labels) or should we fix the hash? Allowing upper-case seems attractive, except that the RFC also says that case doesn't matter, but should be preserved "wherever possible". So we really have three choices, I see. a) only allow lower-case b) allow upper-case and always use caseless comparisons everywhere c) allow upper-case, but normalize to lower-case since it was your hash code
Re: test - sounds like a config source -config integration test, which I'll add. Re: hash - would argue we should fix the hash to generate lowercase for now. I think, but will double check, that the collision space is still improbably low for 63 lowercase chars. I can do that shortly.
And a validate test on all sources.
Lowercase would require base32, rather than base64, no? On Aug 3, 2014 8:20 AM, "Clayton Coleman" wrote:
Yeah, although I don't believe there's a difference to the outcome between base32 and base64(lowercase) given an appropriate hash function.
Well, base32 is reversible and a modified base64is not :) On Aug 3, 2014 9:00 AM, "Clayton Coleman" wrote:
Remind me again why reversibility is important? We discussed it in another issue but I can't find the context.
I'm going to change the generated name to which even with 2^8 files per host and a thousand hosts should be well below 0.% chance of collision
Its only important if we want to use it. We don't AFAIK. On Aug 3, 2014 9:34 AM, "Clayton Coleman" wrote:
Fix makes them unique, somewhat traceable (first 15 safe chars of the name), and generates base32 output.


Question: if pod.Name != "foo" { t.Errorf("Unexpected pod: %#v %s", pod, resp.Node.Value) } <ins> if !(pod.Annotations != nil && pod.Annotations["label1"] == "value1") { t.Fatalf("Pod annotations don't match the expected: %v", pod.Annotations) } } func TestEtcdCreateWithConflict(t *testing.T) { registry, bindingRegistry, _, fakeClient, _ := newStorage(t) ctx := api.NewDefaultContext() fakeClient.TestIndex = true key, _ := registry.store.KeyFunc(ctx, "foo") fakeClient.Data[key] = tools.EtcdResponseWithError{ R: &etcd.Response{ Node: nil, }, E: tools.EtcdErrorNotFound, } _, err := registry.Create(ctx, validNewPod()) if err != nil { t.Fatalf("unexpected error: %v", err) } // Suddenly, a wild scheduler appears: binding := api.Binding{ ObjectMeta: api.ObjectMeta{ Namespace:   api.NamespaceDefault, Name:        "foo", Annotations: map[string]string{"label1": "value1"}, }, Target: api.ObjectReference{Name: "machine"}, } _, err = bindingRegistry.Create(ctx, &binding) if err != nil { t.Fatalf("unexpected error: %v", err) } _, err = bindingRegistry.Create(ctx, &binding) if err == nil || !errors.IsConflict(err) { t.Fatalf("expected resource conflict error, not: %v", err) } </ins> } func TestEtcdCreateWithExistingContainers(t *testing.T) {

Recap of related tangent in follows. Use case: I need to implement mesos task reconciliation in the kubernetes-mesos scheduler. If it crashes and then restarts, I want to be able to recover things like taskID, slaveID, offerID, etc. so as to rebuild the internal state of the scheduler. Once state is recovered, I can reconcile the against the tasks that the mesos master knows about. If I can't use binding annotations for this, then I need to store state some other (super inconvenient) way. Currently the etcd registry assignPod() implementation drops binding annotations on the floor. It should not, and instead they should be preserved for later access. Proposal: Store binding-related annotations in a new field: Pod.Status.Annotations When setting Pod.Status.Host="{some-host-name}" via assignPod(), binding annotations are copied in from Binding When cloning Pod.Status.Host in UpdatePod() (for an already-scheduled pod), the Pod.Status.Annotations are also cloned. I don't think /bindings has to be a readable apiserver endpoint, as long as the binding metadata is exposed some other way, as proposed above. I'd also like to propose that once binding annotations are "bound" to Pod.Status they become read-only. So they're essentially write-once, just like Pod.Status.Host is meant to be. cc xref
There is one situation where we probably want to clear Host, actually: in the case that the pod can't actually be materialized on the selected node for some reason, such as .
Any objection to a PR implementing my proposal? I'm happy to do the work.
You'll probably want to do it on top of (or after it lands) since that code path is changing significantly.
Though it is also represented in PodStatus, Host is properly a field in the PodSpec. Status must be completely reconstructable based on observations. The proposed Pod.Status.Annotations would not be. Furthermore, I don't think we should introduce another annotations field to Pod. I'd prefer to add a podAnnotations field to Binding, and then merge them into the pod. As for clearing the annotations, I'd prefer to simply fail a pod that was rejected by a node so as not to have to deal with that. Also note the discussion here re. moving binding to a pod subresource:
See also .


Question: w.Write(LEVEL_1, "ResourcetRequeststLimitsn") w.Write(LEVEL_1, "--------t--------t------n") reqs, limits := getPodsTotalRequestsAndLimits(nodeNonTerminatedPodsList) <del> cpuReqs, cpuLimits, memoryReqs, memoryLimits := reqs[api.ResourceCPU], limits[api.ResourceCPU], reqs[api.ResourceMemory], limits[api.ResourceMemory] </del> <ins> cpuReqs, cpuLimits, memoryReqs, memoryLimits, ephemeralstorageReqs, ephemeralstorageLimits := reqs[api.ResourceCPU], limits[api.ResourceCPU], reqs[api.ResourceMemory], limits[api.ResourceMemory], reqs[api.ResourceEphemeralStorage], limits[api.ResourceEphemeralStorage] </ins> fractionCpuReqs := float64(0) fractionCpuLimits := float64(0) if allocatable.Cpu().MilliValue() != 0 {

<!-- Please only use this template for submitting enhancement requests --What would you like to be : display EphemeralStorage usage with 'kubectl describe node' , like; Why is this needed: ephemeral-storage is a standard ContainerResources like CPU and Mem now. but now we don't display it in the node state. <!-- DO NOT EDIT BELOW THIS LINE --/kind feature /sig cli cc


Question: continue } } // Delete old endpoints for _, ep := range curEndpoints.Difference(newEndpoints).UnsortedList() { // if curEndpoint is in gracefulDelete, skip

Entry in ipvs real sever remains at weight=0 after kube-proxy restarts. (During 1.21 -1.22 kubernetes cluster upgrade.) Related to As the related issue is considered closed, I will report this issue again as the problem still persists in newer version. Might because of some corner cases. Real server weight should be 1 when endpoints is ready. Use kube-proxy in ipvs mode Deploy a service resource and its backend pods Make real sever not ready (or may need node not ready) so as to trigger graceful delete in kube-proxy Before kube-proxy deletes the entry from ipvs, restart kube-proxy Kube-proxy becomes ready and leave the weight=0 entry as it is Service is no longer accessible through cluster ip When accessed, dmesg shows kernel messge error ofDeletion of pod will trigger kube-proxy to remove weight=0 entry from ipvs so that after the entry is recreated service will become accessible If service is 1. LoadBalancer, and, 2. .spec.externalTrafficPolicy: Local, and, 3, service is accessed from loadbalancer IP, the entry in ipvs for loadbalancer IP will only have endpoints in the same node. If backends is DaemonSet, the only one endpoints will have weight=0 thus causing connection failure After kube-proxy restarts, endpoints will be processed in the following lines. Only when the entry is in the gracefuldeleteManager.InTerminationList will be deleted. Which is not the case if kube-proxy restarts before it deletes the ipvs entry. An existing entry in the ipvs is sufficient for endpoint to be considered healthy. Weight is not considered. kube-proxy should treat ipvs entry properly by the either the following two ways. kube-proxy should initiate InTerminationList and put ipvs entry with weight=0 into the termination list kube-proxy should also look for weight in the ipvs entry instead of only UniqueRSName <details</details<detailson-premises </details<details</details<details</details<details</details<details</details
/sig network
Graceful delete of ipvs is not only triggered when pod is in terminating phase. Graceful delete seems to be triggered when endpointslice status is ready=false. This will happen when node become not ready.

However, after the node become ready again, pods on the node will only be recreated if the duration of NotReady is longer than certain time. Otherwise the pod will just become ready again. If the node will not recreate the pods && kube-proxy set the pods to the graceful delete queue && kube-proxy restarts before deletion and all cache is cleared, kube-proxy will keep ipvs entry with weight=0 even if the real server is ready. Please correct me if I misunderstand something.
/area ipvs /area kube-proxy
Seems like a legit bug to me. I can't triage until wednesday though.
I can't recreate this when I try to follow thre instructions with pod not-ready rather than node not ready. When the pod is made ready again is not stated, so I tried; a connection to a pod it not-ready. The weight is set to 0, the connection prevails -start kube-proxy. The weight is still 0, connection still working the pod ready weight is set to 1 after a while, as it should
In the instruction is different, the pod is made ready while is away. That instruction shows the fault. The reproduce instruction in this issue is incorrect.
I triage this issue as a bug, but it's really a duplicate of which was never fixed (it seems). Don't know if the proper action is to close this issue and re-open . /triage accepted
Please instruct if this issue shall be closed as duplicate and re-opened PTAL at
Thank you for your triage Monitoring IPVS weights is costly, but needed while this problem persists. Any news? Adding the following lines should fix the problem but will also add loads.
That does seem costly to do on every endpoint sync. Is there an alternative solution we can explore where we assess all endpoints once when kube-proxy initially starts up?
This is a startup-only problem. It should be solved with a one-time action.
How about modifying to be ? We could then check and update the real server weights only when is true. Only and will be modified to call , all other users will call the function with false. Since these two functions, are only called during startup, we won't be doing this monitoring every endpoint sync.
cc:


Question: - Score: 2 Utilization: 1 `), <del> wantErr: `decoding .profiles[0].pluginConfig[0]: decoding args for plugin NodeResourcesFit: strict decoding error: unknown field "scoringStrategy.requestedToCapacityRatio.shape[0].Score", unknown field "scoringStrategy.requestedToCapacityRatio.shape[0].Utilization"`, </del> <ins> wantErr: `strict decoding error: decoding .profiles[0].pluginConfig[0]: strict decoding error: decoding args for plugin NodeResourcesFit: strict decoding error: unknown field "scoringStrategy.requestedToCapacityRatio.shape[0].Score", unknown field "scoringStrategy.requestedToCapacityRatio.shape[0].Utilization"`, </ins> }, { name: "v1beta3 NodeResourcesFitArgs resources encoding is strict",

With the addition of server side validation in 1.23, it is now possible for a call to successfully decode it’s data and return a non-nil error when there are strict decoding errors (e.g. duplicate/unknown fields). This means that callers of can distinguish between fatal errors (that cause decoding to fail) and strict decoding errors (that still allow the decode to succeed) by inspecting the target object (to see if it was successfully decoded into) and the error itself (to see if it is a ) With the NestedObjectDecoder however, it only returns an error from giving no indication whether a strict decoding error it has returned comes from an internal decode step that short circuited (by not accounting for strict decoding errors) or actually did successfully decode, but also encountered strict decoding errors. We propose taking a few actions to minimize the possibility of erroneous short-circuiting from Document how all nested decoders should only return strict decoding errors if all decoding is successful or return only non-strict decoding (ie. fatal errors). Document this for all known in-tree and out-of-tree . Update in-tree nested decoders to do the above mentioned behavior (which is just the kube-scheduler) Long-term solutions to better guarantee that no erroneous short-circuiting is taking place could include deprecating the current interface in favor of one that returns both the target object that was decoded into along with an error return a special error type specific to indicating that all internal decoding was successful.
/sig api-machinery
/triage accepted


Question: } } <ins> func TestServiceRegistryCreateMultiNodePortsService(t *testing.T) { storage, registry := NewTestREST(t, nil) testCases := []struct { svc             *api.Service name            string expectNodePorts []int }{ { svc: &api.Service{ ObjectMeta: api.ObjectMeta{Name: "foo1"}, Spec: api.ServiceSpec{ Selector:        map[string]string{"bar": "baz"}, SessionAffinity: api.ServiceAffinityNone, Type:            api.ServiceTypeNodePort, Ports: []api.ServicePort{ { Name:       "port-tcp", Port:       53, NodePort:   30053, TargetPort: intstr.FromInt(6503), Protocol:   api.ProtocolTCP, }, { Name:       "port-udp", Port:       53, NodePort:   30053, TargetPort: intstr.FromInt(6503), Protocol:   api.ProtocolUDP, }, }, }, }, name:            "foo1", expectNodePorts: []int{30053, 30053}, }, { svc: &api.Service{ ObjectMeta: api.ObjectMeta{Name: "foo2"}, Spec: api.ServiceSpec{ Selector:        map[string]string{"bar": "baz"}, SessionAffinity: api.ServiceAffinityNone, Type:            api.ServiceTypeNodePort, Ports: []api.ServicePort{ { Name:       "port-tcp", Port:       54, TargetPort: intstr.FromInt(6504), Protocol:   api.ProtocolTCP, }, { Name:       "port-udp", Port:       54, NodePort:   30054, TargetPort: intstr.FromInt(6504), Protocol:   api.ProtocolUDP, }, }, }, }, name:            "foo2", expectNodePorts: []int{30054, 30054}, }, { svc: &api.Service{ ObjectMeta: api.ObjectMeta{Name: "foo3"}, Spec: api.ServiceSpec{ Selector:        map[string]string{"bar": "baz"}, SessionAffinity: api.ServiceAffinityNone, Type:            api.ServiceTypeNodePort, Ports: []api.ServicePort{ { Name:       "port-tcp", Port:       55, NodePort:   30055, TargetPort: intstr.FromInt(6505), Protocol:   api.ProtocolTCP, }, { Name:       "port-udp", Port:       55, NodePort:   30056, TargetPort: intstr.FromInt(6506), Protocol:   api.ProtocolUDP, }, }, }, }, name:            "foo3", expectNodePorts: []int{30055, 30056}, }, } ctx := api.NewDefaultContext() for _, test := range testCases { created_svc, err := storage.Create(ctx, test.svc) if err != nil { t.Fatalf("Unexpected error: %v", err) } created_service := created_svc.(*api.Service) if !api.HasObjectMetaSystemFieldValues(&created_service.ObjectMeta) { t.Errorf("storage did not populate object meta field values") } if created_service.Name != test.name { t.Errorf("Expected %s, but got %s", test.name, created_service.Name) } serviceNodePorts := CollectServiceNodePorts(created_service) if !reflect.DeepEqual(serviceNodePorts, test.expectNodePorts) { t.Errorf("Expected %v, but got %v", test.expectNodePorts, serviceNodePorts) } srv, err := registry.GetService(ctx, test.name) if err != nil { t.Errorf("unexpected error: %v", err) } if srv == nil { t.Errorf("Failed to find service: %s", test.name) } } } </ins> func TestServiceStorageValidatesCreate(t *testing.T) { storage, _ := NewTestREST(t, nil) failureCases := map[string]api.Service{

Consider a service like The API refuses to accept this service, stating It works fine without . It works when using different ports. The iptables rules installed (in mode) are TCP/UDP specific. I'm not sure if this is deliberate but not well documented, or a bug? I don't see a reason not to allow a UDP and TCP node port to have the same number.
For context, I am trying to make the cluster DNS available to legacy systems with a bit of forwarding, and DNS really really wants to have UDP and TCP on the same port number.
what is your kubernetes version?
This is on 1.1.4. From a quick reading of the code this hasn't changed though.
This was intentionally omitted due to laziness and complexity. We hoped nobody would need it, since NodePorts are generally allocated automatically. It's mostly going to be a matter of adding a switch to look at 2 different saved allocation maps and figuring out how to transition. , Matthias Rampke wrote:
I've also hit this issue (see: ) when trying to expose DNS to queries from outside the cluster. Note that in my setup, the "NodePort" is different for UDP and TCP but only UDP appears to work (it's the first to be defined?).
I also run into this issue. I've got a service that wants to expose TCP and UDP port with the same number. Any plans to include this issue in any roadmap?
Help wanted. We're buried in high-priority bugs and features. I ACK that this behaves imperfectly, but I and my team will not have time to look at it in the immediate future.
FYI
I think the root cause is , when serviceport allocator attempts to reserve the provided port, it just use the port number not consider the port protocol. This is a bug of service port allocator I believe.
"bug", but yes. We'd need to store parallel bitmaps per protocol. Even then, it's not obvious when to use the same port number in different protocols, and whether that is REQUIRED or not.

If you have a Service with port 53 in TCP and UDP, and I can't find any port number that is available in both protocols, should I: a) fail and reject the REST operation b) choose a different port number c) retry forever and hang the REST operation d) accept the REST operation but log an event saying I failed to do my job NodePort was really made for load-balancers, where the port number doesn't matter. Humans using it is "unexpected". , TonyAdo wrote:
this is what I'd expect. Port exhaustion is not really a temporary Matthias Rampke Engineer SoundCloud Ltd. +49 173 Managing Director: Alexander Ljung | Incorporated in England & Wales with Company No. HRB 110657B
I am afraid PR will not fix this issue. We have to store parallel bitmaps per protocol. I will open a new PR to fix this issue.
This alone is insufficient, but we might get away without storing two maps by simply saying the port map is unified across protocols if a service has 2 ports that use the same port number but different protocols, allocate the first nodePort and use the same value for the 2nd port This will prevent the case of ports being used in one protocol but not the other and failing to allocate. Something like: I haven;t tried too hard to break it, but something like that MIGHT work.
Additionally, if you have to store multiple bitmaps, you have a serious hassle on your hands for upgrading cluster in-place. Better to avoid it if you can.
sure, I will think about this more.

Hi, I have created and deployed a service similar to below in minikube environment.. however I am not to connect to the ports sometimes TCP/UDP ` apiVersion: v1 kind: Service metadata: name: dns spec: selector: … clusterIP: … type: NodePort ports: name: grpc protocol: TCP port: targetPort: grpc-port name: port-udp port: targetPort: 5062 protocol: UDP nodePort: name: port-tcp port: targetPort: 5062 protocol: TCP nodePort: Example: $kubectl get all service/-service NodePort 10.106.159.94 <none50051:/TCP,30:/UDP,:/TCP $ telnet 192.168.99.100 telnet: can't connect to remote host (192.168.99.100): Connection refused $ telnet localhost telnet: can't connect to remote host (127.0.0.1): Connection refused $nc -z -v -u 192.168.99.100 Connection to 192.168.99.100 port [udp/] succeeded! $ nc -z -v 192.168.99.100 nc: connect to 192.168.99.100 port (tcp) failed: Connection refused $ kubectl version Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-10-02T17:01:15Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-09-18T14:27:17Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"} $netstat -an tcp 0 0 ::: ::: LISTEN udp 0 0 ::: :::*
!


Question: The address for the proxy server to serve on (set to 0.0.0.0 or "" for all interfaces) .PP <del> fB-configfilefP="/tmp/proxy_config" Configuration file for the proxy .PP </del> fB-etcd_serversfP=[] List of etcd servers to watch ( [la]http://ip:port[ra]), comma separated (optional)

It's not well tested nor widely used; so it should either be removed or have tests and uses .


Question: return err != nil && os.IsNotExist(err) } <ins> // GetCapacity returns node capacity data for "cpu", "memory", "ephemeral-storage", and "huge-pages*" // At present this method is only invoked when introspecting ephemeral storage </ins> func (cm *containerManagerImpl) GetCapacity() v1.ResourceList { <ins> if utilfeature.DefaultFeatureGate.Enabled(kubefeatures.LocalStorageCapacityIsolation) { // We store allocatable ephemeral-storage in the capacity property once we Start() the container manager if _, ok := cm.capacity[v1.ResourceEphemeralStorage]; !ok { // If we haven't yet stored the capacity for ephemeral-storage, we can try to fetch it directly from cAdvisor, if cm.cadvisorInterface != nil { rootfs, err := cm.cadvisorInterface.RootFsInfo() if err != nil { klog.ErrorS(err, "Unable to get rootfs data from cAdvisor interface") // If the rootfsinfo retrieval from cAdvisor fails for any reason, fallback to returning the capacity property with no ephemeral storage data return cm.capacity } // We don't want to mutate cm.capacity here so we'll manually construct a v1.ResourceList from it, // and add ephemeral-storage capacityWithEphemeralStorage := v1.ResourceList{} for rName, rQuant := range cm.capacity { capacityWithEphemeralStorage[rName] = rQuant } capacityWithEphemeralStorage[v1.ResourceEphemeralStorage] = cadvisor.EphemeralStorageCapacityFromFsInfo(rootfs)[v1.ResourceEphemeralStorage] return capacityWithEphemeralStorage } } } </ins> return cm.capacity }

When trying to join a second or sometimes third control plane node, fails with: something that looks like an etcd hangup ("Initial timeout of 40s passed") CrashLoopBackOff for one of the kube-apiserver pods The k8s suggest opening an issue here. Successful join op Here goes... First to init the first control plane node (1 of 3 in an HA cluster). This is successful. Contents of : Apply CNI plugin: Looks fine at this point. shows all Running. Next join 2nd control plane node: This seems to go off without a hitch also. The new node comes up as Ready and all new pods Running. Contents of here: Now for the third, failing node: same exact join command, same exact config. It is really a replica node of the 2nd control plane node, and yet times out. Output: Then going back to initial control node. Output of : Uh-oh. So CrashLoopBackOff shows up and one etcd pod is missing completely. No. Kubernetes version (use ): Cloud provider or hardware configuration: AWS EC2 OS (e.g: ): Ubuntu 18.04.5 LTS Kernel (e.g. ): Network plugin and version (if this is a network-related bug): weave-net
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cluster-lifecycle
/sig usability
Can you check etcd yaml in node in ? Also check the log on node to know why the etcd pod is not created. A known issue is for ephemeral-storage check, you can remove the ephemeral storage requests in and see if the etcd pod can be created.
Okay, I re-ran this exact setup on a newly provisioned set of nodes and ran into the same error.

xx-xx-xx-248: initial control plane node (ok) xx-xx-xx-131: second control plane node (ok) xx-xx-xx-176: third control plane node (fails to join) Also want to add the comment that I am joining the nodes in serial rather than in parallel, since I have in other issues seen parallel joins described as an issue for etcd. With this latest run I ran with and fetched some additional log info as referenced by Contents of on the failing node: Output from the failed on the failing node: is same as original post. I am -ing the output to and guessing I might need to capture stderr too. But that's outside the scope of this issue. And lastly kubelet log on the failing node (from ): As shown above this is littered wth (newlines for readability): Final note: the IP changes in this syslog from .65 because the nodes are rebooted before any init or join operations are ran.
/sig node storage
Going to give a try. In my situation here, each node is 65G disk and 4G RAM with nothing else really going on on them besides the k8s initialization.
It looks like applying as outlined in provided a band-aid and the final control plane node joined successfully. The param is successfully removed from . For other kubeadm users looking to apply this fix, the for is: The key is that the argument is a directory, not the patch file specifically. In my case, the file sat at and the invocation is then
I've updated PR to reflect that it will fix this issue Also, note that was merged yesterday, and will be cherry-picked into future 1.20 and 1.21 releases.
thanks for checking the logs. let's keep a single tracking issue for this problem: /close
Closing this issue. <detailsIn response to : Jun 09 11:49:50 ip-xx-xx-xx-176 kubelet[4641]: I0609 11:49:50.

4641 ] "Failed to admit pod, unexpected error while attempting to recover from admission failure" pod="kube-system/etcd-ip-xx-xx-xx-176" err="preemption: error finding a set of pods to preempt: no set of running pods found to reclaim resources: [(res: ephemeral-storage, q: ), ]" Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

on the new node fails sometimes. The following error appears in the log: on the new node succeeds every time Join a control plane node The node has over 40 GB of free space on its root partition. I some more logging to the following functions: , , , , , , It appears that the ephemeral storage is detected a bit too late: /sig node storage Kubernetes version (use ): 1.20.4, commit Kernel (e.g. ): 4.9.252-1.ph2
/assign I will try to reproduce it today.
LocalStorageCapacityIsolation is a feature gate for kubelet. Can you check if you disable it? ReadyCondition will check ephemeral-storage as required capacities if this feature gate is enabled in kubelet configuration. Would you check it first?
[x] 1.8 alpha ephemeral-storage [x] 1.10 Promote LocalStorageCapacityIsolation feature to beta [x] 1.15 KEPs Quotas for ephemeral-storage for LocalStorageCapacityIsolationFSQuotaMonitoring default to false is Alpha since 1.15 [x] Failed to promote LocalStorageCapacityIsolationFSQuotaMonitoring feature to beta [x] 1.20 Honor disabled LocalStorageCapacityIsolation in scheduling I am no sure whether there is a plan to GA LocalStorageCapacityIsolation?
I tried to fix it in . Need some testing later.
It appears that the pod fails before sees the storage capacity:
Would you try this ? I build it with on PR branch.
I tried to apply the PR to 1.20.4. After four successful s, one failed. I got the following error:
! It means that during the node resource updated the first time, the cadvisor is not available yet. If we wait here, it would be a problem. I would think more of how to fix it.
Hi, Same problem here. I'm using K8s v1.20.4 with CentOS7 (2 cores, 4GB and 20GB storage) Cluster creation with kubeadm init always works, but when I try to add several master nodes with kubeadm join it takes a lot of time waiting for etcd pod to be created. In most of cases, etcd pod never runs.

The rest of components (APIserver, controller, scheduler and proxy) exists and works (except APIServer which reboots trying to connect the missing etcd) Checking the kubelet logs in the missing etcd hosts I have found this: Mar 10 14:23:27 kmaster-02 kubelet[9448]: W0310 14:23:27. 9448 ] Failed to admit pod etcd-kmaster-02_kube-system() - Unexpected error while attempting to recover from admission failure: preemption: error finding a set of pods to preempt: no set of running pods found to reclaim resources: [(res: ephemeral-storage, q: ), ] That node have 17GB storage free. If kubelet is restarted, the etcd (pod and container in docker ps -a) appears. If I could help doing something please tell me. I would like to help
For anyone who is experiencing this issue and still wants to use kubeadm until it is fixed: You can remove the ephemeral-storage request from the etcd static pod and the join will work. Just make sure that your nodes actually have more than 100MB free storage left. You can reinsert the storage request by patching the etcd manifests after the nodes are up. To patch etcd during a create the following file (the filename matters, the file location doesn't) and use to join additional control plane nodes.
Thanks for the workaround I'm encountering this same issue and the workaround worked for me. Edit: the patch I tried doesn't work though (not sure if this is really a problem). I issued the command: and the response shows that the resource can't be patched this way: There may be another way but I'm still fairly new to Kubernetes so it is not obvious to me at the moment. Anyhow, thanks for the suggestion!
I updated Add a start-delay of 10 seconds for a condition that the capacity of node ephemeral-storage is 0. Not sure if this will fix it.
we are starting to see reports about this every week.
cc
cc
From it sounds like the issue is kubeadm etcd ephemeral-storage requests are too big for some environments? So far this is sounding like a kubeadm issue, not a kubernetes issue.
it seems to be something else.

i doubt that users are trying machines that don't satisfy 100Mi. for example, the report in the OP points out that the machine has 40GB.
I can confirm that the VMs on which I installed k8s control plane components had sufficient disk space when I encountered the problem. I'm running Kubernetes v1.20.4 and kubeadm v1.20.4.
I think the root cause here sounds right. There is kind of race here. adding delay will help, but it does not have 100% guarantee. Not sure whether it is easy to add logic to make sure capacity is reported correctly before starting pods..
Did something change in 1.20 regarding the ordering of cadvisor and node startup?
If the reason would be a disk problem(or something we can figure out) and the delay can solve it, I think we should add a warning besides the delay.
We need to figure out what triggers the race condition. This only happens on some machines. I have not seen this myself.
We're able to repro this nearly 100% of the time with Cluster API Provider Azure + k8s 1.20.x + Ubuntu 18.04 + default template, which makes k8s 1.20 pretty much unusable at this point. Trying to see if this repros with other non-Azure cluster-api providers with kubeadm 1.20.
Does anyone know why this only repros on and not ? The ephemeral storage resource requirement here is used in common by both and : In practice I can workaround this issue in cluster-api + Azure by adding a sleep command to my spec, but it requires a significant number to make work ~100% of the time (like 90 seconds). Does the flow take significantly longer before it gets to the part where it submits an etcd pod to the apiserver? Is there any other explanation for this being a -only symptom?
i can see this failing for any static pod with ephemeral storage requests. that is, unless there is proof that this is somehow specific to the kubeadm static pod for etcd that kubeadm deploys.
I agree that the specific kubeadm-delivered etcd pod is not interesting. (In fact it's very permissive, it's only asking for 100Mi.)

What definitely seems to be happening (in Azure) is that kubelet takes a long time to know about its underlying node's ephemeral-storage availability. Kubelet starts: ~2 mins later, it complains that it can't schedule etcd: Again, why does this not repro on nodes created by ?
it also seems to be an even more scoped problem because it's not reproducible for me locally (i create a number of clusters every day, usually), the kubeadm CI or in other high level projects that use kubeadm (e.g. kind, minikube, etc). we should try to find a common reason / condition from all the reporting users that are seeing this.
Experiencing same problem on openSuSE Kubic/Tumbletweed snapshot with kubeadm/kubelet 1.20.2, kernel 5.11.15 running on Xen, 2 CPU, 4GB ram 30GB disk (BRTFS)
Problem consistently appear/disappear when switching between snapshots (Working) and , so a package update in that period has probably triggered this. Following files contains changes between those two snapshots.
are you suggesting a root cause in underlying Linux storage driver surface area that is adding delay/uncertainty to kubelet's ability to introspect available ephemeral-storage on the node host OS?
A bit of luck combined with 6th sense i got it narrowed down to systemd update here Looking at the changelog, one item sticks out: Anyone else enabled cgroupsv2 on i.e. the CentOS cases (it needs to be enabled explicitly for that AFAIK)?
ACK on your latest respond regarding unified cgroup hierarchy. Is there a quick, cross-distro way to verify that any folks who are repro'ing have a similar systemd config?
The precence of file/dir /sys/fs/cgroup/cgroup.controllers should mean you are using cgroupsv2 ( Ref: ) I can confirm that the file is present on the kubic snapshot which is not working. It is not present on the one that does work. ( I'm using cri-o as runtime which should support cgroupsv2 ) cgroup2 support was introduced for K8S in 1.19 ( ).
Thanks. So tl;dr your working theory is that cgroupsv2 + kubelet ephemeral-storage admission has some pathologies.

I'll post quickly with what my repro containerd + Ubuntu 18.04-LTS OS surface area looks like.
I ran into this issue consistently on Ubuntu 20.04.2 LTS using cri-o but is not present on my systems.
I can also confirm that cgroupsv2 is not a factor in my repro'ing environment:
Also, I'm not using cri-o, containerd here:
Ok - so much for that theory. But still: What I did to narrow it to systemd-update was taking the working kubic snapshot () and run zypper update systemd which updates systemd (along with systemd-sysvinit and udev packages to same version). That consistently made it fail. So I think a pointer to the root-cause might still be hiding somewhere around this changelog - even if it is not cgroupsv2. That part was just a theory (seemed like a "big" thing), but the update of systemd package is fairly hard evidence.
I have identified where these particular symptoms (probably) originate, and cursory testing suggests something like this eliminates the issue: I'll be advocating for a few mins in today's sig-node discussion if any folks following this thread want to chime in. cc
/sig cluster-lifecycle /triage accepted
I've also observed that on some occasions the node allocatable storage is in a continual state of reporting "0". Restarting kubelet immediately fixed that. More debugging to do....
did you manage to reproduce this with an arbitrary static pod that is not the etcd one?
I was not able to reproduce in a non-kubeadm test, but I suspect that the way kubelet bootstrap is implemented there (and possibly environmental reasons) the static pod handling was always happening after this code executed: (the above is in the containerManager method) kubeadm w/ ephemeral-storage requirements in etcd is the most effective repro at this time
the writing of the etcd pod manifest in during is after the kubelet has already started and i wonder if that is contributing to the observed failures. probably not given this is a cadvisor related problem. to verify that i would write an arbitrary static pod manifest with the same request in : before (which also starts the kubelet) right after the kubelet has started if is causing such racy problems i don't think it should be graduated to GA.

though, it has been Beta for ~12 releases (on by default), so it's already used everywhere.
Based on your assessment do you think we should remove the ephemeral-storage requirement from default kubeadm etcd static pod? I can make that change.
based on the investigations here (thanks all), it feels like is a feature that may or may not work (for static pods at least). it doesn't feel right to mitigate a kubelet bug on the side of the deployer, but given how many reports there are about this i think it should be done at this point. happy to LGTM approve your PRs. note it must also be backported to all versions in support (back to 1.19). suggesting the following release note: we should keep this issue open, too.
Sounds good. I'll submit a PR soon with the kubeadm change, and I'll continue to drive toward a root cause solution on the kubelet side. I'll bring this up at sig-node tomorrow. Thanks!
is a possible duplicate of this issue - seeing a CrashLoopBackOff failure for kube-apiserver under a similar scenario to those mentioned here, e.g. failure to join a third control plane node with mentions of in the kubelet logs.
An ugly as sin workaroud: (please don't hate me :D )
It seems like this could be closed with and related cherry picks now being part of the newest releases? Bumping to 1.21.2 allowed me to remove the hack for etcd and several test deployments have steered clear entirely of the original issue. Thanks
great news. I'd prefer to pin this issue to this PR: That PR will deliver more race condition resilience for all static pod + ephemeral-storage requirement scenarios.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at .

/lifecycle stale [1]:
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:


Question: local network=$(make-gcloud-network-argument  "${NETWORK_PROJECT}"  "${REGION}"  <del> "${NETWORK}" </del> <ins> "${NETWORK}"  </ins> "${SUBNETWORK:-}"  ""  "${ENABLE_IP_ALIASES:-}"

The following that have been failing on our 5k-node tests (https://k8s-): [HPA] Horizontal pod autoscaling (scale resource: CPU) [sig-autoscaling] ReplicationController light Should scale from 1 pod to 2 pods [HPA] Horizontal pod autoscaling (scale resource: CPU) [sig-autoscaling] ReplicationController light Should scale from 2 pods to 1 pod This is part of the correctness suite and large cluster tests are release-blocking. cc
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/autoscaling sig/scalability: Issue will be escalated to these SIGs if needed. priority/critical-urgent: Never automatically move out of a release milestone; continually escalate to contributor and SIG through all available channels. kind/bug: Fixes a bug discovered during the current release. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details
Could you PTAL into this or redirect as apt? It's a release-blocker.
xRef
The tests seem to be passing on the normal correctness job - https://k8s- (except the latest run) but they're failing on large cluster and the reason seems to be timeout waiting for scaling to happen.
If i have to guess it is heapster who is failing to provide metrics. HPA itself has no direct dependency on the number of nodes in the cluster. Assigning to .
is going to take a look.
From the logs it seems like Heapster is unable to get metrics from nodes. It appears to be a separate issue from addon resizer restarts.
Thanks for checking it. I have got the same logs for smaller cluster (10 nodes), so at least we know that the problem is not related to big number of nodes/pods.
- I'm seeing the test pass on our regular (small cluster) gce job - https://k8s- And seems like it was passing before and then started failing on our 2k-node job - https://k8s-
thanks for pointing on it. The main difference between runs on small and big cluster - special node that we are creating for heapster container. And my local experiment have shown that without this node everything works, but with it - fails.

So I assume that it's more network related. I'll try to figure out what's going on.
Removing this as release-blocker and downing the priority as it's a setup issue specific to use of heapster node. Would continue to work on it (thanks!).


Question: eventType = v1.EventTypeWarning } expc.recorder.Event(pvc, eventType, events.ExternalExpanding, fmt.Sprintf("Ignoring the PVC: %v.", msg)) <del> klog.Infof("Ignoring the PVC %q (uid: %q) : %v.", util.GetPersistentVolumeClaimQualifiedName(pvc), pvc.UID, msg) </del> <ins> klog.Infof("Ignoring the PVC %q (uid: %q) : %v.", key, pvc.UID, msg) </ins> // If we are expecting that an external plugin will handle resizing this volume then // is no point in requeuing this PVC. return nil

I migrated an existing EBS volume from in-tree plugin to external aws-ebs-csi-driver (). Resizing PVC twice in 6 hour window causes kube-controller-manager to crash. Exception message: Kube-controller-manager shouldn't crash in such case. If external aws-ebs-csi-driver fails to provision the pvc it should just log an warning or error and continue operating as usual running cluster version 1.22.9 using in-tree aws ebs driver. pvc: Install aws-ebs-csi-driver following featureGate CSIMigrationAWS on all cluster components (with draining nodes before enabling featureGate on kubelets - according to warning ) we have ebs volume that has been migrated to aws-ebs-csi-driver. pvc by increasing .spec.resources.request.storage to 2Gi. PVC, PV and EBS volume are correctly resized. once again not later than 6 hours after first resize. Increase .spec.resources.request.storage to 3Gi. Now kube-controller-manager throws an exception and crashes: EBS CSI Driver logs: I tried also enabling alpha featureGate InTreePluginAWSUnregister but the issue still occurs. <details</details<detailsAWS </details<details</details<details</details<detailsdocker </details<detailsaws-ebs-csi-driver version 1.10.0 </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cloud-provider
/sig storage some situation where the sync loop passes a nil parameter to
/assign
SetClaimResizer may return nil pvc and non-nil error, ClaimToClaimKey naively assumes non-nil as described above, will create PR
But the posted bug is I think 1.22 specific and should never happen on master or 1.24 and hence this fix against master is kind of okay and should fix the issue but at the same time code in is still kinda broken in 1.22 because it returns if patching fails.


Question: "//pkg/client/clientset_generated/release_1_5:go_default_library", "//pkg/client/record:go_default_library", "//pkg/controller:go_default_library", <ins> "//pkg/conversion:go_default_library", </ins> "//pkg/runtime:go_default_library", "//pkg/util/flowcontrol:go_default_library", "//pkg/watch:go_default_library",

Unit tests on PR are failing due to data race in federation-controller/namespace unittest. More logs: https://k8s-
CC
Sent to fix this


Question: */ // +k8s:conversion-gen=k8s.io/kubernetes/pkg/apis/batch <ins> // +k8s:conversion-gen=k8s.io/kubernetes/pkg/apis/batch/v1 </ins> // +k8s:conversion-gen-external-types=k8s.io/api/batch/v2alpha1 // +k8s:defaulter-gen=TypeMeta // +k8s:defaulter-gen-input=../../../../vendor/k8s.io/api/batch/v2alpha1

it looks like something is up on master. Clean checkout, touch , then -- produces a delta.
There are no sig labels on this issue. Please add a sig label. <detailsA sig label can be by either: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. See the . The in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
That comes out clean. ` That comes out clean. That dir, and who knows how many others, is missing some metadata to allow conversions
looks like it might be the v2alpha1 conversion referencing a v1 type... if the generation processes both the v1 and v2alpha1 packages, it finds the v1 conversion. if it only processes the v2alpha1 package, it doesn't and reverts to the reflection-driven path
this breaks: this works: given a clean build leaves no diff, I think master is fine. the script looks like it doesn't include all required dependencies to rebuild a single package
The tool doesn't know - each pkg i supposed to define enough to make itself hermetic
That are not many input dirs. The generator does not see any custom conversion funcs in v1. As expected, adding v1 to the inputs gives the correct output:
/assign fixed in


Question: assert.Equal(resourceList.APIVersion, "v1") } <del> // TestValidOpenAPISpec verifies that the open api is added // at the proper endpoint and the spec is valid. func TestValidOpenAPISpec(t *testing.T) { _, etcdserver, config, assert := setUp(t) defer etcdserver.Terminate(t) config.GenericConfig.EnableIndex = true config.GenericConfig.OpenAPIConfig = genericapiserver.DefaultOpenAPIConfig(openapigen.OpenAPIDefinitions) config.GenericConfig.OpenAPIConfig.Info = &spec.Info{ InfoProps: spec.InfoProps{ Title:   "Kubernetes", Version: "unversioned", }, } config.GenericConfig.SwaggerConfig = genericapiserver.DefaultSwaggerConfig() master, err := config.Complete().New() if err != nil { t.Fatalf("Error in bringing up the master: %v", err) } // make sure swagger.json is not registered before calling PrepareRun. server := httptest.NewServer(master.GenericAPIServer.HandlerContainer.ServeMux) defer server.Close() resp, err := http.Get(server.URL + "/swagger.json") if !assert.NoError(err) { t.Errorf("unexpected error: %v", err) } assert.Equal(http.StatusNotFound, resp.StatusCode) master.GenericAPIServer.PrepareRun() resp, err = http.Get(server.URL + "/swagger.json") if !assert.NoError(err) { t.Errorf("unexpected error: %v", err) } assert.Equal(http.StatusOK, resp.StatusCode) // as json schema var sch spec.Schema if assert.NoError(decodeResponse(resp, &sch)) { validator := validate.NewSchemaValidator(spec.MustLoadSwagger20Schema(), nil, "", strfmt.Default) res := validator.Validate(&sch) assert.NoError(res.AsError()) } // Validate OpenApi spec doc, err := loads.Spec(server.URL + "/swagger.json") if assert.NoError(err) { validator := validate.NewSpecValidator(doc.Schema(), strfmt.Default) res, warns := validator.Validate(doc) assert.NoError(res.AsError()) if !warns.IsValid() { t.Logf("Open API spec on root has some warnings : %v", warns) } } } </del>

Haven't seen this one before and I am not sure it has any relation with the that hit it. https://k8s-
From what I can gather, it is being killed due to taking over 5 minutes which is roughly how long it takes (~300s) on successful unit builds.
bumping priority. I'm seeing this pretty frequently this morning. I see OpenAPI and associate you. You know something about this one?
Yes, I wrote this test. It was failing (timing out) for quite some time on jenkins but passing quite fast locally. I've disabled it but after a while it started passing and I assumed it was a configuration change on test machines. I will spend some time on it again to see if I can find the root cause and will disable it otherwise. This test is passing locally in about 15 second. Do you have any idea why it is timing out (or passing near to 300s) on jenkins machines? where do you see the successful test stats?
I see what you do. Locally it takes 25s, with bots it is 300s. My guess is that we are constrained by CPU and all the encode/decode is causing strain either on CPU or GC. The bots fail about 75% for me so I just keep retrying until success. Just had 2 this morning pass at 296s.
Curiously, it's taking about 20 seconds with Bazel. this might be worth looking at.
I think we run tests in Jenkins with , which might explain the slowness?
Nevermind, we hit it in a bazel run: https://k8s-
-race does kill the performance, 25s -166s which is on par with bazel.
is it possible to disable -race for this test? and should we do that if it is possible?
Now blocking the submit queue.
I think I am missing something here. I've checked some of the attached PRs. Links to the pkg/master folder in master branch has the change (new file called but non of the recent PRs I checked has the file in their branch: That means they didn't sync with master? What am I missing here?
OK. I see, the tests are patching the change of the PR into the master, so everybody should get that if they retest, but it look like my test is not blocking submit queue. there should be another issue.


Question: "k8s.io/apimachinery/pkg/util/version" "k8s.io/klog/v2" <ins> "k8s.io/kubernetes/pkg/kubelet/userns/inuserns" </ins> utilkernel "k8s.io/kubernetes/pkg/util/kernel" "k8s.io/mount-utils" )

Everything 2024-05-24 No response /sig testing
/sig node
kubelet logs dmesg:
Seems a regression in
/cc
/triage accepted /priority important-longterm


Question: } // TODO: needs to check if newMeta==nil && oldMeta !=nil after the repair logic is removed. <del> if newMeta.DeletionGracePeriodSeconds != nil && oldMeta.DeletionGracePeriodSeconds != nil && *newMeta.DeletionGracePeriodSeconds != *oldMeta.DeletionGracePeriodSeconds { </del> <ins> if newMeta.DeletionGracePeriodSeconds != nil && (oldMeta.DeletionGracePeriodSeconds == nil || *newMeta.DeletionGracePeriodSeconds != *oldMeta.DeletionGracePeriodSeconds) { </ins> allErrs = append(allErrs, field.Invalid(fldPath.Child("deletionGracePeriodSeconds"), newMeta.DeletionGracePeriodSeconds, "field is immutable; may only be changed via deletion")) } <ins> if newMeta.DeletionTimestamp != nil && (oldMeta.DeletionTimestamp == nil || !newMeta.DeletionTimestamp.Equal(*oldMeta.DeletionTimestamp)) { allErrs = append(allErrs, field.Invalid(fldPath.Child("deletionTimestamp"), newMeta.DeletionTimestamp, "field is immutable; may only be changed via deletion")) } </ins> // Reject updates that don't specify a resource version if len(newMeta.ResourceVersion) == 0 {

Honestly I'm not sure if this is a bug or a feature, but didn't expect it. Filing this to track.
I think it's a bug. Nothing in the code base depends on it that I can see (edit: namespace deletion does, but only via an in-server call which I'd like to see fixed), and we don't let other delete-related fields mutate during update (deleteGracePeriodSeconds)


Question: .container-windows-$(ARCH): $(foreach binary, ${BIN}, bin/${binary}-${OS}-${ARCH}) docker buildx build --pull --output=type=${OUTPUT_TYPE} --platform ${OS}/$(ARCH)  <del> -t $(IMAGE):$(TAG)-${OS}-$(ARCH)-${OSVERSION} --build-arg BASE=${BASE}:${OSVERSION} --build-arg ARCH=$(ARCH) -f Dockerfile_windows . </del> <ins> -t $(IMAGE):$(TAG)-${OS}-$(ARCH)-${OSVERSION} --build-arg BASE=${BASE}-windows-${OSVERSION}-${ARCH} --build-arg ARCH=$(ARCH) -f Dockerfile_windows . </ins> touch $@ # Useful for testing, not automatically included in container image

The Windows pause images is built using buildkit on Linux hosts and this configuration does not allow for the Windows registry to me modified during the container build process. When Windows containers are built on Windows host machines part of the build process creates 'delta' registry files when get stored in the layer files. These 'delta' registry hive files are applied to the containers active registry when the containers are created. Chaning the registry files in the container image layers without having the 'delta' files present will result in the changes NOT being present in the container image. We have an increasing demand for the ability to modify registry in the pause image on Windows such as Disable some services like DiagTrack that negatively impact performance Change behavior of RPCSS so it doesn't hijack RPC port (135) for all containers in the pod Things are working as expected given the current build process. There are linux utilities like that can modify Windows registry files that I have used to test this out. After running the container produced from these files you can see the registry changes made by persisted in the container image but these changes are not present in the container's active registry. vs One possible solution is to create an intermediate image based on (the current base image for the Windows pause image) that performs all the desired registry operations that is built on a Windows machine and published someplace (MCR maybe?) and the use that as the container base image for the Windows pause image. <detailsN/A </details<detailsN/A </details<detailsN/A </details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc
cc
my first concern was. will a community member be able to build this image themselves (will the sources be available and up-to-date and the image built from it). We could also do a github hosted windows runner to build / publish this new image as well. let's please revisit this once we wrap 1.24.

/milestone v1.25
We created which has the source/docker files used to build . AKS has made a pause image based on image available to some customers out of necessity to unblock some scenarios. We'd like to upstream the changes - and do so with as little destabilization to Kubernetes project as possible.
Seems like the intermediate base image step would be reasonable. I don't think community members can build the existing base windows images either for the same reason of depending on windows?
The windows pause image is currently based on 'nanoserver' which is one of several container base images provided by Microsoft (and only Microsoft can produce container base images).
yep. i'd expect community downstream teams (for example) would like to start with this nanoserver, have a set of dockerfiles to build this new image ... then build their own pause images based on this new image
or in other words if MSFT wants to roll all the changes into nanoserver itself (since we all already know that only MSFT can produce this), then there is no debate after that
We can't roll these changes into nanoserver because they may break other scenarios. Because of that - having an intermediate layer built somewhere was the best solution we came up with without forking the build process for the pause image. Microsoft is willing to host build/host the intermediate layers/images from open-sourced docker files. If we want to build the intermediate layers somewhere else too (like github hosted Windows runners) that would be fine too.
/sig windows
If this intermediate image used only by k8s then it is a good idea to build the image via GH actions in the kubernetes org so that k8s maintainers have better control on this or if this image is going to be generic for anyone to use then MSFT can build one and host it somewhere.
Picking this back up since 1.24 release is behind everyone. - Do you think a separate github repo for the dockerfile and github actions staging/prod container repos would be appropriate here? If so I can get started on creating these. Also - Is it possible to push to the staging repos from a github action job instead of PROW jobs? We'd need to use github actions to build the Windows image on a Windows machine.

I supposed we can store the .tar files as release artifacts if needed and then use a PROW to job push them to the staging repo if needed but being able to push for the action would be simpler.
/cc as FYI
I see two possible paths forward Continue to host the source for the intermediate layer at , publish the image to , and then update the Windows pause build docker files in k/k to consume this image instead of the or Create a new kubernetes-sigs repo to host the source and build steps for the image, publish the image to , and then update the Windows pause build docker files in k/k to consume this image. I'm fine with either approach but am not sure who the right folks to discuss this with are.
This looks more promising because this image you are creating is not a specific change for the pause container, IMO if we can change that image name to something generic and use the same wherever required.
I think those both sound reasonable personally. I think it's technically feasible to push from GitHub actions to staging GCR with workload identity but nobody has done this yet in Kubernetes.