Question: package apimachinery import ( <ins> "bytes" </ins> "context" <ins> "io" "log" </ins> "net/http/httptest" "net/http/httputil" "net/url" <ins> "strings" </ins> "testing" "time" <ins> "golang.org/x/net/websocket" </ins> corev1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime"

Using the Fabric8 client, which defaults to using WebSockets for Watches, I established a Watch on Service resources. After the timeout (implicit or explicit) rather than the WebSocket seeing a close control frame or even an EOF it continued to be open and fulfill ping/pong requests - but no Service events are sent. The WebSocket would close. Create a WebSocket based Watch with the ListOptions timeoutSeconds set on the Service resource along with a steady production of Service events, such as create/delete of a dummy Service. Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2020-10-27T05:15:27Z", GoVersion:"go1.15.0", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2021-01-13T13:20:00Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"} The behavior has been seen in both local minikube and OpenShift distributions based upon Kubernetes 1.20. OS (e.g: ): Fedora 33
/sig api-machinery
Is this the same issue as ?
It could be related, but in that case the client is calling close. In this case we expect the server to initiate the close.
A couple of things were discussed when we triaged this issue: a) we need to involve the Fabric8 team (~java client?) b) We don't always default to Websocket, in fact we discussed discontinuing it c) But the code is still there, so it's allowed for now and we should support it. Not sure what to do next
a) no, it sounds like an apiserver problem, it should be reproducible with a go client in a test. b) If anything, we should standardize on websockets. Brower based clients depend on them. And when/if we get rid of SPDY, websockets are the replacement. This may be relevant to interests.

I didn't see an option to use websockets with the go client the impl is at Is there another code path or go client that will use a websocket for Watches?
Technically I didn't say our go client, just a go client. I just had in mind using the raw websockets library. must be testing the websocket watches somehow?
I adapted to It will simply keep running - with no events after the timeout. If you switch to configmap, it will terminate.
that's a terrific reproducer, thanks!
Yeah, we should absolutely be able to stick that in a unit / integration test somewhere, thanks!
opened in configmap watch works correctly, service watch hits timeout without the server closing the connection
I suspect the issue is in newDecoratedWatcher. Services implement a decorator and configmaps don't.
yup, the request context wasn't propagated to the decorated watcher
Service decorator was in 1.20 in Fix with unit and integration test in Will pick to 1.21 and 1.20 once merged
/triage accepted


Question: f, err = os.Create(pluginExec) require.NoError(t, err) <ins> // TODO: use mock instead of fake shell script plugin </ins> const execScriptTempl = `#!/usr/bin/env bash <ins> echo -n "{ "cniVersion": "{{.CNIVersion}}", "ip4": { "ip": "{{.PodIP}}/24" } }" if [ "$CNI_COMMAND" = "VERSION" ]; then exit fi </ins> cat > {{.InputFile}} env > {{.OutputEnv}} echo "%@" >> {{.OutputEnv}} export $(echo ${CNI_ARGS} | sed 's/;/ /g') &> /dev/null mkdir -p {{.OutputDir}} &> /dev/null <del> echo -n "$CNI_COMMAND $CNI_NETNS $K8S_POD_NAMESPACE $K8S_POD_NAME $K8S_POD_INFRA_CONTAINER_ID" >& {{.OutputFile}} echo -n "{ "cniVersion": "{{.CNIVersion}}", "ip4": { "ip": "{{.PodIP}}/24" } }"` </del> <ins> echo -n "$CNI_COMMAND $CNI_NETNS $K8S_POD_NAMESPACE $K8S_POD_NAME $K8S_POD_INFRA_CONTAINER_ID" >& {{.OutputFile}}` </ins> inputFile := path.Join(testDataDir, binName+".in") outputFile := path.Join(testDataDir, binName+".out")

The upper test heavily flakes in master:
/cc
Hey I think this might have something to do with your changes. Something is definitely a bit off. It probably is a faulty test...
duplicate of
Looks like an occasional failure, and I don't think this is caused by my code change because I run tests for a few times, and sometime it is passed.... Still I can help to locate the root cause.
I revert my code change, test still fails from time to time...
I think i've found out the root cause, I'll pull a request soon, . cc

Which jobs are failing: pull-kubernetes-bazel-test Which test(s) are failing: //pkg/kubelet/dockershim/network/cni:godefaulttest Since when has it been failing: Unsure, it doesn't appear in triage, but I see it regularly in PRs. Testgrid link: This package is the top flake currently Reason for failure: Anything else we need to know: /sig node /sig network
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/assign
The test results have the following three phenomena. b c


Question: wait.Wait() } <ins> // pastBackoffLimitOnFailure checks if container restartCounts sum exceeds BackoffLimit // this method applies only to pods with restartPolicy == OnFailure func pastBackoffLimitOnFailure(job *batch.Job, pods []*v1.Pod) bool { if job.Spec.Template.Spec.RestartPolicy != v1.RestartPolicyOnFailure { return false } result := int32(0) for i := range pods { po := pods[i] if po.Status.Phase != v1.PodRunning { continue } for j := range po.Status.InitContainerStatuses { stat := po.Status.InitContainerStatuses[j] result += stat.RestartCount } for j := range po.Status.ContainerStatuses { stat := po.Status.ContainerStatuses[j] result += stat.RestartCount } } return result >= *job.Spec.BackoffLimit } </ins> // pastActiveDeadline checks if job has ActiveDeadlineSeconds field set and if it is exceeded. func pastActiveDeadline(job *batch.Job) bool { if job.Spec.ActiveDeadlineSeconds == nil || job.Status.StartTime == nil {

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: When creating a job with and , the pod with the configuration included below continued to restart and was never marked as failed: What you expected to happen: We expected only 2 attempted pod restarts before the job was marked as failed. The pod kept restarting and was never set to failed, remained active. How to reproduce it (as minimally and precisely as possible): Create the above job and observe the number of pod restarts. Anything else we need to know?: The flag works as expected when . Environment: Kubernetes version (use ): 1.8.0 using minikube
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I think this is because for , kubelet will try to restart failed pod in place, and the pod status will be even the container is failed to run, so jobcontroller get a illusion as the pod is still running. Here's my naive thoughts: for short-term to avoid this, add a , for long-term we need consider how to recognize and count this kind of Pod as failed pod. cc
Just ran into the issue as well. It might be useful to amend the documentation to warn about the fact that won't be effective if is chosen as restart policy. At least, it wasn't obvious to me and I didn't catch any relevant note in the docs. Happy to do a quick PR if that's a desirable short-term action. For Pods failing as reported in this issue, what's the difference in behavior between a restart policy and a hypothetical scenario where is respected by ? Anything other than failing Pods being replaced by newly created ones in the former case vs. existing Pods being restarted in the latter?
For OnFailure, the failed Pod will restart on the same kubelet host, so emptyDir/hostPath is still there, for Never the failed Pod got delete and a replacement will create on another(may be the same) kubelet host, if the Pod use emptyDir/hostPath to store some tmp data, the tmp data may be lost.

For Pod don't use emptyDir/hostPath, I think there's no much difference for usage( will involve controller, apiserver, scheduler to do some work as it needs to create a replacement, backoffLimit policy will avoid bad jobSpec & container cause to create too many replacement consume control plane bandwidth).
Might it be a reasonable idea to move the parameter into the Pod? I'd think it could prove valuable to have an upper bound on the restarts on Pods themselves independently of Jobs, and it may solve the "how does the Cronjob controller determine that the limit has been reached" problem more elegantly if the Pod's state automatically transitions to failed once the limit is hit. WDYT?
can anyone tell me if I set the Never and the back off to 3 will it try a restart 3 times? this is very confusing
yes, it will. In fact, setting it to is the only way right now to have a back off limit be applied and effective.
I should mention that my statement is based on Kubernetes 1.8. I don't know if any fixes possibly made it into the upcoming 1.9. Then again, if there were, this issue should have been updated or at least referenced.
this is not working for a cronjob job spec (i am running 1.8.6) in the picture i still get new pods when i kill the previous on (5 created 3 should have been created and the job should have failed) any idea how to get this to actually work? !
what's your spec's ? I don't see it in your manifest.
every minute just for testing
Aren't you risking then that restarting jobs from a failed schedule and those from a succeeding schedule might be overlapping? How does it look like when you set the interval to, say, 10 minutes?
same result with 10 minutes interval
I ran a simplified version of your manifest against my Kubernetes 1.8.2 cluster with the command field replaced by so that every run fails, the schedule set to every 5 minutes, and the back-off limit reduced to 2. Here's my manifest: Initially, I get a series of three failing pods: Then it takes pretty much 5 minutes for the next batch to start up and fail: Interestingly, I only get two new containers instead of three.

The next batch though comes with three again: and so does the final batch I tested: Apart from the supposed off-by-one error, things seem to work for me.
Maybe you can double-check using my manifest?
In the we've stated it should work for both restart policies, I've addressed the problem in It counts pod's restartCount, if overall (iow. all containers in all pods) number exceeds backoff limit the job will fail.
I think the backoffLimit doesn't work at all. My kubernetes version is 1.10.3
See .
NAME READY STATUS RESTARTS AGE job-backofflimit-2-9n74x 0/1 Error 0 34s job-backofflimit-2-bt796 0/1 Error 0 1m job-backofflimit-2-fgktc 0/1 Error 0 49s job-backofflimit-2-gfhlx 0/1 Error 0 1m job-backofflimit-2-gzd54 0/1 ContainerCreating 0 8s job-backofflimit-2-ljlpx 0/1 Error 0 1m job-backofflimit-2-p5fxm 0/1 Error 0 1m job-backofflimit-2-w6s5d 0/1 Error 0 21s apiVersion: batch/v1 kind: Job metadata: name: job-backofflimit-2 spec: backoffLimit: 2 template: spec: containers: name: job-with-python2 image: restartPolicy: Never v1.10.4
9 has the same problem. My cluster is 1.9.6, and i run into the same issue


Question: } pluginStatus.SetFailedPlugin(pl.Name()) statuses[pl.Name()] = pluginStatus <del> if !f.runAllFilters { // Exit early if we don't need to run all filters. return statuses } </del> } }

Since scheduler Policy API was total removed, we don't support any more. Remove unsupported configuration.
/sig scheduling /assign
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scheduling /cc
/sig scheduling (trying to tell the bots to remove )
The label(s) cannot be applied. These labels are supported: <detailsIn response to : /remove-needs sig /remove-label needs-sig Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
It's true scheduler doesn't quite use this field anymore, but I'm wondering if some other project like ClusterAutoscaler or simulator still consumes this via .
There's no place we can config this field now by configure file(before we can apply the change by policy api, but now it's removed). So it seems like unless we support this feature with CC, or this field is useless. To other projects, it's the same. When they are using legacy policy api, they can configure this field, but once they synced with the latest kubernetes, with the policy api removed, they can not configure this anymore. In this situation, we should add support in CC. I don't quite understand the relationship between k/k with k/autoscaling or others, I think they sync with k/k periodic.
As I mentioned above, it's still accessible via a framework runtime programmatically, and thus may be still useful. For example, upon a scheduling failure, a ClusterAutoscaler's implementation may choose to dryrun through all filter plugins to ensure a new provisioned node can fit the pod, instead of resolving the failure while failing at another latter filter plugin.
Oh, I haven't thought of configuration via programmatically, that saved my concern. But a little strange.
That's exactly what I meant. Given it's a public option in framework, and there is no alternative if we disabled it, it'd be good to check with potential consumers.

For now I can think of CA ( and k-sigs/scheduler-simulator ( If they can confirm it's not needed in their usecases, I'm fine with removing .
To k/autoscaler, at least these codes are still there together with the legacy policy api. , looking forward to the response.
is not used in sigs/kube-scheduler-simulator.
this isn't used by cluster autoscaler, we can remove it.


Question: hostPath, cleanupAction, err = mounter.PrepareSafeSubpath(mountutil.Subpath{ VolumeMountIndex: i, Path:             hostPath, <del> VolumeName:       mount.Name, </del> <ins> VolumeName:       vol.InnerVolumeSpecName, </ins> VolumePath:       volumePath, PodDir:           podDir, ContainerName:    container.Name,

Fix status: [x] fixed in master in [x] fixed in release-1.10 branch in , will be in 1.10.0 [x] fixed in release-1.9 branch in , will be in 1.9.7 [x] fixed in release-1.8 branch in , will be in 1.8.11 [x] fixed in release-1.7 branch in , will be in 1.7.16 In SubPath fixes, PR adds bind mount for subpaths. It uses as the path name. The volumename in this path is the container's mount volume name which is also called OuterVolumeSpecName in . This becomes an issue for volume reconstruction when kubelet restarts. Volume scans the volume directories and recover the InnerVolumeSpecName (which is used to construct pod volume paths). Volume reconstruction has not enough information to recover the OuterVolumeSpecName and figure out which volume it maps to. In case kubelet needs to clean up mounts (it happens when volume spec cannot be reconstructed), it will fail to clean up the subpaths because of lacking the OuterVolumeSpecName. We think it is better to use InnerVolumeSpecName for volume subPath instead of OuterVolumeSpecName so that volume clean up can work. The issue we might have is that some users might already deploy a cluster with subpath might already . If the users perform an in-place upgrade, since new version will use InnerVolumeSpecName which might have conflicts with some other older version's OuterVolumeSpecName among different volumes. This change requires clear documentation and communication with the users.
cc
This impacts reconstruction of PVCs, where the inner and outer names are different.
/assign
While testing the e2es against a cluster without the fix, I noticed that subsequent pods using volumes failed to start. My theory is that the unmount volume failed because the subpath was still mounted, and that blocked the volume manager from processing other pods. But I thought that volume manager is supposed to run the unmount volume operation in a separate thread per volume, so this shouldn't happen. Need to inspect the code more carefully to confirm the theory. could the volume reconstruction refactor have a unmount volume call in the main volume manager thread?
runs in reconciler's goroutine - so if that function is blocked, it will not be able to process any more volumes.

Was the problem that failed to unmount volume but got hung/stuck ?
I haven't confirmed for sure, but my guess is that unmount volume failed because the subpath was still mounted
moving into the milestone for discussion /milestone v1.10
/priority critical-urgent /status in-progress
/status approved-for-milestone per discussion in sig-release meeting today
[MILESTONENOTIFIER] Milestone Issue: Up-to-date for process <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
Re-opening to track fixes in release branches
keeping in the v1.10 milestone until merges
since this is now fixed in 1.10, I'm taking it out of the milestone so that the release team doesn't keep checking it.
Can this issue be closed now?


Question: } // Force delete objects, but they still may not be immediately removed <ins> t.Log("Deleting test objects") </ins> deleteTestObjects(config.client, config.ns, deleteOption) }

Which jobs are failing: pull-kubernetes-integration Which test(s) are failing: TestVolumeProvision Example: Since when has it been failing: Not sure, and aren't displaying any results /sig storage
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/assign /remove-lifecycle rotten Still seeing this, although not very frequently. Could indicate a race.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
latest failures:
take as an example the test fails b/c is set to PVC , which is not expected. our volume binder tries to dynamically provision it:
as for the test fails for the same reason when the scheduler volume binder tries to find volumes for PVC , its corresponding volume is not synced to volume binder or synced version is not in Available state so the volume binder will try to dynamically provision it too
I think the root cause is that the scheduler fails to find matching PVs in PV Assume Cache for the PVC because the PV in the Assume Cache is not in Available state or does not exist due to event delay. We should wait for the PV/PVC Assume Cache to be up to date before scheduling the pod.

However, it's not possible in integration tests. I think we can wait for the PV to be available or bound. This should reduce failures.


Question: REGISTRY?=gcr.io/google-containers IMAGE?=debian-hyperkube-base <del> TAG=0.3 </del> <ins> TAG=0.4 </ins> ARCH?=amd64 CACHEBUST?=1

Is this a BUG REPORT or FEATURE REQUEST?: What happened: When I update the kubelet on a node from 1.7.x to 1.8.0, pods on that node stop being able to connect to services whose pods happen to be on the same node. I use the Kubenet network plugin. What you expected to happen: Pods should be able to connect to a service even if the services's pods are on the same node. How to reproduce it (as minimally and precisely as possible): a cluster the DNS addon another pod that runs on the same node as one of the DNS pods resolution starts to fail when the service redirects the requests to the pod that is on the same node. Anything else we need to know?: I have some packet captures that I can show. Kubelet: 1.7.5 DNS pod: 10.2.4.76 Origin pod: 10.2.4.64 Kubelet: 1.8.0 DNS pod: 10.2.1.254 Origin pod: 10.2.1.234 We can see that in the requests that fail the DNS IP is correctly DNATed to the DNS POD destination address but the response is not masqueraded, it still carries the DNS POD source (it should have an origin of 10.3.0.10). Environment: Kubernetes version (use ):Cloud provider or hardware configuration: Azure StandardA4 OS (e.g. from /etc/os-release): "Container Linux by CoreOS 1465.8.0 (Ladybug)" Kernel (e.g. ): Linux node-1-vm 4.12.14-coreos SMP Wed Sep 20 22:20:05 UTC 2017 x8664 Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40GHz GenuineIntel GNU/Linux Install tools:
/sig node
The problem appears when transitioning from kubelet v1.8.0-alpha.2 to v1.8.0-alpha.3.
Well, I've found the problem. Starting from v1.8.0-alpha.3 the "modprobe" binary is not included in the image and so the br-netfilter kernel module is not loaded. I'm using the image, although through "kubelet-wrapper" from CoreOS. Where should this binary reside?

cc/ 1.8 release team.
There is a PR from so it doesn't get lost.
[MILESTONENOTIFIER] Milestone Issue Current <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details


Question: "k8s.io/apimachinery/pkg/util/uuid" "k8s.io/apimachinery/pkg/util/wait" "k8s.io/apiserver/pkg/server/healthz" <ins> utilfeature "k8s.io/apiserver/pkg/util/feature" </ins> cacheddiscovery "k8s.io/client-go/discovery/cached" "k8s.io/client-go/informers" v1core "k8s.io/client-go/kubernetes/typed/core/v1"

As per, , wire up metrics/slis for ccm
/assign /sig instrumentation /sig cloud-provider
GitHub didn't allow me to assign the following users: Richabanker. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign /sig instrumentation /sig cloud-provider Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/triage accepted
/remove-triage accepted
/triage accepted


Question: t.Errorf("expected success or bad request err, got %v", err) } }) <ins> t.Run("StrategicMergePatchType should handle a valid patch just under the max limit", func(t *testing.T) { patchBody := []byte(`{"value":0` + strings.Repeat(" ", 3*1024*1024-100) + `}`) err = rest.Patch(types.StrategicMergePatchType).AbsPath(fmt.Sprintf("/api/v1/namespaces/default/secrets/test")). Body(patchBody).Do().Error() if err != nil { t.Errorf("unexpected error: %v", err) } }) </ins> t.Run("ApplyPatchType should handle a patch just under the max limit", func(t *testing.T) { patchBody := []byte(`{"value":` + strings.Repeat("[", 3*1024*1024/2-100) + strings.Repeat("]", 3*1024*1024/2-100) + `}`) err = rest.Patch(types.ApplyPatchType).Param("fieldManager", "test").AbsPath(fmt.Sprintf("/api/v1/namespaces/default/secrets/test")).

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: It returns an error when I create a pod of yaml file over 1M. I checked the codes and found that the coded wrong. It should be “json.Unmarshal(p.patchBytes, &v)”, not “json.Unmarshal(p.patchBytes, v)” in the 344 and 368 line. What you expected to happen: It can be parsed correctly. How to reproduce it (as minimally and precisely as possible): Look at the code and you'll see or you can test it. Anything else we need to know?: no. Environment: Kubernetes version (use ): Community latest code Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/kind bug
/sig api-machinery
/assign


Question: le.maybeReportTransition() desc := le.config.Lock.Describe() if err == nil { <del> glog.V(4).Infof("successfully renewed lease %v", desc) </del> <ins> glog.V(5).Infof("successfully renewed lease %v", desc) </ins> return } le.config.Lock.RecordEvent("stopped leading")

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: Here's an example from an e2e test: What you expected to happen: leader election success logs to be at a higher log level.
Leader election library is not a part of the scheduler. The scheduler is only a user of the library. It is located in client-go. It is currently . I am not sure if it is a good idea to reduce verbosity further. The log messages seem to be excessive in the scheduler logs when the scheduler is idle in our tests.
I think this is a common problem for all components, not just for scheduler.
In that case, we should consider reducing its verbosity to 5 or 6.
For some reason I thought that this was something new but I went back to 1.9 tests and checked the logs and I also see it printed every 2 seconds. So maybe this is fine for test environments.
This is certainly not new, but I am still in favor of lowering it verbosity unless there is a good reason for having it in tests by default.
I'm ok with changing the verbosity but the code now lives in the api-machinery land.
I'd like to pick this up. Could I do that?
The log level seems correct following these guidelines: I am not sold on the value of glog's infinite log levels feature, however I could be convinced that our logging guidelines need a TRACE level.
Thank you for your response. I'm on your side, How about ?


Question: - context: cluster: my-cluster user: mu-cluster <del> name: my-cluser </del> <ins> name: my-cluster </ins> current-context: minikube kind: Config preferences: {}

/kind bug What happened: Current context was set to cluster1. Ran and got back information about cluster1 instead. What you expected to happen: I expected the output to be the cluster2 information How to reproduce it (as minimally and precisely as possible): Have two contexts in your kube config, A and B Run to set A as current context Run Anything else we need to know?: I'd like to help contribute, so if this is an easy fix, someone point me in the right direction and I'll try to get it done. Environment: Kubernetes version (use ): 1.10.2 OS (e.g. from /etc/os-release): MacOS 10.11.6 Install tools: kubectl was installed via brew
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign Filed


Question: n.node = nil n.allocatableResource = &Resource{} n.allowedPodNumber = 0 <ins> n.taints, n.taintsErr = nil, nil n.memoryPressureCondition = v1.ConditionUnknown n.diskPressureCondition = v1.ConditionUnknown </ins> n.generation++ return nil }

Looking into profiles of scheduler, there shows a bunch of things that we are doing unnecessarily. In particular, instead of computing the same information for a given node multiple times, we should cache it as part of NodeInfo: The information that should be cached and reuse later include: [x] node taints: [x] node conditions: disk/memory/inode pressure
Another quick optimization should be optimizing "selectHost" method: In fact it doesn't need to sort the whole list (which is O(nlgn). All we need to do is to choose the elements that have highest score, and sort only those by node name (to leave the logic that we currently have with round-robic logic of choosing from those).
And we should also limit the amount of places where we call "Parallelize" (because those are relatively expensive). The whole MR-like priorities is a step in this direction too.
OK - I've just sent out a PR fixing the first two things.
can we close this, or are there still work items here?
Some of those are not yet done.
- it's not a high priority for us (for now), so if you want it to be done soon, you probably need to find someone to do it.
- assigned.
Why "Parallelize" is considered as relatively expensive ? (comparing to MR style?)
You do MR when there are certain points that need some synchronizations. Or fully parallelized computation will always be faster.
To clarify, I was referring to the Parallelize function: Using this for many very small functions is pretty expensive - you can take a look into cpu-profiles. It happens that 20% or sth of cpu usage is spent on the operations on the synchronization channel.
Is all of the cacheing part completed? I assume "yes". What is remaining here.... Just an audit of parallelization ? Also, I assume "yes". If so I can audit where the parelellization is getting us hung up and then we can close this one out.
It's hard to tell if caching is done - when adding new priority functions/moving existing ones to MR-framework one pretty often discovers that more things need to be cached. There's my PR , that moves basic spreading priority function to MR-framework, which for some reason makes it worse. I didn't have time to look into that though.

This needs to be done, which may require modifying MR-framework slightly (allowing different type as a Map phase result).
Im not sure what the MR Framework stuff is referring to. Can you provide me a pointer ? Should we close this issue since the original patches for caching were done? Sounds like the work we're referring to now is actually more like a second iteration. if not, what work remains to be done before we can close this ticket? maybe we can make a checklist.
"MR-style" parallelization, that was mentioned before. Yeah, I guess we can close this - this is more like an ongoing process thing, when everyone who adds anything to the scheduler needs to keep it in mind.


Question: "k8s.io/kubernetes/pkg/api/v1" "k8s.io/kubernetes/pkg/apimachinery/registered" "k8s.io/kubernetes/pkg/apis/apps" <ins> "k8s.io/kubernetes/pkg/apis/batch" </ins> "k8s.io/kubernetes/pkg/apis/extensions" clientset "k8s.io/kubernetes/pkg/client/clientset_generated/internalclientset" "k8s.io/kubernetes/pkg/client/clientset_generated/release_1_5"

https://k8s- Failed: [] Job should fail a job {Kubernetes e2e suite} Happened on a presubmit run in . Previous issues for this test:
https://k8s- Failed: [] Job should fail a job {Kubernetes e2e suite}
https://k8s- Failed: [] Job should fail a job {Kubernetes e2e suite} Happened on a presubmit run in .
https://k8s- Failed: [] Job should fail a job {Kubernetes e2e suite}
https://k8s- Failed: [] Job should fail a job {Kubernetes e2e suite}
https://k8s- Failed: [] Job should fail a job {Kubernetes e2e suite}
https://k8s- Failed: [] Job should fail a job {Kubernetes e2e suite}
https://k8s- Failed: [] Job should fail a job {Kubernetes e2e suite}
https://k8s- Failed: [] Job should fail a job {Kubernetes e2e suite}
https://k8s- Failed: [] Job should fail a job {Kubernetes e2e suite}
https://k8s- Failed: [] Job should fail a job {Kubernetes e2e suite}
https://k8s- Failed: [] Job should fail a job {Kubernetes e2e suite}
https://k8s- Failed: [] Job should fail a job {Kubernetes e2e suite}
https://k8s- Failed: [] Job should fail a job {Kubernetes e2e suite}


Question: if err := wait.PollUntilContextTimeout(ctx, waitInterval, timeout, true, func(ctx context.Context) (bool, error) { updatedJob, err := clientSet.BatchV1().Jobs(jobObj.Namespace).Get(ctx, jobObj.Name, metav1.GetOptions{}) if err != nil { <del> t.Fatalf("Failed to get updated Job: %v", err) </del> <ins> t.Logf("Failed to get updated Job: %v", err) return false, nil </ins> } actualCounts = podsByStatus{ Active:      int(updatedJob.Status.Active),

pull-kubernetes-integration TestSuccessPolicy/jobwithsuccessPolicywithsucceededCount;jobhasSuccessCriteriaMetandCompleteconditionsevenifsomeindexesremainpending 2024-09-27 No response /sig apps
/assign
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Thank you for letting me know. I will try to investigate the root cause of the flaky case.
It seems that this failure is not related to the test case. It looks like test infrastructure limitations. What do you think?
Maybe it's a sign that we are putting too short of a deadline somewhere?
I think so too. Actually, we can see the same error in as well. So I would propose increasing timeout here: WDYT?
I don't think that would help:
That makes sense. In that case, can we output the log and retry the listing instead of disrupting polling there?
The root cause is disrupting the polling if the listing fails even once.
My proposal is the following:
I see, yeah. But I wonder if the passed to contains any additional timeout. Maybe not.
It seems that we received dedicated timeout as a last arg and then set the timeout to like this: So, I think that it looks better to retry the listing until reaching the polling deadline.
Throughout more investigation, it seems that the returns err immediately once the condition func returns err there:
I simulated the error situation in my local like this:
I opened the PR, PTAL:


Question: assert.Equal(t, tt.expectedPortMappings, actual, "[%d]", i) } } <ins> func TestHashContainer(t *testing.T) { testCases := []struct { name          string image         string args          []string containerPort int32 expectedHash  uint64 }{ { name:  "test_container", image: "foo/image:v1", args: []string{ "/bin/sh", "-c", "echo abc", }, containerPort: int32(8001), expectedHash:  uint64(0x3c42280f), }, } for _, tc := range testCases { container := v1.Container{ Name:  tc.name, Image: tc.image, Args:  tc.args, Ports: []v1.ContainerPort{{ContainerPort: tc.containerPort}}, } hashVal := HashContainer(&container) assert.Equal(t, tc.expectedHash, hashVal, "the hash value here should not be changed.") } } </ins>

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: During the update from v1.7.1 to v1.8.0 all docker containers were restarted, because container spec hash changed. What you expected to happen: No restart of the docker containers, because they did not change at all. How to reproduce it (as minimally and precisely as possible): get yourself a k8s cluster in v1.7.x update (at least the kubelet) to v1.8.0 Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration**: AWS OS (e.g. from /etc/os-release):Kernel (e.g. ):Install tools: Others:
/area kubelet
I ran into this earlier today. I was able to update the control plane & kube-proxy to 1.8.1 without any problems. Once I upgraded the kubelet though it caused container restarts.
/sig apps /sig api-machinery
the log message one receives is at: The function, that calculates the hashes is at: The hash function seems to not have changed. I suspect changes in the object itself, which is at: One can see: but:
But the new field does not initiated, so the restart seems unreasonable.
the result i tested was that the spew pkg also dumps the new add field(with nil value) to the hasher. is the new field. so, if the container use , then new MountPropagationMode will be included into the hash input, the hash result changed. other new fields are in the same situation.
Do you have any idea about preventing this?
We've hit this as well. Caused a cluster-wide restart of every container.
i also hit this issue as well, when I am upgrading cluster from v1.7.4 to v1.8.5. especially, the kubelet version from v1.7.4 to v1.8.5
Still seems to happen in 1.9.0
I can confirm this. v1.9.0 has introduced a new field attribute for , which is causing the hash value get changed.
It will exist when we use docker. I remember had described it.

There are details in this issues below:
thanks for digging out the old issue! Yes, any new field to the container spec would cause the hash to change. This is not a new issue but new fields are to the container spec rarely (compared with pod spec) so it doesn't happen so often. As discussed in , a new way to generate the hash (e.g., using only updatable fields) or other method for kubelet to examine and compare the state of the container against the spec is needed. Calling this (i.e., container restart) out in the 1.9 release note would be nice.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
/close
Just wondering, why don't we calculate the container hash by since container image is the only mutable field? I know we have to do some refactoring to , but wouldn't it be more reliable?


Question: return KeyError{item, err} } keys.Insert(key) <del> if err := f.queueActionLocked(action, item); err != nil { </del> <ins> if err := f.queueActionInternalLocked(action, Replaced, item); err != nil { </ins> return fmt.Errorf("couldn't enqueue object: %v", err) } }

We use a to nil out the : However on a re-sync, this results in a data race that was observed when running unit tests: The read had occurred previously during a callback. On re-sync, the retrieves every object from the cache, invokes the transform function, and re-queues them. However, the object instances are the same as those stored in the cache so mutating them outside the cache without lock protection is unsafe. It is documented that the "TransformFunc sees the object before any other actor, and it is now safe to mutate the object in place instead of making a copy" however this is not the case on a re-sync. The should make a copy of the objects retrieved from the cache store on a re-sync or perhaps elide the on a re-sync since the objects would already have been transformed. As a workaround, I changed our to only nil the if not already nil but this is a latent bug that others will likely run into.
/assign That means that and introduced possible panic regressions into kube-controller-manager and kube-scheduler /priority critical-urgent /kind regression
/transfer kubernetes
/sig api-machinery
kube-controller-manager and kube-scheduler do the same in 1.29+ and likely have the same issue
I think delete has the same problem, and replace may have the same problem: Passing the same object to the transformer after it is already in the cache is unsafe and we should not do that at all.
/assign /cc
Looking into this. I may take time to come up with a proper fix. But would like to evaluate the risk for the current situation. Not entirely sure how delete and replace would be triggered in normal settings, but resync is fortunately low frequency as the default setting for resync period is between 12h and 24h for (--min-resync-period duration Default: 12h0m0s, see ) . As for the resync function is Just think out aloud, I read a thread discussion somewhere that resync was a mechanism in the past to prevent data inconsistency due to bugs but client-go has been mature these days, resync is no-longer needed in theory but for a safe guard. So maybe disabling resync is a reasonable fix if delete/replace can be fixed.
You're mixing resync with relist - I believe relist is no longer needed. Resync is actually used.

FWIW - I opened an initial version of the fix:
I think it's resync. The thread I mentioned is
/triage accepted


Question: return true, nil } <del> if len(limitRanges.Items) > 0 { if limitRanges.Items[0].ObjectMeta.DeletionTimestamp == nil { framework.Logf("deletion has not yet been observed") return false, nil } return true, nil </del> <ins> for i := range limitRanges.Items { lr := limitRanges.Items[i] framework.Logf("LimitRange %v/%v has not yet been deleted", lr.Namespace, lr.Name) </ins> } return false, nil <del> })) framework.ExpectNoError(err, "kubelet never observed the termination notice") </del> <ins> }) framework.ExpectNoError(err) </ins> ginkgo.By("Creating a Pod with more than former max resources") pod = newTestPod(podName+"2", getResourceList("600m", "600Mi", "600Gi"), v1.ResourceList{})

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: Which test(s) are flaking: Testgrid link: Reason for failure: Anything else we need to know: /sig scheduling /priority critical-urgent /cc /milestone v1.19 This test waits for the limit range to be deleted before attempting to create the so this error is surprising. It appears there are no limit ranges present when they are listed here:
It seems this job failed just once since Jul 23, so I'm wondering is it a must for 1.19?
Given this is a conformance test I would prefer we at least know what is causing the issue
This is still flaking with this same issue. /reopen
Reopened this issue. <detailsIn response to : This is still flaking with this same issue. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/milestone v1.20
This issue has not occurred recently and seems to have already been fixed. I think we can close this issue.
Yes, was expected to get it fixed. /close
Closing this issue. <detailsIn response to : Yes, was expected to get it fixed. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
this is still flaking in the 1.20 branch : confused : as of one hour ago


Question: disk, rerr := c.common.cloud.DisksClient.Get(ctx, resourceGroup, diskName) if rerr != nil { <ins> if rerr.HTTPStatusCode == http.StatusNotFound { klog.V(2).Infof("azureDisk - disk(%s) is already deleted", diskURI) return nil } </ins> return rerr.Error() }

The fix from seems to be incomplete/wrong. When disk is not found, the GET call for the disk fails with and it is not even reaching the DELETE call. PV to be deleted (its finalizer to be removed) when the underlying disk is already gone. a PVC. the created Azure disk and delete it manually. , delete the PVC. sure that corresponding PV fails to be deleted with reason: No response <details</details<details</details<details</details<details</details<details</details<details</details
/sig cloud-provider /sig storage /area provider/azure /cc
this issue was fixed in CSI driver: what k8s version are you using now?
19.15 PS: This detail is provided in the issue itself. :)
should fix this issue, once it's merged into master branch, I will cherry pick to 1.23, 1.22, 1.21, and for the other versions, I think you need to patch with this PR by yourself, thanks.
/triage accepted /assign


Question: } cj.Status.Active = append(cj.Status.Active, *jobRef) cj.Status.LastScheduleTime = &metav1.Time{Time: *scheduledTime} <del> if _, err := jm.cronJobControl.UpdateStatus(ctx, cj); err != nil { </del> <ins> updatedCJ, err = jm.cronJobControl.UpdateStatus(ctx, cj) if err != nil { </ins> klog.InfoS("Unable to update status", "cronjob", klog.KRef(cj.GetNamespace(), cj.GetName()), "resourceVersion", cj.ResourceVersion, "err", err) return cj, nil, fmt.Errorf("unable to update status for %s (rv = %s): %v", klog.KRef(cj.GetNamespace(), cj.GetName()), cj.ResourceVersion, err) } t := nextScheduledTimeDuration(sched, now) <del> return cj, t, nil </del> <ins> return updatedCJ, t, nil </ins> } func getJobName(cj *batchv1.CronJob, scheduledTime time.Time) string {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --This started happening when we upgraded to Kubernetes 1.21. The following message occurs: The Cronjob is pretty simple: No error. But in fact, it does appear jobs ARE being removed properly regardless of the error message. Use the above. We tried deleting and recreating the cronjob both with and and neither seems to work. Note the API version was at batch/v1beta1 before we upgraded, and we did change it to batch/v1 to squelch the deprecation notice. Kubernetes version (use ):Cloud provider or hardware configuration: AWS OS (e.g: ): From nodes (master is identical):Kernel (e.g. ): Install tools: Kops Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig apps
i think it's due to the upgrade of your cluster. The error means resource version can not match? what's your cluster version before? your new created cronjob is working well, right? like below:
The previous cluster version was 1.18, and yes I know we did a bad thing by skipping two minor releases. The newly created cronjob is working fine. And strangely, there are no old jobs that spawned - except for the three most recently run, all of them obviously created in the new cluster (we use the defaults here). So jobs ARE getting cleaned up. That's what makes the error messages baffling.
is set to default 3, is set to default 1. see for details.
It's a bug of controllerv2 for cronjob. It passes a cronjob with old resourceVersion to update its status when cleaning up jobs.

The CronJobControllerV2 feature flag graduates to beta and set to be enabled by default in v1.21. So , this started happening when you upgraded to Kubernetes 1.21. I will try to submit a PR to fix it. /assign
We are also seeing this error message in our cloudwatch logs. Some of our cronjobs say that there's an active job running, yet the job referenced has already been completed. Could that be a result of this bug? For example: As you can see it successfully completed. Yet the cronjob lists it as active: Relevant Cloudwatch logs:
seeing the same thing on GKE 1.21.5-gke.1302
hi! any news on that PR? it seems there is a need for rebase
The whole thing with JobAlreadyActive is that it prevents next schedules of a Job, since we have as our main mechanism to prevent overlapping.
Thanks for reminding me. I've updated the PR.
can you please confirm, that this bug is what causing CronJobs to stuck on JobAlreadyActive reason?
I‘m not sure whether it is also a result of this bug since I didn't reproduce that as you described. But according to the code and logs, they seem to be related. Maybe you can retest it when this fixed.
OK, I'll try to take a closer look at it recently.
we have the same thing with Forbid we use helm templates to deploy using Spinnaker so everytime we recreate cronjobs they have the old . I think they automatically get modified when cronjob objects are created to upgrade to I didn't know that objects were automatically modified when they have the old apiVersion. I noticed Ingress objects were also modified. so I'm updating our helm charts to fix this
/cc this is related to cronjob v2 controller
I wrote a bash script to clean up the jobs affected by this bug it looks for active cronjobs then their active jobs checks the actual status of the jobs deletes the jobs that are complete
the fix is merged and backported to 1.21; are you able to use the latest version to test the fix? It is available in 1.21.9, 1.22.6 and 1.23.3
thanks for confirming the patch versions, I wasn't sure what patch version it would come in.

we're on the latest version of EKS (1.21.5). we'll need to wait for it to come to EKS


Question: return claim }(), }, <ins> "good-parameters-apigroup": { claim: func() *resource.ResourceClaim { claim := testClaim(goodName, goodNS, goodClaimSpec) claim.Spec.ParametersRef = &resource.ResourceClaimParametersReference{ APIGroup: goodAPIGroup, Kind:     "foo", Name:     "bar", } return claim }(), }, "bad-parameters-apigroup": { wantFailures: field.ErrorList{field.Invalid(field.NewPath("spec", "parametersRef", "apiGroup"), badAPIGroup, "a lowercase RFC 1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*')")}, claim: func() *resource.ResourceClaim { claim := testClaim(goodName, goodNS, goodClaimSpec) claim.Spec.ParametersRef = &resource.ResourceClaimParametersReference{ APIGroup: badAPIGroup, Kind:     "foo", Name:     "bar", } return claim }(), }, </ins> "missing-parameters-kind": { wantFailures: field.ErrorList{field.Required(field.NewPath("spec", "parametersRef", "kind"), "")}, claim: func() *resource.ResourceClaim {

Following YAML file is accepted without error. Notice the field, it is an instead of , containing character. This can lead to an issue when resource driver generates structured parameters' in-tree object with being just without version. The DRA scheduler plugin will try to match the parameters object based on the reference here: In latest master the code appears to use an indexer, and the issue might be hidden, if applicable at all. I did not check the reproducibility in master yet. However, main question is whether the should allow value with character in general. An error is expected from the apiserver upon object creation request, reporting that APIGroup containes APIVersion. Maybe. Have any resource driver that supports structured parameters, , for instance, also puts (gpucrd.APIGroup) value into . v1.30.1 <details</details<details</details<details</details<details</details<detailsscheduler's dynamicresources plugin </details
/sig node /triage accepted
/assign


Question: <ins> load("@io_bazel_rules_go//go:def.bzl", "go_test") go_test( name = "go_default_test", srcs = ["compatibility_test.go"], deps = [ "//pkg/apis/core/install:go_default_library", "//pkg/scheduler/algorithmprovider/defaults:go_default_library", "//pkg/scheduler/api:go_default_library", "//pkg/scheduler/api/latest:go_default_library", "//pkg/scheduler/factory:go_default_library", "//staging/src/k8s.io/api/core/v1:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/runtime:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/runtime/schema:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/util/sets:go_default_library", "//staging/src/k8s.io/client-go/informers:go_default_library", "//staging/src/k8s.io/client-go/kubernetes:go_default_library", "//staging/src/k8s.io/client-go/rest:go_default_library", "//staging/src/k8s.io/client-go/util/testing:go_default_library", ], ) filegroup( name = "package-srcs", srcs = glob(["**"]), tags = ["automanaged"], visibility = ["//visibility:private"], ) filegroup( name = "all-srcs", srcs = [":package-srcs"], tags = ["automanaged"], visibility = ["//visibility:public"], ) </ins>

PLEASE COORDINATE WITH SIG-SCHEDULING BEFORE WORKING ON THIS ISSUE. We want to avoid the situation where two people accidentally start work on the same fix. Main tracking issue: If you contributed in phase 1, please give others a chance before claiming this issue. You can also help out by reviewing the phase 2 PRs. The tests in ensure backwards compatibility with the types in . It makes more sense for these tests to live in that package. /sig scheduling /kind cleanup /help /good-first-issue
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : PLEASE COORDINATE WITH SIG-SCHEDULING BEFORE WORKING ON THIS ISSUE. We want to avoid the situation where two people accidentally start work on the same fix. The tests in ensure backwards compatibility with the types in . It makes more sense for these tests to live in that package. /kind cleanup /help /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I would like to take it and thanks
Hi! I am contributer and I would like to take it.
while you're working on do you mind letting take a crack at this one?
OK, go head
, thanks!
/lifecycle active for
Thanks for taking care of this one! :smiley:
You are wellcome


Question: "List of allowed origins for CORS, comma separated.  An allowed origin can be a regular "+ "expression to support subdomain matching. If this list is empty CORS will not be enabled.") <ins> fs.StringSliceVar(&s.HSTSDirectives, "strict-transport-security-directives", s.HSTSDirectives, ""+ "List of directives for HSTS, comma separated. If this list is empty, then HSTS directives will not "+ "be added. Example: 'max-age=31536000,includeSubDomains,preload'") </ins> deprecatedTargetRAMMB := 0 fs.IntVar(&deprecatedTargetRAMMB, "target-ram-mb", deprecatedTargetRAMMB, "DEPRECATED: Memory limit for apiserver in MB (used to configure sizes of caches, etc.)")

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Add the "Strict-Transport-Security" header to the API server responses. Why is this needed: Nessus scanner treats this as a Medium Risk in kubernetes 1.18.10.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig api-machinery
/remove-sig api-machinery /sig auth


Question: forwardedURI string redirect     string redirectWant string <ins> reqHost      string </ins> } table := map[string]Item{

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: When accessing a service (e.g., ) by using the Apiserver proxy feature and the service returns with header, such as The returned header does not preserve the path if the service sent a redirect with an absolute path. Depending on the cluster, the header written by the service itself might have been and the apiserver proxy did rewrite the host:port to match the correct values. Unfortunately, it did not rewrite the path segment. What you expected to happen: I would expect the apiserver to rewrite the path segment of the response header so that it reflects the fact that it's proxied under the path: How to reproduce it (as minimally and precisely as possible): The path was not preserved. Anything else we need to know?: To work around this issue, a service may return strictly a relative url in the header such as: The feature to rewrite headers is quite common. For instance, Apache Httpd does it thanks to: Environment: Kubernetes version (use ):Cloud provider or hardware configuration**: custom_ OS (e.g. from /etc/os-release): Kernel (e.g. ):
/sig api-machinery /sig network
/assign
Thanks for the bug report!
GitHub didn't allow me to assign the following users: roycaihw. Note that only can be assigned. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: EphemeralContainersToStart: []int{0}, }, }, <ins> "Create a new pod sandbox if the pod sandbox is dead, init container failed and RestartPolicy == OnFailure": { mutatePodFn: func(pod *v1.Pod) { pod.Spec.RestartPolicy = v1.RestartPolicyOnFailure }, mutateStatusFn: func(status *kubecontainer.PodStatus) { status.SandboxStatuses[0].State = runtimeapi.PodSandboxState_SANDBOX_NOTREADY status.ContainerStatuses = status.ContainerStatuses[3:] status.ContainerStatuses[0].ExitCode = 137 }, actions: podActions{ KillPod:                  true, CreateSandbox:            true, SandboxID:                baseStatus.SandboxStatuses[0].Id, Attempt:                  uint32(1), NextInitContainerToStart: &basePod.Spec.InitContainers[0], ContainersToStart:        []int{}, ContainersToKill:         getKillMapWithInitContainers(basePod, baseStatus, []int{}), }, }, </ins> "Kill pod and do not restart ephemeral container if the pod sandbox is dead": { mutatePodFn: func(pod *v1.Pod) { pod.Spec.RestartPolicy = v1.RestartPolicyAlways }, mutateStatusFn: func(status *kubecontainer.PodStatus) {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: For a Pod with , kubelet would attempt to create a new sandbox for it even after all container succeeded. Logs as below: This issue is similar to which fixed the case of pods whose RestartPolicy are Never. It caused unnecessary CRI and CNI calls, confusing logs and conflicts between the routine that creates the new sandbox and the routine that kills the Pod. It's likely also the root cause of because sometimes the routine that creates the second sandbox and the routine that kills unwanted pod run concurrently, then the netns would be gone when calling CNI. What you expected to happen: kubelet shouldn't create a new sandbox for pod with RestartPolicyOnFailure if all containers succeeded How to reproduce it (as minimally and precisely as possible): Create a pod like this: Anything else we need to know?: Environment: Kubernetes version (use ): v1.18.2 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig node
Created which may fix this.
/assign


Question: return bitMask1.IsEqual(bitMask2) } <ins> func isAffinityViolatingNUMAAllocations(machineState state.NUMANodeMap, mask bitmask.BitMask) bool { maskBits := mask.GetBits() singleNUMAHint := len(maskBits) == 1 for _, nodeID := range mask.GetBits() { // the node was never used for the memory allocation if machineState[nodeID].NumberOfAssignments == 0 { continue } if singleNUMAHint { continue } // the node used for the single NUMA memory allocation, it cannot be used for the multi NUMA node allocation if len(machineState[nodeID].Cells) == 1 { return true } // the node already used with a different group of nodes, it cannot be used within the current hint if !areGroupsEqual(machineState[nodeID].Cells, maskBits) { return true } } return false } </ins>

Dual socket server with 96 threads total (2242), ~192G of RAM, cpu & memory manager static policy, topologyManagerPolicy best-effort, 10Gi of RAM reserved on NUMA node 0, 1 core (2 threads) reserved on NUMA node 0 kubeadm config: If I try to allocate 2 Guaranteed pod with 85Gi of RAM each, one pod is admitted, and the second one fails with even if it would fit using memory of both NUMA nodes. As I'm using deployments a new pod is recreated right away and you end up with tons of failed pod in state. Error in the logs is: Either the pod becomes pending, or memory from both NUMA nodes is used Have some memory reserved on NUMA 0, launch 2 identical pods with memory limits really close to the max so 1 pod fit on NUMA 1 but the second doesn't fit on NUMA 0 (the sum fits on the server) Here the If I launch 1 pod with 46 CPUs / 170Gi RAM, I get CPUs from 1 NUMA nodes, and memory from both NUMA nodes <details</detailsIf I launch a pod with 46 CPUs / 85Gi RAM, it's properly placed on NUMA 1 <details</detailsIf I try to launch 2 pods with 46 CPUs / 85Gi RAM, one fails with If I try to launch 2 pods with 46 CPUs / 80Gi RAM, everything works 26.8 NONE / bare-metal Alma 8.8 base + rpm-ostree kubeadm containerd 1.7.5 No response
/sig node
/cc Hi! Just to clarify, the memory manager can't keep the pod pending, this is a scheduler feature and builtin kube scheduler (= any of its plugin) is not NUMA-aware atm. So in this case memory manager should extend the pod to both NUMA zones, and failing to do so is likely the issue
Hello Is it possible that you are hitting the exact scenario describe at the KEP: ? Please check the first two steps.
I was going to say I'm hitting the expected unexpected error, but I'm not sure the KEP doc 100% match the current memoryManager behavior (at least when using ).

test 1: the "reject pod2 from the KEP" actually works: start pod 60 cpus / 120Gi RAM wait start pod 20 cpus / 20Gi RAM <details</detailstest 2 (start a 1 NUMA node container then one than spans 2 NUMA node) also works: start pod 20 cpus / 20Gi RAM wait start pod 60 cpus / 120Gi RAM <details</detailsWhat doesn't work is when you get too close to the limits it seems
Interesting those pods are guaranteed I presume? And what about the total node memory capacity? When you check the node status do you have enough memory left to fulfill the second pod request?
click on details you will have the content of yes they are guaranteed else they would not show up in memorymanagerstate it's less then the total node capacity else pod would be pending
/assign
assigned to me for bookkeeping reason but is doing the real work here
Hello so far I didn't manage to reproduce the issue you're describing. In my case I manage to ran both pods, or end up with the second one . However, I managed to find something similar (different affinity error) but with topology manager policy and not as you described which I'm going to open a separate issue for. Could you please share the journal logs while you're reproducing the issue again? Basically on the node where the pod is about to land, share the output
One more thing please share output of the node before and after you reproduce the issue
/triage accepted /priority important-longterm
off until Tuesday, I'll try to reproduce with 1.28.2 then and share the logs next week
NP. I won't be here next week either, so no rush
Hello did you manage to reproduce the issue?
yes, reproduced on my dev machine in 1.28.2, tell me what else you need / if you want me to rebuild kubelet with some debug that's also possible <details</details
Hi I did mange to reproduce the issue, but following the code it seems exactly like I mentioned in my previous I'll try to elaborate: When the first pod gets admitted it placed on NUMA 0 (in my case).

When the second pod gets admitted, there is not enough memory on each NUMA individually (NUMA 0 or NUMA1) to satisfy the memory request, so memory manager tries to span the memory allocation through NUMA 0 and NUMA 1. But it failed short in this line: because NUMA node 0 is now used for single-NUMA memory allocation which align with the KEP . In such case, the pod cannot move to because is done at the scheduler level, while from the scheduler's POV there is enough memory on the node to satisfy the request. Now, about your comment here: wait start pod 20 cpus / 20Gi RAM Make sense because both NUMA 0 and NUMA 1 are already serves mulit-NUMA allocation, since the first pod spans across the two NUMAs, the second pod can squeezed into one of them with no problems. wait start pod 60 cpus / 120Gi RAM This case is more interesting and I didn't manage to reproduce it - I always got which is align with the KEP. Could you please increase Kubelet logs verbosity, reproduce test 2 and attached the following information: logs file before 1 manifest file after pod1 2 manifest file after pod2 Thanks
Hi this is a different server with a bit less CPUs, but test 2 still works for me for verbosity I've used v=99 instead of v=2 Here part of my KubeletConfiguration The logs: And the commands: <details</details
I think I pinpointed a bit more the issue For my test2 pod the cpu manager generated a hint with both NUMA nodes. For test2 pod, if CPU request span both NUMA nodes, then the memory allocation will work whether it requires to use 1 or 2 NUMA nodes If CPU request only requires 1 NUMA node, then it'll fail if memory allocation need to extend to 2 NUMA node.
Hello , After carefully going through the code and the KEP I think that the behavior you're expecting: "Either the pod becomes pending, or memory from both NUMA nodes is used" is not aligned with the architecture.

Let's break it down: "The pod becomes pending" - From the default scheduler POV, the node have enough resources (because the scheduler is not NUMA aware) so the fact that the pod scheduled on this node is align with the expected behavior. Sidenote: if you want to enable NUMA aware scheduling you can take a look at "memory from both NUMA nodes is used" - Because NUMA 0 is already serves as single NUMA memory allocation, it cannot serves as multi NUMA memory allocation as well, otherwise the memory pinning is not guaranteed since linux kernel doesn't support multi-NUMA allocation (pinning). What it can do is grouping NUMA 0 with NUMA 1 into a single group but again this is against the original design: Please check and look for "No spare capacity, reject Pod 7" (last example) And you were right, when the hint generated by cpu manager, spans for both NUMAs the pod gets admitted, and that's exactly where actual bug is - the pod should never get admitted. IOW you were suppose get the error: So the fix here should be preventing the pod admission even when the CPU manager hint spans across both NUMAs. Now, in your case, if you still want and need both pods to get admitted you can change the from to because as I mentioned, the memory pinning in your scenario is not guaranteed anyway.
Hello I still think the bug is that memoryManager doesn't respect topologyManagerPolicy: best-effort. For now I'm exploring using a simple NRI plugin to set the to the nodes of
, The best-effort means that even if the affinity is not preferred, the pod would get admit anyway. But here we're talking about respecting the NUMA node grouping done by memory manager. This is an architectural design we should refer to the original creator memory manager or at least to topology maintainers folks. do you know who can answer on such questions?
some additional info about the limitation i mentioned above


Question: ingressPath := buildHTTPIngressPath(hostSplit[1]) ingressRule := networkingv1.IngressRule{} <del> if host != "_" { </del> <ins> if host != "" { </ins> ingressRule.Host = host }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: kubectl create ingress panics because there is no cmdutil.AddDryRunFlag What you expected to happen: Command to work How to reproduce it (as minimally and precisely as possible): Compile the upstream, run ` Anything else we need to know?: Environment: Kubernetes version (use ): upstream/dev /sig cli /kind bug /priority critical-urgent
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: if err = testutils.WaitForPodToSchedule(testCtx.ClientSet, pod); err != nil { t.Errorf("Expected the pod to be scheduled. error: %v", err) } <ins> select { case <-postBindPlugin.pluginInvokeEventChan: case <-time.After(time.Second * 15): t.Errorf("pluginInvokeEventChan timed out") } </ins> if postBindPlugin.numPostBindCalled == 0 { t.Errorf("Expected the postbind plugin to be called, was called %d times.", postBindPlugin.numPostBindCalled) } } <del> postBindPlugin.reset() preBindPlugin.reset() </del> testutils.CleanupPods(testCtx.ClientSet, t, []*v1.Pod{pod}) }) }

Which jobs are flaking: ci-kubernetes-integration-master pull-kubernetes-integration Which test(s) are flaking: TestPostBindPlugin Testgrid link: Reason for failure: Anything else we need to know: links to appreciated: links to specific failures in spyglass appreciated /sig scheduling
It flaked two times recently:


What happened: kube-controller-manager with enabled option --controllers=,tokencleaner is not removing newly created tokens. It works only after kube-controller-manager restart or after editing secret with token. What you expected to happen: It should remove all expired tokens from kube-system namespace, without restarting or touching secrets. How to reproduce it (as minimally and precisely as possible): enable tokencleaner in kube-controller-manager start controller manager create bootstrap token, you can use script wait 5 minutes token is still there restart kube-controller-manager OR edit newly created token token should be gone after a while Anything else we need to know?: It could be related with Environment: Kubernetes version (use ): Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-12-03T20:56:12Z", GoVersion:"go1.11.2", Compiler:"gc", Platform:"linux/amd64"}
Do you see the following log ? thanks
no errors at all
/sig auth


Question: deps = [ "//pkg/apis/core/install:go_default_library", "//pkg/apis/core/v1/helper:go_default_library", <ins> "//pkg/util/mount:go_default_library", </ins> "//vendor/k8s.io/api/core/v1:go_default_library", "//vendor/k8s.io/apimachinery/pkg/apis/meta/v1:go_default_library", "//vendor/k8s.io/apimachinery/pkg/util/sets:go_default_library", <del> ] + select({ "@io_bazel_rules_go//go/platform:linux": [ "//vendor/k8s.io/client-go/util/testing:go_default_library", ], "//conditions:default": [], }), </del> <ins> "//vendor/k8s.io/client-go/util/testing:go_default_library", ], </ins> ) filegroup(

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): No What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): glusterfs, umount Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-01-12T04:57:25Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-01-12T04:52:34Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"} Environment: Cloud provider or hardware configuration: aws OS (e.g. from /etc/os-release): ubuntu 16.04 Kernel (e.g. ): Linux hostname 4.4.0-59-generic -Ubuntu SMP Fri Jan 6 17:47:47 UTC 2017 x8664 x8664 x86_64 GNU/Linux Install tools: terraform/ansible Others: What happened: Kubernetes allows users to delete a glusterfs pvc before it is has been umounted from all kubelets. This leaves orphaned mount points on the kubelet boxes and the kubelets looping trying and remove them.

What you expected to happen: The kubelet should successfully remove the mountpoints. How to reproduce it (as minimally and precisely as possible): Example deployment with with a heketi/glusterfs storageclass. gluster- To reproduce : kubectl create -f gluster- <wait for all containers to come upkubectl delete -f gluster- This will cause heketi to remove the glusterfs volume before some of the pods have umounted it. All mount points still online will error with: 'transport endpoint is not connected' These can still be umounted, however the way the code is written it will try to "stat" the mount point before trying the umount. This will error with something like the following: This will prevent the umount and pod cleanup. Anything else we need to know: The code looks something like this: I realise the importance of checking to see if a mountpoint is still up before removing it. My first thoughts were to restrict the stat checks to only error if the mount point didn't actually exist. Also checking /proc/mounts seems like a more reliable method for checking if something is mounted on linux.
Would prevent this scenario?
[MILESTONENOTIFIER] Milestone Removed From Issue Important: Code freeze is in effect and only issues with may remain in the v1.9 milestone. <details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details


Question: --public-ip="": Name of a public IP address to set for the service. The service will be assigned this IP in addition to its generated service IP. --selector="": A label selector to use for this service. If empty (the default) infer the selector from the replication controller. --service-name="": The name for the newly created service. <del> -t, --template="": Template string or path to template file to use when -o=template or -o=templatefile. </del> <ins> -t, --template="": Template string or path to template file to use when -o=template or -o=templatefile.  The template format is golang templates [http://golang.org/pkg/text/template/#pkg-overview] </ins> Global Flags: --alsologtostderr=false: log to standard error as well as files

periodic-conformance-main-k8s-main Kubernetes e2e suite.SynchronizedBeforeSuite 07/17 23:52PDT We saw a similar issue in , which was fixed by upgrading the control plane OS from Ubuntu 18.04 to 22.04. /sig cluster-lifecycle
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign /cc
/sig node It seems to be related to SWAP.
There is a bug in containerd that is fixed in v1.6.14. The test here is using v1.6.8. Kubernetes v1.27 supported containerd versions: 1.7.0+, 1.6.15+
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Still failing after is merged. Have we tried bumping the control plane OS image to ubuntu 22.04, like ?
Nevermind, the latest testgrid job might have been triggered before the PR was merged. I am observing this and will close this once I see the job passes.
Containerd version is already v1.6.21 in 's log There maybe still other problems.
can fix a similar failure in .
/close The failure of the swap is fixed by . The current failure is a flake that is mentioned in
Closing this issue. <detailsIn response to : /close The current failure is a flake that is mentioned in Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: klog.Errorf("Error preempting pod %v/%v: %v", victim.Namespace, victim.Name, err) return "", err } <ins> // If the victim is a WaitingPod, send a reject message to the PermitPlugin if waitingPod := fwk.GetWaitingPod(victim.UID); waitingPod != nil { waitingPod.Reject("preempted") } </ins> sched.config.Recorder.Eventf(victim, preemptor, v1.EventTypeNormal, "Preempted", "Preempting", "Preempted by %v/%v on node %v", preemptor.Namespace, preemptor.Name, nodeName) }

<!-- Please only use this template for submitting enhancement requests --What would you like to be : When preempting a pod that is a that permit plugins are waiting for , we need send a message(actually send a ) to the , otherwise permit plugins might wait until time out, and do not know the pod has been preempted. Why is this needed: /sig scheduling /kind cleanup /assign
cc
Interesting interaction between the permit and preempt, I will take a look at the PR you sent, thanks!


Question: cfg := credentialprovider.DockerConfig{} if a.config.UseManagedIdentityExtension { <del> if loginServer := parseACRLoginServerFromImage(image); loginServer == "" { </del> <ins> if loginServer := a.parseACRLoginServerFromImage(image); loginServer == "" { </ins> klog.V(4).Infof("image(%s) is not from ACR, skip MSI authentication", image) } else { if cred, err := getACRDockerEntryFromARMToken(a, loginServer); err == nil {

What happened: An aks-engine cluster with a user assigned managed service cluster identity fails to authenticate to azure container registries in private clouds when the proper (acrpull) azure role assignments are in place and when there are no image pull secrets. What you expected to happen: The cluster should be able to authenticate and pull from an ACR hosted in the same azure cloud without needing additional pull secrets when the cluster identity has the proper acrpull azure role assignment, no matter which azure cloud it is operating in. How to reproduce it (as minimally and precisely as possible): Assume you have a private azure cloud where every url ends with "". That would mean the ACR suffix would be . Let's pretend you have an ACR at and you try to deploy a pod in your cluster referencing the image . This will fail to pull when you have no image pull secrets. Anything else we need to know?: The problem stems from the following lines where expected ACR suffix domains are hardcoded: Environment: Kubernetes version (use ): 1.16.9 Cloud provider or hardware configuration: Azure private OS: ubuntu 16.0.4 Network plugin and version: Azure CNI
/sig provider-azure
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig provider-azure Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/area provider/azure /sig cloud-provider
It looks like we may be able to obtain the acr suffix from the azure environment:
there is a security issue here: if we support self defined ACR domain, e.g. , then service principal secret could be sent to that remote server. The reason why we use hardcoded domains is that we could always make sure the following 4 domains are owned by Azure:


Question: } if len(ui.Extra) > 0 { <del> for k, v := range ui.Extra { </del> <ins> sortedKeys := make([]string, 0, len(ui.Extra)) for k := range ui.Extra { sortedKeys = append(sortedKeys, k) } sort.Strings(sortedKeys) for _, k := range sortedKeys { v := ui.Extra[k] </ins> _, err := fmt.Fprintf(w, "Extra: %st%vn", k, v) if err != nil { return fmt.Errorf("cannot write an extra: %w", err)

Unit tests failed with , see TestWhoAmIRun/successtest None No response No response_ /sig cli
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cli
I think it was caused by
When we loop the golang Map, it's unordered.
I can work on this one.
Thanks.
/priority critical-urgent this is impacting CI,


Question: KubeletPodResources:            {Default: true, PreRelease: featuregate.Beta}, WindowsGMSA:                    {Default: true, PreRelease: featuregate.GA, LockToDefault: true}, // remove in 1.20 WindowsRunAsUserName:           {Default: true, PreRelease: featuregate.GA, LockToDefault: true}, // remove in 1.20 <del> ServiceLoadBalancerFinalizer:   {Default: true, PreRelease: featuregate.GA, LockToDefault: true}, </del> LocalStorageCapacityIsolationFSQuotaMonitoring: {Default: false, PreRelease: featuregate.Alpha}, NonPreemptingPriority:                          {Default: true, PreRelease: featuregate.Beta}, VolumePVCDataSource:                            {Default: true, PreRelease: featuregate.GA, LockToDefault: true}, // remove in 1.20

When should it be removed? /sig network /kind cleanup
It was GAed in 1.17 and should have been removed in 1.18. I will send a PR to remove it retrospectively. /assign
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/remove-triage unresolved
I know a new-contributor looking for good-first-issues. Do you mind if I cc him on this issue since it seems pretty straightforward ?
That sounds perfect, thanks!
cc
I am still planning on working on this, I am just having some dev environment issues I am hoping to resolve in the next few days. I'll update again when I get them resolved.
Hey, just a quick update. I have been fairly busy and am going on vacation next week. I AM still planning on working on this, however since I am obviously very slow if someone comes along and wants to work on this feel free. Just please comment that you will work on it so I know :).


Question: return nil, fmt.Errorf("failed to fetch the objectType: %v", err) } result := &unstructured.Unstructured{} <del> err = managedfields.ExtractInto(object, *objectType, fieldManager, result, subresource) </del> <ins> err = managedfields.ExtractInto(object, *objectType, fieldManager, result, subresource) //nolint:forbidigo </ins> if err != nil { return nil, fmt.Errorf("failed calling ExtractInto for unstructured: %v", err) }

Add a code static check for to report using server-side-apply extract With optimization of and , the will be removed in and , and maybe more of other components in the future, using Extract is problematic. It's safer to add this lint check.
/cc
/sig api-machinery
/sig scheduling /sig apps
/assign /triage accepted


Question: var _ = framework.KubeDescribe("Device Plugin [Feature:DevicePluginProbe][NodeFeature:DevicePluginProbe][Serial]", func() { f := framework.NewDefaultFramework("device-plugin-errors") <del> testDevicePlugin(f, true, "/var/lib/kubelet/plugins/") </del> <ins> testDevicePlugin(f, true, "/var/lib/kubelet/plugins_registry") </ins> }) func testDevicePlugin(f *framework.Framework, enablePluginWatcher bool, pluginSockDir string) {

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug This is more of an enhancement. What happened: The Plugin Watcher is setup to trigger on file activities when they occur in the subdirectory that it scans. During startup, the Watcher is passed (from within the Kubelet) the Kubelet's root plugin directory as a location to scan for newly plugins - Because the kubelet's plugins subdirectory is also used by existing in-tree drivers (including csi) as a location to store plugins artifacts, the Plugin Watcher is getting triggered by directory/file activities occurring in that subdirectory that has nothing to do with plugin registration as shown in the output below: What you expected to happen: The Plugin Watcher should not be getting triggered by file/directory activities not related to plugin registration/unregistration. One possibility is to have the Watcher scan one directory below plugins like or something similar How to reproduce it (as minimally and precisely as possible): Run plugin watcher with a CSI driver. cc
/sig node
I think you should only place the plugin socket file in the plugin registration directory but not the plugin data.
agree with we can only place the registration file in the specified path instead of placing all data into it. On the other hand, you can place data into the /var/lib/kubelet/plugins/registration even if registration created, that's a no-end loop.
As I mentioned in issue, existing in-tree volume plugins were already using that directory as a location to store storage operation artifacts (not just csi). Here is a shortened list of driver locations that use /var/lib/kubelet/plugins in their code already: And several more locations. The fix would be to have plugin watcher scan a subdirectory of plugins (say ) instead of plugins directly as it is doing today. The watcher functions as intended. Except now it picks up all directory activities whether related or not to plugin registration.
Well given the fact that the pluginwatcher is beta this is a bit of an issue. My understanding of the "beta" guarantees is that we want to be backwards compatible for all plugins published with the beta API. Meaning that even if we move the registration directory to a new location, we'll need to keep scanning the old directory (to pick up v1beta plugins).

From a 10 000 feet point of view it seems easier to me if we move the in-tree volume plugin's data. Though it might be a lot of work. Another less painful option, but a bit grey is the following: Enforce plugins to end with this suffix is recommended for device plugins (though not enforced) and I believe is a convention on linux. This is a bit grey because we would still be breaking v1beta users but since we're moving a convention from "recommended" to "enforced" it seems "less" breaking to me (and I haven't seen device plugins not using that extension). What do you think ?
I like the suffix idea. It can be a recommendation (which csi is already doing).
Hmm, I've just deployed a new k8s cluster on 1.12.0 and noticed this repeating in the logs on a few of our pods. Just for my own sanity, is this something I should be worried about or can I ignore it until a fix is released?
you can ignore these errors. if it's a recommendation then we'll keep spamming the logs with all the data files. Another thing to keep in mind, it seems like CSI plugins puts a lot of data (and creates a lot of directories). It might be worth considering limiting the recursion to one level. I'm concerned we might exhaust the amount of fds by tracking the plugins data.
yeah CSI keeps track of volume operation metadata in the plugins dir. This probably will not change. I understand this is beta. However, besides CSI and Device plugin, are there any other implementers out there for this feature ? Would it be possible to propose a new location (i.e. <pluginsDir/reg/?
let me know if you open a PR for this as discussed.
thanks for open the issue with detail logs, i can see two problems by looking at logs and code. 1) plugin watcher trying to dial non-socket files. 2) too many logs generated (CREATE and DELETE). I a suggestion to fix of this issue we can fix those problems by: 1) check file mode when handle CREATE event, only attempt to dial file when file mode is socket.

2) increase log verbose level for both 'handleDeleteEvent()' and 'handleCreateEvent()' plugin watcher still has activity when any file create/delete under the directory, but they become light weighted (CREATE file will only trigger file mode check, DELETE file event will trigger plugin searching) i am not sure we want to go further to limit directory depth where plugin can live. it has two drawbacks IMO: 1) it is hard to enforce, not sure everyone will read code/readme. 2) it provides less flexibility to plugins: if want to organize many plugin socket files with directory.
These are definitely good suggestions! It seems like there might be some confusion as to why we are thinking of doing this: Most plugins use the plugin directory to store data (actually a lot of files / dirs) Watching all the directories and files created inside the plugin directory puts more burden on the Kubelet in terms of cpu / memory consumption Watching all the directories and files created inside the plugin directory quickly raises the number of fd consumed By limiting the search of the plugins to one directory, we would be severely limiting the ability of the module to unintentionally harm the overall Kubelet. It seems actually pretty easy to enforce, as it's a simple condition. As for not everyone reading code / readme, we can add the documentation to the gRPC interface if this is a concern. In terms of flexibility, is there a real need to create a plugin in a lower directory? Is there a plugin that has so many sockets that a single directory underneath the plugin directory isn't enough? Naively it seems to me that is enough for the different plugins we enable.
ping on the above :)
Naively it seems to me that $ is enough for the different plugins we enable. i can not speak from plugin developer, but both below looks reasonable to me , or as long as we can communicate to plugin developers clearly and effectively (gRPC interface documentation is needed), the directory depth restriction is reasonable to me.
I like . However, why not put plugintype one level up as . For instance
Is the suggestion to hard code the number of levels that the recursion algorithm will search to 3? Or just expect a socket only at a hard coded level (e.g. socket must be nested 3 deep)? Copying my comment from for consistency: Another option I can think of: recursively scanning but before scanning a directory check if it is mounted ().

This will be backwards compatible and will prevent scanning of volume data Thoughts?
i like this idea better if it is always the case that: volume data is mounted, we can modify the watcher code to not watch the mounted directory in the first place.
Skipping mounted dir along with restructuring search path to known (shallow) dir sounds like a good plan. Though with the new suggested paths, sockets will be placed at a location where mounts should not exist.
I think that would be a safe assumption. The plugin directory is only used as a target path for global mounts.
Ephemeral volume types may not need to mount, it could just be a symlink or plain directory. Can we assume any ephemeral volume CSI driver won't implement global mount?
Yes, in-tree ephemeral volumes today don't do a global mount so the mounts directory is which is not the plugin dir. So no issue there. For CSI the will be the global mount path (this plugin dir). Emphermal volumes shouldn't implement call -- but nothing prevents them from doing so. Maybe let's start with just checking for mount and ignoring mount directories. If it is still an issue we can consider adding a configurable black list of paths. Thoughts?
That sounds good. Also, putting the socket files in a specific location that does not clash with mounts will also help (i.e. )
/milestone v1.13 /priority important-soon
I just want to point out that one of the issue we have with recursively watching all directories is that it will significantly increase the amount of fd consumed. Given that plugins actually store data there and in a dynamic way my fear is that it will exhaust the available fds for no real good reasons. Limiting the depth sounds like a reasonable solution to that issue.
btw, because of this verbosity in my cluster writes 5G logs every day and filled up my rootfs (which is small). Is there any way to decrease this verbosity?
We can do both. Don't search mounted dirs, and max depth of search. Who is working on this?
That's pretty bad, can you open another issue to track that.
we have a PR in flight discussing this issue (recursive scanning) here: Thinking more about this, the fact that too much logs is generated is clearly linked to this specific issue and how we process plugins.

In any case I'll be fixing that issues too
how confident are we in getting this fixed by Code freeze tomorrow? 5pm PST
I expect things to wind up today, changes in the above PR aren't very contentious and don't involve significant changes


Question: pvNamePrefix := ns + "-" pvHostPathConfig := e2epv.PersistentVolumeConfig{ <del> NamePrefix: pvNamePrefix, Labels:     volLabel, </del> <ins> NamePrefix:       pvNamePrefix, Labels:           volLabel, StorageClassName: ns, </ins> PVSource: v1.PersistentVolumeSource{ CSI: &v1.CSIPersistentVolumeSource{ Driver:       csiDriver.Name,

master-informing: ! Last 2 runs failed consecutively, first failure: No response /sig storage
/cc
I will take a look this week.
/triage accepted
/assign


Question: func TestConfigChange(t *testing.T) { setUp := func() (*os.File, string, error) { <del> tempDir := os.TempDir() file, err := ioutil.TempFile(tempDir, "kube-proxy-config-") </del> <ins> tempDir, err := ioutil.TempDir("", "kubeproxy-config-change") if err != nil { return nil, "", fmt.Errorf("Unable to create temporary directory: %v", err) } fullPath := filepath.Join(tempDir, "kube-proxy-config") file, err := os.Create(fullPath) </ins> if err != nil { return nil, "", fmt.Errorf("unexpected error when creating temp file: %v", err) }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: make test is failing due to build failure What you expected to happen: make test should be successful How to reproduce it (as minimally and precisely as possible): clone kubenetes git and run make test Anything else we need to know?: Environment: OS (e.g. from /etc/os-release):Kernel (e.g. ): GO Version: Other:
/sig testing
does help? are you out of space or have write permission problems in your /tmp directory?
this is happening in a clean environment and I hope got enough disk space! do you see golang/linker issues here?
All I see are missing files, not linker errors.
found the culprit, @ test is deleting the which is actually a - which is removing some of the files which are created by go compile process and failing because of this. Other places in kubernetes used and same logic needs to be used here as well.
good catch. yeah, each test should make its own tmp dir with . can you open a PR sweeping for that?
Just wanted to confirm I'm also experiencing the same build failure on HEAD ().
yes, will be out soon for review.
here is the PR to fix this issue -