Question: "k8s.io/klog/v2" <ins> "k8s.io/kubernetes/test/e2e/framework" </ins> "k8s.io/kubernetes/test/e2e_node/builder" "k8s.io/kubernetes/test/utils" )

Which jobs are failing: Which test(s) are failing: The tests are failing in the . Since when has it been failing: 10/23/2020 Testgrid link: Reason for failure: It started failing at commit , but it's unknown if it's related to that. Anything else we need to know: We discussed in the Kubernetes SIG-Node CI subgroup and will start taking a look at it.
/assign
/sig node
Investigation notes: between successful and unsuccessful runs and found no notable differences. , and had a couple of failed health checks on initialization, which doesn't happen in the successful test runs, though there's only 2 seconds between starting readiness checks and an eventual successful check. config the same between runs looking through the kubelet logs for and , but I haven't found anything interesting yet. kubelet- failed health checks The below is a list of errors that occur in runs that succeeded and failed, which may be due to startup or specific testing scenarios. Bunch of these for numerous containers:
Both new and old test runs have a bunch of logs like this, implying that it's been having issues for a while: The failed one has this additional line: However, the doesn't have the same issues. I don't see the connection timeouts at all. I also don't see anything else in the kubelet logs that appears different between successful and failed runs. Unfortunately, I don't know enough about the infrastructure setup, but here are my current theories: about the infrastructure has gotten progressively more overloaded and now it's falling over. The tests run on 8 nodes versus 1, so they have more capacity there. Not totally buying this because why did it go from mostly successful to 100% failures rather than a mix for a bit. changed in the GCE configuration. Not sure how that is managed.
Diff there suggests three merges from k/k Only merge in test-infra is an image bump <img width="215" alt="image" src="https://user-"Interesting that a similar ci version and pull version of the job passes and the one ci version does not. pull: passing ci: failing ci: The failing ones do not have a node_e2e , but I don't see any of the expected log messages for failure of the services to start.
It is interesting that one passes and one does not.

Primary differences are the number of nodes, resource settings, and the fact that the conformance tests use Docker.
In addition to the services log missing, the overall system run logs are also missing. May be related to any decision we make for
Would we be able to kick off a Prow job with custom configuration? In other words, to rule in or rule out something in the image change, can we kick off a job with the latest everything else but use the older image? In reference to the decision on , are you suggesting we pause investigation and see if we decide to just move or change the conformance tests?
/triage accepted /priority critical-urgent
Catching up on this. qq: has anyone vetted ? Ise that pr went in around the time this job began to fail The beforesuite (AI: we should list out what it does), fails to communicate with nodes the mention of 6443 ports looks interesting here.
Good find. Definitely looks suspicious. I'll dig into that.
From : This comes from , which has some node-conformance specific test setup, so maybe something interesting there.
quick interruption for instructions on how to reproduce this one: I think this may work Focus and skip The FOCUS and SKIP arguments im getting from Image config file The job also uses the following config file Make sure to copy it and name it (the argument to IMAGECONFIG_FILE above). Additional arguments The test args to this job are I think these are the defaults already but just in case may be actually useful to add the following to the above command Miscellaneous If you run into trouble with the above commands try going through The info here came from
You'll also need to use the flag to make sure it runs in Docker, I believe.
ah yeah, good catch! Thank you. Don't know about your availability, but if you are all busy ima allocate sometime later today to dive deeper into the info we have here
I'm going to try to spend time of this in the afternoon (PDT). Specifically, I was going to try to verify that the logs from starting up the test suite look as expected and possibly try to run the case to reproduce.

My first suspicion was to have the server running in another port than 6443 (which Kubelet has configured), this is not the case: Only o.SecureServing.BindAddress is being set explicitly, but the port comes from o.NewServerRunOptions().
I think it's related to the movement of this line. I couldn't figure out how to link to the PR so I made a comment.
as you notice before there's no , and we should be seeing Before:
I can repro the issue now, and I think the line was moved upward to generate the token before starting the kubelet. Moving it up out of the if-statement may help partially, but in the conformance test the kubelet is manually started ahead of this code, so it may not matter.
Make sense. Do you mean moving this out of the conformance check?
Haha, yea, that message is deceptive... I think I'll make a PR to at least let it be more correct and give us more info.
for alerts to subscribers, PR created. Please comment.
looking green!
The output looks very strange, I have no idea why. Output has double-underscore-number for the different runs, instead of the image used. [__08] vs [cos-lts]
Ya, I noticed that. It's weird. Not sure how it's set, but you can see that the junit config names have a similar issue: -junitcos- -
Ok, found the root of the problem. Here's the command for the master suite: Note , which is not present in the conformance tests. but not sure how it eventually gets here.


Question: numErrs: 0, }, { <del> name: "invalid port headless", </del> <ins> name: "invalid port headless 1", </ins> tweakSvc: func(s *api.Service) { s.Spec.Ports[0].Port = 11722 s.Spec.Ports[0].TargetPort = intstr.FromInt(11721) s.Spec.ClusterIP = api.ClusterIPNone }, <del> numErrs: 1, </del> <ins> // in the v1 API, targetPorts on headless services were tolerated. // once we have version-specific validation, we can reject this on newer API versions, but until then, we have to tolerate it for compatibility. // numErrs: 1, numErrs: 0, </ins> }, { <del> name: "invalid port headless", </del> <ins> name: "invalid port headless 2", </ins> tweakSvc: func(s *api.Service) { s.Spec.Ports[0].Port = 11722 s.Spec.Ports[0].TargetPort = intstr.FromString("target") s.Spec.ClusterIP = api.ClusterIPNone }, <del> numErrs: 1, </del> <ins> // in the v1 API, targetPorts on headless services were tolerated. // once we have version-specific validation, we can reject this on newer API versions, but until then, we have to tolerate it for compatibility. // numErrs: 1, numErrs: 0, </ins> }, { name: "invalid publicIPs localhost",

In 1.0 and 1.1, headless services that specified a mismatched port/targetPort were tolerated (the targetPort was ignored). tightened validation to reject these services. That causes problems when upgrading from currently-working clusters.
2 candidate to avoid breaking backwards compatibility
oh hell. Yeah. I don't think this is worth a breaking change. We should either roll that back ASAP or change it to nuke any user-provided value and replace it with the value. I think the latter is potentially dangerous and also not worth the risk, so I think we should rollback and just update the docs for targetPort. objections?
Who has time to do this? It doesn't look like an auto-rollback is doable.
Already opened a PR in
Yes, I agree this isn't worth a breaking change, But could we give an warning log that targetPort isn't equal to port and would be ignored?
There's no facility in validation for warning. I don't think spamming the server log in response to API requests is that helpful here
Yes, that sounds reasonable. I just think that targetPort was ignored is a little unfriendly to user.


Question: // Update actual state of world to reflect volume is globally mounted markDeviceMountedErr := actualStateOfWorld.MarkDeviceAsMounted( <del> volumeToMount.VolumeName) </del> <ins> volumeToMount.VolumeName, devicePath) </ins> if markDeviceMountedErr != nil { // On failure, return error. Caller will log and retry. return volumeToMount.GenerateErrorDetailed("MountVolume.MarkDeviceAsMounted failed", markDeviceMountedErr)

We fixed invalid device path in bunch of places via - but call still could use invalid device path. That check can sometimes use invalid path and hence can cause UnmountDevice operation to fails even though device has already been unmounted. /sig storage /sig openstack /assign
Actually this does not affect version 1.10 onwards. Not sure if 1.9 is still supported. I think I will close this for now. /close
I just noticed that 1.9 is still supported and hence it is probably worth fixing there for now.
How come the check can "sometimes" use invalid device path?
So on Cinder for example - call would return something like as attach device path. But if you see the linked PR(), will do a and use that device path for global mount. In general it is always a good idea to use device path returned by as single source of truth. This applies to something like EBS with NVMe too where device is discovered via call (but nvme disks aren't directly affected by this bug because returned device path on EBS is bogus).
So on EBS with NVMe for example - check will always fail (but still return success) because it will use something like as device path which does not actually exist on the node.
You said the issue only happens "sometimes". Can you clarify why it only happens occasionally?
doesn't the linked PR make it clear? detection based on the drive serial number (aka It's cinder ID) on the kubelet itself. As for why - Cinder sometimes returns wrong device path on call, I don't know. But it happens and we have worked around it.
Fixed by


Question: "k8s.io/kubernetes/plugin/pkg/admission/runtimeclass" "k8s.io/kubernetes/plugin/pkg/admission/security/podsecurity" "k8s.io/kubernetes/plugin/pkg/admission/serviceaccount" <del> "k8s.io/kubernetes/plugin/pkg/admission/storage/persistentvolume/label" </del> "k8s.io/kubernetes/plugin/pkg/admission/storage/persistentvolume/resize" "k8s.io/kubernetes/plugin/pkg/admission/storage/storageclass/setdefault" "k8s.io/kubernetes/plugin/pkg/admission/storage/storageobjectinuseprotection"

In 1.30, the PersistentVolumeLabel admission plugin blocks creation of in-tree AzureDisk and vSphere PVs: The reason is that the cloud provider was removed (), but the admission plugin still calls GetCloudProvider there, which fails: The PV should be admitted + created. Enable PersistentVolumeLabel admission plugin. For example, when running local-up- Create AzureDisk PV on any 1.30 cluster, you don't need Azure cloud: /sig storage /priority important-soon <detailsToday's master </details<detailsAzure </detailsNo response <detailslocal-up- </details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I am fixing the worst bug in and I plan to backport it to 1.30. However, we should consider removing the whole PersistentVolumeLabel admission plugin. Most of the cloud providers were removed, only GCE remains and IMO even that one will be removed sooner or later. The admission plugin has no way to get volume zone + region and is therefore useless. The admission said (in 1.26, where we had all in-tree cloud providers): Such webhook never materialized for AWS, Azure nor vSphere and I am not aware of any plans for GCE.
It did materialize in So removing it from k/k is the best way forward.


Question: } // GetRef returns the container reference of the given ID, or (nil, false) if none is stored. <ins> // TODO: move this to client-go v1.ObjectReference </ins> func (c *RefManager) GetRef(id ContainerID) (ref *v1.ObjectReference, ok bool) { c.RLock() defer c.RUnlock()

Is this a BUG REPORT or FEATURE REQUEST? BUG REPORT Kubernetes version (use ): kubernetes/master using CRI with docker What happened: Create a pod that uses a readiness probe that does an exec to command that doesnt exist See the following in logs: What you expected to happen: No error in log, expected the event to properly be thrown. How to reproduce it (as minimally and precisely as possible): See above. Anything else we need to know: Nope.
this is a regression.
debugging, will post a fix once found.
Other reports: ,
-- thanks for the other reports. at least now we have a reliable reproduction.
The problem appears to be the following: has a that correlates containers to a . It uses the type . event recorder library we use in the prober works with . If it doesn't get one of those, it expects the passed in object to implement List. As a result, we are passing a v1.ObjectReference the recorder was not expecting.
The bug may have been introduced in , which was there since v1.6.0.
Opened for a fix.
Keep this open until 1.6 is patched.
/remove-priority P0 /priority critical-urgent (I'm not actually sure this is the right priority, just trying to remove the old priority/PN labels)
Closing since has been merged.

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): Is this a BUG REPORT or FEATURE REQUEST? (choose one): <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): 6 tree at Environment: Cloud provider or hardware configuration: bare metal OS (e.g. from /etc/os-release): Ubuntu xenial Kernel (e.g. ): Linux fab7-compute-2 4.4.0-70-generic -Ubuntu SMP Wed Mar 22 12:47:43 UTC 2017 x8664 x8664 x86_64 GNU/Linux Install tools: kubeadm What happened: This gets logged ] Could not construct reference to: '&v1.ObjectReference{Kind:"Pod", Namespace:"kube-system", Name:"kube-dns--72wvm", UID:"-1edc-11e7-ac00-", APIVersion:"v1", ResourceVersion:"554", FieldPath:"spec.containers{kubedns}"}' due to: 'object does not implement the List interfaces'. Will not report event: 'Warning' 'Unhealthy' 'Readiness probe failed: Get http://10.1.0.130:8081/readiness: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)' What you expected to happen: Should log correctly How to reproduce it (as minimally and precisely as possible): Create pod that fails readiness probe
cc not sure if this is related to your changes
I don't think so. I didn't change the identification of or which are being used there.

I always get the same message. What does this exactly mean?
We're also seeing this on 1.6.1
Seeing this in as well
How we can come to know which release this is fixed? I am using 1.6.6 still I can see this issue. Thanks in Advance.


Question: existingChains, err := proxier.nftables.List(context.TODO(), "chains") if err == nil { for _, chain := range existingChains { <del> if isServiceChainName(chain) && !activeChains.Has(chain) { tx.Flush(&knftables.Chain{ Name: chain, }) proxier.staleChains[chain] = start </del> <ins> if isServiceChainName(chain) { if !activeChains.Has(chain) { tx.Flush(&knftables.Chain{ Name: chain, }) proxier.staleChains[chain] = start } else { delete(proxier.staleChains, chain) } </ins> } } } else if !knftables.IsNotFound(err) {

While validating nftable mode using conformance test, the following error logs were produced consistently: I thought it was because a known issue described here: However, the error was produced for every run and every kube-proxy instance, which doesn't seem related to the kernel bug. After some investigation, I think it's caused by a bug: in some cases a chain could change from stale to active, but once it's to it will always be deleted once. When the proxier tries to delete a previously stale but currently active chain, it would fail and lead to these errors, though it won't cause real problem thanks to kernel's validation. cc kube-proxy shouldn't try to delete active chains and produce such error logs. The e2e test case can reproduce it consistantly: No response <details</details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign I have a PR to fix it.
/sig network /area proxy
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /sig network /area proxy </details


Question: ### Generate certificates <del> First generate a self signed rsa key and certificate that the server can use for TLS. </del> <ins> First generate a self signed rsa key and certificate that the server can use for TLS. This step invokes the make_secret.go script in the same directory, which uses the kubernetes api to generate a secret json config in /tmp/secret.json. </ins> ```sh $ make keys secret KEY=/tmp/nginx.key CERT=/tmp/nginx.crt SECRET=/tmp/secret.json

I follow to create a nginx example. And I got stuck at the beginning.
I'll send a fix which will supersede .
that would be great! thanks, I just got lost handling other 1.1 stuff and forgot about that pr
Sent , PTAL. Thank you.


Question: // Convert errors into failures to catch timeouts. return probe.Failure, err.Error(), nil } <ins> if _, ok := headers["User-Agent"]; !ok { if headers == nil { headers = http.Header{} } // explicitly set User-Agent so it's not set to default Go value v := version.Get() headers.Set("User-Agent", fmt.Sprintf("kube-probe/%s.%s", v.Major, v.Minor)) } </ins> req.Header = headers if headers.Get("Host") != "" { req.Host = headers.Get("Host")

Grabbed tcpdump of a readiness probe - the src ip was the bridge address (10.244.1.1), but apart from this, there is no identifying information in the HTTP request that this is a Kubernetes daemon requesting readiness checks - this might help QoS/priority assignment by the container application.
IIUC, you mean we should add a better when kubernetes daemon request a readiness check to identify this request comes from kubernetes daemon?
Yes, we should override the default net/http user-agent with our own. Better usability for users and we have no future upgrade hazards if users start hardcoding and expecting this default User-Agent string in their analytic tools.
Any news about this one? It would be nice! I've this little hack in my nginx configuration (in order to not be spamed by health check in logs) :)
This can probably be closed now that probes have support.


Question: srcs = [ "aggregator.go", "apiextensions.go", <del> "plugins.go", </del> "server.go", ], deps = [

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: The current help message from kube-apiserver is wrong. It shows only the 'NamespaceLifecycle' plug-in is available, and the default value is . Users cannot know what are the available plug-ins, and they get confused where comes from. What you expected to happen: The correct output should be this: How to reproduce it (as minimally and precisely as possible): Run using master or at least 1.7.2. Anything else we need to know?: This is a regression introduced when refactoring code into generic server code. Environment: Kubernetes version (use ): 1.7.2+ ? Cloud provider or hardware configuration**: Any OS (e.g. from /etc/os-release): Any Kernel (e.g. ): Any Install tools: Any Others: None
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Yes, it is really exist. If no guys works on this, i will pick it up.
Thanks for confirming this. I'm already working on it. Will submit pr in minutes.
cool. :)


Question: key := getStableKey(pod, container) if backOff.IsInBackOffSince(key, ts) { if containerRef, err := kubecontainer.GenerateContainerRef(pod, container); err == nil { <del> m.recorder.Eventf(containerRef, v1.EventTypeWarning, events.BackOffStartContainer, "Back-off restarting failed container") </del> <ins> m.recorder.Eventf(containerRef, v1.EventTypeWarning, events.BackOffStartContainer, fmt.Sprintf("Back-off restarting failed container %s in pod %s", container.Name, format.Pod(pod))) </ins> } err := fmt.Errorf("back-off %s restarting failed container=%s pod=%s", backOff.Get(key), container.Name, format.Pod(pod)) klog.V(3).InfoS("Back-off restarting failed container", "err", err.Error())

Add back-off restarting failed container name in the event message, like: When there are multiple containers in a pod, we use to show the event, it is not showing the failed container name, but it is useful and needed.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
Maybe we can get the container information from upper, including State, Last State...
Yes, we can see more info on the upper part, but it is more straight-forward to see the container name in the event msg, especially when there are multiple containers in a pod.
I think it is a valuable feature.


Question: externalInformers.Core().V1().Pods().Lister(), ) serviceAccountTokenAuth := serviceaccount.JWTTokenAuthenticator([]string{serviceaccount.LegacyIssuer}, []interface{}{&serviceAccountKey.PublicKey}, nil, serviceaccount.NewLegacyValidator(true, serviceAccountTokenGetter)) <del> authenticator := union.New( </del> <ins> authenticator := group.NewAuthenticatedGroupAdder(union.New( </ins> bearertoken.New(rootTokenAuth), bearertoken.New(serviceAccountTokenAuth), <del> ) </del> <ins> )) </ins> // Set up a stub authorizer: // 1. The "root" user is allowed to do anything

While moving APF to beta (see comments on ), we discovered that the integration tests commonly present users which are not members of any groups. We had thought that the system enforced membership in either the authenticated or unauthenticated group. If this is supposed to be a system invariant (as we thought), we'd like to ask SIG Auth: Please fix the integration tests Please add e.g. logging so that if real, operational clusters hit this there's at least a log message and people will file bugs. /sig auth
Can I suggest also, if this is a real invariant, that it be documented in some place(s) likely to be read?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale /priority important-longterm /triage accepted
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/lifecycle frozen
fixes the authenticator configured in one of the integration test API server startup helpers which resolved this issue for . Remaining tests that still need fixing that are encountering unexpected users look to be ones set up to specifically test auth paths:
open to fix remaining integration test users


Question: action.Value = opts _, _ = c.Fake.Invokes(action, &v1.Pod{}) <del> return &restclient.Request{} </del> <ins> fakeClient := &fakerest.RESTClient{ Client: fakerest.CreateHTTPClient(func(request *http.Request) (*http.Response, error) { resp := &http.Response{ StatusCode: http.StatusOK, Body:       ioutil.NopCloser(strings.NewReader("fake logs")), } return resp, nil }), NegotiatedSerializer: scheme.Codecs.WithoutConversion(), GroupVersion:         podsKind.GroupVersion(), VersionedAPIPath:     fmt.Sprintf("/api/v1/namespaces/%s/pods/%s/log", c.ns, name), } return fakeClient.Request() </ins> } func (c *FakePods) Evict(ctx context.Context, eviction *policy.Eviction) error {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: I tried to use  in a unit test, however I get a nil pointer panic when is invoked. What you expected to happen: would return a that doesn't panic on any method call. Ideally, it would allow me to interact with the client in someway for testing. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: client-go version: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig testing
I believe this also impacts returning logs via a reactor as does not implement
Seems to be a problem with . Since we're creating the request with all empty fields. Not sure what the best way it to resolve this. The best thing I came up with is to use to return a request with everything set up. But I'm a noob here so that's just a shot in the dark :man_shrugging:
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
The request returned has a nil which is used in the
I have filed a PR trying to fix this issue.


Question: return nil, err } <ins> var metricsProvider volume.MetricsProvider if plugin.capabilities.SupportsMetrics { metricsProvider = volume.NewMetricsStatFS(plugin.host.GetPodVolumeDir( pod.UID, utilstrings.EscapeQualifiedNameForDisk(sourceDriver), spec.Name())) } else { metricsProvider = &volume.MetricsNil{} } </ins> return &flexVolumeMounter{ flexVolume: &flexVolume{ driverName:            sourceDriver,

kind feature What happened: Currently there is no mechanism to report metrics as the flex plugins are not enabled to collect metrics. All flex volumes lack this feature as compared to in-tree/built-in drivers even though it is possible to collect such metrics for most of the plugins. What you expected to happen: Depending on the availability of the metrics from the mounted FS, flex plugins should allow to report metrics for the FS. How to reproduce it (as minimally and precisely as possible): The default behavior that can be proved easily by just running a command similar to the following command on a node that is running a flex volume. This will show that there are no metrics collected for flex volumes. kubectl proxy $curl 127.0.0.1:8001/api/v1/nodes/<IP/proxy/stats/summary Environment**: Kubernetes version (use ): 1.12 Cloud provider or hardware configuration: local x86 OS (e.g. from /etc/os-release): Ubuntu 16.04 Kernel (e.g. ): Linux KubeStorage 4.4.0-130-generic -Ubuntu SMP Thu Jun 14 08:53:28 UTC 2018 x8664 x8664 x86_64 GNU/Linux Install tools: Others: /sig storage
+1 Yes, it will be good, if flex volume can support atleast the default metrics( i.e ), which most of the volume plugins support now. Based upon flex plugin "capabilities" parameters, it can be decided, whether the plugin supports "default metrics" or "nil metrics".


Question: klog.V(6).Infof("Optional subsystem not supported: hugetlb") } <del> manager, err := cgroupfs2.NewManager(cgroupConfig, cgroupConfig.Path, false) </del> <ins> manager, err := cgroupfs2.NewManager(cgroupConfig, filepath.Join(cmutil.CgroupRoot, cgroupConfig.Path), false) </ins> if err != nil { return fmt.Errorf("failed to create cgroup v2 manager: %v", err) }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: is created on a cgroup2 host What you expected to happen: Such directory should not be created How to reproduce it (as minimally and precisely as possible): Boot Ubuntu 20.04 with Install moby/moby with containerd/containerd and opencontainers/runc Install kubernetes-sigs/kind . This version uses as the base image. Run with Run Run to drop into the shell of the node The following directories are unexpectedly created. (No process seems to belong to these groups actually) Environment: (See above)
cc
/sig node


Question: var failureMessage string jobHaveNewFailure := failed > job.Status.Failed <del> exceedsBackoffLimit := jobHaveNewFailure && (int32(previousRetry)+1 > *job.Spec.BackoffLimit) </del> <ins> // new failures happen when status does not reflect the failures and active // is different than parallelism, otherwise the previous controller loop // failed updating status so even if we pick up failure it is not a new one exceedsBackoffLimit := jobHaveNewFailure && (active != *job.Spec.Parallelism) && (int32(previousRetry)+1 > *job.Spec.BackoffLimit) </ins> if exceedsBackoffLimit || pastBackoffLimitOnFailure(&job, pods) { // check if the number of pod restart exceeds backoff (for restart OnFailure only)

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: for a Job does not work on Kubernetes 1.10.0 What you expected to happen: can limit the number of time a pod is restarted when running inside a job How to reproduce it (as minimally and precisely as possible): Use this resource file: If the job was created in Kubernetes 1.9, it will soon fail: While creating the job in Kubernetes 1.10, it will be restarted infinitely: Anything else we need to know?: Environment: Kubernetes version (use ): 1.10.0 Cloud provider or hardware configuration: AWS OS (e.g. from /etc/os-release): Ubuntu 16.04 Kernel (e.g. ): Install tools: Kubeadm Others:
/sig scheduling
/sig job /sig lifecycle /sig apps
/sig testing
We just updated to 1.10 and seem to be affected by this, too.
I had to add activeDeadlineSeconds to all Jobs and Hooks, as workaround...
Can confirm with 1.10.1
It seems to be introduced by cc
Cause appears to be that the method gets exactly one attempt to update the status with the failed condition and if it fails due to resource conflict, it never adds the condition. The controller code seems to use requeues to track failures, but if a sync loop executes successfully, it would lose track of the failed item ever having been requeued. Looking into a fix now.
Looking at , we force an immediate resync when we see a pod that has failed which also clears the key in our queue, but that in turn makes us lose state of the number of requeues.
This bug killed my test cluster over the long weekend (with + pods). Luckily we don't yet use CronJobs in prod.
killed my cluster too :( still existing in kubernetes 1.10.2
This issue also can't be mitigated as terminated are not part of the resource quota:
This is a pretty terrible bug and it definitely still exists in v1.10.2
/priority important-soon
can you please take a look?
Looking right now...
I've opened to address the issue. Sorry for the troubles y'all.
pulled the patch into current 1.10.2 release and it works! Thanks :)
I filed (with and a test for Job with backoffLimit=1).

The test exposes some other problems. Here are two failure mode: never retries (): the code uses to see the number of retries. However, the job can fail to be synced just because of an update conflict (). That shouldn't be seen as a retry. retries more than needed (): Job controller creates more pods than needed because it doesn't realize itself has failed and exceeded backoffLimit earlier. This may be due to stale cache or that the job status failed to be updated earlier. I think Job backoffLimit is still broken.
FWIW,(assuming I'm not missing something) I can confirm on GKE 1.10.2-gke.3 (latest as of Jun 2nd) the is still broken. The following job spec endlessly creates pods:
we used the latest v1.10.3 release but backoffLimit is still broken
This issue prevents us from rolling 1.10 to all clusters
/milestone v1.11 Looks like we have a couple of PR(s) in progress and feels like we should get this fixed by the time we ship 1.11, please let me know if this fine or not.
I think that this issue should be fixed in 1.10 regardless of 1.11 as this issue causes serious cluster stability, for a fairly common scenario. Specifically for us, migration to 1.11 adds complexity and risk due to the change in the way nvidia GPUs are managed, so it will take us some time to upgrade to 1.11.
y, guess my question is ... is it fixed in master? (and hence already fixed for 1.11), if not, do we need to get this fixed in master/1.11 and backport to 1.10
Unless I'm mistaken, the objects are considered stable APIs right? (since they are under the group). If so, I also think this should be fixed in the next 1.10 point release. In general, like we would also rather to be able to use s without being forced to upgrade to 1.11.
This is a regression in functionality, and when the proper fix is determined, it should be cherry picked back to 1.10. We need it fixed in master first, however. I'm going to escalate this to sig-apps to see if we can get the right resources.

cc: This will need to merge into master, and then be cherry picked back to 1.10
PTAL
The fix is ready for review and will be cherry-picked for the next 1.10 release.
Reopening for cherry pick.
[MILESTONENOTIFIER] Milestone Issue: Up-to-date for process <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
I didn't see anything in the Changelog for v1.10.4, so i'm guessing this still hasn't been fixed yet?
The fix has landed to master but not to 1.10 yet. See It will get hopefully merged before the next 1.10.x release.
Having the same issue. Waiting for update.
This fix has been cherry picked back to 1.10, and should show up in the next patch release (1.10.5).
Any ETA for 1.10.5 yet? Even a ballpark date would be fine.
I'm planning to cut 1.10.5 tomorrow, assuming all tests will be green. In case of any last minute problems it may slip by a day or two, but so far everything looks good for tomorrow.
Thankyou to all involved! I really appreciate this being rolled back into 1.10.
Given that the fix is already on 1.11 and on 1.10 branches, this issue should be closed. Thanks for fixing this!
/close
I don't believe this is fixed? Unless I'm doing something wrong.. Both resource specs have a and . This is in GKE,
I have the same bug on 1.9, with an openshift cluster: Any plan on backporting it to 1.9 as well?
you likely are encountering a different issue, as the root cause of this bug was introduced in kubernetes 1.10, not 1.9.

Also, kubernetes 1.9 is now outside of the support window, so no fixes will be backported to the upstream branch.
I also find this bug in kubernetes 1.12.1 Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-10-05T16:46:06Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-10-05T16:36:14Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"} In pod failure condition,when i set parallelism 2, backofflimit 0,it will destory the last failure pod,only left one pod remain;And set parallelism 2, backofflimit 1,sometimes it keeps 3 pods,sometimes it kills the last failure pod remains 2
@ kubectl version: Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-11-26T12:57:14Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-03-26T16:44:10Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"} minikube version: v0.30.0 apiVersion: batch/v1 kind: Job metadata: name: whalesay1 spec: template: spec: containers: name: whalesay image: docker/whalesay command: ["cowsa", "This is a Kubernetes Job!"] restartPolicy: Never backoffLimit: 2 NAME READY STATUS RESTARTS AGE pod/whalesay1-6lt4b 0/1 ContainerCannotRun 0 59s

pod/whalesay1-86tlz 0/1 ContainerCannotRun 0 48s pod/whalesay1-8v8nr 0/1 ContainerCannotRun 0 1m pod/whalesay1-8w9k2 0/1 ContainerCannotRun 0 1m pod/whalesay1-8xxtr 0/1 ContainerCannotRun 0 53s pod/whalesay1-dl4l8 0/1 ContainerCannotRun 0 1m pod/whalesay1-fhlhg 0/1 ContainerCannotRun 0 37s pod/whalesay1-fz8mv 0/1 ContainerCannotRun 0 26s pod/whalesay1-g2mt6 0/1 ContainerCannotRun 0 9s pod/whalesay1-hgj95 0/1 ContainerCannotRun 0 1m pod/whalesay1-hrsrd 0/1 ContainerCannotRun 0 1m pod/whalesay1-hzm42 0/1 ContainerCannotRun 0 1m pod/whalesay1-mvctw 0/1 ContainerCannotRun 0 1m pod/whalesay1-n7zq9 0/1 ContainerCannotRun 0 2m pod/whalesay1-nl89l 0/1 ContainerCannotRun 0 31s pod/whalesay1-nqcmg 0/1 ContainerCreating 0 2s pod/whalesay1-p85rm 0/1 ContainerCannotRun 0 2m pod/whalesay1-pjzmd 0/1 ContainerCannotRun 0 2m pod/whalesay1-q2k7p 0/1 ContainerCannotRun 0 43s pod/whalesay1-rhvz7 0/1 ContainerCannotRun 0 1m pod/whalesay1-rzrtg 0/1 ContainerCannotRun 0 1m pod/whalesay1-xp5tr 0/1 ContainerCannotRun 0 1m pod/whalesay1-zn2s6 0/1 ContainerCannotRun 0 1m pod/whalesay1-zq4g9 0/1

ContainerCannotRun 0 14s pod/whalesay1-ztxdv 0/1 ContainerCannotRun 0 19s NAME DESIRED SUCCESSFUL AGE 1 0 2m
Just update to something newer than 1.10.0. That version is ages old and I'm sure it is fixed within the latest 1.10.x release. (Which is also out of support so update to 1.12/1.13 ASAP.)
In version "v1.10.11" the though set to 4 for a Job's pod, the pod eviction happens for 5 times and at few times it was for 3 times. Also to note that we also have applied and Below is the version details of Kubernetes that we are using
Though we have set the backoffLimit of a Job to 4 the pod eviction is still being done more than 4 times i.e. 5 times at one of the instances its three times as well. The eviction of the pod happens for a geniune reason i.e. the disk space of the pod node exhausts when the application within the pod executes, we know the reason why that's happening. The point is that the Job's-Pod should be attempted 4 times and not 5 times. Please note that strangely we don't get to see this error in another environment where we have v1.10.05. providing the version details of that environment below There the pod eviction happens as per the stated literature of Eviction with BackOffLimit. We have no issues there. Now when we have upgraded to v1.10.11 recently [ About two days back ] we are getting the reported problem. Would appreciate if someone in this forum can advise on the root cause and a solution. Do we have any flaw with version "v1.10.5" like security loophole.
10.11 is required to avoid a critical security bug. I imagine we will need to update to fix this, but we will need to find out which of the patched versions has the fix we need. We can then roll this through the environments as a staged update.
As noted above 1.11+ are supported by the community until 1.14 is out:
I experienced the same behavior with . Can somebody confirm that this not happening with = versions?

We are experiencing the same issue with version 1.11.9-docker-1. If it makes any sense we have 2 enviroments. One is Ubuntu 18.04.2 LTS and the other is CentOS Linux release 7.5.1804. It happens only at the Ubuntu enviroment.
In meantime we updated cluster: We still experiencing the issue.
Hi everyone! This issue is for a specific problem in Kubernetes 1.10. Right now, the upstream Kubernetes project supports the following three versions: 1.13, 1.14, 1.15. If you are experiencing a similar issue under any of those three versions, then please open a new issue and provide all the details requested. As the specific bug in this issue is resolved and on an unsupported version, I'm going to lock this closed issue. Thanks!


Question: info, err := dockertools.GetDockerPodInfo(kl.dockerClient, spec, podFullName, uid) if err != nil { <del> glog.Infof("Query docker container info failed with error: %v", err) return podStatus, err </del> <ins> // Error handling glog.Infof("Query docker container info for pod %s failed with error (%v)", podFullName, err) if strings.Contains(err.Error(), "resource temporarily unavailable") { // Leave upstream layer to decide what to do return podStatus, err } else { podStatus.Phase = api.PodPending podStatus.Message = fmt.Sprintf("Query docker container info failed with error (%v)", err) return podStatus, nil } </ins> } <ins> // Assume info is ready to process </ins> podStatus.Phase = getPhase(&spec, info) for _, c := range spec.Containers { containerStatus := info[c.Name]

I have a cluster with ~130 pods. Running back-to-back 'get pods' shows me wildly different understandings of their state, even though I can tell from docker that they have not crashed or anything. thockin kubernetes master /(NF)}' uniq -c Running: ./cluster/../cluster/gce/../../output/dockerized/bin/linux/amd64/kubectl get pods -l app=hostnames 127 Running 1 STATUS 6 Unknown thockin kubernetes master /(NF)}' uniq -c Running: ./cluster/../cluster/gce/../../output/dockerized/bin/linux/amd64/kubectl get pods -l app=hostnames 82 Running 1 STATUS 51 Unknown Nothing changed in between these calls.
cc I blame the pod cache.
has recently changed pod cache? FWIW, I also observed this behavior: 19 pods Running in one call and all Unknown in the next. With 0.10.0.
Yes, I think PR on pod cache should fix the problem reported here. Also it should fix the issue reported by about adds-on services. I have a pending PR to have kubelet deciding on PodStatus too.
Could this have caused ?
I don't see how. I'll comment on that issue.
I don't think this could be the cause of that one.
I don't think PR on pod cache will fix this. That one clears the node status. This issue is with pod going from Running to Unknown even when nodes keep running just fine.
Totally consistent observation based on my personal experience. I just assumed we were shooting for some kind of Hesinberg Uncertainty Principle. S , Tim Hockin wrote:
Agree with clears node status only. One possibility is that we hit transient network problem in between the calls that fails health check, so controller set node condition to not reachable, hence unknown pods. I'll make this more robust anyway. Other than that, it's hard to think of a reason that node status results in the flakiness.
In the absence of node information, the pod status defaults to Unknown. Maybe there's another source of this specific problem, but the pod cache is a source of continuous problems, partly because it's not just a cache, but is implementing or duplicating a significant chunk of system logic in a half-baked manner. My goal is to eliminate it -- see .

Short of that, we need to at least implement , eliminate nodes from the pod cache, and implement a real node cache.
I can look into this. Repro info from Tim, who said he was running on a one-node cluster but has seen the problem with multi-node clusters also: run-container hostnames --image=kubernetes/servehostname --replicas=1 --port=9376 --labels=app=hostnames expose hostnames --port=80 --container-port=9376 --selector="app=hostnames" resize --replicas=150 rc hostnames but I had to hack the node a bit to make it get past 100 on the node: sysctl -w fs.inotify.maxuserinstances=8192 echo "DOCKERNOFILE=" /etc/default/docker; /etc/init.d/docker restart (fails sometimes, dunno why)
I can verify that the situation is rather bad. Lots of fluctuation in the numbers of Pending, Running, and Unknown. serve_hostname containers on the node are exiting; I'm not sure why, or if the pod itself is terminating or just the container. Lots of the following in the log (not sure what this means yet): Sat, 07 Feb 2015 00:06:22 +0000 hostnames-ecd2r Pod failedScheduling {scheduler } Binding rejected: 501: All the given peers are not reachable (Tried to connect to each peer twice and failed) [0]
The pause container is also exiting, so I'm guessing that means the whole pod is exiting, not just serve_hostname...
Ugh, there is an etcd size limit you might be hitting too, I forgot. If you are seeing the fluctuations, maybe you got far enough, though. On Feb 6, 2015 5:13 PM, "David Oppenheimer" wrote:
Just a few more observations: This is reproducible (and a little less chaotic) with one node and 100 pods (but not with 50 pods, so it's definitely load-dependent). You can get to 100 Running pods (0 Pending or Unknown), and then a minute later a dozen or so will become Unknown, and then later go back to Running, etc. As mentioned earlier, the containers continue running on the node all this time (sudo docker ps wc -l), so it is definitely some kind of reporting problem.

As an aside, it looks like there is some kind of race condition with IP address allocation, as during startup (but not at steady state) a number of pods exit with the following in their log (sudo docker logs xxx): listen tcp :9376: bind: address already in use They do restart OK (I'm not sure if this is being restarted locally on the node or making a round-trip back to the scheduler).
Almost certainly the restart is just kubelet retrying after some time. On Feb 8, 2015 2:57 PM, "David Oppenheimer" wrote:
I found that if I comment out the line with the arrow next to it (see below) in , the Pods stop oscillating between Running and Unknown. Commenting out this line changes the logic from "if get an error from Kubelet, change pod status to Unknown" to "if get an error from Kubelet, keep pod status unchanged." I saw in the master logs intermittent errors of this form, which was causing the pods to flip back to Unknown: E0209 04:19:17. 4996 ] Error getting info for pod default/hostnames-v5drx: kubelet "kubernetes-minion-1.c.striped-torus-760.internal" server responded with HTTP error code 500 for pod default/hostnames-v5drx This seems to correspond to lines like this in the kubelet logs: I0209 06:38:11. 4704 ] GET /stats/default/hostnames-gfw5j/-b026-11e4-a070-/hostnames?num_stats=33: (218.857218ms) 500 goroutine 730 [running]: [ ... stack trace, but they're not always the same; I'll dig into this some more, the default stack trace depth wasn't going back past the various functions that just serve up the error ... ] However, pods still flip between Running and Pending, even though no containers are exiting on the minion, so there's another problem here too.
Thanks for investigating this! Pods really shouldn't flip between Running and Pending, as I believe that the only way that they go Pending is if Pod.Status.Host is empty, and this really shouldn't happen post scheduling, unless we are somehow dropping information. Maybe we need to make sure that we don't wipe out status on the error case?

--brendan , David Oppenheimer
I also investigated the same issue last Friday. I observed exact the same as what described above, except the related kubelet log should be: Meanwhile, I observed master is trying to query pod / and failed all the time: Talked to offline, he is going to look into why running pods go back to Pending; meanwhile, I am going to figure out Unknown cases here.
That would be great. It is very hard for me to make sense of any of the scalability experiments that I am doing while the pod status is in hysteresis.
Just to summarize: (1) the Running<-Unknown flapping is caused by the line of code I mentioned earlier; (2) the reason that line of code is being hit on the master is due to two errors on the kubelet (the one I identified earlier was wrong; Dawn found these): I0209 06:42:09. 4704 ] GET /api/v1beta1/podInfo?podID=hostnames-v5drx&podNamespace=default: (8.82543048s) 500 E0209 04:34:13. 4996 ] Error getting info for pod default/hostnames-2y8ny: kubelet "kubernetes-minion-1.c.striped-torus-760.internal" server responded with HTTP error code 500 for pod default/hostnames-2y8ny (3) we don't yet know the cause of the Running<-Pending flapping; I'm going to look into that now
(and Dawn is looking into why we're hitting those two errors on Kubelet)
Are we serving requests concurrently in Kubelet? (Is it possible not to when using go's servemux?) I don't see any locking around the pod map accesses.
This was marked for the fixit but Dawn and I are working on it. Assigned to myself.
I am also investing time writing code to try and report what is going on with unstable cluster status. It would be good to know what you and Dawn are doing in case I am wasting my time. Thanks. , David Oppenheimer
This particular problem is easily reproduced manually using the instructions Tim listed, so I'm not sure it's necessary to write anything sophisticated to drive the cluster into this state.

Anyway, maybe give me and Dawn a couple of days and if we can't figure this out by then we can call in the cavalry.
Am I wrong, or are these instructions also within bounds of an e2e test? Just because our existing e2es only do one or two things doesn't mean we can't do more within O(minutes), assuming you're dealing with the same image.
The "flips back to pending" problem appears to be because updatePodStatus() writes to the cache but not back to the Pod's status. computePodStatus() initializes newStatus based on pod.Status, not based on the cached value of status. This second behavior seems wrong in light of the first behavior. However, I believe this only causes a problem in combination with my fix described earlier, which as a result doesn't actually do what I described (it doesn't "keep the status as-is" but rather copies it out of the pod's status; to "keep the status as-is" it should be copying it out of the cache). I can see if this fixes things. (We also should fix the root cause stuff on the node side, which Dawn is working on.)
Assign the issue to me ... I didn't see the flips between Pending and Running anymore. Don't know why. I consistently see flips between Unknown and Running on those heavily loaded nodes. The root issue is once a while docker inspect API call returns us EAGAIN. Please note that on my test nodes, I already increase NOFILE limit for docker daemon process. The proposed solution is that kubelet deals with all errors internally except EAGAIN one. In this case, computePodStatus logic in pod_cache does nothing when p.containerInfo.GetPodStatus returns an error. Working on a PR to fix this.
Dawn, my suggestion is that computePodStatus() should return the current cached value whenever it gets an error from Kubelet. I can implement this if you want.
, why shouldn't updatePodStatus simply write through to the pod's status? I'd think either we always read the cached value first, or we always write through to the pod and read from there. Only reading the cached value upon an error seems counter-intuitive to me...
why do you computePodStatus returns a cached value here?

The only place calls computePodStatus is updatePodStatus, which triggered by two operations: 1) periodically update all containers 2) the client asks for PodStatus, and there is no cached one, instead calling computePodStatus to retrieve one. I think a cleaner way is 1) Kubelet handles all the errors except this transient one. Of course, we need to make sure docker is auto-restart and there is no too many open files error on node side too. But they are separate issues and should be handled by node side through script or configure etc. 2) Now back to master side, if node status is ok, kubelet should return something; if there is an error from kubelet, it should be transient; and if there is a cached one, simply return cached one upon request; if no cached one, it should be generated by computePodStatus in next round. I have one working PR to avoid the flips between Running and Unknown, but with increasing number of Pending from node side caused by docker failure. Still investigating it...
I didn't understand "if there is an error from kubelet, it should be transient; and if there is a cached one, simply return cached one upon request; if no cached one, it should be generated by computePodStatus in next round." If you're talking about the caching in the podCache on the master, then what you wrote is exactly what I was suggesting. (If you're talking about some kind of caching on the Kubelet, then that's different, because I am not aware of cache for status on the Kubelet.) I think always writing back to the pod's status might be fine but I don't know why the podCache was designed the way it was, so I didn't want to change that without asking Daniel first (I think he's the one who created it). I wasn't suggesting to only read the cached value upon an error; I meant always read the cached value under all circumstances.
and here is my understanding. Correct me if I am wrong. UpdateAllContainers periodically updates all containers, but it gets the initial pods from etcd through this line . each from , UpdateAllContainers calls passes to . computes and returns a new status. Note that the status is computed based on , i.e., the information in etcd. updates the pod cache with the new status.

Note that the new status is not written to etcd. If you repeat the same steps again, you will start with the stale pod status in the etcd because does not check the cached value.
I have tested since last night. The flaps between Running, Pending and Unknown are gone for about 10 hour running 300 pods on 4 nodes except two scenario: 1) etcd issue: cat kube- | grep "status to unknown" I0212 20:18:30. ] node doesn't exist: 501: All the given peers are not reachable (Tried to connect to each peer twice and failed) [0] <nil, setting pod hostnames-yuy8d status to unknown ... This happens a couple of times last 10 hours. We need to figure out why. 2) NodeStatus becomes UNREADY. It happens once last 10 hours, and pods status on that node become Unknown which is expected. I didn't figure out why NodeStatus was UNREADY then, but I heard there is a network earlier this morning.
up to you whether to reopen this issue or file new ones for new issues.
I've also seen the "All the given peers are not reachable" message (see my second entry in this issue). I think that was what was referring to when he said "there is an etcd size limit you might be hitting too" but I may be mis-interpreting.
That error is the etcd client's catch-all for "something went wrong" - it's actually really bad that it obscures errors. , David Oppenheimer
, Tim Hockin wrote:
and Yes, that error message is cache-all error message, which need to investigation. In my particular case, it shouldn't be etcd size limits at all since we have that cluster built and upgraded more than 10 hours before, and all 300 pods were scheduled back then. There is no new events generated either from kubelet or other components since all bound pods are indeed happily running on the nodes. No new writes to etcd except NodeStatus, which is very less frequent and not much data. The problem of etcd here occurs when master tries to read from etcd with an error.


Question: // proxied by this service. Port int `json:"port"` <del> // Optional: Supports "TCP" and "UDP".  Defaults to "TCP". </del> <ins> // Required: Supports "TCP" and "UDP". </ins> Protocol Protocol `json:"protocol,omitempty"` // This service will route traffic to pods having labels matching this selector. If empty or not present,

Many comments in there say "if not specified, default is ..." - but we previously asserted that there are no defaults for the internal API. We should strip those comments out, and leave them in versioned structs only.
cc


Question: "k8s.io/apiserver/pkg/storage/value" kmstypes "k8s.io/apiserver/pkg/storage/value/encrypt/envelope/kmsv2/v2alpha1" "k8s.io/apiserver/pkg/storage/value/encrypt/envelope/metrics" <ins> "k8s.io/klog/v2" </ins> "k8s.io/utils/lru" )

Add additional logging at level that associates UID with the metadata from the request. xref: /sig auth
/triage accepted
/milestone v1.26
Mind if I take a look at this? /assign
Hello I am wondering how's this coming along. Is there anything I can help with?
Sorry. I got a bit occupied with a different project.. Will have a PR opened for this for review tomorrow.. I am trying to find a way to inject the Verbs into log. But haven't found the right way to include that into the context.. Any suggestion on what can be done?
any suggestion on what is the best way to get the additional metadata for verbs into this log? Move that info across using ?
Hello , bug triage team here, just wanted to touch base with you for updates on this ticket. Thanks!
Thanks! This is being worked on and is still on track for .
There is a Pull request opened on this already as well. But Waiting on some clarification to see if that needs additional changes.
/reopen /assign /unassign
Reopened this issue. <detailsIn response to : /reopen /assign /unassign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/milestone v1.27
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
Hi Bug Triage Team for v1.27 here o/ Just wanted to touch base with you for updates on this ticket. Thanks!
I am not working on this anymore Carlos. This was re-assigned
This is tracked for . I've linked the PR.


Question: Portals       []string Iqn           string Lun           string <ins> InitIface     string </ins> Iface         string chapDiscovery bool chapSession   bool

What happened: When creating a block volume, the iSCSI storage plugin stores volume-related data in different directories: As a result, some data is not cleaned up once the volume is detached: How to reproduce it (as minimally and precisely as possible): These are the objects I used to reproduce this issue: pod- /sig storage CC
/assign


Question: // Insert index.html with given content into given volume. It does so by // starting and auxiliary pod which writes the file there. // The volume must be writable. <del> func InjectHtml(client clientset.Interface, config VolumeTestConfig, volume v1.VolumeSource, content string) { </del> <ins> func InjectHtml(client clientset.Interface, config VolumeTestConfig, fsGroup *int64, volume v1.VolumeSource, content string) { </ins> By(fmt.Sprint("starting ", config.Prefix, " injector")) podClient := client.CoreV1().Pods(config.Namespace) podName := fmt.Sprintf("%s-injector-%s", config.Prefix, rand.String(4))

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: Which test(s) are failing: [sig-storage] In-tree Volumes [Driver: local][LocalVolumeType: block] [Testpattern: Pre-provisioned PV (default fs)] volumes should be mountable Since when has it been failing: Testgrid link: Reason for failure: Volume of previous pod is not unmounted before the second pod is trying to mount its volume. Anything else we need to know:
/sig storage /assign
The problem is when client is deleting first pod, it is deleted immediately. This is because when pod status phase is , . But pod is set to before its volumes are unmounted and detached. pod object may be deleted from apiserver before its volumes are unmounted, so it may have conflicts in setting fsGroup for second pod `in-tree-client. If we want to wait for the pod to be deleted by kubelet instead of apiserver, we need to make it a long-running pod. Alternatively, we can make the two pods use the same fsgroup . cc What do you think?
cc is this expected? I thought kubelet is supposed to delete the pod object only after the volumes are unmounted. This also impacts the StorageProtection feature, which assumes that if the Pod object is deleted, then the volume has been unmounted. For a storage backend like nfs, it may result in unmount hanging. cc to fix the flakiness, the simplest thing may be to make the two pods use the same fsgroup. I don't see a strong use case for two pods using different fsgroups.
That is only during graceful deletion. If a pod is Succeeded, do we begin unmounting volumes? Or do we wait until the deletion timestamp is set? Would it be good enough to start unmounting when the pod is failed/succeeded? If we want to be 100% sure this isn't a problem, we would need to always do graceful deletion, even when the pod is failed/succeeded.
Yes, we do start unmounting when pod is . The API is setting grace period to 0 in the case of pod success or failed. Maybe that needs to be removed if we want to use Pod deletion as a signal that volumes are unmounted.

To always do graceful deletion even when the pod is failed/succeeded is simple, then we can assume that normally when pod objects are deleted (except forcefully), volumes are already unmounted by kubelet. But , and it helps to improve performance in deleting jobs in batch. For this flaky test only, what I need is to make sure volumes are unmounted before the second pod is going to mount them, but local volumes are not reported in or other fields of node status, I can't make sure of this by checking node status either. Possible workarounds: make the first a long-running pod (too hacky) wait for a while before starting the second pod (short period may be flaky too, long period increases test time) use same fsGroup for both pods (do not test that fsGroup can be changed) I'm going to use same fsGroup for both pods until we have a reliable way to detect volumes are unmounted or not.


Question: --- - createCoreV1NamespacedServiceAccountToken <del> - createCoreV1Node </del> - createStorageV1CSINode <del> - deleteCoreV1Node </del> - deleteStorageV1CollectionCSINode - deleteStorageV1CSINode - listStorageV1CSINode

[x] two weeks soak end date : 2024-09-08 According to following APIsnoop query, there are two Node endpoints that are untested. Using a number of existing e2e test practices a new has been created to provide future Conformance coverage for the 2 endpoints. The e2e logs for this test are listed below. This query shows the following Node endpoints are hit within a short period of running this e2e test. If a test with these calls gets merged, test coverage will go up by 2 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted
/sig node
/reopen Tracking the next steps to conformance
Reopened this issue. <detailsIn response to : /reopen Tracking the next steps to conformance Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen Tracking the final step
Reopened this issue. <detailsIn response to : /reopen Tracking the final step Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: // We couldn't parse metadata - fallback to computing it. podBestEffort = isPodBestEffort(pod) } // pod is not BestEffort pod if !podBestEffort { return true, nil, nil } <del> // is node under pressure? for _, cond := range node.Status.Conditions { if cond.Type == v1.NodeMemoryPressure && cond.Status == v1.ConditionTrue { return false, []algorithm.PredicateFailureReason{ErrNodeUnderMemoryPressure}, nil } </del> <ins> // is node under presure? if nodeInfo.MemoryPressureCondition() == v1.ConditionTrue { return false, []algorithm.PredicateFailureReason{ErrNodeUnderMemoryPressure}, nil </ins> } return true, nil, nil } // CheckNodeDiskPressurePredicate checks if a pod can be scheduled on a node // reporting disk pressure condition. func CheckNodeDiskPressurePredicate(pod *v1.Pod, meta interface{}, nodeInfo *schedulercache.NodeInfo) (bool, []algorithm.PredicateFailureReason, error) { <del> node := nodeInfo.Node() if node == nil { return false, nil, fmt.Errorf("node not found") </del> <ins> // is node under presure? if nodeInfo.DiskPressureCondition() == v1.ConditionTrue { return false, []algorithm.PredicateFailureReason{ErrNodeUnderDiskPressure}, nil </ins> } <del> // is node under pressure? for _, cond := range node.Status.Conditions { if cond.Type == v1.NodeDiskPressure && cond.Status == v1.ConditionTrue { return false, []algorithm.PredicateFailureReason{ErrNodeUnderDiskPressure}, nil } } </del> return true, nil, nil }

Looking into profiles of scheduler, there shows a bunch of things that we are doing unnecessarily. In particular, instead of computing the same information for a given node multiple times, we should cache it as part of NodeInfo: The information that should be cached and reuse later include: [x] node taints: [x] node conditions: disk/memory/inode pressure
Another quick optimization should be optimizing "selectHost" method: In fact it doesn't need to sort the whole list (which is O(nlgn). All we need to do is to choose the elements that have highest score, and sort only those by node name (to leave the logic that we currently have with round-robic logic of choosing from those).
And we should also limit the amount of places where we call "Parallelize" (because those are relatively expensive). The whole MR-like priorities is a step in this direction too.
OK - I've just sent out a PR fixing the first two things.
can we close this, or are there still work items here?
Some of those are not yet done.
- it's not a high priority for us (for now), so if you want it to be done soon, you probably need to find someone to do it.
- assigned.
Why "Parallelize" is considered as relatively expensive ? (comparing to MR style?)
You do MR when there are certain points that need some synchronizations. Or fully parallelized computation will always be faster.
To clarify, I was referring to the Parallelize function: Using this for many very small functions is pretty expensive - you can take a look into cpu-profiles. It happens that 20% or sth of cpu usage is spent on the operations on the synchronization channel.
Is all of the cacheing part completed? I assume "yes". What is remaining here.... Just an audit of parallelization ? Also, I assume "yes". If so I can audit where the parelellization is getting us hung up and then we can close this one out.
It's hard to tell if caching is done - when adding new priority functions/moving existing ones to MR-framework one pretty often discovers that more things need to be cached. There's my PR , that moves basic spreading priority function to MR-framework, which for some reason makes it worse. I didn't have time to look into that though.

This needs to be done, which may require modifying MR-framework slightly (allowing different type as a Map phase result).
Im not sure what the MR Framework stuff is referring to. Can you provide me a pointer ? Should we close this issue since the original patches for caching were done? Sounds like the work we're referring to now is actually more like a second iteration. if not, what work remains to be done before we can close this ticket? maybe we can make a checklist.
"MR-style" parallelization, that was mentioned before. Yeah, I guess we can close this - this is more like an ongoing process thing, when everyone who adds anything to the scheduler needs to keep it in mind.


Question: // or equal to SystemCriticalPriority. Both the rescheduler(deprecated in 1.10) and the kubelet use this function // to make admission and scheduling decisions. func IsCriticalPod(pod *v1.Pod) bool { <del> return IsCritical(pod.Namespace, pod.Annotations) || (pod.Spec.Priority != nil && IsCriticalPodBasedOnPriority(pod.Namespace, *pod.Spec.Priority)) </del> <ins> return IsCritical(pod.Namespace, pod.Annotations) || (pod.Spec.Priority != nil && IsCriticalPodBasedOnPriority(*pod.Spec.Priority)) </ins> } // IsCritical returns true if parameters bear the critical pod annotation

When we used annotations to mark critical pods, we considered pods with such an annotation critical only when they were created in "kube-system" namespace. Going to the new era of using priority to indicate critical pods, we would like to remove the restriction of "kube-system" namespace and consider pods with critical priority as critical no matter what namespace they are in. ref/ /kind bug /sig scheduling /assign cc/
While I am fine with the approach, it seems the only way to ensure that those critical pods are not coming from some random namespace is whitelisting them using priority admission controller or using quota. Atleast previously we have one more level of security that critical pod has to be created in kube-system namespace.
The requirement to limit the critical pod annotation meaning to the kube-system namespace was a workaround for the lack of access control and quota around the annotation. We actively work to prevent special handling of the kube-system namespace wherever possible.
Yes, which is why those aspects are essential to a functional priority system that works for a variety of cluster configurations
Agree with Ravi and Jordan on what is needed for final solution.
There is no doubt that we need to have quota for priority. It is needed not only for critical priorities, but also for any other high priority classes that users may define. That effort is going forward and we will have it soon.
, I think we also need to update Priority admission controller for that.
see later PR re-restricting this until the quota controls graduate to at least beta state:
Got that, thanks :) That makes sense to me.
If this is closed and there even was a for this type of functionality, what is the current status? And is in sync with current status?
It seems that namespace is still , so I guess the current status is that system critical pods are still limited to namespace. Based on that I think this issue should not be closed.
can we reopen this issue please because as pointed out by the issue still exists. Thanks.
Waiting for this to be available myself. No reason my Node Exporter dataset has to run in the namespace instead of my Grafana namespace.
Confirmed code + behavior. /open
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.

</details
I too am confused based on that documentation. Can we get an update here with an answer and fix the documentation if it's wrong?
This is resolved in 1.17 See


Question:
PLEASE NOTE: This document applies to the HEAD of the source tree
 If you are using a released version of Kubernetes, you should refer to the docs that go with that version.  The latest release of this document can be found [here](http://releases.k8s.io/release-1.1/docs/proposals/choosing-scheduler.md). Documentation for other releases can be found at [releases.k8s.io](http://releases.k8s.io).  --   Scheduler Performance Test ====== Motivation ------ We already have a performance testing system -- Kubemark. However, Kubemark requires setting up and bootstrapping a whole cluster, which takes a lot of time. We want to have a standard way to reproduce scheduling latency metrics result and benchmark scheduler as simple and fast as possible. We have the following goals: - Save time on testing - The test and benchmark can be run in a single box. We only set up components necessary to scheduling without booting up a cluster. - Profiling runtime metrics to find out bottleneck - Write scheduler integration test but focus on performance measurement. Take advantage of go profiling tools and collect fine-grained metrics, like cpu-profiling, memory-profiling and block-profiling. - Reproduce test result easily - We want to have a known place to do the performance related test for scheduler. Developers should just run one script to collect all the information they need. Currently the test suite has the following: - density test (by adding a new Go test) - schedule 30k pods on 1000 (fake) nodes and 3k pods on 100 (fake) nodes - print out scheduling rate every second - let you learn the rate changes vs number of scheduled pods - benchmark - make use of `go test -bench` and report nanosecond/op. - schedule b.N pods when the cluster has N nodes and P scheduled pods. Since it takes relatively long time to finish one round, b.N is small: 10 - 100. How To Run ------ ``` cd kubernetes/test/component/scheduler/perf ./test-performance.sh ```  [![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/test/component/scheduler/perf/README.md?pixel)]()

We have found that on 1k nodes cluster scheduling takes 60-200 ms and becomes the bottleneck. We want to have a standard way to reproduce scheduling latency metrics result and benchmark scheduler as simple and fast as possible. Only with that can we: 1. figure out underlying behaviors and potential bottlenecks quickly; 2. publish results in a standard way and prevent future performance degradation. We have the following goals: Goal 1: Save time on testing. More accurate scheduling latency. We mock up kubelets and only measure scheduling latency and do not care about the latency to start the pod. This is different from Kubemark -- scheduled pods won’t be run by mocked kubelets. We only need to set up the master components. Goal 2: Profiling runtime metrics and find out performance bottleneck. Write scheduler integration test. Take advantage of go test and profiling tools, like cpu-profiling, memory-profiling and block-profiling. This is very fine-grained metrics. Goal 3: Reproduce test result easily. We want to have a known place to do the performance related test for scheduler. Developers should just run one script to collect all the information they need. We have the following plans for writing scheduler tests: density test (by adding a new Go test) schedule 30k pods on 1000 (fake) nodes and 3k pods on 100 (fake) nodes print out scheduling rate every second let you learn the rate changes vs number of scheduled pods benchmark (by modifying the existing benchmark in integration or simply add a new one) make use of and report nanosecond/op. schedule b.N pods when the cluster has N nodes and P scheduled pods. Since it takes relatively long time to finish one round, b.N is small -- level of 10. We are new to k8s codebase. We also want to know where is the best place (directory) to commit in these tests? Any suggestions are welcomed!
/cc
This extends
So density already does this... the one option it looks like you may want is custom control , real or fake is irrelevant as the e2e's just need to point to an api-server. re: benchmarks, seems like we could simply add to the unit-tests.
After chatting with lets add this as an agenda item for the SIG meeting tomorrow.

+1 for adding benchmark (howeveer this should be integration test, not unit-test in my opinion)
cc The next SIG on scheduler meeting is on 12.07, isn't it? If there is one tomorrow, can I have the information please?
retract, +1 to . I was adding this to the list given the context, but it crosses over into both.
Slides from today's meeting:
Here are are the action item notes from the SIG meeting: Everyone agrees this is a good idea and long overdue, but the question is where would this live: New category of tests? (isolated/sandbox’d/component) The current testing structure allows to easily append. create new directory (component) + hack/script to execute Future "we could" gate on known good data, and fail performance regressions
/cc
/cc


Question: profileName: profile, }, } <del> return f.ClientSet.Core().ConfigMaps(f.Namespace.Name).Create(cm) </del> <ins> _, err := f.ClientSet.Core().ConfigMaps(f.Namespace.Name).Create(cm) framework.ExpectNoError(err, "Failed to create apparmor-profiles ConfigMap") </ins> } <del> func createAppArmorProfileLoader(f *framework.Framework) (*extensions.DaemonSet, error) { </del> <ins> func createAppArmorProfileLoader(f *framework.Framework) { </ins> True := true <del> // Copied from https://github.com/kubernetes/contrib/blob/master/apparmor/loader/example-configmap.yaml loader := &extensions.DaemonSet{ </del> <ins> One := int32(1) loader := &api.ReplicationController{ </ins> ObjectMeta: metav1.ObjectMeta{ Name:      "apparmor-loader", Namespace: f.Namespace.Name, }, <del> Spec: extensions.DaemonSetSpec{ Template: api.PodTemplateSpec{ </del> <ins> Spec: api.ReplicationControllerSpec{ Replicas: &One, Template: &api.PodTemplateSpec{ </ins> ObjectMeta: metav1.ObjectMeta{ <del> Labels: map[string]string{"name": "apparmor-loader"}, </del> <ins> Labels: map[string]string{loaderLabelKey: loaderLabelValue}, </ins> }, Spec: api.PodSpec{ Containers: []api.Container{{

The "AppArmor load AppArmor profiles should enforce an AppArmor profile" e2e test in failing on our 5k-node gce clusters - https://k8s- (run no. 8) due to timeout: Seems like the test is trying to load the profiles across all the nodes using a daemonset and then running a test pod to check that. It's passing on our 2k-node clusters (https://k8s-) but failing for 5k - suggesting that it's likely due to size of the cluster. I think it's the ds creation taking longer on large cluster. cc
Seems like you're the most relevant here. Could one of you take a look into it? And does increasing the timeout for the test-pod based on no. of nodes sound reasonable?
Assigning it you since is OOO. Feel free to reassign as apt.
This is part of the correctness suite and large cluster tests are release-blocking.
FYI
Reassigning to who is back on Tuesday. He wrote the AppArmor stuff and should be able to debug this quickly. /assign
Ref
Could you manage to take a look into this?
Sorry, I meant to reply to this! I have a fix, just need to confirm it works. I think you're right about the DaemonSet timing out. There's no reason this needs to load the profiles on every node, so the fix just runs a single loader, and uses pod affinity to co-schedule the test pod.
Smooth!
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/node sig/scalability: Issue will be escalated to these SIGs if needed. priority/important-soon: Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. kind/bug: Fixes a bug discovered during the current release. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details


Question: <ins> /* Copyright 2017 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package kubectl import ( "reflect" "testing" "k8s.io/api/core/v1" extensionsv1beta1 "k8s.io/api/extensions/v1beta1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" ) func TestDeploymentBasicGenerate(t *testing.T) { one := int32(1) tests := []struct { name           string deploymentName string images         []string expected       *extensionsv1beta1.Deployment expectErr      bool }{ { name:           "deployment name and images ok", deploymentName: "images-name-ok", images:         []string{"nn/image1", "registry/nn/image2", "nn/image3:tag", "nn/image4@digest"}, expected: &extensionsv1beta1.Deployment{ ObjectMeta: metav1.ObjectMeta{ Name:   "images-name-ok", Labels: map[string]string{"app": "images-name-ok"}, }, Spec: extensionsv1beta1.DeploymentSpec{ Replicas: &one, Selector: &metav1.LabelSelector{ MatchLabels: map[string]string{"app": "images-name-ok"}, }, Template: v1.PodTemplateSpec{ ObjectMeta: metav1.ObjectMeta{ Labels: map[string]string{"app": "images-name-ok"}, }, Spec: v1.PodSpec{ Containers: []v1.Container{ {Name: "image1", Image: "nn/image1"}, {Name: "image2", Image: "registry/nn/image2"}, {Name: "image3", Image: "nn/image3:tag"}, {Name: "image4", Image: "nn/image4@digest"}, }, }, }, }, }, expectErr: false, }, { name:           "empty images", deploymentName: "images-empty", images:         []string{}, expectErr:      true, }, { name:           "no images", deploymentName: "images-missing", expectErr:      true, }, { name:      "no deployment name and images", expectErr: true, }, } for _, test := range tests { generator := &DeploymentBasicGeneratorV1{ BaseDeploymentGenerator{ Name:   test.deploymentName, Images: test.images, }, } obj, err := generator.StructuredGenerate() if !test.expectErr && err != nil { t.Errorf("unexpected error: %v", err) } if test.expectErr && err != nil { continue } if !reflect.DeepEqual(obj.(*extensionsv1beta1.Deployment), test.expected) { t.Errorf("test: %vnexpected:n%#vnsaw:n%#v", test.name, test.expected, obj.(*extensionsv1beta1.Deployment)) } } } </ins>

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind enhancement What happened: there is no UT for /sig testing
This issue should be closed because the corresponding PR has been merged already


Question: if err != nil { return err } <ins> matches, err = s.PopulateIPs(matches) if err != nil { return err } </ins> nodes, err := s.kubeClient.Nodes().List() if err != nil { return err } nodeMap := make(map[string]*api.Node) <del> for _, node := range nodes.Items { </del> <ins> for i := range nodes.Items { node := nodes.Items[i] </ins> nodeMap[node.Name] = &node }

I see the following in the kube-controller-manager log: - This is happening to me on HEAD and is the reason why pod status is always showing unknown even when they are scheduled and running.
This is with the vagrant cloud provider. It looks like the delta between the new and old minion is the Node.Spec.ExternalID is being set on the incoming minion but is not set in the originally created minion.
I am not sure how this is not affecting other cloud providers. It looks to me that NodeController populated the node ExternalID here: It looks like the Kubelet provided value is missing the ExternalID
Actually, I am now confused on what the NodeController may or may not be doing here that is causing the status to not get updated.
It looks like the client is passing an ExternalID on the incoming request, but the original value that is in etcd does not have an ExternalID set. Not sure what changed that would have caused this issue to appear.
I think this causes problems: The memory is re-used in a for-loop by Go so referring to a pointer value in a map here will cause problems I think.
Ok, the referenced line is the problem. Sending a PR.
Well, I go back on that now. A simple way to reproduce this is to delete all the minions in your cluster, and verify the node controller is able to add them back. For me, this is causing the node controller to add them back in an unknown status, and then all updates to the node fail. The update fails because the node controller is not setting an ExternalID on the original object.
I am looking into this now. cc/ too
Related issue opened while debugging this:
Debugged through here: I can via manual fmt see that the object has an ExternalID set prior to the call Create. When I look at etcd, I end up with this persisted: Note: it is missing an ExternalID somehow ;-)
Ok, I think I see the issue Nodes returned from here: Do not have an ExternalID Which can make Nodes created from here: End up without an External ID. Having two different nodeClient.Create() paths in this code is confusing. Also we are doing bad things in maps holding pointers to memory allocated in for-loop. We need to fix that ;-)
Yeah, this is bug we should fix. Thanks for looking into this.

If you already have a potential fix, please feel free to grab the issue from me.
I think I have a PR that should fix. Testing that now. Will cc you on the PR.
Thanks for fixing - it did end up affecting AWS as well (AWS isn't in a fully working state just yet, particularly with node discovery).


Question: driverName = data[volDataKey.driverName] volID = data[volDataKey.volHandle] } else { <del> klog.Error(log("UnmountDevice failed to load volume data file [%s]: %v", dataDir, err)) // The volume might have been mounted by old CSI volume plugin. Fall back to the old behavior: read PV from API server driverName, volID, err = getDriverAndVolNameFromDeviceMountPath(c.k8s, deviceMountPath) if err != nil { klog.Errorf(log("attacher.UnmountDevice failed to get driver and volume name from device mount path: %v", err)) return err </del> <ins> if errors.Is(err, os.ErrNotExist) { klog.V(4).Info(log("attacher.UnmountDevice skipped because volume data file [%s] does not exist", dataDir)) return nil </ins> } <ins> klog.Errorf(log("attacher.UnmountDevice failed to get driver and volume name from device mount path: %v", err)) return err </ins> } if c.csiClient == nil {

Hello, I've recently been testing the reliability of storage for k8s v1.25.0. A strange phenomenon occurs occasionally during the test: the volume cannot be detached after the PVC and pod are deleted(the reclaimPolicy of pv is delete. So I first went to check the corresponding and objects and found that they both existed. Then I checked the kube-controllerManager and kubelet logs and found that kubelet logs kept printing: But I found that at the beginning, the volume was unmountDevice succeed, as shown in the following kubelet log: it is so wired ! Then I some log variables to the unmountDetachDevices method: The results show that And I've drawn up a time-series chart of the contents of the log. ! Let's go back to the unmountDetachDevices method. According to the above-mentioned log content, the first operation successfully changes of the corresponding attachedVolume to , but the function in the subsequent operation still uses stale DeviceMountState. that the unmountDevice operation is not performed after the first unmountDevice operation succeeds. multiple unmountDevice operations are allowed, ensure that the correct pvname can be obtained in the method. the volume is detached after the pod and PVC objects are deleted This problem can be reproduced by repeatedly deleting and recreating pods that require PVC in k8s 1.25.0 cluster No response <details</details<details</details<details</details<details</details<details</details<details</details
/sig storage
we can't safely assume that the json is always there.
ping
Thanks for the detailed analysis . I'm still sorting through this, do you think that is involved with this?
Yes. look at this log content: The global mount path is changed in the As a result, the method obtains incorrect pvName(e6104b10fea107e004e3307b0444fd7b775fedd804ab47aa997118b7927c32e0). The unmountDevice operation may be performed multiple times in [1.19 - 1.25] kubernetes versions. This problem does not occur before because can still obtain the correct pvName from the global mount path. However, this problem occurs after is merged. https://user- <br</brAfter the first unmountDevice operation is performed successfully, the file is deleted.

So the subsequent unmountDevice operation will perform method because file doesn't exist.
Awesome, thanks. Let me find someone to work on this.
I ll take a look at the issue, since its related to the change I made Note: this change needs the node to drained before upgrading to a version with this change . Please check the release notes
Thanks a lot !
yep, I always reproduce this problem in a newly created k8s 1.25.0 cluster.
Hihas there been any progress on this issue?
/priority important-soon
/triage accepted
Hihas there been any progress?
I also hit this error after pvc and pod deleted on k8s 1.25: and then these two volumes cannot be unmounted on the node forever: I tried to delete pv, it's stuck in terminating state forever
restart the kubelet on the node in problem could mitigate this issue, but we really need to fix this issue ASAP, I could repro this issue easily when running 1000 disk attach/detach test
I think this is again a TOCTOU bug in unmountDetachDevices.
The Unmount operation is , thus the state may not be set as before next reconcile, so it is possible that unmountDevice operation performed multiple times. So the root cause of this issue is that got wrong PV name, since changed global mount path.
I have found a similar issue in our environment, which is based on Kubernetes 1.25.3. If I install a Pod with a PVC, and I force move the Pod with NodeAffinity around the cluster, around 10-20 retries the Pod does not come up: the VolumeAttachment is bound to the old node and is not moved by the system anymore. If this is the same issue as reported here I do not know, but the logs are similar: From the Worker's kubelet where the volumeattachment is bound to: Also I see lots of these: But, if this is the same, I would like to know that the referenced PR was merged into exactly which Kubernetes version, as I have found in the RELEASE NOTES, that it was in 1.24.0-alpha: But we could not reproduce the issue on Kubernetes 1.24.2. How come?
As per the logs, it's the same issue. Kubelet operates in a reconcile loop, and is an async operation.

This bug only happened when is operated for multiple times, so it cannot be 100% reproduced. However, if the possibility of reproduction in 1.25 is much higher than the cluster lower than 1.25, we should figure out whether there are some other changes that makes this bug happened more frequently.
I have 1K disk attach/detach to 40 nodes test, if it runs multiple times, it could 100% repro, we could use this patch for testing, if 100 test runs pass, then it could prove the fix really works.
Sure. I mean not every single Mount/Unmount operation will lead to this error, but if we do it enough times, then it could be 100% to reproduce.
May I ask what could be a workaround in production systems or even CI systems on this issue? Our CI is struggling with this and other random issues, to the point we would like to stabilize it by kicking the system in a way to fix things... I was trying a kubelet restart on the Worker where the Mount is stuck... but many cases the kubelet restart itself hangs or does not produce a correct restart. So what could be a workaround if we find out that there is a hung volumeattachment?
restart kubelet, and if restart kubelet hangs, reboot the node.
Volumeattachment failed to detach from old node When statefulset application with pvc scheduled on worker1 during deployment its volumeattachment bounds to worker1 when same application upgraded scheduled on another worker node say worker2 and struck in ContainerCreating state and volumeattachment failed to detach from old node .

when pod described: Events: Warning FailedMount 23m (x820 over 6d2h) kubelet Unable to attach or mount volumes: unmounted volumes=[datadir], unattached volumes=[ datadir]: timed out waiting for the condition Warning FailedMount 5m22s (x3069 over 6d3h) kubelet Unable to attach or mount volumes: unmounted volumes=[datadir], unattached volumes=[datadir ]: timed out waiting for the condition Environment: NAME="SLES" VERSION="15-SP4" VERSIONID="15.4" PRETTYNAME="SUSE Linux Enterprise Server 15 SP4" ID="sles" IDLIKE="suse" ANSICOLOR="0;32" CPENAME="cpe:/o:suse:sles:15:sp4" DOCUMENTATIONURL=""' Kubernetes version: v1.25.3 kernel: 5.14.21-.24.41-default Storage: Rook: v1.9.12 ceph: 16.2.10 logs of kubelet: Mar 21 11:35:55 kubelet[]: E0321 11:35:55. ] "Unable to attach or mount volumes for pod; skipping pod" err="unmounted volumes=[datadir], unattached volumes=[datadir ]: timed out waiting for the condition" pod="test/ -0" Mar 21 11:35:55 kubelet[]: E0321 11:35:55. ] "Error syncing pod, skipping" err="unmounted volumes=[datadir], unattached volumes=[datadir ]: timed out waiting for the condition" pod="test/ -0" podUID=-1432-473d-af85- Mar 21 11:38:13 kubelet[]: E0321 11:38:13. ] "Unable to attach or mount volumes for pod; skipping pod" err="unmounted volumes=[datadir], unattached volumes=[datadir ]: timed out waiting for the condition" pod="test/ -0" Mar 21 11:38:13 kubelet[]: E0321 11:38:13. ]

"Error syncing pod, skipping" err="unmounted volumes=[datadir], unattached volumes=[datadir ]: timed out waiting for the condition" pod="test/ -0" podUID=-1432-473d-af85- Mar 21 11:40:29 kubelet[]: E0321 11:40:29. ] "Unable to attach or mount volumes for pod; skipping pod" err="unmounted volumes=[datadir], unattached volumes=[datadir ]: timed out waiting for the condition" pod="test/ -0" Mar 21 11:40:29 kubelet[]: E0321 11:40:29. ] "Error syncing pod, skipping" err="unmounted volumes=[datadir], unattached volumes=[datadir ]: timed out waiting for the condition" pod="test/ -0" podUID=-1432-473d-af85-
FYI. this issue is fixed in 1.24.14, 1.25.10, 1.26.5, and has been fixed in v1.27.0
Hello , can you have a look at this reconciler run's operationExecutor fails to umount volume due to missing vol- It appears to be Kube v1.22 not .25 to boot is this the same issue?
I think they are the same issue and I have left some comments in
Issue reoccurs in kubelet version