Question: }}, }, { <ins> name: "pods list logs: default container should not leak across pods", obj: &corev1.PodList{ Items: []corev1.Pod{ { TypeMeta: metav1.TypeMeta{ Kind:       "pod", APIVersion: "v1", }, ObjectMeta: metav1.ObjectMeta{ Name:      "foo", Namespace: "test", Labels:    map[string]string{"test": "logs"}, Annotations: map[string]string{ "kubectl.kubernetes.io/default-container": "c1", }, }, Spec: corev1.PodSpec{ RestartPolicy: corev1.RestartPolicyAlways, DNSPolicy:     corev1.DNSClusterFirst, Containers: []corev1.Container{ {Name: "c1"}, {Name: "c2"}, }, }, }, { TypeMeta: metav1.TypeMeta{ Kind:       "pod", APIVersion: "v1", }, ObjectMeta: metav1.ObjectMeta{ Name:      "bar", Namespace: "test", Labels:    map[string]string{"test": "logs"}, }, Spec: corev1.PodSpec{ RestartPolicy: corev1.RestartPolicyAlways, DNSPolicy:     corev1.DNSClusterFirst, Containers: []corev1.Container{ {Name: "c2"}, }, }, }, }, }, actions: []testclient.Action{ getLogsAction("test", &corev1.PodLogOptions{Container: "c1"}), getLogsAction("test", &corev1.PodLogOptions{Container: "c2"}), }, expectedSources: []corev1.ObjectReference{{ Kind:       "pod", APIVersion: "v1", Name:       "foo", Namespace:  "test", FieldPath:  fmt.Sprintf("spec.containers{%s}", "c1"), }, { Kind:       "pod", APIVersion: "v1", Name:       "bar", Namespace:  "test", FieldPath:  fmt.Sprintf("spec.containers{%s}", "c2"), }}, }, { </ins> name: "pods list logs: all containers", obj: &corev1.PodList{ Items: []corev1.Pod{*testPodWithTwoContainersAndTwoInitContainers()},

/kind bug The logs of pods with label will be output expected without errors. create the following resources: kubectl logs -l Kubernetes version (use ): latest master.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cli
I am able to pull logs from multiple containers in my test setup, this doesn't seem to be an issue with Could you check if all the containers in the specified pod are healthy?
yes, all of these pods' containers are healthy. I the section.
This is working as expected, was not able to find the default container which in both deployments you explicitly said is but only the first deployment has container, the second deployment has . /close
Closing this issue. <detailsIn response to : This is working as expected, was not able to find the default container which in both deployments you explicitly said is but only the first deployment has container, the second deployment has . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: esac done <del> if [ "${golangci_config}" ]; then golangci+=(--config="${golangci_config}") fi </del> # Below the output of golangci-lint is going to be piped into sed to add # a prefix to each output line. This helps make the output more visible # in the Prow log viewer ("error" is a key word there) and ensures that

This is a subset of "make verify doesn't work under anymore" xref: In two places we assume the path for "local builds" But when dockerized it will be at should work.
I'm not sure what the cleanest solution is here. Maybe a symlink in a fixed location that we update to point to whichever we just built ...
cc
Can we set env variables as part of ? golangci-lint doesn't support a search path for plugins yet, but perhaps that can be .
Bit of a blunt instrument but we could do that. Another blunt approach would be picking some unused path under _output to add a symlink to either I guess.
It would be really nice if we could just pass searchpaths for the plugins instead of having to pass a complete path.
relative paths are resolved relative to the config file, so you can indirectly change a root path on a fixed file by using different config paths and not have to edit the file contents ... we could use two symlinks in _output to the actual config file. Still not ideal.
This is one of the few remaining verify scripts that doesnt work on mac or otherwise dockerized.
Let's fix this by making the path absolute in a temporary copy of the golangci-lint config file: Syncing .git is relevant for testing PRs, but it seems that already ensured that it is available.
That sounds reasonable. Yeah, while working on that issue I found the other handful of verify scripts that are also broken when containerized for reasons unrelated to git. This is one of them, the others are linked in


Question: package handlers import ( <del> "encoding/json" </del> "fmt" "k8s.io/apimachinery/pkg/conversion/unstructured" "k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/types" <ins> "k8s.io/apimachinery/pkg/util/json" </ins> "k8s.io/apimachinery/pkg/util/strategicpatch" "github.com/evanphx/json-patch"

When patching, in the type decoded from ToUnstructured mismatch with the one decoded form json.Unmarshal for numbers. Example: port: 80 in original object has type int64 port: 80 in original object has type float64 This is the root cause of


Question: testDeploymentDefaultReplicas := int32(2) testDeploymentMinimumReplicas := int32(1) testDeploymentNoReplicas := int32(0) <ins> testDeploymentAvailableReplicas := int32(0) </ins> testDeploymentLabels := map[string]string{"test-deployment-static": "true"} testDeploymentLabelsFlat := "test-deployment-static=true" w := &cache.ListWatch{

I started missing error checks. When adding error checking to this call here, the test started to fail: The test should have continued to work when checking the error. No response <detailsmaster </details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig apps
/priority important-soon can you or someone in apps PTAL?
The Kubernetes project currently lacks enough contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
/help
This request has been marked as needing help from a contributor. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help?

For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/assign


Question: return strings.Contains(err.Error(), "Retriable: true") } <ins> // HasStatusForbiddenOrIgnoredError return true if the given error code is part of the error message // This should only be used when trying to delete resources func HasStatusForbiddenOrIgnoredError(err error) bool { if err == nil { return false } if strings.Contains(err.Error(), fmt.Sprintf("HTTPStatusCode: %d", http.StatusNotFound)) { return true } if strings.Contains(err.Error(), fmt.Sprintf("HTTPStatusCode: %d", http.StatusForbidden)) { return true } return false } </ins>

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: controller-manager is stuck in deleting an incorrectly configured LB type service. it cannot be recovered until restarted. What you expected to happen: controller-manager should recover. At least it should allow us to create the service without restarting it. How to reproduce it (as minimally and precisely as possible): put a random IP in above yaml and make sure the resource group doen't exist. Anything else we need to know?: Environment: Kubernetes version (use ): 1.12.4 Cloud provider or hardware configuration: AKS OS (e.g: ): Kernel (e.g. ): Install tools: Others:
/sig azure
I also want to add that the above repro is stuck in 403. It's possible for it to be stuck in 404 as well when the SP has the right permission.
Can you share the log for 404?
Thanks for reporting the issue. /assign Do you mean other services couldn't be configured when this happens?
correct.
FYI, we had the same issue when the Idle Timeout annotation was set to "3" which isn't a supported case. Will this PR also cover this edge case ?
Unfortunately , this is still not fixed. The idle timeout is not 403 or 404. We need another fix for this (and possibly more for other annotations)


Question: continue } pv, err = config.client.CoreV1().PersistentVolumes().Get(backgroundCtx, pv.Name, metav1.GetOptions{}) <del> if apierrors.IsNotFound(err) { </del> <ins> if apierrors.IsNotFound(err) || errors.Is(err, context.Canceled) { </ins> continue } // Delete and create a new PV for same local volume storage

master-informing: gce-cos-master-serial ! 2023-06-08 20:01:55 +0000 UTC { failed [FAILED] PV Create API error: Post "https://35.247.13.135/api/v1/persistentvolumes": context canceled In [DeferCleanup (Each)] at: test/e2e/storage/persistentvolumes- @ 06/09/23 18:05:53.165 } No response_ /sig storage cc
I think we did something wrong in /triage accepted
/assign
do you have a link to a full log? It's not clear where comes from (will look closer on Monday), but as the test is shutting down, not allowing new PVs to be created may be normal.
Never mind, the description has links to failing jobs.
ok - this looks like an actual bug with the code as written. what's happening is the "Stop and wait.." at line 492 is calling of the passed to the "goroutine [that] recycle[s] unbound PVs". Ordinarily that signal triggers the and the goroutine exits. In this case (and it will be flaky) it triggers while the case is running and it causes the on line 533 to error with . I don't know what the desired behavior is - should a error be ignored since we're signaling that it's time to leave? or should we actually wait for the to complete and use a different (timeout??) context for that instead.
one potential (if complex) fix is to pass two contexts down to the goroutine. one, to signal “you should stop when you are next at the select statement” and another , that is passed down to CreatePV that signals “stop immediately”. The would then take a Ginkgo context with a timeout. when the first launches it cancels and then if the timeout expires it cancels there might well be a simpler way to track all this but the very nature of what this goroutine is doing seems to need some sort of two-signal approach. alternatively: we can say that within this goroutine is always safe to ignore. it’s simpler as long as there are no test pollution concerns about forcibly canceling the CreatePV.
Patrick is the most knowledgable person in this area, we should wait for his response
Sounds good.

This might be a one off - or they may be a class of flaky tests that have a similar structure and so coming up with, and applying, a common pattern for signaling a background goroutine to exit gracefully could be useful.
I think ignoring the error is fine in this case. There is no reason for it to occur except when the test has already completed. -
sounds good - I suppose the only concern would be that forcibly aborting the might be an issue (e.g. the server's still working on things and we fail to clean up or race with it). but y'al lknow k8s far better than i do so maybe there's reasons to have confidence that it's a nonissue.
If the operation completes on the server side, then the other registered earlier will delete the PV.
last run was green let's see this is stable now, thanks for the help


Question: pleg := &GenericPLEG{ relistPeriod: time.Hour, runtime:      fakeRuntime, <del> eventChannel: make(chan *PodLifecycleEvent, 100), </del> <ins> eventChannel: make(chan *PodLifecycleEvent, eventChannelCap), </ins> podRecords:   make(podRecords), clock:        clock, }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: In our stress testing, we found the deadlock issue in PLEG relist() for healthy checking and Kubelet syncLoop() The goroutine stack trace is as following We did some investigation, and found the following issue. The main loop for event processing in Kubelet will check if there are some runtimeErrors: if yes, wait until runtime back to normal; if no, it will call syncLoopIteration to read and process events. And the runtime healthChecks depends on the But in the relist() impl, it will send the pod lifecycle event to eventChannel which may introduce deadlock when eventChannel is full. And in the above code, the eventChannel will not be consumed if the runtime is not healthy. In some situation, the eventChannel may full, and the relist() will hang in sending new pod lifecycle event. And the runtime status will not be able to recovered. It is related to issues What you expected to happen: No such issue How to reproduce it (as minimally and precisely as possible): It is not easy to reproduce, it happens after few days stress testing (scaling the deployment repeatedly with high load). Anything else we need to know?: Environment: Kubernetes version (use ): 1.11.5 Cloud provider or hardware configuration: Bare metal OS (e.g. from /etc/os-release): CentOS 7.4 Kernel (e.g. ): Linux bjc-qtt-k8s-node-010 3.10.0-693.2.2.el7.x8664 SMP Tue Sep 12 22:26:13 UTC 2017 x8664 x8664 x8664 GNU/Linux Install tools: kubeadm Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
/sig node
Sure, thank you We had some internal fix to break the deadlock, but I am wondering if there are good fix for such kind of issue.
When pleg do relist, can we invoke to judge runtime is ok, if not, not do relist
I submit a PR
Thanks for finding the issue! The issue description makes sense to me. I think PLEG is a core component, we should try our best to unblock it.

The event channel becoming full seems more like an issue in the event consumer side, which shouldn't block PLEG. One solution is to queue the events in PLEG and send later, but that doesn't fundamentally solve the problem, if we don't set a limit on the event number we queue, it seems like a potential memory leak to me; if we set a limit, we may eventually hit this problem again. Another solution is to add a retry for sending event in PLEG, and set a bit in PLEG in between the retry which tells that PLEG is not blocking, it is trying to send out the event. And in the function of PLEG, it returns healthy if the bit is set. Actually, even if we fail to send the event and discard it, since we have already updated the pod cache, the periodical pod resync should eventually pick it up. This may delay pod sync for up to . But if the channel is full, it means that either 1) the syncloop is blocked for some reason, or 2) the node is under heavy loads. In both cases, even the event is successfully sent, the pod sync will likely be delayed anyhow. We can: 1) Enlarge the event channel size to reduce the chance that the channel becoming full. Currently it is arbitrarily set as . 2) Discard the event in PLEG if the channel is already full. In the 3 options above, I'm fine with either 2 or 3. Option 2 doesn't have the 1min pod sync delay, but a bit ad-hoc; option 3 is more generic, but may delay pod sync. There might be other options. /cc Any thoughts?
If PLEG is already blocked by a full event channel, this doesn't help right?
cc/ This looks like the one production issue you mentioned to me a while back.
of options enumerated, option 3 was preferred for me.
Thanks for reporting the issue! what was the load on the node that triggered this condition? If the node wasn't function at all, and the events slowly built up over time, bumping the size of the channel won't help. This should work, and doesn't complicate the logic in PLEG more, so it's my preferred option too.

We need some unit tests, and ideally, a stress test, as mentioned in the sig-node meeting.
Agree, giving a proper size for event channel is tricky as the scale also grows with time. We will ship fix as option 3 () for 1.14. Thanks for summarizing all these up as sig meeting! /assign
If possible, could you share in how large the stress is? It helps understand the issue and kubelet stress handling more. :)
Sure, our product team ( ) will input the stress test data after removing sensitive information in follow up comments.
Thanks for reply. Following is our testcase to reproduce this issue: The test cluster has 5 node on aliyun ECS(like EC2). Three 4C8G machine as master node, Two 96C384G as worker node. Firstly, We create 2 deployment running on the worker node. One deployment is http server pods. Another is siege pods to produce network stress to http server deployment. Those stress test deployments produce 50 CPU Core load on worker node. Then we create a job frequently create/delete 20 pod every 5 seconds. After about 1 day of stess test, The worker node will become NotReady. Dig the pprof goroutine stacktrace found the channel block in .
Yeah, I can see how frequent container state change can cause this issue. The number is very useful, thanks a lot!
cloud you please help to share how to dump the goroutines stack trace of kubelet like what you pasted in your comment. thanks a lot.
This issue is fixed by
using pprof is a answer to my question thank you anyway.


Question: hack/dev-build-and-up.sh ``` <ins> The script above relies on Google Storage to deploy the software to instances running in GCE. It uses the Google Storage APIs so the "Google Cloud Storage JSON API" setting must be enabled for the project in the Google Developers Console (https://cloud.google.com/console#/project). The instances must also be able to connect to each other using their private IP. The script uses the "default" network which should have a firewall rule called "default-allow-internal" which allows traffic on any port on the private IPs. If this rule is missing from the default network or if you change the network being used in `cluster/config-default.sh` create a new rule with the following field values: * Source Ranges: 10.0.0.0/8 * Allowed Protocols or Port: tcp:1-65535;udp:1-65535;icmp </ins> ### Running a container (simple version) Once you have your instances up and running, the `build-go.sh` script sets up

Before enabling the cloud storage API, I'm unable to launch the test cluster. dev-build-and- appears to run successfully, but hangs at the end as the machines have not been properly configured. From on the kubernetes-master: AccessDeniedException: 403 Access Not Configured. Please go to the Google Developers Console () for your pro ject, select APIs and Auth and enable the Google Cloud Storage JSON API.


Question: apiVersion: v1 kind: Service metadata: <del> name: coredns </del> <ins> name: kube-dns </ins> namespace: kube-system labels: <del> k8s-app: coredns </del> <ins> k8s-app: kube-dns </ins> kubernetes.io/cluster-service: "true" addonmanager.kubernetes.io/mode: Reconcile kubernetes.io/name: "CoreDNS" spec: selector: <del> k8s-app: coredns </del> <ins> k8s-app: kube-dns </ins> clusterIP: {{ pillar['dns_server'] }} ports: - name: dns

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: e2e test "gci-gce" is running in 2 versions: with kube-dns as DNS Service : https://k8s- with CoreDNS as DNS Service : https://k8s- In CoreDNS use case the test is failing What you expected to happen: This e2e tests should not depend on what DNS service is running as default in K8s. CoreDNS is currently BETA, will move to GA soon (after v1.10 though) See : How to reproduce it (as minimally and precisely as possible): look at logs of dashboard : https://k8s- go run -v --test --test_args='--[sig-autoscaling]sDNSshorizontalsautoscalings[DisabledForLargeClusters]skube-dns-autoscalersshouldsscaleskube-dnsspodssinsbothsnonfaultysandsfaultysscenarios$' Anything else we need to know?: Environment: Kubernetes version (use ): branch master Cloud provider or hardware configuration: GCE Install tools: deployment intra-test Others: see https://k8s-
/sig autoscaling
/unsig cli
Looks like the test is searching specifically for a deployment labeled "kube-dns". There are several references to "kube-dns" scattered around it, actually.
Right now the tool "dns-horizontal-autoscaller" seems dedicated for kube-dns. This tools is deployed as "add-on" when enabled by this ENV variable : I guess we should have the definition of another dns-horizontal-autoscaller, dedicated to coredns (unless we generate the yaml file). and the choice of deployment is based on the ENV variable . see
Currently, the translation of the deployment of "dns-horizontal-autoscaller" for CoreDNS is in progress by It is expected that the corresponding e2e test will work as is (as the artifacts are selected by labels, which are the same for CroeDNS and kube-dns).


Question: "k8s.io/kubernetes/pkg/features" "k8s.io/kubernetes/pkg/volume" "k8s.io/kubernetes/pkg/volume/csi" <del> "k8s.io/kubernetes/pkg/volume/gcepd" </del> "k8s.io/kubernetes/pkg/volume/util" )

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --Context: When a cluster has dangling s caused by restarts of kube-controller-manager (like described in ), the attach/detach controller tries to clean them up on startup (ref ). For this, it iterates over all s and finds the responsible volume plugin for retrieving the unique volume name. Problem: However, if the volume was provisioned by an in-tree plugin, then later migrated to CSI and now the / feature gate is set, the volume is skipped during , so the will stay in the system and the volume will not be detached from the undesired node and potentially block a new attach operation to a different node. Analysis: The / feature gate causes the respective in-tree volume plugins not to be registered: The spec only contains the in-tree fields (e.g or ), but not the fields: This causes the call in to find no responsible volume plugin for the , which will skip the cleanup: That's why we see a lot of logs like this on startup of kube-controller-manager and dangling s on our clusters: The attach/detach controller should cleanup dangling s for migrated in-tree plugins with unregistered in-tree plugins as well. ref , addresses a similar issue. Though, the fix doesn't come into effect, if the respective / feature gate is enabled. Kubernetes version (use ): / Cloud provider or hardware configuration: AWS, GCP
/sig storage cc
Hey, thanks for reporting this bug!! I will try to come up with a fix. BTW, wondering what's the symptom you observed except for the log. Is there any volume operation(attaching/detaching/mounting/unmounting/creation/deletion) not working? This, to my understanding, will only make dangling attachment not working properly but I guess normal operation should work, is it?
The consequence of the leftover s in our case is that volumes are not detached on infrastructure provider side when moving pods with volumes between nodes, e.g.

because of rolling updates/eviction/etc.. This in turn causes the new s on the new nodes to fail with logs similar to this one: (we observe the same for GCP) I can also work on a fix tomorrow, if you want. I think, we should be able to reuse the fix from and additionally determine the from or similar, instead of only relying on .
That would be nice! Really appreciate it. Yep, sounds like a good idea. If you could also help to verify the fix in your environment that would be awesome
Ok, great. Will try to come up with a fix. /assign Definitely! I'll try my best and let you know if I need some help :)
Yep, we observed this after experiencing the bug. We ended up manually creating a cleanup script to check EC2 <-K8s VA if it was actually attached or not, otherwise delete VA. My understanding is with fixed we will no longer have this issue, but it would be nice to have the garbage collection logic work for migrated/non-migrated volumes regardless
/triage accepted


Question: expectNoError(err, "Failed to create rwPod") expectNoError(framework.WaitForPodRunning(rwPod.Name)) expectNoError(podClient.Delete(rwPod.Name, nil), "Failed to delete host0Pod") <ins> expectNoError(waitForPDDetach(diskName, host0Name)) </ins> By("submitting host0ROPod to kubernetes") _, err = podClient.Create(host0ROPod)

cc
There were: 2 failures as part of "gce-flaky" suite: 2170, 2171 a bunch of failures inside "gce-parallel-flaky" suite
I looked a bit into logs and in all cases it seems that we are simply not able to attach the volume. [Example logs from 2472 run in gce-parallel-flaky]: This log is repeating every ~10 seconds for 5 minutes and then the test fails.
Moving to post since AIUI this is a test stability fix.
I'm not sure - this might be either test problem or a problem with real code. I wasn't able to find the cause of it yet.
Haven't had a chance to dig in to this yet. But it may well be:
I don't think this is because all the failures I've seen here are at the level of attaching PDs, before any dettaching is done. Basically, what is happening here is: create one PD attach this PD in RW mode to pod1 on host1 attach this PD in RO mode to pod2 on host1 attach this PD in RO mode to pod3 on host2 only then do some deletions All the recent failure are before we even start deleting anything - it's just that we cannot start pod3 on host2, because it fails attaching PD to it.
I was trying to reproduce this issue locally, but unsuccessfully :( (although it is still failing on Jenkins). However, I don't think this is a test issue - I think this is a real issue. After looking through the code, I think that what is happening is: we send the request to attach a PD: it doesn't return an error, so it is "accepted" by GCE however, verifyAttached method is not able to verify it (disc is not being attached for the next 100 seconds): retrying that operation doesn't help: The fact that retrying doesn't help is slightly strange - maybe it's something wrong with the PD itself? Any other ideas?
BTW, this test is not unstable on : . Given the chasm, you're probably looking for something since the branchpoint ().
I still suspect it's . Although that is a detach issue, and the symptoms here are on attach, it could still be the same cause: i.e.

the PD fails to actually detach from the first node, therefore it fails to attach to the second node. I'll tackle this while or after I'm done with .
This test is failing on virtually every run in kubernetes-e2e-gce in the last 24 hours (except the ones that were red).
- thanks for working on this - I'm assigning this to you then.
I got a repro of this while testing the fix for by running the test for a few hours. It does appear to be a slightly different issue than that, but my hunch seems to be correct that the disk is not detached from the previous node therefore it can't attach to the new node. Here is what I think is happening: When the test moves the pod with the PD from the first host to the second, the pod on the second host remains in the pending state until the PD is successfully attached to the new host. However, that won't happen until the the PD is successfully detached from the original host. Detaching from the original host can take a variable amount of time and it appears that the test sometimes timesout before this happens. If that is true, then this is a test issue. And the fix would be to increase the amount of time this test waits for the pod to move out of pending state on the second host. I'm trying to capture another repro to verify my hunch. The logs from my repro below. Kubelet logs from Node 1: PD is successfully detached from node 1 @ 05:29:25. Kubelet logs from Node 2: Node 2 tries to attach PD starting at 05:24:02. Node 2 gives up trying to attach PD starting at 05:29:24.


Question: compareProjectsFiles(t, testcase, projectsFile, projidFile, enabled) if len(podQuotaMap) != expectedPodQuotaCount { fail = true <del> t.Errorf("Case %v (%s, %v) podQuotaCount mismatch: got %v, expect %v", seq, testcase.path, enabled, len(podQuotaMap), expectedPodQuotaCount) </del> <ins> t.Errorf("Case %v (%s, %s, %v) podQuotaCount mismatch: got %v, expect %v", seq, testcase.name, testcase.path, enabled, len(podQuotaMap), expectedPodQuotaCount) </ins> } if len(dirQuotaMap) != expectedDirQuotaCount { fail = true <del> t.Errorf("Case %v (%s, %v) dirQuotaCount mismatch: got %v, expect %v", seq, testcase.path, enabled, len(dirQuotaMap), expectedDirQuotaCount) </del> <ins> t.Errorf("Case %v (%s, %s, %v) dirQuotaCount mismatch: got %v, expect %v", seq, testcase.name, testcase.path, enabled, len(dirQuotaMap), expectedDirQuotaCount) </ins> } if len(quotaPodMap) != expectedQuotaPodCount { fail = true <del> t.Errorf("Case %v (%s, %v) quotaPodCount mismatch: got %v, expect %v", seq, testcase.path, enabled, len(quotaPodMap), expectedQuotaPodCount) </del> <ins> t.Errorf("Case %v (%s, %s, %v) quotaPodCount mismatch: got %v, expect %v", seq, testcase.name, testcase.path, enabled, len(quotaPodMap), expectedQuotaPodCount) </ins> } if len(dirPodMap) != expectedDirPodCount { fail = true <del> t.Errorf("Case %v (%s, %v) dirPodCount mismatch: got %v, expect %v", seq, testcase.path, enabled, len(dirPodMap), expectedDirPodCount) </del> <ins> t.Errorf("Case %v (%s, %s, %v) dirPodCount mismatch: got %v, expect %v", seq, testcase.name, testcase.path, enabled, len(dirPodMap), expectedDirPodCount) </ins> } if len(devApplierMap) != expectedDevApplierCount { fail = true <del> t.Errorf("Case %v (%s, %v) devApplierCount mismatch: got %v, expect %v", seq, testcase.path, enabled, len(devApplierMap), expectedDevApplierCount) </del> <ins> t.Errorf("Case %v (%s, %s, %v) devApplierCount mismatch: got %v, expect %v", seq, testcase.name, testcase.path, enabled, len(devApplierMap), expectedDevApplierCount) </ins> } if len(dirApplierMap) != expectedDirApplierCount { fail = true <del> t.Errorf("Case %v (%s, %v) dirApplierCount mismatch: got %v, expect %v", seq, testcase.path, enabled, len(dirApplierMap), expectedDirApplierCount) </del> <ins> t.Errorf("Case %v (%s, %s, %v) dirApplierCount mismatch: got %v, expect %v", seq, testcase.name, testcase.path, enabled, len(dirApplierMap), expectedDirApplierCount) </ins> } if len(podDirCountMap) != expectedPodDirCountCount { fail = true <del> t.Errorf("Case %v (%s, %v) podDirCountCount mismatch: got %v, expect %v", seq, testcase.path, enabled, len(podDirCountMap), expectedPodDirCountCount) </del> <ins> t.Errorf("Case %v (%s, %s, %v) podDirCountCount mismatch: got %v, expect %v", seq, testcase.name, testcase.path, enabled, len(podDirCountMap), expectedPodDirCountCount) </ins> } if len(quotaSizeMap) != expectedQuotaSizeCount { fail = true <del> t.Errorf("Case %v (%s, %v) quotaSizeCount mismatch: got %v, expect %v", seq, testcase.path, enabled, len(quotaSizeMap), expectedQuotaSizeCount) </del> <ins> t.Errorf("Case %v (%s, %s, %v) quotaSizeCount mismatch: got %v, expect %v", seq, testcase.name, testcase.path, enabled, len(quotaSizeMap), expectedQuotaSizeCount) </ins> } if len(supportsQuotasMap) != expectedSupportsQuotasCount { fail = true <del> t.Errorf("Case %v (%s, %v) supportsQuotasCount mismatch: got %v, expect %v", seq, testcase.path, enabled, len(supportsQuotasMap), expectedSupportsQuotasCount) </del> <ins> t.Errorf("Case %v (%s, %s, %v) supportsQuotasCount mismatch: got %v, expect %v", seq, testcase.name, testcase.path, enabled, len(supportsQuotasMap), expectedSupportsQuotasCount) </ins> } if len(backingDevMap) != expectedBackingDevCount { fail = true <del> t.Errorf("Case %v (%s, %v) BackingDevCount mismatch: got %v, expect %v", seq, testcase.path, enabled, len(backingDevMap), expectedBackingDevCount) </del> <ins> t.Errorf("Case %v (%s, %s, %v) BackingDevCount mismatch: got %v, expect %v", seq, testcase.name, testcase.path, enabled, len(backingDevMap), expectedBackingDevCount) </ins> } if len(mountpointMap) != expectedMountpointCount { fail = true <del> t.Errorf("Case %v (%s, %v) MountpointCount mismatch: got %v, expect %v", seq, testcase.path, enabled, len(mountpointMap), expectedMountpointCount) </del> <ins> t.Errorf("Case %v (%s, %s, %v) MountpointCount mismatch: got %v, expect %v", seq, testcase.name, testcase.path, enabled, len(mountpointMap), expectedMountpointCount) </ins> } if fail { <del> logAllMaps(fmt.Sprintf("%v %s", seq, testcase.path)) </del> <ins> logAllMaps(fmt.Sprintf("%v %s %s", seq, testcase.name, testcase.path)) </ins> } } os.Remove(projectsFile)

When sizeLimit is specified subsequent invocations will fail with the message: No error should be raised. Create a pod with emptyDir and sizeLimit with FSQuotaMonitoring enabled, MountVolume.SetUp is called again only if the first iteration takes longer then the actual reconciler loop interval (100ms). No response <detailsversion.Info{Major:"1", Minor:"27+", GitVersion:"v1.27.0-beta.0.87+", GitCommit:"", GitTreeState:"clean", BuildDate:"2023-04-10T09:05:56Z", GoVersion:"go1.20.2", Compiler:"gc", Platform:"linux/amd64"} </details<detailsSelf-managed cluster in Azure </details<detailsNAME="Red Hat Enterprise Linux" VERSION="9.1 (Plow)" ID="rhel" IDLIKE="fedora" VERSIONID="9.1" Linux test-master-0 6.3.0-rc3 SMP PREEMPTDYNAMIC Thu Apr 6 20:44:15 UTC 2023 x8664 x8664 x8664 GNU/Linux </details<detailsContainerd 1.6.15 </details<detailsCNI: CANAL 3.24.1 </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
May you please share more information? Which version of Kubernetes you are using? OS version etc.
The problem occurs with the latest master branch, I updated the initial post with versions.
Thank you.


Question: pvc := createPVC("pvc-rwm", "5G", []api.PersistentVolumeAccessMode{api.ReadWriteMany}) <del> w, _ := testClient.PersistentVolumes().Watch(api.ListOptions{}) </del> <ins> w, err := testClient.PersistentVolumes().Watch(api.ListOptions{}) if err != nil { t.Errorf("Failed to watch PersistentVolumes: %v", err) } </ins> defer w.Stop() <del> _, _ = testClient.PersistentVolumes().Create(pv_rwm) _, _ = testClient.PersistentVolumes().Create(pv_rwo) </del> <ins> _, err = testClient.PersistentVolumes().Create(pv_rwm) if err != nil { t.Errorf("Failed to create PersistentVolume: %v", err) } _, err = testClient.PersistentVolumes().Create(pv_rwo) if err != nil { t.Errorf("Failed to create PersistentVolume: %v", err) } </ins> <del> _, _ = testClient.PersistentVolumeClaims(api.NamespaceDefault).Create(pvc) </del> <ins> _, err = testClient.PersistentVolumeClaims(api.NamespaceDefault).Create(pvc) if err != nil { t.Errorf("Failed to create PersistentVolumeClaim: %v", err) } </ins> // wait until the controller pairs the volume and claim waitForPersistentVolumePhase(w, api.VolumeBound)

PersistentVolume integration tests fail sometimes: There are some interesting reports in the log like: Thans for report.
It turns out that volume integration tests have no logs at all, was not checked and I don't really know what's going on there. adds error checks to help me find out what's wrong there. Still, I suspect there are some troubles with etcd, not related to the test itself.
This flake affects also TestPersistentVolumeMultiPVs:
Extended logging got merged just now, please post links to logs when it breaks again!
Marking as P1 as per
I have no real usable logs, but I think it could be fixed by , it hasn't flaked since it was merged. And I have (in merge queue) to further stabilize the tests.
Not blocking this on 1.3.
It hasn't flaked for almost a month, can we close this?
Closing this issue since previous fixes made the flake go away.


Question: if _, err := fmt.Fprintf( w, "%st%st%dt%st%st%st%st%st%st%s", <del> FirstTimestamp, </del> LastTimestamp, <ins> FirstTimestamp, </ins> event.Count, event.InvolvedObject.Name, event.InvolvedObject.Kind,

Hello Kubernetes, The order of the columns when you get events confused me. I was thinking, Turns out it's just a visual problem where these events are sorted correctly (by ), but the first column is . I think the first column should be because that's the order the events are listed.
This also confused me when I first saw it - I think this ordering would be more intuitive and help users discover the aggregation feature logically, instead of first having to grapple with why events are "out of order"
Should the order be always LASTSEEN,FIRSTSEEN or just when using -w? I have an ugly PR for the latter, but perhaps we should just go for the former.


Question: import ( "fmt" "os" <ins> "os/exec" "path" "runtime" </ins> "strings" "github.com/golang/glog"

The goal of this issue is to design a CephFS driver using FUSE client for Kubernetes.</brExisting Ceph kernel client has the following drawbacks:</br• Kernel client does not support client-enforced quotas. • Kernel CephFS doesn't support quota or capacity, capacity requested by PVC is not enforced or validated. • Kernel CephFS doesn't work with SELinux, setting SELinux label in Pod's security context will not work.
Just a detail. Since we will rely on ceph-fuse, we will need the ceph-fuse client to be available on all kubernetes nodes. We can add a parameter to allow the use of the kernel client as well. I would say better make the fuse client opt-in and keep the kernel one as the default option. The kernel client might support quotas in the future. Thanks
/sig storage
We are also inclined to use fuse instead of kernel client ,any progress on this issue?
I like to continue this job and prepare to send a PR to make kubernetes support cephfs fuse mounting.
/assign I think this task would be valuable. =)
waiting for the PR, do you have a plane to support s3 fuse ?
No, no plan for s3fs-fuse currently
also see and cc
Ceph-fuse support in the command to display the specified secret, it may take some time to support, because it involves LibFUSE(I will follow up this demand). On the other hand, ceph-fuse has stability, quota function, expansion, upgrade and other advantages. Although ceph-fuse only supports the secret file way, but if it can be pushed to the k8s community, there is a very meaningful thing.


Question: } // All pvs and pvcs that we operated on are bound <del> klog.V(4).InfoS("All PVCs for pod are bound", "pod", klog.KObj(pod)) </del> <ins> klog.V(2).InfoS("All PVCs for pod are bound", "pod", klog.KObj(pod)) </ins> return true, nil }

This line reads as if the bound has already happened: But, if I read correctly, the bound happens immediately after: And there is even a wait for confirmation after: The log line to better describe the actual state and another log line at V(2) to confirm that the operation ended. No response all supported versions <details</details<details</details<details</details<details</details<details</details
/sig storage
cc
/triage accepted


Question: Follow         bool         `json:"follow,omitempty"` Previous       bool         `json:"previous,omitempty"` SinceSeconds   *int64       `json:"sinceSeconds,omitempty"` <ins> TailLines      *int64       `json:"tailLines,omitempty"` </ins> SinceTime      *metav1.Time `json:"sinceTime,omitempty"` EmptyTime      *metav1.Time `json:"emptyTime"` NonPointerTime metav1.Time  `json:"nonPointerTime"`

How works: : prints all lines. : prints the last line. : prints no lines. (Usually used with to follow new logs) How works: : prints all lines. : prints the last line. : prints all lines. <-------------- Diffierent. The reason is that the is ignored when converting the parameters Confirmed with What is the expected behavior? /cc
I've also noticed that even if or some arbitrarily low number is specified, it tries to read the whole logfile into memory. If it's a big file it can crash the worker node.
Do you mean the docker implementation? If I read their code correctly, docker read the whole log before tail into memory once tail is specified, which may cause problem if tail is a very large number. So in our own implementation, I've avoided that. ;) However, where do you see the ?
Tail should be *int in the API, is it not? On Oct 21, 2016, at 6:01 PM, Lantao Liu wrote: How docker logs works: docker logs --tail=-1 CONTAINERID: prints all lines. docker logs --tail=1 CONTAINERID: prints the last line. docker logs --tail=0 CONTAINERID: prints no lines. (Usually used with -f to follow new logs) How kubectl logs works: kubectl logs --tail=-1 PODNAME: prints all lines. kubectl logs --tail=1 PODNAME: prints the last line. kubectl logs --tail=0 PODNAME: prints all lines. <-------------- Diffierent. The reason is that the tail=0 is ignored when converting the parameters . Confirmed with /cc — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub , or mute the thread .
Agree tail=0 no logs is intended behavior On Oct 21, 2016, at 6:01 PM, Lantao Liu wrote: How docker logs works: docker logs --tail=-1 CONTAINERID: prints all lines. docker logs --tail=1 CONTAINERID: prints the last line. docker logs --tail=0 CONTAINERID: prints no lines. (Usually used with -f to follow new logs) How kubectl logs works: kubectl logs --tail=-1 PODNAME: prints all lines.

kubectl logs --tail=1 PODNAME: prints the last line. kubectl logs --tail=0 PODNAME: prints all lines. <-------------- Diffierent. The reason is that the tail=0 is ignored when converting the parameters . Confirmed with /cc — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub , or mute the thread .
Yeah, tail is . But if , kubelet will see .
I think the linked to is the problem.
The obvious workaround is to instead use: kubectl logs --tail=1
/sig cli
The line pasted by chao implies omitempty should not be false for tail.
I hate to bump, but 'bump'. Still an issue here. Can confirm works but does not.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale This is still relevant.
Any good news?


Question: visible at runtime in the container. release: v1.9 file: test/e2e/common/node/downwardapi.go <ins> - testname: Ephemeral Container, update ephemeral containers codename: '[sig-node] Ephemeral Containers [NodeConformance] should update the ephemeral containers in an existing pod [Conformance]' description: Adding an ephemeral container to pod.spec MUST result in the container running. There MUST now be only one ephermal container found. Updating the pod with another ephemeral container MUST succeed. There MUST now be two ephermal containers found. release: v1.28 file: test/e2e/common/node/ephemeral_containers.go </ins> - testname: Ephemeral Container Creation codename: '[sig-node] Ephemeral Containers [NodeConformance] will start an ephemeral container in an existing pod [Conformance]'

[x] test approval issue : [x] test pr : 18 May 2023 [x] two weeks soak end date : 01 June 2023 [x] test promotion pr : According to following APIsnoop query, there are still 2 PodEphemeralcontainers endpoints that are not tested for Conformance. Using a number of existing e2e test practices a new has been created to address these 2 endpoints. The e2e logs for this test are listed below. This query shows the following podephemeralcontainers endpoints are hit within a short period of running this e2e test. If a test with these calls gets merged, test coverage will go up by 2 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted cc
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: maxResults = -1 } <del> // NOTE: Since we store the elments in descending timestamp order "start" will // be a higher index than "end". </del> var startIndex int if start.IsZero() { // None specified, start at the beginning.

I have some reports of a Node reporting a a resource capacity for cpu as a negative value. Any idea why cpu could report as a negative value? Is this an expected normal outcome? In its current state, it prevents any pods from being scheduled to that Node. Tips on how to debug further are appreciated. - any ideas?
Can you check machineInfo on Cadvisor or kubelet on the node?
a.k.a:
It is a bug in cAdvisor side. I am assigning it to you :-)
here is contents of /proc/cpuinfo Pertinent pieces of a log:
Machine is AWS instance if that matters....
We build the topology from system/cpu info. I think we can return partial topology information and ignore caches if its not present. This isn't a recent change though. Is it happening on just a single machine? I'll file a cAdvisor issue.
Can you link the referenced issue once created?
Re-assigning to since he's working on google/cadvisor
Could be this was caused by getting a Node on EC2 with old hypervisor, and I am the first victim...
That sounds likely. Most of the cache information is there in /proc/cpuinfo. So looks like the sysfs bits are disabled.
Can you try this cadvisor image on your machine and check /api/v1.3/machine output: sudo docker run  --volume=/:/rootfs:ro  --volume=/var/run:/var/run:rw  --volume=/sys:/sys:ro  --volume=/var/lib/docker/:/var/lib/docker:ro  --publish=8080:8080  --detach=true  --name=cadvisor  jnagal/cadvisor:canary
hmm
ah, selinux :( You'll need to run it with --privileged, I think. who's been dealing with it.
No luck with --privileged either on this box...
This has to do with SELinux, I think we had to disable it for this to work (some of our mounts are being denied). We're working on a better solution... This is not a problem in Kubernetes since we don't run Kubelet/cAdvisor in a Docker container.
ok, unfortunately, i have managed to kill docker on the box in question, and i cannot get the daemon to restart right now so I am not sure when that box will be available to debug against again.

I can provide an environment with a similar issue if required.
The fix is checked in to head. Can you build the binary and try?


Question: RuntimeCgroupsName:    s.RuntimeCgroups, SystemCgroupsName:     s.SystemCgroups, KubeletCgroupsName:    s.KubeletCgroups, <ins> KubeletOOMScoreAdj:    s.OOMScoreAdj, </ins> CgroupsPerQOS:         s.CgroupsPerQOS, CgroupRoot:            s.CgroupRoot, CgroupDriver:          s.CgroupDriver,

runs in a container. Talos sets OOM score to -450 for the kubelet container and passes in the kubelet config. End result is that OOM score adj is -999 though. should have configured OOM score . Source code analysis. First, OOM score is set correctly from the config (flags): But then, in container manager, OOM score is reset to -999 via the following: This function should skip action if the PID is not running on the host: But the function looks to be confusing: When kubelet runs in a container with PID namespace shared with the host, kubelet's PID namespace will be equal to PID 1 namespace. If kubelet runs in a container with unshared PID namespace, kubelet's PID is 1 and its PID namespace is identical to PID 1 namespace. So how this function can return ever? Moreover, function is always setting OOM score to a hardcoded constant -999 ignoring configuation (). No response <details, reproduced with 1.22. </details<details</details<details</details<details</details<details</details<details</details
/sig node
/triage accepted
/assign
I don't think this method needs to change... I have make a draft PR for the default value to use the flag if present, but I would like more time to add some testing. Maybe you can have a look: ?
Thanks, this PR looks good to me, as it's going to set the OOM score to the configured value. And in our case kubelet runs in a container (not on the host, with proper OOM score via OCI spec), so this code should be even skipped, not sure how this can be addressed.
neither do I...
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale


Question: }, expectedErrs: field.ErrorList{}, }, <del> "class field and annotation set": { </del> <ins> "class field and annotation set with same value": { tweakIngress: func(ingress *networking.Ingress) { ingress.Spec.IngressClassName = utilpointer.String("foo") ingress.Annotations = map[string]string{annotationIngressClass: "foo"} }, expectedErrs: field.ErrorList{}, }, "class field and annotation set with different value": { </ins> tweakIngress: func(ingress *networking.Ingress) { ingress.Spec.IngressClassName = utilpointer.String("bar") ingress.Annotations = map[string]string{annotationIngressClass: "foo"} }, <del> expectedErrs: field.ErrorList{field.Invalid(field.NewPath("annotations").Child(annotationIngressClass), "foo", "can not be set when the class field is also set")}, </del> <ins> expectedErrs: field.ErrorList{field.Invalid(field.NewPath("annotations").Child(annotationIngressClass), "foo", "must match `ingressClassName` when both are specified")}, </ins> }, "valid regex path": { tweakIngress: func(ingress *networking.Ingress) {

An ingress resource with both annotation and set cannot be created. Ingress resources can be updated to include both the annotation and the without issue. Kubernetes response: The ingress should be deployed with both annotation and set, as it is supported by updating an existing ingress. Apply this resource: The code causing this issue is here: <details</details<details</details<detailsLinux, Mac, mulitple OSs with the same issue. It's caused by code as linked above </details<details</details<details</details<details</details
/sig network
that code referenced is only used on Create update has a different path is this based on an observation of the code or are you hitting this problem?
, correct. This only happens on create. Our work around now is to create an ingress resource with the annotation, and then update the resource, adding the . This involves 2 MRs and approval processes We are hitting this problem regularly as we scale out the number of kubernetes clusters we operate, and create new ingress resources. It's quite painful as we're starting to use , but cannot drop the ingressClass annotation yet as some third party services have not ported over to use the .
but why do you create it with the annotaiton?
Sorry, I don't understand your question. I need to be able to create an ingress resource with both the ingressClass annotation () and both set
yeah, that is my question, if you are creating the ingress with v1 you don't need the annotation, it was deprecated by the field, the annotation has to be there for the people that upgrades from v1beta1 to v1
As far as I know, External DNS still requires the annotation. We use external DNS, and thus cannot drop the annotation yet
I think what you're doing as a workaround is a valid approach . A better fix would be for the ingress controller you're using to migrate to properly use the v1 API for Ingress and IngressClass. Please consider filing a bug against that ingress controller or other component. We could also return a header when someone sets the annotation on an Ingress that already has set, to make it clear that this is not recommended.
It seems that PR got stuck , but is not a kubernets bug
IMO we should probably not hard-fail if these are both set, but a warning would be appropriate.

, i'd completely agree. Kubernetes is happy to have both set side by side, but the logic linked in the first comment blocks it. Seemingly needlessly.
Discussed at sig-net. We don't object to loosening this rule - an API warning seems sufficient. PRs welcome /triage accepted
Hi , I'm willing to help with this issue. I'm not familiar with , I read the relevant code and I think it seems that to add an API warning, I just need to implement for in . Am I right? And in the same time, don't check these two fields in validation?
Pretty much! See how Service works (though you probably don't need to put it into a different package, not sure why we did that..) kidddddddddddddddddddddd <
/assign


Question: Time metav1.Time `json:"time,omitempty" protobuf:"bytes,1,opt,name=time"` // String detailing the error encountered during Attach or Detach operation. <del> // This string maybe logged, so it should not contain sensitive </del> <ins> // This string may be logged, so it should not contain sensitive </ins> // information. // +optional Message string `json:"message,omitempty" protobuf:"bytes,2,opt,name=message"`

This came up as a follow up to Also rather than opening 5 different issues - we are going to use this github issue to track all follow up items mentioned here -
/sig storage
/assign /assign
: PR
and the rest is fixed in


Question: filegroup( name = "all-srcs", <del> srcs = [":package-srcs"], </del> <ins> srcs = [ ":package-srcs", "//pkg/kubelet/kuberuntime/logs:all-srcs", ], </ins> tags = ["automanaged"], )

We should make the CRI log parsing code in kubelet to be a library. So that container runtimes and external tools don't need to copy the whole code ( vendor it instead ). Referring
/assign
/sig node
I can help on it.
/assign
GitHub didn't allow me to assign the following users: yanxuean. Note that only can be assigned. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Thanks. I have a branch ready for this. Will open a PR later and ping you when it is out.
ok


Question: }, ID:         &ID, InstanceID: &instanceID, <del> Location:   &ss.Cloud.Location, </del> <ins> Location:   &ss.Location, </ins> } } virtualMachineScaleSetVMsClient.setFakeStore(ssVMs) <del> ss.Cloud.VirtualMachineScaleSetsClient = virtualMachineScaleSetsClient ss.Cloud.VirtualMachineScaleSetVMsClient = virtualMachineScaleSetVMsClient </del> <ins> ss.VirtualMachineScaleSetsClient = virtualMachineScaleSetsClient ss.VirtualMachineScaleSetVMsClient = virtualMachineScaleSetVMsClient </ins> } func TestGetScaleSetVMInstanceID(t *testing.T) {

/kind bug What happened: https://k8s-
/sig azure
I find the root cause, and can fix it.
/kind bug
/cc
Cool, thanks


Question: return nil, schema.GroupResource{}, firstErr } <del> // setCurrentReplicasInStatus sets the current replica count in the status of the HPA. func (a *HorizontalController) setCurrentReplicasInStatus(hpa *autoscalingv2.HorizontalPodAutoscaler, currentReplicas int32) { a.setStatus(hpa, currentReplicas, hpa.Status.DesiredReplicas, hpa.Status.CurrentMetrics, false) </del> <ins> // setCurrentReplicasAndMetricsInStatus sets the current replica count and metrics in the status of the HPA. func (a *HorizontalController) setCurrentReplicasAndMetricsInStatus(hpa *autoscalingv2.HorizontalPodAutoscaler, currentReplicas int32, metricStatuses []autoscalingv2.MetricStatus) { a.setStatus(hpa, currentReplicas, hpa.Status.DesiredReplicas, metricStatuses, false) </ins> } // setStatus recreates the status of the given HPA, updating the current and

I utilize two customized metrics, A and B, in my HPA system. A is a gauge-based metric called SLA Metric, while B is a count-based metric that tracks failed requests with HTTP status code 502 or 503 from Istio. These metrics are scraped by Prometheus. To use custom metrics in HPA, we're employing Kube Metrics Adapter . When the application load increases, the value of the SLA Metric also increases, and then the pods scale up until they reach the maximum replica count as expected. However, the problem arises when the load dissipates, and the pods never scale down. Despite the SLA Metric's value being below the target in Prometheus, the HPA description still displays the metric value with a stale value that can be above or below the target. One possible reason for this is that Metric B, which relies on Istio requests, shows up as unknown since there have been no failed requests with the 502 or 503 status codes. Thus, the Prometheus query fails. We have noticed this behavior after upgrading Kube from version 1.21 to 1.24 and changing the HPA version from autoscaling/v2beta2 to autoscaling/v2. kubectl describe hpa my-hpa To troubleshoot this further we checked the metrics value using Output: Although the metric value appears as zero, we can observe that the HPA description displays a stagnant value. Workaround : HPA behaves as expected when the second metric B is completely removed or modified to return 0 when the query fails. HPA should scale down properly based on one of the metrics, even when the other metric value is not available. Set up the Kube metrics adapter . Create a custom-metric-based HPA that uses two metrics among which the value of one is undefined. Increase the load i.e., the value of the other metric so that the HPA kicks in and scales up the pods to the max replica count. Reduce the load i.e. the value of the metric, it will be stuck at a random value. Does anyone similar issues in hpa or is this behavior of hpa for multiple metrics changed recently, especially in scaling down events? Can anyone from the community look into the issue and give some clarity?

<detailsv1.24.7 </details<detailsEKS </details<detailsAlpine Linux </details<details</details<details</details<details</details
/sig autoscaling
About the problem that the HPA description displays a stagnant value, we have seen a 1.24 cluster exposing a similar behavior and here are the notes I took when diagnosing the issue. As far as I can see from reading the code the same behavior is present in more recent k8s versions as well (1.28+). The metric information comes from the hpa.Status.CurrentMetrics field of the hpa object. HPA controller loop updates hpa.Status.CurrentMetrics in setStatus [1] which is at the very end [2] of the hpa reconcile loop reconcileAutoscaler [3]. When a custom metric is in error and can't be retrieved, computeReplicasForMetric [4] will return an error back to computeReplicasForMetrics which in turn returns an error IF the hpa replica count is bellow specReplica based on the remaining available metrics [5]. Due to that error, reconcileAutoscaler returns before it calls setStatus [6] and that's why the currentMetrics field isn't updated. Note that reconcileAutoscaler does not uses hpa.Status.CurrentMetrics to make scaling decisions, the reconcilier pull the metrics from the metric server at each evaluation [7] [8] [9] so scale-ups will keep happening based on the available metrics. On the other hand, HPA not scaling down on incomplete metrics make sense as it cannot know the conditions are met for a scaledown unless it retrieves all the metrics. [1] [2] [3] [4] [5] [6] [7] [8] [9]
/assign
/sig instrumentation
Thanks for the detailed explanation with code snippets. Now I am able to get a high level overview of what had happened after reading your comments. Just update here once the issue is fixed.
/assign /triage accepted
Thanks for the detailed explanation, awaiting for this issue to be resolved.


Question: var err error framework.AllNodesReady(cs, time.Minute) <del> masterNodes, nodeList, err = e2enode.GetMasterAndWorkerNodes(cs) </del> <ins> // NOTE: Here doesn't get nodeList for supporting a master nodes which can host workload pods. masterNodes, _, err = e2enode.GetMasterAndWorkerNodes(cs) </ins> if err != nil { e2elog.Logf("Unexpected error occurred: %v", err) } <ins> nodeList, err = e2enode.GetReadySchedulableNodesOrDie(cs) if err != nil { e2elog.Logf("Unexpected error occurred: %v", err) } </ins> // TODO: write a wrapper for ExpectNoErrorWithOffset() framework.ExpectNoErrorWithOffset(0, err)

What happened: The conformance test "validates resource limits of pods that are allowed to run" is failed if master node doesn't have taints. What you expected to happen: All conformance tests should pass even if a master node doesn't have taints. How to reproduce it (as minimally and precisely as possible): If a master node has taints like many cases, the test can pass like the following: However if the master node doesn't have any taints, the test cannot pass like: Anything else we need to know?: I am guessing this issue can be related to Environment: Kubernetes version (use ): v1.15.1 Cloud provider or hardware configuration: OpenStack OS (e.g: ): Ubuntu 18.04.2 LTS Kernel (e.g. ): 4.15.0-46-generic Install tools: kubeadm Network plugin and version (if this is a network-related bug): Others:
/sig scheduling
The test log contains only, that means the test doesn't create a pod on the master node which doesn't have any taints when this issue happens. The following log is the part of when the issue happened. Important line is which says additional-pod which was expected to be failed to be schedule is assigned to the master node.
nodeList is gotten from as in BeforeEach() step. And as I said on the function is problematic now.
The discussion on the slack:


Question: ExperimentalHostUserNamespaceDefaultingGate: {Default: false, PreRelease: utilfeature.Beta}, ExperimentalCriticalPodAnnotation:           {Default: false, PreRelease: utilfeature.Alpha}, Accelerators:                                {Default: false, PreRelease: utilfeature.Alpha}, <del> DevicePlugins:                               {Default: false, PreRelease: utilfeature.Alpha}, </del> <ins> DevicePlugins:                               {Default: true, PreRelease: utilfeature.Beta}, </ins> TaintBasedEvictions:                         {Default: false, PreRelease: utilfeature.Alpha}, RotateKubeletServerCertificate:              {Default: false, PreRelease: utilfeature.Alpha}, RotateKubeletClientCertificate:              {Default: true, PreRelease: utilfeature.Beta},

/kind feature What happened: The DevicePlugins feature was introduced in 1.8 Kubernetes release as an alpha feature. Since then, we have seen good development work on stabling the feature as well as early adopters on using the framework to support device resources in Kubernetes. This issue is to track the work requirements of entering this feature into beta. FYI, here is list of requirements from the OSS Kubernetes feature stage doc: """ Beta level: Object Versioning: API version name contains beta (e.g. v2beta3) Availability: in official Kubernetes releases, and enabled by default Audience: users interested in providing feedback on features Completeness: all API operations, CLI commands, and UI support should be implemented; end-to-end tests complete; the API has had a thorough API review and is thought to be complete, though use during beta may frequently turn up API issues not thought of during review Upgradeability: the object schema and semantics may change in a later software release; when this happens, an upgrade path will be documented; in some cases, objects will be automatically converted to the new version; in other cases, a manual upgrade may be necessary; a manual upgrade may require downtime for anything relying on the new feature, and may require manual conversion of objects to the new version; when manual conversion is necessary, the project will provide documentation on the process Cluster Reliability: since the feature has e2e tests, enabling the feature via a flag should not create new bugs in unrelated features; because the feature is new, it may have minor bugs Support: the project commits to complete the feature, in some form, in a subsequent Stable version; typically this will happen within 3 months, but sometimes longer; releases should simultaneously support two consecutive versions (e.g. v1beta1 and v1beta2; or v1beta2 and v1) for at least one minor release cycle (typically 3 months) so that users have enough time to upgrade and migrate objects Recommended Use Cases: in short-lived testing clusters; in production clusters as part of a short-lived evaluation of the feature in order to provide feedback """ I think we met most of the requirements here, with the commit that if we need to make incompatible API change in the future, we will document the upgrade path and if necessary, may support multiple API versions in a couple of releases.

Things that I think would be nice to have to enter beta (entering beta may not necessary depends on their completeness) Regarding to the thorough API review requirement, I think we had good discussions during the design proposal but there have been some recent discussions on whether we should use registration vs probe model for the general Kubelet plugin communication. I think we should have a general discussion on this topic with larger audience like people working on CSI plugins. I will create a FR issue to track this discussion soon. We would like to see multiple device plugin implementations. We know some people have started working on device plugin implementations for Solarflare, FPGA, and Infiniband, and probably want to keep closer track on their progresses. We would like to target this FR to 1.10. Please let us know your thoughts or add things that you think are critical to finish. Thanks! What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig node
I'm a fan of probe model. I do feel it unfriendly to plugin developers to let them care about the re-registration after kubelet's restart. And have received complaint from my colleague.
Hi, for cluster reliability, I think we need more test cases, including scale and soak tests, to simulate more production scenarios.
can we consolidate this with ?
Sure. Will do this with Vish after KubeCon.
Sorry to bring up this topic a bit late (feature freeze is the 22nd of Jan). Here are the features we (NVIDIA) consider important or blocker before entering beta: [BLOCKER] Per container allocation Add labels during device plugin registration (e.g: gpu-family: tesla) Add annotations to a container for beta and add an option to CRI for GA to support OCI hooks Things that are not explicitly related to the device plugin: We're also wondering if we should tie up
Thanks a lot for your input, For per container allocation, I think we should be able to implement it in 1.10. I think we should be able to support ER quota in 1.10 as well.

I think allowing device plugins to export device properties is an important feature, especially when we introduce ResourceClass, but I don't think it is a beta blocker. For "Add annotations to a container for beta and add an option to CRI for GA to support OCI hooks", this seems outside the scope of device plugin. There are different ways to implement them (if they have to be implemented) on Kubernetes. We haven't seen strong use cases indicating that they have to go through device plugin.
Sorry, I meant from the device plugin. i.e: the device plugin's returns container annotations for beta Product wise this would allow us to have the nvidia-runtime be supported by CRI-O (which allows hooks through container annotations) as well as docker + nvidia as the default runtime. BTW That would allow a convergence towards a single device plugin in beta. As for GA this would be a different discussion but it would be part of converging towards using any runtime and have them support the nvidia-hooks.
Not saying it's a beta blocker but it would be useful to have labels on homogeneous nodes about what devices are present. Especially: GPU memory GPU compute capability ... It's also not a big change, it just requires consensus
As a heads up, I am going to send a PR soon to change DevicePlugins feature to beta. After introducing the feature in 1.8, we have seen good development work on improving the feature and received many valuable feedbacks from early adopters on how we may extend the framework to support various kinds of resources on Kubernetes. To us, entering beta means: a) We think we have done enough early experiments during alpha phase to get the confidence that this a useful feature that we are willing to carry on to beta and to GA. b) We think the current DevicePlugins is ready to be tried on production. It has good e2e and e2enode test coverage. Some early adopters have tried it on production and didn't report any major breakage. Very likely we will keep seeing new feature requests coming in after beta, as more people try to use DevicePlugins for their specific use cases. Some of these features may even require incompatible API changes. When this happens, the required complexity to support multiple device plugin API versions on Kubelet side should be manageable and mostly local to the devicemanager component.

It should also be manageable for a device plugin to support multiple API versions. As an example, in , I extended gke gpu device plugin to support both v1alpha and v1beta1 versions. For reference, here is list of changes we made on device plugin during 1.10: Support resource quota on extended resources (Issue ). Owner: Added PreStartContainer grpc function to allow device plugins to perform per container start operations. Owner: Add annotations to the device plugin API. Owner: Track unhealthy devices in devicemanager, and have node status capacity to reflect raw capacity but allocatable to reflect healthy device capacity. Owner: Stub device plugin based e2enode test Owner: There are a few pending PRs that would be nice to get into 1.10 but we don't consider as beta blocker: Determine and implement a general Kubelet plugin communication establish model and use it in device plugin. Owner: Uses common checkpointing mechanism in device plugin (PR Issue ) Owner: Device plugin stress test. Owner: Please let us know if anyone has objections on entering the feature to beta in 1.10. Thanks!
Hi, for stress test, here is a PR :)
/cc
Should we reopen this tracing issue?
I think this issue can be closed with We can open separate issues to track future device plugin changes.


Question: _, err = framework.NewKubectlCommand(ns, nsFlag, "run", "-i", "--image="+busyboxImage, "--restart=OnFailure", "failure-2", "--", "/bin/sh", "-c", "cat && exit 42"). WithStdinData("abcd1234"). Exec() <del> framework.ExpectNoError(err) </del> <ins> ee, ok = err.(uexec.ExitError) framework.ExpectEqual(ok, true) if !strings.Contains(ee.String(), "timed out") { framework.Failf("Missing expected 'timed out' error, got: %#v", ee) } </ins> ginkgo.By("running a failing command without --restart=Never, but with --rm") _, err = framework.NewKubectlCommand(ns, nsFlag, "run", "-i", "--image="+busyboxImage, "--restart=OnFailure", "--rm", "failure-3", "--", "/bin/sh", "-c", "cat && exit 42"). WithStdinData("abcd1234"). Exec() <del> framework.ExpectNoError(err) </del> <ins> ee, ok = err.(uexec.ExitError) framework.ExpectEqual(ok, true) if !strings.Contains(ee.String(), "timed out") { framework.Failf("Missing expected 'timed out' error, got: %#v", ee) } </ins> e2epod.WaitForPodToDisappear(f.ClientSet, ns, "failure-3", labels.Everything(), 2*time.Second, wait.ForeverTestTimeout) ginkgo.By("running a failing command with --leave-stdin-open")

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: gce-master-scale-correctness Which test(s) are failing: [sig-cli] Kubectl client Simple pod should return command exit codes Since when has it been failing: 02/05 Testgrid link: https://k8s- Reason for failure: Anything else we need to know: /sig cli
/priority critical-urgent /assign
The test seems to be flaking quite a lot recently ! Friendly ping, sig-cli, could you take a look?
FTR, these are really blocking tests and it's getting more urgent given the forthcoming 1.18 release
Full log of the failure: It looks like the finishes with error. Strange, I thought is supposed to create a pod. Would really appreciate if someone from sig-cli could take a look
4 out of last 8 runs failed: https://k8s- /sig scalability
Another flaky, related test:
SIG CLI Leads -- any update on this? This is affecting scalability signal. /assign
-- Please keep tabs on this one.
yep, monitoring
It looks like there's a race between when the pod gets created and when we try to connect to it, thus we're seeing but further down in the logs the pod is , I'll be looking more into ti.
Thanks, Any update on this?
/cc
Sorry I got sidetracked yesterday and I'm back investigating this just now...
Probable fix in
Will continue to monitor here and reopen if needed :+1:
Unfortunately - it seems that the problem still exists - the test failed today (an hour ago): :( ^^
I'll check it, thx Wojtek for pointing that out.
- I'm not familiar with kubectl code, but I took a quick look and what I found is that this waitForPod() thing is happening only for pods with RestartPolicyNever: I must be missing something here, but for the case that failed it's not really the case, which means we don't wait for it?
This particular one is using , see the actual invocation that failed: That's why I started with this approach.
Sorry - the one for which the bug was originally opened.

But the today`s failure is actually slightly different (the same test, different case - sorry): The restart policy for this one is
Gotcha, so it looks like we progressed, I'll check the possible paths for this one, something tells me we're missing another timeout somewhere :wink:
OK - I think I know what is going on here. Here is what is happening (for the test case from two comments above): the test is setting "-i", so interactive mode is switched on that option triggers that Attach option defaults to true: over the Run() command, we get to the handleAttachPod: handleAttachPod is calling waitForPod And waitForPod is broken. The problem is the preconditionFunc: which is below passed to the UntilWithSync function UntilWithSync underneath is creating an informer and calling that function against its state: But the informer is initialized from apiserver watchcache, so it may be lagging. In large clusters we are observing lags in the order of small hundreds of milliseconds and that seem to be enough. What is happening is that: informer is initialized (but the pod creation is not yet reflected in it) the preconditionFunc is called the state doesn't contain that pod (yet) the whole thing fails with that error I actually believe, that just removing the preconditionFunc should actually solve the problem. I opened for that.
^^ - please take a look at the PR that I assigned to you (it's literally one-liner as of now - if you want to add tests or anything like that, feel free to pick it up).
Actually I went ahead, and fixed couple other commands too - so it's no longer one-liner. But still short one :)


Question: } w.fsWatcher = fsWatcher <ins> // Traverse plugin dir and add filesystem watchers before starting the plugin processing goroutine. if err := w.traversePluginDir(w.path); err != nil { w.Stop() return fmt.Errorf("failed to traverse plugin socket path %q, err: %v", w.path, err) } // Traverse deprecated plugin dir, if specified. if len(w.deprecatedPath) != 0 { if err := w.traversePluginDir(w.deprecatedPath); err != nil { w.Stop() return fmt.Errorf("failed to traverse deprecated plugin socket path %q, err: %v", w.deprecatedPath, err) } } </ins> w.wg.Add(1) go func(fsWatcher *fsnotify.Watcher) { defer w.wg.Done() for { select { case event := <-fsWatcher.Events: //TODO: Handle errors by taking corrective measures <del> w.wg.Add(1) func() { defer w.wg.Done() if event.Op&fsnotify.Create == fsnotify.Create { err := w.handleCreateEvent(event) if err != nil { klog.Errorf("error %v when handling create event: %s", err, event) } } else if event.Op&fsnotify.Remove == fsnotify.Remove { err := w.handleDeleteEvent(event) if err != nil { klog.Errorf("error %v when handling delete event: %s", err, event) } </del> <ins> if event.Op&fsnotify.Create == fsnotify.Create { err := w.handleCreateEvent(event) if err != nil { klog.Errorf("error %v when handling create event: %s", err, event) </ins> } <del> return }() </del> <ins> } else if event.Op&fsnotify.Remove == fsnotify.Remove { err := w.handleDeleteEvent(event) if err != nil { klog.Errorf("error %v when handling delete event: %s", err, event) } } </ins> continue case err := <-fsWatcher.Errors: if err != nil {

What happened: A race condition in the plugin watcher might happen when a is created, started and immediately stopped and cleaned up. If executed several times, the unit test will expose this issue. The PR addressed part of the problem by serializing the handling of CREATE and DELETE events. However, a race might still happen in the function. describes in detail the problem. How to reproduce it (as minimally and precisely as possible): /sig storage /sig node
/assign


Question: }, } <ins> service6 := getTestService("test1", v1.ProtocolUDP, nil, 80) lb6 := getTestLoadBalancer(to.StringPtr("testCluster"), to.StringPtr("testCluster"), to.StringPtr("atest1"), service6, "basic") lb6.FrontendIPConfigurations = &[]network.FrontendIPConfiguration{} lb6.Probes = &[]network.Probe{} expectedLB6 := getTestLoadBalancer(to.StringPtr("testCluster"), to.StringPtr("testCluster"), to.StringPtr("atest1"), service6, "basic") expectedLB6.Probes = &[]network.Probe{} (*expectedLB6.LoadBalancerPropertiesFormat.LoadBalancingRules)[0].Probe = nil (*expectedLB6.LoadBalancerPropertiesFormat.LoadBalancingRules)[0].EnableTCPReset = nil (*expectedLB6.LoadBalancerPropertiesFormat.LoadBalancingRules)[0].DisableOutboundSnat = to.BoolPtr(false) expectedLB6.FrontendIPConfigurations = &[]network.FrontendIPConfiguration{ { Name: to.StringPtr("atest1"), FrontendIPConfigurationPropertiesFormat: &network.FrontendIPConfigurationPropertiesFormat{ PublicIPAddress: &network.PublicIPAddress{ID: to.StringPtr("/subscriptions/subscription/" + "resourceGroups/rg/providers/Microsoft.Network/publicIPAddresses/pipName")}, }, }, } service7 := getTestService("test1", v1.ProtocolUDP, nil, 80) service7.Spec.HealthCheckNodePort = 10081 service7.Spec.ExternalTrafficPolicy = v1.ServiceExternalTrafficPolicyTypeLocal lb7 := getTestLoadBalancer(to.StringPtr("testCluster"), to.StringPtr("testCluster"), to.StringPtr("atest1"), service7, "basic") lb7.FrontendIPConfigurations = &[]network.FrontendIPConfiguration{} lb7.Probes = &[]network.Probe{} expectedLB7 := getTestLoadBalancer(to.StringPtr("testCluster"), to.StringPtr("testCluster"), to.StringPtr("atest1"), service7, "basic") (*expectedLB7.LoadBalancerPropertiesFormat.LoadBalancingRules)[0].Probe = &network.SubResource{ ID: to.StringPtr("/subscriptions/subscription/resourceGroups/rg/providers/Microsoft.Network/loadBalancers/testCluster/probes/atest1-UDP-80"), } (*expectedLB7.LoadBalancerPropertiesFormat.LoadBalancingRules)[0].EnableTCPReset = nil (*expectedLB7.LoadBalancerPropertiesFormat.LoadBalancingRules)[0].DisableOutboundSnat = to.BoolPtr(false) expectedLB7.FrontendIPConfigurations = &[]network.FrontendIPConfiguration{ { Name: to.StringPtr("atest1"), FrontendIPConfigurationPropertiesFormat: &network.FrontendIPConfigurationPropertiesFormat{ PublicIPAddress: &network.PublicIPAddress{ID: to.StringPtr("/subscriptions/subscription/" + "resourceGroups/rg/providers/Microsoft.Network/publicIPAddresses/pipName")}, }, }, } expectedLB7.Probes = &[]network.Probe{ { Name: to.StringPtr("atest1-" + string(service7.Spec.Ports[0].Protocol) + "-" + strconv.Itoa(int(service7.Spec.Ports[0].Port))), ProbePropertiesFormat: &network.ProbePropertiesFormat{ Port:              to.Int32Ptr(10081), RequestPath:       to.StringPtr("/healthz"), Protocol:          network.ProbeProtocolHTTP, IntervalInSeconds: to.Int32Ptr(5), NumberOfProbes:    to.Int32Ptr(2), }, }, } </ins> testCases := []struct { desc                string service             v1.Service

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Refer , health probes are not associated for UDP services on Azure, hence source IPs are not preserved for UDP services on Azure. What you expected to happen: When is set to for UDP services, source IPs should be preserved and the health probe should be associated with UDP load balancer rule. Per the , the health probe only supports TCP, HTTP and HTTPS. UDP is not supported as the probing protocol, hence we can’t enable the health probes for UDP services. However, when the externalTrafficPolicy is Local, Kubernetes would open an HTTP endpoint for UDP services, hence it’s actually supported on Azure. So, should associate the probes on externalTrafficPolicy=Local cases. How to reproduce it (as minimally and precisely as possible): Create a UDP service with : Then check the source IPs. Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /kind bug /sig cloud-provider /area provider/azure


Question: "k8s.io/apimachinery/pkg/fields" "k8s.io/client-go/tools/cache" <del> "encoding/json" </del> appsv1 "k8s.io/api/apps/v1" <ins> autoscalingv1 "k8s.io/api/autoscaling/v1" //Added new </ins> v1 "k8s.io/api/core/v1" apierrors "k8s.io/apimachinery/pkg/api/errors" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"

[x] two weeks soak end date : 26 Feb 2021 According to this APIsnoop query, there are still some remaining Deployment endpoints which are untested. with this query you can filter untested endpoints by their category and eligibility for conformance. e.g below shows a query to find all conformance eligible untested,stable,core endpoints Create a Deployment with a static label Read the Deployment Update the DeploymentScale Get the DeploymentScale to ensure it is updated Patch the DeploymentScale Read the DeploymentScale to ensure it is Patch Delete the Namespace and Deployment Due to the complexity of setting up the resources for APIService we have used the current e2e test as template. It has been extended in a for review. example endpoint | useragent ----------------------------------------|
patchAppsV1NamespacedDeploymentScale | (linux/amd64) kubernetes/$Format -- [sig-apps] Deployment Deployment should have a working scale subresource readAppsV1NamespacedDeploymentScale | (linux/amd64) kubernetes/$Format -- [sig-apps] Deployment Deployment should have a working scale subresource replaceAppsV1NamespacedDeploymentScale | (linux/amd64) kubernetes/$Format -- [sig-apps] Deployment Deployment should have a working scale subresource (3 rows) If a test with these calls gets merged, test coverage will go up by N points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted
Issue was presented in the Conformance meeting of 9 Feb 2021 and it was agreed that that method see to achieve the purpose and a PR can be created. A similar approach would be taken for the DepoymentScale endpoints, and this issue was created for that purpose.
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: errCh := make(chan error, concurrency) ready := newReady() ready.set(false) <ins> wg := sync.WaitGroup{} wg.Add(2 * concurrency) </ins> for i := 0; i < concurrency; i++ { go func() { errCh <- ready.wait(context.Background()) }() go func() { <ins> defer wg.Done() </ins> ready.set(false) }() go func() { <ins> defer wg.Done() </ins> ready.set(true) }() } <ins> // Last one has to be set to true. wg.Wait() </ins> ready.set(true) for i := 0; i < concurrency; i++ { if err := <-errCh; err != nil { t.Errorf("unexpected error %v on channel %d", err, i)

a test that is flaking at a ~3% failure rate with : Originally posted by in /sig api-machinery /assign /cc /kind failing-test
/assign

all the failures I've seen weren't failures, but rather timeouts. I wasn't able to reproduce it either by running the test itself, but I know what the potential failure is here: the and goroutines are somehow finishing the last one that is called is 1+ of goroutine is starting after that In this case the last will never finish. The fix is in
not sure if it's related, but TestGetCurrentResourceVersionFromStorage in the same package started flaking at the same time: looks like the fix in won't address that one
that's a different thing for sure. - can you please take a look at it - it's a newly test from yesterday
Spinned of for the other issue mentioned by Jordan.


Question: framework.DeletePodWithWait(f, c, nfsServerPod) }) <ins> Context("when kube-controller-manager restarts", func() { var ( diskName1, diskName2 string err                  error pvConfig1, pvConfig2 framework.PersistentVolumeConfig pv1, pv2             *v1.PersistentVolume pvSource1, pvSource2 *v1.PersistentVolumeSource pvc1, pvc2           *v1.PersistentVolumeClaim clientPod            *v1.Pod ) BeforeEach(func() { framework.SkipUnlessProviderIs("gce") framework.SkipUnlessSSHKeyPresent() By("Initializing first PD with PVPVC binding") pvSource1, diskName1 = framework.CreateGCEVolume() Expect(err).NotTo(HaveOccurred()) pvConfig1 = framework.PersistentVolumeConfig{ NamePrefix: "gce-", Labels:     volLabel, PVSource:   *pvSource1, Prebind:    nil, } pv1, pvc1, err = framework.CreatePVPVC(c, pvConfig1, pvcConfig, ns, false) Expect(err).NotTo(HaveOccurred()) framework.ExpectNoError(framework.WaitOnPVandPVC(c, ns, pv1, pvc1)) By("Initializing second PD with PVPVC binding") pvSource2, diskName2 = framework.CreateGCEVolume() Expect(err).NotTo(HaveOccurred()) pvConfig2 = framework.PersistentVolumeConfig{ NamePrefix: "gce-", Labels:     volLabel, PVSource:   *pvSource2, Prebind:    nil, } pv2, pvc2, err = framework.CreatePVPVC(c, pvConfig2, pvcConfig, ns, false) Expect(err).NotTo(HaveOccurred()) framework.ExpectNoError(framework.WaitOnPVandPVC(c, ns, pv2, pvc2)) By("Attaching both PVC's to a single pod") clientPod, err = framework.CreatePod(c, ns, []*v1.PersistentVolumeClaim{pvc1, pvc2}, true, "") Expect(err).NotTo(HaveOccurred()) }) AfterEach(func() { // Delete client/user pod first framework.ExpectNoError(framework.DeletePodWithWait(f, c, clientPod)) // Delete PV and PVCs if errs := framework.PVPVCCleanup(c, ns, pv1, pvc1); len(errs) > 0 { framework.Failf("AfterEach: Failed to delete PVC and/or PV. Errors: %v", utilerrors.NewAggregate(errs)) } pv1, pvc1 = nil, nil if errs := framework.PVPVCCleanup(c, ns, pv2, pvc2); len(errs) > 0 { framework.Failf("AfterEach: Failed to delete PVC and/or PV. Errors: %v", utilerrors.NewAggregate(errs)) } pv2, pvc2 = nil, nil // Delete the actual disks if diskName1 != "" { framework.ExpectNoError(framework.DeletePDWithRetry(diskName1)) } if diskName2 != "" { framework.ExpectNoError(framework.DeletePDWithRetry(diskName2)) } }) It("should delete a bound PVC from a clientPod, restart the kube-control-manager, and ensure the kube-controller-manager does not crash", func() { By("Deleting PVC for volume 2") err = framework.DeletePersistentVolumeClaim(c, pvc2.Name, ns) Expect(err).NotTo(HaveOccurred()) pvc2 = nil By("Restarting the kube-controller-manager") err = framework.RestartControllerManager() Expect(err).NotTo(HaveOccurred()) err = framework.WaitForControllerManagerUp() Expect(err).NotTo(HaveOccurred()) framework.Logf("kube-controller-manager restarted") By("Observing the kube-controller-manager healthy for at least 2 minutes") // Continue checking for 2 minutes to make sure kube-controller-manager is healthy err = framework.CheckForControllerManagerHealthy(2 * time.Minute) Expect(err).NotTo(HaveOccurred()) }) }) </ins> Context("when kubelet restarts", func() { var ( clientPod *v1.Pod

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --We should have an e2e to test issue . The test needs to delete a PVC while a pod is still using it, and then restart the controller manager. /kind bug /sig storage
CC
/assign unofficially so I can find this later.


Question: Buckets:   prometheus.ExponentialBuckets(1000, 2, 15), }, ) <ins> SchedulingAlgorithmPredicateEvaluationDuration = prometheus.NewHistogram( prometheus.HistogramOpts{ Subsystem: schedulerSubsystem, Name:      "scheduling_algorithm_predicate_evaluation", Help:      "Scheduling algorithm predicate evaluation duration", Buckets:   prometheus.ExponentialBuckets(1000, 2, 15), }, ) SchedulingAlgorithmPriorityEvaluationDuration = prometheus.NewHistogram( prometheus.HistogramOpts{ Subsystem: schedulerSubsystem, Name:      "scheduling_algorithm_priority_evaluation", Help:      "Scheduling algorithm priority evaluation duration", Buckets:   prometheus.ExponentialBuckets(1000, 2, 15), }, ) </ins> BindingLatency = prometheus.NewHistogram( prometheus.HistogramOpts{ Subsystem: schedulerSubsystem,

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a BUG REPORT or FEATURE REQUEST? (choose one): FEATURE REQUEST <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --As of now in , we have 3 metrics, for end2endLatency, algorithmLatency and BindingLatency. The problem with algorithmLatency is it is a combination of predicate and priority latencies. So breaking down this metric into predicate and priority latencies would be really helpful for performance analysis. For example, as part of the analysis after these metrics were , we were able to find latencies when we have taints & tolerations alone. In fact, ability to drill down on every predicate and priority would be ideal but want to know other's opinions on this.
/cc
+1 please send a PR.
We also have some demonstrations of how this helps with things like highlighting taint and toleration performance and so on. We should update readme info on how to use these along side the scheduler perf stuff as we've done internally .
You need to be careful about what latency means if we are computing multiple different predicates in parallel (if we decide to split it by predicate). Splitting predicate vs priority is definitely a good idea.
Please label this as part of our 1.7 scheduling forensics milestone .
Let's remove the idea of latency entirely, it is confusing - this is just an attempt at measuring predicate and priority timescales, and there is no response latency that we care about in those small (20ms) timescales , (with the exception of maybe the latency of an etcd hit that might happen in a client if the cache is missing information I guess)
Not really - there were cases of predicates which computation could take over a second.

Yes, and even longer on large clusters with things light affinity.... However here I'm just disputing use of the term latency since latency is usually used to refer to the time between the end of a request and the beggining of a response. What we endeavor to do here is really measure round trip execution time, for predicates, in isolation. If we want to measure some kind of latency then I believe we would need to define some notion of a request/response, which has diminished value outside of something like RPC. If we truly were measuring latency (and it might be interesting to do so with tims tracing work) then we might thus look at the time between writing a new unscheduled pod to the system, and the time it takes to start running the scheduler fit functions or something... Tl;dr there are a lot of varying usages and definitions for latency, but almost all of them, with the exception of measurements of virtual function calls, refer to some unavoidable minimum cost that occurs during distributed communication, and in this case, we are simply measuring "execution time of the predicate filter" which is called from inside the same process. (Alternatively I guess it's ok to call it "fitfunctionpredicate_latency", I guess, since then we are using the fit function as the context for a response measurement).
OK, IIUC you object usage of the word 'latency', right? We use it in this context for quite some time, and as I agree that it doesn't fit formal definition, there's no point in changing it now, as it would confuse people even more.
yeah, i really think they should all be refactored to not be named as such, that is my broader point . but if we want to keep those metrics id suggest we at least update the definition in the fields to clarify that latency is defined WRT rather then to a RPC or some other typical latency calc scenario. aplogies for the bike shed, ill move this to a PR :)
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale


Question: } } <ins> func expectTrackedResourceVersion(t *testing.T, tracker *endpointSliceTracker, slice *discovery.EndpointSlice, expectedRV string) { rrv := tracker.relatedResourceVersions(slice) rv, tracked := rrv[slice.Name] if !tracked { t.Fatalf("Expected EndpointSlice %s to be tracked", slice.Name) } if rv != expectedRV { t.Errorf("Expected ResourceVersion of %s to be %s, got %s", slice.Name, expectedRV, rv) } } </ins> func portsAndAddressTypeEqual(slice1, slice2 discovery.EndpointSlice) bool { return apiequality.Semantic.DeepEqual(slice1.Ports, slice2.Ports) && apiequality.Semantic.DeepEqual(slice1.AddressType, slice2.AddressType) }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: With EndpointSlice feature enabled, after EndpointSliceController creates or updates an EndpointSlice when reconciling a Service, the Service would always be processed one extra time, which is triggered by the EndpointSlice /updated event made by itself during the first processing. What you expected to happen: EndpointSliceController shouldn't reprocess a Service due to the EndpointSlice update made by itself. How to reproduce it (as minimally and precisely as possible): EndpointSlice feature, improve kube-controller-manager log to 5. a nginx deployment a Service backed by the above deployment. kube-controller-manager log: the service will be processed twice in succession. Anything else we need to know?: It's because EndpointSliceController tracks stale resource version after creating/updating an EndpointSlice. Environment: Kubernetes version (use ): v1.18.0-alpha.3 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/assign
/sig network
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/remove-triage unresolved