Question: test/e2e/common/init_container.go: "should invoke init containers on a RestartAlways pod" test/e2e/common/init_container.go: "should not start app containers if init containers fail on a RestartAlways pod" test/e2e/common/init_container.go: "should not start app containers and fail the pod if init containers fail on a RestartNever pod" <ins> test/e2e/common/kubelet.go: "should print the output to logs" test/e2e/common/kubelet.go: "should have an terminated reason" test/e2e/common/kubelet.go: "should be possible to delete" test/e2e/common/kubelet.go: "should write entries to /etc/hosts" test/e2e/common/kubelet.go: "should not write to root filesystem" </ins> test/e2e/common/kubelet_etc_hosts.go: "should test kubelet managed /etc/hosts file" test/e2e/common/lifecycle_hook.go: "should execute poststart exec hook properly" test/e2e/common/lifecycle_hook.go: "should execute prestop exec hook properly"

What happened: These tests are errantly listed as (ref: ) but aren't actually run as part of Conformance. They need to be moved to test/e2e/common to be included. To my eyes they may meet the criteria for conformance. There's definitely nothing about them that is node specific. They should be running as part of our regular CI, and they should probably be running as presubmits. Porting them to test/e2e/common would cause this to happen. /kind cleanup /sig node /area test /assign
GitHub didn't allow me to assign the following users: mkurylec. Note that only and repo collaborators can be assigned. For more information please see <detailsIn response to : What happened: These tests are errantly listed as (ref: ) but aren't actually run as part of Conformance. They need to be moved to test/e2e/common to be included. /sig node /area test /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: "context" "fmt" "io" <ins> "sort" </ins> "github.com/spf13/cobra" authenticationv1alpha1 "k8s.io/api/authentication/v1alpha1"

Unit tests failed with , see TestWhoAmIRun/successtest None No response No response_ /sig cli
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cli
I think it was caused by
When we loop the golang Map, it's unordered.
I can work on this one.
Thanks.
/priority critical-urgent this is impacting CI,


Question: // adding TerminationMessagePath on Windows is only allowed if ContainerD is used. Individual files cannot // be mounted as volumes using Docker for Windows. <del> supportsSingleFileMapping := kl.containerRuntime.SupportsSingleFileMapping() </del> if len(container.TerminationMessagePath) != 0 && supportsSingleFileMapping { p := kl.getPodContainerDir(pod.UID, container.Name) if err := os.MkdirAll(p, 0750); err != nil {

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug /sig windows What happened: When deploying Kubernetes pods, the Linux containers will contain the following entries in the file: # Kubernetes-managed hosts file. 127.0.0.1 localhost ::1 localhost ip6-localhost ip6-loopback fe00::0 ip6-localnet fe00::0 ip6-mcastprefix fe00::1 ip6-allnodes fe00::2 ip6-allrouters 10.240.0.67 dns-querier-1.dns-test-service.e2e-tests-dns- dns-querier-1 However, these are not set on Windows containers; the container's file does not contain any entry. This can be attributed to the fact that individual files cannot be mounted on Windows containers the same way they are on Linux containers. It can be fixed with a change to the kubelet once Windows Server 2019 has shipped, and once CRI-ContainerD is enabled on Windows. We cannot fix it for Windows Server 1803 or earlier with dockershim. What you expected to happen: The file should be managed and populated by Kubelet. How to reproduce it (as minimally and precisely as possible): Spawn a Windows pod, and check its container's file. kubectl create namespace e2e-tests-dns-x649p kubectl create -f # kubectl exec -n e2e-tests-dns-x649p dns-test--6d85-11e8-89cc- -c querier -- cmd /S /C type C:WindowsSystem32driversetchosts Compare the result with a Linux pod's container's file. kubectl create namespace e2e-tests-dns-x649p kubectl create -f # kubectl exec -n e2e-tests-dns-x649p dns-test--6d85-11e8-89cc- -c querier -- cat /etc/hosts Anything else we need to know?: Environment: Kubernetes version (use ): 1.12 Cloud provider or hardware configuration: N/A OS (e.g.

from /etc/os-release): Windows Server 1803 (10.0..48) Install tools: acs-engine v0.24.0 Others:
This requires single-file mapping, which won't work in Windows Server version 1803. It should be possible with Windows Server 2019 along with ContainerD
Will it work with Docker on Windows Server 2019?
Not in 18.09 or earlier. I'm not sure what release it will be in because it will require migrating to containerd.
should this issue be tracked for 1.13 in addition to If so please apply the appropriate priority and milestone labels please.
This one needs to be in v1.14. We need to get cri-containerd working first to fix this.
Thanks /milestone v1.14
By default, hosts file are not being used in Windows, it is all registry based. What features is this blocking? CC
The hosts file entries are still taken into account on Windows when it has to resolve a DNS name. This is blocking DNS conformance tests, like.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/lifecycle frozen
Hello! 14 code freeze is coming in about 3 days, is this intended to be implemented in the following week?
No. It is meant to be fixed in the next release, when the Containerd support will be introduced.
/milestone v1.15
Hello! We are starting the code freeze for 1.15 tomorrow EOD. Is this issue still planned for the 1.15 cycle?
This is dependent of the ContainerD implementation. If that hasn't been implemented yet, this can't either.
Is there any update is the containerd implementation finished? Are there any relevant issues or PRs that we can use for tracking?
sounds like a no-go for 1.15, let's just move it out as no one is working on it as well. /milestone v1.16
Hello!

I'm part of the bug triage team for the 1.16 release cycle and considering this issue is tagged for 1.16, but not updated for a long time, I'd like to check its status. The code freeze is starting on August 29th (about 1.5 weeks from now), which means that there should be a PR ready (and merged) until then. Do we still target this issue to be fixed for 1.16? If not please re-tag the issue to the planned milestone.
/milestone v1.17 We won't be able to do this until containerd is ready. It might work in 1.17
Bug triage for 1.17 here. This issue has been open for a significant amount of time and since it is tagged for milestone 1.17, we want to let you know that the 1.17 code freeze is coming in less than one month on Nov. 14th. Will this issue be resolved before then?
We have a PR for it, as you can see above: It;s a pretty small PR too.
The v1.17 code freeze is this Thursday, end of day Pacific time. Will this issue be closed by then?
/milestone v1.18 (code freeze v1.17)
This is going to need a Windows fix, not a ContainerD fix. We won't have that before code freeze on 5 March, so I'm pushing this out to 1.19. /milestone v1.19
This is being tracked with MS internal work item The Windows team thinks an OS fix for Windows Server 2019 will be released with the June patches. I will post updates here as I get them.
Hello from the bug triage team! This issue has not been updated for a long time, so I'd like to check what's the status. The code freeze is starting June 25th (about 4 weeks from now) and while there is still plenty of time, we want to ensure that each PR has a chance to be merged on time. As the issue is tagged for 1.19, is it still planned for this release?
can you comment on when we can expect a Windows fix for Windows Server 2019 that addresses the single file mount issues in containers when a target file already exists in the container?

we are waiting on a Windows OS fix from Microsoft to proceed here. If that fix comes in time for us to validate this functionality in 1.19 we would like it to be included in 1.19.
Hello! As this issue seems to depend on PR which has been pushed to v1.20, should this issue also be pushed to that milestone?
/milestone 1.20 The OS fix is now expect to be available as part of August Windows patches and then changes to the kubelet will need to follow. /cc
The provided milestone is not valid for this repository. Milestones in this repository: [, , , , , , , , , , , ] Use to clear the milestone. <detailsIn response to : /milestone 1.20 Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The provided milestone is not valid for this repository. Milestones in this repository: [, , , , , , , , , , , ] Use to clear the milestone. <detailsIn response to : /milestone 1.20 Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The provided milestone is not valid for this repository. Milestones in this repository: [, , , , , , , , , , , ] Use to clear the milestone. <detailsIn response to : /milestone 1.20 Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The provided milestone is not valid for this repository. Milestones in this repository: [, , , , , , , , , , , ] Use to clear the milestone. <detailsIn response to : /milestone 1.20 Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The provided milestone is not valid for this repository. Milestones in this repository: [, , , , , , , , , , , ] Use to clear the milestone.

<detailsIn response to : /milestone 1.20 Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/milestone v1.20
yes it is expected be part of August Windows release
Hey Bug Triage here. Wanted to follow up on the status of this issue as we're approaching code freeze on 12.11.2020. This issue is tagged for 1.20, is it still planned for this release?
- Yes, we'd like to make it into 1.20. We were waiting on some bugfixes in the Windows OS and those bugfixes are now available (as of Oct 20 2020).
/remove-lifecycle frozen
/priority important-soon
I believe the work involved here is to update to enable this fucntionality for WIndows if container-runtime == remote (still disable this for DockerShim) Update/Add test coverage Update documentation - possible this section ?
/milestone v1.21 Moving to 1.21 since the required OS fixes were not available in time to implement this functionality in v1.20
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
Hello! I am from the Bug Triage team! This issue has not been updated for a long time, so I'd like to check on the status. The code freeze is starting March 9th, 2021 (about 1 week from now). We want to ensure that each PR has a chance to be merged before code freeze. As the Issue is tagged for 1.21, is it still planned for this release?
Can this be reopened? From a related problem I have experienced using HostAlias for Windows pod deployments, this issue does not seem to be resolved. Instead, it looks as if it was closed by accident because the conversation went cold.
Hello. What Kubernetes version are you running? Also, on the Windows nodes, do you use docker, or containerd?

This is only supported on Windows nodes with containerd, since it supports single file mapping, which is required in order to mount single files into Windows containers.
We have experimented with both 1.22.11 and a 1.23 release and yea, the container runtime is containerd. Any other ideas to check up on?


Question: // on its parent directory. func (prober *flexVolumeProber) addWatchRecursive(filename string) error { addWatch := func(path string, info os.FileInfo, err error) error { <del> if info.IsDir() { </del> <ins> if err == nil && info.IsDir() { </ins> if err := prober.watcher.AddWatch(path); err != nil { glog.Errorf("Error recursively adding watch: %v", err) }


Question: batchv1 "k8s.io/api/batch/v1" "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" <ins> "k8s.io/apimachinery/pkg/util/rand" </ins> "k8s.io/kubernetes/test/e2e/framework" )

Which jobs are flaking: pull-kubernetes-e2e-kind Which test(s) are flaking: Testgrid link: Reason for failure: Anything else we need to know: /sig apps
Failure at
Hi I looked into the failures and think it was because the test required the job succeeds 3 times with pseudorandom 50% failure chance within 15 minutes, while there is an exponential back-off delay (10s, 20s, 40s …) capped at 6 minutes before recreating failed pods. As 7 consecutive failures (1/128 chance) could exceed the timeout, the test failed intermittently because of "timed out waiting for the condition". For example, in the failure of , the job pods succeeded and failed in the following orders: After the last failure at , the delay has increased to 640s so no pod was created any more before the test failed at . I think it can be fixed by using a deterministic approach to make the job pods succeed and fail alternately, and its executing time can be reduced to tens of seconds stably. Assigning myself. /assign


Question: <del> /* Copyright 2019 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package service import ( "context" "reflect" "testing" v1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/client-go/kubernetes/fake" ) func addAnnotations(svc *v1.Service) { svc.Annotations["foo"] = "bar" } func TestPatch(t *testing.T) { svcOrigin := &v1.Service{ ObjectMeta: metav1.ObjectMeta{ Name:        "test-patch", Annotations: map[string]string{}, }, Spec: v1.ServiceSpec{ ClusterIP: "10.0.0.1", }, } fakeCs := fake.NewSimpleClientset(svcOrigin) // Issue a separate update and verify patch doesn't fail after this. svcToUpdate := svcOrigin.DeepCopy() addAnnotations(svcToUpdate) if _, err := fakeCs.CoreV1().Services(svcOrigin.Namespace).Update(context.TODO(), svcToUpdate, metav1.UpdateOptions{}); err != nil { t.Fatalf("Failed to update service: %v", err) } // Attempt to patch based the original service. svcToPatch := svcOrigin.DeepCopy() svcToPatch.Finalizers = []string{"foo"} svcToPatch.Spec.ClusterIP = "10.0.0.2" svcToPatch.Status = v1.ServiceStatus{ LoadBalancer: v1.LoadBalancerStatus{ Ingress: []v1.LoadBalancerIngress{ {IP: "8.8.8.8"}, }, }, } svcPatched, err := patch(fakeCs.CoreV1(), svcOrigin, svcToPatch) if err != nil { t.Fatalf("Failed to patch service: %v", err) } // Service returned by patch will contain latest content (e.g from // the separate update). addAnnotations(svcToPatch) if !reflect.DeepEqual(svcPatched, svcToPatch) { t.Errorf("PatchStatus() = %+v, want %+v", svcPatched, svcToPatch) } // Explicitly validate if spec is unchanged from origin. if !reflect.DeepEqual(svcPatched.Spec, svcOrigin.Spec) { t.Errorf("Got spec = %+v, want %+v", svcPatched.Spec, svcOrigin.Spec) } } func TestGetPatchBytes(t *testing.T) { origin := &v1.Service{ ObjectMeta: metav1.ObjectMeta{ Name:       "test-patch-bytes", Finalizers: []string{"foo"}, }, } updated := &v1.Service{ ObjectMeta: metav1.ObjectMeta{ Name:       "test-patch-bytes", Finalizers: []string{"foo", "bar"}, }, } b, err := getPatchBytes(origin, updated) if err != nil { t.Fatal(err) } expected := `{"metadata":{"$setElementOrder/finalizers":["foo","bar"],"finalizers":["bar"]}}` if string(b) != expected { t.Errorf("getPatchBytes(%+v, %+v) = %s ; want %s", origin, updated, string(b), expected) } } </del>

PR service patch methods to Once there is a release with that PR included, the code in service-controller and legacy-cloud-providers/gce can be updated to use those. /area platform/gce /assign
/sig network
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/remove-triage unresolved
Those labels are not set on the issue: <detailsIn response to : /remove-triage unresolved Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Created a PR to use these new functions in Service-controllers.


Question: func (cpo createPodsOp) patchParams(w *workload) (realOp, error) { if cpo.CountParam != "" { <del> var ok bool if cpo.Count, ok = w.Params[cpo.CountParam[1:]]; !ok { return nil, fmt.Errorf("parameter %s is undefined", cpo.CountParam) </del> <ins> var err error cpo.Count, err = w.Params.get(cpo.CountParam[1:]) if err != nil { return nil, err </ins> } } return &cpo, (&cpo).isValid(false)

Note: this issue should be viewed in the context of . If this happens, this is very likely a typo where someone used a parameter in a way they didn't mean to (for example, ). /sig scheduling /assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale
/good-first-issue
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hi this is my first time contributing to kubernetes. can you provide me some guidance on how to to get started?
Hi, is this issue still valid to work on?
yes, this is still valid. I recommend starting with the and reading from there.
are you still working on this issue or should I work on it.
I would like to work on this issue if you haven't started already.
would it be ok if I took a look at this?
/assign
Hi Can you help me in why its not assigning the values to createNodes and createPods directly and using the opcode and countParam in performance-
We use a field to differentiate between different operations. The value of the field (e.g. "createNodes") determines the operation. The field defines the number of pods/nodes to create. What do you mean by "assigning the values to createNodes and createPods directly"?
Thanks for the information.

By assigning, I just meant that whether we can assign createNodes directly like ( createNodes: $initNodes ).
I don't think that's possible since there can potentially be other parameters besides the countParam. For example: Different operations could potentially have different parameters.
createNodes and createPods aren't the only operations. Other operations (e.g. churn and barrier), including future ones, may not fit that format. It's also easier to parse and more explicit this way.
Thanks , Having a doubt more, in performance- under the name:SchedulingPodAntiAffinity, ! why are we giving the same values in 5000Nodes as given in 500Nodes, or this is an error.
that sounds like a typo indeed
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
Hello Are you still going to work on this issue?
Hi I have worked on the issue but not working for a while as was unsure about the exact problem.
, no problem if you are willing to work on this issue, shall unassign myself
/unassign
Thanks :)
/assign
/remove-lifecycle stale
hey! I am beginner and I interested in scheduling and I know basics of scheduling , so how can I start to contribute on this issue.
Sorry, PR for this issue has already created.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:


Question: framework.Failf("Expected `not found` error, got: %v", err) } }) <ins> ginkgo.It("should manage the lifecycle of a ResourceQuota", func() { client := f.ClientSet ns := f.Namespace.Name rqName := "e2e-quota-" + utilrand.String(5) label := map[string]string{"e2e-rq-label": rqName} labelSelector := labels.SelectorFromSet(label).String() ginkgo.By("Creating a ResourceQuota") resourceQuota := &v1.ResourceQuota{ ObjectMeta: metav1.ObjectMeta{ Name:   rqName, Labels: label, }, Spec: v1.ResourceQuotaSpec{ Hard: v1.ResourceList{}, }, } resourceQuota.Spec.Hard[v1.ResourceCPU] = resource.MustParse("1") resourceQuota.Spec.Hard[v1.ResourceMemory] = resource.MustParse("500Mi") _, err := createResourceQuota(client, ns, resourceQuota) framework.ExpectNoError(err) ginkgo.By("Getting a ResourceQuota") resourceQuotaResult, err := client.CoreV1().ResourceQuotas(ns).Get(context.TODO(), rqName, metav1.GetOptions{}) framework.ExpectNoError(err) framework.ExpectEqual(resourceQuotaResult.Spec.Hard[v1.ResourceCPU], resource.MustParse("1")) framework.ExpectEqual(resourceQuotaResult.Spec.Hard[v1.ResourceMemory], resource.MustParse("500Mi")) ginkgo.By("Listing all ResourceQuotas with LabelSelector") rq, err := client.CoreV1().ResourceQuotas("").List(context.TODO(), metav1.ListOptions{LabelSelector: labelSelector}) framework.ExpectNoError(err, "Failed to list job. %v", err) framework.ExpectEqual(len(rq.Items), 1, "Failed to find ResourceQuotes %v", rqName) ginkgo.By("Patching the ResourceQuota") payload := "{"metadata":{"labels":{"" + rqName + "":"patched"}},"spec":{"hard":{ "memory":"750Mi"}}}" patchedResourceQuota, err := client.CoreV1().ResourceQuotas(ns).Patch(context.TODO(), rqName, types.StrategicMergePatchType, []byte(payload), metav1.PatchOptions{}) framework.ExpectNoError(err, "failed to patch ResourceQuota %s in namespace %s", rqName, ns) framework.ExpectEqual(patchedResourceQuota.Labels[rqName], "patched", "Did not find the label for this ResourceQuota. Current labels: %v", patchedResourceQuota.Labels) framework.ExpectEqual(*patchedResourceQuota.Spec.Hard.Memory(), resource.MustParse("750Mi"), "Hard memory value for ResourceQuota %q is %s not 750Mi.", patchedResourceQuota.ObjectMeta.Name, patchedResourceQuota.Spec.Hard.Memory().String()) ginkgo.By("Deleting a Collection of ResourceQuotas") err = client.CoreV1().ResourceQuotas(ns).DeleteCollection(context.TODO(), metav1.DeleteOptions{}, metav1.ListOptions{LabelSelector: labelSelector}) framework.ExpectNoError(err) ginkgo.By("Verifying the deleted ResourceQuota") _, err = client.CoreV1().ResourceQuotas(ns).Get(context.TODO(), rqName, metav1.GetOptions{}) if !apierrors.IsNotFound(err) { framework.Failf("Expected `not found` error, got: %v", err) } }) </ins> }) var _ = SIGDescribe("ResourceQuota [Feature:ScopeSelectors]", func() {

[x] test approval issue : [x] test pr : 7 July 2022 [X] two weeks soak end date : 21 July 2022 [X] test promotion pr : According to this APIsnoop query, there is still some remaining ResourceQuota endpoints which are untested. The Status endpoints will be addressed in another test. listCoreV1ResourceQuotaForAllNamespacespatchCoreV1NamespacedResourceQuotadeleteCoreV1CollectionNamespacedResourceQuota Using a number of existing e2e test practices a new has been created for three ResourceQuota endpoints. The e2e logs for this test are listed below. This query shows the endpoints hit within a short period of running the e2e test If a test with these calls gets merged, test coverage will go up by 3 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: } func TestProxyUpdateDelete(t *testing.T) { <del> port, err := echoServer(t, "127.0.0.1:0") if err != nil { t.Fatal(err) } </del> lb := NewLoadBalancerRR() lb.OnUpdate([]api.Endpoints{{JSONBase: api.JSONBase{ID: "echo"}, Endpoints: []string{net.JoinHostPort("127.0.0.1", port)}}})

https://travis-
Looking.
Another instance, or a different thing? https://travis-


Question: ts := httptest.NewServer(WithPanicRecovery(handler, resolver)) defer ts.Close() <del> _, err = http.Get(ts.URL) </del> <ins> _, err := http.Get(ts.URL) </ins> if err == nil { t.Fatal("expected to receive an error") } <del> capturedOutput := readStdErr() </del> <ins> klog.Flush() klog.SetOutput(&bytes.Buffer{}) // prevent further writes into buf capturedOutput := buf.String() </ins> // We don't expect stack trace from the panic to be included in the log. if isStackTraceLoggedByRuntime(capturedOutput) {

/sig api-machinery /kind bug /priority important-soon
It seems something is racing writhing the logs ? cc:
/cc /triage accepted
Failed with this: pull-kubernetes-unit
this looks like a test-only race where the test overwriting the global os.Stderr in order to capture error output races with an async attempt to log to it from another test in the same package
which still needs to be fixed
It seems to me that this bug can not be fixed with as it is, namely used without a logger object or even a . The smallest tweak that occurs to me would be to add an option to to add an output destination, and later remove that addition. Thus, a test could add logging to its own buffer for the duration of running the code under test.
This may be similar to . How can I reproduce this issue?


Question: ) func TestGetLoadBalancer(t *testing.T) { <del> t.Parallel() </del> vals := DefaultTestClusterValues() gce, err := fakeGCECloud(vals) require.NoError(t, err)

Since merged on Friday night, my PR () has failed several retests of pull-kubernetes-bazel-test. My PR is unrelated to the load balancer so I suspect that introduced a flake. I see several runs of that test against master that aren't failing, so clearly the test works sometimes (and the original PR passed it several times to before it merged). Perhaps the test harness sometimes runs problem tests in parallel and other times it doesn't. Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: My PR had one good run of pull-kubernetes-bazel-test before merged, but has failed several times since with no code changes to my PR. What you expected to happen: The tests should consistently pass unless related problem code is introduced.
/cc /cc /sig gcp /kind flake
My PR started failing recently too. See https://k8s-
Hey, thanks for identifying the issue. contains tests for - but the race condition was caused when tests with modified a singleton variable in a mock. The short-term fix is removing the calls, but the long-term fix is to not use a singleton. Fix is .


Question: } // Complete completes all the required options. <del> func (o *Options) Complete() error { </del> <ins> func (o *Options) Complete(fs *pflag.FlagSet) error { </ins> if len(o.ConfigFile) == 0 && len(o.WriteConfigTo) == 0 { o.config.HealthzBindAddress = addressFromDeprecatedFlags(o.config.HealthzBindAddress, o.healthzPort) o.config.MetricsBindAddress = addressFromDeprecatedFlags(o.config.MetricsBindAddress, o.metricsPort)

introduced configuration of additional logging aspects (e.g. format) through command line flags and config file. This was merged for Kubernetes 1.28. Now the following invocation doesn't work as expected when the config file doesn't configure logging: It should run with verbosity 4. Instead it runs with verbosity 0, from the config. Run with this config file: No response 28 rc n/a No response No response No response No response
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The problem is caused by this line: At that point, is 4 (from the command line flag), and then it gets overwritten with == 0. The normal approach would be: set up config with defaults modify config through flags modify config further by parsing a config file into it. Can kube-proxy do this? It has the downside that a config file can only overwrite fields if they are in the file. A config field doesn't get reset to defaults just by parsing the file. /cc
it looks like kube-proxy special-cases the to let a flag override config... can/should it do the same thing for verbosity? are there any other flags like this?
I don't think that's the normal approach... it's more normal to make flags and config file mutually exclusive. That's difficult for things which started as flags and moved into config
/sig network /sig instrumentation /kind bug /kind regression /milestone v1.28
let's go with and warn it and comment as much as possible, I was going with something like but Patrick is working on it so I assign to him the issue
/assign
Hey folks, v1.28 Bug Triage Lead here Just wanted to know if this is already tracked by SIG leads and someone already working actively fixing it and set a priority?
Perhaps I should have said "this is how component-base/logs was meant to be used". Somehow it worked for kubelet.

Here's my preliminary solution: It needs more testing, but I wanted to share it early because I am running out of time today.
it is , I can take over Patrick's work in case is needed
It would be nice if the logs API could distinguish between and , how about make to a pointer? But this change is supposed to be a breaking change, I'm not sure of the scope of the impact. For side, if set and at the same time, should the command flag override the config file or warn it and be discarded? I think it should be overridden.
xref:


Question: func (dswp *desiredStateOfWorldPopulator) ReprocessPod( podName volumetypes.UniquePodName) { <del> dswp.deleteProcessedPod(podName) </del> <ins> dswp.markPodProcessingFailed(podName) </ins> } func (dswp *desiredStateOfWorldPopulator) HasAddedPods() bool {

It happens during the "delete" phase in the load test. ! In perf-dash we can see that in total there are ~20M requests like that during the load test. ! It started in Sep 2019 (sic!), so probably when we enabled PVs in our load tests. We haven't detected it so far as, strangely, it doesn't seem to be impacting our scalability SLIs. We should investigate this. /sig scalability
I checked and we create 35 PersistentVolumeClaims per 100 nodes. So in case of 5K node test there will be 1750 PVCs in the cluster. This is close to the 2K of GET PVC requests we see.
Let's analyze the last run of 5K node test in more depths - Grafana: ! As I wrote already somewhere here, it happens during the delete phase. We reach the peak of ~2K QPS around 2:30 AM UTC in that run. Let's check the logs (api-server, kubelets) around that time.
Looking at the api-server logs one can easily tell that something is wrong. It's swamped by GET persistentvolumeclaim requests:
Restricting the logs to a single stateful set on a single node (single client ip) - it also doesn't look right. Total such requests from this IP address: So it looks like a single kubelet issued almost 20K GET requests for this PVC within ~75min, which gives us ~4 QPS from a single kubelet...
From the scheduler logs it looks like the statefulset pod for this PVC was bound on the node Let's check the kubelet logs there
Kubelet logs: Bingo, around the time when the issue happens, it's full of the following logs:
Looks like the errors disappeared exactly when the pod was deleted So, what happens here is partially caused by our test as it may happen that we'll first delete PVC and won't delete the corresponding pod for the next O(1h). Nevertheless, the code in the should be fixed so it doesn't cause kubelet to send ~4QPS of GET PVC requests in such case.
/sig node /sig storage
There is a pending discussion about ways to fix it in
Thanks for fixing that! I know how much work was it and I really appreciate!


Question: } } <ins> // checkLeftoverEndpoints lists all currently existing endpoints and adds their // service to the queue. This will detect endpoints that exist with no // corresponding service; these endpoints need to be deleted. We only need to // do this once on startup, because in steady-state these are detected (but // some stragglers could have been left behind if the endpoint controller // reboots). func (e *EndpointController) checkLeftoverEndpoints() { list, err := e.client.Endpoints(api.NamespaceAll).List(labels.Everything()) if err != nil { glog.Errorf("Unable to list endpoints (%v); orphaned endpoints will not be cleaned up. (They're pretty harmless, but you can restart this component if you want another attempt made.)", err) return } for i := range list.Items { ep := &list.Items[i] key, err := keyFunc(ep) if err != nil { glog.Errorf("Unable to get key for endpoint %#v", ep) continue } e.queue.Add(key) } } </ins> func findDefaultPort(pod *api.Pod, servicePort int, proto api.Protocol) int { for _, container := range pod.Spec.Containers { for _, port := range container.Ports {

So we'll leak them and/or reuse them perhaps inappropriately. I imagine this could be pretty hard on kube-proxy after a while. Discovered while working on . was suggesting implementing a generic delete-when-parent-is-deleted mechanism, so I may not try to fix this 100% with my current work on .
Generic mechanism proposal was here:
I think it should be enough to check if endpoints match services once just after we initialize, right?
Yeah that should be enough to close this. Although the general mechanism would be nice, too.


Question: fs.Int32Var(&s.KubeAPIBurst, "kube-api-burst", s.KubeAPIBurst, "Burst to use while talking with kubernetes apiserver") fs.BoolVar(&s.SerializeImagePulls, "serialize-image-pulls", s.SerializeImagePulls, "Pull images one at a time. We recommend *not* changing the default value on nodes that run docker daemon with version < 1.9 or an Aufs storage backend. Issue #10959 has more details. [default=true]") fs.BoolVar(&s.ExperimentalFlannelOverlay, "experimental-flannel-overlay", s.ExperimentalFlannelOverlay, "Experimental support for starting the kubelet with the default overlay network (flannel). Assumes flanneld is already running in client mode. [default=false]") <ins> fs.MarkDeprecated("experimental-flannel-overlay", "Will be removed in a future version.") </ins> fs.DurationVar(&s.OutOfDiskTransitionFrequency.Duration, "outofdisk-transition-frequency", s.OutOfDiskTransitionFrequency.Duration, "Duration for which the kubelet has to wait before transitioning out of out-of-disk node condition status. Default: 5m0s") fs.StringVar(&s.NodeIP, "node-ip", s.NodeIP, "IP address of the node. If set, kubelet will use this IP address for the node") fs.BoolVar(&s.EnableCustomMetrics, "enable-custom-metrics", s.EnableCustomMetrics, "Support for gathering custom metrics.")

From another issue, said: DockerManager.SyncPod() calls the network Plugin and then sets hairpin mode unilaterally. This is wrong. We should either pass the bool down to here or else we should get rid of the default no-op plugin and use the non-existence of a plugin as a flag to control this. This should be a relatively easy one for people who want to contribute but don't know where to start.
I volunteer myself to do this. Actually I have done something about kubernetes network plugin so guess I can do this. I will start to do this when I am on the office tomorrow.
Awesome. I'd advise starting with a sketch so we can all make sure it is a good approach, before jumping in with both feet. , TonyAdo wrote:
ok, I will start with a sketch as soon as possible.
I thought about this for a while: if we pass the bool down to here, of course we can check this flag, if it is set, we should set veth hairpin mode, but problem is that if user set this flag, but still use a custom network plugin just like you mentioned the ovs plugin, I am afraid we will still enable hairpin mode in all interfaces on this host, since we cannot find a peer interface. Is that correct?
Yeah, I think the better fix is to use the existence of a network plugin as an indicator. It sort of makes --configure-cbr0 redundant , TonyAdo wrote:
ok, I will try to get rid of the default no-op plugin and use the non-existence of a plugin as a flag to control this.
I decided to use the existence of a network plugin as an indicator, also I will get ride of the no-op network plugin, I am not going to add new flag to kubelet structure, if the user don't specify network plugin name, the networkPlugin within kubelet will be which just means non-existence of a plugin, and we have to check the existence in the docker manager to avoid nil pointer exception also for setting, I think if a custom network plugin exists, kubelet should not do this. How about this? Should I open a PR?
fyi, saw a hairpin pr today that might be related. ymmv.
what do you think about this?
What you described sounds right to me.

I think the only way to see if there is some detail I am missing is to try it and see if anything pops up that would make this not work. , TonyAdo wrote:
I need to check which part is responsible of what, but what if the hairpin package was just a utility package that should be called by the plugin if it needs to set this up? As long as it is single and simple function call, doing so would avoid adding extra logic here and fix the issue. This is valid only if the aim is to have a "linux bridge plugin", an "OVS plugin", etc.
actually kubelet will call after get called, I think to make the network plugin can call hairpin package is not that easy, of course, I agree that network plugin should setup hairpin mode if it really needs, kubelet should not set hairpin mode unilaterally. Besides, kubelet is not going to provide an "linux bridge plugin" or "OVS plugin".
then the validity conditions of my comment are not met, so I see any simpler way. Thanks for checking! :)
still what do you think about disable hairpin completely and leave this to network plugin?
or if there is a custom network plugin, we don't setup hairpin mode, only do this in default.
Completely? It will break my use case (and anyone using Apache Kafka).
Only by default is what I understood from the previous comments, and LGTM.
Really sorry about that, but I really want to know are you using a kubelet network plugin in your use case? If no, I think we will still setup hairpin, as in default we will do this.
I don't use any plugin, so it is ok if hairpin is set up by default.
ok, I think this will not break your use case.
It's not really the network plugin that needs hairpinmode though, it's the pure-iptables-based kube-proxy code. It's possible the proxying code could change again in the future in such a way that hairpinmode wouldn't be needed any more. So it seems wrong to put that code into the plugin. makes it so that we set hairpinmode on the veth if it's attached to a bridge, but just silently do nothing if it's not.

(Yes, we currently set hairpinmode even if you're using the userspace proxier, but AFAIK it's not needed in that case.)
IMHO... The proxy has the responsibility of service objects, while the kubelet is responsible of pod objects, including their networking. Also, the responsibility of hairpin must be at the same place as the code that creates the network interface, otherwise you have code managing a single object (the pod interface) spread over multiple programs, which will eventually lead to complex problems and a system harder to understand even for the end-user. Thus, it doesn't seems wrong to me that the kubelet does whatever is required to make the pod work in the environment of the cluster. While its true that the hairpin requirement is only for the proxy in iptables mode, a switch to this mode can happen at any time in the life of the cluster. And when it happens, its probably because the userland proxy becomes unresponsive, which means in a situation where the administrator is in a hurry to find a solution. Currently, you just change the proxy mode and restart the proxies. There's little chance the administrator will think about hairpin or restarting the kubelet in this case, so the kubelet should be reacting to this; which means the proxy and the kubelet of a given host should be able to communicate (AFAIK, this doesn't exist now). And all this is currently only to avoid setting up hairpin on the pod's port of standard Linux bridge.
Feel sure I'm missing something but... The CNI bridge plugin now has a flag, so kubenet can use that. Having done that, we can set false when the plugin name is non-blank.
Is this fixed now?
I apologise for leaving a somewhat vague answer, but I have been in that code recently. I don't think it's fixed.
, Bryan Boreham wrote:
Thanks for the PR. CNI Bridge plugin is expecting . Not . Hence the bridge plugin did not recognize it.
EDIT 2016-12-09: The chain below was broken by , so the issue can be closed again (remain closed) I suggest this should be re-opened. The chain of calls is: sets to , e.g. when using CNI calls something goes wrong and it sets the mode on everything. An example of "something goes wrong":


Question: return list } <del> // Contains returns a util.StringSet containing all IDs of stored the items. </del> <ins> // ContainedIDs returns a util.StringSet containing all IDs of stored the items. </ins> // This is a snapshot of a moment in time, and one should keep in mind that // other go routines can add or remove items after you call this. <del> func (c *FIFO) Contains() util.StringSet { </del> <ins> func (c *FIFO) ContainedIDs() util.StringSet { </ins> c.lock.RLock() defer c.lock.RUnlock() set := util.StringSet{}

The client/cache interface has a method which is supposed to return a of the IDs in a store. This is a bit misleading and should probably be renamed to 'Contents' or 'ContainedIDs'.
Thoughts
I tend to agree. contains to me should always return a bool and take an input key. Sent from my iPhone
Yeah, I agree, both of the suggested names would be an improvement.


Question: t.Fatalf("Failed to create Pod %q: %v", pod.Name, err) } <del> // first binding try should fail. err := wait.PollUntilContextTimeout(ctx, 200*time.Millisecond, wait.ForeverTestTimeout, false, testutils.PodSchedulingError(cs, ns, pod.Name)) </del> <ins> // 1. first binding try should fail. // 2. The pod should be enqueued to activeQ/backoffQ without any event. // 3. The pod should be scheduled in the second binding try. // Here, waiting until (3). err := wait.PollUntilContextTimeout(ctx, 200*time.Millisecond, wait.ForeverTestTimeout, false, testutils.PodScheduled(cs, ns, pod.Name)) </ins> if err != nil { <del> t.Fatalf("Expect pod-1 to be rejected by the bind plugin: %v", err) </del> <ins> t.Fatalf("Expect pod-1 to be scheduled by the bind plugin: %v", err) </ins> } <del> // The pod should be enqueued to activeQ/backoffQ without any event. // The pod should be scheduled in the second binding try. err = wait.PollUntilContextTimeout(ctx, 200*time.Millisecond, wait.ForeverTestTimeout, false, testutils.PodScheduled(cs, ns, pod.Name)) if err != nil { </del> <ins> // Make sure the first binding trial was failed, and this pod is scheduled at the second trial. if fakeBind.counter != 1 { </ins> t.Fatalf("Expect pod-1 to be scheduled by the bind plugin in the second binding try: %v", err) } }

pull-kubernetes-integration: scheduler TestRequeueByBindFailure Running with stress reproduces the flake Not sure, seeing a lot of retests on pull-kubernetes-integration Bind logic in scheduler integration tests just changed 3 days ago in , not sure if that's related No response No response /sig scheduling /priority important-soon cc
I doesn't look related to the latest change you were also looking at Bind. Any ideas?
It's a little strange because the error reports but since the bind-plugin will fail for the first calling time and there's no kubelet, that means the pod status will be kept since. I created another PR to help output the error message, which maybe helpful for analysis.
This test is very straightforward: the first try to bind "pod-1", the Pod is rejected by the binding plugin. () scheduling queue puts back this "pod-1" to activeQ/backoffQ directly, and "pod-1" will soon come back to the scheduling cycle. the second try to bind "pod-1", it's successful. And, means that (1) doesn't reject "pod-1" correctly. And I don't have any idea why it can happen. The only scenario I can think of is that there is somehow another Pod created before pod-1. When is executed , is it possible that any other Pods (like system ones) are created automatically?
/triage accepted current top flake of integration
/assign I've got one possible scenario, will have a quick fix.


Question: replicas := 2 // Create a replication controller for a service <del> // that serves its hostname on port 8080. </del> <ins> // that serves its hostname. </ins> // The source for the Docker containter kubernetes/serve_hostname is // in contrib/for-demos/serve_hostname By(fmt.Sprintf("Creating replication controller %s", name))

Some of the e2e tests specify HostPort in their pod configurations. This can cause issues running those tests if the port is already in use.
We've been removing these over the last week and a half and I think that may be the only one left remaining.
We need to decide if HostPort is a supported use case for Kubernetes -- and if it's supported, then it needs e2e test coverage to ensure it remains functional. Note that supported and not-preferred are orthogonal. My understanding is that it's supported-but-not-preferred and that there are reasonable use cases for using it.


Question: // Set IP and hostname addresses for the node. func (kl *Kubelet) setNodeAddress(node *v1.Node) error { if kl.nodeIP != nil { <del> if err := validateNodeIP(kl.nodeIP); err != nil { </del> <ins> if err := kl.nodeIPValidator(kl.nodeIP); err != nil { </ins> return fmt.Errorf("failed to validate nodeIP: %v", err) } glog.V(2).Infof("Using node IP: %q", kl.nodeIP.String())

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: When using the flag on the kubelet, the node's list no longer reports , , addresses. What you expected to happen: That I can explicitly set to an IP, but the other addresses will still show up. How to reproduce it (as minimally and precisely as possible): Create a new kops cluster, modify the kubelet to use the flag with the eth0 on the node. Run Anything else we need to know?: This change will probably take place in  This came up as a result of Environment: Kubernetes version (use ): 1.9.x Cloud provider or hardware configuration: AWS OS (e.g. from /etc/os-release): any Kernel (e.g. ): any Install tools: kops /sig network /sig node I'll make a PR for this change shortly


Question: t.Errorf("Got a nil Unmounter") } <del> // Tear down the volume if err := unmounter.TearDown(); err != nil { t.Errorf("Expected success, got: %v", err) </del> <ins> if !config.readyDirExists { if err := os.RemoveAll(metadataDir); err != nil && !os.IsNotExist(err) { t.Errorf("failed to remove ready dir [%s]: %v", metadataDir, err) } </ins> } <del> if _, err := os.Stat(volPath); err == nil { t.Errorf("TearDown() failed, volume path still exists: %s", volPath) } else if !os.IsNotExist(err) { t.Errorf("TearDown() failed: %v", err) </del> <ins> if !config.volumeDirExists { if err := os.RemoveAll(volPath); err != nil && !os.IsNotExist(err) { t.Errorf("failed to remove ready dir [%s]: %v", metadataDir, err) } } // Tear down the volume if err := testTearDown(unmounter, metadataDir, volPath); err != nil { t.Errorf("Test failed with error %v", err) </ins> } log = physicalMounter.GetLog()

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: There is a node in my cluster whose disk is full on root partition, I cleared the disk and then rebooted the node to recover it. After the reboot, the kube-proxy on that node got stuck in "ContainerCreating" status and never changes, and all other pods can't run properly since the kubeproxy is not runing. What you expected to happen: Kubeproxy on that node can run normally after the disk clean and reboot How to reproduce it (as minimally and precisely as possible): the disk full on root partition in my env, leave roughly 100k space on root partition would make this issue more reproduciblethis issue is kind of tricky, to reproduce it, one need to populate the disk repeatedly since some disk space would be cleared because of pod evictionthe kube-proxy got deleted and created repeatedly because of pod eviction and daemon-set mechanism, it would stuck in ContainerCreating status if the disk space is not enough for creating volume contents of kube-proxy configMap volume, the first pod error event is like: MountVolume.Setup failed for volume "kube-proxy" space left on device After it got stuck, the following pod error event woud repeat over and over, kube-proxy is trying to mount configMap volume but always fail MountVolume.Setup failed for volume "kube-proxy": stat /var/lib/kubelet/pods/{podid} no such file or directory the disk and reboot the reboot is neccesary to reproduce the issue, without the reboot the pod eviction would kick in to delete kube-proxy pod the kube-proxy woud stuck in ContainerCreating Status forever, restart kubelet has no effects on that, kube-proxy pod need to be mannully deleted to get recover kube-proxy pod can't mount configMap volume, the following error keeps repeating MountVolume.Setup failed for volume "kube-proxy": stat /var/lib/kubelet/pods/{podid} no such file or directory Anything else we need to know?

: After digging into this issue for a quite long time, i think the reason for this issue is: The configMap volume uses emptyDir internally to create the volume directory, after the directory is created, configMap volume failed to populate it with configMap content because the disk is full, then configMap volume calls emptyDir.tearDown() to clear the temp directory. The problem is that emptyDir uses a "ready" file under /var/lib/kubelet/pods/{podId}/plugin/.. to determine whether the directory is created or not, if the ready file is present, then it wouldn't create the directory. The tearDown method only remove the directory, it doen't remove the ready file, so after the first out of disk error, the diretory creation always get skipped, hence no such file or directory error repeat over and over. Environment: Kubernetes version (use ): v.1.11.5 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): cent os 7 Kernel (e.g. ): Install tools: kubeadm Others:
/sig storage
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
If it's any help, I just encountered this issue in kubelet v1.13.2 under Debian Stretch 9.5, after a restart. I was not out of disk space. Deleting the ready files from /var/lib/kubelet/pods/{podId}*/ helped.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity.

Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen hit this just now with the same conditions as originally described, but without a reboot.
You can't reopen an issue/PR unless you authored it or you are a collaborator. <detailsIn response to : /reopen </details
/reopen
/remove-lifecycle rotten
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Thanks ! this helped me get my cluster running after all pods were evicted and clearing space alone didn't fix the issue.


Question: t.Errorf("Unexpected UID: %s", update.Pods[0].ObjectMeta.UID) } <del> case <-time.After(2 * time.Millisecond): </del> <ins> case <-time.After(time.Second): </ins> t.Errorf("Expected update, timeout instead") } }

W0128 18:44:26. ] Pod from test failed validation, ignoring: namespace: required value '' E0128 18:44:26. ] Could not construct reference to: '&api.BoundPod{TypeMeta:api.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:api.ObjectMeta{Name:"foo", Namespace:"", SelfLink:"", UID:"foo", ResourceVersion:"", CreationTimestamp:util.Time{Time:time.Time{sec:0, nsec:0x0, loc:(time.Location)(nil)}}, Labels:map[string]string(nil), Annotations:map[string]string{"":""}}, Spec:api.PodSpec{Volumes:[]api.Volume(nil), Containers:[]api.Container(nil), RestartPolicy:api.RestartPolicy{Always:(api.RestartPolicyAlways)(0xe3d670), OnFailure:(api.RestartPolicyOnFailure)(nil), Never:(api.RestartPolicyNever)(nil)}, DNSPolicy:"ClusterFirst", NodeSelector:map[string]string(nil), Host:""}}' due to: 'selfLink was empty, can't make reference'. Will not report event: 'failedValidation' 'Error validating pod foo.<empty-namespacefrom test, ignoring: namespace: required value E0128 18:44:28. ] Unable to read config path "randomnonexistentpath": path does not exist, ignoring --- FAIL: TestReadFromFileWithoutID (0.20 seconds) Expected update, timeout instead


Question: :KUBE-PROXY-FIREWALL - [0:0] :KUBE-SOURCE-RANGES-FIREWALL - [0:0] :KUBE-IPVS-FILTER - [0:0] <ins> :KUBE-IPVS-OUT-FILTER - [0:0] </ins> ` assert.Equal(t, expectedNATChains, fp.natChains.String()) assert.Equal(t, expectedFilterChains, fp.filterChains.String())

When using loadBalancerSourceRanges to allow access only from CIDRs outside the cluster segment, access from the K8s nodes (or pods with hostNetwork:true) should be rejected, but for proxy-mode=ipvs it works. /sig network /area kube-proxy /area ipvs Access from K8s nodes should be rejected in the same way as for proxy-mode=iptables Define a service with loadBalancerSourceRanges that allows access from a cidr outside the cluster, e.g. The K8s nodes have addresses on 192.168.1.0/24 and fd00::c0a8:100/120. Make sure a loadBalancerIP is assigned (in this example 10.0.0.0/32) and try to access it from a K8s node: Same for both ipv4 and ipv6. This is related to since a fix requires that the passed to the is correct. Please see v1.27.4 and master None Linux 6.4.4 None N/A N/A
wanted to work on this
Great! But please coordinate with I hope that one get accepted, and then ~the s (both families) passed to the proxiers should be usable.~ No, they are not, please see
Discussed this one in the SIG Network meeting today: we agree this is something we'd like to implement, however we weren't sure exactly how to do it with ipvs during the meeting. Whoever takes this on needs to come up with and communicate a clear plan of how it will be accomplished. /triage accepted
My comment is not feasible. For proxy-mode=ipvs the src and dst is both loadBalancerIP. So the nodeIP will not help :disappointed:
I did a small experimented with kind + metallb. I created a loadbalancer service backed by . loadBalancerIP: 172.18.255.200 podIP: 10.244.1.3 conntrack picks the src and dst both as loadBalancerIP. Tried to create an iptables rules matching source and destination with loadbalancerIP to drop packets but it didn't work. After some exploration I found and with the following rules I was able to prevent access to lbIP from k8s nodes / host network.

The solution looks hacky though, we only need this rule if source ranges are defined for the service and node ip is not covered by any of them. Will try to program this in the ipvs proxier.
Good find! :+1: I had missed the ipvs extension. I started to check nodeIP and the (flawed) option to experiment with setting a nodeIP as "src" for some routes to dodge the lbip-both-as-src-and-dst problem, and an old "vision" I have to not assign clusterIPs to at all, but have one rule to replace them all, like: It can be done, and in a cluster with 1000's of clusterIP's it would simplify


Question: "k8s.io/apimachinery/pkg/runtime" "k8s.io/apiserver/pkg/registry/generic" genericregistry "k8s.io/apiserver/pkg/registry/generic/registry" <ins> "k8s.io/apiserver/pkg/registry/rest" </ins> "k8s.io/kubernetes/pkg/apis/admissionregistration" "k8s.io/kubernetes/pkg/printers" printersinternal "k8s.io/kubernetes/pkg/printers/internalversion"

What would you like to be : Adding a "webhooks" group to kubectl to show mutating/validating webhooks would be super useful Why is this needed: There are a few reasons this is useful (to me): often cannot remember the right names for these resources, are very long and I often have a typo often want to enumerate all of them on the system. cc
/sig cli
I think this is a great idea, definitely something I could've used in the past. I'd be interested in taking a pass at implementing this if the proposal gets accepted.
Instead of building a category based on the types we have, I would build the category based on why people look at them. I suspect a category like which returned admission config admission config for various CSI types to list in CRD would be very helpful.
Webhook is far too generic. By putting admission webhooks in there we basically steal that word from the ecosystem to be used for other purposes. "admissionwebhooks" I could agree with.
Yeah, there are other webhooks, e.g. CRD conversion webhooks or authz webhooks...
I'd expect the short names "mwc" and "vwc" to be set up, are they not?
Unfortunately, they are not.
/triage accepted /assign
does category tolerate namespacing? i.e. would be tolerated? Still a bit long, but shorter than
It's a string, so you could theoretically namespace that.


Question: // TODO: Maybe we should launch a cleanup of this dir? return nil, fmt.Errorf("can't use volume plugins for %s/%s: %v", podUID, kind, err) } <del> if plugin == nil { // Not found but not an error. return nil, nil } </del> unmounter, err := plugin.NewUnmounter(name, podUID) if err != nil { return nil, fmt.Errorf("failed to instantiate volume plugin for %s/%s: %v", podUID, kind, err)

in there is a generic error variable defined that is used when a failure occurs and no other error is surfaced, it is a catch-all when something goes wrong. Think this should be reworded to something that is more reflective of the true issue and not so confusing for users. This error mostly comes when a builder can not be created and it is typically not true that the volume type is unsupported, but rather some other underlying issue, such as missing secrets or endpoints or etc... propose something like: or
are you working on this?
- yes, I would like to submit the PR for this

fyi When using PV/PVC for Persistent Volume volume mounts, if something goes wrong in the builder, a generic error is surfaced in the 'kubectl describe' with the following event error description *unsupported volume type": This only occurs when using the PV/PVC abstraction and not directly using a volume plugin. These type of errors are common when using volume plugins for mounts and would be useful for users to have an easy way to identify the errors rather than searching the logs for root cause. I think there are two issues: Why is the original error surfaced from the volume plugins (, , etc...) getting eaten when they return back to the I can see in the logs that 2 volume types are processed for any pod - the first is the actual underlying type (i.e. ) and this is where the actual real error is surfaced, then immediately after this the 2nd type is processed which is the general persistent claim type (i.e. ) and this when the first type error is eaten A simple work around would be to simply add a recorder event in the when there is a builder failure. This ensures the error is recorded and displayed as an event for the describer and will handle all like errors from other volume plugin types. full modifed function for reference from This will produce a better user experience when trying to see what went wrong - in this example, I am doing glusterfs PV/PVC but am missing end points which you can clearly see now in the events of the describe: rbd example: (Missing or Couldn't get secret)
I haven't had a chance to look at your proposal in detail yet, but I agree this is one of the most confusing things new users hit
another one is when gluster endpoints are missing, we get 'unsupported volume type' with your work extrapolating the permissions for NFS to automatically set supplemental groups, is there also a way to indicate more properly it's a permissions issue for it not mounting?
- the above discussion highlights the missing endpoints as the first example, it also discusses a similar rbd issue (missing secret file) but the point is, any of those type of volume plugin type of errors would be surfaced by adding the event in the
I like it! Is there a PR?


Question: NumCores:       4, MemoryCapacity: 1.6e+10, MachineID:      "somehostname", <ins> SystemUUID:     "E6C8AC43-582B-3575-4E1F-6DA170888906", </ins> }, nil }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: The windows kubelet does not set .status.nodeInfo.systemUUID on the node object. What you expected to happen: The windows kubelet sets the systemUUID from How to reproduce it (as minimally and precisely as possible): Start the windows kubelet and check for UUID: Anything else we need to know?: This will affect users running the out-of-tree vsphere CCM since that depends on the systemUUID, unlike the in-tree version. Also, looks like cAdvisor is the component responsible for providing the systemUUID so let me know if this issue needs to be moved to that repo. Environment: Kubernetes version (use ): Verified on 1.13.1 and 1.14 Cloud provider or hardware configuration: None OS (e.g: ): Windows Server 2019
/sig windows /sig vmware
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: proxier.StopProxy(service.ID) } glog.Infof("Adding a new service %s on port %d", service.ID, service.Port) <del> listener, err := net.Listen("tcp", fmt.Sprintf(":%d", service.Port)) </del> <ins> listener, err := net.Listen("tcp", net.JoinHostPort(proxier.address, strconv.Itoa(service.Port))) </ins> if err != nil { glog.Infof("Failed to start listening for %s on %d", service.ID, service.Port) continue

kube-proxy listens on all ports, but should be configurable. create a flag to specify which interface to listen on. Thanks for finding this bug.
For context, some providers apparently do external IPs by having a whole interface that gets 1:1 NAT without any firewalling. We do not want service ports on those interfaces. , Brian Ketelsen wrote:


Question: } wp.pendingPlugins = make(map[string]*time.Timer, len(pluginsMaxWaitTime)) <ins> // The time.AfterFunc calls wp.Reject which iterates through pendingPlugins map. Acquire the // lock here so that time.AfterFunc can only execute after newWaitingPod finishes. wp.mu.Lock() defer wp.mu.Unlock() </ins> for k, v := range pluginsMaxWaitTime { plugin, waitTime := k, v wp.pendingPlugins[plugin] = time.AfterFunc(waitTime, func() {

<!-- Please only use this template for submitting enhancement requests --What would you like to be : metric name Description -- -- permitwaitduration_seconds Duration of permit extension point finishes waiting. Why is this needed: In permit extension point, if the status is "Wait", we block until the pod is scheduled or timeout: Blocking is an important behavior that we should monitor. /sig scheduling
/assign


Question: return detailedErr } <del> glog.Infof(volumeToMount.GenerateMsgDetailed("MountVolume.SetUp succeeded", "")) </del> <ins> verbosity := glog.Level(1) if isRemount { verbosity = glog.Level(7) } glog.V(verbosity).Infof(volumeToMount.GenerateMsgDetailed("MountVolume.SetUp succeeded", "")) </ins> // Update actual state of world markVolMountedErr := actualStateOfWorld.MarkVolumeAsMounted(

TL;DR: kubelet keeps trying to remount secrets and configmaps into pods over and over again. My environment: kubernetes 1.3.0 running on Ubuntu 14.04. I used the instructions here to set things up: (i.e. etcd, flanneld and the kube* processes are running on the host OS, not in containers or docker-in-a-vm ala minikube) In trying to track down high disk write rates, I found that the partition I use for my etcd WAL log was getting about 40kb/s write traffic. etcd metrics show that I'm getting 3 qps get and 3qps getrecursive. Write traffic to etcd is essentially nil (pods are all stable). This is a single node system with ~30 user pods, most of which are doing nothing unless I sent traffic to them (which I'm not while doing these measurements). In the upstart kubelet log, I see this over and over again: So it's repeatedly remounting already-mounted secret volumes and configmaps into pods. Analyzing the unique pod IDs, this appears to be affecting all of them: I suspect the 2 missing pods are from old deployments. This goes on ad infinitum every 3-5 seconds, hitting 2-3 pods each time, so presumably some kind of maint loop. The code branch in is this one: and that err is populated from this line: But beyond that I'm not sure why it is consistenly thinking that the mount operation didn't stick. Any ideas where to go from here?
Could it be as simple as that RequiresRemount method in , and are while all the other plugins ?
I can recreate this problem on master with the following sample: Leave it running for 10 mins or so and shows:
Looks like this is by design : The plugin(s) are telling the system to keep remounting to update the contents of the volume. Patching the following to stops the repeated re-mount. Folks are probably using this mechanism to keep the data mounted to be up-to-date. So not sure we should "fix" this behavior. Thoughts please?
Yes, current design keep remounting to update the contents because currently there is no direct way of informing the changes to kubelet.

The remounting normally happens every 10~30 seconds from my observation. In case this behavior cause performance issue, we can consider optimize it.
Could you decrease the verboseness when the volume is already mounted? This is polluting our logs and makes more important failures harder to follow.
Sure. We will work on that. Thanks for posting the issue. Jing 2016721Steffen Hanikel
I'm facing the same issue. Kubernetes 1.3.2 on GKE Container stuck on the ContainerCreating phase since volumes never get mounted. Since default secret keep being remounted all the time, my secret never get mounted:
Hi Taras, I think your problem is different. The log shows that /var/lib/kubelet/pods/ -53de-11e6-b2f2- no such file or directory Looks like it very much related to a name collision bug and there is fix being reviewed Jing , Taras wrote:
Hi Yes, probably it related. Waiting for the fix to get to release to test it on GKE. I needed to downgrade nodes to the 1.2.5 to be able to run pods.
Hello, a parameter to say if the secret need to be updated and remounted periodically would be great :) For most of my pods i just need a secret during the pod first start, and i don't necessarily want the content to stay an be remounted over and over in the pod.
Thanks for your suggestions. I think the log might be a a little bit misleading. If the secret volume is already mounted, it won't mounted ago, But it does rewrite the secret. I think we can avoid rewriting by comparing the contents. Is that good enough? Thanks! Jing , osixia wrote:
yes, how i use secrets not rewriting the content would be great, to avoid rewriting deleted files especially.
do you mean you have some cases that you want to delete some files under secrete directory, and don't want the system to rewrite those files after deletion? It would be great if you could let us know more details about your use case so that we are more clean about what needs to be done. Thanks!

Jing , osixia wrote:
i don't know if it's revelant and may be i just do stupid things just let me know :) a litle example: I have an openldap image, when the container starts for the first time if no database is provided it creates a new one with an admin password. When the database is created the admin password is stored encrypted and i don't want it to stay in the container environment in clear text. (i do the same for mariadb images) So as i don't want the admin password to stay in clear text in the container environment i can't set it via environment variable. I found a solution with secrets : I set the admin password in a configuration file, and put that file in a secret. When a new pod is created the secret was mounted in a volume and the container takes the admin password from the configuration file, create the database and delete the configuration file from the secret's mounted volume. I can start several pods this way with the same secret and admin password and don't have the admin password remaining in the container environment or file system in clear text.
Thank you for posting your use case. We plan to add a feature for addressing it, track it under . Please let me know if you have any suggestions. Thanks! Jing
thank you very mutch :) I have an other thought about the service-account default token, i guess this one will be automatically remounted, but is there any way to not mount this secret to a pod ? I'm concerned about security issues if a pod is compromised, that will allow someone to request the kubernetes api ?
For the problem about excessive disk first raised by I think maybe kubelet should remember which secret UID was last written, and not rewrite if the secret UID has not changed. What do you think For If you want to not have the token mounted, you can do this: It isn't pretty, but it works.
thanks but the trick seems to not work anymore :( returns: if i remove the pod keep crashing with :
What do the pod logs say for the crashing container (e.g. kubectl logs i-have-no-sa)?
The logs are empty
I think your pod is failing because of the combination of image/command.

This gist works: For example, this shows that 's workaround works:
it worked, thanks a lot :)
Could you decrease the verboseness when the volume is already mounted? This is polluting our logs and makes more important failures harder to follow. Sure. We will work on that. Thanks for posting the issue. We're running 1.4.7 now and still seeing a high volume of these messages: With about 125 Pods running in our cluster I'm getting about 6000 log messages per hour similiar to this.
I have similar issue to with fresh kubeadm install on kubernetes 1.5.1
I'm facing the same issue only on one node of my cluster, it fills the number of mounts pretty quick leading to unresponsive OS ( 32k mount )
could you provide some more details about your case? Do you use nfs or gluster in the pods? Could you also give a some of the mounts you see on the node so we can have a better understanding what might cause the issue? Thanks!

Sure sorry, i had lots of tmpfs /home/lib/kubelet/pods/-f25f-11e6-8d1a- tmpfs rw,relatime 0 0 tmpfs /home/lib/kubelet/pods/-f259-11e6-8d1a- tmpfs rw,relatime 0 0 tmpfs /home/lib/kubelet/pods/-f25f-11e6-8d1a- tmpfs rw,relatime 0 0 tmpfs /home/lib/kubelet/pods/-bf6f-11e6-b5ec- tmpfs rw,relatime 0 0 tmpfs /home/lib/kubelet/pods/-f25f-11e6-8d1a- tmpfs rw,relatime 0 0 tmpfs /home/lib/kubelet/pods/-f25f-11e6-8d1a- tmpfs rw,relatime 0 0 tmpfs /home/lib/kubelet/pods/-f25f-11e6-8d1a- tmpfs rw,relatime 0 0 tmpfs /home/lib/kubelet/pods/-ddaa-11e6-8d1a- tmpfs rw,relatime 0 0 tmpfs /home/lib/kubelet/pods/-f25f-11e6-8d1a- tmpfs rw,relatime 0 0 The pod -f25f-11e6-8d1a- was restarting a lot ( a misconfiguration on my part ) i'm using version 1.4.7
which version of kuberentes you are using? I see the exact mounts you posted are repeated many time. In the current mount code path, it should check whether the path is already mounted and if it is yes, it will skip to mount it again. I will check again our code path.
Same here with 1.5.2 on CoreOS - it wasn't the problem with 1.3.7, same setup.
Same issue with 1.5.2_2 on CoreOS latest stable as well.
Same issue with 1.5.4 on Centos 7.2 as well.

because the our log might be a little bit misleading, could you confirm whether you see redundant mounts for a pod volume such as secrete when you run "mount" command on the host? Thanks!
I'm seeing this with v1.5.2 running on AWS, setup via kops: The log has entries from the past ~4 days:


Question: } limit1, ok := obj1.(*example.PodList) if !ok { <del> t.Fatalf("Expected PodList but got %t", limit1) </del> <ins> t.Fatalf("Expected PodList but got %v", limit1) </ins> } if len(limit1.Items) != 2 { t.Errorf("Expected PodList of length 2 but got %d", len(limit1.Items))

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Unit tests are failing with the latest golang(1.12) What you expected to happen: Tests should pass without any fauilures. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Others:
/sig api-machinery


Question: etcdStorageData[gvr("storage.k8s.io", "v1beta1", "csinodes")] = StorageData{ Stub:             `{"metadata": {"name": "csini1"}, "spec": {"drivers": [{"name": "test-driver", "nodeID": "localhost", "topologyKeys": ["company.com/zone1", "company.com/zone2"]}]}}`, ExpectedEtcdPath: "/registry/csinodes/csini1", <ins> ExpectedGVK:      gvkP("storage.k8s.io", "v1", "CSINode"), </ins> } // k8s.io/kubernetes/pkg/apis/storage/v1 etcdStorageData[gvr("storage.k8s.io", "v1", "csinodes")] = StorageData{ Stub:             `{"metadata": {"name": "csini2"}, "spec": {"drivers": [{"name": "test-driver", "nodeID": "localhost", "topologyKeys": ["company.com/zone1", "company.com/zone2"]}]}}`, ExpectedEtcdPath: "/registry/csinodes/csini2", <del> ExpectedGVK:      gvkP("storage.k8s.io", "v1beta1", "CSINode"), </del> } }

<!-- Please only use this template for submitting enhancement requests --What would you like to be : storage/v1.CSINode is being introduced in 1.17 via The storage version should switch to v1 in 1.18. Why is this needed: We need to wait until 1.18 to handle the HA upgrade case where a controller may be 1 version higher than the API server. /assign
These files will have to be updated:


Question: 0, true, }, <ins> { "mirror pod with system priority class", []*scheduling.PriorityClass{}, *pods[5], SystemCriticalPriority, false, }, { "mirror pod with integer priority", []*scheduling.PriorityClass{}, *pods[6], 0, true, }, </ins> } for _, test := range tests {

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Priority admission controller of mirror pods. This is an error and should be removed. What you expected to happen: Integer value of Priority should be populated by the admission controller. How to reproduce it (as minimally and precisely as possible): Create a static pod with PriorityClassName set. The created mirror pod will not get the integer value of priority. /sig scheduling xref
what is the kubelet expected to do with a low priority static pod if it gets evicted?
also doesn't this mean a user can claim their pod is a mirror pod via an annotation and get it admitted with any priority they want?
knows better about kubelet evictions, but in my opinion a static low priority pod should be evicted similar to other pods. Whether it makes sense for static pods to be low priority or not is a relevant question. I would say it doesn't make sense, but one might find a reason to have static low priority pod. Not really. This check effectively bypasses integer resolution of priority causing the priority of all those mirror pods to be considered zero. This if statement is wrong and should be removed.
It also bypasses the check that prevents the pod creator from setting an integer priority themselves. In any case, agree removing the bypass is good
Static pods that fail for any reason (eviction is just one) are not re-run. To avoid evictions entirely as a static pod, users can use the critical pod annotation.
, it seems we just log error if creating mirror pod failed; refer to . Does it make sense?
I am not sure I understand the question entirely. The mirror pod exists only to show the status of the static pod. If the mirror pod doesn't exist, the static pod should still function correctly. Since that is in , it will be retried periodically.
The scheduler dependent on mirror pods for resource usage :). If creating mirror pod failed (e.g. wrong priority class name), there'll be ping-pong between scheduler and kubelet: scheduler send pod to node, kubelet reject it. But we can-t just return error, kubelet can start in standalone mode.
However critical pod annotation is ignored if pod priorities are in use.

This is not the case. See in your issue.
However critical pod annotation is ignored if pod priorities are in use. the intent of pod priorities was to replace the critical pod annotation with something more principled. is that not how it was implemented?
Priority replaces most of the functionality that the critical pod annotation provides. However, the critical pod annotation is still useful for static pods, as it can prevent kubelet evictions. But for all other use-cases, I agree that priority is a much better way to manage which workloads are run, and which remain pending. I am not sure if the scheduler translates a "critical" pod to a "system-priority" pod for scheduling purposes, but the kubelet still prevents the eviction of static critical pods as an escape hatch for important static pods which cannot tolerate kubelet evictions.
Why would that not be modeled as a system priority? I expected critical pod annotations to not be present once the priority feature was completed.
I agree with this. do you think we should not evict static pods which are system priority, so that even users of static pods can move from using the critical pod annotation to use pod priority?
I think Kubelet should not evict static system pods, but even with the current logic of eviction, if these Pods do not use more resources than requested, kubelet never evicts them as their priority is higher than other pods.
I think as a best practice users that write static pod manifests should apply a system priority to prevent eviction, but I do not think the kubelet should do anything magical about that source.


Question: "uniqueItems": true }, { <del> "description": "Redirect the standard output stream of the pod for this call. Defaults to true.", </del> <ins> "description": "Redirect the standard output stream of the pod for this call.", </ins> "in": "query", "name": "stdout", "type": "boolean",

As per the PRs kubernetes/kubernetes kubernetes/kubernetes, the defaulting behaviour of the pod exec parameters had been changed however in the code there is a comment where it is mentioned it defaults to True The comment about the default value must be removed and the openapi spec must be fixed Present in the file as comment and also in the OpenApiSpec swagger If this issue seems appropriate can I take it up please <details</details<details</details<details</details<details</details<details</details<details</details
/sig docs
/sig node
/assign
/assign
/help /triage accepted
This request has been marked as needing help from a contributor. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help /triage accepted </details
/kind documentation


Question: fmt.Sprintf("%s::%s", f.Namespace.Name, pod0): boundedSample(10*e2evolume.Kb, 80*e2evolume.Mb), fmt.Sprintf("%s::%s", f.Namespace.Name, pod1): boundedSample(10*e2evolume.Kb, 80*e2evolume.Mb), }), <del> }) </del> <ins> "pod_swap_usage_bytes": gstruct.MatchElements(podID, gstruct.IgnoreExtras, gstruct.Elements{ fmt.Sprintf("%s::%s", f.Namespace.Name, pod0): boundedSample(0*e2evolume.Kb, 80*e2evolume.Mb), fmt.Sprintf("%s::%s", f.Namespace.Name, pod1): boundedSample(0*e2evolume.Kb, 80*e2evolume.Mb), }), }), haveKeys("scrape_error", "node_cpu_usage_seconds_total", "node_memory_working_set_bytes", "container_cpu_usage_seconds_total", "container_memory_working_set_bytes", "container_start_time_seconds", "pod_cpu_usage_seconds_total", "pod_memory_working_set_bytes"), ) </ins> ginkgo.By("Giving pods a minute to start up and produce metrics") gomega.Eventually(ctx, getResourceMetrics, 1*time.Minute, 15*time.Second).Should(matchResourceMetrics) ginkgo.By("Ensuring the metrics match the expectations a few more times")

Node e2e features is failing after PR for graduating swap to beta is merged. After is merged. Test needs to be adjusted for the extra field for swap usage. No response /sig node
/cc
/assign
/triage accepted /priority critical-urgent


Question: "k8s.io/klog" api "k8s.io/kubernetes/pkg/apis/core" k8s_api_v1 "k8s.io/kubernetes/pkg/apis/core/v1" <ins> persistentvolume "k8s.io/kubernetes/pkg/controller/volume/persistentvolume/util" </ins> kubeapiserveradmission "k8s.io/kubernetes/pkg/kubeapiserver/admission" )

API server latency for PV creation depends on underlying cloud API latency. We have admission plugin that examines incoming PV and adds topology labels and node selector: When the cluster (and cloud) are under pressure, e.g. lot of PVCs are created and dynamically provisioned at one time and attached to nodes, API server latency increases. We've seen cases when we ran out of cloud API request quota and the admission plugin needed to sleep for couple of seconds (on AWS). There should be an easy fix: check, if the created PV already has all necessary labels and topology information and ask the cloud for them only when they're missing. Dynamically provisioned volumes already have these labels. The admission plugin will then ask the cloud only for manually provisioned PVs, which we believe is just a fraction of all PVs in the cluster. There is one glitch: previously, when user created a PV with wrong topology labels, the admission plugin corrected them. With the fix suggested above, this PV won't be corrected. /sig storage cc
Implementation of the above:


Question: if err != nil { t.Fatal(err) } <ins> defer etcdclient.Close() </ins> server, err := apiservertesting.StartTestServer(t, apiservertesting.NewDefaultTestServerOptions(), nil, storageConfig) if err != nil {

The integration framework runs an apiserver and sometimes other components before each test execution, however, the shutdown is not clean and is leaving a lot of goroutines orphan that keep accumulation between different runs. Some orphan goroutines are connections to etcd, and since the etcd process is shared for all test, we don't know if this can cause contention and flakiness on the job. We are logging the number of goroutines leaked per test binary execution, it is important if we can reduce them and more important we don't regress and this number keeps growing. This issue will record the state as today for reference:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig testing
/cc
hmm, not doing better
I'll take the scheduler part for a deep investigate.
For people that want to get access to the stack traces, just apply this patch, so the framework will dump the output of the goroutines after each execution: [x] Apiserver was shutting down the listener before the hooks could finish, hence the controllers couldn't shutdown gracefully since there was no http server [ ] The storage cacher doesn't differentiate between not read and stopping - Missing integration test reproducing the issue, only reproduced with unit tests [ ] admission plugins can't shutdown gracefully - - there is a PR addressing However, most of the contributions to goroutines leaks are grpc connections that are related to the storage cleanup, there is also a TODO in the code and an open issue I've also tried to reduce the number of connections between storage and etcd, right now, each Storage creates an independent conection to etcd, but sharing the etcd client doesn't work
I think I made some progress with it. I was playing with ttlcontroller test (as it's fairly simple one for the playground and I'm familiar with it, but symptoms in many tests are exactly the same) In I actually managed to reduce amount of leaked goroutines from 1023 to 85.

It's not mergable in this state, but it clearly shows that destroy functions for storage are not called in integration tests. I will try to dig deeper into those and clean this up - but I think the above clearly shows where the problem is. - FYI ^^
is out for review and it's making most of the job here There are still leftovers, but those are mostly oneof's here and there - we should address them independently.
And a bit more longerterm cleaner approach is in
With we're down to 14. I admit that this is one of the simplest tests, but apiserver shutdown is pretty clean once that PR merges.
/honk
! <detailsIn response to : /honk Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
a leftover for the storage cleanup in
Re comment above - yeah, I'm aware, we need to clean it up. FTR:
awesome, we are several order of magnitud better
Looking further into it, in different tests I'm seeing a lot of goroutines with the top of the stacktrace like this: should hopefully address those once it merges... ^^
OK - I think I have now the biggest thing contributing factor to the remaining tests, although I don't yet fully understand it and don't know how exactly to fix it. The main problem is basically that TestServer is not finishing its shutdown: So what exactly is happening is: we run TryDown the TestServer there: this underneath is triggering run of PreShutdown hooks: in the meantime ShutdownInitated signal is signaled: as part of running preshutdown hooks, the bootstrap controller is stopping endpoint reconciler: and then as part of doReconcile, the GET requests hangs forever: By adding time.Sleep(5s) (didn't try smaller values TBH) before the ShutdownInitiated is signalled, the TestServer is shutting clearly: Obviously that's not a solution, but I'm wondering why the GET request mentioned above is hanging forever when ShutdownInitiate is signaled and what we can do with this. - thoughts? [The easiest repro is to run: Adding this sleep basically reduces the number of leaked routines from O(1000) to O(20).]

<--- UPDATE ---In fact, setting ShutdownDelayDuration to 5s has the same effect. So there are two main questions: if setting ShutdownDelayDuration is reasonable why the request actually hangs forever instead of finishing with some error... <--- UPDATE ---I think there must have been something different, as I'm no longer able to reproduce the improvement with my current code. Will try to dig deeper. <-- UPDATE ---OK - the last commit in brings it back to fixed.
I thought that the reconciler hangs because of
Not in this place - here the usual GET request hangs - I will keep looking over the next days.
OK - I understand now and the above isn't really true. The request wasn't really hanging forever. What was happening is that it was being rejected by the WaitGroup handler: and retried by the client-go internals... So I guess from that perspective setting non-zero ShutdownDelayDuration actually makes sense.. - thoughts? ^^
mhm, not sure why setting should help. In general, the server shout stop listening and the HTTP std library should instruct the client's stack to tear down the underlying TCP connection. In that case, the client-go should fail on dialing and stop retrying releasing the server. thoughts?
It's that with ShutdownDelay, the "stopCh" is signaling PreShutdown hooks (that still want to do some work, including API calls) and if we immediately start rejecting that, they will not be able to do that work...
Discussed a bit more offline - I'm going to investigate blocking closing the waitGroup filter on preshutdown hooks. Will try to send a PR in upcoming days.
We're getting there - this is where we are now: I have a bit more in my local branch to upstream, but I need to clean them up and may not have time in the next week or so.
where are we standing today?
Getting close:
A bit more coming soon.
This is where we are now: I understand some of them (though fixing is potentially non-trivial) and need to understand the others (in particular serving). And we should prevent regressions by failing if we have more leaking goroutines - will be looking at that over upcoming weeks.

OK - so here is where we are (with the current check set to not leaking more than 4 goroutines): Comparing to the first comment in this issue - that looks like a significant improvement :) Here is the summary of I think the remaining goroutines in the above files: [x] integration/apiserver/tracing: The goroutines are coming from otel exporter (in particular grpc layer). It seems that something is not closed, although required digging deeper into otel exporter. [ ] integration/serving: The goroutines are coming from the failure case: And they basically are coming from the fact that Queue when created is spawning a goroutine: which is only shutdown on Shutdown() call. But Shutdown in all controllers is only called in defer of controllers of the function that is actually not being run in failure cases. The actual queue that is problematic here is coming from here: [x] testing/integration/controlplane/transformation The goroutines are coming from the fact that we're not closing the kms transformer (grpc goroutines) that is being created. We basically create a grpc service and never close that: Fixing would probably require changing transformer interfact to implement Close method or sth like that.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
- can you please help with the apiserver/tracing test mentioned in
looking
Opened to try fixing the goroutine leak.
It seems the fix worked fine - I'm marking the tracing as fixed - thanks I'm also trying to fix couple other smaller things.
I've opened as a final step. With that PR we're in good enough shape that I'm going to close this issue as further improvements would have negligible impact.


Question: MemoryConstraint: 100 * (1024 * 1024), } constraints["l7-lb-controller"] = framework.ResourceConstraint{ <del> CPUConstraint:    0.2 + 0.0001*float64(numNodes), </del> <ins> CPUConstraint:    0.2 + 0.00015*float64(numNodes), </ins> MemoryConstraint: (75 + uint64(math.Ceil(0.8*float64(numNodes)))) * (1024 * 1024), } constraints["influxdb"] = framework.ResourceConstraint{

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug /priority failing-test What happened: https://k8s- failed with: Attached graph shows that it's not a huge increase from normal usage. Dear SIG networking, is this expected? Should we just bump the limit? ! Please note this is in a release-blocking job.
Umbrella bug:
We were seeing a similar issue for memory a while ago () for l7-lb-controller.