Question: // test pods from running, and tests that ensure all pods are running and // ready will fail). if err := waitForPodsRunningReady(api.NamespaceSystem, testContext.MinStartupPods, podStartupTimeout); err != nil { <del> t.Errorf("Error waiting for all pods to be running and ready: %v", err) return </del> <ins> Failf("Error waiting for all pods to be running and ready: %v", err) </ins> } <ins> return nil }, func(data []byte) { // Run on all Ginkgo nodes }) // Similar to SynchornizedBeforeSuite, we want to run some operations only once (such as collecting cluster logs). // Here, the order of functions is reversed; first, the function which runs everywhere, // and then the function that only runs on the first Ginkgo node. var _ = ginkgo.SynchronizedAfterSuite(func() { // Run on all Ginkgo nodes }, func() { // Run only Ginkgo on node 1 if testContext.ReportDir != "" { CoreDump(testContext.ReportDir) } }) // TestE2E checks configuration parameters (specified through flags) and then runs // E2E tests using the Ginkgo runner. // If a "report directory" is specified, one or more JUnit test reports will be // generated in this directory, and cluster logs will also be saved. // This function is called on each Ginkgo node in parallel mode. func TestE2E(t *testing.T) { util.ReallyCrash = true util.InitLogs() defer util.FlushLogs() // We must call setupProviderConfig first since SynchronizedBeforeSuite needs // cloudConfig to be set up already. if err := setupProviderConfig(); err != nil { glog.Fatalf(err.Error()) } gomega.RegisterFailHandler(ginkgo.Fail) // Disable skipped tests unless they are explicitly requested. if config.GinkgoConfig.FocusString == "" && config.GinkgoConfig.SkipString == "" { // TODO(ihmccreery) Remove [Skipped] once all [Skipped] labels have been reclassified. config.GinkgoConfig.SkipString = `[Flaky]|[Skipped]|[Feature:.+]` } </ins> // Run tests through the Ginkgo runner with output to console + JUnit for Jenkins var r []ginkgo.Reporter if testContext.ReportDir != "" { <del> r = append(r, reporters.NewJUnitReporter(path.Join(testContext.ReportDir, fmt.Sprintf("junit_%02d.xml", config.GinkgoConfig.ParallelNode)))) </del> <ins> // TODO: we should probably only be trying to create this directory once // rather than once-per-Ginkgo-node. if err := os.MkdirAll(testContext.ReportDir, 0755); err != nil { glog.Errorf("Failed creating report directory: %v", err) } else { r = append(r, reporters.NewJUnitReporter(path.Join(testContext.ReportDir, fmt.Sprintf("junit_%02d.xml", config.GinkgoConfig.ParallelNode)))) } </ins> } <del> glog.Infof("Starting e2e run; %q", runId) </del> <ins> glog.Infof("Starting e2e run %q on Ginkgo node %d", runId, config.GinkgoConfig.ParallelNode) </ins> ginkgo.RunSpecsWithDefaultAndCustomReporters(t, "Kubernetes e2e suite", r) }

Both the GKE and Parallel tests recently had a run where they timed out. (after 90 minutes!) This is like 10x longer than the parallel run should be taking.
And the build project, too:
any ideas?
p0
yup, investigating this is next on my queue, right after I figure out why the PR jenkins is failing so many runs.
This happened again: In fact I've seen at least few cases like that: "at some point tests just stop and completely nothing is happening to the end" The same happened here - completely nothing happened during last hour of the run,
While the failure rates of the individual test jobs is not very high (e.g. kubernetes-e2e-gce-parallel : MTTF (Last 7 Days) 7 hr 27 min ... the fact that we gate auto-merging on about 7 of these jobs means that a merge fails almost every hour due to this.
I looked at 4 of the builds in kubernetes-e2e-gce-parallel, comparing the test cases marked "N/A" (i.e. no test result given) instead of their usual "PASSED" or "SKIPPED" report. So far, there's been only one test case that's been consistently N/A in all four of the builds I looked at: I'm not sure what might be happening, but I'm tempted to disable this test in parallel runs now while we investigate what's actually wrong.
OK, that test has been disabled. I'm hoping that the PR builds recover (along with some git fixes as referenced in ). Now on to figuring out why this is causing us problems...
Had a successful failure in build 8902. Grepping out the Ginkgo node number shows this test being the problem: So it looks like it got stuck in ? Is there anything in here that should hang without deadline?
I had another run get stuck in exactly the same place: What's worrying is that another parallel build (http://go/k8s-test/job/kubernetes-e2e-gce-parallel/7960/) timed out again early this morning, but this networking test was not enabled. This suggests that there's still something else triggering the problem.
I may have a partial fix for this in .
Have we seen this since merged? Can we close this bug?
- I think we had a lot of other problems during that time.

I would suggest leaving it open during the weekend and close it on Monday if we don't see any problems.
I don't think we've seen this in a while. Please reopen if we're still having mysterious timeouts in Jenkins.
We're seeing this again in . It looks like someone put "Networking should function for intra-pod communication" test back in the parallel run recently. I think that we should revert whatever that PR was.
That test had been fine. More worrying is that my fix from seems to have vanished.
OK, looks like that was removed in , referencing issue . I'm not sure if the same issue is happening again (where operations mysteriously hang forever) or if it's something else. any thoughts?
I think your answer is here: The timeout field, and all references to it in clients were removed, based on the fact that clients should use request.Timeout() instead, but the clients were not updated to do that. I will send a PR shortly.
It seems like very few of the e2e tests are using this method, but we aren't seeing issues on most of the tests. I wonder if there's something in particular we're hitting with the networking test?
Could this be this magic number in ? I guess this means the checks only happen for about a 75 seconds. If one or two of the nodes are spinning up very slowly, this will trip over . Just a thought. Either way I think this needs to be configurable, I'll work on that angle.
I agree this "15" should probably be bigger. However, I'm pretty sure, that this shouldn't cause the test hang forever - it may cause test failures, but hanging forever test shouldn't happen...
Anyway - I've sent out out for review - we will see if that helps...
OK - it won't help, because it doesn't work :( - I asked if there is any reasonably easy way to do it...
Another instance: Again - the "Networking should function for intra-pod communication" is among tests that didn't run...
assigning this issue to you since you're working on a PR
- sure, although I'm not sure if this will solve the issue...
you can assign back to me if it doesn't :)
Unfortunately my PR didn't help. This just happened again: - reassigning back to you
Another instance:
And again:
Are folks still seeing this?

I'm having a really hard time getting it to occur on-demand.
OK, I did find another timeout - The list of suspicious tests does not include the networking test this time: Kubectl client Kubectl patch should add annotations for pods in rc [Conformance] Kubectl client Simple pod should support exec through an HTTP proxy Pods should be submitted and removed [Conformance] Secrets should be consumable from pods [Conformance] Will have to keep searching for builds timing out to see if any new pattern emerges.
Looking more carefully, is another timeout. (I really wish it'd fail on my debug PR...) Suspicious tests: Job should scale a job down Kubectl client Simple pod should support exec through an HTTP proxy I'm gonna start suspecting that Kubectl HTTP proxy test now... are we doing anything in this test that might be blocking?
has a very curious failure: only 3 tests ran at all. It seems we are running the cluster-sanity checks in for every Ginkgo node, so there are probably cases like this run where it'll race and fail in mysterious ways (or possibly even deadlock)? We should probably use and (as documented ) to run the sanity checks, cleanup, and log dumping only once, rather than once per Ginkgo node. This may solve the mysterious timeouts.
Bumping down to P1 because this seems pretty rare nowadays (but I'll still address the issues I noted).
It seems like this issue is way too general: jobs timing out without a clear reason should have an action of making that reason clear. Is there any action we can take there?
I think the fixes I mentioned in will address the mysterious timeouts.
Happened here:
Hoping this one is gone for good, but I'll track and see.
Reopening; saw this again with
Final lines of that before being killed: Another run which timed out is kubernetes-pull-build-test-e2e-gce/. Final lines from that: Plus my earlier suspicions in make me think the kubectl proxy test is lacking a necessary timeout somewhere.
Moving discussion of that test case to .
With open, can we close this in favor of ?
Closing for now. If fixing doesn't fix, I'll keep looking...


Question: return newObj } <ins> // ObjectDiff writes the two objects out as JSON and prints out the identical part of // the objects followed by the remaining part of 'a' and finally the remaining part of 'b'. // For debugging tests. </ins> func ObjectDiff(a, b Object) string { ab, err := json.Marshal(a) if err != nil {

For example: struct field tag not compatible with reflect.StructTag.Get missing argument for Errorf("%v"): format reads arg 2, have only 1 args range variable pod enclosed by function struct field tag not compatible with reflect.StructTag.Get arg minion for printf verb %s of wrong type: unreachable code unreachable code
Those all look really important
I'm going to fix this unless I hear that someone else is working on this in the next couple minutes.


Question: t.Error(err) } } <ins> func TestGetNonExistentImagePullSecret(t *testing.T) { secrets := make([]*v1.Secret, 0) fakeRecorder := record.NewFakeRecorder(1) testKubelet := newTestKubelet(t, false /* controllerAttachDetachEnabled */) testKubelet.kubelet.recorder = fakeRecorder testKubelet.kubelet.secretManager = secret.NewFakeManagerWithSecrets(secrets) defer testKubelet.Cleanup() expectedEvent := "Warning FailedToRetrieveImagePullSecret Unable to retrieve some image pull secrets (secretFoo); attempting to pull the image may not succeed." testPod := &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Namespace:   "nsFoo", Name:        "podFoo", Annotations: map[string]string{}, }, Spec: v1.PodSpec{ ImagePullSecrets: []v1.LocalObjectReference{ {Name: "secretFoo"}, }, }, } pullSecrets := testKubelet.kubelet.getPullSecretsForPod(testPod) assert.Equal(t, 0, len(pullSecrets)) assert.Equal(t, 1, len(fakeRecorder.Events)) event := <-fakeRecorder.Events assert.Equal(t, event, expectedEvent) } </ins>

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --imagePullSecrets can to a pod manifest to reference a secret containing registry credentials. Currently, if you add the name of a secret that does not exist, there is no warning or error. I would expect there to be an error and a failed deployment if the secret does not exist/cannot be found in the namespace. If people are pulling public images, they may still wish to authenticate to a registry, for example dockerhub, before pulling the image, in order to utilise their subscription account rate limits. Currently if they use imagePullSecrets and the referenced secret does not exist, they may assume they are authenticating to their account when they are not, and will still be able to pull public images without any warning that authentication is not taking place. Add to a pod spec, and pull images from
/sig node (based on the field location)
(and/or) /sig auth
correct me if not. Pulling private images with nonexists secret, pod will turn to . However, pulling public image with nonexists secret, no error will display yet.
Similar issue with .
Yes that's correct.
I think so. Working on this later. Will also live up to your expectations, right?
But I check it already have a warnging events if secret not exist and pullImageFail: line:147
Now, only info message when pullsecret not exist
maybe direct return and not enter is better. if pullSecret is not existed
if you pull image from a public repo, the process should not be impacted even no secrets exists. And it's what it is right now.
I think a warning event like could be here /good-first-issue /triage accepted
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /good-first-issue /triage accepted Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.

</details
/assign
GitHub didn't allow me to assign the following users: EricWvi. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
GitHub didn't allow me to assign the following users: EricWvi. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : I would like work on this. /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I would like to work in this. /assign
Thanks for wanting to jump in on this, but it's not clear this is a good first issue… making the proposed change would break deployments/pods that are currently working/running, so it's unlikely this would be acceptable from a compatibility perspective.
Requiring image pull secrets to exist would have to be opt-in, which means an API change. I don't think this is worth the addition, but would like to hear sig-node's perspective. An alternative could be to explore ways to make it more obvious to the user when this situation was occurring without automatically failing the pod
hi , we will not fail the pod, we just want to add some warning to make it obviously, maybe it's the same as you advised. correct me if not.
To be clear here, only adding a warning is marked as good first issue here, not changing any pod behaviour.
gotcha, that seems more reasonable, thanks for clarifying as a note, make sure the warning events are generated at most once in that function, rather than once per missing secret object (if there are multiple missing secrets)
/good-first-issue
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed .

If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Yes, a warning would be good. I feel an error would have be better, but as has already been mentioned above we are too far along now for this change as it could break some users existing deployments, so a warning makes the most sense now.
/assign
based on what you have stated above, I see an INFO log already exists! If I'm understanding this correctly, now we need to raise a warning event in the same code block to fulfill this request. Is that right?
I think has created a above and labeled /lgtm already.
Hey I don't know anything about how to code but i am interested to contribute on kubernetes So help me to contribute
/cc
there is PR already here , just needs approval
it was just FYI =) Reviewed the PR, thank you
Thanks for the review :)
/assign
will like to work on it. /assign
Hey , would love to work on this issue
there is already a PR: do you plan to push the PR forward?
I have been busy with other things recently, if anyone is willing to continue this problem. I will close the PR.
if noone else is working on this task, I would like to take this up.
Feel free to work on this , you can refer to for some advices.
Thanks! Will refer it
/assign
/assign
/assign
Is this issue still open/active?
Hello I was working on it but not able to continue it due to some personal reasons. You can take over if you would like to. You can refer this PR if you want to.
Yes sure!!!
Can I try this if no one is available to work on it ?
Yes go on!!!
/assign
Sorry for keep opening the PR. something went wrong on my fork branch. I eventually needed to re-fork the repo so it caused the PR closed automatically
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
Can I give it a go if no one is working on it?
Thanks for the offer, but it looks like was open and had an update that was waiting for review (which I didn't realize)
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
is this issue still available to pick up?
/remove-lifecycle stale
Hi, is this issue still available?
Hi! It feels that this issue is stalled (although it lacks the label) and the related PRs have been abandoned. I'll work on it if that's ok /assign
As well as an event, we might like to directly report a warning to end users (suggested in ). More complexity though.
should it be part of this issue? it only mentions but as you guys (the contributors) decide. is based on which was labelled as and suggested as a starting point before being closed.


Question: return false } <del> // We don't check TargetPort; that is not relevant for load balancing // TODO: Should we blank it out?  Or just check it anyway? </del> <ins> if x.TargetPort != y.TargetPort { return false } </ins> return true }

Hello , I have saw the commit , can we check the for enqueue to serivce cotnroller work queue? basecause of, In many case for Public Cloud, We always abstract a set for health checks , such as TargetGroup in AWS, instanceGroup in aliyun. We can map to such a collection of instance. If the service controller to check the targetgroup changes, the traffic pass-through pod is easier to implement. In most cases, the instance that in same group has an associated port for external services, and service controller set a target port if which not set in service yaml file. On the other handWe only lose some other performance of service controller. and Implementing targetgroup mapping to targetport depends on the cloud provider. What do you think? /sig cloud-provider /sig network
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.


Question: func init() { utilruntime.Must(logsapi.AddFeatureGates(utilfeature.DefaultMutableFeatureGate)) <ins> utilruntime.Must(features.AddFeatureGates(utilfeature.DefaultMutableFeatureGate)) </ins> } // Option configures a framework.Registry.

As per, , wire up metrics/slis for KCM
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/sig instrumentation


Question: import ( "fmt" <del> "net/http" "net/http/httptest" </del> "testing" <ins> "time" </ins> "github.com/GoogleCloudPlatform/kubernetes/pkg/api" <del> "github.com/GoogleCloudPlatform/kubernetes/pkg/api/testapi" </del> "github.com/GoogleCloudPlatform/kubernetes/pkg/client" <del> "github.com/GoogleCloudPlatform/kubernetes/pkg/cloudprovider/fake" "github.com/GoogleCloudPlatform/kubernetes/pkg/runtime" "github.com/GoogleCloudPlatform/kubernetes/pkg/util" </del> <ins> fake_cloud "github.com/GoogleCloudPlatform/kubernetes/pkg/cloudprovider/fake" </ins> ) <del> func newMinionList(count int) *api.MinionList { </del> <ins> func newMinion(name string) *api.Minion { return &api.Minion{ObjectMeta: api.ObjectMeta{Name: name}} } type FakeMinionHandler struct { client.Fake client.FakeMinions // Input: Hooks determine if request is valid or not CreateHook func(*FakeMinionHandler, *api.Minion) bool Existing   []*api.Minion // Output CreatedMinions []*api.Minion DeletedMinions []*api.Minion RequestCount   int } func (c *FakeMinionHandler) Minions() client.MinionInterface { return c } func (m *FakeMinionHandler) Create(minion *api.Minion) (*api.Minion, error) { defer func() { m.RequestCount++ }() if m.CreateHook == nil || m.CreateHook(m, minion) { m.CreatedMinions = append(m.CreatedMinions, minion) return minion, nil } else { return nil, fmt.Errorf("Create error.") } } func (m *FakeMinionHandler) List() (*api.MinionList, error) { defer func() { m.RequestCount++ }() </ins> minions := []api.Minion{} <del> for i := 0; i < count; i++ { minions = append(minions, api.Minion{ ObjectMeta: api.ObjectMeta{ Name: fmt.Sprintf("minion%d", i), }, }) </del> <ins> for i := 0; i < len(m.Existing); i++ { if !contains(m.Existing[i], m.DeletedMinions) { minions = append(minions, *m.Existing[i]) } </ins> } <del> return &api.MinionList{ Items: minions, </del> <ins> for i := 0; i < len(m.CreatedMinions); i++ { if !contains(m.Existing[i], m.DeletedMinions) { minions = append(minions, *m.CreatedMinions[i]) } </ins> } <ins> return &api.MinionList{Items: minions}, nil </ins> } <del> type serverResponse struct { statusCode int obj        interface{} </del> <ins> func (m *FakeMinionHandler) Delete(id string) error { m.DeletedMinions = append(m.DeletedMinions, newMinion(id)) m.RequestCount++ return nil </ins> } <del> func makeTestServer(t *testing.T, minionResponse serverResponse) (*httptest.Server, *util.FakeHandler) { fakeMinionHandler := util.FakeHandler{ StatusCode:   minionResponse.statusCode, ResponseBody: runtime.EncodeOrDie(testapi.Codec(), minionResponse.obj.(runtime.Object)), } mux := http.NewServeMux() mux.Handle("/api/"+testapi.Version()+"/minions", &fakeMinionHandler) mux.Handle("/api/"+testapi.Version()+"/minions/", &fakeMinionHandler) mux.HandleFunc("/", func(res http.ResponseWriter, req *http.Request) { t.Errorf("unexpected request: %v", req.RequestURI) res.WriteHeader(http.StatusNotFound) }) return httptest.NewServer(mux), &fakeMinionHandler </del> <ins> func TestSyncStaticCreateMinion(t *testing.T) { fakeMinionHandler := &FakeMinionHandler{ CreateHook: func(fake *FakeMinionHandler, minion *api.Minion) bool { return true }, } minionController := NewMinionController(nil, ".*", []string{"minion0"}, &api.NodeResources{}, fakeMinionHandler) if err := minionController.SyncStatic(time.Millisecond); err != nil { t.Errorf("unexpected error: %v", err) } if fakeMinionHandler.RequestCount != 1 { t.Errorf("Expected 1 call, but got %v.", fakeMinionHandler.RequestCount) } if len(fakeMinionHandler.CreatedMinions) != 1 { t.Errorf("expect only 1 minion created, got %v", len(fakeMinionHandler.CreatedMinions)) } if fakeMinionHandler.CreatedMinions[0].Name != "minion0" { t.Errorf("unexpect minion %v created", fakeMinionHandler.CreatedMinions[0].Name) } } func TestSyncStaticCreateMinionWithError(t *testing.T) { fakeMinionHandler := &FakeMinionHandler{ CreateHook: func(fake *FakeMinionHandler, minion *api.Minion) bool { if fake.RequestCount == 0 { return false } return true }, } minionController := NewMinionController(nil, ".*", []string{"minion0"}, &api.NodeResources{}, fakeMinionHandler) if err := minionController.SyncStatic(time.Millisecond); err != nil { t.Errorf("unexpected error: %v", err) } if fakeMinionHandler.RequestCount != 2 { t.Errorf("Expected 2 call, but got %v.", fakeMinionHandler.RequestCount) } if len(fakeMinionHandler.CreatedMinions) != 1 { t.Errorf("expect only 1 minion created, got %v", len(fakeMinionHandler.CreatedMinions)) } if fakeMinionHandler.CreatedMinions[0].Name != "minion0" { t.Errorf("unexpect minion %v created", fakeMinionHandler.CreatedMinions[0].Name) } </ins> } <del> func TestSyncCreateMinion(t *testing.T) { testServer, minionHandler := makeTestServer(t, serverResponse{http.StatusOK, newMinionList(1)}) defer testServer.Close() client := client.NewOrDie(&client.Config{Host: testServer.URL, Version: testapi.Version()}) </del> <ins> func TestSyncCloudCreateMinion(t *testing.T) { fakeMinionHandler := &FakeMinionHandler{ Existing: []*api.Minion{newMinion("minion0")}, } </ins> instances := []string{"minion0", "minion1"} fakeCloud := fake_cloud.FakeCloud{ Machines: instances, } <del> minionController := NewMinionController(&fakeCloud, ".*", nil, nil, client) if err := minionController.Sync(); err != nil { </del> <ins> minionController := NewMinionController(&fakeCloud, ".*", nil, nil, fakeMinionHandler) if err := minionController.SyncCloud(); err != nil { </ins> t.Errorf("unexpected error: %v", err) } <del> data := runtime.EncodeOrDie(testapi.Codec(), &api.Minion{ObjectMeta: api.ObjectMeta{Name: "minion1"}}) minionHandler.ValidateRequest(t, "/api/"+testapi.Version()+"/minions", "POST", &data) </del> <ins> if fakeMinionHandler.RequestCount != 2 { t.Errorf("Expected 2 call, but got %v.", fakeMinionHandler.RequestCount) } if len(fakeMinionHandler.CreatedMinions) != 1 { t.Errorf("expect only 1 minion created, got %v", len(fakeMinionHandler.CreatedMinions)) } if fakeMinionHandler.CreatedMinions[0].Name != "minion1" { t.Errorf("unexpect minion %v created", fakeMinionHandler.CreatedMinions[0].Name) } </ins> } <del> func TestSyncDeleteMinion(t *testing.T) { testServer, minionHandler := makeTestServer(t, serverResponse{http.StatusOK, newMinionList(2)}) defer testServer.Close() client := client.NewOrDie(&client.Config{Host: testServer.URL, Version: testapi.Version()}) </del> <ins> func TestSyncCloudDeleteMinion(t *testing.T) { fakeMinionHandler := &FakeMinionHandler{ Existing: []*api.Minion{newMinion("minion0"), newMinion("minion1")}, } </ins> instances := []string{"minion0"} fakeCloud := fake_cloud.FakeCloud{ Machines: instances, } <del> minionController := NewMinionController(&fakeCloud, ".*", nil, nil, client) if err := minionController.Sync(); err != nil { </del> <ins> minionController := NewMinionController(&fakeCloud, ".*", nil, nil, fakeMinionHandler) if err := minionController.SyncCloud(); err != nil { </ins> t.Errorf("unexpected error: %v", err) } <del> minionHandler.ValidateRequest(t, "/api/"+testapi.Version()+"/minions/minion1", "DELETE", nil) </del> <ins> if fakeMinionHandler.RequestCount != 2 { t.Errorf("Expected 2 call, but got %v.", fakeMinionHandler.RequestCount) } if len(fakeMinionHandler.DeletedMinions) != 1 { t.Errorf("expect only 1 minion deleted, got %v", len(fakeMinionHandler.DeletedMinions)) } if fakeMinionHandler.DeletedMinions[0].Name != "minion1" { t.Errorf("unexpect minion %v created", fakeMinionHandler.DeletedMinions[0].Name) } </ins> } <del> func TestSyncMinionRegexp(t *testing.T) { testServer, minionHandler := makeTestServer(t, serverResponse{http.StatusOK, newMinionList(1)}) defer testServer.Close() client := client.NewOrDie(&client.Config{Host: testServer.URL, Version: testapi.Version()}) </del> <ins> func TestSyncCloudRegexp(t *testing.T) { fakeMinionHandler := &FakeMinionHandler{ Existing: []*api.Minion{newMinion("minion0")}, } </ins> instances := []string{"minion0", "minion1", "node0"} fakeCloud := fake_cloud.FakeCloud{ Machines: instances, } <del> minionController := NewMinionController(&fakeCloud, "minion[0-9]+", nil, nil, client) if err := minionController.Sync(); err != nil { </del> <ins> minionController := NewMinionController(&fakeCloud, "minion[0-9]+", nil, nil, fakeMinionHandler) if err := minionController.SyncCloud(); err != nil { </ins> t.Errorf("unexpected error: %v", err) } <del> // Only minion1 is created. data := runtime.EncodeOrDie(testapi.Codec(), &api.Minion{ObjectMeta: api.ObjectMeta{Name: "minion1"}}) minionHandler.ValidateRequest(t, "/api/"+testapi.Version()+"/minions", "POST", &data) </del> <ins> if fakeMinionHandler.RequestCount != 2 { t.Errorf("Expected 2 call, but got %v.", fakeMinionHandler.RequestCount) } if len(fakeMinionHandler.CreatedMinions) != 1 { t.Errorf("expect only 1 minion created, got %v", len(fakeMinionHandler.CreatedMinions)) } if fakeMinionHandler.CreatedMinions[0].Name != "minion1" { t.Errorf("unexpect minion %v created", fakeMinionHandler.CreatedMinions[0].Name) } } func contains(minion *api.Minion, minions []*api.Minion) bool { for i := 0; i < len(minions); i++ { if minion.Name == minions[i].Name { return true } } return false </ins> }

gives but gives If minions were already in etcd from some previous launch of the apiserver/controller things work just fine no matter what the order. So to reproduce you have to clear everything from etcd before you try...
can you take a look at this? Probably just need to retry in background until etcd comes up?
Yep, there is no control loop for static machine config. Adding one should be trivial.
Blocked by Basically, apiserver returns inconsistent response.
I have observed a similar sort of thing, but what it appeared to be is more like the is not listening on it's socket at the time tries to connect to it and it never retries. So my work-around is to run before it like so: What I then observed is that the connection is indeed being refused at first attempt, but works on the second.
what do you see in the logs? I was seeing that it basically fails to connect to once and doesn't retry (although it said that it will try to retry).
should fix it, sorry for the trouble wondering where did you see that "it will try to retry"?
that wasn't the exact wording there... let me see if I can find out what exactly it said.
Ok, so what I'm seeing now is: I hope that PIDs would be a good enough proof to what order services have started up: I could provide some systemd graph dumps, but that would probably be too verbose, PIDs seem clearer. So really don't see how got //apisevercontroller-managerapiserver` didn't open the socket yet...


Question: opts.AllowInvalidLabelValueInSelector = hasInvalidLabelValueInAffinitySelector(oldPodSpec) // if old spec has invalid labelSelector in topologySpreadConstraint, we must allow it opts.AllowInvalidTopologySpreadConstraintLabelSelector = hasInvalidTopologySpreadConstraintLabelSelector(oldPodSpec) <ins> // if old spec has an invalid projected token volume path, we must allow it opts.AllowNonLocalProjectedTokenPath = hasNonLocalProjectedTokenPath(oldPodSpec) </ins> // if old spec has invalid sysctl with hostNet or hostIPC, we must allow it when update if oldPodSpec.SecurityContext != nil && len(oldPodSpec.SecurityContext.Sysctls) != 0 {

Originally posted by in
Hello, I'm new here. Can I try solving this? it seems like a good first issue
sure, context is the change is needed here: We have to tolerate existing pods / pod specs that fail this validation, so we should only do this additional validation on create or if the existing object passes this validation. To do this: an field to , initialize that field to false, and then conditionally override it to if is non-nil and has a projected token volume path that fails (will have to export the function to check this) , if is false, call the validation on the path:
Hi, I'd like to work on this if there is no one is proceeding on. /assign


Question: // Deletes the PVC/PV, and launches a pod with hostpath volume to remove the test directory. func cleanupLocalVolumeGCELocalSSD(config *localTestConfig, volume *localTestVolume) { By("Removing the test directory") <del> removeCmd := fmt.Sprintf("rm %s", volume.hostDir+"/"+testFile) </del> <ins> file := volume.hostDir + "/" + testFile removeCmd := fmt.Sprintf("if [ -f %s ]; then rm %s; fi", file, file) </ins> err := issueNodeCommand(config, removeCmd, volume.node) Expect(err).NotTo(HaveOccurred()) }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: Since 8/31, these tests have started flaking, and then started failing consistently. https://k8s- https://k8s- /assign
/kind failing-test
/milestone v1.12
Does this need cherry picked back to 1.11? Still have upgrade failures with this symptom, eg: https://k8s-
No, the localssd tests are not failing in the upgrade suite
/reopen I'm going to call this one a partial fix, as we are still seeing the "Set fsGroup for local volume should not set different fsGroups for two pods simultaneously" error on a whole bunch of test suites.
Reopening this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
can you link me to other suites where it is failing?
okay "a whole bunch" was exaggerated, I apologize: https://k8s- https://k8s-
The upgrade tests are not failing with "gce-localssd", which is what this issue is addressing, so is not related. The upgrade tests issues are being tracked in /close
Closing this issue. <detailsIn response to : The upgrade tests are not failing with "gce-localssd", which is what this issue is addressing, so is not related. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: Selector:        map[string]string{"bar": "baz"}, SessionAffinity: api.ServiceAffinityNone, Type:            api.ServiceTypeNodePort, <ins> ClusterIP:       "1.2.3.5", </ins> Ports: []api.ServicePort{{ NodePort:   30020, Port:       6502,

Which test(s) are flaking: TestServiceRegistryUpdateDryRun Reason for failure: This is flaking rarely enough that it is not caught by our CI jobs, which currently tolerate up to 2 unit test failures per run (!). With that toleration removed in , this flake has been seen (). Reproducible with the following steps: the bug in the test that casts the return value even when the create fails: logging on slow etcd writes because of etcd-io/etcd a test binary for the affected package: the affected test using the tool: failures are seen, look at the logs to see the details about the error: /sig network
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
heh this, was a funny one, we had 1/2^8 chances of using a duplicate, with 1/2^16 I've got 2000 runs without any failures
/assign /remove-triage unresolved


Question: } select { case d.resultCh <- send: <del> if send.Type == watch.Error { d.cancel() } </del> <ins> // propagated event successfully </ins> case <-ctx.Done(): <ins> // context timed out or was cancelled, stop the underlying watcher d.w.Stop() return </ins> } case <-ctx.Done(): <ins> // context timed out or was cancelled, stop the underlying watcher </ins> d.w.Stop() <del> close(d.resultCh) </del> return } } } func (d *decoratedWatcher) Stop() { <ins> // stop the underlying watcher d.w.Stop() // cancel our context </ins> d.cancel() }

Using the Fabric8 client, which defaults to using WebSockets for Watches, I established a Watch on Service resources. After the timeout (implicit or explicit) rather than the WebSocket seeing a close control frame or even an EOF it continued to be open and fulfill ping/pong requests - but no Service events are sent. The WebSocket would close. Create a WebSocket based Watch with the ListOptions timeoutSeconds set on the Service resource along with a steady production of Service events, such as create/delete of a dummy Service. Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2020-10-27T05:15:27Z", GoVersion:"go1.15.0", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2021-01-13T13:20:00Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"} The behavior has been seen in both local minikube and OpenShift distributions based upon Kubernetes 1.20. OS (e.g: ): Fedora 33
/sig api-machinery
Is this the same issue as ?
It could be related, but in that case the client is calling close. In this case we expect the server to initiate the close.
A couple of things were discussed when we triaged this issue: a) we need to involve the Fabric8 team (~java client?) b) We don't always default to Websocket, in fact we discussed discontinuing it c) But the code is still there, so it's allowed for now and we should support it. Not sure what to do next
a) no, it sounds like an apiserver problem, it should be reproducible with a go client in a test. b) If anything, we should standardize on websockets. Brower based clients depend on them. And when/if we get rid of SPDY, websockets are the replacement. This may be relevant to interests.

I didn't see an option to use websockets with the go client the impl is at Is there another code path or go client that will use a websocket for Watches?
Technically I didn't say our go client, just a go client. I just had in mind using the raw websockets library. must be testing the websocket watches somehow?
I adapted to It will simply keep running - with no events after the timeout. If you switch to configmap, it will terminate.
that's a terrific reproducer, thanks!
Yeah, we should absolutely be able to stick that in a unit / integration test somewhere, thanks!
opened in configmap watch works correctly, service watch hits timeout without the server closing the connection
I suspect the issue is in newDecoratedWatcher. Services implement a decorator and configmaps don't.
yup, the request context wasn't propagated to the decorated watcher
Service decorator was in 1.20 in Fix with unit and integration test in Will pick to 1.21 and 1.20 once merged
/triage accepted


Question: return deployment, newRS } <ins> func stopDeploymentOverlap(c *clientset.Clientset, oldC client.Interface, ns, deploymentName, overlapWith string) { stopDeploymentMaybeOverlap(c, oldC, ns, deploymentName, overlapWith) } </ins> func stopDeployment(c *clientset.Clientset, oldC client.Interface, ns, deploymentName string) { <ins> stopDeploymentMaybeOverlap(c, oldC, ns, deploymentName, "") } func stopDeploymentMaybeOverlap(c *clientset.Clientset, oldC client.Interface, ns, deploymentName, overlapWith string) { </ins> deployment, err := c.Extensions().Deployments(ns).Get(deploymentName) Expect(err).NotTo(HaveOccurred())

https://k8s- Failed: [] Deployment overlapping deployment should not fight with each other {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment overlapping deployment should not fight with each other {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment overlapping deployment should not fight with each other {Kubernetes e2e suite} Happened on a presubmit run in .
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] Deployment overlapping deployment should not fight with each other {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment overlapping deployment should not fight with each other {Kubernetes e2e suite}
Looking
Flakes happened when stopping overlapping deployments at clean up step.
You probably need to move the check before the deletionTimestamp check. Just make sure that observedGeneration is updated for the first case.
Because the deployments are overlapping, when deleting one deployment and determining if its RSes are deleted, there might be a race when the other deployment creates an RS
[FLAKE-PING] This flaky-test issue would love to have more attention.


Question: ) // Serial because the test restarts Kubelet <del> var _ = framework.KubeDescribe("Device Plugin [Feature:DevicePlugin]", func() { </del> <ins> var _ = framework.KubeDescribe("Device Plugin [Feature:DevicePlugin] [Serial]", func() { </ins> f := framework.NewDefaultFramework("device-plugin-errors") Context("DevicePlugin", func() {

http://k8s- looks like the job has multiple flaky tests :- /sig node /priority failing-test /kind bug /status approved-for-milestone cc any clue :smirk: ?


Question: ginkgo.It("should be able to preserve UDP traffic when server pod cycles for a LoadBalancer service on different nodes", func(ctx context.Context) { // requires cloud load-balancer support <del> e2eskipper.SkipUnlessProviderIs("gce", "gke", "aws", "azure") </del> <ins> e2eskipper.SkipUnlessProviderIs("gce", "gke", "azure") </ins> ns := f.Namespace.Name nodes, err := e2enode.GetBoundedReadySchedulableNodes(ctx, cs, 2) framework.ExpectNoError(err)

When running the e2e tests on AWS, the following two tests are failing: with the stated cause being: these tests were with Expected the tests to pass Run the loadbalancer test suite on AWS While investigating the root cause of these failures, i cam across this function in the cloud-provider-aws: This function checks to ensure that a specific annotation has been applied to the Service object to determine if it is an NLB. The test in question creates the Service object with no annotations placed upon it, this in turn causes the test to fail. This confirms the need for the user to add the annotation. It is a continuation of a discussion that started in the SIG Cloud Provider meeting from 15 Feb 2023. <detailsi don't have a live cluster for this anymore, but it was version 1.26.1 </details<detailsAWS </details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig network /sig cloud-provider
when exploring this issue, i noticed we have this function in a different load balancer test: that function appears to have some specific logic for adding/removing annotations depending on the cloud provider. i thought this might be a place we could also use for AWs and the new tests, but i'm not sure if it is appropriate.
the test should be as much agnostic as possible, theoretically we tried to remove all the checks for cloud providers but Adding annotations and supporting all permutations on each cloud provider will be hard to support in the long term. If AWS doesn't support it we just can skip them in AWS
/assign
thanks , agreed on all points
please go ahead and tag me


Question: // Wait all pods in waitSchedulingPods to be scheduled. wg.Wait() <del> // Ensure that all waitingPods in scheduler can be obtained from any profiles. for _, fwk := range scheduler.Profiles { actualPodNamesInWaitingPods := sets.NewString() fwk.IterateOverWaitingPods(func(pod framework.WaitingPod) { actualPodNamesInWaitingPods.Insert(pod.GetPod().Name) }) // Validate the name of pods in waitingPods matches expectations. if actualPodNamesInWaitingPods.Len() != len(tc.expectPodNamesInWaitingPods) || !actualPodNamesInWaitingPods.HasAll(tc.expectPodNamesInWaitingPods...) { t.Fatalf("Unexpected waitingPods in scheduler profile %s, expect: %#v, got: %#v", fwk.ProfileName(), tc.expectPodNamesInWaitingPods, actualPodNamesInWaitingPods.List()) </del> <ins> utiltesting.Eventually(tCtx, func(utiltesting.TContext) sets.Set[string] { // Ensure that all waitingPods in scheduler can be obtained from any profiles. actualPodNamesInWaitingPods := sets.New[string]() for _, fwk := range scheduler.Profiles { fwk.IterateOverWaitingPods(func(pod framework.WaitingPod) { actualPodNamesInWaitingPods.Insert(pod.GetPod().Name) }) </ins> } <del> } </del> <ins> return actualPodNamesInWaitingPods }).WithTimeout(permitTimeout).Should(gomega.Equal(sets.New(tc.expectPodNamesInWaitingPods...)), "unexpected waitingPods in scheduler profile") </ins> }) } }

/kind failing-test /kind flake First seen in an unrelated PR, but also occurs in periodic jobs. /sig scheduling
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign This feature is reverted once, and I fixed the flake, I'm cherry-pick that back again.


Question: } } Expect(errCount).To(Equal(0), "Found invalid samples") <del> Expect(found).To(BeTrue(), "PVC %s, Namespace %s not found for %s", pvcName, namespace, metricKeyName) </del> <ins> return found </ins> }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: What you expected to happen: Test should not flake /sig storage /cc
/assign
GitHub didn't allow me to assign the following users: vkamra. Note that only can be assigned. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
GitHub didn't allow me to assign the following users: vkamra. Note that only can be assigned. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
- You won't be able to assign this to me (not a kubernetes member). Am taking a look.
volumemetrics.go_ could/should have more logging, either calling or more calls to .
The test appears to fail on this metric: "kubeletvolumestatsused_bytes". So samplePVC != pvcName and/or sampleNS != namespace. It would be nice if the sample name and ns were also logged. Also something is weird here: the framework's is called before the local . Note in the log:
- will add a couple of Logf() statements to log the samples - you're right that will help. Once the volume is attached, volume stats are grabbed every interval. The thing I noticed in the logs is that we issue a delete for the pod at but the volume only got attached/bound successfully at The test assumes blocks till the pod is running and PVC is attached but maybe that's not a valid assumption?
Nevermind - That log message has 2 timestamps in it - the volume was provisioned at .
Pushed The working theory being that volume stats were not collected within the wait earlier being used.
did you figure out if I was correct in thinking that the framework was invoked before the local ? That seems very, very unlikely but that is how it looked in the log.
- I believe you're right that the framework is being invoked before the local one.

I observe the same in a test as well, so I don't think it's specific to this call. Based on the , nested blocks should be invoked inner-outer. It's not clear, however - whether the local is nested - or at the same level as the framework block when I look at


Question: import ( "fmt" <ins> "time" </ins> autoscalingv1 "k8s.io/kubernetes/pkg/apis/autoscaling/v1" "k8s.io/kubernetes/test/e2e/common"

Currently the time it takes for HPA and CA to scale up after load increase is not measured at all, and this can lead to regressions that will be missed. This issue is about adding an e2e test that will prevent such regressions.
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list .
Filip, can you reopen? I cannot do this myself, and this issue is clearly not closed yet.
Thanks Wojtek!


Question: } } <del> func (p *criStatsProvider) makePodStorageStats(s *statsapi.PodStats, rootFsInfo *cadvisorapiv2.FsInfo) { podNs := s.PodRef.Namespace podName := s.PodRef.Name podUID := types.UID(s.PodRef.UID) vstats, found := p.resourceAnalyzer.GetPodVolumeStats(podUID) if !found { return } logStats, err := p.hostStatsProvider.getPodLogStats(podNs, podName, podUID, rootFsInfo) if err != nil { klog.ErrorS(err, "Unable to fetch pod log stats", "pod", klog.KRef(podNs, podName)) // If people do in-place upgrade, there might be pods still using // the old log path. For those pods, no pod log stats is returned. // We should continue generating other stats in that case. // calcEphemeralStorage tolerants logStats == nil. } etcHostsStats, err := p.hostStatsProvider.getPodEtcHostsStats(podUID, rootFsInfo) if err != nil { klog.ErrorS(err, "Unable to fetch pod etc hosts stats", "pod", klog.KRef(podNs, podName)) } ephemeralStats := make([]statsapi.VolumeStats, len(vstats.EphemeralVolumes)) copy(ephemeralStats, vstats.EphemeralVolumes) s.VolumeStats = append(append([]statsapi.VolumeStats{}, vstats.EphemeralVolumes...), vstats.PersistentVolumes...) s.EphemeralStorage = calcEphemeralStorage(s.Containers, ephemeralStats, rootFsInfo, logStats, etcHostsStats, true) } </del> func (p *criStatsProvider) addPodNetworkStats( ps *statsapi.PodStats, podSandboxID string,

I have a cronjob that runs a Python script every 5 minutes. After the cronjob finishes running, I start seeing a bunch of log messages in my systemd journal like the following: Those directories in /var/log don't exist as they belong to old pods that have finished running. These messages keep increasing in volume as time passes (almost received a 1000 of those messages at one time, after which I had to reboot the node to fix the issue). It almost looks like the kubelet is not cleaning up the old pods for some reason. I can't see these old pods when I run , so I'm assuming their is some issue with the kubelet's cleanup process. kubelet (or cadvisor) shouldn't try to read logs for pods that don't exist. Setup a short lived cronjob running every 5 minutes and observe the logs (Kubernetes v1.22.4, CRI-O v1.22.1, Ubuntu 20.04) No response <details</detailsBaremetal cluster <details</details<detailskubeadm (baremetal cluster) </details<detailsCRI-O v1.22.1 </details<details</details
/sig node
/assign for triage
I have something similar on a fresh cluster I use for educational purposes using kubernetes 22.5 with cri-o. Here's an excerpt: This is from the master node. Nodes run Debian bulsseye in KVM provisioned with Vagrant. I have not yet tested it out with different versions. If I can assist in testing, I'd be happy to do so!
/triage accepted /priority important-longterm
I've produced the same issue on my clusters. I'm running Debian 11 on bare metal (homelab) and in VM (test). My clusters are installed using kubeadm, and I'm using multus and calico as CNI. The issue is triggered as soon as a pod is removed. I've triggered it by deleting a simple Pod, scaling down a Deployment, or running a CronJob. On the node where the pod was removed, the logs are then flooded by messages, at the rate of 1 line per pod removed every 10 seconds. I've seen it both on aregular worker nodes, and untainted master nodes.

I've tested under Vagrant with various K8S / CRI-O versions. With CRI-O , I never triggered the issue with Kubernetes : , and . However, with CRI-O = , all my tests triggered the issue : Kubernetes , CRI-O Kubernetes , CRI-O Kubernetes , CRI-O Let me know if I can help by running other tests or capturing specific logs.
Is this only issue with CRI-O? What about containerd? The reports above seem to all be on CRI-O? /cc
openshift seems to have hit this case a couple of times and digging into some bugs leads me to think it will be fixed with and kube 1.24. can you try one of the kube alpha releases? cri-o 1.23.1 should do, or you can try the main branch
it's possible it's unrelated, but is why I am making this guess
I'm used to installing a cluster using Debian packages, so it took me a few tries to have a working v1.24 cluster. I followed the "Without a package manager" to install from the binaries. I used Kubernetes and CRI-O . I'm not certain everything is working fine, but I can schedule a Pod / Deployment / Job, so I assume it's good enough to test this issue. And I still reproduce the issue as soon as I remove a Pod.
I'm also hitting the issue in my lab environment(Vagrant). Running Rocky Linux 8.5 Kubernetes 1.22.6 and CRI-O 1.22.2
ah hah! I found a fix (and found out why it's only being hit in cri-o and not cadvisor)
We're also having this issue on a cluster hosting Gitlab CI jobs, this is quite a trouble since many jobs are spawned and removed daily. Kubernetes version: v1.22.7 CRI-O version: 1.22.2 Is there any news about the fix ? I looked into 's PR, it seemed ok to me but I'm not really experienced in Go... Nevertheless I'd be happy to help
up
I can confirm on a bare metal cluster that I'm running on containerd that I am also seeing these log messages.
Observing similar issues on (k8s and crio ) and (k8s and crio ).

The strange thing is that one cluster just some nodes affected while the other, all of them. Below an overview of 30m occurrences of Cluster running version : Cluster running version : No idea where the discrepancy is coming from.
Tested with and on a fresh cluster bootstraped with kubeadm, all machines run on Debian 11. Ran a job 3 times and there's a log line for every execution, so it spams 3 log lines per kubelet loop.
It looks like cgroups are left behind. Removing them fixes the logspam (temporarily). Can use something like this to find all of them, depending on actual error in journal: Then iterate over the and remove them, while checking that running pods are not considered: It's quick and dirty, but does the job. I've deployed a garbage collector daemonset with the above and until this is properly addressed, I'm quite happy with the fix. The problem is, for a medium to large size cluster with pods coming and going, it's very easy to bust your elasticsearch storage and performance with this log spam.
Yesterday we tested with the newly released version of and the problem was still here. After reading answer I digged a little bit because we use systemd to manage cgroups and it seemed the error was on our side. I updated the kernel boot command in the grub file because in order to have cgroupsv2 unified behavior in systemd you have to set both options: and and we only had : Used the following command for the changes to take effect: And then rebooted the node. At first it seemed like this did the trick as I popped a job on the node I changed and the kubelet didn't spam the logfile anymore. As soon as I deleted the pod from the cluster the kubelet started spamming the logs again with: So this did not resolve the issue. This was tested on a node running , , and .
Looks like the cgroup slices not being cleaned up by systemd is an old issue:
would you mind giving a try to see if it helps?
while your fix does make sense, the root cause (which atm is somewhat unknown) is not really addressed. Something's leaking cgroup slices... and have both been closed off.
I am not sure if they're leaking as much as have yet to be cleaned up. Have those pods been deleted by the kubelet yet?

and by deleted, I mean fully removed (no traces if searched by ). The kubelet will clean up the pod scope when GC kicks in, but will keep them around until then to allow for forensics and inspection to be performed. Since your script uses to find the scopes, I am guessing the pod themselves have yet to be GC'd, and should still exist. Which is why I think that is the correct solution: we expect those scopes to be there, but shouldn't unnecessarily log an error if a pod that's stopped but not yet removed is found
to verify my suspicion, I would be interested in seeing for all the pods that are showing up in your script, as well as to cross reference pods to containers, if you wouldn't mind uploading that somewhere.
I'm sure they're not in crio anymore. I'm only using to ensure I'm never going to touch a scope that's reported under a pod still seen by crio. I'll scale down the DaemonSet on my test cluster, run some workload to generate the leaking pods and then provide you some log snippets and requested output from crictl.
I took one of my clusters that's suffering heavily on this issue (hosting gitlab CI jobs), reactivated the logs on it for a second, doesn't take long for the kubelet to log things like this: Trying to inspect the pod with returns: I tested with a few pods and the result is always the same. This seems logic as gives me the following: But the slices are still here tho: So in my opinion crictl return the right things, it's either the kubelet that doesn't update correctly the list of pods to watch or the systemd problem with the slices.
I've attached the logs and requested outputs here (sanitized hostnames). Let me know if you'd need anything else; for now I'll leave them around and not scale up the DS on the test cluster.
I've posted a quick gist with all the manifests I'm using to deploy a garbage collector daemon set. Nothing fancy, just a bit of patchwork to address the issues I was having.
Here's another really interesting bit of information. In my case, this issue seems to be only affecting VMs and not bare-metal nodes.

It's completely puzzling, but I have 12 bare-metal nodes and 1 VM node in one cluster (the VM is just for testing) and it manifests only on the VM node. As well, my test cluster is made up fully of VMs (ie no bare-metal) and there it happens across the board. The bare metal nodes where the VMs are running are the same are same hardware as the bare-metal nodes of my main cluster (just more NVMe drives). Edit: amended to clarify this is observed in my case running OKD.
Sorry but I have the same issue on bare-metal nodes (x86 ArchLinux kernel 5.17.5 with K8S 1.23.6/Crio 1.23.2)
The issue in title happens on bare metal nodes too.
I've got the same issue on bare-metal nodes(k8s v1.23.0/cri-o v1.22.3) and I'm not sure if it's for the same reason, but I have this problem as well.
I figured out the reason for the discrepancy. The VM node was using cgroups v2, while the bare-metal, older nodes, had the kernel parameter . Removing that and rebooting, triggers the problem.
Did some testing and I can confirm that switching from CRI-O to containerd has stopped these log messages from popping up all the time. For now, this seems like a reasonable solution, at least for me. Using Cgroups v2 Kubernetes Version: v1.23.7 OS: Ubuntu 22.04 Containerd Version: v1.6.4 Runc Version: 1.1.1 EDIT: Kubernetes Version was v1.23, not v1.24
Another option is one could switch to the CRI stats provider by setting kubelet's containerruntimeendpoint to instead of
Is this also an option for OKD/OCP? Is there any progress on a fix or should we look at the above workaround instead?
Out of curiosity, why does this work? /cc
This would work for OKD, but wouldn't be supported for OCP OOTB. should fix, but it needs a reviewer/approver... Fun kubernetes fact! The CRI-O team found performance issues with the CRI stats provider, so there's a hardcoded check to see if the socket is .

However, since is symlinked to , and the check is a string comparison, one can use and not trigger that toggle, turning on CRI stats provider. We're working on fixing the perf troubles with CRI stats KEP 2371
Using doesn't solve the problem on my environment, we're still seeing the same error logs. We're running: Flatcar 3033.2.4 Kubelet 1.22.4 CRI-O 1.22.5 cgroup v2 enabled.
I had this error on Kubernetes version 1.24, with CRI-O 1.24. Today I did a fresh install with Kubernetes 1.25 + CRI-O 1.25 (with the same machine types and OS versions) and the error went away. I tried deployment scale-down, pod deletion, but I no longer see the mentioned "Unable to fetch log stats" messages in the log. VirtualBox machines OS: RHEL 9 with kernel 5.14.0-70.22.1.el90.x8664 Kubernetes version: 1.25.0 CRI-O version: 1.25.0 Runc version 1.1.3 Using cgroups v2
We are running kubernetes 1.24.9 with cri-o 1.24.4 and runc 1.1.4 on Ubuntu 22.04 with cgroups v2 and experienced the same issue. For some time I was running a deployment with a dozen of pods and rotated them every 10 seconds to create a lot of garbage in cgroups. This caused rather slow increase in both mem & cpu usage and a lot of error messages like : Kubelet mem & cpu usage: ! ! After I stopped generating those pods the resource usage stopped. After running script from the CPU&mem usage dropped.


Question: } return migRollingUpdate(tmpl, nt) } <del> // migRollingUpdate starts a MIG rolling update, upgrading the nodes to a new // instance template named tmpl, and waits up to nt times the nubmer of nodes // for it to complete. func migRollingUpdate(tmpl string, nt time.Duration) error { By(fmt.Sprintf("starting the MIG rolling update to %s", tmpl)) id, err := migRollingUpdateStart(tmpl, nt) if err != nil { return fmt.Errorf("couldn't start the MIG rolling update: %v", err) } By(fmt.Sprintf("polling the MIG rolling update (%s) until it completes", id)) if err := migRollingUpdatePoll(id, nt); err != nil { return fmt.Errorf("err waiting until update completed: %v", err) } return nil } // migTemlate (GCE/GKE-only) returns the name of the MIG template that the // nodes of the cluster use. func migTemplate() (string, error) { var errLast error var templ string key := "instanceTemplate" // TODO(mbforbes): Refactor this to use cluster_upgrade.go:retryCmd(...) if wait.Poll(poll, singleCallTimeout, func() (bool, error) { // TODO(mbforbes): make this hit the compute API directly instead of // shelling out to gcloud. o, err := exec.Command("gcloud", "preview", "managed-instance-groups", fmt.Sprintf("--project=%s", testContext.CloudConfig.ProjectID), fmt.Sprintf("--zone=%s", testContext.CloudConfig.Zone), "describe", testContext.CloudConfig.NodeInstanceGroup).CombinedOutput() if err != nil { errLast = fmt.Errorf("gcloud preview managed-instance-groups describe call failed with err: %v", err) return false, nil } output := string(o) // The 'describe' call probably succeeded; parse the output and try to // find the line that looks like "instanceTemplate: url/to/<templ>" and // return <templ>. if val := parseKVLines(output, key); len(val) > 0 { url := strings.Split(val, "/") templ = url[len(url)-1] Logf("MIG group %s using template: %s", testContext.CloudConfig.NodeInstanceGroup, templ) return true, nil } errLast = fmt.Errorf("couldn't find %s in output to get MIG template. Output: %s", key, output) return false, nil }) != nil { return "", fmt.Errorf("migTemplate() failed with last error: %v", errLast) } return templ, nil } // migRollingUpdateStart (GCE/GKE-only) starts a MIG rolling update using templ // as the new template, waiting up to nt per node, and returns the ID of that // update. func migRollingUpdateStart(templ string, nt time.Duration) (string, error) { var errLast error var id string prefix, suffix := "Started [", "]." // TODO(mbforbes): Refactor this to use cluster_upgrade.go:retryCmd(...) if err := wait.Poll(poll, singleCallTimeout, func() (bool, error) { // TODO(mbforbes): make this hit the compute API directly instead of //                 shelling out to gcloud. // NOTE(mbforbes): If you are changing this gcloud command, update //                 cluster/gce/upgrade.sh to match this EXACTLY. o, err := exec.Command("gcloud", "preview", "rolling-updates", fmt.Sprintf("--project=%s", testContext.CloudConfig.ProjectID), fmt.Sprintf("--zone=%s", testContext.CloudConfig.Zone), "start", // Required args. fmt.Sprintf("--group=%s", testContext.CloudConfig.NodeInstanceGroup), fmt.Sprintf("--template=%s", templ), // Optional args to fine-tune behavior. fmt.Sprintf("--instance-startup-timeout=%ds", int(nt.Seconds())), // NOTE: We can speed up this process by increasing //       --max-num-concurrent-instances. fmt.Sprintf("--max-num-concurrent-instances=%d", 1), fmt.Sprintf("--max-num-failed-instances=%d", 0), fmt.Sprintf("--min-instance-update-time=%ds", 0)).CombinedOutput() if err != nil { errLast = fmt.Errorf("gcloud preview rolling-updates call failed with err: %v", err) return false, nil } output := string(o) // The 'start' call probably succeeded; parse the output and try to find // the line that looks like "Started [url/to/<id>]." and return <id>. for _, line := range strings.Split(output, "n") { // As a sanity check, ensure the line starts with prefix and ends // with suffix. if strings.Index(line, prefix) != 0 || strings.Index(line, suffix) != len(line)-len(suffix) { continue } url := strings.Split(strings.TrimSuffix(strings.TrimPrefix(line, prefix), suffix), "/") id = url[len(url)-1] Logf("Started MIG rolling update; ID: %s", id) return true, nil } errLast = fmt.Errorf("couldn't find line like '%s ... %s' in output to MIG rolling-update start. Output: %s", prefix, suffix, output) return false, nil }); err != nil { return "", fmt.Errorf("migRollingUpdateStart() failed with last error: %v", errLast) } return id, nil } // migRollingUpdatePoll (CKE/GKE-only) polls the progress of the MIG rolling // update with ID id until it is complete. It returns an error if this takes // longer than nt times the number of nodes. func migRollingUpdatePoll(id string, nt time.Duration) error { // Two keys and a val. status, progress, done := "status", "statusMessage", "ROLLED_OUT" start, timeout := time.Now(), nt*time.Duration(testContext.CloudConfig.NumNodes) var errLast error Logf("Waiting up to %v for MIG rolling update to complete.", timeout) // TODO(mbforbes): Refactor this to use cluster_upgrade.go:retryCmd(...) if wait.Poll(restartPoll, timeout, func() (bool, error) { o, err := exec.Command("gcloud", "preview", "rolling-updates", fmt.Sprintf("--project=%s", testContext.CloudConfig.ProjectID), fmt.Sprintf("--zone=%s", testContext.CloudConfig.Zone), "describe", id).CombinedOutput() if err != nil { errLast = fmt.Errorf("Error calling rolling-updates describe %s: %v", id, err) Logf("%v", errLast) return false, nil } output := string(o) // The 'describe' call probably succeeded; parse the output and try to // find the line that looks like "status: <status>" and see whether it's // done. Logf("Waiting for MIG rolling update: %s (%v elapsed)", parseKVLines(output, progress), time.Since(start)) if st := parseKVLines(output, status); st == done { return true, nil } return false, nil }) != nil { return fmt.Errorf("timeout waiting %v for MIG rolling update to complete. Last error: %v", timeout, errLast) } Logf("MIG rolling update complete after %v", time.Since(start)) return nil } // parseKVLines parses output that looks like lines containing "<key>: <val>" // and returns <val> if <key> is found. Otherwise, it returns the empty string. func parseKVLines(output, key string) string { delim := ":" key = key + delim for _, line := range strings.Split(output, "n") { pieces := strings.SplitAfterN(line, delim, 2) if len(pieces) != 2 { continue } k, v := pieces[0], pieces[1] if k == key { return strings.TrimSpace(v) } } return "" } </del>

, func restartNodes, assumes that MIG rolling update on GCE will cause node restart. However, there is a feature of MIG rolling update (currently broken) that skips update if the new instance template is the same as the old one. The feature is supposed to work quite soon, so the test will not test anything (reboot will not be performed). A workaround may be to create an identical instance template with a different name. By the way, the current name is misleading. It should be . cc
Hi Marek, Was this changed recently? When we tried this originally it worked. Now, I just tried doing a rolling update to the same template, and it immediately fails with status "PAUSED" and status message "Errors: unknown". The name was chosen to attempt to be future-provider-agnostic, though I agree is better.
Actually, I'm trying it now, and it definitely sometimes works (recreates the instances). If this functionality isn't expected to work (or isn't supported) we can switch this to the equivalent MIG recreate-nodes call. I'll do this.
I've heard that some two months ago MIG update would not recreate new nodes if the template was the same. Then they changed it and instances were recreated (this is when you wrote the test). I think it is considered a bug. And they are going to fix this bug quite soon. So in a week or two (or maybe more - I don't know their release cycle), instances will not be recreated. (sorry for incomprehensible description of this issue - I'm correcting it now).
, I think your error is not related to MIG update - currently instances should be recreated (i.e. the error in MIG rolling update is not fixed). The error may be fixed around Monday (but maybe later - even 2 weeks).
Ack—thanks for the extra detail! I'll just switch this test to and we can stop worrying about the timing. We'll switch this in GKE too. Thanks again, I really appreciate you filing this!
(And by switch I mean just turn off self-upgrades.)
is this v1?
The referenced PR was a refactoring and Github got a bit confused. This issue isn't closed.

I don't think we need a test to recreate the MIG before v1.0 (although if it gets done before the freeze I don't see why we wouldn't merge it).
I'm leaving the project, so reassigning to... Have fun!
/sig testing
Fixed forever ago. /close


Question: return nil, false, fmt.Errorf("Spec does not reference an AzureFile volume type") } <ins> func getAzureCloud(cloudProvider cloudprovider.Interface) (*azure.Cloud, error) { azure, ok := cloudProvider.(*azure.Cloud) if !ok || azure == nil { return nil, fmt.Errorf("Failed to get Azure Cloud Provider. GetCloudProvider returned %v instead", cloudProvider) } return azure, nil } func getStorageEndpointSuffix(cloudprovider cloudprovider.Interface) string { const publicCloudStorageEndpointSuffix = "core.windows.net" azure, err := getAzureCloud(cloudprovider) if err != nil { glog.Warningf("No Azure cloud provider found. Using the Azure public cloud endpoint: %s", publicCloudStorageEndpointSuffix) return publicCloudStorageEndpointSuffix } return azure.Environment.StorageEndpointSuffix } </ins>

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? NO Note: Please file issues for subcomponents under the appropriate repo What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): kubernetes, azure file, chinacloudapi Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): master Environment: Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Ubuntu 16.04 Kernel (e.g. ): Install tools: Others: What happened: We try to mount azure file to pod, and get the following error That is because azure china's storage DNS suffix is rather than What you expected to happen: Kubernetes should provide some method to let user specify storage dns suffix, so we can use Azure file volume How to reproduce it (as minimally and precisely as possible): Any Azure cluster in china can reproduce it. Anything else we need to know: kubernetes has hard coded in shows a relative issue
could you help confirm this?
One possible way is configuring DNS suffix via annotations of PV. If thinks this way okay, I'd like to create a PR to fix it.
azure file was implemented without using resource group or azure cloud provider. The endpoint can be found at One fix is to use azure cloud provider and choose the file service endpoint from azure environment as I did in blob service . Let me know if you can help.
I'm not familiar with azure cloud provider and volume codes. I'd like to try to dig into it to see if I can fix as the way you suggested.


Question: <ins> # Please keep kube-proxy configuration in-sync with: # cluster/addons/kube-proxy/kube-proxy-ds.yaml </ins> apiVersion: v1 kind: Pod metadata:

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: ci-kubernetes-e2e-gci-gce-alpha-features Which test(s) are failing: BeforeSuite Since when has it been failing: Failing since 7/18 The failure seems to have been introduced in is the only PR within that window that modifies any cluster configuration. This PR is related to Testgrid link: Reason for failure: kube-proxy is stuck in a pending state /milestone v1.16 /priority critical-urgent /kind failing-test /sig testing /sig cluster-lifecycle /cc
almost definitely related to the image change E0725 05:54:40. 1368 ] container start failed: ErrImagePull: rpc error: code = Unknown desc = Error response from daemon: manifest for not found
/assign
hmm, we should watch to ensure that actually fixed it /reopen
Reopened this issue. <detailsIn response to : hmm, we should watch to ensure that actually fixed it /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Cluster came up in the most recent run, however it looks like tests may fail (it's not done yet). That should be a new issue though. /close
Closing this issue. <detailsIn response to : Cluster came up in the most recent run, however it looks like tests may fail (it's not done yet). That should be a new issue though. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: ginkgo.By("creating a windows pod and waiting for it to be running") windowsPod = f.PodClient().CreateSync(windowsPod) <del> ginkgo.By("verifying pod external connectivity to the internet") ginkgo.By("checking connectivity to 8.8.8.8 53 (google.com) from Linux") assertConsistentConnectivity(f, linuxPod.ObjectMeta.Name, linuxOS, linuxCheck("8.8.8.8", 53)) ginkgo.By("checking connectivity to www.google.com from Windows") assertConsistentConnectivity(f, windowsPod.ObjectMeta.Name, windowsOS, windowsCheck("www.google.com")) </del> ginkgo.By("verifying pod internal connectivity to the cluster dataplane") ginkgo.By("checking connectivity from Linux to Windows")

We pinging to the original E2Es, however in airgapped environments, these tests fail for obvious reasons: In general, we need to parameterize this if possible. Running the or tests should pass in air-gapped environments, especially those that dont explicitly declare external connectivity. For example: Run the test suites inside of a kuberentes cluster which has no internet connection. pointing finger at :) <details</details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
some of these were recently removed by :) but some more to remove i think
They should have the corresponding tags so they can be filtered
and do we need to add a parameter in function as and replace from link. If not, what are we trying to achieve here ? I would like to collaborate.
independent test cases with their own feature tags as in
/assign
/sig testing windows network
/triage accepted


Question: # Otherwise, we want a completely empty directory. if [[ "${JENKINS_FORCE_GET_TARS:-}" =~ ^[yY]$ ]]; then rm -rf kubernetes* <del> elif [[ $(find . -not -path './.config*' | wc -l) != 1 ]]; then echo $PWD not empty, bailing! </del> <ins> # .config and its children are created by the gcloud call that we use to # get the GCE service account. # console-log.txt is created by Jenkins, but is usually not flushed out # this early in the script. elif [[ $(find . -not -path "./.config*" -not -name "console-log.txt"  | wc -l) != 1 ]]; then echo "${PWD} not empty, bailing!" </ins> find . exit 1 fi

I've already seen it a couple of time. The cluster just don't start with the following logs: The example failure:
+1 I saw this a few times yesterday, but it went away, so I chalked it up to fallout from the GCE outage, and didn't open an issue. If we're still seeing it today, we should definitely try to track down and fix.
Additional failures might be helpful to track this down. I can't see where the extra files came from in the Jenkins log, and the workspace has long since been destroyed and recreated. Created to help debug this.
FYI, I created a Jenkins Failure Cause for this yesterday, so it should be reasonably easy to spot from now on:
Another example from before the debug patch.
Another instance: - this has just happened - you should have more information now
Relevant log: This might be an awkward race condition where most of the time Jenkins hasn't written out to console- by the time this check goes through, but occasionally it has. If that's the case, it might be worth printing a warning instead of failing the job when the workspace isn't empty, or else ignoring console- in the find.
I'm fine with the fix. can you please do it?


Question: go func() { err := cmd.Run() if err != nil { <del> cmdErrorChan <- fmt.Errorf("%s Exited with status %v.  Output:n%v", cmd, err, *cmd.OutputBuffer) </del> <ins> cmdErrorChan <- fmt.Errorf("%s Failed with error "%v".  Command output:n%s", cmd, err, *cmd.OutputBuffer) </ins> } close(cmdErrorChan) }()

Looks like it maybe the ssh connect is getting killed
The relevant log snippet:
And another one, not just coreos.


Question: glog.Warning("Channel nodeCIDRUpdateChannel was unexpectedly closed") return } <del> if err := ca.updateCIDRAllocation(workItem); err != nil { if ca.canRetry(workItem) { time.AfterFunc(updateRetryTimeout, func() { </del> <ins> if err := ca.updateCIDRAllocation(workItem); err == nil { glog.V(3).Infof("Updated CIDR for %q", workItem) ca.removeNodeFromProcessing(workItem) } else { glog.Errorf("Error updating CIDR for %q: %v", workItem, err) if canRetry, timeout := ca.retryParams(workItem); canRetry { glog.V(2).Infof("Retrying update for %q after %v", workItem, timeout) time.AfterFunc(timeout, func() { </ins> // Requeue the failed node for update again. ca.nodeUpdateChannel <- workItem }) continue } <ins> glog.Errorf("Exceeded retry count for %q, dropping from queue", workItem) </ins> } <del> ca.removeNodeFromProcessing(workItem) </del> case <-stopChan: return }

/kind bug /sig network What happened: Networking tight loop causing GCP Cloud rate limiting to kick in. Causing tests to fail. Suspected call stack: Looks like it could be the root cause of these test failures here: https://k8s- These storage tests are hitting rate limit on the cloud api when trying to detach/delete disks. However, we have exponential backoff for all these functions. But the rate limiting from an external source is affecting the tests. See logs snippet: Note: These logs span exactly 1 minute. Try grepping "rateLimit" or "CIDRNotAvailable" -also note that the CIDRNotAvailable error messages are probably not verbose enough as they do not pinpoint what the actual error is, you can only associate them by correlation in timing and then looking at the code. Shorter snippet from above log file to demonstrate: since this is running on hundreds of nodes hitting this API call should involve some sort of exponential backoff as well as jitter. /cc /assign
/assign
/assign
/sig cloud-provider /sig gcp /milestone v1.12 (since is marked as v1.12)
may we have a priority on this since Code Freeze is next week? :)
Quick question, are we using the default API rate limits quota for scalability projects?
I will defer to for priority because he probably has the most context on how serious this is and what it could affect
Which rate limit are you referring to? (I don't remember us changing it any time) Btw - this is the project being used for that job -
I put up a patch for adding exponential retry
Gentle reminder that for this to be considered for 1.12 past Code Freeze, which is next week, it must have a status/critical-urgent label. I see this is being actively worked on


Question: if err != nil { return nil, err } <ins> if !m.checkPodActive(pod) { klog.ErrorS(nil, "pod deleted from activePods, skip to reAllocate", "podUID", podUID) continue } </ins> // This is a device plugin resource yet we don't have cached // resource state. This is likely due to a race during node // restart. We re-issue allocate request to cover this race.

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --Pod worker panic when trying to re-allocating the device plugin resource during the node restart. The issue is potentially caused by an un-matching pod id. The map will be later used in an , which caused panic. ( func by itself is not thread-safe) Stack trace: DeviceManager Allocate() should be able to handle such race safely. Customer reported issue after upgrading to 1.18+. This is the new behavior introduced since 1.18+ from . Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig node
/assign looking into the issue
I think your PR is related to this issue.
/unassign seems pretty relatable to PR cc
/triage accepted /priority critical-urgent
/remove-priority critical-urgent I don't think this is a release blocker at this point given the history reading through ; we can fix this in a cherry-pick.
Hi all, We are the GCP/GKE customer from which this issue originates. I can provide more information: it happened multiple times on our production cluster since we upgraded to 1.18 it happens in 2 scenarios: when GCP has a "host maintenance" event on our VMs with GPU (they don't support VM live migrations when GPU are involved), it forces a VM reboot: when multiple pods on that k8s node request a GPU prior to the reboot, we sometimes get the kubelet panic on node restart. when GKE upgrades our node-pools: new nodes (fresh VMs) get multiple scheduled pods requesting a GPU, sometimes leading to the kubelet panic. =this happens outside of our control The end result is a Pod scheduled on a node, but with no container ever created: our production workload is effectively broken until we manually delete the stuck pods. More details about our device usage: we use a of the each node has one GPU, and our device plugin declares N devices per real GPU; nvidia/cuda supports multi processing so it effectively shares the GPU among N pods/containers.

(our workload is memory-gpu bound, and uses <1/Nth of the gpu memory each, so no conflict) it worked great for almost 3 years now, with N=8
/cc
Now that we have confirmation that it occurs in live cluster I will take a closer look later this week.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale


Question: framework.ExpectEqual(foundCreatedEvent, false, "should not have found test event after deletion") }) <del> ginkgo.It("should delete a collection of events", func() { </del> <ins> /* Release : v1.19 Testname: Event, delete a collection Description: A set of events is created with a label selector which MUST be found when listed. The set of events is deleted and MUST NOT show up when listed by its label selector. */ framework.ConformanceIt("should delete a collection of events", func() { </ins> eventTestNames := []string{"test-event-1", "test-event-2", "test-event-3"} ginkgo.By("Create set of events")

[x] Original approval issue: [x] Two weeks soak end date: 6 July 2020 This issue is created to allow edit by /wip /hold According to this APIsnoop query, there are still an endpoint which is untested. a set of 3 Events with a static label within a Namespace. that all 3 Events with the label in the Namespaces are created. the set of Namespaced Events with a label via DeleteCollection. that all Events with the label have been deleted. Discover useragents: List endpoints hit by the test: Display endpoint coverage change: Two other endpoints tested here have tests in progress to conformance already. If a test with these calls gets merged, test coverage will go up by 1 points This test is also created with the goal of conformance promotion.
/sig testing /sig architecture /area conformance
During Conformance Meeting: Is there a way to write this test so it is resource agnostic? If we are just changing the resource... can be parameterize the resource-type
/assign
GitHub didn't allow me to assign the following users: heyste. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen PR Merge and soaking
Reopened this issue. <detailsIn response to : /reopen PR Merge and soaking Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: }, } <ins> pendingVolumePod := &v1.Pod{ Spec: v1.PodSpec{ Volumes: []v1.Volume{ { VolumeSource: v1.VolumeSource{ PersistentVolumeClaim: &v1.PersistentVolumeClaimVolumeSource{ ClaimName: "csi-4", }, }, }, }, }, } // Different pod than pendingVolumePod, but using the same unbound PVC unboundPVCPod2 := &v1.Pod{ Spec: v1.PodSpec{ Volumes: []v1.Volume{ { VolumeSource: v1.VolumeSource{ PersistentVolumeClaim: &v1.PersistentVolumeClaimVolumeSource{ ClaimName: "csi-4", }, }, }, }, }, } missingPVPod := &v1.Pod{ Spec: v1.PodSpec{ Volumes: []v1.Volume{ { VolumeSource: v1.VolumeSource{ PersistentVolumeClaim: &v1.PersistentVolumeClaimVolumeSource{ ClaimName: "csi-6", }, }, }, }, }, } noSCPVCPod := &v1.Pod{ Spec: v1.PodSpec{ Volumes: []v1.Volume{ { VolumeSource: v1.VolumeSource{ PersistentVolumeClaim: &v1.PersistentVolumeClaimVolumeSource{ ClaimName: "csi-5", }, }, }, }, }, } gceTwoVolPod := &v1.Pod{ Spec: v1.PodSpec{ Volumes: []v1.Volume{ { VolumeSource: v1.VolumeSource{ PersistentVolumeClaim: &v1.PersistentVolumeClaimVolumeSource{ ClaimName: "cs-gce-1", }, }, }, { VolumeSource: v1.VolumeSource{ PersistentVolumeClaim: &v1.PersistentVolumeClaimVolumeSource{ ClaimName: "csi-gce-2", }, }, }, }, }, } </ins> tests := []struct { newPod       *v1.Pod existingPods []*v1.Pod filterName   string maxVols      int <ins> driverNames  []string </ins> fits         bool test         string }{ { newPod:       oneVolPod, existingPods: []*v1.Pod{runningPod, twoVolPod}, <del> filterName:   "csi-ebs", </del> <ins> filterName:   "csi", </ins> maxVols:      4, <ins> driverNames:  []string{"ebs"}, </ins> fits:         true, test:         "fits when node capacity >= new pods CSI volume", }, { newPod:       oneVolPod, existingPods: []*v1.Pod{runningPod, twoVolPod}, <del> filterName:   "csi-ebs", </del> <ins> filterName:   "csi", </ins> maxVols:      2, <ins> driverNames:  []string{"ebs"}, </ins> fits:         false, test:         "doesn't when node capacity <= pods CSI volume", }, <ins> // should count pending PVCs { newPod:       oneVolPod, existingPods: []*v1.Pod{pendingVolumePod, twoVolPod}, filterName:   "csi", maxVols:      2, driverNames:  []string{"ebs"}, fits:         false, test:         "count pending PVCs towards capacity <= pods CSI volume", }, // two same pending PVCs should be counted as 1 { newPod:       oneVolPod, existingPods: []*v1.Pod{pendingVolumePod, unboundPVCPod2, twoVolPod}, filterName:   "csi", maxVols:      3, driverNames:  []string{"ebs"}, fits:         true, test:         "count multiple pending pvcs towards capacity >= pods CSI volume", }, // should count PVCs with invalid PV name but valid SC { newPod:       oneVolPod, existingPods: []*v1.Pod{missingPVPod, twoVolPod}, filterName:   "csi", maxVols:      2, driverNames:  []string{"ebs"}, fits:         false, test:         "should count PVCs with invalid PV name but valid SC", }, // don't count a volume which has storageclass missing { newPod:       oneVolPod, existingPods: []*v1.Pod{runningPod, noSCPVCPod}, filterName:   "csi", maxVols:      2, driverNames:  []string{"ebs"}, fits:         true, test:         "don't count pvcs with missing SC towards capacity", }, // don't count multiple volume types { newPod:       oneVolPod, existingPods: []*v1.Pod{gceTwoVolPod, twoVolPod}, filterName:   "csi", maxVols:      2, driverNames:  []string{"ebs", "gce"}, fits:         true, test:         "don't count pvcs with different type towards capacity", }, { newPod:       gceTwoVolPod, existingPods: []*v1.Pod{twoVolPod, runningPod}, filterName:   "csi", maxVols:      2, driverNames:  []string{"ebs", "gce"}, fits:         true, test:         "don't count pvcs with different type towards capacity", }, </ins> } defer utilfeaturetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, features.AttachVolumeLimit, true)() expectedFailureReasons := []PredicateFailureReason{ErrMaxVolumeCountExceeded} // running attachable predicate tests with feature gate and limit present on nodes for _, test := range tests { <del> node := getNodeWithPodAndVolumeLimits(test.existingPods, int64(test.maxVols), test.filterName) pred := NewCSIMaxVolumeLimitPredicate(getFakeCSIPVInfo("csi-ebs", "csi-ebs"), getFakeCSIPVCInfo("csi-ebs")) </del> <ins> node := getNodeWithPodAndVolumeLimits(test.existingPods, int64(test.maxVols), test.driverNames...) pred := NewCSIMaxVolumeLimitPredicate(getFakeCSIPVInfo(test.filterName, test.driverNames...), getFakeCSIPVCInfo(test.filterName, "csi-sc", test.driverNames...), getFakeCSIStorageClassInfo("csi-sc", test.driverNames[0])) </ins> fits, reasons, err := pred(test.newPod, GetPredicateMetadata(test.newPod, nil), node) if err != nil { t.Errorf("Using allocatable [%s]%s: unexpected error: %v", test.filterName, test.test, err)

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: Volume binding mode doesn't consider the volume attach limit per node. What you expected to happen: volume binding mode should have respected the attach limit, similar to binding mode. How to reproduce it (as minimally and precisely as possible): In CSI driver, configured the volume-attach limit to 1 per node on a 3 node cluster. In volume binding mode, only 3 pods with 3 PV/PVCs runs. In binding mode, all 5 pods runs with 5PV/PVCs on 3 node cluster. (Actually 4 pods scheduled on 1 node and 1 on other node, 3rd didn't have any pod scheduled) Anything else we need to know?: I was able to reproduced it with minimal node and configuration, and also hit this for stress case with 100 Volume limit and 500 pods running on the 5 mode cluster. Environment: Kubernetes version (use ): ~ kubectl version Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-10-30T21:39:38Z", GoVersion:"go1.11.1", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-12-03T20:56:12Z", GoVersion:"go1.11.2", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: On-prem linux server OS (e.g. from /etc/os-release): Centos 7Kernel (e.g. ): 3.10.0 Install tools: Others:
cc
/sig storage
/assign Currently the predicate PVCs that are not bound. It should lookup the storageclass to get the provisioner/driver name. This solution will require a StorageClass object is even if you don't use dynamic provisioning.

I had a thought that it wouldn't be so simple to count unbound PVCs of other pods since binding is still in progress, but I think it's fine actually. We only count other pods that have been assigned to the same node. Those pods are assigned to the node AND still unbound because of one of these reasons: It's actually been scheduled to the node and someone manually came in and unbound the PVC: unlikely and an unsupported sequence since we implemented StorageProtection The Pod was initially created with nodeName already set, bypassing the scheduler The Pod is going to be scheduled (it's been assumed), and it's currently going through binding. This is normal with delayed binding. Either way, the fact that the Pod's nodename is set to this node means that the pod should be counting towards the node's limit.
Hmm, so after opening the PR that fixes this, I realize that - from storageclass we don't really know if the volume we are evaluating is CSI volume or not. I think a similar problem exists for existing predicate though. Like AzureDisk predicate does not know if PVC it is over counting is of type AzureVolume or not. So, I suspect that those 4 hardcoded predicates are counting all delayed binding volumes.
if the storageclass.provisioner name is not found, can we just ignore it?
I think we can ignore it. The PR currently does that.


Question: "secure, no roots, custom dial": { TLSConfig:   &tls.Config{InsecureSkipVerify: false}, Dial:        d.DialContext, <del> ExpectError: "unknown authority", </del> <ins> ExpectError: "unknown authority|not trusted", </ins> }, "secure with roots, custom dial": { TLSConfig: &tls.Config{InsecureSkipVerify: false, RootCAs: roots},

unit tests since switching to go1.18 No response No response Weirdly, the test is passing in CI () and failing locally /sig api-machinery auth testing
seems related Jordan, are you using mac ?
yes, though the description in that issue seems to be not just test-only changes to the errors received from invalid certificates (which is what I saw), but actual certificates that previously validated
marking as release blocker until it is determined that go1.18 is not rejecting valid certificates
this needs an assignee and a reproducer we can file as an upstream issue for golang to ask why previously valid certificates now fail validation
marking critical until root-caused and determined whether this is prod-impacting, since (in my opinion) this should block rc0
On linux it works fine: on MacOS, it fails:
Looks like the change in golang that broke the scenario was:
if the only impact is a different failure message on an already-invalid certificate, I don't care and we can update the test to accept either message is what makes me wonder what previously-accepted certificates are now failing
reproducer test case here. this one has no issues under go 1.17 and fails with go 1.18 (stripped down version of ) : :
the new error message is minted here: i suspect it is related to: So essentially this seems to be something that may hit in our case. right?
yeah, if this is scoped to just darwin clients, and only ones using system roots, that greatly narrows the impact
Asked on gophers slack -
xref:
opened to tolerate the additional errors discussion in indicates the tightened validation is very bounded (darwin-only, so clients-only, system roots only), and certificates issued by public CAs would be expected to pass this validation dropping release-blocking label and priority to important-soon, but should still make it into 1.24 to make unit tests pass locally