Question: func getRESTMappings(mapper meta.RESTMapper, pruneResources *[]pruneResource) (namespaced, nonNamespaced []*meta.RESTMapping, err error) { if len(*pruneResources) == 0 { <del> // default whitelist // TODO: need to handle the older api versions - e.g. v1beta1 jobs. Github issue: #35991 </del> <ins> // default allowlist </ins> *pruneResources = []pruneResource{ {"", "v1", "ConfigMap", true}, {"", "v1", "Endpoints", true},

What happened: Sweeping uses of extensions/v1beta1 in preparation for , the addon manager stood out as using extensions/v1beta1 Ingress by default: What you expected to happen: All components shipped in 1.21 must use Ingress in order to work properly against a 1.22 API server. /sig networking cc per cluster/addons/addon-manager/OWNERS cc I'm not sure what the addon manager does if given a prune list of Ingress objects while still using a manifest containing extensions/v1 Ingress objects... this needs to be tested. /milestone v1.21 /priority important-soon /lifecycle frozen
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : What happened: cc /priority important-soon /lifecycle frozen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig network
/triage accepted
how "urgent" is this? Can wait until next tuesday? Otherwise I can work on the tests this weekend :)
important, not urgent. it should happen early in the 1.21 cycle, but does not require work out of business hours
Hello! I am from the Bug Triage team! This issue has not been updated for a few weeks, so I'd like to check on the status. Code freeze is starting March 9th, 2021 (about 1 week from now). We want to ensure that each PR has a chance to be merged before code freeze. As the Issue is tagged for 1.21, is it still planned for this release?
we're waiting to the PR to get merged :) I guess it's important for the v1.21 release, for the sake of extensions apiGroup deprecation :) Will ping reviewers on the PR


Question: } func (p *singleNumaNodePolicy) CanAdmitPodResult(hint *TopologyHint) lifecycle.PodAdmitResult { <del> if !hint.Preferred || hint.NUMANodeAffinity.Count() > 1 { </del> <ins> if !hint.Preferred { </ins> return lifecycle.PodAdmitResult{ Admit:   false, Reason:  "Topology Affinity Error",

What happened: With the latest , which aims to resolve the issue of deploying pods with single-numa-node policy, it seems to have resulted in failure to deploy pods if the pod spec does not have resources and limits constraints. This is resulting in Kube-dns and other pods which do not have resource constraints mentioned fail to deploy due to Topology Affinity Error. kube-system kube-dns--22dks 0/3 Topology Affinity Error 0 What you expected to happen: Ideally, if there are no resource constraints mentioned we will still need the pod to be deployed. We cannot have Kube-dns failing to deploy due to Topology Affinity issue. How to reproduce it (as minimally and precisely as possible): With the latest fix, enabling single-numa-node policy, Kube-dns errors out on the node. One can just try to deploy simple pod without resource constraints in pod spec. Anything else we need to know?: This happens only when policy is single-numa-node. Pods do not fail to deploy for best-case policy. Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-10-02T17:01:15Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"17+", GitVersion:"v1.17.0-alpha.1.338+-dirty", GitCommit:"", GitTreeState:"dirty", BuildDate:"2019-10-11T10:39:19Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: Baremetal x86 server with 2 NUMA nodes and 32 cpus. OS (e.g: ): Fedora 28 Kernel (e.g. ): 5.0.13-100.fc28.x86_64
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc
/assign
thanks for alerting us to the issue.

Can you try the PR linked?
My guess is that this issue was introduced after landed. Before that, best-effort pods weren't even being considered by the TopologyManager, and the logic for admitting vs. not admitting was skipped. Which means that we need this fix for this on master, but do not need it on the 1.16 branch.
can you verify if it works as expected in the release-1.16 branch. This would verify my claim that it broke with the introduction of not the fix from . The patch for landed just a few hours after , so I could see how its hard to know which one caused the new issue.
PTAL
, That's right. This fix is needed for the master branch and not 1.16. 1.16 works well without .
/cc for visibility. This issue appeared as a result of
Thanks for the ping. So is the proposed fix to always admit best-effort pods in the topology manager?
This only affects the single-numa-node policy because of the "hack" we currently have in place for it: The right long term fix is to address this as part of: However, if we want to put a short term fix in place to "unbreak" master, has a PR linked above that will fix the immediate problem.


Question: w.Write(output) })) defer server.Close() <del> client := NewDiscoveryClientForConfigOrDie(&restclient.Config{Host: server.URL}) </del> for _, test := range tests { <ins> client := NewDiscoveryClientForConfigOrDie(&restclient.Config{Host: server.URL}) </ins> got, err := client.ServerResourcesForGroupVersion(test.request) if test.expectErr { if err == nil {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: With client-go v0.17.0, I observed many "Throttling request" when DiscoveryClient is used to get API group resources from API server. This didn't happen with client-go kubernetes-1.16.0. It looks like originally this kind of discovery requests are not throttled because no rate limiter provided and QPS == 0. But with rate limiter is always set if nil rate limiter is provided. is particularly suspicious because it is always true because DefaultQPS is nonzero. What you expected to happen: Discovery requests should not be throttled as before. How to reproduce it (as minimally and precisely as possible): Create DiscoveryClient and then call with it. Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig api-machinery
The NewTokenBucketRateLimiter was via Always negotiate a decoder using ClientNegotiator /cc
To be clear, that was fixing a bug where the documented defaults for rate limiting were not taking effect. I don't think we should completely disable throttling until more of the server-side priority work is completed, but discovery is expected to be bursty, so I could see defaulting to a different amount in the discovery client constructor, similar to
Yeah, everyone must be rate limited. I think we could give discovery a big burst value.
opened


Question: package cache import ( <del> "k8s.io/api/core/v1" </del> <ins> v1 "k8s.io/api/core/v1" </ins> framework "k8s.io/kubernetes/pkg/scheduler/framework/v1alpha1" )

/sig scheduling What happened: reverted an old behavior to ensure Pod information is not lost when Nodes are recreated without its Pods getting recreated. A ghost Pod is kep in The problem is that we rely in this map for determining deleted Nodes when updating the snapshot in . The good thing is that the nodeinfo's are cloned into the snapshot, so we shouldn't have nil pointer accesses. But we might lose a few cycles trying to schedule on nodes that are deleted How to reproduce it (as minimally and precisely as possible): a node with a few pods. node without deleting pods. Result: The snapshot still has the node. Environment: Kubernetes version (use ): 1.18+
/assign
It actually produces the error: which would cause failed scheduling cycles (the map is not recreated, only the lists).


Question: s.Authorization.RemoteKubeConfigFileOptional = true s.Authorization.AlwaysAllowPaths = []string{"/healthz"} <del> s.SecureServing.ServerCert.CertDirectory = "/var/run/kubernetes" </del> <ins> // Set the PairName but leave certificate directory blank to generate in-memory by default s.SecureServing.ServerCert.CertDirectory = "" </ins> s.SecureServing.ServerCert.PairName = "cloud-controller-manager" s.SecureServing.BindPort = ports.CloudControllerManagerPort

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --BUG REPORT: /kind bug What happened: Kubernetes distro testing v1.12.0-rc.2 shows in setups with self-hosted control planes (i.e. apiserver, controller-manager, scheduler run as pods) that previously worked on v1.11.3 and prior, the kube-controller-manager now crash loops. Notably, the kube-controller-manager is run with: which has been fine through to v1.11.3. Full . What you expected to happen: Controller manager may not assume it has root. Or should clarify if an additional mount is now required. May have been introduced in cc How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): v1.12.0-rc.2 Cloud provider or hardware configuration: GCE, DigitalOcean, AWS OS (e.g. from /etc/os-release): Container Linux 1855.4.0 Kernel (e.g. ): 4.14.67-coreos Install tools: Typhoon
Seems like distros like Typhoon, Tectonic, and Tectonic/Openshift may work around this explicitly passing some other path to kube-controller-manager's or creating an emptyDir mount for . Need to understand the expectations around these certs (multiple pods, restarts, etc.) A release note seems needed to cover the new requirements. Appears this was also noticed by juju in
/sig release
cc
WDYT? Can we get folks to write a note to cover this problem in known issues?
I've an mount to address that crashloop, which seems to work preliminary (). was able to proceed with self-signed cert creation and hit a new bug I've filed separately EDIT: Perhaps a suitable release note for this change is:
the cert should not need to be persisted to disk.

I don't think adding a writeable directory requirement just for a self-signed cert is a very good idea /assign
Possible fixes enable by default if no cert is provided, rather than self-signing the self-sign cert in memory only, no disk persistence needed 't enable by default at all 2 would be ideal but is likely too big a change at this point. 3 would be unfortunate. 1 could be a good compromise if the code change is contained.
It's up to the SIG if this is a 1.12.0 blocker, or a Known Issue and 1.12.1 material. I'm inclined to think the latter.
thanks for the clarifications. I would prefer to avoid my current workaround of giving it a writeable location so those all sound fine to me. is there a full story around these controller-manager changes somewhere? I could mint actual certs, mount from a secret, update clients. If there's merit. Like, how big of a deal is it that this is now always enabled vs before? I'll probably read through the various diffs after vacation.
If this is going to get some text in release notes Known Issues (seems like it should), somebody needs to draft the text and make a suggestion of it in ASAP.
/assign
Do we want to fix this for 1.12.1 (on friday)?
Self signing in memory means you get a different cert each time it starts. It's probably better than nothing but it's not how things are supposed to work (it's too inconvenient to manually verify the cert out of band in places users should trust it). OTOH if you want to do that, maybe you'll provide a writable directory in the first place, or even a real cert. So I guess in-memory is probably OK.
Second thought: is a terrible default for the cert path. It should have the word "certs" in the path somewhere. If folks (inadvisably) run apiserver and controller manager on the same system, they should at least not both try to use certs from the same place, or worse, write them there. I suggest as the default.
What components are now verifying these certs or supposed to be? I'd expect something to be failing since random certs are generated each time the pod restarts, but don't see any impact (implying this isn't yet improving security).

Seems like controller-manager just an HTTPS endpoint, but the HTTP one is still what's used right?
I guess that aligns with the comment
The autogenerated certs only have SANs for localhost and 127.0.0.1.
The auto generated certs are self signed, right? If so, and callers won't be able to validate the signer, which SANs they have don't really matter.
Yeah, they're self-signed. So looks like the 1.12 controller-manager serves on HTTP/ (always on) and HTTPS/ (unless --secure-port=0). I guess the point of the original change was so some client could now choose to use the HTTPS endpoint with insecureSkipVerify. Main reason I was interested in the SAN is to get some clue about how a client would be expected to access these controller-manager pods (for contrast, apiserver SAN has DNS:kubernetes and an internal record that always resolves to a kube-apiserve pod).
is this updated for the openstack cloud provider too?
This issue wasn't related to any particular cloud provider. The cloud provider / hardware config field is just a typical part of filing an issue in case it had been related.
I have a writeable (I've tried other writeable directories too.) I'm encountering this: Should I post elsewhere in a new thread, or continue here?


Question: tsInfo: storageframework.TestSuiteInfo{ Name:         "read-write-once-pod", TestPatterns: patterns, <del> FeatureTag:   "[Feature:ReadWriteOncePod]", </del> }, } }

For 1.27 we should graduate the ReadWriteOncePod feature to beta. This is needed to rollout the feature to GA. In beta this feature will support preemption of pods using PVCs. See:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage /sig scheduling


Question: } wg.Wait() <del> // Expectations have been surpassed podExp, exists, err := e.GetExpectations(rcKey) assert.NoError(t, err, "Could not get expectations for rc, exists %v and err %v", exists, err) assert.True(t, exists, "Could not get expectations for rc, exists %v and err %v", exists, err) add, del := podExp.GetExpectations() assert.Equal(t, int64(-1), add, "Unexpected pod expectations %#v", podExp) assert.Equal(t, int64(-1), del, "Unexpected pod expectations %#v", podExp) assert.True(t, e.SatisfiedExpectations(logger, rcKey), "Expectations are met but the rc will not sync") // Next round of rc sync, old expectations are cleared e.SetExpectations(logger, rcKey, 1, 2) podExp, exists, err = e.GetExpectations(rcKey) assert.NoError(t, err, "Could not get expectations for rc, exists %v and err %v", exists, err) assert.True(t, exists, "Could not get expectations for rc, exists %v and err %v", exists, err) add, del = podExp.GetExpectations() assert.Equal(t, int64(1), add, "Unexpected pod expectations %#v", podExp) assert.Equal(t, int64(2), del, "Unexpected pod expectations %#v", podExp) // Expectations have expired because of ttl fakeClock.Step(ttl + 1) assert.True(t, e.SatisfiedExpectations(logger, rcKey), "Expectations should have expired but didn't") </del> <ins> tests := []struct { name                      string expectationsToSet         []int expireExpectations        bool wantPodExpectations       []int64 wantExpectationsSatisfied bool }{ { name:                      "Expectations have been surpassed", expireExpectations:        false, wantPodExpectations:       []int64{int64(-1), int64(-1)}, wantExpectationsSatisfied: true, }, { name:                      "Old expectations are cleared because of ttl", expectationsToSet:         []int{1, 2}, expireExpectations:        true, wantPodExpectations:       []int64{int64(1), int64(2)}, wantExpectationsSatisfied: false, }, } for _, test := range tests { t.Run(test.name, func(t *testing.T) { if len(test.expectationsToSet) > 0 { e.SetExpectations(logger, rcKey, test.expectationsToSet[0], test.expectationsToSet[1]) } podExp, exists, err := e.GetExpectations(rcKey) assert.NoError(t, err, "Could not get expectations for rc, exists %v and err %v", exists, err) assert.True(t, exists, "Could not get expectations for rc, exists %v and err %v", exists, err) add, del := podExp.GetExpectations() assert.Equal(t, test.wantPodExpectations[0], add, "Unexpected pod expectations %#v", podExp) assert.Equal(t, test.wantPodExpectations[1], del, "Unexpected pod expectations %#v", podExp) assert.Equal(t, test.wantExpectationsSatisfied, e.SatisfiedExpectations(logger, rcKey), "Expectations are met but the rc will not sync") if test.expireExpectations { fakeClock.Step(ttl + 1) assert.True(t, e.SatisfiedExpectations(logger, rcKey), "Expectations should have expired but didn't") } }) } </ins> } func TestUIDExpectations(t *testing.T) { logger, _ := ktesting.NewTestContext(t) uidExp := NewUIDTrackingControllerExpectations(NewControllerExpectations()) <del> rcList := []*v1.ReplicationController{ newReplicationController(2), newReplicationController(1), newReplicationController(0), newReplicationController(5), </del> <ins> type test struct { name        string numReplicas int } shuffleTests := func(tests []test) { for i := range tests { j := rand.Intn(i + 1) tests[i], tests[j] = tests[j], tests[i] } </ins> } <del> rcToPods := map[string][]string{} rcKeys := []string{} for i := range rcList { rc := rcList[i] rcName := fmt.Sprintf("rc-%v", i) </del> <ins> getRcDataFrom := func(test test) (string, []string) { rc := newReplicationController(test.numReplicas) rcName := fmt.Sprintf("rc-%v", test.numReplicas) </ins> rc.Name = rcName rc.Spec.Selector[rcName] = rcName podList := newPodList(nil, 5, v1.PodRunning, rc) rcKey, err := KeyFunc(rc) if err != nil { t.Fatalf("Couldn't get key for object %#v: %v", rc, err) } <del> rcKeys = append(rcKeys, rcKey) </del> rcPodNames := []string{} for i := range podList.Items { p := &podList.Items[i] p.Name = fmt.Sprintf("%v-%v", p.Name, rc.Name) rcPodNames = append(rcPodNames, PodKey(p)) } <del> rcToPods[rcKey] = rcPodNames </del> uidExp.ExpectDeletions(logger, rcKey, rcPodNames) <ins> return rcKey, rcPodNames </ins> } <del> for i := range rcKeys { j := rand.Intn(i + 1) rcKeys[i], rcKeys[j] = rcKeys[j], rcKeys[i] </del> <ins> tests := []test{ {name: "Replication controller with 2 replicas", numReplicas: 2}, {name: "Replication controller with 1 replica", numReplicas: 1}, {name: "Replication controller with no replicas", numReplicas: 0}, {name: "Replication controller with 5 replicas", numReplicas: 5}, </ins> } <del> for _, rcKey := range rcKeys { assert.False(t, uidExp.SatisfiedExpectations(logger, rcKey), "Controller %v satisfied expectations before deletion", rcKey) for _, p := range rcToPods[rcKey] { uidExp.DeletionObserved(logger, rcKey, p) } </del> <ins> shuffleTests(tests) for _, test := range tests { t.Run(test.name, func(t *testing.T) { </ins> <del> assert.True(t, uidExp.SatisfiedExpectations(logger, rcKey), "Controller %v didn't satisfy expectations after deletion", rcKey) </del> <ins> rcKey, rcPodNames := getRcDataFrom(test) assert.False(t, uidExp.SatisfiedExpectations(logger, rcKey), "Controller %v satisfied expectations before deletion", rcKey) </ins> <del> uidExp.DeleteExpectations(logger, rcKey) </del> <ins> for _, p := range rcPodNames { uidExp.DeletionObserved(logger, rcKey, p) } </ins> <del> assert.Nil(t, uidExp.GetUIDs(rcKey), "Failed to delete uid expectations for %v", rcKey) } } </del> <ins> assert.True(t, uidExp.SatisfiedExpectations(logger, rcKey), "Controller %v didn't satisfy expectations after deletion", rcKey) </ins> <del> func TestCreatePods(t *testing.T) { ns := metav1.NamespaceDefault body := runtime.EncodeOrDie(clientscheme.Codecs.LegacyCodec(v1.SchemeGroupVersion), &v1.Pod{ObjectMeta: metav1.ObjectMeta{Name: "empty_pod"}}) fakeHandler := utiltesting.FakeHandler{ StatusCode:   200, ResponseBody: string(body), } testServer := httptest.NewServer(&fakeHandler) defer testServer.Close() clientset := clientset.NewForConfigOrDie(&restclient.Config{Host: testServer.URL, ContentConfig: restclient.ContentConfig{GroupVersion: &schema.GroupVersion{Group: "", Version: "v1"}}}) </del> <ins> uidExp.DeleteExpectations(logger, rcKey) </ins> <del> podControl := RealPodControl{ KubeClient: clientset, Recorder:   &record.FakeRecorder{}, </del> <ins> assert.Nil(t, uidExp.GetUIDs(rcKey), "Failed to delete uid expectations for %v", rcKey) }) </ins> } <ins> } func TestCreatePodsWithGenerateName(t *testing.T) { ns := metav1.NamespaceDefault generateName := "hello-" </ins> controllerSpec := newReplicationController(1) controllerRef := metav1.NewControllerRef(controllerSpec, v1.SchemeGroupVersion.WithKind("ReplicationController")) <del> // Make sure createReplica sends a POST to the apiserver with a pod from the controllers pod template err := podControl.CreatePods(context.TODO(), ns, controllerSpec.Spec.Template, controllerSpec, controllerRef) assert.NoError(t, err, "unexpected error: %v", err) expectedPod := v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Labels:       controllerSpec.Spec.Template.Labels, GenerateName: fmt.Sprintf("%s-", controllerSpec.Name), }, Spec: controllerSpec.Spec.Template.Spec, </del> <ins> type test struct { name            string podCreationFunc func(podControl RealPodControl) error wantPod         *v1.Pod </ins> } <del> fakeHandler.ValidateRequest(t, "/api/v1/namespaces/default/pods", "POST", nil) var actualPod = &v1.Pod{} err = json.Unmarshal([]byte(fakeHandler.RequestBody), actualPod) assert.NoError(t, err, "unexpected error: %v", err) assert.True(t, apiequality.Semantic.DeepDerivative(&expectedPod, actualPod), "Body: %s", fakeHandler.RequestBody) } func TestCreatePodsWithGenerateName(t *testing.T) { ns := metav1.NamespaceDefault body := runtime.EncodeOrDie(clientscheme.Codecs.LegacyCodec(v1.SchemeGroupVersion), &v1.Pod{ObjectMeta: metav1.ObjectMeta{Name: "empty_pod"}}) fakeHandler := utiltesting.FakeHandler{ StatusCode:   200, ResponseBody: string(body), </del> <ins> var tests = []test{ { name: "Create pod", podCreationFunc: func(podControl RealPodControl) error { return podControl.CreatePods(context.TODO(), ns, controllerSpec.Spec.Template, controllerSpec, controllerRef) }, wantPod: &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Labels:       controllerSpec.Spec.Template.Labels, GenerateName: fmt.Sprintf("%s-", controllerSpec.Name), }, Spec: controllerSpec.Spec.Template.Spec, }, }, { name: "Create pod with generate name", podCreationFunc: func(podControl RealPodControl) error { // Make sure createReplica sends a POST to the apiserver with a pod from the controllers pod template return podControl.CreatePodsWithGenerateName(context.TODO(), ns, controllerSpec.Spec.Template, controllerSpec, controllerRef, generateName) }, wantPod: &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Labels:          controllerSpec.Spec.Template.Labels, GenerateName:    generateName, OwnerReferences: []metav1.OwnerReference{*controllerRef}, }, Spec: controllerSpec.Spec.Template.Spec, }, }, </ins> } <del> testServer := httptest.NewServer(&fakeHandler) defer testServer.Close() clientset := clientset.NewForConfigOrDie(&restclient.Config{Host: testServer.URL, ContentConfig: restclient.ContentConfig{GroupVersion: &schema.GroupVersion{Group: "", Version: "v1"}}}) podControl := RealPodControl{ KubeClient: clientset, Recorder:   &record.FakeRecorder{}, } </del> <ins> for _, test := range tests { t.Run(test.name, func(t *testing.T) { body := runtime.EncodeOrDie(clientscheme.Codecs.LegacyCodec(v1.SchemeGroupVersion), &v1.Pod{ObjectMeta: metav1.ObjectMeta{Name: "empty_pod"}}) fakeHandler := utiltesting.FakeHandler{ StatusCode:   200, ResponseBody: string(body), } testServer := httptest.NewServer(&fakeHandler) defer testServer.Close() clientset := clientset.NewForConfigOrDie(&restclient.Config{Host: testServer.URL, ContentConfig: restclient.ContentConfig{GroupVersion: &schema.GroupVersion{Group: "", Version: "v1"}}}) </ins> <del> controllerSpec := newReplicationController(1) controllerRef := metav1.NewControllerRef(controllerSpec, v1.SchemeGroupVersion.WithKind("ReplicationController")) </del> <ins> podControl := RealPodControl{ KubeClient: clientset, Recorder:   &record.FakeRecorder{}, } </ins> <del> // Make sure createReplica sends a POST to the apiserver with a pod from the controllers pod template generateName := "hello-" err := podControl.CreatePodsWithGenerateName(context.TODO(), ns, controllerSpec.Spec.Template, controllerSpec, controllerRef, generateName) assert.NoError(t, err, "unexpected error: %v", err) </del> <ins> err := test.podCreationFunc(podControl) assert.NoError(t, err, "unexpected error: %v", err) </ins> <del> expectedPod := v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Labels:          controllerSpec.Spec.Template.Labels, GenerateName:    generateName, OwnerReferences: []metav1.OwnerReference{*controllerRef}, }, Spec: controllerSpec.Spec.Template.Spec, </del> <ins> fakeHandler.ValidateRequest(t, "/api/v1/namespaces/default/pods", "POST", nil) var actualPod = &v1.Pod{} err = json.Unmarshal([]byte(fakeHandler.RequestBody), actualPod) assert.NoError(t, err, "unexpected error: %v", err) assert.True(t, apiequality.Semantic.DeepDerivative(test.wantPod, actualPod), "Body: %s", fakeHandler.RequestBody) }) </ins> } <del> fakeHandler.ValidateRequest(t, "/api/v1/namespaces/default/pods", "POST", nil) var actualPod = &v1.Pod{} err = json.Unmarshal([]byte(fakeHandler.RequestBody), actualPod) assert.NoError(t, err, "unexpected error: %v", err) assert.True(t, apiequality.Semantic.DeepDerivative(&expectedPod, actualPod), "Body: %s", fakeHandler.RequestBody) </del> } func TestDeletePodsAllowsMissing(t *testing.T) {

During implementation of KEP 3393 (PodReplacementPolicy for Jobs) we found that the unit tests of do not follow common patterns established in k/k. 1) The tests are not written using table driven design (). 2) The tests use rather than . This probably should be done as two separate PRs but wanted to highlight some areas that could use some cleanup. Not using means that we don't see diffs in the unit tests that makes this difficult. And not using table driven tests makes it difficult to add new test cases as it usually is brought up during review that we should use a table-driven test but the entire file does not obey this format.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig apps /kind cleanup /help /good-first-issue
This request has been marked as suitable for new contributors. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /sig apps /kind cleanup /help /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I'd like to work on it /assign
I'd also like to make some contributions. /assign
Looks like this need two PRs. tests are not written using table driven design (). tests use Differences rather than Cmp.Diff. I think I can work on the 2nd one and leave the 1st one to you.

Sounds good?
By looking through the , I found the main function to comapre the results have already used cmp.Diff. Can you elaborate more on which part you expect to use cmp.Diff? func verifyDeletedAndPatchedPods(t testing.T, client fake.Clientset, wantDeletedPodNames, wantPatchedPodNames sets.String) { t.Helper() deletedPodNames := getDeletedPodNames(client) if diff := cmp.Diff(wantDeletedPodNames, deletedPodNames); diff != "" { t.Errorf("Deleted pod names (-want,+got):n%s", diff) } patchedPodNames := getPatchedPodNames(client) if diff := cmp.Diff(wantPatchedPodNames, patchedPodNames); diff != "" { t.Errorf("Patched pod names (-want,+got):n%s", diff) } }
I am really sorry about this. Too much multitasking!! I meant not
Hey ! Sorry, I didn't see this before. No problem at all, but consider that one change depends on the other, and since my task implies a big refactor, there might be conflicts when merging your PR... That's why I see two possible scenarios happening: Your PR gets merged before I raise mine: Then fine, I'll rebase your changes and adapt my code Mine is ready before yours is merged: In this case, I propose you commit your changes to my PR. Both of us will appear as coauthors of the change Getting a PR merged in the Kubernetes project takes quite some time (trust me, I talk by experience and can confirm ) WDYT?
Let's see how it goes. We can always rebase the code change to adapt new.
btw, did you already begin to work on task 1? If not, I may just go ahead and work on it right away since I have already dig into it.
yes, I already started And If you ask me, better to keep it this way so we both have an opportunity to have a contribution
cool. Sounds good.
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hi if this issue needs to be worked on any further, please assign me for the same.
has a PR up that will solve the second part of this issue.


Question: completedRequests     int32 } clients := []client{ <del> // "elephant" refers to a client that creates requests at a much higher </del> <ins> // "highqps" refers to a client that creates requests at a much higher </ins> // QPS than its counter-part and well above its concurrency share limit. <del> // In contrast, the mouse stays under its concurrency shares. // Additionally, the "elephant" client also has a higher matching </del> <ins> // In contrast, "lowqps" stays under its concurrency shares. // Additionally, the "highqps" client also has a higher matching </ins> // precedence for its flow schema. <del> {username: "elephant", qps: 100.0, concurrencyMultiplier: 2.0, matchingPrecedence: 999}, {username: "mouse", qps: 5.0, concurrencyMultiplier: 0.5, matchingPrecedence: 1000}, </del> <ins> {username: "highqps", qps: 100.0, concurrencyMultiplier: 2.0, matchingPrecedence: 999}, {username: "lowqps", qps: 5.0, concurrencyMultiplier: 0.5, matchingPrecedence: 1000}, </ins> } ginkgo.By("creating test priority levels and flow schemas") for i := range clients { clients[i].priorityLevelName = fmt.Sprintf("%s-%s", priorityLevelNamePrefix, clients[i].username) framework.Logf("creating PriorityLevel %q", clients[i].priorityLevelName) <del> _, err := f.ClientSet.FlowcontrolV1beta1().PriorityLevelConfigurations().Create( context.TODO(), &flowcontrol.PriorityLevelConfiguration{ ObjectMeta: metav1.ObjectMeta{ Name: clients[i].priorityLevelName, }, Spec: flowcontrol.PriorityLevelConfigurationSpec{ Type: flowcontrol.PriorityLevelEnablementLimited, Limited: &flowcontrol.LimitedPriorityLevelConfiguration{ AssuredConcurrencyShares: 1, LimitResponse: flowcontrol.LimitResponse{ Type: flowcontrol.LimitResponseTypeReject, }, }, }, }, metav1.CreateOptions{}) framework.ExpectNoError(err) defer func(name string) { framework.ExpectNoError(f.ClientSet.FlowcontrolV1beta1().PriorityLevelConfigurations().Delete(context.TODO(), name, metav1.DeleteOptions{})) }(clients[i].priorityLevelName) </del> <ins> _, cleanup := createPriorityLevel(f, clients[i].priorityLevelName, 1) defer cleanup() </ins> clients[i].flowSchemaName = fmt.Sprintf("%s-%s", flowSchemaNamePrefix, clients[i].username) framework.Logf("creating FlowSchema %q", clients[i].flowSchemaName) <del> _, err = f.ClientSet.FlowcontrolV1beta1().FlowSchemas().Create( context.TODO(), &flowcontrol.FlowSchema{ ObjectMeta: metav1.ObjectMeta{ Name: clients[i].flowSchemaName, }, Spec: flowcontrol.FlowSchemaSpec{ MatchingPrecedence: clients[i].matchingPrecedence, PriorityLevelConfiguration: flowcontrol.PriorityLevelConfigurationReference{ Name: clients[i].priorityLevelName, }, DistinguisherMethod: &flowcontrol.FlowDistinguisherMethod{ Type: flowcontrol.FlowDistinguisherMethodByUserType, }, Rules: []flowcontrol.PolicyRulesWithSubjects{ { Subjects: []flowcontrol.Subject{ { Kind: flowcontrol.SubjectKindUser, User: &flowcontrol.UserSubject{ Name: clients[i].username, }, }, }, NonResourceRules: []flowcontrol.NonResourcePolicyRule{ { Verbs:           []string{flowcontrol.VerbAll}, NonResourceURLs: []string{flowcontrol.NonResourceAll}, }, }, }, }, }, }, metav1.CreateOptions{}) framework.ExpectNoError(err) defer func(name string) { framework.ExpectNoError(f.ClientSet.FlowcontrolV1beta1().FlowSchemas().Delete(context.TODO(), name, metav1.DeleteOptions{})) }(clients[i].flowSchemaName) </del> <ins> _, cleanup = createFlowSchema(f, clients[i].flowSchemaName, clients[i].matchingPrecedence, clients[i].priorityLevelName, clients[i].username) defer cleanup() </ins> } ginkgo.By("getting request concurrency from metrics") for i := range clients { <del> resp, err := f.ClientSet.CoreV1().RESTClient().Get().RequestURI("/metrics").DoRaw(context.TODO()) framework.ExpectNoError(err) sampleDecoder := expfmt.SampleDecoder{ Dec:  expfmt.NewDecoder(bytes.NewBuffer(resp), expfmt.FmtText), Opts: &expfmt.DecodeOptions{}, </del> <ins> realConcurrency := getPriorityLevelConcurrency(f, clients[i].priorityLevelName) clients[i].concurrency = int32(float64(realConcurrency) * clients[i].concurrencyMultiplier) if clients[i].concurrency < 1 { clients[i].concurrency = 1 } framework.Logf("request concurrency for %q will be %d (concurrency share = %d)", clients[i].username, clients[i].concurrency, realConcurrency) } ginkgo.By(fmt.Sprintf("starting uniform QPS load for %s", loadDuration.String())) var wg sync.WaitGroup for i := range clients { wg.Add(1) go func(c *client) { defer wg.Done() framework.Logf("starting uniform QPS load for %q: concurrency=%d, qps=%.1f", c.username, c.concurrency, c.qps) c.completedRequests = uniformQPSLoadConcurrent(f, c.username, c.concurrency, c.qps, loadDuration) }(&clients[i]) } wg.Wait() ginkgo.By("checking completed requests with expected values") for _, client := range clients { // Each client should have 95% of its ideal number of completed requests. maxCompletedRequests := float64(client.concurrency) * client.qps * float64(loadDuration/time.Second) fractionCompleted := float64(client.completedRequests) / maxCompletedRequests framework.Logf("client %q completed %d/%d requests (%.1f%%)", client.username, client.completedRequests, int32(maxCompletedRequests), 100*fractionCompleted) if fractionCompleted < 0.95 { framework.Failf("client %q: got %.1f%% completed requests, want at least 95%%", client.username, 100*fractionCompleted) </ins> } <del> for { var v model.Vector err := sampleDecoder.Decode(&v) if err == io.EOF { break } framework.ExpectNoError(err) for _, metric := range v { if string(metric.Metric[model.MetricNameLabel]) != requestConcurrencyLimitMetricName { continue } if string(metric.Metric[requestConcurrencyLimitMetricLabelName]) != clients[i].priorityLevelName { continue } clients[i].concurrency = int32(float64(metric.Value) * clients[i].concurrencyMultiplier) if clients[i].concurrency < 1 { clients[i].concurrency = 1 } framework.Logf("request concurrency for %q will be %d (concurrency share = %d)", clients[i].username, clients[i].concurrency, int32(metric.Value)) } </del> <ins> } }) // This test has two clients (different usernames) making requests at // different rates. Both clients' requests get mapped to the same flow schema // and priority level. We expect APF's "ByUser" flow distinguisher to isolate // the two clients and not allow one client to drown out the other despite // having a higher QPS. ginkgo.It("should ensure that requests can't be drowned out (fairness)", func() { priorityLevelName := "e2e-testing-prioritylevel" flowSchemaName := "e2e-testing-flowschema" loadDuration := 10 * time.Second framework.Logf("creating PriorityLevel %q", priorityLevelName) _, cleanup := createPriorityLevel(f, priorityLevelName, 1) defer cleanup() framework.Logf("creating FlowSchema %q", flowSchemaName) _, cleanup = createFlowSchema(f, flowSchemaName, 1000, priorityLevelName, "*") defer cleanup() type client struct { username              string qps                   float64 concurrencyMultiplier float64 concurrency           int32 completedRequests     int32 } clients := []client{ {username: "highqps", qps: 100.0, concurrencyMultiplier: 2.0}, {username: "lowqps", qps: 5.0, concurrencyMultiplier: 0.5}, } framework.Logf("getting real concurrency") realConcurrency := getPriorityLevelConcurrency(f, priorityLevelName) for i := range clients { clients[i].concurrency = int32(float64(realConcurrency) * clients[i].concurrencyMultiplier) if clients[i].concurrency < 1 { clients[i].concurrency = 1 </ins> } <ins> framework.Logf("request concurrency for %q will be %d", clients[i].username, clients[i].concurrency) </ins> } ginkgo.By(fmt.Sprintf("starting uniform QPS load for %s", loadDuration.String()))

Should we create tests around the creation, modification and deletion of the objects ? For example, adding an invalid FlowSchema with a misconfigured PriorityLevelConfiguration and verifying that the Status appropriately reflects its invalidity. I’m assuming the controller tests will suffice and we don’t need explicit e2e tests for this. What about CRUD tests wherein we’re validating the debug endpoints are showing the right values ? The first class of tests are the basic ones to verify that the requests are being categorized into the right FlowSchema and Priority level. We’ll set up a new PriorityLevelConfiguration and FlowSchema objects and observe that requests are correctly being classified onto them. Verification that requests are correctly being classified will be tricky. For now, we can scrape the metrics endpoint for ‘’ metric to verify that the expected number of requests for a test FlowSchema is correct. Another option is to utilize the ‘dump_requests’ debug endpoint to verify the correct request classification. If during the course of writing the tests we need additional information, we can extend the debug endpoint. Scraping the metrics endpoints to validate the proper request categorization could lead to a fickle e2e test. So querying the ‘debug’ endpoint seems like the right way to go. This test will also serve as a proxy for an end-to-end test that ensures that requests matching a new FlowSchema are successful. Does this have the propensity to be a flaky test(s) ? This class of tests will focus on validating the behavior this feature promises, viz., requests of low priority won’t be drowned out by requests of higher priority and a single bad actor won’t detrimentally affect the requests from others. The following tests will ensure that high velocity flows don’t affect the low velocity flows. For this test, we’ll have two entities submitting requests at the same time. These requests will map to two different FlowSchemas at different priority levels. The higher priority flow will have a lot of requests going through it, far above its concurrency share. At the same time the low priority flow will be submitting requests as well at a rate that well within the concurrency limits of that Flow. The test will ensure that the low velocity (low priority) flow succeeds. This test will be the same as the above one except that the two flows will map to the same FlowSchema but are separated via their Flow Distinguisher.

One of them will be a high velocity flow which the other is a low velocity one. The test will ensure that the low velocity requests still get through and succeed. Does this have the propensity to be a flaky tests ? If the test fails, will the runner of this test be able to grok where the problem could lie and debug ? Is this an actionable failure ?
cc/
We don't need to test that invalid items are rejected, unit tests are better for that. We should try to exercise the basic features (exempt, queue...) We should not rely on the debug endpoint, since we want this to be a conformance test eventually Metrics scraping is also a little questionable. I think we should exercise fair queueing (make two new PLs at different shares, saturate both, verify the responses follow the share ratio). I think we should exercise the shuffle sharding (make a new PL, saturate it with one/few elephants and many mice, verify the mice get service. (those latter two seem to match your last two.) This would be a pretty high level test that tests things end users will care about, but I think it should still be quite hard to pass without actually running our code.
/sig api-machinery /area test /kind feature
cc
to verify the requests can be correctly classified, how about we make use of the ? we can do a test like: make two PLs, and two clients w/ different identities then make the clients are hitting different FS/PL.
what about using a test aggregated apiserver to serve different PLs, and with different the response latency
No strong opinion on this one. I am not sure if that was a response to the ideas in the "Basic CRUD" section --- there the idea is not about testing validation but rather testing the status setting about referential integrity. The debug endpoints are about what is happening at the moment of the GET. Controlling that tightly enough to be worth a test would be tricky. The only plausible approach in an E2E test would be to register an authorization webhook that synchronizes with the test function. I like the basic idea. For the reasons above, I would not recommend looking at the debug endpoints. I do not understand what is wrong with scraping the metrics. Are they not part of conformance? This feature does add response headers.

Are those part of conformance? I like the basic idea. First, remember that this feature does not control velocity; it controls the product of velocity and execution duration. To get a high signal-to-noise ratio, we would want deterministic and large execution durations. Looking in , you can see that authorization is included in execution. So we could add a controlled large amount of duration in there. I am a little surprised at the question of "If the test fails, will the runner of this test be able to grok where the problem could lie and debug"; why would that be more of a question here than for any other E2E test? Presumably the test failure message will say something about the expected and observed behavior. I think the drown-out scenario involves testing queuing. I think exempt was not covered above, and would be a worthwhile addition. This mixes up two things. Fair queuing is something that happens within one priority level. This sounds like it may be expensive for an E2E test. I think we can make a much more efficient unit test of QueueSet that exercises shuffle sharding. Because the QueueSet implementation can be given a FakeEventClock, we can run the test much faster than real time. To get a high signal-to-noise ratio we want controlled, high latencies. Making an aggregated apiserver do that is possible. Perhaps the simplest approach would be to put a fixed sleep in the registry strategies for whatever custom resources the aggregated apiserver is serving. But this seems like a lot more work than necessary. I think an authorization webhook is a simpler approach.
could you elaborate a bit more about the authorization webhook approach? it is only called in the authorization path, if i understand it correctly.
Authorization, and admission control, happen after the priority and fairness filter.
is this really possible? I am likely missing something, but my understanding from the is that each level is limited to the concurrency shares assigned to it.
: the text should be revised to more clearly express the intent. It is that the driving client has many more requests outstanding at once than the corresponding priority level will allow to be served concurrently.
FYI: I'm working on the second test.


Question: Raw() } if err != nil { <del> Logf("Controller %s: Failed to GET from replica %d [%s]: %v:", r.controllerName, i+1, pod.Name, err) </del> <ins> Logf("Controller %s: Failed to GET from replica %d [%s]: %vnpod status: %#v", r.controllerName, i+1, pod.Name, err, pod.Status) </ins> continue } // The response checker expects the pod's name unless !respondName, in

https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite} Previous issues for this test:
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}


Question: } var tests = map[string]struct { <del> spec        core.PersistentVolumeClaimSpec gateEnabled bool want        *core.TypedLocalObjectReference </del> <ins> spec core.PersistentVolumeClaimSpec want *core.TypedLocalObjectReference </ins> }{ "enabled with empty ds": { <del> spec:        core.PersistentVolumeClaimSpec{}, gateEnabled: true, want:        nil, </del> <ins> spec: core.PersistentVolumeClaimSpec{}, want: nil, </ins> }, "enabled with invalid spec": { <del> spec:        invalidSpec, gateEnabled: true, want:        nil, </del> <ins> spec: invalidSpec, want: nil, </ins> }, "enabled with valid spec": { <del> spec:        validSpec, gateEnabled: true, want:        validSpec.DataSource, }, "disabled with invalid spec": { spec:        invalidSpec, gateEnabled: false, want:        nil, }, "disabled with valid spec": { spec:        validSpec, gateEnabled: false, want:        nil, }, "diabled with empty ds": { spec:        core.PersistentVolumeClaimSpec{}, gateEnabled: false, want:        nil, </del> <ins> spec: validSpec, want: validSpec.DataSource, </ins> }, "enabled with valid spec but nil APIGroup": { <del> spec:        validSpecNilAPIGroup, gateEnabled: true, want:        validSpecNilAPIGroup.DataSource, }, "disabled with valid spec but nil APIGroup": { spec:        validSpecNilAPIGroup, gateEnabled: false, want:        nil, </del> <ins> spec: validSpecNilAPIGroup, want: validSpecNilAPIGroup.DataSource, </ins> }, } for testName, test := range tests { t.Run(testName, func(t *testing.T) { <del> defer featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, features.VolumePVCDataSource, test.gateEnabled)() </del> DropDisabledFields(&test.spec, nil) if test.spec.DataSource != test.want { <del> t.Errorf("expected drop datasource condition was not met, test: %s, gateEnabled: %v, spec: %v, expected: %v", testName, test.gateEnabled, test.spec, test.want) </del> <ins> t.Errorf("expected drop datasource condition was not met, test: %s, spec: %v, expected: %v", testName, test.spec, test.want) </ins> } })

/kind feature /sig storage /sig api PVC Data Sources have been in beta since 1.16 and should be ready to be promoted to GA in the 1.18 release. KEP:
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /kind feature /sig storage /sig api Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: podSchedulingDirty bool mutex sync.Mutex <ins> informationsForClaim []informationForClaim } type informationForClaim struct { // The availableOnNode node filter of the claim converted from the // v1 API to nodeaffinity.NodeSelector by PreFilter for repeated // evaluation in Filter. Nil for claim which don't have it. availableOnNode *nodeaffinity.NodeSelector // The status of the claim got from the // schedulingCtx by PreFilter for repeated // evaluation in Filter. Nil for claim which don't have it. status *resourcev1alpha2.ResourceClaimSchedulingStatus </ins> } func (d *stateData) Clone() framework.StateData {

The node selectors gets parsed by in for each node. That work can be done once in . Better performance.
/sig node /sig scheduling /triage accepted /priority backlog
I can try this one. /assign


Question: }, }, update: networking.NetworkPolicy{ <del> ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "bar"}, </del> <ins> ObjectMeta: metav1.ObjectMeta{Name: "baz", Namespace: "bar"}, </ins> Spec: networking.NetworkPolicySpec{ <del> PodSelector: metav1.LabelSelector{ MatchLabels: map[string]string{"a": "b"}, }, Ingress: []networking.NetworkPolicyIngressRule{}, </del> <ins> PodSelector: metav1.LabelSelector{}, Ingress:     []networking.NetworkPolicyIngressRule{}, </ins> }, }, }, } for testName, errorCase := range errorCases { <ins> errorCase.old.ObjectMeta.ResourceVersion = "1" errorCase.update.ObjectMeta.ResourceVersion = "1" </ins> if errs := ValidateNetworkPolicyUpdate(&errorCase.update, &errorCase.old); len(errs) == 0 { t.Errorf("expected failure: %s", testName) }

I tried to update the NetworkPolicy Spec field with more ingress rules but got It seems NetworkPolicy does not allow update on Spec This means if I want to have more ingress rules, I will have to add those in a new networkpolicy in the same namespace. Any reason why I should not add those in the existing one(update the existing networkpolicy)? Thanks!
This is as designed for now. The expectation is that updates to the spec will be allowed when the NetworkPolicy API moves from v1beta1 to GA.
(And that is planned for Kubernetes 1.6 release.)
good to know, thanks!
Note that the PR to move NetworkPolicy to v1 did not make it into v1.6, but is now planned for 1.7:
Oh, I didn't know about this bug and that didn't get changed in but we can do it as another PR after. (I'm trying to not make any more changes there so we can get it merged.)


Question: NODE_ACCELERATORS=${NODE_ACCELERATORS:-""} REGISTER_MASTER_KUBELET=${REGISTER_MASTER:-true} KUBE_APISERVER_REQUEST_TIMEOUT=300 <ins> # Increase initial delay for the apiserver liveness probe, to avoid prematurely tearing it down KUBE_APISERVER_LIVENESS_PROBE_INITIAL_DELAY_SEC=${KUBE_APISERVER_LIVENESS_PROBE_INITIAL_DELAY_SEC:-45} # Also increase the initial delay for etcd just to be safe ETCD_LIVENESS_PROBE_INITIAL_DELAY_SEC=${ETCD_LIVENESS_PROBE_INITIAL_DELAY_SEC:-45} </ins> PREEMPTIBLE_NODE=${PREEMPTIBLE_NODE:-false} PREEMPTIBLE_MASTER=${PREEMPTIBLE_MASTER:-false} KUBE_DELETE_NODES=${KUBE_DELETE_NODES:-true}

For the cluster upgrade failed. Looking at the failure, it seems to be an inopportune restart of apiserver. The new master comes up, apiserver starts, but then it restarts. The upgrade test hits it during the restart, it looks like. Logs: The reason for the apiserver restart was that kube-apiserver failed its liveness probe The liveness probe for api server is hitting :8080, and indeed it was failing its health check (apiserver log again): I'm guessing this is due to Looking at the next run, I'm hypothesizing that RBAC updates just take too long, and so it's marginal as to whether the bootstrap roles complete in time. /sig api-machinery /kind failing-test cc
/milestone v1.13
I see so many api calls getting clusterroles and clusterrolebingdings. should we run in parallel to accelerate the bootstrap?
I open trying to fix it.
That's great - thank you I do think we should also think about why each call is slow though - running in parallel or just boosting the timeout feels a little bit like cheating, at least until we have justified that this is an intrinsically slow operation. cc (as you know both apimachinery & auth)
I'm seeing a total of 2 seconds to perform all RBAC initialization on initial cluster bring-up, and a total of .25 seconds to verify RBAC initialization on cluster restart. There's nothing inherently slow about the operation. How long is the liveness check configured for in this setup? If many many API calls are being made using the internal client on restart, it could exhaust the assigned QPS and artificially slow down all calls made using it. The apiserver log of calls made during restart (as opposed to initial start) would be helpful in determining if there's a component that is making excessive numbers of API calls.
/priority important-soon
Chatted with directly, and found that while RBAC isn't fast (probably rate-limiting), the're a much bigger delay which is probably more significant: Note the ~ 15 second gap. And this was with other tests as well, e.g. the example above is actually from Going to crank up the log level and see if I get any hints as to what is happening
/assign
GitHub didn't allow me to assign the following users: logicalhan.

Note that only and repo collaborators can be assigned. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I dug into it - the reason for the long delay is that we're waiting for etcd to start. We don't log anything, even at . So maybe we do just increase the wait time :-( An alternative would be to do an initializerPod that waits for etcd, which would be more correct.
I agree with about increasing the wait time for our liveness check (especially considering the fact that we are apparently hitting our liveness failure threshold 75% of the time during out e2e tests).
how did you gather those metrics, by the way?
I'm sure there's a better way, but I ed the logs down with some bash, and then grepped against them. Specifically the file: ( for a count) and to list files without matches
Is this still broken? Trying to figure out if it should be a blocker still, or not. I'm not seeing these test failures.
It’s a bad flake that keeps happening on and off. We can take it off the blocker list, but just find the right owner to investigate and fix for long term. Josh Berkus <:
I discussed this last week with we should increase the livenessProbe delay to avoid the premature kubelet restarts. /assign
GitHub didn't allow me to assign the following users: logicalhan. Note that only and repo collaborators can be assigned. For more information please see <detailsIn response to : I discussed this last week with we should increase the livenessProbe delay to avoid the premature kubelet restarts. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: "k8s.io/apimachinery/pkg/api/resource" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/util/uuid" <del> utilfeature "k8s.io/apiserver/pkg/util/feature" </del> clientset "k8s.io/client-go/kubernetes" <del> "k8s.io/kubernetes/pkg/features" </del> priorityutil "k8s.io/kubernetes/pkg/scheduler/algorithm/priorities/util" "k8s.io/kubernetes/test/e2e/common" "k8s.io/kubernetes/test/e2e/framework"

Testgrid: https://k8s- https://k8s- Example: https://k8s- This seems to be related to /sig scheduling /kind failing-test /priority important-soon /cc
/assign RaviSantosh Gudimetla
GitHub didn't allow me to assign the following users: RaviSantosh, Gudimetla. Note that only and repo collaborators can be assigned. For more information please see <detailsIn response to : /assign RaviSantosh Gudimetla Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
Looked at the kube-scheduler logs for GCE, seems the feature_gate was enabled. I don't see enabled, will setup the cluster locally to test if this is the issue.
have you been able to look into this? These tests are broken right now for unrelated reasons, but once they get fixed I expect to keep seeing the scheduler failure.
/milestone v1.13
is the one open PR expected to fix the remaining failures we are seeing the release dashboards? If not fixed by this week, these failures will block Beta cut targeted for 11/6.
- Apologies for the delay. Yes, this PR should fix the failure, if not, I will disable this test for now and revert the feature promotion PR associated with this test. Also, I need to rethink, if e2e is an appropriate test scenario to handle this feature or should unit tests/integration suffice. I could not find a test which does something like resizing a node in the existing e2e test suite.
I'm looking at this, and I'm not sure this is going to fix the failures. I'm looking at the tests, and it seems to be setting the node status capacity/allocatable to pretend to be a 12 core machine, and then it tries to allocate a pod on every node such that every node is 50% utilized. It thus tries to allocate a pod of ~ 6 cores on the resized node. But that node still only has 2 cores, so it fails to schedule: I think that's why the test has been failing, at least for the past few runs which I checked.

What I have yet to figure out is how this passed here...
is it safe to rephrase this as a pure test design issue then?
the test still failed on the last . Even though it was a couple of hours after PR merged, I dont see it in the delta changes . I will watch for the next run result. Assuming it still doesnt pass (as pointed out about) we need to figure out what this e2e test failure means and should it block 1.13 Beta0 for next Tuesday. Considering the test did pass very few times since its introduction, this seems like an issue with the test design itself and not the feature. Do you have other tests (unit or integration or different approach to e2e) that can infirm us on the health of thew feature? If those exist and are green we can unblock Beta on those results while you determine the right way to author the e2e. Barring those being present right idea, I think we will have to err on the side of caution and roll back the feature itself before Beta. Either way we need to take a call and submit relevant PRs by tomorrow so we get green CI over the weekend, in time for Beta cut on Tuesday.
This is not a blocker, as I told earlier as well, e2e may not be good avenue for this test. I actually thought more about this problem last day. The problem is I updated the node capacity and I was checking if the node could handle the pod based on the scheduler cache which we get from kubelet. But kubelet on the other side is updating it's node information(like CPU, memory etc) in the API server there by causing a race condition. So, it could pass with a very little chance. So, I will remove this test from e2e. Yes, we have unit tests, that help in validating the feature. I will try to get an integration test but this is not a blocker for feature.
Thanks for the explanation and analysis do you plan on adding more e2e tests later or stick to unit and integration for validating this feature? and to weigh in on the path forward i.e., remove e2e tests for this feature and rely on unit and integration.
- We discussed internally in the sig meeting yesterday and it was decided to have an integration test instead of e2e test.

I plan to add an integration test(which we don't have today) later.
since this PR is actually tracking an enhancement work graduating to Beta in 1.13, it should have been tracked via a corresponding issue in k/enhancements repo in 1.13 timeframe. We are now past Feature freeze for 1.13 and its late to open an issue against it. Also we are missing integration tests for this feature, which we are . Given the above two points and from offline discussions with SIG leads regarding the criticality of this feature going to Beta now vs in 1.14 timeframe, I am asking for it to be punted to 1.14. As AIs Please prepare a PR to remove it from Beta in 1.13 Please open a tracking issue in for 1.14 milestone.


Question: // TODO convert this entire proxy to an UpgradeAwareProxy similar to // https://github.com/openshift/origin/blob/master/pkg/util/httpproxy/upgradeawareproxy.go. // That proxy needs to be modified to support multiple backends, not just 1. <ins> if r.tryUpgrade(w, req, newReq, destURL) { return } proxy := httputil.NewSingleHostReverseProxy(&url.URL{Scheme: "http", Host: destURL.Host}) proxy.Transport = &proxyTransport{ proxyScheme:      req.URL.Scheme, proxyHost:        req.URL.Host, proxyPathPrepend: requestInfo.URLPath(), } proxy.FlushInterval = 200 * time.Millisecond proxy.ServeHTTP(w, newReq) } // tryUpgrade returns true if the request was handled. func (r *ProxyHandler) tryUpgrade(w http.ResponseWriter, req, newReq *http.Request, destURL *url.URL) bool { </ins> connectionHeader := strings.ToLower(req.Header.Get(httpstream.HeaderConnection)) <del> if strings.Contains(connectionHeader, strings.ToLower(httpstream.HeaderUpgrade)) && len(req.Header.Get(httpstream.HeaderUpgrade)) > 0 { //TODO support TLS? Doesn't look like proxyTransport does anything special ... dialAddr := netutil.CanonicalAddr(destURL) backendConn, err := net.Dial("tcp", dialAddr) if err != nil { status := errToAPIStatus(err) writeJSON(status.Code, r.codec, status, w) return } defer backendConn.Close() </del> <ins> if !strings.Contains(connectionHeader, strings.ToLower(httpstream.HeaderUpgrade)) || len(req.Header.Get(httpstream.HeaderUpgrade)) == 0 { return false } //TODO support TLS? Doesn't look like proxyTransport does anything special ... dialAddr := netutil.CanonicalAddr(destURL) backendConn, err := net.Dial("tcp", dialAddr) if err != nil { status := errToAPIStatus(err) writeJSON(status.Code, r.codec, status, w) return true } defer backendConn.Close() </ins> <del> // TODO should we use _ (a bufio.ReadWriter) instead of requestHijackedConn // when copying between the client and the backend? Docker doesn't when they // hijack, just for reference... requestHijackedConn, _, err := w.(http.Hijacker).Hijack() if err != nil { status := errToAPIStatus(err) writeJSON(status.Code, r.codec, status, w) return } defer requestHijackedConn.Close() </del> <ins> // TODO should we use _ (a bufio.ReadWriter) instead of requestHijackedConn // when copying between the client and the backend? Docker doesn't when they // hijack, just for reference... requestHijackedConn, _, err := w.(http.Hijacker).Hijack() if err != nil { status := errToAPIStatus(err) writeJSON(status.Code, r.codec, status, w) return true } defer requestHijackedConn.Close() </ins> <del> if err = newReq.Write(backendConn); err != nil { status := errToAPIStatus(err) writeJSON(status.Code, r.codec, status, w) return </del> <ins> if err = newReq.Write(backendConn); err != nil { status := errToAPIStatus(err) writeJSON(status.Code, r.codec, status, w) return true } done := make(chan struct{}, 2) go func() { _, err := io.Copy(backendConn, requestHijackedConn) if err != nil && !strings.Contains(err.Error(), "use of closed network connection") { glog.Errorf("Error proxying data from client to backend: %v", err) </ins> } <ins> done <- struct{}{} }() </ins> <del> done := make(chan struct{}, 2) go func() { _, err := io.Copy(backendConn, requestHijackedConn) if err != nil && !strings.Contains(err.Error(), "use of closed network connection") { glog.Errorf("Error proxying data from client to backend: %v", err) } done <- struct{}{} }() go func() { _, err := io.Copy(requestHijackedConn, backendConn) if err != nil && !strings.Contains(err.Error(), "use of closed network connection") { glog.Errorf("Error proxying data from backend to client: %v", err) } done <- struct{}{} }() <-done } else { proxy := httputil.NewSingleHostReverseProxy(&url.URL{Scheme: "http", Host: destURL.Host}) proxy.Transport = &proxyTransport{ proxyScheme:      req.URL.Scheme, proxyHost:        req.URL.Host, proxyPathPrepend: path.Join(r.prefix, "ns", namespace, resource, id), </del> <ins> go func() { _, err := io.Copy(requestHijackedConn, backendConn) if err != nil && !strings.Contains(err.Error(), "use of closed network connection") { glog.Errorf("Error proxying data from backend to client: %v", err) </ins> } <del> proxy.FlushInterval = 200 * time.Millisecond proxy.ServeHTTP(w, newReq) } </del> <ins> done <- struct{}{} }() <-done return true </ins> } type proxyTransport struct {

The apiserver proxy rewrites links to the form "/proxy/ns/default/...": But this form stopped working recently, seemingly around The "/proxy/ns/default/..." form always returns "404: Page Not Found", similar to the problem seen in If you remove "/ns/default" from the URL, it works.
/cc
I can investigate this as well.
I see the issue, sending a PR.
The proxy path needs to be built with knowledge of the apiversion of the incoming proxy request, and it needs to be updated to use the knowledge from the restmapper. This is a little trickier than I have time for right now, but it should definitely not be hardcode appending the /ns form. Not clear why this is not also caught by our test case. I can try and look at this more in the AM unless someone else wants to take a look in interim. /cc
Let me steal this one from you, I think I can fix it.
Have at it. /cc me on your PR
Sorry, I see you've already got a PR out!
No problem. I would be happy if you finished my PR relative to the commit I started. It will be a few days before I can go back to looking at this issue. I think v1beta1/v1beta2 proxy urls in go-restful need to always but the namespace scope in the path as the ultimate fix. I do want the proxy to have access to the restmapper per my PR. ----- Original Message ----- From: "Daniel Smith" To: "GoogleCloudPlatform/kubernetes" Cc: "Derek Carr" Sent: Friday, February 13, 2015 4:56:19 PM Subject: Re: [kubernetes] apiserver proxy stopped accepting namespaces () Sorry, I see you've already got a PR out! Reply to this email directly or view it on GitHub:
Any updates ?
This is breaking monitoring since it requires the proxy. Bumping up the priority to P0.
Sorry for dropping this. Working on a fix.


Question: func TestEndpointAddressNodeNameUpdateRestrictions(t *testing.T) { oldEndpoint := newNodeNameEndpoint("kubernetes-node-setup-by-backend") updatedEndpoint := newNodeNameEndpoint("kubernetes-changed-nodename") <del> // Check that NodeName cannot be changed during update (if already set) </del> <ins> // Check that NodeName can be changed during update, this is to accommodate the case where nodeIP or PodCIDR is reused. // The same ip will now have a different nodeName. </ins> errList := ValidateEndpoints(updatedEndpoint) errList = append(errList, ValidateEndpointsUpdate(updatedEndpoint, oldEndpoint)...) <del> if len(errList) == 0 { t.Error("Endpoint should not allow changing of Subset.Addresses.NodeName on update") </del> <ins> if len(errList) != 0 { t.Error("Endpoint should allow changing of Subset.Addresses.NodeName on update") </ins> } }

/kind bug What happened: We have noticed that one of the node-exporters is down. It turned out that service has "old" endpoint still. After digging deeper i see following errors in controller manager Turns out controller manager can not update endpoints for node-exporter service, because of this error. In endpoint list is see right IPs for worker nodes, but wrong IP for master node (that's why we detected the problem). Context: two worker nodes were recreated with new hostnames (generated randomly), but they use same ip address one master node were recreated with new hostname and new ip I've tried to manually update hostname and got same error, seems nodeName is not updatable field. It also affects all other endpoints, we were forced to recreated all pods (so they will change IPs) to be able to fix endpoints reconciliation. What you expected to happen: controller manager successfully update endpoints after node recreated with same IP, but different hostname. How to reproduce it (as minimally and precisely as possible): 1 master, 1 worker, service that points to pods with host network. k8s API service (keep it shut down) worker node with same IP and different host name to recreate pod on worker and in controller manager should appear error above. Anything else we need to know?: Environment: Kubernetes version (use ): 1.10.1 Cloud provider or hardware configuration: baremetal OS (e.g. from /etc/os-release): CoreOS 1688.5.3 Kernel (e.g. ): 4.14.32-coreos Install tools: own tool () Others:
/sig node
what is the use case of changing node name?
/sig api-machinery
We have ephemeral immutable nodes in our clusters. So upgrade or any other operation will cause node recreation. Hostname generated automatically, but the IP address depends on the underlay hypervisor node, where node will be scheduled. So it's possible that node stays on the same hypervisor and uses same ip address.
/cc
/assign
/cc do you think it's reasonable to loose endpoints validation?
/sig network this API is owned by sig network


Question: } } <ins> func TestDeleteNoObjects(t *testing.T) { f, tf, codec := NewAPIFactory() tf.Printer = &testPrinter{} tf.Client = &client.FakeRESTClient{ Codec: codec, Client: client.HTTPClientFunc(func(req *http.Request) (*http.Response, error) { switch p, m := req.URL.Path, req.Method; { case p == "/ns/test/pods" && m == "GET": return &http.Response{StatusCode: 200, Body: objBody(codec, &api.PodList{})}, nil default: t.Fatalf("unexpected request: %#vn%#v", req.URL, req) return nil, nil } }), } buf := bytes.NewBuffer([]byte{}) stderr := bytes.NewBuffer([]byte{}) cmd := f.NewCmdDelete(buf) cmd.SetOutput(stderr) cmd.Flags().String("namespace", "test", "") cmd.Run(cmd, []string{"pods"}) if buf.String() != "" { t.Errorf("unexpected output: %s", buf.String()) } if stderr.String() != "No resources foundn" { t.Errorf("unexpected output: %s", stderr.String()) } } </ins> func TestDeleteMultipleObject(t *testing.T) { pods, svc := testData()

We should support multiple objects on the command line, but, failing that, some kind of error message would be useful. /cc
Fixing, just fixed the equiv in create


Question: var ( defaultMetricsCollectorConfig = metricsCollectorConfig{ <del> Metrics: []string{ "scheduler_scheduling_algorithm_predicate_evaluation_seconds", "scheduler_scheduling_algorithm_priority_evaluation_seconds", "scheduler_e2e_scheduling_duration_seconds", "scheduler_pod_scheduling_duration_seconds", </del> <ins> Metrics: map[string]*labelValues{ "scheduler_framework_extension_point_duration_seconds": { label:  extensionPointsLabelName, values: []string{"Filter", "Score"}, }, "scheduler_e2e_scheduling_duration_seconds": nil, "scheduler_pod_scheduling_duration_seconds": nil, </ins> }, } )

<!-- Please only use this template for submitting enhancement requests --What would you like to be : run got the below error message, We should provide the metric data for different extension points from the metric of "schedulerframeworkextensionpointdurationseconds" Why is this needed: Those metrics have been removed by this commit - Those metrics are deprecated cause it is suggested to use - , but simply remove those metrics definitely is not a nice way to address the issue. /sig scheduling
/triage accept
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /triage accept Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage accepted
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale


Question: "k8s.io/kubernetes/pkg/kubelet/status" kubetypes "k8s.io/kubernetes/pkg/kubelet/types" "k8s.io/kubernetes/pkg/kubelet/util/format" <ins> "k8s.io/kubernetes/pkg/util" </ins> "k8s.io/kubernetes/pkg/volume" "k8s.io/kubernetes/pkg/volume/util/volumehelper" "k8s.io/kubernetes/third_party/forked/golang/expansion"

k8s 1.6.3 just landed and broke a lot of our systems. The issue shows as: 2017-05-10 20:07:19. | NAME READY STATUS RESTARTS AGE 2017-05-10 20:07:19. | test-ceph-init-mon-hx3h1 0/1 mkdir /var/lib/kubelet/pods/-35bb-11e7-8ad3- not a directory 0 3m The issue is, we have subPath set to a file in a configmap to map it into the right place in our container. But the new code assumes subPath is a directory and always does a mkdir over it. This causes things to fail. The PR in question is:
cc /assign wongma7
Reiterating the mentions to trigger a notification: <detailsIn response to : cc /assign wongma7 Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. I understand the commands that are listed . </details
Its also related to:
can you ensure your fix is picked to 1.6 branch, too?
yes, I will.
any chance it gets finally committed? our testing gates are all red because of this issue..
I brought it up in the sig-storage meeting today so priority was raised to critical-urgent and sig node was pinged. I guess we could make some more noise, I'll try finding people on slack
Would be great to get it out one way or another. I have heard other people also see this issue.
Status: it's merging to master right now & I just opened a cherry-pick for it into 1.6 I'm not sure what the procedure is for expediting a point release, but I suppose it would be enough if those other people +1 this issue, it may help the folks in charge judge the severity of the bug. and what's the procedure here? The context is that a bug was introduced with 1.6.3 that makes it so that certain pods (that mount a volume's file using the subPath field) fail to start. Thanks!
FYI, closed by mistake, sorry.

We still need to track it 'til it's fixed in 1.6.x
I think an expedited release would be warranted if there are a large number of users who both (1) are affected by this issue, and (2) need something in 1.6.3 so rolling back to 1.6.2 is not good enough. I don't know how common it is for users to rely on this feature, so any signal we can get on that would be helpful.
this feature is used a lot by kolla-kubernetes project. Rolling back is problematic, our gating system works with centos and Ubuntu images, with centos we can go back to 1.6.2 but with Ubuntu, it looks like 1.6.2 binaries have been already removed. We use binaries which are provided by repo and not static files.
Perhaps we should add an e2e test case for subPath.
yeah, i have to agree with and on this one. our projects are very similar, and we're experiencing the same thing on our end w/v1.6.3.
After discussing with I am planning to expedite v1.6.4 to fix this. However, I will wait for the e2e test, which I believe is working on.
Keeping open until v1.6.4 ships with fix.
the e2e test PR is here: . They cover both this issue and the original one I was trying to fix, .
should kubernetes yank the binary packages for kubelet/kubeadm/etc that were released for 1.6.3 (debian/rpm/etc) - until 1.6.4 gets released? Because many of us who are just installing kubeadm/kubelet/etc with 1.6.* have found that things suddenly start break. Right now, its in the release note with a comment like . is not an alpha/beta feature, it's a core feature, and we just can't assume someone "may or may not" use it. Should this be rather more critical than that? 3 is a broken release and nobody should ever install it right?
as a workaround we compiled 1.6.3 kubelet with the one fix. If you are interested, please let me know I will share the link, I have it posted on GitHub.

thank you, we have pinned kubernetes versions for now... I can see that k8s 1.6.4-beta.1 is already being tagged. Great work from the team! Looks like k8s 1.6.4 will supersede 1.6.3 shortly. Thanks!
If you experienced this issue in v1.6.3, please try and report back on this bug. Thanks.
We've got a test all queued up. Just waiting for the rpms/debs to hit.
So I can confirm that the mentioned bug is fixed with 1.6.4. great, thanks.
Closing. Issue is fixed in release
Hi I'm facing this error, I'm using ansible to deploy kubernetes, so it brings the version: 1.5.2 , is their a way to resolve this issue without changing the release ? Thanks.
this specific bug was introduced with 1.6.3 and fixed with 1.6.4. Please open a separate issue with more details about your issue
ok thank u

BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): root kubectl version Client Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-05-10T15:48:59Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.3+coreos.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-05-11T18:15:57Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"} Environment: OS (e.g. from /etc/os-release): root cat /etc/os-release NAME="Ubuntu" VERSION="16.04.2 LTS (Xenial Xerus)" ID=ubuntu IDLIKE=debian PRETTYNAME="Ubuntu 16.04.2 LTS" VERSIONID="16.04" HOMEURL="" SUPPORTURL="" BUGREPORTURL="" VERSIONCODENAME=xenial UBUNTUCODENAME=xenial Kernel (e.g. ): root uname -a Linux ginger 4.4.0-78-generic -Ubuntu SMP Thu Apr 27 15:29:09 UTC 2017 x8664 x8664 x8664 GNU/Linux What happened: Updated latest Ubuntu packages from to Kubelet v1.6.3. Updated to Hyperkube v1.6.3.

Applied deployment from that also creates the following ConfigMaps: apiVersion: v1 kind: ConfigMap metadata: name: salt-syndic-master-config data: master: | autoaccept: True fileserverbackend: git gitfsremotes: extpillar: git: master syndicmaster: gitfssaltenv: kube: ref: master minionid: | playground.cluster.awesome- The pods of deployment are not able to mount the data from the configmaps as file using the following volumeMounts: volumeMounts: mountPath: /etc/salt/master name: salt-syndic-master-config subPath: master Instead I get the following errors from : May 17 13:56:44 ginger kubelet[]: E0517 13:56:44. ] failed to mkdir:/var/lib/kubelet/pods/-3ae8-11e7-ade6- May 17 13:56:44 ginger kubelet[]: E0517 13:56:44. ] container start failed: mkdir /var/lib/kubelet/pods/-3ae8-11e7-ade6- not a directory: Generate Container Config Failed May 17 13:56:44 ginger kubelet[]: E0517 13:56:44. ] Error syncing pod -3ae8-11e7-ade6- ("salt-syndic--kfj8w_awesome(-3ae8-11e7-ade6-)"), skipping: failed to "StartContainer" for "salt-syndic" with mkdir /var/lib/kubelet/pods/-3ae8-11e7-ade6- not a directory: "Generate Container Config Failed" Looking into shows that is a symlink to the which contains the correct data. Any help or tipps for further investigations are appreciated.
Duplicate of ...


Question: import ( "fmt" "net/http" <ins> "time" "k8s.io/api/core/v1" </ins> "k8s.io/apimachinery/pkg/util/sets" "k8s.io/kubernetes/pkg/master/ports" "k8s.io/kubernetes/test/e2e/framework"

Augment to turn on session affinity for a service, lookup the vip from a container/node and make sure we only hit a single endpoint with N ready/running endpoints.
This looks like it is still relevant can anyone confirm if we these? I still see a
/assign
Are you working on it? Could you please assign this issue to me and I am happy to fix it. Thanks a lot!
/assign I have it in my "backlog" behind NetworkPolicy stuff. Feel free to take it over if you have time.
GitHub didn't allow me to assign the following users: m1093782566. Note that only can be assigned. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Would appreciate :)
/assign


Question: effect: "NoExecute" - operator: "Exists" effect: "NoSchedule" <del> hostNetwork: true hostPID: true </del> volumes: - name: device-plugin hostPath:

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: Host Networking is enabled for the nvidia-gpu daemonset. Trying to understand the purpose. What you expected to happen: Disabling hostNetworking seemed to work and GPU workloads are able to be scheduled to the cluster. How to reproduce it (as minimally and precisely as possible): Remove this line and apply the daemonset to a GPU enabled cluster. Anything else we need to know?: I would like to understand the purpose for using hostNetworking for the nvidia-gpu daemonset. It appears nvidia implementation does not use it either:
/sig node /sig network


Question: package config const ( <del> DefaultKubeletPodsDirName             = "pods" DefaultKubeletVolumesDirName          = "volumes" DefaultKubeletVolumeDevicesDirName    = "volumeDevices" DefaultKubeletPluginsDirName          = "plugins" DefaultKubeletContainersDirName       = "containers" DefaultKubeletPluginContainersDirName = "plugin-containers" DefaultKubeletPodResourcesDirName     = "pod-resources" </del> <ins> DefaultKubeletPodsDirName                = "pods" DefaultKubeletVolumesDirName             = "volumes" DefaultKubeletVolumeDevicesDirName       = "volumeDevices" DefaultKubeletPluginsDirName             = "plugins" DefaultKubeletPluginsRegistrationDirName = "plugins_registry" DefaultKubeletContainersDirName          = "containers" DefaultKubeletPluginContainersDirName    = "plugin-containers" DefaultKubeletPodResourcesDirName        = "pod-resources" </ins> )

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug This is more of an enhancement. What happened: The Plugin Watcher is setup to trigger on file activities when they occur in the subdirectory that it scans. During startup, the Watcher is passed (from within the Kubelet) the Kubelet's root plugin directory as a location to scan for newly plugins - Because the kubelet's plugins subdirectory is also used by existing in-tree drivers (including csi) as a location to store plugins artifacts, the Plugin Watcher is getting triggered by directory/file activities occurring in that subdirectory that has nothing to do with plugin registration as shown in the output below: What you expected to happen: The Plugin Watcher should not be getting triggered by file/directory activities not related to plugin registration/unregistration. One possibility is to have the Watcher scan one directory below plugins like or something similar How to reproduce it (as minimally and precisely as possible): Run plugin watcher with a CSI driver. cc
/sig node
I think you should only place the plugin socket file in the plugin registration directory but not the plugin data.
agree with we can only place the registration file in the specified path instead of placing all data into it. On the other hand, you can place data into the /var/lib/kubelet/plugins/registration even if registration created, that's a no-end loop.
As I mentioned in issue, existing in-tree volume plugins were already using that directory as a location to store storage operation artifacts (not just csi). Here is a shortened list of driver locations that use /var/lib/kubelet/plugins in their code already: And several more locations. The fix would be to have plugin watcher scan a subdirectory of plugins (say ) instead of plugins directly as it is doing today. The watcher functions as intended. Except now it picks up all directory activities whether related or not to plugin registration.
Well given the fact that the pluginwatcher is beta this is a bit of an issue. My understanding of the "beta" guarantees is that we want to be backwards compatible for all plugins published with the beta API. Meaning that even if we move the registration directory to a new location, we'll need to keep scanning the old directory (to pick up v1beta plugins).

From a 10 000 feet point of view it seems easier to me if we move the in-tree volume plugin's data. Though it might be a lot of work. Another less painful option, but a bit grey is the following: Enforce plugins to end with this suffix is recommended for device plugins (though not enforced) and I believe is a convention on linux. This is a bit grey because we would still be breaking v1beta users but since we're moving a convention from "recommended" to "enforced" it seems "less" breaking to me (and I haven't seen device plugins not using that extension). What do you think ?
I like the suffix idea. It can be a recommendation (which csi is already doing).
Hmm, I've just deployed a new k8s cluster on 1.12.0 and noticed this repeating in the logs on a few of our pods. Just for my own sanity, is this something I should be worried about or can I ignore it until a fix is released?
you can ignore these errors. if it's a recommendation then we'll keep spamming the logs with all the data files. Another thing to keep in mind, it seems like CSI plugins puts a lot of data (and creates a lot of directories). It might be worth considering limiting the recursion to one level. I'm concerned we might exhaust the amount of fds by tracking the plugins data.
yeah CSI keeps track of volume operation metadata in the plugins dir. This probably will not change. I understand this is beta. However, besides CSI and Device plugin, are there any other implementers out there for this feature ? Would it be possible to propose a new location (i.e. <pluginsDir/reg/?
let me know if you open a PR for this as discussed.
thanks for open the issue with detail logs, i can see two problems by looking at logs and code. 1) plugin watcher trying to dial non-socket files. 2) too many logs generated (CREATE and DELETE). I a suggestion to fix of this issue we can fix those problems by: 1) check file mode when handle CREATE event, only attempt to dial file when file mode is socket.

2) increase log verbose level for both 'handleDeleteEvent()' and 'handleCreateEvent()' plugin watcher still has activity when any file create/delete under the directory, but they become light weighted (CREATE file will only trigger file mode check, DELETE file event will trigger plugin searching) i am not sure we want to go further to limit directory depth where plugin can live. it has two drawbacks IMO: 1) it is hard to enforce, not sure everyone will read code/readme. 2) it provides less flexibility to plugins: if want to organize many plugin socket files with directory.
These are definitely good suggestions! It seems like there might be some confusion as to why we are thinking of doing this: Most plugins use the plugin directory to store data (actually a lot of files / dirs) Watching all the directories and files created inside the plugin directory puts more burden on the Kubelet in terms of cpu / memory consumption Watching all the directories and files created inside the plugin directory quickly raises the number of fd consumed By limiting the search of the plugins to one directory, we would be severely limiting the ability of the module to unintentionally harm the overall Kubelet. It seems actually pretty easy to enforce, as it's a simple condition. As for not everyone reading code / readme, we can add the documentation to the gRPC interface if this is a concern. In terms of flexibility, is there a real need to create a plugin in a lower directory? Is there a plugin that has so many sockets that a single directory underneath the plugin directory isn't enough? Naively it seems to me that is enough for the different plugins we enable.
ping on the above :)
Naively it seems to me that $ is enough for the different plugins we enable. i can not speak from plugin developer, but both below looks reasonable to me , or as long as we can communicate to plugin developers clearly and effectively (gRPC interface documentation is needed), the directory depth restriction is reasonable to me.
I like . However, why not put plugintype one level up as . For instance
Is the suggestion to hard code the number of levels that the recursion algorithm will search to 3? Or just expect a socket only at a hard coded level (e.g. socket must be nested 3 deep)? Copying my comment from for consistency: Another option I can think of: recursively scanning but before scanning a directory check if it is mounted ().

This will be backwards compatible and will prevent scanning of volume data Thoughts?
i like this idea better if it is always the case that: volume data is mounted, we can modify the watcher code to not watch the mounted directory in the first place.
Skipping mounted dir along with restructuring search path to known (shallow) dir sounds like a good plan. Though with the new suggested paths, sockets will be placed at a location where mounts should not exist.
I think that would be a safe assumption. The plugin directory is only used as a target path for global mounts.
Ephemeral volume types may not need to mount, it could just be a symlink or plain directory. Can we assume any ephemeral volume CSI driver won't implement global mount?
Yes, in-tree ephemeral volumes today don't do a global mount so the mounts directory is which is not the plugin dir. So no issue there. For CSI the will be the global mount path (this plugin dir). Emphermal volumes shouldn't implement call -- but nothing prevents them from doing so. Maybe let's start with just checking for mount and ignoring mount directories. If it is still an issue we can consider adding a configurable black list of paths. Thoughts?
That sounds good. Also, putting the socket files in a specific location that does not clash with mounts will also help (i.e. )
/milestone v1.13 /priority important-soon
I just want to point out that one of the issue we have with recursively watching all directories is that it will significantly increase the amount of fd consumed. Given that plugins actually store data there and in a dynamic way my fear is that it will exhaust the available fds for no real good reasons. Limiting the depth sounds like a reasonable solution to that issue.
btw, because of this verbosity in my cluster writes 5G logs every day and filled up my rootfs (which is small). Is there any way to decrease this verbosity?
We can do both. Don't search mounted dirs, and max depth of search. Who is working on this?
That's pretty bad, can you open another issue to track that.
we have a PR in flight discussing this issue (recursive scanning) here: Thinking more about this, the fact that too much logs is generated is clearly linked to this specific issue and how we process plugins.

In any case I'll be fixing that issues too
how confident are we in getting this fixed by Code freeze tomorrow? 5pm PST
I expect things to wind up today, changes in the above PR aren't very contentious and don't involve significant changes


Question: LogSSHResult(result) if result.Code != 0 || err != nil { <del> return nil, fmt.Errorf("failed running %q: %v (exit code %d)", cmd, err, result.Code) </del> <ins> return nil, fmt.Errorf("failed running %q: %v (exit code %d, stderr %v)", cmd, err, result.Code, result.Stderr) </ins> } return &result, nil

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: e2e test "[sig-network] Network [It] should set TCP CLOSEWAIT timeout" requires the test target OS supports /proc/net/nfconntrack, but Ubuntu 16.04 doesn't support it. Then the test fails today like the following: What you expected to happen: The test succeeded or was skipped. How to reproduce it (as minimally and precisely as possible): Just kick Anything else we need to know?: The existence of /proc/net/nfconntrack depends on the Linux kernel config CONFIGNFCONNTRACKPROCFS, but that is disabled on Ubuntu 16.04. In addition, the test is skipped if the node number is less than 2. So I guess that is one of reasons this issue was not present until today. Environment: Kubernetes version (use ): v1.11.3 Cloud provider or hardware configuration: OpenStack OS (e.g. from /etc/os-release): Ubuntu 16.04.4 LTS Kernel (e.g. ): 4.4.0-134-generic Install tools: kubeadm Others:
/sig network
Some kube-proxy functionalities depends on conntrack, is it possible to run a conformance cluster without having conntrack enable?


Question: } func (ds *dockerService) performPlatformSpecificContainerForContainer(containerID string) (errors []error) { <del> if cleanupInfo, present := ds.containerCleanupInfos[containerID]; present { </del> <ins> if cleanupInfo, present := ds.getContainerCleanupInfo(containerID); present { </ins> errors = ds.performPlatformSpecificContainerCleanupAndLogErrors(containerID, cleanupInfo) if len(errors) == 0 { <del> delete(ds.containerCleanupInfos, containerID) </del> <ins> ds.clearContainerCleanupInfo(containerID) </ins> } }

What happened: Windows kubelet panic occasionally on nodes which have high volume of pods run and delete. Full logs: Preliminary investigate result: How to reproduce it (as minimally and precisely as possible): Run high volume of pods and delete them when they completed in Windows nodes. Environment: Kubernetes version (use ): 1.17.7 Cloud provider or hardware configuration: Azure AKS OS (e.g: ): Windows Server 2019 Kernel (e.g. ): 10.0..1282
/assign I would like to give this a shot.
/remove-sig windows I think this is not a windows specific problem. /sig node /area kubelet
Hi! Thanks for your report, I have filed to address this issue.
This issue is a bit urgent for our situation, it occurs every few days in production clusters. Seems the PR is pending a few days, is it possible to increase the priority to merge it trunk and 1.17/1.18 branch? Thanks
I understand how urgent this issue is, but I am not privileged to do anything further :(
Do you know anyone who can help?
looks like a simple fix, add a lock for , I could help.
There is already a fix by , the problem is the PR is pending, not sure if you or someone can increase the priority to merge it trunk and 1.17/1.18 branch asap. Our AKS clusters are affecting by this frequently.
Thanks guys.


Question: printLabelsMultilineWithIndent(w, "  ", "Labels", "t", pvc.Labels, sets.NewString()) printLabelsMultilineWithIndent(w, "  ", "Annotations", "t", pvc.Annotations, sets.NewString()) if capacity, ok := pvc.Spec.Resources.Requests[api.ResourceStorage]; ok { <del> w.Write(LEVEL_1, "Capacity:t%sn", capacity) </del> <ins> w.Write(LEVEL_1, "Capacity:t%sn", capacity.String()) </ins> } else { w.Write(LEVEL_1, "Capacity:t%sn", "<default>") }

Is this a BUG REPORT or FEATURE REQUEST? (choose one): Bug Kubernetes version (use ): master branch What happened: Here's the output when we print as string. What you expected to happen: print How to reproduce it (as minimally and precisely as possible): /assign
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
/sig api-machinery
/king bug
Here's the output I get with with a test PVC: and :
Ditto, not seeing a repro against master head:
please try :).
Ah, brainfart, I thought this was complaining about . Yes, indeed is returning: This should be fixed for 1.7.
Will let add approved-for-milestone
I'm assuming you're working on this?
, yes, the PR is here, waiting for 1.7 milestone to merge :).
, thanks for your verification :). I updated title to include "statefulset".


Question: **-bindaddress**="0.0.0.0" The address for the proxy server to serve on (set to 0.0.0.0 or "" for all interfaces) <del> **-configfile**="/tmp/proxy_config" Configuration file for the proxy </del> **-etcd_servers**=[] List of etcd servers to watch (http://ip:port), comma separated (optional)

It's not well tested nor widely used; so it should either be removed or have tests and uses .