Question: <ins> /* Copyright 2018 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package buffered import ( "testing" "time" "github.com/stretchr/testify/require" "k8s.io/apimachinery/pkg/util/wait" auditinternal "k8s.io/apiserver/pkg/apis/audit" "k8s.io/apiserver/pkg/audit" ) var ( closedStopCh = func() <-chan struct{} { ch := make(chan struct{}) close(ch) return ch }() infiniteTimeCh <-chan time.Time = make(chan time.Time) closedTimeCh                    = func() <-chan time.Time { ch := make(chan time.Time) close(ch) return ch }() ) func newEvents(number int) []*auditinternal.Event { events := make([]*auditinternal.Event, number) for i := range events { events[i] = &auditinternal.Event{} } return events } func TestBufferedBackendCollectEvents(t *testing.T) { config := NewDefaultBatchConfig() testCases := []struct { desc          string timer         <-chan time.Time stopCh        <-chan struct{} numEvents     int wantBatchSize int }{ { desc:          "max batch size encountered", timer:         infiniteTimeCh, stopCh:        wait.NeverStop, numEvents:     config.MaxBatchSize + 1, wantBatchSize: config.MaxBatchSize, }, { desc:   "timer expired", timer:  closedTimeCh, stopCh: wait.NeverStop, }, { desc:   "chanel closed", timer:  infiniteTimeCh, stopCh: closedStopCh, }, } for _, tc := range testCases { tc := tc t.Run(tc.desc, func(t *testing.T) { t.Parallel() backend := NewBackend(&fakeBackend{}, config).(*bufferedBackend) backend.ProcessEvents(newEvents(tc.numEvents)...) batch := backend.collectEvents(tc.timer, tc.stopCh) require.Equal(t, tc.wantBatchSize, len(batch), "unexpected batch size") }) } } func TestBufferedBackendProcessEventsAfterStop(t *testing.T) { t.Parallel() backend := NewBackend(&fakeBackend{}, NewDefaultBatchConfig()).(*bufferedBackend) backend.Run(closedStopCh) backend.Shutdown() backend.ProcessEvents(newEvents(1)...) batch := backend.collectEvents(infiniteTimeCh, wait.NeverStop) require.Equal(t, 0, len(batch), "processed events after the backed has been stopped") } func TestBufferedBackendProcessEventsBufferFull(t *testing.T) { t.Parallel() config := NewDefaultBatchConfig() config.BufferSize = 1 backend := NewBackend(&fakeBackend{}, config).(*bufferedBackend) backend.ProcessEvents(newEvents(2)...) require.Equal(t, 1, len(backend.buffer), "buffed contains more elements than it should") } func TestBufferedBackendShutdownWaitsForDelegatedCalls(t *testing.T) { t.Parallel() delegatedCallStartCh := make(chan struct{}) delegatedCallEndCh := make(chan struct{}) delegateBackend := &fakeBackend{ onRequest: func(_ []*auditinternal.Event) { close(delegatedCallStartCh) <-delegatedCallEndCh }, } config := NewDefaultBatchConfig() backend := NewBackend(delegateBackend, config) // Run backend, process events, wait for them to be batched and for delegated call to start. stopCh := make(chan struct{}) backend.Run(stopCh) backend.ProcessEvents(newEvents(config.MaxBatchSize)...) <-delegatedCallStartCh // Start shutdown procedure. shutdownEndCh := make(chan struct{}) go func() { close(stopCh) backend.Shutdown() close(shutdownEndCh) }() // Wait for some time and then check whether Shutdown has exited. Can give false positive, // but never false negative. time.Sleep(100 * time.Millisecond) select { case <-shutdownEndCh: t.Fatalf("Shutdown exited before delegated call ended") default: } // Wait for Shutdown to exit after delegated call has exited. close(delegatedCallEndCh) <-shutdownEndCh } type fakeBackend struct { onRequest func(events []*auditinternal.Event) } var _ audit.Backend = &fakeBackend{} func (b *fakeBackend) Run(stopCh <-chan struct{}) error { return nil } func (b *fakeBackend) Shutdown() { return } func (b *fakeBackend) ProcessEvents(ev ...*auditinternal.Event) { if b.onRequest != nil { b.onRequest(ev) } } </ins>

We should introduce a buffer (similar to what we have in the webhook output backend) to the log file output backend. Audit events should be queued on a relatively large buffer chanel, and an output go routine should grab events off the queue and write them to disk. If an audit event is output when the channel is full, it should be dropped (this should be counted by a prometheus metric, and an e2e test should ensure that the count is below some threshold). This behavior should be optional (configurable) /cc is tracking the regression
Naturally os will cache , these cached data will be written to disk either by fsync or pdflush periodically. What do you think is the most time-costing procedure in audit?
The problem is not with the time consumed. The problem is that currently write is performed synchronously with processing the request, which slows the request down, no matter how well caching works
The problem is that currently write is performed synchronously with processing the request Sure synchronously write does cost time serializing. Introduce buffer mechanism can really reduce api request latency time.
It doesn't matter that much if it happens asynchronously Indeed
Yes, this is a good point to improve performance.
Now I totally comprehend this. I am pleased to implement this buffer.
I'll be happy to review that
audit log is important sensitive information which we should try to make sure it reaches disk at all cost. Does this effort compromises that in favor of optimizing for latency ?
I'll make sure it's configurable and if the security is critical, you can setup synchronous write in your deployment. Having said that, it's critical to be able to accommodate the case where having 100% calls written does not compensate for increased latencies. Does it make sense?
thanks my preference would be to keep the default as no buffering when making it configurable. I didnt understand how does one setup "synchronous write in your deployment" and how that will affect audit logs reaching disk. My worry is mainly around a crash where the buffered audit logs dont reach the disk and are lost
if use buffer, audit events lost is inevitable when crash. As like webhook backend, still exist this risk. But to improve apiserver qps, something should comprise. By the way, we still reserve the synchronous write mode, which is configurable.

To make it simple, when offered trade-off, people are more likely to give up audit logs reliability than performance. Makes sense?
i am not against this change. i think what would be best is to make the default as no buffering and enable buffering through a parameter. Also for Enterprises, the trade off is not correct, they would prefer to NOT give up audit logs
I understand what you're suggesting and I don't agree :) Switching to buffering will for sure increase the latency, but will only increase the probability of losing logs, which will never be zero. I'm pretty sure we need to have buffered mode by default. However, since this argument is about making a trade-off and since neither of us represents quorum, let's ask more people for their opinion What do you think?
The default is slightly less important here since audit logging as a whole is not enabled by default. If we do turn it on by default, it becomes more important. That said, my hunch is that we should default to the mode with lower performance impact, but I don't have strong feelings either way.
As long as there are options to turn it on of off (the synchronous write) I'm fine with either default. Still, I'd like to see due diligence towards ensuring low lost rate when using buffered write, and sufficient documentation around caveats of using one or the other. Audit, from my personal experience, is crucial for majority of enterprises and they won't tolerate not even single lost event.
That is not possible, there will always be a situation when an audit event can be lost: sudden power loss, disk/network failure, random bit flip, etc. The questions is whether 10^(-6) is low enough loss probability, or we need to get 10^(-14) by sacrificing performance (arbitrary numbers, just to illustrate my point). Note, that reduced performance can increase the number of timeouts and retries, increasing load and leading to failures and even more lost events.
I think this is an important bug that we're not testing audit logging at scale
Marking it as urgent, since it's blocking
If split into 3 prs, I must do it one by one . Now first one is up .
SGTM Status of this issue: ETA: 1/12/2017 Risks: Complicated fix required
has been reviewed for almost 2 months and has not done.

If I raise 3 prs to replace it, I am still worried whether all of them can be merged. They act as a whole.
Now I'm going to devote my time to it, we'll do it
OK.
In progress, reviewed first PR, waiting for applying changes ETA: Dec 01
Per Exception discussion, this portion of the feature is being removed from the milestone. As such, changing the priority. /priority important-soon /remove-priority critical-urgent
[MILESTONENOTIFIER] Milestone Removed From Issue Important: Code freeze is in effect and only issues with may remain in the v1.9 milestone. <details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details


Question: package azure import ( <ins> "fmt" </ins> "path" "strings" "github.com/Azure/azure-sdk-for-go/services/compute/mgmt/2017-12-01/compute" "github.com/Azure/azure-sdk-for-go/services/storage/mgmt/2017-10-01/storage" "github.com/golang/glog" <ins> "k8s.io/apimachinery/pkg/api/resource" </ins> kwait "k8s.io/apimachinery/pkg/util/wait" <ins> "k8s.io/kubernetes/pkg/volume/util" </ins> ) //ManagedDiskController : managed disk controller struct

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: According to , some volumes(e.g. aws, nfs, ceph) has already supported PV resizing, need to add PV resize support for azure disk What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: PV resize doc is here: Update: For azure disk, there is blocking issue: it does not support disk resize on a Running VM. I will try to work on resizing an azure disk which is not in attached state. May dig into more details later. Environment: Kubernetes version (use ): v1.10.0 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: /sig azure /assign
Hi , is PV size grow feature still in alpha in master now? How do I open this feature, set in kubelet, kube-apiserer and kube-controller-manager config?
I have figured out that should be set in kubelet, kube-apiserer, kube-controller-manager, kube-scheduler config, and also need to enable in apiserver admission control. It works well.
should we go ahead and close this issue?
also file needs to be updated to allow resizing. This plugin provides additional checks for resizing.
For azure disk, there is blocking issue: it does not support disk resize on a Running VM. I would suggest work on first, could you review , it works well in my testing env.
Adding priority important-soon to this issue so that it can stay in 1.10. If that is incorrect, please update. priority important-soon
/priority important-soon
PR has been merged, can this issue be closed?
yup, looks like it's in. Thanks for flagging! /close
you can't close an active issue unless you authored it or you are assigned to it, Can only assign issues to org members and/or repo collaborators.. <detailsIn response to : yup, looks like it's in. Thanks for flagging! /close Instructions for interacting with me using PR comments are available .

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Awww shucks, looks like I don't have magic powers. please close this when you have a chance?
has been merged, while for azure disk work is just in investigation state mainly because there is blocking issue: it does not support disk resize on a Running VM.
could someone move this issue out of milestone v1.10, thanks.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Why was this closed? This is not fixed by that crappy PR. This is the kubernetes repo, and this refers to k8s apis being implemented correctly iyam. The PR that you closed this with does NOT support disk resize on a Running VM. Your words Please reopen this.
pls track this issue for online resize feature:


Question: } } <del> func getHostFieldLabel(apiVersion string) string { switch apiVersion { case "v1beta1", "v1beta2": return "DesiredState.Host" default: return "spec.host" } } </del> // nodeEnumerator allows a cache.Poller to enumerate items in an api.NodeList type nodeEnumerator struct { *api.NodeList

My broke it for v1beta3, then fixed it for v1beta3 and broke it for v1beta1. I'm working on a fix.
Honestly I'm very surprised that more people aren't complaining-- integration test logs are full of noise because of this.
Exact text of error log is "Failed to watch *api.Node: field label not supported: "
cc


Question: RunRC(c, name, ns, "gcr.io/google_containers/pause:go", n) podsLaunched += n glog.Info("Launched %v pods so far...", podsLaunched) <ins> err := DeleteRC(c, ns, name) expectNoError(err) </ins> } }() }

Currently the density tests are ok if all the pods come up eventually. We should verify there are no failure/error events while the pods are coming up. This could indicate issues dealing with the number of pods and should be investigated.
has been merged, and there's already an issue filed for followup actions, so I think we can close this.


Question: mv "${temp_file}" /etc/kubernetes/manifests } <ins> function start-etcd-empty-dir-cleanup-pod { cp "${KUBE_HOME}/kube-manifests/kubernetes/gci-trusty/etcd-empty-dir-cleanup/etcd-empty-dir-cleanup.yaml" "/etc/kubernetes/manifests" } </ins> # Starts etcd server pod (and etcd-events pod if needed). # More specifically, it prepares dirs and files, sets the variable value # in the manifests, and copies them to /etc/kubernetes/manifests.

I came upon a cluster that had had the namespace code thoroughly exercised. No namespaces currently existed, but many had been created and deleted in the past. Its etcd had lots of empty directories under each resource that had held items when those namespaces were alive (secrets, serviceaccounts, pods...). Do we need to be doing something to clean these empty directories up? It seems like this etcd was consuming a lot of memory.
or might know the answer to this question.
Right. We should delete them. It might get tricky for concurrent cases like A checks if dir is empty, B creates, then A deletes it. What does it look like currently in creating and deleting namespace?
How much memory does a directory entry take? Does it really sound possible to run out of memory while snapshotting on account of too many empty directories?
-- have you guys encountered this? I think you run with more namespaces than we do. But maybe yours are mostly populated?
Yes, we've seen it. c.f.
Approximate memory use we saw from empty dirs:
cc
The patch we debated carrying is here: Our hope was to just wait to etcd v3 where this would not be a problem.
can you link to your clean-up script here?
I needed another PR to run the pod automatically on master.


Question: By("deleting host1ROPod") framework.ExpectNoError(podClient.Delete(host1ROPod.Name, api.NewDeleteOptions(0)), "Failed to delete host1ROPod") <ins> By("Test completed successfully, waiting for PD to safely detach") waitForPDDetach(diskName, host0Name) waitForPDDetach(diskName, host1Name) </ins> }) It("Should schedule a pod w/ a readonly PD on two hosts, then remove both gracefully. [Slow]", func() {

PD E2E tests have very aggressive clean up routine that does not wait for volumes to safely detached. After the test completes, the test cleanup routine unilaterally detaches the PD (instead of waiting for k8s to do it cleanly). Although this does not cause the test to fail, it results in errors in the kernel logs: We should modify these tests to wait for k8s to safely detach the disk.


Question: ginkgo.It("should reject a pod with an AppArmor profile", func(ctx context.Context) { status := runAppArmorTest(ctx, f, false, v1.DeprecatedAppArmorBetaProfileRuntimeDefault) <del> expectSoftRejection(status) </del> <ins> expectRejection(status) </ins> }) }) }

/kind failing-test <!-- If this is a flake, please add: /kind flake --/sig node
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Failures look like started when landed


Question: } } <ins> func TestOpenAPIApiextensionsOverlapProtection(t *testing.T) { server := kubeapiservertesting.StartTestServerOrDie(t, nil, nil, framework.SharedEtcd()) defer server.TearDownFn() apiextensionsclient, err := apiextensionsclientset.NewForConfig(server.ClientConfig) if err != nil { t.Fatalf("unexpected error: %v", err) } crdPath, exist, err := getOpenAPIPath(apiextensionsclient, `/apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions/{name}`) if err != nil { t.Fatalf("unexpected error getting CRD OpenAPI path: %v", err) } if !exist { t.Fatalf("unexpected error: apiextensions OpenAPI path doesn't exist") } // Create a CRD that overlaps OpenAPI path with the CRD API crd := &apiextensionsv1beta1.CustomResourceDefinition{ ObjectMeta: metav1.ObjectMeta{ Name:        "customresourcedefinitions.apiextensions.k8s.io", Annotations: map[string]string{"api-approved.kubernetes.io": "unapproved, test-only"}, }, Spec: apiextensionsv1beta1.CustomResourceDefinitionSpec{ Group:   "apiextensions.k8s.io", Version: "v1beta1", Scope:   apiextensionsv1beta1.ClusterScoped, Names: apiextensionsv1beta1.CustomResourceDefinitionNames{ Plural:   "customresourcedefinitions", Singular: "customresourcedefinition", Kind:     "CustomResourceDefinition", ListKind: "CustomResourceDefinitionList", }, Validation: &apiextensionsv1beta1.CustomResourceValidation{ OpenAPIV3Schema: &apiextensionsv1beta1.JSONSchemaProps{ Type: "object", Properties: map[string]apiextensionsv1beta1.JSONSchemaProps{ testApiextensionsOverlapProbeString: {Type: "boolean"}, }, }, }, }, } etcd.CreateTestCRDs(t, apiextensionsclient, false, crd) // Create a probe CRD foo that triggers an OpenAPI spec change if err := triggerSpecUpdateWithProbeCRD(t, apiextensionsclient, "foo"); err != nil { t.Fatalf("unexpected error: %v", err) } // Expect the CRD path to not change path, _, err := getOpenAPIPath(apiextensionsclient, `/apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions/{name}`) if err != nil { t.Fatalf("unexpected error: %v", err) } pathBytes, err := json.Marshal(path) if err != nil { t.Fatalf("unexpected error: %v", err) } crdPathBytes, err := json.Marshal(crdPath) if err != nil { t.Fatalf("unexpected error: %v", err) } if !bytes.Equal(pathBytes, crdPathBytes) { t.Fatalf("expected CRD OpenAPI path to not change, but got different results: want %q, got %q", string(crdPathBytes), string(pathBytes)) } // Expect the orphan definition to be pruned from the spec exist, err = specHasProbe(apiextensionsclient, testApiextensionsOverlapProbeString) if err != nil { t.Fatalf("unexpected error: %v", err) } if exist { t.Fatalf("unexpected error: orphan definition isn't pruned") } // Create a CRD that overlaps OpenAPI definition with the CRD API crd = &apiextensionsv1beta1.CustomResourceDefinition{ ObjectMeta: metav1.ObjectMeta{ Name:        "customresourcedefinitions.apiextensions.apis.pkg.apiextensions-apiserver.k8s.io", Annotations: map[string]string{"api-approved.kubernetes.io": "unapproved, test-only"}, }, Spec: apiextensionsv1beta1.CustomResourceDefinitionSpec{ Group:   "apiextensions.apis.pkg.apiextensions-apiserver.k8s.io", Version: "v1beta1", Scope:   apiextensionsv1beta1.ClusterScoped, Names: apiextensionsv1beta1.CustomResourceDefinitionNames{ Plural:   "customresourcedefinitions", Singular: "customresourcedefinition", Kind:     "CustomResourceDefinition", ListKind: "CustomResourceDefinitionList", }, Validation: &apiextensionsv1beta1.CustomResourceValidation{ OpenAPIV3Schema: &apiextensionsv1beta1.JSONSchemaProps{ Type: "object", Properties: map[string]apiextensionsv1beta1.JSONSchemaProps{ testApiextensionsOverlapProbeString: {Type: "boolean"}, }, }, }, }, } etcd.CreateTestCRDs(t, apiextensionsclient, false, crd) // Create a probe CRD bar that triggers an OpenAPI spec change if err := triggerSpecUpdateWithProbeCRD(t, apiextensionsclient, "bar"); err != nil { t.Fatalf("unexpected error: %v", err) } // Expect the apiextensions definition to not change, since the overlapping definition will get renamed. apiextensionsDefinition, exist, err := getOpenAPIDefinition(apiextensionsclient, `io.k8s.apiextensions-apiserver.pkg.apis.apiextensions.v1beta1.CustomResourceDefinition`) if err != nil { t.Fatalf("unexpected error: %v", err) } if !exist { t.Fatalf("unexpected error: apiextensions definition doesn't exist") } bytes, err := json.Marshal(apiextensionsDefinition) if exist := strings.Contains(string(bytes), testApiextensionsOverlapProbeString); exist { t.Fatalf("unexpected error: apiextensions definition gets overlapped") } } // triggerSpecUpdateWithProbeCRD creates a probe CRD with suffix in name, and waits until // the path and definition for the probe CRD show up in the OpenAPI spec func triggerSpecUpdateWithProbeCRD(t *testing.T, apiextensionsclient *apiextensionsclientset.Clientset, suffix string) error { // Create a probe CRD that triggers OpenAPI spec change name := fmt.Sprintf("integration-test-%s-crd", suffix) kind := fmt.Sprintf("Integration-test-%s-crd", suffix) group := "probe.test.com" crd := &apiextensionsv1beta1.CustomResourceDefinition{ ObjectMeta: metav1.ObjectMeta{Name: name + "s." + group}, Spec: apiextensionsv1beta1.CustomResourceDefinitionSpec{ Group:   group, Version: "v1", Scope:   apiextensionsv1beta1.ClusterScoped, Names: apiextensionsv1beta1.CustomResourceDefinitionNames{ Plural:   name + "s", Singular: name, Kind:     kind, ListKind: kind + "List", }, }, } etcd.CreateTestCRDs(t, apiextensionsclient, false, crd) // Expect the probe CRD path to show up in the OpenAPI spec // TODO(roycaihw): expose response header in rest client and utilize etag here if err := wait.Poll(1*time.Second, wait.ForeverTestTimeout, func() (bool, error) { _, exist, err := getOpenAPIPath(apiextensionsclient, fmt.Sprintf(`/apis/%s/v1/%ss/{name}`, group, name)) if err != nil { return false, err } return exist, nil }); err != nil { return fmt.Errorf("failed to observe probe CRD path in the spec: %v", err) } return nil } func specHasProbe(clientset *apiextensionsclientset.Clientset, probe string) (bool, error) { bs, err := clientset.RESTClient().Get().AbsPath("openapi", "v2").DoRaw() if err != nil { return false, err } return strings.Contains(string(bs), probe), nil } func getOpenAPIPath(clientset *apiextensionsclientset.Clientset, path string) (spec.PathItem, bool, error) { bs, err := clientset.RESTClient().Get().AbsPath("openapi", "v2").DoRaw() if err != nil { return spec.PathItem{}, false, err } s := spec.Swagger{} if err := json.Unmarshal(bs, &s); err != nil { return spec.PathItem{}, false, err } if s.SwaggerProps.Paths == nil { return spec.PathItem{}, false, fmt.Errorf("unexpected empty path") } value, ok := s.SwaggerProps.Paths.Paths[path] return value, ok, nil } func getOpenAPIDefinition(clientset *apiextensionsclientset.Clientset, definition string) (spec.Schema, bool, error) { bs, err := clientset.RESTClient().Get().AbsPath("openapi", "v2").DoRaw() if err != nil { return spec.Schema{}, false, err } s := spec.Swagger{} if err := json.Unmarshal(bs, &s); err != nil { return spec.Schema{}, false, err } if s.SwaggerProps.Definitions == nil { return spec.Schema{}, false, fmt.Errorf("unexpected empty path") } value, ok := s.SwaggerProps.Definitions[definition] return value, ok, nil } </ins> // return the unique endpoint IPs func getEndpointIPs(endpoints *corev1.Endpoints) []string { endpointMap := make(map[string]bool)

What happened: Created a CRD with a group/version/kind that defines a CustomResourceDefinition type overlapping with the in-tree kubernetes type (WARNING: do not create this CRD on a cluster you care about, see ) Note that two definitions for the group/version/kind appear in openapi: What you expected to happen: published openapi only includes the in-tree type, which has precedence /cc /area custom-resources /priority important-soon /sig api-machinery
/cc
haven't looked into the code yet. I'd expect the naming controller / apiextensions server to prevent conflicting CR GVK, but they are unaware of builtin API groups
CRD handling is isolated to the apiextensions server... it does not have information about all other built-in and aggregated APIs
mentioned something he was working on to make the aggregation layer honor APIService priorities (so that later definitions would not be included if something had already defined a GVK) edit: possibly
a naive idea is to reserve k8s groups in a hard code way that sounds good the CRD in defines a but the kubectl results are about . Did I miss something?
oops... fixed. I started with an API type under but the issue only surfaces for specific APIs (due to the generated definition name colliding for some types)
that wouldn't address duplicate GVKs from aggregated APIs
Three kinds of openapi paths/definitions that we want to prevent CRDs from overwriting: CRD API the spec aggregation in apiextensions-apiserver allows overwriting. I will send a fix native APIs well protected. The spec aggregation in kube-aggregator: 1) ignores duplicated paths and renames duplicated definitions; 2) treats kube-apiserver with than apiextensions-apiserver APIs the spec aggregation in kube-aggregator allows overwriting. The spec aggregation in kube-aggregator treats aggregated APIs with than apiextensions-apiserver. I'm thinking about giving apiextensions-apiserver the lowest priority, but then the problem is aggregated APIs can potentially overwrite the CRD API Note that none of the above prevents CRDs from adding new subpaths to existing APIs. E.g. a CRD can add a path to the CRD API, or to a aggregated API if it didn't have one


Question: return newPod, nil }) <del> obj, _, err = r.store.Update(ctx, name, updatedPodInfo, createValidation, updateValidation, false, options) </del> <ins> // Validation should be passed the API kind (EphemeralContainers) rather than the storage kind. obj, _, err = r.store.Update(ctx, name, updatedPodInfo, toEphemeralContainersCreateValidation(createValidation), toEphemeralContainersUpdateValidation(updateValidation), false, options) </ins> if err != nil { return nil, false, err } return ephemeralContainersInPod(obj.(*api.Pod)), false, err } <ins> func toEphemeralContainersCreateValidation(f rest.ValidateObjectFunc) rest.ValidateObjectFunc { return func(ctx context.Context, obj runtime.Object) error { return f(ctx, ephemeralContainersInPod(obj.(*api.Pod))) } } func toEphemeralContainersUpdateValidation(f rest.ValidateObjectUpdateFunc) rest.ValidateObjectUpdateFunc { return func(ctx context.Context, obj, old runtime.Object) error { return f(ctx, ephemeralContainersInPod(obj.(*api.Pod)), ephemeralContainersInPod(old.(*api.Pod))) } } </ins> // Extract the list of Ephemeral Containers from a Pod func ephemeralContainersInPod(pod *api.Pod) *api.EphemeralContainers { ephemeralContainers := pod.Spec.EphemeralContainers

What happened: Adding a validating webhook with resources section of or breaks ability to add Ephemeral Containers as described in What you expected to happen: Debug command should attach to the running pod and allow user to interact with the ephemeral container image. This works when no validating webhook is configured. How to reproduce it (as minimally and precisely as possible): Using a K8 cluster with Ephemeral Containers feature gate enabled and v1.18.3 a validating webhook (contents don't matter except that resources is "pods/") <details</summary</detailsa test pod <details</summary</detailsto add an Ephemeral Container using alpha debug command Error: Anything else we need to know?: When using , can see that the api server appears to be throwing the error when trying to convert the object for the webhook invocation. API returning the 500 error: Environment: Kubernetes version (use ): v1.18.3 Cloud provider or hardware configuration: Minikube OS (e.g: ): MacOS Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig api-machinery
This looks like an issue with the ephemeral container storage. It is inheriting the New function from the pod storage to return a core.Pod{}, but that doesn't match the type the ephemeral storage actually expects. I think the ephemeral storage should probably set up its own New method to return the proper type. Enabling the feature gate for the ephemeral container feature in the webhook admission integration test would exercise this with webhooks as well. /sig node /priority important-longterm /assign
/remove-sig api-machinery
ack, thanks for the report & triage! I'm digging into it now.
I thank what matters for proper typing is that for returns the proper type. This is already implemented for and returns an API object. Here's what I think is happening: gets an update for the subresource. This is translated into an update of the pod spec and handed off to the pod store since ephemeral containers do not have separate storage. update logic triggers admission control for the object with the attributes of the subresource. webhook client attempts to convert the stored object (a ) to the Kind of the ephemeralcontainers attributes ( identified by ), but this conversion doesn't exist.

It is actually possible to convert from to , and this fixes the problem except that: The conversion is lossy, which would presumably be a problem for a mutating admission controller that returned an which could not be converted back into a This would result in the being sent in the instead of the , which is not the behavior we want. We could instead implement in , which allows us to override the Kind associated with . This results in no conversion and the is sent to the admission controller, but this also changes discovery info from: to: This fixes the problem, but I'm not sure what other side effects to expect. I imagine if we publish for this subresource then we should accept in addition to . I feel like I might be missing something in the storage configuration, though. do you know of any other possible fixes I could investigate?
For future reference, here's a trace from the validation dispatch that receives the conversion error:
mutating admission is applied here: validating admission methods (createValidation / updateValidation) is called from inside here: You need to wrap the createValidation / updateValidation methods (which expect an EphemeralContainers object) with methods that take a pod (which is what will pass in) and translate it to an EphemeralContainers object. See ScaleREST#Update methods for examples of this for the Scale subresource. Mutating admission is applied earlier, the only problematic call is validating admission Isn't it? The user is submitting an object... so that's what should be sent.
Thanks that set me straight. I have a working fix for this now, but I'm going to make sure it's covered in the e2e tests before sending it. re: for , it was a comment you made long ago that made me think there was a requirement to the contrary, so I must have misunderstood.
I had a chance to re-test this flow with which contains the fix to Although the original issue appears to be fixed, I am hoping to understand what the end goal is for ephemeral containers and their webhook behaviour. In my test, I enabled my to forward all resources . When running to add an ephemeral container to an existing pod, the only webhook that I received was for , which did contain the updated pod spec. However I was hoping that the addition of an ephemeral container would cause a webhook to trigger that wasn't of type . Maybe of type or ?

Is there any plans to add a webhook trigger for ephemeral containers specifically? API server logs after doing :
Hi Your admission controller should receive a review request for resource subresource with a kind . When I run a test on using the test webhook and the config: I see this request
I see it have had a misconfiguration on my previous test. Sorry about that. Thanks for your work on this! Having ephemeral container support is very useful!
Great! Thanks for trying out the feature while still in alpha and reporting the bug!


Question: # See the OWNERS docs at https://go.k8s.io/owners approvers: <del> - bowei - k82cn </del> <ins> - sig-node-approvers </ins> reviewers: <del> - smarterclayton - ingvagabund - k82cn </del> <ins> - sig-node-reviewers </ins> emeritus_approvers: <ins> - bowei - k82cn </ins> - gmarek <ins> labels: - sig/node </ins>

/kind cleanup /kind documentation /sig scheduling /sig node /sig apps The ownership of controller/nodelifecycle is unclear. The currently has some individual names and no labels, although it gets a sig/apps label from the parent pkg/controller OWNERS. This controller has the following functionality: Add taints based on health of the node (from conditions) Evict Pods based on NoExecute taints. The candidates for long term ownership are: : The ones who introduced it and overall maintain taints and tolerations. : Because it's called "node lifecycle", although this could be a misnomer. : Due to implications to the lifecycle of applications and because they are the main shepherds of kcm. I think it makes the most sense under scheduling, but wanted to gather all stakeholder's feedback.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
cc
cc as original author
I think both scheduling and node can own it, if node is too overwhelmed, sig scheduling makes sense as a home.
Thanks, It would be nice to sort out the ownership. It sounds reasonable for sig-scheduling to own it. /cc
Outcome from SIG Scheduling meeting: As already identified, this controller actually has 2 functionalities. From a technical standpoint, the following ownership makes sense to SIG Scheduling leads: Setting taints based on Node conditions -SIG Node Evicting pods that don't tolerate NoExecute taints -SIG Scheduling This ownership structure can be set-up even before is implemented, as the taint functionality has its own folder. I'll create a PR based on the above. But and please raise any concerns.
+1 on this proposal.
+1 on the proposal - I believe if we were to assign responsibility to who owns node level invariants that apply to workloads, it would be sig-node (they generally own node + pod). pod-safety, for instance, I considered owned by sig-node even though it touches storage, apps, api-machinery, and scheduling because the node owns the pod at the time safety is imposed.


Question: } func (cd *cinderVolume) GetPath() string { <del> name := cinderVolumePluginName return cd.plugin.host.GetPodVolumeDir(cd.podUID, kstrings.EscapeQualifiedNameForDisk(name), cd.volName) </del> <ins> return getPath(cd.podUID, cd.volName, cd.plugin.host) </ins> } type cinderVolumeUnmounter struct {

Is this a BUG REPORT or FEATURE REQUEST?: BUG /kind bug What happened: I am trying to get metrics working. According this there should be support after 1.8 kube. However, I am now using kube 1.10.1 and still I cannot see anything in ? Should these be there or not? What you expected to happen: I except that kubelet should expose prometheus metrics to /metrics path as well. Then it is easy to read with prometheus. How to reproduce it (as minimally and precisely as possible): Create new cinder volume, attach it to pod and try to query result from in kubelet Environment: Kubernetes version (use ): 1.10.1 Cloud provider or hardware configuration: openstack OS (e.g. from /etc/os-release): centos 7 Kernel (e.g. ): Install tools: Others:
cc
/sig storage
Prometheus metrics should be available on default port of Kubelet. Can you try ? If you are on a RBAC enabled cluster (such as openshift) you may have to use token or key to authenticate for fetching metrics.
returns many metrics but there is not things. There is one pod which have cinder volume attached running in that node. So there should be some metrics right?
It appears that Cinder volume plugin does not implement metric interface necessary for getting those stat metrics - This seems like an oversight and is fairly easy to fix. I can help getting the PR merged if you can open it.
/sig openstack
Kubernetes v1.19.16 there is no kubelet_volume* metrics in //metrics how to fix it???


Question: { storageClasses: []*storagev1.StorageClass{ makeDefaultStorageClass(classGold, &modeImmediate), <del> makeDefaultStorageClass(classSilver, &modeImmediate)}, </del> <ins> makeStorageClass(classSilver, &modeImmediate), }, </ins> tests: []controllerTest{ { <del> name:            "15-2 - pvc storage class is not assigned retroactively if there are multiple default storage classes", </del> <ins> name:            "15-3 - pvc storage class is not assigned retroactively if claim is already bound", </ins> initialVolumes:  novolumes, expectedVolumes: novolumes, <del> initialClaims:   newClaimArray("claim15-2", "uid15-2", "1Gi", "", v1.ClaimPending, nil), expectedClaims:  newClaimArray("claim15-2", "uid15-2", "1Gi", "", v1.ClaimPending, nil), </del> <ins> initialClaims:   newClaimArray("claim15-3", "uid15-3", "1Gi", "test", v1.ClaimBound, &classCopper, volume.AnnBoundByController, volume.AnnBindCompleted), expectedClaims:  newClaimArray("claim15-3", "uid15-3", "1Gi", "test", v1.ClaimLost, &classCopper, volume.AnnBoundByController, volume.AnnBindCompleted), </ins> expectedEvents:  noevents, errors:          noerrors, test:            testSyncClaim,

Presumably this would occur when changing the default from one class to another. E.g. Class X exists and is marked default Class Y is created Class Y is marked as default Uh oh!! 2 defaults Class X is marked as not default At the case in point, when there are 2 defaults, we currently do not assign default values to new PVCs. We should change this semantic - while there are two, pick one randomly or heuristically. It's still correct. Example: Suppose that you have one user changing the default and another user creating a PVC at exactly the same time. What happens? Depending on network, CPU scheduling, memory allocations, (etc - totally non-deterministic) the API server might see either of those operations first. The PVC creator will either win the race and get the old default class or it will lose the race and get the new default class. The difference could literally be whose network has a faster path to the API server. My point is that whichever one you pick, as long as it is one of those two defaults, it is a correct choice! Any other FooClass things we do should probably change, too. I think that's IngressClass and RuntimeClass.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/assign
GitHub didn't allow me to assign the following users: Octopusjust. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
picking either sounds reasonable but what if we allow only one default to be set and disallow/warn users if they try to create another default.

I can think of issues with this too, mainly when switching over to a new default by removing an existing class A and then setting B as a default but managing that orphaned period while the transition happens, maybe we can have the heuristic there? I'm sure I'm missing something here.
/assign
GitHub didn't allow me to assign the following users: Octopusjust. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
There is a problem when there is no default storage class for a while and another user creates a PVC. Such PVC won't get the new SC. We're trying to fix it in Until then, having two default storage classes for a brief moment is IMO slightly better, because at least users get an error message and can retry PVC creation. In addition, (in-tree) admission plugin in HA environment could be racy, as different API servers are admitting updates of different StorageClasses in parallel.
/assign
I wouldn't make 2 an error - just pick one.
Ok, I'd take a shot at this. Thanks for clearing things up.
+1 Still, having two default classes for a longer time is IMO bad, there should be a metric + alert if there are 2 or more default SCs for X minutes.
it's not ideal - a metric seems OK
hello there, im new to open source. i don't know how to contribute. can anyone hep me please?
I see this issue was labeled as "good first issue" and I would like to start working on it!
I see there is a pr related to this issue , are you still working on this?
Is this issue still open?
/assign
Yes, there's already a . It's linked to the issue itself.
The PR is specifically for StorageClass, but we also need to audit other FooClass things - they should probably change, too. I think that's IngressClass and RuntimeClass. Someone can open new PRs for these.
Oh, I see the second PR, too.

Will look at them ASAP
/remove-sig network
Do we emit a response header when someone makes a request that ends up setting two IngressClasses as default?
that seems a nice thing to do, I didn't know you can add warnings on admissions plugins but seems implemented it here
Is this issue still open??
StorageClass implementation is in I am not sure about those. I'd suggest opening separate issues. /close
Closing this issue. <detailsIn response to : whichever one you pick, as long as it is one of those two defaults, it is a correct choice! </details


Question: if healthCheckNodePort == 0 { framework.Failf("Service HealthCheck NodePort was not allocated") } <del> nodeIP := pickNodeIP(jig.Client) By("hitting the TCP service's NodePort on " + nodeIP + ":" + fmt.Sprintf("%d", tcpNodePort)) jig.TestReachableHTTP(nodeIP, tcpNodePort, kubeProxyLagTimeout) </del> <ins> // TODO(33957): test localOnly nodePort Services. </ins> By("hitting the TCP service's service port, via its external VIP " + ingressIP + ":" + fmt.Sprintf("%d", svcTcpPort)) jig.TestReachableHTTP(ingressIP, svcTcpPort, kubeProxyLagTimeout) <del> By("reading clientIP using the TCP service's NodePort") content := jig.GetHTTPContent(nodeIP, tcpNodePort, kubeProxyLagTimeout, "/clientip") clientIP := content.String() framework.Logf("ClientIP detected by target pod using NodePort is %s", clientIP) </del> By("reading clientIP using the TCP service's service port via its external VIP") <del> content = jig.GetHTTPContent(ingressIP, svcTcpPort, kubeProxyLagTimeout, "/clientip") clientIP = content.String() </del> <ins> content := jig.GetHTTPContent(ingressIP, svcTcpPort, kubeProxyLagTimeout, "/clientip") clientIP := content.String() </ins> framework.Logf("ClientIP detected by target pod using VIP:SvcPort is %s", clientIP) By("checking if Source IP is preserved") if strings.HasPrefix(clientIP, "10.") {

https://k8s- Failed: [] Services should be able to create services of type LoadBalancer and externalTraffic=localOnly [Slow][Feature:ExternalTrafficLocalOnly] {Kubernetes e2e suite}
https://k8s- Failed: [] Services should be able to create services of type LoadBalancer and externalTraffic=localOnly [Slow][Feature:ExternalTrafficLocalOnly] {Kubernetes e2e suite}
https://k8s- Failed: [] Services should be able to create services of type LoadBalancer and externalTraffic=localOnly [Slow][Feature:ExternalTrafficLocalOnly] {Kubernetes e2e suite}
https://k8s- Failed: [] Services should be able to create services of type LoadBalancer and externalTraffic=localOnly [Slow][Feature:ExternalTrafficLocalOnly] {Kubernetes e2e suite}
https://k8s- Failed: [] Services should be able to create services of type LoadBalancer and externalTraffic=localOnly [Slow][Feature:ExternalTrafficLocalOnly] {Kubernetes e2e suite}
https://k8s- Failed: [] Services should be able to create services of type LoadBalancer and externalTraffic=localOnly [Slow][Feature:ExternalTrafficLocalOnly] {Kubernetes e2e suite}
https://k8s- Failed: [] Services should be able to create services of type LoadBalancer and externalTraffic=localOnly [Slow][Feature:ExternalTrafficLocalOnly] {Kubernetes e2e suite}
https://k8s- Failed: [] Services should be able to create services of type LoadBalancer and externalTraffic=localOnly [Slow][Feature:ExternalTrafficLocalOnly] {Kubernetes e2e suite}
https://k8s- Failed: [] Services should be able to create services of type LoadBalancer and externalTraffic=localOnly [Slow][Feature:ExternalTrafficLocalOnly] {Kubernetes e2e suite}
https://k8s- Failed: [] Services should be able to create services of type LoadBalancer and externalTraffic=localOnly [Slow][Feature:ExternalTrafficLocalOnly] {Kubernetes e2e suite}
https://k8s- Failed: [] Services should be able to create services of type LoadBalancer and externalTraffic=localOnly [Slow][Feature:ExternalTrafficLocalOnly] {Kubernetes e2e suite}
This test started getting extremely flaky on all jobs it runs in at around 22:50

on Sept 30. See . I think was the culprit. cc
I'm re-writing it in It's hitting external-ip:nodePort of a loadbalanced service with OnlyLocal annotation. Not sure why, but yeah that will only pass if there is a local endpoint. I'm inclined to let it flake till the pr above goes in, if you prefer I can set a spot fix.
https://k8s- Failed: [] Services should be able to create services of type LoadBalancer and externalTraffic=localOnly [Slow][Feature:ExternalTrafficLocalOnly] {Kubernetes e2e suite}
https://k8s- Failed: [] Services should be able to create services of type LoadBalancer and externalTraffic=localOnly [Slow][Feature:ExternalTrafficLocalOnly] {Kubernetes e2e suite}
https://k8s- Failed: [] Services should be able to create services of type LoadBalancer and externalTraffic=localOnly [Slow][Feature:ExternalTrafficLocalOnly] {Kubernetes e2e suite}
https://k8s- Failed: [] Services should be able to create services of type LoadBalancer and externalTraffic=localOnly [Slow][Feature:ExternalTrafficLocalOnly] {Kubernetes e2e suite}
https://k8s- Failed: [] Services should be able to create services of type LoadBalancer and externalTraffic=localOnly [Slow][Feature:ExternalTrafficLocalOnly] {Kubernetes e2e suite}
https://k8s- Failed: [] Services should be able to create services of type LoadBalancer and externalTraffic=localOnly [Slow][Feature:ExternalTrafficLocalOnly] {Kubernetes e2e suite}
https://k8s- Failed: [] Services should be able to create services of type LoadBalancer and externalTraffic=localOnly [Slow][Feature:ExternalTrafficLocalOnly] {Kubernetes e2e suite}
https://k8s- Failed: [] Services should be able to create services of type LoadBalancer and externalTraffic=localOnly [Slow][Feature:ExternalTrafficLocalOnly] {Kubernetes e2e suite}
https://k8s- Failed: [] Services should be able to create services of type LoadBalancer and externalTraffic=localOnly [Slow][Feature:ExternalTrafficLocalOnly] {Kubernetes e2e suite}
https://k8s- Failed: [] Services should be able to create services of type LoadBalancer and

externalTraffic=localOnly [Slow][Feature:ExternalTrafficLocalOnly] {Kubernetes e2e suite}
https://k8s- Failed: [] Services should be able to create services of type LoadBalancer and externalTraffic=localOnly [Slow][Feature:ExternalTrafficLocalOnly] {Kubernetes e2e suite}
https://k8s- Failed: [] Services should be able to create services of type LoadBalancer and externalTraffic=localOnly [Slow][Feature:ExternalTrafficLocalOnly] {Kubernetes e2e suite}
https://k8s- Failed: [] Services should be able to create services of type LoadBalancer and externalTraffic=localOnly [Slow][Feature:ExternalTrafficLocalOnly] {Kubernetes e2e suite}
https://k8s- Failed: [] Services should be able to create services of type LoadBalancer and externalTraffic=localOnly [Slow][Feature:ExternalTrafficLocalOnly] {Kubernetes e2e suite}
https://k8s- Failed: [] Services should be able to create services of type LoadBalancer and externalTraffic=localOnly [Slow][Feature:ExternalTrafficLocalOnly] {Kubernetes e2e suite}
https://k8s- Failed: [] Services should be able to create services of type LoadBalancer and externalTraffic=localOnly [Slow][Feature:ExternalTrafficLocalOnly] {Kubernetes e2e suite}
https://k8s- Failed: [] Services should be able to create services of type LoadBalancer and externalTraffic=localOnly [Slow][Feature:ExternalTrafficLocalOnly] {Kubernetes e2e suite}
https://k8s- Failed: [] Services should be able to create services of type LoadBalancer and externalTraffic=localOnly [Slow][Feature:ExternalTrafficLocalOnly] {Kubernetes e2e suite}
https://k8s- Failed: [] Services should be able to create services of type LoadBalancer and externalTraffic=localOnly [Slow][Feature:ExternalTrafficLocalOnly] {Kubernetes e2e suite}
given how noisy this thing is, if you have time to add a spot fix, I'd appreciate it.
https://k8s- Failed: [] Services should be able to create services of type LoadBalancer and externalTraffic=localOnly [Slow][Feature:ExternalTrafficLocalOnly] {Kubernetes e2e suite}
https://k8s- Failed: [] Services should be able to create services of

type LoadBalancer and externalTraffic=localOnly [Slow][Feature:ExternalTrafficLocalOnly] {Kubernetes e2e suite}
https://k8s- Failed: [] Services should be able to create services of type LoadBalancer and externalTraffic=localOnly [Slow][Feature:ExternalTrafficLocalOnly] {Kubernetes e2e suite}
https://k8s- Failed: [] Services should be able to create services of type LoadBalancer and externalTraffic=localOnly [Slow][Feature:ExternalTrafficLocalOnly] {Kubernetes e2e suite}
https://k8s- Failed: [] Services should be able to create services of type LoadBalancer and externalTraffic=localOnly [Slow][Feature:ExternalTrafficLocalOnly] {Kubernetes e2e suite}
https://k8s- Failed: [] Services should be able to create services of type LoadBalancer and externalTraffic=localOnly [Slow][Feature:ExternalTrafficLocalOnly] {Kubernetes e2e suite}
https://k8s- Failed: [] Services should be able to create services of type LoadBalancer and externalTraffic=localOnly [Slow][Feature:ExternalTrafficLocalOnly] {Kubernetes e2e suite}
https://k8s- Failed: [] Services should be able to create services of type LoadBalancer and externalTraffic=localOnly [Slow][Feature:ExternalTrafficLocalOnly] {Kubernetes e2e suite}
https://k8s- Failed: [] Services should be able to create services of type LoadBalancer and externalTraffic=localOnly [Slow][Feature:ExternalTrafficLocalOnly] {Kubernetes e2e suite}
https://k8s- Failed: [] Services should be able to create services of type LoadBalancer and externalTraffic=localOnly [Slow][Feature:ExternalTrafficLocalOnly] {Kubernetes e2e suite}
https://k8s- Failed: [] Services should be able to create services of type LoadBalancer and externalTraffic=localOnly [Slow][Feature:ExternalTrafficLocalOnly] {Kubernetes e2e suite}
https://k8s- Failed: [] Services should be able to create services of type LoadBalancer and externalTraffic=localOnly [Slow][Feature:ExternalTrafficLocalOnly] {Kubernetes e2e suite}
https://k8s- Failed: [] Services should be able to create services of type LoadBalancer and externalTraffic=localOnly [Slow][Feature:ExternalTrafficLocalOnly]

{Kubernetes e2e suite}
I will send one tomorrow, in the meanwhile I'll un-assign you (not that it helps because you already commented on it)
https://k8s- Failed: [] Services should be able to create services of type LoadBalancer and externalTraffic=localOnly [Slow][Feature:ExternalTrafficLocalOnly] {Kubernetes e2e suite}
https://k8s- Failed: [] Services should be able to create services of type LoadBalancer and externalTraffic=localOnly [Slow][Feature:ExternalTrafficLocalOnly] {Kubernetes e2e suite}
https://k8s- Failed: [] Services should be able to create services of type LoadBalancer and externalTraffic=localOnly [Slow][Feature:ExternalTrafficLocalOnly] {Kubernetes e2e suite}
https://k8s- Failed: [] Services should be able to create services of type LoadBalancer and externalTraffic=localOnly [Slow][Feature:ExternalTrafficLocalOnly] {Kubernetes e2e suite}
https://k8s- Failed: [] Services should be able to create services of type LoadBalancer and externalTraffic=localOnly [Slow][Feature:ExternalTrafficLocalOnly] {Kubernetes e2e suite}
https://k8s- Failed: [] Services should be able to create services of type LoadBalancer and externalTraffic=localOnly [Slow][Feature:ExternalTrafficLocalOnly] {Kubernetes e2e suite}
https://k8s- Failed: [] Services should be able to create services of type LoadBalancer and externalTraffic=localOnly [Slow][Feature:ExternalTrafficLocalOnly] {Kubernetes e2e suite}
https://k8s- Failed: [] Services should be able to create services of type LoadBalancer and externalTraffic=localOnly [Slow][Feature:ExternalTrafficLocalOnly] {Kubernetes e2e suite}
https://k8s- Failed: [] Services should be able to create services of type LoadBalancer and externalTraffic=localOnly [Slow][Feature:ExternalTrafficLocalOnly] {Kubernetes e2e suite}
https://k8s- Failed: [] Services should be able to create services of type LoadBalancer and externalTraffic=localOnly [Slow][Feature:ExternalTrafficLocalOnly] {Kubernetes e2e suite}
https://k8s- Failed: [] Services should be able to create services of type LoadBalancer and externalTraffic=localOnly

[Slow][Feature:ExternalTrafficLocalOnly] {Kubernetes e2e suite}
https://k8s- Failed: [] Services should be able to create services of type LoadBalancer and externalTraffic=localOnly [Slow][Feature:ExternalTrafficLocalOnly] {Kubernetes e2e suite}
https://k8s- Failed: [] Services should be able to create services of type LoadBalancer and externalTraffic=localOnly [Slow][Feature:ExternalTrafficLocalOnly] {Kubernetes e2e suite}
https://k8s- Failed: [] Services should be able to create services of type LoadBalancer and externalTraffic=localOnly [Slow][Feature:ExternalTrafficLocalOnly] {Kubernetes e2e suite}
https://k8s- Failed: [] Services should be able to create services of type LoadBalancer and externalTraffic=localOnly [Slow][Feature:ExternalTrafficLocalOnly] {Kubernetes e2e suite}
https://k8s- Failed: [] Services should be able to create services of type LoadBalancer and externalTraffic=localOnly [Slow][Feature:ExternalTrafficLocalOnly] {Kubernetes e2e suite}
https://k8s- Failed: [] Services should be able to create services of type LoadBalancer and externalTraffic=localOnly [Slow][Feature:ExternalTrafficLocalOnly] {Kubernetes e2e suite}
https://k8s- Failed: [] Services should be able to create services of type LoadBalancer and externalTraffic=localOnly [Slow][Feature:ExternalTrafficLocalOnly] {Kubernetes e2e suite}
https://k8s- Failed: [] Services should be able to create services of type LoadBalancer and externalTraffic=localOnly [Slow][Feature:ExternalTrafficLocalOnly] {Kubernetes e2e suite}
https://k8s- Failed: [] Services should be able to create services of type LoadBalancer and externalTraffic=localOnly [Slow][Feature:ExternalTrafficLocalOnly] {Kubernetes e2e suite}
https://k8s- Failed: [] Services should be able to create services of type LoadBalancer and externalTraffic=localOnly [Slow][Feature:ExternalTrafficLocalOnly] {Kubernetes e2e suite}
https://k8s- Failed: [] Services should be able to create services of type LoadBalancer and externalTraffic=localOnly [Slow][Feature:ExternalTrafficLocalOnly] {Kubernetes e2e

suite}
https://k8s- Failed: [] Services should be able to create services of type LoadBalancer and externalTraffic=localOnly [Slow][Feature:ExternalTrafficLocalOnly] {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] Services should be able to create services of type LoadBalancer and externalTraffic=localOnly [Slow][Feature:ExternalTrafficLocalOnly] {Kubernetes e2e suite}
https://k8s- Failed: [] Services should be able to create services of type LoadBalancer and externalTraffic=localOnly [Slow][Feature:ExternalTrafficLocalOnly] {Kubernetes e2e suite}
https://k8s- Failed: [] Services should be able to create services of type LoadBalancer and externalTraffic=localOnly [Slow][Feature:ExternalTrafficLocalOnly] {Kubernetes e2e suite}

Some cloudproviders like gce simply forward packets sent to the public lb vip to a vm. The packet shows up at vm's eth0 with a destination ip=lb vip, which the vm accepts as local. Other cloudproviders, like AWS forward packets to the nodePort of the service. The packet shows up at the vm's eth0 with a destination ip=vm ip. Current implementation of source ip preservation targets the former, we need to figure out how to handle this cross platform before it goes beta.
fixed with


Question: <ins> /* Copyright 2023 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package main import ( "fmt" "runtime" "go.uber.org/automaxprocs/maxprocs" ) func main() { maxprocs.Set() fmt.Print(runtime.GOMAXPROCS(0)) } </ins>

<!-- Please only use this template for submitting enhancement requests --What would you like to be : should detect the appropriate GOMAXPROCS based off of the cpu limit supplied to the pod. Why is this needed: If not, test failures can occur (e.g. ) where GOMAXPROCS defaults to the number of cpus on the machine, while the cpu limit is actually far less. Too many compile/test processes launch, and any timing tests can become flaky. The end result is that running the test jobs on a node with more CPUs can make it flaky when running the test on a node with fewer CPUs cleanly passes. It's possible to set GOMAXPROCS=cpu limits in test jobs to avoid this, but it would be nicer if would autodiscover and set GOMAXPROCS appropriately.
/sig testing /sig k8s-infra
/transfer kubernetes
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign dims
xref:
This is already done, which is great, but for some more info I did some benchmarking of setting GOMAXPROCs vs not: Pretty substantially better in some cases


Question: go nc.podController.Run(wait.NeverStop) go nc.daemonSetController.Run(wait.NeverStop) if nc.internalPodInformer != nil { <del> nc.internalPodInformer.Run(wait.NeverStop) </del> <ins> go nc.internalPodInformer.Run(wait.NeverStop) </ins> } // Incorporate the results of node status pushed from kubelet to master.

While following this to setup a kube-mesos integration env on my laptop, I found the default service account and default token are not created when start the cluster. As a result, I can not create a pod or deployment in this environment. Reproduce steps: the latest Mesos code, build and start 1 single master with 1 single agent. the latest Kubernetes code, use to build. the following commands to start apiserver, controller-manager and scheduler respectively: all the components are started, run command and see that there is no default token created for the default namespace. At this moment, run "kubectl create -f " will see the failure of creating a pod.
This introduced to node controller and now when we start a node controller, if is not nil, we will start this Informer controller as well. But the start of the will block main thread from starting other controllers later in the main thread. we can see that in the kube-mesos integration, we started some other controllers, including serviceaccount controller, after we started node controller. I will submit a patch later to run the in a go routine to solve the issue.


Question: return nil, scoreStatus.AsError() } <del> if klog.V(10).Enabled() { for plugin, nodeScoreList := range scoresMap { for _, nodeScore := range nodeScoreList { klog.InfoS("Plugin scored node for pod", "pod", klog.KObj(pod), "plugin", plugin, "node", nodeScore.Name, "score", nodeScore.Score) } } } </del> // Summarize all scores. result := make(framework.NodeScoreList, 0, len(nodes))

/sig scheduling A way to check the scoring of each node to figure out why a node has a high scheduling priority. proposal 1: add some debug level logs to Kube-scheduler, so we can know node score on each scoring step. proposal 2: a simple way to decrease the priority of one node(in my case a node with more than 200 pods is with the highest scheduling priority for an unknown reason) proposal 3:? All new pods are scheduled to one of five nodes in my cluster. I have checked both CPU/memory requested usage and all five nodes are bare metal and running about 150 pods or 200 pods. Pods on node request 80-93% of the memory. We have a similar issue as Currently, I think is the easiest way.
This is useful. how do you suggest we log this information? note that there are two dimensions for each pod: plugin and node score. Also, we need to limit the log length, we can't simply log the score for each node in clusters with large number of nodes.
I prefer to log only the top 5 or 10 node scores in the log. (I have not read the code yet., will dig into it later.)
We actually merged something like this that I wrote in , but had to revert it because of performance issue in large clusters. That could be a good starting point My general approach was to 1) invert the already available to map (rather than , which it currently does) and then 2) log the top 3 highest scoring plugins for each node However discovered performance issues after this in (we believe due to the many writes this process took) and it was reverted in I think we decided that some performance hit is reasonable, especially given log levels =4, but this was ~50% drop.
thanks for the information. So with current kube-scheduler, I can set log level to 10 to get every node score for each plugin. It helps in some cases.
yes, that's the current case. Though creates a lot of other noise in the logs, so I think we agreed that it is okay for us to add this kind of summary at a lower level like
I think we can use this chance to standardize the logging logic to dump scores dynamically depending on log level: for logging level ranged in , dump the scores for topM nodes/plugins.

And make it as performant as possible. for logging level ranged in , dump the scores for topN nodes, and probably show all plugins for logging level = 10, dump the scores for all nodes/plugins.
/assign
/triage accepted
/reopen Some problems that were mentioned by are not fixed yet. /cc
Reopened this issue. <detailsIn response to : /reopen After taking a glance, I can spot several issues: A consensus on the value and . This is not fully discussed yet. Further dig into the performance result. The test runs on v=1 and v=4, but actually they're the same if it's running in the context of integration framework. I'd expect to see the result comparing before/after on v=4 and v=6. Further review on the code, like adding UTs and some logic nits. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Since is merged to revert the change,
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/remove-lifecycle rotten
Just FYI: this may be related to in the future.

In , we cannot add all scores to the event message (because the event msg has the limit on the length), so I am thinking we need to sort node-scores by total score and add the score of top N nodes to the event msg. And, if we do so, we can easily achieve to log the top M Node's score by using this sorted node-scores.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/lifecycle rotten
/remove-lifecycle rotten
/remove-lifecycle rotten
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue with Mark this issue as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close not-planned [1]:
Closing this issue, marking it as "Not Planned". <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue with Mark this issue as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: "encoding/json" "fmt" "io/ioutil" <ins> "reflect" </ins> "strings" "testing"

This tree has a commit that demonstrates two obviously different objects that should not pass the semantic DeepEqual but do.
What shall we do with this issue for triage? Labels? Priority?
P2 for when he returns , Satnam Singh wrote:
OK I am now sufficiently worried about this. Fixes incoming.


Question: import ( "context" <del> "errors" </del> "fmt" <del> "io" "net" </del> "time" "google.golang.org/grpc" grpccodes "google.golang.org/grpc/codes" <del> "google.golang.org/grpc/credentials/insecure" </del> grpcstatus "google.golang.org/grpc/status" "k8s.io/klog/v2"

See comment for details:
/sig node
/area kubelet
/assign
The Kubernetes project currently lacks enough contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
/triage accepted


Question: if err != nil { return nil, err } <ins> readOnly, err := getReadOnlyFromSpec(spec) if err != nil { return nil, err } </ins> // before it is used in any paths such as socket etc addr := fmt.Sprintf(csiAddrTemplate, pvSource.Driver)

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: The field passed to is never set. In where is a object that is initialized in the method in Notice the field is never set. What you expected to happen: The field should be set to How to reproduce it (as minimally and precisely as possible): Create a CSI volume and set to true. Check the value of in the CSI call. It should be true, but is false. CC Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
I think is only used for mounting pod volume path to container, here(for ), we can set to , just like other volume plugins. I can help to fix this.
What about the case when a user wants to mount a volume for one of containers in a POD as read only, but the actual volume has ReadOnly False? I would like to suggest an alternative solution which passes the value of ReadOnly attribute, taken from POD's spec.
Thank you and for jumping on this! I feel like the other volume plugins have the wrong behavior here. The purpose of the call is to mount the volume. Therefore, the should receive the pod volume mount readonly field. That said, as mentioned , I think the fact that " is used for mounting pod volume path to container" might be hiding a bug: that is never set correctly. I haven't had a chance to validate this but my reading of the code suggests that the is never set. But because the container mount uses , even though the plugin did not get the correct readonly value, the mount ends up getting mounted with the correctly readonly value. Thoughts?
[MILESTONENOTIFIER] Milestone Issue: Up-to-date for process Note: This issue is marked as , and must be updated every 1 day during code freeze. Example update: <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release.

</details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
We discussed the proposed fixes in the Kubernetes CSI implementation meeting today (see ). We decided it would be better for CSI to be consistent with the other plugins (and still broken), and have CSI fixed, along with all the other plugins, whenever we fix the issue (likely post-1.10). Therefore, I will accept fix and we will close Then we can remove the 1.10 tag from this issue and track it independent of the 1.10 release.
Spoke with offline. Details here: Based on that conversation, I don't think there is anything else we need to do here.


Question: } // WarningsOnUpdate returns warnings for the given update. <del> func (customResourceStrategy) WarningsOnUpdate(ctx context.Context, obj, old runtime.Object) []string { return nil </del> <ins> func (a customResourceStrategy) WarningsOnUpdate(ctx context.Context, obj, old runtime.Object) []string { return generateWarningsFromObj(obj, old) </ins> } // GetAttrs returns labels and fields of a given object for filtering purposes.

A client is allowed to add custom finalizers to Custom Resources with a finalizer name that's not fully qualified (e.g. or ). However they're prevented from doing the same on builtin resource types, returning the following error: Invalid value: "": name is neither a standard finalizer name nor is it fully qualified Per , custom finalizer names are expected to be in format "consist of a domain name, a forward slash and the name of the finalizer" (e.g. ). Therefore, I expected the enforcement to be the same for CRs and builtin types. a custom resource on the cluster it to add the edit does not fail the same on a builtin resource (e.g. Namespace or Pod), observe it fails No response Reproducible at least starting from 1.21 and still on latest versions. <details</details<detailsN/A kind cluster </details<detailsN/A - kind cluster accessed via macOS </details<details</details<details</details<details</details
/area api /cc /sig api-machinery
/assign /triage accepted
There is no validation of finalizer in and . Please unassign me if you guys have started to work on it. /assign
it is called from here on custom resources:
lacks validation for finalizer name. I have raised a pr for this. PTAL.


Question: }, expectedElements: []string{"test", "State", "Waiting", "Ready", "True", "Restart Count", "7", "Image", "image"}, }, <ins> //env { container: api.Container{Name: "test", Image: "image", Env: []api.EnvVar{{Name: "envname", Value: "xyz"}}}, status: api.ContainerStatus{ Name:         "test", Ready:        true, RestartCount: 7, }, expectedElements: []string{"test", "State", "Waiting", "Ready", "True", "Restart Count", "7", "Image", "image", "envname", "xyz"}, }, </ins> // Using limits. { container: api.Container{

A "kubectl describe" on a pod ID give a lot of informations about that pod but don't you think it should also print the pod's env var ?


Question: } ctx.paramGVK = paramsGVK ctx.paramDeclType = paramsDeclType <ins> ctx.variables = policy.Spec.Variables </ins> return ctx } <ins> func (c *TypeChecker) compiler(ctx *TypeCheckingContext, typeOverwrite typeOverwrite) (*plugincel.CompositedCompiler, error) { envSet, err := buildEnvSet( /* hasParams */ ctx.paramDeclType != nil, /* hasAuthorizer */ true, typeOverwrite) if err != nil { return nil, err } env, err := plugincel.NewCompositionEnv(plugincel.VariablesTypeName, envSet) if err != nil { return nil, err } compiler := &plugincel.CompositedCompiler{ Compiler:       &typeCheckingCompiler{typeOverwrite: typeOverwrite, compositionEnv: env}, CompositionEnv: env, } return compiler, nil } </ins> // CheckExpression type checks a single expression, given the context func (c *TypeChecker) CheckExpression(ctx *TypeCheckingContext, expression string) TypeCheckingResults { var results TypeCheckingResults for i, gvk := range ctx.gvks { declType := ctx.declTypes[i] <del> // TODO(jiahuif) hasAuthorizer always true for now, will change after expending type checking to all fields. issues, err := c.checkExpression(expression, ctx.paramDeclType != nil, true, typeOverwrite{ </del> <ins> compiler, err := c.compiler(ctx, typeOverwrite{ </ins> object: declType, params: ctx.paramDeclType, }) <del> if issues != nil || err != nil { results = append(results, &TypeCheckingResult{Issues: issues, Err: err, GVK: gvk}) </del> <ins> if err != nil { utilruntime.HandleError(err) continue } options := plugincel.OptionalVariableDeclarations{ HasParams:     ctx.paramDeclType != nil, HasAuthorizer: true, } compiler.CompileAndStoreVariables(convertv1beta1Variables(ctx.variables), options, environment.StoredExpressions) result := compiler.CompileCELExpression(celExpression(expression), options, environment.StoredExpressions) if err := result.Error; err != nil { typeCheckingResult := &TypeCheckingResult{GVK: gvk} if err.Type == apiservercel.ErrorTypeInvalid { typeCheckingResult.Issues = err } else { typeCheckingResult.Err = err } results = append(results, typeCheckingResult) </ins> } } return results } <ins> type celExpression string func (c celExpression) GetExpression() string { return string(c) } func (c celExpression) ReturnTypes() []*cel.Type { return []*cel.Type{cel.AnyType} } </ins> func generateUniqueTypeName(kind string) string { return fmt.Sprintf("%s%d", kind, time.Now().Nanosecond()) }

When I load a policy definition like this The object is updated with the following status While this error is reported, the policy does seem to be fully functioning This is similar to the documented examples No error reported on a valid spec Create a a policy that uses variables Observe the status of the newly created policy No response <details</details<detailsn/a </details<details</details<detailskind </details<details</details<details</details
/sig api-machinery
/assign Oops. Good find. Thank you. /triage accepted


Question: featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, features.KubeletSeparateDiskGC, true) mockCadvisor.EXPECT().ImagesFsInfo().Return(imageFsInfo, nil) <del> mockCadvisor.EXPECT().ContainerFsInfo().Return(imageFsInfo, nil) </del> mockRuntime.EXPECT().ImageFsInfo(ctx).Return(imageFsInfoResponse, nil) provider := newCadvisorStatsProvider(mockCadvisor, &fakeResourceAnalyzer{}, mockRuntime, nil, NewFakeHostStatsProvider())

Test is failing on crio conformance (cgroup v1 / cgroup v2) Test doesn't fail. You can reproduce it on a PR with /test pull-kubernetes-node-crio-e2e. I reverted go bump and that was not the case. master CI <details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/priority critical
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /priority critical </details
failed to get imageFS data
Ah. Yea this confirms it.
looks like the KEP-4191 bump to beta is causing the failure.
/priority critical-urgent /sig node
/assign kwilczynski
CRI-O version and runtime configuration from the test run: Version string (from the log)Configuration
CRI-O is not configured to enable the split image file system per: (an excerpt from the runtime configuration)
The code that is returning the error:
Other logs I see:
Are we planning to get this fixed for v1.31 or revert the promotion of KEP 4191 to beta with Since code freeze is in effect now, we'd need approval from the release team either way. If we choose to revert the graduation to beta, the KEP won't be part of v1.31.
I think we have a bug where we have the feature enabled but the container runtime is not configured.
I am looking at fixing this as soon as possible...
That PR was mostly to find out the issue. I'd like to fix it and we will file a release exception as we are past the freeze.
Since you're planning to file an exception request, I'm removing the KEP from the milestone so that the release team can better track KEPs. If the issue is not fixed and the exception request is not raised, we would still need to revert the KEP back into alpha ().

We should either fix the issue and merge any additional code changes with an exception request or revert the feature gate to alpha as done in the above PR.
/cc
this is actually going to be quite a common use case, as users would most likely not have their runtimes configured initially or even at all (if someone does not care about things like a split or separate filesystem). I wonder how we can reliably detect whether a runtime has support enabled or not. At the moment, the error originates from cAdvisor, where we ask for stats under a specific label, which does not exist. Since the feature is enabled in Kubernetes, we assume that the runtime also has it enabled and send the label that pertains to ContainreFs. There is a features discovery bit to the CRI API, which we already use to divulge details about support for Recursive Read-only Mounts, Supplemental Groups, etc. I wonder if we should also convey support about split filesystem support that way?
/milestone v1.31 Not only stat summary, we can see other similar failures.
So this only fails in the case of crio with feature on but system is not configured. I used the ImgaeFsInFo to detect if the system is split so I am curious if something went wrong there. Containerd is not failing at all with this feature on so we are detecting containers lack of support without issue.
this is easily reproducible if you have CRI-O running without the option set. So, is this what you mean by "system not configured"?
Yes.
I see , is this going to be reverted or are you working on a fix?
/assign kannon92
We have open to solve this. Waiting on exception approval and this to the milestone. Confirmed that these tests work now.


Question: <ins> # SaltStack configuration This is the root of the SaltStack configuration for Kubernetes. A high level overview for the Kubernetes SaltStack configuration can be found [in the docs tree.](../../docs/salt.md) This SaltStack configuration currently applies to default configurations for Debian-on-GCE, Fedora-on-Vagrant, Ubuntu-on-AWS and Ubuntu-on-Azure. (That doesn't mean it can't be made to apply to an arbitrary configuration, but those are only the in-tree OS/IaaS combinations supported today.) As you peruse the configuration, these are shorthanded as `gce`, `vagrant`, `aws`, `azure` in `grains.cloud`; the documentation in this tree uses this same shorthand for convenience. See more: * [pillar](pillar/) * [reactor](reactor/) * [salt](salt/) </ins>

We need to document all the configuration derived by Salt. Ideally this would be a markdown doc that we kept as live documentation within the project. The main reasons to bother: It's hard right now for anyone approaching the directory to get a big picture view of what's is going on, especially across providers. If we ever want to do something disciplined w.r.t. upgrade between (non-dev) versions, we're probably going to need a place to look for the state of the world. Decreasing the velocity with which people can add to Salt configuration will encourage people to delete more Salt configuration. cc


Question: type: DirectoryOrCreate name: mountpoint-dir - hostPath: <del> path: /var/lib/kubelet/plugins </del> <ins> path: /var/lib/kubelet/plugins_registry </ins> type: Directory name: registration-dir

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug This is more of an enhancement. What happened: The Plugin Watcher is setup to trigger on file activities when they occur in the subdirectory that it scans. During startup, the Watcher is passed (from within the Kubelet) the Kubelet's root plugin directory as a location to scan for newly plugins - Because the kubelet's plugins subdirectory is also used by existing in-tree drivers (including csi) as a location to store plugins artifacts, the Plugin Watcher is getting triggered by directory/file activities occurring in that subdirectory that has nothing to do with plugin registration as shown in the output below: What you expected to happen: The Plugin Watcher should not be getting triggered by file/directory activities not related to plugin registration/unregistration. One possibility is to have the Watcher scan one directory below plugins like or something similar How to reproduce it (as minimally and precisely as possible): Run plugin watcher with a CSI driver. cc
/sig node
I think you should only place the plugin socket file in the plugin registration directory but not the plugin data.
agree with we can only place the registration file in the specified path instead of placing all data into it. On the other hand, you can place data into the /var/lib/kubelet/plugins/registration even if registration created, that's a no-end loop.
As I mentioned in issue, existing in-tree volume plugins were already using that directory as a location to store storage operation artifacts (not just csi). Here is a shortened list of driver locations that use /var/lib/kubelet/plugins in their code already: And several more locations. The fix would be to have plugin watcher scan a subdirectory of plugins (say ) instead of plugins directly as it is doing today. The watcher functions as intended. Except now it picks up all directory activities whether related or not to plugin registration.
Well given the fact that the pluginwatcher is beta this is a bit of an issue. My understanding of the "beta" guarantees is that we want to be backwards compatible for all plugins published with the beta API. Meaning that even if we move the registration directory to a new location, we'll need to keep scanning the old directory (to pick up v1beta plugins).

From a 10 000 feet point of view it seems easier to me if we move the in-tree volume plugin's data. Though it might be a lot of work. Another less painful option, but a bit grey is the following: Enforce plugins to end with this suffix is recommended for device plugins (though not enforced) and I believe is a convention on linux. This is a bit grey because we would still be breaking v1beta users but since we're moving a convention from "recommended" to "enforced" it seems "less" breaking to me (and I haven't seen device plugins not using that extension). What do you think ?
I like the suffix idea. It can be a recommendation (which csi is already doing).
Hmm, I've just deployed a new k8s cluster on 1.12.0 and noticed this repeating in the logs on a few of our pods. Just for my own sanity, is this something I should be worried about or can I ignore it until a fix is released?
you can ignore these errors. if it's a recommendation then we'll keep spamming the logs with all the data files. Another thing to keep in mind, it seems like CSI plugins puts a lot of data (and creates a lot of directories). It might be worth considering limiting the recursion to one level. I'm concerned we might exhaust the amount of fds by tracking the plugins data.
yeah CSI keeps track of volume operation metadata in the plugins dir. This probably will not change. I understand this is beta. However, besides CSI and Device plugin, are there any other implementers out there for this feature ? Would it be possible to propose a new location (i.e. <pluginsDir/reg/?
let me know if you open a PR for this as discussed.
thanks for open the issue with detail logs, i can see two problems by looking at logs and code. 1) plugin watcher trying to dial non-socket files. 2) too many logs generated (CREATE and DELETE). I a suggestion to fix of this issue we can fix those problems by: 1) check file mode when handle CREATE event, only attempt to dial file when file mode is socket.

2) increase log verbose level for both 'handleDeleteEvent()' and 'handleCreateEvent()' plugin watcher still has activity when any file create/delete under the directory, but they become light weighted (CREATE file will only trigger file mode check, DELETE file event will trigger plugin searching) i am not sure we want to go further to limit directory depth where plugin can live. it has two drawbacks IMO: 1) it is hard to enforce, not sure everyone will read code/readme. 2) it provides less flexibility to plugins: if want to organize many plugin socket files with directory.
These are definitely good suggestions! It seems like there might be some confusion as to why we are thinking of doing this: Most plugins use the plugin directory to store data (actually a lot of files / dirs) Watching all the directories and files created inside the plugin directory puts more burden on the Kubelet in terms of cpu / memory consumption Watching all the directories and files created inside the plugin directory quickly raises the number of fd consumed By limiting the search of the plugins to one directory, we would be severely limiting the ability of the module to unintentionally harm the overall Kubelet. It seems actually pretty easy to enforce, as it's a simple condition. As for not everyone reading code / readme, we can add the documentation to the gRPC interface if this is a concern. In terms of flexibility, is there a real need to create a plugin in a lower directory? Is there a plugin that has so many sockets that a single directory underneath the plugin directory isn't enough? Naively it seems to me that is enough for the different plugins we enable.
ping on the above :)
Naively it seems to me that $ is enough for the different plugins we enable. i can not speak from plugin developer, but both below looks reasonable to me , or as long as we can communicate to plugin developers clearly and effectively (gRPC interface documentation is needed), the directory depth restriction is reasonable to me.
I like . However, why not put plugintype one level up as . For instance
Is the suggestion to hard code the number of levels that the recursion algorithm will search to 3? Or just expect a socket only at a hard coded level (e.g. socket must be nested 3 deep)? Copying my comment from for consistency: Another option I can think of: recursively scanning but before scanning a directory check if it is mounted ().

This will be backwards compatible and will prevent scanning of volume data Thoughts?
i like this idea better if it is always the case that: volume data is mounted, we can modify the watcher code to not watch the mounted directory in the first place.
Skipping mounted dir along with restructuring search path to known (shallow) dir sounds like a good plan. Though with the new suggested paths, sockets will be placed at a location where mounts should not exist.
I think that would be a safe assumption. The plugin directory is only used as a target path for global mounts.
Ephemeral volume types may not need to mount, it could just be a symlink or plain directory. Can we assume any ephemeral volume CSI driver won't implement global mount?
Yes, in-tree ephemeral volumes today don't do a global mount so the mounts directory is which is not the plugin dir. So no issue there. For CSI the will be the global mount path (this plugin dir). Emphermal volumes shouldn't implement call -- but nothing prevents them from doing so. Maybe let's start with just checking for mount and ignoring mount directories. If it is still an issue we can consider adding a configurable black list of paths. Thoughts?
That sounds good. Also, putting the socket files in a specific location that does not clash with mounts will also help (i.e. )
/milestone v1.13 /priority important-soon
I just want to point out that one of the issue we have with recursively watching all directories is that it will significantly increase the amount of fd consumed. Given that plugins actually store data there and in a dynamic way my fear is that it will exhaust the available fds for no real good reasons. Limiting the depth sounds like a reasonable solution to that issue.
btw, because of this verbosity in my cluster writes 5G logs every day and filled up my rootfs (which is small). Is there any way to decrease this verbosity?
We can do both. Don't search mounted dirs, and max depth of search. Who is working on this?
That's pretty bad, can you open another issue to track that.
we have a PR in flight discussing this issue (recursive scanning) here: Thinking more about this, the fact that too much logs is generated is clearly linked to this specific issue and how we process plugins.

In any case I'll be fixing that issues too
how confident are we in getting this fixed by Code freeze tomorrow? 5pm PST
I expect things to wind up today, changes in the above PR aren't very contentious and don't involve significant changes