Question: ErrCodeResourceVersionConflicts: "resource version conflicts", ErrCodeInvalidObj:               "invalid object", ErrCodeUnreachable:              "server unreachable", <del> ErrCodeTimeout:                  "request timeout", </del> } func NewKeyNotFoundError(key string, rv int64) *StorageError {

various Oct 30 2023 6pm pst No response Looks like different test are failing with connection timeout errors: One of the tests triggered a core dump: No response /sig
are both suffers from pkg/registry and is from compoent-base, which likely same reason from `
/retitle failing-test: pkg/kubelet/eviction TestManagerWithLocalStorageCapacityIsolationOpen
/sig node
TestManagerWithLocalStorageCapacityIsolationOpen only failed in one of the PRs, so I don't think that specific test is the problem.
ok, rename as see fit. just put owning sigs please for triage.
TestStoreDeleteCollectionWithContextCancellation is the one that seems to be failing across all the failing PRs, so let's start there first. /retitle failing-test: TestStoreDeleteCollectionWithContextCancellation /sig api-machinery
triage link at shows this just starting (happening in CI jobs as well)
image used in the unit test job didn't change around the timeframe this was seen ()
looking for jumps in package unit test runtimes when this failure started happening to see if something is taking longer or stressing the system: ! - cc ! - cc ! - cc Could any of those have introduced changes that could be stressing or exhausting listener connections in unit tests?
the PR in question introduced a poll function around an API call that is unit tested with a fake client i.e. the poll is unit tested, but the fake client has reactors that return error or success right away, so in theory the poll should not be stretched to 1 retries with 1 connection attempts per fake client. while testing locally i did not notice any significant time difference after or the prior i can do some more checks to see what's going on.
merged right at the time the flakes were seen DeleteCollection invokes List, which was modified in that PR to cancel more aggressively cc
I can reproduce about a 10-20% flake rate at HEAD with: failures look like: or while testing that package with , I see ~1300-1500 open connections with this:
Let's revert it then. I looked into it and I don't know from a quick glance why it's failing and I don't have time to debug it now.

I opened
Okay, it looks like we need more debugging to open this PR
one optimization to strip some ~40 fake client connections and 2 seconds from a test:
sounds good, pointed at that commit as well
/triage accepted


Question: definitions MUST be published for custom resource definitions. release: v1.16 file: test/e2e/apimachinery/crd_publish_openapi.go <ins> - testname: Discovery, confirm the groupVerion and a resourcefrom each apiGroup codename: '[sig-api-machinery] Discovery should locate the groupVersion and a resource within each APIGroup [Conformance]' description: A resourceList MUST be found for each apiGroup that is retrieved. For each apiGroup the groupVersion MUST equal the groupVersion as reported by the schema. From each resourceList a valid resource MUST be found. release: v1.28 file: test/e2e/apimachinery/discovery.go </ins> - testname: Discovery, confirm the PreferredVersion for each api group codename: '[sig-api-machinery] Discovery should validate PreferredVersion for each APIGroup [Conformance]'

[X] test approval issue : [X] test pr : 3 May 2023 [X] two weeks soak end date : 16 May 2023 [X] test promotion pr : [X] remove endpoint form file According to following two APIsnoop queries, there are still 12 APIResources endpoints that are not tested for Conformance. Using a number of existing e2e test practices a new has been created to address these 12 endpoints. The e2e logs for this test are listed below. This query shows the following apiresources endpoints are hit within a short period of running this e2e test. If a test with these calls gets merged, test coverage will go up by 12 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted
/open For tracking the rest of the processes
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: // GetModules returns all installed kernel modules. func (handle *LinuxKernelHandler) GetModules() ([]string, error) { <ins> // Check whether IPVS required kernel modules are built-in kernelVersionFile := "/proc/sys/kernel/osrelease" b, err := ioutil.ReadFile(kernelVersionFile) if err != nil { glog.Errorf("Failed to read file %s with error %v", kernelVersionFile, err) } kernelVersion := strings.TrimSpace(string(b)) builtinModsFilePath := fmt.Sprintf("/lib/modules/%s/modules.builtin", kernelVersion) b, err = ioutil.ReadFile(builtinModsFilePath) if err != nil { glog.Errorf("Failed to read file %s with error %v", builtinModsFilePath, err) } var bmods []string for _, module := range ipvsModules { if match, _ := regexp.Match(module+".ko", b); match { bmods = append(bmods, module) } } </ins> // Try to load IPVS required kernel modules using modprobe first for _, kmod := range ipvsModules { err := handle.executor.Command("modprobe", "--", kmod).Run()

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Deploying kube-proxy with mode ipvs on a system with the required ipvs support compiled into the kernel rather than as modules causes the kube-proxy deployment to fall back to using iptables. tryIPVSProxy is only checking for the presence of kernel modules and not if the running kernel has the required support compiled in. What you expected to happen: tryIPVSProxy should succeed if either the required modules are found or support is compiled into the running kernel. How to reproduce it (as minimally and precisely as possible): Boot a system using a kernel with ipvs support compiled in rather than as a module (such as the kernel used by and a host without iptables installed Add --config=/etc/kubernetes/kubelet- to the kubelet arguments using the following kubelet- contents: Install using kubeadm with the following config: Check the output of the kube-proxy pod: Anything else we need to know?: From , it should be possible to check for the existence of modules.builtin and if present use that in addition to using modprobe and the contents of /proc/modules. Environment: Kubernetes version (use ): v1.10.2 Cloud provider or hardware configuration: local kvm host OS (e.g. from /etc/os-release): custom linuxkit build Kernel (e.g. ): Linux linuxkit- 4.14.40-linuxkit SMP Thu May 10 10:06:57 UTC 2018 x86_64 Linux Install tools: kubeadm
/sig network
/cc
cc
cc please take a look.


Question: cpu: {{hollow_proxy_millicpu}}m memory: {{hollow_proxy_mem_Ki}}Ki - name: hollow-node-problem-detector <del> image: k8s.gcr.io/node-problem-detector:v0.8.0 </del> <ins> image: k8s.gcr.io/node-problem-detector/node-problem-detector:v0.8.5 </ins> env: - name: NODE_NAME valueFrom:

What happened: In a cluster with NPD addon installed and Linux 5.1+ as the kernel OS of the nodes OOMs are not being captured by the NPD. What you expected to happen: NPD should inform about OOMs by emitting appropriate K8s events. How to reproduce it (as minimally and precisely as possible): Create a K8s cluster with NPD addon and Linux 5.1+ installed on the nodes. Schedule a pod which leaks memory. Try to list events with field after an OOM occurs to see there's none. Anything else we need to know?: Fix is already merged into newest (v0.8.5) NPD release: We're currently waiting for the release of Docker image of NPD v0.8.5 - we can proceed with fixing this in k/k afterwards. After it gets merged to the master branch, it should be then cherry-picked to older releases as well. Environment: Any Kubernetes version with NPD addon installed and Linux as the node OS with the kernel version 5.1 or newer.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scalability /sig node I opened a fixing PR in but it's on hold since the Docker image of NPD v0.8.5 doesn't exist yet.


Question: t.Fatalf("unexpected request: %#vn%#v", req.URL, req) return nil, nil } <del> case "PUT": </del> <ins> case "PATCH": </ins> switch req.URL.Path { case "/namespaces/test/pods/foo": return &http.Response{StatusCode: 200, Body: objBody(codec, &pods.Items[0])}, nil

and work in this pattern: first issuing a "GET" to the API server to retrieve the latest object, then adding the label/annotation to the object, and finally issuing an "PUT" with the modified object. The request will fail if the object stored in the server has been updated between the "GET" and "PUT", even if the update has no conflict with the label/annotate request. In such cases, instead of returning an error, should retry without disturbing the end user. There are two ways to implement this: in kubectl: simply retry the label/annotate process several times if the server rejected the request because the object has been updated. The only exception is when the is set, then kubectl shouldn't retry. the strategicMergePatch instead of calling Update at client-side. PATCH is more robust because PATCH does the "get, modify, update" in the server-side, so there is less chance to have a conflict, and PATCH will retry at the server-side (see ). We need to make some adaptions though: kubectl annotate/label has the , when it's false, kubectl should retrieve the latest object to confirm there is no existing annotation/label of the same key, and then proceed with issuing the patch. if the is set, [UPDATE:] we just need to specify the resourVersion in the patch, then it will be treated as a precondition by the UpdateResource() handler in the API server. I plan to take the first approach as an immediate fix.
Immediate need is to fix the flaky e2e test AIUI.
e2e? do we have an issue out there? is an immediate fix to the flake in test-
If this is for 1.1, it needs to be P0 at this point. Should it be?
Fwiw I use annotations both from cmd line as an admin and from scripts, to experiment with loadbalancer settings. In the case of the former, I've never really had a collision, and in the case of the latter my scripts know how to retry like we retry all other updates.
Thank you When used in scripts, the annotate and label commands have collisions frequently, e.g. in the test-, see . IMO, we shouldn't require users to retry in their scripts if we know there is a simple way to avoid the collisions. Anyway, and are sent to fix this.


Question: docs/man/man1/kubectl-set-image.1 docs/man/man1/kubectl-set-resources.1 docs/man/man1/kubectl-set-selector.1 <ins> docs/man/man1/kubectl-set-serviceaccount.1 </ins> docs/man/man1/kubectl-set-subject.1 docs/man/man1/kubectl-set.1 docs/man/man1/kubectl-stop.1

Is this a BUG REPORT or FEATURE REQUEST? (choose one): FEATURE REQUEST Kubernetes version (use ): HEAD Now that RBAC is going to GA, using ServiceAccounts is getting more and more important. Still a lot of users run their apps/addons/whatever in the SA. In order for the admin to be able to change ServiceAccount easily after an application has been installed (which supposedly is crashlooping due to insufficient privs), a command could exist. I know I can do this: but for new users to Kubernetes, it's non-obvious. Proposed command: Tracking issue: WDYT? This would be a good way for a new contributor to get aquainted with the code by contributing this feature.
Is anyone working on this? If not I would like to take this up.
Nobody is working on this. I would be super happy if you could contribute this feature. Be aware of that there isn't consensus around this proposal yet (I proposed it yesterday, saturday, so people haven't had time to catch up on it) However, I do think that this is in line with , so feel free to start working on this if you want to ;) Also, don't hesitate to ask for help if you don't know how to get started and we'll help out! Thanks!
I'm in favor of this command. ?
Thanks, I will start working on this.
Is the SA on a Pod mutable or immutable? What about DaemonSet and StatefulSet? Do those have SA as well? Is there a way to do this so that it dynamically supports anything with a PodTemplate? Maybe this should be done as a subresource that can be dynamically discovered by kubectl using the discovery service.
Maybe helpful:
Pod is immutable. All the other objects (with the exception of stateful set) are mutable.
That pattern is the same for all the things that set aspects of a pod spec (except image, which is also mutable in pods)
Statefulsets are going to loosen their validation soon I expect.
This sounds good to me. just go ahead, I will help review this. Thanks ~
The outout of this command kubectl get rc/nginx-controller -o=yaml , has service acccount at the following path '' , and the corresponding pod has service account at 'spec.serviceAccountName'.

Both refer to the 'serviceAccountName' field of PodSpec struct but can be made to be different by patching the ReplicationController resource. How does ReplicationController and Pod use serviceAccountName differently? Shouldn't the ReplicationControllerSpec have a serviceAccountName field?
ptal, I think you are more familiar with ReplicationController implementation than me.
ReplicationController contains a pod spec object, which contains a serviceAccountName.
Hi , how to go about writing e2e test cases for this. My development is mostly complete.
I wouldn't mind if you posted the PR now and then we'd see what e2e test makes sense to have. In order to move forward...
what are the next steps?


Question: highOOMScoreAdj: 3, }, { <del> pod:             &critical, </del> <ins> pod:             &clusterCritical, memoryCapacity:  4000000000, lowOOMScoreAdj:  1000, highOOMScoreAdj: 1000, }, { pod:             &nodeCritical, </ins> memoryCapacity:  4000000000, lowOOMScoreAdj:  -997, highOOMScoreAdj: -997,

<!-- Feature requests are unlikely to make progress as an issue. Instead, please suggest enhancements by engaging with SIGs on slack and mailing lists. A proposal that works through the design along with the implications of the change can be opened as a KEP: --I'd like only pods to have low OOM score -997. introduced a change which ensured all the critical priority classes will have a low OOM score -997. While this helped in relating OOM score to priority of pod, it also allows containers in pods to have -997 OOM score, making them one of the last processes to be OOMKilled. By definition pods can be scheduled elsewhere if there is crunch on the node where as pods cannot be rescheduled. This was the reason for to have higher priority value than . So, I think it makes sense for only pods to have low OOM score of -997. /sig node cc
/triage accepted
i think this is reasonable since is higher priority than and this would make a distinction between the two making killable but assured to be rescheduled
Note that oomscore has nothing to do with rescheduling. Eviction looks at priority, oomscore simply controls the killer. I think that strengthens the argument - the point of node-critical is makinng sure it's killed last, but even then that should be a rare, rare case (a memory leak in kube-proxy will blow out the workload, but if we just killed kube-proxy everything would heal). We may want to go back to the original justifications for -99x oomscore in system-node-critical and make sure we're still aligning correctly.


Question: // Check all anti-affinity terms. for _, term := range GetPodAntiAffinityTerms(affinity.PodAntiAffinity) { <del> termMatches, _, err := c.anyPodMatchesPodAffinityTerm(pod, filteredPods, node, &term) </del> <ins> termMatches, _, err := c.anyPodMatchesPodAffinityTerm(pod, filteredPods, nodeInfo, &term) </ins> if err != nil || termMatches { glog.V(10).Infof("Cannot schedule pod %+v onto node %v, because of PodAntiAffinityTerm %v, err: %v", podName(pod), node.Name, term, err)

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --It is not a secret that . In my opinion the main reason for the slowness is the flexibility of the inter-pod affinity/anti-affinity rules. However, in most practical scenarios users do not need all the flexibility. For example, people often do not use arbitrary labels in topology key of the affinity rules. They often use node, zone, or region as the topology. Node in particular is used often. That's why I am proposing a fast path for the case that topology key is Node. Today, the affinity/anti-affinity predicate does not have any distinction for topology keys and in the cluster regardless. The logic should recognize "" and scan only pods on the same node for this topology key. Is this a BUG REPORT or FEATURE REQUEST?: FEATURE REQUEST /kind feature /sig scheduling cc/
ref/
/assign
GitHub didn't allow me to assign the following users: misterikkit. Note that only can be assigned. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc
We found some other simple optimizations that had a significant impact on the affinity predicate. I will upload PRs for those as well.
Reopening this as the performance of anti-affinity still doesn't seem to be satisfactory enough on large clusters (i.e 5k nodes) - and it's causing them to fail to startup. Ref: issue I can help with evaluating any performance improvements if needed.
The change sent improved performance of the predicate, but apparently it is not enough to make our large scale scalability tests happy.
This issue is not about general performance improvment of affinity/anti-affinity. It was to implement a particular improvment which is already implemented. There are other performance improvement issues. I am going to close this one.
Sounds good. Thanks.
How come with kops 1.10.0 I still see all kube-dns pods on the same node because the kube-dns deployment doesn't specify any anti-affinity rule?
I think you are correct. you can double-check with the option to . I have a v1.9 cluster in front of me, and kube-dns has no affinity rules. (I didn't use kops though)


Question: t.Log(err) } <del> time.Sleep(100 * time.Millisecond) </del> <ins> time.Sleep(time.Second) </ins> } }() <del> if err := wait.PollImmediate(100*time.Millisecond, 10*time.Second, func() (done bool, err error) { </del> <ins> if err := wait.PollImmediate(time.Second, time.Minute, func() (done bool, err error) { </ins> obj := kubeClientConfigValue.Load() if obj == nil { return false, nil } kubeClientConfig := kubeClientConfigValue.Load().(*restclient.Config) <del> kubeClient, err := kclient.NewForConfig(kubeClientConfig) </del> <ins> // make a copy so we can mutate it to set GroupVersion and NegotiatedSerializer cfg := *kubeClientConfig cfg.ContentConfig.GroupVersion = &schema.GroupVersion{} cfg.ContentConfig.NegotiatedSerializer = kapi.Codecs privilegedClient, err := restclient.RESTClientFor(&cfg) </ins> if err != nil { // this happens because we race the API server start t.Log(err) return false, nil } <del> if _, err := kubeClient.Discovery().ServerVersion(); err != nil { </del> <ins> // wait for the server to be healthy result := privilegedClient.Get().AbsPath("/healthz").Do() if errResult := result.Error(); errResult != nil { t.Log(errResult) </ins> return false, nil } <del> return true, nil </del> <ins> var status int result.StatusCode(&status) return status == http.StatusOK, nil </ins> }); err != nil { t.Fatal(err) }

Seeing a variety of failures Address already in use failures https://k8s- Stack traces: https://k8s- https://k8s- Creation errors: https://k8s-
cc
Here's another false negative in TestEtcdStoragePath: https://k8s-


Question: "io.k8s.api.policy.v1beta1.PodDisruptionBudgetStatus": { "description": "PodDisruptionBudgetStatus represents information about the status of a PodDisruptionBudget. Status may trail the actual state of a system.", "required": [ <del> "disruptedPods", </del> "disruptionsAllowed", "currentHealthy", "desiredHealthy",

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: PodDisruptionBudget provides inappropriate message and cannot be updated as is required filed. What you expected to happen: works fine for editing labels under . Also, produce correct error message when edit field. How to reproduce it (as minimally and precisely as possible): Then, we will get following error after step-2. When edit , it should provide correct message as .When edit especially labels under metadata, they should be updated without error. Anything else we need to know?: According to the code comment, is valid. Environment: Kubernetes version (use ): build on
/sig cli
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale there is a PR open


Question: return params, nil } <del> func cloneIface(b iscsiDiskMounter, newIface string) error { </del> <ins> func cloneIface(b iscsiDiskMounter) error { </ins> var lastErr error <ins> if b.InitIface == b.Iface { return fmt.Errorf("iscsi: cannot clone iface with same name: %s", b.InitIface) } </ins> // get pre-configured iface records <del> out, err := b.exec.Run("iscsiadm", "-m", "iface", "-I", b.Iface, "-o", "show") </del> <ins> out, err := b.exec.Run("iscsiadm", "-m", "iface", "-I", b.InitIface, "-o", "show") </ins> if err != nil { lastErr = fmt.Errorf("iscsi: failed to show iface records: %s (%v)", string(out), err) return lastErr

What happened: When creating a block volume, the iSCSI storage plugin stores volume-related data in different directories: As a result, some data is not cleaned up once the volume is detached: How to reproduce it (as minimally and precisely as possible): These are the objects I used to reproduce this issue: pod- /sig storage CC
/assign


Question: cmd/kube-proxy cmd/kubeadm cmd/kubelet <ins> vendor/k8s.io/component-base/logs/kube-log-runner </ins> ) echo "${targets[@]}" }

/kind feature Hey, in K8s 1.23 we are planning to deprecate multiple klog flags. Deprecated flags: , , , , , , , , , , , One of the alpha milestones is requires us to remove usage of deprecated flags within K8s repo. Examples found for windows [ ] [ ] /sig windows /wg structured-logging
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign I've started looking into this. Part of the problem is that go-runner is not part of kubernetes-node-windows- I suspect that is where the scripts get and kube- from.


Question: } } <ins> func validNewBinding() *api.Binding { return &api.Binding{ ObjectMeta: metav1.ObjectMeta{Name: "foo"}, Target:     api.ObjectReference{Name: "machine", Kind: "Node"}, } } func TestEtcdCreateBindingWithUIDAndResourceVersion(t *testing.T) { originUID := func(pod *api.Pod) types.UID { return pod.UID } emptyUID := func(pod *api.Pod) types.UID { return "" } changedUID := func(pod *api.Pod) types.UID { return pod.UID + "-changed" } originResourceVersion := func(pod *api.Pod) string { return pod.ResourceVersion } emptyResourceVersion := func(pod *api.Pod) string { return "" } changedResourceVersion := func(pod *api.Pod) string { return pod.ResourceVersion + "-changed" } noError := func(err error) bool { return err == nil } conflictError := func(err error) bool { return err != nil && errors.IsConflict(err) } testCases := map[string]struct { podUIDGetter             func(pod *api.Pod) types.UID podResourceVersionGetter func(pod *api.Pod) string errOK                    func(error) bool expectedNodeName         string }{ "originUID-originResourceVersion": { podUIDGetter:             originUID, podResourceVersionGetter: originResourceVersion, errOK:                    noError, expectedNodeName:         "machine", }, "originUID-emptyResourceVersion": { podUIDGetter:             originUID, podResourceVersionGetter: emptyResourceVersion, errOK:                    noError, expectedNodeName:         "machine", }, "originUID-changedResourceVersion": { podUIDGetter:             originUID, podResourceVersionGetter: changedResourceVersion, errOK:                    conflictError, expectedNodeName:         "", }, "emptyUID-originResourceVersion": { podUIDGetter:             emptyUID, podResourceVersionGetter: originResourceVersion, errOK:                    noError, expectedNodeName:         "machine", }, "emptyUID-emptyResourceVersion": { podUIDGetter:             emptyUID, podResourceVersionGetter: emptyResourceVersion, errOK:                    noError, expectedNodeName:         "machine", }, "emptyUID-changedResourceVersion": { podUIDGetter:             emptyUID, podResourceVersionGetter: changedResourceVersion, errOK:                    conflictError, expectedNodeName:         "", }, "changedUID-originResourceVersion": { podUIDGetter:             changedUID, podResourceVersionGetter: originResourceVersion, errOK:                    conflictError, expectedNodeName:         "", }, "changedUID-emptyResourceVersion": { podUIDGetter:             changedUID, podResourceVersionGetter: emptyResourceVersion, errOK:                    conflictError, expectedNodeName:         "", }, "changedUID-changedResourceVersion": { podUIDGetter:             changedUID, podResourceVersionGetter: changedResourceVersion, errOK:                    conflictError, expectedNodeName:         "", }, } storage, bindingStorage, _, server := newStorage(t) defer server.Terminate(t) defer storage.Store.DestroyFunc() for k, testCase := range testCases { pod := validNewPod() pod.Namespace = fmt.Sprintf("namespace-%s", strings.ToLower(k)) ctx := genericapirequest.WithNamespace(genericapirequest.NewDefaultContext(), pod.Namespace) podCreated, err := storage.Create(ctx, pod, rest.ValidateAllObjectFunc, &metav1.CreateOptions{}) if err != nil { t.Fatalf("%s: unexpected error: %v", k, err) } binding := validNewBinding() binding.UID = testCase.podUIDGetter(podCreated.(*api.Pod)) binding.ResourceVersion = testCase.podResourceVersionGetter(podCreated.(*api.Pod)) if _, err := bindingStorage.Create(ctx, binding.Name, binding, rest.ValidateAllObjectFunc, &metav1.CreateOptions{}); !testCase.errOK(err) { t.Errorf("%s: unexpected error: %v", k, err) } if pod, err := storage.Get(ctx, pod.Name, &metav1.GetOptions{}); err != nil { t.Errorf("%s: unexpected error: %v", k, err) } else if pod.(*api.Pod).Spec.NodeName != testCase.expectedNodeName { t.Errorf("%s: expected: %v, got: %v", k, pod.(*api.Pod).Spec.NodeName, testCase.expectedNodeName) } } } </ins> func TestEtcdCreateWithExistingContainers(t *testing.T) { storage, bindingStorage, _, server := newStorage(t) defer server.Terminate(t)

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --The api bind a pod to a node. But the pod is specified by name (not uid), which is not a unique identifier for pod. So the caller of can't make sure the pod is exactly what he want. api can bind a pod with specified uid to a node. If the uid not match, return an error a pending pod by statefulset named busybox, make sure it's pending, the scheduler will not bind it(use non-existed node selector or other method). So we will get a pending pod busybox-0. following code the in the code, delete the pod busybox-0 manually during the sleep or other method. The statefulset busybox will create another pod busybox-0 with different uid , the api in the code above will bind an unexpected pod to node The logic in scheduler is the same. If the pod is recreate with the same name and different resource require or selector between scheduler's and , scheduler will do a wrong bind for the pod. I think the root cause is the api does not check the uid. Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"20+", GitVersion:"v1.20.0-alpha.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2020-10-20T09:01:41Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"16+", GitVersion:"v1.16.9-aliyunedge.1", GitCommit:"", GitTreeState:"", BuildDate:"2021-03-08T09:56:01Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig api-machinery
/sig scheduling for triage (the binding feature of (pod to node) is for scheduling).
For the sake of precedence, are there other APIs that are based on object UID?
uids are used as a precondition is used in many places. if the server started checking that a uid on the binding object (if specified) matched the uid of the pod being modified, that seems sufficient
is there any way for the caller of Bind to confirm that he is binding the right pod? or not uid, use resource version?
that would require: the server to check that bind UID (if specified) matches the pod UID (passing the binding uid from BindingREST#Create → assignPod → setPodHostAndAnnotations and verifying the uid match in setPodHostAndAnnotations) the binding API caller to set in the posted binding object (as illustrated in the description of this issue)
actually, kube-scheduler(the most common binding API caller) has set the in the posted binding object. but the server doesn't check the uid. in my program, i do those to work around(not elegant, but it works...): call the binding API with an uid in annotation(i found that BindingREST#Create → assignPod → setPodHostAndAnnotations will carry annotation). check the matchs , if not match, delete the pod and wait for an new pod ... from your point of view, is it necessary to ensure the pod is the caller want when someone call bind API? what's your advice for this issue
given the scheduler made decisions based on the current shape of the pod spec, I think it makes sense for it to work to make sure it is binding that instance of the pod I would advocate for supporting uid as a precondition in the binding API and starting to use it from the scheduler
i agree with you
/remove-sig api-machinery
I also agree with this and this cannot achieve only on sig/scheduling. Sorry, if it is not suitable, but I'd like to add sig/api-machinery label again.

/sig api-machinery
Either scheduling or node sigs own the pod binding API. Either of those would be a good place to coordinate adding a fix to honor uid as a precondition API machinery owns the generic API plumbing and API server behavior, not implementations of most specific APIs like the pod or pod binding APIs /remove-sig api-machinery
Given you already exercised Jodan's suggestion locally, would you mind raising an offical PR?
how about this PR


Question: // Base64-encoded PKCS#10 CSR data Request []byte <del> // Requested signer for the request. It is a qualified name in the form: // `scope-hostname.io/name`. // Distribution of trust for signers happens out of band. // You can select on this field using `spec.signerName`. </del> <ins> // signerName indicates the requested signer, and is a qualified name. // // List/watch requests for CertificateSigningRequests can filter on this field using a "spec.signerName=NAME" fieldSelector. // // Well-known Kubernetes signers are: //  1. "kubernetes.io/kube-apiserver-client": issues client certificates that can be used to authenticate to kube-apiserver. //   Requests for this signer are never auto-approved by kube-controller-manager, can be issued by the "csrsigning" controller in kube-controller-manager. //  2. "kubernetes.io/kube-apiserver-client-kubelet": issues client certificates that kubelets use to authenticate to kube-apiserver. //   Requests for this signer can be auto-approved by the "csrapproving" controller in kube-controller-manager, and can be issued by the "csrsigning" controller in kube-controller-manager. //  3. "kubernetes.io/kubelet-serving" issues serving certificates that kubelets use to serve TLS endpoints, which kube-apiserver can connect to securely. //   Requests for this signer are never auto-approved by kube-controller-manager, and can be issued by the "csrsigning" controller in kube-controller-manager. // // More details are available at https://k8s.io/docs/reference/access-authn-authz/certificate-signing-requests/#kubernetes-signers // // Custom signerNames can also be specified. The signer defines: //  1. Trust distribution: how trust (CA bundles) are distributed. //  2. Permitted subjects: and behavior when a disallowed subject is requested. //  3. Required, permitted, or forbidden x509 extensions in the request (including whether subjectAltNames are allowed, which types, restrictions on allowed values) and behavior when a disallowed extension is requested. //  4. Required, permitted, or forbidden key usages / extended key usages. //  5. Expiration/certificate lifetime: whether it is fixed by the signer, configurable by the admin. //  6. Whether or not requests for CA certificates are allowed. </ins> SignerName string // usages specifies a set of usage contexts the key will be

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: I spent some time helping a customer figure out they were misusing the recently field in a CSR. They had been using CSRs for several releases to generate server certs for Kubernetes services that were only used within the cluster. They found and read the Signers section of the . The predefined Kubernetes signers didn't seem to fit their use case so they made up their own . When they approved the CSRs no certs were issued; it looked much as if the CSR controller was failing or not even running. It was an area I was unfamiliar with as well and it took a long time to realize the signerName they were using was the cause of the behavior and this was actually working correctly. The PR that this function has a sentence that makes the intent of the signerName field clear: Some ideas coming out of this exercise: Could the signer code add a condition or an event for the CSR resource indicating that the control plane is not issuing a certificate for an approved CSR because the signerName represents an external signer? Could the documentation on clarify that other names are allowed and that those are external signers and as such the control plane signer will not issue certificates - that is function the administrator must provide or install ? For people who have been using the v1beta1 API, would it be appropriate for the "Kubernetes signers" section to suggest that CSRs for internal Kubernetes services "can use as described in " What you expected to happen: How to reproduce it (as minimally and precisely as possible): Create a CSR Send a CertificateSigningRequest resource to Kubernetes using:Approve the CSR: CSR is Approved but not Issued: Anything else we need to know?: Environment: Kubernetes version (use ): I've repeated this on 1.18 through 1.20.1 Cloud provider or hardware configuration: IBM Cloud Kubernetes Service OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance.

The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig auth
I would not expect kubernetes signers to add conditions to CSR objects requesting different signers. This sounds like a good idea to add to the signerName API documentation (in ) and to the website documentation at No, the signerName is only intended for serving certificates issued to kubelets. I know and (and maybe had interest in the "pod/service serving certificate" use case (xref ), but that would be implemented as a (new) distinct signerName.
/assign


Question: hostSystemType             = "HostSystem" ) <del> var ( nameToNodeInfo        = make(map[string]*NodeInfo) vcToZoneDatastoresMap = make(map[string](map[string][]string)) ) </del> <ins> // NewNodeMapper returns a new NodeMapper func NewNodeMapper() *NodeMapper { return &NodeMapper{ nodeInfoRWLock:        &sync.RWMutex{}, nameToNodeInfo:        make(map[string]*NodeInfo), vcToZoneDatastoresMap: make(map[string](map[string][]string)), } } </ins> // GenerateNodeMap populates node name to node info map func (nm *NodeMapper) GenerateNodeMap(vSphereInstances map[string]*VSphere, nodeList v1.NodeList) error {

Command run: vSphere intree volume test suite occasionally failed with the following error: The test suite should always pass. We may have a chance to hit this by running ` No response binary version: 1.27.4 </details<detailsvsphere </details<details</details<details</details<details</details<details</details
/assign
/sig testing storage ?
/triage accepted


Question: * kube-apiserver: the `Priority` admission plugin is now enabled by default when using `--enable-admission-plugins`. If using `--admission-control` to fully specify the set of admission plugins, the `Priority` admission plugin should be added if using the `PodPriority` feature, which is enabled by default in 1.11. ([#65739](https://github.com/kubernetes/kubernetes/pull/65739), [@liggitt](https://github.com/liggitt)) * The `system-node-critical` and `system-cluster-critical` priority classes are now limited to the `kube-system` namespace by the `PodPriority` admission plugin. ([#65593](https://github.com/kubernetes/kubernetes/pull/65593), [@bsalamat](https://github.com/bsalamat)) * kubernetes-worker juju charm: Added support for setting the --enable-ssl-chain-completion option on the ingress proxy.  "action required": if your installation relies on supplying incomplete certificate chains and using OCSP to fill them in, you must set "ingress-ssl-chain-completion" to "true" in your juju configuration. ([#63845](https://github.com/kubernetes/kubernetes/pull/63845), [@paulgear](https://github.com/paulgear)) <ins> * In anticipation of CSI 1.0 in the next release, Kubernetes 1.12 calls the CSI `NodeGetInfo` RPC instead of `NodeGetId` RPC. Ensure your CSI Driver implements `NodeGetInfo(...)` before upgrading to 1.12. [@saad-ali](https://github.com/kubernetes/kubernetes/issues/68688) * Kubernetes 1.12 also enables [Kubelet device plugin registration](https://github.com/kubernetes/features/issues/595) feature by default. Before upgrading to 1.12, ensure the `driver-registrar` CSI sidecar container for your CSI driver is configured to handle plugin registration (set the `--kubelet-registration-path` parameter on `driver-registrar` to expose a new unix domain socket to handle Kubelet Plugin Registration). </ins> ### Other notable changes

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: CSI Plugin calls to get node ID information (see ). If that call is not implemented by driver, registration with kubelet will fail. What you expected to happen: Per CSI Spec, the deprecated call should be supported until CSI 1.0 is released (see ). Therefore if a by Kubernetes fails (because it is not implemented by the driver), we should fall back and try call instead. How to reproduce it (as minimally and precisely as possible): Use any CSI driver that implements but not -- says OpenSDS driver is currently hitting this, and may be used for repro. Anything else we need to know?: When CSI moves to 1.0 we can drop this fall back logic. If a fix is available within , we can include it, but 1.12 release should NOT block on this. /sig storage Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
please verify this is the issue you hit. Specifically see if you can find in your kubelet logs.
CC who volunteered to take a look
I don't see "error during CSI NodeGetInfo" in my kubelet logs. I don't see glog.Infof(log("Register new plugin with name: %s at endpoint: %s", pluginName, endpoint)) so it seems that RegisterPlugin is not called. I do see a message: "Plugin Watcher Start at /var/lib/kubelet/plugins". I don't know why RegisterPlugin is not called. I'll continue to check.
Here's an update. I implemented NodeGetInfo. I followed documentation about the Kubelet Plugin Watcher: https://kubernetes- to get my plugin registered. After that I don’t see the “driver name csi-opensdsplugin not found in the list of registred CSI drivers” any more. I see that now RegisterPlugin gets called and I see the following message in I0914 14:44:50. ]

Register new plugin with name: csi-opensdsplugin at endpoint: /var/lib/kubelet/plugins/csi- So this confirmed that missing NodeGetInfo is the problem. I still have problems but I think that’s a config issue on my side. I the following as cluster role rule for attacher and node: apiGroups: [""] resources: ["csidrivers"] verbs: ["get", "list", "watch", "update"] However I’m still getting the following errors in the logs: I0914 16:12:31. ] Listing and watching v1alpha1.CSIDriver from E0914 16:12:31. ] Failed to list v1alpha1.CSIDriver: is forbidden: User "system:serviceaccount:kube-system:attachdetach-controller" cannot list resource "csidrivers" in API group "" at the cluster scope kube-controller- I0914 16:11:51. ] Listing and watching v1alpha1.CSIDriver from E0914 16:11:51. ] Failed to list v1alpha1.CSIDriver: is forbidden: User "system:node:127.0.0.1" cannot list resource "csidrivers" in API group "" at the cluster scope
Did more investigation on this. Here's an update. I set the following feature gate to true on my local cluster: alpha: CSICRDAutoInstall CSISkipAttach CSIPodInfo I got some new errors in and kube-controller- I0916 07:22:06. ] Listing and watching v1alpha1.CSIDriver from E0916 07:22:06. ] Failed to list v1alpha1.CSIDriver: the server could not find the requested resource (get ) I0916 07:25:24. ] Listing and watching v1alpha1.CSIDriver from E0916 07:25:24. ] Failed to list v1alpha1.CSIDriver: the server could not find the requested resource (get ) In addition, I found the following in kube-controller- E0916 07:22:01. ]

failed to create CSIDrivers CRD: &v1beta1.CustomResourceDefinition{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"", GenerateName:"", Namespace:"", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(time.Location)(nil)}}, DeletionTimestamp:(v1.Time)(nil), DeletionGracePeriodSeconds:(int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1beta1.CustomResourceDefinitionSpec{Group:"", Version:"", Names:v1beta1.CustomResourceDefinitionNames{Plural:"", Singular:"", ShortNames:[]string(nil), Kind:"", ListKind:"", Categories:[]string(nil)}, Scope:"", Validation:(v1beta1.CustomResourceValidation)(nil), Subresources:(v1beta1.CustomResourceSubresources)(nil), Versions:[]v1beta1.CustomResourceDefinitionVersion(nil), AdditionalPrinterColumns:[]v1beta1.CustomResourceColumnDefinition(nil)}, Status:v1beta1.CustomResourceDefinitionStatus{Conditions:[]v1beta1.CustomResourceDefinitionCondition(nil), AcceptedNames:v1beta1.CustomResourceDefinitionNames{Plural:"", Singular:"", ShortNames:[]string(nil), Kind:"", ListKind:"", Categories:[]string(nil)}, StoredVersions:[]string(nil)}}, err: &errors.StatusError{ErrStatus:v1.Status{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ListMeta:v1.ListMeta{SelfLink:"", ResourceVersion:"", Continue:""}, Status:"Failure", Message:" is forbidden: User "system:serviceaccount:kube-system:attachdetach-controller" cannot create resource "customresourcedefinitions" in API group "" at the cluster scope", Reason:"Forbidden", Details:(*v1.StatusDetails)(0xc4212af560), Code:403}} So I made this change to add RBAC rule for customresourcedefinitions: After that, my CSI plugin is functional. I can create/attach/mount volumes successfully now.
/cc
(As we discussed offline) the latest code in master no longer has CSICRDAutoInstall, CSISkipAttach, CSIPodInfo flags.

New CSI features are now flag gated under CSINodeInfo and CSIDriverRegistry flags (both alpha and disabled by default). KubeletPluginsWatcher must be enabled as well, but that's in beta now so it's enabled by default. Since this PR merged: , CRDs must be installed manually, and you can find an example of it here: (note they have to be created with --validation=false kubectl flag for the time being). This PR has not been cherry-picked to 1.12.0-rc.0 yet.
Thanks Tested latest code. If NodeGetInfo is implemented, we just need to make necessary changes when deploying driver-registrar sidecar container so KubeletPluginsWatcher can work. NodeGetInfo is needed for kubelet to register CSI plugins. Also tested with and without "CSINodeInfo=true,CSIDriverRegistry=true" and manually installed CRDs using when they are enabled.
I have the same problem when deploying my CSI plugin based on CSI v0.2.0 on Kubernetes v.1.12.0. Daemonset pods of CSI plugin cannot launch successfully. Kubelet logs:
Considering Kubernetes v1.12 is already out, and we planned to deprecate the old call in CSI 1.0/Kubernetes 1.13 anyway, we'll pull of the bandaid and deprecate it now. If you are running Kubernetes 1.13, make sure your CSI driver implements the .
looks like you might be running in to a different issue: Kubernetes v1.13 also enables Kubelet device plugin registration by default. Before upgrading to v1.13, ensure the CSI sidecar container for your CSI driver is configured to handle plugin registration (set the parameter on to expose a new unix domain socket to handle Kubelet Plugin Registration). From
Hi Because goes Beta and is enabled by default in Kubernetes v1.12. After disabling this feater gate, I can run CSI plugin based on CSI spec v0.2.0 in Kubernetes v1.12.2. When Kubelet enable , Kubelet would call gRPC call to register CSI plugin. gRPC call was first proposed at CSI spec v0.3.0. Therefore, user should disable feature gate when running CSI plugin based on CSI spec v0.2.0 in Kubernetes v1.12.

I have implement the NodeGetInfo call. But it still has this issue. My kubernetes version is 1.14.3.


Question: # Increase the allowed number of open file descriptors ulimit -n 65536 <ins> mount-master-pd </ins> tar xzf kubernetes-server-linux-amd64.tar.gz kubernetes/server/bin/kube-scheduler --master=127.0.0.1:8080 $(cat scheduler_flags) &> /var/log/kube-scheduler.log &

While running kubemark tests we are facing "no space left on device" problems, because master-disk is not mounted in kubemark.
I'm reopening it for now, since it seems that we are still facing "no space left on device" issues in kubemark scale.
Seems to be fixed by


Question: "k8s.io/apimachinery/pkg/api/resource" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/labels" <ins> "k8s.io/apimachinery/pkg/util/errors" </ins> "k8s.io/apimachinery/pkg/util/uuid" "k8s.io/apimachinery/pkg/util/wait" clientset "k8s.io/client-go/kubernetes"

From After , Multi-AZ Clusters tests in 1.19 and 1.20 started timing out as createBalancedPodForNodes is extremely inefficient in scale: for each node (out of 5k), it lists all pods in a cluster serially (each list takes ~5s), which translates to ~7 hours. Optimizing this function by listing pods once and then in memory by Spec.NodeName /kind bug /priority critical-urgent /sig scheduling
/triage accepted
/assign
/reopen Looks that ComputeCPUMemFraction took only 12 minutes, but it stuck later, on It is scheduling one pod per each node: Which is waiting 5s to be ready: 5k nodes * 5s = ~7h
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign Can you do the pod creations in parallel?
Yes, good idea. I'll update to do that


Question: }, } podWithHook := getPodWithHook("pod-with-poststart-exec-hook", imageutils.GetE2EImage(imageutils.Agnhost), lifecycle) testPodWithHook(podWithHook) }) /*

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --k8s-e2e-ltsc2019-containerd-flannel-sdnoverlay-master [] Container Lifecycle Hook when create a pod with lifecycle hook should execute prestart http hook properly [NodeConformance] [Conformance] [] Container Lifecycle Hook when create a pod with lifecycle hook should execute poststart http hook properly [NodeConformance] [Conformance] Note: the tests have been excluded until they are fixed. In the case of multinode clusters, the http server pod and the test cluster can spawn on different nodes, which can be problematic for poststart / prestop hooks, as they are executed by the kubelet itself, and the cross-node lifecycle hook might fail (according to the [Kubernetes network model], it is not mandatory for kubelet to be able to access pods on a different node). However, that is not always the case, different Networking setups allow cross-node communication between kubelet and pods, which means that the above tests are passing properly (for example: ), which begs the question if this behaviour should be universal or not, especially since the tests are marked as , which would imply that it should work in all cases. [Kubernetes network model]: links to appreciated links to specific failures in spyglass appreciated Slack: /sig testing /sig networking /sig windows <!-- Please see the deflaking doc () for more guidance! --
/sig network /sig node /area conformance /assign /priority critical-urgent /cc This is important since affects conformance and defines the networking behaviour for the poststart/ prestop hooks
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /sig network /sig node /sig conformance /assign /priority critical-urgent /cc Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
i'd personally let folks have first dibs on what we need to do here. Though from the name of the test itself, it seems like the test is NOT a sig-network test :)
Indeed, it does not have any tag in its name, but the tests rely on networking and on how CNIs work.

As I've stated above, it's not a problem for flannel sdnbridge, but it is for flannel sdnoverlay; other CNIs might have a similar issue too.
because the network is not a thing :) let´s tag the sig owner then /sig node I think that the key here is to decide if kubelet should be able to access pod in other nodes, and based on the link Claudiu is part of the Kubernetes networking model. I saw users that doesn't want kubelet to access even the pods on same host, so I think that is legit to say kubelet MAY reach pods in other nodes but not MUST ... but I'm not familiar with the to fully understand the context. If kubelet "may" reach pods in the other nodes, the test must only test lifecycle hooks for pods in the same node as the kubelet under test, right?
This is correct. The model also says: "For those platforms that support Pods running in the host network (e.g. Linux): pods in the host network of a node can communicate with all pods on all nodes without NAT" I believe that Windows host network is not supported, right?
Right, and there are specific tests that checks that exact scenario you've mentioned: However, the flaky tests I've mentioned in the issue require kubelet to be able to dial-in into a webserver pod for a different pod's poststart / prestop HTTP lifecycle hooks. There are couple of things regarding this: of all, in order to ensure kubelet would be able to run the HTTP lifecycle hooks, we'd have to make sure both pods are on the same node, as that is a supported scenario, according to the Kubernetes Network Model. I have sent a PR for this above. Alternatively, a Cluster IP could probably work as well, but IMO, ensuring that both pods end up on the same node is simpler and easier to clean up. , it's interesting that those HTTP lifecycle hook tests work on different networking setups, without any pods having HostNetwork enabled, even though you've mentioned they should (e.g.: Linux / Windows with flannel l2bridge, and even Linux with flannel overlay). Now, my question is: is this behaviour a feature, or a bug, especially since the tests are marked as `?

oh, I missed that part, so in Linux kubelet must be able to communicate with pods on all nodes (becuase it is running in the hostnetwork) If that is correct, it seems that one solution is to keep the test for Linux and tag it as LinuxOnly, and a new test for WindowsOnly that only spawn the pods under test in the same node EDIT Discussed in sig-net meeting , no clear answer... can be kubelet considered as a hostnetwork pod?
For the purposes of testing, I don't mind forcing the pods onto the same node., but let's document the heck out of that at the code-site?
correct but it will be supported once HostProcess containers are merged in 1.22: Only in Overlay networks (ie Flannel where these tests are flaking) does the pod have to be on the same node. It doesn't require host network mode and these tests are . I would be more inclined to disable this test for Windows Overlay test setups and have a sig-windows test for the overlay functionality. I would also question if the lifecycle hooks only running on the same node is useful and if it should just be documented that it doesn't work in the Windows Overlay setup.
is looking at test PRs from SIG Node. I will let sig-net triage this issue
Next steps on this?
/triage accepted
We still haven't merged this: I also had this, but it has been decided to be blocked:
requires conformance approval, at this point I think that has to be postponed to 1.23 right, since that behavior is not required, we shouldn't test it, because is legit to fail and having a test for something that can fail can create confusion :smile:


Question: Minify       bool RawByteData  bool <ins> Context      string </ins> OutputFormat string genericclioptions.IOStreams

/kind bug What happened: Current context was set to cluster1. Ran and got back information about cluster1 instead. What you expected to happen: I expected the output to be the cluster2 information How to reproduce it (as minimally and precisely as possible): Have two contexts in your kube config, A and B Run to set A as current context Run Anything else we need to know?: I'd like to help contribute, so if this is an easy fix, someone point me in the right direction and I'll try to get it done. Environment: Kubernetes version (use ): 1.10.2 OS (e.g. from /etc/os-release): MacOS 10.11.6 Install tools: kubectl was installed via brew
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign Filed


Question: PVCs:          []*v1.PersistentVolumeClaim{pvc}, SeLinuxLabel:  e2epv.SELinuxLabel, NodeSelection: node, <ins> PVCsReadOnly:  readOnly, </ins> } pod, err := e2epod.CreateSecPodWithNodeSelection(cs, &podConfig, framework.PodStartTimeout) defer func() {

What happened: Running two pods that use the same iSCSI volume, one of them fails with: What you expected to happen: Both pods should be running Anything else we need to know?: It's caused by iscsi volume plugin persiting iSCSI settings into the mount directory itself (e.g. ): volume plugin creates the global mount directory. volume plugin creates there. volume plugin mounts the volume there. This shadows the file, but it's fine, as long as no other pod uses the same volume on the same node. creates a second pod that uses the volume. volume plugin tries to create in the global volume directory. Since the volume is mounted, it will either writes the file to the volume (!) or fails, if the volume is read-only.
/sig storage /assign /priority impotant-soon
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig storage /assign /priority impotant-soon Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/priority important-soon
Our e2e did not catch this, because they always create only one pod when testing read-only volumes.


Question: package flock <del> import ( "os" "sync" "golang.org/x/sys/unix" ) var ( // lock guards lockfile. Assignment is not atomic. lock sync.Mutex // os.File has a runtime.Finalizer so the fd will be closed if the struct // is garbage collected. Let's hold onto a reference so that doesn't happen. lockfile *os.File ) </del> <ins> import "syscall" </ins> // Acquire acquires a lock on a file for the duration of the process. This method // is reentrant. func Acquire(path string) error { <del> lock.Lock() defer lock.Unlock() var err error if lockfile, err = os.OpenFile(path, os.O_RDWR|os.O_CREATE, 0600); err != nil { return err } defer lockfile.Close() opts := unix.Flock_t{Type: unix.F_WRLCK} if err := unix.FcntlFlock(lockfile.Fd(), unix.F_SETLKW, &opts); err != nil { </del> <ins> fd, err := syscall.Open(path, syscall.O_CREAT|syscall.O_RDWR, 0600) if err != nil { </ins> return err } <del> return nil </del> <ins> // We don't need to close the fd since we should hold // it until the process exits. return syscall.Flock(fd, syscall.LOCK_EX) </ins> }

BUG REPORT: Kubernetes version latest master: Environment: Ubuntu 16.04 LTS, Linux 4.2.0-42-generic: What happened: The doesn't provide mutual exclusion. Two processes can both grab the same flock at the same time. What you expected to happen: Only one process should be able to grab the flock. How to reproduce it (as minimally and precisely as possible): Anything else we need to know: Although the flock doesn't work, the does work. Since we usually specify together with , so the 1st process will exit anyway. However we still need to fix the flock..
cc
Sent a PR to fix this ()


Question: ) // NewProxyServer returns a new ProxyServer. <del> func NewProxyServer(config *proxyconfigapi.KubeProxyConfiguration, cleanupAndExit bool, cleanupIPVS bool, scheme *runtime.Scheme, master string) (*ProxyServer, error) { </del> <ins> func NewProxyServer(o *Options) (*ProxyServer, error) { return newProxyServer(o.config, o.CleanupAndExit, o.CleanupIPVS, o.scheme, o.master) } func newProxyServer( config *proxyconfigapi.KubeProxyConfiguration, cleanupAndExit bool, cleanupIPVS bool, scheme *runtime.Scheme, master string) (*ProxyServer, error) { </ins> if config == nil { return nil, errors.New("config is required") }

https://k8s- https://k8s- cc: /kind bug /priority failing-test
This is at least twice in a week we've had cross break due to a public method signature change not being brought to all platforms () We should do:
I'm lost - the PR in question adds an argument and fixes the call-site - what is broken?
Ahh, I see.
How did that PR get through, if it is so obviously broken on Cross?
Cross build doesn't run 100% of the time as a pre-submit (because it takes so long), and it can't block. It's something sig-testing is tracking, but there isn't a quick fix yet.
Looks like was skipped for that PR: "pull-kubernetes-cross Skipped" not sure why yet.
Cross runs when a file matching: changes, (which needs to be fixed, opening a test-infra issue to expand the regex), but also some files could have build tags without the file name matching () Additionally cross is not blocking (), the underlying bug should be fixed now though so once we get the fix in to k/k we can make it blocking when it does run.
I find has already sent a PR. Thanks!


Question: for _, container := range allContainers { containerID, err := findContainerIDByName(&pstatus, container.Name) if err != nil { <del> klog.InfoS("ReconcileState: skipping container; ID not found in pod status", "pod", klog.KObj(pod), "containerName", container.Name, "err", err) </del> <ins> klog.V(4).InfoS("ReconcileState: skipping container; ID not found in pod status", "pod", klog.KObj(pod), "containerName", container.Name, "err", err) </ins> failure = append(failure, reconciledContainer{pod.Name, container.Name, ""}) continue } cstatus, err := findContainerStatusByName(&pstatus, container.Name) if err != nil { <del> klog.InfoS("ReconcileState: skipping container; container status not found in pod status", "pod", klog.KObj(pod), "containerName", container.Name, "err", err) </del> <ins> klog.V(4).InfoS("ReconcileState: skipping container; container status not found in pod status", "pod", klog.KObj(pod), "containerName", container.Name, "err", err) </ins> failure = append(failure, reconciledContainer{pod.Name, container.Name, ""}) continue } if cstatus.State.Waiting != nil || (cstatus.State.Waiting == nil && cstatus.State.Running == nil && cstatus.State.Terminated == nil) { <del> klog.InfoS("ReconcileState: skipping container; container still in the waiting state", "pod", klog.KObj(pod), "containerName", container.Name, "err", err) </del> <ins> klog.V(4).InfoS("ReconcileState: skipping container; container still in the waiting state", "pod", klog.KObj(pod), "containerName", container.Name, "err", err) </ins> failure = append(failure, reconciledContainer{pod.Name, container.Name, ""}) continue }

kubernetes 1.19.2 seeing tons of these, feels like something that should be debug or info ... not really actionable or something that it should warn about ? /sig node
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
I am also seeing this same issue what's odd is that I cannot find the container it's looking for which is probably why its giving this message, so my question would be why is it looking for this container ID?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/lifecycle frozen
I agree this should be lower than warning... /triage accepted /help
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : I agree this should be lower than warning... /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I think all InfoS messages under the should be moved to debug messages because it flood the log when the CPU manager enabled with the static policy.
/assign


Question: }() } <ins> // skipPodSchedule returns true if we could skip scheduling the pod for specified cases. func (sched *Scheduler) skipPodSchedule(pod *v1.Pod) bool { // Case 1: pod is being deleted. if pod.DeletionTimestamp != nil { sched.Recorder.Eventf(pod, nil, v1.EventTypeWarning, "FailedScheduling", "Scheduling", "skip schedule deleting pod: %v/%v", pod.Namespace, pod.Name) klog.V(3).Infof("Skip schedule deleting pod: %v/%v", pod.Namespace, pod.Name) return true } // Case 2: pod has been assumed and pod updates could be skipped. // An assumed pod can be added again to the scheduling queue if it got an update event // during its previous scheduling cycle but before getting assumed. if sched.skipPodUpdate(pod) { return true } return false } </ins> type podConditionUpdaterImpl struct { Client clientset.Interface }

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Similar problem as , howerver in our case, pod's annotation was updated at scheduling cycle, so it had not been assumed when its annotation was updated. We also need to deal with this case. Detailed steps: a pod . tried to schedule the pod. Then it was prediated by a filter plugin that we write. This plugin modified the pod's annotation to record something. Because pod's annotation changed, scheduuler received a pod update event, and put the pod to the schedule queue again. scheduled the pod successfully for this schedule cycle. tried to schedule the updated pod again(it was put to the schedule queue at step 2). If the pod was scheduled sucessfully, we would see the error log() in scheduler. If scheduler failed to schedue the pod, scheduler would send a failed scheduler event to pod. They are both confusing because the pod had already been successfully. ! Why is this needed: It will be better to handle pod annotation updates during scheduling more gracefully. /kind cleanup /sig scheduling /assign
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : <!-- Please only use this template for submitting enhancement requests --1. Created a pod . Scheduler tried to schedule the pod. Then it was prediated by a filter plugin that we write. This plugin modified the pod's annotation to record something. Because pod's annotation changed, scheduuler received a pod update event, and put the pod to the schedule queue again. Scheduler scheduled the pod successfully for this schedule cycle. Scheduler tried to schedule the updated pod again(it was put to the schedule queue at step 2). If the pod was scheduled sucessfully, we would see the error log() in scheduler. If scheduler failed to schedue the pod, scheduler would send a failed scheduler event to pod. They are both confusing because the pod had already been successfully. /sig scheduler /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: } // optimization: avoid invoking the pod worker if no further changes are possible to the pod definition <del> if podWorkerTerminal { klog.V(4).Infof("Pod %q has completed, ignoring remaining sync work: %s", format.Pod(pod), syncType) </del> <ins> // (i.e. the pod has completed and its containers have been terminated) if podWorkerTerminal && containersTerminal { klog.V(4).InfoS("Pod has completed and its containers have been terminated, ignoring remaining sync work", "pod", klog.KObj(pod), "syncType", syncType) </ins> return }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When pod exceeds we find that, The pod is correctly states with the reason but as you can see above the container state is still reported as What you expected to happen: Container state should be not because pod has exceeded How to reproduce it (as minimally and precisely as possible): Create the pod described above and wait till it exceeds listed . Then get the container status using, Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
I would like to address this :) /assign
thanks.


Question: t.Errorf("unexpected lack of error") } } <ins> { // Case 7: now is way way ahead of last start time, but there is a short deadline. sj.ObjectMeta.CreationTimestamp = metav1.Time{Time: T1.Add(-2 * time.Hour)} sj.Status.LastScheduleTime = &metav1.Time{Time: T1.Add(-1 * time.Hour)} now := T2.Add(10 * 24 * time.Hour) // Deadline is short deadline := int64(2 * 60 * 60) sj.Spec.StartingDeadlineSeconds = &deadline _, err := getRecentUnmetScheduleTimes(sj, now) if err != nil { t.Errorf("unexpected error") } } </ins> }

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): No What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): , , Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Environment: Cloud provider or hardware configuration: bare metal OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: , deployed with Others: What happened: I am using a scheduledjob with . An instance of my job has taken a long time to finish, thus skipping many start times. Ever since, the scheduledjob will not schedule anything new. Controller manager log has: What you expected to happen: I expect the scheduledjob not to be stuck not scheduling anything anymore. With , I expect that it should just re-schedule something as soon as the previously running job has completed. The "Too many missed start times to list" safeguard appears to apply mostly to cases where job concurrency is allowed to avoid creating a lot of jobs in case of timestamp issues. ` How to reproduce it (as minimally and precisely as possible): The issue shows up after about 100 minutes of this running. Anything else do we need to know:
cc
sorry for the long delay in response, there were a few issues solved in later versions of ScheduledJob, can you verify this against latest 1.4.x? I've tried cherry-picking all of the crucial ones, like: and I'll try to verify this on my own as well.

I am still getting the same issue with
thanks for the info, will dig deeper into this one.
Thanks a lot any chance a potential fix could make it in 1.5?
if possible - yes, I'll try to backport it to 1.5
did you look into this? If not, let me know if you want me to pick it up and work on a PR
yeah, go ahead and ping me in the PR.
I just thought of something which might make sense. We've modified the error message to make it more "actionable", and the user can set a deadline to avoid running into this. However, the error message is still only controller level AFAIK and thus not exposed to the user. From the user's perspective, there is not way to tell why the CronJob is not scheduling the next instance. For these reasons, maybe we should also emit an event in the corresponding CronJob? I can send a small PR for this modification.


Question: package main import ( <ins> "context" </ins> "flag" "fmt" "io/ioutil"

We should support image family in node e2e image config. It makes image validation much easier on GCE.
Cool idea :) Adding a link to for anyone like me who hadn't heard of them before :)


Question: "k8s.io/kubernetes/pkg/api/errors" "k8s.io/kubernetes/pkg/api/testapi" "k8s.io/kubernetes/pkg/api/unversioned" <ins> "k8s.io/kubernetes/pkg/client/restclient" </ins> "k8s.io/kubernetes/pkg/client/unversioned/fake" )

When users try to label/annotate multiple resources, we should tell them to add . Current output: Expected output:
and too
Instead of "--all is false", say "multiple resources were provided, but no label selector or --all flag specified"
Description updated based on suggestion
I think "resource(s) were provided, but no name, label selector, or --all flag specified" is better. I will send a pull request for this as my first PR.


Question: if plugin, ok := inTreePlugins[pluginName]; ok { return plugin.GetInTreePluginName(), nil } <del> return "", fmt.Errorf("could not find In-Tree driver name for CSI plugin %v", pluginName) </del> <ins> return "", fmt.Errorf("could not find In-Tree driver name for CSI plugin %s", pluginName) </ins> } // IsPVMigratable tests whether there is migration logic for the given Persistent Volume

I noticed that kube-controller-manager is consuming visible amount of CPU in completely idle 5k node cluster (~7 cores/s) while for comparison kube-apiserver was using much less that time. I fetched pprof and noticed that we are burning significant in metrics.(attachDetachStateCollector).CollectWithStability:! Going further, most of the cpu is used for fmt.Errorf for formatting v1.Volume in line 160: which is never actually used by the caller We do not use significant cpu for preparing error messages that are then ignored. pprof cluster with some pods and volumes. No response_ <details26.1 </details<detailsgke, any? </details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
A trivial fix would be actually to change GetInTreePluginNameFromSpec to write only PV/Volume names instead of entire specs in error message.
/assign In scheduler we use this function to check if we can translate the in-tree PV to CSI, I think we also only need the name of the pv. I'll create a pr for this one line change.
/sig storage Instrumentation ?


Question: continue } <del> pvName := pvc.Spec.VolumeName // TODO - the actual handling of unbound PVCs will be fixed by late binding design. if pvName == "" { klog.V(4).Infof("Persistent volume had no name for claim %s/%s", namespace, pvcName) </del> <ins> driverName, volumeHandle := c.getCSIDriver(pvc) // if we can't find driver name or volume handle - we don't count this volume. if driverName == "" || volumeHandle == "" { </ins> continue } <del> pv, err := c.pvInfo.GetPersistentVolumeInfo(pvName) if err != nil { klog.V(4).Infof("Unable to look up PV info for PVC %s/%s and PV %s", namespace, pvcName, pvName) continue } csiSource := pv.Spec.PersistentVolumeSource.CSI if csiSource == nil { klog.V(4).Infof("Not considering non-CSI volume %s/%s", namespace, pvcName) continue } driverName := csiSource.Driver </del> volumeLimitKey := volumeutil.GetCSIAttachLimitKey(driverName) <del> result[csiSource.VolumeHandle] = volumeLimitKey </del> <ins> result[volumeHandle] = volumeLimitKey </ins> } return nil } <ins> func (c *CSIMaxVolumeLimitChecker) getCSIDriver(pvc *v1.PersistentVolumeClaim) (string, string) { pvName := pvc.Spec.VolumeName namespace := pvc.Namespace pvcName := pvc.Name placeHolderCSIDriver := "" placeHolderHandle := "" if pvName == "" { klog.V(5).Infof("Persistent volume had no name for claim %s/%s", namespace, pvcName) return c.getDriverNameFromSC(pvc) } pv, err := c.pvInfo.GetPersistentVolumeInfo(pvName) if err != nil { klog.V(4).Infof("Unable to look up PV info for PVC %s/%s and PV %s", namespace, pvcName, pvName) // If we can't fetch PV associated with PVC, may be it got deleted // or PVC was prebound to a PVC that hasn't been created yet. // fallback to using StorageClass for volume counting return c.getDriverNameFromSC(pvc) } csiSource := pv.Spec.PersistentVolumeSource.CSI if csiSource == nil { klog.V(5).Infof("Not considering non-CSI volume %s/%s", namespace, pvcName) return placeHolderCSIDriver, placeHolderHandle } return csiSource.Driver, csiSource.VolumeHandle } func (c *CSIMaxVolumeLimitChecker) getDriverNameFromSC(pvc *v1.PersistentVolumeClaim) (string, string) { namespace := pvc.Namespace pvcName := pvc.Name scName := pvc.Spec.StorageClassName placeHolderCSIDriver := "" placeHolderHandle := "" if scName == nil { // if StorageClass is not set or found, then PVC must be using immediate binding mode // and hence it must be bound before scheduling. So it is safe to not count it. klog.V(5).Infof("pvc %s/%s has no storageClass", namespace, pvcName) return placeHolderCSIDriver, placeHolderHandle } storageClass, err := c.scInfo.GetStorageClassInfo(*scName) if err != nil { klog.V(5).Infof("no storage %s found for pvc %s/%s", *scName, namespace, pvcName) return placeHolderCSIDriver, placeHolderHandle } // We use random prefix to avoid conflict with volume-ids. If PVC is bound in the middle // predicate and there is another pod(on same node) that uses same volume then we will overcount // the volume and consider both volumes as different. volumeHandle := fmt.Sprintf("%s-%s/%s", c.randomVolumeIDPrefix, namespace, pvcName) return storageClass.Provisioner, volumeHandle } </ins>

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: Volume binding mode doesn't consider the volume attach limit per node. What you expected to happen: volume binding mode should have respected the attach limit, similar to binding mode. How to reproduce it (as minimally and precisely as possible): In CSI driver, configured the volume-attach limit to 1 per node on a 3 node cluster. In volume binding mode, only 3 pods with 3 PV/PVCs runs. In binding mode, all 5 pods runs with 5PV/PVCs on 3 node cluster. (Actually 4 pods scheduled on 1 node and 1 on other node, 3rd didn't have any pod scheduled) Anything else we need to know?: I was able to reproduced it with minimal node and configuration, and also hit this for stress case with 100 Volume limit and 500 pods running on the 5 mode cluster. Environment: Kubernetes version (use ): ~ kubectl version Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-10-30T21:39:38Z", GoVersion:"go1.11.1", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-12-03T20:56:12Z", GoVersion:"go1.11.2", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: On-prem linux server OS (e.g. from /etc/os-release): Centos 7Kernel (e.g. ): 3.10.0 Install tools: Others:
cc
/sig storage
/assign Currently the predicate PVCs that are not bound. It should lookup the storageclass to get the provisioner/driver name. This solution will require a StorageClass object is even if you don't use dynamic provisioning.

I had a thought that it wouldn't be so simple to count unbound PVCs of other pods since binding is still in progress, but I think it's fine actually. We only count other pods that have been assigned to the same node. Those pods are assigned to the node AND still unbound because of one of these reasons: It's actually been scheduled to the node and someone manually came in and unbound the PVC: unlikely and an unsupported sequence since we implemented StorageProtection The Pod was initially created with nodeName already set, bypassing the scheduler The Pod is going to be scheduled (it's been assumed), and it's currently going through binding. This is normal with delayed binding. Either way, the fact that the Pod's nodename is set to this node means that the pod should be counting towards the node's limit.
Hmm, so after opening the PR that fixes this, I realize that - from storageclass we don't really know if the volume we are evaluating is CSI volume or not. I think a similar problem exists for existing predicate though. Like AzureDisk predicate does not know if PVC it is over counting is of type AzureVolume or not. So, I suspect that those 4 hardcoded predicates are counting all delayed binding volumes.
if the storageclass.provisioner name is not found, can we just ignore it?
I think we can ignore it. The PR currently does that.


Question: return lvl < zapcore.ErrorLevel }) core = zapcore.NewTee( <del> zapcore.NewCore(encoder, flushError, highPriority), </del> <ins> zapcore.NewCore(encoder, errorStream, highPriority), </ins> zapcore.NewCore(encoder, infoStream, lowPriority), ) }

When working on I have found a regression in JSON logging format that happend in 1.23. In v1.23 JSON logging introduced two new features, log stream splitting and info log buffering. When I started measuring throughput it looked very promising, for example for info logs showed 10x improvement when using JSON split stream and 100Mi info log buffer. However when I introduced mixed logs (ever X info logs write 1 error logs), I noticed a large regression. For 10% of error logs written (apiserver during upgrade can write even 30% errors), I have found that just switching json on, results in log throughput being reduced from 25 MiB/s to 0.7MiB. I have bisected git repo and found that was the root cause. I expect that this regression would be noticed if we run scalability tests with JSON enabled. Loader code: /wg structured-logging /cc /sig scalability JSON should generate similar or greater log throughput when compared to text format. Run loader as described in from No response N/A N/A N/A N/A N/A N/A
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
So to reproduce the regression, I need to do (default for error rate is 10%, 300 for message length, 10s duration) for old and new code, right?
Yes, let me give my results. Before text: 30.7MiB/s, json:39.6MiB/s After, text: 30.6MiB/s, json: 6.5MiB/s
should we try to get this in 1.23.1?
Yes, we just moved JSON and Structured Logging to Beta in v1.23. This bug should cause performance issues when there is non trivial amount of errors (1%) written by control plane. I expect that this has really bad impact on upgrades which I have seen can reach 30% of error logs from apiserver.

It's worth pointing out that this regression only occurs when writing to a disk-based file (see ). That also provides a very likely explanation for the huge performance difference: I bet zap not only flushes its buffered info messages, it also syncs the file. That is fast for tmpfs, but slow for real disks.
do you know how the Kubernetes components are run in those cases where logging performance matters? Do they directly write to a real disk or does it all go through pipes? If it is the latter, then this issue matters a lot less in practice. you mentioned at some point that you had scalability issues with logging that went away when using go-runner as wrapper instead of using klog's internal output redirection. I bet the root cause is the same: fast vs. slow sync of the output fd. That's because go-runner always runs output through one or more pipes. I don't know how realistic the scalability setup was, but it must have written log files to some real disk.


Question: --pod-manifest-path="${POD_MANIFEST_PATH}"  ${auth_args}  ${dns_args}  <del> ${net_plugin_dir_args}  </del> <ins> ${cni_conf_dir_args}  ${cni_bin_dir_args}  </ins> ${net_plugin_args}  ${container_runtime_endpoint_args}  ${image_service_endpoint_args}

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Environment: Cloud provider or hardware configuration: bare-metal OS (e.g. from /etc/os-release): Ubuntu 14.04 Kernel (e.g. ): Linux dev-k8s-controller-p1 4.4.0-53-generic ~14.04.1-Ubuntu SMP Fri Dec 2 03:43:31 UTC 2016 x8664 x8664 x86_64 GNU/Linux Install tools: chef cookbook Others: Docker 1.12.6, calico 2.1 CNI binaries: kubelet config: What happened: When trying to run pod error message is generated: So it seems that kubelet is trying to find CNI binaries in paths: /opt/loopback/bin /etc/cni/net.d /opt/calico/bin But it should look at since this is the default valur for parameter. What you expected to happen: Pod starts. How to reproduce it (as minimally and precisely as possible): Run pod: with kubelet 1.6 and CNI binaries in those path (default for calico): Anything else we need to know: After copy to and pod starts fine.
Good, very helpful to me
When set and not set , default path of CNI binaries will not take effect, the path specified by will be used instead. We should use and instead of if use CNI plugin.
I will push a PR to deprecate the flag. It is confusing. Closing this issue.
Was that ? It was closed without merging. I agree should be deprecated.
I think is needed before is removed. The flag is still not deprecated, I will send a PR to deprecate it.


Question: "Time":           recent(maxStatsAge), "AvailableBytes": fsCapacityBounds, "CapacityBytes":  fsCapacityBounds, <del> // we assume we are not running tests on machines < 10tb of disk </del> <ins> // we assume we are not running tests on machines more than 10tb of disk </ins> "UsedBytes":  bounded(e2evolume.Kb, 10*e2evolume.Tb), "InodesFree": bounded(1e4, 1e8), "Inodes":     bounded(1e4, 1e8),

Which jobs are failing: ci-cri-containerd-node-e2e Which test(s) are failing: Summary API [NodeConformance] when querying /stats/summary should report resource usage through the stats api Since when has it been failing: 2020 10-12 09:04 PDT Since Testgrid link: Reason for failure: Capability set incorrectly. CAPNETRAW should be NETRAW. I1013 18:39:19.058] Oct 13 18:10:51.403: INFO: At 2020-10-13 18:09:34 +0000 UTC - event for stats-busybox-0: {kubelet tmp-node-e2e--cos-81--1210-0} Failed: Error: failed to create containerd task: OCI runtime create failed: starting container process caused: unknown capability "CAPCAPNET_RAW": unknown Anything else we need to know: I have no idea where this is documented. It's a best guess.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
it's a node_e2e test so /sig node
Seems like there's some dispute on who is doing the correct thing.
I would like to have a try.
please see discussion in


Question: Name:            d.Name + "-" + podTemplateSpecHash, Namespace:       d.Namespace, OwnerReferences: []metav1.OwnerReference{*newControllerRef(d)}, <del> Finalizers:      []string{metav1.FinalizerDeleteDependents}, </del> }, Spec: extensions.ReplicaSetSpec{ Replicas:        new(int32),

Seeing this in v1.6.1. a Deployment and a Service the Deployment add an ownerRef that points back to the Service delete the Service The Service and the Deployment are deleted, but the ReplicaSet and pods remain. This is neither Orphan nor Foreground/Background policy. This behavior should never happen. Note: if I make a delete call to the API with propagationPolicy set explicitly then things behave as expected, so this only occurs when the resource's default propagation policy is being used.
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
cc
This is expected. The default garbage collection policy for old controller types (rc, rs, deployment) etc. is orphaning, for backward compatibility. And when doing BackgroundGC, the GC controller doesn't set the deleteOptions.propagationPolicy. We met a similar problem for deleting deployment -RS -pods. Because that is a very common use case, we made a special fix in As you noticed, if you set deleteOptions.propagationPolicy to Orphan or ForegroundGC, it works as you expected.
that seems wrong. if I'm using the garbage collection feature to propagate (whether a user specified it or it was a default of a parent resource), I expect GC to set propagation policy.
rather than a one-off fix in the RS controller, why not address the break in the propagation chain in the GC controller?
In the backgroundGC, the owner is deleted immediately, so unlike orphaning or the foregroundGC, there is no finalizer left for the GC controller to determine the intention of the user. In the absence of the user's intention, the GC controller doesn't set deleteOptions.PropagationPolicy when deleting dependents.
If it's not orphaning, and it's not foregroundGC, the only logical option that remains is backgroundGC
There is another case, user might intentionally choose not to set DeleteOptions.Propagation policy and wants to respect the existing finalizers of the dependents or the default GCstrategy defined by the REST storage of the dependents.

except that it isn't... if I delete resource X and specify no propagation policy, that resource strategy's default propagation policy should take effect just as if the user had specified it.
Add CronJobs to the list of affected resources, since you have CronJob-Job-Pod chain. :+1: to what Jordan said just above.

BUG REPORT: <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version: 1.6.1 Environment: Hardware configuration: 2 identical machines, with i7-2600K CPUs and 4GB RAM. OS: CentOS 7 Kernel: 3.10.0-514.10.2.el7.x86_64 Install tools: KubeADM What happened: Deleting a deployment via the API with the DeleteOptions settings or deletes the Deployment and the replica set but NOT the pods. Deleting it with the DeleteOptions setting works as intended. What you expected to happen: The replica set deletion should propagate to the pods. I've read that the garbage collection should work for Deployments in 1.6. How to reproduce it: Anything else we need to know: I am using the and at the moment it only supports the orphanDependants option, since it's still at api version 1.5.
cc:
We discussed the problem in I'll change the GC to make sure it's cascading.
Sent
Any progress on this?
Still trying to figure out a sane fix:
I've encountered the same issue. My pods are still running even though I deleted the deployment.
lets come up with a fix for 1.7. This should work from the first time we introduced cascading deletion in Deployments.
Yes, we should. Do you have any comments on
replied. Let's move this forward. If the fix is small, I am going to backport it to 1.6.
Offline discussion with we punt this to v1.8.
Discussed with if we can get the PR fixed soon, it can still go in 1.7. It's a real bug fix.
Yes, we need to fix this for 1.7 and also backport to 1.6.

Should this be targeting 1.7?
yes, the fix that went in in is going to cause us data issues we don't want
Are you working on this? What's the timeline to resolution?
Based on above comment, looks like actually introduced a new regression. Am I right?
In , I reverted what's introduced in .
- is working on it
I approved this for 1.7 based on comment at We should revert , but are we ok with since it also introducing a new behavior. I will leave to decide.
replaces with a fix that addresses this issue systemically. While this bug did exist in 1.6.x, more things rely on GC in 1.7 than in 1.6, and without that fix, GC leaves more resources vulnerable to unintentional orphaning. Since the issue is actually with general GC behavior, should weigh in.


Question: <del> [CHANGELOG/README.md](./CHANGELOG/README.md) </del> <ins> CHANGELOG/README.md </ins>  No newline at end of file

The change of this PR has resulted in the exception of the file. The file was originally a symbol file, but now other content has been , resulting in the exception of the file symbol. Rollback to the correct symbol file No response <details</details<details</details<details</details<details</details<details</details<details</details
/kind documentation
/triage accepted /priority important-soon /sig release


Question: cmdNamespace, err := f.DefaultNamespace() cmdutil.CheckErr(err) mapper, typer := f.Object() <del> r := resource.NewBuilder(mapper, typer, f.ClientMapperForCommand(cmd)). </del> <ins> r := resource.NewBuilder(mapper, typer, f.ClientMapperForCommand()). </ins> ContinueOnError(). NamespaceParam(cmdNamespace).RequireNamespace(). ResourceTypeOrNameArgs(false, args...).

I think the need for this argument to the factory methods has passed - it means that callers who want to provide configurable behavior have to do it through the factory itself, which is acceptable. We can remove it whenever.
I removed most of the unused arguments. But, There are a few methods still using the argument. Namely, , , and . Before proceeding any further I wanted to ask what should be done about these methods? Deprecated/Refactored/Removed ?
The factory methods were the most important. The pattern before was people using the flags by name (cmd.Flags().Get("foo")), but now factory methods must use pointer flag vars, so there's nothing that should need the command there. The printer methods are optional units that can be used when you add the flags for them to your command. Unless we refactor the definition of those flags to use pointers and return a printer struct (something I had started a while back and never finished), it's not worth changing them right now. The other consumer of factory is OpenShift - we wrap the factory to add our additional types (implementations of things like describers, action verbs, etc): - but we will pick up and remove use of cmd when we rebase.
Thanks for the explanation. I've cleaned the as well. Print method APIs are untouched as you suggested.
cc
Looks like this was done by PR


Question: vm, err := as.getVirtualMachine(nodeName) if err != nil { // if host doesn't exist, no need to detach <del> klog.Warningf("azureDisk - cannot find node %s, skip detaching disk %s", nodeName, diskName) </del> <ins> klog.Warningf("azureDisk - cannot find node %s, skip detaching disk(%s, %s)", nodeName, diskName, diskURI) </ins> return nil }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: add more logging in azure disk attach/detach What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Others: /sig azure /assign
The issue message here doesn't say why a change was desired. The PR "Original logging is not clear when dealing with managed disk, since diskName is empty". That's more informative.