Question: eventRecorder: eventBroadcaster.NewRecorder(scheme.Scheme, v1.EventSource{Component: "daemonset-controller"}), podControl: controller.RealPodControl{ KubeClient: kubeClient, <del> Recorder:   eventBroadcaster.NewRecorder(scheme.Scheme, v1.EventSource{Component: "daemon-set"}), </del> <ins> Recorder:   eventBroadcaster.NewRecorder(scheme.Scheme, v1.EventSource{Component: "daemonset-controller"}), </ins> }, crControl: controller.RealControllerRevisionControl{ KubeClient: kubeClient,

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: kubect create -f to create statefulset, and describe the statefulset, output the event likes <preEvents: FirstSeen LastSeen Count From SubObjectPath Type Reason Message --------- -------- ----- ---- ------------- -------- ------ ------- 1m 1m 1 statefulset Normal SuccessfulCreate create Claim www-web-0 Pod web-0 in StatefulSet web success 1m 1m 1 statefulset Normal SuccessfulCreate create Pod web-0 in StatefulSet web successful </preWhat you expected to happen: the From of event change to 'statefulset-controller' in output, let know the source of event more obvious, likes the event when create job and other resource(replicaset,replication). <preEvents: FirstSeen LastSeen Count From SubObjectPath Type Reason Message --------- -------- ----- ---- ------------- -------- ------ ------- 3m 3m 1 job-controller Normal SuccessfulCreate Created pod: pi-7gd2m </preHow to reproduce it (as minimally and precisely as possible): create a statefulset and describe it to see the event Anything else we need to know?: Environment: Kubernetes version (use ):kubernetes 1.7 Cloud provider or hardware configuration**: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/assign
/area controller-manager
/sig node


Question: // TestEntry is used to check whether the specified entry is in the set or not. func (runner *runner) TestEntry(entry string, set string) (bool, error) { if out, err := runner.exec.Command(IPSetCmd, "test", set, entry).CombinedOutput(); err == nil { <del> reg, e := regexp.Compile("NOT") </del> <ins> reg, e := regexp.Compile("is NOT in set " + set) </ins> if e == nil && reg.MatchString(string(out)) { return false, nil } else if e == nil {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--The method only finds string in output of , but when the name of the set contains , it will return no matter the entry exists in ipset or not. What happened: returns false whenever entry exists if the name of set contains , e,g: , , What you expected to happen: returns whether the entry exists whatever the set name is How to reproduce it (as minimally and precisely as possible): replace the set name to any name containing , for example in of , then cd to and run , and it fails on . Anything else we need to know?: Environment: Kubernetes version (use ): v1.11.6, but the bug still exists in Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: /sig network /kind bug
/assign
This page says that returns non-zero when entry does not exist in set, and zero when entry exists: and I tested the output of (ipset v6.29 on Ubuntu 16.04 LTS, kernel 4.4.0-116-generic),the result goes as follow: so maybe we can judge by return value of command and the separated output of stdout and stderr,


Question: }}, }) addPods(endpoints.podStore, ns, 1, 1, 0, false) <del> endpoints.serviceStore.Add(&v1.Service{ ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: ns}, </del> <ins> service := &v1.Service{ ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: ns, Labels: map[string]string{"a": "b"}}, </ins> Spec: v1.ServiceSpec{ Selector:  map[string]string{}, ClusterIP: api.ClusterIPNone, Ports:     []v1.ServicePort{}, }, <del> }) </del> <ins> } originalService := service.DeepCopy() endpoints.serviceStore.Add(service) </ins> endpoints.syncService(ns + "/foo") data := runtime.EncodeOrDie(testapi.Default.Codec(), &v1.Endpoints{ ObjectMeta: metav1.ObjectMeta{

bug (): Originally posted by in cc /sig network /priority critical-urgent
fix in
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.


Question: return client } <del> func newOIDCAuthProvider(_ string, cfg map[string]string, persister restclient.AuthProviderConfigPersister) (restclient.AuthProvider, error) { </del> <ins> func newOIDCAuthProvider(clusterAddress string, cfg map[string]string, persister restclient.AuthProviderConfigPersister) (restclient.AuthProvider, error) { </ins> issuer := cfg[cfgIssuerUrl] if issuer == "" { return nil, fmt.Errorf("Must provide %s", cfgIssuerUrl)

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: We using go-code kubectl cli package. We are a long running app serving to many different Kubernetes clusters. Each cluster has a different user credentials and bearer token to access it. But they all use the same issuer and clientid. The oidc cache is keyed off of these two only. So what happens is when we switch to another configuration it pulls in the oidc provider for the other cluster and that bearer token does not have authority to this other cluster. The cluster config has the correct info in it but the oidc provider ignores that and used the stale data. What you expected to happen: oidc provider would always use the appropriate current config context to access the cluster How to reproduce it (as minimally and precisely as possible): In same app instance create two cluster configs to two different clusters with same oidc issuer and clientid by DIFFERENT credentials. Where the credentials on one are not authorized for the other cluster. Try to get namespaces on each, one will work and the other won't. Anything else we need to know?: Environment: Kubernetes version (use ): latest go-code from as a package called from go application.Cloud provider or hardware configuration: IBM Kubernetes OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten


Question: return fmt.Errorf("only cifs mount is supported now, fstype: %q, mounting source (%q), target (%q), with options (%q)", fstype, source, target, options) } <ins> // lock smb mount for the same source getSMBMountMutex.LockKey(source) defer getSMBMountMutex.UnlockKey(source) </ins> if output, err := newSMBMapping(options[0], options[1], source); err != nil { if isSMBMappingExist(source) { klog.V(2).Infof("SMB Mapping(%s) already exists, now begin to remove and remount", source)

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: I am using Windows containers (Server Core ltsc 2019). These all have volumes mounted to three different Azure file shares under a single storage account. Multiple instances of these containers run on each node. On deploying the containers I get the following errors. Sometimes this does manage to resolve itself. MountVolume.SetUp failed for volume "log-share" : Remove-SmbGlobalMapping failed: exit status 1, output: "Remove-SmbGlobalMapping : No MSFTSmbGlobalMapping objects found with property 'RemotePath' equal to rn'storageaccounturllogs'. Verify the value of the property and retry.rnAt line:1 char:1rn+ Remove-SmbGlobalMapping -RemotePath $Env:smbremotepath -Forcern+ ~rn + CategoryInfo : ObjectNotFound: (storageaccounturllogs:String) [Remove-SmbGlobalMapping], Ci rn mJobExceptionrn + FullyQualifiedErrorId : CmdletizationQueryNotFoundRemotePath,Remove-SmbGlobalMappingrn rn" MountVolume.SetUp failed for volume "config-share" : New-SmbGlobalMapping failed: exit status 1, output: "New-SmbGlobalMapping : Generic failure rnAt line:1 char:190rn+ ... ser, Env:smbremotepath -Cred ...rn+ ~~~rn + CategoryInfo : NotSpecified: (MSFTSmbGlobalMapping:ROOT/Microsoft/...mbGlobalMapping) [New-SmbGlobalMa rn pping], CimExceptionrn + FullyQualifiedErrorId : HRESULT 0x80041001,New-SmbGlobalMappingrn rn" What you expected to happen: For the volumes to be mounted. This all worked ok in k8s 1.12.1 using Windows 1803. I have now upgraded to 1.13.4 and Windows ltsc2019.

How to reproduce it (as minimally and precisely as possible): Use aks-engine to create a 1.13.4 cluster with Windows nodes Create a storage account Create multiple file shares under the account. Add access credentials to the storage account as a secret on k8s Create a windows container with folders that will be used to mount the Azure storage account file shares. Deploy to k8s defining volume mounts and volumes in the container section of the deployment e.g. volumeMounts: name: folder1-share mountPath: "c:appfolder1" name: folder2-share mountPath: "c:appfolder2" volumes: name: folder1-share azureFile: secretName: azure-file-share-cred shareName: share1 readOnly: false name: folder2-share azureFile: secretName: azure-file-share-cred shareName: share2 readOnly: true Anything else we need to know?: This maybe related to the fix for Environment: Kubernetes version (use ): 1.13.4 Cloud provider or hardware configuration: Azure OS (e.g: ): Windows 2019 Kernel (e.g. ): 10.0..316 Install tools: aks-engine 0.13.2 Others:
/sig Azure
/sig windows
/assign
hi in your logs, the remotepath is , so is the share name? also, could you log on to that agent node, and provide , there are logs like
I think this is a race condition issue, let me file a PR to fix this, thanks for reporting


Question: func formatResourceUsageStats(containerStats framework.ResourceUsagePerContainer) string { // Example output: // <del> // Resource usage for node "e2e-test-foo-node-abcde": // container        cpu(cores)  memory(MB) // "/"              0.363       2942.09 // "/docker-daemon" 0.088       521.80 // "/kubelet"       0.086       424.37 // "/system"        0.007       119.88 </del> <ins> // Resource usage: //container cpu(cores) memory_working_set(MB) memory_rss(MB) //"kubelet" 0.068      27.92                  15.43 //"runtime" 0.664      89.88                  68.13 </ins> buf := &bytes.Buffer{} w := tabwriter.NewWriter(buf, 1, 0, 1, ' ', 0) fmt.Fprintf(w, "containertcpu(cores)tmemory_working_set(MB)tmemory_rss(MB)n")

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: Now, benchmark test make docker as the only runtime. What you expected to happen: benchmark test should be generic to all CRI container runtimes Related to How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/area test /area node-e2e
/sig node


Question: gce, err := fakeGCECloud(vals) // Loadbalancer controller code expects basepath to contain the projects string. expectBasePath := "https://compute.googleapis.com/compute/v1/projects/" <ins> // See https://github.com/kubernetes/kubernetes/issues/102757, the endpoint can have mtls in some cases. expectMtlsBasePath := "https://compute.mtls.googleapis.com/compute/v1/projects/" </ins> require.NoError(t, err) <del> if gce.service.BasePath != expectBasePath { t.Errorf("Compute basePath has changed. Got %q, want %q", gce.service.BasePath, expectBasePath) </del> <ins> if gce.service.BasePath != expectBasePath && gce.service.BasePath != expectMtlsBasePath { t.Errorf("Compute basePath has changed. Got %q, want %q or %q", gce.service.BasePath, expectBasePath, expectMtlsBasePath) </ins> } }

/sig cloud-provider /area provider/gcp run (or just . observe failure: <details</detailsalso the logs from this failure are pretty low signal/noise ... xref:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
cc ref:
I see the same failure too.. I think somehow the mtls endpoint is being generated on a call to which calls A previous method generates non-nil Client cert resource if the file is present in the home directory. . It depends on which machine the test is being run from. Deleting the file will probably make the test pass, but I dunno what else depends on that file. Is it ok if I changed the test to look for either or
Verified that the test passes with fix.
does this test depend on a live network connection then? or GCP credentials? either would be ... not ideal for a unit test.
thank you for looking into this.
it does not require either. The test is to make sure the go/compute API does not change the basePath syntax. We saw some version drop the "projects/" suffix, so want to make sure upgrading to a new compute API version does not break basePath assumptions.
Ok great, thanks. Do we know why it is changing locally? Ideally we want tests to reproduce locally, yet this dependency seems to be aware of CI vs local. EDIT: explains
thanks!


Question: taintsToAdd    []*v1.Taint expectedTaints []v1.Taint requestCount   int <ins> expectedErr    error </ins> }{ { name: "add one taint on node",

pkg/controller in I transformed that testcase into a fuzz driver and tested it with ]() It crash with "invalid memory address or nil pointer dereference" when some argument related data are fuzzed. Always No response No response Simplified PoC code is here: and it crashed as: /sig
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/kind cleanup /sig node /priority backlog This nil pointer is never reported by end users which means that before we call AddOrUpdateTaintOnNode, the node existence is checked. To make the code more stable, fixing it is valid. This would avoid future bugs if somewhere calls AddOrUpdateTaintOnNode without checking whether the node exists. I opened for this. /remove-kind failing-test (which means falling test ci )
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:


Question: By("deleting the pod") podClient.Delete(pod.Name) }() <del> _, err := podClient.Create(pod) </del> <ins> pod, err := podClient.Create(pod) </ins> if err != nil { <del> Fail(fmt.Sprintf("Failed to create pod: %v", err)) </del> <ins> Failf("Failed to create pod: %v", err) </ins> } expectNoError(waitForPodRunning(c, pod.Name))

This is wrong because something else could have updated the pod in the mean time, invalidating your resource version. (Specifically, kubelet writes statuses.) It should do a get/update loop that retries as long as it gets a conflict error. to find an owner
PR almost ready - just struggling to get e2e tests to run :-)
At last. SUCCESS! -- 5 Passed 34 Skipped I0424 09:04:47. ] All tests pass PR incoming in a few minutes.


Question: if l := len(user); l == 0 { errMsg := fmt.Sprintf("runAsUserName's User cannot be empty") allErrs = append(allErrs, field.Invalid(fieldPath.Child("runAsUserName"), windowsOptions.RunAsUserName, errMsg)) <del> } else if l >= maxRunAsUserNameUserLength { errMsg := fmt.Sprintf("runAsUserName's User length must be under %d characters", maxRunAsUserNameUserLength) </del> <ins> } else if l > maxRunAsUserNameUserLength { errMsg := fmt.Sprintf("runAsUserName's User length must not be longer than %d characters", maxRunAsUserNameUserLength) </ins> allErrs = append(allErrs, field.Invalid(fieldPath.Child("runAsUserName"), windowsOptions.RunAsUserName, errMsg)) }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Currently, the character limit for the usernames set in the field is 20, which is too low, considering that "ContainerAdministrator" is a valid username and it is longer than 20 characters. A user should be able to run containers as Administrator, if needed. What you expected to happen: Users should be able to spawn pods which run as. How to reproduce it (as minimally and precisely as possible): WindowsRunAsUserName feature gate. () Anything else we need to know?: /sig windows /priority important-soon /milestone 1.16 Environment: Kubernetes version (use ): 1.16 Cloud provider or hardware configuration: N/A OS (e.g: ): Windows Server 2019 Kernel (e.g. ): N/A Install tools: N/A Network plugin and version (if this is a network-related bug): N/A Others:
You must be a member of the GitHub team to set the milestone. If you believe you should be able to issue the /milestone command, please contact your and have them propose you as an additional delegate for this responsibility. <detailsIn response to : <!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! --2. ` /priority important-soon /milestone 1.16 Kubernetes version (use ): 1.16 Cloud provider or hardware configuration: N/A OS (e.g: ): Windows Server 2019 Kernel (e.g. ): N/A Install tools: N/A Network plugin and version (if this is a network-related bug): N/A Others: </details
/milestone v1.16


Question: if !api.Semantic.DeepDerivative(job, testJob) { t.Errorf("Expected %#v, but got %#v", testJob, job) } <del> received <- key </del> <ins> close(received) </ins> return nil } // Start only the job watcher and the workqueue, send a watch event,

Ran a test executable with more than 20,000 times and no failures. Any hints as to how to get it to show up? I'm running 100 jobs at a time, so my system is stressed already.
I to am unable to reproduce this. I get similar errors if I reduce the timeout at the top of the file. I spent a bit looking at how to rewrite the tests without the timeout, but nothing jumped out at me. Looking at it though, it's really just a 500ms timeout for the test to complete. I'm just going to remove them. Go has a default timeout of 10 minutes. If it's going to pass, it will pass then. If not, Go will give you a nice traceback where it deadlocked.


Question: ginkgo.By(fmt.Sprintf("Checking if the volume in pod%d exists as expected volume mode (%s)", index, *pvc.Spec.VolumeMode)) utils.CheckVolumeModeOfPath(f, pod, *pvc.Spec.VolumeMode, path) <ins> if readOnly { ginkgo.By("Skipping volume content checks, volume is read-only") continue } </ins> if i != 0 { ginkgo.By(fmt.Sprintf("From pod%d, checking if reading the data that pod%d write works properly", index, index-1)) // For 1st pod, no one has written data yet, so pass the read check

What happened: Running two pods that use the same iSCSI volume, one of them fails with: What you expected to happen: Both pods should be running Anything else we need to know?: It's caused by iscsi volume plugin persiting iSCSI settings into the mount directory itself (e.g. ): volume plugin creates the global mount directory. volume plugin creates there. volume plugin mounts the volume there. This shadows the file, but it's fine, as long as no other pod uses the same volume on the same node. creates a second pod that uses the volume. volume plugin tries to create in the global volume directory. Since the volume is mounted, it will either writes the file to the volume (!) or fails, if the volume is read-only.
/sig storage /assign /priority impotant-soon
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig storage /assign /priority impotant-soon Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/priority important-soon
Our e2e did not catch this, because they always create only one pod when testing read-only volumes.


Question: } state, exists := lb.services[svcPort] <del> if !exists { </del> <ins> if !exists || state == nil { </ins> return } for _, existingEndpoint := range state.endpoints {

As suggested in , some portions of the code contains the map checking for nil values in map, and others don't have. Even though we're never assigning nil values to this map, we should standardize the portions that does not have this check in module. Example: Should be like this: I can take care of this issue.
/sig network
There have been discussions of deprecating the user-space proxier. Maybe transfer it to . A question is if anybody still uses it. Seems like you do, so if possible please explain if you have a use-case where the user-space proxier is wanted/needed.
Hey Actually, I was casually reading the source code and I saw a over there regarding the nil checks. Regarding your question, we should check if anyone actually uses it though, or maybe discuss its usage or deprecation a little further.
Quick question though: I saw in the news that the user-space proxy porting was near completion, maybe it's worth talking to them in order to check upon the status.
I don't object to making it clearer or more consistent, if you want to spend time on it, but it's not a high priority. That said, I think a nil value in that map would be really bad (and should be impossible) and I don't know how we would raise a signal about it.
let us know if you want to try to fix it, otherwise we can just close this.
I already opened a PR with the clearer nil checks for this specific module in , so you can review it. I think whether this would have nil values into the service map we should discuss this apart and try to reproduce this case (if any would happen).
PR submitted, so let's review and discuss further there. /assign dcbw /triage accepted


Question: framework.ExpectEqual(updatedNamespace.ObjectMeta.Labels[ns], "updated", "Failed to update namespace %q. Current Labels: %#v", ns, updatedNamespace.Labels) framework.Logf("Namespace %q now has labels, %#v", ns, updatedNamespace.Labels) }) <ins> ginkgo.It("should apply a finalizer to a Namespace", func() { fakeFinalizer := v1.FinalizerName("e2e.example.com/fakeFinalizer") var updatedNamespace *v1.Namespace nsName := "e2e-ns-" + utilrand.String(5) ginkgo.By(fmt.Sprintf("Creating namespace %q", nsName)) testNamespace, err := f.CreateNamespace(nsName, nil) framework.ExpectNoError(err, "failed creating Namespace") ns := testNamespace.ObjectMeta.Name nsClient := f.ClientSet.CoreV1().Namespaces() framework.Logf("Namespace %q has %#v", testNamespace.Name, testNamespace.Spec.Finalizers) ginkgo.By(fmt.Sprintf("Adding e2e finalizer to namespace %q", ns)) err = retry.RetryOnConflict(retry.DefaultRetry, func() error { updateNamespace, err := nsClient.Get(context.TODO(), ns, metav1.GetOptions{}) framework.ExpectNoError(err, "Unable to get Namespace %q", ns) updateNamespace.Spec.Finalizers = append(updateNamespace.Spec.Finalizers, fakeFinalizer) updatedNamespace, err = nsClient.Finalize(context.TODO(), updateNamespace, metav1.UpdateOptions{}) return err }) framework.ExpectNoError(err, "failed to add finalizer to the namespace: %q", ns) var foundFinalizer bool for _, item := range updatedNamespace.Spec.Finalizers { if item == fakeFinalizer { foundFinalizer = true break } } framework.ExpectEqual(foundFinalizer, true, "Finalizer %q was not found. Namespace %q has %#v", fakeFinalizer, updatedNamespace.Spec.Finalizers) framework.Logf("Namespace %q has %#v", updatedNamespace.Name, updatedNamespace.Spec.Finalizers) ginkgo.By(fmt.Sprintf("Removing e2e finalizer from namespace %q", ns)) err = retry.RetryOnConflict(retry.DefaultRetry, func() error { updatedNamespace, err = nsClient.Get(context.TODO(), ns, metav1.GetOptions{}) framework.ExpectNoError(err, "Unable to get namespace %q", ns) var finalizerList []v1.FinalizerName for _, item := range updatedNamespace.Spec.Finalizers { if item != fakeFinalizer { finalizerList = append(finalizerList, item) } } updatedNamespace.Spec.Finalizers = finalizerList updatedNamespace, err = nsClient.Finalize(context.TODO(), updatedNamespace, metav1.UpdateOptions{}) return err }) framework.ExpectNoError(err, "failed to remove finalizer from namespace: %q", ns) foundFinalizer = false for _, item := range updatedNamespace.Spec.Finalizers { if item == fakeFinalizer { foundFinalizer = true break } } framework.ExpectEqual(foundFinalizer, false, "Finalizer %q was found. Namespace %q has %#v", fakeFinalizer, updatedNamespace.Spec.Finalizers) framework.Logf("Namespace %q has %#v", updatedNamespace.Name, updatedNamespace.Spec.Finalizers) }) </ins> }) func unstructuredToNamespace(obj *unstructured.Unstructured) (*v1.Namespace, error) {

[X] Test approval issue: [X] Test PR: 15 Oct 2022 [X] Two weeks soak end date: 28 Oct 2022 [X] Test promotion PR: According to this APIsnoop query, there are still one remaining Namespace endpoint which is untested. replaceCoreV1NamespaceFinalize Note: The test also removes the fake finalize from before calling the finalize action on the test namespace a second time. This will allow the test framework to clean up all namespaces when the e2e finishes, else the test namespace will be stuck in the state. Using a number of existing e2e test practices a new has been created for one Namespace endpoint. The e2e logs for this test are listed below. This query shows that the Finalize endpoint is hit within a short period of running this e2e test If a test with these calls gets merged, test coverage will go up by 1 point This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: if c.Started != nil && *c.Started { // Stop probing for startup once container has started. <ins> // we keep it running to make sure it will work for restarted container. </ins> if w.probeType == startup { <del> return false </del> <ins> return true </ins> } } else { // Disable other probes until container has started.

We detected behavior change in v1.21.0 Example: POD has and for some containers Behavior in v1.20.5: starts applied startupProbe success, then start checks for livenessProbe working failed restarted startupProbe applied startupProbe success, then start checks for livenessProbe working Behavior in v1.21.0: starts applied startupProbe success, then start checks for livenessProbe working failed restarted applied, startupProbe skipped not ready yet, killed by liveness check in restart loop So, in v1.21 is applied to "fresh" starts only and not used when container restarted. There is not mentions in changelog that behavior of the is changed. should run on container restarts also. POD with and fail that doesn't apply after container restart Kubernetes version (use ):Cloud provider or hardware configuration: baremetal
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
/assign
Thanks for the quick fix
it was fixed in master (1.22) but not for 1.21, does it need a backport? /reopen
Reopened this issue. <detailsIn response to : it was fixed in master (1.22) but not for 1.21, does it need a backport? /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/close never mind i just saw the backports.
Closing this issue. <detailsIn response to : /close </details
can you summarize the impacted patch releases (looks like the regression was backported to 1.18-1.20 as well)
These 4 patch releases are impacted, 1.18.18, 1.19.10, 1.20.6, 1.21.0


Question: "testing" "time" <del> "go.etcd.io/etcd/api/v3/mvccpb" clientv3 "go.etcd.io/etcd/client/v3" </del> storagetesting "k8s.io/apiserver/pkg/storage/testing" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"

While working on generalizing tests to somehow better define spec for storage interface (and share them across all implementation of storage.interface - in particular etcd3 and watchcache), I found an unexpected inconsistency in Delete operation in etcd3. What is happening is that it effectively returns the state of the object from just before the deletion: However, this is inconsistent with how our watch is working, which is returning the state of the object from just before the deletion, but it sets the RV to the moment of deletion (which Delete call isn't doing). This sounds like a bug to me, but given this is a long-standing behavior we may need to take additional care when fixing this. /kind bug /sig api-machinery /priority important-soon /cc
I opened as a fix for it.
/triage accepted


Question: its label selector. release: v1.20 file: test/e2e/instrumentation/core_events.go <del> - testname: Event resource lifecycle codename: '[sig-instrumentation] Events should ensure that an event can be fetched, patched, deleted, and listed [Conformance]' description: Create an event, the event MUST exist. The event is patched with a new message, the check MUST have the update message. The event is deleted and MUST NOT show up when listing all events. release: v1.20 </del> <ins> - testname: Event, manage lifecycle of an Event codename: '[sig-instrumentation] Events should manage the lifecycle of an event [Conformance]' description: Attempt to create an event which MUST succeed. Attempt to list all namespaces with a label selector which MUST succeed. One list MUST be found. The event is patched with a new message, the check MUST have the update message. The event is updated with a new series of events, the check MUST confirm this update. The event is deleted and MUST NOT show up when listing all events. release: v1.25 </ins> file: test/e2e/instrumentation/core_events.go - testname: DNS, cluster codename: '[sig-network] DNS should provide /etc/hosts entries for the cluster [Conformance]'

[X] test approval issue : [X] test pr : [x] two weeks soak end date : 15 Jul 2022 [x] test promotion pr : According to this APIsnoop query, there are still a single remaining Event endpoint which is untested. replaceCoreV1NamespacedEvent Using a number of existing e2e test practices a new has been created for this pod status endpoint. The e2e logs for this test are listed below. This query shows the event endpoints hit within a short period of running the e2e test. If a test with these calls gets merged, test coverage will go up by 1 point. This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: PREEMPTIBLE_NODE=${PREEMPTIBLE_NODE:-false} PREEMPTIBLE_MASTER=${PREEMPTIBLE_MASTER:-false} <del> OS_DISTRIBUTION=${KUBE_OS_DISTRIBUTION:-debian} MASTER_IMAGE=${KUBE_GCE_MASTER_IMAGE:-container-vm-v20160321} </del> <ins> OS_DISTRIBUTION=${KUBE_OS_DISTRIBUTION:-gci} # For GCI, leaving it blank will auto-select the latest GCI image on the `dev` # channel. MASTER_IMAGE=${KUBE_GCE_MASTER_IMAGE:-} </ins> MASTER_IMAGE_PROJECT=${KUBE_GCE_MASTER_PROJECT:-google-containers} <del> NODE_IMAGE=${KUBE_GCE_NODE_IMAGE:-"${MASTER_IMAGE}"} NODE_IMAGE_PROJECT=${KUBE_GCE_NODE_PROJECT:-"${MASTER_IMAGE_PROJECT}"} </del> <ins> NODE_IMAGE=${KUBE_GCE_NODE_IMAGE:-container-vm-v20160321} NODE_IMAGE_PROJECT=${KUBE_GCE_NODE_PROJECT:-google-containers} </ins> CONTAINER_RUNTIME=${KUBE_CONTAINER_RUNTIME:-docker} RKT_VERSION=${KUBE_RKT_VERSION:-0.5.5}

We recently switched GKE's master node to GCI image, but not GCE master node. This discrepancy causes a lot of issues for us, especially some prs merged might work on GCE, but totally break GKE. We should get rid of that difference. cc/
/cc (two build cops who have recently suffered the effects of this).
The master image for test is defined here: We want to switch all test jobs? Or just one particular job, e.g., k8s ci testing k8s master head? I am afraid that simply replacing the image in that script will affect all test jobs
I think this is a bad plan unless we continue to test container VM start up somehow. We can't break that while people still use it!
An alternative: Set up another jenkins job dedicated for the purpose and mark it as commit queue blocker.
It may be better to just switch GKE masters back to containerVM for now, then redo the sequence (i.e. switch GCE to GCI, then GKE master to GCI).
if we switch both the test and non-test default from cvm -gci then starting with the 1.3 release people won't be using it (by default), so testing becomes less important.
I don't think we have to. The GKE CI job is already a CQ blocker, but it runs after submit so bad PRs still get merged (though they get identified quickly). What we need is a PR builder that runs tests with GCI, so we can block bad changes in the first place.
Definition of PR builers: cc/
Yes. It could be even simpler, it just has to start up a cluster and tear it down. Just run like 1 fast test.
We've got to do something though, the queue has been blocked for like 3 hours on GKE tests not passing.
I'd still like to run the e2e tests as some tests make assumptions of the OS distro and thus fail on GCI. Can we duplicate (that runs gce-e2e on every PR), replacing ContainerVM with GCI as the master and (or) node image?
It's a bit of a pain to do this, since that job also does a full build and it feels like a major waste to build each PR again. The Jenkins agents probably can't handle that much load, anyway.

at some point we probably need to bite the bullet and set up pipelines so we only build kubernetes once, rather than separately in the unit/integration, e2e, and node e2e jobs. I have no idea how well any of that interacts with the pull request plugin, of course.
Thanks for the information . So looks like the first alternative, running tests with GCI/GKE and block bad changes in the first place, is hard to get done due to some infrastructure challenges (as mentioned in ). It's not impossible though, just not prioritized right now. The second alternative would be, keep running the post-submit tests (the GKE e2e tests), and fix issues as they appear, which is not too bad imo since they run every 30 min and block submit queue on failure. At the end of the day, you cannot prevent 100% of breakages in presubmit tests, and that's why we have to have postsubmit tests to complement.
I think start another jenkins job as one more pr builder, which have the master node on GCI image and rest of nodes on ContainerVM is the simplest thing unblock us. The newly created job should only run the basic test suites. After 1.3, I suggested we do
No, this is not acceptable! we've been blocked for 10 out of the last 24 hours due to this! :)
I agree that we need to catch this before merging instead of after merging. The question is how best to do so.
The problem is that the submit-queue puts them in at odd hours, and we don't expect OSS oncall to be watching at 2 AM.
Well, the flip side is you don't have to run all tests (on all providers) so that all your PRs get merged faster. I am not saying I prefer postsubmit tests, it just something you have to face, no matter its gke tests or gce tests. I would prefer that too.
I think the solution to this is to run more tests when the PR is created, but then when the submit queue asks for a retest before merging only run a smoke test. I think that's a good tradeoff.
That's a good topic to talk and I have some ideas about that too, but I'd stop dragging the focus too far.

:smile:
I don't see any compelling reasons to have a different master image-type for GCE and GKE -- there are strong benefits to them being identical, as we've observed. We want 1.3+ to run GCI for GKE, therefore I propose we switch both GCE and GKE to use GCI for the master on all 1.3+ tests by default, including PR and post-submit. If there is a testing gap in that situation, because we think it's important to support containerVM for 1.3+ masters on GCE (but not GKE), then add coverage for that gap. I don't have enough context to know whether that's pre-submit or post-submit or both.
(To append: I think post-submit coverage is horrible and would support a P0 of any test tiger team and/or code-yellow exit criteria being that we have pre-submit coverage of all the various test matrix points. I realize that'll be expensive in both time and machine resources, but it'll be less expensive than our current approach and I suspect generate significant velocity benefits.)
This is a decent idea.
is out for vacation, could you please pick up this one? I think we agreed to have a separate jenkins job as one more pr builder, which have the master node on GCI image and rest of nodes on ContainerVM. The newly created job should only run the basic test suites. If you need some help, please ask
After talking with and we came to the following agreement: will switch the default base image for master nodes running on GCE from debian container-vm to GCI. will add a jenkins job that runs periodically (post-submit) that will continue to test master nodes running the debian container-vm, initially running against the master branch. will pin this job to the 1.3 release branch once it is cut, and no longer run it against the master branch. will delete the new jenkins job once we have full confidence that we will not roll back the master image from GCI to the debian container-vm, sometime between now and the 1.4 release.
I am going to work on a PR for no. 1. cc/
no. 2 is being in
The build broke again because of this:


Question: allowedReplacementPods = make([]string, 0, len(nodeToDaemonPods)) } allowedReplacementPods = append(allowedReplacementPods, oldPod.Name) <ins> numUnavailable++ </ins> case numUnavailable >= maxUnavailable: // no point considering any other candidates continue

Starting with a cluster of three nodes, with a healthy deamonset pod running on each, with . When performing three updates to the daemonset, each with some reason that prevents the new pods from becoming ready, I ended up with zero ready pods. I would expect that when I apply an update, and the new pods fail to become ready, when I apply a second update the "new" failing pods are replaced before considering any of the oldest healthy pods. This is similar behavior to how Deployments work. On a three node (or any count really) cluster, create this daemonset, which is just an nginx pod with a startup probe hitting port 80. Update the startup probe port to something other than port 80, which should cause the pod to never become ready, and apply again. You can see the new unhealthy pod, and the generation of each pod Repeat this process for as many nodes/pods as you have (2 more in this example), changing anything that would cause the pods to be recreated (in this case I just cycled the startup probe port a few times). After the 2nd update, you can see that it has replaced both the 2nd generation pod (the one that was failing), and one of the first generation pods. Im now down to 1 out of 3, when I had specified . One more apply / update, and all generation 1 pods are gone, and all pods are unhealthy. No response <details</details<detailsReproduced this on bare metal, and GKE/GCP </detailsNo response No response No response No response
/sig apps
The Kubernetes project currently lacks enough contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/triage accepted /priority important-longterm


Question: return "", "" } <ins> // Removes a scsi device based upon /dev/sdX name func removeFromScsiSubsystem(deviceName string, io ioHandler) { fileName := "/sys/block/" + deviceName + "/device/delete" glog.V(4).Infof("fc: remove device from scsi-subsystem: path: %s", fileName) data := []byte("1") io.WriteFile(fileName, data, 0666) } </ins> // rescan scsi bus func scsiHostRescan(io ioHandler) { scsi_path := "/sys/class/scsi_host/"

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: When user uses fc volume plugin, device file on operating system such as /dev/sdX remains after deleting a Pod. After deleting the Pod, PVC and PV, cluster admin or external-provisioner might delete the disk from the storage, therefore the block device on the worker node should be cleaned up. The photon volume plugin already has this functionality in the removeFromScsiSubsystem(). Also the gce volume plugin has similar functionality in the udevadmChangeToDrive() too. photon: echo 1 /sys/block/<dev name/device/delete gce: udevadm trigger --action=change --property-match=DEVNAME=%s Also os-brick library in the OpenStack Cinder component uses same way with photon driver. What you expected to happen: The device file on operating system such as /dev/sdX will be cleaned up after deleting PVs. Environment: Kubernetes version (use ):Cloud provider or hardware configuration**: KVM OS (e.g. from /etc/os-release): Fedora 25 Kernel (e.g. ):Install tools: local-up- Others:
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig storage
I'll work on this.
Reiterating the mentions to trigger a notification: <detailsIn response to : I'll work on this. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: By("updating the Service to use the standard tier with a requested IP") svc = jig.UpdateServiceOrFail(ns, svc.Name, func(svc *v1.Service) { svc.Spec.LoadBalancerIP = requestedIP <del> setNetworkTier(svc, gcecloud.NetworkTierAnnotationStandard.ToGCEValue()) </del> <ins> setNetworkTier(svc, string(gcecloud.NetworkTierAnnotationStandard)) </ins> }) // Verify that service has been updated properly. Expect(svc.Spec.LoadBalancerIP).To(Equal(requestedIP))

Is this a BUG REPORT or FEATURE REQUEST?: /kind e2e-test-failure /sig network /area platform/gce What happened: From https://k8s-, network tier test is failing since 2/24. From the , seems a bit suspicious. Example failure:
/assign agau4779
GitHub didn't allow me to assign the following users: agau4779. Note that only and repo collaborators can be assigned. <detailsIn response to : /assign agau4779 Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: <ins> // +build !windows /* Copyright 2019 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package metrics import ( "os" "github.com/prometheus/procfs" ) func getProcessStart() (float64, error) { pid := os.Getpid() p, err := procfs.NewProc(pid) if err != nil { return 0, err } if stat, err := p.Stat(); err == nil { return stat.StartTime() } return 0, err } </ins>

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Start kubelet on a Windows node and you will get this error: What you expected to happen: The process start time to be read appropriately for Windows How to reproduce it (as minimally and precisely as possible): Start kubelet on a Windows node Anything else we need to know?: The gauge falls back to but we get this error. This information is avaliable via the windows package so we should be able to get an accurate time. Environment: Kubernetes version (use ): 1.20.1 Cloud provider or hardware configuration: azure OS (e.g: ): Windows 2019 Kernel (e.g. ): Version 10.0..1637 Install tools: aks-engine (but would be any) Network plugin and version (if this is a network-related bug): Others: /sig windows /assign
/triage accepted


Question: // TestClientReceivedGOAWAY tests the in-flight watch requests will not be affected and new requests use a new // connection after client received GOAWAY. func TestClientReceivedGOAWAY(t *testing.T) { <del> t.Skip("disabled because of https://github.com/kubernetes/kubernetes/issues/94622") </del> s, err := newTestGOAWAYServer() if err != nil { t.Fatalf("failed to set-up test GOAWAY http server, err: %v", err)

Before merged, TestClientReceivedGOAWAY failed with data race on . However, as far as I know, it's not the root cause of TestClientReceivedGOAWAY flake, because dial a new connection only when client send a new request, and in this case, requests are sent one by one. In other words, if data race on , then the additional unexpected connection set up, which means the test case or the configuration has problem. After merged, TestClientReceivedGOAWAY still failed in many jobs, such as , , which may be a good proof of my suspect. And I got a failed job with data race issue that read at (main testing goroutine) and write at , which told us the http client try to set-up a new connection even when we had already finished all requests(may caused by goroutine leak/block in http client). I'm still debug the root cause of TestClientReceivedGOAWAY flake, for now, I doubt that I used an incorrect http2 server/client configuration( or SETTINGSMAXCONCURRENT_STREAMS limited), or the dial function ignored context.Context which will cause more than one goroutine dial the connection when the first canceled(not found any where will cancel the requests). I still can not reproduce this failed case and it's hard to find the root cause. Couple of days may be needed to find the root cause, and I will post a PR to skip TestClientReceivedGOAWAY before I solved the problem. Sorry for any inconvenient.
/assign /sig testing
cc
Thank you for working on this!! How often does it flake? I'm just wondering if we should disable it in the mean time to avoid angry contributors.
On my PC, about 1/1000. I will post a PR to disable it for now.
Thank you so much! That's not a bad rate at all, but we run a lot of tests. This is super helpful!
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or .

/lifecycle stale
/lifecycle stale I'm still working on it, a new golang version(1.16 or 1.15.7+) release required
/sig api-machinery /remove-sig testing AFAICT this is specific to these tests, not the test framework or CI infrastructure.
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
the tests re-enabled in started flaking again immediately Example failure: /milestone v1.21 the tests should be deflaked or disabled for 1.21
/cc
/triage accepted
+1 in recent PR tests
Sorry for my slow response. I'm working on it, looks like I need to debug golang HTTP client again :-P
Shall we revert in the meantime?
No, has already fixed by golang 1.15.8 I think. This one is caused by another problem.
I will disable TestClientReceivedGOAWAY case again if I can not find the root cause this night.
Can not reproduce on my PC, disable this test case before I figure out the root cause:
I run this test case with this command and can not reproduce. Does any thing different with command ?
moving out of the milestone and lowering priority becuase merged, leaving open because the fundamental issue is unresolved /milestone clear
will do something comparable under make. There's a bunch of stuff in there, generally it's not relevant for a unit test bug like this but the answer to what exactly it does is not easily concise. Note that CI may have more load than your local machine running it once, and also currently runs as root.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/lifecycle active /assign ref:

This fails close to 100% of the time when run with race mode: Disabled in It is unclear if the test or the component needs to be corrected /sig api-machinery /priority critical-urgent
/assign /cc
GitHub didn't allow me to assign the following users: answer1991. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
The test is also flaky. but this one is hard to reproduce locally.
I will debug and try to fix it later, thanks for the report.
I doubt data race is caused by golang http/http2 pkg, let me try to make some reproduce cases and may open an issue to golang repo.
data race is caused by golang http/http2 pkg, I have already opened an issue at golang repo. I will continue to debug tomorrow.
another case:
For case, change to be would be a solution: But I would rather figure out why client will set-up the additional TCP connections.
Fix TestGOAWAYConcurrency flaky may need a new golang version after golang/go fixed. TestClientReceivedGOAWAY will be tracked at .
Great find-- can we skip the test for now while we wait for the go fix to land?
Yes, I think so.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/lifecycle stale I'm still working on it, a new golang version(1.16 or 1.15.7+) release required
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with .

Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
As clients in k/k repo use to config http2 transports instead of use golang SDK h2_bundle, a new version of (with ) will fix this issue. However, for other clients which use default http2 configuration in golang SDK, golang 1.15.8 or 1.16+ required to avoid HTTP2 un-expected error when received GOAWAY frame.
15.8 should be landed now
yes, I have already post a pr to enable goaway feature test cases:


Question: // TestClientReceivedGOAWAY tests the in-flight watch requests will not be affected and new requests use a new // connection after client received GOAWAY. func TestClientReceivedGOAWAY(t *testing.T) { <del> t.Skip("disabled because of https://github.com/kubernetes/kubernetes/issues/94622") </del> s, err := newTestGOAWAYServer() if err != nil { t.Fatalf("failed to set-up test GOAWAY http server, err: %v", err)

Before merged, TestClientReceivedGOAWAY failed with data race on . However, as far as I know, it's not the root cause of TestClientReceivedGOAWAY flake, because dial a new connection only when client send a new request, and in this case, requests are sent one by one. In other words, if data race on , then the additional unexpected connection set up, which means the test case or the configuration has problem. After merged, TestClientReceivedGOAWAY still failed in many jobs, such as , , which may be a good proof of my suspect. And I got a failed job with data race issue that read at (main testing goroutine) and write at , which told us the http client try to set-up a new connection even when we had already finished all requests(may caused by goroutine leak/block in http client). I'm still debug the root cause of TestClientReceivedGOAWAY flake, for now, I doubt that I used an incorrect http2 server/client configuration( or SETTINGSMAXCONCURRENT_STREAMS limited), or the dial function ignored context.Context which will cause more than one goroutine dial the connection when the first canceled(not found any where will cancel the requests). I still can not reproduce this failed case and it's hard to find the root cause. Couple of days may be needed to find the root cause, and I will post a PR to skip TestClientReceivedGOAWAY before I solved the problem. Sorry for any inconvenient.
/assign /sig testing
cc
Thank you for working on this!! How often does it flake? I'm just wondering if we should disable it in the mean time to avoid angry contributors.
On my PC, about 1/1000. I will post a PR to disable it for now.
Thank you so much! That's not a bad rate at all, but we run a lot of tests. This is super helpful!
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or .

/lifecycle stale
/lifecycle stale I'm still working on it, a new golang version(1.16 or 1.15.7+) release required
/sig api-machinery /remove-sig testing AFAICT this is specific to these tests, not the test framework or CI infrastructure.
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
the tests re-enabled in started flaking again immediately Example failure: /milestone v1.21 the tests should be deflaked or disabled for 1.21
/cc
/triage accepted
+1 in recent PR tests
Sorry for my slow response. I'm working on it, looks like I need to debug golang HTTP client again :-P
Shall we revert in the meantime?
No, has already fixed by golang 1.15.8 I think. This one is caused by another problem.
I will disable TestClientReceivedGOAWAY case again if I can not find the root cause this night.
Can not reproduce on my PC, disable this test case before I figure out the root cause:
I run this test case with this command and can not reproduce. Does any thing different with command ?
moving out of the milestone and lowering priority becuase merged, leaving open because the fundamental issue is unresolved /milestone clear
will do something comparable under make. There's a bunch of stuff in there, generally it's not relevant for a unit test bug like this but the answer to what exactly it does is not easily concise. Note that CI may have more load than your local machine running it once, and also currently runs as root.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/lifecycle active /assign ref:


Question: } klog.V(3).InfoS("Starting watch", "path", req.URL.Path, "resourceVersion", opts.ResourceVersion, "labels", opts.LabelSelector, "fields", opts.FieldSelector, "timeout", timeout) ctx, cancel := context.WithTimeout(ctx, timeout) <del> defer cancel() </del> <ins> defer func() { cancel() }() </ins> watcher, err := rw.Watch(ctx, &opts) if err != nil { scope.err(err, w, req)

Optimize the memory usage of goroutine stack for serving watch requests. According to the of handling a watch request, the stack memory is 0x3768, which is very close to 16 KiB and Golang runtime will double it to 32 KiB once exceeded. In fact, it does exceed if we print the traceback deeper and take the fact that extra stack memory reserved by Golang itself into consideration. A naive optimization is to run as much as preparation handlers in a separate goroutine and then move the actual serving to another goroutine, so the former routine can free the stack memory earlier and make it more memory efficient. Given the extensive use of watch requests, 32 KiB per watch could add up to a very high amount of memory overhead for . A typical watch request commonly lasts for 30s or longer, which means this memory will be active for quite long. I see a great improvement in memory efficiency to do this optimization.
/cc
/sig api-machinery
/cc if you would like to take a look when you have time :) Thank you /triage accepted


Question: gotNames.Insert(pod.Name) } <del> assert.Equal(t, 0, expectedNames.Difference(gotNames).Len(), "expected %v, got %v", expectedNames.List(), gotNames.List()) assert.Equal(t, 0, gotNames.Difference(expectedNames).Len(), "expected %v, got %v", expectedNames.List(), gotNames.List()) </del> <ins> if diff := cmp.Diff(expectedNames.List(), gotNames.List()); diff != "" { t.Errorf("Active pod names (-want,+got):n%s", diff) } </ins> } func TestSortingActivePods(t *testing.T) {

During implementation of KEP 3393 (PodReplacementPolicy for Jobs) we found that the unit tests of do not follow common patterns established in k/k. 1) The tests are not written using table driven design (). 2) The tests use rather than . This probably should be done as two separate PRs but wanted to highlight some areas that could use some cleanup. Not using means that we don't see diffs in the unit tests that makes this difficult. And not using table driven tests makes it difficult to add new test cases as it usually is brought up during review that we should use a table-driven test but the entire file does not obey this format.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig apps /kind cleanup /help /good-first-issue
This request has been marked as suitable for new contributors. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /sig apps /kind cleanup /help /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I'd like to work on it /assign
I'd also like to make some contributions. /assign
Looks like this need two PRs. tests are not written using table driven design (). tests use Differences rather than Cmp.Diff. I think I can work on the 2nd one and leave the 1st one to you.

Sounds good?
By looking through the , I found the main function to comapre the results have already used cmp.Diff. Can you elaborate more on which part you expect to use cmp.Diff? func verifyDeletedAndPatchedPods(t testing.T, client fake.Clientset, wantDeletedPodNames, wantPatchedPodNames sets.String) { t.Helper() deletedPodNames := getDeletedPodNames(client) if diff := cmp.Diff(wantDeletedPodNames, deletedPodNames); diff != "" { t.Errorf("Deleted pod names (-want,+got):n%s", diff) } patchedPodNames := getPatchedPodNames(client) if diff := cmp.Diff(wantPatchedPodNames, patchedPodNames); diff != "" { t.Errorf("Patched pod names (-want,+got):n%s", diff) } }
I am really sorry about this. Too much multitasking!! I meant not
Hey ! Sorry, I didn't see this before. No problem at all, but consider that one change depends on the other, and since my task implies a big refactor, there might be conflicts when merging your PR... That's why I see two possible scenarios happening: Your PR gets merged before I raise mine: Then fine, I'll rebase your changes and adapt my code Mine is ready before yours is merged: In this case, I propose you commit your changes to my PR. Both of us will appear as coauthors of the change Getting a PR merged in the Kubernetes project takes quite some time (trust me, I talk by experience and can confirm ) WDYT?
Let's see how it goes. We can always rebase the code change to adapt new.
btw, did you already begin to work on task 1? If not, I may just go ahead and work on it right away since I have already dig into it.
yes, I already started And If you ask me, better to keep it this way so we both have an opportunity to have a contribution
cool. Sounds good.
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hi if this issue needs to be worked on any further, please assign me for the same.
has a PR up that will solve the second part of this issue.


Question: // Fit is determined based on whether a pod can tolerate all of the node's taints factory.RegisterMandatoryFitPredicate(predicates.PodToleratesNodeTaintsPred, predicates.PodToleratesNodeTaints) <ins> // Fit is determined based on whether a pod can tolerate unschedulable of node factory.RegisterMandatoryFitPredicate(predicates.CheckNodeUnschedulablePred, predicates.CheckNodeUnschedulablePredicate) </ins> // Insert Key "PodToleratesNodeTaints" and "CheckNodeUnschedulable" To All Algorithm Provider // The key will insert to all providers which in algorithmProviderMap[] // if you just want insert to specific provider, call func InsertPredicateKeyToAlgoProvider() factory.InsertPredicateKeyToAlgorithmProviderMap(predicates.PodToleratesNodeTaintsPred) <ins> factory.InsertPredicateKeyToAlgorithmProviderMap(predicates.CheckNodeUnschedulablePred) </ins> glog.Warningf("TaintNodesByCondition is enabled, PodToleratesNodeTaints predicate is mandatory") }

https://k8s- This test is a persistent fail in this one test bucket only, and aside from one flake it appears to consistently pass everywhere else in sig-release-master-upgrade and in sig-release-1.12-all. /kind bug /priority failing-test /sig storage /assign /cc /milestone v1.12
looks like some permissions issue after upgrade /assign
has triaged this to an understanding that the failures happen because the pod is scheduled on the master and getting logs through the method they're using is disabled on master nodes. This makes me think this is similar to and we have non-release-blocking issues in test cases not being quite robust enough now in the face of node taints.
It is interesting that in some places this test is always failing, some it is always succeeding, and even in one it flakes but with a high success rate: https://k8s-
Adding on to comment: The pod that we are creating does not have any special "master schedulable" annotations or tolerations, so from my understanding this should still be not scheduled to the master for any reason. The Pod Spec in question: Potentially relevant scheduler logs.
The fix mentioned in is only handling DaemonSets. I don't think it will fix this issue.
Do we need to get sig-scheduling involved to see why the master is showing up as a choice for this pod?
/sig scheduling Can you triage this for 1.12 release blocking?
It seems that this pvc-tester- pod is only scheduled to master in every run which caused the test to fail consistently.
However in this test suite, it seems like the test passes most of the time (pod is scheduled to a node), then occasionally flakes (pod scheduled to master).: https://k8s- Whereas in this one it doesn't seem to fail at all!: https://k8s- I talked to about why the behavior might be different across different test suites but we were unable to come up with anything concrete.
Is the master node in your test environment tainted properly?
When we were doing unrelated development with DaemonSets, we noticed a similar issue where our DaemonSet pod would get scheduled on gce environment using hack/cluster-, and we noticed masters brought up using that script were not tainted
What release was the tainting master logic ?
cc we had similar issue that gpu driver installer is being scheduled on master at , can also be related?

Not sure if this is a source of truth on the cluster but the test artifacts here on a successful run show that the master node doesn't have a taint: and on a failed run ALSO no taint: But I'm not really sure where this file comes from. And whether this is the cluster before or after the upgrade
ie:
Don't think will help. It looks specific to gpu labeling
Also ^^ is GKE specific, and GKE docu says no taints are inherited to a pool from the cluster, so shouldn't be something from that direction.
Some log snippets around the area of interest:
/priority critical-urgent
I think the fix is obvious: we need to taint the master node on the gce master setup in cluster/. What I still don't understand is how did we get this far without anything else failing, and why only this one test?
Do taints persist across master upgrades, or do they need to be re-applied again?
Based on Slack discussion and the triage so far, this looks like it's a case of unfortunate cluster/* directory bit rot. We can add the taint and have time before the release still to see clear CI signal, but I would not rate this as a release blocker, especially considering node-"" appears to have been introduced in 1.6.
/assign I'm going to contribute in my daytime :)
I think there's what happened: only update status when upgrading from 1.11 to 1.13, as the node was previously registered (1.11); so is not updated, including unschedulable taint. scheduler is 1.13 now, TaintNodesByCondition is enabled by default and NodeConditionCheck is disabled by default, so it ignores controller manager is 1.13 now, it'll taint nodes according to node's configurations, e.g. . Because (including unschedulable) is not updated by kubelet when upgrading, it has to dependent on controller manager (nodelifecycle) to update it which has race condition with kube-scheduler.

There's an another issue that, we'll reject taints update to Node in noderestriction admission controller :( For the solution, I think we need to patch (at least ) and add a whitelist to noderestriction admission controller for known taints key; we can not use to start the kubelet, as it also be skipped (only update node status). Refer to the following logs for what happened: All logs are from https://k8s- Build- The taint of master; the taint will include if tainted by controller manager. scheduler-
/sig node
if no comments on the solution, i will open a pr for it.
cc on relaxing node restriction for this upgrade case Could another potential solution be to keep the previous scheduler node.unschedulable logic for one more release in order to handle this 2 version upgrade scenario?
Actually to clarify, is this both a 1.10 to 1.12 and 1.11 to 1.12 issue?
please get a PR started so we can get review happening ASAP
Thanks, for the investigation. Initially, I wanted to suggest to keep the logic of checking node condition in the scheduler along side TaintNodeByCondition for one more release, but I guess there would be conflicts. For example, a pod that tolerates certain node conditions might not be scheduled if the scheduler checked both node taints and node condition. So, looks like we'll have to fix the tainting logic.
can you clarify on what fixing the tainting logic would entail? If I understand your example, wouldn't the Pod object need to be updated to also tolerate the taint that the condition got converted to?
I was specifically referring to what suggested: "I think we need to patch node (at least node.Spec.Taints) and add a whitelist to noderestriction admission controller for known taints key." Yes, the pod needs to have the right toleration for the taint. We already create our critical system pods with proper tolerations for certain node taints, such as unreachable, etc.
I'm going back to feeling this is actually a release blocking upgrade issue. please clarify in more detail your concern for breakages, the possible paths for resolution, and preferred path.
I agree with you that this is release blocking. We commented at the same time.

I mentioned above that I think we need to address tainting of the nodes in kubelet.
Who's the right owner for this now? I'm assuming we've lost Klaus to the weekend in his timezone, but some of us still have a few work hours yet on Friday...
I already pinged Dawn as our Node guru, but didn't get any reply. She might be on vacation. Let me see if I can find other folks here.
cc for what we can do on the node side. cc for the security/policy related matters associated with adding taints for node condition
help please
regarding your example "a pod that tolerates certain node conditions might not be scheduled if the scheduler checked both node taints and node condition", could this be an API compatibility issue? Will a pod that tolerates a node condition in 1.11 fail to schedule in 1.12 because the scheduler no longer looks at node conditions and only node taints?
Pods were not able to tolerate node conditions by setting some API fields. We had some annotations, such as pod critical annotation and other hard-coded logic in DS controller, etc. that would have the same effect of tolerating conditions, but they were not in our API.
Thanks for the explanation. It looks like the critical pod annotation was alpha, so does not have a backwards compatibility requirement, and we've fixed all the known users of it.
xref: - cc Should this be handled by the node controller? I.e. if the Unschedulable bool is set on the node, add the unschedulable taint?
I don't believe the kubelet has the ability to update its taints anymore. Assuming this is the case, this seems like a version skew problem between master components. I.E. the 1.11 controller manager isn't compatible with the 1.13 scheduler because it doesn't add taints that the scheduler requires. This means that if the scheduler is upgraded before the controller manager (or before it has the chance to perform updates), we can run into issues. I think making the scheduler respect conditions in 1.13, and thus making it compatible with the 1.11 controller manager, seems like the best remedy to this.

Regardless of this issue, user pods would need to include both the critical annotation as well as the toleration in order to maintain the same behavior across releases that transition from conditions to taints.
Oh, I see that the node controller is already doing that, but there's a race condition (), sorry I missed that earlier.
Yes, translation of conditions and unschedulable field to taints is done by the controller
I wonder if we can change our upgrade flow to ensure that the controller manager is upgraded and becomes stable (adds the taints) before the scheduler is upgraded.
The kubelet has never required the ability to update taints. Was that a typo, supposed to be 1.12? Have we ever supported two version skew between those components? Under what circumstances do we upgrade to 1.13 scheduler while a 1.11 controller manager is still running? Same question about two version skew between these two components. That's more concerning… is that something all user pods would be exposed to and required to change?
+1 to suggestion in to unblock the release
What about HA master upgrade? scheduler and controller-manager each have independent leader elections, so during a HA rolling upgrade, the current leaders for scheduler and controller managers could be on different releases in either direction
I wouldn't expect an upgrade from 1.12 to 1.13 to begin until the upgrade from 1.11 to 1.12 was complete/successful
Thanks for correcting me. It wasn't a typo, but just a lack of knowledge about how multi-version upgrades are performed. In that case we need to support 1 version skew. Yeah, it should be 1.12, not 1.13. But generally, the newer scheduler should still be compatible with the state left by the previous version's controller manager, even if it isn't running anymore. That's more concerning… is that something all user pods would be exposed to and required to change? Actually, I agree. have we officially deprecated the critical pod annotation? We should ideally add the correct tolerations to pods with the critical pod annotation to ensure backwards compatibility.
Bobby commented that the system pods have already been corrected. I also noted that the critical pod annotation is alpha.
I mean doing an HA rolling upgrade from 1.11 to 1.12.

Let's say we have 3 masters, A, B, C, all on 1.11. is upgraded to 1.12, B and C are still on 1.11. leader election could pick scheduler A as the leader, controller-manager leader election could pick controller-manager B as the leader. Now we have version skew between master components.
That feature (critical pods) was an alpha feature and we have already replaced it. The dilemma is that I am not so sure if critical pods were the only kind of pods with a "hack" for tolerating node conditions. Let me dig a bit more.
One version of skew, absolutely. comment made me concerned two versions of skew support were being claimed/proposed
I looked the scheduler code and the DaemonSet controller code to find if adding "CheckNodeCondition" could cause issues in presence of TaintNodeByCondition and I didn't find anything particularly concerning. So, I created . Let's see if it breaks tests.
that will break schedule daemonset by scheduler. and registerwithtaints of kubelet is not worked in upgrade.
I suspected that it would conflict with pods which now have tolerations and the scheduler should ignore node conditions for, but I thought it would be limited to critical pods. Thanks for pointing it out. I closed the PR. As an alternative solution, I suggested changing the order of upgrading server components in our upgrade process, so that the controller manager is upgraded before the scheduler, but I am not sure how easy it is to implement. Do you have any alternative fix in mind?
unschedulable is taint by controller & kubelet; taints node with unschedulable to avoid race condition between controller and scheduler, it dependents on of kubelet to taint when start up; but does not work in upgrade case as mentioned above (we only update status when upgrading). IMO, it should be fixed; it'll also fix upgrade e2e test but may not work in real scenario which upgrade master firstly. let me think about it, I'll propose one today.
The race that is important to resolve at creation time is that of a new node object appearing untainted. That race does not apply in an upgrade scenario, where the node object already exists, and can already be observed and acted upon. Relying on the condition-to-taint controller in upgrade scenarios is reasonable.

The race after kubelet start up is well known; but we did not handle startup case well. I think what we're doing is to deprecate , and replace it with unschedulable taint. We need to keep backward compatibility for it.


Question: cmd.Flags().StringVar(&o.APIGroup, "api-group", o.APIGroup, "Limit to resources in the specified API group.") cmd.Flags().BoolVar(&o.Namespaced, "namespaced", o.Namespaced, "If false, non-namespaced resources will be returned, otherwise returning namespaced resources by default.") cmd.Flags().StringSliceVar(&o.Verbs, "verbs", o.Verbs, "Limit to resources that support the specified verbs.") <ins> cmd.Flags().StringVar(&o.SortBy, "sort-by", o.SortBy, "If non-empty, sort nodes list using specified field. The field can be either 'name' or 'kind'.") </ins> cmd.Flags().BoolVar(&o.Cached, "cached", o.Cached, "Use the cached list of resources if available.") return cmd }

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Add a sort parameter (or similar concept) to the command. Why is this needed: With the numerous amount of API resources (not counting custom ones), it is sometimes difficult locating them in the output. By default, API resources are not sorted by the name column (the first on the left) and often this is the column holding the information one wishes to query. A sort parameter would help more easily identify the object to be understood.
/sig usability


Question: echo "minion-$((${BASH_REMATCH[1]} - 1))" } <del> # Find the vagrant machien name based on the host name of the minion </del> <ins> # Find the vagrant machine name based on the host name of the minion </ins> function find-vagrant-name-by-minion-name { local ip="$1" if [[ "$ip" == "${INSTANCE_PREFIX}-master" ]]; then

This is an umbrella issue to track changes in the . It is required to update the guide based on the suggestion we've received so far after officially migrating , , and (ongoing migration). The guide can be improved by adding more relevant details about structured logging migration, examples, common mistakes, and easy navigation in the guide using appropriate referencing. We've observed common mistakes made by contributors while doing the migration. To avoid those "common mistakes", more examples are needed in the structured logging migration guide: approach could be manually looking at various review comments that we've received so far while performing migration of , , and and figure out common mistakes done by contributors. Add examples to avoid those mistakes, and enhance the guidelines present in addition to this, people involved in the previous migration process, either as approver/reviewer or as contributors, can suggest changes in this umbrella issue or open a pull request towards the necessary changes referencing this issue. As we advance, we need to make the migration guide suitable for both "contributors" and "users" of Kubernetes. Since slowly we are progressing and covering more components of Kubernetes, it is required to improve the guide so that both contributors and Kubernetes users can effectively use it.
Suggestions in form of review comments from previous migrations are here:
/sig instrumentation /wg structured-logging /area logging /priority important-longterm /kind documentation /cc
Do you intend to update the documentation yourself, or are you looking for others to help here?
Yes, 1) I'm looking for contributors and reviewers to suggest more changes (I may not be able to cover all the required changes) and I'll add those changes in the table above. 2) Contributors can submit PRs for these changes in the migration guide, I can definitely submit PRs if we see that the issue is not progressing enough
Thanks. I'm going to assign this to you for now then. /assign shivanshu1333
/assign
/triage accepted
/sig docs
/assign
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale


Question: } } <ins> // Process Volume-Attachment objects. // Should be called only after populating attached volumes in the ASW. // For each VA object, this function checks if its present in the ASW. // If not, adds the volume to ASW as an "uncertain" attachment. // In the reconciler, the logic checks if the volume is present in the DSW; //   if yes, the reconciler will attempt attach on the volume; //   if not (could be a dangling attachment), the reconciler will detach this volume. func (adc *attachDetachController) processVolumeAttachments() error { vas, err := adc.volumeAttachmentLister.List(labels.Everything()) if err != nil { klog.Errorf("failed to list VolumeAttachment objects: %v", err) return err } for _, va := range vas { nodeName := types.NodeName(va.Spec.NodeName) pvName := va.Spec.Source.PersistentVolumeName if pvName == nil { // Currently VA objects are created for CSI volumes only. nil pvName is unexpected, generate a warning klog.Warningf("Skipping the va as its pvName is nil, va.Name: %q, nodeName: %q", va.Name, nodeName) continue } pv, err := adc.pvLister.Get(*pvName) if err != nil { klog.Errorf("Unable to lookup pv object for: %q, err: %v", *pvName, err) continue } volumeSpec := volume.NewSpecFromPersistentVolume(pv, false) plugin, err := adc.volumePluginMgr.FindAttachablePluginBySpec(volumeSpec) if err != nil || plugin == nil { // Currently VA objects are created for CSI volumes only. nil plugin is unexpected, generate a warning klog.Warningf( "Skipping processing the volume %q on nodeName: %q, no attacher interface found. err=%v", *pvName, nodeName, err) continue } volumeName, err := volumeutil.GetUniqueVolumeNameFromSpec(plugin, volumeSpec) if err != nil { klog.Errorf( "Failed to find unique name for volume:%q, va.Name:%q, nodeName:%q: %v", *pvName, va.Name, nodeName, err) continue } attachState := adc.actualStateOfWorld.GetAttachState(volumeName, nodeName) if attachState == cache.AttachStateDetached { klog.V(1).Infof("Marking volume attachment as uncertain as volume:%q (%q) is not attached (%v)", volumeName, nodeName, attachState) err = adc.actualStateOfWorld.MarkVolumeAsUncertain(volumeName, volumeSpec, nodeName) if err != nil { klog.Errorf("MarkVolumeAsUncertain fail to add the volume %q (%q) to ASW. err: %s", volumeName, nodeName, err) } } } return nil } </ins> var _ volume.VolumeHost = &attachDetachController{} var _ volume.AttachDetachVolumeHost = &attachDetachController{}

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Background: The attach/detach controller create objects for CSI Volumes to indicate that a volume needs to be attached to a node. It deletes the object to indicate that the volume should be detached. Problem: If the Attach/Detach controller crashes, and in the meantime the pod using that volume is deleted, when the AD controller restarts, it won't know that the volume was previously attached and should be cleaned up. What you expected to happen: Proposal: We should have the AD controller periodically list all objects and for any that are not accounted for, update its internal actual/desired state caches so that the volume is gracefully detached. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: /sig storage Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
could I take it?
/assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available .

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/lifecycle frozen /reopen
Reopened this issue. <detailsIn response to : /lifecycle frozen /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
should address this


Question: return obj3 } <ins> func TestSetDefaultReplicationController(t *testing.T) { tests := []struct { rc             *current.ReplicationController expectLabels   bool expectSelector bool }{ { rc: &current.ReplicationController{ Spec: current.ReplicationControllerSpec{ Template: &current.PodTemplateSpec{ ObjectMeta: current.ObjectMeta{ Labels: map[string]string{ "foo": "bar", }, }, }, }, }, expectLabels:   true, expectSelector: true, }, { rc: &current.ReplicationController{ ObjectMeta: current.ObjectMeta{ Labels: map[string]string{ "bar": "foo", }, }, Spec: current.ReplicationControllerSpec{ Template: &current.PodTemplateSpec{ ObjectMeta: current.ObjectMeta{ Labels: map[string]string{ "foo": "bar", }, }, }, }, }, expectLabels:   false, expectSelector: true, }, { rc: &current.ReplicationController{ ObjectMeta: current.ObjectMeta{ Labels: map[string]string{ "bar": "foo", }, }, Spec: current.ReplicationControllerSpec{ Selector: map[string]string{ "some": "other", }, Template: &current.PodTemplateSpec{ ObjectMeta: current.ObjectMeta{ Labels: map[string]string{ "foo": "bar", }, }, }, }, }, expectLabels:   false, expectSelector: false, }, { rc: &current.ReplicationController{ Spec: current.ReplicationControllerSpec{ Selector: map[string]string{ "some": "other", }, Template: &current.PodTemplateSpec{ ObjectMeta: current.ObjectMeta{ Labels: map[string]string{ "foo": "bar", }, }, }, }, }, expectLabels:   true, expectSelector: false, }, } for _, test := range tests { rc := test.rc obj2 := roundTrip(t, runtime.Object(rc)) rc2, ok := obj2.(*current.ReplicationController) if !ok { t.Errorf("unexpected object: %v", rc2) t.FailNow() } if test.expectSelector != reflect.DeepEqual(rc2.Spec.Selector, rc2.Spec.Template.Labels) { if test.expectSelector { t.Errorf("expected: %v, got: %v", rc2.Spec.Template.Labels, rc2.Spec.Selector) } else { t.Errorf("unexpected equality: %v", rc.Spec.Selector) } } if test.expectLabels != reflect.DeepEqual(rc2.Labels, rc2.Spec.Template.Labels) { if test.expectLabels { t.Errorf("expected: %v, got: %v", rc2.Spec.Template.Labels, rc2.Labels) } else { t.Errorf("unexpected equality: %v", rc.Labels) } } } } </ins> func TestSetDefaultService(t *testing.T) { svc := &current.Service{} obj2 := roundTrip(t, runtime.Object(svc))

It doesn't make sense to define pod template with a set of labels that is strict subset of replicaSelector (or at least I can't find an use-case). For that reason I think that the labels in pod template part of replication controller should be optional and user should care about them only if he wants to define a superset of what replicaSelector already defines.
I've heard the same comments from a client. I think that it'd be more intuitive if we always specify pod labels in the template (so that we can copy&paste pod definition), and replicaSelector is optional (by default pod labels).
It would be reasonable for the selector to be a subset. I'm fine with intelligent defaulting but we need to be careful not to conflate the 2 concepts, which I think would increase the amount of confusion.
Setting RC selector from pod template labels SGTM. Any concerns It wouldn't have to work with separate templates, though it would be desirable if it did. It would be nice if we could add preconditions based on other objects (resource version of pod template in this case) to etcd transactions.
I think we agreed in another issue that defaulting the selector from the pod template labels is reasonable. No argument from me. One problem with pod templates is that we lose the strict relationship between selector and template, so if you change the template the RC needs to behave intelligently (I refuse to create more pods because the template selector doesn't match mine, etc)
We should do this, and we should default replicas to 1. We could also consider defaulting replication controller labels from pod template labels. That way, essentially the only thing that would need to be specified for a singleton RC would be the pod template.
I don't think we can default replicas to 1, since we can't tell the difference between an explicit zero and a default zero, and I think there are lots of reasonable cases where you want to set replicas == 0 on create. (esp. in the context of socket activation)
Responded on the PR: We can distinguish default from 0 if we make it a pointer.


Question: return len(s) } <del> func less[T ordered](lhs, rhs T) bool { </del> <ins> func less[T cmp.Ordered](lhs, rhs T) bool { </ins> return lhs < rhs }

Package use a custom-defined type . Go 1.21 support an official package cmp which has .
/sig api-machinery
cc
/assign xgp01
/triage accepted
/kind cleanup


Question: - [Deprecation of PodSecurityPolicy](#deprecation-of-podsecuritypolicy) - [Kubernetes API Reference Documentation](#kubernetes-api-reference-documentation) - [Kustomize Updates in Kubectl](#kustomize-updates-in-kubectl) <del> - [Default Container Labels](#default-container-labels) </del> <ins> - [Default Container Annotation](#default-container-annotation) </ins> - [Immutable Secrets and ConfigMaps](#immutable-secrets-and-configmaps) - [Structured Logging in Kubelet](#structured-logging-in-kubelet) - [Storage Capacity Tracking](#storage-capacity-tracking)

is wrong as it says default label can be specified, but KEP and other lines of changelog mentions annotation. Changelog is correct and consistent about default container feature for kubectl. I'm still not able to get this feature to work actually for . Kubernetes version (use ): v1.21.0
/triage accepted thanks for reporting early!
/remove-kind bug /kind documentation
/sig release


Question: // This call is idempotent, and must not return an error if the sandbox has // already been removed. RemovePodSandbox(context.Context, *RemovePodSandboxRequest) (*RemovePodSandboxResponse, error) <del> // PodSandboxStatus returns the status of the PodSandbox. </del> <ins> // PodSandboxStatus returns the status of the PodSandbox. If the PodSandbox is not // present, returns an error. </ins> PodSandboxStatus(context.Context, *PodSandboxStatusRequest) (*PodSandboxStatusResponse, error) // ListPodSandbox returns a list of PodSandboxes. ListPodSandbox(context.Context, *ListPodSandboxRequest) (*ListPodSandboxResponse, error)

The behavior when requested id doesn't exist is not defined clearly in CRI. Currently, we define that should return when requested image doesn't exist (), and kubelet is relying on this behavior now (). However, we haven't clearly defined the behavior of and . Currently, they return error when requested sandbox/container doesn't exist, and kubelet is also relying on this behavior. Not super urgent, just track the issue here. /cc
/assign
Added for clarifying the behavior.