Question: framework.ExpectNoError(sr.CleanupResource(f.Timeouts)) }) vs := sr.Vs <del> vsc := sr.Vsclass // Get new copy of the snapshot ginkgo.By("checking the snapshot") vs, err = dc.Resource(storageutils.SnapshotGVR).Namespace(vs.GetNamespace()).Get(context.TODO(), vs.GetName(), metav1.GetOptions{}) framework.ExpectNoError(err) // Get the bound snapshotContent snapshotStatus := vs.Object["status"].(map[string]interface{}) snapshotContentName := snapshotStatus["boundVolumeSnapshotContentName"].(string) vscontent, err := dc.Resource(storageutils.SnapshotContentGVR).Get(context.TODO(), snapshotContentName, metav1.GetOptions{}) framework.ExpectNoError(err) snapshotContentSpec := vscontent.Object["spec"].(map[string]interface{}) volumeSnapshotRef := snapshotContentSpec["volumeSnapshotRef"].(map[string]interface{}) // Check SnapshotContent properties ginkgo.By("checking the SnapshotContent") // PreprovisionedCreatedSnapshot do not need to set volume snapshot class name if pattern.SnapshotType != storageframework.PreprovisionedCreatedSnapshot { framework.ExpectEqual(snapshotContentSpec["volumeSnapshotClassName"], vsc.GetName()) } framework.ExpectEqual(volumeSnapshotRef["name"], vs.GetName()) framework.ExpectEqual(volumeSnapshotRef["namespace"], vs.GetNamespace()) </del> <ins> // get the snapshot and check SnapshotContent properties vscontent := checkSnapshot(dc, sr, pattern) </ins> ginkgo.By("Modifying source data test") var restoredPVC *v1.PersistentVolumeClaim

The changes to support snapshot tests for Generic Ephemeral Volumes generated some duplicated code in the file as commented here: , it'd be great to create functions that are reused in both and tests. Things that are needed: A kubernetes dev cluster (I follow to create a GCE cluster) Knowledge about how to /sig storage /area testing /good-first-issue Improve readability of the storage VolumeSnapshot tests
This request has been marked as suitable for new contributors. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : ### What would you like to be ? /area testing /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : ### What would you like to be ? /area testing /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
Hi Could you please give me some more details about what needs to be done in this issue?
I modified the description, it's all about making changes in a single file creating functions that reduce the duplicated code following the thread also linked in the description.
Thanks!

Hi, I'm currently in the process of working on this issue. I'm adding the functions to the file.
is this issue solved because now I am unable to see the duplication which I saw day before yesterday.
Hi When I try to run the snapshottable tests using and I get 8 of the following failures: I have setup and as their documentation suggested, but seem to not be able to run the tests. Do you have an idea why this happens? Thanks!
what was the command that you used? The csi-hostpath driver is embedded into the kubernetes codebase so you don't need to add anything else to your kind? cluster.
The command I used to run the snapshottable tests in a local kind cluster is: . Does that seem OK?
do you have an idea about why this happens? Thanks!
hey sorry for the late response and thanks for the reminder, I always run my tests with and with a GCE cluster (not a kind cluster) for example following . The VolumeSnapshot feature requires a CSI Driver, in our e2e tests we use a CSI Driver that's embedded in the kubernetes codebase called csi-hostpath, from the logs I saw that it used csi-hostpath which I think it's ok, ideally we should test it with a real CSI Driver like PD CSI Driver but that's going to provision storage in GCP. Have you tried with ? I think that there's more info in that page about kubetest than kubetest2
Yes, I tried , but got the same errors as well. Do you think it's maybe a problem with ?
/assign
/assign
hey and you guys have solved this issues or still working on it, because i'm considering to solve this issue if you haven't solved it
/assign
thank you for signing up for this task, I believe should work, we're waiting for additional review


Question: } } <ins> var debug = false </ins> func dumpEtcdKVOnFailure(t *testing.T, kvClient clientv3.KV) { <del> if t.Failed() { </del> <ins> if t.Failed() && debug { </ins> response, err := kvClient.Get(context.Background(), "/", clientv3.WithPrefix()) if err != nil { t.Fatal(err)

Improve the to ensure that the expected storage version for a resource is one understood by previous releases. We have API testdata from prior versions available in When checking each group/version/kind in the test: accumulate all the versions of the group/kind available in previous (non-HEAD) releases if there is a beta or stable version of the group/kind available in previous releases, then the storage version must be in the set of previously available versions It is too easy to accidentally change the storage version prematurely and edit the test/integration/etcd test to match the bug cc
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: func getFlexDir(c clientset.Interface, node *v1.Node, vendor, driver string) string { volumePluginDir := defaultVolumePluginDir if framework.ProviderIs("gce") { <del> if node == nil && framework.MasterOSDistroIs("gci") { </del> <ins> if node == nil && framework.MasterOSDistroIs("gci", "ubuntu") { </ins> v, err := getMasterVersion(c) if err != nil { framework.Failf("Error getting master version: %v", err)

https://k8s- https://k8s- It passes on COS. /cc
I can look into this, since likely this was introduced because of latest changes I made.
This looks like a timing thing - because I can see message in logs , but I think 5 minutes is not enough for the pod to come up after installing flexvolume.
That's strange, the dummy driver just sets up a tmpfs so it should be fairly quick. I wonder if there's something else blocking pod creation?
the driver is not being installed on kubelet I am doing some more investigating, have my own cluster up.
On ubuntu, we set the flexvolume install dir to --volume-plugin-dir=/home/kubernetes/fl exvolume Small oversight, I'll make a PR shortly
They are still failing, I am having another look e: Same as before, it is not installing on the node. I started my test cluster using (basically...) the same kubetest command sooo allow me to try again. e2: it's a false alarm. The tests are not using a commit with my fix in it yet zzzz


Question: "//pkg/cloudprovider:go_default_library", "//pkg/types:go_default_library", "//pkg/util/errors:go_default_library", <ins> "//pkg/volume:go_default_library", </ins> "//vendor:github.com/Azure/azure-sdk-for-go/arm/compute", "//vendor:github.com/Azure/azure-sdk-for-go/arm/network", "//vendor:github.com/Azure/azure-sdk-for-go/arm/storage",

Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"6+", GitVersion:"v1.6.0-alpha.0.1011+", GitCommit:"", GitTreeState:"clean", BuildDate:"2016-11-30T07:19:12Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"6+", GitVersion:"v1.6.0-alpha.0.1011+", GitCommit:"", GitTreeState:"clean", BuildDate:"2016-11-30T07:19:12Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"} Environment: Cloud provider or hardware configuration: Azure OS (e.g. from /etc/os-release): RHEL 7.3 Install tools: Ansible What happened: PVC keeps pending with a new created storage account What you expected to happen: Storage class can find storage account and then create a dynamic pv for pvc to bind How to reproduce it (as minimally and precisely as possible): a new storage account without create any container a storage class as below a pvc annotating this storage class pvc status is pending Anything else do we need to know: After I create a new container "vhds" in storage account of "qewestus", storage class works and pvc is bound
VHD container is a rerequiste. Will update the doc.
.. I was about to open this exact bug. Shouldn't the provisoner create the vhds container ?
yes, there is a todo for this case (also missing storage account)
Another minor issue is, new volumes take a lot of time (tens of mins) to be ext4 formatted. This causes the pod to timeout and fail upon first use! Any way to make this better
fix is in
Can anyone share how to create the vhds container? For those that don't yet have this fix
Easiest is through the portal, browse to your storage account and create a container called
Thats great thanks for the quick response. The automator in me now wants to see if I can automate the creation of that container through powershell ;)
Does the container have to be of type 'Container' ? IE Full public read access??

Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"6+", GitVersion:"v1.6.0-alpha.0.345+-dirty", GitCommit:"", GitTreeState:"dirty", BuildDate:"2016-11-10T06:07:31Z", GoVersion:"go1.7.3", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"6+", GitVersion:"v1.6.0-alpha.0.345+-dirty", GitCommit:"", GitTreeState:"dirty", BuildDate:"2016-11-10T06:07:31Z", GoVersion:"go1.7.3", Compiler:"gc", Platform:"linux/amd64"} Environment: Linux Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): RHEL 7.3 Install tools: hack/local-up- What happened: storage class pvc annotated with this storage class a pod consume this pvc read/write function with the mounted path pod/pvc the pv status which is "Failed" What you expected to happen: PV should not have "Failed" status before deleted How to reproduce it (as minimally and precisely as possible): Always Anything else do we need to know:
cc
Per message The disk is still being attached. As a result, PV cannot be deleted.
This condition is handled by the PV framework. See However the volume plugin must return which it looks like the Azure volume plugin is not doing.
please help remove label, kubectl can't do anything with this.


Question: } if err != nil { klog.V(1).Infof("Failed to bind pod: %v/%v", assumed.Namespace, assumed.Name) <del> if err := sched.SchedulerCache.ForgetPod(assumed); err != nil { klog.Errorf("scheduler cache ForgetPod failed: %v", err) } </del> return }

This is a bug found in The scheduler fails to schedule the pod on the first try due to PV is out of sync. This may happen because the PV/PVC may be modified before the scheduler binder assumes them. However, the scheduler skips the pod update event and never try to scheduler the pod again: This is because when the reserve plugins fails, we didn't try to forget the pod. After the is called, the pod is in the state.
/sig scheduling /sig storage /assign /priority critical-urgent
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig scheduling /sig storage /assign /priority critical-urgent Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: GetContext:         ctxFn, SelfLinker:         a.group.Linker, ClusterScoped:      true, <del> SelfLinkPathPrefix: gpath.Join(a.prefix, resourcePath, "/"), </del> <ins> SelfLinkPathPrefix: gpath.Join(a.prefix, resourcePath) + "/", </ins> SelfLinkPathSuffix: suffix, }

Bug report Kubernetes version Client Version: version.Info{Major:"1", Minor:"5+", GitVersion:"v1.5.0-beta.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2016-11-24T22:35:03Z", GoVersion:"go1.7.3", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"5+", GitVersion:"v1.5.0-beta.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2016-11-24T22:30:23Z", GoVersion:"go1.7.3", Compiler:"gc", Platform:"linux/amd64"} When you curl the api you get a list of third party resources, however their selflinks are missing a forward slash:
fixed with , you may take a look.
This looks like it affects root scoped resources in general -- both custom ones (e.g. ServiceCatalog brokers, as noted in kubernetes-incubator/service-catalog) and "built-in" ones, like namespaces and nodes:
cc


Question: ginkgo.By("wait for the deleted pod to be cleaned up from the state file") waitForStateFileCleanedUp() ginkgo.By("the deleted pod has already been deleted from the state file") <ins> }) ginkgo.AfterEach(func() { </ins> setOldKubeletConfig(f, oldCfg) }) }

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: ci-kubernetes-node-kubelet-serial node-kubelet-serial job Which test(s) are failing: Hard to tell, but everything at one point or another. Since when has it been failing: A month or two, by recollection. would indicate several years... ci-kubernetes-node-kubelet-serial | failing_days | 992 Testgrid link: Reason for failure: Hard to tell, some is timeout, but I think timeouts due to test failures. Anything else we need to know: Nothing consistent about the failures that I can tell besides timeouts. This is the only place these serial tests are running.
/sig node
/cc
I will take a look at what happens with the serial lane.
So the kubelet failed to restart because of I think it can be related to the topology manager scope PR, I will dig into the code tomorrow. FYI
It actually looks like it was introduced by: With the same issue reported here: And Alexey has a fix for it here (I haven't reviewed it):
Thanks for the update.
Some of these failures are definitely due to , which is causing tests to fail since pods aren't terminating in time. See e.g.
/assign
Under the latest run, I saw that the kubelet failed to restart because of The problem that one of the topology manager tests failed in the middle and did not remove the state file, I can not guarantee that it a single problem, but let's start from it.


Question: manager.RegisterPod(podWithConfigMaps("ns1", "other-name", s2)) manager.UnregisterPod(podWithConfigMaps("ns1", "other-name", s2)) <ins> s5 := configMapsToAttach{ containerEnvConfigMaps: []envConfigMaps{ {envVarNames: []string{"s7"}}, {envFromNames: []string{"s70"}}, }, } // Check the no-op update scenario manager.RegisterPod(podWithConfigMaps("ns1", "noop-pod", s5)) manager.RegisterPod(podWithConfigMaps("ns1", "noop-pod", s5)) </ins> refs := func(ns, name string) int { store.lock.Lock() defer store.lock.Unlock()

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Occasionally (more frequently if the node is heavily loaded) the node fails to mount a secret or configmap for a pod. Error recovery fails and the pod must be replaced with a new pod to have any hope of getting the secret or configmap volume mounted. The problem appears to be that if a pod is updated, its secrets are removed from the secretStore's secrets map. See: Normally the effects of the bug are not felt since the usual order of operations masks the bug: 1) The pod is created and registered the first time with the secrets manager 2) The volume mounter gets the secret 3) The pod is updated and registered again with the secrets manager, which deletes the secrets (the bug) If the race is lost, though, then the bug symptoms are manifest: 1) The pod is created and registered the first time with the secrets manager 2) The pod is updated and registered again with the secrets manager, which deletes the secrets (the bug) 3) The volume mounter tries to get the secret and fails with the message What you expected to happen: On update, the secrets manager should only delete secrets if they are not present in the latest pod definition (rather than deleting all prior secrets). Whether the pod update happens before or after the volume mounter attempts to get the secret, it should succeed. How to reproduce it (as minimally and precisely as possible): Because it is a race condition, it's difficult to reproduce. We've most often seen the bug on systems that were under load. 1) Create a pod that has a secrets or configmap volume 2) See if the pod fails to mount the volume. Anything else we need to know?: If the above analysis is correct, this bug has been present since 1.6 for secret volumes, and 1.7 for configmap volumes. Environment: We've seen the bug in software based upon the 1.6 release.
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list .

The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
The PR addresses this issue. /sig node
This may be a false alarm. I've never actually reproduced it in Kubernetes proper, only in OpenShift. Since our smoking gun was anything but, I'm going to close this until I manage to find an actual problem in Kubernetes.
To close the loop on this for any that find it via future searches: It turns out that this was a false alarm. The above analysis was flawed and this turns out not to be a bug (or maybe a very minor one if we consider it a bug to log a an error on a (semantic) non-error). The error messages like were not an indicator of a failure to mount a secret volume, but instead a failure to update a secret volume during pod teardown. Here's how the error message is triggered: pod is created volume manager gets the secret and mounts the volume the lifetime of the pod, the volume manager periodically re-checks the secret to make sure that it's still in a good state. pod goes away volume is unmounted If one of the refreshes from step 3 happens after step 4 but before step 5, the log message will appear.


Question: ginkgo.By("waiting for RC to be modified") eventFound = false <del> ctx, cancel = context.WithTimeout(context.Background(), 30*time.Second) </del> <ins> ctx, cancel = context.WithTimeout(context.Background(), 60*time.Second) </ins> defer cancel() _, err = watchUntilWithoutRetry(ctx, retryWatcher, func(watchEvent watch.Event) (bool, error) { if watchEvent.Type != watch.Modified {

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: - Triage link for more jobs that are affected: Which test(s) are failing: Since when has it been failing: Since October 23 Testgrid link: Reason for failure: Anything else we need to know: Example spyglass link: /sig apps /cc /priority important-soon
/cc missed this in the main body
/assign /triage accepted
/remove-priority important-soon /priority critical-urgent top flake for pull-kubernetes-conformance-kind-ipv6-parallel, 2nd top flake for per /area conformance this is a conformance test /remove-kind failing-test /kind flake this is not failing continuously, but is instead flaking
/assign
I increased the timeout while waiting for available Replicas.
Test grid green since Merge 5 Nov 2020 /cc


Question: import ( "encoding/json" "errors" <ins> "fmt" </ins> "net/http" "strconv" "strings"

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When using kubectl, authentication fails with the below error For some reason kubectl is adding a lot of s to the start of the SPN - this can be observed in What you expected to happen: Authentication works as in previous versions of Kubectl How to reproduce it (as minimally and precisely as possible): Create a cluster using Azure AD authentication. Authenticate from a client and wait until the token expires. Use Kubectl and have it automatically refresh the token; it will fail due to the extra 'spn:' entries. Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: on-premise OS (e.g: ): macOS 10.14.6 Kernel (e.g. ): Darwin Hostname 18.7.0 Darwin Kernel Version 18.7.0: Mon Feb 10 21:08:45 PST 2020; root:xnu-4903.278.28~1/RELEASEX8664 x86_64 Install tools: N/A Network plugin and version (if this is a network-related bug): N/A Others:
/sig cli
Kubectl version 1.18.0 Kubernetes version 1.16.6
can you confirm whether your original kubeconfig already has in apiserver-id or not? in the mean time, you may try it can convert your kubeconfig from azure module to exec module and do the token thing outside kubectl
At this point if I remove the tokens and try to reauth i get the error described above. I've tried it both ways with apiserver-id having an initially and not, both lead to the same behavior in the end.
i know what's going on now. will prepare a fix. the root cause is that token's target resource (adal.Token.Resource) needs to have prefix and this prefix gets unexpectedly to as well, which gets written back to kubeconfig. So every time kubectl is run, it adds to apiserver-id
/priority important-soon
I'm seeing this with kubectl 1.17.3 and kubernetes 1.16.7.

It's unclear to me what versions are impacted and in what version it is fixed, since only version 1.18 is mentioned.
This still happens on kubectl 1.16.12 and 1.17.8
This still happens on kubectl v1.18.5
same
can you guys share your kubectl version --client and kubeconfig with me? you can find me on kubernetes slack. the slack user name is same as my github handle.
Version 1.17.6 works fine.
Since I was mistaken thinking was still causing this for me, I figured I would just drop a note for others in case it's helpful. Turns out my problem was actually due to and looks to have been resolved as of the v3.3.0 release.
This issue has now reappeared for me - although not necessarily the same root cause. AKS 1.19.0 Authentication works, but it spams frequently.


Question: TargetCPUUtilizationPercentage: &cpu, }, } <del> _, errHPA := rc.framework.ClientSet.Autoscaling().HorizontalPodAutoscalers(rc.framework.Namespace.Name).Create(hpa) </del> <ins> hpa, errHPA := rc.framework.ClientSet.Autoscaling().HorizontalPodAutoscalers(rc.framework.Namespace.Name).Create(hpa) </ins> framework.ExpectNoError(errHPA) <ins> return hpa } func DeleteHorizontalPodAutoscaler(rc *ResourceConsumer, autoscalerName string) { rc.framework.ClientSet.Autoscaling().HorizontalPodAutoscalers(rc.framework.Namespace.Name).Delete(autoscalerName, nil) </ins> }

Currently the time it takes for HPA and CA to scale up after load increase is not measured at all, and this can lead to regressions that will be missed. This issue is about adding an e2e test that will prevent such regressions.
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list .
Filip, can you reopen? I cannot do this myself, and this issue is clearly not closed yet.
Thanks Wojtek!


Question: import ( "errors" "fmt" <ins> "net" </ins> "net/http" "regexp" "strings"

/kind bug What happened: Continuation of . The fix to that bug only takes the first subnetwork in the correct region. However, if the user manual subnets in the region that the cluster exists, it's possible the cloudprovider could choose the wrong subnet. What you expected to happen: If no subnet is specified for a non-legacy network, always take the subnet which was automatically created. /assign /sig network /area platform/gce


Question: // Waits for the deployment to reach desired state. // Returns an error if minAvailable or maxCreated is broken at any times. func waitForDeploymentStatus(c *client.Client, ns, deploymentName string, desiredUpdatedReplicas, minAvailable, maxCreated int) error { <del> return wait.Poll(poll, 2*time.Minute, func() (bool, error) { </del> <ins> return wait.Poll(poll, 5*time.Minute, func() (bool, error) { </ins> deployment, err := c.Deployments(ns).Get(deploymentName) if err != nil {

cc
Thanks for flagging it! Its not flaky, it is broken as we did not enable deployments for that test suite. Sent to fix that.


Question: }, "targetPort": { "type": "string", <del> "description": "Number or name of the port to access on the pods targeted by the service. Number must be in the range 1 to 65535. Name must be an IANA_SVC_NAME. If this is a string, it will be looked up as a named port in the target Pod's container ports. If this is not specified, the value of Port is used (an identity map). Defaults to the service port. More info: http://releases.k8s.io/HEAD/docs/user-guide/services.md#defining-a-service" </del> <ins> "description": "Number or name of the port to access on the pods targeted by the service. Number must be in the range 1 to 65535. Name must be an IANA_SVC_NAME. If this is a string, it will be looked up as a named port in the target Pod's container ports. If this is not specified, the value of the 'port' field is used (an identity map). This field is ignored for services with clusterIP=None, and should be omitted or set equal to the 'port' field. More info: http://releases.k8s.io/HEAD/docs/user-guide/services.md#defining-a-service" </ins> }, "nodePort": { "type": "integer",

In 1.0 and 1.1, headless services that specified a mismatched port/targetPort were tolerated (the targetPort was ignored). tightened validation to reject these services. That causes problems when upgrading from currently-working clusters.
2 candidate to avoid breaking backwards compatibility
oh hell. Yeah. I don't think this is worth a breaking change. We should either roll that back ASAP or change it to nuke any user-provided value and replace it with the value. I think the latter is potentially dangerous and also not worth the risk, so I think we should rollback and just update the docs for targetPort. objections?
Who has time to do this? It doesn't look like an auto-rollback is doable.
Already opened a PR in
Yes, I agree this isn't worth a breaking change, But could we give an warning log that targetPort isn't equal to port and would be ignored?
There's no facility in validation for warning. I don't think spamming the server log in response to API requests is that helpful here
Yes, that sounds reasonable. I just think that targetPort was ignored is a little unfriendly to user.


Question: } func TestAttacherWaitForVolumeAttachment(t *testing.T) { <del> plug, fakeWatcher, tmpDir := newTestWatchPlugin(t) defer os.RemoveAll(tmpDir) attacher, err := plug.NewAttacher() if err != nil { t.Fatalf("failed to create new attacher: %v", err) } csiAttacher := attacher.(*csiAttacher) </del> nodeName := "test-node" testCases := []struct { name                 string initAttached         bool

/kind bug /sig storage What happened: CSI unit tests occasionally end up with data race: How to reproduce it (as minimally and precisely as possible): to get CSI unit tests into a binary: the binary 500 times in parallel until it breaks: . Use different argument to really load your machine, it should not be idle! . On my Linux, I got 3 failures in ~1000 runs. puts stderr of failed runs into .
/assign
I make a Fix: PTAL


Question: # Windows Stage <ins> FROM --platform=linux/amd64 $REGISTRY/windows-servercore-cache:1.0-linux-amd64-$OS_VERSION as servercore-helper </ins> FROM e2eteam/powershell-helper:6.2.7 as ps-helper FROM $BASEIMAGE

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI -- [sig-windows] [Feature:Windows] GMSA Kubelet [Slow] kubelet GMSA support when creating a pod with correct GMSA credential specs passes the credential specs down to the Pod's containers [sig-windows] [Feature:Windows] GMSA Full [Serial] [Slow] GMSA support works end to end ** pause image using in image was transitioned to the official pause image which no longer has in the image related to /sig windows /triage accepted


Question: return false, nil } <del> // UnbindAddress unbind address from the interface </del> <ins> // UnbindAddress makes sure IP address is unbound from the network interface. </ins> func (h *netlinkHandle) UnbindAddress(address, devName string) error { dev, err := h.LinkByName(devName) if err != nil {

https://k8s- https://k8s- cc: /priority failing-test
Sorry, but I will fix it ASAP.
/assign
PR is already sent. Wating for CI report...


Question: // Configure reflector's pager to for an appropriate pagination chunk size for fetching data from // storage. The pager falls back to full list if paginated list calls fail due to an "Expired" error. reflector.WatchListPageSize = storageWatchListPageSize <ins> // When etcd loses leader for 3 cycles, it returns error "no leader". // We don't want to terminate all watchers as recreating all watchers puts high load on api-server. // In most of the cases, leader is reelected within few cycles. reflector.MaxInternalErrorRetryDuration = time.Second * 30 </ins> cacher.watchCache = watchCache cacher.reflector = reflector

When etcd is unable to elect leader for 3 election cycles, etcd returns "no leader" error through watches. This can be seen in api-server logs as . Also, when apiserver receives this error, it terminates all watchers: . In large clusters, this can destabilize master, as all these watchers need to be recreated almost at the same time, overwhelming api-server quite often. I propose adding new api-server flag that allows to specify timeout, during which we are retrying to reestablish apiserver <-etcd watches, without terminating all api-server watchers. If we are unable to reestablish watches within this specified timeout, we start terminating all watchers. Losing etcd leader for ~2-3s does not destabilize api-server. Trigger prolonged reelection of etcd leader. No response Observed in all kubernetes versions 1.19-1.24 N/A <details</details<details</details<details</details<details</details
/sig scalability /sig api-machinery /priority important-soon
/wg reliability
/assign
/triage accepted
indeed, this is a source of instability. Is this coming from ?
I believe it's coming from this change. As I mentioned, the idea to mitigate it is to add some parameter that will allow us to ignore this error coming from etcd for specific duration without terminating all watchers, for example 10s be default.
With this comment thread in mind: It would probably be cleaner to allow for simply setting the "timeout" as part of WithRequireLeader option (instead of using the constant of 3). But that would require etcd changes, including the API and we want to workaround that in much smaller timeframe. But a comment about how to make that cleaner with some additional context from here would help.


Question: return resList } <del> func calculateScaleUpLimit(currentReplicas int32) int32 { return int32(math.Max(scaleUpLimitFactor*float64(currentReplicas), scaleUpLimitMinimum)) } </del>

We recently had 2 serious bugs caused by non-trivial logic calculating desiredReplicas in hpa reconcileAutoscaler method (, ). I think we should extract this part of logic (starting around ) to a separate function and make sure it's covered with unittests for different possible edge condition. The logic itself takes a few ints as input (calculated desiredReplicas, min/max replicas, currentReplicas, ...) to produce a single integer value and set some status conditions, making it very easily unittestable if we extract it to a separate function. WDYT
+100. It would be super unfortunate if we had yet another issue in this area of the code.
agreed. you've been going over these pieces of code recently, you want to tackle this?
Yes, I'd love to! Thanks for the ping. I'll tackle this this weekend :)


Question:
Warning! This document might be outdated.
  # Horizontal Pod Autoscaling ## Preface

could you please update Scale Subresource section? cc
To be honest I think that we shouldn't update design docs. Overtime we'd have to update too many of them. As a workaround I'd suggest to: a warning at the top that this design doc might be updated "last update" timestamp to the design doc.
I'd rather not have obsolete docs in the tree. We need to ensure that everything in this doc that's still relevant is covered and up to date somewhere. We definitely need a doc for the scale API. (And for the rest of the API ) The kubectl details needs to be covered in kubectl user docs. Other user-facing details need to be in an autoscaling guide and example in the user guide. We need documentation for the HPA controller-manager flags and other details, in the code. Ensure there are github issues for the incomplete roadmap items. cc
SGTM. I guess it's a policy decision so if we decide to update all design docs I'm fine with this :)
Yes, please file an issue that we should update proposals and design docs. Some of them should really be converted to user docs, too.
I don't think it's a good use of time to update design docs. Features morph infinitely over time; it's not practical to keep their original design doc up to date. I don't think people expect that a design doc will even reflect what was originally implemented, much less how the feature exists at the time they are reading the design doc. Now, I do agree "We need to ensure that everything in this doc that's still relevant is covered and up to date somewhere." When something is implemented, we should write proper user guide documentation for it, which we keep up-to-date indefinitely, I don't think anything contradicts this, except "I'd rather not have obsolete docs in the tree." I think there's no way around that with design docs (and proposals), and people should just understand that's how such docs work. We can put a in proposals/ and design/ to explain this.
AFAIU you're both saying different things here :) We have already updated user guides. Should we update the ?
Please hold off on doing it until I discuss with Brian. I definitely don't think this is a good policy.
Any update here?
I talked to about this just now.

He was OK with just putting a big warning at the top of any design docs that contain out-of-date information, warning that the doc is just for historical purposes and pointing to the up-to-date docs. This assumes we actually document our APIs properly and write user-guide docs for the feature that we keep up-to-date. So we need to get better at that -- I agree with him that we should not have any features that are only documented in a design doc that is not up-to-date.
And I reserve the right to gut/delete out-of-date docs at some point.
SGTM. Can you please look into this?


Question: // If the current endpoints is updated we track the old resource version, so // if we obtain this resource version again from the lister we know is outdated // and we need to retry later to wait for the informer cache to be up-to-date. <del> if !createEndpoints { </del> <ins> // there are some operations (webhooks, truncated endpoints, ...) that can potentially cause endpoints updates became noop // and return the same resourceVersion. // Ref: https://issues.k8s.io/127370 , https://issues.k8s.io/126578 if updatedEndpoints != nil && updatedEndpoints.ResourceVersion != currentEndpoints.ResourceVersion { </ins> e.staleEndpointsTracker.Stale(currentEndpoints) } return nil

This issue was supposed to have fixed the issue where endpoint stay out of sync I also wrote a small script which would get me the out of sync endpoints compared to the endpointslices I expect the endpoints to eventually sync and reflect the most upto date information. I have just deployed the newer patch to our cluster and that has resulted in endpoints never ending up being updated if the status goes out of sync. No response Client Version: v1.29.7 Kustomize Version: v5.0.4-0.- Server Version: v1.29.7 <details</detailsalmalinux-9 <details</detailscri-o No response
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig network
from the logs, endpoint controller was retrying test1/test-qa in the expected way. It has retried 5 times and the next retry should be after 320ms, which should long enough for the informer cache to get the updated endpoints. If there is no more logs related to the failure, I think it may have synced it successfully. The python script used to detect the failure seems capturing diff between endpoints and endpointslice at a moment, but that could happen if there is any Pod change in the cluster, and technically the script is comparing endpoints of one moment with endpointslice of another moment. So did you see the endpoints not being synced after endpoint controller stopped retrying it?
Yeah i still see it and see similar messages continuously. Let me post some additional log messages after the one i posted above.
Observed the same issue in 1.29 and 1.30. I keep seeing the same error then new pods were behind a service. And the endpoint never catch up after all pods are settled.
could you provide complete log of kube-controller-manager and the yamls of the related endpoints and endpointslice?
/assign Are you able to triage this since you are already looking at this?

Yes i can get that over by EOD today
Sure, I'm waiting for more logs and the content of the endpoints to understand how it happened. Currently there are only retry logs which is expected in some cases.
We found a customer that triggered an edge case in your change of , which will cause Endpoints never updated. They have a mutatingWebhook on Endpoints object that automatically injects a label, let's say . syncServices, the endpoints controller , and they won't match since Endpoints have the extra "dummy-k: value". endpoints controller will by removing the label. mutatingWebhook will automatically adds back the label as part of the update request. From the APIServer's POV, the Endpoints object don't have any changes, thus the update request will succeed but returns the same resourceVersion as before. change in will mark that resourceVersion as stale. future pods events won't be handled correctly due to the step didn't result in a new resourceVersion. I think we should fix this by check the resulting resourceVersion from the update request, and only mark the old resourceVersion as stale when a new resourceVersion is generated. Not sure whether this is the same case others in this thread encountered :D
thanks for explaining the use case and the analysis. I did analyze the code after the issue was opened and suspected the only possibility is the returned resource version somehow equals to the one in the request but didn't know how it could happen after it has checked the desired endpoints and the current endpoints are different: Now I understand this could happen when the endpoints appear different on client side but are mutated to be same on server side. Do you plan to create a PR since you have proposed a fix? could you confirm if this is also your case, i.e. there is a mutatingWebhook for endpoints objects that could make the endpoints different from what endpoints controller expects?
Yeah, i can do a fix for this
- can we assign to you?
Sure, I'm already assigned. Given that we've had three user reports on this issue, I think it's important to address it as soon as possible. Could you please let me know when you plan to create the PR? If you're currently occupied, I'm happy to handle it and submit the PR.
Sorry, were occupied by some internal incidents last week. Just created a WIP PR for it.

Will test it in kops for this specific webhook scenario in the following two days and and remove the WIP flag
/assign


Question: func (rc *ResourceConsumer) makeConsumeCPURequests() { defer GinkgoRecover() <ins> rc.stopWaitGroup.Add(1) defer rc.stopWaitGroup.Done() </ins> sleepTime := time.Duration(0) millicores := 0 for {

https://k8s-
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
might be a culprit, but maybe not
This could be related to how the test initializes the framework or cleans it up, maybe.
Another failure in the same run, not just this one test:
I'm not positive but the stack trace might be wrong due to inlining optimizations. rc.framework == nil was my first guess, but that would fail earlier in that method.
Framework AfterEach runs namespace deletion, which may mean that .Namespace is nil (if inlining is indeed present).
I am looking into this right now.
It's unrelated. I reverted from the current master HEAD locally (via ), then did and finally and it did not help. The test still fails with the same message "Expected <*v1.Initializers | 0x0: nil not to be nil" (location: )
Note that I have executed one of the initializers e2e test (not autoscaling), because it is one of the tests reported by https://k8s- as panicked.
Looking a bit further I found out that commit the test above as already failing.
I have just realized that I have discovered yesterday a race condition in ResourceConsumer implementation, and might actually make it worse. Let me quickly make a PR out of a fix.
Let me try to recap what I have found out so far: The test that is being referred to at the top of this issue () fails "out of the box" and probably deserves a separate issue to track it Other failing tests that look like they have a similar failure pattern (NPE) fail because of trying to send a "consume CPU" request to a non-existent pod (unrelated to above, as far as I can tell) PR might cause the effect above (and probably does) because it actually deletes the pods that should have been deleted However, the root cause is that there is a race condition inside ResourceConsumer that makes "consume CPU" requests being sometimes sent already after "CleanUp" got called (fix prepared in PR )
Initializer was fixed in a different PR, depending on what cluster you test on.

, Andrzej Wasylkowski < :
Yeah, confirmed initializers test is now passing on GCE and disabled on GKE (which doesn't enable initializers configuration group yet) , Clayton Coleman <: The test that is being referred to at the top of this issue () fails "out of the box" and probably deserves a separate issue to track it Other failing tests that look like they have a similar failure pattern (NPE) fail because of trying to send a "consume CPU" request to a non-existent pod (unrelated to above, as far as I can tell) PR cause the effect above (and probably does) because it actually deletes the pods that should have been deleted However, the root cause is that there is a race condition inside ResourceConsumer that makes "consume CPU" requests being sometimes sent already after "CleanUp" got called (fix prepared in PR ) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub , or mute the thread
Clayton, can you assign this to me? I cannot do this myself.


Question: KUBELET_CERT_BASE64=$(get-env-val "${master_env}" "KUBELET_CERT") KUBELET_KEY_BASE64=$(get-env-val "${master_env}" "KUBELET_KEY") } <ins> # Update or verify required gcloud components are installed # at minimum required version. # Assumed vars #   KUBE_PROMPT_FOR_UPDATE function update-or-verify-gcloud() { local sudo_prefix="" if [ ! -w $(dirname `which gcloud`) ]; then sudo_prefix="sudo" fi # update and install components as needed if [[ "${KUBE_PROMPT_FOR_UPDATE}" == "y" ]]; then ${sudo_prefix} gcloud ${gcloud_prompt:-} components install alpha ${sudo_prefix} gcloud ${gcloud_prompt:-} components install beta ${sudo_prefix} gcloud ${gcloud_prompt:-} components update else local version=$(${sudo_prefix} gcloud version --format=json) python -c' import json,sys from distutils import version minVersion = version.LooseVersion("1.3.0") required = [ "alpha", "beta", "core" ] data = json.loads(sys.argv[1]) rel = data.get("Google Cloud SDK") if rel != "HEAD" and version.LooseVersion(rel) < minVersion: print "gcloud version out of date ( < %s )" % minVersion exit(1) missing = [] for c in required: if not data.get(c): missing += [c] if missing: for c in missing: print ("missing required gcloud component "{0}"".format(c)) exit(1) ' """${version}""" fi } </ins>

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): Is this a BUG REPORT or FEATURE REQUEST? (choose one): FEATURE REQUEST <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): all :smile: Environment: Cloud provider or hardware configuration: GCE OS (e.g. from /etc/os-release): all Kernel (e.g. ): all Install tools: Others: What happened: uses the cli a lot and does a (in the commonly used function ) before doing things like . updates can introduce breaking changes (for example in the recent 132.0.0 release it removed a bunch of flags). This has caused cross-the-board test failures like . But this is not limited to just tests. Any kubernetes GCE users who rely on the and commands could have been broken. What you expected to happen: Stop updating gcloud automatically. We could either warn user to have the required dependencies installed, or pin to specific versions of dependencies like gcloud. How to reproduce it (as minimally and precisely as possible): download 1.4.4 release tarball run and Anything else do we need to know:
cc
/cc
cc I think it's reasonable to remove gcloud update calls from scripts, as I believe gcloud is updated out of band by k8s continuous testing. The scripts will still need to work with latest, so maybe they should check for a minimum version.
Good idea.
Yeah, we definitely want to continue using latest in CI so that we know if things break, but I think it's reasonable to remove the update itself from the kubernetes cluster util scripts.


Question: name = "go_default_test", size = "large", srcs = [ <ins> "crd_overlap_storage_test.go", </ins> "etcd_cross_group_test.go", "etcd_storage_path_test.go", "main_test.go",

What happened: Created a CRD: Created another CRD defining an overlapping type with CustomResourceDefinition: Verify both exist: Deleted the one overlapping with the in-tree CRD type: Verify only the overlapping CRD remains: and is stuck deleting: What you expected to happen: Creation and deletion of the overlapping type CRD would have no effect on the in-tree data. Anything else we need to know?: The only types in the kube-apiserver (currently) that store data under a group prefix are: APIService () CustomResourceDefinition () Custom resources () When a CRD is deleted, all instances of the type it defines are first removed. That means that defining and deleting a CustomResourceDefinition that defines an overlapping APIService or CustomResourceDefinition type will remove existing APIService or CustomResourceDefinition data. We can special-case these two resources in the CRD deletion code, but we need a more robust solution before CRD v1. /cc /sig api-machinery /priority important-soon /area custom-resources
/sub
Is anyone working on this? I'll like to take a stab at this.
Not currently, no. Great. I'd suggest discussion a potential solution here before diving into code. The two main aspects are: -time - Should we forbid creating CRDs that overlap with built-in types? -time - How do we detect a CRD overlapping with a built-in type and skip the etcd cleanup for it
I think we should forbid creating CRDs that overlap (option 1).
That can help, but is not sufficient. Imagine: n forbids creating overlapping CRDs on version n creates MyType, which is allowed because that is not a built-in type upgrades to version n+1 which adds MyType as a built-in type deletes CRD What should happen?
For that, we may need delete-time detection of overlapping types as well (option 2), which may make option 1 redundant. Do you have any advice for detecting overlapping CRDs?
for now, just delete-time detection in


Question: roundedVal: int64(1073742), }, { <del> name:        "round overflowed quantity to int64", resource:    resource.MustParse("73786976299133170k"), </del> <ins> name:       "round decimal to MB", resource:   resource.MustParse("1.2Gi"), roundedVal: int64(1289), }, { name:       "round big value to MB", resource:   resource.MustParse("8191Pi"), roundedVal: int64(9222246136948), }, { name:        "round quantity to MB that would lead to an int64 overflow", resource:    resource.MustParse("8192Pi"), </ins> roundedVal:  int64(0), expectError: true, },

After upgrading our clusters to Kubernetes 1.19, the provisioning of PersistentVolumes started failing with the following Events: The value matches the actual storage request of 3.2Gi. It doesn't seem special in any way, not too big, not too small, and it always worked until 1.19. We look into the issue with and we believe the PR introduced a change that broke the PersistentVolume provisioning even when it doesn’t overflow int64. Amounts of resource in Kubernetes are represented with the . In the PersistentVolume drivers, there are some functions that manipulate storage request in order to allocate a disk with the right size. Before the mentioned PR, those quantities were retrieved by calling which relies on to return the value as an int64. This worked probably fine in most of the cases, but it could happen that the represented value would overflow an int64, and in this case the result of would be wrong. The function’s description states that it’s the caller’s responsibility to check this: This is likely what the PR wanted to handle. The change replaced all calls with which has a different signature. From where it’s used now, you would think that the second return value signals an overflow: : This is probably where the error we’re seeing is generated. I believe that the signature of was misinterpreted. From my understanding of its description, returning doesn’t mean we’re overflowing . It only means that internally, the quantity is represented as a decimal amount: Internally, depending on their values, quantities are stored/represented in : for DecimalExponent (e.g. ) or DecimalSI (e.g. ) for BinarySI that don’t have a decimal part but a unit (simplified) (e.g. ) for all the rest With the PR, Kubernetes now fails to provision volumes that have a storage request represented as . Many of the storage requests are represented this way and still perfectly valid in our opinion. I would have expected the provisioning to continue to work as those requests seemed valid and were always accepted by our underlying cloud-provider in earlier version of Kubernetes.

Create a StatefulSet with a PersistentVolumeClaim that uses vSphere storage: Kubernetes version (use ): 1.19.7 Cloud provider or hardware configuration: vSphere, but I think it impacts other cloud-provider that were changed in the PR: /sig storage /kind bug /cc Possibly related to
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I think the only reason we do not use the quantity library for scaling and why we tried to roll our own rounding code is because when scaling stuff in AsScaledInt64(3) or ScaledValue(3), the library assumes decimal SI (M, 1000), not binary SI units (Mi, 1024): You are right, our self-rolled rounding code doesn't handle values with millibyte or higher precision like 3.2Gi. Our code uses AsInt64 and then performs its own scaling. If AsInt64 returns false, the int64 returned is not in bytes (it must be in millibytes or higher precision), so our rounding code won't work, so it returns that " is too great, overflows int64". In 1.18, we used Value() to round to bytes but we didn't check it against ScaleValue as the signature suggested. So the PR in question fixed cases where Value() overflowed but inadvertently broke your case. The correct way to round, which the resource library's rounding does, is to drop the useless precision and round up. , but as I said it doesn't seem to handle binary SI quantities correctly. For decimal SI quantities, we should rely on the resource library's AsScaledInt64 and be done with it. (We could also use Value() and ScaledValue as the signature suggests but calling AsScaledInt64 directly seems simpler). Like this: For binary SI quantities, we need to do something fancier involving CanonicalizeBytes I think, I'm not sure yet.
I guess for binary SI we can go back to using Value but actually check the overflow this time.

Hi Thanks for the quick feedback on the problem's description already. Indeed, the Scaled() functions don't handle BinarySI and that's why I quickly discarded the idea of using them. Since this is about storage, I suppose most people are using this format. I just finished wrapping up a PR that I hope address both the issue we are facing, but also what was meant to be fixed in the PR your reviewed back then. It doesn't make any difference between the 3 formats however.
Yeah I commented within seconds of your PR. I glanced at it , will review more soon, but it makes sense to me to just use VAlue and check ofr overflow for both decimal SI and binary SI rather than using Scaled() for one and not the other :+1:
I think it means both. i.e - when incoming value is decimal or if value is so big that type can't store it as it switches to decimal. So while my PR fixed later case, it broke the former. :/


Question: return allErrs } <del> if params.Scope != nil || params.Namespace != nil { scope := utilpointer.StringDeref(params.Scope, "") </del> <ins> scope := utilpointer.StringDeref(params.Scope, "") </ins> <del> if !supportedIngressClassParametersReferenceScopes.Has(scope) { allErrs = append(allErrs, field.NotSupported(fldPath.Child("scope"), scope, supportedIngressClassParametersReferenceScopes.List())) } else { if scope == networking.IngressClassParametersReferenceScopeNamespace { if params.Namespace == nil { allErrs = append(allErrs, field.Required(fldPath.Child("namespace"), "`parameters.scope` is set to 'Namespace'")) } else { for _, msg := range apivalidation.ValidateNamespaceName(*params.Namespace, false) { allErrs = append(allErrs, field.Invalid(fldPath.Child("namespace"), *params.Namespace, msg)) } </del> <ins> if !supportedIngressClassParametersReferenceScopes.Has(scope) { allErrs = append(allErrs, field.NotSupported(fldPath.Child("scope"), scope, supportedIngressClassParametersReferenceScopes.List())) } else { if scope == networking.IngressClassParametersReferenceScopeNamespace { if params.Namespace == nil { allErrs = append(allErrs, field.Required(fldPath.Child("namespace"), "`parameters.scope` is set to 'Namespace'")) } else { for _, msg := range apivalidation.ValidateNamespaceName(*params.Namespace, false) { allErrs = append(allErrs, field.Invalid(fldPath.Child("namespace"), *params.Namespace, msg)) </ins> } } <ins> } </ins> <del> if scope == networking.IngressClassParametersReferenceScopeCluster && params.Namespace != nil { allErrs = append(allErrs, field.Forbidden(fldPath.Child("namespace"), "`parameters.scope` is set to 'Cluster'")) } </del> <ins> if scope == networking.IngressClassParametersReferenceScopeCluster && params.Namespace != nil { allErrs = append(allErrs, field.Forbidden(fldPath.Child("namespace"), "`parameters.scope` is set to 'Cluster'")) </ins> } }

This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
It looks like you are correct. Please feel free to submit a pull request to fix it. For information on contribute, please refer to:
/sig network /sig auth
/assign
when reporting issues please give more context to explain it, in this case it is required so it never can be nil in this comparison
Who wants to fix this? This whole block is suspect - it checks for nil later, too.
I'm working on it in !


Question: rc=0 # find test files accessing the mutable global feature gate or interface <del> direct_sets=$(find -L . -name '*_test.go' -exec grep -Hn 'MutableFeatureGate' {} ; 2>/dev/null) || true </del> <ins> direct_sets=$(git grep MutableFeatureGate -- '*_test.go') || true </ins> if [[ -n "${direct_sets}" ]]; then echo "Test files may not access mutable global feature gates directly:" >&2 echo "${direct_sets}" >&2

PR was reverted by because it (erroneously) relied on OpenAPIV3 feature to be enabled, but would non-deterministically leave the feature disabled after running. I tracked it down to the fact that this test disables all or features in a loop which iterates over a , which are known to have unstable iteration order: When key is processed before key , then would disable before s default value is to be saved. Normally, ignores features that have been explicitly set, so it only affects those which fall back on the value. But this issue does not just affect but any non-GA feature. I suggest the loop in be made to skip the special feature keys such as . /cc /cc The value before every feature gate should be restored after running . This test always fails on my machine (just pasted it into ): No response <details</details<details</details<details</details<details</details<details</details<details</details
/assign
Wow, good catch, sorry to make you track down what turned out to be a bug in our test
/sig auth
/area test
/triage accepted


Question: <del> /* Copyright 2017 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package eventratelimit import ( "time" ) // realClock implements flowcontrol.Clock in terms of standard time functions. type realClock struct{} // Now is identical to time.Now. func (realClock) Now() time.Time { return time.Now() } // Sleep is identical to time.Sleep. func (realClock) Sleep(d time.Duration) { time.Sleep(d) } </del>

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --While working on , I realized that implements another rather than using the one already implemented in I'm not sure why this was re-implemented, but if we can migrate to using itself, then the code in the package can be cleaned up significantly (as part of ). I expect areas of the codebase that make use of a real clock to consistently use instead of re-implementing it NA Another (slightly) related issue: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/remove-kind bug /kind cleanup /sig auth
/cc fyi
/assign
Better yet, use the one in
/sig api-machinery
/assign /triage accepted


Question: return nil, err } <ins> // We have to find the plugins by volume spec (NOT by plugin name) here // in order to correctly reconstruct ephemeral volume types. // Searching by spec checks whether the volume is actually attachable // (i.e. has a PV) whereas searching by plugin name can only tell whether // the plugin supports attachable volumes. attachablePlugin, err := rc.volumePluginMgr.FindAttachablePluginBySpec(volumeSpec) if err != nil { return nil, err } deviceMountablePlugin, err := rc.volumePluginMgr.FindDeviceMountablePluginBySpec(volumeSpec) if err != nil { return nil, err } </ins> var uniqueVolumeName v1.UniqueVolumeName if attachablePlugin != nil || deviceMountablePlugin != nil { uniqueVolumeName, err = util.GetUniqueVolumeNameFromSpec(plugin, volumeSpec)

When a pod is marked as deleted while kubelet is down / being restarted, newly started kubelet does not clean up CSI filesystem volumes of the pod. Newly started kubelet tries to reconstruct the volume using CSI's function. This part looks working, CSI volume plugin loads its json file. But then VolumeManager checks if the volume is still mounted in directory. There are two issues: does not require volumes to be presented as mounts. They can be just directories with files on them. This will be case of the most of in-line volumes. Kubelet checks the right directory given by
/sig storage related to cc
Maybe a stupid idea... Reconciler sync has found a directory in . Reconciler has no way how to check that CSI volume (or any other volume) is really mounted / published there. Depending on DSW: The volume is in DSW -from comment here it seems that the SetUp (=NodePublish) will be called on the volume: The volume is not in DSW and it will be to ASW, so it's unmounted / unpublished in the next sync: In both cases, we can expect that the volume plugin / CSI driver is idempotent and SetUp() won't do anything if the volume is already set up / TearDown() will not do anything if the volume has been torn down already. Are our volume plugins really idempotent? IMO they are. We perhaps don't need to check for presence of mount point anywhere, just presence of directory should be enough to reconstruct the volume and set it up / tear it down. Adding to the loop.
Filed , PTAL
I'm curious how the reconstruction e2e tests passed
https://k8s-*force
They start kubelet with a pod already deleted. If the pod still exists, volume reconstruction can see it in DSW and does nothing when fails: [Reconstruction fails because it checks for a mount in a wrong directory.] In this case, kubelet waits for the volume to appear in VolumesInUse before calling . The pod is deleted before that and kubelet then just deletes DSW entry and the volume is never cleaned up.
They pass because their container does not handle SIGTERM and it takes 30 seconds to kill them. Kubelet has enough time to the volume during reconciliation.

It turns out that I tested with broken version of and reconstruction is broken only for ephemeral volumes that don't mount.
So the conclusion is that we only need to fix for ephemeral volumes? Thanks!
Yes, sorry again about the noise.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or .

/lifecycle stale
/remove-lifecycle rotten
/lifecycle frozen
From this issue surfaced through e2e tests using the hostPath CSI driver. The cause is (which is used in for reconstruction) - it doesn't play well with volumes that bind mount from a directory instead of a device.
To work around this bug here, volume life cycle checking was disabled for all tests, not just the subpath test: This check is useful and should be enabled again.
Hey everyone! I arrived at this conversation by investigating on issue (seems to me like a duplicate of ). I'm wondering if there's something I can help with? I'm still a bit new to the codebase, though. I was able to reproduce the problem by using the following ginkgo focus command in the e2e framework: And do see that if we manually add the --check-volume-lifecycle=false to the statefulset/csi-hostpathplugin tests will pass correctly. I also see that during the process, something in the cleanup function doesn't correctly delete the pv's after the tests are done, so they remain alive if we execute a kubectl get pv command.
I think I found a bit of more info on the issue probably could be of some help. I think that the volume is not being correclty marked as unmounted inside in GenerateUnmountVolumeFunc. I'm not sure why but I'll continue to investigate on it. I think this error is coming from DeletePodFromVolume in
I a couple of extra debug lines to that lines in the file, and can see that when DeletePodFromVolume tries to see if the volume exists, it doesn't find it in the asw.attachedVolumes struct, like so: Probably we should search for the name of the attached volume differently for ephemeral volumes?
/assign
There are two problems: 1) The IsLikelyNotMountPoint() call that , which is known not to work for bind mounts. The mountpoint never gets found, so volume reconstruction fails with "... is not mounted", the PVC never never gets to the list of attached volumes, and that's why we end up hitting this error: 2) CSI inline volumes hit a different error (in addition to the first problem). The call path looks like this (starting in ): Inline volumes do not have a PV spec, so it fails . We should be calling instead.

There is an ongoing that changes to utilize syscall to detect mount point by using . By using , bind mount will be properly detected fast, but it requires kernel version is v5.6 or later. So, we would be able to also utilize in . Then, the issue of bind mount can be resolved at least for kernel v5.6+, and we will be able to focus on how we solve this issue for old kernels.


Question: import ( "fmt" <ins> "net" </ins> "os" "path/filepath" "strings"

/kind bug What happened: Currently, if the user sets the cidr with --pod-network-cidr with kubeadm, the kube-controller-manager is passed the cidr with the --cluster-cidr arg. --node-cidr-mask-size is not passed to the kube-controller-manager and the controller-manager used the default value of 24 The default value of 24 if good for IPv4, but for IPv6 the controller-manager throws an error because the node-cidr-mask-size is incorrect. What you expected to happen: The user should be able to specify the node-cidr-mask-size for IPv6 How to reproduce it (as minimally and precisely as possible): Use Kubeadm specify an IPv6 address with --pod-network-cidr. with a subnet greater than 24. For example kubeadm init --pod-network-cidr=2001:101::0/64 Anything else we need to know?: Environment: IPv6 Kubernetes version (use ): Cloud provider or hardware configuration**: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/area ipv6 /area kubeadm
/assign rpothier


Question: } func removeMissingExtendedResources(pod *v1.Pod, nodeInfo *schedulerframework.NodeInfo) *v1.Pod { <del> podCopy := pod.DeepCopy() for i, c := range pod.Spec.Containers { // We only handle requests in Requests but not Limits because the // PodFitsResources predicate, to which the result pod will be passed, // does not use Limits. podCopy.Spec.Containers[i].Resources.Requests = make(v1.ResourceList) for rName, rQuant := range c.Resources.Requests { if v1helper.IsExtendedResourceName(rName) { if _, found := nodeInfo.Allocatable.ScalarResources[rName]; !found { continue </del> <ins> filterExtendedResources := func(containers []v1.Container) { for i, c := range containers { // We only handle requests in Requests but not Limits because the // PodFitsResources predicate, to which the result pod will be passed, // does not use Limits. filteredResources := make(v1.ResourceList) for rName, rQuant := range c.Resources.Requests { if v1helper.IsExtendedResourceName(rName) { if _, found := nodeInfo.Allocatable.ScalarResources[rName]; !found { continue } </ins> } <ins> filteredResources[rName] = rQuant </ins> } <del> podCopy.Spec.Containers[i].Resources.Requests[rName] = rQuant </del> <ins> containers[i].Resources.Requests = filteredResources </ins> } } <ins> podCopy := pod.DeepCopy() filterExtendedResources(podCopy.Spec.Containers) filterExtendedResources(podCopy.Spec.InitContainers) </ins> return podCopy }

The function removeMissingExtendedResources() located at , is designed to remove any extended resources from a container’s requests that are not found in nodeInfo.Allocatable before the pod is admitted. This is necessary to support cluster-level resources, which are extended resources that are unknown to nodes. However, this function only removes missingExtendedResources from the container’s requests, and does not process the requests of initContainers. This could lead to issues, as initContainers might request resources that do not exist on the node, which could result in the pod failing to be admitted or other unforeseen issues. As such, it may be necessary to refine the removeMissingExtendedResources() function to also handle initContainer’s requests. Refine the removeMissingExtendedResources() function to also handle initContainer’s requests. Create a pod that requests cluster-level extended resources in both the container and InitContainer. No response The problem persists in the latest version. <details</details<details</details<details</details<details</details
/sig node
/triage accepted /priority important-longterm


Question: "//pkg/master/ports:go_default_library", "//pkg/scheduler/nodeinfo:go_default_library", "//staging/src/k8s.io/api/apps/v1:go_default_library", <ins> "//staging/src/k8s.io/api/autoscaling/v1:go_default_library", </ins> "//staging/src/k8s.io/api/batch/v1:go_default_library", "//staging/src/k8s.io/api/batch/v1beta1:go_default_library", "//staging/src/k8s.io/api/core/v1:go_default_library",

According to this APIsnoop query, there are still some remaining ReplicationController endpoints which are untested. Create a ReplicationController with a static label Patch the ReplicationController with a new Label and updated data Patch the ReplicationController's status Patch/Scale the ReplicationController's Replica count to 2 Replicas Get the ReplicationController Replace the ReplicationController's Status List all ReplicationControllers in all Namespaces find the ReplicationControllers(1) ensure that the ReplicationController is found and is patched Delete Namespaced ReplicationControllers(1) via a Collection with a LabelSelector Discover useragents: See all endpoints hit from this test Display test coverage change If a test with these calls gets merged, test coverage will go up by 7 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
Deployment replaced ReplicationController a while back. It should be part of conformance. We need to check more than the data-aspect (beyond crud), we need to verify the data-plane. In this case we need to check the pods as well and not just the ReplicationController status. Moved to backlog by conformance subgroup.
PR:
/open /reopen We need this open to track the test promotion and flakyness. We can't write a promotion PR until the test has no flakes for 2 weeks.
Reopened this issue. <detailsIn response to : /open /reopen We can't write a promotion PR until the test has no flakes for 2 weeks. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/retitle Promote CoreV1ReplicationController lifecycle mock test - +7 coverage Need to verify two weeks on non-flakeyness.
/retitle Write CoreV1ReplicationController lifecycle test+promote - +7 coverage
Closing in favour of /close
Closing this issue. <detailsIn response to : Closing in favour of /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: "os" "strconv" "time" <ins> "golang.org/x/text/encoding/unicode" "golang.org/x/text/transform" </ins> ) const VHD_COOKIE = "636f6e6563746978"     // conectix

I ran an OSS license scan of a codebase that has this project as a dependency and came across an interesting hit. This go-vhd dependency specified here: has a snippet of code from StackOverflow which is license under creative commons sharealike. Here's the code in question: I'm not a license expert, so I don't know if there are any real issues/conflicts here or not, but wanted to share to see what you all think. Here's SO's license info for reference: My understanding is that the potential issue would be with this clause: which could potentially require go-vhd and projects that depend on it to use the Creative Commons license as well.
Thanks for the report! can you please take a look if this is concerning? If so, we'll check if we can avoid this dependency (looks unlikely unless we duplicate what it's ) or we'll follow up upstream to update the code. /cc OWNERS for this repo
CC-BY-SA can be problematic in Apache-2.0 projects like k8s because of the ShareAlike aspect as noted. I'd recommend working with upstream to rewrite just that one function particularly since it looks like it's fairly limited.
let's please move this issue to kubernetes/kubernetes?
/provider azure
/sig release /area code-organization
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig release /area code-organization Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
cc /assign
/area code-organization
/assign
/milestone v1.19 /priority important-soon
Not a huge difference, but this is likely actually licensed under , not 4.0. See
the k/k code actually does not reference , since that function is in , do you mean we should remove from k/k totally?
Yes, we would either need to remove from k/k or ask the maintainers of to update . This means that (and k/k if we end up using it) would need to be licensed until CC-BY-SA.
Would this ultimately be backported to older versions of k8s or would you just do it in 1.19?

thanks, Let me remove the code in v1.19, actually that part of functionality is deprecated in Azure cloud provider already. My initial thought is only do this in v1.19
/kind bug /priority important-soon /sig cloud-provider /area provider/azure
Since it isn't trivial to remove as a dependency entirely (), I've created upstream to track updating the code there.


Question: } // no cached CIDR is Ok during teardown <del> if cidr, ok := plugin.podCIDRs[id]; ok { </del> <ins> cidr, hasCIDR := plugin.podCIDRs[id] if hasCIDR { </ins> glog.V(5).Infof("Removing pod CIDR %s from shaper", cidr) // shaper wants /32 if addr, _, err := net.ParseCIDR(cidr); err != nil {

Kubelet log is full of errors from tearing down network at HEAD. From the log, a lot of them were the result of trying to access . Note that if you an exited container, the returned PID will be 0. Therefore, my guess is that kubelet tried to tear down the network after the container has already exited. I didn't dig further. could you take a look?
/cc
Is this a dup of ? We can add a safe guard to prevent TearDown to proceed, but it will not solve the root cause. IIUC,the root cause is the race between syncPod and podKiller.
See also
I don't think failed because of this. this is probably caused by , like you said. The fact that we are seeing more errors than before is likely due to , which enforces graceless namespace deletions. Before that, most objects in the e2e were deleted gracefully and the race condition was rarely hit. Now most objects will hit this path. It'd still be a good idea to add a safeguard to check whether the pid is 0.


Question: } } <ins> func TestCreateDeploymentRollbackValidation(t *testing.T) { storage, server := newStorage(t) rollbackStorage := storage.Rollback rollback := apps.DeploymentRollback{ Name:               name, UpdatedAnnotations: map[string]string{}, RollbackTo:         apps.RollbackConfig{Revision: 1}, } ctx := genericapirequest.WithNamespace(genericapirequest.NewContext(), namespace) if _, err := storage.Deployment.Create(ctx, validNewDeployment(), rest.ValidateAllObjectFunc, &metav1.CreateOptions{}); err != nil { t.Fatalf("Unexpected error: %v", err) } validationError := fmt.Errorf("admission deny") alwaysDenyValidationFunc := func(obj runtime.Object) error { return validationError } _, err := rollbackStorage.Create(ctx, &rollback, alwaysDenyValidationFunc, &metav1.CreateOptions{}) if err == nil || validationError != err { t.Errorf("expected: %v, got: %v", validationError, err) } storage.Deployment.Store.DestroyFunc() server.Terminate(t) } </ins> // Ensure that when a deploymentRollback is created for a deployment that has already been deleted // by the API server, API server returns not-found error. func TestEtcdCreateDeploymentRollbackNoDeployment(t *testing.T) {

What happened: I'm unable to configure the api-server through a to pass admission requests to my for the resource. I'm adding the following rule to the When performing rollbacks, I'm not seeing any calls to the admission webhook. What you expected to happen: To receive validation requests when performing rollbacks. How to reproduce it (as minimally and precisely as possible): Create a ValidatingAdmissionWebhook and perform a rollback request. Anything else we need to know?: Mentioned this on the #sig-api-machinery channel on Slack, and suggested the error was that the -object never is used: Environment: Kubernetes version (use ): 12.3 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Container Linux by CoreOS 1911.5.0 (Rhyolite) 4.14.84-coreos docker://18.6.1 Kernel (e.g. ): Install tools: Others: /kind bug
/sig apps
/sig api-machinery
Good point, should call .


Question: return false, nil } <ins> // nodeIsHealthy returns true if the node looks healthy. func (rc *reconciler) nodeIsHealthy(nodeName types.NodeName) (bool, error) { node, err := rc.nodeLister.Get(string(nodeName)) if err != nil { return false, err } return nodeutil.IsNodeReady(node), nil } </ins> func (rc *reconciler) reconcile() { // Detaches are triggered before attaches so that volumes referenced by // pods that are rescheduled to a different node are detached first.

A/D controller tries to detach volumes that are still mounted after 6 minutes. The conditions to trigger the detach are: I.e. all containers are finished. But that does not guarantee that volumes are unmounted. When a CSI driver can't unmount a volume for 6 minutes, the volume is force detached by A/D controller, corrupting the volume and leaving volume mounts on the node in a bad state. Note that the node is perfectly healthy, it just can't unmount a volume from whatever reason. A/D controller detaches volumes after a Pod is deleted from the API server, i.e. the volumes are unmounted or someone force-deleted the pods. Very artificial reproducer: a Pod with an attachable CSI volume. the node, make the volume mount busy, e.g.: Every unmount of the volume fails and kubelet/the CSI driver periodically retry. the pod 6 minutes, see the volume is detached without unmounting first. KCM logs: Note that the Pod is still in the API server: And the node is healthy: Note that some CSI drivers (e.g. the cloud ones) do not allow detaching a volume while it's still mounted. Not all CSI driver have such possibility and they rely on Kubernetes not to call ControllerUnpublish before NodeUnpublish / NodeUnstage succeeds. <detailsToday's master </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage /priority important-longterm
Right now, a volume is detached from a node when: (all pods that use the volume are deleted from the API server OR all their container are stopped) AND (the volume is unmounted OR 6 minutes passed). Potential solution 1: Issue detach of a mounted volume only when all pods are force-deleted from the API server.

More formally: Detach when [(All pods that use the volume are deleted from the API server OR all their container are stopped) AND the volume is unmounted] OR [all pods that use the volume are deleted from the API server AND 6 minutes passed]. Is this change subtle enough that it does not break existing behavior (e.g. tooling on top of Kubernetes)?
Potential solution 2: Add a new flag to CSIDriver, say . We would need to extend internal Detach() call to include a flag if the volume to detach is mounted or not (A/D controller knows that) and the CSI volume plugin would check CSIDriver and potentially reject detach of a mounted volume. This way, it would not be possible to ever force detach a mounted volume, even when a pod is force-deleted! Potential solution 3: As a variant, we could add a cmdline option to the external-attacher, like . When that cmdline option is specified, the external attacher would check , if the volume to detach is still mounted, and refuse such detach (with a reasonable error message). CSI driver vendors could opt-in into this behavior. This way, it would not be possible to ever force detach a mounted volume, even when a pod is force-deleted! Potential solution 4: As a variant, we could add to CSIDriver, where each driver could put their own timeout before force-detaching a volume, with an option to use infinity.
Note to self: check , it might help on the CSI side.
Solution 3 will not work well with ungraceful node shutdown - attach/detach controller does not watch pods and force-deleting pods is the main point of ungraceful node shutdown KEP. /me will try if solution 1 would work when NodeStage times out (is Pod deleted from the API server before or after NodeUnstage is called?)
was right, a pod is deleted (without force) from the API server after NodeUnpublish succeeds, it does not wait for NodeUnstage to finish. Therefore we cannot distinguish if a pod was force-deleted (and we should start 6 minute timer) or pod was deleted without any force and only NodeUnstage is taking more time than expected. Solution 1 is then not possible.
Now we are down to solution 2 and 4.

For solution 2, if driver does not have a way to check whether volume is mounted or not, the behavior will not change and we still force detach after 6 mins. And it might happen in the middle of unstage ..
Summing up yesterday's call: A/D controller should check if a node is unhealthy before force detaching volumes from it. It should never force detach a volume from a healthy node, not even after 6 minutes. Question what healthy means is open and we should talk to sig-node. is a working assumption.
I agree with both of these. I'd just emphasize that that particular definition of unhealthy definitely includes perfectly healthy nodes in a network partition. This change is no worse than the status quo ante for those nodes, and a definite improvement on the healthy nodes, so I wholeheartedly support it.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/remove-lifecycle rotten
/assign


Question: logger.Error(err, "Scheduler cache UpdatePod failed", "pod", klog.KObj(oldPod)) } <del> sched.SchedulingQueue.AssignedPodUpdated(logger, oldPod, newPod) </del> <ins> // SchedulingQueue.AssignedPodUpdated has a problem: // It internally pre-filters Pods to move to activeQ, // while taking only in-tree plugins into consideration. // Consequently, if custom plugins that subscribes Pod/Update events reject Pods, // those Pods will never be requeued to activeQ by an assigned Pod related events, // and they may be stuck in unschedulableQ. // // Here we use MoveAllToActiveOrBackoffQueue only when QueueingHint is enabled. // (We cannot switch to MoveAllToActiveOrBackoffQueue right away because of throughput concern.) if utilfeature.DefaultFeatureGate.Enabled(features.SchedulerQueueingHints) { sched.SchedulingQueue.MoveAllToActiveOrBackoffQueue(logger, queue.AssignedPodUpdate, oldPod, newPod, nil) } else { sched.SchedulingQueue.AssignedPodUpdated(logger, oldPod, newPod) } </ins> } func (sched *Scheduler) deletePodFromCache(obj interface{}) {

/sig scheduling /kind cleanup Ref: A test helper function creates a goroutine to pop a Pod from the scheduling queue. This goroutine never stops until the scheduling queue is ed or is somehow stopped for waiting for a new Pod. Ideally, when reaching timeout, we should somehow stop from waiting for Pod to be enqueued to activeQ.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
We can replace this function with and it should be enough? I think we no longer need to maintain this function.
Not enough, the core of this problem is that we cannot cancel once we start . Even if we change function to whatever, we will hit this problem. So, we need:
Or, alternatively and much more easily, maybe we can just replace all with and remove . kills the entire test if reaching timeout and we could care less about leaked goroutine then.
I don't think we can do this. From the test case: The expected behavior of this case is that we receive a timeout to make sure no pods are to . So, reaching timeout is acutally a success. So, I think: option 1: using sth like this in : option2: do sth in the test code This part is where it confused me. The current code is fine as we closed schdulerQueue after called , and after I think there is no test case using this this function anymore(seeems that you refactored the test), so are we trying to solve a potential problem?
Oh, then we can just remove NextPod in the PR.
/assign
The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/close
Closing this issue. <detailsIn response to : /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

/sig scheduling /kind bug AssignedPodAdded(): AssignedPodUpdated(): When assigned Pods are created/updated, cluster events (Pod/Add or Pod/Update) should be delivered for all plugins that subscribe those events. But, actually pre-filters Pods to receive those events based on Pods' affinity and . - meaning only PodAffinity/NodeResourceFit receive them actually. The problem here is similar to - in in-tree plugins, PodAffinity, PodTopologySpread and NodeResourceFit subscribe Pod add and/or update events. And, Pods rejected by PodTopologySpread will never be requeued to activeQ by Pod-related events. The same happens for custom plugins registering pod add and/or update events. All Pods will get chances to get back to activeQ by Pod-related events. Run ↓ in No response <detailsmaster </details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
The Kubernetes project currently lacks enough contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale


Question: // IngressClass resource create/read/update/watch verbs ginkgo.By("creating") <del> ingressClass1, err := createIngressClass(cs, "ingressclass1", true, f.UniqueName) </del> <ins> ingressClass1, err := createIngressClass(cs, "ingressclass1", false, f.UniqueName) </ins> framework.ExpectNoError(err) <del> _, err = createIngressClass(cs, "ingressclass2", true, f.UniqueName) </del> <ins> _, err = createIngressClass(cs, "ingressclass2", false, f.UniqueName) </ins> framework.ExpectNoError(err) <del> _, err = createIngressClass(cs, "ingressclass3", true, f.UniqueName) </del> <ins> _, err = createIngressClass(cs, "ingressclass3", false, f.UniqueName) </ins> framework.ExpectNoError(err) ginkgo.By("getting")

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: gce-master-scale-correctness Which test(s) are flaking: [sig-network] Ingress API should support creating Ingress API operations [Conformance] Testgrid link: https://k8s- Reason for failure: Anything else we need to know: /kind flake /sig scalability /sig network /cc
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
this is caused by the issue described in a different test is creating a default ingressclass, which breaks creation of ingress objects if there was already a default ingressclass set up in the cluster /assign
this should resolve the issue:
I take it back, looks like the conformance test is also creating default ingressclass objects... it should not fix in