Question: } } <ins> func TestInitialPendingVolumesForPodAndGetVolumesInUse(t *testing.T) { tmpDir, err := utiltesting.MkTmpdir("volumeManagerTest") if err != nil { t.Fatalf("can't make a temp dir: %v", err) } defer os.RemoveAll(tmpDir) podManager := kubepod.NewBasicPodManager(podtest.NewFakeMirrorClient(), secret.NewFakeManager(), configmap.NewFakeManager()) node, pod, pv, claim := createObjects() claim.Status = v1.PersistentVolumeClaimStatus{ Phase: v1.ClaimPending, } kubeClient := fake.NewSimpleClientset(node, pod, pv, claim) manager := newTestVolumeManager(tmpDir, podManager, kubeClient) stopCh := runVolumeManager(manager) defer close(stopCh) podManager.SetPods([]*v1.Pod{pod}) // Fake node status update go simulateVolumeInUseUpdate( v1.UniqueVolumeName(node.Status.VolumesAttached[0].Name), stopCh, manager) // delayed claim binding go delayClaimBecomesBound(kubeClient, claim.GetNamespace(), claim.ObjectMeta.Name) err = manager.WaitForAttachAndMount(pod) if err != nil { t.Errorf("Expected success: %v", err) } } </ins> func TestGetExtraSupplementalGroupsForPod(t *testing.T) { tmpDir, err := utiltesting.MkTmpdir("volumeManagerTest") if err != nil {

Deployment/Service manifest: logs from events: In aws, the volume is attached, On the box: In only one is mounted: Where can I debug this further?
Take a look at the on your master for events related to your volume . That will show you when the volume was attached/detached. It appears that your volume attached without issue so there should be nothing unusual here. Then take a look at on the node that the volume is supposed to be attached to. Looks for the UID of your pod (). I suspect one of your mounts is failing. It looks like you are using hyperkube so it might be a mount namespace propagation issue.
I believe this happens when the RC/Pod is being created shortly after the PV or PVC. The latter two appear to need several seconds to be fully available (even if the status is already set to "Bound"). I ran into the same issue with a NFS PV. When I delete and create PV+PVC+SVC+RC all in one go, the associated Pod will be in ContainerCreating state for several seconds (10,<60), then log the warning event and afterwards be created without any problems or further delays. When I first create PV+PVC, wait for several seconds (30) and only then create SVC+RC, the pod will be created immediately and no warning event will be logged. One oddity I noticed is that the PVC will have capacity 0 and empty access modes (as shown by ), even though it was setup to request ROX for 1Gi and the PV provides 1Gi with ROX. This might be a different bug, though.
Thanks for digging into it The capacity 0 issue () was fixed by PR and is in v1.3.5. This is more or less expected behavior. The mount logic has exponential backoff so if it rapidly encounters PVC/PV does not exist, it'll back off the retries resulting in a longer mount time. We can consider optimizing this particular scenario. But I'll lower the priority on this for now and remove the milestone.
Maybe the "Bound" status could be delayed, so the RC would be held back for a while and be scheduled only when the PVC is actually ready?
I don't think the RC actually waits for PVC to be bound before scheduling.

I have also this problem using deployment with HELM, I deploy all my pv,pvc,deploy,svc in one Charts, and my pod who requiere a volume from the PVC get scheduled and stay in ContainerCreating mode. I need to wait minutes or delete pods for they correctly get rescheduled using the volume. Here is the log when the pod get automaticly deleted (after minutes) : It would be awesome if the deployment or RC wait for the PVC to be bound before schedule !
may slightly improve this situation as the maximum backoff time of 2 minutes will not fire anymore.
Note that the same error is thrown when node-size == C5 or M5 for kubectl and with kops . Solution is to set node as t2 machines. Due to it seems.
Reading from the new solution is:


Question: } // setting a threshold to 0% disables; non-empty map overrides default value (necessary due to omitempty) initialConfig.EvictionHard = map[string]string{"memory.available": "0%"} <ins> if initialConfig.FeatureGates == nil { initialConfig.FeatureGates = map[string]bool{} } </ins> initialConfig.FeatureGates[string(LSCIQuotaFeature)] = quotasRequested }) runEvictionTest(f, evictionTestTimeout, noPressure, noStarvedResource, logDiskMetrics, []podEvictSpec{

Seen in this job failing on multiple tests, this is just one of the stack traces it seems that for a long time :smile: it panics here I don't have a good understanding of those tests mechanics, but it seems a complex setup embedding functions and goroutines before the test setup, I suggest to move to something more simple No response_ /sig node
ping I don't have time for working on this unfortunately but happy to help on reviews, this can also be a good task for new contributors, also I don't consider a bad idea to rework the tests a bit to make them easier to read and simpler, I did that for sig-network and we have a pretty low flake rate since then
/assign /cc This probably came out of /triage accepted
/priority important-soon /milestone v1.23
Ooops - I'm assuming it's going through one of the paths where Features or whatever isn't initialized.


Question: "-m", "mark", "!", "--mark", fmt.Sprintf("%s/%s", masqueradeMark, masqueradeMark), "-j", "RETURN"); err != nil { klog.Errorf("Failed to ensure filtering rule for %v: %v", KubePostroutingChain, err) <del> return </del> <ins> return false </ins> } // Clear the mark to avoid re-masquerading if the packet re-traverses the network stack. // We know the mark bit is currently set so we can use --xor-mark to clear it (without needing

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: ci-kubernetes-e2e-gci-gce-alpha-features Which test(s) are failing: Networking should recreate its iptables rules if they are deleted [Disruptive] Since when has it been failing: Commit Testgrid link: Reason for failure: Anything else we need to know: Possibly related to this? It's listed in the git diff /sig network
/assign /cc Thanks for detecting it so quicklly
most probably is related to
/unassign /assign he has already submitted a fix
/triage accepted
Reviewed. Marking as resolved on the CI Signal Board. Once was merged, failures stopped and they have not re-occurred in the last 8 days (Oct 10th)
Yay


Question: go_test( name = "go_default_test", <del> srcs = [ "controller_test.go", "patch_test.go", ], </del> <ins> srcs = ["controller_test.go"], </ins> embed = [":go_default_library"], deps = [ "//pkg/controller:go_default_library",

PR service patch methods to Once there is a release with that PR included, the code in service-controller and legacy-cloud-providers/gce can be updated to use those. /area platform/gce /assign
/sig network
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/remove-triage unresolved
Those labels are not set on the issue: <detailsIn response to : /remove-triage unresolved Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Created a PR to use these new functions in Service-controllers.


Question: lastErr = err return false, nil } <ins> if warningWriter.WarningCount() != 1 { lastErr = fmt.Errorf("expected one warning, got: %v", warningWriter.WarningCount()) return false, nil } if !strings.Contains(recordedWarnings.String(), expectedWarning) { lastErr = fmt.Errorf("unexpected warning, expected: n%vn, got: n%v", expectedWarning, recordedWarnings.String()) return false, nil } lastErr = nil </ins> return true, nil <del> }); err != nil { </del> <ins> }); err != nil || lastErr != nil { </ins> t.Fatalf("failed to wait for apiserver handling webhook mutation: %v, last error: %v", err, lastErr) } <del> if lastErr != nil { t.Fatal(lastErr) } if warningWriter.WarningCount() != 1 { t.Errorf("expected one warning, got: %v", warningWriter.WarningCount()) } if !strings.Contains(recordedWarnings.String(), expectedWarning) { t.Errorf("unexpected warning, expected: n%vn, got: n%v", expectedWarning, recordedWarnings.String()) } recordedWarnings.Reset() </del> // Make sure reset happens in update requests pod, err = client.CoreV1().Pods("default").Update(context.TODO(), pod, metav1.UpdateOptions{})

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --pull-kubernetes-integration TestMutatingWebhookResetsInvalidManagedFields Not sure. Example failures from recent PRs: (PR: ) (PR: ) /sig api-machinery /kind flake
There's even a few failures in the post-submit master integration tests job: (ignore the test/integration/job failures, I have an open PR to fix that).
cc
Thanks! /cc /wg api-expression
Oh no, what did I do :-/ I'll have a look. Might be that the admission webhook is not up quick enough. /assign
/triage accepted
I tried this locally and it passes, so maybe I can try stress testing this
Would increasing the timelimit here give more time to the admissionhook to come up ?
I think so. Looked at it a few days ago, I'm not sure how much we should increase it. I don't want to slow down the tests a lot more if we can avoid it. Do we know if the other admission tests (duplicate owner ref) have the same issue?
Can we make the webhook blocking so that the request will wait for it to be ready? Can we make sure that it's up-and-running somehow before performing the actual test? Having an arbitrary "wait" before we start the test seems wrong.
The poll and timeout are fine as is. lastErr is only getting set, never unset. If validateManagedFieldsAndDecode returns nil, lastErr should get cleared.
Hmm.. actually, the one failing is the Update request, after the Patch request has already been checked. The patch request passing means the webhook was effective and working, the update request should work consistently the first time. If the patch request really is ensuring the webhook was installed, then a timeout or retry on the update request is not the right approach.
But isnt the patch request alos failing to report the adequate warnings
this continues to be the flakiest test in the integration job in by a wide margin
So sorry for letting this slip for so long :-/ I'll work on it right when I get back home in an hour.


Question: csipbv1 "github.com/container-storage-interface/spec/lib/go/csi" api "k8s.io/api/core/v1" "k8s.io/apimachinery/pkg/api/resource" <ins> "k8s.io/apimachinery/pkg/util/wait" </ins> "k8s.io/kubernetes/pkg/volume" "k8s.io/kubernetes/pkg/volume/csi/fake" volumetypes "k8s.io/kubernetes/pkg/volume/util/types"

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Kubelet dies with a panic exception if CSI plugin returns NodeId == "" and nil error. Exception log see at the end of the message What you expected to happen: Kubelet should just raise an error to the plugin and ignore the registration. How to reproduce it (as minimally and precisely as possible): Create a CSI plugin with only Node that would implement GetNodeInfo as follows: Register it as a DaemonSet and see kublet crush Anything else we need to know?: Environment: Kubernetes version (use ): 1.14.10-gke.17 Cloud provider or hardware configuration: GKE OS (e.g: ): Container-Optimized OS (cos) Kernel (e.g. ): - Install tools: Network plugin and version (if this is a network-related bug): Others: Related exception Related line of code: - no check whether error is not nil before calling Error() on it.
/sig storage
/sig node is this the version of the kubelet too? 14 is out of support. is the issue reproducible with a 1.15+ kubelet?
/assign
I checked the code - same defect still exists in current codebase.
thanks /priority important-soon
Wow, that was fast. Thanks!


Question: srcs = [ "container_gc_test.go", "convert_test.go", <ins> "docker_manager_linux_test.go", </ins> "docker_manager_test.go", "docker_test.go", "images_test.go",

go test -v This is because there are a few test cases in dockertools are linux-specific.


Question: if err != nil { return err, retry } <del> // Always update the cache upon success </del> <ins> // Always update the cache upon success. // NOTE: Since we update the cached service if and only if we successully // processed it, a cached service being nil implies that it hasn't yet // been successfully processed. </ins> cachedService.service = service s.cache.set(namespacedName.String(), cachedService) case cache.Deleted:

In , I found a logic error (when I had a broken cloud implementation): If createLoadBalancerIfNeeded returns err, then cachedService will have service = nil. This causes updateLoadBalancerHosts to panic. It would be easy to skip cacheService if service=nil, but I wasn't sure whether this should then be included in servicesToRetry or not. Another problem (by inspection) is that I think there's a race condition where updateLoadBalancerHosts can run immediately after getOrCreate, before the service field has been set.
/cc
Great catches, Justin, thanks! I believe that simply not processing cached services with a nil api.Service object in the node pool reconciler is the correct approach. A nil Service object implies that the load balancer reconciler still intends to process it, and the LB reconciler always uses the most up-to-date list of nodes. I'll send a PR ASAP.


Question: # https://bazel-review.googlesource.com/#/c/10390/ http_archive( name = "io_bazel", <del> sha256 = "667d32da016b1e2f63cf345cd3583989ec4a165034df383a01996d93635753a0", </del> <ins> sha256 = "892a84aa1e7c1f99fb57bb056cb648745c513077252815324579a012d263defb", </ins> strip_prefix = "bazel-df2c687c22bdd7c76f3cdcc85f38fefd02f0b844", urls = ["https://github.com/ixdy/bazel/archive/df2c687c22bdd7c76f3cdcc85f38fefd02f0b844.tar.gz"], )

I don't see an existing issue, so opening one for tracking. PR builds are currently failing due to the SHA of a remote archive changing. ... $ sha256sum 27a13726ff8621cfa4592fbef116b253043c2b093c8814265dc4489a0e67d229 cc cc cc
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/release: Issue will be escalated to these SIGs if needed. priority/critical-urgent: Never automatically move out of a release milestone; continually escalate to contributor and SIG through all available channels. kind/bug: Fixes a bug discovered during the current release. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details
cc
/assign We can probably just bump the sha, but I'm trying to figure out what changed...
I found a copy of the which matched the sha256sum (thanks There's no diff when I compare the extracted trees. There's also no diff if I compare the output on each. The new and old gzip'd tarballs are slightly different sizes (old bytes, new bytes). If I ungzip them, they are the same size ( bytes) but they still have different sha256sums. My best guess at this point is that there was some library change recently which is creating slightly different versions of these tarballs, but I don't know how it's different. I also don't know why the archive tarballs for other repos don't seem to have changed.
go 1.8 vs 1.9 gzip library maybe?
It seems like the order of some files in the tarball may have changed.
Yeah, we're not alone:
I guess we should start mirroring these archives somewhere rather than depending on GitHub.


Question: // AddWatchReactor appends a reactor to the end of the chain. func (c *Fake) AddWatchReactor(resource string, reaction WatchReactionFunc) { <ins> c.Lock() defer c.Unlock() </ins> c.WatchReactionChain = append(c.WatchReactionChain, &SimpleWatchReactor{resource, reaction}) } // PrependWatchReactor adds a reactor to the beginning of the chain. func (c *Fake) PrependWatchReactor(resource string, reaction WatchReactionFunc) { <ins> c.Lock() defer c.Unlock() </ins> c.WatchReactionChain = append([]WatchReactor{&SimpleWatchReactor{resource, reaction}}, c.WatchReactionChain...) }

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI -- ci-kubernetes-generate-make-test-count10 pr:pull-kubernetes-unit ci-kubernetes-unit ci-kubernetes-generate-make-test-count10 TestAttacherMountDeviceWithInline links to appreciated links to specific failures in spyglass appreciated <!-- Please see the deflaking doc () for more guidance! --
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /sig sig-storage </details
/sig storage


Question: return namespaces } <ins> func createClients(numberOfClients int) ([]*client.Client, error) { clients := make([]*client.Client, numberOfClients) for i := 0; i < numberOfClients; i++ { config, err := framework.LoadConfig() Expect(err).NotTo(HaveOccurred()) config.QPS = 100 config.Burst = 200 if framework.TestContext.KubeAPIContentType != "" { config.ContentType = framework.TestContext.KubeAPIContentType } // For the purpose of this test, we want to force that clients // do not share underlying transport (which is a default behavior // in Kubernetes). Thus, we are explicitly creating transport for // each client here. transportConfig, err := config.TransportConfig() if err != nil { return nil, err } tlsConfig, err := transport.TLSConfigFor(transportConfig) if err != nil { return nil, err } config.Transport = utilnet.SetTransportDefaults(&http.Transport{ Proxy:               http.ProxyFromEnvironment, TLSHandshakeTimeout: 10 * time.Second, TLSClientConfig:     tlsConfig, MaxIdleConnsPerHost: 100, Dial: (&net.Dialer{ Timeout:   30 * time.Second, KeepAlive: 30 * time.Second, }).Dial, }) // Overwrite TLS-related fields from config to avoid collision with // Transport field. config.TLSClientConfig = restclient.TLSClientConfig{} c, err := client.New(config) if err != nil { return nil, err } clients[i] = c } return clients, nil } </ins> func computeRCCounts(total int) (int, int, int) { // Small RCs owns ~0.5 of total number of pods, medium and big RCs ~0.25 each. // For example for 3000 pods (100 nodes, 30 pods per node) there are:

https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite} Previous issues for this test:
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
We understand the problem here - I'm working on the fix.
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}


Question: containerList, _ := runtimeService.ListContainers(nil) for _, c := range containerList { if _, exists := podSandboxMap[c.PodSandboxId]; !exists { <del> return nil, fmt.Errorf("no PodsandBox found with Id '%s' for container with ID '%s' and Name '%s'", c.PodSandboxId, c.Id, c.Metadata.Name) </del> <ins> klog.InfoS("no PodSandBox found for the container", "podSandboxId", c.PodSandboxId, "containerName", c.Metadata.Name, "containerId", c.Id) continue </ins> } containerMap.Add(podSandboxMap[c.PodSandboxId], c.Metadata.Name, c.Id) } <del> return containerMap, nil </del> <ins> return containerMap </ins> } func isProcessRunningInHost(pid int) (bool, error) {

What happened: with --cpu-manager-policy=static kubelet will panic when pause container lose during kubelet restarting What you expected to happen: kubelet can start normally How to reproduce it (as minimally and precisely as possible): 1,config kubelet with --cpu-manager-policy=static 2, Run a large number of pods on the test node 3,stop kubelet use command: 4,kill and remove all the pause container like this: ! 5,start kubelet use command: and then kubelet maybe panic, log is : Anything else we need to know?: When we actually encountered this problem, only one pause container was lost. In order to reproduce this bug, I deleted all pause containers. Environment: Kubernetes version (use ): 1.20.1 Cloud provider or hardware configuration: x8664 OS (e.g: ):18.04.5 LTS (Bionic Beaver) Kernel (e.g. ): Linux ecs-linux-0008-43ee 4.15.0-123-generic -Ubuntu SMP Wed Oct 21 09:40:11 UTC 2020 x8664 x8664 x8664 GNU/Linux Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig node /assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-contributor-experience at . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available .

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I think it makes sense to reopen this issue as the referenced PR is still open.
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-lifecycle rotten
/triage accepted
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
I am seeing this issue with a customer also. What is the timeline for this PR to be reviewed and merged?


Question: readOnly:     readOnly, mounter:      &mount.SafeFormatAndMount{Interface: mounter, Exec: exec}, deviceUtil:   util.NewDeviceHandler(util.NewIOHandler()), <del> mountOptions: []string{}, </del> <ins> mountOptions: util.MountOptionFromSpec(spec), </ins> }, nil } return &fcDiskMounter{

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Currently many in-tree storage plugins, including the iSCSI plugin, support mount options. The FC plugin does not. I have poked around the code enough to know where the plugin reports that mount options are not supported, but it isn't clear to me WHY they aren't supported. Why is this needed: Mount options can drastically change the performance and data persistence characteristics of a filesystem. For example, "noatime" can reduce metadata operations on frequently read data or "sync" can ensure all writes go immediately down to a backing storage array. I am working on a deployment across an FC backed cluster for which the the containerized application benefits from specific mount options, and I would guess other users might find similar use cases. It seems to me that if mount options are supportable on iSCSI, they should be equally supportable on FC, but I certainly recognize the possibility that I may be missing something. Otherwise, I'm guessing they simply haven't been a priority?
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/good-first-issue /help
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /good-first-issue /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Yeah, we just miss a volunteer that would implement this feature. It should be relatively easy and straightforward, following iscsi as an example.
I would like to work on it! As this my first issue I would like to get some help.
/assign sathish-thangaraj
GitHub didn't allow me to assign the following users: sathish-thangaraj. Note that only and repo collaborators can be assigned and that issues/PRs can only have 10 assignees at the same time.

For more information please see <detailsIn response to : /assign sathish-thangaraj Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
that's great! I was going to write detailed info how to fix this, however, I found out that mount options should work, at least in Kubernetes 1.13, see this commit: So there is a simple bug somewhere. Looking around the code, this code smells fishy: . Can you check with "BlockVolume" feature turned off in kubelet, i.e. ? Mount options should work there. And then it should be fairly easy to fix it in .
It definitely looks like there's an issue there. I don't think the kubelet ever made it that far in my initial testings, as mounting (with mountOptions listed in the PersistentVolume object) failed with the following message: I assume this is related to the following code: Is there a way to disable a check like this without rebuilding all of Kubernetes?
Good catch! No. IMO it should be enough to rebuild kubelet here and it's fast and easy: and find it somewhere in .
Thanks for the help so far I flipped the SupportsMountOption() bool to "true" and the BlockVolume feature gate to false. I was then able to mount an ext4 volume with the "noatime" and "nodiratime" option (confirmed on the worker node). However, I attempted to use the "nobarrier" option but the worker node mounted the volume without it. Finally, I attempted to use a completely made-up option ("hello"). The volume did not fail to mount (as is suggested in the Kubernetes documentation. Instead, it mounted without the "hello" options. I am attempting to locate useful logs to try to understand why some mount options are ignored or why incorrect mount options aren't causing failures, but the kubelet logs haven't been helpful thus far. Unfortunately I don't currently have any easy access to any iSCSI equipment for comparison.
anyone working on this?
is there anywhere that I could help on the issue?
First time contributor here - I just changed 2 lines of code to make FC similar to other volume types (eg. iSCSI).

I'm not sure about the mount option validation that mentions. Should that be a separate issue?
Apologies for the extreme delay in keeping up with this issue. I believe the issue with ignored mount options may be more generally related to the way the Linux mounter (/pkg/utils/mount/mount-) handles mount options, although I don't have enough experience with other volume plugins to speak authoritatively on the issue. I recommend we try to get the basic changes suggested above into the code, as my limited testing shows them to be effective for at least some mount options. I'd try to submit a pull request myself, but I don't want to take it from as they've already put forth the work to get it started.
To clarify my previous comment, during Attach, the FC volume plugin mounts an FC volume to a global path with no mount options. During Mount, the Linux mounter bind mounts (-o bind) and bind remounts (-o bind,remount) an FC volume to a Pod specific path with mount options. However, the bind remount action is only capable of applying a limited number of mount options (those not on the list are silently ignored). In order to expand our mount option capabilities, we would (I believe) need to use mount options in the Attach phase.
First time contributor, taking this one /assign
you are, of course, welcome to work on this issue. However, I wanted to let you know that I made code changes back in November that (I believe) address my previous comments. I only received approval from my employer in the last couple of days to make the contributions, or I would have submitted them already. I plan to clean things up and get a PR in sometime this afternoon. I am also a first time contributor, so we'll see how it goes. /assign
No problem at all dude. You take this one since you already made the changes. On Thu, 23 Jan 2020 at 18:47 Eric Weber <:


Question: metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime" "k8s.io/apiserver/pkg/authentication/user" <ins> utilfeature "k8s.io/apiserver/pkg/util/feature" </ins> rbac "k8s.io/kubernetes/pkg/apis/rbac" <ins> "k8s.io/kubernetes/pkg/features" </ins> ) var (

/kind feature From: Would it make sense to add these rules to the bootstrappolicy? To me they seem generic enough, and they wouldn't bind to something. Consumers can do the binding to their subjects as they like. I'm asking because it's already duplicated twice for two early adopters: I don't think those ClusterRoles are specific to kubeadm and GKE at all; but the ClusterRoleBindings will surely be, so therefore we'd create the roles but not the bindings.
Wouldn't we expect and encourage people using the CSR capability for their nodes to use node identities that match the node authorizer, in which case we won't need the client or server renewal CSR roles?
The node authorizer doesn't grant these, since otherwise you could never prevent a compromised node from renewing its credentials
Realistically, with these roles you still wouldn't. People would grant them to the node group, not each individual node, but I guess that would be a tradeoff for the deployment mechanism.
You could individually revoke this role from the group and stop renewals without disrupting regular operation
I can definitely see myself revoking a binding from to multiple specific bindings in case of a compromise. Anyway, this issue is more "should we create these by default or not". I've already mentioned my opinion.
Creating the roles seems reasonable. I don't think I want bindings. I think the binding should be up the particular deployer.
Perfect, that's exactly my proposal. I'll go ahead and make this change...
/assign
are you working on this or should I send a PR?


Question: - endpoint: replaceNetworkingV1NamespacedNetworkPolicyStatus reason: endpoints is currently feature gated and and will only receive e2e & conformance test in 1.25 link: https://github.com/kubernetes/kubernetes/pull/107963 <ins> - endpoint: readCoreV1NodeStatus reason: Kubernetes distribution would reasonably not allow this action via the API link: https://github.com/kubernetes/kubernetes/issues/109379 - endpoint: replaceCoreV1NodeStatus reason: Kubernetes distribution would reasonably not allow this action via the API link: https://github.com/kubernetes/kubernetes/issues/109379 - endpoint: deleteCoreV1CollectionNode reason: Kubernetes distribution would reasonably not allow this action via the API link: https://github.com/kubernetes/kubernetes/issues/109379 </ins>

In the K8s Conformance meeting of 6 April the testing of Node endpoints and their eligibility for conformance testing was discussed. This issue is created to get more community input on the topic. Should node endpoints be tested? Can we accualy test these endpoints with the current test framework /sig testing /sig architecture /area conformance
/triage accepted /cc
I have lost track of this issue where it have already been discussed. Let's continue to discuss further here. Apologies for the confusion /cc
Clayton We have historically focused conformance on the perceived end user experience deploying applications to a conformant Kubernetes cluster as being consistently successful. From that perspective, while these are actions that are indeed key to conformant behavior on the majority of distributions, the first two are not invoked by end users, only by kubelets and potentially administrators in rare cases. I would expect to be able to create a node object, delete it, mutate its status, and delete a portion of them on a conformant Kubernetes cluster as an admin - but I would not expect a similar end user to do so. If we can make a case that a conformant Kubernetes distribution would reasonably not allow any of these actions via the API, then we could make a case to omit them from conformance. I can see a small path to adding these to conformance under a "they may be necessary as administrative action on the vast majority of kubernetes clusters", but I can't necessarily bump that to 100% of Kube clusters, especially if we put the application deployer as the center of our story (an application deployer does not need to delete node collections in order to succeed at running an application on conformant kubernetes)
Addressed the question in SIG Node
Created to make the remaining node endpoints ineligible. readCoreV1NodeStatus replaceCoreV1NodeStatus deleteCoreV1CollectionNode


Question: // perform any validation. // According to our measurements this is significantly faster // in codepaths that matter at high scale. <ins> // Note: this method copies the Set; if the Set is immutable, consider wrapping it with ValidatedSetSelector // instead, which does not copy. </ins> func (ls Set) AsSelectorPreValidated() Selector { return SelectorFromValidatedSet(ls) }

A common pattern in Kubernetes (both internally and in third party usage) is something like: (this is from the ). This is commonly called at a fairly high rate; in our case we would actually end up seeing this account for significant portions of CPU/memory to compute the and run on it. In our controller, as an optimization, we replaced usage of and just used our own simpler label matching code, leading to a 70% speed up in our function (which does more than just match) and 96% allocation size improvement; see I believe similar could be done in Kubernetes directly by having a specialized Selector implementation for simple labels. This would transparently given substantial performance benefits to anyone using the library
/sig api-machinery
is the change here a new kind of selector matching or a tweak to something existing? It's hard to tell if this is proposing a new design to partition types of selectors that needs a kep versus a perf tweak to existing code that is straight-forward.
The proposal is an internal performance improvement to make a specialized implementation of the selector matching internally. So two concrete implementations for 1 shared interface. is a PoC of this. I think it can be extended (by me, or others if someone wants to) to complete it
We discussed this one during our bug triage. There seemed to be the feeling that this is large enough to need a KEP? Would you be up for that? Additionally you could bring it up in our SIG meeting to socialize / discuss it more if you wanted.
The patch shared in helps to explain, it's a lot smaller than the description made it sound. This is a special case for equality-only selectors. Probably still worth bringing up at a sig-meeting, but if if the patch is a good representation, it is more bounded than expected from the issue description.
/triage accepted


Question: nodeMaxAllocatable = allocatable.MilliValue() } } <del> framework.WaitForStableCluster(cs, masterNodes) </del> <ins> // Clean up added labels after this test. defer func() { for nodeName := range nodeToAllocatableMap { framework.RemoveLabelOffNode(cs, nodeName, "node") } }() </ins> pods, err := cs.CoreV1().Pods(metav1.NamespaceAll).List(metav1.ListOptions{}) framework.ExpectNoError(err)

cc cc Seeing a failure on our downgrade test. Can someone please look at this? https://k8s- Example failure https://k8s- Seeing timeouts on the downgrade test
Since we're so late in the release process and this is the first time we're seeing this fail, going to mark as non-release-blocking for now.
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/scheduling: Issue will be escalated to these SIGs if needed. priority/critical-urgent: Never automatically move out of a release milestone; continually escalate to contributor and SIG through all available channels. kind/bug: Fixes a bug discovered during the current release. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details
It fails in more places than just the downgrade job, but when the downgrade job fails that test, that timeout appears to be the consistent error that it fails with
/assign I looked at the logs. The test creates a few pods to saturate the existing nodes and then creates an additional pod and ensures that this last pod stays pending. The problem we observe is that those saturating pods are often not scheduled, causing a test timeout. Those pods use all the resources on the nodes. So, any small changes to the amount of available node resources or even rounding errors could cause those saturating pods to not schedule. I will test this theory by cutting down the amount of resources that the saturating pods use, but I would like to check if the problem is fixed before submitting the PR. Where can I get the steps these cluster downgrade tests take?
The test setups are in the test-infra repo. E.g.
see my analysis in .


Question: { Key:      nodeAffinityKey, Operator: v1.NodeSelectorOpIn, <del> Values:   []string{string(i)}, </del> <ins> Values:   []string{fmt.Sprintf("%v", i)}, </ins> }, }, },

What keywords did you search in Kubernetes issues before filing this one? scheduler_perf. Is this a BUG REPORT or FEATURE REQUEST? Bug Kubernetes version (use ): Master Looks like perf test is broke. I'll dig into the details. Leads to Will add more details later or issue a PR if its something trivial.
i think your affinity cleanup patch, might be the source :) .... [ deleted above comment not relevant ] some other change, possibly in the affinity cleanup may have caused this, but i dont think its the stuff, that looks ok to me. Possibly just a typo might have been introduced in that patch? If not maybe its just an issue with the way we need to update the labelling json or something. I can take another look in the morning.
ok, i think i see... it wants to match the label ... from this node: into this set so the set is empty. I guess it could be a bug in the way the match selector is being produced, but the code looks clean to me. will double check.
fixed


Question: } t.Logf("Found ExternalID(%s) = %sn", srvs[0], externalId) <ins> nonExistingVM := rand.String(15) externalId, err = i.ExternalID(nonExistingVM) if err == cloudprovider.InstanceNotFound { t.Logf("VM %s was not found as expectedn", nonExistingVM) } else if err == nil { t.Fatalf("Instances.ExternalID did not fail as expected, VM %s was found", nonExistingVM) } else { t.Fatalf("Instances.ExternalID did not fail as expected, err: %v", err) } </ins> instanceId, err := i.InstanceID(srvs[0]) if err != nil { t.Fatalf("Instances.InstanceID(%s) failed: %s", srvs[0], err) } t.Logf("Found InstanceID(%s) = %sn", srvs[0], instanceId) <ins> instanceId, err = i.InstanceID(nonExistingVM) if err == cloudprovider.InstanceNotFound { t.Logf("VM %s was not found as expectedn", nonExistingVM) } else if err == nil { t.Fatalf("Instances.InstanceID did not fail as expected, VM %s was found", nonExistingVM) } else { t.Fatalf("Instances.InstanceID did not fail as expected, err: %v", err) } </ins> addrs, err := i.NodeAddresses(srvs[0]) if err != nil { t.Fatalf("Instances.NodeAddresses(%s) failed: %s", srvs[0], err)

ExternalID/InstanceID do not return 'cloudprovider.InstanceNotFound' error when the VM is not found. This can occur when a node VM is deleted entirely. In this case InstanceID method would throw an nil pointer exception here: since returned error here: is not being handled, vm would be nil. vm object is then dereferenced in getVirtualMachineManagedObjectReference func. In ExternalID method, the error is being handled by just passing it up. Node controller then prints out error and continues: e.g. ExternalID/InstanceID should handle the 'nod found' error being returned from vCenter and return 'cloudprovider.InstanceNotFound'.


Question: func TestPatchPodStatus(t *testing.T) { tests := []struct { <del> name           string pod            v1.Pod </del> <ins> name   string pod    v1.Pod client *clientsetfake.Clientset // validateErr checks if error returned from PatchPodStatus is expected one or not. // (true means error is expected one.) validateErr    func(goterr error) bool </ins> statusToUpdate v1.PodStatus }{ { <del> name: "Should update pod conditions successfully", </del> <ins> name:   "Should update pod conditions successfully", client: clientsetfake.NewSimpleClientset(), </ins> pod: v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Namespace: "ns",

If there is a connection failure when updating the Pod status, we don't retry. Furthermore, due to , we won't have another chance until much later. The problem is that if we don't mark the Pod as Unschedulable, other controllers (such as cluster-autoscaler) wouldn't react to these pods. We should have stronger guarantees to mark a Pod as Unschedulable. You need an unschedulable Pod (for example, a Pod with a non matching node affinity) and a flaky connection to apiserver. No response master This is particularly problematic in 1.24, due to . Any <details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scheduling
That is bad, retrying in place on transient errors should probably be ok.
I'll address it. /assign
Thanks!


Question: if err != nil { t.Errorf("Unxpected error when syncing jobs, got %v", err) } <del> ch := make(chan interface{}) go func() { item, _ := manager.queue.Get() ch <- item }() select { case key := <-ch: expectedKey := getKey(job, t) if key != expectedKey { t.Errorf("Expected requeue of job with key %s got %s", expectedKey, key) } case <-time.After(controllerTimeout): manager.queue.ShutDown() t.Errorf("Expected to find a job in the queue, found none.") </del> <ins> t.Log("Waiting for a job in the queue") key, _ := manager.queue.Get() expectedKey := getKey(job, t) if key != expectedKey { t.Errorf("Expected requeue of job with key %s got %s", expectedKey, key) </ins> } }

Ran a test executable with more than 20,000 times and no failures. Any hints as to how to get it to show up? I'm running 100 jobs at a time, so my system is stressed already.
I to am unable to reproduce this. I get similar errors if I reduce the timeout at the top of the file. I spent a bit looking at how to rewrite the tests without the timeout, but nothing jumped out at me. Looking at it though, it's really just a 500ms timeout for the test to complete. I'm just going to remove them. Go has a default timeout of 10 minutes. If it's going to pass, it will pass then. If not, Go will give you a nice traceback where it deadlocked.


Question: if len(list.Items) > 1 { return cmdutil.UsageError(cmd, "%s specifies multiple items", filename) } <ins> if len(list.Items) == 0 { return cmdutil.UsageError(cmd, "please make sure %s exists and is not empty", filename) } </ins> obj = list.Items[0] } newRc, ok = obj.(*api.ReplicationController)

For example
This is with recent kubectl:
I think this is a duplicate, but just in case.
I will double check this.
Thanks - it isn't a big deal, because I obviously didn't intend to rolling update to an empty manifest :-) (I was experimenting with templating my manifests and made an error) But let me know if you need any more info to reproduce!


Question: // need more than one per sidecar and kubelet. Keeping this reasonably // small ensures that we don't establish connections through the apiserver // and the remote kernel which then aren't needed. <ins> // // The proxy code below establishes this many connections in advance, // without waiting for a client on the remote side. On the local side // a gRPC server will accept the same number of connections and then wait // for data from a future client. // // This approach has the advantage that a client on the remote side can // immediately start communicating, without the delay caused by establishing // the connection. That delay is large enough that clients like the // node-driver-registrar with a very small timeout for gRPC did indeed // time out unnecessarily. </ins> const maxConcurrentConnections = 10 <ins> // This delay determines how quickly we notice when someone has // connected inside the cluster. With socat, we cannot make this too small // because otherwise we get many rejected connections. With the mock // driver as proxy that doesn't happen as long as we don't // ask for too many concurrent connections because the mock driver // keeps the listening port open at all times and the Linux // kernel automatically accepts our connection requests. const connectionPollInterval = 100 * time.Millisecond </ins> // Listen creates a listener which returns new connections whenever someone connects // to a socat or mock driver proxy instance running inside the given pod. //

ci-kubernetes-csi-canary-test-on-kubernetes-master, [It] [sig-storage] CSI mock volume CSI Volume Snapshots [Feature:VolumeSnapshotDataSource] volumesnapshotcontent and pvc in Bound state with deletion timestamp set should not get deleted while snapshot finalizer exists [It] [sig-storage] CSI mock volume CSI Volume Snapshots secrets [Feature:VolumeSnapshotDataSource] volume snapshot create/delete with secrets 25-08-2022? No response Looking at "volume snapshot create/delete with secrets" test, it finishes OK, but the "Cleaning up resources" freezes for ~30 minutes: After these 30 minutes, the test timeouts and kills API server and etcd a and everything fails miserably. Stack trace from "Cleaning up resources", it's stuck at shutting down gRPC server: /sig storage
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I reproduced this issue in somewhat hacked from external-provisioner (or any other kubernetes-csi repo). The script compiles Kubernetes master from a fresh checkout, runs it in KIND, installs hostpath CSI driver with canary images and runs e2e tests for the CSI driver like this: "hostpath CSI driver with canary images" is the interesting part - when I run the test with the latest tagged images, it succeeds, when using canaries it freezes at the cleanup. I found out that using the latest tagged images for everything except is enough to freeze the cleanup. And further I was able to check that node-driver-registrar at commit works, while master does not. There are just 2 commits in between: new release-tools and update to v1.25 libraries. I suspect it's the new libraries. I still don't understand how node-driver-registrar and its libraries could affect gRPC between the hostpath CSI driver and binary.
does this ring a bell? I though node-driver-registrar does not do anything with proxied "mock" CSI calls between and the CSI driver.

In addition, why only these two tests fail reliably, while the rest works?
For the record, this is diff in k/k that adds canary node-driver-registrar to This diff is enough to make the tests fail:
This does not ring a bell at all. You are right that csi-node-driver-registrar should not be involved in the gRPC communication between and the driver. Does it communicate with the driver after the initial registration? If yes, then that will lead into connections that need to be proxied, which might prevent the shutdown. Just a wild guess. But even then it's not clear how grpc.Server.Stop can get stuck. According to its documentation, "It immediately closes all open connections and listeners.". Hmm, could the proxy code get stuck when its listener is closed?
There is in that gets stuck, I did not check what is in that group.
Well, of course it fails :sweat_smile: There is no such canary image. Hmm, but then why do other tests pass? They all use .
There is some magic in your script that replaces with , the registrar runs and registers the driver in the end.
Yep:
One difference is that the driver-registrary canary no longer detects the loss of connection to the mock proxy. Previously it did: It could be that the proxy code relies on the remote side to close their side of the connection before considering everything shut down. That is necessary because the remote side might still send some data that needs to be received. If the remote side keeps the connection open, everything freezes. One can argue that it shouldn't block forever, but the bigger question is why "Lost connection" is not detected anymore with the more recent gRPC.
The "exit on connection loss" depends on our dialer being called again by gRPC:
/assign No idea yet, but I'll keep looking...
I can reproduce the same failure to shut down also in other tests. Must be a timing issue that it only occurred for the same two tests in the CI - not a very satisfying explanation, unfortunately. I have something that works for me locally: closing connections when shutting down the listener -see


Question: "still active, is time, suspended":         {A, T, onTheHour, noDead, T, T, justAfterTheHour(), F, F, 1}, "still active, is time, past deadline":     {A, F, onTheHour, shortDead, T, T, justAfterTheHour(), F, F, 1}, "still active, is time, not past deadline": {A, F, onTheHour, longDead, T, T, justAfterTheHour(), T, F, 2}, <ins> // Controller should fail to schedule these, as there are too many missed starting times // and either no deadline or a too long deadline. "prev ran but done, long overdue, not past deadline, A": {A, F, onTheHour, longDead, T, F, weekAfterTheHour(), F, F, 0}, "prev ran but done, long overdue, not past deadline, R": {R, F, onTheHour, longDead, T, F, weekAfterTheHour(), F, F, 0}, "prev ran but done, long overdue, not past deadline, F": {f, F, onTheHour, longDead, T, F, weekAfterTheHour(), F, F, 0}, "prev ran but done, long overdue, no deadline, A":       {A, F, onTheHour, noDead, T, F, weekAfterTheHour(), F, F, 0}, "prev ran but done, long overdue, no deadline, R":       {R, F, onTheHour, noDead, T, F, weekAfterTheHour(), F, F, 0}, "prev ran but done, long overdue, no deadline, F":       {f, F, onTheHour, noDead, T, F, weekAfterTheHour(), F, F, 0}, "prev ran but done, long overdue, past medium deadline, A": {A, F, onTheHour, mediumDead, T, F, weekAfterTheHour(), T, F, 1}, "prev ran but done, long overdue, past short deadline, A":  {A, F, onTheHour, shortDead, T, F, weekAfterTheHour(), T, F, 1}, "prev ran but done, long overdue, past medium deadline, R": {R, F, onTheHour, mediumDead, T, F, weekAfterTheHour(), T, F, 1}, "prev ran but done, long overdue, past short deadline, R":  {R, F, onTheHour, shortDead, T, F, weekAfterTheHour(), T, F, 1}, "prev ran but done, long overdue, past medium deadline, F": {f, F, onTheHour, mediumDead, T, F, weekAfterTheHour(), T, F, 1}, "prev ran but done, long overdue, past short deadline, F":  {f, F, onTheHour, shortDead, T, F, weekAfterTheHour(), T, F, 1}, </ins> } for name, tc := range testCases { sj := cronJob()

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): No What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): , , Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Environment: Cloud provider or hardware configuration: bare metal OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: , deployed with Others: What happened: I am using a scheduledjob with . An instance of my job has taken a long time to finish, thus skipping many start times. Ever since, the scheduledjob will not schedule anything new. Controller manager log has: What you expected to happen: I expect the scheduledjob not to be stuck not scheduling anything anymore. With , I expect that it should just re-schedule something as soon as the previously running job has completed. The "Too many missed start times to list" safeguard appears to apply mostly to cases where job concurrency is allowed to avoid creating a lot of jobs in case of timestamp issues. ` How to reproduce it (as minimally and precisely as possible): The issue shows up after about 100 minutes of this running. Anything else do we need to know:
cc
sorry for the long delay in response, there were a few issues solved in later versions of ScheduledJob, can you verify this against latest 1.4.x? I've tried cherry-picking all of the crucial ones, like: and I'll try to verify this on my own as well.

I am still getting the same issue with
thanks for the info, will dig deeper into this one.
Thanks a lot any chance a potential fix could make it in 1.5?
if possible - yes, I'll try to backport it to 1.5
did you look into this? If not, let me know if you want me to pick it up and work on a PR
yeah, go ahead and ping me in the PR.
I just thought of something which might make sense. We've modified the error message to make it more "actionable", and the user can set a deadline to avoid running into this. However, the error message is still only controller level AFAIK and thus not exposed to the user. From the user's perspective, there is not way to tell why the CronJob is not scheduling the next instance. For these reasons, maybe we should also emit an event in the corresponding CronJob? I can send a small PR for this modification.


Question: return configCmd } <del> func generateSecureToken(tokenLen int) (string, error) { </del> <ins> // GenerateSecureToken returns a string of length tokenLen, consisting // of random bytes encoded as base64 for use as a Bearer Token during // communication with an APIServer func GenerateSecureToken(tokenLen int) (string, error) { </ins> // Number of bytes to be tokenLen when base64 encoded. tokenSize := math.Ceil(float64(tokenLen) * 6 / 8) rawToken := make([]byte, int(tokenSize))

Which jobs are failing: Which test(s) are failing: The tests are failing in the . Since when has it been failing: 10/23/2020 Testgrid link: Reason for failure: It started failing at commit , but it's unknown if it's related to that. Anything else we need to know: We discussed in the Kubernetes SIG-Node CI subgroup and will start taking a look at it.
/assign
/sig node
Investigation notes: between successful and unsuccessful runs and found no notable differences. , and had a couple of failed health checks on initialization, which doesn't happen in the successful test runs, though there's only 2 seconds between starting readiness checks and an eventual successful check. config the same between runs looking through the kubelet logs for and , but I haven't found anything interesting yet. kubelet- failed health checks The below is a list of errors that occur in runs that succeeded and failed, which may be due to startup or specific testing scenarios. Bunch of these for numerous containers:
Both new and old test runs have a bunch of logs like this, implying that it's been having issues for a while: The failed one has this additional line: However, the doesn't have the same issues. I don't see the connection timeouts at all. I also don't see anything else in the kubelet logs that appears different between successful and failed runs. Unfortunately, I don't know enough about the infrastructure setup, but here are my current theories: about the infrastructure has gotten progressively more overloaded and now it's falling over. The tests run on 8 nodes versus 1, so they have more capacity there. Not totally buying this because why did it go from mostly successful to 100% failures rather than a mix for a bit. changed in the GCE configuration. Not sure how that is managed.
Diff there suggests three merges from k/k Only merge in test-infra is an image bump <img width="215" alt="image" src="https://user-"Interesting that a similar ci version and pull version of the job passes and the one ci version does not. pull: passing ci: failing ci: The failing ones do not have a node_e2e , but I don't see any of the expected log messages for failure of the services to start.
It is interesting that one passes and one does not.

Primary differences are the number of nodes, resource settings, and the fact that the conformance tests use Docker.
In addition to the services log missing, the overall system run logs are also missing. May be related to any decision we make for
Would we be able to kick off a Prow job with custom configuration? In other words, to rule in or rule out something in the image change, can we kick off a job with the latest everything else but use the older image? In reference to the decision on , are you suggesting we pause investigation and see if we decide to just move or change the conformance tests?
/triage accepted /priority critical-urgent
Catching up on this. qq: has anyone vetted ? Ise that pr went in around the time this job began to fail The beforesuite (AI: we should list out what it does), fails to communicate with nodes the mention of 6443 ports looks interesting here.
Good find. Definitely looks suspicious. I'll dig into that.
From : This comes from , which has some node-conformance specific test setup, so maybe something interesting there.
quick interruption for instructions on how to reproduce this one: I think this may work Focus and skip The FOCUS and SKIP arguments im getting from Image config file The job also uses the following config file Make sure to copy it and name it (the argument to IMAGECONFIG_FILE above). Additional arguments The test args to this job are I think these are the defaults already but just in case may be actually useful to add the following to the above command Miscellaneous If you run into trouble with the above commands try going through The info here came from
You'll also need to use the flag to make sure it runs in Docker, I believe.
ah yeah, good catch! Thank you. Don't know about your availability, but if you are all busy ima allocate sometime later today to dive deeper into the info we have here
I'm going to try to spend time of this in the afternoon (PDT). Specifically, I was going to try to verify that the logs from starting up the test suite look as expected and possibly try to run the case to reproduce.

My first suspicion was to have the server running in another port than 6443 (which Kubelet has configured), this is not the case: Only o.SecureServing.BindAddress is being set explicitly, but the port comes from o.NewServerRunOptions().
I think it's related to the movement of this line. I couldn't figure out how to link to the PR so I made a comment.
as you notice before there's no , and we should be seeing Before:
I can repro the issue now, and I think the line was moved upward to generate the token before starting the kubelet. Moving it up out of the if-statement may help partially, but in the conformance test the kubelet is manually started ahead of this code, so it may not matter.
Make sense. Do you mean moving this out of the conformance check?
Haha, yea, that message is deceptive... I think I'll make a PR to at least let it be more correct and give us more info.
for alerts to subscribers, PR created. Please comment.
looking green!
The output looks very strange, I have no idea why. Output has double-underscore-number for the different runs, instead of the image used. [__08] vs [cos-lts]
Ya, I noticed that. It's weird. Not sure how it's set, but you can see that the junit config names have a similar issue: -junitcos- -
Ok, found the root of the problem. Here's the command for the master suite: Note , which is not present in the conformance tests. but not sure how it eventually gets here.


Question: // UnmountDevice is part of the volume.Detacher interface. func (d *flexVolumeDetacher) UnmountDevice(deviceMountPath string) error { <del> if pathExists, pathErr := util.PathExists(deviceMountPath); pathErr != nil { return fmt.Errorf("Error checking if path exists: %v", pathErr) } else if !pathExists { </del> <ins> pathExists, pathErr := util.PathExists(deviceMountPath) if !pathExists { </ins> glog.Warningf("Warning: Unmount skipped because path does not exist: %v", deviceMountPath) return nil } <ins> if pathErr != nil && !util.IsCorruptedMnt(pathErr) { return fmt.Errorf("Error checking path: %v", pathErr) } </ins> notmnt, err := isNotMounted(d.plugin.host.GetMounter(d.plugin.GetPluginName()), deviceMountPath) if err != nil { <del> return err </del> <ins> if util.IsCorruptedMnt(err) { notmnt = false // Corrupted error is assumed to be mounted. } else { return err } </ins> } if notmnt { glog.Warningf("Warning: Path: %v already unmounted", deviceMountPath) } else {

/kind bug What happened: Kubelet failed to unmount volumes if the backend ISCSI volume is not available. This also applies to other network attached volumes. What you expected to happen: Kubelet should be able to unmount the volumes. A manual unmount works even when the backend network volume is not available. How to reproduce it (as minimally and precisely as possible): Mount an ISCSI/network volume and disconnect/de-provision ISCSI target. Anything else we need to know?: XFS shuts down the filesystem due to I/O errors. Kubelet uses "stat" calls to figure out if it is a valid mount point or it is already unmounted. Stat calls fail with EIO and Kubelet never proceeds to unmount the volume. Environment: Kubernetes version (use ): Kubernetes 1.10.5 Cloud provider or hardware configuration: Bare metal OS (e.g. from /etc/os-release): Centos 7.4 Kernel (e.g. ): Linux appserv37 3.10.0-693.el7.x8664 SMP Tue Aug 22 21:09:27 UTC 2017 x8664 x8664 x8664 GNU/Linux Install tools: Manual bring up Others:
/sig storage
IsLikelyNotMountPoint is using stat call to figure the filesystem is mounted or not. Stat calls are failing with EIO when there are I/O errors on the backend and Kubelet never proceeds to unmount the filesystem. Also, isLikelyNotMountPoint is not looking for "isCorruptedMnt" errors. All other calls have it.
looks like the nfs plugin is calling . Can iscsi use that?
didn't include error "Input/output error" () We should add it. I make a fix, PTAL, Thank you!
Thanks. Trying out the change. Will keep you posted if we need any other changes.
Stock ISCSI driver is working fine with EIO change. But more changes are required for making it work with Flexvolume drivers. I have the change list. I can club your change in or let me know if you want to go ahead and push this fix. I will push another PR to fix flexvolume issues.
Please club my change, I will close my PR. Thanks!


Question: <ins> /* Copyright 2018 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package admissionwebhook import ( "fmt" "testing" "time" admissionregistrationv1beta1 "k8s.io/api/admissionregistration/v1beta1" appsv1 "k8s.io/api/apps/v1" corev1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/client-go/kubernetes" kubeapiservertesting "k8s.io/kubernetes/cmd/kube-apiserver/app/testing" "k8s.io/kubernetes/test/integration/framework" ) var ( brokenWebhookName    = "integration-broken-webhook-test-webhook-config" deploymentNamePrefix = "integration-broken-webhook-test-deployment" ) func TestBrokenWebhook(t *testing.T) { var tearDownFn kubeapiservertesting.TearDownFunc defer func() { if tearDownFn != nil { tearDownFn() } }() etcdConfig := framework.SharedEtcd() server := kubeapiservertesting.StartTestServerOrDie(t, nil, nil, etcdConfig) tearDownFn = server.TearDownFn client, err := kubernetes.NewForConfig(server.ClientConfig) if err != nil { t.Fatalf("unexpected error: %v", err) } t.Logf("Creating Deployment to ensure apiserver is functional") _, err = client.AppsV1().Deployments("default").Create(exampleDeployment(generateDeploymentName(0))) if err != nil { t.Fatalf("Failed to create deployment: %v", err) } t.Logf("Creating Broken Webhook that will block all operations on all objects") _, err = client.AdmissionregistrationV1beta1().ValidatingWebhookConfigurations().Create(brokenWebhookConfig(brokenWebhookName)) if err != nil { t.Fatalf("Failed to register broken webhook: %v", err) } // There is no guarantee on how long it takes the apiserver to honor the configuration and there is // no API to determine if the configuration is being honored, so we will just wait 10s, which is long enough // in most cases. time.Sleep(10 * time.Second) // test whether the webhook blocks requests t.Logf("Attempt to create Deployment which should fail due to the webhook") _, err = client.AppsV1().Deployments("default").Create(exampleDeployment(generateDeploymentName(1))) if err == nil { t.Fatalf("Expected the broken webhook to cause creating a deployment to fail, but it succeeded.") } t.Logf("Restarting apiserver") tearDownFn = nil server.TearDownFn() server = kubeapiservertesting.StartTestServerOrDie(t, nil, nil, etcdConfig) tearDownFn = server.TearDownFn client, err = kubernetes.NewForConfig(server.ClientConfig) if err != nil { t.Fatalf("unexpected error: %v", err) } // test whether the webhook still blocks requests after restarting t.Logf("Attempt again to create Deployment which should fail due to the webhook") _, err = client.AppsV1().Deployments("default").Create(exampleDeployment(generateDeploymentName(2))) if err == nil { t.Fatalf("Expected the broken webhook to cause creating a deployment to fail, but it succeeded.") } t.Logf("Deleting the broken webhook to fix the cluster") err = client.AdmissionregistrationV1beta1().ValidatingWebhookConfigurations().Delete(brokenWebhookName, nil) if err != nil { t.Fatalf("Failed to delete broken webhook: %v", err) } // The webhook deletion is honored in 10s. time.Sleep(10 * time.Second) // test if the deleted webhook no longer blocks requests t.Logf("Creating Deployment to ensure webhook is deleted") _, err = client.AppsV1().Deployments("default").Create(exampleDeployment(generateDeploymentName(3))) if err != nil { t.Fatalf("Failed to create deployment: %v", err) } } func generateDeploymentName(suffix int) string { return fmt.Sprintf("%v-%v", deploymentNamePrefix, suffix) } func exampleDeployment(name string) *appsv1.Deployment { var replicas int32 = 1 return &appsv1.Deployment{ TypeMeta: metav1.TypeMeta{ Kind:       "Deployment", APIVersion: "apps/v1", }, ObjectMeta: metav1.ObjectMeta{ Namespace: "default", Name:      name, }, Spec: appsv1.DeploymentSpec{ Replicas: &replicas, Selector: &metav1.LabelSelector{ MatchLabels: map[string]string{"foo": "bar"}, }, Template: corev1.PodTemplateSpec{ ObjectMeta: metav1.ObjectMeta{ Labels: map[string]string{"foo": "bar"}, }, Spec: corev1.PodSpec{ Containers: []corev1.Container{ { Name:  "foo", Image: "foo", }, }, }, }, }, } } func brokenWebhookConfig(name string) *admissionregistrationv1beta1.ValidatingWebhookConfiguration { var path string var failurePolicy admissionregistrationv1beta1.FailurePolicyType = admissionregistrationv1beta1.Fail return &admissionregistrationv1beta1.ValidatingWebhookConfiguration{ ObjectMeta: metav1.ObjectMeta{ Name: name, }, Webhooks: []admissionregistrationv1beta1.Webhook{ { Name: "broken-webhook.k8s.io", Rules: []admissionregistrationv1beta1.RuleWithOperations{{ Operations: []admissionregistrationv1beta1.OperationType{admissionregistrationv1beta1.OperationAll}, Rule: admissionregistrationv1beta1.Rule{ APIGroups:   []string{"*"}, APIVersions: []string{"*"}, Resources:   []string{"*/*"}, }, }}, // This client config references a non existent service // so it should always fail. ClientConfig: admissionregistrationv1beta1.WebhookClientConfig{ Service: &admissionregistrationv1beta1.ServiceReference{ Namespace: "default", Name:      "invalid-webhook-service", Path:      &path, }, CABundle: nil, }, FailurePolicy: &failurePolicy, }, }, } } </ins>

Please write a test that: Starts kube-apiserver Adds a webhook which blocks everything it is possible to block. Re-starts kube-apiserver Verifies that kube-apiserver comes back up Verifies that it can remove the broken webhook. This would have caught
cc
Hi I'm new to kubernetes but eager to contribute. I think this will be a good first issue for me but I'll need some guide on how to proceed I will need the first two points explained and also which file I'm supposed to be making my changes to.
Here is a good example of how to start a realistic kube-apiserver in integration tests:
"Verifies that it can remove the broken webhook." sounds dangerous. The apiserver should come up and it should be possible to remove a webhook via the API. But removing a potentially broken (or intentionally DoS'ed) webhook asks for security attacks.
I interpreted "Verifies that it can remove the broken webhook" as just ensuring that no webhook can be registered which could potentially deny its own (or another webhook in the same namespace's) deletion. Broken webhooks definitely shouldn't be treated differently in terms of determining who can delete them, but validation should probably prevent webhooks from applying to creates, updates and deletes of webhook objects, wildcard selectors should exclude webhook objects, etc. Also, this behavior would need to be documented. I think the safest way to ensure that a webhook does not get deleted is with RBAC rules, rather than using a validating webhook to prevent its deletion. I can't think of a legitimate use case for webhooks which affect the admission of webhook objects, it seems like any workflow which relies on that could be implemented in another, easier to reason about way. thoughts?
I have the same understanding. Excluding webhooks from webhooks seems reasonable. But it's only one piece in the (restart when webhook broke) puzzle. We reconcile RBAC rules, create namespaces (or keep them when they exist already). So I expect some more exceptions.
Is this still something you are working on?
If you can make a webhook which prevents the apiserver from starting at all, this is a problem, no matter what the webhook is verifying/doing, because you can't recover from that state.

I'm not suggesting that the system should automatically remove a broken webhook, which would be a security problem, rather that it needs to be possible for an appropriately authenticated sysadmin to do so. Especially since making a broken webhook non-broken may require executing API calls that the broken webhook is preventing.
/assign
as you assigned this to you, can you sketch what the plan is for this?
The plan was to use an integration test, (i.e. ) which I implemented in but it turns out that since the integration test apiserver does not follow exactly the same startup chain as the real one, that it is insufficient to ensure the apiserver can actually come up with a broken webhook installed. The integration test doesn't even fail against the code before merged. So currently the plan is to do it as an e2e test, probably with the [Slow] and [Destructive] tags.
it should be 99.9% the same. What are you missing? I think an integration test would be much better than an e2e test. We have much more control there.
I think the bug fixed was something a controller (bootstrap controller?) was doing, not the apiserver
those bootstrap controller are also executed by the testserver. So I would expect that the bug could be hit in testserver integration tests too. Compare with the real start code and for the test server start code. They are the same. The only difference is that we tweak the options beforehand a bit to run with a random port and random etcd prefix.
Hmm, then I think the reason that the test doesn't fail pre- is because the post start hook changed by that PR isn't called unless api-aggregation is being used. I guess the integration test for this will have to do that as well. Although the one in is still somewhat valuable to make sure new startup namespace creations in non aggregated apiservers are done correctly.
- you probably already thought about it, but one more thing to ensure here is that: apiserver can starte responding "OK" to "/healthz" even if all webhooks are broken [that means e.g. finishing all post-startup hooks] Without that, apiserver would be consistently being killed + restarted by Kubelet on which it is running (at least in some setups).

Yeah, the post-start hooks are the thing that inspired this issue :) Wojciech Tyczynski <:
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Reopened this issue. <detailsIn response to : /reopen : — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub , or mute the thread </details
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available .

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Reopened this issue. <detailsIn response to : /reopen /remove-lifecycle rotten Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: expected: map[string]map[string]bool{"machine1": {"b": true, "c": true}}, }, { <ins> name:       "mixed priority pods are preempted, pick later StartTime one when priorities are equal", predicates: map[string]algorithmpredicates.FitPredicate{"matches": algorithmpredicates.PodFitsResources}, nodes:      []string{"machine1", "machine2"}, pod:        &v1.Pod{ObjectMeta: metav1.ObjectMeta{Name: "machine1", UID: types.UID("machine1")}, Spec: v1.PodSpec{Containers: largeContainers, Priority: &highPriority}}, pods: []*v1.Pod{ {ObjectMeta: metav1.ObjectMeta{Name: "a", UID: types.UID("a")}, Spec: v1.PodSpec{Containers: smallContainers, Priority: &lowPriority, NodeName: "machine1"}, Status: v1.PodStatus{StartTime: &startTime20190107}}, {ObjectMeta: metav1.ObjectMeta{Name: "b", UID: types.UID("b")}, Spec: v1.PodSpec{Containers: smallContainers, Priority: &lowPriority, NodeName: "machine1"}, Status: v1.PodStatus{StartTime: &startTime20190106}}, {ObjectMeta: metav1.ObjectMeta{Name: "c", UID: types.UID("c")}, Spec: v1.PodSpec{Containers: mediumContainers, Priority: &midPriority, NodeName: "machine1"}, Status: v1.PodStatus{StartTime: &startTime20190105}}, {ObjectMeta: metav1.ObjectMeta{Name: "d", UID: types.UID("d")}, Spec: v1.PodSpec{Containers: smallContainers, Priority: &highPriority, NodeName: "machine1"}, Status: v1.PodStatus{StartTime: &startTime20190104}}, {ObjectMeta: metav1.ObjectMeta{Name: "e", UID: types.UID("e")}, Spec: v1.PodSpec{Containers: largeContainers, Priority: &highPriority, NodeName: "machine2"}, Status: v1.PodStatus{StartTime: &startTime20190103}}}, expected: map[string]map[string]bool{"machine1": {"a": true, "c": true}}, }, { </ins> name:       "pod with anti-affinity is preempted", predicates: map[string]algorithmpredicates.FitPredicate{"matches": algorithmpredicates.PodFitsResources}, nodes:      []string{"machine1", "machine2"},

<!-- Please only use this template for submitting enhancement requests --What would you like to be : When selecting victims on node in (), take pod's into account. When two pod's Priority is equal, preempt pod whose is later first. Why is this needed: If a pod's has been running for a longer time, it might finish earlier, or it might have done more things. It might make more sense to preempt them later, e.g. a user submitted a job A at time 9:00 am, then he submitted a job B at time 10:00 am, he might expect that scheduler preempts pods for job B first. <!-- DO NOT EDIT BELOW THIS LINE --/kind feature
/sig scheduling /assign cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/close
Closing this issue. <detailsIn response to : /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
PR implements this feature. We are currently in code freeze for 1.14. The feature will be in 1.15.
Thanks for it. It is similar with a little difference, the PR aims to select a node, and this issue aims to pick a pod whose StartTime is latter after the node has been selected.
I just sent a PR for it. PTAL :)


Question: // nothing provided by user, use default range (only applies to the Primary) if len(serviceClusterIPRangeList) == 0 { var primaryServiceClusterCIDR net.IPNet <del> serviceIPRange, apiServerServiceIP, err = master.DefaultServiceIPRange(primaryServiceClusterCIDR) </del> <ins> serviceIPRange, apiServerServiceIP, err = master.ServiceIPRange(primaryServiceClusterCIDR) </ins> if err != nil { return options, fmt.Errorf("error determining service IP ranges: %v", err) }

API server defaults the service CIDR. We should instead just fail if it is not specified. Defaulting is a wild guess and very likely to be wrong. /sig network /kind bug
For existing clusters that don't currently set it, is setting it to anything other than the current default a breaking change?
Yes. You can not really change it (well, you can expand it, but not totally move it), unless you delete ALL services, including kubernetes.default. I filed this aspirationally. It's not on fire, it's just a dumb thing to have provided a default for.
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/remove-triage unresolved
Those labels are not set on the issue: <detailsIn response to : /remove-triage unresolved Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: minAffinitySize = mask.Count() } <ins> // the node already in group with another node, it can not be used for the single NUMA node allocation if singleNUMAHint && len(machineState[maskBits[0]].Cells) > 1 { return } for _, nodeID := range maskBits { // the node already used for the memory allocation if !singleNUMAHint && machineState[nodeID].NumberOfAssignments > 0 { // the node used for the single NUMA memory allocation, it can not be used for the multi NUMA node allocation if len(machineState[nodeID].Cells) == 1 { return } // the node already used with different group of nodes, it can not be use with in the current hint if !areGroupsEqual(machineState[nodeID].Cells, maskBits) { return } } } </ins> // verify that for all memory types the node mask has enough free resources for resourceName, requestedSize := range requestedResources { podReusableMemory := p.getPodReusableMemory(pod, mask, resourceName)

Setup: 3 Master nodes and 2 Worker Nodes. Version: v1.22.0-rc.0+ Memory Manager on Kubernetes Version (v1.22.0-rc.0+) and Nodes configured with Hugepages of size 2M on 2 Worker nodes. Worker node having 2 Numa Nodes (numa node0 and numa node1) of Hugepages of size 2M are 10 on each Numa node sh-4.4# cat /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nrhugepages 10 sh-4.4# cat /sys/devices/system/node/node1/hugepages/hugepages-2048kB/nrhugepages 10 After configuring Memory , where Hugepages are 20M (2M * 10) on each Numa node. we create QoS pod1 pod which consumes 24Mi Hugepages (of size 2M) Pod Spec: List pods Memory Manager state file: Now create a QoS Guaranteed Pod2 requesting 16Mi . On the same worker node. Pod spec: List pods: Memory Manager state file after pod2 is deployed. Pod2 should be rejected. Steps provided above Kubernetes version (use ): v1.22.0-rc.0+ Cloud provider or hardware configuration: Openshift OS (e.g: ):Red Hat Enterprise Linux CoreOS release 4.9 Kernel (e.g. ): Linux 4.18.0-305.12.1.rt7.84.el84.x8664 SMP PREEMPTRT Thu Jul 29 14:18:12 EDT 2021 x8664 x8664 x8664 GNU/Linux Kubelet configuration
/triage accepted /priority important-longterm /sig node /area kubelet
/assign


Question: func ResizeInstanceGroup(asg ASG, instanceGroupName string, size int) error { request := &autoscaling.UpdateAutoScalingGroupInput{ AutoScalingGroupName: aws.String(instanceGroupName), <del> MinSize:              aws.Int64(int64(size)), MaxSize:              aws.Int64(int64(size)), </del> <ins> DesiredCapacity:      aws.Int64(int64(size)), </ins> } if _, err := asg.UpdateAutoScalingGroup(request); err != nil { return fmt.Errorf("error resizing AWS autoscaling group: %q", err)

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: After the test ASG min/max were reset to the desired node count. What you expected to happen: Min/Max shouldn't be reset, desired should. How to reproduce it (as minimally and precisely as possible): Run this command twice: Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: AWS OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/assign ellis-tarn
GitHub didn't allow me to assign the following users: ellis-tarn. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign ellis-tarn Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig autoscaling


Question: } } <del> func setAffinity(d *appsv1.Deployment) { d.Spec.Template.Spec.Affinity = &v1.Affinity{ PodAffinity: &v1.PodAffinity{ PreferredDuringSchedulingIgnoredDuringExecution: []v1.WeightedPodAffinityTerm{ </del> <ins> // setAffinities set PodAntiAffinity across pods from the same generation // of Deployment and if, explicitly requested, also affinity with pods // from other generations. // It is required to make those "Required" so that in large clusters where // scheduler may not score all nodes if a lot of them are feasible, the // test will also have a chance to pass. func setAffinities(d *appsv1.Deployment, setAffinity bool) { affinity := &v1.Affinity{ PodAntiAffinity: &v1.PodAntiAffinity{ RequiredDuringSchedulingIgnoredDuringExecution: []v1.PodAffinityTerm{ </ins> { <del> Weight: int32(100), PodAffinityTerm: v1.PodAffinityTerm{ TopologyKey: "kubernetes.io/hostname", LabelSelector: &metav1.LabelSelector{ MatchExpressions: []metav1.LabelSelectorRequirement{ { Key:      "name", Operator: metav1.LabelSelectorOpIn, Values:   []string{d.Spec.Template.Labels["name"]}, }, { Key:      "iteration", Operator: metav1.LabelSelectorOpNotIn, Values:   []string{d.Spec.Template.Labels["iteration"]}, }, </del> <ins> TopologyKey: "kubernetes.io/hostname", LabelSelector: &metav1.LabelSelector{ MatchExpressions: []metav1.LabelSelectorRequirement{ { Key:      "name", Operator: metav1.LabelSelectorOpIn, Values:   []string{d.Spec.Template.Labels["name"]}, }, { Key:      "iteration", Operator: metav1.LabelSelectorOpIn, Values:   []string{d.Spec.Template.Labels["iteration"]}, </ins> }, }, }, }, }, }, <del> PodAntiAffinity: &v1.PodAntiAffinity{ </del> <ins> } if setAffinity { affinity.PodAffinity = &v1.PodAffinity{ </ins> RequiredDuringSchedulingIgnoredDuringExecution: []v1.PodAffinityTerm{ { TopologyKey: "kubernetes.io/hostname",

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: gce-master-scale-correctness Which test(s) are failing: [sig-network] Loadbalancing: L7 [Slow] Nginx should conform to Ingress spec [sig-network] Services should handle load balancer cleanup finalizer for service [sig-network] Services should only allow access from service loadbalancer source ranges Since when has it been failing: 2019-10-19 Testgrid link: https://k8s- Reason for failure: and Anything else we need to know: /sig scalability /sig network could you triage?
It seems that new test was at that point: [sig-apps] Deployment should not disrupt a cloud load-balancer's connectivity during rollout My personal guess is that we started doing too much load-balancer-related stuff at the same time.
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
I think this is my fault. should fix it. Is there some way to those tests to run against a PR? /remove-triage unresolved
At the moment the fastest way would be to merge your fix and wait for the next run tomorrow.
Reopening just to keep this tracking - will close once we have green run.
the fix helped for a two tests, but one of them is still failing: https://k8s-
I disagree - there are couple loadbalancing-related tests that are flaking with high flakiness ration - I think this PR didn't change anything. So my feeling is that is may still be something like:
Still, the conclusion is that unfortunately more work in needed to put us back in a good shape.
I looked a bit into it and the conclusions are: "Services should handle load balancer cleanup finalizer for service [Slow]" test is consistently failing in small clusters too with: IIUC, the reason is that if we downgrade load-balancer service to non-loadbalaced one, it takes some time to remove the ingress from Status, but it is failing immediately due to this: - can you please take a look? the apps test (and I think couple more) are failing due to incorrect timeouts - I will send out a fix for it soon
Actually, this one will be still failing as of now - the reason for it is described here:


Question: } if len(sorted) != len(graph) { <del> fmt.Printf("%s", (sorted)) </del> return nil // Cycle detected }

ci-kubernetes-unit TestPickBestStrategy shows the jobs are success. but in test-grid and ci report, it shows a failure. After This failed the {Failed;Failed;Failed;Failed;Failed;Failed;Failed; === RUN TestPickBestStrategy === RUN TestPickBestStrategy/singlecandidate,singlepreferredstrategy === RUN TestPickBestStrategy/multiplecandidates,differentpreferredstrategiesshouldfail === RUN TestPickBestStrategy/multiplecandidates,multipleresolvedpreferredstrategy No response_ /sig testing
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc
Thanks for opening this Paco, just noticed.
cc
! It is weird as the test passed indeed and I cannot reproduce it.
Fixed in


Question: {Name: "node-x", Score: 100}, }, }, <ins> { name: "existing pods in a different namespace do not count", pod: st.MakePod().Name("p").Label("foo", ""). SpreadConstraint(1, "node", softSpread, st.MakeLabelSelector().Exists("foo").Obj()). Obj(), existingPods: []*v1.Pod{ st.MakePod().Name("p-a1").Namespace("ns1").Node("node-a").Label("foo", "").Obj(), st.MakePod().Name("p-a2").Node("node-a").Label("foo", "").Obj(), st.MakePod().Name("p-b1").Node("node-b").Label("foo", "").Obj(), st.MakePod().Name("p-b2").Node("node-b").Label("foo", "").Obj(), }, nodes: []*v1.Node{ st.MakeNode().Name("node-a").Label("node", "node-a").Obj(), st.MakeNode().Name("node-b").Label("node", "node-b").Obj(), }, want: []framework.NodeScore{ {Name: "node-a", Score: 100}, {Name: "node-b", Score: 50}, }, }, { name: "terminating Pods should be excluded", pod: st.MakePod().Name("p").Label("foo", "").SpreadConstraint( 1, "node", softSpread, st.MakeLabelSelector().Exists("foo").Obj(), ).Obj(), nodes: []*v1.Node{ st.MakeNode().Name("node-a").Label("node", "node-a").Obj(), st.MakeNode().Name("node-b").Label("node", "node-b").Obj(), }, existingPods: []*v1.Pod{ st.MakePod().Name("p-a").Node("node-a").Label("foo", "").Terminating().Obj(), st.MakePod().Name("p-b").Node("node-b").Label("foo", "").Obj(), }, want: []framework.NodeScore{ {Name: "node-a", Score: 100}, {Name: "node-b", Score: 0}, }, }, </ins> } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) {

How to reproduce it (as minimally and precisely as possible): pre-requisites: a multi-az cluster, a deployment with 1 pod in each az. a with and . the node(s) in 1 az. the deployment's pod(s) from the cordoned az. What happened: A new pod was created and scheduled in a form that What you expected to happen: I expected that a new pod will be created and be "stuck" in status as it shouldn't have nowhere be scheduled. Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: AWS OS (e.g: ):Kernel (e.g. ): 4.19.84-coreos Others: I created a with how I reproduce the bug
/sig scheduling
cc/
Thanks for raising this issue. I feel like it's a general timing issue - when you delete a Pod, that Pod may still exist for a very short time (in graceful deletion period), and at the same time, the controller backfills a Pod for it. So when scheduling tries to schedule the backfill Pod, it may read the cluster as: [1, 1, 1(deleting)]. To confirm this, can you try deleting the Pod using , and see if the problem goes away?
Yes sure I will try that. It sounds reasonable, thats why in the first demonstration I deleted all the pods and waited for all of them to completely disappear. Nevertheless, this is still something that I believe needs to be fixed. The pod's evaluation should take in consideration pods that are in status and don't count them as pods.
I do believe we exclude the Pods. (if not, it should be a serious regression). What concerns me is that at that moment, scheduler may treats that (deleting) Pod as running in its cache. I will try to debug into this.
Yep, this time the new pod was in status until I uncordoned the relevant nodes.


Question: <del> v1.6.3-7 </del> <ins> v1.6.3-8 </ins>

See
CI has built and uploaded unofficial prerelease darwin binaries for 1.4 using go1.7.1: cc Official binaries will ship as soon as we cut 1.4.1.
We have official v1.4.1 binaries available now, too.
Hooray


Question: <del> v1.6.3-7 </del> <ins> v1.6.3-8 </ins>

See
CI has built and uploaded unofficial prerelease darwin binaries for 1.4 using go1.7.1: cc Official binaries will ship as soon as we cut 1.4.1.
We have official v1.4.1 binaries available now, too.
Hooray


Question: } if pvc.Status.Phase != v1.ClaimBound || pvc.Spec.VolumeName == "" { return "", "", fmt.Errorf( "PVC %s/%s has non-bound phase (%q) or empty pvc.Spec.VolumeName (%q)", namespace,

Deployment/Service manifest: logs from events: In aws, the volume is attached, On the box: In only one is mounted: Where can I debug this further?
Take a look at the on your master for events related to your volume . That will show you when the volume was attached/detached. It appears that your volume attached without issue so there should be nothing unusual here. Then take a look at on the node that the volume is supposed to be attached to. Looks for the UID of your pod (). I suspect one of your mounts is failing. It looks like you are using hyperkube so it might be a mount namespace propagation issue.
I believe this happens when the RC/Pod is being created shortly after the PV or PVC. The latter two appear to need several seconds to be fully available (even if the status is already set to "Bound"). I ran into the same issue with a NFS PV. When I delete and create PV+PVC+SVC+RC all in one go, the associated Pod will be in ContainerCreating state for several seconds (10,<60), then log the warning event and afterwards be created without any problems or further delays. When I first create PV+PVC, wait for several seconds (30) and only then create SVC+RC, the pod will be created immediately and no warning event will be logged. One oddity I noticed is that the PVC will have capacity 0 and empty access modes (as shown by ), even though it was setup to request ROX for 1Gi and the PV provides 1Gi with ROX. This might be a different bug, though.
Thanks for digging into it The capacity 0 issue () was fixed by PR and is in v1.3.5. This is more or less expected behavior. The mount logic has exponential backoff so if it rapidly encounters PVC/PV does not exist, it'll back off the retries resulting in a longer mount time. We can consider optimizing this particular scenario. But I'll lower the priority on this for now and remove the milestone.
Maybe the "Bound" status could be delayed, so the RC would be held back for a while and be scheduled only when the PVC is actually ready?
I don't think the RC actually waits for PVC to be bound before scheduling.

I have also this problem using deployment with HELM, I deploy all my pv,pvc,deploy,svc in one Charts, and my pod who requiere a volume from the PVC get scheduled and stay in ContainerCreating mode. I need to wait minutes or delete pods for they correctly get rescheduled using the volume. Here is the log when the pod get automaticly deleted (after minutes) : It would be awesome if the deployment or RC wait for the PVC to be bound before schedule !
may slightly improve this situation as the maximum backoff time of 2 minutes will not fire anymore.
Note that the same error is thrown when node-size == C5 or M5 for kubectl and with kops . Solution is to set node as t2 machines. Due to it seems.
Reading from the new solution is: