Question: test := t It(test.name, func() { if test.driverExists { <del> csiDriver := createCSIDriver(csics, "csi-hostpath-"+f.UniqueName, test.driverAttachable) </del> <ins> csiDriver := createCSIDriver(csics, drivers.GetUniqueDriverName(driver), test.driverAttachable) </ins> if csiDriver != nil { defer csics.CsiV1alpha1().CSIDrivers().Delete(csiDriver.Name, nil) }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: Probably caused by <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
https://k8s- /assign
I understand that moved crd creation logic to e2e test's k8s cluster deployment code. On the other hand, I tested it with manually deployed k8s cluster. So, in my case, I guess that in addition to enable and feature gates enabled, I need to create crd manually before tests, like below: I could make two of three SkipAttach tests pass in my environment by above way. For "non-attachable volume does not need VolumeAttachment" case, I'm still looking into it. (It doesn't seem to skip creating volumeAttachment, properly)
I found a typo in 's argument. While hostpath driver's connects driver name and unique name without "-", it connects them with "-". This makes "non-attachable" test fail, due to the mismatch between volume name in crd and provisioner name, as a result, creating volumeAttachment isn't skipped. I will create a PR to fix it. For above bug, is unrelated. It just changed the way in how to test in manually deployed k8s cluster as I mentioned in above comment. So, there should be no problem for the PR, at least for the CI testing if it works well in CI environments.
I can verify that the CRDs are not getting installed on the alpha cluster. Investigating why.
/reopen since I see the Alpha tests failing
/milestone v1.13 /priority important-soon /kind failing-tests
/kind failing-test
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I don't think this is fully resolved. There are still failures on https://k8s- /reopen
Reopened this issue. <detailsIn response to : I don't think this is fully resolved. There are still failures on https://k8s- Instructions for interacting with me using PR comments are available .

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
can you help take a look at these failures? It's testing CSI filesystem volume mode. I think it is unrelated to the crd installation issue that Saad fixed.
I tested with AllAlpha flag and manually created csi crds and confirmed that filesystem volume mode test passed in my environment and also other tests passed there. If I delete csi crds, other tests also fail, so I agree that the issue that Saad fixed is not related. However, above also means that this issue can't be reproduced in my environment.
I1109 14:56:01.141] Nov 9 14:56:01.131: INFO: At 2018-11-09 14:50:54 +0000 UTC - event for security-context--e42e-11e8-9b95-: {attachdetach-controller } FailedAttachVolume: AttachVolume.Attach failed for volume "pvc--e42e-11e8-880b-" : node "bootstrap-e2e-minion-group-3wtg" has no NodeID annotation
Hostpath also seems to be failing because of driver registration issues? Not sure why other alpha tests are passing if that's the case
This test is the only test that uses  to create pod with CSI driver volume and expects the result to be success. One of the pod parameters specified there might affect this failure.
Those 2 failing tests are newly introduced in and started to fail from the beginning. Is that PR blocking 1.13? If not, can we revert and fix later? I have recently set up optional presubmit tests for alpha features at , which will pick up those 2 tests. Currently this failure is blocking
apart from the 2 failing tests, I see which addressed with is also flaky. These failures are now blocking PRs as mentioned above. As we enter code slush today, its very critical to have the PRs get in sooner than later. Can we plz revert the feature and the tests until they are fully resolved and ready to go. Thanks
I prefer disabling the specific tests instead of reverting the entire pr. We will lose important signal for csi
That works for me.

Can you cc me on the PR that disables those tests? When you re-enable the tests, remember to try them out on . :)
Let me consider if we can just skip these test for CSI, instead of reverting the entire patch.
you mean just skipping them in Alpha cluster right?
I was thinking about skipping the below test with the combination of below drivers: [Test] [Testpattern: Dynamic PV (filesystem volmode)] volumeMode[Feature:BlockVolume] should create sc, pod, pv, and pvc, read/write to the pv, and delete all created resources [Driver] csi-hostpath Won't it enough? Or, do we need to skip it only with alpha cluster?
If we just disable the tests alone and not revert the feature, we are just masking the issue. Ideally if we are not able to fix the tests and reenable them the feature should be reverted from 1.13 as well. Are you also actively working on fixing the tests and this skip is just to unblock PRs in the meantime? if so can we timebox by when we can have the tests fixed and re-enabled / the feature reverted? My fear is we might forget the fact that its disabled since there's no active reporting on it.
Actually, there is no feature in either of culprit commits. It tests or intended to improve the way it is tested. Therefore, it must be a test issue or another issue which is found by the test , not an issue introduced by the commits themselves. In the code, there is a clear comment that the test is skipped due to this issue with the reference to this issue#, so this won't be forgot. If you still worry about this issue is not tracked, we might be able to keep this issue open.
IIUC the test that were failing (and are being reverted) were for the new CSI festures going to Alpha in 1.13, right? so even these 2 culprit commits do not contain feature work, it is testing some new code additions to 1.13 and now those will be untested until we re-enable these tests. Is there an ETA to re-enable them, preferably sometime early next week? Yes +1 to keeping this issue open to track the skipped tests.
The two failing tests themselves are not for new alpha features that are in v1.13.

Instead, they are just to test csi driver works well as it works in existing in-tree driver. (And failed because they might be affected by some other alpha features or some of specific configurations in gce.) In addition, ~50 tests for csi driver, so it would be better not to remove all of the tests by reverting the PR, just for the 2 failing tests. I think that just reverting the patch won't help improving quality of k8s. As for ETA to solve the issue, I found difficulty in debuging in my local environment. It seems that the two tests fail in gce environment w/ AllAlpha=true, however, as I've already mentioned, at least csi-hostpath tests in my local environment w/ AllAlpha=true doesn't fail.(And I don't have test environment for gce.) Of course, I will help solving the issue next week. However, I will need someone's help for testing combinations of alpha features in gce environment to narrow down which flag really affects this issue. At least I would like to know the results: w/ MountPropagation=true,CSIPersistentVolume=true,BlockVolume=true,CSIBlockVolume=true all alpha flags except CSIDriverRegistry and CSINodeInfo
/reopen Thanks for the detailed explanation. I think the revert is fine in this case then. I am keeping this issue open however to track and fix the failures in these 2 tests can any of you help answer w.r.t alpha features on gce environment. thanks
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I can help debug when I'm back on Tuesday. In the meantime, I would suggest looking through the failing test logs to try to triage what's going on
Here's one failing run:
I looked into the csi-hostpath failure log. As seen in the for , where the pod which has a csi-hostpath volume is being attached is scheduled, mount volume fails with these errors (1). Related code to this log is (2), so the driver doesn't seem to be registered to . As for the registration code for csiDriver, it is called as (3).

Therefore, will be logged if the driver is triggered for registration. As seen (4), registration is only triggered in , which is different from the one the pod is scheduled. In addition, driver-registrar seems to be deployed only on the node where registration is triggered (5). Is it right behaviour, while registrar is deployed as daemonset? I guess that just this fact seems to cause an issue to make csi-hostpath driver work properly. (Actually, I only tested with 1 node environment, so this might only happen in multiple nodes environment.) <details- (1) (2) Result of grep to check whether registrar is deployed to each node [ for bootstrap-e2e-minion-group-50r7][ for bootstrap-e2e-minion-group-k781][ for bootstrap-e2e-minion-group-z5cg] </details
The hostpath plugin should be deployed as a DaemonSet, so it's strange that it's only getting registered on only one node.
/assign looking into this as well
could you link the PR for skipping [external-gcepd] tests into this issue when you do that as well
skipping gcePD-external is here:
Since gce-cos-alphafeatures is now passing, do we need to keep this issue open? Should it morph into a non-blocking issue for longterm work on a better test?
We have root caused 2 of the 3 failures, which were problems in the test and test conifg. Fixes will be out soon. Still triaging the 3rd failure, which could be a real bug in alpha clusters
I've root caused csi-hostpath failure issue. It just failed to set the node properly in test code side. I will make a PR to fix it.
Or should it be done in your PR too, to avoid modifying twice in the separate PR? Actually, the fix are as below: (Failing to set node before pod creation)
Created for csi-hostpath fix, to prepare for the case that we need a separate PR. We can discuss more on this there, if needed.
/unassign found possible root cause for GCE issue: CSI NodeID annotation is not present on the Node object, could be caused by failed retries on updating Node object.

(apparently when alpha features are open we shouldn't be looking at the Node object anyway except we seem to be, so that might also be a bug). has the most context for the fix /assign
The symptom of the remaining test failure is the test pod fails to create because it fails to attach with the error: . The test succeeds the first time it's run in a cluster, and always fails after that. Two issues caused this error message. the CSI external-attacher tries to fetch the object to get the NodeID, it does so through an informer. The first time the test is run, objects are created for the very first time, and the informer returns the correct object. In subsequent test runs, s from the first run is kept, and for some reason the informer fails to detect the object and returns empty. Making the API call directly works, I'll patch the external-provisioner with this to mitigate the issue, and this is sufficient to get the test to pass. The informer bug will need to be eventually fixed. fails to post the NodeID annotation sometimes, likely due to node status patch not being retried correctly. This is crucial for clusters without feature flag set, and this is the default for 1.13. Will add the appropriate retry mechanism.
Opened to track the informer issue
Are and potential PR for the only 2 fixes being actively worked on for this issue?
Only is needed. We have a workaround for
will you have a followup PR to enable the CSI alpha tests back? or is that already in flight?
also reenables the tests
OK with merged we should see the CSI alpha tests re-enable. to keep an eye out for CI signal
The tests got reenabled 5 days ago, and alpha tests are passing. Closing this issue. Thanks everyone for your rapid turnaround! /close
Closing this issue. <detailsIn response to : The tests got reenabled 5 days ago, and alpha tests are passing. Closing this issue. Thanks everyone for your rapid turnaround! Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The tests are actually running (and passing) in the normal suite now, now that the raw block feature moved to beta.


Question: return fmt.Errorf("failed os init: %w", err) } <del> if err := opts.Complete(); err != nil { </del> <ins> if err := opts.Complete(cmd.Flags()); err != nil { </ins> return fmt.Errorf("failed complete: %w", err) }

introduced configuration of additional logging aspects (e.g. format) through command line flags and config file. This was merged for Kubernetes 1.28. Now the following invocation doesn't work as expected when the config file doesn't configure logging: It should run with verbosity 4. Instead it runs with verbosity 0, from the config. Run with this config file: No response 28 rc n/a No response No response No response No response
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The problem is caused by this line: At that point, is 4 (from the command line flag), and then it gets overwritten with == 0. The normal approach would be: set up config with defaults modify config through flags modify config further by parsing a config file into it. Can kube-proxy do this? It has the downside that a config file can only overwrite fields if they are in the file. A config field doesn't get reset to defaults just by parsing the file. /cc
it looks like kube-proxy special-cases the to let a flag override config... can/should it do the same thing for verbosity? are there any other flags like this?
I don't think that's the normal approach... it's more normal to make flags and config file mutually exclusive. That's difficult for things which started as flags and moved into config
/sig network /sig instrumentation /kind bug /kind regression /milestone v1.28
let's go with and warn it and comment as much as possible, I was going with something like but Patrick is working on it so I assign to him the issue
/assign
Hey folks, v1.28 Bug Triage Lead here Just wanted to know if this is already tracked by SIG leads and someone already working actively fixing it and set a priority?
Perhaps I should have said "this is how component-base/logs was meant to be used". Somehow it worked for kubelet.

Here's my preliminary solution: It needs more testing, but I wanted to share it early because I am running out of time today.
it is , I can take over Patrick's work in case is needed
It would be nice if the logs API could distinguish between and , how about make to a pointer? But this change is supposed to be a breaking change, I'm not sure of the scope of the impact. For side, if set and at the same time, should the command flag override the config file or warn it and be discarded? I think it should be overridden.
xref:


Question: cmdArgs = append(cmdArgs, "--runtime-cgroups=/docker-daemon", "--kubelet-cgroups=/kubelet", <ins> "--cgroup-root=/", "--system-cgroups=/system", </ins> ) } cmdArgs = append(cmdArgs,

In node-e2e-test on Containervm image, the containers of the pods created by Kubernetes are subcontainers of docker-daemon/docker/. It means that the runtime resource usage (docker-daemon) is not correct, because all pods are included. This problem maybe due to container misconfiguration in node-e2e-test.
The kubelet on debian-based containervm image is misconfigured. Will fix it shortly.


Question: t.Errorf("nexpected:n%vnsaw:n%v", expected, params) } } <ins> func TestGetBool(t *testing.T) { testCases := []struct { name         string parameters   map[string]string key          string defaultValue bool expected     bool expectError  bool }{ { name: "found key in parameters, default value is different from key value", parameters: map[string]string{ "foo": "false", }, key:          "foo", defaultValue: false, expected:     false, expectError:  false, }, { name: "found key in parameters, default value is same with key value", parameters: map[string]string{ "foo": "true", }, key:          "foo", defaultValue: true, expected:     true, expectError:  false, }, { name: "key not found in parameters, default value is true", parameters: map[string]string{ "foo": "true", "far": "false", }, key:          "bar", defaultValue: true, expected:     true, expectError:  false, }, { name: "key not found in parameters, default value is false", parameters: map[string]string{ "foo": "true", "far": "false", }, key:          "bar", defaultValue: false, expected:     false, expectError:  false, }, { name:         "parameters is empty", parameters:   map[string]string{}, key:          "foo", defaultValue: true, expected:     true, expectError:  false, }, { name: "parameters key is not a valid bool value", parameters: map[string]string{ "foo": "error", }, key:          "foo", defaultValue: true, expected:     false, expectError:  true, }, } for _, test := range testCases { got, err := GetBool(test.parameters, test.key, test.defaultValue) if err != nil && test.expectError == false { t.Errorf("%s: unexpected error: %v", test.name, err) } if err == nil && test.expectError == true { t.Errorf("%s: expect error, got nil", test.name) } if got != test.expected { t.Errorf("%s: expect %v, got %v", test.name, test.expected, got) } } } func TestMakeParseLabels(t *testing.T) { successCases := []struct { labels   map[string]string expected map[string]string }{ { labels: map[string]string{ "foo": "false", }, expected: map[string]string{ "foo": "false", }, }, { labels: map[string]string{ "foo": "true", "bar": "123", }, expected: map[string]string{ "foo": "true", "bar": "123", }, }, } for _, test := range successCases { labelString := MakeLabels(test.labels) got, err := ParseLabels(labelString) if err != nil { t.Errorf("unexpected error :%v", err) } if !reflect.DeepEqual(test.expected, got) { t.Errorf("nexpected:n%vngot:n%v", test.expected, got) } } errorCases := []struct { name   string labels interface{} }{ { name:   "non-string", labels: 123, }, { name:   "empty string", labels: "", }, { name:   "error format", labels: "abc=456;bcd=789", }, { name:   "error format", labels: "abc=456.bcd=789", }, { name:   "error format", labels: "abc,789", }, { name:   "error format", labels: "abc", }, { name:   "error format", labels: "=abc", }, } for _, test := range errorCases { _, err := ParseLabels(test.labels) if err == nil { t.Errorf("labels %s expect error, reason: %s, got nil", test.labels, test.name) } } } func TestMakeParseProtocols(t *testing.T) { successCases := []struct { protocols map[string]string expected  map[string]string }{ { protocols: map[string]string{ "101": "TCP", }, expected: map[string]string{ "101": "TCP", }, }, { protocols: map[string]string{ "102": "UDP", "101": "TCP", }, expected: map[string]string{ "102": "UDP", "101": "TCP", }, }, } for _, test := range successCases { protocolString := MakeProtocols(test.protocols) got, err := ParseProtocols(protocolString) if err != nil { t.Errorf("unexpected error :%v", err) } if !reflect.DeepEqual(test.expected, got) { t.Errorf("nexpected:n%vngot:n%v", test.expected, got) } } errorCases := []struct { name      string protocols interface{} }{ { name:      "non-string", protocols: 123, }, { name:      "empty string", protocols: "", }, { name:      "error format", protocols: "123/TCP;456/UDP", }, { name:      "error format", protocols: "123/TCP.456/UDP", }, { name:      "error format", protocols: "123=456", }, { name:      "error format", protocols: "123", }, { name:      "error format", protocols: "123=", }, { name:      "error format", protocols: "=TCP", }, } for _, test := range errorCases { _, err := ParseProtocols(test.protocols) if err == nil { t.Errorf("protocols %s expect error, reason: %s, got nil", test.protocols, test.name) } } } </ins>

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: construct a string for labels, and let consume it. For example, suppose label string is , then this function will return a map which indicates a label . construct a string for portProtocols, and let consume it. For example, suppose portProtocols string is , then this function will return a map which indicates a pair . What you expected to happen: I assume the key of a label should not be an empty string since it makes no sense. I assume neither the nor the of pair should not be an empty string since it makes no sense. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration**: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: /area kubectl
If it's a bug, I can help fix it.
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig cli


Question: finalAttached:        false, finalAttachErr:       &storage.VolumeError{Message: "missing volume"}, trigerWatchEventTime: 5 * time.Millisecond, <del> sleepTime:            10 * time.Millisecond, </del> timeout:              30 * time.Millisecond, shouldFail:           true, },

/kind bug /sig storage What happened: CSI unit tests occasionally end up with data race: How to reproduce it (as minimally and precisely as possible): to get CSI unit tests into a binary: the binary 500 times in parallel until it breaks: . Use different argument to really load your machine, it should not be idle! . On my Linux, I got 3 failures in ~1000 runs. puts stderr of failed runs into .
/assign
I make a Fix: PTAL


Question: HardPodAffinitySymmetricWeight: 1, HealthzBindAddress:             "0.0.0.0:10251", MetricsBindAddress:             "0.0.0.0:10251", <ins> DebuggingConfiguration: componentbaseconfig.DebuggingConfiguration{ EnableProfiling:           true, EnableContentionProfiling: true, }, </ins> LeaderElection: kubeschedulerconfig.KubeSchedulerLeaderElectionConfiguration{ LeaderElectionConfiguration: componentbaseconfig.LeaderElectionConfiguration{ LeaderElect:       false,

What would you like to be : Enable profiling by default in the scheduler. We can do this by defaulting the CC flags to true. Why is this needed: It is useful when debugging the performance of a running cluster. API server already enables profiling by default. /sig scheduling /help /milestone v1.17
/assign /remove-help


Question: **Note**: You will want to set the Docker VM to have at least 8GB of initial memory or building will likely fail. (See: [#11852]( http://issue.k8s.io/11852)). * **Linux with local Docker**  Install Docker according to the [instructions](https://docs.docker.com/installation/#installation) for your OS. * **Windows with Docker Desktop WSL2 backend**  Install Docker according to the [instructions](https://docs.docker.com/docker-for-windows/wsl-tech-preview/). Be sure to store your sources in the local Linux file system, not the Windows remote mount at `/mnt/c`. <del> **Note**: You will need to check if Docker CLI plugin buildx is properly installed. You can install buildx according to the [instructions](https://github.com/docker/buildx/blob/master/README.md#installing). </del> <ins> **Note**: You will need to check if Docker CLI plugin buildx is properly installed (`docker-buildx` file should be present in `~/.docker/cli-plugins`). You can install buildx according to the [instructions](https://github.com/docker/buildx/blob/master/README.md#installing). </ins> 2. **Optional** [Google Cloud SDK](https://developers.google.com/cloud/sdk/) You must install and configure Google Cloud SDK if you want to upload your release to Google Cloud Storage and may safely omit this otherwise.

does not explicitly mention that buildx cli plugin is required for this operation. If buildx is not present, running gives the following error (which is not very helpful.) Running the manual command also results in the same error. I think that the doc should mention that buildx cli plugin should be present for the Docker Image to successfully build, how users can check if buildx is installed, and if not already present. (I'm very new to kubernetes community, and I spent 2-3 days trying to figure this out, so this could really help beginners who get stuck here).
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig contributor-experience
/kind documentation
this seems like a valid improvement. if someone wants to take this, PRs are welcome. EDIT: it seems the buildx cli plugin installation is a bit vague on the side of docker too. thus i will not add the label.
Happy to help here Docker Buildx installation can be verified by checking if file is present in . Buildx can be installed by the following sequence of commands (): Also, should this issue be left open for community (such as first time contributors) or should I make the changes and raise a PR (if we need it quickly)?
if you think you can make the change, please go ahead. thanks


Question: State:  status.WorkType(), Orphan: orphan, } <del> if status.activeUpdate != nil && status.activeUpdate.Pod != nil { sync.HasConfig = true sync.Static = kubetypes.IsStaticPod(status.activeUpdate.Pod) </del> <ins> switch { case status.activeUpdate != nil: if status.activeUpdate.Pod != nil { sync.HasConfig = true sync.Static = kubetypes.IsStaticPod(status.activeUpdate.Pod) } case status.pendingUpdate != nil: if status.pendingUpdate.Pod != nil { sync.HasConfig = true sync.Static = kubetypes.IsStaticPod(status.pendingUpdate.Pod) } </ins> } workers[uid] = sync }

The following log line is often (~30times) logged during our tests in : . This suggests a possible bug as it was assumed to do not happen. No log line saying . Look at artifacts of all (or almost all) builds of: No response <details</details<detailsReproducible in tests </details<details</details<details</details<details</details<details</details
/sig node FYI
/triage accepted since it's a new error, it is a regression. So adding a corresponding priority /priority important-soon
are you working on this? Should I assign bug to you?
Yes, I'm taking a look at this. /assign
/cc
This is a metrics only error (we don’t report a fraction of metrics) and has no functional impact.
That makes sense, but is it expected that this is a valid state in first place? . My assumption was that if static is false, (i.e. API pod) then we should have a config?
I took a closer look at the logs, it seems that this is happening when we enter while a static pod is being synced which has UID reuse and another pod which has the same full name did not terminate yet. In this case, we block the static pod from starting i.e. () returns false: Because the pod can't start, we don't set : which means that when we log metrics there is no set and thus or are both defaulted to false:
Based on the above, it looks like this case is expected. As a result, maybe the simplest option is to simply drop the log statement since by design we only want to into account a subset of the states for the metrics?


Question: func (a *azureDriver) CreateVolume(config *testsuites.PerTestConfig, volType testpatterns.TestVolType) testsuites.TestVolume { ginkgo.By("creating a test azure disk volume") <ins> if volType == testpatterns.InlineVolume { // Volume will be created in framework.TestContext.CloudConfig.Zone zone, // so pods should be also scheduled there. config.ClientNodeSelector = map[string]string{ v1.LabelZoneFailureDomain: framework.TestContext.CloudConfig.Zone, } } </ins> volumeName, err := framework.CreatePDWithRetry() framework.ExpectNoError(err) return &azureVolume{

I am trying to run Azure e2e tests: The test creates a volume in the cloud + runs a pod with the volume as in-line. The pod never starts: The test creates a "blob" disk, while the Azure machine expects "managed" disk. Does anyone run storage e2e tests on Azure regularly? Was it ever working? Managed disks are provisioned . Should the test suite create "managed" disks by default too? Environment: Kubernetes version (use ): v1.17.0-alpha.0.1089+-dirty
/sig azure
/sig cloud-provider
managed disk should be the default, the azure disk storage class in the test case may have
The volume is created outside of kubernetes dynamic provisioning to test pre-provisioned and inline volumes: I wonder if anyone actually runs the tests.


Question: var map_EndpointConditions = map[string]string{ "":            "EndpointConditions represents the current condition of an endpoint.", <del> "ready":       "ready indicates that this endpoint is prepared to receive traffic, according to whatever system is managing the endpoint. A nil value indicates an unknown state. In most cases consumers should interpret this unknown state as ready. For compatibility reasons, ready should never be "true" for terminating endpoints.", </del> <ins> "ready":       "ready indicates that this endpoint is prepared to receive traffic, according to whatever system is managing the endpoint. A nil value indicates an unknown state. In most cases consumers should interpret this unknown state as ready. For compatibility reasons, ready should never be "true" for terminating endpoints, except when the normal readiness behavior is being explicitly overridden, for example when the associated Service has set the publishNotReadyAddresses flag.", </ins> "serving":     "serving is identical to ready except that it is set regardless of the terminating state of endpoints. This condition should be set to true for a ready endpoint that is terminating. If nil, consumers should defer to the ready condition.", "terminating": "terminating indicates that this endpoint is terminating. A nil value indicates an unknown state. Consumers should interpret this unknown state to mean that the endpoint is not terminating.", }

The EndpointSlice API says and it works as expected in normal circumstances I delete the pod and the endpoints conditions are the expected However, if during this time I set we run into an incoherence with the API doc, but it seems that we already documented this edge case in the code I think that at this point is just to amend this information to the API, no? /kind bug /kind documentation
/sig network /assign
I would like to work on this. I just want a bit more clarification on what needs to be done!
I proposed a solution, that is clarify it in the doc ... once we reach an agreement in this issue we implement it ...
+1 to docs. I'd make it a little more vague, like: """ when the normal readiness behavior is being explicitly overridden, for example when the associated Service has set the publishNotReadyAddresses flag. """
so, the thing is to update the docs here with Tim's suggestion and here and do to update the generated code and submit a PR
/assign
I am not able to run command in windows, I am checking it. This error message is coming: *
/triage accepted
Is there a PR for this? Happy to review
I will raise the PR if my above problem gets solved. I am trying to make an update but the error mentioned above is coming.
What platform are you on? If the Makefile were really broken EVERYTHING would be failing.
he's on windows ... you will need a linux/macos. see
Let me know if you want someone to take it over.
I think in that case, you can assign this to someone else.


Question: # reset manifest list; needed in case multiple images are being built / pushed. manifest=() # Make os_archs list into image manifest. Eg: 'linux/amd64 linux/ppc64le' to '${REGISTRY}/${image}:${TAG}-linux-amd64 ${REGISTRY}/${image}:${TAG}-linux-ppc64le' <del> while IFS='' read -r line; do manifest+=("$line"); done < <(echo "$os_archs" | ${SED} "s~/~-~g" | ${SED} -e "s~[^ ]*~$REGISTRY/$image:$TAG-&~g") </del> <ins> while IFS='' read -r line; do manifest+=("$line"); done < <(echo "$os_archs" | "${SED}" "s~/~-~g" | "${SED}" -e "s~[^ ]*~$REGISTRY/$image:$TAG-&~g") </ins> docker manifest create --amend "${REGISTRY}/${image}:${TAG}" "${manifest[@]}" # We will need the full registry name in order to set the "os.version" for Windows images.

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: when building images on non-amd platform, arm64 for example, and the target image is arm64 too, the code still reset . What you expected to happen: There is no need to reset anything, the shell code is purely for amd64 and haven't considered the non-amd platforms. How to reproduce it (as minimally and precisely as possible): on any platform other than amd, cd test/images build all WHAT=echoserver (or any image under the dir) when coming to image which is exactly the same as the build machine, following is seen from console Anything else we need to know?: Environment: Kubernetes version (use ): latest Cloud provider or hardware configuration: OS (e.g: ): all Kernel (e.g. ): all Install tools: Network plugin and version (if this is a network-related bug): Others:
/assign
/sig testing
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten this issue still stands.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/lifecycle frozen
/remove-lifecycle rotten
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
triage/accepted
/triage accepted

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: some of base images support multi-arch, if no platform specified, the image pulled will actually reflect where the operation is performed, current implementation always assume the default platform is amd64 which make the tags is not correctly defined. for example, on arm64 platform, here is what I got after build the image of ipc-utils, What you expected to happen: tag should be defined correctly How to reproduce it (as minimally and precisely as possible): on arm64 platform, Anything else we need to know?: Environment: Kubernetes version (use ): latest Cloud provider or hardware configuration: all arch except x86 OS (e.g: ): all Kernel (e.g. ): all Install tools: Network plugin and version (if this is a network-related bug): Others:
/assign
/sig testing release
This is the side effect of command, you can fix it by specifying tag while building the image.
The tag is still experimental and introduced in so I really doubt about using this anywhere, I feel it is just a cosmetic issue and only applies for the images with tag. As we are pushing fat manifest images with proper archs so we don't have to worry much about it.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with .

Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/lifecycle frozen
/remove-lifecycle rotten
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available .

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage accepted


Question: invalidPredicates.Insert("MatchInterPodAffinity") } // NoVolumeZoneConflict will only be affected by zone related label change <del> if k == kubeletapis.LabelZoneFailureDomain || k == kubeletapis.LabelZoneRegion { </del> <ins> if isZoneRegionLabel(k) { </ins> if v != newNode.GetLabels()[k] { invalidPredicates.Insert("NoVolumeZoneConflict") }

What happened: When cleanup the existing equivalence class code. I noticed there may be duplication in handlers of Pod, PV, PVC change. For now we will invalidate volume related predicates for all nodes on PV and PVC add. Can we assume that volumes will only be attached to Node during Pod add? So we can just deal with those predicates on Pod add/delete. Don't need to bother PV/PVC at all. Any comments? Anything else we need to know?: Part of Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
Sorry, I'm really busy this week, but will think about this next week. Please ping me if I forget.
Let me think about this one predicate at a time: Volume zone predicate doesn't exist - predicate will fail, not cached a. PV doesn't exist - if you're using the new volume scheduling feature, predicate will pass. I think it is ok to ignore this predicate if you're using that feature, because your volume should be using VolumeBind predicate instead. So nothing to do here. b. PV doesn't exist - normal scenario, predicate will fail, not cached was modified to change its zone/region labels. This should not happen in normal operation, but theoretically it's possible for the user to change it. The predicate should be invalidated in this case (if we want to support it). cannot modify its VolumeName once its set. So we don't have to worry about this. /PV previously existed, but is deleted. We should start failing the predicate again. Summary: Volume zone predicate should be invalidated if: PV update modifies the zone/region labels. PVC, PV is deleted
Volume count predicate: /PV doesn't exist - predicate will succeed, maybe even incorrectly (ie PV is not one of the counted types, like NFS). Ideally, we want the count to become more accurate when the error condition is fixed, so the predicate should be invalidated. not bound - same as above volume type cannot be modified. Don't need to worry about this. /PV previously existed, but is deleted. I think this is ok to not invalidate the predicate in this case, because it would only increase the count (and the count would become less accurate).

is deleted from a node. Predicate should be invalidated for that node if it uses volumes, because it will impact the node volume count. is . Predicate should be invalidated because the node volume count may have increased from the last time we scheduled a pod. Summary: Volume count predicate should be invalidated on: PVC/PV add PVC modify volumeName Pod add if it uses volumes Pod delete for node
Volume bind predicate: PVC doesn't exist: predicate will fail PV add/delete can cause volume selection to choose something different PV modify node affinity can affect volume selection or already bound node selection PVC not bound will trigger PV selection. Once it's bound, then it should be invalidated because the other nodes should fail instead of succeed. PVC/PV deletion can cause predicate to fail when it previously succeeded Summary: Volume bind predicate should be invalidated on: PVC modify volumeName' PVC delete PV add/delete PV modify node affinity
I think in general, we need to have separate invalidation handlers for PVC and PV events because they can occur independently of Pod events. For example, the volume predicates could succeed and get cached, but because of some other unrelated predicate, the Pod won't get scheduled. In the meantime, PVC/PV events could happen that may change the results of the volume predicates.
Thank you very much for such a detailed explanation, it's really appreciated! I am writhing up a patch to fix volume related predicates based on this thread. Will ping you there very soon.


Question: # Set artifacts directory export ARTIFACTS_DIR=${WORKSPACE}/artifacts <del> retry go get github.com/tools/godep && godep version </del> export LOG_LEVEL=4 cd /go/src/k8s.io/kubernetes

Didn't see an issue for this. Godep recently updated its version to v80. This broke several builds that directly godep, grabbing latest. /kind bug cc Example https://k8s- Here are some examples where we grab master:
We have helpers to grab godep. We should use them everywhere.
/cc
/assign


Question: <del> ## Contributor guidelines 1. Please read our [contributor guidelines](https://github.com/kubernetes/kubernetes/blob/master/CONTRIBUTING.md). 1. See our [developer guide](https://github.com/kubernetes/kubernetes/blob/master/docs/devel/development.md). 1. Follow the instructions for [labeling and writing a release note for this PR](https://github.com/kubernetes/kubernetes/blob/master/docs/devel/pull-requests.md#release-notes) in the block below. (_this would also contain updated detail later about this new template_) ```release-note * Use the release-note-* labels to set the release note state * Clear this block to use the PR title as the release note -OR- * Enter your extended release note here ``` </del>

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Using example for from documentation failed with: What you expected to happen: Example from documentation works. How to reproduce it (as minimally and precisely as possible): How should configuration look like for ? Environment: Kubernetes version (use ):OS (e.g: ):Kernel (e.g. ):
/sig scheduling
/triage accepted oh, I missed a dash. It should be: Would you mind sending a PR to kubernetes/website?


Question: { Name:    "busybox-container", Image:   "gcr.io/google_containers/busybox:1.24", <del> Command: []string{"sh", "-c", "ping -c 1 google.com; while true; do echo 'hello world' | tee /test-empty-dir-mnt/file ; sleep 1; done"}, </del> <ins> Command: []string{"sh", "-c", "ping -c 1 google.com; while true; do echo 'hello world' >> /test-empty-dir-mnt/file ; sleep 1; done"}, </ins> Resources: api.ResourceRequirements{ Limits: api.ResourceList{ // Must set memory limit to get MemoryStats.AvailableBytes

https://k8s- Failed: [] Summary API when querying /stats/summary should report resource usage through the stats api {E2eNode Suite}
The test failed because the busybox container was crash-looping, but I'm not sure why.
https://k8s- Failed: [] Summary API when querying /stats/summary should report resource usage through the stats api {E2eNode Suite}
https://k8s- Failed: [] Summary API when querying /stats/summary should report resource usage through the stats api {E2eNode Suite}
https://k8s- Failed: [] Summary API when querying /stats/summary should report resource usage through the stats api {E2eNode Suite}
https://k8s- Failed: [] Summary API when querying /stats/summary should report resource usage through the stats api {E2eNode Suite}
https://k8s- Failed: [] Summary API when querying /stats/summary should report resource usage through the stats api {E2eNode Suite}
https://k8s- Failed: [] Summary API when querying /stats/summary should report resource usage through the stats api {E2eNode Suite}
https://k8s- Failed: [] Summary API when querying /stats/summary should report resource usage through the stats api {E2eNode Suite}
- Any ideas what changed in the past week that would cause the Kubelet usage/working set memory usage to suddenly increase? It may be a non-issue, but it's concerning that this seems to have recently changed. - This is only happening on GCI. Did we change the GCI image being tested recently?
https://k8s- Failed: [] Summary API when querying /stats/summary should report resource usage through the stats api {E2eNode Suite}
Last image version update for testing was 11 days ago () though we are soon going to update it again ().
Since we increased the bound on (), we should also increase the bound on ( is usage - total inactive file). I think this may be the cause of this flake.
I agree that the bounds should match, but the cause of both flakes (why usage & workingset suddenly increased) is still unknown.
Usage workingset so it may not be necessary to bump both ;) I think this may be the caused by the gci mounter.

/cc
https://k8s- Failed: [] Summary API when querying /stats/summary should report resource usage through the stats api {E2eNode Suite}
https://k8s- Failed: [] Summary API when querying /stats/summary should report resource usage through the stats api {E2eNode Suite}
I checked the log from the last failure, didn't see anything related to gci mounter. The test failed on trusty node which mounter should not be used at all.
Yes, the latest failures are just a flake introduced by The bounds need to be adjusted for the misc container. I'll send a PR soon.


Question: // of volume creation. epServiceName := dynamicEpSvcPrefix + p.options.PVC.Name epNamespace := p.options.PVC.Namespace <del> endpoint, service, err := p.plugin.createEndpointService(epNamespace, epServiceName, dynamicHostIps, p.options.PVC.Name) </del> <ins> endpoint, service, err := p.createEndpointService(epNamespace, epServiceName, dynamicHostIps, p.options.PVC.Name) </ins> if err != nil { glog.Errorf("glusterfs: failed to create endpoint/service: %v", err) deleteErr := cli.VolumeDelete(volume.Id)

several operations to the glusterfs mount operation that nodes are not permitted to perform: looks up arbitrary secrets based on storage class parameters. this will not succeed, as the node authorizer will only allow nodes access to secrets referenced by pods or persistent volumes used by the node attempts to create a service and endpoint using the node's API client. nodes do not have permission to create arbitrary endpoints and services. For both of those reasons, that recreate operation will fail when attempted from the node. The recreate operation should probably move to one of the storage-related controllers (provisioner or attach/detach) cc
Marking this with 1.7 milestone until we have a better idea of whether this is blocking or not for the 1.7 release.
CC Is the endpoint originally created by the user or by the volume plugin creating/deleting Endpoint objects? In addition to the concerns raised, having a volume create new API objects seems like a bad idea (abstraction layer violation?).
From on Slack: Unless there is justification otherwise, I will roll back the PR on Monday.
cc/
revert button doesn't work, have to do it manually
Revert PR is approved and awaiting merge (). Once that merges this issue can be closed.
Opened an E2E pr to catch this kind of behavior in the future. WIP, still exploring a method to reproduce and catch the behavior.


Question: <ins> #!/bin/bash # Copyright 2014 Google Inc. All rights reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # #     http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # This command builds and runs a local kubernetes cluster. It's just like # local-up.sh, but this one launches the three separate binaries. # You may need to run this as root to allow kubelet to open docker's socket. if [ "$(which etcd)" == "" ]; then echo "etcd must be in your PATH" exit 1 fi # Stop right away if the build fails set -e $(dirname $0)/build-go.sh echo "Starting etcd" ETCD_DIR=$(mktemp -d -t kube-integration.XXXXXX) trap "rm -rf ${ETCD_DIR}" EXIT etcd -name test -data-dir ${ETCD_DIR} &> /tmp/etcd.log & ETCD_PID=$! sleep 5 # Shut down anyway if there's an error. set +e API_PORT=8080 KUBELET_PORT=10250 $(dirname $0)/../output/go/apiserver  --address="127.0.0.1"  --port="${API_PORT}"  --etcd_servers="http://127.0.0.1:4001"  --machines="127.0.0.1" &> /tmp/apiserver.log & APISERVER_PID=$! $(dirname $0)/../output/go/controller-manager  --etcd_servers="http://127.0.0.1:4001"  --master="127.0.0.1:${API_PORT}" &> /tmp/controller-manager.log & CTLRMGR_PID=$! $(dirname $0)/../output/go/kubelet  --etcd_servers="http://127.0.0.1:4001"  --hostname_override="127.0.0.1"  --address="127.0.0.1"  --port="$KUBELET_PORT" &> /tmp/kubelet.log & KUBELET_PID=$! echo "Local Kubernetes cluster is running. Press enter to shut it down." read unused kill ${APISERVER_PID} kill ${CTLRMGR_PID} kill ${KUBELET_PID} kill ${ETCD_PID} </ins>

It would be great to have an easy way to run a kubernetes master/minion locally. Currently, it's not that easy to set up. We should have a convenient script to make it work.
The integration test binary is pretty close. It'd be pretty simple to glue in the kubelet to that binary. Brendan On Jun 10, 2014 2:53 PM, "Daniel Smith" wrote:
I'll take a look.
+1 for this -- especially in script form rather than a go executable.
what exactly do you mean by "script form"? Were you thinking of something like a version of cloudcfg that can be run without needing to have started up a daemon first?
Basically, what I had in mind, is a shell script that looks something like
Ah, ok. That's very similar to local-; the only difference is that everything is packaged into a single go binary. We probably will switch to using individual binaries for this when we start running our components inside docker containers.
The script just makes things more obvious how to run things for those who are looking to integrate them into non-GCE environments.
That's a good point. I'll look into this some more later.
Create a pad as follows: Name: non-persistent-redis Container image: redis Named-volum with name: cache-control Mount path: /data/redis It should launch in the pre-prod namespace and the volume MUST NOT be persistent. apiVersion: v1 kind: Pod metadata: name: non-persistent-redis namespace: pre-prod spec: containers: name: non-persistent-redis image: redis volumeMounts: name: cache-control mountPath: "/data/redis" volumes: name: cache-control emptyDir: {}

I cannot find such a service when . Same thing applies for . My cluster runs fine. cc:
Ah, good catch... everything is run as a container now, will update the documentation on where to get the logs.

When I'm accessing persistentVolumes and persistentVolumeClaims, I'm getting nil instead of empty list of items, as happening in other entities when there are no items. { "kind": "PersistentVolumeList", "apiVersion": "v1", "metadata": { "selfLink": "/api/v1/persistentvolumes", "resourceVersion": "" } }
Yes, this seems like a bug. I see an empty list when I access pods: And items is nil when I access persistent volumes:
Items shouldn't be tagged with omitempty:
cc
PersistentVolumeList and PersistentVolumeClaimList appear to be the only resource types with this problem.
Please see for fix.


Question: // attributes. func FsUserFrom(pod *v1.Pod) *int64 { var fsUser *int64 <del> // Exclude ephemeral containers because SecurityContext is not allowed. podutil.VisitContainers(&pod.Spec, podutil.InitContainers|podutil.Containers, func(container *v1.Container, containerType podutil.ContainerType) bool { </del> <ins> podutil.VisitContainers(&pod.Spec, podutil.AllFeatureEnabledContainers(), func(container *v1.Container, containerType podutil.ContainerType) bool { </ins> runAsUser, ok := securitycontext.DetermineEffectiveRunAsUser(pod, container) // One container doesn't specify user or there are more than one // non-root UIDs.

projected volumes exclude ephemeral containers in calculating fs user because "SecurityContext is not allowed" : This is no longer technically correct -- ephemeral containers are allowed to specify security context -- but it's a no-op because is guaranteed to be empty at pod creation when fs user is calculated. Still, the comment is wrong so let's clean it up to avoid confusion. /kind cleanup
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage /assign
/priority backlog
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue with Mark this issue as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close not-planned [1]:
Closing this issue, marking it as "Not Planned". <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue with Mark this issue as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available .

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-lifecycle rotten


Question: } } <del> // Regularly send heartbeat event to APIServer so that the cluster doesn't enter fullyDisruption mode. // TODO(Huang-Wei): use "NodeDisruptionExclusion" feature to simply the below logic when it's beta. var heartbeatChans []chan struct{} for i := 0; i < nodeCount; i++ { heartbeatChans = append(heartbeatChans, make(chan struct{})) } for i := 0; i < nodeCount; i++ { // Spin up <nodeCount> goroutines to send heartbeat event to APIServer periodically. go func(i int) { for { select { case <-heartbeatChans[i]: return case <-time.Tick(2 * time.Second): nodes[i].Status.Conditions = []v1.NodeCondition{ { Type:              v1.NodeReady, Status:            v1.ConditionTrue, LastHeartbeatTime: metav1.Now(), }, } updateNodeStatus(cs, nodes[i]) } } }(i) } </del> neededNode := nodes[1] if test.pod != nil { test.pod.Name = fmt.Sprintf("testpod-%d", i)

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: Which test(s) are failing: TaintBasedEvictions Since when has it been failing: probably since it was in this PR? Testgrid link: Reason for failure: Anything else we need to know: /sig scheduling /kind flake /priority important-soon /milestone v1.17
You must be a member of the GitHub team to set the milestone. If you believe you should be able to issue the /milestone command, please contact your and have them propose you as an additional delegate for this responsibility. <detailsIn response to : <!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs -- TaintBasedEvictions probably since it was in this PR? Anything else we need to know: /sig scheduling /kind flake /priority important-soon /milestone v1.17 </details
cc.
Looking into this. We moved it to integration because it was flaking in e2e too
/assign damemi
Not sure yet, but I do see this in the logs of the failing tests: which, if I understand this, would remove all the taints and mark nodes as reachable (which would explain why the test is failing waiting for node taints: ). But I need to figure out why all the nodes are becoming unavailable
is that a full fix for this flake or just partial? I still saw it failed yesterday so just checking... :)
I think I also run into this on 13th Oct: Full logs:
Now gets merged, which might fix the flake. Let's keep an eye on this.
reported it still flaked: Can you continue looking into this?
I'll keep trying, from my previous debugging it seems that in some cases the test cluster is entering full disruption mode and removing all of the taints from the nodes. I know we specifically made changes to the test (using multiple nodes) to ensure it wouldn't become fully disrupted, do you have any idea what could still be causing that?
There is some flag controlling how the dispution thing is managed internally, maybe you can check how a node lifecycle manager is constructed.

The test still flakes after we merge , we still need to investigate the cause ,
/assign
Another flake:
These flakes are consistently (always) timeouts waiting on node taints:
I'm working on this. Will send out a PR soon.
/reopen We see another flake since got merged: However, don't panic :) The original "Entering master disruption mode" cause has been fixed - you won't see that message in the above log. This time it's a smaller issue. I will explain it in another PR.
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen The job is still flaky
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/close But for another test :)
Closing this issue. <detailsIn response to : /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: // the specified zone. It stores specified tags encoded in JSON in Description // field. func (gce *GCECloud) CreateDisk(name string, diskType string, zone string, sizeGb int64, tags map[string]string) error { <ins> // Do not allow creation of PDs in zones that are not managed. Such PDs // then cannot be deleted by DeleteDisk. isManaged := false for _, managedZone := range gce.managedZones { if zone == managedZone { isManaged = true break } } if !isManaged { return fmt.Errorf("kubernetes does not manage zone %q", zone) } </ins> tagsStr, err := gce.encodeDiskTags(tags) if err != nil { return err

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"4+", GitVersion:"v1.4.0-alpha.3.317+", GitCommit:"", GitTreeState:"clean", BuildDate:"2016-09-01T05:48:36Z", GoVersion:"go1.7", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"4+", GitVersion:"v1.4.0-alpha.3.317+", GitCommit:"", GitTreeState:"clean", BuildDate:"2016-09-01T05:48:36Z", GoVersion:"go1.7", Compiler:"gc", Platform:"linux/amd64"} What happened: Dynamic provisioned volumes in different zones with master failed to deleted What you expected to happen: Since the volume can be dynamic provisioned when the zones differs from master, then it should be deleted after delete the PVC which created them. How to reproduce it (as minimally and precisely as possible): up an all-in-one cluster. The instance is created in zone us-central1-a. several storageClass. One in zone us-central1-a. Another in zone us-central1-b. several PVC. pvc1 using storageClass1(zone us-central1-a), pvc2 using storageClass2(zone us-central1-b). PV/volume were provisioned in proper size/mode/zone. PVCs. with volume in zone us-central1-b failed with error can not find the volume.

Anything else do we need to know: Name: pvc--70d1-11e6-b17e- Labels: failure- failure- Status: Failed Claim: default/pvc5 Reclaim Policy: Delete Access Modes: RWO Capacity: 4Gi Message: Delete of volume "pvc--70d1-11e6-b17e-" failed: GCE persistent disk not found: "kubernetes-dynamic-pvc--70d1-11e6-b17e-" Source: Type: GCEPersistentDisk (a Persistent Disk resource in Google Compute Engine) PDName: kubernetes-dynamic-pvc--70d1-11e6-b17e- FSType: Partition: 0 ReadOnly: false Events: FirstSeen LastSeen Count From SubobjectPath Type Reason Message 11m 11m 1 {persistentvolume-controller } Warning VolumeFailedDelete Delete of volume "pvc--70d1-11e6-b17e-" failed: GCE persistent disk not found: "kubernetes-dynamic-pvc--70d1-11e6-b17e-" creationTimestamp: '2016-09-01T22:49:26.439-07:00' description: '{"":"pvc--70d1-11e6-b17e-","":"pvc5","":"default"}' id: '' kind: compute#disk name: kubernetes-dynamic-pvc--70d1-11e6-b17e- selfLink: sizeGb: '4' status: READY type: zone:
Kubernetes can delete GCE PDs only in zones it manages. I.e. either in the same zone as the master or, when multizone is enabled in GCE config, in all zones in the same region. This should not be allowed in the first place. Kubernetes should not create disks outside zones it manages and report an error instead. So... please use following , it will be allow creation/deletion of disks in all zones in us-central1 region: And I'll fix Kubernetes to report an error when creating a disk in all other zones.


Question: ## Version 1.6 (Wed June 29 2016 Girish Kalele <gkalele@google.com>) - Godeps update for vendor code (skydns/mux) <ins> ## Version 1.7 (Wed August 24 2016 Zihong Zheng <zihongz@google.com>) - Add support for ExternalName services (pr #31159) </ins>

Now that merged, we need to build a new kube-dns release.
I just put in a PR yesterday to fix the missing changelog entry for kube-dns:1.6


Question: http_archive( name = "io_bazel_rules_go", <del> sha256 = "f9cc31e9d66dad1154de3c214158916411040e2c2614e4fbc2bab67330e95f82", </del> <ins> sha256 = "27a13726ff8621cfa4592fbef116b253043c2b093c8814265dc4489a0e67d229", </ins> strip_prefix = "rules_go-82483596ec203eb9c1849937636f4cbed83733eb", urls = ["https://github.com/bazelbuild/rules_go/archive/82483596ec203eb9c1849937636f4cbed83733eb.tar.gz"], )

I don't see an existing issue, so opening one for tracking. PR builds are currently failing due to the SHA of a remote archive changing. ... $ sha256sum 27a13726ff8621cfa4592fbef116b253043c2b093c8814265dc4489a0e67d229 cc cc cc
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/release: Issue will be escalated to these SIGs if needed. priority/critical-urgent: Never automatically move out of a release milestone; continually escalate to contributor and SIG through all available channels. kind/bug: Fixes a bug discovered during the current release. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details
cc
/assign We can probably just bump the sha, but I'm trying to figure out what changed...
I found a copy of the which matched the sha256sum (thanks There's no diff when I compare the extracted trees. There's also no diff if I compare the output on each. The new and old gzip'd tarballs are slightly different sizes (old bytes, new bytes). If I ungzip them, they are the same size ( bytes) but they still have different sha256sums. My best guess at this point is that there was some library change recently which is creating slightly different versions of these tarballs, but I don't know how it's different. I also don't know why the archive tarballs for other repos don't seem to have changed.
go 1.8 vs 1.9 gzip library maybe?
It seems like the order of some files in the tarball may have changed.
Yeah, we're not alone:
I guess we should start mirroring these archives somewhere rather than depending on GitHub.


Question: if client.IsUnexpectedStatusError(err) { glog.FatalDepth(1, fmt.Sprintf("Unexpected status received from server: %s", err.Error())) } <del> glog.FatalDepth(1, fmt.Sprintf("Error: %s", err.Error())) </del> <ins> glog.FatalDepth(1, fmt.Sprintf("Client error: %s", err.Error())) </ins> } }

On a cluster running 0.13.2, I ran , picked out the name of a pod that only contains a single container, let's call it foo, and then ran . It prints nothing other than the message "Unexpected path for command running". If I add on the container name (), then it prints nothing, presumably because the pod has no logs. The docs for kubectl log say that the container name is optional if the pod only has one container, so I wouldn't expect this response.
+1, we're having to work around this on guides


Question: defer cancel() vsi, err := vs.getVSphereInstance(nodeName) if err != nil { <ins> // If node doesn't exist, disk is already detached from node. if err == vclib.ErrNoVMFound { glog.Infof("Node %q does not exist, disk %s is already detached from node.", convertToString(nodeName), volPath) return nil } </ins> return err } // Ensure client is logged in and session is valid

/kind bug What happened: We had a pod with persistent storage. We drained the node. The node VM was deleted. The PV was not attached to the node that the Pod migrated too. The kube-controller-manager logs contained the following. What you expected to happen: The PV can be attached to the node it was migrated to. How to reproduce it (as minimally and precisely as possible): Create a 2 node cluster on vSphere Deploy a pod with persistent storage attached Drain the node Delete the node VM immediately after draining Anything else we need to know?: The issue seems to be in the call to getVSphereInstance. This will return an error if the Node doesn't exist. Precluding the logic on line 804-813 from being executed. A restart of the kube-controller-manager resolved the issue. The following e2e test is not triggered in vSphere environments - Which might expose the issue. Environment: Kubernetes version (use ): 1.9.5 Cloud provider or hardware configuration: vSphere OS (e.g. from /etc/os-release): ubuntu Kernel (e.g. ): Install tools: CFCR Others: /sig storage /sig vmware
More logs:
/assign


Question: return nil, errors.New("must specify at least one ExtKeyUsage") } <ins> RemoveDuplicateAltNames(&cfg.AltNames) </ins> certTmpl := x509.Certificate{ Subject: pkix.Name{ CommonName:   cfg.CommonName,

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Duplicate altnames in certs which generated by kubeadm ! What you expected to happen: No Duplicate altnames in certs which generated by kubeadm How to reproduce it (as minimally and precisely as possible): Anytime Anything else we need to know?: ! Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig cluster-lifecycle
/area kubeadm
Can you describe your detailed usage scenarios and reproduction steps, and explain the kubernetes version so that we can quickly locate
Thank for your response! I have found why and will fix it soon.
i have commented on the PR. /priority awaiting-more-evidence
/remove-priority awaiting-more-evidence /priority backlog


Question: Expect(err).NotTo(HaveOccurred()) Expect(tolePod.Spec.NodeName).To(Equal(nodeName)) }) <del> It("Pod should be preferably scheduled to nodes which satisfy its limits", func() { if !utilfeature.DefaultFeatureGate.Enabled(features.ResourceLimitsPriorityFunction) { framework.Skipf("ResourceLimits Priority function is not enabled, so skipping this test") } var podwithLargeRequestedResource *v1.ResourceRequirements = &v1.ResourceRequirements{ Requests: v1.ResourceList{ v1.ResourceMemory: resource.MustParse("100Mi"), v1.ResourceCPU:    resource.MustParse("100m"), }, Limits: v1.ResourceList{ v1.ResourceMemory: resource.MustParse("3000Mi"), v1.ResourceCPU:    resource.MustParse("5000m"), }, } // Update one node to have large allocatable. lastNode := nodeList.Items[len(nodeList.Items)-1] nodeName := lastNode.Name nodeOriginalMemory, found := lastNode.Status.Allocatable[v1.ResourceMemory] nodeOriginalCPU, found := lastNode.Status.Allocatable[v1.ResourceCPU] nodeOriginalCPUCapacity, found := lastNode.Status.Capacity[v1.ResourceCPU] nodeOriginalMemoryCapacity, found := lastNode.Status.Capacity[v1.ResourceMemory] Expect(found).To(Equal(true)) nodeOriginalMemoryVal := nodeOriginalMemory.Value() nodeOriginalCPUVal := nodeOriginalCPU.MilliValue() nodeOriginalMemoryCapacityVal := nodeOriginalMemoryCapacity.Value() nodeOriginalCPUCapacityVal := nodeOriginalCPUCapacity.MilliValue() err := updateNodeResources(cs, nodeName, int64(10737418240), int64(12000), int64(10737418240), int64(12000)) Expect(err).NotTo(HaveOccurred()) defer func() { // Resize the node back to its original allocatable values. if err := updateNodeResources(cs, nodeName, nodeOriginalMemoryVal, nodeOriginalCPUVal, nodeOriginalMemoryCapacityVal, nodeOriginalCPUCapacityVal); err != nil { framework.Logf("Failed to revert node resources with %v", err) } // Reset the node list with its old entry. nodeList.Items[len(nodeList.Items)-1] = lastNode }() // update nodeList with newNode. newNode, err := cs.CoreV1().Nodes().Get(nodeName, metav1.GetOptions{}) if err != nil { framework.Logf("Failed to get node", err) } nodeList.Items[len(nodeList.Items)-1] = *newNode err = createBalancedPodForNodes(f, cs, ns, nodeList.Items, podRequestedResource, 0.5) framework.ExpectNoError(err) // After the above we should see 50% of node to be available which is 5000MiB memory, 6000m cpu for large node. By("Create a pod with unusual large limits") podWithLargeLimits := "with-large-limits" pod := createPausePod(f, pausePodConfig{ Name:      podWithLargeLimits, Resources: podwithLargeRequestedResource, }) framework.ExpectNoError(f.WaitForPodRunning(pod.Name)) By("Pod should preferably scheduled to nodes which satisfy its limits") // The pod should land onto large node(which has 5000MiB free) which satisfies the pod limits which is 3000MiB. podHighLimits, err := cs.CoreV1().Pods(ns).Get(podWithLargeLimits, metav1.GetOptions{}) Expect(err).NotTo(HaveOccurred()) Expect(podHighLimits.Spec.NodeName).To(Equal(nodeName)) }) </del> }) // createBalancedPodForNodes creates a pod per node that asks for enough resources to make all nodes have the same mem/cpu usage ratio.

Testgrid: https://k8s- https://k8s- Example: https://k8s- This seems to be related to /sig scheduling /kind failing-test /priority important-soon /cc
/assign RaviSantosh Gudimetla
GitHub didn't allow me to assign the following users: RaviSantosh, Gudimetla. Note that only and repo collaborators can be assigned. For more information please see <detailsIn response to : /assign RaviSantosh Gudimetla Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
Looked at the kube-scheduler logs for GCE, seems the feature_gate was enabled. I don't see enabled, will setup the cluster locally to test if this is the issue.
have you been able to look into this? These tests are broken right now for unrelated reasons, but once they get fixed I expect to keep seeing the scheduler failure.
/milestone v1.13
is the one open PR expected to fix the remaining failures we are seeing the release dashboards? If not fixed by this week, these failures will block Beta cut targeted for 11/6.
- Apologies for the delay. Yes, this PR should fix the failure, if not, I will disable this test for now and revert the feature promotion PR associated with this test. Also, I need to rethink, if e2e is an appropriate test scenario to handle this feature or should unit tests/integration suffice. I could not find a test which does something like resizing a node in the existing e2e test suite.
I'm looking at this, and I'm not sure this is going to fix the failures. I'm looking at the tests, and it seems to be setting the node status capacity/allocatable to pretend to be a 12 core machine, and then it tries to allocate a pod on every node such that every node is 50% utilized. It thus tries to allocate a pod of ~ 6 cores on the resized node. But that node still only has 2 cores, so it fails to schedule: I think that's why the test has been failing, at least for the past few runs which I checked.

What I have yet to figure out is how this passed here...
is it safe to rephrase this as a pure test design issue then?
the test still failed on the last . Even though it was a couple of hours after PR merged, I dont see it in the delta changes . I will watch for the next run result. Assuming it still doesnt pass (as pointed out about) we need to figure out what this e2e test failure means and should it block 1.13 Beta0 for next Tuesday. Considering the test did pass very few times since its introduction, this seems like an issue with the test design itself and not the feature. Do you have other tests (unit or integration or different approach to e2e) that can infirm us on the health of thew feature? If those exist and are green we can unblock Beta on those results while you determine the right way to author the e2e. Barring those being present right idea, I think we will have to err on the side of caution and roll back the feature itself before Beta. Either way we need to take a call and submit relevant PRs by tomorrow so we get green CI over the weekend, in time for Beta cut on Tuesday.
This is not a blocker, as I told earlier as well, e2e may not be good avenue for this test. I actually thought more about this problem last day. The problem is I updated the node capacity and I was checking if the node could handle the pod based on the scheduler cache which we get from kubelet. But kubelet on the other side is updating it's node information(like CPU, memory etc) in the API server there by causing a race condition. So, it could pass with a very little chance. So, I will remove this test from e2e. Yes, we have unit tests, that help in validating the feature. I will try to get an integration test but this is not a blocker for feature.
Thanks for the explanation and analysis do you plan on adding more e2e tests later or stick to unit and integration for validating this feature? and to weigh in on the path forward i.e., remove e2e tests for this feature and rely on unit and integration.
- We discussed internally in the sig meeting yesterday and it was decided to have an integration test instead of e2e test.

I plan to add an integration test(which we don't have today) later.
since this PR is actually tracking an enhancement work graduating to Beta in 1.13, it should have been tracked via a corresponding issue in k/enhancements repo in 1.13 timeframe. We are now past Feature freeze for 1.13 and its late to open an issue against it. Also we are missing integration tests for this feature, which we are . Given the above two points and from offline discussions with SIG leads regarding the criticality of this feature going to Beta now vs in 1.14 timeframe, I am asking for it to be punted to 1.14. As AIs Please prepare a PR to remove it from Beta in 1.13 Please open a tracking issue in for 1.14 milestone.


Question: return } <ins> // A Pod delete event followed by an immediate Pod add event may be merged // into a Pod update event. In this case, we should invalidate the old Pod, and // then add the new Pod. if oldPod.UID != newPod.UID { sched.deletePodFromCache(oldObj) sched.addPodToCache(newObj) return } </ins> // NOTE: Updates must be written to scheduler cache before invalidating // equivalence cache, because we could snapshot equivalence cache after the // invalidation and then snapshot the cache itself. If the cache is

What happened: pod can't be removed from scheduler cache when be deleted, logs in kube-scheduler is below: I1128 16:41:50. ] lock is held by shaphicpra1490764ccb0c5-aaa1-435d-a4b4- and has not yet expired I1128 16:41:50. ] failed to acquire lease kube-system/kube-scheduler I1128 16:41:53. ] lock is held by shaphicpra1490764ccb0c5-aaa1-435d-a4b4- and has not yet expired I1128 16:41:53. ] failed to acquire lease kube-system/kube-scheduler I1128 16:41:57. ] successfully acquired lease kube-system/kube-scheduler I1128 16:41:57. ] caches populated E1128 16:42:11. ] scheduler cache UpdatePod failed: pod -decb-4abd-a1d3- is not to scheduler cache, so cannot be updated E1128 16:42:12. ] scheduler cache UpdatePod failed: pod -decb-4abd-a1d3- is not to scheduler cache, so cannot be updated E1128 16:42:12. ] scheduler cache UpdatePod failed: pod -decb-4abd-a1d3- is not to scheduler cache, so cannot be updated E1128 16:42:20. ] scheduler cache UpdatePod failed: pod -decb-4abd-a1d3- is not to scheduler cache, so cannot be updated E1128 16:42:20. ] scheduler cache RemovePod failed: pod -decb-4abd-a1d3- is not found in scheduler cache, so cannot be removed from it What you expected to happen: pod in scheduler cache can be deleted successfully when pod is deleted How to reproduce it (as minimally and precisely as possible): main steps to reproduce this bug is these: scheduler run in leader-elect=true modeschedulerA is the leader scheduler, and schedulerB is an assistant.

pod1 with "namespace1/name1" and it runs on Node1 down the network between schedulerB and kube-apiserver pod1 and create pod2 with same name, ie "namespace1/name1" and pod2 been assigned to Node1 by schedulerA again. off schedulerA and recover the network between schedulerB and kube-apiserver, and then schedulerB become the leader scheduler. pod2. this time log in schedulerB will print this: E1128 16:42:11. ] scheduler cache UpdatePod failed: pod -decb-4abd-a1d3- is not to scheduler cache, so cannot be updated E1128 16:42:20. ] scheduler cache RemovePod failed: pod -decb-4abd-a1d3- is not found in scheduler cache, so cannot be removed from it this time, if I create a new pod3 with podAntiAffinity to pod1. the result is pod3 can't been assigned to Node1 because pod1 is still in schedulerB's scheduler cache. Anything else we need to know?: the reason of this bug is below: since PR(), podKey in scheduler cache use pod's UID instead of namespace/name however, the key in SharedIndexInformer still use namespace/name in this upper case, delete the pod and create a same name pod may be delivered as one update notification.( see disscusion ) scheduler received pod update notification and then print below log. E1128 16:42:11. ]

scheduler cache UpdatePod failed: pod -decb-4abd-a1d3- is not to scheduler cache, so cannot be updated Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-06-19T16:40:16Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-06-19T16:40:16Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig release
/sig scheduling
Since has been marked resolved (fix available in 1.17), does that mean this issue is expected to go away in 1.17? Could you please give a try?
, fix for just resolved the case add event may be lost after deletion event. It doesn't solve the problem that a delete followed by an add may be delivered as one update notification and it's thought normal from MikeSpreitzer's comment. The main cause of this problem is scheduler cache use Pod UID as key while sharedInformer use namespace/name. Still, I also test with the latest kubernetes code. The bug of this issue reappeared.
Thanks I a comment in the PR, and a test is necessary to reproduce the issue and verify the fix.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale ping there is an unresolved comment in the PR, do you have a chance to take a look?


Question: } } <ins> type vaTest struct { testName          string volName           string podName           string podNodeName       string pvName            string vaName            string vaNodeName        string vaAttachStatus    bool expected_attaches map[string][]string expected_detaches map[string][]string } func Test_ADC_VolumeAttachmentRecovery(t *testing.T) { for _, tc := range []vaTest{ { // pod is scheduled testName:          "Scheduled pod", volName:           "vol1", podName:           "pod1", podNodeName:       "mynode-1", pvName:            "pv1", vaName:            "va1", vaNodeName:        "mynode-1", vaAttachStatus:    false, expected_attaches: map[string][]string{"mynode-1": {"vol1"}}, expected_detaches: map[string][]string{}, }, { // pod is deleted, attach status:true, verify dangling volume is detached testName:          "VA status is attached", volName:           "vol1", pvName:            "pv1", vaName:            "va1", vaNodeName:        "mynode-1", vaAttachStatus:    true, expected_attaches: map[string][]string{}, expected_detaches: map[string][]string{"mynode-1": {"vol1"}}, }, { // pod is deleted, attach status:false, verify dangling volume is detached testName:          "VA status is unattached", volName:           "vol1", pvName:            "pv1", vaName:            "va1", vaNodeName:        "mynode-1", vaAttachStatus:    false, expected_attaches: map[string][]string{}, expected_detaches: map[string][]string{"mynode-1": {"vol1"}}, }, } { t.Run(tc.testName, func(t *testing.T) { volumeAttachmentRecoveryTestCase(t, tc) }) } } func volumeAttachmentRecoveryTestCase(t *testing.T, tc vaTest) { fakeKubeClient := controllervolumetesting.CreateTestClient() informerFactory := informers.NewSharedInformerFactory(fakeKubeClient, time.Second*1) plugins := controllervolumetesting.CreateTestPlugin() nodeInformer := informerFactory.Core().V1().Nodes().Informer() podInformer := informerFactory.Core().V1().Pods().Informer() pvInformer := informerFactory.Core().V1().PersistentVolumes().Informer() vaInformer := informerFactory.Storage().V1().VolumeAttachments().Informer() // Create the controller adcObj, err := NewAttachDetachController( fakeKubeClient, informerFactory.Core().V1().Pods(), informerFactory.Core().V1().Nodes(), informerFactory.Core().V1().PersistentVolumeClaims(), informerFactory.Core().V1().PersistentVolumes(), informerFactory.Storage().V1().CSINodes(), informerFactory.Storage().V1().CSIDrivers(), informerFactory.Storage().V1().VolumeAttachments(), nil, /* cloud */ plugins, nil, /* prober */ false, 1*time.Second, DefaultTimerConfig, nil, /* filteredDialOptions */ ) if err != nil { t.Fatalf("NewAttachDetachController failed with error. Expected: <no error> Actual: <%v>", err) } adc := adcObj.(*attachDetachController) // Add existing objects (created by testplugin) to the respective informers pods, err := fakeKubeClient.CoreV1().Pods(v1.NamespaceAll).List(context.TODO(), metav1.ListOptions{}) if err != nil { t.Fatalf("Run failed with error. Expected: <no error> Actual: %v", err) } for _, pod := range pods.Items { podToAdd := pod podInformer.GetIndexer().Add(&podToAdd) } nodes, err := fakeKubeClient.CoreV1().Nodes().List(context.TODO(), metav1.ListOptions{}) if err != nil { t.Fatalf("Run failed with error. Expected: <no error> Actual: %v", err) } for _, node := range nodes.Items { nodeToAdd := node nodeInformer.GetIndexer().Add(&nodeToAdd) } // Create and add objects requested by the test if tc.podName != "" { newPod := controllervolumetesting.NewPodWithVolume(tc.podName, tc.volName, tc.podNodeName) _, err = adc.kubeClient.CoreV1().Pods(newPod.ObjectMeta.Namespace).Create(context.TODO(), newPod, metav1.CreateOptions{}) if err != nil { t.Fatalf("Run failed with error. Failed to create a new pod: <%v>", err) } podInformer.GetIndexer().Add(newPod) } if tc.pvName != "" { newPv := controllervolumetesting.NewPV(tc.pvName, tc.volName) _, err = adc.kubeClient.CoreV1().PersistentVolumes().Create(context.TODO(), newPv, metav1.CreateOptions{}) if err != nil { t.Fatalf("Run failed with error. Failed to create a new pv: <%v>", err) } pvInformer.GetIndexer().Add(newPv) } if tc.vaName != "" { newVa := controllervolumetesting.NewVolumeAttachment("va1", "pv1", "mynode-1", false) _, err = adc.kubeClient.StorageV1().VolumeAttachments().Create(context.TODO(), newVa, metav1.CreateOptions{}) if err != nil { t.Fatalf("Run failed with error. Failed to create a new volumeAttachment: <%v>", err) } vaInformer.GetIndexer().Add(newVa) } // Makesure the informer cache is synced stopCh := make(chan struct{}) informerFactory.Start(stopCh) if !kcache.WaitForNamedCacheSync("attach detach", stopCh, informerFactory.Core().V1().Pods().Informer().HasSynced, informerFactory.Core().V1().Nodes().Informer().HasSynced, informerFactory.Core().V1().PersistentVolumes().Informer().HasSynced, informerFactory.Storage().V1().VolumeAttachments().Informer().HasSynced) { t.Fatalf("Error waiting for the informer caches to sync") } // Populate ASW err = adc.populateActualStateOfWorld() if err != nil { t.Fatalf("Run failed with error. Expected: <no error> Actual: <%v>", err) } // Populate DSW err = adc.populateDesiredStateOfWorld() if err != nil { t.Fatalf("Run failed with error. Expected: <no error> Actual: %v", err) } // Run reconciler and DSW populator loops go adc.reconciler.Run(stopCh) go adc.desiredStateOfWorldPopulator.Run(stopCh) defer close(stopCh) // Verify if expected attaches and detaches have happened testPlugin := plugins[0].(*controllervolumetesting.TestPlugin) for tries := 0; tries <= 10; tries++ { // wait & try few times before failing the test expected_op_map := tc.expected_attaches plugin_map := testPlugin.GetAttachedVolumes() verify_op := "attach" volFound, nodeFound := false, false for i := 0; i <= 1; i++ { // verify attaches and detaches if i == 1 { expected_op_map = tc.expected_detaches plugin_map = testPlugin.GetDetachedVolumes() verify_op = "detach" } // Verify every (node, volume) in the expected_op_map is in the // plugin_map for expectedNode, expectedVolumeList := range expected_op_map { var volumeList []string volumeList, nodeFound = plugin_map[expectedNode] if !nodeFound && tries == 10 { t.Fatalf("Expected node not found, node:%v, op: %v, tries: %d", expectedNode, verify_op, tries) } for _, expectedVolume := range expectedVolumeList { volFound = false for _, volume := range volumeList { if expectedVolume == volume { volFound = true break } } if !volFound && tries == 10 { t.Fatalf("Expected %v operation not found, node:%v, volume: %v, tries: %d", verify_op, expectedNode, expectedVolume, tries) } } } } if nodeFound && volFound { break } time.Sleep(time.Second * 1) } if testPlugin.GetErrorEncountered() { t.Fatalf("Fatal error encountered in the testing volume plugin") } } </ins>

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Background: The attach/detach controller create objects for CSI Volumes to indicate that a volume needs to be attached to a node. It deletes the object to indicate that the volume should be detached. Problem: If the Attach/Detach controller crashes, and in the meantime the pod using that volume is deleted, when the AD controller restarts, it won't know that the volume was previously attached and should be cleaned up. What you expected to happen: Proposal: We should have the AD controller periodically list all objects and for any that are not accounted for, update its internal actual/desired state caches so that the volume is gracefully detached. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: /sig storage Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
could I take it?
/assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available .

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/lifecycle frozen /reopen
Reopened this issue. <detailsIn response to : /lifecycle frozen /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
should address this