Question: return pongHandler(msg) }) go heartbeat.start() <ins> var wg sync.WaitGroup wg.Add(1) </ins> go func() { <ins> defer wg.Done() </ins> for { _, _, err := client.ReadMessage() if err != nil {

FAIL: TestWebSocketClientHeartbeatSucceeds also flake in ci-kubernetes-unit use of closed network connection /cc /sig api-machinery network cli
/cc /triage accepted
seems it got better since
The new flake in this test is like I also see the failure in some PR: for instance.
yeah, it is racing somewhere , at least we have a different symptom now
/help someone want a great challenge and make a very useful contribution?
This request has been marked as needing help from a contributor. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
something odd is with the test, the panic is because it logs on the goroutine, but I'm not completely sure the test is working as expected
my apologies, I found the problem and I'm submitting a patch /assign /unassign you can help me with the review if you want


Question: "k8s.io/api/core/v1" "k8s.io/apimachinery/pkg/types" utiltesting "k8s.io/client-go/util/testing" <ins> "k8s.io/kubernetes/pkg/cloudprovider/providers/openstack" </ins> "k8s.io/kubernetes/pkg/util/mount" "k8s.io/kubernetes/pkg/volume" volumetest "k8s.io/kubernetes/pkg/volume/testing" <ins> "k8s.io/kubernetes/pkg/volume/util" </ins> ) func TestCanSupport(t *testing.T) {

We merged to allow configurable volume limits for Cinder. But proposed adding such a limit to as well, among other things this will enable us to support node specific limits more easily - /sig openstack /sig storage /sig scheduling
/assign
cc
/kind feature
This will require updating in-tree Openstack cloudprovider code. are we onboard with that?
What would be the extent of the modification? It must not break current compatibility, and the in-tree provider is scheduled for removal this year. I'm ok with it as long as existing users aren't broken by it.
it should be a change into which would add a new configurable field. It would be backward compatible but would allow admins to configure different limits on a node if they must.
/milestone v1.14 We're entering and this looks relevant to the 1.14 release. Please if I am incorrect


Question: # you are updating the os image versions, update this variable. # Also please update corresponding image for node e2e at: # https://github.com/kubernetes/kubernetes/blob/master/test/e2e_node/jenkins/image-config.yaml <del> GCI_VERSION=${KUBE_GCI_VERSION:-cos-77-12371-114-0} </del> <ins> GCI_VERSION=${KUBE_GCI_VERSION:-cos-77-12371-175-0} </ins> MASTER_IMAGE=${KUBE_GCE_MASTER_IMAGE:-} MASTER_IMAGE_PROJECT=${KUBE_GCE_MASTER_PROJECT:-cos-cloud} NODE_IMAGE=${KUBE_GCE_NODE_IMAGE:-${GCI_VERSION}}

Which jobs are flaking: pull-kubernetes-e2e-gce ci-cos-containerd-e2e-ubuntu-gce ci-kubernetes-e2e-kops-gce Changes needed to get the fix into official containerd releases: [x] runc race condition fixed - [x] runc release - [x] cri bump of runc - [x] containerd bump of runc/cri - [x] containerd 1.2.x release - [x] containerd 1.3.x release - As I understand it, this is what we need to get the fix into our CI environments: For kind: [x] kind containerd build with updated runc - [x] kind containerd-nightly release include patch - [x] kind containerd-nightly release - [x] kind pick up new containerd release - For GCE / kube-up: [x] [x] cri bump of runc - [x] containerd bump of runc/cri - [x] containerd 1.2.x release - [x] containerd 1.3.x release - [x] COS image update with new runc version - Which test(s) are flaking: [sig-storage] ConfigMap should be consumable from pods in volume as non-root with FSGroup [LinuxOnly] [NodeFeature:FSGroup] [sig-storage] Projected downwardAPI should provide podname as non-root with fsgroup [LinuxOnly] [NodeFeature:FSGroup] [sig-storage] Projected configMap should be consumable from pods in volume with mappings as non-root [NodeConformance] [Conformance] [sig-storage] EmptyDir volumes should support (root,0644,tmpfs) [LinuxOnly] [NodeConformance] [Conformance] [sig-storage] In-tree Volumes [Driver: hostPathSymlink] [Testpattern: Inline-volume (default fs)] subPath should support file as subpath [LinuxOnly] [sig-storage] In-tree Volumes [Driver: local][LocalVolumeType: blockfs] [Testpattern: Pre-provisioned PV (default fs)] subPath should support readOnly file specified in the volumeMount [LinuxOnly] Testgrid link: Reason for failure: runc race condition fixed in opencontainers/runc Anything else we need to know: Test containers appear to be exiting with a CRI error?

/sig node for CRI expertise /sig storage for test ownership
From in
I think we have a general failure to start containers reliably. Almost every instance of the "In-tree volume" test flakes referenced from contained the pod status error of "OCI runtime start failed: container process is already dead: unknown"
seen in [] Docker Containers should be able to override the image's default arguments (docker cmd) [NodeConformance] [Conformance] in
also seen in containerd CI runs -
This seems to be a race condition in runc. It only happened once in 2 weeks for containerd. In the runc code, when starting a container, will be called: There are 2 cases: : returns and throws out error, when 1) exec fifo is opened; 2) after 100 milliseconds and the init process is zombie or gone. : returns when the exec fifo is opened. If I understand the code correctly, I can see a potential race condition: 1) opens the fifo, but hasn't informed yet; 2) Container process starts running after the exec fifo is open; 3) Container process finishes; 4) checks process after 100 milliseconds, and found the process is already gone; 5) The error is returned unexpectedly. At least for the containerd failure, the test is a short running container.
thanks for digging.

at first glance, it looks more frequent than that in some of the jobs - ci-cos-containerd-e2e-ubuntu-gce - Failed in 5% (18/348) of builds: 12/17/2019, 10:14:46 AM 12/15/2019, 2:28:43 PM 12/14/2019, 7:09:32 PM 12/14/2019, 12:02:33 PM 12/14/2019, 4:55:35 AM 12/13/2019, 2:42:28 PM 12/13/2019, 12:40:08 PM 12/13/2019, 5:33:22 AM 12/12/2019, 8:24:21 PM 12/10/2019, 2:20:14 AM 12/9/2019, 6:01:32 AM 12/8/2019, 12:44:23 PM 12/7/2019, 8:27:44 PM 12/6/2019, 9:04:54 PM 12/6/2019, 1:58:22 PM 12/5/2019, 12:33:37 PM 12/4/2019, 4:01:54 AM 12/3/2019, 7:42:04 AM
/remove-sig storage
opened
cc
fix in progress at
the runc race condition fix has been merged at I think the next steps are getting a runc release cut and propagating that fix into the container runtimes we use in Kube CI (docker and containerd). Who on the sig-node side would be the right people to drive those?
For the GCE tests, we'd need a new COS image once the new runc release is available.

once a runc release is available, we'll need to update our e2e images /assign for COS image updates /assign for kind updates
as soon as there's a runc release we'll roll it out in kind (preferably if containerd 1.3.X can pick it up officially... we already have infra building that), tempted to go ahead and grab a runc binary at some commit ...
looks like runc rc10 is blocked on a fix for (which should have a PR open tomorrow per )
We will have to request containerd folks bump opencontainers/runc here - this is a nasty bug, hope we get a runc release quickly (cc
and then request docker folks after that -
Also affects (I guess)
Still waiting for but there's recent activity :crossed_fingers: On the KIND side: When this goes in and there is a runc release, next up after containerd bumping to it we'll (I'll) ensure kind picks up the new containerd release and builds new images with it. Right now we consume the 1.3.X / release-1.3 branch, but we are good to upgrade to HEAD if necessary.
On the GCE side: I'm still not sure who handles COS etc. currently, but someone from node on our end (=Google) will need to get this ported into COS and into our CI.. I'm going to poke as a starting point
runc rc10 is cut. containerd and docker are up next?
On Thu, Jan 23, 2020, 19:10 Jordan Liggitt <:
edit: Hoisted checklist of remaining work to the issue description
KIND won't actually need a containerd release, just a build of the commit with updated runc. Additionally though: per discussion we've noted that the runc bump is mostly information to downstream packaging of containerd, there are no code changes. So we're tentatively forging ahead with bumping runc in our "nightly" build ahead of that landing in containerd.
looks clean after merged. I verified the only flakes occurring after that was merged were on past branches still using the older COS versions. Next week, we can update the non-test kube-up config file to use the fixed version of COS as well if signal still looks good.


Question: # you are updating the os image versions, update this variable. # Also please update corresponding image for node e2e at: # https://github.com/kubernetes/kubernetes/blob/master/test/e2e_node/jenkins/image-config.yaml <del> GCI_VERSION=${KUBE_GCI_VERSION:-cos-77-12371-114-0} </del> <ins> GCI_VERSION=${KUBE_GCI_VERSION:-cos-77-12371-175-0} </ins> MASTER_IMAGE=${KUBE_GCE_MASTER_IMAGE:-} MASTER_IMAGE_PROJECT=${KUBE_GCE_MASTER_PROJECT:-cos-cloud} NODE_IMAGE=${KUBE_GCE_NODE_IMAGE:-${GCI_VERSION}}

Which jobs are flaking: pull-kubernetes-e2e-gce ci-cos-containerd-e2e-ubuntu-gce ci-kubernetes-e2e-kops-gce Changes needed to get the fix into official containerd releases: [x] runc race condition fixed - [x] runc release - [x] cri bump of runc - [x] containerd bump of runc/cri - [x] containerd 1.2.x release - [x] containerd 1.3.x release - As I understand it, this is what we need to get the fix into our CI environments: For kind: [x] kind containerd build with updated runc - [x] kind containerd-nightly release include patch - [x] kind containerd-nightly release - [x] kind pick up new containerd release - For GCE / kube-up: [x] [x] cri bump of runc - [x] containerd bump of runc/cri - [x] containerd 1.2.x release - [x] containerd 1.3.x release - [x] COS image update with new runc version - Which test(s) are flaking: [sig-storage] ConfigMap should be consumable from pods in volume as non-root with FSGroup [LinuxOnly] [NodeFeature:FSGroup] [sig-storage] Projected downwardAPI should provide podname as non-root with fsgroup [LinuxOnly] [NodeFeature:FSGroup] [sig-storage] Projected configMap should be consumable from pods in volume with mappings as non-root [NodeConformance] [Conformance] [sig-storage] EmptyDir volumes should support (root,0644,tmpfs) [LinuxOnly] [NodeConformance] [Conformance] [sig-storage] In-tree Volumes [Driver: hostPathSymlink] [Testpattern: Inline-volume (default fs)] subPath should support file as subpath [LinuxOnly] [sig-storage] In-tree Volumes [Driver: local][LocalVolumeType: blockfs] [Testpattern: Pre-provisioned PV (default fs)] subPath should support readOnly file specified in the volumeMount [LinuxOnly] Testgrid link: Reason for failure: runc race condition fixed in opencontainers/runc Anything else we need to know: Test containers appear to be exiting with a CRI error?

/sig node for CRI expertise /sig storage for test ownership
From in
I think we have a general failure to start containers reliably. Almost every instance of the "In-tree volume" test flakes referenced from contained the pod status error of "OCI runtime start failed: container process is already dead: unknown"
seen in [] Docker Containers should be able to override the image's default arguments (docker cmd) [NodeConformance] [Conformance] in
also seen in containerd CI runs -
This seems to be a race condition in runc. It only happened once in 2 weeks for containerd. In the runc code, when starting a container, will be called: There are 2 cases: : returns and throws out error, when 1) exec fifo is opened; 2) after 100 milliseconds and the init process is zombie or gone. : returns when the exec fifo is opened. If I understand the code correctly, I can see a potential race condition: 1) opens the fifo, but hasn't informed yet; 2) Container process starts running after the exec fifo is open; 3) Container process finishes; 4) checks process after 100 milliseconds, and found the process is already gone; 5) The error is returned unexpectedly. At least for the containerd failure, the test is a short running container.
thanks for digging.

at first glance, it looks more frequent than that in some of the jobs - ci-cos-containerd-e2e-ubuntu-gce - Failed in 5% (18/348) of builds: 12/17/2019, 10:14:46 AM 12/15/2019, 2:28:43 PM 12/14/2019, 7:09:32 PM 12/14/2019, 12:02:33 PM 12/14/2019, 4:55:35 AM 12/13/2019, 2:42:28 PM 12/13/2019, 12:40:08 PM 12/13/2019, 5:33:22 AM 12/12/2019, 8:24:21 PM 12/10/2019, 2:20:14 AM 12/9/2019, 6:01:32 AM 12/8/2019, 12:44:23 PM 12/7/2019, 8:27:44 PM 12/6/2019, 9:04:54 PM 12/6/2019, 1:58:22 PM 12/5/2019, 12:33:37 PM 12/4/2019, 4:01:54 AM 12/3/2019, 7:42:04 AM
/remove-sig storage
opened
cc
fix in progress at
the runc race condition fix has been merged at I think the next steps are getting a runc release cut and propagating that fix into the container runtimes we use in Kube CI (docker and containerd). Who on the sig-node side would be the right people to drive those?
For the GCE tests, we'd need a new COS image once the new runc release is available.

once a runc release is available, we'll need to update our e2e images /assign for COS image updates /assign for kind updates
as soon as there's a runc release we'll roll it out in kind (preferably if containerd 1.3.X can pick it up officially... we already have infra building that), tempted to go ahead and grab a runc binary at some commit ...
looks like runc rc10 is blocked on a fix for (which should have a PR open tomorrow per )
We will have to request containerd folks bump opencontainers/runc here - this is a nasty bug, hope we get a runc release quickly (cc
and then request docker folks after that -
Also affects (I guess)
Still waiting for but there's recent activity :crossed_fingers: On the KIND side: When this goes in and there is a runc release, next up after containerd bumping to it we'll (I'll) ensure kind picks up the new containerd release and builds new images with it. Right now we consume the 1.3.X / release-1.3 branch, but we are good to upgrade to HEAD if necessary.
On the GCE side: I'm still not sure who handles COS etc. currently, but someone from node on our end (=Google) will need to get this ported into COS and into our CI.. I'm going to poke as a starting point
runc rc10 is cut. containerd and docker are up next?
On Thu, Jan 23, 2020, 19:10 Jordan Liggitt <:
edit: Hoisted checklist of remaining work to the issue description
KIND won't actually need a containerd release, just a build of the commit with updated runc. Additionally though: per discussion we've noted that the runc bump is mostly information to downstream packaging of containerd, there are no code changes. So we're tentatively forging ahead with bumping runc in our "nightly" build ahead of that landing in containerd.
looks clean after merged. I verified the only flakes occurring after that was merged were on past branches still using the older COS versions. Next week, we can update the non-test kube-up config file to use the fixed version of COS as well if signal still looks good.


Question: wantFailed:    4, }, "deleted pods": { <ins> pods: []*v1.Pod{ buildPod().uid("a").phase(v1.PodSucceeded).deletionTimestamp().Pod, buildPod().uid("b").phase(v1.PodFailed).deletionTimestamp().Pod, buildPod().uid("c").phase(v1.PodRunning).deletionTimestamp().Pod, buildPod().uid("d").phase(v1.PodPending).deletionTimestamp().Pod, }, wantSucceeded: 1, wantFailed:    1, }, "deleted pods, tracking with finalizers": { </ins> job: batch.Job{ Status: batch.JobStatus{ Succeeded:               1,

Since last Friday scalability test become red/very flaky Affected jobs: Those failure are related to failed jobs. Sample test output: Working hypothesis is that code change from , but this requires confirmation.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scalability
This particular regression is purely Job-related (we only see problems with batchv1.Job objects). /sig apps /priority critical-urgent - FYI /milestone v1.22
/assign
After internal testing it seems that can be a fix. Added big scale presubmits to


Question: } func (i *iSCSIDriver) CreateDriver() { <del> // Reset config. It might have been modified by a previous CreateVolume call. i.driverInfo.Config.ServerConfig = nil </del> } func (i *iSCSIDriver) CleanupDriver() {

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: gce/gke multizone/regional jobs Which test(s) are failing: Only PD tests Since when has it been failing: Around 12/21 with the 16:49 run Testgrid link: https://k8s- Reason for failure: TBD Anything else we need to know: <!-- DO NOT EDIT BELOW THIS LINE --/kind failing-test
The runs started flaking after merged. cc Here's one failing run: Pod pod-subpath-test-gcepd-dynamicpv-pfsm failed to schedule because of conflicting node selectors: We shouldn't need to set the node selector for PD tests using PVs.
The PR broke the subpath test because the modification to the config struct embedded in the driver info now persists across tests: A quick fix would be to reset that field in CreateDriver. The long-term fix is the change discussed in My preference is to do the quick fix now, then either merge or close PRs in this order: (needs a decision about logging of tests that aren't getting defined, see ) a future PR for What I'd like to avoid is having to rebase PR on top of the long-term solution for issue - that'll be lots of code conflicts.
/assign PR is here: PR


Question: } return errKeyIDOKCode, nil } <ins> func getRequestInfoFromContext(ctx context.Context) *genericapirequest.RequestInfo { if reqInfo, found := genericapirequest.RequestInfoFrom(ctx); found { return reqInfo } return &genericapirequest.RequestInfo{} } </ins>

Add additional logging at level that associates UID with the metadata from the request. xref: /sig auth
/triage accepted
/milestone v1.26
Mind if I take a look at this? /assign
Hello I am wondering how's this coming along. Is there anything I can help with?
Sorry. I got a bit occupied with a different project.. Will have a PR opened for this for review tomorrow.. I am trying to find a way to inject the Verbs into log. But haven't found the right way to include that into the context.. Any suggestion on what can be done?
any suggestion on what is the best way to get the additional metadata for verbs into this log? Move that info across using ?
Hello , bug triage team here, just wanted to touch base with you for updates on this ticket. Thanks!
Thanks! This is being worked on and is still on track for .
There is a Pull request opened on this already as well. But Waiting on some clarification to see if that needs additional changes.
/reopen /assign /unassign
Reopened this issue. <detailsIn response to : /reopen /assign /unassign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/milestone v1.27
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
Hi Bug Triage Team for v1.27 here o/ Just wanted to touch base with you for updates on this ticket. Thanks!
I am not working on this anymore Carlos. This was re-assigned
This is tracked for . I've linked the PR.


Question: // Get all the zones that the nodes are in expectedZones, err := gceCloud.GetAllZonesFromCloudProvider() Expect(err).NotTo(HaveOccurred()) <del> framework.Logf("Expected zones: %vn", expectedZones) </del> <ins> framework.Logf("Expected zones: %v", expectedZones) </ins> // Get all the zones in this current region region := gceCloud.Region()

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: is always appending a new line to its message: In some places of code, the message is already has a newline character and it leads to doubled new lines in the logs. What you expected to happen: I'm not sure what is the correct behavior here. One of the options could be to append a new line conditionally.
/sig testing
I'm new to Kubernetes and would like to start contributing. Is it ok if I assign myself this issue?
/assign MorrisLaw
GitHub didn't allow me to assign the following users: MorrisLaw. Note that only can be assigned. <detailsIn response to : /assign MorrisLaw Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I can't assign this to myself, as I am not a member yet. But I am still going to take this issue.
I'll help him on reviews so assigning.


Question: return allErrs } <del> // This validate will make sure targetPath: </del> <ins> // ValidateLocalNonReservedPath makes sure targetPath: </ins> // 1. is not abs path // 2. does not contain any '..' elements // 3. does not start with '..' <del> func validateLocalNonReservedPath(targetPath string, fldPath *field.Path) field.ErrorList { </del> <ins> func ValidateLocalNonReservedPath(targetPath string, fldPath *field.Path) field.ErrorList { </ins> allErrs := field.ErrorList{} allErrs = append(allErrs, validateLocalDescendingPath(targetPath, fldPath)...) // Don't report this error if the check for .. elements already caught it.

Originally posted by in
Hello, I'm new here. Can I try solving this? it seems like a good first issue
sure, context is the change is needed here: We have to tolerate existing pods / pod specs that fail this validation, so we should only do this additional validation on create or if the existing object passes this validation. To do this: an field to , initialize that field to false, and then conditionally override it to if is non-nil and has a projected token volume path that fails (will have to export the function to check this) , if is false, call the validation on the path:
Hi, I'd like to work on this if there is no one is proceeding on. /assign


Question: }) ginkgo.It("should support inline execution and attach", func() { <ins> waitForStdinContent := func(pod, content string) string { var logOutput string err := wait.Poll(10*time.Second, 5*time.Minute, func() (bool, error) { logOutput = framework.RunKubectlOrDie(ns, "logs", pod) return strings.Contains(logOutput, content), nil }) gomega.Expect(err).To(gomega.BeNil(), fmt.Sprintf("unexpected error waiting for '%v' output", content)) return logOutput } </ins> ginkgo.By("executing a command with run and attach with stdin") // We wait for a non-empty line so we know kubectl has attached <del> runOutput := framework.NewKubectlCommand(ns, "run", "run-test", "--image="+busyboxImage, "--restart=OnFailure", "--attach=true", "--stdin", "--", "sh", "-c", "while [ -z "$s" ]; do read s; sleep 1; done; echo read:$s && cat && echo 'stdin closed'"). </del> <ins> framework.NewKubectlCommand(ns, "run", "run-test", "--image="+busyboxImage, "--restart=OnFailure", "--attach=true", "--stdin", "--", "sh", "-c", "echo -n read: && cat && echo 'stdin closed'"). </ins> WithStdinData("valuenabcd1234"). ExecOrDie(ns) <ins> runOutput := waitForStdinContent("run-test", "stdin closed") </ins> gomega.Expect(runOutput).To(gomega.ContainSubstring("read:value")) gomega.Expect(runOutput).To(gomega.ContainSubstring("abcd1234")) gomega.Expect(runOutput).To(gomega.ContainSubstring("stdin closed"))

The test case "executing a command with run and attach without stdin" is inherently racy. The test runs, and expected to see in the output. The case when the test passes: case 1: Kubernetes creates and starts the container without stdin; The container exits before attach runs; kubectl falls back to get container log, which contains . case 2: Kubernetes creates and starts the container without stdin; kubectl attach before "stdin closed" is printed; kubectl attach contains . The case when the test fails: Kubernetes creates and starts the container without stdin; The container prints "stdin closed", but hasn't exited yet; kubectl attach contains nothing. One example failure: Related issue: Can we modify this test not to check the output? Or use to check output instead? /cc
Uploaded to try to address
/reopen This test is still inherently racy, even after
Fix in (I hope!)
Moving into the current milestone as it was not fixed in 1.14
Hello I'm a bug triage shadow for the 1.16 release cycle and considering this issue is tagged for 1.16, I'd like to check its status. Code freeze is starting on August 29th (about 3 days from now). Do we still target this issue to be fixed for 1.16?
Thanks . PR is available and awaiting review!
What is the PR in question? Is it targeting 1.16 or 1.17?
/reopen This test is having a lot of failures these days
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
is right, it seems to be present in kind jobs now only
/assign let's see if we start to recover a bit of stability :smile:
The rate of failures decreased a lot since last changes Checking the logs of the failures it seems that the problem is that the pod has not finished before checking the output, as Lantao described The test fails Collecting events from namespace The framework logs the pods the kubelet thinks is on the node
note that it's still racy, but it's hopefully somewhat mitigated.
/reopen
Reopened this issue.

<detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
any updates here?
nope, totally agree with the description, is racy by design, couldn't find any way or trick to avoid the randomness
EDIT: It's late and I'm actually not sure I'm remembering this correctly, take this comment with a heap of salt: Specific sub-tests were fixed previously I think, but this one sub-case: That one we can't use the "Wait for stdin input to signal we're attached" hack, so it's just racy and broken. We could remove that case maybe. I'm not sure who I'd want to weigh in on that, probably someone more active in node / CLI.
not super common atm but still happening. discussing
/unassign has a fix for this


Question: library = ":go_default_library", tags = ["automanaged"], deps = [ <ins> "//pkg/cloudprovider/providers/azure:go_default_library", "//pkg/cloudprovider/providers/fake:go_default_library", </ins> "//pkg/util/mount:go_default_library", "//pkg/volume:go_default_library", "//pkg/volume/testing:go_default_library",

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? NO Note: Please file issues for subcomponents under the appropriate repo What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): kubernetes, azure file, chinacloudapi Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): master Environment: Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Ubuntu 16.04 Kernel (e.g. ): Install tools: Others: What happened: We try to mount azure file to pod, and get the following error That is because azure china's storage DNS suffix is rather than What you expected to happen: Kubernetes should provide some method to let user specify storage dns suffix, so we can use Azure file volume How to reproduce it (as minimally and precisely as possible): Any Azure cluster in china can reproduce it. Anything else we need to know: kubernetes has hard coded in shows a relative issue
could you help confirm this?
One possible way is configuring DNS suffix via annotations of PV. If thinks this way okay, I'd like to create a PR to fix it.
azure file was implemented without using resource group or azure cloud provider. The endpoint can be found at One fix is to use azure cloud provider and choose the file service endpoint from azure environment as I did in blob service . Let me know if you can help.
I'm not familiar with azure cloud provider and volume codes. I'd like to try to dig into it to see if I can fix as the way you suggested.


Question: "//pkg/kubelet/cm:go_default_library", "//pkg/kubelet/eviction/api:go_default_library", "//pkg/kubelet/lifecycle:go_default_library", <ins> "//pkg/kubelet/metrics:go_default_library", </ins> "//pkg/kubelet/pod:go_default_library", "//pkg/kubelet/qos:go_default_library", "//pkg/kubelet/server/stats:go_default_library",

We should keep track of how stale stats used for evictions are. This will help track improvements in eviction latency, and help uncover regressions. cc If someone could assign this to me, I would appreciate that.
closing via


Question: } } } <ins> func TestValidateFlags(t *testing.T) { tests := []struct { taintOpts   TaintOptions description string expectFatal bool }{ { taintOpts:   TaintOptions{selector: "myLabel=X", all: false}, description: "With Selector and without All flag", expectFatal: false, }, { taintOpts:   TaintOptions{selector: "", all: true}, description: "Without selector and All flag", expectFatal: false, }, { taintOpts:   TaintOptions{selector: "myLabel=X", all: true}, description: "With Selector and with All flag", expectFatal: true, }, { taintOpts:   TaintOptions{selector: "", all: false, resources: []string{"node"}}, description: "Without Selector and All flags and if node name is not provided", expectFatal: true, }, { taintOpts:   TaintOptions{selector: "", all: false, resources: []string{"node", "node-name"}}, description: "Without Selector and ALL flags and if node name is provided", expectFatal: false, }, } for _, test := range tests { sawFatal := false err := test.taintOpts.validateFlags() if err != nil { sawFatal = true } if test.expectFatal { if !sawFatal { t.Fatalf("%s expected not to fail", test.description) } } } } </ins>

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): No What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): taint, selector Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-03-28T16:36:33Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-04-03T20:33:27Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"} Environment: Cloud provider or hardware configuration: GKE OS (e.g. from /etc/os-release): MacOS client, linux server Kernel (e.g. ): Install tools:gcloud Others: What happened: I tried to kubectl taint some nodes using a label selector: kubectl taint nodes -l dedicated=foo dedicated=foo:PreferNoSchedule and got the error: error: at least one resource name must be specified since 'all' parameter is not set What you expected to happen: Those nodes would be tainted with dedicated=foo:PreferNoSchedule. Trying to use --all along with the selector is also not supported.

How to reproduce it (as minimally and precisely as possible): kubectl taint nodes -l <any label selectordedicated=foo:PreferNoSchedule Anything else we need to know:
can you post the output of kubectl taint nodes -l myLabel=X dedicated=foo:PreferNoSchedule (((And first of course Make sure that myLabel is on the cluster ....)))
Would you like me to explicitly use the label "myLabel"? Otherwise that's the command and output I posted in "What happened" above. (I had set a label dedicated=foo ahead of time on two nodes.)
Ah ok so the taint k/v is the exact same as the label k/v, that makes the example tricky to parse.... but I agree .... this should work .. I can try to reproduce this....
Yup, the same k/v. (I wanted both labels and taints to determine this set of nodes, and it seemed like it would be confusing to have different k/v's for each. I kinda thought this would be the common case...)
Ah , yeah i see it to ... Even though clearly, the selector exists.
Is the taints and tolerations expert on our team. ravi can you look at ? this logic might be shadowing the label selector, such that its impossible to use labels to select the nodes you're trying to taint.
cc. m i definitely think this is a bug ... ? Looks like the taints maybe are shadowing the label selectors above in
/assign
GitHub didn't allow me to assign the following users: ravisantoshgudimetla. Note that only can be assigned.. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. I understand the commands that are listed . </details
/assign
It seems the line is the culprit. Here we are checking for resources if they are 2, the resources allowed are node,nodes,no so if it needs it has to be 1 and this check is already available So, I guess its safe to remove that. I tested that functionality after removing this and it fixed the issue. I will submit a PR for this unless someone has objections.


Question: // forbidden renders a simple forbidden error func forbidden(attributes authorizer.Attributes, w http.ResponseWriter, req *http.Request, reason string) { msg := forbiddenMessage(attributes) <ins> w.Header().Set("Content-Type", "text/plain") w.Header().Set("X-Content-Type-Options", "nosniff") </ins> w.WriteHeader(http.StatusForbidden) fmt.Fprintf(w, "%s: %q", msg, reason) }

The error reporting functions here all repeat a URI without sanitizing it. An XSS attacker could use this to get the server to embed a reference to a malicious script. (As always, please note that kubernetes apiserver is NOT xss hardened and we do NOT recommend using the api directly from a browser.)
The error correctly sets the content type to , so this is only an issue using a browser that does content sniffing. Would it be sufficient to just add the header here?
Yes, that will work for browsers that honor the header. It's probably sufficient-- we could do that globally, we set Content-Type correctly everywhere, I think.


Question: if err != nil { allErrs = append(allErrs, err) } <del> if len(infos) == 0 { </del> <ins> if len(infos) == 0 && len(allErrs) == 0 { </ins> outputEmptyListWarning(errOut) }

Is this a BUG REPORT or FEATURE REQUEST? (choose one): bug Kubernetes version (use ): master What happened: is printed when error. What you expected to happen: Only error message was printed.
I believed this has been fixed
, it seems not; in , it's not necessary to print if errs is not empty. Refer to the PR on the proposal :).
ok, seems other issue.


Question: // This is very low impact. The NodePort range is intentionally obscure, and unlikely to actually collide with real Services. // This only affects UDP connections, which are not common. // See issue: https://github.com/kubernetes/kubernetes/issues/49881 <del> isIPv6 := svcInfo.clusterIP.To4() != nil </del> <ins> isIPv6 := utilproxy.IsIPv6(svcInfo.clusterIP) </ins> err := utilproxy.ClearUDPConntrackForPort(proxier.exec, lp.Port, isIPv6) if err != nil { glog.Errorf("Failed to clear udp conntrack for port %d, error: %v", lp.Port, err)

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: There is a bug in ipv6, see: What you expected to happen: The judgement logic should be: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: /sig network /area ipv6
/assign cc


Question: c.reflectorMutex.Unlock() var wg wait.Group <del> defer wg.Wait() </del> wg.StartWithChannel(stopCh, r.Run) wait.Until(c.processLoop, time.Second, stopCh) <ins> wg.Wait() </ins> } // Returns true once this controller has completed an initial resource listing

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When use client-go to build the client, we implement the event handler for some k8s resourses. Sometime it will panic (maybe cause by the unexpected nil pointer bug), the process will not crash after panic. What you expected to happen: When we use event handler with informer in client-go, sometime it will panic in handler, we expect the process will crash. How to reproduce it (as minimally and precisely as possible): I run process which just panic simpliy in an event handler , use to produce the coredump, and use to check the call stack. The sample code I run is: The call stack of paniced goroutine: We can see the panic was called at frame 16 , and was catched by at 10. It printed the call stack in and panic again. The defer func was call at frame 8, it's (at 5). Panic was blocked at here. Finally, this goroutine was scheduled to , would not exit or crash the whole process. The go code in is : Environment: Kubernetes version (use ):OS (e.g: ):Kernel (e.g. ):
/sig api-machinery
Thanks for finding this! Personally I think any lines of the form are a bug waiting to happen, and in all the cases introduced by are unnecessary - we can simply call at the end.
/assign
Your fix looks better. I think using is easy to make mistakes . It not only defers the execution of a function, but also has some side effects ...


Question: TEST_ETCD_VERSION="2.2.1" fi <del> retry() { </del> <ins> function retry() { </ins> for i in {1..4}; do "$@" && return 0 || sleep $i done "$@" } <ins> function mount-master-pd() { if [[ ! -e /dev/disk/by-id/google-master-pd ]]; then echo "Can't find master-pd. Skipping mount." return fi device_info=$(ls -l "/dev/disk/by-id/google-master-pd") relative_path=${device_info##* } pd_device="/dev/disk/by-id/${relative_path}" echo "Mounting master-pd" local -r pd_path="/dev/disk/by-id/google-master-pd" local -r mount_point="/mnt/disks/master-pd" # Format and mount the disk, create directories on it for all of the master's # persistent data, and link them to where they're used. mkdir -p "${mount_point}" device=$1 mountpoint=$2 # Format only if the disk is not already formatted. if ! tune2fs -l "${pd_path}" ; then echo "Formatting '${pd_path}'" mkfs.ext4 -F -E lazy_itable_init=0,lazy_journal_init=0,discard "${pd_path}" fi mkdir -p "${mount_point}" echo "Mounting '${pd_path}' at '${mount_point}'" mount -o discard,defaults "${pd_path}" "${mount_point}" echo "Mounted master-pd '${pd_path}' at '${mount_point}'" # Contains all the data stored in etcd. mkdir -m 700 -p "${mount_point}/var/etcd" ln -s -f "${mount_point}/var/etcd" /var/etcd mkdir -p /etc/srv # Contains the dynamically generated apiserver auth certs and keys. mkdir -p "${mount_point}/srv/kubernetes" ln -s -f "${mount_point}/srv/kubernetes" /etc/srv/kubernetes # Directory for kube-apiserver to store SSH key (if necessary). mkdir -p "${mount_point}/srv/sshproxy" ln -s -f "${mount_point}/srv/sshproxy" /etc/srv/sshproxy if ! id etcd &>/dev/null; then useradd -s /sbin/nologin -d /var/etcd etcd fi } </ins> if [ "${EVENT_STORE_IP}" == "127.0.0.1" ]; then # Retry starting etcd to avoid pulling image errors. retry sudo docker run --net=host

While running kubemark tests we are facing "no space left on device" problems, because master-disk is not mounted in kubemark.
I'm reopening it for now, since it seems that we are still facing "no space left on device" issues in kubemark scale.
Seems to be fixed by


Question: }, { desc: "getServiceLoadBalancerStatus shall return nil if lb.FrontendIPConfigurations.name != " + <del> "az.getFrontendIPConfigName(service, subnet(service))", </del> <ins> "az.getFrontendIPConfigName(service)", </ins> service: &internalService, lb:      &lb3, },

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When the specified internal LB uses a subnet with longer name, the LB configuration will fail with following error: What you expected to happen: When constructing the frontend IP configuration name and the load balancing rules name, the subnet name should be truncated if too long. How to reproduce it (as minimally and precisely as possible): Use a subnet that has long name (70 charactors) and create ILB service. Anything else we need to know?: Environment: Kubernetes version (use ): 1.15 Cloud provider or hardware configuration: azure OS (e.g: ): Ubuntu 16.04 Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig cloud-provider /area provider/azure /assign


Question: // Total number of available pods (ready for at least minReadySeconds) targeted by this StatefulSet. // This is a beta field and enabled/disabled by StatefulSetMinReadySeconds feature gate. <ins> // +optional </ins> optional int32 availableReplicas = 11; }

Looking at the for , it defines as a required attribute. But looking at the of , it states . Should this in fact be a required attribute? I come about this from using , a python package that generates models from the swagger definitions and thus builds a that requires an attribute that is not enabled. That only truly required attributes be required in the swagger definition Reproduction is not applicable as this is an issue with a spec No response Appeared in 1.22 (I believe) and affects current versions. Does not affect 1.21 n/a No response No response No response No response
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
alpha: has that is planned for removal in beta. ( results in fields as non-required in the OpenAPI) beta: removes the this might be a question whether feature gate controlled fields must remain until the feature gate is removed (and how would validation work in that case). ~old feature gates graduations to beta enabled a feature by default.~ a topic related to ? but it does not apply to feature gates. /kind documentation /sig apps
cc (since you are the author)
new fields to existing types must always be marked as +optional, since clients are not required to specify them
please open a PR adding back the +optional tag, regenerate openapi, and open picks back to 1.23 and 1.22


Question: package create import ( <ins> "k8s.io/cli-runtime/pkg/genericclioptions" restclient "k8s.io/client-go/rest" cmdtesting "k8s.io/kubectl/pkg/cmd/testing" </ins> "testing" v1 "k8s.io/api/core/v1"

I'm writing a small bash script which generates some k8s resources in a namespace (other than ). When using to generate a service, the yaml which is generated does not include the defined namespace. I've also tried to move around the argument to the beginning (i.e. , but that doesn't work as well. Generated yaml should include the , when it's defined as an option. Kubernetes version (use ): Cloud provider or hardware configuration: n/a (not really important) OS (e.g: ): Kernel (e.g. ): Install tools: kubectl Network plugin and version (if this is a network-related bug): n/a Others: /sig cli
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cli
/assign


Question: import ( "reflect" "testing" <ins> "time" </ins> "k8s.io/api/core/v1" "k8s.io/apimachinery/pkg/api/resource" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" <ins> "k8s.io/kubernetes/pkg/scheduler/internal/cache" </ins> fakecache "k8s.io/kubernetes/pkg/scheduler/internal/cache/fake" <ins> "k8s.io/kubernetes/pkg/scheduler/internal/queue" </ins> ) func TestSkipPodUpdate(t *testing.T) {

What happened: pod can't be removed from scheduler cache when be deleted, logs in kube-scheduler is below: I1128 16:41:50. ] lock is held by shaphicpra1490764ccb0c5-aaa1-435d-a4b4- and has not yet expired I1128 16:41:50. ] failed to acquire lease kube-system/kube-scheduler I1128 16:41:53. ] lock is held by shaphicpra1490764ccb0c5-aaa1-435d-a4b4- and has not yet expired I1128 16:41:53. ] failed to acquire lease kube-system/kube-scheduler I1128 16:41:57. ] successfully acquired lease kube-system/kube-scheduler I1128 16:41:57. ] caches populated E1128 16:42:11. ] scheduler cache UpdatePod failed: pod -decb-4abd-a1d3- is not to scheduler cache, so cannot be updated E1128 16:42:12. ] scheduler cache UpdatePod failed: pod -decb-4abd-a1d3- is not to scheduler cache, so cannot be updated E1128 16:42:12. ] scheduler cache UpdatePod failed: pod -decb-4abd-a1d3- is not to scheduler cache, so cannot be updated E1128 16:42:20. ] scheduler cache UpdatePod failed: pod -decb-4abd-a1d3- is not to scheduler cache, so cannot be updated E1128 16:42:20. ] scheduler cache RemovePod failed: pod -decb-4abd-a1d3- is not found in scheduler cache, so cannot be removed from it What you expected to happen: pod in scheduler cache can be deleted successfully when pod is deleted How to reproduce it (as minimally and precisely as possible): main steps to reproduce this bug is these: scheduler run in leader-elect=true modeschedulerA is the leader scheduler, and schedulerB is an assistant.

pod1 with "namespace1/name1" and it runs on Node1 down the network between schedulerB and kube-apiserver pod1 and create pod2 with same name, ie "namespace1/name1" and pod2 been assigned to Node1 by schedulerA again. off schedulerA and recover the network between schedulerB and kube-apiserver, and then schedulerB become the leader scheduler. pod2. this time log in schedulerB will print this: E1128 16:42:11. ] scheduler cache UpdatePod failed: pod -decb-4abd-a1d3- is not to scheduler cache, so cannot be updated E1128 16:42:20. ] scheduler cache RemovePod failed: pod -decb-4abd-a1d3- is not found in scheduler cache, so cannot be removed from it this time, if I create a new pod3 with podAntiAffinity to pod1. the result is pod3 can't been assigned to Node1 because pod1 is still in schedulerB's scheduler cache. Anything else we need to know?: the reason of this bug is below: since PR(), podKey in scheduler cache use pod's UID instead of namespace/name however, the key in SharedIndexInformer still use namespace/name in this upper case, delete the pod and create a same name pod may be delivered as one update notification.( see disscusion ) scheduler received pod update notification and then print below log. E1128 16:42:11. ]

scheduler cache UpdatePod failed: pod -decb-4abd-a1d3- is not to scheduler cache, so cannot be updated Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-06-19T16:40:16Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-06-19T16:40:16Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig release
/sig scheduling
Since has been marked resolved (fix available in 1.17), does that mean this issue is expected to go away in 1.17? Could you please give a try?
, fix for just resolved the case add event may be lost after deletion event. It doesn't solve the problem that a delete followed by an add may be delivered as one update notification and it's thought normal from MikeSpreitzer's comment. The main cause of this problem is scheduler cache use Pod UID as key while sharedInformer use namespace/name. Still, I also test with the latest kubernetes code. The bug of this issue reappeared.
Thanks I a comment in the PR, and a test is necessary to reproduce the issue and verify the fix.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale ping there is an unresolved comment in the PR, do you have a chance to take a look?


Question: import ( "github.com/onsi/ginkgo" <ins> v1 "k8s.io/api/core/v1" </ins> "k8s.io/apimachinery/pkg/util/sets" "k8s.io/kubernetes/test/e2e/framework" e2enetwork "k8s.io/kubernetes/test/e2e/framework/network"

What would you like to be : Right now we poll one node at a time for intra-pod tests. We ultimately fail a test if any one node is unreachable. If we instead polled all nodes, multiple times, and collected the number of times a node was unreachable, this would tell us what % of nodes on the cluster were broken, rather then just telling us the first node that was broken. not break the testing contract (were still verifying intra-pod connectivity, just doing so in a way that gives us holistic cluster info) Why is this needed: I described why this would make triage alot easier, and in general make it easier to see why tests are failing, here ... i'll collect a little more data on this to justify the decision .... as this is a conformance test so the bar for modification is higher (even if the test itself isnt changing) then regular tests.
/sig network
/assign
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/remove-triage unresolved


Question: p.lock.Lock() defer p.lock.Unlock() // Map accesses must be locked. <del> p.podStatus[objKey{pod.Namespace, pod.Name}] = newStatus </del> <ins> if err == nil { p.podStatus[objKey{pod.Namespace, pod.Name}] = newStatus } </ins> return err }

I have a cluster with ~130 pods. Running back-to-back 'get pods' shows me wildly different understandings of their state, even though I can tell from docker that they have not crashed or anything. thockin kubernetes master /(NF)}' uniq -c Running: ./cluster/../cluster/gce/../../output/dockerized/bin/linux/amd64/kubectl get pods -l app=hostnames 127 Running 1 STATUS 6 Unknown thockin kubernetes master /(NF)}' uniq -c Running: ./cluster/../cluster/gce/../../output/dockerized/bin/linux/amd64/kubectl get pods -l app=hostnames 82 Running 1 STATUS 51 Unknown Nothing changed in between these calls.
cc I blame the pod cache.
has recently changed pod cache? FWIW, I also observed this behavior: 19 pods Running in one call and all Unknown in the next. With 0.10.0.
Yes, I think PR on pod cache should fix the problem reported here. Also it should fix the issue reported by about adds-on services. I have a pending PR to have kubelet deciding on PodStatus too.
Could this have caused ?
I don't see how. I'll comment on that issue.
I don't think this could be the cause of that one.
I don't think PR on pod cache will fix this. That one clears the node status. This issue is with pod going from Running to Unknown even when nodes keep running just fine.
Totally consistent observation based on my personal experience. I just assumed we were shooting for some kind of Hesinberg Uncertainty Principle. S , Tim Hockin wrote:
Agree with clears node status only. One possibility is that we hit transient network problem in between the calls that fails health check, so controller set node condition to not reachable, hence unknown pods. I'll make this more robust anyway. Other than that, it's hard to think of a reason that node status results in the flakiness.
In the absence of node information, the pod status defaults to Unknown. Maybe there's another source of this specific problem, but the pod cache is a source of continuous problems, partly because it's not just a cache, but is implementing or duplicating a significant chunk of system logic in a half-baked manner. My goal is to eliminate it -- see .

Short of that, we need to at least implement , eliminate nodes from the pod cache, and implement a real node cache.
I can look into this. Repro info from Tim, who said he was running on a one-node cluster but has seen the problem with multi-node clusters also: run-container hostnames --image=kubernetes/servehostname --replicas=1 --port=9376 --labels=app=hostnames expose hostnames --port=80 --container-port=9376 --selector="app=hostnames" resize --replicas=150 rc hostnames but I had to hack the node a bit to make it get past 100 on the node: sysctl -w fs.inotify.maxuserinstances=8192 echo "DOCKERNOFILE=" /etc/default/docker; /etc/init.d/docker restart (fails sometimes, dunno why)
I can verify that the situation is rather bad. Lots of fluctuation in the numbers of Pending, Running, and Unknown. serve_hostname containers on the node are exiting; I'm not sure why, or if the pod itself is terminating or just the container. Lots of the following in the log (not sure what this means yet): Sat, 07 Feb 2015 00:06:22 +0000 hostnames-ecd2r Pod failedScheduling {scheduler } Binding rejected: 501: All the given peers are not reachable (Tried to connect to each peer twice and failed) [0]
The pause container is also exiting, so I'm guessing that means the whole pod is exiting, not just serve_hostname...
Ugh, there is an etcd size limit you might be hitting too, I forgot. If you are seeing the fluctuations, maybe you got far enough, though. On Feb 6, 2015 5:13 PM, "David Oppenheimer" wrote:
Just a few more observations: This is reproducible (and a little less chaotic) with one node and 100 pods (but not with 50 pods, so it's definitely load-dependent). You can get to 100 Running pods (0 Pending or Unknown), and then a minute later a dozen or so will become Unknown, and then later go back to Running, etc. As mentioned earlier, the containers continue running on the node all this time (sudo docker ps wc -l), so it is definitely some kind of reporting problem.

As an aside, it looks like there is some kind of race condition with IP address allocation, as during startup (but not at steady state) a number of pods exit with the following in their log (sudo docker logs xxx): listen tcp :9376: bind: address already in use They do restart OK (I'm not sure if this is being restarted locally on the node or making a round-trip back to the scheduler).
Almost certainly the restart is just kubelet retrying after some time. On Feb 8, 2015 2:57 PM, "David Oppenheimer" wrote:
I found that if I comment out the line with the arrow next to it (see below) in , the Pods stop oscillating between Running and Unknown. Commenting out this line changes the logic from "if get an error from Kubelet, change pod status to Unknown" to "if get an error from Kubelet, keep pod status unchanged." I saw in the master logs intermittent errors of this form, which was causing the pods to flip back to Unknown: E0209 04:19:17. 4996 ] Error getting info for pod default/hostnames-v5drx: kubelet "kubernetes-minion-1.c.striped-torus-760.internal" server responded with HTTP error code 500 for pod default/hostnames-v5drx This seems to correspond to lines like this in the kubelet logs: I0209 06:38:11. 4704 ] GET /stats/default/hostnames-gfw5j/-b026-11e4-a070-/hostnames?num_stats=33: (218.857218ms) 500 goroutine 730 [running]: [ ... stack trace, but they're not always the same; I'll dig into this some more, the default stack trace depth wasn't going back past the various functions that just serve up the error ... ] However, pods still flip between Running and Pending, even though no containers are exiting on the minion, so there's another problem here too.
Thanks for investigating this! Pods really shouldn't flip between Running and Pending, as I believe that the only way that they go Pending is if Pod.Status.Host is empty, and this really shouldn't happen post scheduling, unless we are somehow dropping information. Maybe we need to make sure that we don't wipe out status on the error case?

--brendan , David Oppenheimer
I also investigated the same issue last Friday. I observed exact the same as what described above, except the related kubelet log should be: Meanwhile, I observed master is trying to query pod / and failed all the time: Talked to offline, he is going to look into why running pods go back to Pending; meanwhile, I am going to figure out Unknown cases here.
That would be great. It is very hard for me to make sense of any of the scalability experiments that I am doing while the pod status is in hysteresis.
Just to summarize: (1) the Running<-Unknown flapping is caused by the line of code I mentioned earlier; (2) the reason that line of code is being hit on the master is due to two errors on the kubelet (the one I identified earlier was wrong; Dawn found these): I0209 06:42:09. 4704 ] GET /api/v1beta1/podInfo?podID=hostnames-v5drx&podNamespace=default: (8.82543048s) 500 E0209 04:34:13. 4996 ] Error getting info for pod default/hostnames-2y8ny: kubelet "kubernetes-minion-1.c.striped-torus-760.internal" server responded with HTTP error code 500 for pod default/hostnames-2y8ny (3) we don't yet know the cause of the Running<-Pending flapping; I'm going to look into that now
(and Dawn is looking into why we're hitting those two errors on Kubelet)
Are we serving requests concurrently in Kubelet? (Is it possible not to when using go's servemux?) I don't see any locking around the pod map accesses.
This was marked for the fixit but Dawn and I are working on it. Assigned to myself.
I am also investing time writing code to try and report what is going on with unstable cluster status. It would be good to know what you and Dawn are doing in case I am wasting my time. Thanks. , David Oppenheimer
This particular problem is easily reproduced manually using the instructions Tim listed, so I'm not sure it's necessary to write anything sophisticated to drive the cluster into this state.

Anyway, maybe give me and Dawn a couple of days and if we can't figure this out by then we can call in the cavalry.
Am I wrong, or are these instructions also within bounds of an e2e test? Just because our existing e2es only do one or two things doesn't mean we can't do more within O(minutes), assuming you're dealing with the same image.
The "flips back to pending" problem appears to be because updatePodStatus() writes to the cache but not back to the Pod's status. computePodStatus() initializes newStatus based on pod.Status, not based on the cached value of status. This second behavior seems wrong in light of the first behavior. However, I believe this only causes a problem in combination with my fix described earlier, which as a result doesn't actually do what I described (it doesn't "keep the status as-is" but rather copies it out of the pod's status; to "keep the status as-is" it should be copying it out of the cache). I can see if this fixes things. (We also should fix the root cause stuff on the node side, which Dawn is working on.)
Assign the issue to me ... I didn't see the flips between Pending and Running anymore. Don't know why. I consistently see flips between Unknown and Running on those heavily loaded nodes. The root issue is once a while docker inspect API call returns us EAGAIN. Please note that on my test nodes, I already increase NOFILE limit for docker daemon process. The proposed solution is that kubelet deals with all errors internally except EAGAIN one. In this case, computePodStatus logic in pod_cache does nothing when p.containerInfo.GetPodStatus returns an error. Working on a PR to fix this.
Dawn, my suggestion is that computePodStatus() should return the current cached value whenever it gets an error from Kubelet. I can implement this if you want.
, why shouldn't updatePodStatus simply write through to the pod's status? I'd think either we always read the cached value first, or we always write through to the pod and read from there. Only reading the cached value upon an error seems counter-intuitive to me...
why do you computePodStatus returns a cached value here?

The only place calls computePodStatus is updatePodStatus, which triggered by two operations: 1) periodically update all containers 2) the client asks for PodStatus, and there is no cached one, instead calling computePodStatus to retrieve one. I think a cleaner way is 1) Kubelet handles all the errors except this transient one. Of course, we need to make sure docker is auto-restart and there is no too many open files error on node side too. But they are separate issues and should be handled by node side through script or configure etc. 2) Now back to master side, if node status is ok, kubelet should return something; if there is an error from kubelet, it should be transient; and if there is a cached one, simply return cached one upon request; if no cached one, it should be generated by computePodStatus in next round. I have one working PR to avoid the flips between Running and Unknown, but with increasing number of Pending from node side caused by docker failure. Still investigating it...
I didn't understand "if there is an error from kubelet, it should be transient; and if there is a cached one, simply return cached one upon request; if no cached one, it should be generated by computePodStatus in next round." If you're talking about the caching in the podCache on the master, then what you wrote is exactly what I was suggesting. (If you're talking about some kind of caching on the Kubelet, then that's different, because I am not aware of cache for status on the Kubelet.) I think always writing back to the pod's status might be fine but I don't know why the podCache was designed the way it was, so I didn't want to change that without asking Daniel first (I think he's the one who created it). I wasn't suggesting to only read the cached value upon an error; I meant always read the cached value under all circumstances.
and here is my understanding. Correct me if I am wrong. UpdateAllContainers periodically updates all containers, but it gets the initial pods from etcd through this line . each from , UpdateAllContainers calls passes to . computes and returns a new status. Note that the status is computed based on , i.e., the information in etcd. updates the pod cache with the new status.

Note that the new status is not written to etcd. If you repeat the same steps again, you will start with the stale pod status in the etcd because does not check the cached value.
I have tested since last night. The flaps between Running, Pending and Unknown are gone for about 10 hour running 300 pods on 4 nodes except two scenario: 1) etcd issue: cat kube- | grep "status to unknown" I0212 20:18:30. ] node doesn't exist: 501: All the given peers are not reachable (Tried to connect to each peer twice and failed) [0] <nil, setting pod hostnames-yuy8d status to unknown ... This happens a couple of times last 10 hours. We need to figure out why. 2) NodeStatus becomes UNREADY. It happens once last 10 hours, and pods status on that node become Unknown which is expected. I didn't figure out why NodeStatus was UNREADY then, but I heard there is a network earlier this morning.
up to you whether to reopen this issue or file new ones for new issues.
I've also seen the "All the given peers are not reachable" message (see my second entry in this issue). I think that was what was referring to when he said "there is an etcd size limit you might be hitting too" but I may be mis-interpreting.
That error is the etcd client's catch-all for "something went wrong" - it's actually really bad that it obscures errors. , David Oppenheimer
, Tim Hockin wrote:
and Yes, that error message is cache-all error message, which need to investigation. In my particular case, it shouldn't be etcd size limits at all since we have that cluster built and upgraded more than 10 hours before, and all 300 pods were scheduled back then. There is no new events generated either from kubelet or other components since all bound pods are indeed happily running on the nodes. No new writes to etcd except NodeStatus, which is very less frequent and not much data. The problem of etcd here occurs when master tries to read from etcd with an error.


Question: {% set override_deb_sha1='81abef31dd2c616883a61f85bfb294d743b1c889' %} {% endif %} <ins> {% if override_deb_url == '' %} {% set override_deb_url=storage_base + override_deb %} {% endif %} </ins> {% if override_docker_ver != '' %} purge-old-docker-package: pkg.removed:

GCE updated docker, but only via a base-image update. AWS should match versions of docker, to slipstream off all the testing that GCE gets. This will be slightly complicated because docker 1.8.3 isn't in the bucket AFAICT. Thanks to for the tip.
This is done by


Question: } resyncPeriod := 1 * time.Millisecond r := NewReflector(lw, &api.Pod{}, s, resyncPeriod) <del> r.ListAndWatch(stopCh) </del> <ins> err := r.ListAndWatch(stopCh) if err != rerr { t.Errorf("expected exiting from err %v, got: %v", rerr, err) } </ins> if iteration != 2 { t.Errorf("exactly 2 iterations were expected, got: %v", iteration) }

I saw this as well. Log: Brief discussion in
I think I have a fix for this
Just submitted a PR that I believe fixes this. I was able to get the flakes locally w/o my PR and haven't seen any with it. PTAL!
how did you repro locally? I tried ran this test 1,000 times last night and it passed 1,000 times.
Change the resync period from 1 milli to 1 microsecond. You can also add 'stress -c 8' to the mix. On Friday, May 20, 2016, Matt Liggett wrote:
- FYI
sent which should fix this


Question: } // Update pod condition to unschedulable. podutil.UpdatePodCondition(&p1.Status, &v1.PodCondition{ <del> Type:    v1.PodScheduled, Status:  v1.ConditionFalse, Reason:  v1.PodReasonUnschedulable, Message: "fake scheduling failure", </del> <ins> Type:          v1.PodScheduled, Status:        v1.ConditionFalse, Reason:        v1.PodReasonUnschedulable, Message:       "fake scheduling failure", LastProbeTime: metav1.Now(), </ins> }) // Put in the unschedulable queue. q.AddUnschedulableIfNotPresent(p1)

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: In large clusters, events that trigger rescheduling of the unschedulable pods are frequent. For example, in a cluster with 1000 nodes, node updates, which are often just heartbeats, may happen at an average rate of 100/s. These events trigger rescheduling of unschedulable pods. The scheduler moves unschedulable pods to its active scheduling queue when such events are observed. There are two particularly problematic scenarios: are high-priority unschedulable pods and preemption does not help make them schedulable. For example, they may have node affinity that cannot be satisfied, or they may have pod affinity to other pods that do not exist. Such high priority pods may go to the head of the active scheduling queue and block everything behind them. are many pending pods with the same priority, but a number of them are unschedulable. When these unschedulable pods are moved to the active queue, they are expected to go behind other pods in the queue, but since our queue sorts pods by priority and these pods have the same priority, their order is unspecified. The Pop() algorithm of the heap swaps the head of the queue with the tail, effectively causing the heap to act similar to a LIFO. So, pods which are tried recently, go to the head with a high probability and are retried. This starves pods behind them in the queue. Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug /sig scheduling /assign
For case 1: How about add an field to record how many times scheduler have tried to schedule it? The bigger it is, the latter we should schedule it. Then we could consider to add it as a factor for sorting pods.
Yes, we basically need a . Another approach is to . We will pursue both of these approaches.
Well differentiating optionsounds better
Agree. On the one hand we need to reduce the frequency of unnecessary rescheduling, on the other hand we need to optimize the sorting algorithm.

And the would help to reduce the frequency. But I think "differentiate between node heartbeats vs. actual node updates" is a better approach.
Yes its a better approach indeed but what if we use the recursion there
We should have all three changes to address all known scenarios efficiently. The three changes are: back off mechanism fair sorting of pods in the scheduling queue smarter processing of node updates Node updates are not the only trigger for the scheduler to retry unschedulable pods. The scheduler also retries when running pods terminate, among a couple of other events. So, without having backoff mechanism a high priority unschedulable pod could still block head of the queue in clusters with high churn, for example, in clusters where many pods terminate frequently. Back off mechanism by itself addresses the issue in all scenarios, but the other two changes make the scheduling queue smarter and more efficient.
The three prs above are quite comprehensive : ).
All of the above items are done already.


Question: ( # ensures KUBECONFIG is set get-kubeconfig-basicauth <ins> get-kubeconfig-bearertoken </ins> echo echo "Kubernetes cluster is running." echo

I have a running cluster in vagrant, had to make some code changes and push back but fails with:
cc:
Adding to queue of vagrant stuff to look at tomorrow.
Fixed by


Question: numPDBViolatingPods := victims.numPDBViolations if numPDBViolatingPods < minNumPDBViolatingPods { minNumPDBViolatingPods = numPDBViolatingPods <del> minPDBViolatingNodes = nil </del> <ins> minNodes1 = nil lenNodes1 = 0 </ins> } if numPDBViolatingPods == minNumPDBViolatingPods { <del> minPDBViolatingNodes = append(minPDBViolatingNodes, node) </del> <ins> minNodes1 = append(minNodes1, node) lenNodes1++ </ins> } } <del> if len(minPDBViolatingNodes) == 1 { return minPDBViolatingNodes[0] </del> <ins> if lenNodes1 == 1 { return minNodes1[0] </ins> } // There are more than one node with minimum number PDB violating pods. Find // the one with minimum highest priority victim. minHighestPriority := int32(math.MaxInt32) <del> var minPriorityNodes []*v1.Node for _, node := range minPDBViolatingNodes { </del> <ins> var minNodes2 = make([]*v1.Node, lenNodes1) lenNodes2 := 0 for i := 0; i < lenNodes1; i++ { node := minNodes1[i] </ins> victims := nodesToVictims[node] // highestPodPriority is the highest priority among the victims on this node. highestPodPriority := util.GetPodPriority(victims.pods[0]) if highestPodPriority < minHighestPriority { minHighestPriority = highestPodPriority <del> minPriorityNodes = nil </del> <ins> lenNodes2 = 0 </ins> } if highestPodPriority == minHighestPriority { <del> minPriorityNodes = append(minPriorityNodes, node) </del> <ins> minNodes2[lenNodes2] = node lenNodes2++ </ins> } } <del> if len(minPriorityNodes) == 1 { return minPriorityNodes[0] </del> <ins> if lenNodes2 == 1 { return minNodes2[0] </ins> } // There are a few nodes with minimum highest priority victim. Find the // smallest sum of priorities. minSumPriorities := int64(math.MaxInt64) <del> var minSumPriorityNodes []*v1.Node for _, node := range minPriorityNodes { </del> <ins> lenNodes1 = 0 for i := 0; i < lenNodes2; i++ { </ins> var sumPriorities int64 <ins> node := minNodes2[i] </ins> for _, pod := range nodesToVictims[node].pods { // We add MaxInt32+1 to all priorities to make all of them >= 0. This is // needed so that a node with a few pods with negative priority is not

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: Fix the following : The basic logic in of using the is continuing reducing the nodes slices through , , and . It needs 4 slices to do this. This is my thought to reuse the slices: Use 2 nodes slices, such as , The reducing process becomes: -Clear -Clear - please give some suggestion, thanks :smile:
/sig scheduling
The challenge is that clearing a slice and reusing it still needs GC clean up, so it may not be effective. The size of slices shrink as we proceed through the function. So, having two slices and over-writing the elements in them can help reduce GC time, but this approach needs some additional checks to ensure we are not using stale elements.
To avoid clearing the slices, this is my thought: Declare to indicate the actual length of the 2 node slices, instead of . Appending a node to a node slice will add 1 to . Clearing a node slice will set to 0. WDYT
Yes, I can't think of any better solution either, but remember that you should append only to the slice in the first for loop of the function. The second slice should be initialized with the same number of elements as the slice used in the first for loop. Setting elements of the slices should be done by indexing them (not by append) in the second, third, and forth loop.
OK, I will submit a PR to implement it.


Question: validArgs := cmdutil.ValidArgList(f) cmd := &cobra.Command{ <del> Use: "get [(-o|--output=)json|yaml|wide|custom-columns=...|custom-columns-file=...|go-template=...|go-template-file=...|jsonpath=...|jsonpath-file=...] (TYPE [NAME | -l label] | TYPE/NAME ...) [flags]", </del> <ins> Use: "get [(-o|--output=)json|yaml|wide|custom-columns=...|custom-columns-file=...|go-template=...|go-template-file=...|jsonpath=...|jsonpath-file=...] (TYPE[.VERSION][.GROUP] [NAME | -l label] | TYPE[.VERSION][.GROUP]/NAME ...) [flags]", </ins> DisableFlagsInUseLine: true, Short:   i18n.T("Display one or many resources"), Long:    getLong + "nn" + cmdutil.SuggestApiResources(parent),

Is this a BUG REPORT or FEATURE REQUEST?: /kind feature /sig cli What happened: I have two custom resources in my cluster called Cluster, and couldn't figure out how to ask for one or the other. was of no help, and various Google searches for things like "kubectl apigroup" or "kubectl crd" revealed nothing of use. Worse, some bugs that came up like suggested that the syntax was , which produced entirely incorrect output (e.g. results in . Eventually, I stumbled by accident on the correct way to do it, which is to do and then , where is the name of the custom resource exactly as listed in the output of . I still don't know how to request a specific API version using this syntax, if that is even possible. What you expected to happen: I expected to document how to query custom resources, given that they're increasingly central to running production quality clusters with various operators. Failing that, I expected the k8s website to have documentation on querying custom resources that is discoverable via Google search. How to reproduce it (as minimally and precisely as possible): Install something that uses custom resources (e.g. Calico in k8s datastore mode, Rook), and try to one of the custom resources by precise name (e.g. assume that there are multiple objects defined by different CRDs, so you can't just , you have to be more precise). Environment: Kubernetes version (use ): 1.10.2 client+cserver
Opened with examples


Question: } } } <ins> func TestDeploymentController_cleanupDeploymentOrder(t *testing.T) { selector := map[string]string{"foo": "bar"} now := metav1.Now() duration := time.Minute newRSWithRevisionAndCreationTimestamp := func(name string, replicas int, selector map[string]string, timestamp time.Time, revision string) *apps.ReplicaSet { rs := rs(name, replicas, selector, metav1.NewTime(timestamp)) if revision != "" { rs.Annotations = map[string]string{ deploymentutil.RevisionAnnotation: revision, } } rs.Status = apps.ReplicaSetStatus{ Replicas: int32(replicas), } return rs } // for all test cases, creationTimestamp order keeps as: rs1 < rs2 < rs3 < r4 tests := []struct { oldRSs               []*apps.ReplicaSet revisionHistoryLimit int32 expectedDeletedRSs   sets.String }{ { // revision order: rs1 < rs2, delete rs1 oldRSs: []*apps.ReplicaSet{ newRSWithRevisionAndCreationTimestamp("foo-1", 0, selector, now.Add(-1*duration), "1"), newRSWithRevisionAndCreationTimestamp("foo-2", 0, selector, now.Time, "2"), }, revisionHistoryLimit: 1, expectedDeletedRSs:   sets.NewString("foo-1"), }, { // revision order: rs2 < rs1, delete rs2 oldRSs: []*apps.ReplicaSet{ newRSWithRevisionAndCreationTimestamp("foo-1", 0, selector, now.Add(-1*duration), "2"), newRSWithRevisionAndCreationTimestamp("foo-2", 0, selector, now.Time, "1"), }, revisionHistoryLimit: 1, expectedDeletedRSs:   sets.NewString("foo-2"), }, { // rs1 has revision but rs2 doesn't have revision, delete rs2 oldRSs: []*apps.ReplicaSet{ newRSWithRevisionAndCreationTimestamp("foo-1", 0, selector, now.Add(-1*duration), "1"), newRSWithRevisionAndCreationTimestamp("foo-2", 0, selector, now.Time, ""), }, revisionHistoryLimit: 1, expectedDeletedRSs:   sets.NewString("foo-2"), }, { // rs1 doesn't have revision while rs2 has revision, delete rs1 oldRSs: []*apps.ReplicaSet{ newRSWithRevisionAndCreationTimestamp("foo-1", 0, selector, now.Add(-1*duration), ""), newRSWithRevisionAndCreationTimestamp("foo-2", 0, selector, now.Time, "2"), }, revisionHistoryLimit: 1, expectedDeletedRSs:   sets.NewString("foo-1"), }, { // revision order: rs1 < rs2 < r3, but rs1 has replicas, delete rs2 oldRSs: []*apps.ReplicaSet{ newRSWithRevisionAndCreationTimestamp("foo-1", 1, selector, now.Add(-1*duration), "1"), newRSWithRevisionAndCreationTimestamp("foo-2", 0, selector, now.Time, "2"), newRSWithRevisionAndCreationTimestamp("foo-3", 0, selector, now.Add(duration), "3"), }, revisionHistoryLimit: 1, expectedDeletedRSs:   sets.NewString("foo-2"), }, { // revision order: rs1 < rs2 < r3, both rs1 && rs2 have replicas, don't delete oldRSs: []*apps.ReplicaSet{ newRSWithRevisionAndCreationTimestamp("foo-1", 1, selector, now.Add(-1*duration), "1"), newRSWithRevisionAndCreationTimestamp("foo-2", 1, selector, now.Time, "2"), newRSWithRevisionAndCreationTimestamp("foo-3", 0, selector, now.Add(duration), "3"), }, revisionHistoryLimit: 1, expectedDeletedRSs:   sets.NewString(), }, { // revision order: rs2 < rs4 < rs1 < rs3, delete rs2 && rs4 oldRSs: []*apps.ReplicaSet{ newRSWithRevisionAndCreationTimestamp("foo-1", 0, selector, now.Add(-1*duration), "3"), newRSWithRevisionAndCreationTimestamp("foo-2", 0, selector, now.Time, "1"), newRSWithRevisionAndCreationTimestamp("foo-3", 0, selector, now.Add(duration), "4"), newRSWithRevisionAndCreationTimestamp("foo-4", 0, selector, now.Add(2*duration), "2"), }, revisionHistoryLimit: 2, expectedDeletedRSs:   sets.NewString("foo-2", "foo-4"), }, } for i := range tests { test := tests[i] t.Logf("scenario %d", i) fake := &fake.Clientset{} informers := informers.NewSharedInformerFactory(fake, controller.NoResyncPeriodFunc()) controller, err := NewDeploymentController(informers.Apps().V1().Deployments(), informers.Apps().V1().ReplicaSets(), informers.Core().V1().Pods(), fake) if err != nil { t.Fatalf("error creating Deployment controller: %v", err) } controller.eventRecorder = &record.FakeRecorder{} controller.dListerSynced = alwaysReady controller.rsListerSynced = alwaysReady controller.podListerSynced = alwaysReady for _, rs := range test.oldRSs { informers.Apps().V1().ReplicaSets().Informer().GetIndexer().Add(rs) } stopCh := make(chan struct{}) defer close(stopCh) informers.Start(stopCh) d := newDeployment("foo", 1, &test.revisionHistoryLimit, nil, nil, map[string]string{"foo": "bar"}) controller.cleanupDeployment(test.oldRSs, d) deletedRSs := sets.String{} for _, action := range fake.Actions() { deleteAction, ok := action.(testclient.DeleteActionImpl) if !ok { t.Logf("Found not-delete action with verb %v. Ignoring.", action.GetVerb()) continue } if deleteAction.GetResource().Resource != "replicasets" { continue } deletedRSs.Insert(deleteAction.GetName()) } t.Logf("&test.revisionHistoryLimit: %d, &test.deletedReplicaSets: %v", test.revisionHistoryLimit, deletedRSs) if !test.expectedDeletedRSs.Equal(deletedRSs) { t.Errorf("expect to delete old replica sets %v, but got %v", test.expectedDeletedRSs, deletedRSs) continue } } } </ins>

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: when deployment controller cleans up old ReplicaSet, it sorts all old ReplicaSets by createTimeStamp and delete the oldest one, which may cause Deployment roll back to a wrong version. How to reproduce it (as minimally and precisely as possible): Suppose we set the revisionHistoryLimit of a Deployment to 3, and currently we have 3 Replicas: r1, r2, r3 (the createTimeStamp and Revision both are r3 r2 r1); if we upgrade and roll back 3 times continuously, there will be an expected case: upgrade:   ReplicaSet will be: r4, r3, r2   createTimeStamp: r4 r3 r2   reversion: r4 r3 r2 rollback: should rollback to v3   ReplicaSet will be: r4, r3, r2   createTimeStamp: r4 r3 r2   reversion: r3 r4 r2 upgrade:   ReplicaSet will be: r5, r4, r3   createTimeStamp: r5 r4 r3   reversion: r5 r3 r4 rollback: should rollback to v3   ReplicaSet will be: r4, r3, r2   createTimeStamp: r5 r4 r3   reversion: r3 r5 r4 upgrade:   ReplicaSet will be: r6, r5, r4   createTimeStamp: r6 r5 r4   reversion: r6 r5 r4 rollback: should rollback to r3, but instead, we rollbacked to r5, and this may cause error. What you expected to happen: when deployment controller cleans up old ReplicaSet, it should sort old ReplicaSets by Revision
/sig api-machinery /sig apps
does this make sense to you? /assign /cc
?
/remove-sig api-machinery
This requirement makes sense. Unassign API machinery - SIG Apps should own this.
Issues go stale after 90d of inactivity. Mark the issue as fresh with .

Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: Resource("pods"). Name(pod.Name). Namespace(pod.Namespace). <del> SubResource("exec"). Param("container", containerName) </del> <ins> SubResource("exec") </ins> req.VersionedParams(&corev1.PodExecOptions{ Container: containerName, Command:   p.Command,

<!-- --What happened: when use command, we found multi query param in the request, because we set the param twice in What you expected to happen: How to reproduce it (as minimally and precisely as possible): use command and set the log level to Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig cli
It seems a duplicate 'container' param are set, thanks for your report. I will fix it later.


Question: "storage.k8s.io/v1/csidrivers":                                     "hL6j/rwBV5w=", "storage.k8s.io/v1/csinodes":                                       "Pe62DkZtjuo=", "storage.k8s.io/v1/storageclasses":                                 "K+m6uJwbjGY=", <del> "storage.k8s.io/v1/csistoragecapacities":                           "4as6MA/kOg0=", </del> <ins> "storage.k8s.io/v1/csistoragecapacities":                           "xeVl+2Ly1kE=", </ins> "storage.k8s.io/v1/volumeattachments":                              "tJx/ezt6UDU=", <del> "storage.k8s.io/v1beta1/csistoragecapacities":                      "4as6MA/kOg0=", </del> <ins> "storage.k8s.io/v1beta1/csistoragecapacities":                      "xeVl+2Ly1kE=", </ins> "apps/v1/controllerrevisions":                                      "85nkx63pcBU=", "apps/v1/daemonsets":                                               "dd7pWHUlMKQ=", "apps/v1/deployments":                                              "8aSe+NMegvE=",

The override in can be removed in Kubernetes v1.25 because then the oldest supported kube-apiserver for downgrades will be v1.24 which will understand the v1 format. Also need to adapt See and Code cleanup...
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/sig storage
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale /lifecycle frozen


Question: } // Cloning fails if the source disk is still in the process of detaching, so we wait for the VolumeAttachment to be removed before cloning. volumeAttachment := e2evolume.GetVolumeAttachmentName(ctx, f.ClientSet, testConfig, l.testCase.Provisioner, dataSourceRef.Name, l.sourcePVC.Namespace) <del> e2evolume.WaitForVolumeAttachmentTerminated(ctx, volumeAttachment, f.ClientSet, f.Timeouts.DataSourceProvision) </del> <ins> framework.ExpectNoError(e2evolume.WaitForVolumeAttachmentTerminated(ctx, volumeAttachment, f.ClientSet, f.Timeouts.DataSourceProvision)) </ins> l.testCase.TestDynamicProvisioning(ctx) })

This line here lacks error checking: It shouldn't continue when that function times out. Add . No response <details</details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage
in StorageClassTest.TestDynamicProvisioning also fails in that test.
/assign Looks like it only times out on my local branch. I'll investigate that. However, the error check is missing on master.
Fixed, and I'll add the error check in
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/remove-lifecycle rotten Fix is in


Question: if err != nil { return err } <del> c.HealthzChecks = append(c.HealthzChecks, healthz.NamedCheck("etcd", func(r *http.Request) error { </del> <ins> c.AddHealthChecks(healthz.NamedCheck("etcd", func(r *http.Request) error { </ins> return healthCheck() }))

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: The and urls do not provide as much information about the cluster as much as does. It's expected to see and (if kms is enabled) but does not. What you expected to happen: and url to include and How to reproduce it (as minimally and precisely as possible): any 1.16 pre-release cluster the server url for , and Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: IBM OS (e.g: ):Kernel (e.g. ):Install tools: Network plugin and version (if this is a network-related bug): Others: Logs: healthz readyz livez /sig api-machinery
/sig api-machinery
/assign