Question: buildBackOffDuration = time.Minute syncLoopFrequency    = 10 * time.Second maxBackOffTolerance  = time.Duration(1.3 * float64(kubelet.MaxContainerBackOff)) <del> // maxReadyStatusUpdateTolerance specifies the latency that allows kubelet to update pod status. // When kubelet is under heavy load (tests may be parallelized), the delay may be longer, hence // causing tests to be flaky. maxReadyStatusUpdateTolerance = 10 * time.Second </del> ) // testHostIP tests that a pod gets a host IP

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: Which test(s) are failing: Since when has it been failing: It has been flaking more frequently since 3/13. Testgrid link: Reason for failure: The issue seems to be a timeout cc /kind flake /sig node /priority important-soon /milestone v1.14
Latest PR to touch that test May be due to a readiness probe failing.
/remove-kind failing-test sending this your way for triage too - could you have a look please? I wonder if it's a node problem or if timeouts are originating elsewhere fwiw, it's failing about 10%-15% of the time in e2e tests:
It looks like updating the status after the patch to the readiness gate takes longer than expected.

Looking at this failure: From the test logs: [It] should support pod readiness gates [NodeFeature:PodReadinessGate] ï¿½[1mSTEPï¿½[0m: submitting the pod to kubernetes ï¿½[1mSTEPï¿½[0m: patching pod status with condition "" to true ï¿½[1mSTEPï¿½[0m: patching pod status with condition "" to true Mar 14 01:52:56.720: INFO: Expect the Ready condition of pod "pod-ready" to be true, but got false Mar 14 01:52:57.731: INFO: Expect the Ready condition of pod "pod-ready" to be true, but got false Mar 14 01:52:58.725: INFO: Expect the Ready condition of pod "pod-ready" to be true, but got false Mar 14 01:52:59.707: INFO: Expect the Ready condition of pod "pod-ready" to be true, but got false Mar 14 01:53:00.721: INFO: Expect the Ready condition of pod "pod-ready" to be true, but got false Mar 14 01:53:01.708: INFO: Expect the Ready condition of pod "pod-ready" to be true, but got false Mar 14 01:53:02.753: INFO: Expect the Ready condition of pod "pod-ready" to be true, but got false Mar 14 01:53:03.707: INFO: Expect the Ready condition of pod "pod-ready" to be true, but got false Mar 14 01:53:04.713: INFO: Expect the Ready condition of pod "pod-ready" to be true, but got false Mar 14 01:53:05.709: INFO: Expect the Ready condition of pod "pod-ready" to be true, but got false Mar 14 01:53:05.767: INFO: Expect the Ready condition of pod "pod-ready" to be true, but got false From the kubelet logs: The pod goes to Ready at 1:53:12, and we stopped checking at 1:53:05 This looks expected, based on the description of the parameter in the test. I don't think this should be release-blocking.

/assign who owns this feature to determine how we can make the test less flaky. Feel free to close if this is already tracked somewhere else /unassign
Thanks, good to know. if it could be worked on, it would really help stabilize CI - which would be super appreciated, especially for release-blocking jobs.
When this tests was (2 or 3 release back), I remember the 10 second waiting timeout is enough for kubelet to react and update the pod ready condition. IIRC, kubelet reconciles pod status every 15s. But if pod readiness gate condition is updated, kubelet should reconcile ASAP skipping the 15s period. There might be a bug here.
Thanks - do you have time to take a look? It would help to know (especially if we need to include something in known issues).
I just did a few manual tests. Kubelet reacted very quickly. Not able to reproduce the update latency more than 10s.
I ran the test in a loop more than 50 times now on my own e2e cluster. Not a single failure. Is it because there are many tests running in parallel causing high load on kubelet, hence longer update latency?
Interesting, could well be the case Is there a way to validate that? And, is there something here we should raise in terms of performance/scalability (e.g. "update latency varies with pod load")?
That is a well-known result of the rate limiting of the kubelet. The kubelet can only send 5 qps of updates to the API Server, so the kubelet is limited in its ability to update statuses when there are lots of updates to send, which occurs when there is high pod churn.
This issue seems to not be a problem anymore. /close
Closing this issue. <detailsIn response to : This issue seems to not be a problem anymore. /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Should this test be marked [Serial] then? (Or maybe flaky?) running into this again
Can you share the test grid link where it is flaky?
I've been testing extensively myself offline to re-evaluate tests to use in presubmit.

this sounds like inherently this test is not safe to run with lots of other tests in parallel, which would make it a candidate for
/reopen This is the flakiest test for PRs Triage link: Since this is a GA feature it should have conformance tests covering it, this specific test was prevented from promotion to conformance due to flakiness issues called out here One of the comments suggested rewriting to use an informer or poll for longer than 10s, and not use an explicit sleep. Tagging with [Serial] seems like it would work from a brute force perspective. Is the 10s number an important or documented SLO somewhere?
Reopened this issue. <detailsIn response to : /reopen This is the flakiest test for PRs Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
another data point of sorts: IIRC I disabled this test in kind because it was too flaky, i wasn't aware that it was also flaky on GCE.
opened to deflake
Still seen after disabling auto flake retry:
top-ten flake according to
Confirming this flake is still top-ten after we merged Thanks for all your work here !
no longer top-ten, but still flaky -
back in the top ten according to
/assign assigning sig-node leads for visibility/routing
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale /remove-priority critical-urgent /priority important-longterm Not in the top ten, but it still flakes occasionally
I'm pretty sure we still blacklist this test in most kind CI because it's too flaky. Aaron Crickenberger < :
/milestone v1.19
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or .

/lifecycle stale
Gonna push this with sig node /cc /remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale It's still failing across a number of jobs but I can't tell that it's as painful a flake today
/milestone clear
/assign /triage accepted
GitHub didn't allow me to assign the following users: fromanirh. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign /triage accepted Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
working to reproduce outside CI - way easier to debug
still working my way towards a reproducer.
has been unable to reproduce locally. We are still seeing occasional timeouts in CI: Since this was the only timeout on that test run, we may want to consider tweaking the timeout thresholds?
I'm happy I have time again to work on this issue. It doesn't reproduce locally on BM, and this seems indeed to point to a too busy worker in CI and/or too short timeout. So let's start with the simplest fix:
Today in , it is the top 1 flake for "pr:pull-kubernetes-e2e-gce-ubuntu-containerd".
ok, time for me to prioritize this work
Do we have a testgrid link? I'm looking at to try to understand better the flakes and getting too little from here (but looks relatively good from here?)

More recent triage link: Appears to be failing in a number of containerd and crio jobs. /unassign /priority important-soon /remove-priority important-longterm
/milestone v1.22
Did we vet that this is fixed after ?


Question: return err } <ins> getPodFn := func(namespace, name string) (*api.Pod, error) { return o.client.Core().Pods(namespace).Get(name) } if len(policyGroupVersion) > 0 { return o.evictPods(pods, policyGroupVersion, getPodFn) } else { return o.deletePods(pods, getPodFn) } } func (o *DrainOptions) evictPods(pods []api.Pod, policyGroupVersion string, getPodFn func(namespace, name string) (*api.Pod, error)) error { doneCh := make(chan bool, len(pods)) errCh := make(chan error, 1) </ins> for _, pod := range pods { <del> if len(policyGroupVersion) > 0 { err = o.evictPod(pod, policyGroupVersion) } else { err = o.deletePod(pod) } if err != nil { </del> <ins> go func(pod api.Pod, doneCh chan bool, errCh chan error) { var err error for { err = o.evictPod(pod, policyGroupVersion) if err == nil { break } else if apierrors.IsTooManyRequests(err) { time.Sleep(5 * time.Second) } else { errCh <- fmt.Errorf("error when evicting pod %q: %v", pod.Name, err) return } } podArray := []api.Pod{pod} _, err = o.waitForDelete(podArray, kubectl.Interval, time.Duration(math.MaxInt64), true, getPodFn) if err == nil { doneCh <- true } else { errCh <- fmt.Errorf("error when waiting for pod %q terminating: %v", pod.Name, err) } }(pod, doneCh, errCh) } doneCount := 0 // 0 timeout means infinite, we use MaxInt64 to represent it. var globalTimeout time.Duration if o.Timeout == 0 { globalTimeout = time.Duration(math.MaxInt64) } else { globalTimeout = o.Timeout } for { select { case err := <-errCh: </ins> return err <ins> case <-doneCh: doneCount++ if doneCount == len(pods) { return nil } case <-time.After(globalTimeout): return fmt.Errorf("Drain did not complete within %v", globalTimeout) </ins> } } <ins> } </ins> <del> getPodFn := func(namespace, name string) (*api.Pod, error) { return o.client.Core().Pods(namespace).Get(name) </del> <ins> func (o *DrainOptions) deletePods(pods []api.Pod, getPodFn func(namespace, name string) (*api.Pod, error)) error { // 0 timeout means infinite, we use MaxInt64 to represent it. var globalTimeout time.Duration if o.Timeout == 0 { globalTimeout = time.Duration(math.MaxInt64) } else { globalTimeout = o.Timeout } for _, pod := range pods { err := o.deletePod(pod) if err != nil { return err } </ins> } <del> _, err = o.waitForDelete(pods, kubectl.Interval, o.Timeout, getPodFn) </del> <ins> _, err := o.waitForDelete(pods, kubectl.Interval, globalTimeout, false, getPodFn) </ins> return err } <del> func (o *DrainOptions) waitForDelete(pods []api.Pod, interval, timeout time.Duration, getPodFn func(string, string) (*api.Pod, error)) ([]api.Pod, error) { </del> <ins> func (o *DrainOptions) waitForDelete(pods []api.Pod, interval, timeout time.Duration, usingEviction bool, getPodFn func(string, string) (*api.Pod, error)) ([]api.Pod, error) { var verbStr string if usingEviction { verbStr = "evicted" } else { verbStr = "deleted" } </ins> err := wait.PollImmediate(interval, timeout, func() (bool, error) { pendingPods := []api.Pod{} for i, pod := range pods { p, err := getPodFn(pod.Namespace, pod.Name) if apierrors.IsNotFound(err) || (p != nil && p.ObjectMeta.UID != pod.ObjectMeta.UID) { <del> cmdutil.PrintSuccess(o.mapper, false, o.out, "pod", pod.Name, false, "deleted") </del> <ins> cmdutil.PrintSuccess(o.mapper, false, o.out, "pod", pod.Name, false, verbStr) </ins> continue } else if err != nil { return false, err

In writing some documentation for I noticed some issues with the implementation. I think the simplest user experience is that the command doesn't return until all the user pods are drained off the machine. In particular this means wait until all graceful termination is finished infinitely retry evictions that are rejected due to insufficient PodDisruptionBudget The current implementation does not do this, in particular wait for graceful termination appears to be at 10 sec number of pods, which may be much shorter than the max graceful termination we only retry evictions number of times My suggestion is to change the behavior so that we compute how long to wait for graceful termination, by taking max(grace periods of all pods that we want to delete) (or the value of --grace-period command line argument instead, if that is specified), and wait exactly that amount of time plus a small buffer we retry evictions infinitely WDYT? I think we should do this in 1.5 because the current behavior is very non-obvious.
The problem here is how long should we wait, if the user doesn't specify a timeout. The default timeout used in drain is following how long the reapers have in , , and . Maybe this is not long enough. I can make the changes. Do we still need exponential back off?
To summarize discussion with there are two possible approaches (pseudocode below) 1) Goroutine approach: For each pod P that meets the , do the following: 2) Non-goroutine approach: where "not timed out yet" means you haven't finished waiting max(--grace period argument to kubectl drain, max graceful termination spec across all pods on the node)
Could you please triage this for 1.5 and apply blocker or non-blocker labels as appropriate along with priority.
The PR for this issue is waiting to be merged.
marking this as a release blocker due to being marked as such. Please update if needed cc:


Question: validSeccomp := validPSP() validSeccomp.Annotations = map[string]string{ seccomp.DefaultProfileAnnotationKey:  "docker/default", <del> seccomp.AllowedProfilesAnnotationKey: "docker/default,unconfined,localhost/foo", </del> <ins> seccomp.AllowedProfilesAnnotationKey: "docker/default,unconfined,localhost/foo,*", </ins> } validDefaultAllowPrivilegeEscalation := validPSP()

It should be possible to create a PodSecurityPolicy with zero restrictions. This was the intent with the (a.k.a. "AllowAny") allowed seccomp profile, but it is rejected during the validation phase. /kind bug
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/auth: Issue will be escalated to these SIGs if needed. priority/important-soon: Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. kind/bug: Fixes a bug discovered during the current release. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details


Question: } } <ins> sort.Slice(volumesToReportInUse, func(i, j int) bool { return string(volumesToReportInUse[i]) < string(volumesToReportInUse[j]) }) </ins> return volumesToReportInUse }

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: is not sent in a stable sort order. this makes node status patch requests larger than needed, and makes debugging nodes harder than necessary. What you expected to happen: stable sort of volumesInUse How to reproduce it (as minimally and precisely as possible): always Anything else we need to know?: nope Environment: n/a
/sig node
- this is a good getting started item.
I've already submitted a PR in . PTAL. Thanks.


Question: } func newTestGenericPLEG() *TestGenericPLEG { <ins> return newTestGenericPLEGWithChannelSize(largeChannelCap) } func newTestGenericPLEGWithChannelSize(eventChannelCap int) *TestGenericPLEG { </ins> fakeRuntime := &containertest.FakeRuntime{} clock := clock.NewFakeClock(time.Time{}) // The channel capacity should be large enough to hold all events in a

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: In our stress testing, we found the deadlock issue in PLEG relist() for healthy checking and Kubelet syncLoop() The goroutine stack trace is as following We did some investigation, and found the following issue. The main loop for event processing in Kubelet will check if there are some runtimeErrors: if yes, wait until runtime back to normal; if no, it will call syncLoopIteration to read and process events. And the runtime healthChecks depends on the But in the relist() impl, it will send the pod lifecycle event to eventChannel which may introduce deadlock when eventChannel is full. And in the above code, the eventChannel will not be consumed if the runtime is not healthy. In some situation, the eventChannel may full, and the relist() will hang in sending new pod lifecycle event. And the runtime status will not be able to recovered. It is related to issues What you expected to happen: No such issue How to reproduce it (as minimally and precisely as possible): It is not easy to reproduce, it happens after few days stress testing (scaling the deployment repeatedly with high load). Anything else we need to know?: Environment: Kubernetes version (use ): 1.11.5 Cloud provider or hardware configuration: Bare metal OS (e.g. from /etc/os-release): CentOS 7.4 Kernel (e.g. ): Linux bjc-qtt-k8s-node-010 3.10.0-693.2.2.el7.x8664 SMP Tue Sep 12 22:26:13 UTC 2017 x8664 x8664 x8664 GNU/Linux Install tools: kubeadm Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
/sig node
Sure, thank you We had some internal fix to break the deadlock, but I am wondering if there are good fix for such kind of issue.
When pleg do relist, can we invoke to judge runtime is ok, if not, not do relist
I submit a PR
Thanks for finding the issue! The issue description makes sense to me. I think PLEG is a core component, we should try our best to unblock it.

The event channel becoming full seems more like an issue in the event consumer side, which shouldn't block PLEG. One solution is to queue the events in PLEG and send later, but that doesn't fundamentally solve the problem, if we don't set a limit on the event number we queue, it seems like a potential memory leak to me; if we set a limit, we may eventually hit this problem again. Another solution is to add a retry for sending event in PLEG, and set a bit in PLEG in between the retry which tells that PLEG is not blocking, it is trying to send out the event. And in the function of PLEG, it returns healthy if the bit is set. Actually, even if we fail to send the event and discard it, since we have already updated the pod cache, the periodical pod resync should eventually pick it up. This may delay pod sync for up to . But if the channel is full, it means that either 1) the syncloop is blocked for some reason, or 2) the node is under heavy loads. In both cases, even the event is successfully sent, the pod sync will likely be delayed anyhow. We can: 1) Enlarge the event channel size to reduce the chance that the channel becoming full. Currently it is arbitrarily set as . 2) Discard the event in PLEG if the channel is already full. In the 3 options above, I'm fine with either 2 or 3. Option 2 doesn't have the 1min pod sync delay, but a bit ad-hoc; option 3 is more generic, but may delay pod sync. There might be other options. /cc Any thoughts?
If PLEG is already blocked by a full event channel, this doesn't help right?
cc/ This looks like the one production issue you mentioned to me a while back.
of options enumerated, option 3 was preferred for me.
Thanks for reporting the issue! what was the load on the node that triggered this condition? If the node wasn't function at all, and the events slowly built up over time, bumping the size of the channel won't help. This should work, and doesn't complicate the logic in PLEG more, so it's my preferred option too.

We need some unit tests, and ideally, a stress test, as mentioned in the sig-node meeting.
Agree, giving a proper size for event channel is tricky as the scale also grows with time. We will ship fix as option 3 () for 1.14. Thanks for summarizing all these up as sig meeting! /assign
If possible, could you share in how large the stress is? It helps understand the issue and kubelet stress handling more. :)
Sure, our product team ( ) will input the stress test data after removing sensitive information in follow up comments.
Thanks for reply. Following is our testcase to reproduce this issue: The test cluster has 5 node on aliyun ECS(like EC2). Three 4C8G machine as master node, Two 96C384G as worker node. Firstly, We create 2 deployment running on the worker node. One deployment is http server pods. Another is siege pods to produce network stress to http server deployment. Those stress test deployments produce 50 CPU Core load on worker node. Then we create a job frequently create/delete 20 pod every 5 seconds. After about 1 day of stess test, The worker node will become NotReady. Dig the pprof goroutine stacktrace found the channel block in .
Yeah, I can see how frequent container state change can cause this issue. The number is very useful, thanks a lot!
cloud you please help to share how to dump the goroutines stack trace of kubelet like what you pasted in your comment. thanks a lot.
This issue is fixed by
using pprof is a answer to my question thank you anyway.


Question: flags.StringVar(&TestContext.RepoRoot, "repo-root", "../../", "Root directory of kubernetes repository, for finding test files.") flags.StringVar(&TestContext.Provider, "provider", "", "The name of the Kubernetes provider (gce, gke, local, skeleton (the fallback if not set), etc.)") flags.StringVar(&TestContext.Tooling, "tooling", "", "The tooling in use (kops, gke, etc.)") <del> flags.StringVar(&TestContext.KubectlPath, "kubectl-path", "kubectl", "The kubectl binary to use. For development, you might use 'cluster/kubectl.sh' here.") </del> flags.StringVar(&TestContext.OutputDir, "e2e-output-dir", "/tmp", "Output directory for interesting/useful test data, like performance data, benchmarks, and other metrics.") flags.StringVar(&TestContext.Prefix, "prefix", "e2e", "A prefix to be added to cloud resources created during testing.") flags.StringVar(&TestContext.MasterOSDistro, "master-os-distro", "debian", "The OS distribution of cluster master (debian, ubuntu, gci, coreos, or custom).")

Which jobs are failing: ci-kubernetes-node-kubelet-orphans Which test(s) are failing: [sig-storage] EmptyDir volumes pod should support shared volumes between containers Since when has it been failing: Testgrid link: Reason for failure:
/sig testing
/assign
Thank for comment
Tests are now passing, /close
Closing this issue. <detailsIn response to : Tests are now passing, Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: fmt.Sprintf("the type of the condition is not unique, it also appears in conditions[%d]", ii))) } seenType[condition.Type] = i <del> for _, msg := range utilvalidation.IsQualifiedName(string(condition.Type)) { allErrs = append(allErrs, field.Invalid(fldPath.Index(i).Child("type"), string(condition.Type), msg)) } for _, msg := range utilvalidation.IsQualifiedName(string(condition.Status)) { allErrs = append(allErrs, field.Invalid(fldPath.Index(i).Child("status"), string(condition.Type), msg)) } </del> <ins> allErrs = append(allErrs, apivalidation.ValidateQualifiedName(string(condition.Type), fldPath.Index(i).Child("type"))...) allErrs = append(allErrs, apivalidation.ValidateQualifiedName(string(condition.Status), fldPath.Index(i).Child("status"))...) </ins> if condition.Reason == "" { allErrs = append(allErrs, field.Required(fldPath.Index(i).Child("reason"), "reason cannot be empty")) }

After got merged, we have new helper function here: It will be good for us to use this simpler helper function to replace previous function. /good-first-issue
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : After got merged, we have new helper function here: </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig usability
I would like to take this one /assign
I worked in on this as well, but kept your changed files untouched (as written in my PR description). same for you. I hope this is OK for you.
This should be reopened until and got merged (my PR excluded the changes from the other PRs).
/reopen per last comment
Reopened this issue. <detailsIn response to : This should be reopened until and got merged (my PR excluded the changes from the other PRs). per last comment Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
can you assign me
/assign
/assign
I'd love to work on this issue.

/assign
Hi, , I am a first commitor for kubernetes and I have committed a PR in , could you help me to review it, or give me some suggestions. Thanks a lot.
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: var _ = framework.KubeDescribe("Pods", func() { f := framework.NewDefaultFramework("pods") <ins> var dc dynamic.Interface </ins> var podClient *framework.PodClient ginkgo.BeforeEach(func() { podClient = f.PodClient() <ins> dc = f.DynamicClient </ins> }) /*

This Issue replace Watch events may be missed and the test may still pass. We need to ensure that all expected watch events are seen. Watch events are not collected and checked at the end of the test. In order to solve the problem, we must collect all watch events as they come in and perform a check at the end of the test to ensure that the order is correct and they contain no errors.
/sig testing /area conformance /assign /cc
This Issue replace
AI: Must allow for other events to occur through the expected watch events stream expected watch events: must allow the following to be valid:
[x] Set up a watch for resource through a static label to the resource with DynamicClient [x] Manage the watch if it dies [x] Feed the watch to the function [x] Read all the watch events from the at the end of the test, ensuring that the events occur question: What does doing this mean to tests which check data? (e.g: - waits for values, sourced from watch) [x] Retry the test, upon failure [x] Manage the resources on retry (delete)
More or less, yes. If we need to read results mid-test, I think that will be OK, rather than waiting for the end. The main point is to wrap the watch such that it gets restarted if it fails. That wrapper will need to sit between the raw watch channel and the test: raw watch -channel -wrapper (restarts watch if closes/fails prematurely) -channel consumed in test I'm a little surprised there isn't already a sort of "buffered watch" object that does this already ? You'll also need a stop channel for the go routine started by the wrapper, so that at the end of the test you can stop that go routine and watch.
Also, if we have the "protected watch" wrapper, not sure we need to retry the test on failure. AFAIU, the only reason we wouldn't see the watch events we expected would be because of etcd compaction, which would not be expected to happen in e2e clusters, and even if it did it would not happen to event histories that are brand new (i.e., it only compacts old history). Again would like to confirm.
/reopen While we did push through this first iteration, this ticket is for ongoing discussion of watch tooling and refactoring of the approach.
Reopened this issue.

<detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen While we did push through this first iteration, this ticket is for ongoing discussion of watch tooling and refactoring of the approach.
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

[x] Original approval issue: - Reverted due to flakes [x] Two weeks soak end date: 20 November 2020 This issue is created to allow edit by /wip /hold According to this APIsnoop query, there are still some remaining Pod endpoints which are untested. a Pod with a static label the Pod with a new Label and updated data the Pod to ensure it's patched the Pod's status the Pod's status Ready condition to False the Pod's status to check if Ready is False all Pods in all Namespaces find the Pod(1) ensure that the Pod is found and is patched Namespaced Pod(1) via a Collection with a LabelSelector the Pod to check that it's deleted Discover useragents: List endpoints hit by the test: Display endpoint coverage change: If a test with these calls gets merged, test coverage will go up by 4 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/reopen Test PR Merge Promotion PR to be created next
Reopened this issue. <detailsIn response to : /reopen Test PR Merge Promotion PR to be created next Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
reverted by due to flaky test
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
/reopen PR on test grid for 2 weeks before promotion to conformance
Reopened this issue. <detailsIn response to : /reopen PR on test grid for 2 weeks before promotion to conformance Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage accepted


Question: livenessProbe: httpGet: path: /healthz <del> port: 8080 </del> <ins> port: 8082 </ins> scheme: HTTP initialDelaySeconds: 180 timeoutSeconds: 5

liveness probe was introduced in Heapster seems to work correctly so we should either fix it or revert liveness probe.
cc
port seems to be wrong


Question: obj := fixtures.NewNoxuInstance("default", "foo") unstructured.SetNestedField(obj.UnstructuredContent(), int64(42), "metadata", "unknown") <ins> unstructured.SetNestedField(obj.UnstructuredContent(), nil, "metadata", "generation") </ins> unstructured.SetNestedField(obj.UnstructuredContent(), map[string]interface{}{"foo": int64(42), "bar": "abc"}, "metadata", "labels") _, err = instantiateCustomResource(t, obj, noxuResourceClient, noxuDefinition) if err == nil {

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: CRDs created with old Kubernetes version are missing : Example: CRD created recently: $ curl -s '…' {creationTimestamp, generation}' { "creationTimestamp": "2018-07-23T17:24:58Z", "generation": 1 } CRD created with one of the old versions: $ curl -s 'https://…' {creationTimestamp, generation}' { "creationTimestamp": "2018-04-11T12:38:47Z", "generation": null } What you expected to happen: I'd expect to be always present so I can use it in controllers and monitoring without resorting to ugly hacks. How to reproduce it (as minimally and precisely as possible): Create a CRD with an old version of Kubernetes with CRD subresources disabled (not sure which one, 1.10 or maybe 1.9). Update the cluster to 1.11. Create another CRD. Observe that is always present in the new one, but doesn't show up in the old one. Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-09-09T18:02:47Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-09-09T17:53:03Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: AWS OS (e.g. from /etc/os-release): CoreOS 1800.7.0 Kernel (e.g. ): 4.14.63-coreos Install tools:
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available .

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Custom resources start setting as 1 during creation from v1.10 because that's when the subresources feature was introduced. The subresources feature says that I want to follow the spec/status conventions and hence, want generation to be initialized to 1 and be incremented whenever spec changes. In <=v1.9, even if we initialized it to 1, it does not hold much meaning because is not allowed to be modified during updates. It is also not incremented at all, since we have no way of knowing if the user wants to follow spec/status conventions. So it will always remain as 1 and won't be helpful in controllers. The condition is surprising. In the older version, if it is absent with , it should also be absent with . I am not able to reproduce this though. would you know on which version you tested the export against?
/area custom-resources /cc
I don't expect anything from old clusters, that's completely understandable. However, if I have a 1.11 cluster with subresources enabled, I expect all CRDs to behave in the same way, not just the ones that were created after 1.10. This doesn't seem to be the case. Version information is in the original issue. All our clusters are currently running v1.11.3. I think that some of the buggy CRDs were created with v1.10 (without CRD subresources enabled), but it might've been v1.9.
I think the older cluster version here is 1.9 because we always set generation to 1 during creation from 1.10: 10 did change the default to 1. However, I am not sure if we should be changing the default values for a 1.9 cluster now. Moreover, we maintain only 3 releases at a time - so I think we only cherry-pick to 1.12, 1.11 and 1.10 now (?). thoughts?
What I'm trying to say is that clusters running 1.11 should automatically handle resources with missing or unset by updating it to 1. It doesn't make sense to keep them in this broken state forever just because they were created in one of the previous versions.

Actually, after editing the resource the is updated to 1, so I can just default to in the controller. For some reason I thought that the resource stays broken forever, which is not the case. Sorry about that!
We can still do this, but I'm wondering if that's ok or not because that wouldn't represent the "true" generation value for that resource. Have the same questions for , which initializes and increments generation for all other resources. :thinking:
I've updated the issue since export is not relevant (and I think the logic resets generation to 1). However, the fact that for old CRDs is (internally) set to 0 is still problematic. Any custom controllers or monitoring code that checks will not be able to tell whether they've been processed successfully by a controller or not (because both and are missing).
/assign
Are you suggesting we set generation on read if it is missing?
That would be one option, or just patch them once after update.
it would probably make sense to make this part of metadata coercion on read, based on whether the CRDs has spec/status enabled
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
Hello! I'm bug triage lead for the 1.16 release cycle and considering this issue is tagged for 1.16, but not updated for a long time, I'd like to check its status. The code freeze is starting on August 29th (about 1.5 weeks from now), which means that there should be a PR ready (and merged) until then. Do we still target this issue to be fixed for 1.16?
No idea, I'm not a maintainer.
Yes


Question: return err } } <ins> for _, memberId := range pool.MemberIDs { err = members.Delete(lb.network, memberId).ExtractErr() if err != nil && !isNotFound(err) { return err } } </ins> err = pools.Delete(lb.network, pool.ID).ExtractErr() if err != nil && !isNotFound(err) { return err

I am using openstack as the cloud provider, and the error could be reproduced like this: It looks so far so good, but after restarting the control plane.
logs from kube-controller-manager:
When checking the function here: I doubt that the process about deleting a vip, the deletion order expressed there is: get the vip. delete the vip. disassociate monitor. delete monitor. delete pool. why don't we check the members of the pool, and delete them before we delete the pool?
Hrm, that error message looks like it's generated only when a vip refers to the pool: (and from the URL I can see you're using lbaas v1) If it isn't too difficult, could you manually create a vip+pool+member and confirm exactly what conditions produce that error? (I'm afraid I only easily have access to lbaasv2 setups atm) Another possibility is that we didn't wait long enough after deleting the vip, etc before attempting to delete the pool.
Thanks for your attention, I already fixed it by removing the members, and the issue was gone after that. :) as far as I know, before all the references are removed, the pool would not be able to delete, so either the vip refers to it or the member refers to it. I do not have a deployment of LBaaS v2, but by reading the code, I am pretty sure that it deletes the member before deleting the pool as well. checking here:
Meh, if it works, then I'm good with that. LGTM.


Question: <del> 1.0 </del> <ins> 1.10 </ins>

[sig-api-machinery] Aggregator Should be able to support the 1.7 Sample API Server using the current Aggregator More nuanced logs and details can be found at gave up waiting for apiservice wardle to come up successfully Expected error: <errors.errorString | 0xc420131150: { s: "timed out waiting for the condition", } timed out waiting for the condition not to have occurred /priority failing-test /priority important-soon /kind flake /sig-api-machinery /assign can you please help triage this issue?
has made changes that could have affected this recently. made one that'd be even more suspicious but I don't think it has merged yet :) :)
flakes show steady 2-3/day over the past week: which changes are you referring to? the line that's failing isn't doing any discovery, restmapping, etc, at all... just a straight get to the API:
audit log from the test run shows the requests are returning 404s, not 503s
ah, sorry, I misread the log and thought it was a discovery error. Hm.
hmm, there's a second smaller set of failures that are discovery related: looks like those started flaking after which went in on 5/4 looks like some discovery paths had internal retries that could have masked those 404 errors before. since the error that test is encountering is for an unrelated CRD API that is being created/deleted as part of another test, it makes more sense to switch that check to just ensure the extension API group under test is discovered, rather than assert there are no errors looking for other API groups that we know are being dynamically /removed opened to deflake the aggregator e2e check
/assign resolved the discovery issue. now the failure is the one noted in where calls to the extension API are responding with 404 errors
Kubelet log is reporting that the sample-apiserver exited with a return code of 2 and it also shows with a high restart count for the sample-apiserver.

When I extract the sample-apiserver log I'm seeing multiple "panic: Get https://10.0.0.1:443/api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentication: dial tcp 10.0.0.1:443: getsockopt: connection refused" Both failures today are in ipvs clusters which seems to have extensive networking problems. /assign
We are looking into finding the root cause into the problems with the ipvs clusters. I am pretty confident they were triggered by a bad PR, just need to identify which one.
Initial hypothesis is that PR caused it.
We think should fix the IPVS issues.
I see merged and this test is now passing in both pull-kubernetes-e2e-gce and ci-kubernetes-e2e-gci-gce. Hence closing the issue. Thanks all
/reopen The test is failing for the same error again in the following jobs https://k8s- It is one of the top flake from last week failing 47 jobs.
can you please take a look and triage? Thanks
this flake is now blocking this test from being promoted to Conformance suite in 1.11. Is it something we can investigate and resolve soon (today or tomorrow) to increase changes of getting this into 1.11? If not this test has to wait until 1.12 to be promoted.
all I see in the test run logs is 404 errors coming back from that check. can you see anything useful in the container logs for the aggregated server?
/cc
Update: I have been trying to reproduce this failure on my own test cluster (go run ...) and it has passed 128 times in a row so far. I think this means that the issue must be with either an interaction between e2e tests or an issue with the job running the tests.
Looking through recent results this crops up as a flake in presubmit and CI across many configurations... I don't think how we run the test jobs should be affecting this in the slightest. Not sure what's wrong here, but it looks like a pretty typical flakey test (some race / buggy setup/teardown?) nb that may not trivially map to the same way the tests are run though, and itself is a wrapper over
does the API server logs throw any light on whats happening on CI test cluster.

The teardown for the aggregator test is wrong, it tries to delete the deployment "sample-apiserver" when the deployment is actually named "sample-apiserver-deployment", but I don't think that's the issue. It's been like that since the test was . I'll look for conflicts with other tests.
We should probably go ahead and fix that anyway, it might be stomping on another test... Jenny Buckley <:
The namespace controller polls each of the api groups periodically, and it gets this internal error back from the endpoint a couple of times. Maybe related?
unclear... I opened to capture the state of the APIService, extension server pods, and those pod logs in failures cases
It flaked again since your PR merged https://k8s-
All I'm seeing is
I don't think that's related... I see 10-30 of those same rejections in apiserver logs of successful test runs (for example, search for ). I don't see anything in the test setup that grants the sample apiserver permission to run those checks, so it doesn't seem like that would be intermittent or flaky.
I am re-opening this issue since I still see intermittent failure of this test are your pending PRs ( and ) targeted towards these remaining issues? /reopen
this looks like 1.7 API Server image is causing more flakes other than just this test - The test has become lot less flaky after increasing the timeout. Since the official policy for GKE is 2 previous versions, and 1.11 is almost baked, this is the plan discussed with offline: leave the 1.7 test as it is right now create a similar test to the 1.7, but with 1.9 (implies creating a new sample-sever, uploading image, and creating the test). get that new test PR merged and running let it soak for several days, see if in 1.7 - 1.9 the flakiness (latency or whatever is causing it) went away.

if it did, evaluate removing the 1.7 and using the 1.9 let us know what you think of this approach
I think you misunderstood my comment... that only related to kubectl being used against a 1.7-level kube-apiserver serving discovery docs for the v1 API, not a generic 1.7-level extension API server
unrelatedly, moving the test to 1.9-level code once 1.11 is released seems reasonable to me
Thank you Jordan, that's the current plan! We will actually create a parallel 1.9-level test, to see if between 1.7 and 1.9 the internals of API Server fixed what's generating the flakiness and see next steps from there.
Sorry I didn't read the previous discussion close enough. The idea of creating a 1.9 test makes sense.
Pinging to see what are the next steps. Who is taking point to create a parallel test?
last I chated with someone on his team will build 1.9 API server image and push it to GCR. Once that image is available then can help writing the parallel test. I am not sure if the image step is done.
It's on the works. Buiding the sample-image showed to be more complicated than what it seemed to be. If it becomes a blocker let me know. Aish Sundar <:
I'd like to know where we are on this, because the 1.7 thing has come up in the context of multi-arch image builds
and
Hi Aaron and Aish, sorry for the delay. The process to compile, create and upload the sample API Server was not properly documented / maintained, and that plus more burning things is what made it take so long. We are trying to finally have it back (and documented by end of this week). Will keep you posted.
Image created with all the refactor happened on that area, trying upgrading the test at present from 1.7 to 1.9
Update: Upgraded the test, and now the test is not passing... working on this.

• Failure [38.067 seconds] [sig-api-machinery] Aggregator Should be able to support the 1.9 Sample API Server using the current Aggregator [It] creating a new flunders resource Expected error: <errors.StatusError | 0xc420c90cf0: { ErrStatus: { TypeMeta: {Kind: "", APIVersion: ""}, ListMeta: {SelfLink: "", ResourceVersion: "", Continue: ""}, Status: "Failure", Message: " "rest-flunder-" is forbidden: not yet ready to handle request", Reason: "Forbidden", Details: { Name: "rest-flunder-", Group: "", Kind: "flunders", UID: "", Causes: nil, RetryAfterSeconds: 0, }, Code: 403, }, } "rest-flunder-" is forbidden: not yet ready to handle request not to have occurred
not sure if it's related, but I don't think the test currently gives sufficient permissions to the aggregated server to perform delegated authn/authz checks. we might want to look at merging
tried your fix, same result. The image of the sample-apiserver 1.9 based that I've created is uploaded to If someone want's to give it a try, simply by upgrading 1.0 to 1.1 here:
/assign Can you try to figure out why the test is failing w/ the new image?
/cc
Now waiting on
Is there an ETA on ? This test is flaking quite a bit offlate - https://k8s-
upgrades to 1.10. however it requires more rbac permissions than before which is not fully understood. We will either resolve the permission issue or leave it for later investigation. The upgrade to 1.10 should be merged in a couple of days, if not today.
what issue are you seeing? is relevant?
Yes, it is relevant. It does not fix the issue by itself though.
As for should be ready to be merged.
/reopen I would like some manual confirmation this has been addressed
Reopening this issue. <detailsIn response to : /reopen I would like some manual confirmation this has been addressed Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.

</details
after some in-depth investigations, it turns out the 1.10 sample-apiserver would need the following rbac permissions (in addition to the system:auth-delegator role). I'm updating the PR to add these. apiVersion: kind: ClusterRole metadata: name: sample-apiserver rules: apiGroups: resources: '*' verbs: list apiGroups: '' resources: namespaces verbs: list
This test is failing again, FYI: https://k8s-
Hi Josh, TL;DR: we are upgrading to 1.10 and that's what people is talking about in the previous comments.
/reopen I would like to see the test pass with the fix before closing this issue out. thanks
Reopening this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hi please notice the test was renamed to: [sig-api-machinery] Aggregator Should be able to support the 1.10 Sample API Server using the current Aggregator, and has been passing since merged? https://k8s-
Ah cool thanks that explains it :) I was looking at in release-master-blocking and looks like it hasn't picked by the latest test since its 10/13 run. Closing this issue now based on the api-machinery dashboard now. Will reopen if the release job doesnt turn green the next run. Thanks much.
Is it worth removing the copy of this test that lives in ? Or is it still a release-blocking test?
I think the test will automatically pick the new version of the test in its next run (which should be sometime today, 10/20?). As to whether the test should be in the scale-correctness job , it is a ques for and


Question: case ok && cache.assumedPods[key]: if currState.pod.Spec.NodeName != pod.Spec.NodeName { // The pod was added to a different node than it was assumed to. <del> glog.Warningf("Pod %v assumed to a different node than added to.", key) </del> <ins> glog.Warningf("Pod %v was assumed to be on %v but got added to %v", key, pod.Spec.NodeName, currState.pod.Spec.NodeName) </ins> // Clean this up. cache.removePod(currState.pod) cache.addPod(pod)

/kind bug There are several error messages in the scheduler cache that are confusing to read, at least one of these surfaced up to end users via Events. I noticed these two in particular but there may be others: Whether or not these should actually be surfacing as events, the messages should be cleaned up.
cc
Seems that these are internal or implement details in scheduler, do our end users can fully understand what the events really mean? I think it is depend on our own implement. idea?
I believe, the problem is with in the message but I am not able to reproduce it, can you please share steps to reproduce this.
No unfortunately, one of our users reported seeing it when there system was in a bad state, not in a reproducible way. The point of the issue is just that the grammar in those messages is bad, which should be fixable even if you can't get them to fire.


Question: apiVersion: v1 kind: Pod metadata: <del> name: l7-lb-controller-v0.7.1 </del> <ins> name: l7-lb-controller-v0.8.0 </ins> namespace: kube-system labels: k8s-app: glbc <del> version: v0.7.1 </del> <ins> version: v0.8.0 </ins> kubernetes.io/cluster-service: "true" kubernetes.io/name: "GLBC" spec: terminationGracePeriodSeconds: 600 hostNetwork: true containers: <del> - image: gcr.io/google_containers/glbc:0.7.1 </del> <ins> - image: gcr.io/google_containers/glbc:0.8.0 </ins> livenessProbe: httpGet: path: /healthz

0 picks up a godep update with an int overflow fix to the upstream workqueue package. This bug tracks cherrypicking the version bump into 1.4.1.
closed via


Question: if ok { t.Fatalf("expected result channel to be closed") } <del> case <-time.After(1 * time.Millisecond): </del> <ins> case <-time.After(1 * time.Second): </ins> t.Fatalf("watch should have closed channel: %#v", watching) }

From: https://travis- a test failure on a PR that documentation, so no failure expected. E0813 16:12:46. ] unknown action: update E0813 16:12:46. ] unexpected nil node: &etcd.Response{Action:"set", Node:(etcd.Node)(nil), PrevNode:(etcd.Node)(nil), EtcdIndex:0x0, RaftIndex:0x0, RaftTerm:0x0} E0813 16:12:46. ] failure to decode api object: 'foobar' from &etcd.Response{Action:"set", Node:(etcd.Node)(0xc2100d4720), PrevNode:(etcd.Node)(nil), EtcdIndex:0x0, RaftIndex:0x0, RaftTerm:0x0} &etcd.Node{Key:"", Value:"foobar", Dir:false, Expiration:(time.Time)(nil), TTL:0, Nodes:etcd.Nodes(nil), ModifiedIndex:0x0, CreatedIndex:0x0} E0813 16:12:46. ] etcd.Watch stopped unexpectedly: Injected error ("/some/key") E0813 16:12:46. ]

watch was unable to retrieve the current index for the provided key: 101: () --- FAIL: TestWatchFromOtherError (0.00 seconds) returning /some/key: &{ <nil<nil0 0 0} &etcd.EtcdError{ErrorCode:101, Message:"", Cause:"", Index:0x2} watch should have closed channel: &tools.etcdWatcher{encoding:(_conversion.Scheme)(0xc21001fa50), versioner:(api.jsonBaseResourceVersioner)(0xc586f0), transform:(tools.TransformFunc)(nil), list:false, filter:(tools.FilterFunc)(nil), etcdIncoming:(chan etcd.Response)(0xc2100bf240), etcdStop:(chan bool)(0xc2100bf2a0), etcdCallEnded:(chan struct {})(0xc2100bf300), outgoing:(chan watch.Event)(0xc2100bf360), userStop:(chan struct {})(0xc2100bf3c0), stopped:false, stopLock:sync.Mutex{state:0, sema:0x0}, emit:(func(watch.Event))(0x457940)} FAIL FAIL 0.474s ok 1.179s coverage: 83.6% of statements ok 1.286s coverage: 92.9% of statements ok 1.165s coverage: 84.4% of statements ok 1.283s coverage: 100.0% of statements The command "./hack/test-" exited with 123.


Question: // If the -exist option is specified, ipset ignores the error otherwise raised when // the same set (setname and create parameters are identical) already exists. func (runner *runner) AddEntry(entry string, set *IPSet, ignoreExistErr bool) error { <del> args := []string{"add", set.Name, entry, "comment", set.Comment} </del> <ins> args := []string{"add", set.Name, entry} </ins> if ignoreExistErr { args = append(args, "-exist") }

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug /sig network /area kube-proxy /area ipvs /assign I'll send a fixing pull request tomorrow. What happened: On centos 7.2 with ipset 6.29, kube-proxy in ipvs mode failed to create ipset sets. After some dig, it turns out to be problem of ipset on centos/rh. See I tried ipset 6.29 on ubuntu, it's OK. But when I tried ipset 6.29 on centos, I got errors: What you expected to happen: Since the problematic ipset versions have very large installation base, we could remove ipset commet, or remove ipset commet with problematic ipset version. The problematic ipset also says it support comment by --help. The problem could be detected by a test run of "ipset create foo hash:ip comment". How to reproduce it (as minimally and precisely as possible): Create some service in kubernetes, run kube-proxy with ipvs mode on Centos 7.x node. Anything else we need to know?: Environment: Kubernetes version (use ): commit (actually all commits after ) Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): CentOS Linux release 7.2.1511 (Core) Kernel (e.g. ): Linux VM34centos 3.10.0-514.26.2.el7.x8664 SMP Tue Jul 4 15:04:05 UTC 2017 x8664 x8664 x8664 GNU/Linux Install tools: Others: ipset v6.29, protocol version: 6
could you test it before commit , that's ok?
, yes, I did. It's OK. But I think the we don't need to revert the iptables comments.
if ipset package not installed, will set module or match-set option in rule be weird and now seems not.


Question: "k8s.io/api/core/v1" "k8s.io/apimachinery/pkg/types" "k8s.io/apimachinery/pkg/util/sets" <del> "k8s.io/apimachinery/pkg/util/validation/field" </del> "k8s.io/client-go/tools/record" <del> "k8s.io/kubernetes/pkg/api" k8s_api_v1 "k8s.io/kubernetes/pkg/api/v1" "k8s.io/kubernetes/pkg/api/validation" </del> kubecontainer "k8s.io/kubernetes/pkg/kubelet/container" "k8s.io/kubernetes/pkg/kubelet/events" kubetypes "k8s.io/kubernetes/pkg/kubelet/types"

To read a pod manifest file from disk (or from the http source), kubelet execute the following code to Decode a on-disk v1 pod manifest to an internal pod object (the defaults are applied in this process). Validate the internal pod. Convert the internal pod back to a v1 pod. The downstream pod cache (which gets v1 pods from different sources) performs similar operations: Convert the v1 pod to an internal pod. Validate the internal pod. One of two conversion/validation should be removed, and some of them should be simplified.
/cc for suggestions on defaulting and conversion.
If no one is working on this, I will take it.


Question: // TODO: return a ListerWatcher interface instead? func (factory *ConfigFactory) createAssignedPodLW() *cache.ListWatch { return cache.NewListWatchFromClient(factory.Client, "pods", api.NamespaceAll, <del> parseSelectorOrDie(getHostFieldLabel(factory.Client.APIVersion())+"!=")) </del> <ins> parseSelectorOrDie(client.PodHost+"!=")) </ins> } // createMinionLW returns a cache.ListWatch that gets all changes to minions.

My broke it for v1beta3, then fixed it for v1beta3 and broke it for v1beta1. I'm working on a fix.
Honestly I'm very surprised that more people aren't complaining-- integration test logs are full of noise because of this.
Exact text of error log is "Failed to watch *api.Node: field label not supported: "
cc


Question: next_node=$(increment_ipv4 $next_node $node_subnet_count) done <del> CLUSTER_IP_RANGE="10.244.0.0/16" </del> <ins> CLUSTER_IP_RANGE="${KUBE_GCE_CLUSTER_CLASS_B:-10.244}.0.0/16" </ins> MINION_IP_RANGES=($(eval echo "${subnets[@]}")) MINION_SCOPES=("storage-ro" "compute-rw" "https://www.googleapis.com/auth/monitoring")

This is an offshoot of . cc Here's a log from a single showing that it's getting it's context screwed over by the other clusters coming up at the same time, even those its local environment never changed: Here's the relevant part of the new script: ( is in a separate commit and allows the override of on the GCE so that this script can work.)
I think this can be resolved by having kube- set the context by exporting KUBECONFIG, so a subsequent run doesn't overwrite it.
After some digging with it's a bit trickier than I thought. If we want to be able to run kube-up in parallel from the same branch/working directory, needs to create a separate kubeconfig for each run. But we also want kube-up to set config such that you can immediately run against the created cluster. Probably best to defer until is resolved.
cc And the script is sitting in


Question: return nil, err } pod := obj.(*v1.Pod) <del> p.receivedMoveRequest = false </del> return pod, err }

What happened: Kubernetes scheduler may leave a single pod stuck in pending state after a worker node is reloaded/reinstalled. This problem appears to have been introduced in Kubernetes v1.12.4 and v1.13.1. We have not been able to recreate the problem in our testing on earlier releases. The key to the failure is that the worker being reloaded/reinstalled must be the only scheduling option available. The pending pod is eventually scheduled if a new scheduler leader is elected or the node status changes. This appears to be a timing bug since the problem does not always occur. What you expected to happen: No pods are stuck in pending state. How to reproduce it (as minimally and precisely as possible): Start with a single node cluster running a handful of pods. Cordon, drain and then reload/reinstall the node. Since this is a timing bug, it may take several iterations to recreate the problem. Environment: Kubernetes version (use ): 1.12.4, 1.13.1 and 1.13.2 Cloud provider or hardware configuration: VMs running in VirtualBox OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Custom ansible scripts Others: N/A
/sig scheduling
Events from the pending pod: Events from the node:
does "reload/reinstall" mean info is still kept in etcd? or (or equivalent steps) is executed to remove that node entirely? And I assume the "reloaded" node keeps the same spec and IP/node name with previous one?
the info is still kept in etcd. And the node keeps the same spec, IP and name as it had before. The reload does the following: and drain the node the VM the VM again with the same spec the node
What's the taint/toleration accordingly?
there were no taints to the node during this process. The pods are part of a deployment with only basic tolerations provided by Kubernetes. Deleting the pod will get it scheduled even though the taints and tolerations are unchanged. Simply restarting the scheduler leader will get the pod scheduled as well.
Can you share and ?
The symptom (esp. "a single pod stuck in pending state") is quite similar as issue .

I think why this occurs since v1.12.4 is related the PR - which was delivered to master branch, and backported to v1.12.4, and it appears to reveal higher chance of racing. cc/ Basically we have 2 solutions to fix this: the real root cause of the racing condition and give fix (I haven't got luck to reproduce this yet) that we have a solution () to alleviate this kind of problem, could you try to run your test on a) master branch or b) v1.12.4 with cherrypicked to see if the issue is resolved? and does this fits your business need? (in other words, the single pod may be pending for 30 seconds at most)
I'll recreate and provide you the details. However, the pod that is stuck pending is different every time.
Thanks, I'll work on the recreate and test of the cherrypick. That PR certainly seems promising.
Node YAML after recreate:
Pod YAML after recreate:
the cherrypick doesn't look trivial for 1.12 or 1.13. Give me some time to setup an environment to test on 1.14. Unless you have another way?
Yeap, it's not a trivial task to rebuild the source code. Let me know if you need any help. I suppose you're going to try on latest 1.14 tag? e.g. v1.14.0-alpha.1? Right now I'm working on the 1st solution I mentioned in The challenge is I can't reproduce this in my local dev env (I'm running on source code branch of v1.12.4) yet, so my current plan is to retry reproducing your issue to make my solution more convincing. If still no luck, I will send out the PR for reviewing early next week.
And let me confirm with your statement: When the issue occurs, everytime there is only one single pod stuck in pending state, right?
Still don't have luck in reproducing the issue. But I've prepared a PR and put detailed analysis there. Any comments are welcome.
No, the problem does not occur every time. It can take as many as 7 retries before reproducing it.

Sry, I should have made it more clear - I wanted to confirm that among all the times the issue is reproducible, is that everytime there is only one single pod in pending state?
correct, every time that I've reproduced the problem there has been only one pod in pending state.
Thanks. To some extent, it's consistent with the deduction in my PR.
is it possible to have the necessary fixes cherry-picked to at least 1.12 and 1.13? That would make it easier for me to build and test?
Yeap, cherrypick PRs ( onto 1.13, onto 1.12) were created and will be reviewed/approved soon.
Thank you for the cherrypick PRs. I rebuilt 1.13 with the cherrypick PR and still saw the problem on 2 of 10 recreate attempts. Without the fix, I saw a similar failure rate.
the fix is only ready in master right now. Did you rebuild 1.13 by manually running “git cherrypick <this commit id” on release-1.13? Or directly built from latest src code of release-1.13?
I manually rebuilt 1.13.2 with
/reopen I will take another look.
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I've verified is able to fix your issue thoroughly. And cherrypick PRs have been raised (1.12 , 1.13 ).
Thank you.
Thanks, and K8s 13.3 already has another change () that should alleviate the problem significantly. K8s 13.4 will have the definitive fix. Given that the root-cause is fixed, we can close this issue.


Question: import ( "fmt" "os" <ins> "os/signal" </ins> "runtime" "runtime/pprof"

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: While investigating I've found that kubectl doesn't handle sigterm with profiling. This way, when profiling a long run command (as wait, drain, etc) and interrupting the command in the middle (ctrl-c, as an example) the profile file is not available. What you expected to happen: Profiling file exists if the command exists correctly or if the command is interrupted with ctrl-c How to reproduce it (as minimally and precisely as possible): Execute a long run command (as kubectl drain, kubectl wait) with --profile=cpu and then interrupt the command before it finishes the execution Anything else we need to know?: Because kubectl deals with errors with an util (cmdutil.CheckErr) that in some specific time exists with os.Exit(1), it's not possible yet to also profile programs that exits. Environment: Kubernetes version (use ): kubectl compiled from upstream Cloud provider or hardware configuration: OS (e.g: ): Ubuntu 20.04
/sig cli /assign
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: }) framework.ExpectNoError(err) <del> ginkgo.By("waiting for the no-snat-test-proxy Pod to be scheduled and running") err = wait.PollImmediate(10*time.Second, 1*time.Minute, func() (bool, error) { pod, err := pc.Get(context.TODO(), "no-snat-test-proxy", metav1.GetOptions{}) if err != nil { return false, err } if pod.Status.Phase != v1.PodRunning { if pod.Status.Phase != v1.PodPending { return false, fmt.Errorf("expected pod to be in phase "Pending" or "Running"") } return false, nil // pod is still pending } return true, nil // pod is running }) framework.ExpectNoError(err) </del> ginkgo.By("sending traffic from each pod to the others and checking that SNAT does not occur") <del> pods, err := pc.List(context.TODO(), metav1.ListOptions{LabelSelector: "no-snat-test"}) </del> <ins> pods, err := pc.List(context.TODO(), metav1.ListOptions{LabelSelector: noSNATTestName}) </ins> framework.ExpectNoError(err) <del> // collect pod IPs podIPs := []string{} for _, pod := range pods.Items { podIPs = append(podIPs, pod.Status.PodIP+":"+strconv.Itoa(testPodPort)) } // hit the /checknosnat endpoint on each Pod, tell each Pod to check all the other Pods </del> <ins> // hit the /clientip endpoint on every other Pods to check if source ip is preserved </ins> // this test is O(n^2) but it doesn't matter because we only run this test on small clusters (~3 nodes) <del> errs := []string{} client := http.Client{ Timeout: 5 * time.Minute, } for _, pip := range podIPs { ips := []string{} for _, ip := range podIPs { if ip == pip { </del> <ins> for _, sourcePod := range pods.Items { for _, targetPod := range pods.Items { if targetPod.Name == sourcePod.Name { </ins> continue } <del> ips = append(ips, ip) </del> <ins> targetAddr := net.JoinHostPort(targetPod.Status.PodIP, testPodPort) sourceIP, execPodIP := execSourceIPTest(sourcePod, targetAddr) ginkgo.By("Verifying the preserved source ip") framework.ExpectEqual(sourceIP, execPodIP) </ins> } <del> // hit /checknosnat on pip, via proxy resp, err := client.Get(checknosnatURL(proxyNodeIP, pip, ips)) framework.ExpectNoError(err) // check error code on the response, if 500 record the body, which will describe the error if resp.StatusCode == 500 { body, err := ioutil.ReadAll(resp.Body) framework.ExpectNoError(err) errs = append(errs, string(body)) } resp.Body.Close() } // report the errors all at the end if len(errs) > 0 { str := strings.Join(errs, "n") err := fmt.Errorf("/checknosnat failed in the following cases:n%s", str) framework.ExpectNoError(err) </del> } }) })

What would you like to be : The test currently only works on Cloud Providers or environments that set the External IP on nodes, because it is using the framework directly to check the IPs of the pods, using a no-snat-test-proxy to check if the Pods are not using SNAT. However, this can be simplified, removing the no-snat-test-proxy Pod and executing the queries directly in a Pod. Why is this needed: The test should work in all environments, and it's failing https://k8s-,%20master&include-filter-by-regex=NoSNAT
/sig network /help wanted /failing test
/remove kind-feature /kind failing-test
/help
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove kind-feature
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
I would like to investigate this problem. /assign
I have filed , how could I verify if my patch fixes the problem?
I run them locally with KIND


Question: linux/amd64 linux/arm linux/arm64 <del> #linux/ppc64le # temporarily disabled due to a linking error </del> <ins> linux/ppc64le # note: hyperkube is temporarily disabled due to a linking error </ins> ) # If we update this we should also update the set of golang compilers we build

disabled PPC builds due to the following error: Additional comments here
cc
The issue on PowerPC is both for cross and native-builds. The issue appears to be with the toolchain (golang) and how the linking is being done. The object sections have grown too large to link. We are looking at it cc
Have you tried to upgrade to a newer version? It might or might not help... You might also try to look at what in the gcc linker differs from for example, where it doesn't fail
We are looking into this and it might take some more time. In the meanwhile is it possible to remove hyperkube from the ppc64le builds? All other binaries are building perfectly fine. We can resume the hyperkube for ppc64le build as soon as this issue resolves. cc:
Ah, this is only partially fixed. Reopening...
The error occurs because the hyperkube binary is very large and when building the binary, golang is generating a single go.o file with a single text section that is larger than 2^26 on ppc64le. When there is a call instruction within that section where the call target offset is beyond what can fit in the instruction the relocation truncation error will be generated by the GNU linker. When there are multiple, smaller text sections sent to the GNU linker for linking, it can generate appropriate call instructions even the call target is beyond that limit. I'm working on a fix to do that but hitting other issues as a result which I'm working through. I suspect that this same error will be hit at some point on arm64 unless their linker has another way to handle this situation. On ppc64le we are very close to the limit and from build to build it varies so that some days it will build and others it won't.
Interesting. How does amd64 handle it?
I don't know for sure how arm64 handles it because I don't have access to a machine that is set up to build for arm64. I suspect their text section size is not as large as it is on ppc64le so they haven't hit the limit yet.
Ok, you may easily see what's going on for arm64 by investigating what does in kube-cross.

But I asked about what amd64 (x86_64) does to prevent this :) (I'm assuming this will never happen to amd64)
Sorry I misread your question about amd64... The call instruction in amd64 has a 32 bit field instead of a 24 bit field in ppc64le for the call target address. That means they wouldn't hit this error when linking hyperkube until its text section was larger than 2^32. If you search on "relocation truncated to fit" you will see that this error has happened in the past on x86 with other programming languages.
Once a fix is ready, how does it need to be provided?
I saw https://go- probably won't make 1.7, and that's sad. We would just upgrade golang version I think, not really sure what you're asking for
I assume you only use official golang versions or those provided with distros for your builds? Are you doing all your builds as cross builds from a golang on an x86?
Official versions, yes. We're using the docker image. All official builds are built on amd64 hosts.
Where are you getting this docker image from? Will you be moving to golang:1.7 when it is released?
We're using the official images, and will pick up as soon as it is released for testing I think. Then if the testing and scalability teams think it's safe to use, we'll switch to 1.7 BTW, where is your patch for this? Can you link to it?
is also broken right now - see .
Hi, I'm sorry I missed your last question to me. Here is the CL for my patch for the PPC64 relocations truncation error. https://go- The error with kubemark is also size related but somewhat different because that one is being built as static. Just curious, why are some built static and some non-static?
just to tie everything together, disabled linux/ppc64le compilation entirely. hopefully by go1.8 we can reenable it.
Well, we've hit 1.8 I think with Want to link to any other relevant issues to kinda tie this one together, and close this if it seems appropriate?
yeah, ppc64le builds have been re-enabled for k8s 1.6.

The issue on PowerPC is both for cross and native builds. The issue appears to be with the toolchain (golang) and how the linking is being done. The object sections have grown too large to link. We are looking at it cc
this issue is quite old. Can you provide more context to your comment? Specifically, more detail to help me reproduce. I was unable to find anything related to ppc64le on .


Question: var ipsetInfo = []struct { name    string setType utilipset.Type <del> isIPv6  bool </del> comment string }{ <del> {kubeLoopBackIPSet, utilipset.HashIPPortIP, true, kubeLoopBackIPSetComment}, {kubeClusterIPSet, utilipset.HashIPPort, true, kubeClusterIPSetComment}, {kubeExternalIPSet, utilipset.HashIPPort, true, kubeExternalIPSetComment}, {kubeLoadBalancerSet, utilipset.HashIPPort, true, kubeLoadBalancerSetComment}, {kubeLoadbalancerFWSet, utilipset.HashIPPort, true, kubeLoadbalancerFWSetComment}, {kubeLoadBalancerLocalSet, utilipset.HashIPPort, true, kubeLoadBalancerLocalSetComment}, {kubeLoadBalancerSourceIPSet, utilipset.HashIPPortIP, true, kubeLoadBalancerSourceIPSetComment}, {kubeLoadBalancerSourceCIDRSet, utilipset.HashIPPortNet, true, kubeLoadBalancerSourceCIDRSetComment}, {kubeNodePortSetTCP, utilipset.BitmapPort, false, kubeNodePortSetTCPComment}, {kubeNodePortLocalSetTCP, utilipset.BitmapPort, false, kubeNodePortLocalSetTCPComment}, {kubeNodePortSetUDP, utilipset.BitmapPort, false, kubeNodePortSetUDPComment}, {kubeNodePortLocalSetUDP, utilipset.BitmapPort, false, kubeNodePortLocalSetUDPComment}, {kubeNodePortSetSCTP, utilipset.BitmapPort, false, kubeNodePortSetSCTPComment}, {kubeNodePortLocalSetSCTP, utilipset.BitmapPort, false, kubeNodePortLocalSetSCTPComment}, </del> <ins> {kubeLoopBackIPSet, utilipset.HashIPPortIP, kubeLoopBackIPSetComment}, {kubeClusterIPSet, utilipset.HashIPPort, kubeClusterIPSetComment}, {kubeExternalIPSet, utilipset.HashIPPort, kubeExternalIPSetComment}, {kubeLoadBalancerSet, utilipset.HashIPPort, kubeLoadBalancerSetComment}, {kubeLoadbalancerFWSet, utilipset.HashIPPort, kubeLoadbalancerFWSetComment}, {kubeLoadBalancerLocalSet, utilipset.HashIPPort, kubeLoadBalancerLocalSetComment}, {kubeLoadBalancerSourceIPSet, utilipset.HashIPPortIP, kubeLoadBalancerSourceIPSetComment}, {kubeLoadBalancerSourceCIDRSet, utilipset.HashIPPortNet, kubeLoadBalancerSourceCIDRSetComment}, {kubeNodePortSetTCP, utilipset.BitmapPort, kubeNodePortSetTCPComment}, {kubeNodePortLocalSetTCP, utilipset.BitmapPort, kubeNodePortLocalSetTCPComment}, {kubeNodePortSetUDP, utilipset.BitmapPort, kubeNodePortSetUDPComment}, {kubeNodePortLocalSetUDP, utilipset.BitmapPort, kubeNodePortLocalSetUDPComment}, {kubeNodePortSetSCTP, utilipset.BitmapPort, kubeNodePortSetSCTPComment}, {kubeNodePortLocalSetSCTP, utilipset.BitmapPort, kubeNodePortLocalSetSCTPComment}, </ins> } // ipsetWithIptablesChain is the ipsets list with iptables source chain and the chain jump to

/kind bug /area ipv6 /area ipvs /sig network What happened: In an ipv6-only cluster the in mode creates a number of ipset's with . This causes problems and fault logs when they are used with ipv6 addresses. What you expected to happen: The ipset's should be created with . How to reproduce it (as minimally and precisely as possible): Start a ipv6-only cluster Execute; on any node and check the output Anything else we need to know?: The problem seems to be This code seems pretty weird to me. I mean why a flag in the structure at all? I ignored the flag in the structure and just passed the in the directly to and it seem to work. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: Kvm on Ubuntu 16.04 OS (e.g. from /etc/os-release): Own BusyBox system Kernel (e.g. ):Install tools: None Others:
Did you make changes like this Earlier error related with ipsets whose is ?
Exactly. I have no idea why any particular ipset would be ipv4 in an ipv6 cluster. But the flag is false for NodePort set's and there is another bug that NodePort addresses are ipv4 (and faulty). So there might have been some special handling of nodeports at some time. IMHO there shoudn't be. FYI, I can't send a PR myself just yet but expect to be able to do that in September.
/cc
I can confirm that, in Ipv6 only cluster, is reporting family inet for many of the entries (KUBE-LOOP-BACK, KUBE-CLUSTER-IP, KUBE-LOAD-BALANCER-FW...).
There is clearly a bug in this code, as it forces IPV4: Should have been an else clause, or just: Even with this, there area entries that are using IPv4. Mostly NodePort related entries. Also see iptables error still:
Correction to above... the ipv6 flag should be based on node's IP family and not the isIPv6 field in the ipsetInfo struct, which has true for all entries, except the nodeport entries. Is anyone planning on creating a PR for this?

Looking forward to PRs
The following codes actually act the same because are of type , which have only one argument (no : or ). I think they are old ipsets. Are there ipsets with ?
I actually forgot to mention that traffic does not work. It is not just flooded logs and printout from that is the problem. Below is printout from in an ipv6-only cluster. As you can see all set's that have an "ip" have "family inet". Those without "ip", for instance "bitmap:port" has no family at all.
And I hope to send a PR for this in a few days.
I think the changes you made won't work. The problem is failing to create new ipsets. It should work after deleting the old ipsets.
They do work, I have tested. There are no old ipset's since the cluster is started fresh with an ipv6-only configuration. Do you upgrade your cluster from ipv4-only to ipv6-only without rebooting the nodes? That is a very complex upgrade and IMHO it should not be supported. When dual-stack comes I think it will be easier, like; to k8s with dual-stack support while still using ipv4-only /enable ipv6
I think I understand what you mean by "existing" sets now. Some sets are created twice since there are no clause in the if-statement. The ipset command is called with which I think causes the second "create" to actually become a "modify". And in that call the ipIpv6 flag is hard-coded to . I see no purpose for the isIpv6 flag in the structure and will remove it in my PR.


Question: if test.Response.Body == nil { test.Response.Body = ioutil.NopCloser(bytes.NewReader([]byte{})) } <del> body, err := ioutil.ReadAll(test.Response.Body) if err != nil { t.Errorf("failed to read body of response: %v", err) } response, created, err := r.transformResponse(test.Response, &http.Request{}, body) </del> <ins> result := r.transformResponse(test.Response, &http.Request{}) response, created, err := result.body, result.created, result.err </ins> hasErr := err != nil if hasErr != test.Error { t.Errorf("%d: unexpected error: %t %v", i, test.Error, err)

Storing r.req / r.resp internally (for use from DoRaw and Do) is probably not the right factoring. There should be a common method doRaw that returns request and body.


Question: // Register regions, in particular for ECR credentials once.Do(func() { <del> RecognizeWellKnownRegions() </del> <ins> recognizeWellKnownRegions() </ins> }) return awsCloud, nil

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: From /kind cleanup What happened: What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig cloud-provider /assign


Question: framework.ExpectEqual(*(ss.Spec.Replicas), int32(4), "statefulset should have 4 replicas") }) <ins> ginkgo.It("should list, patch and delete a collection of StatefulSets", func() { ssPatchReplicas := int32(2) ssPatchImage := imageutils.GetE2EImage(imageutils.Pause) one := int64(1) ssName := "test-ss" // Define StatefulSet Labels ssPodLabels := map[string]string{ "name": "sample-pod", "pod":  WebserverImageName, } ss := e2estatefulset.NewStatefulSet(ssName, ns, headlessSvcName, 1, nil, nil, ssPodLabels) setHTTPProbe(ss) ss, err := c.AppsV1().StatefulSets(ns).Create(context.TODO(), ss, metav1.CreateOptions{}) framework.ExpectNoError(err) e2estatefulset.WaitForRunningAndReady(c, *ss.Spec.Replicas, ss) waitForStatus(c, ss) ginkgo.By("patching the StatefulSet") ssPatch, err := json.Marshal(map[string]interface{}{ "metadata": map[string]interface{}{ "labels": map[string]string{"test-ss": "patched"}, }, "spec": map[string]interface{}{ "replicas": ssPatchReplicas, "template": map[string]interface{}{ "spec": map[string]interface{}{ "TerminationGracePeriodSeconds": &one, "containers": [1]map[string]interface{}{{ "name":  ssName, "image": ssPatchImage, }}, }, }, }, }) framework.ExpectNoError(err, "failed to Marshal StatefulSet JSON patch") _, err = f.ClientSet.AppsV1().StatefulSets(ns).Patch(context.TODO(), ssName, types.StrategicMergePatchType, []byte(ssPatch), metav1.PatchOptions{}) framework.ExpectNoError(err, "failed to patch Set") ss, err = c.AppsV1().StatefulSets(ns).Get(context.TODO(), ssName, metav1.GetOptions{}) framework.ExpectNoError(err, "Failed to get statefulset resource: %v", err) framework.ExpectEqual(*(ss.Spec.Replicas), ssPatchReplicas, "statefulset should have 2 replicas") framework.ExpectEqual(ss.Spec.Template.Spec.Containers[0].Image, ssPatchImage, "statefulset not using ssPatchImage. Is using %v", ss.Spec.Template.Spec.Containers[0].Image) e2estatefulset.WaitForRunningAndReady(c, *ss.Spec.Replicas, ss) waitForStatus(c, ss) ginkgo.By("Listing all StatefulSets") ssList, err := c.AppsV1().StatefulSets("").List(context.TODO(), metav1.ListOptions{LabelSelector: "test-ss=patched"}) framework.ExpectNoError(err, "failed to list StatefulSets") framework.ExpectEqual(len(ssList.Items), 1, "filtered list wasn't found") ginkgo.By("Delete all of the StatefulSets") err = c.AppsV1().StatefulSets(ns).DeleteCollection(context.TODO(), metav1.DeleteOptions{GracePeriodSeconds: &one}, metav1.ListOptions{LabelSelector: "test-ss=patched"}) framework.ExpectNoError(err, "failed to delete StatefulSets") ginkgo.By("Verify that StatefulSets have been deleted") ssList, err = c.AppsV1().StatefulSets("").List(context.TODO(), metav1.ListOptions{LabelSelector: "test-ss=patched"}) framework.ExpectNoError(err, "failed to list StatefulSets") framework.ExpectEqual(len(ssList.Items), 0, "filtered list should have no Statefulsets") }) </ins> ginkgo.It("should validate Statefulset Status endpoints", func() { ssClient := c.AppsV1().StatefulSets(ns) labelSelector := "e2e=testing"

[X] test pr : kuberenetes/kubernetes 17 June 2021 [x] two weeks soak end date : 1 July 2021 [X] test promotion pr : kubernetes/kubernetes According to this APIsnoop query, there are still some remaining StatafulSet endpoints that are untested. with this query, you can filter untested endpoints by their category and eligibility for conformance. e.g below shows a query to find all conformance eligible untested, stable, core endpoints Create a StatefulSet Patch the StatefulSet List all StatefulSet in the namespace Delete the collection if StatefulSets in the namespace Verify that all StatefulSet was deleted Due to the complexity of setting up the resources for APIService we have used the current e2e test as a template. It has been extended in a for review.

example endpoint | useragent ---------------------------------------------|
createAppsV1NamespacedStatefulSet | (linux/amd64) kubernetes/$Format -- [sig-apps] StatefulSet Basic StatefulSet functionality [StatefulSetBasic] should list, patch and delete a collection of StatefulSets deleteAppsV1CollectionNamespacedStatefulSet | (linux/amd64) kubernetes/$Format -- [sig-apps] StatefulSet Basic StatefulSet functionality [StatefulSetBasic] should list, patch and delete a collection of StatefulSets listAppsV1NamespacedStatefulSet | (linux/amd64) kubernetes/$Format -- [sig-apps] StatefulSet Basic StatefulSet functionality [StatefulSetBasic] should list, patch and delete a collection of StatefulSets listAppsV1StatefulSetForAllNamespaces | (linux/amd64) kubernetes/$Format -- [sig-apps] StatefulSet Basic StatefulSet functionality [StatefulSetBasic] should list, patch and delete a collection of StatefulSets patchAppsV1NamespacedStatefulSet | (linux/amd64) kubernetes/$Format -- [sig-apps] StatefulSet Basic StatefulSet functionality [StatefulSetBasic] should list, patch and delete a collection of StatefulSets readAppsV1NamespacedStatefulSet | (linux/amd64) kubernetes/$Format -- [sig-apps] StatefulSet Basic StatefulSet functionality [StatefulSetBasic] should list, patch and delete a collection of StatefulSets replaceAppsV1NamespacedStatefulSetStatus | kube-controller-manager/v1.20.4 (linux/amd64) kubernetes//system:serviceaccount:kube-system:statefulset-controller (7 rows) If a test with these calls gets merged, test coverage will go up by 3 points This test is also created with the goal of conformance promotion. /sig apps /sig testing /sig architecture /area conformance
/triage accepted
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: } func NewFakeVolumeHost(rootDir string, kubeClient clientset.Interface, plugins []VolumePlugin) *fakeVolumeHost { <del> host := &fakeVolumeHost{rootDir: rootDir, kubeClient: kubeClient, cloud: nil} </del> <ins> return newFakeVolumeHost(rootDir, kubeClient, plugins, nil) } func NewFakeVolumeHostWithCloudProvider(rootDir string, kubeClient clientset.Interface, plugins []VolumePlugin, cloud cloudprovider.Interface) *fakeVolumeHost { return newFakeVolumeHost(rootDir, kubeClient, plugins, cloud) } func newFakeVolumeHost(rootDir string, kubeClient clientset.Interface, plugins []VolumePlugin, cloud cloudprovider.Interface) *fakeVolumeHost { host := &fakeVolumeHost{rootDir: rootDir, kubeClient: kubeClient, cloud: cloud} </ins> host.mounter = &mount.FakeMounter{} host.writer = &io.StdWriter{} host.pluginMgr.InitPlugins(plugins, host)

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? NO Note: Please file issues for subcomponents under the appropriate repo What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): kubernetes, azure file, chinacloudapi Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): master Environment: Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Ubuntu 16.04 Kernel (e.g. ): Install tools: Others: What happened: We try to mount azure file to pod, and get the following error That is because azure china's storage DNS suffix is rather than What you expected to happen: Kubernetes should provide some method to let user specify storage dns suffix, so we can use Azure file volume How to reproduce it (as minimally and precisely as possible): Any Azure cluster in china can reproduce it. Anything else we need to know: kubernetes has hard coded in shows a relative issue
could you help confirm this?
One possible way is configuring DNS suffix via annotations of PV. If thinks this way okay, I'd like to create a PR to fix it.
azure file was implemented without using resource group or azure cloud provider. The endpoint can be found at One fix is to use azure cloud provider and choose the file service endpoint from azure environment as I did in blob service . Let me know if you can help.
I'm not familiar with azure cloud provider and volume codes. I'd like to try to dig into it to see if I can fix as the way you suggested.


Question: return cmdutil.AddSourceToErr(fmt.Sprintf("applying patch:n%snto:n%vnfor:", patchBytes, info), info.Source, err) } <del> if cmdutil.ShouldRecord(cmd, info) { if patch, patchType, err := cmdutil.ChangeResourcePatch(info, f.Command(cmd, true)); err == nil { if recordedObject, err := helper.Patch(info.Namespace, info.Name, patchType, patch); err != nil { glog.V(4).Infof("error recording reason: %v", err) } else { patchedObject = recordedObject } } } </del> info.Refresh(patchedObject, true) if uid, err := info.Mapping.UID(info.Object); err != nil {

What happened: What you expected to happen: DS history revision 2 should have change cause from the second command "kubectl apply -f ds-2.yaml --record", not the first /assign


Question: "context" "errors" "fmt" <ins> "sync" </ins> "time" v1 "k8s.io/api/core/v1"

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: during phase from the plugin when the is updated: What you expected to happen: Not crash. How to reproduce it (as minimally and precisely as possible): I am suspecting that the issue is in the naive implementation of the 's method: Most likely the is cloned during filter phase, e.g. in the called during in the : ... and when is cloned during filtering, new mutex is created, but is not cloned properly, this mean that potentially there are two or more different mutexes protecting the same map from writes. Anything else we need to know?: I think potential solution is to have mutex in the or change the 's . Environment: Kubernetes version (use ): 1.19.0 Cloud provider or hardware configuration: not relevant OS (e.g: ): not relevant Kernel (e.g. ): not relevant Install tools: not relevant Network plugin and version (if this is a network-related bug): not relevant Others: not relevant /sig scheduling
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scheduling
It looks like relevant code is from and mutex per was considered intially. CC
IIUC, the may (edit: if the node has nominated pods) be duplicated in for every node before passing to extension point. So we cann't rely on 's lock to prevent the state from being modified concurrently. Am I right?
I may have given an incorrect analysis as VolumeBinding's plugin-specific state are always using the same pointer, sorry for the confusion. I removed the confusing comments. It's actually different from what was discussed in . Your and 's analysis make more sense to me.

For nodes that have nominated nodes (cloned CycleState) or not (original CycleState), the original mutex may not be able to protect the CycleState: I will take another look tomorrow.
can you please post the actual core dump, the description in the original comment doesn't seem complete. If the "concurrent map" fatal error is indeed happening at , then it means we are operating on two different CycleStates instances at the same time, which may happen when adding nominated pods () Using the CycleState lock to synchronize access to the plugin's state in the map is not proper, the plugin state itself should have its private lock, this is certainly a bug.
/cc