Question: } } <del> // runStaticPodTest is disabled until #6651 is resolved. func runStaticPodTest(c *client.Client, configFilePath string) { var testCases = []struct { desc         string fileContents string }{ { desc: "static-pod-from-manifest", fileContents: `version: v1beta2 id: static-pod-from-manifest containers: - name: static-container image: gcr.io/google_containers/pause:2.0`, }, { desc: "static-pod-from-spec", fileContents: `{ "kind": "Pod", "apiVersion": "v1", "metadata": { "name": "static-pod-from-spec" }, "spec": { "containers": [{ "name": "static-container", "image": "gcr.io/google_containers/pause:2.0" }] } }`, }, } for _, testCase := range testCases { func() { desc := testCase.desc manifestFile, err := ioutil.TempFile(configFilePath, "") defer os.Remove(manifestFile.Name()) ioutil.WriteFile(manifestFile.Name(), []byte(testCase.fileContents), 0600) // Wait for the mirror pod to be created. podName := fmt.Sprintf("%s-localhost", desc) namespace := kubetypes.NamespaceDefault if err := wait.Poll(time.Second, longTestTimeout, podRunning(c, namespace, podName)); err != nil { if pods, err := c.Pods(namespace).List(api.ListOptions{}); err == nil { for _, pod := range pods.Items { glog.Infof("pod found: %s/%s", namespace, pod.Name) } } glog.Fatalf("%s FAILED: mirror pod has not been created or is not running: %v", desc, err) } // Delete the mirror pod, and wait for it to be recreated. c.Pods(namespace).Delete(podName, nil) if err = wait.Poll(time.Second, longTestTimeout, podRunning(c, namespace, podName)); err != nil { glog.Fatalf("%s FAILED: mirror pod has not been re-created or is not running: %v", desc, err) } // Remove the manifest file, and wait for the mirror pod to be deleted. os.Remove(manifestFile.Name()) if err = wait.Poll(time.Second, longTestTimeout, podNotFound(c, namespace, podName)); err != nil { glog.Fatalf("%s FAILED: mirror pod has not been deleted: %v", desc, err) } }() } } </del> func runReplicationControllerTest(c *client.Client) { t := time.Now() clientAPIVersion := c.APIVersion().String()

The integration test is hard to debug -- it runs tests concurrently with interspersed logs. I doubt that it's catching any bug that neither unit test nor a e2e test can catch, and both unit and e2e tests are easier to debug. I'd vote for deprecating this file and remove/move all the tests. Any objections? Here is a list of tests that it curretnly runs:
Yeah the rc and services tests are redundant, I think we exercise them enough in e2es. Don't know about: It quicky creates/deletes pods with the same hostport that get scheduled onto a single node. Don't see a reason why we can't move it to an e2e, but I don't think it's been done yet.
since you are the original author of , do you have any suggestions/preferences about this test?
That test is important, it has caught regressions two or three times. It should be movable to the other integration test section. , Yu-Ju Hong wrote:
looks like you're working on this already. Feel free to unassign if that's incorrect.
I uploaded a PR to remove rc and service tests based on I'll let each team decide whether they want to migrate or delete the .
maybe someone with spare time can rewrite that scheduler test as an e2e, and we can kill integration? I continue to see integration flake that's hard to debug, current SQ closing failure: Why did that fail? Ctrl+f "fail" gives me 700 matches, scrolling through the logs i see all sorts of nonsense. I'm going to guess that scheduler test wasn't able to put pods on nodes because of the kubeletNotReady condition, but that's just a guess.
Those failures even existed in passing integration test as well (I can feel your pain). This is due to . It would be great if original author of those tests can clarify what each test does and maintain it. Otw those tests are just legacy obstacles that we should get rid of.
the rc and services tests have been removed. The only ones remaining are put/patch/apiserver tests and the scheduler phantom pod tests IIRC. The former is relatively straightforward and only needs the apiserver (segmented failure mode), the latter would be much easier to maintain as an e2e. Bandwidth issue.
I see.

I can definitely help with it. Let me spend some time figuring them out. Meanwhile, anyone familiar with the scheduler tests should feel free to open another issue and track them.
What's your opinion on this?
I agree that integration either needs to be killed or massively overhauled
This issue describes the circumstances leading up to that scheduler integration test, for anyone attempting to port it: Though IIRC we got rid of the modeler, so things might be different now.
I think we actually have a spike in integration test failures:
unfortunately the failure you linked was not from this file () Those tests were in , and were considered easier to debug (at least from doc last time). Personally I don't have much experiences debugging this new category of integration tests, so I can't say much. They do seem to flake a lot recently in per-PR tests as well.
I concur on this regarding the "fail" message seen on scheduler test too. Re. "SchedulerNoPhantomPodsTest", it assumes two kubelets with predefined resources, which is easier to maintain in simulated environment. Right? If testing in is to hard to debug, we could make it a lightweight unit test. IIRC, it's testing one specific race condition that existed before in scheduler.
oh, hmm, different bug then Re shceduler test: If you can manage with a unittest that would be great. If not I'm fine running the integration tests serially. I don't think it'll take that much longer, and flakes that only show up in parallel runs are hopefully covered in gce-e2e.


Question: metadata, _ := meta.Accessor(info.Object) annotationMap := metadata.GetAnnotations() if _, ok := annotationMap[corev1.LastAppliedConfigAnnotation]; !ok { <del> fmt.Fprintf(o.ErrOut, warningNoLastAppliedConfigAnnotation, o.cmdBaseName) </del> <ins> fmt.Fprintf(o.ErrOut, warningNoLastAppliedConfigAnnotation, info.ObjectName(), corev1.LastAppliedConfigAnnotation, o.cmdBaseName) </ins> } patcher, err := newPatcher(o, info, helper)

<!-- Please only use this template for submitting enhancement requests --What would you like to be : show a clearer message that better informs the user of a potential conflict between changes done declaratively vs imperatively on the same object, inform whether the changes have been applied, what may have been overwritten based on closed issue resource that has been directly modified imperatively (for example via a kubectl edit command) try to update it again declaratively (for example by kubectl apply -f yamlfile) The following confusing info message appears: "Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply... " Why is this needed: the error message is confusing. is it really an error message or just an informative message? Did the command successfully execute the changes being asked? Were the changes applied successfully? Was there anything overwritten? If changes were successfully applied, what is the reason for this message to show?
/sig cli
background references: and
/assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Agree that this is a confusing message. Could probably use some love. First-round search indexing brought me here.


Question: } path := hostPathVolumeSource.Path <ins> pathType := new(v1.HostPathType) if hostPathVolumeSource.Type == nil { *pathType = v1.HostPathUnset } else { pathType = hostPathVolumeSource.Type } </ins> return &hostPathMounter{ <del> hostPath: &hostPath{path: path, pathType: hostPathVolumeSource.Type, containerized: opts.Containerized}, </del> <ins> hostPath: &hostPath{path: path, pathType: pathType, containerized: opts.Containerized}, </ins> readOnly: readOnly, }, nil }

/kind bug What happened: I ran kubelet from HEAD and tried to make a normal cluster using kubeadm. I saw errors like these in the kubelet's log: (I think is the cause) What you expected to happen: No panic How to reproduce it (as minimally and precisely as possible): kubeadm the normal way kubelet, kubeadm from CI HEAD and a cluster like normal crashes Seems related to
cc I think the only thing we have to do here is to check for nil before trying to dereference the pointer Adding the v1.8 milestone to this bug to make we track it in a release-blocking sense, this is probably a release blocker. cc
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/node sig/storage: Issue will be escalated to these SIGs if needed. priority/critical-urgent: Never automatically move out of a release milestone; continually escalate to contributor and SIG through all available channels. kind/bug: Fixes a bug discovered during the current release. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details
Since we've already set default value for , the value is a non-nil pointer. No obvious errors found in are related to .
Yeah, but somehow it seems to be nil here... otherwise it wouldn't have paniced
Before we merge, we need to understand how it got to be nil. It's supposed to be defaulted.
Absolutely. has a prototype of how it could be nil now in , let's verify if it fixes the issue...
I think volume manager reconstructs the specs from reading the directories on the disk and these specs do not have the defaults applied. would know better
Using api machinery validation without accompanying api machinery decoding/defaulting is dangerous... many paths make assumptions that the objects they are given have passed through standard decoding/defaulting
Agreed with both and above. can you assign this to the owner of the volume manager in sig-storage? cc/
The root cause is apiserver version < kubelet version, as I stated in .
Even with an old api server, it concerns me that there is apparently a code path that causes the kubelet to run validation on internal objects that did not go through a defaulting pass as part of being converted from v1 objects to internal objects.

This issue is being discussed on , see the latest on that PR. At the moment, it appears to be caused by kubeadm using a newer version of the k8s client then then the server. Awaiting word from and If that is the case, we should be able to close this bug. If not, we can merge as a temporary mitigation to unblock release.
Do you mean kubelet? We explicitly support clients being ahead of the server by a minor version.
Also, your comment doesn't address concern above.
Yes, referencing this discussion (): liggitt commented 17 hours ago Even with an old api server, it concerns me that there is apparently a code path that causes the kubelet to run validation on internal objects that did not go through a defaulting pass as part of being converted from v1 objects to internal objects. I don't think that is the case. See comment here: Based on that discussion, it looks like volume reconstruction was a red herring. In general volume reconstruction creates internal objects that are only used internally, they are not run through validation.
Still awaiting response on CC
Volume reconstruction is not entirely guilt free here: it constructs an internal object, does not run the object through validation or defaulting logic, so if a new field is introduced (like hostpath type in this case), the default value is not set resulting in issues like this one. To handle this case gracefully I'm approving PR For Q4 (1.9) we will follow up on this and see what we can do to handle this more gracefully. In general, volume reconstruction is a thorny bit of code and deserves a thorough redesign.
I have filed an issue related to internal volume reconstruction process during kublet restarts. The new proposed approach is very different from the current design. Will work on it in detail for 1.9
I got the same bug this is log Oct 24 14:45:27 node-23-40 kubelet[]: Oct 24 14:45:27 node-23-40 kubelet[]: /usr/local/go/src/runtime/asmamd64.s:2361 Oct 24 14:45:27 node-23-40 kubelet[]: E1024 14:45:27. ]

Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference) Oct 24 14:45:27 node-23-40 kubelet[]: Oct 24 14:45:27 node-23-40 kubelet[]: Oct 24 14:45:27 node-23-40 kubelet[]: Oct 24 14:45:27 node-23-40 kubelet[]: /usr/local/go/src/runtime/asmamd64.s:573 Oct 24 14:45:27 node-23-40 kubelet[]: Oct 24 14:45:27 node-23-40 kubelet[]: Oct 24 14:45:27 node-23-40 kubelet[]: Oct 24 14:45:27 node-23-40 kubelet[]: Oct 24 14:45:27 node-23-40 kubelet[]: Oct 24 14:45:27 node-23-40 kubelet[]: Oct 24 14:45:27 node-23-40 kubelet[]: Oct 24 14:45:27 node-23-40 kubelet[]: Oct 24 14:45:27 node-23-40 kubelet[]: Oct 24 14:45:27 node-23-40 kubelet[]: Oct 24 14:45:27 node-23-40 kubelet[]:
what version did you see the panic on?
kubectl version Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-09-19T07:01:43Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"11+", GitVersion:"v1.11.0-168+", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-08-25T21:05:52Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}
What flex driver are you using? It looks like it's not returning any capabilities.

Here's the panic line: cc should we fallback to default capabilities if the driver doesn't return any?
This stacktrace is unrelated to the original failure. Can you open a new issue for this?
We already default it, unless the plugin overrides it with nil:


Question: `), }, }, <ins> { Description: "$deleteFromPrimitiveList should delete item from a list with merge patch strategy", StrategicMergePatchRawTestCaseData: StrategicMergePatchRawTestCaseData{ Original: []byte(` mergingIntList: - 1 - 2 - 3 `), TwoWay: []byte(` $deleteFromPrimitiveList/mergingIntList: - 2 `), Modified: []byte(` mergingIntList: - 1 - 3 `), }, }, { Description: "$deleteFromPrimitiveList should delete item from a list without merge patch strategy", StrategicMergePatchRawTestCaseData: StrategicMergePatchRawTestCaseData{ Original: []byte(` nonMergingIntList: - 1 - 2 - 3 `), TwoWay: []byte(` $deleteFromPrimitiveList/nonMergingIntList: - 2 `), Modified: []byte(` nonMergingIntList: - 1 - 3 `), }, }, </ins> } func TestCustomStrategicMergePatch(t *testing.T) {

This page explains that the deleteFromPrimitiveList should delete the specified items, but the specified items are kept, and the other items are deleted. The specified items in the deleteFromPrimitiveList should be removed. The result should be instead: No response <details</details<details</details<details</details<details</details<details</details<details</details
/sig api-machinery
Related:
/assign Could you take a look Antoine please? Thanks! /triage accepted
Simplest way to reproduce: It doesn't work with any kubectl version from 1.18 to 1.23 that I've tested.
I tried the exact same object, changing the finalizers works like a charm.
cc Want to give this one a try? :-)
Sure, I'll take a look at it... /assign
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale


Question: <ins> #!/usr/bin/env bash # Copyright 2017 The Kubernetes Authors. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # #     http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. DEB_ARCH=$(dpkg --print-architecture) # http://security.debian.org/debian-security/dists/jessie/updates/InRelease is missing # entries for some platforms, so we just remove the last line in sources.list in # /etc/apt/sources.list which is "deb http://deb.debian.org/debian jessie-updates main" case ${DEB_ARCH} in arm64|ppc64el) sed -i '/debian-security/d' /etc/apt/sources.list ;; esac exit 0 </ins>

[X] "goproxy" to "" [X] "jessie-dnsutils" to "" [X] "redis" to "" [X] "serve-hostname" to ""
/assign /assign
we can use to iterate
/sig testing
tip for uploading is here -
should be published to now. I'm rerunning for the other 3 images.
I think the and images were published successfully this time. :
can you please try ?
should now be published too thanks to PR.
related to


Question: queue internalqueue.SchedulingQueue, pdbs []*policy.PodDisruptionBudget, ) ([]*v1.Pod, int, bool) { <ins> if nodeInfo == nil { return nil, 0, false } </ins> potentialVictims := util.SortableList{CompFunc: util.HigherPriorityPod} nodeInfoCopy := nodeInfo.Clone()

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: kube-scheduler crashes (panics) in rare circumstances. Stack trace: What you expected to happen: flawless operation ;) How to reproduce it (as minimally and precisely as possible): Unclear. The error above looks like it was associated with some cloning of node metadata. Perhaps due to a race during node scale-up/scale-down and cache interactions. Anything else we need to know?: Environment: Kubernetes version (use ): <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
/sig scheduling
are you using hostPort for your workload?
is there a specific workload you had in mind? The does, as well as daemonsets for our service mesh, which are in host networking mode.
There was a PR () in v1.12/v1.11.3 fixing issue when workloads uses hostNetwork, and the solution is to correct the NodeInfo cloning logic. So it's quite likely related with the symptom you described. Could you give a try on v1.11.3?
BTW: the issue that PR fixed is only reproducible on some circumstances.
Thanks - I'll give this a whirl. I think I can optimistically close this issue - we'll reopen if it persists with a more recent autoscaler release. Cheers!
Sounds good! Thanks for the feedback.
we just had a similar issue this morning and our cluster is running kube 1.11.4: and these are the error logs : we do have hostNetwork work load running inside our cluster but not sure if that's caused by other issues since you mentioned that hostNetwork issue has been fixed since 1.11.3.
could you re-open this issue ? or I can raise a new issue for it.
/reopen I will take a look.
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
thanks
are the worker nodes auto scaled out/in? e.g. by an auto-scaler component?

yes, we do have an autoscaler to auto scale our worker nodes and we are using since most of our workloads are job based.
I see. It's likely an internal cache inconsistency. (not related with the hostPort issue I fixed earlier) /assign
One thing I'm curious is from the log it points to L386 of Then it means is nil, but should't give a panic trace at the first call inside ? Say: I looked through the code, but can't find a place that could be nil. So are you using raw v1.11.4-beta.0 branch, or did some modification? As in my opinion, it makes sense the panic comes from L385:
No, in Go if n.requestedResource is nil, you will get a stack trace similar to what is given in this issue.
I read it differently. I wrote an example , it shows deepest trace at L12, rather than L17.
I was actually wondering if it's possible that the stack trace isn't 100% line accurate.. Either that it systematically hits the next line in this case, or that the order in which field copies are executed isn't a simple 1:1 (linear) relationship to the code.
it's interesting, maybe it's some trick (or bug) in apimachinery's runtime.HandleCrash()? And I wrote a test to reproduce the issue, even the same line on the deepest stack trace: (although I think it should be line above)
/reopen I will raise cherrypick PRs to 1.12 and 1.11, and close this issue by then.
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hi i have scaled my cluster to 100 node from 40 nodes as there were 400 pending pods and for some of the nodes all pods get schedule properly while around 28 nodes have only dnsmasq and kube-proxy was running. after around 40 min those node had weave and other pods started running. i coundn't able to find any other issue in my cluster apart from kube-scheduler got restarted because same error mentino in the issue. though it restarted in few second.

i was just wondering that if after restart would it takes 30-40 min to schedule pods? cause after 40 min everything seems to be working properly. I am running kubernetes 1.12.1 and i have installed it using kops 1.10.0
fix of this issue is cherrypicked (PR ) into v1.12.4. Could you give a try on v1.12.4?
Close this as fix has been applied to 1.11+ k8s versions. /close
Closing this issue. <detailsIn response to : Close this as fix has been applied to 1.11+ k8s versions. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: // We omit creation of pretty much everything if we run in cleanup mode if cleanupAndExit { <del> return &ProxyServer{IptInterface: iptInterface}, nil </del> <ins> return &ProxyServer{IptInterface: iptInterface, CleanupAndExit: cleanupAndExit}, nil </ins> } client, eventClient, err := createClients(config.ClientConnection, master)

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: bug What happened: Got panic after running: Panic: What you expected to happen: No panic and exit successfully. How to reproduce it (as minimally and precisely as possible): Run the following command on both CoreOS v1298.7.0(bare metal) and Ubuntu 16.04.2 LTS(OpenStack) Happened with the following versions: v1.7.0-alpha.3 v1.7.0-alpha.4 v1.7.0-beta.0 v1.7.0-beta.1 v1.7.0-beta.2 v1.7.0-rc.1 v1.8.0-alpha.0 v1.8.0-alpha.1 Anything else we need to know?: I guess it caused by . Environment: Kubernetes version (use ): Cloud provider or hardware configuration**: OS (e.g. from /etc/os-release): Both CoreOS v1298.7.0(bare metal) and Ubuntu 16.04.2 LTS(OpenStack) Kernel (e.g. ): CoreOS v1298.7.0: Linux node- 4.9.16-coreos-r1 SMP Tue Apr 4 19:09:38 UTC 2017 x8664 Intel(R) Core(TM) i5-6500 CPU @ 3.20GHz GenuineIntel GNU/Linux Ubuntu 16.04.2 LTS: Linux ubuntu- 4.4.0-62-generic -Ubuntu SMP Wed Jan 18 14:10:15 UTC 2017 x8664 x8664 x8664 GNU/Linux Install tools: Others:
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <bre.g., for API Machinery<br(2) specifying the label manually: <bre.g., for sig/scalability<br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
/sig network /area kube-proxy
/cc
cc/ Looks like there is a new regression in v1.7. Please triage this to decide if it is a release blocker. If not, please provide the workaround for the users. Thanks!

cc/
Digging into the code, will send a PR.
PR sent. Folks review it please.
Added the release-blocker label. We got a PR for it very fast so it won't delay the release anyway, thanks


Question: if m.invoked <= 1+m.failures+m.successes { return &m.configurationList, nil } <del> m.t.Fatalf("unexpected call to List, stopCh has been closed at the %d time call", 1+m.successes+m.failures) </del> <ins> m.t.Fatalf("unexpected call to List, should only be called %d times", 1+m.successes+m.failures) </ins> return nil, nil }

Found in I can reproduce locally using
/assign (via git blame)
cc
Thanks. I will take a look.
Got a fix locally. Testing it.


Question: # Enable metadata concealment by firewalling pod traffic to the metadata server # and run a proxy daemonset on nodes. ENABLE_METADATA_CONCEALMENT="${ENABLE_METADATA_CONCEALMENT:-true}" # true, false <ins> METADATA_CONCEALMENT_NO_FIREWALL="${METADATA_CONCEALMENT_NO_FIREWALL:-false}" # true, false </ins> if [[ ${ENABLE_METADATA_CONCEALMENT:-} == "true" ]]; then # Put the necessary label on the node so the daemonset gets scheduled. NODE_LABELS="${NODE_LABELS},beta.kubernetes.io/metadata-proxy-ready=true"

you this variable - PTAL
/sig gcp
Ping? I see that in all failure log messages for . Not sure if thats the cause or just an error message I can ignore?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale /assign since I bumped into this


Question: fmt.Fprintf(out, "Name:t%sn", pv.Name) fmt.Fprintf(out, "Labels:t%sn", formatLabels(pv.Labels)) fmt.Fprintf(out, "Status:t%dn", pv.Status.Phase) <del> fmt.Fprintf(out, "Claim:t%dn", pv.Spec.ClaimRef.UID) </del> <ins> if pv.Spec.ClaimRef != nil { fmt.Fprintf(out, "Claim:t%dn", pv.Spec.ClaimRef.Namespace+"/"+pv.Spec.ClaimRef.Name) } else { fmt.Fprintf(out, "Claim:t%dn", "") } </ins> return nil }) }

After I created a persistent volume, it throws an error on "describe".
Thanks for reporting this. will fix this nil pointer bug. The problem is your volume is not yet bound, but the command line wants to show you some info from the bound claim, except there isn't one.
do you want to break out as a separate pr to fix this, since the nil deref sort of a standalone bug?
done. .


Question: - ./hack/build-go.sh script: <del> - KUBE_RACE="-race" KUBE_COVER="-cover -covermode=atomic" KUBE_TIMEOUT='-timeout 60s' ./hack/test-go.sh "" -p=4 </del> <ins> - KUBE_RACE="-race" KUBE_COVER="y" KUBE_GOVERALLS_BIN="$HOME/gopath/bin/goveralls" KUBE_TIMEOUT='-timeout 60s' KUBE_COVERPROCS=8 ./hack/test-go.sh -- -p=2 </ins> - PATH=$HOME/gopath/bin:./third_party/etcd:$PATH ./hack/test-cmd.sh - PATH=$HOME/gopath/bin:./third_party/etcd:$PATH ./hack/verify-gendocs.sh - PATH=$HOME/gopath/bin:./third_party/etcd:$PATH ./hack/test-integration.sh

Our test- script invokes 'go test' serially for each pkg - currently taking 2 minutes on my laptop. Running a single 'go test' for all pkgs cuts the time down to 45 seconds. Even running with -race is only 1:15. As far as I can tell, the only time we HAVE to run serially is when coverage is on. Does anyone have time to fix this?
I'd created on Friday to fix this, though it's not yet been reviewed.
I cobbled together but I'll review yours since it is probably better :) , Jeff Grafton wrote:


Question: } }, "required": [ <del> "replicas", "availableReplicas" </del> <ins> "replicas" </ins> ], "type": "object" },

Looking at the for , it defines as a required attribute. But looking at the of , it states . Should this in fact be a required attribute? I come about this from using , a python package that generates models from the swagger definitions and thus builds a that requires an attribute that is not enabled. That only truly required attributes be required in the swagger definition Reproduction is not applicable as this is an issue with a spec No response Appeared in 1.22 (I believe) and affects current versions. Does not affect 1.21 n/a No response No response No response No response
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
alpha: has that is planned for removal in beta. ( results in fields as non-required in the OpenAPI) beta: removes the this might be a question whether feature gate controlled fields must remain until the feature gate is removed (and how would validation work in that case). ~old feature gates graduations to beta enabled a feature by default.~ a topic related to ? but it does not apply to feature gates. /kind documentation /sig apps
cc (since you are the author)
new fields to existing types must always be marked as +optional, since clients are not required to specify them
please open a PR adding back the +optional tag, regenerate openapi, and open picks back to 1.23 and 1.22


Question: } // GetNodeIP returns the ip of node with the provided hostname <ins> // If required, wait for the node to be defined. </ins> func GetNodeIP(client clientset.Interface, hostname string) net.IP { var nodeIP net.IP <del> node, err := client.CoreV1().Nodes().Get(hostname, metav1.GetOptions{}) if err != nil { klog.Warningf("Failed to retrieve node info: %v", err) return nil </del> <ins> backoff := wait.Backoff{ Steps:    5, Duration: 1 * time.Second, Factor:   2.0, Jitter:   0.2, </ins> } <del> nodeIP, err = GetNodeHostIP(node) if err != nil { klog.Warningf("Failed to retrieve node IP: %v", err) return nil </del> <ins> err := wait.ExponentialBackoff(backoff, func() (bool, error) { node, err := client.CoreV1().Nodes().Get(hostname, metav1.GetOptions{}) if err != nil { klog.Errorf("Failed to retrieve node info: %v", err) return false, nil } nodeIP, err = GetNodeHostIP(node) if err != nil { klog.Errorf("Failed to retrieve node IP: %v", err) return false, err } return true, nil }) if err == nil { klog.Infof("Successfully retrieved node IP: %v", nodeIP) </ins> } return nodeIP }

During kube-proxy bootstrap, kube-proxy retrieves node IP by accessing node object. Because kube-proxy is a static pod, kubelet may start kube-proxy before completing self registration. Hence kube-proxy may start up without knowing the correct node IP.
to confirm.
I'm hitting this as well, for the same reason - kube-proxy comes up before node registration is complete. I'm wondering what are the consequences of this?
I completely missed the issue :- kube-proxy uses the host network, so it's not blocked by kubelet registering and getting the pod CIDR. Can't kube-proxy wait and/or retry? is the bug still valid or has it been fixed?
This is not fixed. We are moving kube-proxy to daemonset. I am wondering if there is some easy config to make sure kube-proxy starts up after node status becomes ready or some sort.
yay! Is making kube-proxy blocked until node is ready not an option?
/assign
GitHub didn't allow me to assign the following users: MikeSpreitzer. Note that only can be assigned. <detailsIn response to : /assign </details
/assign
Update?
I got derailed by ; clues there would be nice. I will get back to this today, one way or another.
I have been reading about tests, wondering where to add a test case that covers this problem. It is not clear to me that Kubernetes has a test suite that can test race conditions in cluster creation.
There are some reboot/restart test.
FYI:
A test case that probes this issue would be even more intimate with the way the cluster is deployed, needing to inject a controlled delay in the node startup sequence.
Will this make 1.7?
move to the next milestone
BTW, always makes the kube-proxy before the kubelet, so it is easy to probe this issue in manual testing.
Next milestone reason: it seems that there are still some conversations about this the PR fix around whether or not we want to make require the presence of . It seems warranted then to accept this fix in the next release.
We need to address this in a patch release, but it should not block 1.7.
This is impacting one of our worker nodes as well, which was working fine and then after a restart, the API server communication was impacted (for POST requests, e.g. , ).

We noticed the the error in the kube-proxy logs. The workaround is to detect the failure via the logs, and restart the kube-proxy pod.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
As far as I can tell, the nodeIP is used in one particular case and nowhere else. Specifically a service of type=LoadBalancer, which also uses LoadBalancerSourceRanges field AND specifies the node itself as a valid source range for the LB. That's a pretty corner case. I bet we could find a better way to express that, and eliminate that parameter and error message entirely. I doubt it is actually causing anyone problems unless I totally misunderstand what that field is doing
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Since the PR is merged to master, is also used in . Now if this race condition happens, kube-proxy is not able to determine the correct nodeIP. This affects all the nodeport services with kube-proxy mode ipvs. Please reopening the issue.
You can't reopen an issue/PR unless you authored it or you are a collaborator. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
looks like this shouldn't have gotten closed anyway /reopen /remove-lifecycle rotten /lifecycle frozen
Reopened this issue.

<detailsIn response to : looks like this shouldn't have gotten closed anyway /reopen /remove-lifecycle rotten /lifecycle frozen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Can't we use the similar fix given in PR .


Question: const ( requestKind   = "request_kind" <del> priorityLevel = "priorityLevel" flowSchema    = "flowSchema" </del> <ins> priorityLevel = "priority_level" flowSchema    = "flow_schema" </ins> phase         = "phase" mark          = "mark" )

<!-- Please only use this template for submitting enhancement requests --What would you like to be : API priority and fairness metrics have labels such as "priorityLevel" and "flowSchema". These should be "prioritylevel" and "flowschema" respectively. These metrics have been to an exception list: Why is this needed: snake_case is consistent with other metrics. /sig api-machinery /kind cleanup /cc Do we want to do this before 1.20 beta?
yes, lets do that.
/triage accepted Yes, we do not want to go to beta with known wrongness in the metrics.


Question: const noConnectionToDelete = "0 flow entries have been deleted" <del> func isIPv6(ip string) bool { netIP := net.ParseIP(ip) </del> <ins> func IsIPv6(netIP net.IP) bool { </ins> return netIP != nil && netIP.To4() == nil } <ins> func IsIPv6String(ip string) bool { netIP := net.ParseIP(ip) return IsIPv6(netIP) } </ins> func parametersWithFamily(isIPv6 bool, parameters ...string) []string { if isIPv6 { parameters = append(parameters, "-f", "ipv6")

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: There is a bug in ipv6, see: What you expected to happen: The judgement logic should be: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: /sig network /area ipv6
/assign cc


Question: } for _, c := range cases { <del> ssName, instanceID, err := ss.extractVmssVMName(c.vmName) </del> <ins> ssName, instanceID, err := extractVmssVMName(c.vmName) </ins> if c.expectError { assert.Error(t, err, c.description) continue

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: When useInstanceMetadata is set to true for vmss nodes, their providerID is wrong. What you expected to happen: Their providerID should be in format azure:///subscriptions/<subscription-id/resourceGroups/<rg-name/providers/Microsoft.Compute/virtualMachineScaleSets/<scaleset-name/virtualMachines/<instance-id How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig azure
adding priority important-soon so that this issue can stay in 1.10. Please update if incorrect. /priority important-soon


Question: } return nil } <ins> // parseKVLines parses output that looks like lines containing "<key>: <val>" // and returns <val> if <key> is found. Otherwise, it returns the empty string. func parseKVLines(output, key string) string { delim := ":" key = key + delim for _, line := range strings.Split(output, "n") { pieces := strings.SplitAfterN(line, delim, 2) if len(pieces) != 2 { continue } k, v := pieces[0], pieces[1] if k == key { return strings.TrimSpace(v) } } return "" } </ins>

, func restartNodes, assumes that MIG rolling update on GCE will cause node restart. However, there is a feature of MIG rolling update (currently broken) that skips update if the new instance template is the same as the old one. The feature is supposed to work quite soon, so the test will not test anything (reboot will not be performed). A workaround may be to create an identical instance template with a different name. By the way, the current name is misleading. It should be . cc
Hi Marek, Was this changed recently? When we tried this originally it worked. Now, I just tried doing a rolling update to the same template, and it immediately fails with status "PAUSED" and status message "Errors: unknown". The name was chosen to attempt to be future-provider-agnostic, though I agree is better.
Actually, I'm trying it now, and it definitely sometimes works (recreates the instances). If this functionality isn't expected to work (or isn't supported) we can switch this to the equivalent MIG recreate-nodes call. I'll do this.
I've heard that some two months ago MIG update would not recreate new nodes if the template was the same. Then they changed it and instances were recreated (this is when you wrote the test). I think it is considered a bug. And they are going to fix this bug quite soon. So in a week or two (or maybe more - I don't know their release cycle), instances will not be recreated. (sorry for incomprehensible description of this issue - I'm correcting it now).
, I think your error is not related to MIG update - currently instances should be recreated (i.e. the error in MIG rolling update is not fixed). The error may be fixed around Monday (but maybe later - even 2 weeks).
Ack—thanks for the extra detail! I'll just switch this test to and we can stop worrying about the timing. We'll switch this in GKE too. Thanks again, I really appreciate you filing this!
(And by switch I mean just turn off self-upgrades.)
is this v1?
The referenced PR was a refactoring and Github got a bit confused. This issue isn't closed.

I don't think we need a test to recreate the MIG before v1.0 (although if it gets done before the freeze I don't see why we wouldn't merge it).
I'm leaving the project, so reassigning to... Have fun!
/sig testing
Fixed forever ago. /close


Question: } }) <del> ginkgo.It("should not set default value if no default IngressClass", func() { </del> <ins> ginkgo.It("should not set default value if no default IngressClass [Serial]", func() { </ins> ingressClass1, err := createIngressClass(cs, "ingressclass1", false, f.UniqueName) framework.ExpectNoError(err) defer deleteIngressClass(cs, ingressClass1.Name)

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: gce-master-scale-correctness Which test(s) are flaking: [sig-network] Ingress API should support creating Ingress API operations [Conformance] Testgrid link: https://k8s- Reason for failure: Anything else we need to know: /kind flake /sig scalability /sig network /cc
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
this is caused by the issue described in a different test is creating a default ingressclass, which breaks creation of ingress objects if there was already a default ingressclass set up in the cluster /assign
this should resolve the issue:
I take it back, looks like the conformance test is also creating default ingressclass objects... it should not fix in


Question: } } } <del> } else { // TODO(dchen1107): Separate issue docker/docker#8294 was filed result.status.State.Waiting = &api.ContainerStateWaiting{ Reason: ErrContainerCannotRun.Error(), } </del> } return &result }

If you create a pod with a restart policy of never, and you have a container that fails to start (e.g. you set its command to a nonexistent file), the Kubelet currently tries to restart it infinitely because the container never started successfully, and therefore never received an exit code. You end up with something like this from Docker: Is it worth modifying the code to consider a container that failed to start as actually failed, so the pod's restart policy can govern it? Originally reported at
+1. Also, I'm not sure why we would even apply a restart policy, since the container never started. , Andy Goldstein wrote:
One issue that comes up with marking it failed is if its governed by a Replication Controller, a consequence of marking it Failed is a proliferation of Pods being rapidly created and then failed. I know pod garbage collection would help that, but it would be good if we can update the dependent controller to recognize in common cases to back-off if pods keep failing because of failure to start scenarios. , Vish Kannan wrote:
I thought Kubelet already surfaces both creation and start failure from docker. Let me find the old issue I filed.
Here is the issue and the initial fix is PR: This is a bug we should fix anyway.
what is the output of kubectl describe pods? Are container at waiting state with proper error message from docker?
With respect to replication controller, see also . It should back off.
kubelet does surface the error (see the status below). The problem is that it doesn't categorize the container as dead/terminated, which affects the restart policy. A container would be considered "terminated" in kubelet if its state is not Running and the . In the case where a container failed to start, the FinishedAt time is zero (see example below), and the container state would be . This may cause kubelet to not even back off restarting at all (didn't check thoroughly), let alone complying to the restart policy. We should make sure kubelet categorize these containers properly.
Ok, I reproduced the issue easily. Here are the current situation: When container cannot start by docker due to various reasons, Kubelet marked the container as Waiting state with proper reason (RunContainerError) and detail message for docker, so Pod will stuck at Pending state.

I copied & pasted a PodStatus example below: When above situation occurs, Kubelet will retry in next syncLoop, which is the original issue reported here. Please note that I call this retry, instead of restart, which arguably align with our current , and do nothing. So that upstream layer can pick up such issue, for example, job controller in this case, and delete such bad pods. Or Kubelet can introspect ContainerStatus's reason: With restartPolicy never, Kubelet marks such Container as terminated with above proper reason. With restartPolicy onFailure, Kubelet marks such Container as terminated with above proper reason, and pod stays Running. Retry / Restart will be backoff by the same backoff policy on the node. With restartPolicy Always, Kubelet marks such Container as terminated with above proper reason, and pod stays Running. etry / Restart will be backoff by the same backoff policy on the node. For replication controlled pods, since we didn't mark Pod as Failed, there is no issue mentioned at
I think kubelet should not rely on the reason (which is best-effort), and simply stops relying solely on to categorize "terminated" containers.
what would you look at in addition to ?
in my example above, the container which failed to start had . To me, it's clear that there was a failed attempt trying to start the container. Even though it's finish time is zero, this container should be considered "terminated".
ah, good point! I hadn't noticed that ExitCode was nonzero.


Question: } for i, tc := range tests { <del> t.Run(string(i), func(t *testing.T) { </del> <ins> t.Run(strconv.Itoa(i), func(t *testing.T) { </ins> roleBinding, err := tc.options.createRoleBinding() if err != nil { t.Errorf("unexpected error:n%#vn", err)

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which test(s) are failing: Since when has it been failing: since upgrading golang: Reason for failure: Since Go 1.15, type casting from int to string using is not allowed, output: Anything else we need to know: related links: golang/go golang/go
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
the fix should be straight forward, so I can create a pull request to fix this. /sig testing /sig cli /assign
/assign


Question: gvr("flowcontrol.apiserver.k8s.io", "v1beta2", "prioritylevelconfigurations"): { Stub:             `{"metadata": {"name": "conf3"}, "spec": {"type": "Limited", "limited": {"assuredConcurrencyShares":3, "limitResponse": {"type": "Reject"}}}}`, ExpectedEtcdPath: "/registry/prioritylevelconfigurations/conf3", <ins> ExpectedGVK:      gvkP("flowcontrol.apiserver.k8s.io", "v1beta3", "PriorityLevelConfiguration"), </ins> }, // --

adds new version for in 1.26. For 1.26, we are setting the storage version to because vN-1 understands that level. In 1.27, we should change the storage version from to .
/sig api-machinery
/cc
/cc
/triage accepted
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:


Question: fs.StringVar(&c.HairpinMode, "hairpin-mode", c.HairpinMode, "How should the kubelet setup hairpin NAT. This allows endpoints of a Service to loadbalance back to themselves if they should try to access their own Service. Valid values are "promiscuous-bridge", "hairpin-veth" and "none".") fs.Int32Var(&c.MaxPods, "max-pods", c.MaxPods, "Number of Pods that can run on this Kubelet.") <del> fs.StringVar(&c.PodCIDR, "pod-cidr", c.PodCIDR, "The CIDR to use for pod IP addresses, only used in standalone mode.  In cluster mode, this is obtained from the master. For IPv6, the minimum length of the cluster or pod CIDR is /66.") </del> <ins> fs.StringVar(&c.PodCIDR, "pod-cidr", c.PodCIDR, "The CIDR to use for pod IP addresses, only used in standalone mode.  In cluster mode, this is obtained from the master. For IPv6, the maximum number of IP's allocated is 65536") </ins> fs.Int64Var(&c.PodPidsLimit, "pod-max-pids", c.PodPidsLimit, "<Warning: Alpha feature> Set the maximum number of processes per pod.") fs.StringVar(&c.ResolverConfig, "resolv-conf", c.ResolverConfig, "Resolver configuration file used as the basis for the container DNS resolution configuration.")

/kind bug What happened: The kubelet command option description for pod-cidr shows the following "For IPv6, the minimum length of the cluster or pod CIDR is /66" however, this /66 restriction was removed with PR What you expected to happen: The command option should not display this restriction. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/assign rpothier
/sig node


Question: docker_pull( name = "debian-iptables-amd64", <del> digest = "sha256:bfc7cc030258f53495b5dacf1e1d750db9b8687577a8648a3c8e245f8d7d2c52",  # v7 </del> <ins> digest = "sha256:bc20977ac38abfb43071b4c61c4b7edb30af894c05eb06758dd61d05118d2842",  # v7 </ins> registry = "gcr.io", repository = "google-containers/debian-iptables-amd64", )

A number of addon images have CVEs that have been fixed in the upstream base images. We should update these images before the final 1.7 release. cc
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
/cc
cc
The new image appears to have removed . I'm not sure if this was expected, but it caused at least fluentd to stop working on COS (). This might also affect kubeproxy, which uses (a derivative of ).
also the metadata-proxy, prometheus-to-sd, the fluentd-event-exporter, and ip-masq-agent.
anything more to do for this issue?
yes, I need to get merged and update those images again. I also need to bump a few other images (dns and ip-masq-agent).
should be the last update needed.


Question: * [kubectl](kubectl.md)	 - kubectl controls the Kubernetes cluster manager <del> ###### Auto generated by spf13/cobra at 2015-07-16 22:39:16.132838722 +0000 UTC </del> <ins> ###### Auto generated by spf13/cobra at 2015-08-01 00:12:16.314376117 +0000 UTC </ins> <!-- BEGIN MUNGE: GENERATED_ANALYTICS -->

The current documentation is very vague and doesn't explain what can be updated in this way. Can I rename or stop the pod, change number of replicas in RC etc. For example it seems to be impossible to change pod name:
cc
After an in person discussion with we think the docs should point people to the swagger-ui to find if a field is mutable. I'll add this point into the docs.


Question: $(warning ***** finding all +k8s:deepcopy-gen tags) endif DEEPCOPY_DIRS := $(shell                                <del> grep -l '+k8s:deepcopy-gen=' $(ALL_K8S_TAG_FILES)   </del> <ins> grep --color=never -l '+k8s:deepcopy-gen=' $(ALL_K8S_TAG_FILES)   </ins> | xargs -n1 dirname                             | sort -u                                       )

Using the grep from OSX, fails when GREPOPTIONS/GREPCOLOR is set as shown below (). Workaround : unset GREPOPTIONS/GREPCOLOR Note that using GNU grep using brew install will show that GREPOPTIONS shows immediately that GREP_OPTIONS may be interfering with make ()
oh, here's badly manged conversion-
Will look, thanks. For make-related bugs, please @ me directly.
Will do. thanks
This breaks building kubernetes inside Alpine Linux:
I'm pressing more and more on the idea that the only supported build is the dockerized build. Trying to accomodate every random environment is just too taxing. , Marc Villacorta wrote:
I agree with We can support the stuff in to run on mac to get you dockerized but everything else is only supported in there. I've worked around a few BSD/OSX things in but it isn't fun. We should try to keep that stuff isolated.
I agree, with the usual caveat that we need to be doing all of the build in one container with as little as possible happening outside. OS X tends to have old and broken versions of important programs, which is terrible. We discussed setting up CI for it but it's not going to happen in the near term. Alpine/Busybox purposefully omit features from their versions for good reasons, but it's still more work to support than it's worth.
Ok, I will re-think my workflow.
How did you find yourself building in Alpine, in I may ask? , Marc Villacorta wrote:
nothing is forcing me to use Alpine I just like the distro. I run stuff on CoreOS and my aim is to solve the build-deliver-run pipeline with a single small container. By the way, I have the problem installing the GNU package :-)


Question: "sync" "time" <del> "k8s.io/api/core/v1" </del> <ins> v1 "k8s.io/api/core/v1" </ins> metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/labels" "k8s.io/apimachinery/pkg/runtime"

Kubelet and System-reserve are implemented on Windows along with pod/container reserves which can help prevent overprovisioning, but we still need to test OOM behavior. We need a test case that will: kubelet & system reserve pods until node is full something that will take up system ram so that OOM is triggered /sig windows /milestone v1.14
I have started to look into this test case and have a few questions: What is the expected behavior we are testing for? What does "run something that will take up system ram" mean? Does that mean run a pod that uses up memory or run a system service that uses memory? After setting the reserves and as mentioned in we should be able to schedule pods until full and then once we trigger OOM (either by pod or system service) kubelet should continue running becuase of the reserves? Do the two settings need to be tested independently or they should be tested together? I found that seems to be doing something similar but it is in node-e2e. Is sig-windows running node-e2e tests currently?
/assign
GitHub didn't allow me to assign the following users: jsturtevant. Note that only and repo collaborators can be assigned and that issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
After researching this further and talking with I found that the does test the allocatable memory but uses CGroups to set reserves to test the reported vs actual. As windows does not have CGroups this test will not port as is. There is and the is not implemented so will not work either. The proposed testing solution would be two parts: kubelet & system reserve and verify allocatable is reporting properly. Schedule pods with known memory consumption values and verify the allocatable is correclty updated.Schedule until full and verify scheduling is not allowed. kubelet & system reserve and verify allocatable is reporting properly. Schedule Schedule until full and verify scheduling is not into machine and run process that consumes memory and verify kubelet and pods remain running. There is an open question on the second test as to whether kubelet and will be have properly.

If it does not we will submit test 1 and open an issue to resolve 2.
I would suggest adding a third case that would test the behavior when excessive memory pressure is exerted on the node from the pods. This is perhaps a more common use case than 2 since the pod workloads are more likely to exhibit greater resource variability than node system processes. It would also carry over as a test for when/if node allocatable enforcement get implemented for windows nodes. Additionally, to complement this work would it be suitable to test windows pod memory limit enforcement? Or is this tested elsewhere?
Agreed this is a more common case. I am not sure that it is. Is this tested on linux somewhere?
I am seeing strange numbers reported when performing a test case. I scheduled a few pods to test memory limits and then schedule a pod with no memory limit and a process that will consume all of the available memory 500mb at a time: After the pod consumes all available memory, When I compare Node Allocatable/Node usage/Pod Usage I am not seeing the numbers add up properly: My node is staying online and kubelet is reporting no memory pressure. The total node memory consumption is over the Allocatable memory. What is the expected behavior and what should the test look like?
I believe node allocatable will always just be capacity minus reserves (system and kubelet) - So in this scenario it looks like you have and/or set? The memory % on the node is relative to the allocatable amount, and still seems to be under the total memory capacity - are you sure that all available memory has been consumed by your oom pod? The node should be tainted with memory-pressure when a hard or soft eviction threshold is met (). What do you have set for and ?
Thanks for the clarification. I currently have these settings: And on the machine I am seeing most memory being used: If I understand correctly, I start to see memory-pressure once I get into the setting above. In that case I should start to see pods get evicted, so for a valid test I could adjust the values to be easier to hit then test to see that pods get evicted properly which would be similar the .

I am going to adjust a few of the settings and manually test that this is working then see what it will take to port the PriorityMemoryEviction, which currently uses cgroups which won't work on windows. This combined with testing that Memory Allocatable is being properly set should give us a good test coverage for windows on this scenario.
I performed manual testing and found that even when memory was used up on the machine the "Memory Pressure" notification was not fired. After looking deeper I found is not implemented for windows. The , which uses cgroups, provides the . This means even though is being it isn't used. On Windows, Node allocatable when used with limits and requests does prevent the node from over scheduling. I believe the the test and  (this only shows cpu and I could not find a test that demonstrated memory) demonstrate this works. But this doesn't stop a pod from consuming all the memory on the node and since eviction (hard/soft/node allocatable) is not implemented on windows this means a node can end up resource starved. I think the only valid test to be written for this scenario, as it stands now, is validating that Node Allocatable is being reported properly. Beyond that there is no functionality that can be validated around OOM behavior for the windows node.
Yeah, that sounds to be in agreement with what I've observed on the windows nodes that I run. To prevent resource starvation (as a workaround), would setting mem/cpu requests equal to mem/cpu limits for all windows pods work? I think the request / limit values get passed down into the docker daemon when it starts a container, so is it possible that docker could enforce the memory limit?
Yes, setting limits==requests would help the scheduler not over provision the nodes. I think the best way to approach this particular issue for 1.14 is to make sure docs call out what works/doesn't work and give guidance to on how to prevent resource starvation on nodes. Is there a place in the docs that would make sense for this information?

The call outs would be: , , and flags do update Node Allocatable is not implemented and are not implemented Condition is not implemented in Windows There are no OOM actions taken by kubelet As a work around to previous, and should always be equal to prevent memory resource starvation as this allows the scheduler to to use limits and Kubelet running on the windows node does not have memory restrictions. and system-reserve does not set limits on kubelet or processes running the host. This means kubelet or a process on the host could cause memory resource starvation outside the and scheduler
thank you for the thorough investigation


Question: } return ipnets, nil } <ins> // RequestsOnlyLocalTraffic checks if service requests OnlyLocal traffic. func RequestsOnlyLocalTraffic(service *v1.Service) bool { if service.Spec.Type != v1.ServiceTypeLoadBalancer && service.Spec.Type != v1.ServiceTypeNodePort { return false } // First check the alpha annotation and then the beta. This is so existing // Services continue to work till the user decides to transition to beta. // If they transition to beta, there's no way to go back to alpha without // rolling back the cluster. for _, annotation := range []string{AlphaAnnotationExternalTraffic, BetaAnnotationExternalTraffic} { if l, ok := service.Annotations[annotation]; ok { switch l { case AnnotationValueExternalTrafficLocal: return true case AnnotationValueExternalTrafficGlobal: return false default: glog.Errorf("Invalid value for annotation %v: %v", annotation, l) } } } return false } // NeedsHealthCheck Check if service needs health check. func NeedsHealthCheck(service *v1.Service) bool { if service.Spec.Type != v1.ServiceTypeLoadBalancer { return false } return RequestsOnlyLocalTraffic(service) } // GetServiceHealthCheckNodePort Return health check node port annotation for service, if one exists func GetServiceHealthCheckNodePort(service *v1.Service) int32 { if !NeedsHealthCheck(service) { return 0 } // First check the alpha annotation and then the beta. This is so existing // Services continue to work till the user decides to transition to beta. // If they transition to beta, there's no way to go back to alpha without // rolling back the cluster. for _, annotation := range []string{AlphaAnnotationHealthCheckNodePort, BetaAnnotationHealthCheckNodePort} { if l, ok := service.Annotations[annotation]; ok { p, err := strconv.Atoi(l) if err != nil { glog.Errorf("Failed to parse annotation %v: %v", annotation, err) continue } return int32(p) } } return 0 } // GetServiceHealthCheckPathPort Return the path and nodePort programmed into the Cloud LB Health Check func GetServiceHealthCheckPathPort(service *v1.Service) (string, int32) { if !NeedsHealthCheck(service) { return "", 0 } port := GetServiceHealthCheckNodePort(service) if port == 0 { return "", 0 } return "/healthz", port } </ins>

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Environment: Cloud provider or hardware configuration: AWS m4.2xlarge OS (e.g. from /etc/os-release): Ubuntu 16.04 Kernel (e.g. ): 4.4.0-36-generic Install tools: kubeadm Others: 1 master and 1 minion What happened: This states that if using kube-proxy will only proxy requests to local endpoints, never forwarding traffic to other nodes and thereby preserving the original source IP address. However when I annotate my NodePort type service with this annotation I see the following error in the logs of kube-proxy of the node where the only pod of the service is running: After this message the service stops working outside of the cluster on the specified NodePort. What you expected to happen: I expected to see that only the with the pod on it serves traffic. How to reproduce it (as minimally and precisely as possible): Anything else we need to know:
Sorry, I have found the issue, I had a setup where the none of the node names were the same as the hostname of the kube-proxy pod and this caused the issue, where none of the Pods running on a hosts were recognised as local pods. I have fixed this issue by manually setting the hostname with in kube-proxy. However I still insist that this error message is misleading and shouldn't be printed in case of NodePort type services, but only for just LoadBalancers.
I met the same problem in the aws cloud providers, by adding - the hostname-override to solve this problem. good


Question: // RemoveStatusCondition removes the corresponding conditionType from conditions. // conditions must be non-nil. func RemoveStatusCondition(conditions *[]metav1.Condition, conditionType string) { <del> if conditions == nil { </del> <ins> if conditions == nil || len(*conditions) == 0 { </ins> return } newConditions := make([]metav1.Condition, 0, len(*conditions)-1)

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --Attempting to remove a status condition from an already empty slice panics. An empty slice should be removed (i.e. removed idempotently) and without panicking. Originally found with , reproduced with ( at the time of writing) Kubernetes version (use ): N/A Cloud provider or hardware configuration: N/A OS (e.g: ): Pop!OS 20.04 LTS Kernel (e.g. ): Install tools: N/A Network plugin and version (if this is a network-related bug): N/A Others: N/A
/sig api-machinery
cc
/assign
/triage accepted


Question: len(nodes.Items)) } <del> ips := e2enode.CollectAddresses(nodes, v1.NodeInternalIP) </del> type NodeInfo struct { node   *v1.Node name   string nodeIP string } <ins> var family v1.IPFamily if framework.TestContext.ClusterIsIPv6() { family = v1.IPv6Protocol } else { family = v1.IPv4Protocol } ips := e2enode.GetAddressesByTypeAndFamily(&nodes.Items[0], v1.NodeInternalIP, family) </ins> clientNodeInfo := NodeInfo{ node:   &nodes.Items[0], name:   nodes.Items[0].Name, nodeIP: ips[0], } <ins> ips = e2enode.GetAddressesByTypeAndFamily(&nodes.Items[1], v1.NodeInternalIP, family) </ins> serverNodeInfo := NodeInfo{ node:   &nodes.Items[1], name:   nodes.Items[1].Name, <del> nodeIP: ips[1], </del> <ins> nodeIP: ips[0], </ins> } // Create a pod to check the conntrack entries on the host node

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: sig-network-kind-ipvs,dual,master sig-network-kind-ipvs,IPv6,master Which test(s) are failing: [x] KubeProxy should set TCP CLOSE_WAIT timeout [x] Conntrack should be able to preserve UDP traffic when server pod cycles for a NodePort Service [x] KubeProxy should resorve connection reset issue (tracking in ) Since when has it been failing: Oct 28 Testgrid link: sig-network-kind-ipvs,dual,master  Reason for failure: Both appear to be failing on missing conntrack entries ex. Anything else we need to know:
/sig network /cc /assign
I'm spinning up a kind cluster with ipv6+ipvs to see if I can replicate this test locally
/triage accepted
heh, the job was not running it, it started to run when we changed the tag I meant this test and the PR where we renamed the kube-proxy test
this one has an issue already open, and I have a PR for fixing it
reached out to me on Slack and we identified as the high priority. I have tested this locally and it fails on both IPTables and IPVS kind clusters locally. I'm digging through the cluster logs now.
I' ve logging to conntrack with level v4 recently just for these problems, try to bump your kube-proxy logging to v4
seems that there are less test failures on the IPv6 jobs, let's keep watching them :eyes: However, the dual-stack jobs are legit failures, basically because on dualstack clusters, the node has 2 internal IPs, and those tests use the internal IP for asserting some operations, and the problem is that they are picking the internalIP of the wrong IP family, hence the test fails The solution should be to add the InernalIp correspoding to the cluster primary IP familty, you can obtain it with but it seems that we have to modify the function or add a new one that returns the nodes and the corresponding ips
Hey Thanks for all the help, will look into the areas of the code and try to bump the logging.


Question: } // We want to add the pod to the model iff the bind succeeds, but we don't want to race <del> // with any deletions, which happen asyncronously. </del> <ins> // with any deletions, which happen asynchronously. </ins> s.config.Modeler.LockedAction(func() { bindingStart := time.Now() err := s.config.Binder.Bind(b)

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: Which test(s) are failing: [] Container Runtime blackbox test on terminated container should report termination message from log output if TerminationMessagePolicy FallbackToLogsOnError is set [NodeConformance] Since when has it been failing: Failing under PR Testgrid link: Reason for failure: Unable to match the inconsistent termination message with an expected value. Anything else we need to know: <!-- DO NOT EDIT BELOW THIS LINE --/kind failing-test
/sig testing /area conformance
Just for quick update. From the latest of PR , following points are observed from - PASSING under local cluster (hack/local-up-) tmp-node-e2e--coreos-beta-1883-1-0-v20180911 tmp-node-e2e--cos-stable-60-9592-84-0 tmp-node-e2e--cos-stable-63--71-0 FAILING under tmp-node-e2e--ubuntu-gke-1604-xenial-v20170420-1
Seeking attention from for troubleshooting issue. test is running successful on various platforms but not able to retrieve test's execution logs in kubelet's log. Tried investigating issue (ref ) from various execution platform where test is successful but couldn't find sufficient log entries. Seeking SIG help to troubleshoot test failure observed in PR
there are two problems here: 1) so far the broken tests never get run because a bug in the test code (this explains why pull-kubernetes-node-e2e never fails on our PRs), which should be fixed by . 2) With , I can reproduce your problem, and verify that the ubuntu image used in our test is broken. We will update the ubuntu image.
Thanks for looking into this long-awaited issue.


Question: } // Expect EndpointSlice resource to be deleted when Service is. <del> if err := wait.PollImmediate(2*time.Second, 12*time.Second, func() (bool, error) { </del> <ins> if err := wait.PollImmediate(2*time.Second, wait.ForeverTestTimeout, func() (bool, error) { </ins> endpointSliceList, err := cs.DiscoveryV1beta1().EndpointSlices(svc.Namespace).List(context.TODO(), metav1.ListOptions{ LabelSelector: "kubernetes.io/service-name=" + svc.Name, })

Which jobs are flaking: ci-kubernetes-kind-e2e-parallel pr:pull-kubernetes-e2e-gce pr:pull-kubernetes-e2e-gce-ubuntu-containerd pr:pull-kubernetes-e2e-gce-ubuntu-containerd ci-kubernetes-kind-ipv6-e2e-parallel Which test(s) are flaking: Second flakiest test in the e2e job according to Testgrid link: Reason for failure: Anything else we need to know: /sig network /assign
needs investigation since we're switching kube-proxy to endpointslice in this release
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
made me think of , but that's in already.
It's possible it's a test timeout that's just too close, but we need verification of that checking the controller and apiserver logs to make sure there isn't a latent bug
Thanks for catching this! It looks like the timeouts are actually associated with waiting for 2 pods to be ready, not actually anything related to EndpointSlice functionality - timeout is here: Given that, it seems like the obvious solution would be to update the Services to set to and update this block to wait for the Pods to have IPs, not for them to be ready. If that sounds reasonable, I'll work on getting a fix in tomorrow.
/remove-triage unresolved
If that is still exercises the same conditions (I vaguely remember publishNotReadyAddresses changing code paths in the endpoints controller... does it do the same in the endpointslice controller?), that seems reasonable to me.
I think the code path is very similar here, the only real difference should be how the EndpointSlice controller calculates readiness. I've gone ahead and filed a PR that uses this approach, hopefully it helps:
This flaked again this morning /reopen
Reopened this issue. <detailsIn response to : This flaked again this morning Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This should be /priority critical-urgent given
The top level error message does not make this clear but the issue here is that too many EndpointSlices are being created - 3 instead of 2.
Interesting... that condition should not persist for O(minutes), right?

that seems like it might be a bug, not just a test issue
There's potential that it's not consistently merging 2 ports with 1 endpoint into a single slice and instead creating a slice per port. It also could represent a larger bug. Trying to find a way to consistently reproduce this.
Despite my best efforts I have been unable to reproduce this on my own e2e cluster. I've had 10 test runs going at the same time repeatedly without any luck, any ideas for how to reproduce flaky test failures local would be welcome. In the meantime, I've created a follow up PR that will provide more helpful information if/when this test fails again: My leading theory right now is that this could be a caching issue. The EndpointSlice controller runs a sync, sees that we don't have an EndpointSlice for the given port combination in cache and then goes ahead and creates one, even though a matching EndpointSlice already exists and has not been cached yet. That would result in an annoying situation where an essentially duplicate EndpointSlice was created due to a stale cache. I would think the EndpointSliceTracker would resolve this issue for us though, so not sure. Getting the above PR in will help validate this theory. It's worth noting that if this is actually a duplicate EndpointSlice, clients like kube-proxy will handle this seamlessly because they already have to handle potential duplication in EndpointSlices. That would potentially explain why a test failure like this does not seem to be causing other things to break.
It appears that my theory was correct as witnessed here: I create a .

The key points I've noticed so far: 1 EndpointSlice references pod2 (example-named-port-f7vq5) 2 EndpointSlices reference pod1 (example-named-port-fkgcc and example-named-port-vfp2q) 2 EndpointSlices were created at "2020-07-28T04:57:51Z": example-named-port-f7vq5 (references pod2) example-named-port-vfp2q (references pod1) 1 EndpointSlice was created 1 second later ("2020-07-28T04:57:52Z") example-named-port-fkgcc (references pod1) All 3 EndpointSlices are: On generation 1 - no updates have occurred With a state of ready: true The 2 EndpointSlices referencing pod1 are effectively identical  Of course this is something the EndpointSliceTracker should be handling for us, but even that approach seems less than ideal here. It wouldn't actually prevent the duplicate EndpointSlices from being created, it should just be cleaning up duplicates after they happen. This combined with the race conditions referenced in make me think we may want to find a way to slow down the sync cycle for each Service here. This also feels at least tangentially related to the need for better rate limiting in both this and the Endpoints controller discussed in . /cc
Hmm - actually I don't understand why this is happening. Maybe I'm missing something, but I don't think slowing down controller is needed to solve this problem. Even though the controller usually is not lagging my more than 1s, it is possible that it can be lagging by a minute or more in pathological cases. So we can't just rely on the fact that it's guaranteed to be synced. Can you explain it a bit deeper why this is happening? i.e. some more detailed timeline like: T1: state of cache is XYZ T2: controller computes the ES x1, x2 T3: state of cache in ABC T4: controller computes ES y1 [why?]
Agree, I'm not in favor of slowing down the controller globally to address this. Delaying resync of a particular service until we observe the existence of new endpointslices we know we created (with some max delay) could be reasonable.

That's exactly where I was hoping to end up here, although I'm not quite sure how to do that cleanly yet. To be clear, this is not just to fix this flaky test. This flaky test is a symptom of a much larger and more annoying issue with the current implementation. The EndpointSlice controller is consistently running into errors where it's trying to update EndpointSlices that are stale. This is caused by follow up calls to running before the informer cache is up to date, which happens all the time during rolling updates where rapid churn of EndpointSlices is expected. Although this eventually resolves itself, it's causing a lot more load on API Server than necessary and can potentially result in some bad states where an update fails but a delete succeeds. In this specific instance, it looks like a syncService run is occurring after the first 2 EndpointSlices have been created but before the one referencing pod1 is in the informer cache. That results in the controller creating a new EndpointSlice to reference pod1 and thus we end up with a duplicate. If anything in the future triggers another syncService call that would be resolved. I'd initially thought that EndpointSliceTracker would be thing to trigger that follow up syncService, but I was wrong. That tracker just tracks the latest resource version for each EndpointSlice the controller has created/updated and compares that with EndpointSlices received through create/update events. If the controller doesn't think that EndpointSlice should exist or the controller's resource version for that slice does not match it triggers another syncService. In this case neither of those conditions would be met so no additional syncService call would be triggered. So that's a long way of saying that I think our best possible solution here is to introduce some delay between when Services can be synced. I think the easiest solution that would solve 99% of our problems would be to add something like a 1s delay in between syncs simply by delaying when was called. This would not be ideal, but it would result in an immediate improvement and be easy to backport. It's worth noting that although duplicate EndpointSlices like this or even attempts to update a stale EndpointSlice are not ideal, they are not actually invalid and will eventually resolve themselves. Consumers of this API already have to do some form of deduping.

With that said, I'm hopeful that there's a clean way to implement a solution that actually waited for each EndpointSlice change the controller has made to be reflected in the cache because that would be a better solution. I'll spend some time looking into possible implementations here.
I've created a draft PR that contains my initial progress towards the approach I described above. I don't particularly love this approach so I'm very open to other ways we could solve this issue. It's also still very much a WIP and has significant work remaining. I opened the PR early just so I could get some initial feedback on the direction and potentially inspire other ideas around approaches we could take here.
is still the best approach I can come up with here. It is ready for review if anyone has time to take a look.
Quick update here, I've limited the scope here to the following changes as discussed in Update the flaky test to consider duplicate EndpointSlices valid. Update the docs to make it very clear that consumers of the EndpointSlice API will have to do deduplication (as kube-proxy already does). Update the controller error handling to return immediately if a create, update, or delete errors (in that order). Reviews would be appreciated on all of the above.
Reopening since is not yet merged.
Thanks for catching that Closing now that is merged. /close
Closing this issue. <detailsIn response to : Thanks for catching that Closing now that is merged. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: "k8s.io/apimachinery/pkg/util/wait" federationapi "k8s.io/kubernetes/federation/apis/federation/v1beta1" fedclientset "k8s.io/kubernetes/federation/client/clientset_generated/federation_clientset" <del> "k8s.io/kubernetes/pkg/api" </del> "k8s.io/kubernetes/pkg/api/v1" kubeclientset "k8s.io/kubernetes/pkg/client/clientset_generated/clientset" <ins> "k8s.io/kubernetes/pkg/cloudprovider" </ins> "k8s.io/kubernetes/test/e2e/common" "k8s.io/kubernetes/test/e2e/framework" fedframework "k8s.io/kubernetes/test/e2e_federation/framework"

In federation tests, if we observe the build logs. we see cleaning up LoadBalancer from cloud provider seems not working. It has 2 effects: delays the tests by waiting to delete but always returns error. so we waste like 5 minutes. maybe we are leaking the LoadBalancer in the project.
we can see these errors in every build. but just for reference here are some build logs https://k8s-
cc
All the errors in logs are . It doesn't necessarily indicate we are leaking resource, just now we must have clusterID initialized before calling EnsureLoadBalancerDeleted(). Would be better to not call this function directly in e2e test :)
Federation cleanup should use the same implementation as non-federated e2e test:
function looks like only specific to GCE. in future we would run the e2e tests in AWS too. it would be better to use generic functions if any.
using is working for us. here is the test logs https://k8s-
EnsureLoadBalancerResourcesDeleted() actually cleans up nothing, it only lists forwarding rules and probably logging :)


Question: } } <ins> for i := 0; i < nodeCount; i++ { // Stop the neededNode's heartbeat goroutine. if neededNode.Name == fmt.Sprintf("node-%d", i) { heartbeatChans[i] <- struct{}{} break } } </ins> neededNode.Status.Conditions = test.nodeConditions // Update node condition. err = updateNodeStatus(cs, neededNode)

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: Which test(s) are failing: TaintBasedEvictions Since when has it been failing: probably since it was in this PR? Testgrid link: Reason for failure: Anything else we need to know: /sig scheduling /kind flake /priority important-soon /milestone v1.17
You must be a member of the GitHub team to set the milestone. If you believe you should be able to issue the /milestone command, please contact your and have them propose you as an additional delegate for this responsibility. <detailsIn response to : <!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs -- TaintBasedEvictions probably since it was in this PR? Anything else we need to know: /sig scheduling /kind flake /priority important-soon /milestone v1.17 </details
cc.
Looking into this. We moved it to integration because it was flaking in e2e too
/assign damemi
Not sure yet, but I do see this in the logs of the failing tests: which, if I understand this, would remove all the taints and mark nodes as reachable (which would explain why the test is failing waiting for node taints: ). But I need to figure out why all the nodes are becoming unavailable
is that a full fix for this flake or just partial? I still saw it failed yesterday so just checking... :)
I think I also run into this on 13th Oct: Full logs:
Now gets merged, which might fix the flake. Let's keep an eye on this.
reported it still flaked: Can you continue looking into this?
I'll keep trying, from my previous debugging it seems that in some cases the test cluster is entering full disruption mode and removing all of the taints from the nodes. I know we specifically made changes to the test (using multiple nodes) to ensure it wouldn't become fully disrupted, do you have any idea what could still be causing that?
There is some flag controlling how the dispution thing is managed internally, maybe you can check how a node lifecycle manager is constructed.

The test still flakes after we merge , we still need to investigate the cause ,
/assign
Another flake:
These flakes are consistently (always) timeouts waiting on node taints:
I'm working on this. Will send out a PR soon.
/reopen We see another flake since got merged: However, don't panic :) The original "Entering master disruption mode" cause has been fixed - you won't see that message in the above log. This time it's a smaller issue. I will explain it in another PR.
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen The job is still flaky
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/close But for another test :)
Closing this issue. <detailsIn response to : /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: expansionFailed expansionFailedOnController expansionFailedOnNode <ins> expansionFailedMissingStagingPath </ins> ) const (

When a pod is quickly deleted and re-created on a node, it may not be necessary to unstage the volume and in which case only node-publish is called for the volume. When this happens - stagepath won't be set for RPC call because volume was not staged even though, volume does have capability. We should try and fix this behaviour so that - nodestagepath is always set in RPC call. We should try and fix this behaviour so that - nodestagepath is always set in RPC call. Quickly delete and recreate a pod No response master Any No response_ <details</details<details</details<details</details
/sig storage /triage accepted


Question: }, } <del> time.Sleep(10 * time.Millisecond) if receivedCount != 1 { t.Errorf("Expected 1 call but got %v", receivedCount) </del> <ins> select { case <-received: case <-time.After(10 * time.Millisecond): t.Errorf("Expected 1 call but got 0") </ins> } // Test error case fakeEtcd.WatchInjectError <- fmt.Errorf("Injected error") <del> time.Sleep(10 * time.Millisecond) </del> // Did everything shut down? if _, open := <-fakeEtcd.WatchResponse; open {

Detected by
Create a Pod as follows: Name: jenkins Using image: jenkins In a new Kubenetes namespace named website-frontend kubectl create ns website-frontend apiVersion: v1 kind: Pod metadata: name: jenkins namespace: website-frontend spec: containers: name: jenkins image: jenkins kubectl create -f


Question: devID1 := parseLog(f, pod1.Name, pod1.Name, deviceIDRE) gomega.Expect(devID1).To(gomega.Not(gomega.Equal(""))) <del> v1alphaPodResources, err := getV1alpha1NodeDevices() </del> <ins> v1alphaPodResources, err = getV1alpha1NodeDevices() </ins> framework.ExpectNoError(err) <del> v1PodResources, err := getV1NodeDevices() </del> <ins> v1PodResources, err = getV1NodeDevices() </ins> framework.ExpectNoError(err) framework.Logf("v1alphaPodResources.PodResources:%+vn", v1alphaPodResources.PodResources)

Device Plugin [Feature:DevicePluginProbe][NodeFeature:DevicePluginProbe][Serial] DevicePlugin [Serial] [Disruptive] Can schedule a pod that requires a device It appears a bit flaky on testgrid, but more recently been failing more often No response example - No response /sig node
/cc
/triage accepted /assign /priority important-soon
I ran this test locally 15 times back to back and did not run into this failure. The test passed locally. I suspect this to be a case where pod cleanup is not happening properly. In this test, we are relying on the number of PodResources and based on the error it looks like we have an addtional pod running other than the two expected pods (device plugin pod and test pod requesting device) but can't be 100% sure because we don't have enough logs to support that claim. Going to add more logs to expose output obtained from the podresource API endpoint.


Question: Failf("got unexpected len(Status.LoadBalancer.Ingresss) for NodePort service: %v", service) } ingress2 := service.Status.LoadBalancer.Ingress[0] <del> // TODO: This is a problem on AWS; we can't just always be changing the LB Expect(ingress1).To(Equal(ingress2)) </del> <ins> // TODO: Fix the issue here: https://github.com/GoogleCloudPlatform/kubernetes/issues/11002 if providerIs("aws") { // TODO: Make this less of a hack (or fix the underlying bug) time.Sleep(time.Second * 120) service, err = waitForLoadBalancerIngress(c, serviceName, ns) Expect(err).NotTo(HaveOccurred()) // We don't want the ingress point to change, but we should verify that the new ingress point still works ingress2 = service.Status.LoadBalancer.Ingress[0] Expect(ingress1).NotTo(Equal(ingress2)) } else { Expect(ingress1).To(Equal(ingress2)) } </ins> By("hitting the pod through the service's updated NodePort") testReachable(ip, nodePort2)

Currently, when we change a service with a LoadBalancer, we will often delete the load balancer and create a new one. If possible, we should not do that: on GCE it may involve temporary service disruption, although the new LB has the same IP. On AWS a new ELB is created with a different DNS name, so the user must update their DNS settings. (We have a PR that fixes this , but it is post-v1; I am filing this bug to document the improvement needed, and then changing the tests to test the current state of the world)
How is the LB preserved? indicates it is NOT preserved, making this higher prio.
We do have an e2e test for this, but perhaps it isn't testing what I think it is. highlights that we definitely don't preserve it on AWS, but the test should be checking that it is preserved on GCE.