Question: workerControllerJson := mkpath("storm-worker-controller.json") nsFlag := fmt.Sprintf("--namespace=%v", ns) zookeeperPod := "zookeeper" <ins> nimbusPod := "nimbus" </ins> By("starting Zookeeper") framework.RunKubectlOrDie("create", "-f", zookeeperPodJson, nsFlag) framework.RunKubectlOrDie("create", "-f", zookeeperServiceJson, nsFlag) <del> err := framework.WaitForPodNameRunningInNamespace(c, zookeeperPod, ns) </del> <ins> err := f.WaitForPodRunningSlow(zookeeperPod) </ins> Expect(err).NotTo(HaveOccurred()) By("checking if zookeeper is up and running")

https://k8s- Failed: [] [Feature:Example] [] Storm should create and stop Zookeeper, Nimbus and Storm worker servers {Kubernetes e2e suite}
https://k8s- Failed: [] [Feature:Example] [] Storm should create and stop Zookeeper, Nimbus and Storm worker servers {Kubernetes e2e suite}
https://k8s- Failed: [] [Feature:Example] [] Storm should create and stop Zookeeper, Nimbus and Storm worker servers {Kubernetes e2e suite}
https://k8s- Failed: [] [Feature:Example] [] Storm should create and stop Zookeeper, Nimbus and Storm worker servers {Kubernetes e2e suite}
https://k8s- Failed: [] [Feature:Example] [] Storm should create and stop Zookeeper, Nimbus and Storm worker servers {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention...
This has been marked with the "needs attention" label and needs to be addressed for the 1.4 release. Could you please close if it's fixed update on your progress to fix it make a case for it not to block the release or find a new owner who can do one of the above soon. Once you or the new owner update this issue and are actively working on it, you can remove the "needs attention" label.
the following message looks suspicious, looks a DNS issue?
Actually more like a network connectivity issue
do you happen to have the Dockerfile for these images? We better push them into
I don't think this is a blocking issue. It looks a transient issue. Since the example is largely dependent on whether the image can be fetched and how long it takes to pull the image, it is very environment sensitive. I would recommend removing label. Longer term, we should move the images used in examples to gcr.
https://k8s- Failed: [] [Feature:Example] [] Storm should create and stop Zookeeper, Nimbus and Storm worker servers {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.

for the last flake, based on , image pull (mattf/zookeeper:latest) started at 42:40, finished at 48:31, about 5min51sec, well passed 5 min
storm-nimbus, storm-worker and zookeeper are from they're all automated builds from github (quick link: and )
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] [Feature:Example] [] Storm should create and stop Zookeeper, Nimbus and Storm worker servers {Kubernetes e2e suite}


Question: "type": "array" } }, <del> "required": [ "acceptedNames", "storedVersions" ], </del> "type": "object" }, "io.k8s.apiextensions-apiserver.pkg.apis.apiextensions.v1beta1.CustomResourceDefinitionVersion": {

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Currently, the field on must be explicitly set to their zero-values due to the fact that they are not tagged . Ref: Why is this needed: If using a system that automatically compares the provided configuration to the current, it will produce a diff on the updated status after creation. In addition, making sure to explicitly set to zero-values is not the best user experience. I am happy to provide the fix if this change is approved! /cc /sig api-machinery /area custom-resources
marking them as seems ok, so that client-side validation doesn't complain if they are absent when submitting manifests, but they shouldn't need to be set to omitempty (and at this point, we likely wouldn't change the serialization) Those fields are changed by controllers after creation, so expecting the live API object to exactly match the status is not going to go well anyway.
Good point! I’ll open a fix for this. Yep! The specific system in question ignores fields that are not set at creation time by default when doing comparison, so if status fields are not included then it will not report a diff.
/assign
I opened a fix for this but CC'ed both you and Let me know if any additional effort is required. Thanks!


Question: } func Convert_autoscaling_PodsMetricSource_To_v2beta1_PodsMetricSource(in *autoscaling.PodsMetricSource, out *autoscalingv2beta1.PodsMetricSource, s conversion.Scope) error { <del> targetAverageValue := *in.Target.AverageValue out.TargetAverageValue = targetAverageValue </del> <ins> if in.Target.AverageValue != nil { targetAverageValue := *in.Target.AverageValue out.TargetAverageValue = targetAverageValue } </ins> out.MetricName = in.Metric.Name out.Selector = in.Metric.Selector

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: The cluster started giving a lot of NODE DENY error in the log corresponding to restarting kube-apiserver make another one crash in a 3 apiserver setup. (say systemctl restart kube-apiserver on controller-1 will crash controller 2 or 3) It's as if node were show as but were not registered anymore?! What you expected to happen: The cluster to work as it did the weeks before How to reproduce it (as minimally and precisely as possible): I haven't fixed this yet and pod are stock in pending most of the time even after restarting controllers, workers, etc. Anything else we need to know?: Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-10-24T06:54:59Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-10-24T06:43:59Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: 10 workers with 64GB on 3 controllers VM. OS (e.g. from /etc/os-release): NAME="Ubuntu" VERSION="16.04.5 LTS (Xenial Xerus)" ID=ubuntu IDLIKE=debian PRETTYNAME="Ubuntu 16.04.5 LTS" VERSIONID="16.04" HOMEURL="" SUPPORTURL="" BUGREPORTURL="" VERSIONCODENAME=xenial UBUNTUCODENAME=xenial Kernel (e.g.

): Linux staging-controller-002 4.4.0-139-generic -Ubuntu SMP Wed Oct 24 10:58:50 UTC 2018 x8664 x8664 x86_64 GNU/Linux Install tools: ansible Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
/sig api-machinery
/cc seems that it's the graph in node authz is not ready to hande requests. probably respecting infomer's func can make it initialized gracefully. while the log you provides didn't show why the apiserver's crushed. could you provide more detail?
if this occurred right at server start, it is expected the graph will take a short amount of time to populate
If you tell me what more info you need, I'll be glad to provide you with it, maybe the apiserver flags used? Also, I restart the one crash-dumping, one of the other two will start outputting the same dump. Looking back at the logs, this seems to have started when I moved from 1.10.3 to 1.12.1 (now 1.12.2), I just hadn't realized it until recently as we had a big cluster wide problem yesterday and had to investigate some more. But it is still outputting this same dump non-stop even though it seems to be working at the moment and that API server does respond to kubectl commands without issues. Any specific info you'd like me to get?
from the stack trace, sadly we can't find out which exact line trigger that nil pointer panic. the behavior of apiserver could be unstable on the fresh start (due to rbac authz, crd-apiservice registration, etc). are we planing a graceful start mechanism for the server?
the apiserver endpoint is used to determine if the server is ready to accept traffic, and will not return a 200 response until those startup tasks have completed
/healthz does respond (200), while still crash-dump-looping. EDIT: anything I can test specifically to help differentiate between RBAC authz, crd-apiservice reg, etc?
right, I was saying that is the mechanism for graceful startup. unfortunately, the way golang works, if you propagate a panic (), you lose all stack information from it by default. Some of our panic propagation logs or preserves that, but not all.

I'll open a fix to preserve that information. we can see the request that errored, though: can you see if the same resource (autoscaling/v2beta1/horizontalpodautoscalers) is what causes the panic each time? if so, can you check that resource per namespace to isolate a particular problematic object? those are not related to the panic, as far as I can tell
Still reading, but: Did you skip 1.11? Skipping releases isn't actually supported.
Right now, having been up for a while, it alternates almost perfectly (E114 sometime coming twice in a row) between E114 and I114, basically once per second. These two errors correspond to the one you quoted (HPA) & the other to the HTTP2 panic. The cluster was upgraded from 1.10.3 to 1.12.1 (to 1.12.2) and from what I could gather last week, this started after the upgrade to 1.12.x branch but we didn't realize it until we had to investigate a cluster issue 3 weeks after the upgrade.
Yes, I did, I didn't know that... should I roll back to 1.11 and than back again to 1.12? Or what are the effects and what can be done?
There's at least two things going wrong, one is: the other is whatever is actually crashing. At this point I'm not sure if rolling back to 1.11 would make things worse or better, if your cluster is functioning I probably wouldn't do; if your cluster is nonfunctional and you need it fixed then I might roll the dice. I think we should be able to get a fix for the first thing out soon, but it'll be a while before you'd be able to deploy it unless you're willing to build your own apiserver & run it.
So, any way to diagnose what should have been updated in, I guess, etcd? I mean, is this similar to a schema change that is missing some steps? Thanks!
callstack logging fixed in and
Any chance this will also get merged in 1.12.3 so I could get a meaningful crash dump with the fix?

:)
sure, this will get into 1.12.3 via
So here is the new dump I've got under 1.12.3: Does it make more sense to anyone and is it pointing to the reason for the stack dump?
apologise, it didn't get into 1.12.3. (i thought it would). if you don't feel like waiting until next release, how about building ur own apiserver from release-1.12 HEAD as suggests? (run under this project)
Oh well, I can wait for 1.12.4... (Building it didn't work out and I'd have to build a custom docker, etc. I'll wait for the next release...)
We have the same issue and we reproduced with new kubernetes api-server Stacktrace:
thx for providing the clue! fixed via


Question: "type": "string" }, "volumeAttributesClassName": { <del> "description": "volumeAttributesClassName may be used to set the VolumeAttributesClass used by this claim. If specified, the CSI driver will create or update the volume with the attributes defined in the corresponding VolumeAttributesClass. This has a different purpose than storageClassName, it can be changed after the claim is created. An empty string value means that no VolumeAttributesClass will be applied to the claim but it's not allowed to reset this field to empty string once it is set. If unspecified and the PersistentVolumeClaim is unbound, the default VolumeAttributesClass will be set by the persistentvolume controller if it exists. If the resource referred to by volumeAttributesClass does not exist, this PersistentVolumeClaim will be set to a Pending state, as reflected by the modifyVolumeStatus field, until such as a resource exists. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#volumeattributesclass (Alpha) Using this field requires the VolumeAttributesClass feature gate to be enabled.", </del> <ins> "description": "volumeAttributesClassName may be used to set the VolumeAttributesClass used by this claim. If specified, the CSI driver will create or update the volume with the attributes defined in the corresponding VolumeAttributesClass. This has a different purpose than storageClassName, it can be changed after the claim is created. An empty string value means that no VolumeAttributesClass will be applied to the claim but it's not allowed to reset this field to empty string once it is set. If unspecified and the PersistentVolumeClaim is unbound, the default VolumeAttributesClass will be set by the persistentvolume controller if it exists. If the resource referred to by volumeAttributesClass does not exist, this PersistentVolumeClaim will be set to a Pending state, as reflected by the modifyVolumeStatus field, until such as a resource exists. More info: https://kubernetes.io/docs/concepts/storage/volume-attributes-classes/ (Alpha) Using this field requires the VolumeAttributesClass feature gate to be enabled.", </ins> "type": "string" }, "volumeMode": {

in link is out of date. the new one is fix link none No response <details</details<detailsnone </details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage /kind documentation /remove-kind bug


Question: // We seem to need to do additional waiting until the extension api service is actually up. err = wait.Poll(100*time.Millisecond, 30*time.Second, func() (bool, error) { <del> request := client.Discovery().RESTClient().Get().AbsPath("/apis/wardle.k8s.io/v1alpha1/namespaces/default/flunders") </del> <ins> request := restClient.Get().AbsPath("/apis/wardle.k8s.io/v1alpha1/namespaces/default/flunders") </ins> request.SetHeader("Accept", "application/json") _, err := request.DoRaw() if err != nil {

1 tests failed, 11 jobs failed, 241 builds failed. Failure stats cover 1 day time range '17 Aug 2017 22:57 UTC' to '18 Aug 2017 22:57 UTC'.
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
Seeing this failure on the e2e release-master-blocking https://k8s- Example failure https://k8s- Looks like an RBAC issue cc
this is currently blocking the alpha.3 release
looks like this new test was by in which was merged about 6 days ago and is failing consistently
This has started failing again on our GKE test suite https://k8s- https://k8s- cc
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/api-machinery: Issue will be escalated to these SIGs if needed. priority/critical-urgent: Never automatically move out of a release milestone; continually escalate to contributor and SIG through all available channels. kind/bug: Fixes a bug discovered during the current release. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details
/assign
Seems to be flaking now with the error: https://k8s-
- we need to determine (today if possible) if this is truly release blocking. If so, please add the release-blocker label. And, if not, how do we best continue work on this for 1.8.x/1.9.0?
this test has two passes and four failures on the same commit I'm seeing gke-specific authz grants in that test that are incorrect: That commit removed the wait for the authz grant to become effective (which can lead to flakes), and granted superuser permissions to all users, which is incorrect and invalidates any other authz-related tests run in parallel with this test
cc
I can add the wait back.

edit: and fix the test given admin to all authenticated users.
Actually after staring at this test for about half an hour I can't figure out what different users exist or what permissions they're being granted. ClientSet, InternalClientset, and AggregatorClient are all initialized from the same config so I don't see how one would be able to create an RBAC binding but another would fail later. any thoughts here?
I honestly think the gke specific BindClusterRole is a red-herring. It is needed so the client has permission to perform one of the setup steps. (I think it was either to create the wardler cluster role or to bind that role to the anonymous user) Once that setup step is complete we no longer need the that cluster role bound and so I don't think its related.
The gke authorizer allows the “bind” verb, so the client can create a binding to the cluster-admin. It cannot create a role directly unless it has permissions via RBAC. Since we don’t have a way to determine the username associated with , binding to all authenticated users is what was done as a workaround.
I agree that the point at which the tests are failing indicate that the previous authorization issues are not the cause.
/open
/reopen
So a lot more information to work with now but the error is still occurring. I am still looking into this.
any update on the investigation?
Friendly v1.8 release team ping. This failure still seems to be happening, is this actively being worked? Does this need to be in the v1.8 milestone?


Question: match: __default_go_runner_version= - name: "k8s.gcr.io/pause" <del> version: 3.5 </del> <ins> version: 3.6 </ins> refPaths: - path: build/pause/Makefile match: TAG =

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --When including windows in a pause build I get a rate limit error as it tries to pull dockcross/windows-static-x64:latest. This causes the build to fail. This image should be stored and pulled from as there should be a copy of this in to mimic kube-cross for Linux images cd build/pause make all-push Kubernetes version : v1.20.1 Cloud provider or hardware configuration: N/A OS (e.g: ): Build running on Linux (includes cross compiling for windows) Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig windows
Are you either of you able help with getting this image into the k8s gcr or if there is a reason why it can't be put there??
context - this might be due to use pulling from a small # of vmware VIPs
goal from sig-windows ~ move this into GCR eventually ~ would that be ok ?
do you know any history for this image?
That image carried over from the original work from . I have asked sig-testing previously if we could include that image's functionality into ` into the image. I can try this out in the morning, and if it works, I can send some PRs.
Seems to work. I have sent 2 PRs for this:


Question: "k8s.io/client-go/rest" ) <del> var scheme = runtime.NewScheme() </del> <ins> var configScheme = runtime.NewScheme() </ins> func init() { <del> apiserverapi.AddToScheme(scheme) apiserverapiv1alpha1.AddToScheme(scheme) </del> <ins> apiserverapi.AddToScheme(configScheme) apiserverapiv1alpha1.AddToScheme(configScheme) </ins> } // AdmissionOptions holds the admission options

/kind bug What happened: When I want to enable both plugins by passing below flag, as they both need config file, and I find I donot know what to set If I set , then I get an error in apiserver
/kind bug /sig auth
/cc
/cc
first, you can't put comments in json. the error was complaining about your embedded comments. this would let the server start: second, the docs are really, really, not right (opened with fixes). if you want to specify config for multiple plugins, should point to a config that references individual config files for each plugin (which there's a bug in loading... fixed in ): example
/assign


Question: ARCH?=amd64 HYPERKUBE_BIN?=_output/dockerized/bin/linux/$(ARCH)/hyperkube <del> BASEIMAGE=gcr.io/google-containers/debian-hyperkube-base-$(ARCH):0.7 </del> <ins> BASEIMAGE=gcr.io/google-containers/debian-hyperkube-base-$(ARCH):0.8 </ins> TEMP_DIR:=$(shell mktemp -d -t hyperkubeXXXXXX) all: build

the new hyperkube image has changed without notifying the bash environment now links to sh, there is only one thing worst than an error, and it is a SILENT ERROR this means that anyone that execute kubectl inside the hyperkube and leverage from a bash environment now get the weirdest exceptions , in my case troubles where caused by flexvolume scripted in bash which stopped working without reason and with weird errors logged ( but also there are some other cluster-images based on your hyperkube image, and if you link bash to sh it will spread headache around :) ) while for version 1.7.7 the result was different so , or remove bash, or repristinate bash, link bash-sh is a NO-NO
/assign /sig node


Question: return false, err } return true, nil <del> }) </del> <ins> }, "Waited %s for the sample-apiserver to be ready to handle requests.") </ins> if err != nil { currentAPIServiceJSON, _ := json.Marshal(currentAPIService) framework.Logf("current APIService: %s", string(currentAPIServiceJSON))

[sig-api-machinery] Aggregator Should be able to support the 1.7 Sample API Server using the current Aggregator More nuanced logs and details can be found at gave up waiting for apiservice wardle to come up successfully Expected error: <errors.errorString | 0xc420131150: { s: "timed out waiting for the condition", } timed out waiting for the condition not to have occurred /priority failing-test /priority important-soon /kind flake /sig-api-machinery /assign can you please help triage this issue?
has made changes that could have affected this recently. made one that'd be even more suspicious but I don't think it has merged yet :) :)
flakes show steady 2-3/day over the past week: which changes are you referring to? the line that's failing isn't doing any discovery, restmapping, etc, at all... just a straight get to the API:
audit log from the test run shows the requests are returning 404s, not 503s
ah, sorry, I misread the log and thought it was a discovery error. Hm.
hmm, there's a second smaller set of failures that are discovery related: looks like those started flaking after which went in on 5/4 looks like some discovery paths had internal retries that could have masked those 404 errors before. since the error that test is encountering is for an unrelated CRD API that is being created/deleted as part of another test, it makes more sense to switch that check to just ensure the extension API group under test is discovered, rather than assert there are no errors looking for other API groups that we know are being dynamically /removed opened to deflake the aggregator e2e check
/assign resolved the discovery issue. now the failure is the one noted in where calls to the extension API are responding with 404 errors
Kubelet log is reporting that the sample-apiserver exited with a return code of 2 and it also shows with a high restart count for the sample-apiserver.

When I extract the sample-apiserver log I'm seeing multiple "panic: Get https://10.0.0.1:443/api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentication: dial tcp 10.0.0.1:443: getsockopt: connection refused" Both failures today are in ipvs clusters which seems to have extensive networking problems. /assign
We are looking into finding the root cause into the problems with the ipvs clusters. I am pretty confident they were triggered by a bad PR, just need to identify which one.
Initial hypothesis is that PR caused it.
We think should fix the IPVS issues.
I see merged and this test is now passing in both pull-kubernetes-e2e-gce and ci-kubernetes-e2e-gci-gce. Hence closing the issue. Thanks all
/reopen The test is failing for the same error again in the following jobs https://k8s- It is one of the top flake from last week failing 47 jobs.
can you please take a look and triage? Thanks
this flake is now blocking this test from being promoted to Conformance suite in 1.11. Is it something we can investigate and resolve soon (today or tomorrow) to increase changes of getting this into 1.11? If not this test has to wait until 1.12 to be promoted.
all I see in the test run logs is 404 errors coming back from that check. can you see anything useful in the container logs for the aggregated server?
/cc
Update: I have been trying to reproduce this failure on my own test cluster (go run ...) and it has passed 128 times in a row so far. I think this means that the issue must be with either an interaction between e2e tests or an issue with the job running the tests.
Looking through recent results this crops up as a flake in presubmit and CI across many configurations... I don't think how we run the test jobs should be affecting this in the slightest. Not sure what's wrong here, but it looks like a pretty typical flakey test (some race / buggy setup/teardown?) nb that may not trivially map to the same way the tests are run though, and itself is a wrapper over
does the API server logs throw any light on whats happening on CI test cluster.

The teardown for the aggregator test is wrong, it tries to delete the deployment "sample-apiserver" when the deployment is actually named "sample-apiserver-deployment", but I don't think that's the issue. It's been like that since the test was . I'll look for conflicts with other tests.
We should probably go ahead and fix that anyway, it might be stomping on another test... Jenny Buckley <:
The namespace controller polls each of the api groups periodically, and it gets this internal error back from the endpoint a couple of times. Maybe related?
unclear... I opened to capture the state of the APIService, extension server pods, and those pod logs in failures cases
It flaked again since your PR merged https://k8s-
All I'm seeing is
I don't think that's related... I see 10-30 of those same rejections in apiserver logs of successful test runs (for example, search for ). I don't see anything in the test setup that grants the sample apiserver permission to run those checks, so it doesn't seem like that would be intermittent or flaky.
I am re-opening this issue since I still see intermittent failure of this test are your pending PRs ( and ) targeted towards these remaining issues? /reopen
this looks like 1.7 API Server image is causing more flakes other than just this test - The test has become lot less flaky after increasing the timeout. Since the official policy for GKE is 2 previous versions, and 1.11 is almost baked, this is the plan discussed with offline: leave the 1.7 test as it is right now create a similar test to the 1.7, but with 1.9 (implies creating a new sample-sever, uploading image, and creating the test). get that new test PR merged and running let it soak for several days, see if in 1.7 - 1.9 the flakiness (latency or whatever is causing it) went away.

if it did, evaluate removing the 1.7 and using the 1.9 let us know what you think of this approach
I think you misunderstood my comment... that only related to kubectl being used against a 1.7-level kube-apiserver serving discovery docs for the v1 API, not a generic 1.7-level extension API server
unrelatedly, moving the test to 1.9-level code once 1.11 is released seems reasonable to me
Thank you Jordan, that's the current plan! We will actually create a parallel 1.9-level test, to see if between 1.7 and 1.9 the internals of API Server fixed what's generating the flakiness and see next steps from there.
Sorry I didn't read the previous discussion close enough. The idea of creating a 1.9 test makes sense.
Pinging to see what are the next steps. Who is taking point to create a parallel test?
last I chated with someone on his team will build 1.9 API server image and push it to GCR. Once that image is available then can help writing the parallel test. I am not sure if the image step is done.
It's on the works. Buiding the sample-image showed to be more complicated than what it seemed to be. If it becomes a blocker let me know. Aish Sundar <:
I'd like to know where we are on this, because the 1.7 thing has come up in the context of multi-arch image builds
and
Hi Aaron and Aish, sorry for the delay. The process to compile, create and upload the sample API Server was not properly documented / maintained, and that plus more burning things is what made it take so long. We are trying to finally have it back (and documented by end of this week). Will keep you posted.
Image created with all the refactor happened on that area, trying upgrading the test at present from 1.7 to 1.9
Update: Upgraded the test, and now the test is not passing... working on this.

• Failure [38.067 seconds] [sig-api-machinery] Aggregator Should be able to support the 1.9 Sample API Server using the current Aggregator [It] creating a new flunders resource Expected error: <errors.StatusError | 0xc420c90cf0: { ErrStatus: { TypeMeta: {Kind: "", APIVersion: ""}, ListMeta: {SelfLink: "", ResourceVersion: "", Continue: ""}, Status: "Failure", Message: " "rest-flunder-" is forbidden: not yet ready to handle request", Reason: "Forbidden", Details: { Name: "rest-flunder-", Group: "", Kind: "flunders", UID: "", Causes: nil, RetryAfterSeconds: 0, }, Code: 403, }, } "rest-flunder-" is forbidden: not yet ready to handle request not to have occurred
not sure if it's related, but I don't think the test currently gives sufficient permissions to the aggregated server to perform delegated authn/authz checks. we might want to look at merging
tried your fix, same result. The image of the sample-apiserver 1.9 based that I've created is uploaded to If someone want's to give it a try, simply by upgrading 1.0 to 1.1 here:
/assign Can you try to figure out why the test is failing w/ the new image?
/cc
Now waiting on
Is there an ETA on ? This test is flaking quite a bit offlate - https://k8s-
upgrades to 1.10. however it requires more rbac permissions than before which is not fully understood. We will either resolve the permission issue or leave it for later investigation. The upgrade to 1.10 should be merged in a couple of days, if not today.
what issue are you seeing? is relevant?
Yes, it is relevant. It does not fix the issue by itself though.
As for should be ready to be merged.
/reopen I would like some manual confirmation this has been addressed
Reopening this issue. <detailsIn response to : /reopen I would like some manual confirmation this has been addressed Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.

</details
after some in-depth investigations, it turns out the 1.10 sample-apiserver would need the following rbac permissions (in addition to the system:auth-delegator role). I'm updating the PR to add these. apiVersion: kind: ClusterRole metadata: name: sample-apiserver rules: apiGroups: resources: '*' verbs: list apiGroups: '' resources: namespaces verbs: list
This test is failing again, FYI: https://k8s-
Hi Josh, TL;DR: we are upgrading to 1.10 and that's what people is talking about in the previous comments.
/reopen I would like to see the test pass with the fix before closing this issue out. thanks
Reopening this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hi please notice the test was renamed to: [sig-api-machinery] Aggregator Should be able to support the 1.10 Sample API Server using the current Aggregator, and has been passing since merged? https://k8s-
Ah cool thanks that explains it :) I was looking at in release-master-blocking and looks like it hasn't picked by the latest test since its 10/13 run. Closing this issue now based on the api-machinery dashboard now. Will reopen if the release job doesnt turn green the next run. Thanks much.
Is it worth removing the copy of this test that lives in ? Or is it still a release-blocking test?
I think the test will automatically pick the new version of the test in its next run (which should be sometime today, 10/20?). As to whether the test should be in the scale-correctness job , it is a ques for and


Question: echo "Download kubernetes release v${K8S_VERSION} ..." curl -L ${K8S_DOWNLOAD_URL} -o ${RELEASES_DIR}/kubernetes.tar.gz <del> echo "Download docker-latest ..." curl -L https://get.docker.com/builds/Linux/x86_64/docker-latest -o ${RELEASES_DIR}/docker </del> <ins> echo "Download docker release v${DOCKER_VERSION} ..." curl -L ${DOCKER_DOWNLOAD_URL} -o ${RELEASES_DIR}/docker.tar.gz </ins> } function unpack-releases() {

Did a git clone and trying to build on centos. [root kubernetes]# cat /etc/redhat-release CentOS Linux release 7.2.1511 (Core) [root kubernetes]# make release +++ [0616 17:42:23] Verifying Prerequisites.... +++ [0616 17:42:57] Building Docker image kube-build:build-. +++ [0616 17:44:44] Running build command.... Usage of loopback devices is strongly discouraged for production use. Either use or use to suppress this warning. Go version: go version go1.6.2 linux/amd64 +++ [0616 17:45:01] Multiple platforms requested, but available 0G < threshold 11G, building platforms in serial +++ [0616 17:45:01] Building the toolchain targets: +++ [0616 17:45:04] Building go targets for linux/amd64: cmd/kube-dns cmd/kube-proxy cmd/kube-apiserver cmd/kube-controller-manager cmd/kubelet cmd/kubemark cmd/hyperkube federation/cmd/federation-apiserver federation/cmd/federation-controller-manager plugin/cmd/kube-scheduler go build /usr/local/go/pkg/tool/linuxamd64/compile: signal: killed !!! Error in 'go install "{goflags[ -ldflags "{nonstatics[ exited with status 1 Call stack: 1: kube::golang::buildbinariesforplatform(...) 2: kube::golang::buildbinaries(...) 3: hack/build- main(...) Exiting with status 1 !!! Error in '( kube::golang::setupenv; echo "Go version: (kube::golang::hostplatform); local goflags goldflags; eval "goflags=({KUBEGOLDFLAGS:-} do if [[ "${arg}" == "--usegobuild" ]]; then usegobuild=true; else if [[ "${arg}" == - ]]; then goflags+=("${arg}"); else targets+=("${arg}"); fi; fi; done; if [[ ${#targets[ -eq 0 ]]; then targets=("${KUBEALLTARGETS[ fi; local -a platforms=("{KUBEBUILDPLATFORMS[ if [[ ${#platforms[ -eq 0 ]]; then platforms=("${hostplatform}"); fi; local binaries;

binaries=({targets[ local parallel=false; if [[ ${#platforms[ -gt 1 ]]; then local gigs; gigs={gigs} -ge ${KUBEPARALLELBUILDMEMORY} ]]; then kube::log::status "Multiple platforms requested and available {KUBEPARALLELBUILDMEMORY}G, building platforms in parallel"; parallel=true; else kube::log::status "Multiple platforms requested, but available {KUBEPARALLELBUILDMEMORY}G, building platforms in serial"; parallel=false; fi; fi; kube::golang::buildkubetoolchain; if [[ "${parallel}" == "true" ]]; then kube::log::status "Building go targets for {targets[ local platform; for platform in "${platforms[ do ( kube::golang::setplatformenvs "{platform}: go build started"; kube::golang::buildbinariesforplatform {usegobuild:-}; kube::log::status "{platform////}.build" & done; local fails=0; for job in $(jobs -p); do wait ${job} let "fails+=1"; done; for platform in "${platforms[ do cat "/tmp//${platform////}.build"; done; exit ${fails}; else for platform in "${platforms[ do kube::log::status "Building go targets for {targets[ kube::golang::setplatformenvs "{platform} ${usegobuild:-}; done; fi )' exited with status 1 Call stack: 1: kube::golang::buildbinaries(.) 2: hack/build- main(.) Exiting with status 1 !!! Error in '" exited with status 1 Call stack: 1: kube::build::runbuildcommand(...) 2: main(...) Exiting with status 1 make:  [release] Error 1
Note that Centos downloads a bad "docker" binary that is an image with "404 Error". Centos seems abandonned by Kubernetes team... so I needed to make the entire operation with Ansible. And that is not simple at all :)
There was below info from your log: It seems like that the compiler was OOM killed due to memory pressure. AFAIK, consumes a lot of memory. I used to encounter when running it. We're going to close this issue.

Feel free to file a new issue if you fail to run at latest HEAD and you think there is a bug. Thanks!


<!-- Please only use this template for submitting enhancement requests --What would you like to be : let's say I run If the deployment is comes up in time it tells me which deployment is up, but if it times out it doesn't tell me on which deployment/resource. Could it be Why is this needed: That would make it easier to use, and the information provided is the same irregardless on whether the condition is met or not
/sig cli
Sounds reasonable for me, i can take care of this one if someone from owners will approve it as a feature.
/assign Zyqsempai
GitHub didn't allow me to assign the following users: Zyqsempai. Note that only and repo collaborators can be assigned and that issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Zyqsempai Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: }, }, { <del> name: "add a pod with mis-matched namespace doesn't change topologyKeyToMinPodsMap", </del> <ins> name: "add a pod in a different namespace doesn't change topologyKeyToMinPodsMap", </ins> preemptor: st.MakePod().Name("p").Label("foo", ""). SpreadConstraint(1, "node", hardSpread, st.MakeLabelSelector().Exists("foo").Obj()). Obj(),

How to reproduce it (as minimally and precisely as possible): pre-requisites: a multi-az cluster, a deployment with 1 pod in each az. a with and . the node(s) in 1 az. the deployment's pod(s) from the cordoned az. What happened: A new pod was created and scheduled in a form that What you expected to happen: I expected that a new pod will be created and be "stuck" in status as it shouldn't have nowhere be scheduled. Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: AWS OS (e.g: ):Kernel (e.g. ): 4.19.84-coreos Others: I created a with how I reproduce the bug
/sig scheduling
cc/
Thanks for raising this issue. I feel like it's a general timing issue - when you delete a Pod, that Pod may still exist for a very short time (in graceful deletion period), and at the same time, the controller backfills a Pod for it. So when scheduling tries to schedule the backfill Pod, it may read the cluster as: [1, 1, 1(deleting)]. To confirm this, can you try deleting the Pod using , and see if the problem goes away?
Yes sure I will try that. It sounds reasonable, thats why in the first demonstration I deleted all the pods and waited for all of them to completely disappear. Nevertheless, this is still something that I believe needs to be fixed. The pod's evaluation should take in consideration pods that are in status and don't count them as pods.
I do believe we exclude the Pods. (if not, it should be a serious regression). What concerns me is that at that moment, scheduler may treats that (deleting) Pod as running in its cache. I will try to debug into this.
Yep, this time the new pod was in status until I uncordoned the relevant nodes.


Question: MutatingKind = "mutating" ) <del> func init() { // Register all metrics. prometheus.MustRegister(requestCounter) prometheus.MustRegister(longRunningRequestGauge) prometheus.MustRegister(requestLatencies) prometheus.MustRegister(requestLatenciesSummary) prometheus.MustRegister(responseSizes) prometheus.MustRegister(DroppedRequests) prometheus.MustRegister(currentInflightRequests) </del> <ins> var registerMetrics sync.Once // Register all metrics. func Register() { registerMetrics.Do(func() { for _, metric := range metrics { prometheus.MustRegister(metric) } }) } // Reset all metrics. func Reset() { for _, metric := range metrics { metric.Reset() } </ins> } func UpdateInflightRequestMetrics(nonmutating, mutating int) {

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: endpoint of different k8s components expose Prometheus metrics that are not for them. For example, exposes metrics: or : What you expected to happen: Prometheus metrics only for to appear on kubelet's endpoint. How to reproduce it (as minimally and precisely as possible): Access the endpoint of all Kubernetes components Anything else we need to know?: This problem comes from usage of functions in different packages that automatically register Prometheus metrics. Just importing that package or some package that depends on it can cause those metrics to appear in unexpected places. In the list bellow you can find the different packages that have init functions and affected binaries: [ ] () cloud-controller-manager hyperkube kube-apiserver kube-controller-manager kube-proxy kube-scheduler [ ] () cloud-controller-manager hyperkube kube-apiserver kube-controller-manager kube-proxy kube-scheduler [ ] cloud-controller-manager hyperkube kube-apiserver kube-controller-manager [ ] cloud-controller-manager hyperkube kube-apiserver kube-controller-manager kubelet [ ] cloud-controller-manager hyperkube kube-apiserver kube-controller-manager kube-proxy kube-scheduler [ ] cloud-controller-manager hyperkube kube-apiserver kube-controller-manager kube-proxy kube-scheduler kubectl kubelet [ ] cloud-controller-manager hyperkube kube-apiserver kube-controller-manager kube-proxy kube-scheduler kubelet [ ] hyperkube kube-apiserver kube-controller-manager [ ] hyperkube kube-apiserver [ ] cloud-controller-manager hyperkube kube-apiserver kube-controller-manager kubelet [ ] cloud-controller-manager hyperkube kube-apiserver kube-controller-manager kube-proxy kube-scheduler kubelet Environment: Kubernetes version (use ):Cloud provider or hardware configuration: N/A OS (e.g. from /etc/os-release): N/A Kernel (e.g. ): N/A Install tools: N/A Others: N/A /sig instrumentation
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: It("Should be able to support the 1.7 Sample API Server using the current Aggregator", func() { // Make sure the relevant provider supports Agggregator <ins> framework.SkipUnlessServerVersionGTE(serverAggregatorVersion, f.ClientSet.Discovery()) </ins> framework.SkipUnlessProviderIs("gce", "gke") // Testing a 1.7 version of the sample-apiserver

1 tests failed, 11 jobs failed, 241 builds failed. Failure stats cover 1 day time range '17 Aug 2017 22:57 UTC' to '18 Aug 2017 22:57 UTC'.
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
Seeing this failure on the e2e release-master-blocking https://k8s- Example failure https://k8s- Looks like an RBAC issue cc
this is currently blocking the alpha.3 release
looks like this new test was by in which was merged about 6 days ago and is failing consistently
This has started failing again on our GKE test suite https://k8s- https://k8s- cc
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/api-machinery: Issue will be escalated to these SIGs if needed. priority/critical-urgent: Never automatically move out of a release milestone; continually escalate to contributor and SIG through all available channels. kind/bug: Fixes a bug discovered during the current release. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details
/assign
Seems to be flaking now with the error: https://k8s-
- we need to determine (today if possible) if this is truly release blocking. If so, please add the release-blocker label. And, if not, how do we best continue work on this for 1.8.x/1.9.0?
this test has two passes and four failures on the same commit I'm seeing gke-specific authz grants in that test that are incorrect: That commit removed the wait for the authz grant to become effective (which can lead to flakes), and granted superuser permissions to all users, which is incorrect and invalidates any other authz-related tests run in parallel with this test
cc
I can add the wait back.

edit: and fix the test given admin to all authenticated users.
Actually after staring at this test for about half an hour I can't figure out what different users exist or what permissions they're being granted. ClientSet, InternalClientset, and AggregatorClient are all initialized from the same config so I don't see how one would be able to create an RBAC binding but another would fail later. any thoughts here?
I honestly think the gke specific BindClusterRole is a red-herring. It is needed so the client has permission to perform one of the setup steps. (I think it was either to create the wardler cluster role or to bind that role to the anonymous user) Once that setup step is complete we no longer need the that cluster role bound and so I don't think its related.
The gke authorizer allows the “bind” verb, so the client can create a binding to the cluster-admin. It cannot create a role directly unless it has permissions via RBAC. Since we don’t have a way to determine the username associated with , binding to all authenticated users is what was done as a workaround.
I agree that the point at which the tests are failing indicate that the previous authorization issues are not the cause.
/open
/reopen
So a lot more information to work with now but the error is still occurring. I am still looking into this.
any update on the investigation?
Friendly v1.8 release team ping. This failure still seems to be happening, is this actively being worked? Does this need to be in the v1.8 milestone?


Question: if utilfeature.DefaultFeatureGate.Enabled(features.CSINodeInfo) && utilfeature.DefaultFeatureGate.Enabled(features.CSIMigration) { <del> // This function prevents Kubelet from posting Ready status until CSINodeInfo </del> <ins> // This function prevents Kubelet from posting Ready status until CSINode </ins> // is both installed and initialized if err := initializeCSINode(host); err != nil { <del> return errors.New(log("failed to initialize CSINodeInfo: %v", err)) </del> <ins> return errors.New(log("failed to initialize CSINode: %v", err)) </ins> } }

What happened: Upgraded from 1.16.6 to 1.17.2 and will no longer start on control-plane hosts. It fails with: What you expected to happen: to start, static pods to launch, to connect to in the static pod How to reproduce it (as minimally and precisely as possible): Have the , , , etc in the static pod manifest folder. Try to start the with the container images not preloaded on the host. Just in case it is needed: is used as the backend is run using Anything else we need to know?: This looks to be two things interacting poorly: 1- It takes a long time for static pods to be created for the first time when the can't talk to yet. 2- The new CSI migration code force terminates causing these log lines to appear in : It looks like introduced this new fatal error. Environment: Kubernetes version (use ): 1.17.2 Cloud provider or hardware configuration: Bare-metal OS (e.g: ): Fedora 31..1 (CoreOS preview)" Kernel (e.g. ): 5.3.12-300.fc31.x86_64 Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig cluster-lifecycle
Actually, probably: /sig node
The referenced PR was in 1.14, so that interaction is not new. What is likely new is the size increase of hyperkube in 1.17 (xref ). When not pre-pulled, this can increase the time it takes to start hyperkube for the first time. Does the image pull make progress as the kubelet restarts/retries?
It might make sense for the initializeCSINode() method to: first wait to successfully talk to the kube apiserver at all, perhaps indefinitely after successfully talking to the kube apiserver, then try to register the CSINode object with backoff, and fail if it cannot do that successfully
/sig storage /remove-sig cluster-lifecycle
It doesn't seem to. Once the connection to containerd is closed, it cancels the context. Otherwise I would expect this to succeed eventually if the pull happened in the background. That sounds reasonable to me assuming it isn't blocking the static pod creation code. Do you happen to know why it takes a very long time for the static pod creation code to be called?

I would expect the pods to be created very soon after the kubelet is started.
Pulling the image and starting the processes for those pods happens immediately (from the containerd logs, you can see the image pull happening... it just doesn't complete in time before the independent kubelet logic to register the CSI node info times out).
Ah, yes. I just upgraded a couple of days ago so I didn't check whether I still saw the 2 minute delay in creating the "pod sandboxes" (this is before the pull even happens). It now is only a 23 second delay: containerd: kubelet:
ack, I like the idea Will work on a fix today and cherry-pick it as necessary


Question: // RunningReady checks whether pod p's phase is running and it has a ready // condition of status true. func RunningReady(p *v1.Pod) bool { <del> return p.Status.Phase == v1.PodRunning && podutil.IsPodReady(p) </del> <ins> return p.Status.Phase == v1.PodRunning && podutils.IsPodReady(p) </ins> } // WaitForPodsRunning waits for a given `timeout` to evaluate if a certain amount of pods in given `ns` are running.

Several test/e2e/framework/* packages have as their only dependency on k/k. The following functions are shared that way: IsPodAvailable returns true if a pod is available; false otherwise. IsPodReady returns true if a pod is ready; false otherwise. VisitContainers invokes the visitor function with a pointer to every container spec in the given pod spec with type set in mask. If visitor returns false, visiting is short-circuited. VisitContainers returns true if visiting completes, false if visiting was short-circuited. AllFeatureEnabledContainers returns a ContainerType mask which includes all container types except for the ones guarded by feature gate. The feature gate check in AllFeatureEnabledContainers is problematic. E2E tests don't know what the feature gates in the cluster are. The rest of the functions look like something that could be moved into == Having the functions in makes it clearer that the checks only depend on aspects of the public API. Reduces dependencies of e2e/framework on k/k, in case that we want to move it into staging. Prevents accidentally using checks that only work in-tree (like feature gates).
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig testing
/assign


Question: } } <ins> if s.CloudProvider == kubeletconfigv1alpha1.AutoDetectCloudProvider { glog.Warning("--cloud-provider=auto-detect is deprecated. The desired cloud provider should be set explicitly") } </ins> if kubeDeps.Cloud == nil { if !cloudprovider.IsExternal(s.CloudProvider) && s.CloudProvider != kubeletconfigv1alpha1.AutoDetectCloudProvider { cloud, err := cloudprovider.InitCloudProvider(s.CloudProvider, s.CloudConfigFile)

/kind feature What happened: Continuing the discussion from , we should deprecate (or remove since its alpha) the option to set . See comment and for more details. related issues: related kubernetes-dev thread: !topic/kubernetes-dev/UAxwa2inbTA EDIT: Deprecation plan: [x] v1.8 - deprecate auto-detect cloud provider feature in kubelet [ ] v1.9 - set no cloud provider as the default in kubelet but still allow opt in for auto-detect [ ] v1.10 - completely remove the option to use
/sig cluster-lifecycle
/area cloudprovider
cc
/assign wlan0
cc - Wanted to ask you if you had any thoughts for/against this issue?
/assign andrewsykim As discussed in our meeting I'll be following up with sig-node to see if we can deprecate this as it's going against what we want for external cloud providers
FYI
Would have been nice to keep this issue open to track progress in the next releases
Indeed, should be kept open. Next task is now targeting v1.9
Thanks I edited the issue description to include the deprecation plan going forward. Open to changes on the deprecation plan if we think it's too aggressive.
Great, I think this is a good timeline. We need to make sure we communicate it very well though can you add this to the draft release notes in kubernetes/features please?
Looks like you beat me to it , gave it a review.
hehe, yes, it at the same time as everything else ;)
Can you create a PR for this soon to get this early into v1.9 please?
[MILESTONENOTIFIER] Milestone Issue Current <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Escalate to the issue owners; move out of the milestone after 1 attempt. : Adding tests, refactoring, fixing old bugs. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
i took the liberty to move this along (filed ). Hope that's ok.
Thanks appreciate it. I think there were some discussions in the mailing list thread about v1.9 being too soon.

I think it largely depends on how many users actually take advantage of the auto-detect feature. It's not many from what I've seen but we should follow up and be sure.
Any recent discussion about that? I haven't seen any activity recently but I might just be subject to email overflow. I hope we can get this into v1.9 cc
I didn't follow any discussion related to this topic, but thought this is right move to decouple cloudprovider components from the core system. If I remembered correctly, only cloudproviders for GCE, AWS and Azura depend on the feature, which can be easily override. Any concerns for v1.9?
yep, GCE/AWS/Azure -
Please look at and give your LGTM
Done. Just better document in 1.9 release notes. LGTM
Thank you!
Reopening for the removal of the option in v1.10
please move this to 1.10 milestone


Question: }) } } <ins> func customExit(exitCode int) { panic(strconv.Itoa(exitCode)) } func safeExtractKubeVersionTag(tagName string, t *types.Type) (value *tagValue, major int, minor int, err error, localErr error) { defer func() { if e := recover(); e != nil { localErr = fmt.Errorf("extractKubeVersionTag returned error: %v", e) } }() value, major, minor, err = extractKubeVersionTag(tagName, t) return } func safeExtractTag(t *testing.T, tagName string, comments []string) (value *tagValue, err error) { defer func() { if e := recover(); e != nil { err = fmt.Errorf("extractTag returned error: %v", e) } }() value = extractTag(tagName, comments) return } func Test_extractTag(t *testing.T) { oldKlogOsExit := klog.OsExit defer func() { klog.OsExit = oldKlogOsExit }() klog.OsExit = customExit comments := []string{ "+variable=7", "+anotherVariable=8", "+yetAnotherVariable=9", "variableWithoutMarker=10", "+variableWithoutValue", "+variable=11", "+multi-valuedVariable=12,13,14", "+strangeVariable=15,=16", } tests := []struct { name         string tagComments  []string variableName string wantError    bool wantValue    *tagValue }{ { name:         "variable with explicit value", tagComments:  comments, variableName: "anotherVariable", wantValue:    &tagValue{value: "8"}, }, { name:         "variable without explicit value", tagComments:  comments, variableName: "variableWithoutValue", wantValue:    &tagValue{value: ""}, }, { name:         "variable not present in comments", tagComments:  comments, variableName: "variableOutOfNowhere", wantValue:    nil, }, { name:         "variable without marker test", tagComments:  comments, variableName: "variableWithoutMarker", wantValue:    nil, }, { name:         "abort duplicated variable", tagComments:  comments, variableName: "variable", wantError:    true, wantValue:    nil, }, { name:         "abort variable with multiple values", tagComments:  comments, variableName: "multi-valuedVariable", wantError:    true, wantValue:    nil, }, { name:         "this test documents strange behaviour", // TODO: Is this behaviour intended? tagComments:  comments, variableName: "strangeVariable", wantValue:    &tagValue{value: "15"}, }, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { gotTag, err := safeExtractTag(t, tt.variableName, tt.tagComments) if (err != nil) != tt.wantError { t.Errorf("extractTag() err = %v, wantError = %v.", gotTag, tt.wantError) return } if tt.wantError { return } if !reflect.DeepEqual(gotTag, tt.wantValue) { t.Errorf("extractTag() got = %v, want %v", gotTag, tt.wantValue) } }) } } func Test_extractEnabledTypeTag(t *testing.T) { someComments := []string{ "+variable=7", "+k8s:prerelease-lifecycle-gen=8", } moreComments := []string{ "+yetAnotherVariable=9", "variableWithoutMarker=10", "+variableWithoutValue", "+variable=11", "+multi-valuedVariable=12,13,14", } tests := []struct { name      string mockType  *types.Type wantValue *tagValue }{ { name:      "desired info in main comments", mockType:  &types.Type{CommentLines: someComments, SecondClosestCommentLines: moreComments}, wantValue: &tagValue{value: "8"}, }, { name:      "secondary comments empty", mockType:  &types.Type{CommentLines: someComments}, wantValue: &tagValue{value: "8"}, }, { name:      "main comments empty", mockType:  &types.Type{SecondClosestCommentLines: someComments}, wantValue: &tagValue{value: "8"}, }, { name:      "lack of desired info, empty secondary comments", mockType:  &types.Type{CommentLines: moreComments}, wantValue: nil, }, { name:      "lack of desired info, empty main comments", mockType:  &types.Type{SecondClosestCommentLines: moreComments}, wantValue: nil, }, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { gotTag := extractEnabledTypeTag(tt.mockType) if !reflect.DeepEqual(gotTag, tt.wantValue) { t.Errorf("extractEnabledTypeTag() got = %v, want %v", gotTag, tt.wantValue) } }) } } func Test_extractReplacementTag(t *testing.T) { replacementTag := "+k8s:prerelease-lifecycle-gen:replacement" tests := []struct { name               string mainComments       []string secondaryComments  []string wantGroup          string wantVersion        string wantKind           string wantHasReplacement bool wantErr            bool }{ { name:               "no replacement tag", mainComments:       []string{"randomText=7"}, secondaryComments:  []string{"importantFlag=8.8.8.8"}, wantGroup:          "", wantVersion:        "", wantKind:           "", wantHasReplacement: false, wantErr:            false, }, { name:               "replacement tag correct", mainComments:       []string{fmt.Sprintf("%v=my_group,v1,KindOf", replacementTag)}, secondaryComments:  []string{}, wantGroup:          "my_group", wantVersion:        "v1", wantKind:           "KindOf", wantHasReplacement: true, wantErr:            false, }, { name:               "correct replacement tag in secondary comments", mainComments:       []string{}, secondaryComments:  []string{fmt.Sprintf("%v=my_group,v1,KindOf", replacementTag)}, wantGroup:          "my_group", wantVersion:        "v1", wantKind:           "KindOf", wantHasReplacement: true, wantErr:            false, }, { name:               "4 values instead of 3", mainComments:       []string{fmt.Sprintf("%v=my_group,v1,KindOf,subKind", replacementTag)}, secondaryComments:  []string{}, wantGroup:          "", wantVersion:        "", wantKind:           "", wantHasReplacement: false, wantErr:            true, }, { name:               "4 values instead of 3 in secondary comments", mainComments:       []string{}, secondaryComments:  []string{fmt.Sprintf("%v=my_group,v1,KindOf,subKind", replacementTag)}, wantGroup:          "", wantVersion:        "", wantKind:           "", wantHasReplacement: false, wantErr:            true, }, { name:               "2 values instead of 3", mainComments:       []string{fmt.Sprintf("%v=my_group,v1", replacementTag)}, secondaryComments:  []string{}, wantGroup:          "", wantVersion:        "", wantKind:           "", wantHasReplacement: false, wantErr:            true, }, { name:               "group name not all upper", mainComments:       []string{fmt.Sprintf("%v=myGroup,v1,KindOf", replacementTag)}, secondaryComments:  []string{}, wantGroup:          "", wantVersion:        "", wantKind:           "", wantHasReplacement: false, wantErr:            true, }, { name:               "version name does not start with v", mainComments:       []string{fmt.Sprintf("%v=my_group,bestVersion,KindOf", replacementTag)}, secondaryComments:  []string{}, wantGroup:          "", wantVersion:        "", wantKind:           "", wantHasReplacement: false, wantErr:            true, }, { name:               "kind name does not start with capital", mainComments:       []string{fmt.Sprintf("%v=my_group,v1,kindOf", replacementTag)}, secondaryComments:  []string{}, wantGroup:          "", wantVersion:        "", wantKind:           "", wantHasReplacement: false, wantErr:            true, }, { name:               "empty group name", // TODO: is it a valid input or a bug? mainComments:       []string{fmt.Sprintf("%v=,v1,KindOf", replacementTag)}, secondaryComments:  []string{}, wantGroup:          "", wantVersion:        "v1", wantKind:           "KindOf", wantHasReplacement: true, wantErr:            false, }, { name:               "empty version", mainComments:       []string{fmt.Sprintf("%v=my_group,,KindOf", replacementTag)}, secondaryComments:  []string{}, wantGroup:          "", wantVersion:        "", wantKind:           "", wantHasReplacement: false, wantErr:            true, }, { name:               "empty kind", mainComments:       []string{fmt.Sprintf("%v=my_group,v1,", replacementTag)}, secondaryComments:  []string{}, wantGroup:          "", wantVersion:        "", wantKind:           "", wantHasReplacement: false, wantErr:            true, }, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { replacementGroup, replacementVersion, replacementKind, hasReplacement, err := extractReplacementTag(&types.Type{ CommentLines:              tt.mainComments, SecondClosestCommentLines: tt.secondaryComments, }) if replacementGroup != tt.wantGroup { t.Errorf("extractReplacementTag() group got = %v, want %v", replacementGroup, tt.wantGroup) } if replacementVersion != tt.wantVersion { t.Errorf("extractReplacementTag() version got = %v, want %v", replacementVersion, tt.wantVersion) } if replacementKind != tt.wantKind { t.Errorf("extractReplacementTag() kind got = %v, want %v", replacementKind, tt.wantKind) } if hasReplacement != tt.wantHasReplacement { t.Errorf("extractReplacementTag() hasReplacement got = %v, want %v", hasReplacement, tt.wantHasReplacement) } if (err != nil) != tt.wantErr { t.Errorf("extractReplacementTag() err got = %v, want %v", err, tt.wantErr) } }) } } func Test_isAPIType(t *testing.T) { tests := []struct { name string t    *types.Type want bool }{ { name: "private name is not apitype", want: false, t: &types.Type{ Name: types.Name{ Name: "notpublic", }, }, }, { name: "non struct is not apitype", want: false, t: &types.Type{ Name: types.Name{ Name: "Public", }, Kind: types.Slice, }, }, { name: "contains member type", want: true, t: &types.Type{ Name: types.Name{ Name: "Public", }, Kind: types.Struct, Members: []types.Member{ { Embedded: true, Name:     "TypeMeta", }, }, }, }, { name: "contains no type", want: false, t: &types.Type{ Name: types.Name{ Name: "Public", }, Kind: types.Struct, }, }, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { if got := isAPIType(tt.t); got != tt.want { t.Errorf("isAPIType() = %v, want %v", got, tt.want) } }) } } </ins>

The commands under will benefit of a batter unit test coverage as seen in Originally posted by in
/sig api-machinery /assign
GitHub didn't allow me to assign the following users: damiansawicki. Note that only with read permissions, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /sig api-machinery Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage accepted
Hi I would like to work on this issue. Could you tell me how many test files are needed per PR under this issue?
Hi the issue is already assigned to me. Would you consider contributing to other issues to avoid duplication?
Hi do you mind if I work on this issue along with you? I would only like to write tests on limited number of files, if that's okay with you...
Hi thanks for your engagement! If you also want to work on this particular issue, we should coordinate. Are you working on something other than ?
Hey currently I'm thinking of shifting to Do let me know if you are already working on that particular directory. Also I could assist you in any of your work on this issue as I'm not preoccupied at the moment.


Question: fi } <ins> # Create a fake git repo for staging to prevent godeps from complaining pushd "${KUBE_ROOT}" >/dev/null git init >/dev/null git config --local user.email "nobody@k8s.io" git config --local user.name "$0" git add . >/dev/null git commit -q -m "Snapshot" >/dev/null popd >/dev/null </ins> # move into staging and save the dependencies for everything in order mkdir -p "${TMP_GOPATH}/src/k8s.io" for repo in $(ls ${KUBE_ROOT}/staging/src/k8s.io); do

/kind bug What happened: After k8s master started rsyncing directory, causing hundreds of megabytes to be transferred, making some workflows that utilize remote docker engines impractical. E.g. I use which can start docker-in-docker test cluster on the same remote docker where k8s is built, and currently my Internet connection can only give me about 1 Mb/s, so ~430Mb transfer caused by .git is impractical. What you expected to happen: is ignored as before, unless it's needed by the current script. In case is about to be ed, a warning needs to be printed. How to reproduce it (as minimally and precisely as possible): Just run . Doing so against a remote docker over slow(ish) Internet connection will make the behavior apparent. Anything else we need to know?: For now perhaps only script needs to rsync IIUC (see ). Environment: Kubernetes version (use ): I'll work on this issue.
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig contributor-experience /area build-release
/assign Feel free to ping for review when you have a PR :)


Question: Status: v1.NodeStatus{ Conditions: []v1.NodeCondition{ { <ins> Type:               v1.NodeOutOfDisk, Status:             v1.ConditionFalse, Reason:             "KubeletHasSufficientDisk", Message:            fmt.Sprintf("kubelet has sufficient disk space available"), LastHeartbeatTime:  metav1.Date(2012, 1, 1, 0, 0, 0, 0, time.UTC), LastTransitionTime: metav1.Date(2012, 1, 1, 0, 0, 0, 0, time.UTC), }, { </ins> Type:               v1.NodeMemoryPressure, Status:             v1.ConditionFalse, Reason:             "KubeletHasSufficientMemory",

Both https://k8s- and https://k8s- started failing recently. Their 1.7 counterparts are passing. The difference between the last passing and the first failing tests is: Probably something in broke the tests. /sig scheduling /cc /kind bug
It looks like there isn't enough cpu on the master to run the cos-nvidia-installer daemonset. Did we add any addons recently?
My previous comment is a red herring. It is an issue with . Ill post a fix soon. From the scheduler logs on the master:
that's a benign error. The daemonset is not expected to run on the master. But there is no easy way to express that request in our APIs.
Yes, the pod not scheduling on the master is benign. The actual error is that we do not report node condition , so it is left as . The fix is to report that is false.
The tests are still failing. /reopen
/assign
I will be updating to track progress.
tests are green again! /close


Question: loadBalancerName := c.GetLoadBalancerName(ctx, clusterName, apiService) serviceName := types.NamespacedName{Namespace: apiService.Namespace, Name: apiService.Name} <del> securityGroupIDs, err := c.buildELBSecurityGroupList(serviceName, loadBalancerName, annotations) </del> <ins> securityGroupIDs, setupSg, err := c.buildELBSecurityGroupList(serviceName, loadBalancerName, annotations) </ins> if err != nil { return nil, err }

What happened: Create SG outside of K8s - a SG rule has a range of ports which are opened via K8s Add this into Ensure load balancer switches from the range back to individual ports, the CIDR will be 0.0.0.0/0 if loadBalancerSourceRanges was removed too What you expected to happen: K8S doesn't modify the SG How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): v1.13.7 Cloud provider or hardware configuration: EKS OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig aws
Note - I'm not sure if this is related to the aws cloud config, which I can't change as I'm using EKS Add aws cloud config: [global] disableSecurityGroupIngress = true
/sig cloud-provider
We're also having this issue which makes kinda unusable. shouldn't ensure permissions if an SG is a custom one here . could you please take a look?
This is also an issue for us, using these service annotations should have been straightforward. A custom security group id used in the should ensure it's attached to the ELB/LB, and remain untouched. Instead, this customer security group gets purged, leaving the port mapping untouched, but all sources become 0.0.0.0/0.
unfortunately I can't, I don't have access at AWS resources to work on that. To be honest, it was pointed out a number of times that the load balancer code will need a refactor and better tests, I would suggest pinging AWS directly or bringing it up during the meetings.
Suppose you have an existing EKS cluster, gateways, the whole nine yards... Also, suppose you have a custom security group with entries already defined, for example, an SG that holds the rules of your corporate office network ranges, or even customer-specific IPs. You expect, by using this SG id (sg-) in conjunction with this service annotation that: All other security groups kubernetes created during init would be "wiped away", and your custom SG would be the load-balancer's only security group (assuming you only had one).

All entries in your custom security group are intact, and would be retained by the control plane. Gotcha, looks like it doesn't. It's reverted back to 0.0.0.0/0 as the source for all rules. Temporary solution: Run the override script (in our case, assign the serviceAnnotations to the gateway). Still use a custom security group, but leave it empty. When all and done, add your custom entries to the SG. As of 08/22, this cheap workaround, works. Still needs investigating.
Any subsequent updates to the cluster, reverts all configurations back to 0.0.0.0/0. EKS was updated from eks.2 to eks.3. Which somehow triggered the changes with istio.
Hi, It looks I have the same issue. Company policy dictates that all SG's have a specific name. Before: we just had service deployment which result in an auto-generated ELB with an auto-generated SG with the 'wrong' name, but everything was working fine. I'm now in the process of getting the correct name. So I created a custom SG "A" which has ingress rules SG "A1" and "A2". I add this with Now the elb is still auto-generated (as expected), but it only has 1 of the SG's from "A", being "A2" and a autogenerated one. I the a file with contents in the location. But I cannot get it to work. We use as , which probably creates this extra rule. I cannot find ( or am probably not able to read correctly) the correct config. Should the aws-load-balancer-internal be used or not? Do I need another directive? Maybe someone can help me out with some pointers? Thanks in advance.
It's possible to work around this by setting to something bogus (i.e. ) and putting your desired SG in the annotation instead (k8s won't modify this one).
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale


Question: "metadata": { %s "name": "subobject", <del> "namespace": "my-ns" </del> <ins> "namespace": "%s" </ins> } } } }` meta := unknownFieldMetadataJSON(gvk, "test-cr") <del> unknownRootMetaCR := fmt.Sprintf(embeddedCRPattern, meta, "") </del> <ins> unknownRootMetaCR := fmt.Sprintf(embeddedCRPattern, meta, "", ns) </ins> _, err = framework.RunKubectlInput(ns, unknownRootMetaCR, "create", "--validate=true", "-f", "-") if err == nil { framework.Failf("unexpected nil error when creating CR with unknown root metadata field")

/kind failing-test <!-- If this is a flake, please add: /kind flake --/sig cli
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign


Question: Corefile: | .:53 { errors <del> log stdout </del> <ins> log </ins> health kubernetes {{ .DNSDomain }} {{ .ServiceCIDR }} { pods insecure

Is this a BUG REPORT or FEATURE REQUEST?: BUG What happened: When CoreDNS is deployed by default using feature-gates of kubeadm, it does not log anything when processing a DNS query. That is due to change of log option of CoreDNS in latest version. that version was bumped up in kubeadm, but the manifest was not adapted to the new log option. initial change of 'log stdout' in CoreDNS : version of CoreDNS 1.0.0 bumped into Kubeadm : NOTE: I guess we should also changed manifests for kube-up : see The change in manifest expected is to remove the option "stdout" in the line "log ...". As it is now interpreted as a scope of domain to log. Invalid manifest: Environment: Kubernetes version (use ): 1.9 and MAIN Cloud provider or hardware configuration: any Install tools: kubeadm, kube-up the configmap for coredns kubectl get configmaps -n kube-system coredns -o yaml kubectl replace -f - the coredns deployment kubectl scale deployment --replicas=0 -n kube-system coredns kubectl scale deployment --replicas=1 -n kube-system coredns
/cc /sig network
NOTE: only log is impacted. CoreDNS working as expected.
/cc
I have a question : the initial manifest is done for the image 0.9.10 of CoreDNS. Bumping the version of CoreDNS was done by changing the version in file (see ) which means that the image's version is STILL 0.9.10 in the manifests ... and the corefile config is correct against the version of the manifest. QUESTION: when modifying the , is there a way to upgrade the manifest by program ? I guess at the time the new version number is applied on the manifest ...
thanks for reporting this! I send PR to update the Corefile. I don't think of a way so far. I guess we can't unless there is a way to get the Corefile from some place of upstream (CoreDNS).


Question: // the binding (PV->PVC or PVC->PV) was installed by the controller.  The // absence of this annotation means the binding was done by the user (i.e. // pre-bound). Value of this annotation does not matter. <ins> // External PV binders must bind PV the same way as PV controller, otherwise PV // controller may not handle it correctly. </ins> const annBoundByController = "pv.kubernetes.io/bound-by-controller" // This annotation is added to a PV that has been dynamically provisioned by

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: create a pod with local volume scheduler start to do bind, it will make pv = controller will call syncVolume to make pv bound , then syncClaim will make pvc bound. But from the log, I find that after step 2, pv controller start to reclaim the pv because pvc not in cache. logs: We can see after pv reclaimed, then the logs related with pvc start to print out. I know kube-scheduler will wait for the pvc to be bound, then it will be scheduled again in this situation. But is there a better way to handle this situation at pv_controller? What you expected to happen: PV not reclaimed. How to reproduce it (as minimally and precisely as possible): Create pod and local volume pvc in large scale Anything else we need to know?: Environment: Kubernetes version (use ): k8s 1.9
/sig storage Maybe syncVolume could wait to make PV bound until PVC exists. wdyt?
/assign
/assign
Perhaps we cannot wait to make PV bound until PVC exists, because syncVolume needs to release PV if PVC does not exist. Even if scheduler pre-binds a PV, when its PVC is deleted, PV controller should release PV. The problem is that after binding a PV scheduler waits for PV controller to bind PVC, but PV controller may not have corresponding PVC (not synced or deleted), so it will release PV. PV controller pre-binds a PV the same way as scheduler, but PV/PVC binding processes share same cache, so it will not have problem. To fix this, we can check PVC existence in apiserver instead of local cache in at the cost of performance. Possible optimizing is we can add a specific annotation for scheduler, then can check PVC existence in apiserver only when volume is bound by scheduler.
Another optimizing is that we can get PVC from apiserver only if PVC does not exist in cache. Double-check PVC from apiserver to make sure PV is not reclaimed on API sync delay.
We are doing the same way at our prod env.
Fix with the second optimizing is much simpler. I created a PR:
Next time, you can submit a PR if you have a fix for issues you encountered ;-) By the way, how many nodes in your large scale cluster?

For this issue, I think there should be solution which no need to touch api-server to fix this issue. That's why I didn't submit the PR. Our cluster has more than 1k BM nodes, and we have more than 10K local volume PV. This issue happens when I create 1000 pods(which use local volume) with a for loop script. It is easy to reproduce.
This issue happens when cache is out of sync (perhaps heavy load on apiserver), that would be better if we can fix it without touching apiserver.
I am not sure I fully understand the issue. How is it possible that PV controller does not see the PVC in its cache? It must have been created before pod was created and scheduler bound it to a PV. That's IMO long enough for a pod to be in the cache, especially when PV got there quick enough.
The issue is syncing PVC/PV caches across scheduler and pv controller. The PVs have been created long ago. Now a bunch of PVCs and Pods are created with delayed binding. Now, the scheduler will process the pod -pvc, and then bind PV by setting PV.ClaimRef. However, the PV controller could be slow to get API updates, and does not know about the new PVC object yet, but it got the PV update from the scheduler. So it sees that PV is bound and pointing to a ClaimRef, but PVC doesn't exist, so it goes into Release state.


Question: ns := f.Namespace n.externalPluginName = fmt.Sprintf("example.com/nfs-%s", ns.Name) <del> // Reset config. It might have been modified by a previous CreateVolume call. n.driverInfo.Config.ServerConfig = nil </del> // TODO(mkimuram): cluster-admin gives too much right but system:persistent-volume-provisioner // is not enough. We should create new clusterrole for testing. framework.BindClusterRole(cs.RbacV1beta1(), "cluster-admin", ns.Name,

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: gce/gke multizone/regional jobs Which test(s) are failing: Only PD tests Since when has it been failing: Around 12/21 with the 16:49 run Testgrid link: https://k8s- Reason for failure: TBD Anything else we need to know: <!-- DO NOT EDIT BELOW THIS LINE --/kind failing-test
The runs started flaking after merged. cc Here's one failing run: Pod pod-subpath-test-gcepd-dynamicpv-pfsm failed to schedule because of conflicting node selectors: We shouldn't need to set the node selector for PD tests using PVs.
The PR broke the subpath test because the modification to the config struct embedded in the driver info now persists across tests: A quick fix would be to reset that field in CreateDriver. The long-term fix is the change discussed in My preference is to do the quick fix now, then either merge or close PRs in this order: (needs a decision about logging of tests that aren't getting defined, see ) a future PR for What I'd like to avoid is having to rebase PR on top of the long-term solution for issue - that'll be lots of code conflicts.
/assign PR is here: PR


Question: # exit on any error set -e <ins> gsutil_version=$(gsutil version | awk '{split($0,a," "); print a[3]}') # Warning! uses lexical comparison.  This really only works for major versions, or minor versions up to x.9 min_gsutil_version="4.0" if [[ "$gsutil_version" < "$min_gsutil_version" ]]; then echo "gsutil version $min_gsutil_version or greater is required, please run 'gcloud components upgrade'" exit 1 fi </ins> SCRIPT_DIR=$(CDPATH="" cd $(dirname $0); pwd) source $SCRIPT_DIR/config.sh

As I discovered the hard way, if the json GCS api is not turned on for a project, will not surface this error, but just loop 'forever'. API access errors should be easy to catch, so it would be great to surface them on cluster startup and abort with a useful message. (I might take a look at this).
This is described in step 3 of But I agree that it would be better if the tool could catch this case and let the user know (it bit me the first time I tried to run that script as well).


Question: return "localhost/" + seccompLocalhostRef(profileName) } <ins> func TestIsInitContainerFailed(t *testing.T) { tests := []struct { status      *kubecontainer.Status isFailed    bool description string }{ { status: &kubecontainer.Status{ State:    kubecontainer.ContainerStateExited, ExitCode: 1, }, isFailed:    true, description: "Init container in exited state and non-zero exit code should return true", }, { status: &kubecontainer.Status{ State: kubecontainer.ContainerStateUnknown, }, isFailed:    true, description: "Init container in unknown state should return true", }, { status: &kubecontainer.Status{ Reason:   "OOMKilled", ExitCode: 0, }, isFailed:    true, description: "Init container which reason is OOMKilled should return true", }, { status: &kubecontainer.Status{ State:    kubecontainer.ContainerStateExited, ExitCode: 0, }, isFailed:    false, description: "Init container in exited state and zero exit code should return false", }, { status: &kubecontainer.Status{ State: kubecontainer.ContainerStateRunning, }, isFailed:    false, description: "Init container in running state should return false", }, { status: &kubecontainer.Status{ State: kubecontainer.ContainerStateCreated, }, isFailed:    false, description: "Init container in created state should return false", }, } for i, test := range tests { isFailed := isInitContainerFailed(test.status) assert.Equal(t, test.isFailed, isFailed, "TestCase[%d]: %s", i, test.description) } } </ins> func TestStableKey(t *testing.T) { container := &v1.Container{ Name:  "test_container",

Init container exited with OOMKilled reason, but init container was marked succesful, and main app container was allowed to be started If non init process gets killed the container should not be marked OOMKilled or If OOMKilled is terminate reason main app contaienr should not be started. According to Container should only be marked OOMKilled if init process is killed. the container the When checking init container status: But main app container is running. Kubernetes version (use ): Dockerfile:
According to previous issue will add sig label: /sig node
/assign
I have a quesion. when the oom killer, why your init container exit code is 0. normal is 137.
the init process is the shell script. the oom killer kills the forked process of stress tester, and it continues the script. next step is to exit 0 in the if process. wth error handling this can be caught, but my problem is, if there is a 0 exit code, and init process isnt oom killed, why is there a reason of oomkilled.
when you use shell scripts, you should add . kubelet will collect the status of init container. when init container is oomkilled, the container status will save the oom kill message.
I understand that. but according to the behaviour that is written, this is not expected outcome. The init container was not killed because of oom, just a single process. There should be 2 expected behaviours: Either there should be no printed, since it should only be when the main init process gets killed Or the whole init container should fail if any process is caught with oom.
I think you are right.
/assign
/triage accepted


Question: // HPAScaleTest struct is used by the scale(...) function. type HPAScaleTest struct { <del> initPods                    int32 totalInitialCPUUsage        int32 </del> <ins> initPods                    int totalInitialCPUUsage        int </ins> perPodCPURequest            int64 targetCPUUtilizationPercent int32 minPods                     int32

Test: https://k8s- Example: https://k8s- The HPA tests for gce-serial are flaking rather regularly; they fail about 2/3 of the time for the last week, and are often the cause of a failed test run. Can we de-flake these? Over multiple test runs the problem seems to be the number of replicas jumping up to 4 unexpectedly: Is the limit not getting set correctly here? Is this an actual bug? /sig autoscaling /priority important-soon /kind failing-test /kind flake
cc:
Recent failures fall into three categories: not starting (, , , , , , ). They fail with error message: increasing unexpectedly (, , ). They fail with error message: waiting for scale down (). This one failed with error message: I suspect 2. and 3. may be caused by test using correct amount of CPU when averaged over minute-long window but spiky (for example using 100% cpu for some time then 0% cpu for rest of the minute). I saw similar behavior when validating recent changes manually. This would result in HPA: behaving as the test expects with before recent changes (because it ignored any spikes during scale*forbidden windows, longer sample windows masking the spike), HPA (correctly) increasing size if spikes align after those changes (failures in group 2.), HPA (correctly) not decreasing size if spikes align after those changes (failure in group 3.), I'll take a look at code of those tests and try to responsible for most of the flakes but I don't have any ideas for a cause yet.
/remove-kind failing-test
I was busy this week but I think I'll be able to prepare a fix for this tomorrow.
/milestone v1.13 /kind failing-test
this tests is failing consistently in
On Friday I verified that CPU usage generated by resource consumer stays really close to the target value but oscillates slightly. To fix this I: Increase CPU usage in the test (resource consumer is implemented in a way that makes me think that deviation fro target are of a fixed size so higher target would make deviation a smaller percent of the target). Lower generated load (the test right now is targeting border between recommendation of 3 instances and 4 instances, I will change the load to something between 2 and 3 instances).

I'm checking if this helps the test.
It seems to have helped. I've sent a PR with those changes for review.
Re-opening until we have at least 3 green runs: https://k8s- /reopen
Reopening this issue. <detailsIn response to : Re-opening because we just had another flake last night: Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
But it doesn't work on test grid, the test now fails with: I'll look into this.
Yes, and it's now causing another test to fail as well: https://k8s- Escalating to critical-urgent. This test has been failing for 3 weeks now, and is potentially blocking visibility into other failures. /priority critical-urgent /remove-priority important-soon
This test had a couple of successful runs yesterday but is now flaking/failing in master-blocking: https://k8s- It seems to be passing in master-upgrade. Any chance we could prioritize work on it? It's preventing us from seeing whether fixes for other test failures are working.
There was only 1 flake since merged on Thursday (and ~15 green runs). was an unexpected scale up happening during stabilization. I'll check if lowering CPU utilization a bit lower deflakes the test more but I think it's ok to close this issue now.
Lowering CPU utilization didn't help.
Test jobs where these tests have flaked or failed: https://k8s- (today) https://k8s- (last 3 days ago) https://k8s- (yesterday) https://k8s- (yesterday) https://k8s- (today) https://k8s- (yesterday) ... basically, everywhere the tests run, they're flaky as recently as today. More on Triage: You can see here that across all jobs the HPA tests are flaky, with a "worst flake" perioud of 10/23-10/25, but by no means cleared up.
SIG-Autoscaling's boards are also showing flakiness: https://k8s-, although that looks better than the sig-release test boards. Any idea why?

Thanks for the links I've been looking only at one of those dashboards. I've sent that should reduce flakiness of our tests.
Thanks!
The PR merged. Now let's wait and see if it solves the problem.
The test named in the tile of this issue<sup1</supgot better. Last failures at the dashboards you listed: , (before merged), error before taht was on 10-25), , , . So it looks like the test didn't flake in the last ~2 days. Another two tests<sup2</supare flaking , I'll send a PR that should fix it. I think those flakes are result of behavior of HPA intentionally changing<sup3</supso the PR will just relax expectations of the test. <b1</b[sig-autoscaling] [HPA] Horizontal pod autoscaling (scale resource: CPU) [sig-autoscaling] [Serial] [Slow] ReplicationController Should scale from 5 pods to 3 pods and from 3 to 1 and verify decision stability <b2</b[sig-autoscaling] [HPA] Horizontal pod autoscaling (scale resource: Custom Metrics from Stackdriver) should scale down with External Metric with target value from Stackdriver [Feature:CustomMetricsAutoscaling] and [sig-autoscaling] [HPA] Horizontal pod autoscaling (scale resource: Custom Metrics from Stackdriver) should scale down with External Metric with target average value from Stackdriver [Feature:CustomMetricsAutoscaling] <b3</bHPA no longer delays scale ups based on metrics other than CPU. Flakes are when HPA scales deployment to 2 or 3 so test doesn't observe it having size 1 and times out waiting for that size.
I misread titles of custom metric tests as tests for scaling up somehow. Those flaking tests are tests for scaling down and scale downs sometimes don't happen. I'm looking into why.
While this test has gotten better, it's still flaking some of the time: https://k8s-
I looked at "Latest failures" in the two top categories according to (all the other categories have almost no recent flakes). All of the failures log as if they were not affected by : instead of .

Tests results also say they are running version 1.11... or 1.12... (or more rarely 1.9... or 1.10... but I'm ignoring those for now - they are not related to any changes I made). So it looks like I need to cherry pick the PR into 1.11 and 1.12 branches.
... when I look at git branches it looks like my PR is there. There was with logs that looking like affected it. I'll wait to see if there are more failures in test runs including that change. In case there are I'll add some more logging (I have no ideas for cause of the one failure I saw so far).
/assign
GitHub didn't allow me to assign the following users: jbartosik. Note that only and repo collaborators can be assigned. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
any further update on this? I see it turning into more of a failure than a flake and I see another test failing as well in latest runs
We are due to cut Beta for 1.13 today and would like to understand if this should be a blocker or not.
Also are there cherrypick PRs to backport your change to 1.11 and 1.12 branches?
Potentially related:
6 (consecutive) most recent failures of test do not have judging by their logs. That PR would change them from failures (unexpected scale from 3 to 4 instances during stabilisation) to success (expected 3 to 4 instances during stabilisation). One of the 2 recent failures of looks wrong. One failure was scaling up immediately to 5 instances (wrong). The other was scale up to 4 instances (ok, I'll send a PR relaxing that condition tomorrow). I didn't send PRs cherrypicking test changes to 1.12 (because when I checked it looked like the change was there) not to 1.11 (my changes aren't there so old tests should keep working). merged and it ads more logging, which will help me debug this. Failures in the look like the PR relaxing stability requirement should have helped there too.

the , which has the 6 (consecutive) most recent failures , upgrades the master alone to 1.13(master) and and keep the nodes in old version and then run the old test suite. The latest failing runs show that the tests are run from 1.11.5 branch Doesn't this mean your changes need to be backported to 1.11 branch? That is the reason why you dont see you change in those tests logs
FWIW, the same test also flakes in job which upgrade both master and nodes to 1.13 and runs 1.11 tests.
this failure is currently blocking Beta for 1..13. Can you please investigate these and let us know the path forward.
Fixes done by are all test fixes, not feature fixes. So the problem should be fixed once cherry-pick gets to 1.11. That being said - why are we running 1.11 e2e tests against 1.13 cluster? That assumes full compatibility of both API and implementation. We can never remove a feature without breaking this suite (the tests for the feature will still exist on older branches after all). And the tests are often tweaked to the specifics of implementation. Whenever we make any change to autoscaling algorithm tests need to be updated accordingly. Making tests on 1.11 work well with 1.13 risks they will stop working well with 1.11. Shouldn't the upgrade job create 1.11 cluster, upgrade it to 1.13 and run 1.13 e2e tests on that cluster?
I see these 2 pass in the latest run of https://k8s- Over to to close when he sees fit
We have other flakes on that job now, but HPA is no longer flaking. Thanks so much for the dilligence in getting these flakes cleared up, /close
Closing this issue. <detailsIn response to : We have other flakes on that job now, but HPA is no longer flaking. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details