Question: kube-apiserver,busybox kube-controller-manager,busybox kube-scheduler,busybox <del> kube-proxy,gcr.io/google_containers/debian-iptables-amd64:v3 </del> <ins> kube-proxy,gcr.io/google_containers/debian-iptables-amd64:v4 </ins> );; "arm") local targets=( kube-apiserver,armel/busybox kube-controller-manager,armel/busybox kube-scheduler,armel/busybox <del> kube-proxy,gcr.io/google_containers/debian-iptables-arm:v3 </del> <ins> kube-proxy,gcr.io/google_containers/debian-iptables-arm:v4 </ins> );; "arm64") local targets=( kube-apiserver,aarch64/busybox kube-controller-manager,aarch64/busybox kube-scheduler,aarch64/busybox <del> kube-proxy,gcr.io/google_containers/debian-iptables-arm64:v3 </del> <ins> kube-proxy,gcr.io/google_containers/debian-iptables-arm64:v4 </ins> );; "ppc64le") local targets=( kube-apiserver,ppc64le/busybox kube-controller-manager,ppc64le/busybox kube-scheduler,ppc64le/busybox <del> kube-proxy,gcr.io/google_containers/debian-iptables-ppc64le:v3 </del> <ins> kube-proxy,gcr.io/google_containers/debian-iptables-ppc64le:v4 </ins> );; esac

From another issue, said: DockerManager.SyncPod() calls the network Plugin and then sets hairpin mode unilaterally. This is wrong. We should either pass the bool down to here or else we should get rid of the default no-op plugin and use the non-existence of a plugin as a flag to control this. This should be a relatively easy one for people who want to contribute but don't know where to start.
I volunteer myself to do this. Actually I have done something about kubernetes network plugin so guess I can do this. I will start to do this when I am on the office tomorrow.
Awesome. I'd advise starting with a sketch so we can all make sure it is a good approach, before jumping in with both feet. , TonyAdo wrote:
ok, I will start with a sketch as soon as possible.
I thought about this for a while: if we pass the bool down to here, of course we can check this flag, if it is set, we should set veth hairpin mode, but problem is that if user set this flag, but still use a custom network plugin just like you mentioned the ovs plugin, I am afraid we will still enable hairpin mode in all interfaces on this host, since we cannot find a peer interface. Is that correct?
Yeah, I think the better fix is to use the existence of a network plugin as an indicator. It sort of makes --configure-cbr0 redundant , TonyAdo wrote:
ok, I will try to get rid of the default no-op plugin and use the non-existence of a plugin as a flag to control this.
I decided to use the existence of a network plugin as an indicator, also I will get ride of the no-op network plugin, I am not going to add new flag to kubelet structure, if the user don't specify network plugin name, the networkPlugin within kubelet will be which just means non-existence of a plugin, and we have to check the existence in the docker manager to avoid nil pointer exception also for setting, I think if a custom network plugin exists, kubelet should not do this. How about this? Should I open a PR?
fyi, saw a hairpin pr today that might be related. ymmv.
what do you think about this?
What you described sounds right to me.

I think the only way to see if there is some detail I am missing is to try it and see if anything pops up that would make this not work. , TonyAdo wrote:
I need to check which part is responsible of what, but what if the hairpin package was just a utility package that should be called by the plugin if it needs to set this up? As long as it is single and simple function call, doing so would avoid adding extra logic here and fix the issue. This is valid only if the aim is to have a "linux bridge plugin", an "OVS plugin", etc.
actually kubelet will call after get called, I think to make the network plugin can call hairpin package is not that easy, of course, I agree that network plugin should setup hairpin mode if it really needs, kubelet should not set hairpin mode unilaterally. Besides, kubelet is not going to provide an "linux bridge plugin" or "OVS plugin".
then the validity conditions of my comment are not met, so I see any simpler way. Thanks for checking! :)
still what do you think about disable hairpin completely and leave this to network plugin?
or if there is a custom network plugin, we don't setup hairpin mode, only do this in default.
Completely? It will break my use case (and anyone using Apache Kafka).
Only by default is what I understood from the previous comments, and LGTM.
Really sorry about that, but I really want to know are you using a kubelet network plugin in your use case? If no, I think we will still setup hairpin, as in default we will do this.
I don't use any plugin, so it is ok if hairpin is set up by default.
ok, I think this will not break your use case.
It's not really the network plugin that needs hairpinmode though, it's the pure-iptables-based kube-proxy code. It's possible the proxying code could change again in the future in such a way that hairpinmode wouldn't be needed any more. So it seems wrong to put that code into the plugin. makes it so that we set hairpinmode on the veth if it's attached to a bridge, but just silently do nothing if it's not.

(Yes, we currently set hairpinmode even if you're using the userspace proxier, but AFAIK it's not needed in that case.)
IMHO... The proxy has the responsibility of service objects, while the kubelet is responsible of pod objects, including their networking. Also, the responsibility of hairpin must be at the same place as the code that creates the network interface, otherwise you have code managing a single object (the pod interface) spread over multiple programs, which will eventually lead to complex problems and a system harder to understand even for the end-user. Thus, it doesn't seems wrong to me that the kubelet does whatever is required to make the pod work in the environment of the cluster. While its true that the hairpin requirement is only for the proxy in iptables mode, a switch to this mode can happen at any time in the life of the cluster. And when it happens, its probably because the userland proxy becomes unresponsive, which means in a situation where the administrator is in a hurry to find a solution. Currently, you just change the proxy mode and restart the proxies. There's little chance the administrator will think about hairpin or restarting the kubelet in this case, so the kubelet should be reacting to this; which means the proxy and the kubelet of a given host should be able to communicate (AFAIK, this doesn't exist now). And all this is currently only to avoid setting up hairpin on the pod's port of standard Linux bridge.
Feel sure I'm missing something but... The CNI bridge plugin now has a flag, so kubenet can use that. Having done that, we can set false when the plugin name is non-blank.
Is this fixed now?
I apologise for leaving a somewhat vague answer, but I have been in that code recently. I don't think it's fixed.
, Bryan Boreham wrote:
Thanks for the PR. CNI Bridge plugin is expecting . Not . Hence the bridge plugin did not recognize it.
EDIT 2016-12-09: The chain below was broken by , so the issue can be closed again (remain closed) I suggest this should be re-opened. The chain of calls is: sets to , e.g. when using CNI calls something goes wrong and it sets the mode on everything. An example of "something goes wrong":


Question: kubeClientConfig.QPS = 99999 kubeClientConfig.Burst = 9999 <ins> // we make requests to all resources, don't log warnings about deprecated ones restclient.SetDefaultWarningHandler(restclient.NoWarnings{}) </ins> kubeClient := clientset.NewForConfigOrDie(kubeClientConfig) go func() {

Improve the to ensure that the expected storage version for a resource is one understood by previous releases. We have API testdata from prior versions available in When checking each group/version/kind in the test: accumulate all the versions of the group/kind available in previous (non-HEAD) releases if there is a beta or stable version of the group/kind available in previous releases, then the storage version must be in the set of previously available versions It is too easy to accidentally change the storage version prematurely and edit the test/integration/etcd test to match the bug cc
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: expectErrors: false, validate: (&JobControllerOptions{ &jobconfig.JobControllerConfiguration{ <del> ConcurrentJobSyncs: 5, </del> <ins> ConcurrentJobSyncs: 10, </ins> }, }).Validate, },

What would you like to be : The ability to set for the kube-controller-manager at runtime. It looks like the option exists in the code, but a mechanism is not available to set it via kops. If I understand the code correctly, a flag needs to be in order to be able to set the value using kops. Why is this needed: We want to shift a lot of our batch processing jobs from running directly on our CI servers to running as jobs on Kubernetes. Unfortunately if too many jobs are launched on kubernetes, they begin to back up. We've noticed up to a 10 minute delay between when the logs indicate that a job has finished and when the kubernetes API actually reflects the same on our small ~30 node cluster.
/sig scalability
I think scalability is the best fit since this is impacting our ability to fully utilize and scale how we use our kubernetes cluster.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
/sig api-machinery
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with .

/close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Heyo -- we're running into a similar issue as this with our Kubernetes cluster. Is there anything I can do to help get landed?
You can't reopen an issue/PR unless you authored it or you are a collaborator. <detailsIn response to : Heyo -- we're running into a similar issue as this with our Kubernetes cluster. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign /remove-sig api-machinery /sig apps
/triage accepted /remove-lifecycle rotten
can you please describe your use case in a more detail? how How this affects you, and how this flag would help to achieve your goals?
-- I left a comment describing my use case in the PR:


Question: "//pkg/util/sysctl:go_default_library", "//pkg/util/version:go_default_library", "//vendor/github.com/golang/glog:go_default_library", <del> "//vendor/golang.org/x/sys/unix:go_default_library", </del> "//vendor/k8s.io/api/core/v1:go_default_library", "//vendor/k8s.io/apimachinery/pkg/types:go_default_library", "//vendor/k8s.io/apimachinery/pkg/util/sets:go_default_library",

https://k8s- https://k8s- cc: /priority failing-test
Sorry, but I will fix it ASAP.
/assign
PR is already sent. Wating for CI report...


Question: Into(result) return } <ins> // CreateScale takes the representation of a scale and creates it.  Returns the server's representation of the scale, and an error, if there is any. func (c *clusterTestTypes) CreateScale(ctx context.Context, clusterTestTypeName string, scale *autoscalingv1.Scale, opts metav1.CreateOptions) (result *autoscalingv1.Scale, err error) { result = &autoscalingv1.Scale{} err = c.client.Post(). Resource("clustertesttypes"). Name(clusterTestTypeName). SubResource("scale"). VersionedParams(&opts, scheme.ParameterCodec). Body(scale). Do(ctx). Into(result) return } </ins>

What happened: Test create method for non-namespaced subresources generate invalid code. I have the following code generation annotations: which generated test method with generated fake client: What you expected to happen: It should be: How to reproduce it (as minimally and precisely as possible): Generate clients for non-namespaced resources with subresources (like scale) like this: /sig api-machinery /area code-generator /assign
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : What happened: What you expected to happen: It should be: How to reproduce it (as minimally and precisely as possible): Generate clients for non-namespaced resources with subresources (like scale) like this: /sig api-machinery /area code-generator /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/triage accepted


Question: // Find all sibling pods in the service and post to their /write handler. func contactOthers(state *State) { <del> const waitTimeout = 2 * time.Minute </del> <ins> const waitTimeout = 5 * time.Minute </ins> defer state.doneContactingPeers() client, err := client.NewInCluster() if err != nil {

cc
This has failed The two failures are different, and both seemingly totally legitimate. The first one is simply a pod never entering running state within the 5 minute timeout. Expected error: <*errors.errorString | 0xc208475150: { s: "gave up waiting for pod 'nettest-dtwk3' to be 'running' after 5m0s", } gave up waiting for pod 'nettest-dtwk3' to be 'running' after 5m0s not to have occurred The second is not being reachable: Dec 7 01:22:47.275: Unable to connect/talk to the internet: Get dial tcp 74.125.202.147:80: i/o timeout The latter could be a result of a missing MASQ rule on the node, or GCP egress problems. I don't think that this is worth trying to debug further in the context of this test for intra-pod communication. I'll check more generally whether we have a current intermittent problems with respect to pods entering running state, or reaching the internet.
I think this test may be causing builds to time out, too. See comments from me in .
For the future, I actually don't think we need the test at all in the networking test. Maybe even move it do a different test altogether.
This is clearly a bug. I'll send a PR shortly: Regarding the ping to , why did you put in in the first place?
hmmm, not sure what you mean by "in the first place" ? I did not write the test...
sorry, I thought that you'd written that part of the test. I will find out who wrote that, figure out why they put it there in the first place, and then either remove it, or rewrite it to work correctly. Clearly that part of the test needs at least some basic retry to work consistently.
ok great, no problem just wasnt sure what you meant. SGTM. Id be okay w/ deleting the ping entirely regardless of who wrote it, its a very old snippet which doesnt really add much to the overall test suite (and in fact is questionable wether it should even be an assertion at all )
somebody put it there for some reason. Better to understand why before just deleting it.

i looked, its actually rob who i work with :) i will ask him in the morning unless you find him first :) (rrati)
Regarding my alleged "clear bug" above, I was completely wrong. There's nothing wring with that code. And unfortunately, when this test goes into a presumed endless loop (see comment above) it does not produce any output (as the output is cached by Ginkgo until the test completes). So it's very difficult to debug. But in summary I agree with this test is probably never completing, and causing tests to time out. Someone will need to isolate this test in a environment where it produces output and can be debugged. I'll keep trying, but need to head home soon, so if anyone wants to pick it up overnight, go ahead.
Smoking gun:
hmm - I'm just wondering why only this particular test is hanging forever (if we agree that's the case). I don' think that some requests are hanging forever in this test, but not in any other test...
Closing in favor of which should fix this. Or at least some way to set client timeout, which and are still figuring out.
It's happening again.
Talked to offline. I am assigning this one to you for triage. Let me know if you need any help.
AFAICT the last failure was just the pod nettest-7h5jj not coming up
Looks like this should be resolved by your pr: too. :-)
Maybe. Though the webserver only deadlocks in the failure case, so the test is probably still flaky...
Looks like this test is now explicitly flaking by failing, as I expected.
Based on the , it took 15 minutes to pull the image. closing this one as dup of
New Failure:
This looks like a race between populating the endpoints list and the /status, /read methods of the service exposed through the master proxy. It looks like the webserver pod is just trying to confirm that it can reach all peers. We can do this synchronously after having observed all expected endpoints in the test (or re-write the test to use the same pattern we do in services e2e).
The test looks really racy to me. I think the following 15 retries prevents it to fail more often, but not completely.

I will submit a pr to enfore webservers to wait till alll endpoints are up before they start to contact each other.
New failure
I am making this P0
It took more than 4 minutes for a nettest pod to come up. The nettest container is waiting for all endpoints to be ready for 2 minutes plus 15 attempts (5 sec interval) to contact its peers. This happens in less than 4 minutes. Hence the first 2 nettest pods finished contacting peers and declared failure.
The node you want is: I see: that's too much silence ...
From the It looks like kubelet is slow to start the container. Due to the lower logging level in the gke, it's hard to guess what could've slowed down the goroutine...


Question: // The Kubelet panics if either kube-reserved or system-reserved is not set // when CPU Manager is enabled. Set cpu in kube-reserved > 0 so that // kubelet doesn't panic. <ins> if newCfg.KubeReserved == nil { newCfg.KubeReserved = map[string]string{} } </ins> if _, ok := newCfg.KubeReserved["cpu"]; !ok { newCfg.KubeReserved["cpu"] = "200m" }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Test failed when running CPU Manager node e2e test on Ubuntu 16.04 locally, and show this panic message: What you expected to happen: Tests passed. How to reproduce it (as minimally and precisely as possible): Run this node e2e test, use the following command: Anything else we need to know?: I think this is because the following codes: is . I will send a PR to fix this after it is confirmed. /sig node /assign PTAL, Thanks!
Thanks for the bug report. Reproduced successfully. It looks like the response body may have changed from empty map to null, which caused this to break. In any case, suggested adding the fix to his pending PR here:
Hi, have fixed this, lgtm.


Question: else for platform in "${platforms[@]}"; do kube::log::status "Building go targets for ${platform}:" "${targets[@]}" <del> kube::golang::set_platform_envs "${platform}" kube::golang::build_binaries_for_platform ${platform} ${use_go_build:-} </del> <ins> ( kube::golang::set_platform_envs "${platform}" kube::golang::build_binaries_for_platform ${platform} ${use_go_build:-} ) </ins> done fi )

Similar to this is to test the first contributor experience fails on ubuntu 16.04, docker version 1.12.3:
Got the same error on Linux ubuntu 4.2.0-27-generic ~14.04.1-Ubuntu SMP Fri Jan 22 15:32:26 UTC 2016 x8664 x8664 x86_64 GNU/Linux Docker Client version: 1.6.2 Client API version: 1.18 Go version (client): go1.2.1 Git commit (client): OS/Arch (client): linux/amd64 Server version: 1.6.2 Server API version: 1.18 Go version (server): go1.2.1 Git commit (server): OS/Arch (server): linux/amd64
What git sha of were you building from?
I have cloned the head. I think the reason might be missing package: gccgo-4.8-aarch64-linux-gnu I'm compiling right now and update on result.
were you running or ? runs inside docker, so the gcc error is confusing.
I run: sudo make release in Ubuntu VM running in VMWare workstation if that matters. Are you in kubernetes-users slack channel?
does fail in the same way?
Do you need to run , instead of just ? Can you try running the following:
well, if I don't do sudo docker does not work :) I need to run as root. I'll try the commands as soon as quick-release ends.
ah, gotcha. you can probably fix the docker access issue by following in the docker manual. that's probably not why it's failing, though.
make release still does not work :(
can you please run and then paste the results? ( turns up the debugging verbosity.)
for me: works (using your PR to fixes the gobindata dep) -local Go build- works (docker build) works if you have Docker running. does not work: on OSX did not seem to give more verbosity.
V=4 make release did not help :(
oops, that was not the correct environment variable, sorry. I meant .
here we go:
Aha, I think I figured it out. Sorry for making you rebuild so many times; the issue still isn't super obvious from the logs.

Basically, the bug only occurs if you don't have enough memory to do the docker builds in parallel; if that happens, you fall into this . On the parallel side of the branch, we set the platform env vars within a subshell, so each of the cross-compilation builds won't affect each other. On the non-parallel side of the branch, we don't use a subshell. Inside , there's no overrides done for . As a result, we end up incorrectly reusing the value for , , which doesn't recognize the flag being applied for the darwin build. The fix should be easy; I'll get that sent out ASAP. we probably want this fix for 1.5, since if you fall back to non-parallel crossbuilds we're doing the wrong thing.
Ack. Will keep an eye out for the fix. Please mark it as cherrypick-candidate
Thank you guys!
I think it is almost there. Got much further in the build (4 GB of RAM). Towards the end there is a failed test that makes the build fail: This is probably a different bug, as your PR got passed the misconfigured compilation.
FWIW, the worked on ubuntu 16.06. It did ran some tests apparently but the build succeeded.
I just tried with and it worked fine
since we have a PR ready and working, can we please switch this to non-release-blocker?


Question: echo 'No need to refresh sudo credentials' else echo 'Updating sudo credentials' <del> sudo --validate || exit 1 </del> <ins> sudo -v || exit 1 </ins> fi # FOCUS is ginkgo focus to select which tests to run. By default, FOCUS is

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: when run , I founded failed. What you expected to happen: Test passed How to reproduce it (as minimally and precisely as possible): Sudo version 1.8.6p7 Anything else we need to know?: Environment: Kubernetes version (use ): 1.17 Cloud provider or hardware configuration: OS (e.g: ):centos Kernel (e.g. ):Linux elk-141 4.9.6- SMP Wed Mar 1 18:39:23 CST 2017 x8664 x8664 x8664 GNU/Linu Install tools: Network plugin and version (if this is a network-related bug): Others:
/cc
/sig node


Question: return schema, nil } <ins> // validateList unpack a list and validate every item in the list. // It return nil if every item is ok. // Otherwise it return an error list contain errors of every item. func (s *SwaggerSchema) validateList(obj map[string]interface{}) errs.ValidationErrorList { allErrs := errs.ValidationErrorList{} items, exists := obj["items"] if !exists { return append(allErrs, fmt.Errorf("no items field in %#v", obj)) } itemList, ok := items.([]interface{}) if !ok { return append(allErrs, fmt.Errorf("items isn't a slice")) } for i, item := range itemList { fields, ok := item.(map[string]interface{}) if !ok { allErrs = append(allErrs, fmt.Errorf("items[%d] isn't a map[string]interface{}", i)) continue } groupVersion := fields["apiVersion"] if groupVersion == nil { allErrs = append(allErrs, fmt.Errorf("items[%d].apiVersion not set", i)) continue } itemVersion, ok := groupVersion.(string) if !ok { allErrs = append(allErrs, fmt.Errorf("items[%d].apiVersion isn't string type", i)) continue } if len(itemVersion) == 0 { allErrs = append(allErrs, fmt.Errorf("items[%d].apiVersion is empty", i)) } kind := fields["kind"] if kind == nil { allErrs = append(allErrs, fmt.Errorf("items[%d].kind not set", i)) continue } itemKind, ok := kind.(string) if !ok { allErrs = append(allErrs, fmt.Errorf("items[%d].kind isn't string type", i)) continue } if len(itemKind) == 0 { allErrs = append(allErrs, fmt.Errorf("items[%d].kind is empty", i)) } version := apiutil.GetVersion(itemVersion) errs := s.ValidateObject(item, "", version+"."+itemKind) if len(errs) >= 1 { allErrs = append(allErrs, errs...) } } return allErrs } </ins> func (s *SwaggerSchema) ValidateBytes(data []byte) error { var obj interface{} out, err := yaml.ToJSON(data)

Using kubectl create with a list no longer works. We get the output:
In kubernetes 1.0 with a List would create all items in the List.
Test data to duplicate the problem:
CC
FYI in case you run into this.
Another feature that needs a test.
the swagger spec may only include object definitions that are registered as API request/response types... since isn't actually submitted to the server (that I know of), that's probably why it didn't get included. Enabling validation by default () probably exposed the bug with a List not being able to validate
We need to make validation on List work. We should test multiple resources w/o List as well.
Naive question: Could we just call Schema() after Flatten() but before Do()?
Hmm. No, validation isn't actually called there.
Each input-source visitor (URLVisitor, FileVisitor, StreamVisitor) has its own call to ValidateSchema. We need to validate before decoding, but we don't unpack the list until after decoding.
We could do something similar to FlattenListVisitor in ValidateSchema: But that's pretty ugly.
Do you have time to take a look at what the most straightforward solution/workaround is for this? Longer-term we should maybe push this into the json decoder, as proposed here:
The simplest thing might be to add List to the swaggerapi models list… I was looking into that yesterday and got sidetracked
I thought about adding List, but it didn't look like the validator would recurse into the individual resource models, hence the need to special-case List.
Nobody is working on this at the moment, if you'd like to take a look.
OK, I will take a look.
If add List to the swaggerapi, I think it need to add a new api . If we post request to to handler the List, need to unpack the list in the server. Or just add an "empty" api only list the List kind ? I think that might be a little strange all above. Please correct me.
I recommended not adding List to the swagger. I don't think it would help. I think we need to detect the case of a List and extract the resources within, similar to FlattenListVisitor.
I agree. I think doing something similar to FlattenListVisitor in ValidateSchema maybe a little ugly. But it's straightforward.


Question: "net/http" "os" "reflect" <ins> goruntime "runtime" </ins> "sort" "strconv" "strings"

TestUpdateNewNodeStatus TestUpdateExistingNodeStatus TestUpdateNodeStatusWithRuntimeStateError The above tests seem to be architecture specific now (and fail on unexpected objects "linux" vs "darwin"). Per I would expect unit tests to still pass on OSX


Question: g.eventRecorder.Event(svc, v1.EventTypeWarning, "ILBOptionsIgnored", "Internal LoadBalancer options are not supported with Legacy Networks.") options = ILBOptions{} } <ins> if !g.AlphaFeatureGate.Enabled(AlphaFeatureILBCustomSubnet) { if options.SubnetName != "" { g.eventRecorder.Event(svc, v1.EventTypeWarning, "ILBCustomSubnetOptionIgnored", "Internal LoadBalancer CustomSubnet options ignored as the feature gate is disabled.") options.SubnetName = "" } } </ins> loadBalancerName := g.GetLoadBalancerName(context.TODO(), clusterName, svc) sharedBackend := shareBackendService(svc)

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: ILB ip addresses come from the subnet that the K8s cluster uses. There is no ability to specify a different subnet. What you expected to happen: ILB ip addresses should be allocated from a different subnet if specified. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: GCE OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/area platform/gce /assign
/sig network


Question: e2e_storage_test_environment: '$(echo "$E2E_STORAGE_TEST_ENVIRONMENT" | sed -e "s/'/''/g")' kube_uid: '$(echo "${KUBE_UID}" | sed -e "s/'/''/g")' initial_etcd_cluster: '$(echo "${INITIAL_ETCD_CLUSTER:-}" | sed -e "s/'/''/g")' <del> etcd_quorum_read: '$(echo "${ETCD_QUORUM_READ:-}" | sed -e "s/'/''/g")' </del> hostname: $(hostname -s) enable_default_storage_class: '$(echo "$ENABLE_DEFAULT_STORAGE_CLASS" | sed -e "s/'/''/g")'

Slack user "nalin" reported an issue Looking at the code this makes sense. Will fix.
For anyone following along, the fix for this bug unfortunately didn't get cherrypicked because of a process issue, and is broken until 1.5.2. cc
I just the new lines from As a temperatur fix ONLY. If anyone like me dont want to wait. Worked like at charm for AWS cluster setup. :)
A quick note, just as confirmation: the above temporary fix does resolve the issue for AWS - even though it's a change to a GCE-related file (cluster/gce/configure-). It seems that the GCE files are used for certain aws functions.
In based AWS deployment, the script is shared between AWS and GCE.
Also, this fix was cherry-picked into the branch a while ago, so instead of manually adding the lines, you can also just checkout the branch and run from there.


Question: --master="http://${API_HOST}:${API_PORT}" >"${SCHEDULER_LOG}" 2>&1 & SCHEDULER_PID=$! <del> echo "Local Kubernetes cluster is running. Press Ctrl-C to shut it down." echo "Logs: " echo "  ${APISERVER_LOG}" echo "  ${CTLRMGR_LOG}" echo "  ${KUBELET_LOG}" echo "  ${PROXY_LOG}" echo "  ${SCHEDULER_LOG}" </del> <ins> cat <<EOF Local Kubernetes cluster is running. Press Ctrl-C to shut it down. Logs: ${APISERVER_LOG} ${CTLRMGR_LOG} ${KUBELET_LOG} ${PROXY_LOG} ${SCHEDULER_LOG} To start using your cluster, open up another terminal/tab and run: export KUBERNETES_PROVIDER=local cluster/kubectl.sh EOF </ins> cleanup() {

When starting locally a cluster from source by running hack/local-up-, kubecfg and kubectl require gcloud in order to work. It doesn't make sense to depend on gcloud when running a cluster from source locally in a dev env, hence please provide instructions to make kubectl/kubecfg work locally without gcloud in this local setup.
Doesn't make sense for vagrant either
Have you read getting started guide? In section. I think it's recently :)
I think the issue is that kube up local / vagrant shouldn't require gcloud at all ----- Original Message -----
It shouldn't. I suspect that it is that defaults to gce. I'm going to take a run through the experience/guides and make sure this is clear and there is no gcloud required.
I had done some work on making a real cluster provider for local, but didn't merge it, since it didn't look like it would be sufficient to make e2e work. However, if it would make it easier to run locally, I could check it in. It's weird that the local setup lives in hack.
I think it'd be great to have a real provider for local, but I'm not sure it is a super high pri. I'm going to clarify the local script a little to close out this. -- let us know if doesn't fix it for you and we will investigate further.


Question: // SetNewReplicaSetAnnotations sets new replica set's annotations appropriately by updating its revision and // copying required deployment annotations to it; it returns true if replica set's annotation is changed. <del> func SetNewReplicaSetAnnotations(deployment *apps.Deployment, newRS *apps.ReplicaSet, newRevision string, exists bool) bool { </del> <ins> func SetNewReplicaSetAnnotations(deployment *apps.Deployment, newRS *apps.ReplicaSet, newRevision string, exists bool, revHistoryLimitInChars int) bool { </ins> // First, copy deployment's annotations (except for apply and revision annotations) annotationChanged := copyDeploymentAnnotationsToReplicaSet(deployment, newRS) // Then, update replica set's revision annotation

What happened: We created a Deployment and continuously updated an annotation on it periodically for a week. Eventually, we noticed errors in the controller-manager logs due to the annotation growing too large. Example annotations: controller-manager error: What you expected to happen: that do not change the revision number should not be to the revision history. revision history should be capped at a fixed number of elements, it should not be possible to have unbounded growth for Deployments that are frequently updated. How to reproduce it (as minimally and precisely as possible): a Deployment modify an annotation on it and note that the annotation gets the same number appended every time. Environment: Kubernetes version (use ):
/sig scalability Not 100% sure where this one fits, I couldn't determine which group owns the Deployment/ReplicaSet controllers, but this is a scalability problem
/sig apps
Agree with these, and if the limit is hit, the beginning of the history can be removed.
/assign


Question: fs.Var(componentconfig.IPVar{Val: &options.config.BindAddress}, "bind-address", "The IP address for the proxy server to serve on (set to `0.0.0.0` for all IPv4 interfaces and `::` for all IPv6 interfaces)") fs.StringVar(&options.master, "master", options.master, "The address of the Kubernetes API server (overrides any value in kubeconfig)") fs.Int32Var(&options.healthzPort, "healthz-port", options.healthzPort, "The port to bind the health check server. Use 0 to disable.") <del> fs.Var(componentconfig.IPVar{Val: &options.config.HealthzBindAddress}, "healthz-bind-address", "The IP address and port for the health check server to serve on (set to `0.0.0.0` for all IPv4 interfaces and `::` for all IPv6 interfaces)") fs.Var(componentconfig.IPVar{Val: &options.config.MetricsBindAddress}, "metrics-bind-address", "The IP address and port for the metrics server to serve on (set to `0.0.0.0` for all IPv4 interfaces and `::` for all IPv6 interfaces)") </del> <ins> fs.Var(componentconfig.IPPortVar{Val: &options.config.HealthzBindAddress}, "healthz-bind-address", "The IP address and port for the health check server to serve on (set to `0.0.0.0` for all IPv4 interfaces and `::` for all IPv6 interfaces)") fs.Var(componentconfig.IPPortVar{Val: &options.config.MetricsBindAddress}, "metrics-bind-address", "The IP address and port for the metrics server to serve on (set to `0.0.0.0` for all IPv4 interfaces and `::` for all IPv6 interfaces)") </ins> fs.Int32Var(options.config.OOMScoreAdj, "oom-score-adj", utilpointer.Int32PtrDerefOr(options.config.OOMScoreAdj, int32(qos.KubeProxyOOMScoreAdj)), "The oom-score-adj value for kube-proxy process. Values must be within the range [-1000, 1000]") fs.StringVar(&options.config.ResourceContainer, "resource-container", options.config.ResourceContainer, "Absolute name of the resource-only container to create and run the Kube-proxy in (Default: /kube-proxy).") fs.MarkDeprecated("resource-container", "This feature will be removed in a later release.")

Whenever I try to change kube proxy metrics bind using flag , it gives me: Then, I the port , the error changes. I expect the metrics server to listen to 0.0.0.0:. Do I miss something? Environment: Kubernetes version: v1.7.8 Cloud provider or hardware configuration**: on-premise OS (e.g. from /etc/os-release): debian/stretch Kernel (e.g. ): Linux R11U2-7U20N-1 4.9.0-3-amd64 SMP Debian 4.9.30-2+deb9u3 (2017-08-06) x86_64 GNU/Linux Install tools: Others:
/sig network /area kube-proxy
The is being and does NOT allow specifying a port. Have you tried using a config file? The field allows you to specify a port (an example ).
/assign
Turned out we use a wrong type when validating and . Sent for fixing this. Will try to cherrypick the fix as well.
New finding, is broken on HEAD in a different way: Seems to be introduced by
Humm..We have some default logic for : Seems like for some reason it does not auto append the default port number OR the validation happens before defaulting.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
/remove-lifecycle rotten
I got the same issue 0.0.0.0 0.0.0.0: both not work with kube-proxy 1.9.3 amd64
It is a BUG. Conflict between componentconfig.IPVar.Set(s string) and validation.validateHostPort.

Could you elaborate the issue a bit more? Which k8s version are you on?
I also meet the same issue with kubernetes 1.8. It's impossibe for Prometheus to fetch metrics from 127.0.0.1: in my case, any workaround? I think we'd better reopen this issue.
metrics binds to by default, are you trying to modify port as well? Thanks for reporting, I confirm the issue is remaining. (or any ip:port) does not work as intended even after the fix. Reopening this and filed a new issue for conflict kubeproxyconfig validation. /reopen
Thanks for reopening the issue. I want to bind metrics to 0.0.0.0: as well. With address 127.0.0.1:, Prometheus cannot pull metrics from this address unless they are on the same node.
is fixing this.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale (I just got bitten by this on kube-proxy v1.9.10)
should fix this.
Closing this as is merged. /close
Closing this issue. <detailsIn response to : Closing this as is merged. /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


https://k8s- /sig auth /kind failing-test
Adding a priority to this: /priority important-soon Please raise the priority if you consider this a release blocker. Given the test job, it normally wouldn't be.
It's a test issue. I'm hoping to fix this before 1.13 but I don't consider it a release blocker.


Question: - *Flocker Dataset Agent(s)*: a convergence agent that modifies the cluster state to match the desired configuration; - *Flocker Container Agent(s)*: a convergence agent that modifies the cluster state to match the desired configuration (unused in this configuration but still required in the cluster). <del> Read more about the [Flocker Cluster Architecture](https://docs.clusterhq.com/en/latest/concepts/architecture.html) at the [Flocker Documentation](https://docs.clusterhq.com/). </del> <ins> The Flocker cluster can be installed on the same nodes you are using for Kubernetes. For instance, you can install the Flocker Control Service on the same node as Kubernetes Master and Flocker Dataset/Container Agents on every Kubernetes Slave node. </ins> It is recommended to follow [Installing Flocker](https://docs.clusterhq.com/en/latest/install/index.html) and the instructions below to set-up the Flocker cluster to be used with Kubernetes.

We found that some users coming to the document were getting a little confused. It should include more detail about where to install Flocker services, how to set environment variables, that you can create volumes now with Docker CLI and give more detail in example including how to use it with a spec that includes a replication controller.


Question: "io/ioutil" "net/http" "os" <del> goruntime "runtime" </del> "strings" "time"

A new flag indicating how kube-proxy gets the number of CPUs, either from goruntime or . In kube-proxy, we can use to set node's , the resulting is . We have nodes where pods with none CPU policy and static CPU policy are running together. Pods like kube-proxy are running in none CPU policy by default, and they only gets CPUs left by static CPU policy pods. If static CPU policy pods take away most CPUs cores, kube-proxy can only get one or two cores, then will be set to a very small values. In this case, a lot packet failure can occur. When we set kube-proxy to work in static CPU policy, kube-proxy only gets the number of CPUs selected by the CPU policy, not the number of total CPUs on the node. This may surprise users who are only trying to limit kube-proxy's CPU usages, but resulting in some conntrack fulls faults. Please let me know if this enhancement is reasonable, so I can send a pr later. Thanks!
/sig network
Is it refer to deploy order of kube-proxy? If kube-proxy deploy first in node, will it get just one or two core?
By one or two I mean kube-proxy gets a small number of CPU cores, because most of CPU cores are allocated to static policy pods, sorry for the misunderstanding. The number of conntrack-max depends on the CPU cores when kube-proxy started, even kube-proxy was first deployed, it can get restarted or redeployed by some reasons. Then the conntrack-max will change.
yeah, when kube-proxy deploy firstly, Maybe it can see the whole cpu nums from shared cpu(/var/lib/kubelet/cpumanagestate when kubelet start with static cpu policy). But I think it is a bug for kube-proxy. The cpu info can also get from node resource info. So will it be suitable to add a flag to set cpu num?
Maybe, or we can add a flag for user to choose where kube-proxy should get CPU informations from. There maybe some multi-tenant scenarios where kube-proxy are deployed for per tenant so they would like to limit conntrack-max by CPU cores allocated to each tenant.
I think the simple fix is to replace with something that counts /proc/cpuinfo. I don't know of anyone running multiple proxies, and I don't think it would work well. The intention was "all the CPUs on the node". PRs welcome!
/triage accepted


Question: package common import ( <ins> "encoding/json" </ins> "fmt" "k8s.io/api/core/v1"

This issue contains a very basic mock test in order to start the conversation about what a good e2e test would look like for this endpoint. The code below is verified to be hitting the intended endpoint, show as per the queries in APIsnoop's live view of the cluster's audit logs. According to these query results from APIsnoop, we are lacking in tests for the follow endpoints related to Secrets. We chose the follow endpoints for this mock test. Test outline: a Secret in the default namespace with custom values Secrets in all namespaces search for Secret created in  the data in Secret  Secret  Secrets in all namespaces ensure that Secret  doesn't exist Discover useragents Display endpoints hit by the test Review the change in coverage If a test with these calls gets merged in, conformance will go up by 3 endpoints. /sig architecture /area conformance
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Yeah! We got a test written and a promotion (and we linked them to the mock-ticket that started it all). /close
Closing this issue. <detailsIn response to : Yeah! We got a test written and a promotion (and we linked them to the mock-ticket that started it all). /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: // kubectl command (begining with a space). func kubectlArgs() string { if *checkVersionSkew { <del> return " --match-server-version --v=4" </del> <ins> return " --match-server-version" </ins> } <del> return " --v=4" </del> <ins> return "" </ins> } func bashWrap(cmd string) string {

In case it isn't clear why this happened, it's because gcloud ships with an older version of kubectl (currently version 0.7.0). We keep it as up to date as we can, but it will always lag a release or so behind head.
Fixed by the revert.


Question: package v1 import ( <ins> "fmt" </ins> "reflect" "strings" "testing"

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: After creating a DaemonSet that uses node affinity to request only creating pods on nodes with a label matching a set of candidate values, pods get created on the appropriate nodes—and, fortunately, only on the appropriate nodes—but the pods get terminated and deleted quickly before they can start running. The cycle takes about ten seconds from pod creation through completion of deletion; the pod appears to be deleted within one second after its creation, but it takes about ten seconds for the deletion to complete. If I vary the DaemonSet's scheduling predicates, I find the following work to get pods running successfully: Use to choose a specific node. Use with an operator of "Exists" or "NotIn" to choose some nodes. Use with an operator of "In," but with only one value to choose some nodes. Use with a label that is not applied by the cloud provider. If I come up with my own label such as "special-hardware" and use kubectl label node to apply it, and use that label as a match expression key, it seems to work fine. It's the dynamically applied labels like "" and "" that trigger this problem. Without the above concessions, if I use with an operator of "In," but with more than one value, the pods don't start correctly. Note that this behavior is a regression from Kubernetes version 1.10.4. The same configuration works as intended in a cluster running that earlier version. What you expected to happen: After creating the DaemonSet, pods would start successfully on the nodes with label values matching one of the DaemonSet's candidates values. How to reproduce it (as minimally and precisely as possible): a DaemonSet that uses node affinity to request only creating pods on nodes with a label matching a set of candidate values. that at least one node has a label with a value that matches a member of that candidate set. kubectl get pods -o wide to observe that there are pods created on behalf of the DaemonSet on nodes that match the predicate, but they have status "Terminating." By watching the pods, you can see new pods arrive with status "Pending," then "ContainerCreating," then "Terminating," which they'll retain until deletion completes and a replacement arrives. a similar pod directly, without a supervising DaemonSet.

If the "" label is inconvenient within your cluster, instead consider using "" and a few hostnames as the candidate values. that one pod starts running successfully on one of the nodes that match that the predicate. that lone pod (for clarity), and vary the predicate on the DaemonSet to see which changes allow pods to start running successfully. Try each of these individually: Remove one of the values in the sequence. That is, leave only one candidate value. Change the operator to "NotIn," and adjust the values to select some subset of the nodes. Alternately, try "Exists" with no values. Remove the node affinity stanza's and add a node selector in its place: Anything else we need to know?: Here is the DaemonSet object as captured via kubectl get daemonset -o yaml: Here is one of the pods created on behalf of the DaemonSet: Here is one of the nodes on which pods like these should run: There some preceding discussion in the "sig-node" channel of the "Kubernetes" Slack team . Possibly related issues: , Possibly related PRs: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: AWS EC2 (g3.8xlarge instance, but other instance types exhibit the same behavior) OS (e.g. from /etc/os-release):Kernel (e.g. ):Install tools: kubeadm init, kubeadm join
/sig node /sig apps
To work around this problem, I found that I can use multiple match expressions, with one candidate value in each. This works because sibling match expressions are a disjunction.
if the kubelet is rejecting the pod, then this belongs to sig-node it might be reusing logic from the scheduler library, so looping in sig-scheduling for reference: /sig scheduling
I can reproduce it on kubeadm v1.11.0. Let me try to give a fix. /assign
GitHub didn't allow me to assign the following users: Huang-Wei. Note that only and repo collaborators can be assigned. <detailsIn response to : I can reproduce it on kubeadm v1.11.0. Let me try to give a fix. </details
I've found the root cause, and should be able to send out a PR next week.


Question: transformContext := authenticatedDataString(key) for { if err := preconditions.Check(key, origState.obj); err != nil { <del> return err </del> <ins> // If our data is already up to date, return the error if !mustCheckData { return err } // It's possible we were working with stale data // Actually fetch origState, err = getCurrentState() if err != nil { return err } mustCheckData = false // Retry continue </ins> } ret, ttl, err := s.updateState(origState, tryUpdate)

Which jobs are failing: pull-kubernetes-integration Which test(s) are failing: https://k8s- Not showing up in triage, oddly, but seen in individual failures: not sure if this specific to custom resources, but that's where I'm seeing the flakes /sig api-machinery /priority important-soon /area custom-resources
adding to the milestone until root-caused
/cc
I suspect there is some cache/race that keeps the old CR information around during the update from the CR that fails UID check during an update etcd log: few read (other other version defined by CRD. the port difference indicates a different client) the timestamp, the failure happened after the write a successful CR from another test in that file (TestValidateOnlyStatus, TestGeneration, etc) (different etcd key prefix from the same etcd log) few read (update) few read (the second version. I'm surprised that the port didn't change) few read
leaving in the milestone for visibility as we root cause whether this is a test-only issue or not
I spent a while trying to recreate this and could not (again). I made one of the tests hotloop with QPS 100 doing get/update of the status subresource, then ran that in parallel with the other integration tests in the package. No failures found after running for ~10 minutes. Updated description with testgrid link to locate recent flakes (https://k8s-), found a couple in the last week.
iiuc, the error comes from apiserver the UID against the . The requested obj is up-to-date since it was from etcd, but I didn't see mechanism that guarantees the watch cache to be up-to-date after a "delete-create" or "delete-create-get" sequence (the cacher implementation for Delete, Create and Get(rv="") go through etcd directly). Is this failure an expected behavior, and should we just retry on precondition failure?
the triage page seems to be stale, filed
talked with offline, we get it reproduced by adding some delay to the watch cache: as suggested preconditions should retry on stale watch data instead of surfacing such error, this is not a test issue. I will send a fix


Question: } cmd.Flags().BoolVar(&makeSymlinksFlag, "make-symlinks", makeSymlinksFlag, "create a symlink for each server in current directory") cmd.Flags().MarkHidden("make-symlinks") // hide this flag from appearing in servers' usage output <ins> cmd.Flags().MarkDeprecated("make-symlinks", "This feature will be removed in a later release.") </ins> for i := range commandFns { cmd.AddCommand(commandFns[i]())

Hyperkube supports a number of short aliases for commands, such as , , , , etc, rather than the longer , , , etc. There are comments indicating this is "unfortunate" and we still currently create these aliases in the hyperkube docker image: It'd be nice to simplify this, assuming we can find all uses of the short aliases and remove them. /kind cleanup Not sure which sig should own this, so tagging a few. /sig architecture cluster-lifecycle cli cc
interestingly, the command doesn't create the short aliases, though we also don't really use anywhere either.
I think we can just go ahead and start the deprecation cycle for this. WDYT? cc
cc as well
+1 to nuke it
/assign :wink: ok?
SGTM
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
/help
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I would like to work on it!
It seems like this issue was already solved in ?
I don't think the hyperkube code was changed in that PR, only the image?
Yes, thanks for your comment, can i try to fix it ?
I made a PR here: . Review/feedback/suggestions appreciated! :)
/assign /unassign
GitHub didn't allow me to assign the following users: Rand01ph.

Note that only and repo collaborators can be assigned and that issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign /unassign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-sig architecture cli /sig release Does anyone even still use hyperkube? Wouldn't they use kubeadm or k3s instead?
AFAIK, SAP (the company) still use hyperkube.
aks-engine deployer ( ) still uses hyperkube
Since this ticket is completed now. Can this todo be removed?
we are removing the whole thing :)


Question: srcs = [ "controller_test.go", "import_known_versions_test.go", <ins> "master_openapi_test.go", </ins> "master_test.go", ], library = ":go_default_library",

Haven't seen this one before and I am not sure it has any relation with the that hit it. https://k8s-
From what I can gather, it is being killed due to taking over 5 minutes which is roughly how long it takes (~300s) on successful unit builds.
bumping priority. I'm seeing this pretty frequently this morning. I see OpenAPI and associate you. You know something about this one?
Yes, I wrote this test. It was failing (timing out) for quite some time on jenkins but passing quite fast locally. I've disabled it but after a while it started passing and I assumed it was a configuration change on test machines. I will spend some time on it again to see if I can find the root cause and will disable it otherwise. This test is passing locally in about 15 second. Do you have any idea why it is timing out (or passing near to 300s) on jenkins machines? where do you see the successful test stats?
I see what you do. Locally it takes 25s, with bots it is 300s. My guess is that we are constrained by CPU and all the encode/decode is causing strain either on CPU or GC. The bots fail about 75% for me so I just keep retrying until success. Just had 2 this morning pass at 296s.
Curiously, it's taking about 20 seconds with Bazel. this might be worth looking at.
I think we run tests in Jenkins with , which might explain the slowness?
Nevermind, we hit it in a bazel run: https://k8s-
-race does kill the performance, 25s -166s which is on par with bazel.
is it possible to disable -race for this test? and should we do that if it is possible?
Now blocking the submit queue.
I think I am missing something here. I've checked some of the attached PRs. Links to the pkg/master folder in master branch has the change (new file called but non of the recent PRs I checked has the file in their branch: That means they didn't sync with master? What am I missing here?
OK. I see, the tests are patching the change of the PR into the master, so everybody should get that if they retest, but it look like my test is not blocking submit queue. there should be another issue.


Question: // no-op, we've been upgraded case resp.StatusCode < http.StatusOK || resp.StatusCode > http.StatusPartialContent: if !isStatusResponse { <del> return nil, false, r.transformUnstructuredResponseError(resp, req, body) </del> <ins> return Result{err: r.transformUnstructuredResponseError(resp, req, body)} </ins> } <del> return nil, false, errors.FromObject(&status) </del> <ins> return Result{err: errors.FromObject(&status)} </ins> } // If the server gave us a status back, look at what it was. success := resp.StatusCode >= http.StatusOK && resp.StatusCode <= http.StatusPartialContent if isStatusResponse && (status.Status != api.StatusSuccess && !success) { <del> // "Working" requests need to be handled specially. </del> // "Failed" requests are clearly just an error and it makes sense to return them as such. <del> return nil, false, errors.FromObject(&status) </del> <ins> return Result{err: errors.FromObject(&status)} </ins> } <del> created := resp.StatusCode == http.StatusCreated return body, created, nil </del> <ins> return Result{ body:    body, created: resp.StatusCode == http.StatusCreated, codec:   r.codec, } </ins> } // transformUnstructuredResponseError handles an error from the server that is not in a structured form.

Storing r.req / r.resp internally (for use from DoRaw and Do) is probably not the right factoring. There should be a common method doRaw that returns request and body.


Question: Selector                     string LastAppliedConfigurationList []string OutputFormat                 string <ins> All                          bool </ins> Factory                      cmdutil.Factory Out                          io.Writer ErrOut                       io.Writer

It actually doesn't support --all.
/assign
ohh, thanks, will make a pr to address that right now..


Question: // Validate compute resource typename. // Refer to docs/design/resources.md for more details. func validateResourceName(value string, fldPath *field.Path) field.ErrorList { <del> allErrs := field.ErrorList{} for _, msg := range validation.IsQualifiedName(value) { allErrs = append(allErrs, field.Invalid(fldPath, value, msg)) } </del> <ins> allErrs := apivalidation.ValidateQualifiedName(value, fldPath) </ins> if len(allErrs) != 0 { return allErrs }

After got merged, we have new helper function here: It will be good for us to use this simpler helper function to replace previous function. /good-first-issue
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : After got merged, we have new helper function here: </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig usability
I would like to take this one /assign
I worked in on this as well, but kept your changed files untouched (as written in my PR description). same for you. I hope this is OK for you.
This should be reopened until and got merged (my PR excluded the changes from the other PRs).
/reopen per last comment
Reopened this issue. <detailsIn response to : This should be reopened until and got merged (my PR excluded the changes from the other PRs). per last comment Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
can you assign me
/assign
/assign
I'd love to work on this issue.

/assign
Hi, , I am a first commitor for kubernetes and I have committed a PR in , could you help me to review it, or give me some suggestions. Thanks a lot.
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: }, nil, NewDefaultClientConfigLoadingRules(), promptedCredentials{}} ) <ins> // getDefaultServer returns a default setting for DefaultClientConfig // DEPRECATED func getDefaultServer() string { if server := os.Getenv("KUBERNETES_MASTER"); len(server) > 0 { return server } return "http://localhost:8080" } </ins> // ClientConfig is used to make it easy to get an api server client type ClientConfig interface { // RawConfig returns the merged result of all overrides

What happened: I am using a command like below in GitHub actions: I get the following error: What you expected to happen: Since I am using flag, it should work without requiring a valid kubeconfig file. How to reproduce it (as minimally and precisely as possible): Just delete your local kubeconfig file and try the above command. Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig cli
/assign /kind regression
This likely breaks some scripted usage of kubectl
This is related to /assign
Or
Yes, sorry I mentioned you in this issue. Because my first thought is this brings and which will check the config file existence but in fact that's not the root cause. Actually the root cause This checks config has the server or not. And with this , this function will return error when there's no local config file. I think there might be two solutions? the similar logic like to every , , ..... when or is set. Seems not a good idea. reverse this to have the default server which is better than ? By the way, is equals ? Since This works only for .
The plan I agreed on with Jordan is that I will revert since that seems to be the main culprit behind all these problems. I'll add a test ensuring this works as required. This will allow us to backport the fix to 1.18. For 1.19 I'll put together a proper fix supporting and flags.


Question: nodes:        []string{"machine1", "machine2"}, services:     []*v1.Service{{Spec: v1.ServiceSpec{Selector: map[string]string{"baz": "blah"}}}}, sss:          []*apps.StatefulSet{{Spec: apps.StatefulSetSpec{Selector: &metav1.LabelSelector{MatchLabels: map[string]string{"foo": "bar"}}}}}, <del> expectedList: []schedulerapi.HostPriority{{Host: "machine1", Score: 0}, {Host: "machine2", Score: 5}}, name:         "service with partial pod label matches with service and replica set", </del> <ins> expectedList: []schedulerapi.HostPriority{{Host: "machine1", Score: 0}, {Host: "machine2", Score: 0}}, name:         "service with partial pod label matches with service and stateful set", </ins> }, { pod: &v1.Pod{ObjectMeta: metav1.ObjectMeta{Labels: map[string]string{"foo": "bar", "bar": "foo"}, OwnerReferences: controllerRef("ReplicationController", "name", "abc123")}},

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: /kind bug What you expected to happen: Problem 1: Predicates filter nodes. Existing pods on those nodes will be not be counted when calculating max pods per (zone or node) resulting in imbalanced cluster. Eg: removes nodes which cannot . If any of the pods are already scheduled on this node, they are not considered when counting in . Problem 2: When there are 2 selectors(service and replication controller), it is sufficient to match any one selector for distribution. This creates imbalance ]. Pods from previous deploys matches and are counted when distributing pods across zones/nodes (Even though they do not match ) . These pods will be deleted. After the deploy completes, the cluster is imbalanced - by zone and/or pods per node. How to reproduce it (as minimally and precisely as possible): Problem 1: Get pods to get scheduled on nodes with very high cpu utilization [Existing cpu utilization + 1 new pod will result in almost equal to 0]. GeneralPredicate will drop this node after scheduling the first pod. This will lead to over utilization of an already loaded zone. If there are enough nodes like this one, it will create a pile on effect resulting in most pods scheduled in an already loaded zone. Problem 2: Have 2 selectors and deploy 30 pods across 3 zones. Twice. Anything else we need to know?: Environment: Kubernetes version (use ): /sig scheduling <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
re Problem 1: for per node, we only need to compare current nodes as they will share the same maxPods. for per zone, we need to account other nodes, e.g. because of resource predicates. re problem 2: a little about your case :(
Problem 1: : It does not matter if the denominator is all nodes or current nodes. One with higher utilization will still end up with a lower score. : It is critical that the denominator is all nodes. Else we are ending up with highly imbalanced cluster. Problem 2: Here is an example list of selectors.

All pods matching any one service selector are counted in So in our case, old pods are counted during scheduling. But they get killed during the deployment. If the maxSurge and maxUnavailable is high enough, you will end up with an imbalanced zone after deployment.
Gentle ping.
/assign bsalamat
Re: Problem 1 - You can also reproduce this by using pod anti-affinity. Zone score is calculated only based on nodes filtered by predicates. IMO zone score should be calculated using all nodes.
I will try to separate this into two PRs. Any other suggestions welcome.
I am working Problem 1. Looks like I have to open a new issue.


Question: timeoutErr, ok := err.(*TimeoutError) if ok { if utilfeature.DefaultFeatureGate.Enabled(features.ExecProbeTimeout) { <del> return probe.Failure, string(data), nil </del> <ins> // When exec probe timeout, data is empty, so we should return timeoutErr.Error() as the stdout. return probe.Failure, timeoutErr.Error(), nil </ins> } klog.Warningf("Exec probe timed out after %s but ExecProbeTimeout feature gate was disabled", timeoutErr.Timeout())

I was trying a scenario where liveness/readiness probe command run inside the container for long time('sleep 10 && exit 10') with shorter timeoutSecod value say 2 Sec. As expected container restarted because of timeout but when i check the events in pod describe it stated only unhealthy and in events actual reason for probe failure is not printed "liveness probe failed:<'no reason printed'". Without clear logs or pod events description it is very difficult to debug and understand what is the failure. Below is the example manifest file to reproduce the issue. apiVersion: v1 kind: Pod metadata: labels: test: liveness name: liveness-exec spec: containers: name: liveness image: args: /bin/sh -c touch /tmp/healthy; sleep 30; rm -rf /tmp/healthy; sleep 600 readinessProbe: exec: command: [/bin/sh, -c , 'sleep 10 && exit10'] initialDelaySeconds: 5 periodSeconds: 5 timeoutSeconds: 2 livenessProbe: exec: command: [/bin/sh, -c ,'sleep 5 && exit10'] initialDelaySeconds: 5 periodSeconds: 5 timeoutSeconds: 2 It would be better if actual reason or something for probe failure printed in pod describe events incase of liveness/readiness probe gets timeout. Example like below in pod describe events: Warning Unhealthy 3s (x2 over 8s) kubelet Liveness probe failed: "probe failed due to timeout " Warning Unhealthy 3s (x2 over 8s) kubelet Readiness probe failed: "probe failed due to timeout " ! It would be better if actual reason or something for probe failure printed in pod describe events incase of liveness/readiness probe gets timeout. Example like below in pod describe events: Warning Unhealthy 3s (x2 over 8s) kubelet Liveness probe failed: "probe failed due to timeout " Warning Unhealthy 3s (x2 over 8s) kubelet Readiness probe failed: "probe failed due to timeout " Below is the example manifest file to reproduce the issue.

apiVersion: v1 kind: Pod metadata: labels: test: liveness name: liveness-exec spec: containers: name: liveness image: args: /bin/sh -c touch /tmp/healthy; sleep 30; rm -rf /tmp/healthy; sleep 600 readinessProbe: exec: command: [/bin/sh, -c , 'sleep 10 && exit10'] initialDelaySeconds: 5 periodSeconds: 5 timeoutSeconds: 2 livenessProbe: exec: command: [/bin/sh, -c ,'sleep 5 && exit10'] initialDelaySeconds: 5 periodSeconds: 5 timeoutSeconds: 2 Create a pod using above manifest file and do pod describe (kubectl describe pod <pod-name ) and check for the events. ! No response kubectl version Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2021-08-31T06:59:53Z", GoVersion:"go1.16.4", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2021-08-31T06:52:44Z", GoVersion:"go1.16.4", Compiler:"gc", Platform:"linux/amd64"} openstack <details</details<details</detailscontainerd No response
I tried to fix one similar issue in
/sig node
/triage accepted
can I work on this issue? Thanks very much. /assign
We've experienced this issue with clusters on v1.20. The for that version mention the following change: Since v1.20 is seemingly when exec probe timeouts started being enforced, would it be possible to have this fix for this issue backported to all minor versions since then? Thanks!
bump on that question
What do you think of? Thanks very much.


Question: return nil, false, fmt.Errorf("Spec does not reference a NFS volume type") } <ins> func getServerFromSource(source *v1.NFSVolumeSource) string { if netutil.IsIPv6String(source.Server) { return fmt.Sprintf("[%s]", source.Server) } return source.Server } </ins>

In bare-metal IPv6 environment, any pod consuming storage resource is not able to run. The storage should be consumable, and the pod should become in status. Running the in a bare-metal IPv6 configurations. I have replaced the cloud-based storage resource with a mounted to a folder on my localhost. Steps alongside logs are below :- $kubectl get svc $kubectl describe svc nfs-server $kubectl describe pv nfs $kubectl get pod -w $kubectl get pod -w $kubectl describe pod nfs-busybox-6r96h I have built the master branch to make sure it was not fixed up to this moment.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage
/sig storage /tirage accepted /assign


Question: ginkgo.By("verifying that pod has the correct nodeSelector") framework.ExpectEqual(pod.Spec.NodeSelector["kubernetes.io/os"], "windows") <del> cmd := []string{"cmd", "/c", "echo windows-volume-test", ">", filePath} stdoutRW, stderrRW, errRW := f.ExecCommandInContainerWithFullOutput(podName, rwcontainerName, cmd...) msg := fmt.Sprintf("cmd: %v, stdout: %q, stderr: %q", cmd, stdoutRW, stderrRW) </del> <ins> ginkgo.By("verifying that pod can write to a volume with read/write access") writecmd := []string{"cmd", "/c", "echo windows-volume-test", ">", filePath} stdoutRW, stderrRW, errRW := f.ExecCommandInContainerWithFullOutput(podName, rwcontainerName, writecmd...) msg := fmt.Sprintf("cmd: %v, stdout: %q, stderr: %q", writecmd, stdoutRW, stderrRW) </ins> framework.ExpectNoError(errRW, msg) <del> _, stderr, _ := f.ExecCommandInContainerWithFullOutput(podName, containerName, cmd...) </del> <ins> ginkgo.By("verifying that pod will get an error when writing to a volume that is readonly") _, stderr, _ := f.ExecCommandInContainerWithFullOutput(podName, containerName, writecmd...) </ins> framework.ExpectEqual(stderr, "Access is denied.") <ins> ginkgo.By("verifying that pod can read from the the volume that is readonly") </ins> readcmd := []string{"cmd", "/c", "type", filePath} readout, readerr, err := f.ExecCommandInContainerWithFullOutput(podName, containerName, readcmd...) readmsg := fmt.Sprintf("cmd: %v, stdout: %q, stderr: %q", readcmd, readout, readerr)

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI -- Kubernetes e2e suite: [sig-windows] [Feature:Windows] Windows volume mounts check volume mount permissions container should have readOnly permissions on hostMapPath Feb 23 14:59:47.591: FAIL: cmd: [cmd /c echo windows-volume-test C:tmptest-file], stdout: "", stderr: "Access is denied."
/sig windows
/assign
Looks like it's a flaky test instead of a failing test /cc
mentioned this could be related to permissions: There might be two things happening, the test isn't actually waiting for pod to start (which is why it passes sometimes) and but when it does start it fails the test due to the permissions
` is used for creating the pods. According to the comments, the Pods should be up and running:
On successful run, I see the following in the kubelet logs: I believe we're execing into the pod when the container is actually getting deleted by containerd but the pod is running which is not giving a correct indication of the command. I am modifying the test to ensure we would wait till all the containers are Ready and log the stdout and stderr to see if we're noticing issues. I tried manually running this test on a Windows node but couldn't reproduce it.
More examples of flaking runs:
/assign
I ran the test continuously: I am guessing this might be due to some of resources constraints we have seen when other tests runs at the same time (such as ). looking into it further...
Maybe we can convert it to a serial test case?
maybe, slowly looking into is some more... I find it interesting that which essentially does the same thing
Here's another:
Figured out what is happening: If the test runs before it leaves a file behind that the new test doesn't have permissions to. To repo: Then run:
/triage accepted


Question: <del> serve_hostname ============== </del> <ins> ## serve_hostname </ins> <del> Util app to serve your hostname on TCP and/or UDP.  Useful for testing. </del> <ins> This is a small util app to serve your hostname on TCP and/or UDP.  Useful for testing. The `serve_hostname` Makefile supports multiple architectures, which means it may cross-compile and build an docker image easily. Arch-specific busybox images serve as base images. If you are releasing a new version, please bump the `TAG` value in the `Makefile` before building the images. ## How to release: ``` # Build cross-platform binaries $ make all-push # Build for linux/amd64 (default) $ make push ARCH=amd64 # ---> gcr.io/google_containers/serve_hostname-amd64:TAG $ make push ARCH=arm # ---> gcr.io/google_containers/serve_hostname-arm:TAG $ make push ARCH=arm64 # ---> gcr.io/google_containers/serve_hostname-arm64:TAG $ make push ARCH=ppc64le # ---> gcr.io/google_containers/serve_hostname-ppc64le:TAG ``` Of course, if you don't want to push the images, run `make all-container` or `make container ARCH={target_arch}` instead. </ins> [![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/contrib/for-demos/serve_hostname/README.md?pixel)]()

cross-build , Supported arch are: amd64(default) arm arm64 ppc64le PR is . This issue and PR() is an effort to achieve multiarch Kubernetes()
Done.


Question: "github.com/coreos/go-etcd/etcd" ) <del> type fakeClientGetSet struct { get func(key string, sort, recursive bool) (*etcd.Response, error) set func(key, value string, ttl uint64) (*etcd.Response, error) } </del> type TestResource struct { api.TypeMeta   `json:",inline"` api.ObjectMeta `json:"metadata"`

kubectl returns resources in a random order, which makes it hard to look for things when users have a lot resources. Returning resources in deterministic order should be a helpful and trivial change (hopefully). We have an issue about pagination , but that's a larger and separate issue.
We should be cause to not do this on the * namespace - I think thats a special case. No disagreement for other list calls
What is the default etcd response order? Last created or last modified? Sent from my iPhone
I think the order is by key if sort param is given. If not given, the order is by last created (maybe, but we shouldn't trust it). It seems we are not trying to get sorted result from etcd.
Today, I think it's random order in the child key space in etcd (because the node structure uses a map), so you get golang map iteration order without sort. I don't know if sort on server is any cheaper than sort on client today, so we may want to do this in rest / registry rather than by passing etcd.
Apart from performance, sorting in clients means we can also take control of how responses are sorted, so for * namespace, we can sort by namespace, then by resource ID. , Clayton Coleman wrote: Master of Electrical and Computer Engineering Carnegie Mellon University
We can always hint sort to the registry storage layer in the future, so sorting on the client now won't kill us.
Just took a closer look at our rest/registry code to implement the sort (in client), but implementation-wise, it seems much easier to pass sort to etcd (one line change). Also, namespace is a prefix on resource, so sort across all namespace with etcd key will essentially sort resources by namespace then by Name. If we are to implement sort on client side, we can either: sort package: have each list object, ie.e PodList, ControllerList, implement sort interface. (We won't do this). have ListMeta implement sort interface, if there is a way to find enclosing structs (so we can use items field) have api.Object implement sort interface, verify object is list before sorting. sort algorithm ourself, and do it in rest. Is there other golang or kubernetes idom that I haven't been aware of that can be applied to this use case?

----- Original Message ----- I think sorting it on etcd is fine for now - that's the simplest possible thing that works.
Is there any material perf impact in etcd to enable it? Sent from my iPhone
Probably not material. Can always switch away if necessary
Famous last words :-) Sent from my iPhone
Got my hat ready


Question: }, } <del> preconditionFunc := func(store cache.Store) (bool, error) { _, exists, err := store.Get(&metav1.ObjectMeta{Namespace: info.Namespace, Name: info.Name}) if err != nil { return true, err } if !exists { // We need to make sure we see the object in the cache before we start waiting for events // or we would be waiting for the timeout if such object didn't exist. return true, apierrors.NewNotFound(mapping.Resource.GroupResource(), info.Name) } return false, nil } </del> // if the rollout isn't done yet, keep watching deployment status ctx, cancel := watchtools.ContextWithOptionalTimeout(context.Background(), o.Timeout) intr := interrupt.New(nil, cancel) return intr.Run(func() error { <del> _, err = watchtools.UntilWithSync(ctx, lw, &unstructured.Unstructured{}, preconditionFunc, func(e watch.Event) (bool, error) { </del> <ins> _, err = watchtools.UntilWithSync(ctx, lw, &unstructured.Unstructured{}, nil, func(e watch.Event) (bool, error) { </ins> switch t := e.Type; t { case watch.Added, watch.Modified: status, done, err := statusViewer.Status(e.Object.(runtime.Unstructured), o.Revision)

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: gce-master-scale-correctness Which test(s) are failing: [sig-cli] Kubectl client Simple pod should return command exit codes Since when has it been failing: 02/05 Testgrid link: https://k8s- Reason for failure: Anything else we need to know: /sig cli
/priority critical-urgent /assign
The test seems to be flaking quite a lot recently ! Friendly ping, sig-cli, could you take a look?
FTR, these are really blocking tests and it's getting more urgent given the forthcoming 1.18 release
Full log of the failure: It looks like the finishes with error. Strange, I thought is supposed to create a pod. Would really appreciate if someone from sig-cli could take a look
4 out of last 8 runs failed: https://k8s- /sig scalability
Another flaky, related test:
SIG CLI Leads -- any update on this? This is affecting scalability signal. /assign
-- Please keep tabs on this one.
yep, monitoring
It looks like there's a race between when the pod gets created and when we try to connect to it, thus we're seeing but further down in the logs the pod is , I'll be looking more into ti.
Thanks, Any update on this?
/cc
Sorry I got sidetracked yesterday and I'm back investigating this just now...
Probable fix in
Will continue to monitor here and reopen if needed :+1:
Unfortunately - it seems that the problem still exists - the test failed today (an hour ago): :( ^^
I'll check it, thx Wojtek for pointing that out.
- I'm not familiar with kubectl code, but I took a quick look and what I found is that this waitForPod() thing is happening only for pods with RestartPolicyNever: I must be missing something here, but for the case that failed it's not really the case, which means we don't wait for it?
This particular one is using , see the actual invocation that failed: That's why I started with this approach.
Sorry - the one for which the bug was originally opened.

But the today`s failure is actually slightly different (the same test, different case - sorry): The restart policy for this one is
Gotcha, so it looks like we progressed, I'll check the possible paths for this one, something tells me we're missing another timeout somewhere :wink:
OK - I think I know what is going on here. Here is what is happening (for the test case from two comments above): the test is setting "-i", so interactive mode is switched on that option triggers that Attach option defaults to true: over the Run() command, we get to the handleAttachPod: handleAttachPod is calling waitForPod And waitForPod is broken. The problem is the preconditionFunc: which is below passed to the UntilWithSync function UntilWithSync underneath is creating an informer and calling that function against its state: But the informer is initialized from apiserver watchcache, so it may be lagging. In large clusters we are observing lags in the order of small hundreds of milliseconds and that seem to be enough. What is happening is that: informer is initialized (but the pod creation is not yet reflected in it) the preconditionFunc is called the state doesn't contain that pod (yet) the whole thing fails with that error I actually believe, that just removing the preconditionFunc should actually solve the problem. I opened for that.
^^ - please take a look at the PR that I assigned to you (it's literally one-liner as of now - if you want to add tests or anything like that, feel free to pick it up).
Actually I went ahead, and fixed couple other commands too - so it's no longer one-liner. But still short one :)


Question: "bytes" "encoding/json" "fmt" <ins> "github.com/pkg/errors" "math" </ins> "strconv" "strings" "k8s.io/kubernetes/test/e2e/framework" ) <del> // IPerfResults is a struct that stores some IPerfResult </del> <ins> const ( megabyte = 1024 * 1024 ) // IPerfResults is a struct that stores some IPerfCSVResult </ins> type IPerfResults struct { BandwidthMap map[string]int64 } <del> // IPerfResult struct modelling an iperf record.... </del> <ins> // IPerfCSVResult struct modelling an iperf record.... </ins> // 20160314154239,172.17.0.3,34152,172.17.0.2,5001,3,0.0-10.0,33843707904,27074774092 <del> type IPerfResult struct { </del> <ins> type IPerfCSVResult struct { </ins> date          string // field 1 in the csv cli           string // field 2 in the csv cliPort       int64  // ...

What happened: The iperf tests were written a while ago but i think havent been updated. They measure bandwidth over multiple pods, but they probably should also have a test which reports individual pod's bandwidth between one pod over a SEP. They should pretty print a table so its easy to see exactly what is going on with the bandwidth They should use iperf3 (need to check that) Maybe we should have permutations for udp and tcp? The current output is something like: What you expected to happen: That running the tests would result in a table such as those that you can get from iperf3, i.e. How to reproduce it: run the e2es on any cluster, Environment: Kubernetes version (use ): master, any version
/sig network
/assign
/assign
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/remove-triage unresolved
I've taken a first pass at this , working on resolving a couple issues then we can circle back to determine how else it could be improved!
Cool c/p the new test output so we can compare when u get a chance
ah yess. much better then what we have now.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: framework.ExpectEqual(resp.StatusCode, 301, "The resp.StatusCode returned: %d", resp.StatusCode) } <del> func checkPodStatus(f *framework.Framework, label string) func() (bool, error) { return func() (bool, error) { var err error list, err := f.ClientSet.CoreV1().Pods(f.Namespace.Name).List(context.TODO(), metav1.ListOptions{ LabelSelector: label}) if err != nil { return false, err } if list.Items[0].Status.Phase != "Running" { framework.Logf("Pod Quantity: %d Status: %s", len(list.Items), list.Items[0].Status.Phase) return false, err } framework.Logf("Pod Status: %v", list.Items[0].Status.Phase) return true, nil } } </del> // validateProxyVerbRequest checks that a http request to a pod // or service was valid for any http verb. Requires agnhost image // with porter --json-response

pull-kubernetes-e2e-kind-ipv6 - Not sure. Time out on waiting Pod to be started. No response /sig network
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: r, err := d.serverResourcesForGroupVersion(gv) if err != nil { <del> utilruntime.HandleError(fmt.Errorf("couldn't get resource list for %v: %v", gv, err)) </del> <ins> // Don't log "empty response" as an error; it is a common response for metrics. if _, emptyErr := err.(*emptyResponseError); emptyErr { // Log at same verbosity as disk cache. klog.V(3).Infof("%v", err) } else { utilruntime.HandleError(fmt.Errorf("couldn't get resource list for %v: %v", gv, err)) } </ins> } resultLock.Lock()

I think this got stale and closed because it was posted on but is still very relevant: The created by memory.NewMemCacheClient(...) returns an error when calling ServerResourcesForGroupVersion if the APIResources list returned is empty, as seen here: Our particular case came up due to the API group being registered with the API server, but having no resources. If using like to discover with zero current resources there should not be an error returned. Do a discovery with a against a Kubernetes cluster that has at least one with zero resources. The condition on the server that triggers the described issue can be expressed as e.g.: I would be good to get to know the reasoning behind the emptiness check in the , maybe there is good reason for it. <detailsClient Version: v1.20.15 Server Version: v1.22.8-gke.202 </details<detailsGoogle Cloud (GKE) </detailsNo response No response No response No response
/sig api-machinery
/cc /triage accepted
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at .

/lifecycle stale [1]:
Reviving this issue based on This behavior has become quite annoying for the custom/external metrics aggregated APIs since after , client-go is now producing a very spammy error log when an aggregated API is found without resource. This is a very common scenario with the metric APIs since metrics are dynamically registered as Resources to the aggregated APIs so on startup the aggregated apiserver may not expose any resource and that's completely normal. It is even more common for implementations of these APIs that register metric Resources from CRDs as CRs tend to be created later than APIServices and the aggregated apiserver. I can see a few path forward to improve the situation for us: the error if we consider that having no resources exposed by an aggregated API is acceptable the verbosity of the log to just show it in debugging scenarios somehow update to defer registration until we have at least one Metric resource. could you perhaps share your insight on that one please? /remove-lifecycle stale
Hello folks! Any update on this topic? The message is really spammy
Removing triage accepted manually since there was no update on this for more than 6 months. /remove-triage accepted
For context, I traced it back and check has been considered an error since the first iteration of the in-memory discovery client in : It is unexpected for a built-in or CRD groupversion to have no resources, but for aggregated-apiservice I can see it being possible that the object is created and set up for discovery before the aggregated-apiservice is ready.
I think this is which introduces (but maybe I'm wrong). There is also in that repo about the same issue after upgrading the kube-client pkg. in custom-metrics repo be correct from Kubernetes pov? IMO we need to find a solution soon or the amount of noise will increase once people bump their deps and tooling (, kubectl is affected, and any tool which uses kube-client is affected)
/triage accepted