Question: "github.com/GoogleCloudPlatform/kubernetes/pkg/clientauth" nodeControllerPkg "github.com/GoogleCloudPlatform/kubernetes/pkg/cloudprovider/controller" "github.com/GoogleCloudPlatform/kubernetes/pkg/controller" <ins> "github.com/GoogleCloudPlatform/kubernetes/pkg/credentialprovider" </ins> "github.com/GoogleCloudPlatform/kubernetes/pkg/kubelet" "github.com/GoogleCloudPlatform/kubernetes/pkg/kubelet/config" "github.com/GoogleCloudPlatform/kubernetes/pkg/kubelet/dockertools"

As per We should look for the .dockercfg in three places: ./.dockercfg, ~/.dockercfg, and $rootDir/.dockercfg
I'll tackle this.
Bunch of questions: we also add a flag that overrides the search mechanism suggested in this issue? Perhaps, ? to the user's homedir requires we use os/user [0] . The os/user package and cross-compiling don't play nice [1]. So, if we add to the search path, cross-compiled kubelet binaries will not actually be able to take advantage of it. Is this an issue? is the proper lookup order? I suggest: if provided, , , . walking through the lookups, do we stop at the first file that exists, or the first file that exists and is parsed as a valid ? [0] [1] /cc
I would absolutely love a --dockercfg option, and it would also be helpful if it logged which .dockercfg it ultimately used.
1) Seems like YAGNI to me 2) Is all of os/user broken? We will eventually want actual user support. I'm fine with dropping it for now. 3) I would say $rootDir/.dockercfg, ./.dockercfg (assuming we drop ~) 4) First successful parse sounds good to me. And can we NOT log errors if files don't exist :)
would it make sense to default to the kubelet config directory if provided rather than the working directory of the kubelet process? then no extra flag required.
What ever happened with this idea?
/cc
It never got high enough on my weekend TODO list :( I can keep telling myself that I need to do it, or we can find someone who has a bit more time than me.
I can probably get to this today.
Cool, thank you! , Mike Danese wrote:


Question: The example combines a web frontend, a redis master for storage and a replicated set of redis slaves. ### Step Zero: Prerequisites <del> This example assumes that you have forked the repository and [turned up a Kubernetes cluster](https://github.com/GoogleCloudPlatform/kubernetes-new#setup). </del> <ins> This example assumes that you have forked the repository and [turned up a Kubernetes cluster](https://github.com/GoogleCloudPlatform/kubernetes-new#setup): $ cd kubernetes $ hack/dev-build-and-up.sh </ins> ### Step One: Turn up the redis master. <del> Create a file named `redis-master.json`, this file is describes a single pod, which runs a redis key-value server in a container. </del> <ins> Create a file named `redis-master.json` describing a single pod, which runs a redis key-value server in a container. </ins> ```javascript {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: I am running an admission webhook using If there it has been hours since the last mutating request to the webhook, then the next mutating request will take many seconds to be processed. This may even take long enough to have the request time out. What you expected to happen: A request takes approximately the same amount of time to be processed regardless of how long it has been since the last request. How to reproduce it (as minimally and precisely as possible): Create an API server for handling a new API group. Wait a couple hours. Send a CREATE request to the API server. Observe that the CREATE request takes a few seconds to complete. Anything else we need to know?: This is the offending code. It must loop through once for every second since the last request. Environment: Kubernetes version (use ): 1.19.0
/sig api-machinery
I thought we'd already merged a fix for this from before the release?
I assume that the fix was supposed to be From my understanding, those changes only are in effect if the apiserver is running with priority and fairness enabled. In my situation, I do not have that enabled.
Ack, if we have the problem in both places we need the fix in both places :)
/triage accepted
I think we have fixed this in , do you guys agree?


Question: // to avoid potential orphaned devices upon re-registration devices := make(map[string]pluginapi.Device) for _, device := range old.getDevices() { <ins> device.Health = pluginapi.Unhealthy </ins> devices[device.ID] = device } existingDevs = devices

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug /sig node What happened: i register device by my deviceplugin. It works good. Then i stopmy plugin, k8s mark resources Unhealthy as expect. But when i restart plugin. The resource does not recover, remain allocatable 0. What you expected to happen: Resource turns healthy when plugin restart. How to reproduce it (as minimally and precisely as possible): k8s device plugin device plugin device plugin Anything else we need to know?: Environment: Kubernetes version (use ): v1.10.1 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Ubuntu Kernel (e.g. ): Install tools: Others:
devicemanger set devices unhealthy when socket closed. But devices in endpoint, devices remain health, and will not call callback function when watched new device request (due to same health), and devicemanger' devices will never recover.
I was able to reproduce this. would this be a proper fix or we need something more sophisticated?
May be less destructive variant would be better?
/area hw-accelerators /cc
I thought when the new device plugin re-registers, we would create a new endpoint.
Ah I think I see your point. The fix you proposed in sounds reasonable, but we may need to expose a function from with proper locking.
Btw, when we are fixing this, I think we should extend the existing test/e2enode/[gpu] to verify the fix by restarting the device plugin and waiting for resource allocatable change. We once had that test coverage in test/e2e/scheduling/nvidia- but somehow lost it through some code changes.
/cc
[Posting offline question] Why is state stored both in endpoints and device manager?
I have a vague memory that we had some similar discussion before and am trying to remember the reason why the code was structured this way. I think one reason is perhaps that we want to prevent a single buggy device plugin from DOS device manager with flooding ListAndWatch updates by having its endpoint filter out unnecessary updates. do you remember any other reasons?
Endpoints stores device state whereas manager stores "API server" state. These two states are very similar but different.

One always has the latest device state. The other is in charge of reporting what devices changed to an unhealthy state and to delete the resource after a certain set of time. So we're not really storing the state twice :)
It might be simpler to make endpoints lean and store all state within the device manager to avoid synchronisation issues. Renaud Gaubert <:
I doubt that any of the fix that mentioned will work. Problem is this -- devices are being copied with health as "Healthy" from old endpoint to new endpoint, And then marking those devices unhealthy in device manager. So what was saying is correct that we create a new endpoint and in the fix , we should update devices health in new endpoint. This can be done at two places-- 1. before copying devices to new endpoint or 2. in the goroutine after e.stop() as is suggesting. I would prefer option 1 because in option2 there could be a race if device health gets updated to "unhealthy" after got invoked in . I will post a fix soon today.
/assign
will handle e2e in a separate PR shortly.
device manager to avoid synchronisation issues. then grpc calls will have to be protected with mutex.
Your fix looks quite implicit in my opinion. It sets device health to Unhealthy in addEndPoint, which is called from Register. This might look confusing. It looks like we make new endpoint unhealthy when we register new resource. My fix synchronises device manager and endpoint health exactly when resource is set to unhealthy state. The code is covered by mutex, so I don't see why it would race. Even if it would I'd argue that your fix could cause future confusions and should be reworked/ thanks for the suggestion to move this functionality into I agree with this. Also calling it from markResourceUnhealthy may look better than my sketchy proposal. Thoughts?
I tested my fix before posting here. It worked. As for your comment - devices are copied from old to new endpoint with the old state: and then passed to newEndpointImpl as existingDevs variable.
My guess is that you might not have hit the race condition in your local testing. If devices are marked unhealthy from within goroutine, like you suggested, this will race against the thread which will invoke for the new endpoint.

And if first thread got late in marking devices unhealthy, callback from the will not be invoked. mutex that you talked about is not protecting from the race i described. This mutex is protecting device manager database only. Though i dont think it is confusing to mark devices from old endpoint to be unhealthy because anyways old endpoint is going to be deleted and new endpoint may really not be having devices in healthy state. But still if there is any confusion may be we can extend existing comment smthng like this:
See my comment on your PR :
Is this fixed in 1.10 and/or 1.11?
I believe 1.9.5, 1.9.6, 1.10.0 1.10.1, 1.10.2 are the only releases affected by this bug.
Thanks


Question: import ( "fmt" "io" <ins> "time" "github.com/jonboulle/clockwork" </ins> "github.com/spf13/cobra" "k8s.io/kubernetes/pkg/api" "k8s.io/kubernetes/pkg/api/errors" <ins> "k8s.io/kubernetes/pkg/api/meta" </ins> "k8s.io/kubernetes/pkg/kubectl" cmdutil "k8s.io/kubernetes/pkg/kubectl/cmd/util" "k8s.io/kubernetes/pkg/kubectl/resource"

From an e2e test failure: The command should retry resource version mismatches like scale and rollingupdate. cc
/cc fyi.
From perspective , Isn't the failure intentional (i.e. optimistic mutate) ? or... maybe I'm (probably) missing something. I would think its the user's job to retry... ?
The desired behavior that we settled on previously is to detect version mismatches at the client and error. User can force replacement using --force. See , and .
What is the mismatch here? If it is just resourceVersion, kubectl should retry. It should only bail on a meaningful conflict. We're working on eliminating flaky tests (and flaky parts of the system). Also, that log message isn't very useful.
per discussion on , we can't determine the correct behavior when the resource version changes without more information, so best to catch version mismatch at the client and bail, and let the user replace the object with --force if they want. We can also make the error message less verbose.
I don't see any mention of resourceVersion in . It shouldn't be serialized in the annotation, because the user shouldn't specify it in their config. The resourceVersion-based precondition is intended to guard against concurrent read-modify-write operations. kubectl apply should be able retry the entire diff&patch in the case of this type of conflict. That said, it's not obvious to me from the log message what the actual problem was in this case.
I'm sorry, my mistake. You're right that resourceVersion is not in the annotation, and that we should retry on resourceVersion failure. I was confusing this with the api version change issue discussed in . This is actually a much simpler issue, in the sense that it's an optimistic lock conflict. If we encounter a lock conflict, then we should fetch a fresh copy of the object, and try again to apply the new configuration supplied by the user. If we don't get a merge conflict when we resubmit, then all's well. If we get a merge conflict because the object has changed out from under the user, then we report it in those terms, and the user can decide what to do (e.g., replace using --force). Does this fit your expectation of how it should be handled?
This is causing test flake:
yes, your understanding is correct.

Do you have time to fix this? We're trying to get all our tests fixed. Additionally this sounds like a legit system bug, not just a flaky test problem.
Maybe. Swamped right now, but can fit it in given enough notice. When is the fix needed?
asap. We'll find someone else.
This should just be a generic GET/Update retry loop, eg:
Go for it. Jack Greenfield | Google , Prashanth B wrote:
Chao, do you have cycles to take a look?
I thought Patch is retried in the server. I'll investigate.
Patch is retried at server-side five times before an error is returned: , so I expect this issue to occur very rarely. In fact, it has passed last ~400 kubernetes-gce-e2e tests. If we want to make this test less flaky, I'd recommend increase the number of the retries at the server-side. What do you guys think?
It'll fail if an actual conflicting change was made, though. Do we know what the nature of the conflict was? Maybe first we need to add some log messages to figure out what the conflict was next time it occurs. , Chao Xu wrote:
This happened last week:
you are right. I thought an actual conflicting change would result in a different error message but it's not. I'll add the logs.
In I a log entry if there is a meaningful conflict. That said, I don't think we have a meaningful conflict in this particular e2e test. After taking a closer look at the implementation of , I think the submitted is precise, so we shouldn't retry at the client side if server reports a meaningful conflict. [edit] And if the conflict is not meaningful, we should just rely on the server-side patch handler to retry.
does the milestone on this need to be changed to v1.2?
How often is the test flaking?
Last time I checked, I'll check it again.
Last time I checked the results was two days ago, in the past 48 hours, the test has consecutively passed another 124 builds ( to ).
I only see 3 reports of failures since October, so I'm going to say, no, it doesn't need to be fixed in 1.2.
any progress on this now?

I returned back to the kubectl apply issues, and hope I can fix all apply related issues before 1.3.0 release.
I'll try to check the flake rate again. IMO if the flake rate is non-negligible, we should increase the retry limits of Patch, or more aggressively, just remove the limit.
Infinite retry still scares me, mostly because we may have failure modes that could race for a very long time. , Chao Xu wrote:
I don't think anyone has worked on this. Though the original failing test/logs weren't recorded, it appears to be a failure of "should apply a new configuration to an existing RC": There is a retry loop in the Patch handler in apiserver, but if resourceVersion is specified, the retry loop will always fail in the case of a conflict. (We should detect that and bail out, but I don't think we do.) In the case of such a conflict, apply should diff again, create another patch, and try again. I agree with that it shouldn't retry forever. We should use the same retry limits we use for other commands.
we have an infinite loop in GuaranteedUpdate (), maybe we should apply a retry limit there as well?
Agree. Just note that the test flake doesn't specify resourceVersion.
Since this is causing flakey test failures, it is something we should focus on. Is this something I should pick up?
FYI: has a PR out for this under review
Making this a P3 (no release block) since I looks like we haven't seen many instances since Feb 2016


Question: return true, nil } <ins> // Ensure that a resource has the correct tags // If it has no tags, we assume that this was a problem caused by an error in between creation and tagging, // and we add the tags.  If it has a different cluster's tags, that is an error. func (s *AWSCloud) ensureClusterTags(resourceID string, tags []*ec2.Tag) error { actualTags := make(map[string]string) for _, tag := range tags { actualTags[aws.StringValue(tag.Key)] = aws.StringValue(tag.Value) } addTags := make(map[string]string) for k, expected := range s.filterTags { actual := actualTags[k] if actual == expected { continue } if actual == "" { glog.Warningf("Resource %q was missing expected cluster tag %q.  Will add (with value %q)", resourceID, k, expected) addTags[k] = expected } else { return fmt.Errorf("resource %q has tag belonging to another cluster: %q=%q (expected %q)", resourceID, k, actual, expected) } } if err := s.createTags(resourceID, addTags); err != nil { return fmt.Errorf("error adding missing tags to resource %q: %v", resourceID, err) } return nil } </ins> // Makes sure the security group exists // Returns the security group id or error func (s *AWSCloud) ensureSecurityGroup(name string, description string, vpcID string) (string, error) {

After setting up a few services, I noticed the following output in : I assume the service tried to configure an ELB in the past (not sure when or how or what details it tried to use) and has kept trying and now can't even use the AWS API. How can I either: 1) tell it to please stop so that I can use my AWS account again 2) fix it so that it does configure the ELB /cc
Sorry about the problem. Looks like we need a sleep / backoff policy here! To get you up and running again, if you redeploy the nginx service without Type=LoadBalancer, we will (should) stop trying to create a load balancer, and you should get out of the AWS API penalty-box. Sorry about this, and thanks for the report. Looks like this is pretty important to fix. Do you happen to know why you can't create an ELB? My guess is that you have hit your quota?
I'll remove the service, wait for the penalty-box to expire, then build it again so that we can see what the real error was.
Adding the LoadBalancer service back in, I'm seeing:
I took a few different steps and seem to have gotten it both working in different ways: First I deleted the service because it was erroring, but I noticed the count was still going up. So I deleted the security group it said was a duplicate and then it finished setting the ELB up! That meant I needed to manually clean up the ELB resources and security groups because k8 thought it was already deleted. After cleaning up all the resources in AWS again, I the service again and it seemed to work fine! I'm not sure how/why it initially confused itself with the duplicate SG, but it doesn't necessarily seem to happen every time. The code could likely just, in the event that it sees the SG already exists, continue on with the setup. I could send you a PR to handle that if you'd like!
I have a PR that I think covers this: . I thought it was a one-off error, but it sounds like it wasn't. Thanks so much for pausing & retrying also - that was super-helpful. Even if we think covers this, let's leave this issue open to track the need for a backoff to avoid overloading the AWS API.

(And presumably this aggressive retrying happens on other clouds also, though maybe they have overload a sleep built in).
Certainly looks like should fix it. I'm still wondering why I'd see it failing literally for hours in a loop though. If it was a problem with AWS being eventually consistent, then after an hour I'd think that first check, looking to see if the securityGroup existed, would start to return the group! I don't know the k8 code well yet, though so perhaps it was looping in a different place that wasn't running that code to see if the security group already existed.
you're right; my explanation wasn't entirely correct... So security group names are unique per VPC. We also have a per-cluster name, and we filter all our AWS API interactions to only select resources tagged with this name. If we had a security group name collision across two k8s clusters in the same VPC, that would be bad. Thankfully the security group name we're creating includes a UUID, so this "won't happen". However, the problem is that we created the security group, and then failed to tag it. But we were filtering based on our cluster. So our "describe security groups" operation would never see the SG, because it wasn't appropriately tagged. We have a few workarounds to make this much less likely, and I have opened the "big picture" issue . And I'm going to think about whether we should in fact be filtering when we DescribeSecurityGroups - I think the answer is no in the case where the name has a UUID in it.
Ah! That totally makes sense. The race between resource creation and resource tagging can created "invisible" resources if the process of tagging fails. Maybe an anti-entropy function that can look at all the resources and normalize them. In the case of invisible resources, you can just delete them and hope that recreating them won't fail the tagging process.
I see this as well when frequently deploying/testing. Unfortunately, it happens far too often (running v1.0.0).
+1
I've been having a similar issue to this, Kubernetes can't create a load balancer as it says the security group already exists. The workaround I've found is to manually delete the security group and wait for Kubernetes to successfully create it again.

I've generally had the problem when quickly deleting a service with a and then creating it again (I see now I should use ). I wonder if it is something to do with AWS reassigning the same load balancer back to you if you haven't finished deleting it coupled with the tagging issue mentioned earlier.
+1
cc
I had an email discussion with on this and he said that what's remaining shouldn't be considered a blocker for 1.1, so I'll move it to 1.2-candidate.
Can you summarize what's remaining to do for this issue? Should we just close it in favor of ?
I see you the v1.2 milestone -- will you be able to look into this in the next few days?


Question: } } <ins> // TODO: Remove in 1.6. Returns if kubectl is older than v1.5.0 func isOldKubectl(userAgent string) bool { // example userAgent string: kubectl-1.3/v1.3.8 (linux/amd64) kubernetes/e328d5b if !strings.Contains(userAgent, "kubectl") { return false } userAgent = strings.Split(userAgent, " ")[0] subs := strings.Split(userAgent, "/") if len(subs) != 2 { return false } kubectlVersion, versionErr := version.Parse(subs[1]) if versionErr != nil { return false } return kubectlVersion.LT(version.MustParse("v1.5.0")) } // TODO: Remove in 1.6. This is for backward compatibility with 1.4 kubectl. // See https://github.com/kubernetes/kubernetes/issues/35791 var groupsWithNewVersionsIn1_5 = sets.NewString("apps", "policy") // TODO: Remove in 1.6. func filterAPIGroups(req *restful.Request, groups []unversioned.APIGroup) []unversioned.APIGroup { if !isOldKubectl(req.HeaderParameter("User-Agent")) { return groups } // hide API group that has new versions added in 1.5. var ret []unversioned.APIGroup for _, group := range groups { if groupsWithNewVersionsIn1_5.Has(group.Name) { continue } ret = append(ret, group) } return ret } </ins> // RootAPIHandler returns a handler which will list the provided groups and versions as available. func RootAPIHandler(s runtime.NegotiatedSerializer, f func(req *restful.Request) []unversioned.APIGroup) restful.RouteFunction { return func(req *restful.Request, resp *restful.Response) { <del> writeNegotiated(s, unversioned.GroupVersion{}, resp.ResponseWriter, req.Request, http.StatusOK, &unversioned.APIGroupList{Groups: f(req)}) </del> <ins> writeNegotiated(s, unversioned.GroupVersion{}, resp.ResponseWriter, req.Request, http.StatusOK, &unversioned.APIGroupList{Groups: filterAPIGroups(req, f(req))}) </ins> } }

Is this expected? Does it mean that we cannot drop and replace alpha api without waiting for the new gcloud version?
cc:
cc
cc:
My money is on the TPR discovery code that is trying to identify TPR versions and hacks them into the list of registered APIs.
I guess it's not on HEAD? Do you have the PR#?
had a PR moving StatefulSet from v1alpha1 to v1beta1 and she didn't have the same problem. It's interesting why graduating PDB will have the problem
My PR has similar failure but different symptom. I created a GCE cluster locally. doesn't give me that error, but addon-manager isn't working properly. From addon-manager container log, I saw similar error:
Ah, I was using new kubectl
was using the newly built kubectl so we didn't see the problem at first. Addon-manager is using 1.4.4 kubectl, so we observed problem later.
Hmm, if the problem is the old versions of kubectl unable to dealing with new REST endpoints, then the problem is hard. I'll take a look.
is correct. See this .
I thought the TPR code was inert by default and required a flag to kick in
It seems the flag is enabled for most commands:
We need to support 1 minor version skew for kubectl, so we have to workaround in the server. We can hide those new APIVersions in the discovery API from old kubectl. I'm going to send a PR. In the mean time, if someone can come up with a better idea.
kubectl bug is fixed in , need to cherrypick the change to 1.4. is working on server side temp fix
should fix the problem.
The issue was "fixed" because the "unable to register" error was ignored (not returned, just logged), see:
that line in doesn't seem like a fix... it seems like it's leaving the client restmapper in an unknown state, which seems bad to me
Yeah it's not really a fix
Looks like kubectl should try to register the version to the existing group. IMO this isn't a blocker for
can you make sure there's a 1.5 issue tracking the failure to register the new versions into existing groups correctly, or we'll have a related issue in the future
Filed to track this


Question: fmt.Sprintf("%s::%s::%s", f.Namespace.Name, pod1, "busybox-container"): boundedSample(0, 100), }), <del> "container_memory_working_set_bytes": gstruct.MatchAllElements(containerID, gstruct.Elements{ </del> <ins> "container_memory_working_set_bytes": gstruct.MatchElements(containerID, gstruct.IgnoreExtras, gstruct.Elements{ </ins> fmt.Sprintf("%s::%s::%s", f.Namespace.Name, pod0, "busybox-container"): boundedSample(10*e2evolume.Kb, 80*e2evolume.Mb), fmt.Sprintf("%s::%s::%s", f.Namespace.Name, pod1, "busybox-container"): boundedSample(10*e2evolume.Kb, 80*e2evolume.Mb), }),

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: Which test(s) are flaking: Testgrid link: Reason for failure: Anything else we need to know: /cc /sig node /priority important-soon /milestone v1.20 Looks like this might be related to
looks like my fault, whoops. /assign As far as I can tell the test is sensitive to the environment it runs in. I'll try to make it match less, and if that doesn't work it may need to become serial.
attempt based on my understanding,


Question: case cache.Deleted: err := s.balancer.EnsureTCPLoadBalancerDeleted(s.loadBalancerName(service), s.zone.Region) if err != nil { <ins> s.eventRecorder.Event(service, "deleting loadbalancer failed", err.Error()) </ins> return err, retryable } s.cache.delete(namespacedName.String())

If a LoadBalancer cannot be created (e.g. quota limit hit), there is currently no way to know why other than reviewing the logs (?). We could put an status message / object into LoadBalancerStatus.
We could make an event for this too Justin Santa Barbara < :
+1 for event
I think that is reasonable, but we need to be careful how we define it. Event also sounds good. , Justin Santa Barbara < :
I'm in favor of status as well as an event. The pattern I recommend is an array of conditions, similar to NodeCondition, where each contains at least a Status, Reason, and Message.
+1 for an event. I'd argue this is important enough for usability that we should get it into v1.
How often does this failure mode occur?
I unfortunately don't have stats, but it can happen for a handful of reasons. I think these are the most likely: Cloud provider unavailability/errors Ran out of quota from the cloud provider The cluster is using a cloud provider that doesn't support external load balancers I expect the third one is most common by far. We could perhaps move that into the API object validation rather than writing events to handle that particularly case, but using events would solve all the cases together.


Question: "//vendor/gopkg.in/gcfg.v1:go_default_library", "//vendor/k8s.io/api/core/v1:go_default_library", "//vendor/k8s.io/apimachinery/pkg/types:go_default_library", <del> "//vendor/k8s.io/apimachinery/pkg/util/wait:go_default_library", </del> "//vendor/k8s.io/client-go/informers:go_default_library", "//vendor/k8s.io/client-go/tools/cache:go_default_library", ],

In 1.9 vSphere cloud provider creates a new client, new shared informer factory and lists nodes IMO, the cloud provider should not create it's own client at all. GCE uses , called by controller-manager to initialize the cloud provider. This way, GCE does not need extra policy and it shares the informer with all the controllers. is called only in controller-manager process, i.e. for provisioning and attach/detach. Do you need the informer really in kubelet and/or API server? It's quite odd to list all nodes there. /kind bug /sig storage /sig auth
issue regarding is getting fixed with creating and binding with ServiceAccount. See is there a way in kubernetes, we can fix this without assigning permissions to serviceaccount? We will look into SetInformer interface GCE uses.
cloud provider specific roles should be applied externally as we prepare for moving cloud provider specific components out of tree. had feedback on the way that was wired in that would be good to adjust before copying that pattern
thanks for the pointer to old issue. I reworked this one to remove vsphere's own client (and policy).
/assign rohitjogvmw
GitHub didn't allow me to assign the following users: rohitjogvmw. Note that only and repo collaborators can be assigned. <detailsIn response to : /assign rohitjogvmw Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Running into this same issue (manifested as "Kubernetes node nodeVmDetail details is empty" error upon pvc build using vsphere). Commenting to watch progress. Happy to help debug if needed.
/assign divyenpatel

Here if we with , then we are only able to watch the latest change. In this case if the key is modified after the returns but before the we call the , then we will miss the event. To avoid the window time, we'd better specify the to be the index in the response of the . BTW, I think if we use with , we can probably eliminate the first call.
This is fixed in


Question: "//vendor/gopkg.in/gcfg.v1:go_default_library", "//vendor/k8s.io/api/core/v1:go_default_library", "//vendor/k8s.io/apimachinery/pkg/types:go_default_library", <del> "//vendor/k8s.io/apimachinery/pkg/util/wait:go_default_library", </del> "//vendor/k8s.io/client-go/informers:go_default_library", "//vendor/k8s.io/client-go/tools/cache:go_default_library", ],

Here if we with , then we are only able to watch the latest change. In this case if the key is modified after the returns but before the we call the , then we will miss the event. To avoid the window time, we'd better specify the to be the index in the response of the . BTW, I think if we use with , we can probably eliminate the first call.
This is fixed in


Question: { name: "valid Merge", primaryRegistry: Registry{ <del> "pluginFactory1": NewMockNoopPluginFactory(), </del> <ins> "pluginFactory1": m1, </ins> }, registryToMerge: Registry{ <del> "pluginFactory2": NewMockNoopPluginFactory(), </del> <ins> "pluginFactory2": m2, </ins> }, expected: Registry{ <del> "pluginFactory1": NewMockNoopPluginFactory(), "pluginFactory2": NewMockNoopPluginFactory(), </del> <ins> "pluginFactory1": m1, "pluginFactory2": m2, </ins> }, shouldError: false, }, { name: "Merge duplicate factories", primaryRegistry: Registry{ <del> "pluginFactory1": NewMockNoopPluginFactory(), </del> <ins> "pluginFactory1": m1, </ins> }, registryToMerge: Registry{ <del> "pluginFactory1": NewMockNoopPluginFactory(), </del> <ins> "pluginFactory1": m2, </ins> }, expected: Registry{ <del> "pluginFactory1": NewMockNoopPluginFactory(), </del> <ins> "pluginFactory1": m1, </ins> }, shouldError: true, },

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --Below are the failures while running unit test with golang in master branch. Reference : $ go version go version devel go1.17- Thu Apr 22 01:40:02 2021 +0000 linux/ppc64le Note: Occurs in x86 architecture as well. The tests should Pass as with stable go git clone cd kubernetes make test KUBERACE=-race WHAT=./pkg/scheduler/framework/runtime/ Master branch of kubernetes repo. OS: Linux Observed failure on ppc64le and x86 architectures Below is the attached log:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I request help on getting this triaged. Is there a procedure to follow to get this accepted as a bug?
The Go compiler developer has identified the bugs in the test that is causing it to fail. They are listed in the last comment in the Go issue before closing it.
So the correct way to compare function would be something like would you test it in your env?
The issue still exists. Can you please reopen?
I am not sure if you can access this link for our CI job. Can you please check? https://prow.ppc64le- The above tests have been failing every day in our jobs!
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
downloads go from url="${GOLANGVERSION}.linux-" downloads go from "curl -X GET http://build-?project=build_commit -o " I suspect the postsubmit-master-golang-kubernetes-unit-test-ppc64le failed for wrong golang version.

failed for other problems. The TestMerge is not failed. For ppc64le failures, there are several issues in golang: (fixed yesterday)
Yes the go versions for both the jobs are different and it is intentional: : It is meant to run Kubernetes unit tests with the stable version of golang : It is meant to run Kubernetes unit tests with golang built from the master branch of As I mentioned in the description of this issue, these unit test cases are failing only on golang that is built from master branch and passing with stable versions of golang. And as Lynn mentioned in , the golang team has concluded that these failures are due to the bugs in the test case. And the golang issues that you mentioned do not seem to be related to the failures mentioned in this issue.
With , is false Per , do you mean that we should change to ?
No. In general comparing is the right thing to do. But if the Value is actually a , which is not comparable, comparing will panic.
/sig scheduling
Note from golang team about this bug is here:


Question: less, err := isLess(iField, jField) if err != nil { <del> klog.Fatalf("Field %s in %T is an unsortable type: %s, err: %v", t.field, t.parsedRows, iField.Kind().String(), err) </del> <ins> klog.Exitf("Field %s in %T is an unsortable type: %s, err: %v", t.field, t.parsedRows, iField.Kind().String(), err) </ins> } return less }

I am getting a crash when trying to sort pod with kubectl on Not crashing and at least an error message <details<summaryCrash log</summary</details- Kubernetes version (use ): Cloud provider or hardware configuration: tried on kind latest and openshift 4.7 OS (e.g: ): debian sid Kernel (e.g. ): 5.11.x Install tools: NA Network plugin and version (if this is a network-related bug): NA Others: kubectl compiled from tip branch (as of this bug report) with GO version 1.16.5
/sig cli
Hey I'm not able to reproduce the crash, although I do get an error on running However, the following works: Could you try this and check if this crashes as well?
It definitively crashes it, did you test it in a repo where there is actual pod in there? that error would not show if there is nothing.
Yep Hmm, interesting. Will try to reproduce it again
Sorry, not sure how to remove the label but this is expected. When sorting if the type is not expected it will output errors using over which will logs to the FATAL, ERROR, WARNING, and INFO logs, including a stack trace of all running goroutines, then calls os.Exit(255). Arguments are handled in the manner of fmt.Printf; a newline is appended if missing.
Those labels are not set on the issue: <detailsIn response to : /remove-kind bug Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
You need to have multiple pods to reproduce this error. It won't happen with just one pod. For example: The error says the type can't be sorted, so that is the problem: As said, this error is generated intentionally.

Here is the code: The problem stems from the fact that  has no way to return an error, so it logs a fatal error which os.Exits. It seems like the problem is that the error message is too verbose. One solution could be to change the code to only print the error message and exit, instead of using which is what is outputting the stack trace. It probably doesn't make sense to dump a stack trace for something that is a user error.


Question: cd ${TEMP_DIR} && sed -i.back "s|BASEIMAGE|${BASEIMAGE}|g" Dockerfile <del> docker build --pull -t ${REGISTRY}/conformance-${ARCH}:${VERSION} ${TEMP_DIR} rm -rf "${TEMP_DIR}" push: build docker push ${REGISTRY}/conformance-${ARCH}:${VERSION} </del> <ins> DOCKER_CLI_EXPERIMENTAL=enabled docker buildx build --pull --platform linux/$(ARCH) $(OUTPUT) -t ${REGISTRY}/conformance-${ARCH}:${VERSION} ${TEMP_DIR} </ins> ifeq ($(ARCH),amd64) docker rmi ${REGISTRY}/conformance:${VERSION} 2>/dev/null || true <del> docker tag ${REGISTRY}/conformance-${ARCH}:${VERSION} ${REGISTRY}/conformance:${VERSION} docker push ${REGISTRY}/conformance:${VERSION} </del> <ins> DOCKER_CLI_EXPERIMENTAL=enabled docker buildx build --pull --platform linux/$(ARCH) $(OUTPUT) -t ${REGISTRY}/conformance:${VERSION} ${TEMP_DIR} </ins> endif <ins> rm -rf "${TEMP_DIR}" push: OUTPUT=--push push: build </ins> .PHONY: build push all

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: produces an error: What you expected to happen: Conformance tests should be launched. Examining image contents indicates executables are of the correct architecture. How to reproduce it (as minimally and precisely as possible): From aarch64 machine: Anything else we need to know?: Works with v1.17.4: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: OS (e.g: ):Kernel (e.g. ):Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig Architecture /sig Testing
/sig release /remove-sig architecture
any thoughts on this issue at all? We use this to test K3d and are blocked on releasing anything in the 1.18 line because of it.
I don't have expertise in the image release pipeline, I'd suggest reaching out to
Hi Did you run this test in an arm64 virtual machine? If so, I have fixed this issue in cadvisor side. This issue is caused by cadvisor. Currently, the version of cadvisor in kubernetes is v0.36.1-0.-. There is a defect in this version, it gets the wrong machine info and reports num_cores=0 on Arm64 guest. I have delivered the PR into cadvisor in 22 days ago. And this patch was merged in 10 days ago. Please see my pr as reference: Currently, I'm waiting a new tag of cadvisor, so that I can use this new tag in kubernetes and fix this issue in k8s side. Please see my issue: Thanks.
Currently we only have one Arm64 conformance test is alive, this conformance test was run on 2 AWS arm64 instances https://k8s- All other VM-based Arm64 conformance tests are down.
/area conformance
Thanks for taking a look just to be sure it is not a VM issue I fired up an a1-metal arm64 aws instance.

Please note I am using sonobuoy for running tests rather than kubetest, I tried to find if kubetest is using the conformance image for testing and from a quick check maybe not. The error is not in the conformance test itself, but launching the conformance test image on arm64. The is . Tried again with with 1.18.2 and got the same error: still works ok for the v1.17 branch: It looks like the build chain for images has changed: So there are now six less layers of the conformance image for the 1.18 branch. Please also note how the arm64 images show amd64 as the architecture for both branches. This has been problematic for trying to air-gap arm64 properly and is a systemic issue for many of the projects on gcr (and elsewhere).
Hi I see. I'll check it. Thanks.
/reopen This issue is still unfixed as of today. here is a reproducer that helps doing the reverse test on a x8664 machine: So this is an image where the base os is actually arm64. starting with v1.18.0 and still as of today (v1.18.5-rc.0 and v1.19.0-beta.2) the uploaded -arm64 image is actually an amd64 image. There must be a typo somewhere. testable this way: This is supposed to generate the 'exec file error', but it doesn't, which indicates that it is actually an x8664 image (amd64).
You can't reopen an issue/PR unless you authored it or you are a collaborator. <detailsIn response to : /reopen x8664 # podman pull # podman run 66652b9ed7f16a9bc848038ddb00159f3e2d22d8c2bb1ba050913309daf101c9 cat /etc/os-release exec user process caused "exec format error" Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
IIRC the fix was reverted due to not working in the release pipeline. I did say that would hurt :upsidedownface:
/reopen
Reopened this issue.

<detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten This is still blocking conformance testing on arm64, so I hope it can be considered open.
Looks like to me this is a very complicated issue which is related with GCB, Moby Buildkit, it's not easy for me to reproduce and some trouble shooting by myself since I am not able to access GCB. Just wanna know is there any follow up to make this PR merge again? , we do have some needs to run the conformance testing on ARM64 .
SIG testing does not use this image (we prefer not to run the test runner as a pod given our clusters may be unreliable since we're testing changes and we want to avoid e.g. the pod being evicted =data loss), however this is currently used by the CNCF conformance program (which is CNCF / SIG arch) and sonobuoy (which is a vmware project). A bunch of our images are shipping with buildkit / buildx currently at HEAD, something like number could probably be brought back if someone wanted to work on this. Currently I'm not going to have time myself.
/cc
cc for arm64 image :(
xref /assign
It looks like that the issue is not in k/release, since we only build/stage the images via . Then we push the images and annotate them to their corresponding architecture.

Looking at the staged artifacts, for example for v1.20.1 arm64: Then we can see that also kube-proxy seems to have this issue: I assume that the fix is somewhere in the build scripts, where I'm now looking into. Beside that, all other architectures (arm, ppc64le, ...) are affected by this bug as well (for the same images only). Edit: Looks like the distroless images are fine, whereas the debian based images (proxy / conformance) are pointing to the wrong architecture. Conformance seems to always use , which might be the issue here. Edit: Done with my experiments. will change the already open PR according to our findings from the slack conversation (), whereas we have to change the used base images for kube-proxy to match the corresponding target architecture (like using for )


Question: $ kubectl proxy --port=8011 --www=./local/www/ // Run a proxy to kubernetes apiserver, changing the api prefix to k8s-api <del> // This makes e.g. the pods api available at localhost:8011/k8s-api/v1beta1/pods/ $ kubectl proxy --api-prefix=k8s-api` </del> <ins> // This makes e.g. the pods api available at localhost:8011/k8s-api/v1beta3/pods/ $ kubectl proxy --api-prefix=/k8s-api` </ins> ) func NewCmdProxy(f *cmdutil.Factory, out io.Writer) *cobra.Command {

Without , works just fine, With '--api-prefix=xxx-api', it stops work: My version: could you take a look? Thanks.
--api-prefix was in . could you take a look? This appears to be completely broken, so we should fix it or rip it out.
Reading the code and I find the way to fix it. You need to use , note the slashes. I will fix the man page.
What's an example command line that works?
then
Ah, the default flag value has leading and trailing slashes: The example in the documentation is wrong: Re-opening for doc fix.
The trailing slash is not necessary. and duplicate each other.


Question: generate-etcd-cert "${KUBE_TEMP}/cfssl" "${host}" "peer" "peer" pushd "${KUBE_TEMP}/cfssl" <del> ETCD_CA_KEY_BASE64=$(base64 "ca-key.pem" | tr -d 'rn') </del> <ins> ETCD_CA_KEY_BASE64=$(base64 $BASE64_INPUT_FLAG "ca-key.pem" | tr -d 'rn') </ins> ETCD_CA_CERT_BASE64=$(gzip -c "ca.pem" | base64 | tr -d 'rn') <del> ETCD_PEER_KEY_BASE64=$(base64 "peer-key.pem" | tr -d 'rn') </del> <ins> ETCD_PEER_KEY_BASE64=$(base64 $BASE64_INPUT_FLAG "peer-key.pem" | tr -d 'rn') </ins> ETCD_PEER_CERT_BASE64=$(gzip -c "peer.pem" | base64 | tr -d 'rn') popd }

under kubernetes project and running get error: ! I am running on darwin arm64 running get this: if running on linux:
/area test
/sig cli
/sig testing /area test-infra /remove-sig cli /remove-area test
FWIW You pretty much have to install gnu coreutils to develop here, many scripts will actually refuse to run if the gnu version of utils are not available. That said, we should fix this.
/triage accepted


Question: // interesting OS-based errors happen earlier. crossPlatforms = []string{ "linux/amd64", "windows/386", <del> "darwin/amd64", "linux/arm", "linux/386", "windows/amd64", "linux/arm64", "linux/ppc64le", "linux/s390x", </del> <ins> "darwin/amd64", "darwin/arm64", "linux/arm", "linux/386", "windows/amd64", "linux/arm64", "linux/ppc64le", "linux/s390x", </ins> } // directories we always ignore

<!-- Please only use this template for submitting enhancement requests --What would you like to be : When trying to build kubectl from tip () on a M1 Macbook: Why is this needed: M1 MacBooks are a new major client platform that many developers will use. brew listed kubectl (kubernetes-cli) is one of the major packages still not supported on M1 () Since it's supported by go, it's only required to add it to the supported client list Locally I managed to add support using the following patch:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cli architecture
/sig release
Any updates on this? cheers
you can watch the PR:
it works, nice!!!
Done!!!


Question: <ins> apiVersion: v1 kind: List items: - apiVersion: v1 kind: Service metadata: name: test-service labels: name: test-service spec: ports: - port: 80 selector: name: test-rc - apiVersion: v1 kind: ReplicationController metadata: name: test-rc labels: name: test-rc spec: replicas: 1 template: metadata: labels: name: test-rc spec: containers: - name: test-rc image: nginx ports: - containerPort: 80 </ins>

kubectl create, delete, etc. can already take a List object. kubectl apply can take the list object when it does creation, but it gives an (ugly) error when it's configuring the objects. Then... expected: Actual:
what's you ?
0-alpha.5
Sounds like a bug. Are you interested in taking a look
sure I will take a look during this weekend.
I think the root cause is , the actually is , in the origin , is not , but after the , this object is nil. Would you like to double check this?
I'll take a look.
if you have make sure about this, please let me know. Thanks :)
Is there any ways we could get this into 1.2? It's actually making our config generation quite painful to have to make a separate file for every object. It looks like got pretty far? would you like to make a PR that we can review?
I don't see how. It could have made 1.2 if a PR had been sent within a few days of the issue being filed.
I unassigned all issues assigned to you. Please only assign issues that are actively being worked on, so that others can know which issues are available to be picked up.
My apologies. If I'm not mistaken, this issue was filed after the code freeze for 1.2. I didn't realize that it was time critical or that it could make the release at that time. However, I should have asked. Sorry for the inconvenience. Thank you. I still plan to address those issues for the next release, as discussed, but will assign each one to myself as I work on it.
I created which fixes this. I'm really hoping we can get this into 1.2 since it's a pretty small, isolated fix but a pretty big bug that is complicating our config generation system. Let me know what else I can do to help.
Thanks for the PR. We're short on reviewer bandwidth at the moment. I assigned it to Clayton for review or delegation.


Question: // DefaultOffAdmissionPlugins get admission plugins off by default for kube-apiserver. func DefaultOffAdmissionPlugins() sets.String { defaultOnPlugins := sets.NewString( <del> lifecycle.PluginName,                //NamespaceLifecycle limitranger.PluginName,              //LimitRanger serviceaccount.PluginName,           //ServiceAccount setdefault.PluginName,               //DefaultStorageClass resize.PluginName,                   //PersistentVolumeClaimResize defaulttolerationseconds.PluginName, //DefaultTolerationSeconds mutatingwebhook.PluginName,          //MutatingAdmissionWebhook validatingwebhook.PluginName,        //ValidatingAdmissionWebhook resourcequota.PluginName,            //ResourceQuota </del> <ins> lifecycle.PluginName,                    //NamespaceLifecycle limitranger.PluginName,                  //LimitRanger serviceaccount.PluginName,               //ServiceAccount setdefault.PluginName,                   //DefaultStorageClass resize.PluginName,                       //PersistentVolumeClaimResize defaulttolerationseconds.PluginName,     //DefaultTolerationSeconds mutatingwebhook.PluginName,              //MutatingAdmissionWebhook validatingwebhook.PluginName,            //ValidatingAdmissionWebhook resourcequota.PluginName,                //ResourceQuota storageobjectinuseprotection.PluginName, //StorageObjectInUseProtection </ins> ) if utilfeature.DefaultFeatureGate.Enabled(features.PodPriority) {

What happened: The preparation step of the e2e tests were failed sometimes on my local environment like What you expected to happen: These tests can be passed without any errors. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): v1.13.1 Cloud provider or hardware configuration: OpenStack OS (e.g. from /etc/os-release): 16.04.4 LTS (Xenial Xerus) Kernel (e.g. ): 4.4.0-138-generic Install tools: kubeadm Others:
/sig storage
After discussing on the PR I have known I need to enable StorageObjectInUseProtection plugin of admission controller which is disabled by default. The corresponding e2e tests require the plugin is enabled, so I think it is better to enable the plugin by default for most users.
Investigation of StorageObjectInUseProtection plugin of admission controller under kubernetes/kubernetes: cluster/centos/config- Enabled cluster/gce/config- Enabled cluster/gce/config- Enabled cluster/kubemark/gce/config- Enabled hack/local-up- Enabled cluster/juju/layers/kubernetes- Disabled This is the reason why we don't face this issue on the upstream CI system, and I it on my local environment which is deployed with kubeadm.


Question: }, DeleteFunc: ssc.enqueueStatefulSet, }, <del> statefulSetResyncPeriod, </del> ) ssc.setLister = setInformer.Lister() ssc.setListerSynced = setInformer.Informer().HasSynced

<!-- Please only use this template for submitting enhancement requests --/area controller-manager What would you like to be : Don't resync statefulset in statefulset controller. Why is this needed: There are 2000+ sts in my k8s cluster, thus when i create a new sts, it takes a lot of time to create it's pod, i found that every sts will cost 40ms to sync it, then 2000 sts will cost 80s, and there is also a statefulSetResyncPeriod to resync all sts, I don't know why there need to have statefulSetResyncPeriod to resync all sts? Can we not resync sts and just watch it's changes?
/sig api-machinery
/remove-sig api-machinery /sig apps /sig scalability
the resync pattern is followed by all controllers to account for bugs in watch mechanism or missed notification due to any other reason. I dont think the individual controllers resync period is configurable or we could increase it . Also i am not sure if the current scale limits of 2000 StatefulSets is supported FYI
I have three cluster, A has 20 sts, B has 1300 sts and C has 2300 sts. I set kube-controller-manager loglevel to 4 and find that the cost of processing every sts is 2ms,30ms,45ms. Can we record the processed object into a processed cache and every time we get a new object to precess, we can compare it with the item in the cache, if they are same, then we don't need to sync it.
The "resync pattern" doesn't help with watch bugs (resync is done purely from internal cache, so in hypothetical situation that we missed watch event, it won't help at all). Resync-period should basically be used only in controllers that also sync some external things (like load-balancers), and watching changes of those is impossible. Unless I'm not missing anything, resync period can be safely dropped in this case.
I think all the controllers have the same problem because of the sharedinformerfactory has a default resync period if we don't set it to 0 in the config file. Also if there are more sts, the consumption for every sts will be larger.
interesting, but the resync is only happening for k8s objects.

Why is watching changes on those impossible ? And when you say external things you mean loadbalancers in gcloud ? resync is only happening for k8s services corresponding to those loadbalancers , right ? Or may be you are saying that if a k8s services is linked to a gcloud loadbalancer, the resync enables the controller code to go and check gcloud if the loadbalancing is provisioned or has anything changed there ? We need to document this in somwhere much like the apiconventions doc
- I mean loadbalancer in GCP as an example (but can also be other load balancer, persistent volumes in cloud or things like that). Yes - we resync on k8s objects, but that that let's you sync those. As an example - imagine service of type LoadBalancer - if you're periodically syncing the k8s service object, this triggers the operation of syncing the load-balancer (and we have an ability to check if it exists, is healthy etc.). - hmm, I would need to double check it, but I thought the default resync period is pretty long...
The default resync period is 12~24 hours. It is long but the bad experience will still happen every day.


Question: "fmt" "io" "os" <del> "strings" </del> "github.com/GoogleCloudPlatform/kubernetes/pkg/api" cmdutil "github.com/GoogleCloudPlatform/kubernetes/pkg/kubectl/cmd/util"

Right now at the end of I get We should update the output to point to the appropriate v1beta3 URL endpoints instead.
/cc
Ref and . cc
And as a minor nit, not all of those URLs need a trailing slash hack e.g. I don't think it is required for Elasticsearch. I think most recently worked on this code?
Thanks for fixing this


Question: configs[VolumeISCSIServer] = Config{e2eRegistry, "volume/iscsi", "2.0"} configs[VolumeGlusterServer] = Config{e2eRegistry, "volume/gluster", "1.0"} configs[VolumeRBDServer] = Config{e2eRegistry, "volume/rbd", "1.0.1"} <del> configs[WindowsNanoServer] = Config{e2eRegistry, "windows-nanoserver", "v1"} </del> <ins> configs[WindowsNanoServer] = Config{e2eGcRegistry, "windows-nanoserver", "v1"} </ins> return configs }

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: All sig-windows jobs (master): Which test(s) are failing: [] Container Runtime blackbox test when running a container with a new image should be able to pull from private registry with secret [NodeConformance] [] Container Runtime blackbox test when running a container with a new image should be able to pull image from [NodeConformance] Since when has it been failing: Since this PR merged: Testgrid link: Reason for failure: The mentioned PR removed hardcoded image names from tests, centralizing them into , but there were a few omissions: entry was written for. registry set for ` is because that registry contains the Windows images needed for all the Kubernetes E2E tests. Anything else we need to know: /sig testing /sig windows /area conformance /priority important-soon


Question: e2fsprogs  ebtables  ethtool  <ins> kmod  </ins> ca-certificates  conntrack  util-linux

Is this a BUG REPORT or FEATURE REQUEST?: What happened: When I update the kubelet on a node from 1.7.x to 1.8.0, pods on that node stop being able to connect to services whose pods happen to be on the same node. I use the Kubenet network plugin. What you expected to happen: Pods should be able to connect to a service even if the services's pods are on the same node. How to reproduce it (as minimally and precisely as possible): a cluster the DNS addon another pod that runs on the same node as one of the DNS pods resolution starts to fail when the service redirects the requests to the pod that is on the same node. Anything else we need to know?: I have some packet captures that I can show. Kubelet: 1.7.5 DNS pod: 10.2.4.76 Origin pod: 10.2.4.64 Kubelet: 1.8.0 DNS pod: 10.2.1.254 Origin pod: 10.2.1.234 We can see that in the requests that fail the DNS IP is correctly DNATed to the DNS POD destination address but the response is not masqueraded, it still carries the DNS POD source (it should have an origin of 10.3.0.10). Environment: Kubernetes version (use ):Cloud provider or hardware configuration: Azure StandardA4 OS (e.g. from /etc/os-release): "Container Linux by CoreOS 1465.8.0 (Ladybug)" Kernel (e.g. ): Linux node-1-vm 4.12.14-coreos SMP Wed Sep 20 22:20:05 UTC 2017 x8664 Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40GHz GenuineIntel GNU/Linux Install tools:
/sig node
The problem appears when transitioning from kubelet v1.8.0-alpha.2 to v1.8.0-alpha.3.
Well, I've found the problem. Starting from v1.8.0-alpha.3 the "modprobe" binary is not included in the image and so the br-netfilter kernel module is not loaded. I'm using the image, although through "kubelet-wrapper" from CoreOS. Where should this binary reside?

cc/ 1.8 release team.
There is a PR from so it doesn't get lost.
[MILESTONENOTIFIER] Milestone Issue Current <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details


Question: go copyBytes("from backend", in, out, &wg) go copyBytes("to backend", out, in, &wg) wg.Wait() <del> in.Close() out.Close() </del> } func copyBytes(direction string, dest, src *net.TCPConn, wg *sync.WaitGroup) {

While investigating a socket leak in our kubernetes-deployed application, I have found out that the kube-proxy also seems to have a serious leak of its own. In two of my three nodes, out of the ~44K connections of the kube-proxy .... ... about 65% are legit established connections caused by the aforementioned leak ... ... but ~14K are in FINWAIT2 state ... ... and ~1K are in CLOSEWAIT. Here's some debug info: Gist with service descriptions, node descriptions and iptables rules from node (we are running kubernetes 1.1 in AWS): Goroutine dump of kube-proxy at node , obtained with : at node : has a value of 60 in all my nodes. After deploying the fix for application-level leak and waiting 10 minutes, according to the number of ESTABLISHED connections is ~100 in all nodes as expected but, in the two affected nodes: The CLOSEWAIT connections only halve from 1K to 500 The FINWAIT2 ones also halve from 14K to 7K I obtained new goroutine and netstat dumps, in case it helps: Also, it may be valuable to know that the application-level leaks were caused by a websocket proxy (part of the pods) which failed to clean up connections when the client-side closed the underlying TCP socket. Both the app-mapper and the destination are accessed through kubernetes services, which means that the kube-proxy is caught up in-between: The two kube-proxy instances affected live in the same nodes where the two app-mapper pods live: and
Uhm, I have noticed that netstat somehow wasn't reporting all the sockets. Whilst lsof does: The extra entries seem to correspond to : So, I think my analysis above still applies. Here's the lsof listing in case it helps:
Can you post anything interesting from kube- Does anything appear to be spinning? Maybe try bumping the kube-proxy --v flag to 4 and reproducing , Alfonso Acosta wrote:
For record: The majority of hung goroutines (about 2/3 of them) are at go copyBytes("from backend", in, out, &wg)go copyBytes("to backend", out, in, &wg), both of which call down to . I note that we half-close (CloseWrite(), CloseRead()) connections in each direction.

I'm not sure I recall why exactly. I could try some tweaks around here to get more info - are you willing to run a test binary for me? , Tim Hockin wrote:
The kube-proxy logs were empty (rotated by I guess) but I can try to reproduce in a separate cluster. It would be good to know, 7K are a lot plus they escape netstat (and I would bet that the same happens with most monitoring systems). Sure
I have reproduced it with our unfixed, leaky app and kube-proxy running with (in fact I can reproduce it systematically). I can confirm that, for each descriptor pair our application leaks, kube-proxy gets one socket in CLOSEWAIT and another one in FINWAIT2. Again, to give more context on our application: What's happening is that the frontend closes the client socket, but the app-mapper (which is proxying the frontend connection to the destination) doesn't close its connection pair. The CLOSEWAIT socket in kube-proxy corresponds to the frontend ("from" side) socket and the FINWAIT2 to the app-mapper ("to" side). My theory is that, when cleaning up a connection pair (due to the client closing its connection), kube-proxy expects the other end in the "to" connection to close it right away, but when it doesn't (due to the leak in this case) it stays in the FINWAIT2 state forever and leaving the "from" side in CLOSEWAIT. I think that, unlike what stated in , kube-proxy should probably timeout the connections on the "to" end after certain time. I don't see a point to wait for the "to" end if the "from" end was closed. I am not familiar with kube-proxy's internals so my reasoning may be an oversimplification or simply wrong. Here is the netstat listing from one of the affected nodes: Here are the logs of the kube-proxy running in the same node: Here is the kube-proxy go-routine dump: Here is the node description: Here are the service descriptions:
After having a look at the code, causing a couple of leaks and removing all pods to reduce the connections of the kube-proxy to a minimum, I have a pretty good idea of what's happening. The exits, but its doesn't.

I need to confirm this, but my guess is that the in the copier are not enough to guarantee that in the routine returns if the remote end doesn't close the socket. You may find the full go-routing stack here:
Good news - I have a clue what is happening (I think). I am pretty sure it is the half-close logic we have - it's technically possible to close just one direction of a TCP session. If the client closes the socket, the proxy gets a signal that the client read-side closed and closes the backend's write-side. But it does not get a signal that the client's write side closed (because POSIX, that's why). If the backend responds by closing the socket (since it knows if it has more data to send or not), the proxy gets a signal that the backend read-side closed and closes the client's write side. If the backend DOES NOT close the socket, the proxy waits forever. It does not get a signal that the client's write side closed. So the easy fix is to not try the half-close stuff and just say that closed means closed, lingering data be damned. The more robust fix might be to start a timeout on the backend once the client disconnects. Here is a kube proxy binary which just closes the connection - try it out? , Alfonso Acosta wrote:
Specifically run with --v=4 and let me see logs :) , Tim Hockin wrote:
Yep, that matches my observations above. I haven't run your binary but in parallel to your investigations I have come up with a simple repro which does exactly that. For every leak created by the service I get a / pair in the kube-proxy: I can run your binary tomorrow if you still need me to, but I think this confirms the theory. I believe that the opposite timer (on the client once the backend disconnects) would also be necessary
I found some time and got the logs from a recent unmodified kube-proxy binary and the one provided while causing a leak in the leaker service: They confirm that the problem comes from the current use of and .
Thanks for the confirmation. I can't say for sure whether these are the exact same problem, but they are suspiciously similar. I'll try to work up a real patch tomorrow. , Alfonso Acosta wrote:
Patch is out for review. Thanks for the help!


Question: // Returns the prefix set via the ETCD_PREFIX environment variable (if any). func PathPrefix() string { <del> return path.Join("/", os.Getenv("ETCD_PREFIX")) </del> <ins> pref := os.Getenv("ETCD_PREFIX") if pref == "" { pref = "registry" } return path.Join("/", pref) </ins> } // Adds the ETCD_PREFIX to the provided key

The prefix should probably default to the empty string or something like "". When running in the , third party objects are stored to but the double slash is eaten on the HTTP Get:
"registry" would also be a sane default.
Possibly even validate the the specified prefix doesn't end in a slash too.
+1, I encountered this problem last night, too.


Question: # Do not use any network plugin by default. User could override the flags with # test_args. <del> test_args='--kubelet-flags="--network-plugin= --network-plugin-dir=" '$test_args </del> <ins> test_args='--kubelet-flags="--network-plugin= --cni-bin-dir=" '$test_args </ins> # Runtime flags test_args='--kubelet-flags="--container-runtime='$runtime'" '$test_args

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Environment: Cloud provider or hardware configuration: bare-metal OS (e.g. from /etc/os-release): Ubuntu 14.04 Kernel (e.g. ): Linux dev-k8s-controller-p1 4.4.0-53-generic ~14.04.1-Ubuntu SMP Fri Dec 2 03:43:31 UTC 2016 x8664 x8664 x86_64 GNU/Linux Install tools: chef cookbook Others: Docker 1.12.6, calico 2.1 CNI binaries: kubelet config: What happened: When trying to run pod error message is generated: So it seems that kubelet is trying to find CNI binaries in paths: /opt/loopback/bin /etc/cni/net.d /opt/calico/bin But it should look at since this is the default valur for parameter. What you expected to happen: Pod starts. How to reproduce it (as minimally and precisely as possible): Run pod: with kubelet 1.6 and CNI binaries in those path (default for calico): Anything else we need to know: After copy to and pod starts fine.
Good, very helpful to me
When set and not set , default path of CNI binaries will not take effect, the path specified by will be used instead. We should use and instead of if use CNI plugin.
I will push a PR to deprecate the flag. It is confusing. Closing this issue.
Was that ? It was closed without merging. I agree should be deprecated.
I think is needed before is removed. The flag is still not deprecated, I will send a PR to deprecate it.


Question: glog.V(4).Infof("Service type: %v does not need health check node port", service.Spec.Type) return false } <ins> // Loop through the service ports list, find one with the same port number and // NodePort specified, return this NodePort otherwise return 0. func findRequestedNodePort(port int, servicePorts []api.ServicePort) int { for i := range servicePorts { servicePort := servicePorts[i] if port == int(servicePort.Port) && servicePort.NodePort != 0 { return int(servicePort.NodePort) } } return 0 } </ins>

Consider a service like The API refuses to accept this service, stating It works fine without . It works when using different ports. The iptables rules installed (in mode) are TCP/UDP specific. I'm not sure if this is deliberate but not well documented, or a bug? I don't see a reason not to allow a UDP and TCP node port to have the same number.
For context, I am trying to make the cluster DNS available to legacy systems with a bit of forwarding, and DNS really really wants to have UDP and TCP on the same port number.
what is your kubernetes version?
This is on 1.1.4. From a quick reading of the code this hasn't changed though.
This was intentionally omitted due to laziness and complexity. We hoped nobody would need it, since NodePorts are generally allocated automatically. It's mostly going to be a matter of adding a switch to look at 2 different saved allocation maps and figuring out how to transition. , Matthias Rampke wrote:
I've also hit this issue (see: ) when trying to expose DNS to queries from outside the cluster. Note that in my setup, the "NodePort" is different for UDP and TCP but only UDP appears to work (it's the first to be defined?).
I also run into this issue. I've got a service that wants to expose TCP and UDP port with the same number. Any plans to include this issue in any roadmap?
Help wanted. We're buried in high-priority bugs and features. I ACK that this behaves imperfectly, but I and my team will not have time to look at it in the immediate future.
FYI
I think the root cause is , when serviceport allocator attempts to reserve the provided port, it just use the port number not consider the port protocol. This is a bug of service port allocator I believe.
"bug", but yes. We'd need to store parallel bitmaps per protocol. Even then, it's not obvious when to use the same port number in different protocols, and whether that is REQUIRED or not.

If you have a Service with port 53 in TCP and UDP, and I can't find any port number that is available in both protocols, should I: a) fail and reject the REST operation b) choose a different port number c) retry forever and hang the REST operation d) accept the REST operation but log an event saying I failed to do my job NodePort was really made for load-balancers, where the port number doesn't matter. Humans using it is "unexpected". , TonyAdo wrote:
this is what I'd expect. Port exhaustion is not really a temporary Matthias Rampke Engineer SoundCloud Ltd. +49 173 Managing Director: Alexander Ljung | Incorporated in England & Wales with Company No. HRB 110657B
I am afraid PR will not fix this issue. We have to store parallel bitmaps per protocol. I will open a new PR to fix this issue.
This alone is insufficient, but we might get away without storing two maps by simply saying the port map is unified across protocols if a service has 2 ports that use the same port number but different protocols, allocate the first nodePort and use the same value for the 2nd port This will prevent the case of ports being used in one protocol but not the other and failing to allocate. Something like: I haven;t tried too hard to break it, but something like that MIGHT work.
Additionally, if you have to store multiple bitmaps, you have a serious hassle on your hands for upgrading cluster in-place. Better to avoid it if you can.
sure, I will think about this more.

Hi, I have created and deployed a service similar to below in minikube environment.. however I am not to connect to the ports sometimes TCP/UDP ` apiVersion: v1 kind: Service metadata: name: dns spec: selector: … clusterIP: … type: NodePort ports: name: grpc protocol: TCP port: targetPort: grpc-port name: port-udp port: targetPort: 5062 protocol: UDP nodePort: name: port-tcp port: targetPort: 5062 protocol: TCP nodePort: Example: $kubectl get all service/-service NodePort 10.106.159.94 <none50051:/TCP,30:/UDP,:/TCP $ telnet 192.168.99.100 telnet: can't connect to remote host (192.168.99.100): Connection refused $ telnet localhost telnet: can't connect to remote host (127.0.0.1): Connection refused $nc -z -v -u 192.168.99.100 Connection to 192.168.99.100 port [udp/] succeeded! $ nc -z -v 192.168.99.100 nc: connect to 192.168.99.100 port (tcp) failed: Connection refused $ kubectl version Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-10-02T17:01:15Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-09-18T14:27:17Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"} $netstat -an tcp 0 0 ::: ::: LISTEN udp 0 0 ::: :::*
!


Question: // HasExpired will identify whether the secret expires func HasExpired(secret *v1.Secret, currentTime time.Time) bool { <ins> _, expired := GetExpiration(secret, currentTime) return expired } // GetExpiration checks if the secret expires // isExpired indicates if the secret is already expired. // timeRemaining indicates how long until it does expire. // if the secret has no expiration timestamp, returns 0, false. // if there is an error parsing the secret's expiration timestamp, returns 0, true. func GetExpiration(secret *v1.Secret, currentTime time.Time) (timeRemaining time.Duration, isExpired bool) { </ins> expiration := GetData(secret, api.BootstrapTokenExpirationKey) <del> if len(expiration) > 0 { expTime, err2 := time.Parse(time.RFC3339, expiration) if err2 != nil { klog.V(3).Infof("Unparseable expiration time (%s) in %s/%s Secret: %v. Treating as expired.", expiration, secret.Namespace, secret.Name, err2) return true } if currentTime.After(expTime) { klog.V(3).Infof("Expired bootstrap token in %s/%s Secret: %v", secret.Namespace, secret.Name, expiration) return true } </del> <ins> if len(expiration) == 0 { return 0, false } expTime, err := time.Parse(time.RFC3339, expiration) if err != nil { klog.V(3).Infof("Unparseable expiration time (%s) in %s/%s Secret: %v. Treating as expired.", expiration, secret.Namespace, secret.Name, err) return 0, true } timeRemaining = expTime.Sub(currentTime) if timeRemaining <= 0 { klog.V(3).Infof("Expired bootstrap token in %s/%s Secret: %v", secret.Namespace, secret.Name, expiration) return 0, true </ins> } <del> return false </del> <ins> return timeRemaining, false </ins> } // ParseName parses the name of the secret to extract the secret ID.

What happened: kube-controller-manager with enabled option --controllers=,tokencleaner is not removing newly created tokens. It works only after kube-controller-manager restart or after editing secret with token. What you expected to happen: It should remove all expired tokens from kube-system namespace, without restarting or touching secrets. How to reproduce it (as minimally and precisely as possible): enable tokencleaner in kube-controller-manager start controller manager create bootstrap token, you can use script wait 5 minutes token is still there restart kube-controller-manager OR edit newly created token token should be gone after a while Anything else we need to know?: It could be related with Environment: Kubernetes version (use ): Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-12-03T20:56:12Z", GoVersion:"go1.11.2", Compiler:"gc", Platform:"linux/amd64"}
Do you see the following log ? thanks
no errors at all
/sig auth


Question: podName := "busybox-scheduling-" + string(uuid.NewUUID()) /* <del> Release : v1.9 </del> <ins> Release : v1.13 </ins> Testname: Kubelet, log output, default Description: By default the stdout and stderr from the process being executed in a pod MUST be sent to the pod's logs. */ <del> It("should print the output to logs [NodeConformance]", func() { </del> <ins> framework.ConformanceIt("should print the output to logs [NodeConformance]", func() { </ins> podClient.CreateSync(&v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name: podName,

What happened: These tests are errantly listed as (ref: ) but aren't actually run as part of Conformance. They need to be moved to test/e2e/common to be included. To my eyes they may meet the criteria for conformance. There's definitely nothing about them that is node specific. They should be running as part of our regular CI, and they should probably be running as presubmits. Porting them to test/e2e/common would cause this to happen. /kind cleanup /sig node /area test /assign
GitHub didn't allow me to assign the following users: mkurylec. Note that only and repo collaborators can be assigned. For more information please see <detailsIn response to : What happened: These tests are errantly listed as (ref: ) but aren't actually run as part of Conformance. They need to be moved to test/e2e/common to be included. /sig node /area test /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: Program            cel.Program Error              *apiservercel.Error ExpressionAccessor ExpressionAccessor <ins> OutputType         *cel.Type </ins> } // Compiler provides a CEL expression compiler configured with the desired admission related CEL variables and

Variables field is to ValidatingAdmissionPolicy in 1.28. Currently the usage of variables would expect a operator associated. e.g. (thanks to Mariam Fahmy for raising the issue) Above will fail with error: Meanwhile the will pass. The reason is when variables be evaluated to non dyn type(like bool or string), it will require special handling(like casting) since we use dyn type everywhere. It might confuse user.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage accepted /sig api-machinery /cc
Tests I'd expect to see verify this behavior: PASS: A variable is of type bool, an expression returns the variable PASS: A variable is of type string, a messageExpression returns the variable PASS: A variable is of type dyn, at runtime the dyn is a bool, an expression returns the variable PASS: A variable is of typedyn, at runtime the dyn is a string, an messageExpression returns the variable PASS: A expression is of type dyn, at runtime the dyn is of type bool PASS: A messageExpression is of type dyn, at runtime the dyn is of type string COMPILE ERROR: A variable is of type bool, an expression tries to perform a non-bool (e.g. string replace) operation on the variable, this should resultsin a compile error COMPILE ERROR: A variable is of type string, an expression tries to perform a non-string (e.g. less than) operation on the variable, this should result in a compile error RUNTIME ERROR: A variable is of type dyn, at runtime the dyn is a bool, an expression tries to perform a non-bool (e.g. string replace) operation on the variable, this should result in a runtime error I'd also expect to see some cases covering variables of other types (maps, lists, duration, quantity, ...) to make sure we're over-fitting for bool and string.
/retitle ValidatingAdmissionPolicy: variable expression type not recognized as return type in expression that evaluates the variable
/assign


Question: // GetCandidateZonesForDynamicVolume retrieves  a list of all the zones in which nodes are running // It currently involves querying all instances func (c *Cloud) GetCandidateZonesForDynamicVolume() (sets.String, error) { <del> zones := sets.NewString() </del> <ins> // We don't currently cache this; it is currently used only in volume // creation which is expected to be a comparatively rare occurrence. // TODO: Caching / expose v1.Nodes to the cloud provider? // TODO: We could also query for subnets, I think // Note: It is more efficient to call the EC2 API twice with different tag // filters than to call it once with a tag filter that results in a logical // OR. For really large clusters the logical OR will result in EC2 API rate // limiting. instances := []*ec2.Instance{} </ins> <del> // TODO: list from cache? nodes, err := c.kubeClient.CoreV1().Nodes().List(metav1.ListOptions{}) </del> <ins> baseFilters := []*ec2.Filter{newEc2Filter("instance-state-name", "running")} filters := c.tagging.addFilters(baseFilters) di, err := c.describeInstances(filters) </ins> if err != nil { <del> klog.Errorf("Failed to get nodes from api server: %#v", err) </del> return nil, err } <del> for _, n := range nodes.Items { if !c.isNodeReady(&n) { klog.V(4).Infof("Ignoring not ready node %q in zone discovery", n.Name) continue } // In some cluster provisioning software, a node can be both a minion and a master. Therefore we white-list // here, and only filter out node that is not minion AND is labeled as master explicitly if c.isMinionNode(&n) || !c.isMasterNode(&n) { if zone, ok := n.Labels[v1.LabelZoneFailureDomain]; ok { zones.Insert(zone) } else { klog.Warningf("Node %s does not have zone label, ignore for zone discovery.", n.Name) } } else { klog.V(4).Infof("Ignoring master node %q in zone discovery", n.Name) </del> <ins> instances = append(instances, di...) if c.tagging.usesLegacyTags { filters = c.tagging.addLegacyFilters(baseFilters) di, err = c.describeInstances(filters) if err != nil { return nil, err </ins> } <ins> instances = append(instances, di...) </ins> } <del> klog.V(2).Infof("Found instances in zones %s", zones) return zones, nil } </del> <ins> if len(instances) == 0 { return nil, fmt.Errorf("no instances returned") } zones := sets.NewString() </ins> <del> // isNodeReady checks node condition and return true if NodeReady is marked as true func (c *Cloud) isNodeReady(node *v1.Node) bool { for _, c := range node.Status.Conditions { if c.Type == v1.NodeReady { return c.Status == v1.ConditionTrue </del> <ins> for _, instance := range instances { // We skip over master nodes, if the installation tool labels them with one of the well-known master labels // This avoids creating a volume in a zone where only the master is running - e.g. #34583 // This is a short-term workaround until the scheduler takes care of zone selection master := false for _, tag := range instance.Tags { tagKey := aws.StringValue(tag.Key) if awsTagNameMasterRoles.Has(tagKey) { master = true } </ins> } <del> } return false } // isMasterNode checks if the node is labeled as master func (c *Cloud) isMasterNode(node *v1.Node) bool { // Master node has one or more of the following labels: // // 	kubernetes.io/role: master //	node-role.kubernetes.io/master: "" //	node-role.kubernetes.io/master: "true" if val, ok := node.Labels[labelKeyNodeMaster]; ok && val != "false" { return true } else if role, ok := node.Labels[labelKeyNodeRole]; ok && role == nodeMasterRole { return true } return false } </del> <ins> if master { klog.V(4).Infof("Ignoring master instance %q in zone discovery", aws.StringValue(instance.InstanceId)) continue } </ins> <del> // isMinionNode checks if the node is labeled as minion func (c *Cloud) isMinionNode(node *v1.Node) bool { // Minion node has one or more oof the following labels: // // 	kubernetes.io/role: "node" //	node-role.kubernetes.io/compute: "true" //	node-role.kubernetes.io/node: "" if val, ok := node.Labels[labelKeyNodeMinion]; ok && val != "false" { return true } else if val, ok := node.Labels[labelKeyNodeCompute]; ok && val != "false" { return true } else if role, ok := node.Labels[labelKeyNodeRole]; ok && role == nodeMinionRole { return true </del> <ins> if instance.Placement != nil { zone := aws.StringValue(instance.Placement.AvailabilityZone) zones.Insert(zone) } </ins> } <del> return false </del> <ins> klog.V(2).Infof("Found instances in zones %s", zones) return zones, nil </ins> } // GetZone implements Zones.GetZone

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: We've seen scalability issues with the current aws-ebs volume provisioner impl, which is querying possible zones from cloud provider directly (). The improvement proposed in did make api calls more efficient, but also raises people's concern about whether it will create regression or not, or is there any more reliable ways to improve. To summarize, there are 2 major concerns: zone info in Node label trustworthy? My answer is yes, since currently the zone labels (beta version only) are maintained by node controller, the information is retrieved from cloud provider directly and is always updated during node reconciliation, it can be trusted. Even if kubelet can be misconfigured by wrong zone information, node controller is going to override them. See: Once we have the GA version of zone labels pushed out, we should update node controller to maintain both zone labels. It's fair enough to assume that if user wants kubelet to report zone/region, the information should be correct. the master/minion labels trustworthy? My answer is "semantic wide they are equivalent to checking node labels from cloud provider" - just as people are not required to set master/minion labels in v1.Node, well-known labels in cloud provider is not guaranteed to be set as well. Therefore besides mainstream cluster provisioners, which maintains these sets of labels in both Kubernetes and cloud provider, the function's behavior will remain unchanged: the original impl skips nodes that are explicitly labeled with well-known master labels, same here. Please suggest. /sig aws /sig cloud-provider /kind bug /priority important-soon /cc
for item 2, an alternative is to first call a List() of all v1.Nodes, get all providerID, and make a DescribeInstance call by passing all these IDs, which might still be more efficient than running the filter, and still very heavy-weighted with large scale. Current logic can be applied to the results though. Need input from aws side.
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available .

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
One of the quick optimizations we can do is, to remove those API calls if pod placement has already been decided because of delayed binding. I think, using delayed binding works well in practice and should eliminate calls being made.
Another thing is - since EBS volume plugin is being targeted for CSI migration, we should be super careful approving any changes into in-tree plugins which is not available in EBS CSI plugin. cc
Thanks ! yes I do agree that for topo aware storage class, when we create volume, the node is already bound, so we just get the zone information from the selected node, and this is a good to have short-term improvement. In long term, as cloud providers are going to be out of tree, we have more freedom to introduce informer to the aws library (gce is already doing that), and get zone information from cache. Since the path we are going is to make kubernetes components node role neutral, it's fair enough to say that we don't have to filter out master only zone in volume provisioner, and user should leverage topology aware storage class along with labels, taints, tolerations to achieve that. what do you think?
I believe in long term we want to default all clusters to use policy. That is the direction sig-storage wants to take(IIRC ), so IMO it may not be worth fixing general immediate binding use case for in-tree EBS driver where cloudprovider itself is deprecated (and any proposed change is non-trivial).
if the direction sig-storage is going is to always use , I agree with you that fixing a deprecated path of a deprecated EBS driver would be not worthwhile. Meanwhile, since in-tree EBS driver will still stay for a few more versions, I'm proposing for optimizing code path. Could you please provide inputs? thanks! cc
Related to
Hello! I'm the bug triage lead for the 1.16 release cycle and considering this issue is tagged for 1.16, I'd like to check its status. The code freeze is starting on August 29th EOD PST (about 5 days from now) and once the code freeze starts, only release-blocking changes will be considered to be merged. Is this issue still planned to be fixed for the 1.16 release?

I have addressed this issue via , will close this issue /close
Closing this issue. <detailsIn response to : I have addressed this issue via , will close this issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: tc.runTest(t) } <del> func TestScaleDownOneMetricInvalid(t *testing.T) { tc := testCase{ minReplicas:             2, maxReplicas:             6, specReplicas:            5, statusReplicas:          5, expectedDesiredReplicas: 3, CPUTarget:               50, metricsTarget: []autoscalingv2.MetricSpec{ { Type: "CheddarCheese", }, }, reportedLevels:      []uint64{100, 300, 500, 250, 250}, reportedCPURequests: []resource.Quantity{resource.MustParse("1.0"), resource.MustParse("1.0"), resource.MustParse("1.0"), resource.MustParse("1.0"), resource.MustParse("1.0")}, useMetricsAPI:       true, recommendations:     []timestampedRecommendation{}, } tc.runTest(t) } func TestScaleDownOneMetricEmpty(t *testing.T) { tc := testCase{ minReplicas:             2, maxReplicas:             6, specReplicas:            5, statusReplicas:          5, expectedDesiredReplicas: 3, CPUTarget:               50, metricsTarget: []autoscalingv2.MetricSpec{ { Type: autoscalingv2.ExternalMetricSourceType, External: &autoscalingv2.ExternalMetricSource{ Metric: autoscalingv2.MetricIdentifier{ Name:     "qps", Selector: &metav1.LabelSelector{}, }, Target: autoscalingv2.MetricTarget{ Type:         autoscalingv2.AverageValueMetricType, AverageValue: resource.NewMilliQuantity(1000, resource.DecimalSI), }, }, }, }, reportedLevels:      []uint64{100, 300, 500, 250, 250}, reportedCPURequests: []resource.Quantity{resource.MustParse("1.0"), resource.MustParse("1.0"), resource.MustParse("1.0"), resource.MustParse("1.0"), resource.MustParse("1.0")}, useMetricsAPI:       true, recommendations:     []timestampedRecommendation{}, } _, _, _, testEMClient, _ := tc.prepareTestClient(t) testEMClient.PrependReactor("list", "*", func(action core.Action) (handled bool, ret runtime.Object, err error) { return true, &emapi.ExternalMetricValueList{}, fmt.Errorf("something went wrong") }) tc.testEMClient = testEMClient tc.runTest(t) } </del> func TestScaleDownIgnoreHotCpuPods(t *testing.T) { tc := testCase{ minReplicas:             2,

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --If you have 2 or more metrics defined on an HPA and one metric is suggesting to scale down and the other metric is unavailable, the HPA will scale down despite missing metrics. Example: Imagine the following Horizontal Pod Autoscaler resource: And the deployment is currently scaled to replicas. If the metric is suggesting to scale to 1 AND the metric is unavailable e.g. HPA can't fetch the metrics. Then it will happily scale down to despite not knowing what the currently unavailable metric would suggest. It's clearly stated in the docs the scale down should not happen if one or more metrics are unavailable: Until Kubernetes v1.15.x the behavior was as expected. However with the . The logic was removing the special handling of stopping scale down in case of unavailable metrics. Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /bug /sig autoscaling
/triage accepted


Question: return *node, nil } <ins> func (nm *NodeManager) getNodes() map[string]*v1.Node { nm.registeredNodesLock.RLock() defer nm.registeredNodesLock.RUnlock() registeredNodes := make(map[string]*v1.Node, len(nm.registeredNodes)) for nodeName, node := range nm.registeredNodes { registeredNodes[nodeName] = node } return registeredNodes } </ins> func (nm *NodeManager) addNode(node *v1.Node) { nm.registeredNodesLock.Lock() nm.registeredNodes[node.ObjectMeta.Name] = node

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Quote I took goroutine dump of controller-managers that were stuck in volume provisioning and I think we have a deadlock in code: I think we have a problem introduced when we merged - and what is happening is, when in is called it acquires a lock on , but same function ends up calling which acquires another lock on object . Now the problem is, function gets called in a loop and it will try to acquire lock for each node it processes and hence two operations could result in deadlock. This results into inability to provision a with a default StorageClass. The volume does not get to call the API endpoint, just hangs after call, which is clear in the goroutine dump. What you expected to happen: 1) Provision a from successfully. 2) Don't be blocked by credentials secret change. How to reproduce it (as minimally and precisely as possible): 1) Provision a vSphere cluster with credentials secret set 2) Create volume 3) Wait... Anything else we need to know?: Environment: Kubernetes version (use ): v1.19.0-rc.2+-dirty Cloud provider or hardware configuration: vSphere OS (e.g: ): Linux Kernel (e.g. ): GNU/Linux Install tools: openshift-install (kubeadm) Network plugin and version (if this is a network-related bug): Others:
/sig cloud-provider /area provider/vmware
This should be a blocker bug for 1.19 at least as far as vsphere cloudprovider integration is concerned, because it breaks cloudprovider integration/volume provisioning. cc
cc:
/milestone v1.19
Something makes me think this issue existed before, and just escalated to a larger scope after adding informers, demanding refresh for all objects. This is used within plugin in and it is mounting the volume if it is required, calling - it is locking on all the nodes at once. If the pod was just restarted (say just changed, or the health probes demanded a restart), it would process all existing at the same time as trying to mount/create the volume.

And both threads will clang on the same lock, as the would probably not be populated at that moment, so will go the path of - same which is used in . So we get lost volume data, pending etc... And the links I'm sharing are from Kubernetes, where issues with similar footprint randomly arose.