Question: // waiting in its queue, or `Time{}` if this did not happen. func (cfgCtlr *configController) startRequest(ctx context.Context, rd RequestDigest, queueNoteFn fq.QueueNoteFn) (fs *flowcontrol.FlowSchema, pl *flowcontrol.PriorityLevelConfiguration, isExempt bool, req fq.Request, startWaitingTime time.Time, flowDistinguisher string) { klog.V(7).Infof("startRequest(%#+v)", rd) <del> cfgCtlr.lock.Lock() defer cfgCtlr.lock.Unlock() </del> <ins> cfgCtlr.lock.RLock() defer cfgCtlr.lock.RUnlock() </ins> var selectedFlowSchema, catchAllFlowSchema *flowcontrol.FlowSchema for _, fs := range cfgCtlr.flowSchemas { if matchesFlowSchema(rd, fs) {

Two flakes recently: They fail with error like: or /assign
/sig scalability
In logs I have found the following slow calls: It's a bug that we use exempt priority level for clusterloader: I created It's quite worrisome that APF handler takes 1s for exempt request.
I checked a APF code and I think the flow for exempt requests is: genericfilters.WithPriorityAndFairness() workEstimate := workEstimator.EstimateWork(r) fcIfc.Handle() cfgCtlr.startRequest() () looks for matching flowschemas in cfgCtrl.flowSchemas returns for exempt for other requests it calculates sha256 for flowDistinguisher and queues request req.Finish updates metrics calls execFn updates headers handler.ServeHTTP which stops the timer for priorityandfairness For me, the most likely reason for slowness is lock contention on We know that the test is super cpu-intensive in that part and other requests are e.g. calculating sha256 keeping that lock. A potential improvement would be to change this lock to RWLock or return the lock before starting calculating sha256 (if we believe that this is the most expensive operation there). /cc
/assign
I think the lock-contention is even worse than what you wrote, as e.g.: is happening under this lock (which is also dispatching requests). I will take a deeper look later this week, but I think that switching to RWLock might do the job without any invasive changes.
may trigger the execution of some requests that were waiting, but they are not executed in the same goroutine. Yes, I think changing that Mutex to a RWMutex would be an easy way to greatly reduce contention.
I agree switching to RWMutex is likely to help a lot, if that's not enough I think with a little work we could e.g. switch to s.
(perhaps because that's what my code originally did? Reviewers thought the retirement dance was too tricky.)
/triage accepted /priority important-soon I will work on the fix.
/sig api-machinery
Wow! Thanks a lot for rapid help here! Let's wait for the next test run to see if it helped. /reopen
Reopened this issue. <detailsIn response to : Wow! Thanks a lot for rapid help here! Let's wait for the next test run to see if it helped.

/reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Yes, can you please report the filter latency regardless of whether it is bad or good? I would like to know how much of a difference that PR made.
Sure, will report the latency change in both cases.
I compared (the one where we saw issue) (first with the lock improvements) I tried to find requests that: use exempt PL (so shouldn't be throttled) has fl_priorityandfairness, key which means that priority and fairness filter took more than 100ms Sample matching request: In the old run, I see ~6k such requests, with the locking improvements there are no such requests. The latency of PATCH jobs seems to improve significantly: ! So overall it looks great. The only worrisome metric I see is that actual P&F throttling for other requests has grown: With locking improvements: ! Old: ! E.g. 'node-high' has now throttling latency of ~1.5s in spikes where in the previous test run it was more like ~800ms. I don't think the lock improvements touched this directly. I expect that this is either: per-test variance (if we rerun the old run again we would see different throttling latencies as well) we throttle exempt requests less (as expected) giving less resources to other requests. Anyway, I don't think this change is meaningful as per-test latency aggregates for PATCH nodes/status and PUT leases (the most common requests in node-high) look ~same as previously: ! ! I think we can close this for now as the issue seems to be resolved. Thanks a lot for your help here! /close
Closing this issue. <detailsIn response to : I compared In the old run, I see ~6k such requests, with the locking improvements there are no such requests. The latency of PATCH jobs seems to improve significantly: ! ! ! I expect that this is either: ! ! /close </details
: thanks for reporting!


Question: // Create an API Server. controlPlaneConfig := framework.NewIntegrationTestControlPlaneConfig() controlPlaneConfig.GenericConfig.Authorization.Authorizer = newRBACAuthorizer(t, controlPlaneConfig) <del> controlPlaneConfig.GenericConfig.Authentication.Authenticator = bearertoken.New(tokenfile.New(map[string]*user.DefaultInfo{ </del> <ins> controlPlaneConfig.GenericConfig.Authentication.Authenticator = group.NewAuthenticatedGroupAdder(bearertoken.New(tokenfile.New(map[string]*user.DefaultInfo{ </ins> superUser:                          {Name: "admin", Groups: []string{"system:masters"}}, "any-rolebinding-writer":           {Name: "any-rolebinding-writer"}, "any-rolebinding-writer-namespace": {Name: "any-rolebinding-writer-namespace"},

While moving APF to beta (see comments on ), we discovered that the integration tests commonly present users which are not members of any groups. We had thought that the system enforced membership in either the authenticated or unauthenticated group. If this is supposed to be a system invariant (as we thought), we'd like to ask SIG Auth: Please fix the integration tests Please add e.g. logging so that if real, operational clusters hit this there's at least a log message and people will file bugs. /sig auth
Can I suggest also, if this is a real invariant, that it be documented in some place(s) likely to be read?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale /priority important-longterm /triage accepted
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/lifecycle frozen
fixes the authenticator configured in one of the integration test API server startup helpers which resolved this issue for . Remaining tests that still need fixing that are encountering unexpected users look to be ones set up to specifically test auth paths:
open to fix remaining integration test users


Question: } } } <ins> func TestSecurityGroupFiltering(t *testing.T) { grid := []struct { in          []*ec2.SecurityGroup name        string expected    int description string }{ { in: []*ec2.SecurityGroup{ { IpPermissions: []*ec2.IpPermission{ { IpRanges: []*ec2.IpRange{ { Description: aws.String("an unmanaged"), }, }, }, }, }, }, name:        "unmanaged", expected:    0, description: "An environment without managed LBs should have %d, but found %d SecurityGroups", }, { in: []*ec2.SecurityGroup{ { IpPermissions: []*ec2.IpPermission{ { IpRanges: []*ec2.IpRange{ { Description: aws.String("an unmanaged"), }, { Description: aws.String(fmt.Sprintf("%s=%s", NLBClientRuleDescription, "managedlb")), }, { Description: aws.String(fmt.Sprintf("%s=%s", NLBHealthCheckRuleDescription, "managedlb")), }, }, }, }, }, }, name:        "managedlb", expected:    1, description: "Found %d, but should have %d Security Groups", }, } for _, g := range grid { actual := len(filterForIPRangeDescription(g.in, g.name)) if actual != g.expected { t.Errorf(g.description, actual, g.expected) } } } </ins>

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: After a rolling update of the cluster, NLB kept failing health checks because security group did not have rules for new EC2 instance. What you expected to happen: Security group should have been updated with both TCP rules for NLBs to work correctly. How to reproduce it (as minimally and precisely as possible): Create a service object with type LoadBalancer and adding the annotation for NLBs. FWIW, I had 6 services and each had their own NLBs. It’s possible that the bug happens when there are multiple NLBs causing multiple AWS updates. Perform a rolling update of the instance group. This causes the instances to be terminated and new ones to be brought up. AFAICT, this should result in two changes to the security group per EC2 instance. When the instance is terminated, the rules for that nodePort should be deleted. When the new node is created, the rule should be again. However, after the rolling update, the rule doesn’t get . As a result, the NLB’s health checks fail. Anything else we need to know?: As mentioned earlier, this happened on a cluster where there were multiple NLBs. This doesn’t seem to be a deterministic failure. The log has the following error message:I manually edited the Service object to see if it would simply retrigger an update on the cloud resources. It actually did. The rules got to the security group. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: AWS OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: kops 1.8.1 Others:
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
did you find any workarounds? we've run into this issue, where the security group rules got deleted. Specifically, we found that the security group missed the following: 1) TCP rule to allow traffic from NLB to Node Port 2) TCP rule to allow traffic from NLB to Healthcheck Port We these manually back in to fix, but were surprised these were deleted in the first place.

This stackoverflow also is relevant:
One workaround was to simply refresh the Kubernetes service object. We /updated a label in the Kubernetes service object and found that it would make the corresponding changes in AWS.
thanks have you been able to root cause why the rule doesn't get ? We're trying to re-produce the failure ourselves, too.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Any update on this bug fix release please?
Sorry to bother you with commenting on this issue. But I cannot find out in what version of Kubernetes this is expected to be released / was released. Any clarification highly appreciated.


Question: handler = genericapifilters.WithTracing(handler, c.TracerProvider) } handler = genericapifilters.WithLatencyTrackers(handler) <ins> // WithRoutine will execute future handlers in a separate goroutine and serving // handler in current goroutine to minimize the stack memory usage. It must be // after WithPanicRecover() to be protected from panics. if utilfeature.DefaultFeatureGate.Enabled(genericfeatures.APIServingWithRoutine) { handler = genericfilters.WithRoutine(handler, c.LongRunningFunc) } </ins> handler = genericapifilters.WithRequestInfo(handler, c.RequestInfoResolver) handler = genericapifilters.WithRequestReceivedTimestamp(handler) handler = genericapifilters.WithMuxAndDiscoveryComplete(handler, c.lifecycleSignals.MuxAndDiscoveryComplete.Signaled())

Optimize the memory usage of goroutine stack for serving watch requests. According to the of handling a watch request, the stack memory is 0x3768, which is very close to 16 KiB and Golang runtime will double it to 32 KiB once exceeded. In fact, it does exceed if we print the traceback deeper and take the fact that extra stack memory reserved by Golang itself into consideration. A naive optimization is to run as much as preparation handlers in a separate goroutine and then move the actual serving to another goroutine, so the former routine can free the stack memory earlier and make it more memory efficient. Given the extensive use of watch requests, 32 KiB per watch could add up to a very high amount of memory overhead for . A typical watch request commonly lasts for 30s or longer, which means this memory will be active for quite long. I see a great improvement in memory efficiency to do this optimization.
/cc
/sig api-machinery
/cc if you would like to take a look when you have time :) Thank you /triage accepted


Question: for ix := range ports { allowedPorts[ix] = strconv.Itoa(int(ports[ix].Port)) } <del> hostTags, err := gce.computeHostTags(hosts) if err != nil { return nil, err </del> <ins> // If the node tags to be used for this cluster have been predefined in the // provider config, just use them. Otherwise, invoke computeHostTags method to get the tags. hostTags := gce.nodeTags if len(hostTags) == 0 { var err error if hostTags, err = gce.computeHostTags(hosts); err != nil { return nil, fmt.Errorf("No node tags supplied and also failed to parse the given lists of hosts for tags. Abort creating firewall rule.") } </ins> } firewall := &compute.Firewall{ Name:         makeFirewallName(name), Description:  desc,

In some custom setup cases the prefix-matching logic for GCE load-balancers can break and end up creating a firewall rule with no tags. We should instead refuse to create the rule. In fact, if we find it, we should probably delete the rule, but that's maybe too aggressive. Looking at the logic, if we return error in the tag-selection, we'll abort the whole load-balancer creation, which is unfortunate, but not TERRIBLE. We could just silently not create the firewall but not return error, which will just retry over and over.
/cc
Are you active on this? may have some starter-project bandwidth to help.
Not active. It would indeed be a good starter project :)


Question: t.Errorf("Expected match but received nil volume for scenario: %s", name) } if scenario.expectedMatch != "" && volume != nil && string(volume.UID) != scenario.expectedMatch { <del> t.Errorf("Expected %s but got volume %s instead", scenario.expectedMatch, volume.UID) </del> <ins> t.Errorf("Expected %s but got volume %s in scenario %s", scenario.expectedMatch, volume.UID, name) </ins> } if scenario.expectedMatch == "" && volume != nil { t.Errorf("Unexpected match for scenario: %s", name)

E0513 22:55:45. 957 ] unable to sync kubernetes RO service: endpoints "kubernetes-ro" cannot be updated: the resource was updated to 266 E0513 22:55:45. 957 ] unable to sync kubernetes RO service: endpoints "kubernetes-ro" cannot be updated: the resource was updated to 266 E0513 22:55:48. 957 ] watch of api.Secret ended with: Stop requested E0513 22:55:48. 957 ] watch of api.ServiceAccount ended with: Stop requested E0513 22:55:48. 957 ] watch of api.Namespace ended with: Stop requested E0513 22:55:48. 957 ] watch of api.ServiceAccount ended with: Stop requested E0513 22:55:48. 957 ] watch of api.Secret ended with: Stop requested E0513 22:55:48. 957 ] watch of api.ServiceAccount ended with: Stop requested W0513 22:55:49. 957 ] Resetting endpoints for master service "kubernetes" to &{{ } {kubernetes default 0001-01-01 00:00:00 +0000 UTC <nilmap[] map[]} [{[{10.0.0.22 <nil}] [{ 6443 TCP}]}]} W0513 22:55:50. 957 ] Resetting endpoints for master service "kubernetes-ro" to &{{ } {kubernetes-ro default 0001-01-01 00:00:00 +0000 UTC <nilmap[] map[]} [{[{10.0.0.22 <nil}] [{ 7080 TCP}]}]} W0513 22:55:50. 957 ] Resetting endpoints for master service "kubernetes-ro" to &{{ } {kubernetes-ro default 0001-01-01 00:00:00 +0000 UTC <nilmap[] map[]} [{[{10.0.0.22 <nil}] [{ 7080 TCP}]}]} W0513 22:55:50.

957 ] Resetting endpoints for master service "kubernetes" to &{{ } {kubernetes default 0001-01-01 00:00:00 +0000 UTC <nilmap[] map[]} [{[{10.0.0.22 <nil}] [{ 6443 TCP}]}]} W0513 22:55:50. 957 ] Resetting endpoints for master service "kubernetes-ro" to &{{ } {kubernetes-ro default 0001-01-01 00:00:00 +0000 UTC <nilmap[] map[]} [{[{10.0.0.22 <nil}] [{ 7080 TCP}]}]} W0513 22:55:50. 957 ] Resetting endpoints for master service "kubernetes-ro" to &{{ } {kubernetes-ro default 0001-01-01 00:00:00 +0000 UTC <nilmap[] map[]} [{[{10.0.0.22 <nil}] [{ 7080 TCP}]}]} W0513 22:55:50. 957 ] Resetting endpoints for master service "kubernetes-ro" to &{{ } {kubernetes-ro default 0001-01-01 00:00:00 +0000 UTC <nilmap[] map[]} [{[{10.0.0.22 <nil}] [{ 7080 TCP}]}]} E0513 22:55:50. 957 ] unable to sync kubernetes service: endpoints "kubernetes" cannot be updated: the resource was updated to 279 E0513 22:55:50. 957 ] unable to sync kubernetes RO service: endpoints "kubernetes-ro" cannot be updated: the resource was updated to 280 E0513 22:55:50. 957 ] unable to sync kubernetes RO service: endpoints "kubernetes-ro" cannot be updated: the resource was updated to 280 E0513 22:55:50. 957 ] unable to sync kubernetes RO service: endpoints "kubernetes-ro" cannot be updated: the resource was updated to 280 E0513 22:55:50. 957 ] unable to sync kubernetes RO service: endpoints "kubernetes-ro" cannot be updated: the resource was updated to 280 W0513 22:55:51. 957 ] Portal net unspecified.

Defaulting to 10.0.0.0/24. I0513 22:55:51. 957 ] Will report 10.0.0.22 as public IP address. E0513 22:56:00. 957 ] Unable to authenticate the request due to an error: Token has been invalidated E0513 22:56:00. 957 ] Unable to authenticate the request due to an error: Token has been invalidated E0513 22:56:00. 957 ] Unable to authenticate the request due to an error: Token has been invalidated E0513 22:56:01. 957 ] Unable to authenticate the request due to an error: Token has been invalidated E0513 22:56:08. 957 ] watch of api.Secret ended with: Stop requested E0513 22:56:08. 957 ] watch of api.ServiceAccount ended with: Stop requested E0513 22:56:08. 957 ] watch of api.Namespace ended with: Stop requested E0513 22:56:08. 957 ] watch of api.ServiceAccount ended with: Stop requested E0513 22:56:08. 957 ] watch of api.Secret ended with: Stop requested E0513 22:56:08. 957 ] watch of api.ServiceAccount ended with: Stop requested FAIL FAIL 111.315s
Do we know which test is failing? The last messages in the log are from , but I don't actually see a test failure message printed, and I can't recreate any failures.

The "Unable to authenticate the request due to an error: Token has been invalidated" messages are expected since the test exercises attempting to use invalid tokens () Also, the "Stop requested" messages indicate that particular test completed and invoked the deferred stopFunc, which shuts down the controllers and closes the test server
Here is the failing test: --- FAIL: TestPersistentVolumeClaimBinder (3.45s) Expected claim to be bound: &{TypeMeta:{Kind: APIVersion:} ObjectMeta:{Name:claim02 GenerateName: Namespace:default SelfLink:/api/v1beta1/persistentVolumeClaims/claim02?namespace=default UID:-f9c3-11e4-96ec- ResourceVersion:172 CreationTimestamp:2015-05-13 22:54:47 +0000 UTC DeletionTimestamp:<nilLabels:map[] Annotations:map[]} Spec:{AccessModes:[ReadOnlyMany ReadWriteOnce ReadWriteMany] Resources:{Limits:map[] Requests:map[storage:{Amount: Format:BinarySI}]} VolumeName:} Status:{Phase:Pending AccessModes:[] Capacity:map[]}}
you aware of the issue with TestPersistentVolumeClaimBinder?
I ran into above issue a lot.
Sorry, hadn't seen this sooner. Lost in the flurry of email traffic. I am looking into it now.
I cannot reproduce on F20, Go 1.3.1 I'll try 1.4 next. Any other info would be appreciated.
Here is the output of docker version from my desktop. Client version: 1.6.2 Client API version: 1.18 Go version (client): go1.4.2 Git commit (client): OS/Arch (client): linux/amd64 Server version: 1.6.2 Server API version: 1.18 Go version (server): go1.4.2 Git commit (server): OS/Arch (server): linux/amd64 I started to run into this issue two days ago, and just ran into it half hour ago right after I sync-ed to HEAD again. Any more information you need here?
Thanks :)
I cannot reproduce this on F20, Go 1.4.2 Any runtime settings I might be missing? I see hack/test- is set to "-race" .


Question: kube::util::download_file "${url}" "${download_file}" tar xzf "${download_file}" ln -fns "etcd-v${ETCD_VERSION}-linux-amd64" etcd <ins> rm "${download_file}" </ins> fi kube::log::info "etcd v${ETCD_VERSION} installed. To use:" kube::log::info "export PATH=$(pwd)/etcd:${PATH}"

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: During k8s cluster launch, is checked and if any etcd binary is found regardless of current OS Distribution, expected etcd binary is not downloaded and etcd soft link is not updated to point desired etcd binary, so cluster creation fails. What you expected to happen: Along with expected etcd version, OS Distribution should also be checked for successfully and efficiently installing etcd during setting up local cluster. How to reproduce it (as minimally and precisely as possible): Checkout k8s codbase on MacOS and launch cluster. working k8s codebase under virtual machine to launch cluster in VM. (Cluster creation would fail, prompting error that etcd needs to be installed and update in {KUBEROOT}/thirdparty/etcd-v{ETCDVERSION}-linux-* Anything else we need to know?: It's an optimization on managing thirdparty etcd binary for local cluster setup. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
do you have a patch that you are working on? we don't really support local-up-cluster on macos yet. /sig testing
is fixing this issue. Though I use Mac as my workstation but sometimes need local-cluster to test codebase so I mapped (softlinked) my GOPATH in virtual machine keeping separate bin directory. Everything is running perfectly fine after making changes mentioned in this PR. So thought to submit small optimization on etcd installation logic.
/close


Question: f := framework.NewDefaultFramework("fake-node") f.NamespacePodSecurityLevel = admissionapi.LevelPrivileged <del> ginkgo.It("should run through the lifecycle of a node", func(ctx context.Context) { </del> <ins> /* Release: v1.32 Testname: Node, resource lifecycle Description: Creating and Reading a Node MUST succeed with required name retrieved. Patching a Node MUST succeed with its new label found. Listing Nodes with a labelSelector MUST succeed with only a single node found. Updating a Node MUST succeed with its new label found. Deleting the Node MUST succeed and its deletion MUST be confirmed. */ framework.ConformanceIt("should run through the lifecycle of a node", func(ctx context.Context) { </ins> // the scope of this test only covers the api-server nodeClient := f.ClientSet.CoreV1().Nodes()

[x] two weeks soak end date : 2024-09-08 According to following APIsnoop query, there are two Node endpoints that are untested. Using a number of existing e2e test practices a new has been created to provide future Conformance coverage for the 2 endpoints. The e2e logs for this test are listed below. This query shows the following Node endpoints are hit within a short period of running this e2e test. If a test with these calls gets merged, test coverage will go up by 2 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted
/sig node
/reopen Tracking the next steps to conformance
Reopened this issue. <detailsIn response to : /reopen Tracking the next steps to conformance Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen Tracking the final step
Reopened this issue. <detailsIn response to : /reopen Tracking the final step Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: } // New initializes a new plugin and returns it. <del> func New(plArgs *runtime.Unknown, handle framework.FrameworkHandle) (framework.Plugin, error) { </del> <ins> func New(_ *runtime.Unknown, handle framework.FrameworkHandle) (framework.Plugin, error) { </ins> return &PrioritySort{}, nil }

has compatibility tests for Policy v1. We are lacking similar tests for other options of , such as: [x] : This can be right away to test that default plugins can be disabled, changed order, etc. [x] : This one is trickier, as it requires a new method in to expose all configurations and a new method in so that they can say which args they used. /sig scheduling /help /kind cleanup
/assign
you can work on the first one. I'll have a PR soon that should clarify how the second one should be done.
I have sent a pr for the first one. And I'm working on the second one, I think it's better to keep 2 PRs for this issue? If so, PTAL at the first one, thanks.
Yes, 2 PRs is ideal. I'll review later today.
I have one small question about do I need to mock some fake plugins for testing , or just testing the default plugins?
You should use the existing plugins, because these are compatibility tests.
Okay
Actually, I think adding a new method such as to interface is a little redundant. In fact, most plugins don't need any argument for building them, so we are talking about adding a new method that needs to be implemented by all of the existing and coming plugins. And users who write out-of-tree plugins also hardly have the need for using this method. IMO, it would be more meaningful to move up to the interface. Then by this method, we can simply implement for default plugins who actually need args. WDYT?
should remain unexported because it provides pointers. However, I agree that might be too much to ask for all plugins. Instead, we can add a new interface and then plugins can opt-int. Not sure if we should follow this extension pattern Any thoughts
On the other hand, simply returning doesn't seem like much.
Sure, then I will add like what we want before(add this method to interface), and if there is any better idea later, please let me know.
I think we should just add it to the plugins that actually have args, this will require custom code in the test, but I think it is better than having all plugins implement a function means something to only 1% of them.
Thanks, but now there is a problem that we can't get the registered plugins with current API.

So, I'm not sure if we can just implement the method in with some reflect code for checking if one plugin has implement method? This could help us with keeping the interface clean. cc
you don't need to check if it implements BuildArgs, since this is for testing, we can hardcode which plugins should have their args verified.
The test will not be pretty, but I think it is worth the tradeoff.
Do you mean that we don't test the default plugins, but define some custom plugins in the test file? And we don't need the ListPluginArgs method anymore, am I right?
and I discussed this offline and arrived to this conclusion: For the compatiblity test, we can mock ( in ) to capture the arguments the configuration that is being passed. This can also be used by with We want another test for that checks the build args for the in-tree plugins accessing the internal pointers. These 2 tests should cover the whole stack. Sorry for the back-and-forth. Sometimes it's useful to see a full PR to realize that the initial idea wasn't good enough. Thanks a lot for it.
Sure, that sounds much more clear, thanks. :D
One more question, the is under package , and the compatibility test is under package . So for mocking , we need to change to a function variable in , and add a method for it, right It will look like: Maybe we can add something like to , but that's the implementation details. I want to make sure if the above design is proper.
Or rather, you could add something like: . Then you can call the capturer from if there is one defined.
Got your message in slack, let me reply here. sounds good, and do I need to revert the change in ? I think is not necessary now.
I think they will still be useful for the other test:
Hello are you still working on this?
Ah, I'm so sorry that I'm so busy in the last few weeks, but I will send prs fot this issue in this weekend. Sorry for my delay again.


Question: create-node-instance-template $template_name } <del> function create-nodes() { local template_name="${NODE_INSTANCE_PREFIX}-template" </del> <ins> # Assumes: # - MAX_INSTANCES_PER_MIG # - NUM_NODES # exports: # - NUM_MIGS function set_num_migs() { </ins> local defaulted_max_instances_per_mig=${MAX_INSTANCES_PER_MIG:-500} if [[ ${defaulted_max_instances_per_mig} -le "0" ]]; then echo "MAX_INSTANCES_PER_MIG cannot be negative. Assuming default 500" defaulted_max_instances_per_mig=500 fi <del> local num_migs=$(((${NUM_NODES} + ${defaulted_max_instances_per_mig} - 1) / ${defaulted_max_instances_per_mig})) local instances_per_mig=$(((${NUM_NODES} + ${num_migs} - 1) / ${num_migs})) local last_mig_size=$((${NUM_NODES} - (${num_migs} - 1) * ${instances_per_mig})) </del> <ins> export NUM_MIGS=$(((${NUM_NODES} + ${defaulted_max_instances_per_mig} - 1) / ${defaulted_max_instances_per_mig})) } # Assumes: # - NUM_MIGS # - NODE_INSTANCE_PREFIX # - NUM_NODES # - PROJECT # - ZONE function create-nodes() { local template_name="${NODE_INSTANCE_PREFIX}-template" local instances_per_mig=$(((${NUM_NODES} + ${NUM_MIGS} - 1) / ${NUM_MIGS})) local last_mig_size=$((${NUM_NODES} - (${NUM_MIGS} - 1) * ${instances_per_mig})) </ins> #TODO: parallelize this loop to speed up the process <del> for i in $(seq $((${num_migs} - 1))); do </del> <ins> for i in $(seq $((${NUM_MIGS} - 1))); do </ins> gcloud compute instance-groups managed  create "${NODE_INSTANCE_PREFIX}-group-$i"  --project "${PROJECT}"

cc:
It was broken by . I'll send a fix in a sec.
Sorry & thanks for fixing.


Question: shift $((OPTIND - 1)) # Use eval to preserve embedded quoted strings. <del> eval "goflags=(${KUBE_GOFLAGS:-})" </del> <ins> eval "goflags=(${GOFLAGS:-})" </ins> eval "testargs=(${KUBE_TEST_ARGS:-})" # Used to filter verbose test output.

We should just use GOFLAGS
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
/sig contributor-experience
/assign


Question: package services import ( <del> "flag" </del> "fmt" "io/ioutil" "math/rand" <del> "net/http" </del> "os" "os/exec" <del> "os/signal" </del> "path" "path/filepath" <del> "reflect" "strconv" </del> "strings" <del> "syscall" "time" </del> "github.com/golang/glog" "github.com/kardianos/osext"

In node e2e, we have the option to keep the services running after test for further debug. However, we set to for the services, which means that once the parent exits, they will receive and exit with the parent if they don't trap the signal. With this, option will never work. If we still need this feature, we should not set if is false. /cc
We still need this feature. It helps with debugging test failures by keeping all the daemons running upon failure. , Lantao Liu wrote:


Question: return dial(updatedReq, h.UpgradeTransport) } <ins> // getResponseCode reads a http response from the given reader, returns the status code, // the bytes read from the reader, and any error encountered func getResponseCode(r io.Reader) (int, []byte, error) { rawResponse := bytes.NewBuffer(make([]byte, 0, 256)) // Save the bytes read while reading the response headers into the rawResponse buffer resp, err := http.ReadResponse(bufio.NewReader(io.TeeReader(r, rawResponse)), nil) if err != nil { return 0, nil, err } // return the http status code and the raw bytes consumed from the reader in the process return resp.StatusCode, rawResponse.Bytes(), nil } </ins> // dial dials the backend at req.URL and writes req to it. func dial(req *http.Request, transport http.RoundTripper) (net.Conn, error) { conn, err := DialURL(req.Context(), req.URL, transport)

(9.8, critical) With a specially crafted request, users that are authorized to establish a connection through the Kubernetes API server to a backend server can then send arbitrary requests over the same connection directly to that backend, authenticated with the Kubernetes API server’s TLS credentials used to establish the backend connection. Thanks to Darren Shepherd for reporting this problem. CVE-2018- is fixed in the following Kubernetes releases: Kubernetes API server Kubernetes v1.0.x-1.9.x Kubernetes v1.10.0-1.10.10 (fixed in ) Kubernetes v1.11.0-1.11.4 (fixed in ) Kubernetes v1.12.0-1.12.2 (fixed in ) Note: If you are using binaries or packages provided by a distributor (not the ones provided in the open source release artifacts), you should contact them to determine what versions resolve this CVE. Distributors may choose to provide support for older releases beyond the ones maintained by the open source project. Clusters = 1.6.x that run aggregated API servers (like the metrics server) that are directly accessible from the Kubernetes API server’s network. If there are aggregated API servers configured in a cluster, the following command will return the names of the associated APIService objects (if no names are listed, or the kube-apiserver is an older version that does not have the apiservices API, then the cluster has no aggregated API servers configured): Clusters = 1.0.x that grant pod exec/attach/portforward permissions to users that are not expected to have full access to kubelet APIs An API call to any aggregated API server endpoint can be escalated to perform any API request against that aggregated API server, as long as that aggregated API server is directly accessible from the Kubernetes API server’s network. allows all users (authenticated and unauthenticated) to perform discovery API calls that allow this escalation against any aggregated API servers configured in the cluster. A pod exec/attach/portforward API call can be escalated to perform any API request against the kubelet API on the node specified in the pod spec (e.g. listing all pods on the node, running arbitrary commands inside those pods, and obtaining the command output). Pod exec/attach/portforward permissions are included in the admin/edit RBAC roles intended for namespace-constrained users. This section lists possible mitigations to use prior to upgrading.

Note that many of the mitigations are likely to be disruptive, and upgrading to a fixed version is strongly recommended. suspend use of aggregated API servers (note that this will disrupt users of the APIs provided by the aggregated server) disable anonymous requests by passing to the kube-apiserver (note that this may disrupt load balancer or kubelet health checks of the kube-apiserver, and breaks setup flows) remove all anonymous access to all aggregated APIs (including discovery permissions granted by the ) suspend use of aggregated API servers (note that this will disrupt users of the APIs provided by the aggregated server) remove all access to all aggregated APIs (including discovery permissions granted by the ) from users that should not have full access to the aggregated APIs (note that this may disrupt users and controllers that make use of discovery information to map API types to URLs) Remove pod exec/attach/portforward permissions from users that should not have full access to the kubelet API There is no simple way to detect whether this vulnerability has been used. Because the unauthorized requests are made over an established connection, they do not appear in the Kubernetes API server audit logs or server log. The requests do appear in the kubelet or aggregated API server logs, but are indistinguishable from correctly authorized and proxied requests via the Kubernetes API server.
, why doesn't kubelet's webhook authorization help here? Shouldn't it reject subsequent requests on still open TCP connection from APIServer?
Because the kube-apiserver <-kubelet connection was established with the kube-apiserver's TLS credentials, which are broadly authorized against the kubelet API. The kubelet would authorize the kube-apiserver to make that request, and allow it.
Hmm, I misunderstood how webhook authorization works then, I thought that no requests are trusted in that mode and even for /proxy requests, initiator's credentials are passed by APIServer to kubelet, which then comes back to apiserver for SubjectAccessReview. So I thought scheme is like following: Initiator -APIServer (authn/authz) -kubelet -APIServer (SubjectAccessReview) But seems it is not the case :(
the credentials authorized by the kubelet in that case are the apiserver's
Thanks for the fix and the detailed summary! Just for clarification - this vulnerability can't be exploited by anonymous users from outside of the cluster, as long as we disable , right?

I'm asking because my users tend to open up K8S API endpoint to the Internet, occasionally but not always restricting accesses to the endpoint at L4 by the source IP.
Correct, escalations start with a single authorized request through kube-apiserver to the backend. Preventing that (either by disallowing anonymous requests with --anonymous-auth=false or by removing the default discovery permissions granted to anonymous users) mitigate the anonymous escalation. Be aware that can disrupt normal operations of a cluster, including load balancer and kubelet detection of kube-apiserver health (via anonymous requests to /healthz), and kubeadm join flows that depend on public access to a config map containing cluster information in the namespace.
Initiator -APIServer (authn/authz) -kubelet -APIServer (SubjectAccessReview) Interesting, are there any drawbacks of passing initiator's credentials all the way to kubelet? Would it allow to eliminate one more "superuser" in the form of APIServer?
Exposing initiator's credentials broadly to the cluster is a non-goal. Additionally, many API callers have non-forwardable credentials (TLS client certificates). The apiserver could forward information about the initiator to the kubelet, but that information would have to be trusted (e.g. initiator info in headers, verified by mTLS connection proving the apiserver provided the info). Interestingly, that is exactly the scenario exercised by kube-apiserver -aggregated API server connections, where the kube-apiserver acts as an authenticating proxy, and the aggregated API server trusts the identity assertions sent to it by the kube-apiserver. not really, as long as kube-apiserver is responsible for terminating and asserting authentication information to backends.
- I'm having a hard time understanding the full extent of this. If the API is in a private network (no public exposure), with RBAC enabled, but still around, would it still be vulnerable to this?
The first step in the escalation is a properly authorized request to the kube-apiserver that gets proxied to a backend server. A malicious request must be able to reach the kube-apiserver for the vulnerability to be exploited.
That's what I thought, thanks so much for taking the time to answer so quickly!
Thanks for the clarification !
Hi, I want to know why the issue isn't fixed in 1.9.x?

We have a lot of clusters running the 1.9.x k8s.
Pardon stupid question, but is Heapster considered aggregated API server same as metrics server?
The Kubernetes project supports the most recent three minor releases with bug and security fixes. The currently supported releases are 1.10.x, 1.11.x, and 1.12.x. If you are running builds provided by a distributor (not the binaries or packages provided by the Kubernetes project), contact their support to determine which of their versions are vulnerable. Some distributors may apply fixes to versions prior to the open-source-supported versions. No, Heapster is not an aggregated API server. Aggregated API servers register objects with non-nil fields, and requests to endpoints on the kube-apiserver are proxied to the appropriate aggregated server.
Are there any detail procedures to reproduce the issue? So we can verify that if we have fixed it on 1.9.x. Thanks a lot.
How to reproduce the issue, so we can check if the kubernetes version we're using is affected by this issue? An example script/command shows how gets all pods information on that node can be helpful.
Is it enough to just update the API Server to a fixed version, or should all the components (scheduler, kubelet, etc.) be updated to the fixed Kubernetes version to be secure?
, fix is in the apiserver codebase, so it should be enough to update just it
Does this affect ELB/ALB/Ingress controllers in any way? I mean if K8s API is hidden in a private network but load balancers for services (not for K8s API) are wide open, is it possible to use this vulnerability?
If the request cannot reach the API servers, the exploit cannot work. Even if the request can reach the API servers, but not have the permission to establish backend/proxy connection, it is also fine. But if a user in your cluster can use kubectl "freely", it means that they can connect to K8s API server, and the CVE may be applied here. Or, if the services that run on your cluster has a role that allows pod exec/attach/port-forward, it can proxy the request from the service to the API server.

Or if one of your services are API server extension, and your users can interact with it directly, then he/she can craft the exploiting request
Could someone give an example of an aggregated API endpoint that is usually enabled by default?
the metrics server is an aggregated API server, so is often a valid endpoint
What is the safest way to "suspend use of aggregated API servers" and to "remove all anonymous access to all aggregated APIs" () ? Is it sufficient to move aside the proxy-client-key-file mentioned here: ?
Don't laugh at this question. is Kubernetes version 1.7.11 affected? I would assume so. But if anyone can confirm this will help me out considerably. Thanks James Sorry I won't add another comment, I'll edit this one to thank you for your reply below, I mis-read the post on affected versions, I did not see 1.0.x-1.9.x, Sorry for wasting your time with the question.
I would say Yes, as Kubernetes v1.0.x-1.9.x 1.7.x falls within that range.
Sorry to bug you, but it would be a huge help to get an answer to my questions in .
I know the official docs on upgrading state to to from 1.7 to 1.8 then to 1.9 and then to 1.10, Would anyone know of a way to from 1.7.11 to 1.10.11. If not we will just figure out the steps and do the 3 separate upgrades. If something similar has been done before we'd like to hear about it as we have a few customers on 1.7.11 and all need to get upgraded. We will of course do our own trial and error in a lab and try and script much of it.
I don't understand why this leads to security issues. what is the "In default configurations"?
Without knowledge of your particular cluster and use, that is difficult to say. The safest way to suspend use of aggregated API servers might be to leave their objects registered, and scale down the deployments backing those objects, so that no aggregated server instances are available to receive requests.

Some potential disruptions that would cause: If portions of the cluster depend on an aggregated server (like HPA depending on metrics-server), this would break those dependencies aggregated servers that cannot be reached cause the garbage collector and namespace cleanup controllers to wait to perform some operations until the unavailable server can be reached again. This could prevent deletion of namespaces, and prevent foreground deletion of objects until the servers are made available again Removing anonymous access would either consist of disabling anonymous requests entirely with (see issue description for possible problems with that approach) or removing authorization permissions from anonymous users (specifically the user and group). If using , that includes removing the anonymous subjects from the and on restart. Keep in mind that preventing anonymous access still leaves the cluster vulnerable to escalation by authenticated users. Moving the would just prevent your kube-apiserver from starting Skip-version upgrading is not supported. A low-privilege request (like an anonymous call to a discovery API listing available resource types, like ) can be escalated to make any API call (including reading and writing API objects) to the aggregated server (like ). The default configuration refers to the discovery permissions granted to all users by .
on a 1.9.5 version i have it as --anonymous-auth=false and i deny all unauthenticated requests
In our Kubernetes cluster (Azure ACS, Kubernetes version# 1.8.4), we are using kubeconfig in order to establish proxy connection with kubernetes cluster. This is limited to only certain operations team members. It means in order to set connection with kubernetes api server, certificate from kubeconfig is used. Also, all the apis hosted on this cluster are exposed via azure api management service. In that case, is our kubernetes cluster still applicable to above vulnerability?
you mentioned "Aggregated API servers register APIService objects with non-nil spec.service fields" ... so is this a good query to see if someone has one of those in their environment?
This would work without jq: Added to the description
Gravitational has created a vulnerability test utility:
Under "affected configurations", it seems to suggest that if you run the given kubectl command and have no output, you do not have any exploitable endpoints for the aggregation api portion of this vulnerability. However, in "vulnerability impact" immediately below it states "In default configurations, all users (authenticated and unauthenticated) are allowed to perform discovery API calls that allow this escalation.".

Does this actually more precisely mean that the ability to discover vulnerable api calls that might exist is enough access to perform the exploit, if you happen to have exploitable endpoints? Asking because we appear to have no output from the kubectl command, but I can anonymously call "get --raw /apis" with . Unsure if this cluster is currently exploitable or not via this mechanism.
- clarified the "Affected configurations" and "Vulnerability impact" sections
thanks
Great thanks to !
there are two tests circulating for configurations that are vulnerable to the aggregated API exploits: the test you to the issue description, and which posted above. As mentioned in this issue: their results aren't always the same. In fact, we have clusters that are not affected according to your test but are affected according to . Could you clarify whether a negative from your test plus a positive from Gravitational's test indicates that we are not exposed?
the gravitational tests for ... however, if there are NO aggregated apis (as shown by 's command) then the cluster is still ok (hopefully can confirm this). You are still vulnerable to the other code path so you will have to review the RBAC permissions.
If there are no objects that route to a service (the check), that means no aggregated servers are configured in the cluster, and the anonymous escalation via aggregated discovery API requests is not possible. That said, if you are still running a vulnerable version and later add an aggregated API server, it would become exploitable.
Thanks very much for the clarity.
What about Kubernetes <1.5.x that no have "apiservices" daemon running/installed? And Could you please explain "user", which user do you mean. User that have shell access without root-access to kube-masters (what i think) and/or user from outside through an hosted application (ex. apache server on running pod-API-Server) and/or user that have general access to kube machines but without shell access. Thanks in advance
The APIService API was alpha in 1.6.x, and beta and enabled by default in 1.7.x+. kube-apiserver releases without an APIService API do not have the ability to proxy to aggregated API servers, so anonymous escalation via aggregated discovery API requests is not possible on those versions. Any actor that can send a network request to the kube-apiserver.

Escalation paths for this vulnerability begin with a network request to the kube-apiserver, which must first pass authorization, then be proxied to a backend (either to a kubelet in the case of pod exec/attach/portforward, or to an aggregated API server in the case of aggregated APIs).
The pod exec/attach/portforward escalation path applies back to 1.0.x
Fixed in for
Updated the description with a link to the document and recording of the post-mortem for this incident. Links to action items are in the document and reference this issue.
/label official-cve-feed (Related to )


Question: } crd, err := crd.CreateMultiVersionTestCRD(f, group, func(crd *v1beta1.CustomResourceDefinition) { <del> apiVersions := []v1beta1.CustomResourceDefinitionVersion{} for _, version := range versions { v := v1beta1.CustomResourceDefinitionVersion{ </del> <ins> var apiVersions []v1beta1.CustomResourceDefinitionVersion for i, version := range versions { apiVersions = append(apiVersions, v1beta1.CustomResourceDefinitionVersion{ </ins> Name:    version, Served:  true, <del> Storage: false, } apiVersions = append(apiVersions, v) </del> <ins> Storage: i == 0, }) </ins> } <del> apiVersions[0].Storage = true </del> <ins> crd.Spec.Versions = apiVersions </ins> crd.Spec.Validation = &v1beta1.CustomResourceValidation{ OpenAPIV3Schema: props,

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: ci-kubernetes-e2e-gci-gce-alpha-features Which test(s) are failing: [sig-api-machinery] CustomResourcePublishOpenAPI [Feature:CustomResourcePublishOpenAPI] works for multiple CRDs of same group but different versions [sig-api-machinery] CustomResourceConversionWebhook [Feature:CustomResourceWebhookConversion] Should be able to convert a non homogeneous list of CRs [sig-api-machinery] CustomResourcePublishOpenAPI [Feature:CustomResourcePublishOpenAPI] works for CRD without validation schema` [sig-api-machinery] CustomResourcePublishOpenAPI [Feature:CustomResourcePublishOpenAPI] works for multiple CRDs of same group and version but different kinds [sig-api-machinery] CustomResourcePublishOpenAPI [Feature:CustomResourcePublishOpenAPI] updates the published spec when one versin gets renamed [sig-api-machinery] CustomResourceConversionWebhook [Feature:CustomResourceWebhookConversion] Should be able to convert from CR v1 to CR v2 [sig-api-machinery] CustomResourcePublishOpenAPI [Feature:CustomResourcePublishOpenAPI] works for multiple CRDs of different groups [sig-api-machinery] CustomResourcePublishOpenAPI [Feature:CustomResourcePublishOpenAPI] removes definition from spec when one versin gets changed to not be served Since when has it been failing: Failing since 5/16 . There were two PRs related to API machinery and tests that were merged during this period and Testgrid link: Reason for failure: The first failure seems to be this: [sig-api-machinery] CustomResourcePublishOpenAPI [Feature:CustomResourcePublishOpenAPI] works for multiple CRDs of same group but different versions /cc /priority critical-urgent /kind failing test /milestone v1.15 /cc
/assign sttts
/assign
looks like there were some alpha gated tests that need adjusting after
/unassign
On it.
another PR touching CRD OpenAPI publishing was merged yesterday and we didn't run alpha presubmit test let me know if you think it's specific to CRD openapi publishing and I can look into it
debugging now
is now green /close
Closing this issue. <detailsIn response to : is now green Instructions for interacting with me using PR comments are available .

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: // Ignores claims that already have a storage class. // TODO: if resync is ever changed to a larger period, we might need to change how we set the default class on existing unbound claims func (ctrl *PersistentVolumeController) assignDefaultStorageClass(claim *v1.PersistentVolumeClaim) (bool, error) { <del> if claim.Spec.StorageClassName != nil { </del> <ins> if storagehelpers.GetPersistentVolumeClaimClass(claim) != "" { </ins> return false, nil }

We observed that when deplying manifests to a cluster running 1.26.1-gke.200, the StorageClass and PVC get created successfully. However, the PVC is stuck in a Pending state and no PV is provisioned. The workaround is to add storageClassName to one of the pending PVCs on cluster, and it did go from Pending to properly provisioned a PV and reaching Bound state. We looked at the logs and found that according to the doc: , Is this annotation is fully deprecated in the new 1.26.1 release? If yes can we add this detail to documentation or release note? the PVC should not stuck in a Pending state and PV is provisioned in 1.26.1 a 1.26.1-gke.200 cluster in GKE via yaml The error comes form the code: and There are some hypotheses here: The annotation does not work on 1.26.1 any more, so a default storage class is being populated to the PVC. For some reason, the did not work, so the PVC was deployed with an empty StorageClass nil. Then the kube controller manager tried to update the PVC with the default StorageClass, which triggered the error: <details</details<detailsGKE </details<details</details<details</details<details</details<details</details
/sig storage
Annotation support seem to be still intact. Check the e2e test
From the point of view of the code, this annotation is still in use. I'll try to see if I can reproduce it.
It's a bug introduced in The feature is enabled in 1.26 by default.
/triage accepted


Question: roundedVal: int64(1024000), }, { <del> name:        "round overflowed quantity to int64", resource:    resource.MustParse("73786976299133170k"), </del> <ins> name:       "round decimal to MiB", resource:   resource.MustParse("1.2Gi"), roundedVal: int64(1229), }, { name:       "round big value to MiB", resource:   resource.MustParse("8191Pi"), roundedVal: int64(8795019280384), }, { name:        "round quantity to MiB that would lead to an int64 overflow", resource:    resource.MustParse("8192Pi"), </ins> roundedVal:  int64(0), expectError: true, },

After upgrading our clusters to Kubernetes 1.19, the provisioning of PersistentVolumes started failing with the following Events: The value matches the actual storage request of 3.2Gi. It doesn't seem special in any way, not too big, not too small, and it always worked until 1.19. We look into the issue with and we believe the PR introduced a change that broke the PersistentVolume provisioning even when it doesn’t overflow int64. Amounts of resource in Kubernetes are represented with the . In the PersistentVolume drivers, there are some functions that manipulate storage request in order to allocate a disk with the right size. Before the mentioned PR, those quantities were retrieved by calling which relies on to return the value as an int64. This worked probably fine in most of the cases, but it could happen that the represented value would overflow an int64, and in this case the result of would be wrong. The function’s description states that it’s the caller’s responsibility to check this: This is likely what the PR wanted to handle. The change replaced all calls with which has a different signature. From where it’s used now, you would think that the second return value signals an overflow: : This is probably where the error we’re seeing is generated. I believe that the signature of was misinterpreted. From my understanding of its description, returning doesn’t mean we’re overflowing . It only means that internally, the quantity is represented as a decimal amount: Internally, depending on their values, quantities are stored/represented in : for DecimalExponent (e.g. ) or DecimalSI (e.g. ) for BinarySI that don’t have a decimal part but a unit (simplified) (e.g. ) for all the rest With the PR, Kubernetes now fails to provision volumes that have a storage request represented as . Many of the storage requests are represented this way and still perfectly valid in our opinion. I would have expected the provisioning to continue to work as those requests seemed valid and were always accepted by our underlying cloud-provider in earlier version of Kubernetes.

Create a StatefulSet with a PersistentVolumeClaim that uses vSphere storage: Kubernetes version (use ): 1.19.7 Cloud provider or hardware configuration: vSphere, but I think it impacts other cloud-provider that were changed in the PR: /sig storage /kind bug /cc Possibly related to
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I think the only reason we do not use the quantity library for scaling and why we tried to roll our own rounding code is because when scaling stuff in AsScaledInt64(3) or ScaledValue(3), the library assumes decimal SI (M, 1000), not binary SI units (Mi, 1024): You are right, our self-rolled rounding code doesn't handle values with millibyte or higher precision like 3.2Gi. Our code uses AsInt64 and then performs its own scaling. If AsInt64 returns false, the int64 returned is not in bytes (it must be in millibytes or higher precision), so our rounding code won't work, so it returns that " is too great, overflows int64". In 1.18, we used Value() to round to bytes but we didn't check it against ScaleValue as the signature suggested. So the PR in question fixed cases where Value() overflowed but inadvertently broke your case. The correct way to round, which the resource library's rounding does, is to drop the useless precision and round up. , but as I said it doesn't seem to handle binary SI quantities correctly. For decimal SI quantities, we should rely on the resource library's AsScaledInt64 and be done with it. (We could also use Value() and ScaledValue as the signature suggests but calling AsScaledInt64 directly seems simpler). Like this: For binary SI quantities, we need to do something fancier involving CanonicalizeBytes I think, I'm not sure yet.
I guess for binary SI we can go back to using Value but actually check the overflow this time.

Hi Thanks for the quick feedback on the problem's description already. Indeed, the Scaled() functions don't handle BinarySI and that's why I quickly discarded the idea of using them. Since this is about storage, I suppose most people are using this format. I just finished wrapping up a PR that I hope address both the issue we are facing, but also what was meant to be fixed in the PR your reviewed back then. It doesn't make any difference between the 3 formats however.
Yeah I commented within seconds of your PR. I glanced at it , will review more soon, but it makes sense to me to just use VAlue and check ofr overflow for both decimal SI and binary SI rather than using Scaled() for one and not the other :+1:
I think it means both. i.e - when incoming value is decimal or if value is so big that type can't store it as it switches to decimal. So while my PR fixed later case, it broke the former. :/


Question: exit 1 fi fi <del> if [[ "${KUBE_SKIP_UPDATE}" == "y" ]]; then return fi # update and install components as needed if [[ "${KUBE_PROMPT_FOR_UPDATE}" != "y" ]]; then gcloud_prompt="-q" fi local sudo_prefix="" if [ ! -w $(dirname `which gcloud`) ]; then sudo_prefix="sudo" fi ${sudo_prefix} gcloud ${gcloud_prompt:-} components install alpha || true ${sudo_prefix} gcloud ${gcloud_prompt:-} components install beta || true ${sudo_prefix} gcloud ${gcloud_prompt:-} components install kubectl|| true ${sudo_prefix} gcloud ${gcloud_prompt:-} components update || true </del> <ins> update-or-verify-gcloud </ins> } # Validate a kubernetes cluster

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): Is this a BUG REPORT or FEATURE REQUEST? (choose one): FEATURE REQUEST <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): all :smile: Environment: Cloud provider or hardware configuration: GCE OS (e.g. from /etc/os-release): all Kernel (e.g. ): all Install tools: Others: What happened: uses the cli a lot and does a (in the commonly used function ) before doing things like . updates can introduce breaking changes (for example in the recent 132.0.0 release it removed a bunch of flags). This has caused cross-the-board test failures like . But this is not limited to just tests. Any kubernetes GCE users who rely on the and commands could have been broken. What you expected to happen: Stop updating gcloud automatically. We could either warn user to have the required dependencies installed, or pin to specific versions of dependencies like gcloud. How to reproduce it (as minimally and precisely as possible): download 1.4.4 release tarball run and Anything else do we need to know:
cc
/cc
cc I think it's reasonable to remove gcloud update calls from scripts, as I believe gcloud is updated out of band by k8s continuous testing. The scripts will still need to work with latest, so maybe they should check for a minimum version.
Good idea.
Yeah, we definitely want to continue using latest in CI so that we know if things break, but I think it's reasonable to remove the update itself from the kubernetes cluster util scripts.


Question: pod, err = c.CoreV1().Pods(ns).Get(pod.Name, metav1.GetOptions{}) Expect(err).NotTo(HaveOccurred()) <del> // Wait for `VolumeStatsAggPeriod' to grab metrics time.Sleep(1 * time.Minute) // Grab kubelet metrics from the node the pod was scheduled on kubeMetrics, err := metricsGrabber.GrabFromKubelet(pod.Spec.NodeName) Expect(err).NotTo(HaveOccurred(), "Error getting kubelet metrics : %v", err) framework.Logf("Deleting pod %q/%q", pod.Namespace, pod.Name) framework.ExpectNoError(framework.DeletePodWithWait(f, c, pod)) </del> // Verify volume stat metrics were collected for the referenced PVC volumeStatKeys := []string{ kubeletmetrics.VolumeStatsUsedBytesKey,

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: What you expected to happen: Test should not flake /sig storage /cc
/assign
GitHub didn't allow me to assign the following users: vkamra. Note that only can be assigned. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
GitHub didn't allow me to assign the following users: vkamra. Note that only can be assigned. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
- You won't be able to assign this to me (not a kubernetes member). Am taking a look.
volumemetrics.go_ could/should have more logging, either calling or more calls to .
The test appears to fail on this metric: "kubeletvolumestatsused_bytes". So samplePVC != pvcName and/or sampleNS != namespace. It would be nice if the sample name and ns were also logged. Also something is weird here: the framework's is called before the local . Note in the log:
- will add a couple of Logf() statements to log the samples - you're right that will help. Once the volume is attached, volume stats are grabbed every interval. The thing I noticed in the logs is that we issue a delete for the pod at but the volume only got attached/bound successfully at The test assumes blocks till the pod is running and PVC is attached but maybe that's not a valid assumption?
Nevermind - That log message has 2 timestamps in it - the volume was provisioned at .
Pushed The working theory being that volume stats were not collected within the wait earlier being used.
did you figure out if I was correct in thinking that the framework was invoked before the local ? That seems very, very unlikely but that is how it looked in the log.
- I believe you're right that the framework is being invoked before the local one.

I observe the same in a test as well, so I don't think it's specific to this call. Based on the , nested blocks should be invoked inner-outer. It's not clear, however - whether the local is nested - or at the same level as the framework block when I look at


Question: namespace, name, err := namer.Name(req) if err != nil { <del> notFound(w, req.Request) </del> <ins> errorJSON(err, codec, w) </ins> return }

e.g. REST.Update should never return an arbitrary error value. Maybe we should change the return code to something stricter than generic error.
/cc
FYI: is for remaining cases where the API doesn't return JSON/Status errors.
Re-opening this until we
I think etcd errors require more abstraction, also:
There are some admission control plugins that did not always follow that practice which I will resolve as part of
Are there any known instances of this remaining?
None that I know of.


Question: } // NewRCByName creates a replication controller with a selector by name of name. <del> func NewRCByName(c clientset.Interface, ns, name string, replicas int32, gracePeriod *int64) (*v1.ReplicationController, error) { </del> <ins> func NewRCByName(c clientset.Interface, ns, name string, replicas int32, gracePeriod *int64, containerArgs []string) (*v1.ReplicationController, error) { </ins> ginkgo.By(fmt.Sprintf("creating replication controller %s", name)) <ins> if containerArgs == nil { containerArgs = []string{"serve-hostname"} } </ins> return c.CoreV1().ReplicationControllers(ns).Create(framework.RcByNamePort( <del> name, replicas, framework.ServeHostnameImage, 9376, v1.ProtocolTCP, map[string]string{}, gracePeriod)) </del> <ins> name, replicas, framework.ServeHostnameImage, containerArgs, 9376, v1.ProtocolTCP, map[string]string{}, gracePeriod)) </ins> } // RestartNodes restarts specific nodes.

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: ci-kubernetes-e2e-gci-gce-serial Which test(s) are failing: [sig-cluster-lifecycle] Nodes [Disruptive] Resize [Slow] should be able to add nodes [sig-cluster-lifecycle] Nodes [Disruptive] Resize [Slow] should be able to delete nodes Since when has it been failing: These tests have been failing since 6/28 Failures coincide with the following changes: Testgrid link: Reason for failure: The failure messages look as follows: [sig-cluster-lifecycle] Nodes [Disruptive] Resize [Slow] should be able to add nodes [sig-cluster-lifecycle] Nodes [Disruptive] Resize [Slow] should be able to delete nodes /cc /milestone v1.16 /priority critical-urgent /kind failing-test /cc
/sig cluster-lifecycle
For more context, the code for the failing tests is here The error for both comes through  - some Pod is not responding. It is failing to find pods prefixed with "my-hostname-add-node". Within the logs we can see this: So the Pods are stuck in a Pending state. The Pods are supposed to be created by a ReplicationController in , which itself uses this method from the e2e framework . However, after being in a Pending state (after the "failed to wait for pods responding: timed out waiting for the condition" error is registered), they do seem to reach a Ready state
/assign for triage.
I think agnhost needs to be passed , but did not do so in /assign sig-testing
GitHub didn't allow me to assign the following users: sig-testing. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : I think agnhost needs to be passed , but did not do so in Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Reopen to monitor job /reopen
Reopened this issue.

<detailsIn response to : Reopen to monitor job /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Thank you for working on the PR to solve this. indeed solved the issue reported here. /close
Closing this issue. <detailsIn response to : Thank you for working on the PR to solve this. /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: testingclock "k8s.io/utils/clock/testing" "k8s.io/utils/pointer" <ins> "github.com/google/go-cmp/cmp" </ins> "github.com/stretchr/testify/assert" )

During implementation of KEP 3393 (PodReplacementPolicy for Jobs) we found that the unit tests of do not follow common patterns established in k/k. 1) The tests are not written using table driven design (). 2) The tests use rather than . This probably should be done as two separate PRs but wanted to highlight some areas that could use some cleanup. Not using means that we don't see diffs in the unit tests that makes this difficult. And not using table driven tests makes it difficult to add new test cases as it usually is brought up during review that we should use a table-driven test but the entire file does not obey this format.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig apps /kind cleanup /help /good-first-issue
This request has been marked as suitable for new contributors. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /sig apps /kind cleanup /help /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I'd like to work on it /assign
I'd also like to make some contributions. /assign
Looks like this need two PRs. tests are not written using table driven design (). tests use Differences rather than Cmp.Diff. I think I can work on the 2nd one and leave the 1st one to you.

Sounds good?
By looking through the , I found the main function to comapre the results have already used cmp.Diff. Can you elaborate more on which part you expect to use cmp.Diff? func verifyDeletedAndPatchedPods(t testing.T, client fake.Clientset, wantDeletedPodNames, wantPatchedPodNames sets.String) { t.Helper() deletedPodNames := getDeletedPodNames(client) if diff := cmp.Diff(wantDeletedPodNames, deletedPodNames); diff != "" { t.Errorf("Deleted pod names (-want,+got):n%s", diff) } patchedPodNames := getPatchedPodNames(client) if diff := cmp.Diff(wantPatchedPodNames, patchedPodNames); diff != "" { t.Errorf("Patched pod names (-want,+got):n%s", diff) } }
I am really sorry about this. Too much multitasking!! I meant not
Hey ! Sorry, I didn't see this before. No problem at all, but consider that one change depends on the other, and since my task implies a big refactor, there might be conflicts when merging your PR... That's why I see two possible scenarios happening: Your PR gets merged before I raise mine: Then fine, I'll rebase your changes and adapt my code Mine is ready before yours is merged: In this case, I propose you commit your changes to my PR. Both of us will appear as coauthors of the change Getting a PR merged in the Kubernetes project takes quite some time (trust me, I talk by experience and can confirm ) WDYT?
Let's see how it goes. We can always rebase the code change to adapt new.
btw, did you already begin to work on task 1? If not, I may just go ahead and work on it right away since I have already dig into it.
yes, I already started And If you ask me, better to keep it this way so we both have an opportunity to have a contribution
cool. Sounds good.
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hi if this issue needs to be worked on any further, please assign me for the same.
has a PR up that will solve the second part of this issue.


Question: BlockVolume:                                 {Default: false, PreRelease: utilfeature.Alpha}, StorageObjectInUseProtection:                {Default: true, PreRelease: utilfeature.Beta}, ResourceLimitsPriorityFunction:              {Default: false, PreRelease: utilfeature.Alpha}, <del> SupportIPVSProxyMode:                        {Default: false, PreRelease: utilfeature.Beta}, </del> <ins> SupportIPVSProxyMode:                        {Default: true, PreRelease: utilfeature.Beta}, </ins> SupportPodPidsLimit:                         {Default: false, PreRelease: utilfeature.Alpha}, HyperVContainer:                             {Default: false, PreRelease: utilfeature.Alpha}, NoDaemonSetScheduler:                        {Default: false, PreRelease: utilfeature.Alpha},

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug /sig network /area ipvs What happened: Per , kubeadm init doesn't support the --feature-gates=SupportIPVSProxyMode=true flag anymore since IPVS is in beta and not alpha anymore. We should enable IPVS feature gateway by default to make kubeadm set up IPVS proxy. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/assign


Question: "volumeName":       "VolumeName is the binding reference to the PersistentVolume backing this claim.", "storageClassName": "Name of the StorageClass required by the claim. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#class-1", "volumeMode":       "volumeMode defines what type of volume is required by the claim. Value of Filesystem is implied when not included in claim spec.", <del> "dataSource":       "This field requires the VolumeSnapshotDataSource alpha feature gate to be enabled and currently VolumeSnapshot is the only supported data source. If the provisioner can support VolumeSnapshot data source, it will create a new volume and data will be restored to the volume at the same time. If the provisioner does not support VolumeSnapshot data source, volume will not be created and the failure will be reported as an event. In the future, we plan to support more data source types and the behavior of the provisioner may change.", </del> <ins> "dataSource":       "This field can be used to specify either: * An existing VolumeSnapshot object (snapshot.storage.k8s.io/VolumeSnapshot - Beta) * An existing PVC (PersistentVolumeClaim) In order to use VolumeSnapshot object types, the appropriate feature gate must be enabled (VolumeSnapshotDataSource) If the provisioner can support the specified data source, it will create a new volume based on the contents of the specified PVC or Snapshot. If the provisioner does not support the specified data source, the volume will not be created and the failure will be reported as an event. In the future, we plan to support more data source types and the behavior of the provisioner may change.", </ins> } func (PersistentVolumeClaimSpec) SwaggerDoc() map[string]string {

/kind feature /sig storage /sig api PVC Data Sources have been in beta since 1.16 and should be ready to be promoted to GA in the 1.18 release. KEP:
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /kind feature /sig storage /sig api Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: } } <ins> func (e *eventBroadcasterImpl) Shutdown() { e.Broadcaster.Shutdown() } </ins> // refreshExistingEventSeries refresh events TTL func (e *eventBroadcasterImpl) refreshExistingEventSeries() { // TODO: Investigate whether lock contention won't be a problem

In order to initialize a client-go , you need an . and friends initialize a new : launches a goroutine: This goroutine is stoppable with . However, the interface returned above does not expose anything that calls that shutdown method, AFAICT. This means that that goroutine cannot ever be cleaned up. That's not great. It's fine if you're using in like k/k does and never stopping anything, but if you ever want to stop your controllers and start up new ones, it becomes an issue. Tested/tracked down against master (as of ). /sig apimachinery /kind bug
The label(s) cannot be applied. These labels are supported: <detailsIn response to : In order to initialize a client-go , you need an . and friends initialize a new : /kind bug </details
/sig api-machinery I always forget the dash


Question: p.t.Errorf("invariant violated, specifying resource version (%s) is not allowed when using continue (%s).", options.ResourceVersion, options.Continue) return nil, fmt.Errorf("invariant violated") } <ins> if options.Continue != "" && options.ResourceVersionMatch != "" { p.t.Errorf("invariant violated, specifying resource version match type (%s) is not allowed when using continue (%s).", options.ResourceVersionMatch, options.Continue) return nil, fmt.Errorf("invariant violated") } </ins> var list metainternalversion.List total := options.Limit if total == 0 {

We recently started using the list pager from client-go as part of Calico, and have seen reports of the following error showing up: No error, just beautiful paged list calls. Create 1,000 resources of a particular type, such as a NetworkPolicy. (Need 500 to trigger paging) Do a of all those resources with set. Originally filed here (but maybe that repo is not monitored): Reference issue in Calico: projectcalico/calico (comment) Reference Calico list pager code: We think this is the required fix: <detailsclient-go version (but bug seems to be in master too). </details<details</details<details</details<details</details<details</details<details</details
/sig api-machinery
/triage accepted


Question: CPUQuota:   rOpts.CpuQuota, CPUPeriod:  rOpts.CpuPeriod, CpusetCpus: rOpts.CpusetCpus, <ins> CpusetMems: rOpts.CpusetMems, </ins> } createConfig.HostConfig.OomScoreAdj = int(rOpts.OomScoreAdj) }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --The memory manager fails to set the when used with the docker shim The memory manager should succeed to set the when used with the docker shim. Run memory manager e2e test on the environment with more than one NUMA node. Kubernetes version (use ): master Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node


Question: if [[ -n "${KUBE_DOCKER_IMAGE_TAG-}" && -n "${KUBE_DOCKER_REGISTRY-}" ]]; then for arch in "${KUBE_SERVER_PLATFORMS[@]##*/}"; do <del> kube::log::status "Building hyperkube image for arch: ${arch}" REGISTRY="${KUBE_DOCKER_REGISTRY}" VERSION="${KUBE_DOCKER_IMAGE_TAG}" ARCH="${arch}" make -C cluster/images/hyperkube/ build </del> <ins> # TODO(IBM): Enable hyperkube builds for ppc64le again if [[ ${arch} != "ppc64le" ]]; then kube::log::status "Building hyperkube image for arch: ${arch}" REGISTRY="${KUBE_DOCKER_REGISTRY}" VERSION="${KUBE_DOCKER_IMAGE_TAG}" ARCH="${arch}" make -C cluster/images/hyperkube/ build fi </ins> done fi }

disabled PPC builds due to the following error: Additional comments here
cc
The issue on PowerPC is both for cross and native-builds. The issue appears to be with the toolchain (golang) and how the linking is being done. The object sections have grown too large to link. We are looking at it cc
Have you tried to upgrade to a newer version? It might or might not help... You might also try to look at what in the gcc linker differs from for example, where it doesn't fail
We are looking into this and it might take some more time. In the meanwhile is it possible to remove hyperkube from the ppc64le builds? All other binaries are building perfectly fine. We can resume the hyperkube for ppc64le build as soon as this issue resolves. cc:
Ah, this is only partially fixed. Reopening...
The error occurs because the hyperkube binary is very large and when building the binary, golang is generating a single go.o file with a single text section that is larger than 2^26 on ppc64le. When there is a call instruction within that section where the call target offset is beyond what can fit in the instruction the relocation truncation error will be generated by the GNU linker. When there are multiple, smaller text sections sent to the GNU linker for linking, it can generate appropriate call instructions even the call target is beyond that limit. I'm working on a fix to do that but hitting other issues as a result which I'm working through. I suspect that this same error will be hit at some point on arm64 unless their linker has another way to handle this situation. On ppc64le we are very close to the limit and from build to build it varies so that some days it will build and others it won't.
Interesting. How does amd64 handle it?
I don't know for sure how arm64 handles it because I don't have access to a machine that is set up to build for arm64. I suspect their text section size is not as large as it is on ppc64le so they haven't hit the limit yet.
Ok, you may easily see what's going on for arm64 by investigating what does in kube-cross.

But I asked about what amd64 (x86_64) does to prevent this :) (I'm assuming this will never happen to amd64)
Sorry I misread your question about amd64... The call instruction in amd64 has a 32 bit field instead of a 24 bit field in ppc64le for the call target address. That means they wouldn't hit this error when linking hyperkube until its text section was larger than 2^32. If you search on "relocation truncated to fit" you will see that this error has happened in the past on x86 with other programming languages.
Once a fix is ready, how does it need to be provided?
I saw https://go- probably won't make 1.7, and that's sad. We would just upgrade golang version I think, not really sure what you're asking for
I assume you only use official golang versions or those provided with distros for your builds? Are you doing all your builds as cross builds from a golang on an x86?
Official versions, yes. We're using the docker image. All official builds are built on amd64 hosts.
Where are you getting this docker image from? Will you be moving to golang:1.7 when it is released?
We're using the official images, and will pick up as soon as it is released for testing I think. Then if the testing and scalability teams think it's safe to use, we'll switch to 1.7 BTW, where is your patch for this? Can you link to it?
is also broken right now - see .
Hi, I'm sorry I missed your last question to me. Here is the CL for my patch for the PPC64 relocations truncation error. https://go- The error with kubemark is also size related but somewhat different because that one is being built as static. Just curious, why are some built static and some non-static?
just to tie everything together, disabled linux/ppc64le compilation entirely. hopefully by go1.8 we can reenable it.
Well, we've hit 1.8 I think with Want to link to any other relevant issues to kinda tie this one together, and close this if it seems appropriate?
yeah, ppc64le builds have been re-enabled for k8s 1.6.

The issue on PowerPC is both for cross and native builds. The issue appears to be with the toolchain (golang) and how the linking is being done. The object sections have grown too large to link. We are looking at it cc
this issue is quite old. Can you provide more context to your comment? Specifically, more detail to help me reproduce. I was unable to find anything related to ppc64le on .


Question: ) var ( <del> // Use buckets ranging from 5 ms to 2.5 seconds (admission webhooks timeout at 30 seconds by default). latencyBuckets       = []float64{0.005, 0.025, 0.1, 0.5, 1.0, 2.5} </del> latencySummaryMaxAge = 5 * time.Hour // Metrics provides access to all admission metrics.

I would like to propose the graduation of the following apiserver metrics from alpha to stable: With the exception of the metric these metrics have been stable for 2 years. The metric has been stable for 12 months.
/sig api-machinery /sig instrumentation
/assign /triage accepted
I'm okay with promoting all of these except I am hesitant about since it is relatively new. Can we omit that one?
I agree with the metric was patched a year ago as part of , so it would be best to wait a bit more until the majority of Kubernetes users catch up to the patched version and start using the new version of the metric in production.
A couple of renaming suggestions, since it is important to get it right before graduation: apiserverresponsesizes =apiserverresponsesizebytes aggregatorunavailableapiservice =aggregatorunavailableapiservices One concern regarding . It was very recently changed but the change was later reverted by In the end, the metric wasn't really changed, but it might raise the concern that some users may still need additional information from this metric so I am on the fence with this one. Side implementation note that we will need to keep in mind if we graduate is that the graduation should only affect the webhook admission metric and not the other, so we will need to pass the stability level to
Thank you and for the feedback. To clarify then, are the decisions to not graduate and final?
I'm actually okay with , the namespace addition was recent but it was a bad idea and was reverted. It has otherwise been in use for quite some time, so I'd be comfortable with promoting it.
I'm willing to make a PR for this one
+1 it wasn't a hard no from my side, just wanted to raise some concerns in case anyone had an objection.
/assign /unassign
Thank you for the feedback, and To summarize, we are in agreement to graduate all but the metric. Please correct me if I'm wrong.
/cc
since all the three metrics that are created with this function are graduating to stable, do you think I should still pass the to the function or should I just set the metrics created by this functions as stable by default? update: Seems like is designed to parse whatever's in front of . So putting a variable there (that is passed to the function) will cause the script to fail. Please correct me if I'm wrong.

update: Even setting the metrics to be stable by default will cause the script to fail and you'll get an error saying for the following line: It appears that the name of the stable metrics should be given to that object as a string and using the output of a function ( in this case) will cause the script to fail.
The PR is ready to be reviewed:


Question: "github.com/GoogleCloudPlatform/kubernetes/cmd/kubelet/app" "github.com/GoogleCloudPlatform/kubernetes/pkg/api" <ins> "github.com/GoogleCloudPlatform/kubernetes/pkg/credentialprovider" </ins> _ "github.com/GoogleCloudPlatform/kubernetes/pkg/healthz" "github.com/GoogleCloudPlatform/kubernetes/pkg/kubelet" "github.com/GoogleCloudPlatform/kubernetes/pkg/master/ports"

As per We should look for the .dockercfg in three places: ./.dockercfg, ~/.dockercfg, and $rootDir/.dockercfg
I'll tackle this.
Bunch of questions: we also add a flag that overrides the search mechanism suggested in this issue? Perhaps, ? to the user's homedir requires we use os/user [0] . The os/user package and cross-compiling don't play nice [1]. So, if we add to the search path, cross-compiled kubelet binaries will not actually be able to take advantage of it. Is this an issue? is the proper lookup order? I suggest: if provided, , , . walking through the lookups, do we stop at the first file that exists, or the first file that exists and is parsed as a valid ? [0] [1] /cc
I would absolutely love a --dockercfg option, and it would also be helpful if it logged which .dockercfg it ultimately used.
1) Seems like YAGNI to me 2) Is all of os/user broken? We will eventually want actual user support. I'm fine with dropping it for now. 3) I would say $rootDir/.dockercfg, ./.dockercfg (assuming we drop ~) 4) First successful parse sounds good to me. And can we NOT log errors if files don't exist :)
would it make sense to default to the kubelet config directory if provided rather than the working directory of the kubelet process? then no extra flag required.
What ever happened with this idea?
/cc
It never got high enough on my weekend TODO list :( I can keep telling myself that I need to do it, or we can find someone who has a bit more time than me.
I can probably get to this today.
Cool, thank you! , Mike Danese wrote: