Question: return true } <ins> // Zeros out extended resource capacity during reconciliation. func (kl *Kubelet) reconcileExtendedResource(initialNode, node *v1.Node) bool { requiresUpdate := false for k := range node.Status.Capacity { if v1helper.IsExtendedResourceName(k) { node.Status.Capacity[k] = *resource.NewQuantity(int64(0), resource.DecimalSI) node.Status.Allocatable[k] = *resource.NewQuantity(int64(0), resource.DecimalSI) requiresUpdate = true } } return requiresUpdate } </ins> // updateDefaultLabels will set the default labels on the node func (kl *Kubelet) updateDefaultLabels(initialNode, existingNode *v1.Node) bool { defaultLabels := []string{

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: In 1.11, removed externalID from Node.Spec which used to be used by kubelet to detect node recreation in certain environments during node upgrade. This changed node upgrade behavior in those environments because in tryRegisterWithAPIServer(node *v1.Node), without externalID checking, we now consider the recreated node after upgrade as a previously registered node and thus will get old node state from API server instead of regenerating them based on node local state. For extended resource reported by a device plugin, this causes problem because after node upgrade/recreation, the device plugin DaemonSet pod needs to be restarted by Kubelet, finishes certain setup, and then report to kubelet that its resource is available at the node. However, the node status capacity and allocatable from the API server still contains the old state before the node upgrade. After kubelet syncs up with the API server, the node status field gets overwritten and previously reported extended resources will appear on the node status capacity/allocatable even though the node is not ready for pods to consume such resources. As the result, Kubelet may start a pod requesting such resource without proper container runtime setup it needs to learn from device plugin. Note to cope with the externalID removal change, has been updated () to explicitly uncordon the node after checking it is restarted and becomes ready. This however doesn't help device plugin case because node readiness doesn't depend on individual extended resource readiness. This issue is to explore how we can fix this problem on head and 1.11. In particular, I wonder whether we may switch to the model that we consider Kubelet as the only source to update extended resource capacity/allocatable in node status. I.e., don't support manually updating node status capacity/allocatable fields as documented in This would allow us to re-generate node status capacity/allocatable every time in node status update, which I think is much simpler and more robust. As I heard, some folks have been using this mechanism to do simple resource exporting/accounting through a central controller. I wonder whether people are open to switch to the device plugin model in those cases.

Even though those extended resources may not require special container setup, it seems a more secure model to have Kubelet totally own its node status. FYI, here is the related OSS issue: where we first introduced extended resource concept. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Running the gpu upgrade test from with --upgrade-target set to any 1.11+ version would hit the issue. Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig node
/cc
Having extended resources be a concept encapsulating both device plugin resources and user set resources (opaque int resources) has been a hassle to deal with (as it introduced unexpected behavior in the device plugin lifecycle). We should also consider a few alternatives : separating them from a code perspective (i.e: different fields in node status) . separating them from a name (/dns) perspective.
Agree with If we do need to support the use case of exporting a resource in node status capacity/allocatable not through Kubelet, seems better to introduce special naming convention for such resources.
We had a discussion in another issue about using the check point to keep the node in not ready until all expected device plugins have re-registered. Is that option still on the table? I’m hesitant to suggest changing the semantics of extended resources again, at least until we are sure it’s a valid long term solution.
We did implement the mechanism to use check point data to determine previously registered device plugin resource and keep the resource not ready till the expected device plugin re-register. It helps normal node reboot case but not for node upgrade/recreation case, because boot disk is recreated in the later case and the check point data is also gone.
Kubelet's device manager should recover gracefully from a deleted checkpoint. It should be able to identify which resources in the node object are extended resources and it should remove them if the checkpoint disappears. Checkpoints can disappear for any number of reasons and kubelet needs to react appropriately. cc since this is fallout from . Device manager was relying on node object deletion/recreation to function after GCE upgrades.

+1 Node API object deletion/recreation was only for the case where you have a completely different instance (and deletion was only done because the external ID was made immutable). Deletion/recreation in response to an upgrade or machine restart is a form of data loss (admin- taints, labels, etc, are unrecoverable). Additionally, depending on Node object deletion on upgrade as previously seen in GCE is highly environment-dependent and will not be consistent between cloud-providers, environments, etc.
how about during node registration reconciliation, we reconcile extended resources by setting their capacity to zero? That means any external controller that exports extended resources by directly patching node status capacity/allocatable needs to re-do that after any kubelet restart, which I guess is already what they are doing to handle node upgrade. The required change should be small:
what are the implications of doing that on every kubelet process restart? does available resource flutter impact already-running workloads on a node (does the kubelet evaluation of PodFitsResources evict running workloads? does any controller notice workloads running on a node are using resources the kubelet says it doesn't have, and evict them?)
For extended resources exported through device plugin, kubelet device manager will handle their restarts, currently based on checkpoint data and in the future, we will explore whether we may store this information in some form of API. If the checkpoint data is gone, pods will fail kubelet admission handling and the pod's controller should recreate a new pod. For extended resources exported by an external controller through direct node status capacity patching, the controller should watch for kubelet readiness change, and repatch the field after kubelet becomes ready again.
Having an external controller watching the nodes status update to repatch the nodes sounds like a bad "user" experience. Maybe looking at it from a user's perspective and then figuring out what we'll do is a better way to look at this?

The behavior I'd expect to see is the following: For non device plugin extended resources A user patches the node status with the extended resource If the node restarts I don't expect the Allocatable / Capacity fields to change If the node is upgraded I don't expect the Allocatable / Capacity fields to change If the node is recreated I don't expect the Allocatable / Capacity fields to change For device plugin extended resources Kubelet advertises the extended resources when the device plugin registers itself against kubelet If the node restarts I expect the Allocatable field to be 0 until the resource is ready but still advertise its last known number of devices If the node is upgraded I expect the Allocatable field to be 0 until the resource is ready but still advertise its last known number of devices If the node is recreated I expect the Allocatable field to be 0 until the resource is ready but still advertise its last known number of devices As for the case where the checkpoint data is gone, should we expect the pod's controller to recreate a new pod? That seems like a surprising behavior are we doing this somewhere else? Does the general behavior outlined here seem sane to the people in this thread?
I think in 1.12 time frame, we should start to scope the work on improving extended resource life cycle. Right now, as we are approaching 1.11 code freeze timeline, I am trying to figure out a solution that is least surprise to any existing workloads. Till 1.11, if there is indeed a real use case where an external component directly patches node status capacity to export a resource, the solution already needs to handle the capacity loss after node upgrade by repatching the field. That is the least surprising solution I can think of now. If folks have concerns on the breaking behavior this may introduce, we should seriously consider reverting part of in 1.11.
+1 for considering reverting for v1.11 timeframe. What would be the side-effect of that ? Ownership of fields in the Node object has been a long standing issue. For example node labels suffered from the same issue. I do agree that kubelet has to be resilient and has to recover from control plane data loss with minimal impact on running applications. Having to expose arbitrary extended resources as device plugin resources is also not ideal.

For instance, if one wants to expose inodes or network bandwidth as a resource, device plugins would not be an ideal interface (we can consider fixing this). k8s node upgrades should ideally involve cordoning (or tainting) the node during the duration of the upgrade. During this upgrade phase the node is primarily expected to keep existing pods running (if the upgrades are in place) and the system (scheduler) is not expected to place additional workloads. Since the node object is a shared resource, every component that relies on the node object must ensure eventual consistency. Arbitrary users are not expected to update/patch the node object since it is core to k8s functionality. does node feature discovery continually update the state on the node object to reflect the current state on the node? Until we have a better resource representation format, kubelet will not be able to tag ownership for individual extended resources attached to the node object. I think this limitation is acceptable and the users who introduce additional extended compute resources are expected to be sophisticated enough that they can run long running processes to ensure their additional compute resources are accurately reflected in the node object. As part of Resource Classes, and v2 compute resource APIs, we should consider incorporating a owner reference into concrete compute resource objects which can help with the ownership issues. Node upgrades also need non-trivial automation today to ensure that all node dependencies (runtime, kernel, OS, network, storage plugins, etc.) are logically ready prior to adding a node back to the scheduling pool. Assuming patches to node compute resources are continually updated and nodes are cordoned during the upgrade phase, node upgrade automation can ensure that all the expected extended compute resources on certain nodes are updated to a non-zero capacity prior to adding a node back to the scheduling pool (uncordon) after an upgrade. In addition, additional automation can be employed to auto-taint nodes that do not expose non-zero capacity for expected extended resources. For the near future, kubelet can on-startup can only try to restore state on a best effort basis. If checkpoints are not around, kubelet may override existing patched fields. Device plugins and other extended resource plugins will have to ideally update capacity/allocatable to reflect the state on the local node prior to marking an upgrade as complete.

it would delay removal of the self-deletion permission for yet another release (already pushed to 1.12 because of version-skew), meaning node taints would not be usable for isolating compromised nodes until at least 1.14 (because the associated kubelet could simply delete/re-register their Node object). I also don't think it would actually solve this issue. According to , even were we to revert and restore kubelet self-deletion, changes made in stabilized the computation of external id in GCE such that it no longer changes as part of node upgrades in 1.11+. Undoing that, destabilizing the GCE computed external id, and reintroducing self-deletion cases to solve the stale resource issue in one particular cloud environment doesn't seem like a good approach.
The feature that would be slipping is which is particularly critical for and because it's required to run a cluster with dedicated nodes securely. We would be slipping another release on something that is already painfully slow due to backward compatibility considerations. The PR () that changed the behavior device manager was depending on merged two months ago. It's really unfortunate that this report came in so late. I'm pretty convinced that this is a device manager issue that is only mitigated by clearing the node object. Is there anything I can do to help fix this? I'm happy to argue that changes here should be exempt from the codefreeze deadline.
The node feature discovery project only deals with node labels, and not extended resources. NFD fairly recently the ability to run as a daemon, but also retained the option to publish labels once and exit.
So far, it sounds is perhaps the most realistic solution for 1.11 time frame. Although we want to support good user experience, I do feel we need to balance the complexity we need to add into the system v.s. the use cases we want to optimize for. I feel the ask for the external component to repatch node status capacity after kubelet restart seems reasonable and perhaps is already implemented by any serious use cases.
The change that was backported to 1.10 (and maybe all versions?) for this seems to have had the unexpected consequence of zeroing out all of our extended resources upon kubelet restart and causing a large number of our pods to get evicted in our QA cluster since they all use extended resources.

I filed an issue here: if you have any advice on intentions/workarounds. For now, we reverted that commit in our internal build as a workaround. Thanks!
How is your extended resource exported on a node? Are they being exported by a device plugin or do you directly update your node objects? If it is the later case, do you have a central controller that may watch for node updates and re-export your resources?
When the host is first configured, our configuration management system running on the node starts the kubelet service, waits for kubelet to register itself in the API by polling and then patches the node capacity directly. We do not have a central controller to rexport the resources. It also sounds like watching and rexporting is not sufficient, since the value would still hit zero for some brief period of time before the central controller could react, at which point kubelet will have already begun hard evicting all the pods. The only central solution I think would work is doing it from an admission controller that ignores the kubelet's attempt to set the extended resources in our namespace to zero.
Can the configuration management system re-patch node capacity after each Kubelet registration?
The kubelet service can restart at any time, whether the config management system is currently configuring the system or not. Ex: kubelet could get OOM-killed, crash, fail healthchecks, etc. If every kubelet restart is a kubelet registration, then no, unless we made the kubelet service also launch a script every time it restarts. So lets say that we hypothetically did configure a script to re-patch the node capacity every time kubelet restarts: there is still an unsolvable race condition where the node temporarily has its capacity set to zero. Consider the situation: is up and healthy. The value for . kubelet service restarts reconciliation logic sets . kubelet immediately begins hard-evicting every pod that requested a . script patches the node capacity back to , but it's too late, every pod has already been killed non-gracefully by kubelet due to the resource having been reset back to 0 at step 3.
Sigh, this is the deficiency we documented in Unfortunately, I couldn't think of a clean solution without API change. One thing we may consider is to instead of zeroing out extended resource capacity, we delete them.

Because we are currently taking an open policy on extended resource admission, pods already assigned on the node wouldn't be affected, and your configuration manager can re-patch node later. Another possible workaround you may consider is to implement device plugin API in your configuration manager. Because Kubelet maintains device plugin checkpoint, it would allow pods to pass admission based on the checkpointed state. cc for thoughts on this issue.
Note for the first mentioned workaround, i.e., instead of zeroing out extended resource capacity, we delete them, I think we will rely on to re-populate device plugin resource runtime states during container restart retries. This is not ideal but we already need this to handle the edge cases that node API object may get deleted but pods are still assigned to node, or pods can be admitted before node re-register with API server.
Unfortunately, I couldn't think of a clean solution without API change. The effect of this change as it stands is that kubernetes users run the risk of having 100% of pods in their cluster completely wiped out with no respect for PDB, terminationGracePeriodSeconds, and preStop hooks just by restarting the kubelet processes. This really feels like a critical bug that has much worse consequences than the bugs the change sought to solve. If this is going to continue being the behavior, there should be a really loud disclaimer on which currently makes it appear that extended resources are trivial to manage. I had a team of 5 people restoring 350 stateful database clusters to a clean state for 3+ hours on Monday night. Restarting kubelets was never a destructive action before, so we weren't restarting them as slowly and carefully as if we were rolling out a config change. If this had happened in our production cluster instead, it could have brought the company down for hours. So you're saying that simply removing the entry from the capacity map instead of zeroing it would allow pods using those resources to continue running? When you re-patch the capacity, does that break the accounting at all? (Ex if 3 pods were consuming an extended resource, and the resource is deleted and re-, does the scheduler realize that 3 of that resource are still consumed by pods, or would it reset to thinking zero pods are using it or would it count them correctly?) I'm willing to do that if it's really necessary.

That said, I couldn't find the code that does that during kubelet registration in 1.10, what happens if the device plugin is down during a kubelet restart? Does kubelet then zero out the resource? Does the device plugin need to be running for a container using the resource to successfully start? As a possible suggestion that doesn't require an (http) API change or the cluster admins to run a device plugin daemon, has it been considered to implement similar to ? on kubelet. That way the admin would have a trivial way to explicitly tell kubelet to always register with a static extended resource capacity value, making the reconciliation process much more straightforward.
Paul Furtado <: state. It would see three pods are already assigned to the node and wouldn't assign more. being scheduled on the node till the resource is ready and device plugin re-registers. However, in Kubelet device plugin manager, we use check-pointed state to bypass predicate admission handler so running pods wouldn't be affected. The function that does this is at: Jiaying
Could we add a whitelist configration to skip reconcile specified extended resource? Or skip a specific prefix
That would require changing kubelet flags, which I am hoping to avoid.
but we have no way to distinguish the extended resource is reported by device plugin or patched by users directly. Seems we can't use non device plugin extended resources anymore. b/c we can't guarantee to repatch resources before pods evicted.
It is indeed a question whether we want to support the use case of directly patching node object to export a resource. Capacity/Allocatable are node status fields. By definition, they should be observable from the node. I.e., even when Kubelet is isolated from the API server, it should be able to reconstruct its status fields. I think even without zeroing out extended resource capacity during API registration, there is perhaps already a race between pod admission and API registration. They are performed by separate go routines on Kubelet and there is no guaranteed ordering between these two processes. This wan't a problem in early versions of 1.10 because we changed extended resource admission into open policies. It remains a question whether we want to keep this open admission policies or not.

Adding an explicit Kubelet flag to specify what extended resources (and perhaps also their capacities) it should export may be a better way than directly patching node API object, because it would make that information truly observable. Whether we have enough use cases to justify this flag is perhaps a question we should discuss in sig-node.
Could we have a "generic device plugin" / "extended resource device plugin"? e.g: You deploy this device plugin and start interacting with a CRD rather than patching the node object in the spec?
For this case, shall we take the following ways to avoid pod restarts when kubelet restart? (1) we can use the default namespace prefix ,"" to avoid being zero out (2) remove extended resource before kubelet start, and patch add again when kubelet started Both of the two ways can make pod with extended resource not evicted then kubelet restart


Question: <ins> /* Copyright 2016 The Kubernetes Authors All rights reserved. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package net import ( "net" "reflect" ) // IPNetEqual checks if the two input IPNets are representing the same subnet. // For example, //	10.0.0.1/24 and 10.0.0.0/24 are the same subnet. //	10.0.0.1/24 and 10.0.0.0/25 are not the same subnet. func IPNetEqual(ipnet1, ipnet2 *net.IPNet) bool { if ipnet1 == nil || ipnet2 == nil { return false } if reflect.DeepEqual(ipnet1.Mask, ipnet2.Mask) && ipnet1.Contains(ipnet2.IP) && ipnet2.Contains(ipnet1.IP) { return true } return false } </ins>

This is at least partially responsible for - some tests can't run when dns is down. Logs from just before the crash: It got killed because Kubelet decided to kill it (readiness probe failed): It started happening quite recently - I'm trying to locate a build which caused this. Repro: run "Kubelet should not restart containers across restart" test - it caused this behavior for me multiple times today. cc - I applied the P0/v1.3 label/milestone, as I think it should be a blocker. Feel free to decrease the priority if you disagree
gce-serial 1511 seems to be the first one with this problem.
Or 1510 - hard to tell, as it was a broken run. There's no PR around that looks too suspicious though:/
I locally reverted and it seems that DNS is no longer failing. Sadly I have to run to catch a train, but maybe someone can pick it up here ?
Taking a look now. The PR should be unrelated...I am checking to see if the Federation DNS changes went in yet. Add nethealth container to prepull manifest
I search the restart counts for jobs from 1510-latest and gce-serial Job 1540 is the first one to show a non-zero kubedns restart count. If there is a commit responsible, it would be between 1539 and 1540 (June 21 7 AM -11 AM) 1539 end of run 1540 end of run Latest
The kubedns:1.3 container hasn't been updated since (Sat, 04 Jun 2016 03:29:37.847 GMT). GET
cc The kubedns container code is from June 04th. Dawn suggested that its networking issues that are causing kubedns container to fail its readiness check. I will need to add logic to print kubedns container logs during e2e tests, I am trying to search for any logs that might help, otherwise I will have to debug the next run live.
As reported, a simple kubelet restart triggers this issue - it looks like 10.0.0.1 is unreachable from the kubedns container.

The kubedns container blocks indefinitely waiting for a connection to the kubernetes service at 10.0.0.1 before setting up the healthz handlers - the readiness probe will fail until kubedns successfully connects to the API server.
- you can try reverting and running it again - it worked for me for some reason (it might be by accident though)
I think the root cause is that cbr0 doesn't have an IP address assigned after kubelet restart - all the pods running behind cbr0 are isolated from the network. Not only kube-dns, but after my kubelet restart, the other addon pods are also stuck in a crash loop.
This sounds like a blocker even more now.
Looks like a regression. I am investigating.
Found the root cause. There is a subnet ip mismatch. vs . Hence kubenet thinks they are different and clears the address. Will submit a fix soon.
i found coredns status is CrashLoopBackOff and i check the logs : i got tips from so i restart my docker , i check the logs , there is no error in logs again


Question: } } <ins> // Enforce CLI specified namespace on server request. if o.EnforceNamespace { o.VisitedNamespaces.Insert(o.Namespace) } </ins> // Generates the objects using the resource builder if they have not // already been stored by calling "SetObjects()" in the pre-processor. infos, err := o.GetObjects()

What happened: While testing out the flag, I created a manifest with the following contents: I ran , and the resources appeared on my minikube cluster: Then, I removed the deployment from the manifest: And re-ran the same command, . To my surprise, however, the deployment wasn't removed; the only output from kubectl was . What you expected to happen: I expected the pruner to find the deployment with the matching label, , and prune it. How to reproduce it (as minimally and precisely as possible): I believe my example above is minimal and easily reproducible. Anything else we need to know?: This small change to the code for resolves the issue: Environment: Kubernetes version (use ): v1.16.3 Cloud provider or hardware configuration: N/A OS (e.g: ): N/A Kernel (e.g. ): N/A Install tools: N/A Network plugin and version (if this is a network-related bug): N/A Others: N/A
/sig cli
/assign cc


Question: // And Go client will follow redirects automatically so the 301 check is probably unnecessary. if resp.StatusCode == http.StatusTooManyRequests || resp.StatusCode == http.StatusServiceUnavailable { retryAfter := resp.Header.Get("Retry-After") <del> if seconds, err := strconv.Atoi(retryAfter); err != nil { </del> <ins> if seconds, err := strconv.Atoi(retryAfter); err == nil { </ins> backoff = seconds + 10 } fmt.Fprintf(os.Stderr, "Got %d visiting %s, retry after %d seconds.n", resp.StatusCode, string(URL), backoff)

This issue was reported in the Description Kubernetes includes a linkcheck command, intended to check the correctness of links within Kubernetes documentation. The command uses the strconv.Atoi function to parse the Retry-After response header. However, the backoff value that is later used to wait between making another linkcheck attempt is updated only if strconv.Atoi returns an error. In other words, a valid value of Retry-After header value is not used by linkcheck: Figure 34.1: Retry-After header parsing in linkcheck command Furthermore, note that: .Atoi uses strconv.ParseInt, which has edge cases for parser failures, which will return MAXINT for values too large and zero for values that fail to parse as integers. does not set an explicit redirect policy via CheckRedirect, meaning that Go’s core HTTP library will follow redirects. While linkcheck has a whitelist for URLs, redirects are not checked against the whitelist. Retry-After response header is used only in the case of “Too many Requests” (HTTP status code 429). However, the Retry-After header may also be returned with both 301 (Redirect) and 503 (Service Unavailable) status codes as well. Exploit Scenario Alice wishes to build a copy of Kubernetes. Unbeknownst to Alice, Eve has inserted a link into the documentation that returns a redirect to a URL that was initially blocked by the linkcheck link regex whitelist. Recommendation Short term, use the retryAfter value and always include reasonable minimum and maximum values for all untrusted data. These should meet the roughly-expected timelines of an operation. For example, if a server responds with a Retry-After header that is longer than 1 minute, mark the link as inactive and continue on. Define a CheckRedirect policy for HTTP clients, and ensure that developers and end-users may control if they want redirects to be followed. Long term, ensure that all timeouts and similar operations have both a maximum and a minimum value. This will prevent events from happening more frequently than developers expect, or from taking too long under imperfect situations. Additionally, ensure that any code that relies on standards such as HTTP adequately follows the standard. Anything else we need to know?: See for current status of all issues created from these findings. The vendor gave this issue an ID of TOB-K8S-003 and it was finding 35 of the report.

The vendor considers this issue Informational Severity. To view the original finding, begin on page 83 of the Environment: Kubernetes version: 1.13.4
/cc
/sig release /area security /kind bug /remove-kind feature
header would be date time as well.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/lifecycle frozen
/assign


Question: }) } } <ins> func TestCleanScope(t *testing.T) { testCases := []struct { name          string requestInfo   *request.RequestInfo expectedScope string }{ { name:          "empty scope", requestInfo:   &request.RequestInfo{}, expectedScope: "", }, { name: "resource scope", requestInfo: &request.RequestInfo{ Name:              "my-resource", Namespace:         "my-namespace", IsResourceRequest: false, }, expectedScope: "resource", }, { name: "namespace scope", requestInfo: &request.RequestInfo{ Namespace:         "my-namespace", IsResourceRequest: false, }, expectedScope: "namespace", }, { name: "cluster scope", requestInfo: &request.RequestInfo{ Namespace:         "", IsResourceRequest: true, }, expectedScope: "cluster", }, } for _, test := range testCases { t.Run(test.name, func(t *testing.T) { if CleanScope(test.requestInfo) != test.expectedScope { t.Errorf("failed to clean scope: %v", test.requestInfo) } }) } } </ins>

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --the issue stays in the CleanScope function where all GET calls will end up with scope "namespace" and end up with verb "LIST" Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /assign /sig api-machinery
/triage accepted


Question: pods, err := podClient.List(labels.SelectorFromSet(labels.Set(map[string]string{"time": value})), fields.Everything()) Expect(len(pods.Items)).To(Equal(1)) <del> By("retrieving the pod") podOut, err := podClient.Get(pod.Name) if err != nil { Fail(fmt.Sprintf("Failed to get pod: %v", err)) } By("updating the pod") value = "time" + value pod.Labels["time"] = value pod.ResourceVersion = podOut.ResourceVersion pod.UID = podOut.UID pod.Spec.Host = podOut.Spec.Host pod, err = podClient.Update(pod) if err != nil { Fail(fmt.Sprintf("Failed to update pod: %v", err)) } </del> <ins> // Standard get, update retry loop expectNoError(wait.Poll(time.Millisecond*500, time.Second*30, func() (bool, error) { By("updating the pod") value = strconv.Itoa(time.Now().Nanosecond()) if pod == nil { // on retries we need to re-get pod, err = podClient.Get(name) if err != nil { return false, fmt.Errorf("failed to get pod: %v", err) } } pod.Labels["time"] = value pod, err = podClient.Update(pod) if err == nil { Logf("Successfully updated pod") return true, nil } if errors.IsConflict(err) { Logf("Conflicting update to pod, re-get and re-update: %v", err) pod = nil // re-get it when we retry return false, nil } return false, fmt.Errorf("failed to update pod: %v", err) })) </ins> expectNoError(waitForPodRunning(c, pod.Name)) By("verifying the updated pod is in kubernetes") pods, err = podClient.List(labels.SelectorFromSet(labels.Set(map[string]string{"time": value})), fields.Everything()) Expect(len(pods.Items)).To(Equal(1)) <del> fmt.Println("pod update OK") </del> <ins> Logf("Pod update OK") </ins> }) It("should contain environment variables for services", func() {

This is wrong because something else could have updated the pod in the mean time, invalidating your resource version. (Specifically, kubelet writes statuses.) It should do a get/update loop that retries as long as it gets a conflict error. to find an owner
PR almost ready - just struggling to get e2e tests to run :-)
At last. SUCCESS! -- 5 Passed 34 Skipped I0424 09:04:47. ] All tests pass PR incoming in a few minutes.


Question: }, "completionTime": { "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.Time", <del> "description": "Represents time when the job was completed. It is not guaranteed to be set in happens-before order across separate operations. It is represented in RFC3339 form and is in UTC." </del> <ins> "description": "Represents time when the job was completed. It is not guaranteed to be set in happens-before order across separate operations. It is represented in RFC3339 form and is in UTC. The completion time is only set when the job finishes successfully." </ins> }, "conditions": { <del> "description": "The latest available observations of an object's current state. More info: https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/", </del> <ins> "description": "The latest available observations of an object's current state. When a job fails, one of the conditions will have type == "Failed". More info: https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/", </ins> "items": { "$ref": "#/definitions/io.k8s.api.batch.v1.JobCondition" },

The API docs for jobs are unclear as to what fields a caller would use to determine job success or failure in 1.11 (and we haven't updated them in a while) A casual reader would infer that completionTime is set when the job fails (but it isn't). Linking out to the docs is ok, but I shouldn't have to click through the links to understand how to use the Job object. The Job description and the job status fields should clearly indicate what fields are set in completion states on the job, because that's 100% of the point of docs. Taking a summary of the jobs-run-to-completion link and putting it in , , and job would improve the actual utility of our API doc. We should also do a pass on other objects and make the actual in server docs more useful.
as a consumer of the API, I had to read way too much to know how to use jobs correctly and how to write an API client that properly handled the conditions. Also, we should consider either setting completionTime on failures, or adding failedTime (the former may not be forwards compatible), because reading the conditions is too complex for a fundamental use case like this. This matches our decision w.r.t. conditions vs. fields - conditions are useful for extension and extra data, but status should have fields that allow an API consumer to easily understand state.
Also... our docs don't actually explain how to determine the job is complete via the API, so there's arguably no place this is captured.
I have the same problem right now, of correctly identifying a finished job, when it does not complete successfully and after reading the documentation and the reference and all the examples I could find, it's still not clear how I can check the conditions and make sure the job is finished when I do so: -what does each mean? can I rely on , is that set only when the job finishes, whether via or ? what other reasons are possible? It would definitely help to have specified on failure as well, and the differentiation to be made via the presence of .
/assign soltysh
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with .

Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
I have to say this issue is still very relevant. As of now, it is very hard to design something around Job Statuses. Anyway to move this forward?
Completely agree with previous comments! Even I found the way how to detect completed/failed jobs, still not sure am I covered all use cases. This part definitely needs clarifications/improvements/better documentation.
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
With regards to: If you actually follow this link and then search on that page for "conditions" you will get exactly zero hits. This is extremely unhelpful.
This documentation really, really needs work. (So does the API, for that matter.)
So what is the correct way to determine if a job is finished and whether it completed successfully?
When a Job completes (finished successfully), its will be set, and Job conditions will have one with . When a Job fails, Job conditions will have one with .
Hi, if nobody has been working on it, can I take this one? I have gone through the issue which is mentioned here and I would like to work on this. Can you please help me to get started as I am beginner to the project.
I totally agree with this- currently the only way to get the full picture of a Job is to read its conditions along with status fields which is error prone and a lot of extra work. I think a good start would be to add a new field as suggested by so as to not break compatibility for users already depending on
This sound reasonable. I'm still unclear when can contain more than one element. Also what does a or mean?
Still, open -- anyone actively working on this? I would like to attempt if not. :)
Are you still working on this?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
bump
This does not match what I just witnessed in my cluster… digging deeper to see if I need to file a bug but I have a Job with a single Pod that was executed. The Pod failed but the Job was marked as complete. This is the one (and only) pod that executed as my job: And this is the job that owns it (from kubectl describe): This is the job status object: on v1.16.7 but not finding any similar reports. just found this page as my only relevant search result.
I think it not only needs documentation, but also a change: It would be good to have 2 fields (probably needs better names), this way it's easy and clear when a job is finished and if it failed or not and when (someone can look at failed, succeeded and conditions for details): finishedtime: only set when the job finished (either completely failed or completely finished) finishtype: 'failed or completed'


Question: return fmt.Errorf("cannot autoscale a %v: %v", mapping.GroupVersionKind.Kind, err) } <del> generator, err := o.generatorFunc(info.Name, mapping) </del> <ins> generator, err := o.generatorFunc(o.Name, info.Name, mapping) </ins> if err != nil { return err }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: the hpa name is always info.Namenot --name What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/assign
/sig cli


Question: } } func (h *holder) expect(gvr schema.GroupVersionResource, gvk, optionsGVK schema.GroupVersionKind, operation v1beta1.Operation, name, namespace string, object, oldObject, options bool) { <del> // Special-case namespaces, since the object name shows up in request attributes for update/delete requests if len(namespace) == 0 && gvk.Group == "" && gvk.Version == "v1" && gvk.Kind == "Namespace" && operation != v1beta1.Create { </del> <ins> // Special-case namespaces, since the object name shows up in request attributes if len(namespace) == 0 && gvk.Group == "" && gvk.Version == "v1" && gvk.Kind == "Namespace" { </ins> namespace = name }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: It appears that validating admission requests for server-side apply are populating the for requests to create Namespace objects. Non server-side apply requests don't do this, and a missing namespace field is the only way to distinguish cluster-scoped resources from namespace-scoped resources. This appears to be a breaking behavioral change. See What you expected to happen: Requests to create namespaces would have a consistent signature between server-side apply and non-server-side apply contexts. How to reproduce it (as minimally and precisely as possible): See Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: kind OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig sig-api-machinery
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig sig-api-machinery Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig api-machinery
/wg api-expression
/assign
/assign
Requests made to namespaced URLs (URLs which contain a segment) populate the attribute in AdmissionReview requests. The documentation of the AdmissionReview field is as follows: From Update/Patch/Delete requests to specific namespaces have consistently populate this attribute for modifications to Namespace objects. Server-side apply is a type of patch request, so even when it is creating an object, that attribute will be populated. The difference is caused by a difference in the URL used by POST and apply-based creates. Creation via POST requests are made against the endpoint, so no attribute was populated. Namespace computed from request path: Namespace admission attributes for POST-based creates: Creation via server-side apply is done via a PATCH request to , so the attribute is populated. Namespace computed from request path: Namespace admission for apply-based creates: This is currently working as intended. If we want to make the behavior consistent between the two, I'd consider populating the namespace attribute for POST-based creation of Namespace objects.

Thanks for the thorough analysis Jordan, the problem OP is trying to solve is that they want to know if the object is namespaced or not, and used the presence of the field to detect that. According to your analysis, that seems wrong. The typical way to know if an object is namespaced is to use the discovery information, would that be the right way to solve that?
They could use discovery, or can continue to use the presence of the attribute, special-casing requests (namespaces are the only cluster-scoped resource which can have the attribute populated in AdmissionReview requests).
the documentation should be clearer, and I'd consider updating the POST-based create behavior to include this information to be consistent with update/patch/delete requests
do you want to update the doc and/or the post-based create behavior?
Sure! I am happy to take care of both.
where do you think it makes sense to add the namespace to the post-based behavior? We could add it to , but that might interfere with authorization, right? Would it be better to add the namespace directly to the ?
The info from the request body is not available when parsing requestinfo, only data in the URL. That's sort of what I expected, around the point we do this to pull the name from the incoming object:
Makes sense, I will give it a go, thank you!
that will also require sweeping admission plugins that check the namespace attribute. examples:
I think that we don't need to change anymore now. Do you know if there are other places in the docs where I should check?
yeah, I think we're good now. thanks for helping identify and clean this up


Question: go_rules_dependencies() go_register_toolchains( <del> go_version = "1.12.4", </del> <ins> go_version = "1.12.5", </ins> ) http_archive(

Something has changed between 320 and 321 runs and the request latency has increased significantly. The graphs below shows request latency (non watch) over time for a single test run: : ! : ! In my opinion they looks significantly different: in 320 most of the spikes are lower than 0.8s while 321 most of the spikes exceeds 1.6s. I checked few runs below 320 and have latency pattern similar to 320 and few runs after 321 and they have similar pattern to the 321. We are seeing similar latency growth in kubemark tests. The most significant change between 321 and 320 was go 1.11.5 -1.12 migration. /cc
/sig scalability
With we are trying to prepare small scale repro that we can use for further debugging.
was also in that range (cc for verification that nothing in that PR increased server load in non-alpha clusters)
did tests at head with reverting to go1.11 and metrics seemed better He is working on confirming that and should paste results then, but it seems to be pretty clear that it's go regression. We're also working on providing smaller-scale repro case to work closer with golang team (they used to be really helpful when debugging regressions in previous versions (IIRC 1.9 is where there were previous one).
I double checked and didn't find anything suspicious. The alpha function wasn't invoked in that window, and the PR is a no-op to non-alpha clusters
Regarding what wrote: ! I was mostly investigating the regression on kubemark-5000. Some of its latencies increased x3. I run two tests: one with checkout, and the other one with some commit checkout but without (Update to use golang 1.12). The 99th latency percentile of node listing was 3400.152ms in first case and f 952.576ms in second one.
Look into other metrics (I can't remember which exactly, delete pods? post pods? or sth like that) - which increased by almost 10x.
This is the graph that I had on my mind, but yeah, it's more like 4x not 10x. But still quite huge drop: !

Reproducing the issue: kubernetes repo - First broken version (go1.12) Last working version (go1.11) perf-tests repo - This repo is home for clusterloader2 - tool used for running the scalability tests. kubernetes directory run “make quick-release”. This will build kubernetes based on checkout version. GCE cluster. kubemark test. This step creates kubemark cluster (on top of the cluster created in step 4), run cluster loader test and tear down the cluster. The test run metrics will be exported to directory provided by the --report-dir flag. Inside this directory there will be APIResponsivenessload file containing 50, 90, and 90 percentiles of all apiserver call latencies. The latencies affected the most are: Listing nodes { "Resource": "nodes", "Scope": "cluster", "Subresource": "", "Verb": "LIST" } Pod deletion { "Resource": "pods", "Scope": "namespace", "Subresource": "", "Verb": "DELETE" } Listing services { "Resource": "services", "Scope": "cluster", "Subresource": "", "Verb": "LIST" } I would suggest their 99th percentiles as the issue metric. As for comparison: -Go1.11 run: Listing nodes - 162.409ms Pod deletion - 41.302ms Listing services - 28.277ms -Go1.12 run: Listing nodes - 293.724ms Pod deletion - 56.2ms Listing services - 46.183ms down GCE cluster
We're trying to dig deeper into it to both root cause it to a smaller part of the system, but it is quite tricky. Unfortunately, we don't have much more than guesses right now, though. Some observations: I looked into kubemark-5000 runs into DELETE pods calls, and it seems that all of them for which we have traces (500ms+) - there are ~5000 of those in each of recent runs [previously 99th percentile was <200ms), there is a significant big of "Object deleted from database". That may suggest some etcd issues, but we don't see logs related to long-running-calls on etcd side.

That may suggest some issues on the etcd client side, but we don't have anything concrete yet.It's also possible that golang upgrade changes something in serialization/deserialization - that's a second hypothesis (we've seen a bunch of traces of "Decoded delete options" (deserialization is also done in etcd client)), but I'm not convinced this is the case as of now
We've spent significant time with golang folks and already have some findings. To summarize: the microbenchmark from enabled Go team to find some regression on their side - the faulty commit from golang is: to cite Go folks: "To summarize the change: I removed the "fast" path for small page-level allocations (which currently always need to acquire the heap lock) in order to simplify code for subsequent changes. In effect, span allocations now all go through a balanced tree rather than just a linked list for small spans (I think it was all spans under 128 pages?)." We aren't fully convinced that this is for sure the same thing that is causing regression in our scalability tests, but analyzing traces, profiles, golang tracesetc. suggests that problems are indeed around memory allocations or GC. So with high probabilty this is the same thing. We are discussing with the way to prove it's the first thing and working with the to see what we can do with it.
It appeared that the commit isn't the most faulty one for us - running scalability tests against that commit doesn't show significant drop in metrics. In the lack of better ideas, we decided to bisect changes in golang by running full scalability suite on 5k-node cluster against them. This will take couple more days though...
Just to update this thread - we're bisecting Go 1.12 branch and there aren't that many remaining commits. We already know that: is a good one is a bad one I hope that we will know the exact faulty commit on Monday.
Looking through the following stand out for me /me places bets on one of those 2 :)
That is our bet too - we're working with Go folks to confirm and fix/mitigate this.
Any luck? (curious!)

I'm sorry for delay with update here - the current update is as following: after the next round of bisections, we know that: is a good one is a bad one During my discussions with golang folks, the outcome was that we believe that is the faulty one, as it adds a bunch of syscall to second-slowest allocation path (built on top of the previous one) should only help So the next step was to verify and indeed it is a bad one (and actually as expected, the metrics were worse than in the so far confirmed oldest bad commit: ) This is what we already know. What we're doing now is that golang folks provided us two patches that they think should mitigate the issue and I'm trying to confirm if that is really true. We should have results of this one later today. The two potential outcomes are: it solves our issue; then I will be working with golang folks to patch 1.12 with those it doesn't solve our issue: then I will be working on final confirmation that it's exactly this commit (by checking the previous one, and the next one to confirm it really helps) and working further with golang folks on what else we can do to mitigate it (and also I will try to understand deeper what exactly is happening there, I just have some rough understanding now).
Very very cool! thanks
cc
Just to give an update on that.

we fully confirmed that the regression is due to (using go1.12.1 with manually reverted that commit reverts the behavior) however, according to golang fokls that commit was fixing some things, and there are couple commits built on top of them after 1.12, so it's not-trivial to revert it (we are still looking for other options)the patches that golang team provided helps, but don't fix the problem (they roughly reduce the problem to 50%) yesterday I got another two patches that may help and we're validating them nowin the meantime we're also working with golang folks to try gathering more useful data for them for debugging - they already have some patches to apply in golang and I'm working on piping them through k8s and our build scripts - I hope I will have them ready today for an over-night run
To give an update on that - the current state of it is as following: that particular commit can't be just reverted, as it was fixing some issue however, golang folks have a PR that is mitigating that issue in a different way we have verified with custom build of that fully mitigates regression on our side The current plan is as following: has been merged to head of Go [that is the mitigation commit] it will be cherrypicked to 1.12 branch additionally in 1.12 we will revert the offending commit: We expect a new patch release of Go with those two commits to be released on May 1st. is currently my main point-of-contact coordinating that on golang side.
thanks for the update ! we will still be in 1.15 alpha* phases so that's great news.
- the trick is that go 1.12 is also in 1.14, so we should also cherrypick 1.14 branch - currently there is visible regression in 1.14
agree. i'd support that
The commits that we were waiting for are already there: Now we are waiting for next 1.12 patch release of Go to update Kubernetes to use, the ETA is May 1st. - given that I will be on vacation for the rest of this week, would you be able to help with that?
yes i can, please let me know what needs to be done (so i can figure out if i have access before you head out)
- thanks a lot.

I didn't do upgrade of go for 2 years or so, so I'm not entirely sure how (if?) the process changed in the meantime. - was doing the last one: , so I'm sure he will be able to help with that.
Ah just update to latest golang when it gets out. yes, i can do that. i was wondering if you wanted me to kick off some CI runs or something. Have a good vacation
- thanks a lot! Yes - it wasn't clear, but I was testing that and it helps us, so no need to run another tests (we will get final confirmation from postsubmits).
I can also help out to get this upgrade in when it’s released. Thanks for all your work on this
Looks like 1.12.5 went out this afternoon. PR open (), but expect tests to fail: Waiting for the docker hub image to publish. The PR is merged, so this should happen shortly. Need a new version of the bazel rules_go to be published.
Just to clarify the above (from : "oh, so I saw the auto update ran here: but it looks like the update for the tags hasn't merged yet: "
Yea, it takes some time for the images to get published.
Looks like new images are live.
Reopening this so we can verify that everything is back to normal in CI
/assign
For me it's pretty clear that this was resolved: http://perf- http://perf- Now there is one more thing we should do, which is cherrypick it to 1.14 branch. ^^
Opened pick:
has SIG Scalability done any one-off custom scale/perf testing with the new golang patch release and the k8s 1.14 branch content? Do you see this cherry pick as a 1.14.2 release blocker? Or would a week or two of focused testing with updated golang in the 1.14 k8s branch and a 1.14.3 release by end of May be a sufficiently safe yet expedient response?
We were doing a bunch of tests to discover the real regression in golang 1.12 - the regression happened within 1.14 release timeframe and the bisection was done on that branch. So yes - I'm 100% confident that it helps.

Tough question - I would love to see it in 1.14.2, otoh given how long it took to debug and fix it, I can see release team pushing it to 1.14.3. But if we could have that in 1.14.2, it would be extremely helpful.
just merged and we let 1.14 soak for one or two days before we cut . Let's keep this issue open until we have some data on the latencies after that merge. SIG Scalability, can you please provide some data / graphs about the latency in a (couple of) days for the branch?
is there any evidence we'll be able to observe in for validation? Or is it possible for SIG Scalability to trigger any custom runs now that the cherry-pick has merged to the 1.14 branch?
We aren't running large tests on release branches due to cost of them... We don't have any free quota to run it now, due to huge number of regressions that we have at head that we're currently debugging. But I'm 100% confident with this change. What I was personally doing in the past: I was running tests against commit when we bumped to 1.12.0 with 1.12.5 (which was from beginning of March, so still in 1.14 timeline) and the effect was exactly the same as the effect against head we got after merging it. I also understand changes in Go between 1.12.4 and 1.12.5 and their effect on performance And we also understand this regression, and why the fixes help, which has nothing to do with any changes that are happening to kubernetes. So I'm entirely sure that if you didn't observe any correctness issues, from performance perspective it is good-to-go.
Thanks for your answer!
With this being fixed in head and 1.14.2 released with the fix, I'm closing this one as fixed.


Question: # do_start() { <ins> # Raise the file descriptor limit - we expect to open a lot of sockets! ulimit -n 65536 </ins> # Return #   0 if daemon has been started #   1 if daemon was already running

When I run a density test in a large cluster I've noticed the following error in scheduler and kubelet: E0402 13:42:04. 4061 ] apiserver received an error that is not an api.Status: Put http://127.0.0.1:4001/v2/keys/registry/pods/default/my-hostname-density30--d93e-11e4-8f0b-bc30 5be02f04mlp5u?prevExist=false: dial tcp 127.0.0.1:4001: too many open files This could be either because either etcd or apiserver has too low ulimit -n
cc
Since etcd is running now in a pod it has a limit for fd set to (10e6) so I'll see if increasing this for apiserver helps.
etcd is running in a pod since yesterday afternoon. (It was introduced first on friday evening, rolledback on sunday, and reintroduced yesterday afternoon.) Just FYI, to help co-relate timelines..
I increased the limit for apiserver to 10e6 and started counting opened fd using a simple bash loop: datels /proc/4048/fd | wc -l where 4048 is a PID of apiserver. I haven't noticed any more "too many" errors in few consecutive runs of density test on 50 node cluster with 30 pods per node. At the same time I've noticed that number of opened fd reached ~2000. I'm gonna send a PR to increase the limit to something more reasonable.


Question: <ins> ### Version 8.8  (Mon October 1 2018 Zihong Zheng <zihongz@google.com>) - Update to use debian-base:0.3.2. </ins> ### Version 8.7  (Tue September 4 2018 Zihong Zheng <zihongz@google.com>) - Support extra `--prune-whitelist` resources in kube-addon-manager. - Update kubectl to v1.10.7.

Is this a BUG REPORT or FEATURE REQUEST?: /kind feature Alpine fixed a RCE vulnerability in apk in 3.8.1 () Update the base image used for addon-manager () to pick it up. Note: I'd send a PR if it depended directly on alpine image, not bashell/alpine.
There are no sig labels on this issue. Please add a sig label by either: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. See the . The in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assing
Ping


Question: // TestCoSchedulingWithPermitPlugin tests invocation of permit plugins. func TestCoSchedulingWithPermitPlugin(t *testing.T) { // Create a plugin registry for testing. Register only a permit plugin. <del> registry := framework.Registry{permitPluginName: NewPermitPlugin} </del> <ins> perPlugin := &PermitPlugin{} registry := framework.Registry{permitPluginName: NewPermitPlugin(perPlugin)} </ins> // Setup initial permit plugin for testing. plugins := &schedulerconfig.Plugins{

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: pull-kubernetes-integration Which test(s) are failing: TestPreemptWithPermitPlugin Since when has it been failing: 13:30 PST, Sep 9, 2019 Testgrid link: Reason for failure: Need investigations. Anything else we need to know: /sig scheduling
Another flaky It happens at 2019-08-06 05:43:47 +0000 UTC
The above repeated for 30 seconds.
I checked the code and log again. I think the reason is as following: When scheduler preempted , it failed to delete the pod(same as the case for , there might be something wrong with etcd), so it could not schedule because had not been deleted successfully.
/assign Find the root cause now, will send a PR soon.
Thanks for your work! Not sure whether you are still investigating it, the flaky test is impacting many PRs now, so I send a PR for it now, sorry for it.


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: We've seen scalability issues with the current aws-ebs volume provisioner impl, which is querying possible zones from cloud provider directly (). The improvement proposed in did make api calls more efficient, but also raises people's concern about whether it will create regression or not, or is there any more reliable ways to improve. To summarize, there are 2 major concerns: zone info in Node label trustworthy? My answer is yes, since currently the zone labels (beta version only) are maintained by node controller, the information is retrieved from cloud provider directly and is always updated during node reconciliation, it can be trusted. Even if kubelet can be misconfigured by wrong zone information, node controller is going to override them. See: Once we have the GA version of zone labels pushed out, we should update node controller to maintain both zone labels. It's fair enough to assume that if user wants kubelet to report zone/region, the information should be correct. the master/minion labels trustworthy? My answer is "semantic wide they are equivalent to checking node labels from cloud provider" - just as people are not required to set master/minion labels in v1.Node, well-known labels in cloud provider is not guaranteed to be set as well. Therefore besides mainstream cluster provisioners, which maintains these sets of labels in both Kubernetes and cloud provider, the function's behavior will remain unchanged: the original impl skips nodes that are explicitly labeled with well-known master labels, same here. Please suggest. /sig aws /sig cloud-provider /kind bug /priority important-soon /cc
for item 2, an alternative is to first call a List() of all v1.Nodes, get all providerID, and make a DescribeInstance call by passing all these IDs, which might still be more efficient than running the filter, and still very heavy-weighted with large scale. Current logic can be applied to the results though. Need input from aws side.
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available .

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
One of the quick optimizations we can do is, to remove those API calls if pod placement has already been decided because of delayed binding. I think, using delayed binding works well in practice and should eliminate calls being made.
Another thing is - since EBS volume plugin is being targeted for CSI migration, we should be super careful approving any changes into in-tree plugins which is not available in EBS CSI plugin. cc
Thanks ! yes I do agree that for topo aware storage class, when we create volume, the node is already bound, so we just get the zone information from the selected node, and this is a good to have short-term improvement. In long term, as cloud providers are going to be out of tree, we have more freedom to introduce informer to the aws library (gce is already doing that), and get zone information from cache. Since the path we are going is to make kubernetes components node role neutral, it's fair enough to say that we don't have to filter out master only zone in volume provisioner, and user should leverage topology aware storage class along with labels, taints, tolerations to achieve that. what do you think?
I believe in long term we want to default all clusters to use policy. That is the direction sig-storage wants to take(IIRC ), so IMO it may not be worth fixing general immediate binding use case for in-tree EBS driver where cloudprovider itself is deprecated (and any proposed change is non-trivial).
if the direction sig-storage is going is to always use , I agree with you that fixing a deprecated path of a deprecated EBS driver would be not worthwhile. Meanwhile, since in-tree EBS driver will still stay for a few more versions, I'm proposing for optimizing code path. Could you please provide inputs? thanks! cc
Related to
Hello! I'm the bug triage lead for the 1.16 release cycle and considering this issue is tagged for 1.16, I'd like to check its status. The code freeze is starting on August 29th EOD PST (about 5 days from now) and once the code freeze starts, only release-blocking changes will be considered to be merged. Is this issue still planned to be fixed for the 1.16 release?

I have addressed this issue via , will close this issue /close
Closing this issue. <detailsIn response to : I have addressed this issue via , will close this issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: expect := api.PodList{ ListMeta: api.ListMeta{ResourceVersion: "10"}, Items: []api.Pod{ <ins> // We expect list to be sorted by directory (e.g. namespace) first, then by name. {ObjectMeta: api.ObjectMeta{Name: "baz", ResourceVersion: "1"}}, </ins> {ObjectMeta: api.ObjectMeta{Name: "foo", ResourceVersion: "1"}}, {ObjectMeta: api.ObjectMeta{Name: "bar", ResourceVersion: "2"}}, },

kubectl returns resources in a random order, which makes it hard to look for things when users have a lot resources. Returning resources in deterministic order should be a helpful and trivial change (hopefully). We have an issue about pagination , but that's a larger and separate issue.
We should be cause to not do this on the * namespace - I think thats a special case. No disagreement for other list calls
What is the default etcd response order? Last created or last modified? Sent from my iPhone
I think the order is by key if sort param is given. If not given, the order is by last created (maybe, but we shouldn't trust it). It seems we are not trying to get sorted result from etcd.
Today, I think it's random order in the child key space in etcd (because the node structure uses a map), so you get golang map iteration order without sort. I don't know if sort on server is any cheaper than sort on client today, so we may want to do this in rest / registry rather than by passing etcd.
Apart from performance, sorting in clients means we can also take control of how responses are sorted, so for * namespace, we can sort by namespace, then by resource ID. , Clayton Coleman wrote: Master of Electrical and Computer Engineering Carnegie Mellon University
We can always hint sort to the registry storage layer in the future, so sorting on the client now won't kill us.
Just took a closer look at our rest/registry code to implement the sort (in client), but implementation-wise, it seems much easier to pass sort to etcd (one line change). Also, namespace is a prefix on resource, so sort across all namespace with etcd key will essentially sort resources by namespace then by Name. If we are to implement sort on client side, we can either: sort package: have each list object, ie.e PodList, ControllerList, implement sort interface. (We won't do this). have ListMeta implement sort interface, if there is a way to find enclosing structs (so we can use items field) have api.Object implement sort interface, verify object is list before sorting. sort algorithm ourself, and do it in rest. Is there other golang or kubernetes idom that I haven't been aware of that can be applied to this use case?

----- Original Message ----- I think sorting it on etcd is fine for now - that's the simplest possible thing that works.
Is there any material perf impact in etcd to enable it? Sent from my iPhone
Probably not material. Can always switch away if necessary
Famous last words :-) Sent from my iPhone
Got my hat ready


Question: if registered.Has(nodeID) { continue } <del> _, err := s.kubeClient.Nodes().Create(&api.Node{ </del> <ins> node := &api.Node{ </ins> ObjectMeta: api.ObjectMeta{Name: nodeID}, Spec: api.NodeSpec{ Capacity: s.staticResources.Capacity, }, <del> }) </del> <ins> } addr := net.ParseIP(nodeID) if addr != nil { node.Status.HostIP = nodeID } else { addrs, err := net.LookupIP(nodeID) if err != nil { glog.Errorf("Can't get ip address of node %v", nodeID) } else if len(addrs) == 0 { glog.Errorf("No ip address for node %v", nodeID) } else { node.Status.HostIP = addrs[0].String() } } _, err := s.kubeClient.Nodes().Create(node) </ins> if err == nil { registered.Insert(nodeID) }

Why doesn't the 'node' entity have a hostname and/or IP fields? Currently its id field (soon to become name field) holds the IP but is that something that a REST api consumer can firmly rely on to get the node's IP/hostname so this field 'name/id' will always hold IP/hostname and not other information? (IP/hostname is useful for example for visualization of the minions available in a k8s cluster in UI and showing their IP/hostnames)
Minion does have a HostIP field in v1beta1, and NodeStatus has a HostIP field in v1beta3. When you GET the resource, is the value for that field not set? If so, on what cloudprovider? It's true that there is no hostname field, but the minion/node name itself is the hostname.
, I'm currently using the hack/local cluster setup , this is the output of GET for minion 127.0.0.1 with v1beta1, seems that HostIP is not returned at all (even not with empty value) { "kind": "Node", "id": "127.0.0.1", "uid": "-8527-11e4-8f4b-", "creationTimestamp": "2014-12-16T15:31:50+02:00", "selfLink": "/api/v1beta1/minions/127.0.0.1", "resourceVersion": 7, "apiVersion": "v1beta1", "resources": { "capacity": { "cpu": 1000, "memory": } } } Can you please elaborate/point to docs about the change in v3? so HostIP field is moved into a NodeStatus section/internal to Node entity? BTW why is it considered part of a status? thank you
The best documentation of v1beta3 is unfortunately the source right now: It adheres to the conventions described here: The fact that HostIP is not reported is probably a quirk with static minions. In reality, hosts can have multiple IP addresses, some private and some public, v4 and v6, dynamically assigned (e.g., using DHCP), etc. For now, we support single, private, static IP addresses, but in cases where hosts are specified using hostnames, the IP addresses are looked up.

So, status makes the most sense to me.
Yeah, HostIP is only queried from cloudprovider. you won't see hostIP in json if it's empty (see the tag)
why HostIP is not returned for local static setup? the id reflects the IP anyhow. it sounds like a bug, but this issue is tagged with "documentation".
Yes, I agree that it's a bug that HostIP isn't populated. The documentation aspect is to explain somewhere why it's in Status rather than Spec.


Question: glog.Warningf("No API client: %v", err) } <del> cloud, err := cloudprovider.InitCloudProvider(s.CloudProvider, s.CloudConfigFile) if err != nil { return err </del> <ins> if s.CloudProvider == options.AutoDetectCloudProvider { kcfg.AutoDetectCloudProvider = true } else { cloud, err := cloudprovider.InitCloudProvider(s.CloudProvider, s.CloudConfigFile) if err != nil { return err } glog.V(2).Infof("Successfully initialized cloud provider: %q from the config file: %qn", s.CloudProvider, s.CloudConfigFile) kcfg.Cloud = cloud </ins> } <del> glog.V(2).Infof("Successfully initialized cloud provider: %q from the config file: %qn", s.CloudProvider, s.CloudConfigFile) kcfg.Cloud = cloud </del> } if kcfg.CAdvisorInterface == nil {

It appears that the kubelet will detect and use the cloud provider as detected by cadvisor thus overriding the cloud provider passed in at runtime: This will break our private cloud configuration where we do not use our cloud provider's load balancing and instead use our own load balancing tier. In my opinion, the cloud provider specified at kubelet runtime should be respected for the node. Am I reading this correctly? Any thoughts on how to handle this properly? Thank you!
If you specify any cloud provider through flags, it will be used by kubelet (see ). It's only when you don't specify any cloud provider that kubelet will detect the cloud provider from the machine info, as described by the comment.
that's exactly my situation, I want to run in our private cloud environment without any of the cloud integration. If I specify cloud-provider="" at runtime it will get overridden.
Ah I see. I guess specifying an empty string is the same as not specifying any cloud provider. This auto-detection feature was introduced by /cc the author and reviewer to see if this is an intended behavior:
Legitimate concern. I'm posting a fix. Do you consider this to be a regression or are you just getting started with k8s.
I am sorry, This was not the original intention. We can keep the auto detection with a flag, or make a flag to ask kubelet not to do it. Unless there is a way to see if the variable was assigned to that I missed. I can post a fix tomorrow if it won't exit by then.
Thank you. We're running on 1.2.4 and won't be running on 1.3 for a few months yet anyways and was just testing to look for future issues when I found this.
I assigned this issue to you since , and I can't assign this to anyway :-)
Thanks for testing our 1.3 release candidate, and reporting this regression to us. We are including the fix to 1.3.


Question: func addNamespaceRoleBinding(namespace string, roleBinding rbac.RoleBinding) { if !strings.HasPrefix(namespace, "kube-") { <del> glog.Fatalf(`roles can only be bootstrapped into reserved namespaces starting with "kube-", not %q`, namespace) </del> <ins> glog.Fatalf(`rolebindings can only be bootstrapped into reserved namespaces starting with "kube-", not %q`, namespace) </ins> } existingRoleBindings := namespaceRoleBindings[namespace]

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?:BUG REPORT What happened: the err message in addNamespaceRoleBinding is wrong. ref to
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig auth


Question: Install-Module -Name powershell-yaml -Force } <del> # Disables Windows Defender realtime scanning. # TODO: remove this workaround once the fix is rolled out the Windows image # https://github.com/kubernetes/kubernetes/issues/75148 function Disable-WindowsDefender { # Windows Defender periodically consumes 100% of the CPU, so disable realtime # scanning. Uninstalling the Windows Feature will prevent the service from # starting after a reboot. # TODO(pjh): move this step to image preparation, since we don't want to do a # full reboot here. if ((Get-WindowsFeature -Name 'Windows-Defender').Installed) { Log-Output "Disabling Windows Defender service" Set-MpPreference -DisableRealtimeMonitoring $true Uninstall-WindowsFeature -Name 'Windows-Defender' } } </del> # Creates directories where other functions in this module will read and write # data. # Note: C:tmp is required for running certain kubernetes tests.

Two weeks ago we were observing Windows nodes crashing and rebooting during our e2e test runs on GCE (). Analyzing the crash dumps showed a fault in the NTFS driver while the process was running. appears to be part of the Windows Defender anti-malware component. To address the crashes we did two things: 1) disabled Windows Defender on our Windows e2e test nodes (), and 2) included February's Windows updates on our Windows nodes. After these two things the crashes seem to have stopped. Two days ago we re-enabled Windows Defender on our Windows nodes: Unfortunately we started seeing the Windows node crashes again, starting with run 228 on (To determine if a node crash occurred we open the results link for the test run, open the artifacts for each Windows node and look for a file (e.g. )). We analyzed one of these crash dump files and it looked the same as what we saw a couple weeks ago: fault in NTFS while was running. For now we've merged we re-disables Windows Defender on our e2e Windows nodes. Since that PR merged early this morning (starting with run 246) none of our Windows nodes have crashed, strongly implicating Windows Defender as contributing to the crashes. Since the failure is happening in the NTFS driver we'll need help from Microsoft folks to debug it. The memory dump files are publicly available from the test artifacts (runs 228 - 245), and I've copied a WinDbg crash analysis below. Additionally, some Windows version information is available in the serial-1.log file for each Windows node, also copied below. I don't have any great ideas about why the combination of K8s e2e tests and Windows Defender seems to cause the crash; we don't have any other reports of this failure happening in Windows VMs on GCE. Version info: Analysis:
/assign pjh cc is there anyone on your side who can look at this? It's fairly easily reproducible in our e2e test environment. We're working around it for now by disabling Windows Defender; I think we're ok with this for now, but it would be good to find the cause.
/sig windows Not blocking for 1.14.

/cc /cc in case you've seen the same crash
did do some deeper analysis of one of the crash dumps early last week and found this which might be helpful:
cc as well
I fwd'ed the dmp internally
Thanks! also passed this along internally. I'll leave this issue assigned to me while awaiting updates from Microsoft.
The folks at Microsoft root-caused this issue and we've verified a fix for it. The fix will make it into Windows Server 2019 at some point, not sure about the timing for SAC releases.
/assign craiglpeters For tracking.
I believe this issue should be fixed via KB4501371, can you verify once it becomes available?
/assign pjh Yes, I'll verify the fix when that KB is available (end of June).
There's an optional fix out: It will be included in the July cumulative update too