Question: t.Errorf("Unexpected diff after updating pods (-want, +got):n%s", diff) } <ins> // Attempt to nominate a pod that was deleted from the informer cache. // Nothing should change. q.AddNominatedPod(nonExistentPodInfo, &framework.NominatingInfo{NominatingMode: framework.ModeOverride, NominatedNodeName: "node1"}) if diff := cmp.Diff(q.PodNominator, expectedNominatedPods, cmp.AllowUnexported(nominator{}), cmpopts.IgnoreFields(nominator{}, "podLister", "RWMutex")); diff != "" { t.Errorf("Unexpected diff after nominating a deleted pod (-want, +got):n%s", diff) } // Attempt to nominate a pod that was already scheduled in the informer cache. // Nothing should change. scheduledPodCopy := scheduledPodInfo.Pod.DeepCopy() scheduledPodInfo.Pod.Spec.NodeName = "" q.AddNominatedPod(framework.NewPodInfo(scheduledPodCopy), &framework.NominatingInfo{NominatingMode: framework.ModeOverride, NominatedNodeName: "node1"}) if diff := cmp.Diff(q.PodNominator, expectedNominatedPods, cmp.AllowUnexported(nominator{}), cmpopts.IgnoreFields(nominator{}, "podLister", "RWMutex")); diff != "" { t.Errorf("Unexpected diff after nominating a scheduled pod (-want, +got):n%s", diff) } </ins> // Delete a nominated pod that doesn't have nominatedNodeName in the pod // object. It should be deleted. q.DeleteNominatedPodIfExists(highPriorityPodInfo.Pod)

The scheduler cache entered a corrupted state where a Pod, according to the cache dump, was nominated to node A, while the status said it was nominated to node B. This lead to unschedulable low priority pods due to the phantom pod in the nominator cache. The cache to match the Pod status Setup: two nodes with all the resources used. A high priority pod comes in that is able to preempt at least one pod. This might cause these sequence of events X doesn't fit, it gets nominated to and back to scheduling queue. start a second scheduling attempt for Pod X. get a Pod update for the . This puts the pod back into the scheduling queue. second scheduling attempt finishes and schedules the pod successfully in start a third scheduling attempt for Pod X (due to 3) get a Pod update for the . This should have cleared the nominated node in the cache, but there is nothing currently there. the third scheduling attempt, the pod doesn't fit, so the scheduler nominates it to . Nothing clears this nomination afterwards Usually step 2 happens after 3, so everything works as expected. But in a busy cluster, 3 might be delayed. No response <detailsGKE </details<details</details<details</details<details</details<details</details
/sig scheduling cc
This Pod actually got scheduled: The cache dump was obtained much later at : I think the problem is that we got an update for the Pod (probably the nominated node itself), scheduled it again and assigned a second nominated node for it. The second scheduling attempt failed, but we didn't clear the cache.
/triage accepted
/priority important-soon
Sounds like is related?
It could be the same, or it could be another bug. Note that this is a second scheduling attempt for a Pod that was already scheduled.
My theory of events: X doesn't fit, it gets nominated to and back to scheduling queue. start a second scheduling attempt for Pod X. get a Pod update for the . This puts the pod back into the scheduling queue. second scheduling attempt finishes and schedules the pod successfully in start a third scheduling attempt for Pod X (due to 3) get a Pod update for the . This should have cleared the nominated node in the cache, but there is nothing currently there.

the third scheduling attempt, the pod doesn't fit, so the scheduler nominates it to . Not sure why this pod update fails, but nevertheless, we have a corrupted cache at this point.
thanks for raising up this and the fix. Could you update the "How can we reproduce it (as minimally and precisely as possible)?" section a bit? Per our offline conversation, the issue will leave a "phantom" pod holding a number of resources in the internal nominator cache. And due to that unnecessary resource reservation, low-priority pods may not get scheduled while they should have been. This might be the key trigger of this issue. In practice, it can be caused by a (way-too-quick) movePodBackToActiveQ call (due to relevant/irrelevant events, or backoff's completion) after step 1 and before step 3.
Updated the description. I will update the PRs on Monday.


Question: <ins> #!/usr/bin/env bash # Copyright 2016 The Kubernetes Authors. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # #     http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. METADATA_ENDPOINT="http://metadata.google.internal/computeMetadata/v1/instance/attributes/kube-master-internal-ip" METADATA_HEADER="Metadata-Flavor: Google" ip=$(curl -s --fail ${METADATA_ENDPOINT} -H "${METADATA_HEADER}") if [ -n "$ip" ]; then # Check if route is already set if not set it if ! sudo ip route show table local | grep -q "$(echo "$ip" | cut -d'/' -f 1)"; then sudo ip route add to local "${ip}/32" dev "$(ip route | grep default | awk '{print $5}')" fi fi </ins>

If you create a private cluster via kube-up using provider GCE, you won't be able to restart your master nodes. If you restart you master, the nodes won't be able to connect to your master. This is due to this line - It gets executed on master on cluster-up, but it won't be re-executed on master restart. We should fix that, especially if we're to implement /sig scalability /priority important-soon
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale /good-first-issue I'll ask whether someone in SIG Scalability public meeting would like to take on this one
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /remove-lifecycle stale /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hello, Can I start work on this one as my first issue? :)
I don't see anyone else claiming it, so your contributions are welcome :)
let me know if you need any explanations here
Thank you :) I've made some changes and assign you can you tell me is this will be good approach?
Thanks, I replied in the PR


Question: }, "selector": { "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.LabelSelector", <del> "description": "Label query over pods whose evictions are managed by the disruption budget. A null selector selects no pods. An empty selector ({}) also selects no pods, which differs from standard behavior of selecting all pods. In policy/v1, an empty selector will select all pods in the namespace.", "x-kubernetes-patch-strategy": "replace" </del> <ins> "description": "Label query over pods whose evictions are managed by the disruption budget. A null selector selects no pods. An empty selector ({}) also selects no pods, which differs from standard behavior of selecting all pods. In policy/v1, an empty selector will select all pods in the namespace." </ins> } }, "type": "object"

PDB was upgraded to v1 in 1.21. Despite the changelog saying that the behavior of the v1beta1 implementation should have stayed the same, it appears that the for the field was set to . I believe this is the change in question: When running with a starting config that had a with both a key and a key, and a patch that had a with just a key, the behavior between 1.20 and 1.21 differs. What we get is the key gets deleted, rather than merged with the patch's key to make one unified selector. In 1.20, the behavior we'd get is thus, which is desirable for us because it lets us inject a base assumption about our PDBs into everything we do: This is my repro case: : Compare by using the command from earlier in this report: I also brought this up in Slack: I don't think we want to switch to using just for PDBs, because that wouldn't solve issues where our users want to use without having to re-specify the from our base. Putting aside the fact that changing the behavior of the v1beta1 implementation was probably not intentional, it also seems undesirable to change how selectors work in general. I don't know how widely scoped this change will be across various resource types, too, so we're starting down quite a bit of nasty discovery time. <detailsThis is the version I used to demonstrate my repro case, but I confirmed that this occurs even in kubectl v1.21.0. For my "happy" repro case, this is what the version output looks like: </details<detailsAWS (the hard way) </detailsNo response No response No response No response
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment.

<detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
as requested, I am tagging you and on this one
We should narrowly revert the strategic merge patch selector behavior for v1beta1 to match pre-1.21 behavior for backwards compatibility. More broadly, it is unexpected for multiple actors to be writing to a single selector. The pre-1.21 strategic merge patch behavior for v1beta1 PDB was already inconsistent w.r.t. merging portions of the selector and replacing others (it would merge and replace ) For that reason, server-side apply treats selector writes as owning the entire selector, and the v1 PodDisruptionBudget API will do the same for strategic merge patch. edit: reworded the inconsistencies with the < 1.21 selector handling
More broadly, it is unexpected for multiple actors to be writing to a single selector. The pre-1.21 strategic merge patch behavior for v1beta1 PDB has problems with data getting left in the selector incorrectly if the same user submits a patch with two keys, and then submits a patch with only a single key. Would that be considered a bug in the implementation, or was there some inherent, design-caused race condition in patch application that I don't understand? Cuz if I were to sequentially apply patches with the old strategic merge, I would think the selector would just accumulate keys, which seems fine to me–but I can see how a race condition would turn it into chaos
edit: I was wrong about the previous behavior... the issue was that it already inconsistently merged the selector... it would merge and replace , which is still pretty incoherent If you really want to tweak field-level things, jsonpatch can still do surgical insertion/appending/removal of specific fields.
Should we expect to use jsonpatch for anything that can have a selector, going forward? If it changed for PDBs then it should change for everything else, I would assume edit: we discussed more in slack. The answer is no, the fields for other resource types' label selectors are not changing, so we only need to treat PDBs as a special case.


Question: } count++ <del> shortOutput := cmdutil.GetFlagString(cmd, "output") == "name" </del> <ins> shortOutput := output == "name" if len(output) > 0 && !shortOutput { return cmdutil.PrintResourceInfoForCommand(cmd, info, f, out) } </ins> if !shortOutput { f.PrintObjectSpecificMessage(info.Object, out) }

What happened: What you expected to happen: yaml output How to reproduce it (as minimally and precisely as possible): Anything else do we need to know: Related PR


Question: s.GenericServerRunOptions.AddUniversalFlags(fs) s.Etcd.AddFlags(fs) s.SecureServing.AddFlags(fs) <del> s.SecureServing.AddDeprecatedFlags(fs) </del> s.InsecureServing.AddFlags(fs) s.InsecureServing.AddDeprecatedFlags(fs) s.Audit.AddFlags(fs)

Currently as authentication and authorization are both standard config for all clusters, the kubeapiserver insecure http handler is unsecure, which means the cluster is vulnerable to the attack. So it should be deprecated, even removed totally.
/sig api-machinery
I think removing the insecure option is a good idea. Can you send an email the sig-apimachinery mailing list with a proposed timeline? I think this should be like other GA features and have a one year clock.
Sure, I will send it ASAP.
:+1:


Question: package kubectl import ( <ins> "fmt" </ins> "strings" "github.com/GoogleCloudPlatform/kubernetes/pkg/api"

kubectl 0.17.0: curl http://localhost:8080/api/v1beta3/minions { "kind": "Status", "apiVersion": "v1beta3", "metadata": {}, "status": "Failure", "message": "the server could not find the requested resource", "reason": "NotFound", "details": {}, "code": 404 } master3 core # /opt/bin/kubectl get mi master3 core # kubectl 0.16.0: uses: http://localhost:8080/api/v1beta1/minions master3 core # ./kubectl get mi NAME LABELS STATUS 192.168.10.70 <noneReady 192.168.10.71 <noneReady in the same time curl http://localhost:8080/api/v1beta3/nodes works well.
Shall we just remove "minion" from kubectl now and throw an error?
remove "minions"?
Yes, that should be part of . cc
Presumably works?
Assigning who's working on
Ref
Could you please take a look at this?

I know that you can get nodes / a node, but the help doesn't say you can or provide examples: To resolve this issue: the help for this command to include nodes (and any other resources we're missing). we generate the help text in a way that ensures this (and probably the many others that are drifting) don't get out of date? Or at least give us a heuristic as to whether this might be true, even if we have to fix it manually?
Can this be closed?
I'll close this one with ; we want to get rid of the minions (mi) resource name, since it doesn't work for v1beta3.


Question: } } <del> select { case <-portForwardFuncDone: case <-time.After(100 * time.Millisecond): t.Fatalf("%d: timed out waiting for portForwardFuncDone", i) } </del> <ins> <-portForwardFuncDone </ins> } }

this is you
Hmm, I had upped waiting from 100ms to 500ms and that seemed to fix it. Do you have any better ideas?
Don't wait? :)
If the only reason you would timeout would be the code is broken, that's why the test runs have -timeout set.
Then the test fails immediately. It's waiting for the net.Listen calls to finish so it can connect to the specified local ports.
*finish enough so that socket connections work
So I can remove the timeout gating in my tests?
Yes ----- Original Message -----
Just failed again: https://travis-
Seems to be failing frequently now.
If this is test is failing again, my gut says it's not the same root cause. The original failure was a timeout issue. This new one is connection reset by peer. I've got the test running in a loop waiting for a failure but haven't seen one yet. is this still failing frequently?
No. It had failed twice in ~10 builds when I reopened this issue, but hasn't failed since. Reclosing. Will file a new issue if it recurs.


Question: VOLUME_PLUGIN_DIR: $(yaml-quote ${VOLUME_PLUGIN_DIR}) KUBELET_ARGS: $(yaml-quote ${KUBELET_ARGS}) EOF <ins> if [[ "${master}" == "true" && "${MASTER_OS_DISTRIBUTION}" == "gci" ]] ||  [[ "${master}" == "false" && "${NODE_OS_DISTRIBUTION}" == "gci" ]]  ||  [[ "${master}" == "true" && "${MASTER_OS_DISTRIBUTION}" == "cos" ]] ||  [[ "${master}" == "false" && "${NODE_OS_DISTRIBUTION}" == "cos" ]]; then cat >>$file <<EOF REMOUNT_VOLUME_PLUGIN_DIR: $(yaml-quote ${REMOUNT_VOLUME_PLUGIN_DIR:-true}) EOF fi </ins> if [ -n "${KUBE_APISERVER_REQUEST_TIMEOUT:-}" ]; then cat >>$file <<EOF KUBE_APISERVER_REQUEST_TIMEOUT: $(yaml-quote ${KUBE_APISERVER_REQUEST_TIMEOUT})

/kind bug https://k8s- This is likely caused by a missing REMOUNTVOLUMEPLUGIN_DIR flag in kube-env construction. /assign /sig storage
xref /priority failing-test /priority critical-urgent any update on this one?
/status approved-for-milestone
I'm aware of the solution, but I'm trying to understand how to limit the scope of the change to just COS images, instead of all VM distros.
xref
Fix submitted, awaiting review:
PR approved; awaiting 1.10 milestone marker
/status in-progress
You must be a member of the github team to add status labels.
/status in-progress
[MILESTONENOTIFIER] Milestone Issue Needs Attention Action Required: This issue has not been updated since Mar 11. Please provide an update. Note: This issue is marked as , and must be updated every 1 day during code freeze. Example update: <details<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details


Question: // // This method is a getter but it can cause side-effect of updating NodeInfo objects. func (nm *NodeManager) GetNodeDetails() ([]NodeDetails, error) { <del> nm.registeredNodesLock.Lock() defer nm.registeredNodesLock.Unlock() </del> var nodeDetails []NodeDetails <del> for nodeName, nodeObj := range nm.registeredNodes { </del> <ins> for nodeName, nodeObj := range nm.getNodes() { </ins> nodeInfo, err := nm.GetNodeInfoWithNodeObject(nodeObj) if err != nil { return nil, err

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Quote I took goroutine dump of controller-managers that were stuck in volume provisioning and I think we have a deadlock in code: I think we have a problem introduced when we merged - and what is happening is, when in is called it acquires a lock on , but same function ends up calling which acquires another lock on object . Now the problem is, function gets called in a loop and it will try to acquire lock for each node it processes and hence two operations could result in deadlock. This results into inability to provision a with a default StorageClass. The volume does not get to call the API endpoint, just hangs after call, which is clear in the goroutine dump. What you expected to happen: 1) Provision a from successfully. 2) Don't be blocked by credentials secret change. How to reproduce it (as minimally and precisely as possible): 1) Provision a vSphere cluster with credentials secret set 2) Create volume 3) Wait... Anything else we need to know?: Environment: Kubernetes version (use ): v1.19.0-rc.2+-dirty Cloud provider or hardware configuration: vSphere OS (e.g: ): Linux Kernel (e.g. ): GNU/Linux Install tools: openshift-install (kubeadm) Network plugin and version (if this is a network-related bug): Others:
/sig cloud-provider /area provider/vmware
This should be a blocker bug for 1.19 at least as far as vsphere cloudprovider integration is concerned, because it breaks cloudprovider integration/volume provisioning. cc
cc:
/milestone v1.19
Something makes me think this issue existed before, and just escalated to a larger scope after adding informers, demanding refresh for all objects. This is used within plugin in and it is mounting the volume if it is required, calling - it is locking on all the nodes at once. If the pod was just restarted (say just changed, or the health probes demanded a restart), it would process all existing at the same time as trying to mount/create the volume.

And both threads will clang on the same lock, as the would probably not be populated at that moment, so will go the path of - same which is used in . So we get lost volume data, pending etc... And the links I'm sharing are from Kubernetes, where issues with similar footprint randomly arose.


Question: return nil, err } <del> oauthConfig, err := adal.NewOAuthConfigWithAPIVersion(env.ActiveDirectoryEndpoint, token.tenantID, nil) </del> <ins> oauthConfig, err := adal.NewOAuthConfig(env.ActiveDirectoryEndpoint, token.tenantID) </ins> if err != nil { return nil, fmt.Errorf("building the OAuth configuration for token refresh: %v", err) }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Kubernetes 1.17.1 included a breaking change to how Azure AD may be used for OIDC authentication in Kubernetes: For all teams which were successfully using Azure AD's implementation of OIDC in Kubernetes previously, this change means: - Notably, this includes not only teams who directly use AAD, but also all AKS clusters. See What you expected to happen: Associated with a breaking change, a migration path should be provided. In this case (and as currently released), both the apiserver and all clients of the apiserver need to simultaneously change versions to avoid impact. In larger deployments of Kubernetes, a single cluster may have hundreds (or more) of disparate clients. Such a large change cannot be reasonably executed without impacting customers and causing critical service outages. This is a request for details on how cluster operators and Kubernetes users (that use Azure AD) should migrate from 1.17.0 to 1.17.1 without impact. How to reproduce it (as minimally and precisely as possible): a cluster on 1.17.0 instructions here to tie into Azure AD: valid authn/authz with invalid authn/authz with test with a 1.17.1 cluster and versions below 1.17.1 Anything else we need to know?: The fact that this underlying functionality should change is not disputed. (see the evidence in the original ticket ). Azure AD has improperly injected into all JWT token fields for a long time now. The issue is not with the new functionality itself, but rather the manner in which the change was implemented. Without a designated migration path, this change is highly impactful to clients. Environment: Kubernetes version (use ): apiserver: 1.17.1, kubectl: 1.17.0 (and vice versa) Cloud provider or hardware configuration: N/A OS (e.g: ): N/A Kernel (e.g.

): N/A Install tools: N/A Network plugin and version (if this is a network-related bug): N/A Others: Tagging original author and approvers of the PR: /sig cloud-provider /area provider/azure
do you mean we should revert this change? cc currently it's only in k8s v1.17.1 release
To help in determining that, I'd suggest the following: This is a major functionality change which was introduced in a patch release. It would be more appropriate for such a change to be introduced in a major Kubernetes release version. This is a breaking change with high impact to both operations teams and end users of Kubernetes. Such a change should be introduced with a clear migration strategy that does not impact cluster operations. I am supportive of this functionality change being made to Kubernetes, however this PR as implemented requires an unreasonably impactful cost to end users. I recommend it be reworked into a more appropriate and non-impactful PR that provides a solid migration path for those already using the "old" functionality. With that, I'll defer to maintainers to determine the proper next steps.
thanks, that's the case, if there is one AAD enabled cluster with both 1.17.0 and 1.17.1 agent nodes , there is no way to use one kubectl verison to operate on this cluster.
Since there is already a breaking change on 1.17.1, revert means the change can only be done in 1.17.2, that would make the situation more complex. I would suggest we add a doc: Need to check whether this will break sth, e.g. upgrade AKS cluster from 1.17.0 to 1.17.1
it's not related to server side version, but apiserver configuration: Current situation is A workable fix should be allow multiple '--oidc-client-id' to be passed in, while the server switch to
When you're running a shared cluster this situation is really bad. Many people/teams are now complaining not being able to login to the clusters anymore so they now have to keep multiple versions of kubectl which is breaking a lot of scripts. Please create a migration path for companies which have to run multiple clusters in different versions or roll back the change. The support overhead is huge.
This isn't a feasible solution for users.

As it stands, users using Azure AD OIDC have no upgrade path from 1.17.0. Users operating a single high-availability cluster will have a window where both 1.17.1 and 1.17.0 apiservers are running and either version of kubectl will randomly fail depending on which apiserver responds. For users operating a single cluster, asking thousands of clients to switch over at the exact same instant the new apiserver is up is unreasonable- especially for clients using a Kubernetes client library for their programming language of choice, since changing the client version may require deploying a different build. For users operating multiple clusters, it is impossible for clients to switch over without breakage because different clusters would likely be upgraded at different times (possibly spanning days or weeks). The only reasonable solution is to provide an "overlapping" release where both OIDC client ID formats are accepted. I think a reasonable solution would be: Revert this change for both 1.17.2 and 1.18.0 Publish notifications that users using Azure AD OIDC should skip 1.17.1 Rework this change to allow both the old and new format of client ID to be used. Add warning messages in the apiserver startup logs and release changelog that the old format will go away in a future release. This change would ideally be in a new minor release of Kubernetes rather than a patch release. In a future minor release at least one minor version past the one that allowed the overlap, remove support for the old OIDC version.
/sig auth
/sig cli
Note that this change also invalidates the i.e. 1.17.X clusters must support kubelet 1.16.X and 1.18.X. So the OIDC change would have to support a total of three minor versions, instead of a single patch version as it stands.
sorry about that my change caused such problem. originally i thought a configuration change is an acceptable migration path. I will submit PR to revert it.
another thought: revert the change in and keep the azure auth module as before add auth module which supports new oidc behavior this would prevent breaking change and allow new scenario to be enabled immediately the reason i suggest to add module is that right now it's the bug in kubectl which dictates the AAD oidc argument in api-server.

It seems weird to fix kube-apiserver for a bug in kubectl. There will always be a time where kubectl is updated and breaks existing AAD OIDC flow when api-server is not updated accordingly as people have voiced here. Which suggest a solution requiring no change to api-server OIDC argument. At the same time, we also should not do too many hacky thing in api-server to intelligently omit prefix because authentication is security critical. Hence, we might as well just implement a new auth module... thoughts?
I'm not sure if proposed is helpful or not. It's only helpful to deal with different versions of kubectl. But it doesn't help in the case where new kubectl talking to existing api-server. As indicated in the comments, people would need to update their clusters as soon as kubectl is updated. That's not much different than what currently is right now. Right?
While this migration path is also technically out of compliance with the kubectl version skew policy, it is at least technically possible if users upgrade apiserver prior to upgrading kubectl. The separate auth module for AAD v2 is an interesting thought. Is it possible for an apiserver to use multiple auth modules simultaneously? If both auth modules could be accepted at the same time, there would be a clean migration path by keeping both modules available over a minor release.
But it doesn't help in the case where new kubectl talking to existing api-server. As indicated in the comments, people would need to update their clusters as soon as kubectl is updated This is technically not true. As people have commented here, there will be a window of time where older kubectl is unable to talk to api-server with newer configuration. I think it's possible. we will need to 1) implement two auth modules in kubectl, and 2) enable multiple in api server like what proposed. 3) migrate kubeconfig to use new auth module.
does that sound like a plausible path? Should I just open issues to track these tasks while I'm preparing to revert?
We've identified a lot of use cases here, each with differing requirements. I believe to entirely remove impact, it's a requirement for both a server-side solution and a client-side solution to be provided. Of these two sides -- client and server -- a solution in one means that the "other doesn't need to change".

ie. If we change the apiserver to support multiple ID's, no clients will need to change how they authenticate as all old and new OIDC client ID's will be accepted. And vice versa, if we change kubectl to include a second auth option, cluster operators don't need to reconfigure apiservers to change what ID's they accept. Having both of these solutions allows all users and operations teams involved the flexibility to select which option fits their needs best. It also allows for swapping between and the lack of as needed (which moving off is desirable for the many reasons stated in the original git issue). Having said all of this, and as an operator of (very many) clusters, the primary cost I'm looking to avoid is the requirement that every user of a k8s cluster my team operates needs to make a change to their client libraries/kubectl version, and associated kubeconfig. Since that affects thousands of users and deployment tools in my organization, that is the primary point of impact I'm attempting to avoid. For that reason I'm much more interested in the serverside apiserver modification, yet others likely feel differently based on their use case. As such, the proper solution is likely to support both serverside and clientside solutions. Hopefully this all makes sense.
I was referring to this paragraph. this is aligned to your thought: we need both server side and client side change to have proper migration The separate auth module for AAD v2 is an interesting thought. Is it possible for an apiserver to use multiple auth modules simultaneously? If both auth modules could be accepted at the same time, there would be a clean migration path by keeping both modules available over a minor release.
The first priority should be to restore compatibility with the oldest existing releases. I would expect the breaking change to be reverted on 1.15/1.16/1.17 branches.
/assign
yes. will cherry pick once it's merged. thank you!


Question: DEFAULTER_FILES := $(addsuffix /$(DEFAULTER_FILENAME), $(DEFAULTER_DIRS)) <ins> RUN_GEN_DEFAULTER :=                                                          function run_gen_defaulter() {                                            if [[ -f $(META_DIR)/$(DEFAULTER_GEN).todo ]]; then                   ./hack/run-in-gopath.sh $(DEFAULTER_GEN)                          --v $(KUBE_VERBOSE)                                           --logtostderr                                                 -i $$(cat $(META_DIR)/$(DEFAULTER_GEN).todo | paste -sd, -)   --extra-peer-dirs $$(echo $(addprefix $(PRJ_SRC_PATH)/, $(DEFAULTER_DIRS)) | sed 's/ /,/g')  -O $(DEFAULTER_BASENAME)                                      "$$@";                                                        fi                                                                    };                                                                        run_gen_defaulter </ins> # This rule aggregates the set of files to generate and then generates them all # in a single run of the tool. .PHONY: gen_defaulter <del> gen_defaulter: $(DEFAULTER_FILES) if [[ -f $(META_DIR)/$(DEFAULTER_GEN).todo ]]; then                   ./hack/run-in-gopath.sh $(DEFAULTER_GEN)                          --v $(KUBE_VERBOSE)                                           --logtostderr                                                 -i $$(cat $(META_DIR)/$(DEFAULTER_GEN).todo | paste -sd, -)   --extra-peer-dirs $$(echo $(addprefix $(PRJ_SRC_PATH)/, $(DEFAULTER_DIRS)) | sed 's/ /,/g')  -O $(DEFAULTER_BASENAME);                                     fi </del> <ins> gen_defaulter: $(DEFAULTER_FILES) $(DEFAULTER_GEN) $(RUN_GEN_DEFAULTER) .PHONY: verify_gen_deepcopy verify_gen_defaulter: $(DEFAULTER_GEN) $(RUN_GEN_DEFAULTER) --verify-only </ins> # For each dir in DEFAULTER_DIRS, this establishes a dependency between the # output file and the input files that should trigger a rebuild.

Right now on head () if you run it will end up having a modified file. This happened because some type has been changed and this file should have been generated before commit, and verification should have failed but they didn't.
This is not only a problem with openapi-gen. deepcopy-gen also has the same problem. I've a field to a type (autoscaling/v1) and verify-all passed for the commit (without adding generated code to it). those files will be generated on verify-codegen and it will not fail and the commit can be merged without generated code.
Yeah. So when the Makefile stuff originally went in those files were not checked in. Then we un-did that part, but I didn't re-add them to verify...


Question: fi } <ins> # Use gsutil to get the md5 hash for a particular tar function gsutil_get_tar_md5() { # location_tar could be local or in the cloud # local tar_location example ./_output/release-tars/kubernetes-server-linux-amd64.tar.gz # cloud tar_location example gs://kubernetes-staging-PROJECT/kubernetes-devel/kubernetes-server-linux-amd64.tar.gz local -r tar_location=$1 #parse the output and return the md5 hash #the sed command at the end removes whitespace local -r tar_md5=$(gsutil hash -h -m ${tar_location} 2>/dev/null | grep "Hash (md5):" | awk -F ':' '{print $2}' | sed 's/^[[:space:]]*//g') echo "${tar_md5}" } </ins> # Copy a release tar and its accompanying hash. function copy-to-staging() { local -r staging_path=$1 local -r gs_url=$2 local -r tar=$3 local -r hash=$4 <ins> local -r basename_tar=$(basename ${tar}) #check whether this tar alread exists and has the same hash #if it matches, then don't bother uploading it again #remote_tar_md5 checks the remote location for the existing tarball and its md5 #staging_path example gs://kubernetes-staging-PROJECT/kubernetes-devel #basename_tar example kubernetes-server-linux-amd64.tar.gz local -r remote_tar_md5=$(gsutil_get_tar_md5 "${staging_path}/${basename_tar}") if [[ -n ${remote_tar_md5} ]]; then #local_tar_md5 checks the remote location for the existing tarball and its md5 hash #tar example ./_output/release-tars/kubernetes-server-linux-amd64.tar.gz local -r local_tar_md5=$(gsutil_get_tar_md5 "${tar}") if [[ "${remote_tar_md5}" == "${local_tar_md5}" ]]; then echo "+++ ${basename_tar} uploaded earlier, cloud and local file md5 match (md5 = ${local_tar_md5})" return 0 fi fi </ins> echo "${hash}" > "${tar}.sha1" gsutil -m -q -h "Cache-Control:private, max-age=0" cp "${tar}" "${tar}.sha1" "${staging_path}" gsutil -m acl ch -g all:R "${gs_url}" "${gs_url}.sha1" >/dev/null 2>&1 <del> echo "+++ $(basename ${tar}) uploaded (sha1 = ${hash})" </del> <ins> echo "+++ ${basename_tar} uploaded (sha1 = ${hash})" </ins> } # Given the cluster zone, return the list of regional GCS release # bucket suffixes for the release in preference order. GCS doesn't # give us an API for this, so we hardcode it.

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help?: No What keywords did you search in Kubernetes issues before filing this one? : copy-to-staging Is this a BUG REPORT or FEATURE REQUEST? (choose one): Feature Request / Refactoring. <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Environment: Cloud provider or hardware configuration: Google Cloud (gce) What happened: Whenever I execute cluster/kube- it copies my tar files to google cloud, even if the files haven't changed. These files are a few hundred megs and can take a long time to upload. What you expected to happen: I'd like copy-to-staging function to check whether a matching tar already exist, and avoid uploading it again in this situation. How to reproduce it: execute cluster/kube- Here is the current behavior: cluster/kube- ... Starting cluster in us-central1-b using provider gce ... calling verify-prereqs ... calling verify-kube-binaries ... calling kube-up Project: <PROJECT-NAMEZone: us-central1-b +++ Staging server tars to Google Storage: gs://kubernetes-staging-<PROJECT-NAME/kubernetes-devel +++ kubernetes-server-linux- uploaded (sha1 = <HASH) Anything else we need to know: I have written the code to deal with this issue, and I'll submit a related PR shortly.
- PR coming...
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list .
/sig cluster-lifecycle
/assign


Question: w.resultsManager.Set(w.containerID, result, w.pod) <ins> if w.probeType == liveness && result == results.Failure { // The container fails a liveness check, it will need to be restared. // Stop probing until we see a new container ID. This is to reduce the // chance of hitting #21751, where running `docker exec` when a // container is being stopped may lead to corrupted container state. w.onHold = true } </ins> return true }

Happened in GKE suite:
Does it need a liveness probe, maybe?
to delegate
Saw
Can you take a look at this one? Thanks!
FWIW, I saw some fishy messages in in (node: 104.197.48.84) It seems like docker was giving conflicting view of the container status, i.e., whether the container was running or not (and whether kubelet should kill it). kubelet's first attempt to kill the container : All the exec well after the grace period still shows that the container is the "Running: true" state () Later, kubelet tried killing the container again and failed because container does not exist. Note that after the failed kill also shows that the container was in the Running state. At the same time, cadvisor observed the container being stopped but then detected the container and then tried adding it again, but failed to find the /cc UPDATE: According to supervisord, docker was not killed during this time.
I didn't find result for the first Killing, does that really return? What is the log level, I remember that you told me before the log level in GKE is 2?
AFAICT, the pod worker may have never returned because the specific pod worker didn't log anything beyond that point. I was looking at (docker stop hangs) to see if it's relevant. didn't print any error though.
Yeah, the last SyncPod log was which is before the first Killing. Then the pod worker was stuck in
This really looks similar with I tried to reproduce it following the comment. The container was not actually killed. Docker stop hangs. Docker ps shows the container is still running. But the difference is that when I tried to kill it with another , it still hung there.
I was able to reproduce this quite easily by running 30 pods with the same spec in the test and with :
From
My theory is that during may cause docker to leave the container in a unrecoverable state (yep, a bug). I can consistently reproduce this by simply creating (which always fails), and leave the cluster for < 10 min. The restart counts of some of the pods will stop increasing after sometime, and after ssh'ing in to the node, one can easily confirm the same symptoms posted above. If pods use the http probe, they do not have this issue.

For the workaround, we can instruct the probe workers to stop probing after detecting a liveness check failure, until the container has been restarted (with a new container ID). Note that this is just a workaround, as it doesn't stop other components or users to run during this window. I've verified that with this fix, there are no zombie docker containers anymore. Will upload a PR.
FWIW, I also tested it with docker 1.8.3, which has the same problem.
Thanks !


Question: const ServiceAnnotationLoadBalancerBEProtocol = "service.beta.kubernetes.io/aws-load-balancer-backend-protocol" const ( <del> // volumeAttachmentStatusTimeout is the maximum time to wait for a volume attach/detach to complete volumeAttachmentStatusTimeout = 30 * time.Minute </del> // volumeAttachmentConsecutiveErrorLimit is the number of consecutive errors we will ignore when waiting for a volume to attach/detach volumeAttachmentStatusConsecutiveErrorLimit = 10 <del> // volumeAttachmentErrorDelay is the amount of time we wait before retrying after encountering an error, // while waiting for a volume attach/detach to complete volumeAttachmentStatusErrorDelay = 20 * time.Second // volumeAttachmentStatusPollInterval is the interval at which we poll the volume, // while waiting for a volume attach/detach to complete volumeAttachmentStatusPollInterval = 10 * time.Second </del> <ins> // volumeAttachmentStatus* is configuration of exponential backoff for // waiting for attach/detach operation to complete. Starting with 10 // seconds, multiplying by 1.2 with each step and taking 21 steps at maximum // it will time out after 31.11 minutes, which roughly corresponds to GCE // timeout (30 minutes). volumeAttachmentStatusInitialDelay = 10 * time.Second volumeAttachmentStatusFactor       = 1.2 volumeAttachmentStatusSteps        = 21 // createTag* is configuration of exponential backoff for CreateTag call. We // retry mainly because if we create an object, we cannot tag it until it is // "fully created" (eventual consistency). Starting with 1 second, doubling // it every step and taking 9 steps results in 255 second total waiting // time. createTagInitialDelay = 1 * time.Second createTagFactor       = 2.0 createTagSteps        = 9 </ins> ) // Maps from backend protocol to ELB protocol

In a cloud provider environment it's possible to use up the API quota grinding the cluster to a halt: In a multi tenant environment, one user could consume the quota for an entire cluster. Some cloud providers allow throttling of API calls by type but Kube does not differentiate on the call types, so DOS still possible.
Adding some colors. Cloud providers like AWS/GCE/Azure have rate limits on API calls. GCE and Azure have fixed API calls per second, while AWS doesn’t offer fixed quota. Once API call quota are exhausted, no more API calls are allowed. Consequences include: If the error code path are not well tested, unexpected behavior will happen. A malicious user may launch DoS by triggering frequent API calls. The entire cluster would be impacted AWS cloudprovider has implemented API throttling at global level as well as on per call type level most recently. However, at least on AWS, quota for call types are different: query type API calls have higher quota while modification type API calls have lower quota. There should be strategies to differentiate these calls. Quota for Cloudprovider API calls should be supported on per account/project basis, so DoS can be averted. (Per This needs significant enhancements in Attach/Detach logic, volume plugin gets only pv.Spec, which does not include information about namespace of the PVC bound to it nor the user that initiated the operation. And only volume plugin (on top of cloud provider) can throttle API calls.) Throttling at per call type level should be supported.
I understand there is a problem, but I am not convinced on any particular solution. Can we get a more detailed enumeration on the code paths that invoke cloud provider backed interfaces, and where we would want some bucket to control rate for those calls? I don't think the existing object should bleed concepts from the underlying cloud provider, and I don't think its setup to support rate based quota models at this time.
/cc
I went through storage parts of the cloud provider and on AWS it looks like this: Deleter (in persistent volume controller) calls ec2.DeleteVolume() once per delete attempt Provisioner (in persistent volume controller) calls ec2.DescribeInstances() with filter "instance-state-name" == "running" to get all zones that contain a node. Once per provisioning attempt.

calls ec2.CreateVolume() once per provisioning attempt calls ec2.CreateTags() - up to 60 times per provisioning call (until it succeeds). Typically one attempt needed. calls ec2.DescribeVolumes() once per provisioning attempt When provisioning or deletion of a volume fails, the controller re-tries every 15 seconds until it succeeds. Typically it succeeds on the first attempt. PersistentVolume admission plugin: ec2.DescribeVolumes() once per new PV instance created (to fill its labels) Attach (in attach/detach controller) ec2.DescribeInstances() with filterprivate-dns-name" = nodeName once per attach attempt (to get list of already attached volumes) ec2.AttachVolume() once per attempt ec2.DescribeVolumes() with filter volumeId = volume every 10 seconds until it's attached Detach (in attach/detach controller) ec2.DescribeInstances() with filterprivate-dns-name" = nodeName once per attempt ec2.DetachVolume once per attempt ec2.DescribeVolumes() with filter volumeId = volume every 10 seconds until it's detached please confirm how often the controller re-tries on failure. Is it ? With many volumes being attached/detached at the same time, each volume polls its own DescribeVolumes() every 10 seconds while waiting for the operation to finish. We could save a lot of calls if we had one DescribeVolumes() call for all "interesting" volumes and distribute the results to proper places. In addition, some exponential backoff would be helpful, AWS is pretty slow when attaching/detaching volumes. Only storage was investigated!!!
The same issue also applies to ELB on AWS. Service controller invokes which calls:
It should be noted exponential backoff is disabled in - , I am not sure if it was intentional. Also, we should consider case where k8s cluster is in, inconsistent state. For example - if volume being deleted is still mounted to a pod, there would be a flood of API requests for deleting volume in pvcontroller (without any exponential backoff)
You're right, PV controller does not use retry logic of GoRoutineMap and retries provisioning and deletion on its own and currently without exponential backoff. The reason is that PV controller can react to deletion of object, e.g. when user removes a PVC then the controller does not need to retry provisioning a volume for it.

PV controller retries failed operation every 15 seconds by default. Some exponential backoff could be if 15 seconds are too short.
If Attach or Detach operation fails (e.g. due to throttling), there already is exponential backoff, starting with 0.5 seconds and doubling with each failed attempt, capped at 2 minutes. Summary: there already is exponential backoff for Attach/DetachVolume with we will have the same exponential backoff for Create/DeleteVolume the only fixed is DescribeVolumes() when we wait for a volume to get really attached/detached. I am going to fix it.


Question: COPY cni-conf /etc/cni/net.d # Create symlinks for each hyperkube server <del> # TODO: this is unreliable for now (e.g. running "/kubelet" panics) # Also, it doesn't work for other architectures </del> <ins> # TODO: replace manual symlink creation with --make-symlink command once # cross-building with qemu supports go binaries. See #28702 </ins> # RUN /hyperkube --make-symlinks <ins> RUN ln -s /hyperkube /apiserver  && ln -s /hyperkube /controller-manager  && ln -s /hyperkube /federation-apiserver  && ln -s /hyperkube /federation-controller-manager  && ln -s /hyperkube /kubectl  && ln -s /hyperkube /kubelet  && ln -s /hyperkube /proxy  && ln -s /hyperkube /scheduler </ins>

The command was removed from the dockerfile in this PR: This addition was originally proposed here: I'd like to fix the problem or re-enable the creation of symlinks if none exists. Currently there is not enough information from the PR that removed them to know what the problem is. I've tested the symlink command and successfully have built hyperkube images with it enabled. Calling the kubelet symlink doesn't panic for me. Do you know how to reproduce this issue?
You're right, the panic I observed when giving no arguments is gone with v1.3.0 It doesn't work when cross-building Qemu doesn't play very well with Go binaries Can't we just for all binaries in the hyperkube image? I can make a patch with that if you like
It was discussed in that the idea with the command was there to obviate the need to maintain all the commands in the dockerfile since the hyperkube binary already knows all the registered servers. Do you have more information about why this doesn't work with cross-building? I don't quite understand why Go binaries not playing nicely with qemu matter, since, at some point hyperkube is full of a bunch of go binaries that have to play nicely with arm.
Qemu can't run Go binaries, that's the problem. We can use virtually anywhere else Until Qemu releases a version that's compatible with Go, we can't run Go binaries when cross-building
Ooh I understand now, basically any use of go during cross building will not work. Do you happen to know if this more a Go issue or qemu issue? Which upstream project needs to make a fix?
It's a qemu issue, talked to the golang folks Some qemu maintainer said they've improved stuff at HEAD, but haven't tested it Anyway, we have to wait until the next qemu release. If we want it now, we initially have to go with


Question: dswp.kubeClient = fakeClient } <del> func prepareDswpWithVolume(t *testing.T) (*desiredStateOfWorldPopulator, kubepod.Manager) { </del> <ins> func prepareDswpWithVolume(t *testing.T) (*desiredStateOfWorldPopulator, kubepod.Manager, *fakePodStateProvider) { </ins> // create dswp mode := v1.PersistentVolumeFilesystem pv := &v1.PersistentVolume{

We observed that while a container is in the process of being torn down, but not completed, the volume gets unmounted and then subsequently detached, causing the fs to enter read-only mode. This happened after kubelet restarted. We observed that on the restart, the volume was not to DSW because the Pod was marked for deletion. In the normal case, we only remove the pod from DSW when the containers have finished teardown: However, in the reconstruction case, it looks like if a pod has started termination, but not necessarily completed, then we don't add the pod to DSW: We probably need to be consistent and change the add pod criteria to also add even if it's started terminating. The side effect is that in the normal case we may do unnecessary mounts if a pod is quickly created and deleted. We can also consider a flag to only use the stricter check during reconstruction. What happened in our env was that kubelet was crash looping, and then all the pods on the node were deleted. It looks like there was a good 30 seconds between volume unmount and container teardown. I think this may be easily reproducible if the pod has a long graceful termination period and the container ignores the stop signal. No response <details22 </details<details</details<details</details<details</details<details</details<details</details
/triage accepted
Yeah this looks problematic. What was the exact kube version btw? So it was possible that container is still writing to the volume? Do we know why fs enters read-only mode when this happens? I know the behaviour of kubelet is problematic regardless, but still. I think adding volumes belonging to terminating pod doesn't sound so bad. I wonder if we could change this to .
Does this mean instead of checking ShouldPodContainersBeTerminating to decide if we add the pod or not, we should check IsPodTerminationRequested or not to decide? Or we should use the same check ShouldPodRuntimeBeRemoved with reconstruction(this one is more consistent)? pointed out that there were already scaling challenges with volume manager processing pods it doesn't need to.
For reconstruction, we need to add the volume in DSW if there is a chance that containers are running. So is the best call to determine that.
We observed a similar issue but could only reproduce it with emptydir volumes. Do you think is the same issue?

/assign sunnylovestiramisu
thanks for the report. I think this issue could impact emptydir volumes too.
Is there any recommended way to make a pod having a long graceful termination period? Tried with some simple pods but it just terminated right away.
You can use the sample pod and instructions from here
cc
Here is my repro pod and steps: the pod shell, another window (window 2), ssh to the node another window (window 3), window 2, in window 1, exec back into the pod . The file will be gone. try . It will return When looking at kubelet logs, you'll see the log messages that indicate the volume was reconstructed without being to dsw. and UnmountVolume happen soon after: Pod lifecycle logs:
Tested locally via kind cluster. /k master is showing error of touch: cannot touch '/scratch/test': No such file or directory. cluster with the fix in this pr shows no error. And running shows no result in the log.


Question: if err != nil { return errors.Wrap(err, "unable to fetch CoreDNS current installed version and ConfigMap.") } <ins> var hasCoreDNSMigrationFailed bool </ins> if IsCoreDNSConfigMapMigrationRequired(corefile) { if err := migrateCoreDNSCorefile(client, coreDNSConfigMap, corefile, currentInstalledCoreDNSVersion); err != nil { <del> return err </del> <ins> // Errors in Corefile Migration is verified during preflight checks. This part will be executed when a user has chosen // to ignore preflight check errors. hasCoreDNSMigrationFailed = true klog.Warningf("the CoreDNS Configuration was not migrated: %v. The existing CoreDNS Corefile configuration has been retained.", err) if err := apiclient.CreateOrRetainConfigMap(client, coreDNSConfigMap, kubeadmconstants.CoreDNSConfigMap); err != nil { return err } </ins> } } else { if err := apiclient.CreateOrUpdateConfigMap(client, coreDNSConfigMap); err != nil {

What happened: Upgrade using kubeadm failed because of CoreDNS migration enchancement What you expected to happen: Upgrade passed correctly How to reproduce it (as minimally and precisely as possible): any 1.15.x Kubernetes with CoreDNS that deployed with any unusual image (self built for example). We using commits counter as part of image tag, e.g - 1.6.2-100. to upgrade to 1.16.x with kubeadm. Anything else we need to know?: Logs part without ignoring preflight: Logs part with ignoring preflight: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: OpenStack OS (e.g: ):Kernel (e.g. ):Install tools: Kubeadm
/sig cluster-lifecycle
The error came from It seems you should mention this issue on coredns project
True. Will notify them also.
/cc
/assign In such cases where the migration isn't supported and the user chooses to ignore the warnings from the preflight check, the migration should be skipped. wdyt? /cc
Yes. IMO, the warnings should be phrased like "CoreDNS configuration migration will not occur: reason"... so that a user expects the config migration will not be done.
/area kubeadm
SGTM
Upgrade passed correctly I think the best we can do here is to allow the upgrade to pass, but this will, in most practical cases, result in a cluster without a functioning DNS. The most common practical use-case for using a self built image of CoreDNS is to use external plugins. During the kubeadm upgrade, CoreDNS will be replaced with a stock release version of CoreDNS. When that new version of CoreDNS starts, it will not be able to load the existing Corefile's external plugins, and exit. ... or, we could finish the upgrade, leaving the existing CoreDNS deployment untouched, which I think works better.
Yea, it's kinda strong words that "Upgrade passed correctly". Probably, I think the best solution will be to make possible skip Corefile migration and make kubeadm finish the upgrade without Corefile changes. Guys that using custom images of CoreDNS should care about Corefile for new version by themselves, IMO.
Yes, I think that would work best...


Question: kubecontainer "k8s.io/kubernetes/pkg/kubelet/container" "k8s.io/kubernetes/pkg/kubelet/network" utilexec "k8s.io/kubernetes/pkg/util/exec" <ins> "k8s.io/kubernetes/pkg/util/wait" </ins> ) const (

Host network pods should be able to be created before the container network exists. Pod syncs for pods on the container network should fail until the container network is configured. cc
ref:
I think this is a thing we should make working before v1.4, so adding the milestone. Complain if you disagree. Also, does volunteer to make a PR? :)
Going to make a first pass at this in
Great!
/cc


Question: // enabled. func TestPreferNominatedNode(t *testing.T) { defer featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, features.PreferNominatedNode, true)() <del> testCtx := initTestPreferNominatedNode(t, "perfer-nominated-node") t.Cleanup(func() { testutils.CleanupTest(t, testCtx) }) cs := testCtx.ClientSet defaultPodRes := &v1.ResourceRequirements{Requests: v1.ResourceList{ v1.ResourceCPU:    *resource.NewMilliQuantity(100, resource.DecimalSI), v1.ResourceMemory: *resource.NewQuantity(100, resource.DecimalSI)}, } </del> defaultNodeRes := map[v1.ResourceName]string{ v1.ResourcePods:   "32", v1.ResourceCPU:    "500m", v1.ResourceMemory: "500", } <del> type nodeConfig struct { name string res  map[v1.ResourceName]string </del> <ins> defaultPodRes := &v1.ResourceRequirements{Requests: v1.ResourceList{ v1.ResourceCPU:    *resource.NewMilliQuantity(100, resource.DecimalSI), v1.ResourceMemory: *resource.NewQuantity(100, resource.DecimalSI)}, </ins> } tests := []struct { name         string <del> nodes        []*nodeConfig </del> <ins> nodeNames    []string </ins> existingPods []*v1.Pod pod          *v1.Pod runnningNode string }{ { <del> name: "nominated node released all resource, preemptor is scheduled to the nominated node", nodes: []*nodeConfig{ {name: "node-1", res: defaultNodeRes}, {name: "node-2", res: defaultNodeRes}, }, </del> <ins> name:      "nominated node released all resource, preemptor is scheduled to the nominated node", nodeNames: []string{"node-1", "node-2"}, </ins> existingPods: []*v1.Pod{ initPausePod(&pausePodConfig{ Name:      "low-pod1", <del> Namespace: testCtx.NS.Name, </del> Priority:  &lowPriority, NodeName:  "node-2", Resources: defaultPodRes, }), }, pod: initPausePod(&pausePodConfig{ <del> Name:      "preemptor-pod", Namespace: testCtx.NS.Name, Priority:  &highPriority, </del> <ins> Name:     "preemptor-pod", Priority: &highPriority, </ins> Resources: &v1.ResourceRequirements{Requests: v1.ResourceList{ v1.ResourceCPU:    *resource.NewMilliQuantity(500, resource.DecimalSI), v1.ResourceMemory: *resource.NewQuantity(200, resource.DecimalSI)},

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Integration tests are sometimes flaking locally: PreferNominatedNode Once the node gets deleted, the nodelifecycle controller is racing to update pod status and the pod deletion logic is failing causing tests to flake. Ultimately, we call testCtx.CleanUp() which cleans the nodes and the pods involved the test namespace so we are not going to leak any resources. links to appreciated links to specific failures in spyglass appreciated <!-- You can identify the SIG from the "prowjobconfigurl" on the testgrid dashboard for a test --/sig scheduling <!-- Please see the deflaking doc () for more guidance! --/assign
cc Tried addressing it via
/triage accepted thanks!


Question: package(default_visibility = ["//visibility:public"]) <del> load("@io_bazel_rules_go//go:def.bzl", "go_library") </del> <ins> load("@io_bazel_rules_go//go:def.bzl", "go_library", "go_test") </ins> go_library( name = "go_default_library",

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Setup a kubernetes cluster with dual stack enabled, set kube-proxy to "ipvs" proxy mode, many warnings are flooding in kube-proxy pod log, W0303 22:11:22. 1 ] failed to update endpoints kube-system/kube-scheduler with error failed to identify ipfamily for endpoints (no subsets) W0303 22:11:22. 1 ] failed to update endpoints kube-system/kube-controller-manager with error failed to identify ipfamily for endpoints (no subsets) W0303 22:11:24. 1 ] failed to update endpoints kube-system/kube-scheduler with error failed to identify ipfamily for endpoints (no subsets) W0303 22:11:24. 1 ] failed to update endpoints kube-system/kube-controller-manager with error failed to identify ipfamily for endpoints (no subsets) W0303 22:11:26. 1 ] failed to update endpoints kube-system/kube-scheduler with error failed to identify ipfamily for endpoints (no subsets) What you expected to happen: No such error logs How to reproduce it (as minimally and precisely as possible): Here is my setup step, I'm using a single master node setup, kubeadm init --pod-network-cidr=192.168.0.0/16,fc00::/112 --service-cidr=10.96.0.0/16,fd00::/112 --feature-gates=IPv6DualStack=true Change controller-manager cidr configuration --node-cidr-mask-size-ipv6=120 install calico 3.6.4 with dual stack enabled kube-proxy proxy mode set to "ipvs" featureGates: IPv6DualStack: true healthzBindAddress: "" hostnameOverride: "" iptables: masqueradeAll: false masqueradeBit: null minSyncPeriod: 0s syncPeriod: 0s ipvs: excludeCIDRs: null minSyncPeriod: 0s scheduler: "" strictARP: false syncPeriod: 0s kind: KubeProxyConfiguration metricsBindAddress: "" mode: "ipvs" Anything else we need to know?

: Linux Redhat 7.7 Environment: Kubernetes version (use ):v1.17.3 Cloud provider or hardware configuration: OS (e.g: ): Redhat 7.7 Kernel (e.g. ): 3.10.0-1062.4.1.el7.x8664 Install tools: kubeadm Network plugin and version (if this is a network-related bug): calico 3.6.4 Others:
/sig network
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/assign
/remove-triage unresolved those warnings are harmless because those objects doesn't have endpoints addresses, thus no IP family, as you can see below: however, I think that if there are no endpoint addresses we should skip instead of logging, what do you think ?
It seems these printouts can be caused by headless services, please see;
Can endpoints support dual stack? From my understanding, it cannot support dual stack in v1.17. Why kube-proxy will check another ipFamily which is not configured? Here is the endpoints info under default ns from my cluster,
Another observation about dual stack supporting is I don't see ipv6 interface address in node status. Here is my interface configuration, and here is pod configuration, I see kubelet has already set ipv6 network in podCIDRs
The is reported by if you specify . Currently kubelet does not report both ipv4 and ipv6 addresses, that is not implemented yet. But public clouds do not let report the node but defines them themselves. At least set both addresses. So it depends on your environment.
endpoints on services with ClusterIP services are single family, with the same IPFamily than the ClusterIP. However, you can create endpoints manually, not associated to a service, or to a headless service (ClusterIP=None) with a mix of families. You opened the issue asking why do you have these errors logs If you check the endpoints of the services mentioned in the logs there are no addresses, hence the kube-proxy logs a WARNING because is not able to identify the IP family. Technically the warning is correct, but I think that if there are no addresses we shouldn't log the warning.

In order to not convert this issue in a placeholder for dual stack let's try to stick to the topic, for more general things we have a #k8s-dual-stack channel in slack.
Ok, I understand what you mentioned. From the implementation, this log behavior is correct. The other problem is kube-proxy is constantly logging the same error.
I think (please somebody correct me if I'm wrong) this is because the endpoints are checked periodically, per the control-loop / reconcile paradigm that kubernetes uses, so it tries to add those endpoints periodically
nodeIPs are only dual stack if the cloud-controller-manager updates it, on bare metal or without cloud-controller-manager you only see one address. I tried cloud-controller-manager in my openstack cluster and can get ipv4/ipv6 interface.
Thanks, I will create another issue for document.


Question: return diskUUID, nil } <ins> // GetHost returns host of the virtual machine func (vm *VirtualMachine) GetHost(ctx context.Context) (mo.HostSystem, error) { host, err := vm.HostSystem(ctx) var hostSystemMo mo.HostSystem if err != nil { klog.Errorf("Failed to get host system for VM: %q. err: %+v", vm.InventoryPath, err) return hostSystemMo, err } s := object.NewSearchIndex(vm.Client()) err = s.Properties(ctx, host.Reference(), []string{"summary"}, &hostSystemMo) if err != nil { klog.Errorf("Failed to retrieve datastores for host: %+v. err: %+v", host, err) return hostSystemMo, err } return hostSystemMo, nil } </ins> // DetachDisk detaches the disk specified by vmDiskPath func (vm *VirtualMachine) DetachDisk(ctx context.Context, vmDiskPath string) error { device, err := vm.getVirtualDeviceByPath(ctx, vmDiskPath)

Currently whenever a vSphere volume is provisioned, we are looking at all nodes in the cluster and then fetching their hosts and then their datastores and then ensuring that selected datastore in present on all the nodes. This is done every time a new PVC is provisioned. It can be pretty expensive in large clusters and number of involved API calls can be big. /sig storage /area vsphere cc
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : Currently whenever a vSphere volume is provisioned, we are looking at all nodes in the cluster and then fetching their hosts and then their datastores and then ensuring that selected datastore in present on all the nodes. This is done every time a new PVC is provisioned. /area vsphere Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage accepted
/assign


Question: // Volume binder only wants to keep unassigned pods sched.VolumeBinder.DeletePodBindings(pod) } <ins> sched.Framework.RejectWaitingPod(pod.UID) </ins> } func (sched *Scheduler) addPodToCache(obj interface{}) {

What would you like to be : When the pod is scheduled to the phase and I delete the pod, the pod still remains in the until the plugin allow/reject/timeout(Especially when waiting for a long time). Why is this needed: We need to take the initiative to delete the data in the and terminate the process in waiting state of plugin to ensure data accuracy, if the pod has been deleted /cc /priority important-soon /assign /sig scheduling
I think we can update the event handler for deleting pods to reject the pod. Can you send a PR for that?
OKI will send a pr for it.


Question: c, _ := storage.Delete(svc.ID) <-c <del> if len(fakeCloud.Calls) != 1 || fakeCloud.Calls[0] != "delete" { </del> <ins> if len(fakeCloud.Calls) != 2 || fakeCloud.Calls[0] != "get-zone" || fakeCloud.Calls[1] != "delete" { </ins> t.Errorf("Unexpected call(s): %#v", fakeCloud.Calls) } srv, err := memory.GetService(svc.ID)

pull-kubernetes-e2e-azure-disk-windows pull-kubernetes-e2e-azure-disk-windows It seems caused by connection failed: from kubelet logs: This repeats until timeout
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig windows
/area providers/azure
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /area providers/azure Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Last run (): The run before (): no code changes in between
Now my PR passed the lane without any code changes (just yet another retest):
note that is not a presubmit for merge, or else it would show: <img width="89" alt="Screen Shot 2021-03-08 at 11 30 57 PM" src="https://user-"non-required presubmits can be bypassed with , but are generally worth looking at first
/assign Can you take a look?
There's something odd going on here. As the test description indicates, it deletes the pod for the StatefulSet and expects it to be recreated. When the kubelet attempts to recreate the pod, however, the mount point for the volume is no longer there. From the log above, we can see the pod was deleted succesfully. It was then rescheduled but failed because the volume mount point does not exist. In the kubelet logs, we see the successful mount for the initial pod creation: We then see the delete of the pod and the failed attempt to recreate it: Finally during cleanup, we see a failure during unmount the volume because the mount point does not exist: None of the other collected logs show what happened to the original mount point.


Question: if err != nil { klog.Error(log("registrationHandler.RegisterPlugin failed at CSI.NodeGetInfo: %v", err)) if unregErr := unregisterDriver(pluginName); unregErr != nil { <del> klog.Error(log("registrationHandler.RegisterPlugin failed to unregister plugin due to previous: %v", unregErr)) </del> <ins> klog.Error(log("registrationHandler.RegisterPlugin failed to unregister plugin due to previous error: %v", unregErr)) </ins> return unregErr } return err

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: If kubelet plugin registration fails during the registration process, the kubelet never attempts to register the socket again (assuming the error was transient), unless the socket for the driver is deleted & recreated to trigger a new registration. What you expected to happen: Maybe retry with exponential backoff when kubelet plugin registration fails instead of just logging and waiting for another filesystem event like it does today. Of course, if a socket returns a terminal error (does not implement the kubelet plugin registration service, for example), we should not retry. Follow up on /milestone v1.14 /sig storage How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
I'd like to work on this issue
/assign
Re opening. was a temporary work around for this. is the permanent fix.
Hello! We are starting Bug Freeze for 1.15. Is this issue still planned on being incorporated into 1.15?
the dependent PR is not yet merged. can we get this in for 1.15?
said that is the "permanent fix." and that has merged. So let's close this out /close
Closing this issue. <detailsIn response to : said that is the "permanent fix." and that has merged. So let's close this out Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: }}, metav1.CreateOptions{}) framework.ExpectNoError(err, "failed to create pod") framework.Logf("created %v", podTestName) <ins> framework.ExpectNoError(e2epod.WaitForPodNameRunningInNamespace(f.ClientSet, podTestName, f.Namespace.Name)) framework.Logf("running and ready %v", podTestName) </ins> } // wait as required for all 3 pods to be found

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI -- [sig-node] Pods should delete a collection of pods [Conformance] TBD links to appreciated links to specific failures in spyglass appreciated <!-- Please see the deflaking doc () for more guidance! --
cc /sig windows
I've looked through the logs for a bit and the test itself. What the test is doing: create 3 pods. expect to find 3 pods (it doesn't seem to wait for the to start, they only need to exist). delete all 3 of them as a collection. expect all 3 pods to dissapear. All 3 pods ended up on the same node, plus some other tests were running in parallel, which could also contribute to the failure. Looking at the kubelet logs, it seems that was the remaining pod to be deleted. Filtering the logs, we can see: The test failed at, while the other 2 pods did: Those times correspond with the number of pods observed by the test: Now, my question is, why are there such big gaps between the ` events? It's 12s for test-pod-2 and 21s for test-pod-3? Seems like a long time for pods / containers to linger after they've been killed. PS: It seems that Containers are still being created / started / stopped, even if the delete request came less than 0.2s after their creation. It doesn't seem to me like a reasonable thing to do, especially if that Pod would have to pull a huge image; I would have expected to be able to cancel that.
/triage accepted
Let's discuss at the 8/17 SIG-Windows community meeting
Hit again on 8/15
I think was planning on opening up a PR to use longer timeouts in this test. We see that some Windows containers do not get scheduled until 1+ minutes into the test - especially when there are many tests running in parallel.


Question: return nil } <ins> // TODO: This should probably not be a 403 </ins> return admission.NewForbidden(a, fmt.Errorf("Unable to create new content in namespace %s because it is being terminated.", a.GetNamespace())) }

Currently the admission controller returns a forbidden (403) error if the namespace doesn't exist. We should return a 404 error instead.


Question: } if err != nil { if ctx.Err() != nil { <del> Failf("Controller %s: Failed to Get from replica %d [%s]: %vn pod status: %#v", r.controllerName, i+1, pod.Name, err, pod.Status) return false, err </del> <ins> // We may encounter errors here because of a race between the pod readiness and apiserver // proxy. So, we log the error and retry if this occurs. Logf("Controller %s: Failed to Get from replica %d [%s]: %vn pod status: %#v", r.controllerName, i+1, pod.Name, err, pod.Status) return false, nil </ins> } Logf("Controller %s: Failed to GET from replica %d [%s]: %vnpod status: %#v", r.controllerName, i+1, pod.Name, err, pod.Status) continue

https://k8s- Failed: [] Network Partition [Disruptive] [Slow] [] [ReplicationController] should recreate pods scheduled on the unreachable node AND allow scheduling of pods on a node after it rejoins the cluster {Kubernetes e2e suite}
Builds: ci-kubernetes-e2e-gci-gke-serial ci-kubernetes-e2e-gke-serial-release-1.5 ci-kubernetes-e2e-gke-serial Failed: [] Network Partition [Disruptive] [Slow] [] [ReplicationController] should recreate pods scheduled on the unreachable node AND allow scheduling of pods on a node after it rejoins the cluster {Kubernetes e2e suite}
Builds: ci-kubernetes-e2e-gci-gke-serial-release-1.5 ci-kubernetes-e2e-gke-containervm-1.4-gci-1.5-upgrade-cluster-new ci-kubernetes-e2e-gke-serial-release-1.5 Failed: [] Network Partition [Disruptive] [Slow] [] [ReplicationController] should recreate pods scheduled on the unreachable node AND allow scheduling of pods on a node after it rejoins the cluster {Kubernetes e2e suite}
this is currently assigned to you -- can you take a look? I guess it has not failed individually recently, only as part of large "broken test runs" so maybe we should close it? Please advise.
This has only failed once in the last couple of weeks in https://k8s- and zero times in https://k8s- so I think we should close it. WDYT?
Yes, fixed a regression in this recently. Closing this.
It seems like this test is still flaking a lot on GKE
should this block 1.6?
Re-opening until we've verified it's fixed. It seems to have just failed again
I think the serial tests haven't picked up the fix yet.
Tests are looking good, closing if I see another success.
We need to cherrypick the fix in into 1.6. I've filed a separate issue to track the hanging network calls on the GKE apiserver. /cc
I believe this was fast forwarded onto the branch by
-- can we take this off of the 1.6 release now? It seems to be a test setup issue
Yes, this is not a blocker anymore.

this can be removed from the 1.6 milestone
This Issue hasn't been active in 49 days. It will be closed in 40 days (Jun 14, 2017). cc You can add 'keep-open' label to prevent this from happening, or add a comment to keep it open another 90 days


Question: cpu: "2" memory: 1Gi min: <del> cpu: 250m </del> <ins> cpu: 200m </ins> memory: 6Mi type: Pod - default: <del> cpu: 250m </del> <ins> cpu: 300m memory: 200Mi defaultRequest: cpu: 200m </ins> memory: 100Mi max: cpu: "2" memory: 1Gi min: <del> cpu: 250m memory: 6Mi </del> <ins> cpu: 100m memory: 3Mi </ins> type: Container

Since resource QoS is implemented, limitrange also supports request constraint, we can add an example for that.
cc


Question: var ignoreNotReady bool badPods := []v1.Pod{} desiredPods := 0 <ins> notReady := int32(0) </ins> if wait.PollImmediate(Poll, timeout, func() (bool, error) { // We get the new list of pods, replication controllers, and

Is this a BUG REPORT or FEATURE REQUEST?: /kind cleanup What happened: This log seems to imply that we wait until less than some number of pods are un-ready, and yet the loop succeeds if there is no improvement. What you expected to happen: Either the loop should terminate earlier (if the condition is acceptable) or fail (if it's unacceptable).
/sig scalability since this probably most affects us
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close


Question: return class }(), }, <ins> "good-parameters-apigroup": { class: func() *resource.ResourceClass { class := testClass(goodName, goodName) class.ParametersRef = goodParameters.DeepCopy() class.ParametersRef.APIGroup = goodAPIGroup return class }(), }, "bad-parameters-apigroup": { wantFailures: field.ErrorList{field.Invalid(field.NewPath("parametersRef", "apiGroup"), badAPIGroup, "a lowercase RFC 1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*')")}, class: func() *resource.ResourceClass { class := testClass(goodName, goodName) class.ParametersRef = goodParameters.DeepCopy() class.ParametersRef.APIGroup = badAPIGroup return class }(), }, </ins> "missing-parameters-name": { wantFailures: field.ErrorList{field.Required(field.NewPath("parametersRef", "name"), "")}, class: func() *resource.ResourceClass {

Following YAML file is accepted without error. Notice the field, it is an instead of , containing character. This can lead to an issue when resource driver generates structured parameters' in-tree object with being just without version. The DRA scheduler plugin will try to match the parameters object based on the reference here: In latest master the code appears to use an indexer, and the issue might be hidden, if applicable at all. I did not check the reproducibility in master yet. However, main question is whether the should allow value with character in general. An error is expected from the apiserver upon object creation request, reporting that APIGroup containes APIVersion. Maybe. Have any resource driver that supports structured parameters, , for instance, also puts (gpucrd.APIGroup) value into . v1.30.1 <details</details<details</details<details</details<details</details<detailsscheduler's dynamicresources plugin </details
/sig node /triage accepted
/assign


Question: status := sched.bindingCycle(bindingCycleCtx, state, fwk, scheduleResult, assumedPodInfo, start, podsToActivate) if !status.IsSuccess() { sched.handleBindingCycleError(bindingCycleCtx, state, fwk, assumedPodInfo, start, scheduleResult, status) <ins> return </ins> } // Usually, DonePod is called inside the scheduling queue, // but in this case, we need to call it here because this Pod won't go back to the scheduling queue.

test/integration/volumescheduling triggered a race condition when run with . From No race detected. , or testing with /sig scheduling master n/a No response No response No response No response
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc was merged instead of to fix a similar problem.
Read: Write:
cc (although I think he's OOO) Are you planning to work on a fix, Patrick, or should we find a volunteer?
I don't know how to fix this except via additional locking for that field (e.g. via ), so I wasn't planning to work on this myself because someone else might have a better idea.
Yes, sorry I'm on vacation and unfortunately I'm not sure when I'm back to work because the is blocking all flights from the vacation place. If no one else has time/a good idea for a quick bug fix, given we're close to the release, can we change the feature gate of the scheduling hint to alpha/disabled by default? And then run done() in the gate, where causing data race. That should be one of easy fixes to resolve issue in the default scheduler.
I had an idea: if we add to , then the code which currently needs to go through the pod pointer no longer needs that pointer. I'll give that a try. /assign
Or even simpler, replace with . It must be the same value, and is immutable in .
That would fix this particular race. But also passes to various other functions which also read the pointer. If using that pointer is unsafe, then those other functions also need to be fixed somehow. is this code safe?
That should be safe because of this: I'm not understanding what's different
Oh right, a Pod update would still change what's in the "assumed cache". I guess this is not a regression then.
So if makes a copy that isn't shared with anything, why is the event handler able to write into that same copy?

The write is done for a PodInfo found in the backoff queue:
Could it be that one of the steps in put the into the backoff queue because of a failure, then the event occurred, and then the code running after incorrectly used the which it no longer owns exclusively? If yes, then the proposed fix in is correct and complete. FWIW, I've tested it in and there are no more race reports. I'm just cautious because "no race report" does not mean "no races"...
Hmm, that theory doesn't pan out.: was only called in case of a success, so it's still unclear to me why that instance can be in the backoff queue.
It's not, and maybe that's the bug. already calls . So we shouldn't call again if there was a failure during binding. There should probably be an here:
/assign I want to help fix it
Oh, right, there's no after .
I appreciate your enthusiasm, but... This issue was already assigned to me. I did the work of finding the issue, reporting it, and analyzing the problem. The remaining work is to create a proper PR (good description and commit message with an explanation of the change, not just "fixes xyz") and to test that the race is indeed gone. Can we agree on letting me finish what I started and focus on that I just created? I verified that this particular commit fixes the race ().
sorry for that. I have close my pr
/unassign


Question: func initStatusForbiddenControlPlaneConfig() *controlplane.Config { controlPlaneConfig := framework.NewIntegrationTestControlPlaneConfig() <ins> controlPlaneConfig.GenericConfig.Authentication.Authenticator = authenticatorunion.New( authauthenticator.RequestFunc(func(req *http.Request) (*authauthenticator.Response, bool, error) { return &authauthenticator.Response{ User: &user.DefaultInfo{ Name:   "unprivileged", Groups: []string{user.AllAuthenticated}, }, }, true, nil })) </ins> controlPlaneConfig.GenericConfig.Authorization.Authorizer = authorizerfactory.NewAlwaysDenyAuthorizer() return controlPlaneConfig }

/kind failing-test <!-- If this is a flake, please add: /kind flake --/kind flake <!-- Please assign a SIG using: /sig SIG-NAME --/sig api-machinery
/assign I want to take a try
2 but actual 0 till timeout resource services, expected 2, actual 0 error waiting or ResourceQuota status: timed out waiting for the condition time later, actual state became 2 Seems polls a ResourceQuota status with timeout 10s may be not enough, we should enlarge it into 300s, that is poll for 300s timeout instead of 10s.
/triage accepted


Question: export ARTIFACTS_DIR=${WORKSPACE}/artifacts # Save the verbose stdout as well. export KUBE_KEEP_VERBOSE_TEST_OUTPUT=y <del> export KUBE_TIMEOUT='-timeout 300s' </del> export KUBE_INTEGRATION_TEST_MAX_CONCURRENCY=4 export LOG_LEVEL=4

/kind test-flake https://k8s-
/kind flake
cc
Saw this on
Sorry, I haven't had time to look into this. Let's see if can get to this earlier than I do.
I'm seeing this a lot in the rush to code freeze.
if this is actually a 1.10 bug, can you add status/approved-for-milestone to it? Thanks.
Something suspicious from the , but probably it's not related to the failure:
The garbage collector's behavior looks right, it does delete the dependent configmap: The tested start at 16:44:27., and failed with "test timed out after 5m0s" at 16:44:50.. I'll check if other failed cases have the same causes.
The "Kubernetes Aggregated Failures" shows there is no "TestCRDDeletionCascading" failure in the last tests (). Maybe I used the wrong tool? cc
cc because triage does not support pull-kubernetes-unit? I cannot find in as well
gave a new failure sample: https://k8s- (And the triage dashboard can't find it). Anyway, the test showed similar behavior. The tested started at I0226 20:57:02.. The gc behaved correctly: And the 5min timeout occurred 23s after the test began. I'll check what's the 5min timeout for.
The 5 min timeout is for all the tests in the test/integration/garbagecollector package. The tests are run sequentially, so is the last one to run, and unfortunately uses up the 5 min quota. The two logs I looked into showed the test itself was healthy. We just need to increase the timeout limit for this package.
found a lot of failure samples here: https://k8s- The first 11 of them failed because the 5m timeout.
Indicates this is still present and severe.
I was trying to get recent failures, it seems we recently moved intergration tests out of the unit test suites, https://k8s-, where there is no failure yet. I'll try again tomorrow.

(we rename the job this afternoon, it was named kubernetes-unit but only runs integration test)
https://k8s- that looks all green so far
Can we also check run time of the tests in that package over the past couple weeks? It seems like this just spiked. Did our tests slow down, or did we add tests to that package?
as far as I can tell the runtime of the test didn't spike ( shows the test running in just over 300s and also included the original bump of the timeout from 300s to 600s), but the timeout was accidentally decreased from 600s back to 300s as an unintended side effect of
ah, ok
Since it has passed the last 26 times, is this safe to reclose?
/close


Question: mounter := mount.New() if s.Containerized { glog.V(2).Info("Running kubelet in containerized mode (experimental)") <del> mounter = &mount.NsenterMounter{} </del> <ins> mounter = mount.NewNsenterMounter() </ins> } var dockerExecHandler dockertools.ExecHandler

What happened: Windows documentation claims: V1.VolumeMount.mountPropagation - only MountPropagationHostToContainer is available. However, the internals suggest otherwise; is always returned on Windows: Either the docs should be corrected, or the Kubelet should be updated, or we should have a comment on why this does in fact enable MountPropagationHostToContainer on Windows. /cc /sig windows
/cc
/cc
I think the doc is not accurate. The doc claims the same thing. For MountPropagationHostToContainer option, we can test it whether it is working or not.
i created two PRs to address the documentation gap. we will also test this and get back to this thread.
For 1.15 we may need to update Kubelet code to allow HostToContainer.


Question: NodeName:            c.NodeName, KubeletPort:         c.KubeletPort, KubeletReadOnlyPort: c.KubeletReadOnlyPort, <del> MaxPods:             maxPods, </del> <ins> MaxPods:             c.MaxPods, </ins> PodsPerCore:         podsPerCore, NodeLabels:          c.NodeLabels, RegisterWithTaints:  c.RegisterWithTaints,

A kubernetes cluster is normally constructed from a variety of machine templates. Different instance groups may have different CPU, memory capacity. They may be equipped with specialized accelerators, network devices. I think we should have an extensible way for users to custom node configuration to simulate real production cluster states. Currently, nodelabel and nodetaints are supported. I am consider to accept a template with more configuration like resources so we won't grow option list in the future. Without this feature, it's hard for users to simulate heterogeneous cluster.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/sig scalability