Question: } // Cloning fails if the source disk is still in the process of detaching, so we wait for the VolumeAttachment to be removed before cloning. volumeAttachment := e2evolume.GetVolumeAttachmentName(ctx, f.ClientSet, testConfig, l.testCase.Provisioner, dataSourceRef.Name, l.sourcePVC.Namespace) <del> e2evolume.WaitForVolumeAttachmentTerminated(ctx, volumeAttachment, f.ClientSet, f.Timeouts.DataSourceProvision) </del> <ins> framework.ExpectNoError(e2evolume.WaitForVolumeAttachmentTerminated(ctx, volumeAttachment, f.ClientSet, f.Timeouts.DataSourceProvision)) </ins> t.TestDynamicProvisioning(ctx) }(i) }

This line here lacks error checking: It shouldn't continue when that function times out. Add . No response <details</details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage
in StorageClassTest.TestDynamicProvisioning also fails in that test.
/assign Looks like it only times out on my local branch. I'll investigate that. However, the error check is missing on master.
Fixed, and I'll add the error check in
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/remove-lifecycle rotten Fix is in


Question: return &etcd.Response{}, f.Err } <ins> func (f *FakeEtcdClient) WaitForWatchCompletion() { f.condLock.Lock() defer f.condLock.Unlock() f.condWatchCompleted.Wait() } </ins> func (f *FakeEtcdClient) Watch(prefix string, waitIndex uint64, recursive bool, receiver chan *etcd.Response, stop chan bool) (*etcd.Response, error) { f.WatchResponse = receiver f.WatchStop = stop injectedError := make(chan error) defer close(injectedError) f.WatchInjectError = injectedError <ins> f.condWatchCompleted.Broadcast() </ins> select { case <-stop: return nil, etcd.ErrWatchStoppedByUser

Detected by
Create a Pod as follows: Name: jenkins Using image: jenkins In a new Kubenetes namespace named website-frontend kubectl create ns website-frontend apiVersion: v1 kind: Pod metadata: name: jenkins namespace: website-frontend spec: containers: name: jenkins image: jenkins kubectl create -f


Question: verbs:     "get,watch,list", resources: "replicasets/scale", expectedRole: &rbac.Role{ <ins> TypeMeta: v1.TypeMeta{APIVersion: "rbac.authorization.k8s.io/v1", Kind: "Role"}, </ins> ObjectMeta: v1.ObjectMeta{ Name: roleName, },


Question: return pod.Status.Phase != "" } <ins> // isPending returns true if pod has a Phase of PodPending func isPending(pod *v1.Pod) bool { return pod.Status.Phase == v1.PodPending } </ins> // isFailed returns true if pod has a Phase of PodFailed func isFailed(pod *v1.Pod) bool { return pod.Status.Phase == v1.PodFailed

User cordons a node with a pvc bound to a local persistent volume User deletes a stateful set pod from that node The pod is in pending because the pvc is still bound to a local pv on the cordoned node (expected) The node dies and the pvc is deleted The pod is still stuck in pending because the pvc is missing (bug) The pod now needs to be manually deleted so that the pod+pvc can be recreated After the pvc was deleted, the statefulset controller should have recreated it and bound it to a new node's local pv so that the pending pod could be scheduled. Currently the only way to get the pod out of pending is to manually delete it so that it recreates itself and its pvc We used for the local PVs, but the problem is not limited to this particular provisioner create a stateful set with a pvc that is bound to a local PV use to cordon a node with a local pv delete a statefulset pod from that node delete the pvc from that node the pod is now stuck in pending due to a missing pvc There was a related issue but this only fixed the issue when the order of operations was delete pvc -delete pod. but in this case, it is delete pod -delete pvc <details</details<detailsTested on aws and gcp </details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig apps storage
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The Kubernetes project currently lacks enough contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
This issue was fixed by The new issues look different. /close
Closing this issue. <detailsIn response to : This issue was fixed by The new issues look different. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: return v1.PodSpec{ Containers: []v1.Container{ { <del> Name:  "iperf-client", Image: imageutils.GetE2EImage(imageutils.Agnhost), </del> <ins> Name:    "iperf-client", Image:   imageutils.GetE2EImage(imageutils.Agnhost), Command: []string{"/bin/sh"}, </ins> Args: []string{ <del> "/bin/sh", </del> "-c", "/usr/local/bin/iperf " + familyStr + "-c service-for-" + appName + " -p 8002 --reportstyle C && sleep 5", },

What happened: test failing My cluster is based on 1.16.3 release with multus CNI, when I ran networkingperf suite from , it is failing out with below error. Apr 17 07:46:25.532: INFO: At 2020-04-17 07:36:40 +0000 UTC - event for iperf-e2e-pod-0: {kubelet worker-pool1-09u5e8jl-esiperu-demo-cluster} FailedCreatePodSandBox: Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "fb85ec94a3d2f884902dc4ccc5cf05cf9c50bb4f53d4f14903fcad3302fd18d8" network for pod "iperf-e2e-pod-0": networkPlugin cni failed to set up pod "iperf-e2e-pod-0network-perf-3964" network: Multus: [network-perf-3964/iperf-e2e-pod-0]: error adding container to network "k8s-pod-network": delegateAdd: cannot set "" interface name to " eth0": validateIfName: no net namespace /proc//ns/net found: failed to Statfs "/proc//ns/net": no such file or directory What you expected to happen: The networkingperf test should work. How to reproduce it (as minimally and precisely as possible): ./hack/ginkgo- -- -- Anything else we need to know?: This issue is there in current master branch also Both IPv4 and IPv6 Fails. The issue is container configuration in the test, the ENTRYPOINT for the agnhost image is agnhost , which will not take the argument as ""/bin/sh", "-c", "/usr/local/bin/iperf " + familyStr + "-c service-for-" + appName + " -p 8002 --reportstyle C && sleep 5"," hence container is coming out early, thus CNI is complaining. container configuration should take command as "/bin/sh" and argument as "-c", "/usr/local/bin/iperf " + familyStr + "-c service-for-" + appName + " -p 8002 --reportstyle C && sleep 5" .

Environment: Kubernetes version (use ): 1.16.3 Cloud provider or hardware configuration: OS (e.g: ): SUSE Linux Enterprise Server 15 SP1 Kernel (e.g. ):4.12.14-197.29-default Install tools: Network plugin and version (if this is a network-related bug): Others: /sig network /sig testing
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : What happened: My cluster is based on 1.16.3 release with multus CNI, when I ran networkingperf suite from , it is failing out with below error. Apr 17 07:46:25.532: INFO: At 2020-04-17 07:36:40 +0000 UTC - event for iperf-e2e-pod-0: {kubelet worker-pool1-09u5e8jl-esiperu-demo-cluster} FailedCreatePodSandBox: Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "fb85ec94a3d2f884902dc4ccc5cf05cf9c50bb4f53d4f14903fcad3302fd18d8" network for pod "iperf-e2e-pod-0": networkPlugin cni failed to set eth0": validateIfName: no net namespace /proc//ns/net found: failed to Statfs "/proc//ns/net": no such file or directory The networkingperf test should work. The issue is container configuration in the test, the ENTRYPOINT for the agnhost image is agnhost , which will not take the argument as ""/bin/sh", Kubernetes version (use ): 1.16.3 Cloud provider or hardware configuration: OS (e.g: ): SUSE Linux Enterprise Server 15 SP1 Kernel (e.g. ):4.12.14-197.29-default Install tools: Network plugin and version (if this is a network-related bug): Others: /sig testing /kind/failing-test Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.


Question: <del> apiVersion: v1 kind: Namespace metadata: name: kube-system </del>

What happened: Add-on manager uses kubectl apply to create the kube-system namespace at start-up. This is redundant because the kube-apiserver creates the namespace in NewBootstrapController. This is a problem because if someone uses apply to make changes to the kube-system namespace, the add-on manager will wipe out these changes. What you expected to happen: Kube-APIServer has the sole responsibility of creating the kube-system namespace. How to reproduce it (as minimally and precisely as possible): kubectl apply a label to the kube-system namespace. restart the add-on manager. Note the labels have been removed. Anything else we need to know?: /sig api-machinery Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Others:
/sig api-machinery
the contention script wipes out namespace ^^
/cc
i guess it is intentional to have the addon manager to ensure kube-system is not modified, it tries to avoid the case that some cluster admin accidentally delete kube-system that result in a non-functional cluster.
The add-on manager will only perform the check when it restarts. The API Server will also perform this check when it restarts. I think we should only have 1 system responsible for this and I believe that system should be the API server.
trying to have my first contribute here, so is it proper to have the kube-system existence checked before applying the yaml file to create the kube-system namespace once again?
The Kube API Server will check for the existence of the kube-system namespace and create it if absent. It does all of this using client-go requests. The Add-On manager is the only system creating the kube-system by applying (a private) yaml file. I believe the correct fix here is to remove the Add-On managers attempts to create the kube-system namespace entirely. This is especially important if a user writes a CI/CD controller to apply their desired state to the kube-system namespace. I would argue the CI/CD is correct but the add-on manager will randomly wipe out the users changes. The API Server on the other hand will not.
, thanks for your help, so i will remove that line and pull a request later
Trying to get my first PR here. Is the issue still open ?
The issue is still open. You may want to check on though as was trying to fix the issue with that PR.


Question: type cniNetworkPlugin struct { network.NoopNetworkPlugin <del> loNetwork      *cniNetwork </del> <ins> loNetwork *cniNetwork sync.RWMutex </ins> defaultNetwork *cniNetwork <del> host           network.Host execer         utilexec.Interface nsenterPath    string </del> <ins> host        network.Host execer      utilexec.Interface nsenterPath string </ins> } type cniNetwork struct {

Host network pods should be able to be created before the container network exists. Pod syncs for pods on the container network should fail until the container network is configured. cc
ref:
I think this is a thing we should make working before v1.4, so adding the milestone. Complain if you disagree. Also, does volunteer to make a PR? :)
Going to make a first pass at this in
Great!
/cc


Question: if !hasTerminalPhase { var names []string for _, status := range pod.Status.ContainerStatuses { <del> if status.State.Terminated != nil || status.State.Running != nil { </del> <ins> if status.State.Running != nil { </ins> names = append(names, status.Name) } }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --When a pod is evicted under resource pressure, it goes into an error state. The pod terminates normally. Note that the 130GB file is based on 90% of total disk space - current utilization to hit the default eviction threshold. Full pod spec after going into error state: <details</detailsSee also Eviction tests () are currently broken (), there's a PR up to fix: Kubernetes version (use ): HEAD/1.23.x (kubeletVersion="v1.23.0-alpha.3.47+") Cloud provider or hardware configuration: local-up-cluster OS (e.g: ), Kernel (e.g. ):Install tools: local-up- Network plugin and version (if this is a network-related bug): N/A Others: Also reproducible on OpenShift 4.9 (1.23-based) /sig node /cc
mentions this looks very similar to
Confirmed that this is a regression from 1.21. When running the reproducer against the release-1.21 branch, the pod above correctly stays in Evicted status (since pods are mortal). On 1.22, it tries to restart, entering either an Error or ContainerCreating state. /kind regression
/assign
/triage accepted
From : is the first bad commit This was
Seems likely that wasn't actually fixed. The broken eviction tests made it much harder to verify.
Attaching logs from HEAD from reproducer: v=4 Let me also try to get a repro where it goes into Error
Logs from HEAD for reproducer when pod goes into Error: v=4
Roughly, kubelet syncTerminatingPod sends "Failed" to status manager (which looks like it generates the correct apiserver patch for phase running -failed), then somehow by the time Kubelet syncTerminatedPod is called it sends "Running but with errors" to status manager, and status manager allows that transition (internally) and then sends it to the apiserver. So the pod goes Running -Failed -Running which should never happen (we "lost" that the pod was stopped).

Bug 1: pod patch status probably shouldn't allow terminal -nonterminal phase transitions (didn't we fix that??) Bug 2: pod worker is supposed to be getting most up to date status from status manager before calling syncTerminatedPod, which should include "Failed", but that isn't happening? thus the status manager then sends a "non-terminal state" (shouldn't ever happen) Roughly, figure out why status passed to syncTerminatedPod is older than what was sent by syncTerminatingPod, then figure out why status manager allowed that transition back to running, then figure out why apiserver didn't reject.
Staring at logs, I've figured out where the illegal phase transition is happening: If the container failed to properly terminate and generate a status saying as much, generateAPIPodStatus won't have any hints available to it that the pod died because it was killed. (This is happening in the reproducer case because the container won't exit before the termination grace period, and needs to be force killed.) Thus, it will assume that since the container used to be running, its current status is running. This results in an illegal phase transition.
From what I can tell, the issue is that is being passed a pod that still has running containers. From kubelet logs: which gets set as part of . It looks like that will always run so long as succeeds. Thus, I feel like syncTerminatingPod must have the bug: it returns successfully, but the containers of the pod aren't all actually terminated.
/priority critical-urgent based on priority of and the impact of this regression (Kubelet-initiated evictions frequently result in error state pods)
The containers ARE terminated, we just haven't retrieved their status. This is because of a decision to not refresh the container status cache in syncTerminatingPods. So syncTerminatingPod is not a bug.
So then perhaps the right thing to do here would be to refresh the container status cache before syncing another status to the API server in ? Then we can avoid syncing the outdated cached status
Updating from an internal discussion last night - bug 2 isn't actually a bug.

Lantau convinced me we wouldn't refresh the status cache, so what happens is we call generatedAPIStatus with: API status (mostly irrelevant) status manager status to date container cache (but 2s) after The key thing is that phase from runtime status is irrelevant when 1 or 2 are terminal - because runtime has no idea about whether a pod is terminal. So roughly we should be ignoring phase from 3 except when determining when to go from pending to running. The bug here is that we're thinking that this is an "illegal transition" but still copying the latest reason and message (runtime status doesn't get to weigh in on overal terminal phases). We shouldn't need to refresh status at all.
Okay, so there is definitely a bug in what's being synced to the API server for evicted pod statuses right now, possibly for other terminated pods. In 1.21 and earlier, when a pod is evicted, it looks like this: <details</detailsNote that the status field is very minimal, and there are no container statuses. In 1.22+, we attempt to sync container statuses on termination, and sometimes they are wrong (e.g. changing phase from Failed -Running). Here's an example of what an evicted pod could look like on 1.22: <details</details
This appears to be broken even for runOnce pods, tested against latest HEAD (). With spec: After scheduling the pod... And the final pod status: <details</details
Elana identified the issue on status here - previously we NEVER reported container status after an eviction, so the refactor (which correctly ensured container exit codes showed up) was returning containers with state=waiting,lastState=terminated, which is incorrect. generateAPIStatus needs to know the difference between "current container status, waiting for more to show up" and "any status we have is the best we have". So once killPods succeeds in syncTerminatingPods and in syncTerminatedPods, we're no longer "waiting". So generating pod status should correctly summarize the container status as and not inject .


Question: return false, cloudprovider.NotImplemented } <ins> // InstanceShutdown returns true if the instance is in safe state to detach volumes func (g *Cloud) InstanceShutdown(ctx context.Context, node *v1.Node) (bool, error) { return false, cloudprovider.NotImplemented } </ins> func nodeAddressesFromInstance(instance *compute.Instance) ([]v1.NodeAddress, error) { if len(instance.NetworkInterfaces) < 1 { return nil, fmt.Errorf("could not find network interfaces for instanceID %q", instance.Id)

For external cloud provider - Kubelet setting the TaintExternalCloudProvider taint: The results in calling function in cloud node-controller: This (in a healthy case) sends 4 GCE API calls to: node addresses: (as part of getNodeModifier call) instance type: (as part of getNodeModifiers call) instance: (as part of updateNodeAddresses call) node addresses (again): (as part of updateNodeAddresses call) All those requests are read requests and could be replaced by a single call. We could do that by introducing a simple struct like: and adding a method (or rather replacing the existing InstanceTypeByProviderID which is used only for this purpose) to Instances cloud-provider interface; With that change, we would be able to replace all that 4 calls with just a single API call. This would be useful especially for creating large clusters (like 5k nodes), which may saturate the quota for API calls for quite some time. This is quite significant size of refactoring, though it should be relatively mechanical task to do. /sig cloud-provider /sig scalability /priority important-longterm /help-wanted - FYI
The InstanceMetadataByProviderID would be to every cloud provider, such as aws, azure, etc. right ?
/assign Hi , if you're interested in it, I can work with you.
Yes - we need to do that (which is why I wrote it's quite significant refectoring). It also makes reviews harder, because we will need to ask maintainers of all cloud-providers. So it would be good to separate changes to different cloud providers into separate commits.
I have one question: and first query by providerID, if failed then query by nodeName. While the logic in is converse(if providerID is nil, get providerID by nodeName, then ensure its existence by providerID). If we merge these three method into , how to deal with the query sequence of nodeName and providerID?
I bet it's a consequence of different people adding different code paths. Those conceptually seem to be the same to me. Unifying this will be additional benefit from this change imho. So I would say that choose whatever is easier to implement.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
I think and are the only PRs left for this. One other optimization that can be made is to cache the instance metadata but I think that's a provider specific implementation
Plus enabling it for GCE if we decide to remove it from the first PR. But yeah, in general I agree.
- can you please address ?
Yes, I have seen that.
I think this issue has been resolved, feel free to reopen it. /close
Closing this issue. <detailsIn response to : I think this issue has been resolved, feel free to reopen it. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: diskName:         fakeCreateDiskFailed, oldSize:          *resource.NewQuantity(2*(1024*1024*1024), resource.BinarySI), newSize:          *resource.NewQuantity(3*(1024*1024*1024), resource.BinarySI), <del> existedDisk:      compute.Disk{Name: to.StringPtr(fakeCreateDiskFailed), DiskProperties: &compute.DiskProperties{DiskSizeGB: &diskSizeGB}}, </del> <ins> existedDisk:      compute.Disk{Name: to.StringPtr(fakeCreateDiskFailed), DiskProperties: &compute.DiskProperties{DiskSizeGB: &diskSizeGB, DiskState: compute.Unattached}}, </ins> expectedQuantity: *resource.NewQuantity(2*(1024*1024*1024), resource.BinarySI), expectedErr:      true, expectedErrMsg:   fmt.Errorf("Retriable: false, RetryAfter: 0s, HTTPStatusCode: 0, RawError: Create Disk failed"), }, <ins> { desc:             "an error shall be returned if disk is not in Unattached state", diskName:         fakeCreateDiskFailed, oldSize:          *resource.NewQuantity(2*(1024*1024*1024), resource.BinarySI), newSize:          *resource.NewQuantity(3*(1024*1024*1024), resource.BinarySI), existedDisk:      compute.Disk{Name: to.StringPtr(fakeCreateDiskFailed), DiskProperties: &compute.DiskProperties{DiskSizeGB: &diskSizeGB, DiskState: compute.Attached}}, expectedQuantity: *resource.NewQuantity(2*(1024*1024*1024), resource.BinarySI), expectedErr:      true, expectedErrMsg:   fmt.Errorf("azureDisk - disk resize is only supported on Unattached disk, current disk state: Attached, already attached to "), }, </ins> } for i, test := range testCases {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: resize Azure disk issue when it's in attached state: What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /priority important-soon /sig cloud-provider /area provider/azure /triage accepted /kind bug /assign


Question: defer cleanup() // Check different-node test requirement <ins> if l.driver.GetDriverInfo().Capabilities[CapSingleNodeVolume] { framework.Skipf("Driver %s only supports %v -- skipping", l.driver.GetDriverInfo().Name, CapSingleNodeVolume) } </ins> nodes := framework.GetReadySchedulableNodesOrDie(l.cs) if len(nodes.Items) < 2 { framework.Skipf("Number of available nodes is less than 2 - skipping")

What happened: When we run csi hostpath driver e2e tests on a multi-node cluster some tests failed because they assume that volumes are node independent but this is not the case for some drivers like hostpath. Related PR; What you expected to happen: Skip the tests that assume volumes are node independent for the drivers that don't support node independent volumes. /sig storage /assign


Question: Err                error Command            string TmpDir             string <ins> CategoryExpander   resource.CategoryExpander </ins> ClientForMappingFunc             func(mapping *meta.RESTMapping) (resource.RESTClient, error) UnstructuredClientForMappingFunc func(mapping *meta.RESTMapping) (resource.RESTClient, error)

Is this a BUG REPORT or FEATURE REQUEST? (choose one): FEATURE REQUEST Kubernetes version (use ): HEAD Now that RBAC is going to GA, using ServiceAccounts is getting more and more important. Still a lot of users run their apps/addons/whatever in the SA. In order for the admin to be able to change ServiceAccount easily after an application has been installed (which supposedly is crashlooping due to insufficient privs), a command could exist. I know I can do this: but for new users to Kubernetes, it's non-obvious. Proposed command: Tracking issue: WDYT? This would be a good way for a new contributor to get aquainted with the code by contributing this feature.
Is anyone working on this? If not I would like to take this up.
Nobody is working on this. I would be super happy if you could contribute this feature. Be aware of that there isn't consensus around this proposal yet (I proposed it yesterday, saturday, so people haven't had time to catch up on it) However, I do think that this is in line with , so feel free to start working on this if you want to ;) Also, don't hesitate to ask for help if you don't know how to get started and we'll help out! Thanks!
I'm in favor of this command. ?
Thanks, I will start working on this.
Is the SA on a Pod mutable or immutable? What about DaemonSet and StatefulSet? Do those have SA as well? Is there a way to do this so that it dynamically supports anything with a PodTemplate? Maybe this should be done as a subresource that can be dynamically discovered by kubectl using the discovery service.
Maybe helpful:
Pod is immutable. All the other objects (with the exception of stateful set) are mutable.
That pattern is the same for all the things that set aspects of a pod spec (except image, which is also mutable in pods)
Statefulsets are going to loosen their validation soon I expect.
This sounds good to me. just go ahead, I will help review this. Thanks ~
The outout of this command kubectl get rc/nginx-controller -o=yaml , has service acccount at the following path '' , and the corresponding pod has service account at 'spec.serviceAccountName'.

Both refer to the 'serviceAccountName' field of PodSpec struct but can be made to be different by patching the ReplicationController resource. How does ReplicationController and Pod use serviceAccountName differently? Shouldn't the ReplicationControllerSpec have a serviceAccountName field?
ptal, I think you are more familiar with ReplicationController implementation than me.
ReplicationController contains a pod spec object, which contains a serviceAccountName.
Hi , how to go about writing e2e test cases for this. My development is mostly complete.
I wouldn't mind if you posted the PR now and then we'd see what e2e test makes sense to have. In order to move forward...
what are the next steps?


Question: } func (server *KubeDNSServer) Run() { <ins> glog.Infof("%+v", version.Get()) pflag.VisitAll(func(flag *pflag.Flag) { glog.Infof("FLAG: --%s=%q", flag.Name, flag.Value) }) </ins> setupSignalHandlers() server.startSkyDNSServer() server.kd.Start()

This makes it so much easier to tell what you are debugging. is a not great version--the stringification elides some useful info--but it's a start.
And also flag values.
is the same thing but adds flags.
will handle flags for everything that uses .
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
/remove-lifecycle rotten
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
/cc I think you might be interested in this. Its absence makes debugging harder.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten Adding a good first issue label on this because the idea is clear. If anyone wants to work on this, please see and as examples on how to add the version info. go through all binaries and check if these exist.

If they don't, add these in the glog lines. /good-first-issue
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /remove-lifecycle rotten go through all binaries and check if these exist. If they don't, add these in the glog lines. /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hey may I take this one?
We would love to get this fixed but would prefer if you choose one good-first-issue out of this and first. Feel free to send a PR for one issue and if the other issue hasn't been claimed yet, feel free to send a PR for that one as well. :)
/kind cleanup
please, sync up together, since your PRs are overlapping from kubeadm's perspective :)
, I remove kubeadm's perspective , thanks.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.

</details
/remove-lifecycle rotten
+1
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
/assign
I believe this issue can be closed now, from what I could see all the intended binaries now include version logging.
thanks! Were there some other CLs for binaries besides kubemark?
I count/see the following: Which I think covers everything that makes sense; notable exclusions being hyperkube, kubeadm, kubectl, and a few of the other misc things under /cmd
Seems the list is complete. Should this issue be closed rather still open?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available .


Question: }, }, }, <del> } for _, successCase := range successCases { successCase.old.ObjectMeta.ResourceVersion = "1" successCase.update.ObjectMeta.ResourceVersion = "1" if errs := ValidateNetworkPolicyUpdate(&successCase.update, &successCase.old); len(errs) != 0 { t.Errorf("expected success: %v", errs) } } errorCases := map[string]npUpdateTest{ "change name": { </del> <ins> "change spec": { </ins> old: networking.NetworkPolicy{ ObjectMeta: metav1.ObjectMeta{Name: "foo", Namespace: "bar"}, Spec: networking.NetworkPolicySpec{

I tried to update the NetworkPolicy Spec field with more ingress rules but got It seems NetworkPolicy does not allow update on Spec This means if I want to have more ingress rules, I will have to add those in a new networkpolicy in the same namespace. Any reason why I should not add those in the existing one(update the existing networkpolicy)? Thanks!
This is as designed for now. The expectation is that updates to the spec will be allowed when the NetworkPolicy API moves from v1beta1 to GA.
(And that is planned for Kubernetes 1.6 release.)
good to know, thanks!
Note that the PR to move NetworkPolicy to v1 did not make it into v1.6, but is now planned for 1.7:
Oh, I didn't know about this bug and that didn't get changed in but we can do it as another PR after. (I'm trying to not make any more changes there so we can get it merged.)


Question: return boolValue, nil } <ins> func helpErrorf(cmd *cobra.Command, format string, args ...interface{}) error { cmd.Help() msg := fmt.Sprintf(format, args...) return fmt.Errorf("%sn", msg) } </ins>

The '-context' does not seem to add any value
add priority and team please.
Blocked on
/sig cli


Question: // calculatePriority returns the priority of a node. Given the sumScores of requested images on the node, the node's // priority is obtained by scaling the maximum priority value with a ratio proportional to the sumScores. <del> func calculatePriority(sumScores int64) int64 { </del> <ins> func calculatePriority(sumScores int64, numContainers int) int64 { maxThreshold := maxContainerThreshold * int64(numContainers) </ins> if sumScores < minThreshold { sumScores = minThreshold } else if sumScores > maxThreshold {

<!-- Please only use this template for submitting enhancement requests --What would you like to be : redefine the formula, the score should be capped per images instead of pod. Why is this needed: For the pod that going to be scheduled, calculate the score based on the below steps, Scaled score base on the image size and it's spread over the cluster, image[i].nodes: number of nodes that already have image ! Capped score based on the assumption 90% image size on dockerhub falls into the range [23M ~ 1000M] lowerbound: 23M upperbound:1000M ! Final scores returned for scheduling, maxscore: 100 ! The second step is not well guarded, the assumption is based on the image size while the pod may hold multiple images, it is not surprise to see that total image size of the pods is larger than 1000M, while each of might be only around 300M. In this below case, following the above formula, NODE1 and NODE2 willl get the same scores(100), since both of them are capped by the max score (100), the ideal result is score of NODE2 should be larger than NODE1 (all requested images are ready on NODE2) and is preferred, the larger the sample1 here the more scores should be expected for NODE2. !
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig scheduler Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scheduling
cc
Thanks a lot for summarizing the problem. I agree that the formula needs improvement. ~I think doing the capping for each image solves the original problem and what said in There are two options: only the total from to for each container. We have to ask ourselves how much signal each formula gives and depending on what. Think about different scenarios: a pod with a lot of very small containers, a pod with a few big containers, etc.
/assign
Here is my thoughts, step1: calculate image level priority, for each image (should be same as exsting func ) step2: cap the each image's score with thresholds, minThreshold and maxThreshold.

(should be same as existing func except that it's per image instead of pod) step3: score normalization per image. step4: calculate pod level priority, and return the final score, range (0~100) step1 and step2 primarily update the exiting code to make it is image level priorities instead of pod level. Based on the above steps, consider the below two cases, a pod with four images, case1: All images are ready on node2 node1 (0, 80, 90, 60) - final score is: (80+90+60) / 4 = 57 node2 (90, 80, 90, 60) - final score is: (90+80+90+60) / 4 = 80 √ case2: each node has three images ready but image1 is larger than image2. node1 (0, 80, 70, 60) - final score is: (80+70+60) / 4 = 52 node2 (90, 0, 70, 60) - final score is: (90 +70+60) / 4 = 55 √ thoughts? cc
Thanks can you update the the examples with the score that we would have gotten with the current function?
For reference, see this analysis for a different plugin
Also it's worth to see the difference with just capping per pod, instead of per container.
can you also describe the image sizes you used? the example in the original post assumed three images, while your analysis is assuming four.
agree with and the explanation in looks good but just to be sure it would be nice to see the math worked out with image sizes, and compared to the current formula
Sorry for the confusion, let me switch back to the original three images, update the image size a little bit to make it easier to understand, and also fit to the case of and . Here is the pod, ! and three nodes in the cluster, ! Let's see the difference between two approaches, capping per pod, step1: ! NODE1: scores = 30 2/3 + 1500 2/3 = 1020 NODE2: scores = 30 2/3 + 600 2/3 + 1500 2/3 = 1420 step2: !

NODE1: 1020 1000 (maxThreshold), cappedscore = 1000 NODE2: 1420 1000 (maxThreshold), cappedscore = 1000 step3: ! upperbound = 100 / lowerbound = 23 / maxscore = 100 NOTE1: score = 100 (1000 - 23) / (1000 - 23) = 100 NODE2: score = 100 (1000 - 23) / (1000 - 23) = 100 potentially, NODE1 will be chose for scheduling. capping per images, step1: calculate image level priority, NODE1: image1: score = 30 2/3 = 20 image2: score = 0 image3: score = 1500 2/3 = 1000 NODE2: image1: score = 30 2/3 = 20 image2: score = 600 2/3 = 400 image3: score = 1500 2/3 = 1000 step2/3: cap each image's score with thresholds and calculate the image score, ! NODE1: image1: score = 100 (23 - 23) / ( 1000 - 23) = 0 image2: score = 0 image3: score = 100 (1000 - 23) / (1000 - 23) = 100 NODE2: image1: score = 100 (23 - 23) / ( 1000 - 23) = 0 image2: score = 100 (400 - 23) / (1000 - 23) = 35 image3: score = 100 * (1000 - 23) / (1000 - 23) = 100 step4: final score NODE1: finalscore = 100 / 3 = 33 NODE2: final_score = 135 / 3 = 45 NODE2 will be chosen for scheduling.
Computing the score per image and normalizing by the number of images make sense to me at first glance, but I would like to see more examples.

It would be much easier to digest the analysis if it was summarized in tabular form comparing three scoring functions: (1) current one, (2) one in and (3) multiple big images, multiple small images (total less than the threshold) and a mix. Also, compare cases where the spread is high vs low.
Yes, we don't need to see the individual steps, but a good set of scenarios with final scores. I didn't mean just the old function. We know it's very ineffective. What I meant is capping per pod using the strategy you used in
Did some comparison here, spread has been included based on the layout of each node, it's not easy for me to pull this factor out to focus on this factor only. I will design more testcases when the patch is ready for review which I hope could be more clear. column 1: pod with three images column 2: three nodes' images layout column 3 / 4 / 5: score of each node for different approaches. pod cap and normalizing per image
[10, 20, 30] & small [0,0,0] [800, 1200, 1500] & big [33, 59, 67] [30, 600, 1500] & mix [33, 45, 0]
Normalizing by the number of images gives each image the same "weight". Another variation of the third option is to try and multiply each image score with its size divided by the total size of all images, can you add that as another option?
I don't understand that suggestion. The score of an image is already proportional to its size. I would say the difference between the two suggestions is: Capping per pod cares about total downloaded size Capping per image gives roughly the same importance to each image. I think capping per pod makes more sense. However, there is some information we are losing in both suggestions, as seen in the first case. Even if a lot of small images need to be downloaded, we are giving a 0 score to all nodes. I think a cap from makes more sense, as suggested in the initial PR.
It helps giving more importance to larger images.
If the statistics are accurate with regards to image size distribution, then having many small images all scoring zero isn't common.

For the more common cases (sizes between 32-1000), it seems that both approaches will likely give similar results, so I am fine with both approaches.
That would make the score proportional to the square of the size. Just to play safe (ie, not make the score too strong in a way that could affect spreading or resource allocation) I suggest we go with capping per pod. And then we can re-evaluate next release.
Speaking of statistics. They might have changed since this plugin was first developed. Is there a reliable source?
True for uncapped images, but everything we are doing is just heuristics, and the suggestion is just for the sake of exploring options, not advocating a specific one. Playing it safe sounds good to me.
This approach needs the smallest change and the minimum impact.
+1, this is not common case we can see that a pod with lots of small images (all < 23M), but a corner case that might be improved. No perfect solution so far, but either one is better than the original one IMHO.
/cc , Do you have any comments so far?
btw, has been updated to the second approach (capping and normalization per image).
I would go with this approach as well, it preserves the behavior for multiple small images, and improve it for multiple big ones.
seems like a simple enough compromise for now, and running through some of the examples with that formula it does get more of the specificity you're looking for with large images without normalizing all smaller images right to 0 (so we keep some of the detail there too). I would like to see some updated statistics on the thresholds, and while 90% of the images might fall between them I'm wondering if that's a totally normal distribution. suggestion in got me (tangentially) thinking if we could use thresholds that are more relevant to the pod in question, rather than all of Dockerhub. Because as it is now hypothetical pods (and clusters) which have all small images don't get much benefit from ImageLocality. But I recognize that could be intentional since download resources for small images is ultimately negligible.
I also believe that scoring of small images by zero is intentional, but seems like making lots of small images in pod accountable for the scheduling is slightly better.
It looks like we have consensus. Can you update the PR?


Question: <del> load("@io_bazel_rules_go//go:def.bzl", "go_library") </del> <ins> load("@io_bazel_rules_go//go:def.bzl", "go_library", "go_test") </ins> go_library( name = "go_default_library",

In we observed a crashlooping controller cause a cluster cold boot failure, as controller-manager wasn't able to stay up long enough to run the deployment that would have fixed the problem. We should audit everything that runs in kube-controller-manager and ensure that they all fail gracefully enough to prevent crashloops, or make them individually restart in that case, instead of the process. Longer term, we should consider classifying the controllers by run level or bootstrap order (), and splitting them into separate binaries or changing the start-up order in kcm to make sure this sort of thing doesn't happen in the future.
Same issue with metrics-server, using minikube. Also reproduced on our test cluster. Steps to reproduce: Set up localkube Install metrics-server Stop minikube: sudo minikube stop Start minikube: sudo -E minikube start --vm-driver=none Expected: Kubernetes is running Actual: Kubernetes is dead Logs: Workaround: To fix the system, we must delete the metrics-server: Now kubernetes will start normally. After that we can reinstall the metrics server, and it will be ok.
Related:
This will be a hard problem to fix permanently. In all likelihood, we should stop allowing Fatalf() in kubernetes/kubernetes, but as you can see we do this to ourselves right here...
/kind bug
I also think this bug would benefit from an e2e test where we misconfigure things in a variety of ways and verify that KCM still comes up and stays up: [ ] misconfigured apiservice [ ] broken service attached to that apiservice [ ] RBAC rules that disallow things. maybe everything! [ ] webhooks that disallow everything [ ] yr example here
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: <del> load("@io_bazel_rules_go//go:def.bzl", "go_library") </del> <ins> load("@io_bazel_rules_go//go:def.bzl", "go_library", "go_test") </ins> go_library( name = "go_default_library",

What would you like to be : Actually disable scheduler plugins according to the disabled list in the scheduler's component config. The semantics are defined here: Part of /sig scheduling /priority important-soon /assign
The following semantics seems vague, and I think we should reach consensus on some cases when disabled plugins are involved. Suppose we have two default plugins, and , the runs before , here are the cases we could meet with the scheduling framework, plugins which run after all the default plugins plugins which should run ahead of the plugins plugin which should run ahead of all the plugins -enable default or custom plugin should return errors
Do you have any other cases not mentioned here?
Those cases sound right, I don't have others.
Is this actually what we want? I don't think so. The comment needs rewording.
I plan to revisit the comments after


Question: <del> load("@io_bazel_rules_go//go:def.bzl", "go_library") </del> <ins> load("@io_bazel_rules_go//go:def.bzl", "go_library", "go_test") </ins> go_library( name = "go_default_library",

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Better testing of monotonic time in Why is this needed:
/sig api-machinery
/triage accepted
The label cannot be applied. Only GitHub organization members can add the label. <detailsIn response to : /triage accepted </details
/triage accepted


Question: e2enode "k8s.io/kubernetes/test/e2e/framework/node" e2epod "k8s.io/kubernetes/test/e2e/framework/pod" e2eskipper "k8s.io/kubernetes/test/e2e/framework/skipper" <del> e2etestfiles "k8s.io/kubernetes/test/e2e/framework/testfiles" testutils "k8s.io/kubernetes/test/utils" </del> "github.com/onsi/ginkgo/v2" "github.com/onsi/gomega" <del> "github.com/onsi/gomega/gcustom" "github.com/onsi/gomega/types" </del> ) const (

Failing: ` Failing since . Last passing: First failing:
The diff () shows that went in which looks related
/sig node
/kind regression /kind bug /triage accepted


Question: nodeName := types.NodeName(a.plugin.host.GetHostName()) diskName := volumeSource.DiskName <del> glog.V(2).Infof("azureDisk - WaitForAttach: begin to GetDiskLun by diskName(%s), DataDiskURI(%s), nodeName(%s), devicePath(%s)", diskName, volumeSource.DataDiskURI, nodeName, devicePath) lun, err := diskController.GetDiskLun(diskName, volumeSource.DataDiskURI, nodeName) if err != nil { return "", err </del> <ins> var lun int32 if runtime.GOOS == "windows" { glog.V(2).Infof("azureDisk - WaitForAttach: begin to GetDiskLun by diskName(%s), DataDiskURI(%s), nodeName(%s), devicePath(%s)", diskName, volumeSource.DataDiskURI, nodeName, devicePath) lun, err = diskController.GetDiskLun(diskName, volumeSource.DataDiskURI, nodeName) if err != nil { return "", err } glog.V(2).Infof("azureDisk - WaitForAttach: GetDiskLun succeeded, got lun(%v)", lun) } else { lun, err = getDiskLUN(devicePath) if err != nil { return "", err } </ins> } <del> glog.V(2).Infof("azureDisk - WaitForAttach: GetDiskLun succeeded, got lun(%v)", lun) </del> exec := a.plugin.host.GetExec(a.plugin.GetPluginName()) io := &osIOHandler{}

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: There is a drawback after PR: following code will fetch vmCache may lead to below error and the vmCache refresh time period would be 1 min, and may cost 1 min which is too long time How to fix this At least in Linux, we may still use parameter to get LUN, which is almost zero time. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): v1.12 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: /sig azure /assign
k8s 1.10.4, Azure (deployed via ACS-engine) do you know about any directly applicable workaround? This is hitting us quite hard. logs: logs: The last Attach operation reports success, yet we enter the same error loop over and over again.
thanks for reporting, I am fixing this issue, code should be ready in this week.
We have seen what looks like this issue - possible regression? It took 11m for the volume to attach.
it's more likely that disk could not be detected under dir inside vmss instance, is it a transient issue? Can this issue repro easily?
Seems to happen consistently on the 2nd pod of a new deploy. We have seen 3-4 times in a row deleting and re-deploying this project. Doesn't seem to be an issue for the kafka-0 and kafka-2 pods, just kafka-1. And only on a fresh deploy of the helm chart, deleting the pod does not reproduce.
what's your region and k8s version? it only happened on vmss, right?
The version we’re running is k8s 1.13.12 in useast2. And right this is on vmss. EDIT: I should also note that we're also using multi fault zones for each of the vmss pools we're deploying.
I'm having a similar issue, but I'm on 1.14.8.

I don't know if is related, but these disk attachments seem to not be working properly on aks with vmms.
confirm volume attach with vmss is failing bad..
pls open a separate issue, provide detailed error info, e.g. k8s version, pod error description, etc. thanks.
This took too long AKS - kubernetes version -1.14.8 location -west europe
thanks for the reporting, I could confirm that the is due to PR: , which is going to mitigate vmss API throttling issue, in that PR, it enlarge the getVMSS TTL from 1min to 10min. Fortunately, from k8s 1.15.0, won't have this issue, since we have refactored the disk detect logic, it won't depend on getVMSS operation to get the disk LUN. In brief, pls upgrade to latest version 1.15.5, 1.15.x won't have this issue, thanks.
cherry-pick PR: to fix this issue on 1.14
You mention a 1.14 cherry pick but I'm also seeing Is this correct to assume 1.13 will have this fix as well ?
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : <!-- This form is for bug reports and feature requests ONLY! --/kind bug Uncomment only one, leave it on its own line: /kind bug /kind feature What happened: There is a drawback after PR: following code will fetch vmCache may lead to below error and the vmCache refresh time period would be 1 min, and may cost 1 min which is too long time At least in Linux, we may still use parameter to get LUN, which is almost zero time. Anything else we need to know?: Kubernetes version (use ): v1.12 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.

</details
1.13 don't accept any cherry-pick, so only 1.14 could be cherry-picked now, target fixed version is 1.14.10
the delay still can be very big this is because azure api call for disk attach returns after at 35s-2min and kubelet goes to its exp. backoff retries so it can reach 2min timeout adding attached recording: called 3 disk simultaneous attach with interval ~2s and saw and So actual attachment takes ~10-30s (device appears in /dev/disk/azure/scsi1) , but due to azure api we can loop on it I saw also that disk attach timing is unstable and somtemes it can take 2.5min , what do you think about logic like below? is not waiting for attachdisk to return, instead it calls GET until it returns lun id (we saw it takes few seconds after attachdisk). after GET returns lun id , it updates node's disks with lunId and kubelet can continue with mount after waiting for /dev/disk/azure/scsi1/lun*
same here
It takes about 10 minutes to recover from the FailedMount messages.
what's your k8s version?
there is a new vhd disk feature based on azure file which could attach/detach disk < 1s, consider that as an option if you have concern about disk attach/detach time cost:
AKS v1.14.8
for this specific issue, aks 1.15.x won't have this issue.
hi ,I am not dare to upgrade.because my stateful rabbitmq with pvc can not started because of failmount of PVC!
you could scale your statefulset to 0 replica, and pvc disk would be detached. After upgrade, scale back to original count.
okay,let me have a try.
I use helm to can I restore to two replicas after upgrading k8s version?


Question: } pod := api.Pod{ JSONBase: api.JSONBase{ <del> ID: fmt.Sprintf("%x", rand.Int()), </del> <ins> ID: fmt.Sprintf("%08x", rand.Uint32()), </ins> }, DesiredState: controllerSpec.DesiredState.PodTemplate.DesiredState, Labels:       controllerSpec.DesiredState.PodTemplate.Labels,

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: Which test(s) are failing: Since when has it been failing: Testgrid link: Reason for failure: Anything else we need to know: /sig cluster-lifecycle /cc /priority critical-urgent /milestone v1.19


Question: KUBECTL_EXTERNAL_DIFF environment variable can be used to select your own diff command. By default, the "diff" command available in your path will be run with "-u" (unified diff) and "-N" (treat absent files as empty) options. Exit status: 0 No differences were found.

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: fails. What you expected to happen: I would expect the PatchOptions to not include FieldManager. The following fix will eliminate the problem. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): 1.16 and 1.17 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig cli
cc
/wg apply
cc
I agree, we clearly missed that. Would you mind sending the fix in the form of a PR? Also please include a test, because that shouldn't have happened if I had tested my code properly :-(
Clearly we should also cherry-pick that into 1.17. I'll send the PR on Monday if you prefer! Thanks.
Ok, I will send a PR. However, I can't include a test immediately because I don't understand how the test works.
Thanks, I’ll try to show you how to get started!
Any update we clearly need this promptly since code-freeze is coming! Thanks.
I'll send a PR on Monday if you haven't! Thanks.
Sorry, I haven't started working on this test. Thanks.


Question: return nil, err } <del> i := sort.Search(len(volumes), func(i int) bool { return matchPredicate(pv, volumes[i]) }) if i < len(volumes) { return volumes[i], nil </del> <ins> // volumes are sorted by size but some may be bound. // remove bound volumes for easy binary search by size unboundVolumes := []*api.PersistentVolume{} for _, v := range volumes { if v.Spec.ClaimRef == nil { unboundVolumes = append(unboundVolumes, v) } } i := sort.Search(len(unboundVolumes), func(i int) bool { return matchPredicate(pv, unboundVolumes[i]) }) if i < len(unboundVolumes) { return unboundVolumes[i], nil </ins> } return nil, nil }

E0513 22:55:45. 957 ] unable to sync kubernetes RO service: endpoints "kubernetes-ro" cannot be updated: the resource was updated to 266 E0513 22:55:45. 957 ] unable to sync kubernetes RO service: endpoints "kubernetes-ro" cannot be updated: the resource was updated to 266 E0513 22:55:48. 957 ] watch of api.Secret ended with: Stop requested E0513 22:55:48. 957 ] watch of api.ServiceAccount ended with: Stop requested E0513 22:55:48. 957 ] watch of api.Namespace ended with: Stop requested E0513 22:55:48. 957 ] watch of api.ServiceAccount ended with: Stop requested E0513 22:55:48. 957 ] watch of api.Secret ended with: Stop requested E0513 22:55:48. 957 ] watch of api.ServiceAccount ended with: Stop requested W0513 22:55:49. 957 ] Resetting endpoints for master service "kubernetes" to &{{ } {kubernetes default 0001-01-01 00:00:00 +0000 UTC <nilmap[] map[]} [{[{10.0.0.22 <nil}] [{ 6443 TCP}]}]} W0513 22:55:50. 957 ] Resetting endpoints for master service "kubernetes-ro" to &{{ } {kubernetes-ro default 0001-01-01 00:00:00 +0000 UTC <nilmap[] map[]} [{[{10.0.0.22 <nil}] [{ 7080 TCP}]}]} W0513 22:55:50. 957 ] Resetting endpoints for master service "kubernetes-ro" to &{{ } {kubernetes-ro default 0001-01-01 00:00:00 +0000 UTC <nilmap[] map[]} [{[{10.0.0.22 <nil}] [{ 7080 TCP}]}]} W0513 22:55:50.

957 ] Resetting endpoints for master service "kubernetes" to &{{ } {kubernetes default 0001-01-01 00:00:00 +0000 UTC <nilmap[] map[]} [{[{10.0.0.22 <nil}] [{ 6443 TCP}]}]} W0513 22:55:50. 957 ] Resetting endpoints for master service "kubernetes-ro" to &{{ } {kubernetes-ro default 0001-01-01 00:00:00 +0000 UTC <nilmap[] map[]} [{[{10.0.0.22 <nil}] [{ 7080 TCP}]}]} W0513 22:55:50. 957 ] Resetting endpoints for master service "kubernetes-ro" to &{{ } {kubernetes-ro default 0001-01-01 00:00:00 +0000 UTC <nilmap[] map[]} [{[{10.0.0.22 <nil}] [{ 7080 TCP}]}]} W0513 22:55:50. 957 ] Resetting endpoints for master service "kubernetes-ro" to &{{ } {kubernetes-ro default 0001-01-01 00:00:00 +0000 UTC <nilmap[] map[]} [{[{10.0.0.22 <nil}] [{ 7080 TCP}]}]} E0513 22:55:50. 957 ] unable to sync kubernetes service: endpoints "kubernetes" cannot be updated: the resource was updated to 279 E0513 22:55:50. 957 ] unable to sync kubernetes RO service: endpoints "kubernetes-ro" cannot be updated: the resource was updated to 280 E0513 22:55:50. 957 ] unable to sync kubernetes RO service: endpoints "kubernetes-ro" cannot be updated: the resource was updated to 280 E0513 22:55:50. 957 ] unable to sync kubernetes RO service: endpoints "kubernetes-ro" cannot be updated: the resource was updated to 280 E0513 22:55:50. 957 ] unable to sync kubernetes RO service: endpoints "kubernetes-ro" cannot be updated: the resource was updated to 280 W0513 22:55:51. 957 ] Portal net unspecified.

Defaulting to 10.0.0.0/24. I0513 22:55:51. 957 ] Will report 10.0.0.22 as public IP address. E0513 22:56:00. 957 ] Unable to authenticate the request due to an error: Token has been invalidated E0513 22:56:00. 957 ] Unable to authenticate the request due to an error: Token has been invalidated E0513 22:56:00. 957 ] Unable to authenticate the request due to an error: Token has been invalidated E0513 22:56:01. 957 ] Unable to authenticate the request due to an error: Token has been invalidated E0513 22:56:08. 957 ] watch of api.Secret ended with: Stop requested E0513 22:56:08. 957 ] watch of api.ServiceAccount ended with: Stop requested E0513 22:56:08. 957 ] watch of api.Namespace ended with: Stop requested E0513 22:56:08. 957 ] watch of api.ServiceAccount ended with: Stop requested E0513 22:56:08. 957 ] watch of api.Secret ended with: Stop requested E0513 22:56:08. 957 ] watch of api.ServiceAccount ended with: Stop requested FAIL FAIL 111.315s
Do we know which test is failing? The last messages in the log are from , but I don't actually see a test failure message printed, and I can't recreate any failures.

The "Unable to authenticate the request due to an error: Token has been invalidated" messages are expected since the test exercises attempting to use invalid tokens () Also, the "Stop requested" messages indicate that particular test completed and invoked the deferred stopFunc, which shuts down the controllers and closes the test server
Here is the failing test: --- FAIL: TestPersistentVolumeClaimBinder (3.45s) Expected claim to be bound: &{TypeMeta:{Kind: APIVersion:} ObjectMeta:{Name:claim02 GenerateName: Namespace:default SelfLink:/api/v1beta1/persistentVolumeClaims/claim02?namespace=default UID:-f9c3-11e4-96ec- ResourceVersion:172 CreationTimestamp:2015-05-13 22:54:47 +0000 UTC DeletionTimestamp:<nilLabels:map[] Annotations:map[]} Spec:{AccessModes:[ReadOnlyMany ReadWriteOnce ReadWriteMany] Resources:{Limits:map[] Requests:map[storage:{Amount: Format:BinarySI}]} VolumeName:} Status:{Phase:Pending AccessModes:[] Capacity:map[]}}
you aware of the issue with TestPersistentVolumeClaimBinder?
I ran into above issue a lot.
Sorry, hadn't seen this sooner. Lost in the flurry of email traffic. I am looking into it now.
I cannot reproduce on F20, Go 1.3.1 I'll try 1.4 next. Any other info would be appreciated.
Here is the output of docker version from my desktop. Client version: 1.6.2 Client API version: 1.18 Go version (client): go1.4.2 Git commit (client): OS/Arch (client): linux/amd64 Server version: 1.6.2 Server API version: 1.18 Go version (server): go1.4.2 Git commit (server): OS/Arch (server): linux/amd64 I started to run into this issue two days ago, and just ran into it half hour ago right after I sync-ed to HEAD again. Any more information you need here?
Thanks :)
I cannot reproduce this on F20, Go 1.4.2 Any runtime settings I might be missing? I see hack/test- is set to "-race" .


Question: return fmt.Errorf("%v: %v", s.url, resp.Status) } if len(data) == 0 { <ins> // Emit an update with an empty PodList to allow HTTPSource to be marked as seen s.updates <- kubelet.PodUpdate{[]api.BoundPod{}, kubelet.SET, kubelet.HTTPSource} </ins> return fmt.Errorf("zero-length data received from %v", s.url) } // Short circuit if the manifest has not changed since the last time it was read.

To reproduce: Start a cluster via hack/local-up- Start some pods Kill the cluster "docker ps -a" should show the pod containers running still Start the cluster again "etcdctl ls /registry" should show that there is no boundpods directory -- only minions and services "docker ps -a" should show the old pod containers still running Start a new pod After, say, 10s, "docker ps -a" should show the old containers dead /cc
I suspect the cause of this might be : which guarantees that the pod sync logic is never called if is nil. Need to get a repro to confirm.


Question: Write a JSON pod description to a local file: ```bash <del> $ cat <<EOPOD >nginx.json { "kind": "Pod", "apiVersion": "v1beta1", "id": "nginx-id-01", "desiredState": { "manifest": { "version": "v1beta1", "containers": [{ "name": "nginx-01", "image": "nginx", "ports": [{ "containerPort": 80, "hostPort": 31000 }], "livenessProbe": { "enabled": true, "type": "http", "initialDelaySeconds": 30, "httpGet": { "path": "/index.html", "port": "8081" } } }] } }, "labels": { "name": "foo" } } </del> <ins> $ cat <<EOPOD >nginx.yaml apiVersion: v1 kind: Pod metadata: name: nginx spec: containers: - name: nginx image: nginx ports: - containerPort: 80 </ins> EOPOD ``` Send the pod description to Kubernetes using the `kubectl` CLI: ```bash <del> $ bin/kubectl create -f nginx.json nginx-id-01 </del> <ins> $ kubectl create -f nginx.yaml pods/nginx </ins> ``` Wait a minute or two while `dockerd` downloads the image layers from the internet. We can use the `kubectl` interface to monitor the status of our pod: ```bash <del> $ bin/kubectl get pods POD          IP           CONTAINER(S)  IMAGE(S)          HOST                       LABELS                STATUS nginx-id-01  172.17.5.27  nginx-01      nginx             10.72.72.178/10.72.72.178  cluster=gce,name=foo  Running </del> <ins> $ kubectl get pods NAME      READY     REASON    RESTARTS   AGE nginx     1/1       Running   0          14s </ins> ``` Verify that the pod task is running in the Mesos web GUI. Click on the Kubernetes framework. The next screen should show the running Mesos task that started the Kubernetes pod. <del> ## Run the Example Guestbook App Following the instructions from the kubernetes-mesos [examples/guestbook][6]: ```bash $ export ex=k8sm/examples/guestbook $ bin/kubectl create -f $ex/redis-master.json $ bin/kubectl create -f $ex/redis-master-service.json $ bin/kubectl create -f $ex/redis-slave-controller.json $ bin/kubectl create -f $ex/redis-slave-service.json $ bin/kubectl create -f $ex/frontend-controller.json $ cat <<EOS >/tmp/frontend-service { "id": "frontend", "kind": "Service", "apiVersion": "v1beta1", "port": 9998, "selector": { "name": "frontend" }, "publicIPs": [ "${servicehost}" ] } EOS $ bin/kubectl create -f /tmp/frontend-service ``` </del> <ins> ## What next? </ins> <del> Watch your pods transition from `Pending` to `Running`: </del> <ins> Try out some of the standard [Kubernetes examples][9]. </ins> <del> ```bash $ watch 'bin/kubectl get pods' ``` </del> <ins> **NOTE:** Some examples require Kubernetes DNS to be installed on the cluster. Future work will add instructions to this guide to enable support for Kubernetes DNS. </ins> <del> Review your Mesos cluster's tasks: ```bash $ mesos ps TIME   STATE    RSS     CPU    %MEM  COMMAND  USER                   ID 0:00:05    R    41.25 MB  0.5   64.45    none   root  0597e78b-d826-11e4-9162-42010acb46e2 0:00:08    R    41.58 MB  0.5   64.97    none   root  0595b321-d826-11e4-9162-42010acb46e2 0:00:10    R    41.93 MB  0.75  65.51    none   root  ff8fff87-d825-11e4-9162-42010acb46e2 0:00:10    R    41.93 MB  0.75  65.51    none   root  0597fa32-d826-11e4-9162-42010acb46e2 0:00:05    R    41.25 MB  0.5   64.45    none   root  ff8e01f9-d825-11e4-9162-42010acb46e2 0:00:10    R    41.93 MB  0.75  65.51    none   root  fa1da063-d825-11e4-9162-42010acb46e2 0:00:08    R    41.58 MB  0.5   64.97    none   root  b9b2e0b2-d825-11e4-9162-42010acb46e2 ``` The number of Kubernetes pods listed earlier (from `bin/kubectl get pods`) should equal to the number active Mesos tasks listed the previous listing (`mesos ps`). Next, determine the internal IP address of the front end [service][7]: ```bash $ bin/kubectl get services NAME            LABELS                                    SELECTOR            IP             PORT kubernetes      component=apiserver,provider=kubernetes   <none>              10.10.10.2     443 redismaster     <none>                                    name=redis-master   10.10.10.49    10000 redisslave      name=redisslave                           name=redisslave     10.10.10.109   10001 frontend        <none>                                    name=frontend       10.10.10.149   9998 ``` Interact with the frontend application via curl using the front-end service IP address from above: ```bash $ curl http://${frontend_service_ip_address}:9998/index.php?cmd=get&key=messages {"data": ""} ``` Or via the Redis CLI: ```bash $ sudo apt-get install redis-tools $ redis-cli -h ${redis_master_service_ip_address} -p 10000 10.233.254.108:10000> dump messages "x00x06,worldx06x00xc9x82x8eHjxe5xd1x12" ``` #### Test Guestbook App Or interact with the frontend application via your browser, in 2 steps: First, open the firewall on the master machine. ```bash # determine the internal port for the frontend service $ sudo iptables-save|grep -e frontend  # -- port 36336 in this case -A KUBE-PORTALS-CONTAINER -d 10.10.10.149/32 -p tcp -m comment --comment frontend -m tcp --dport 9998 -j DNAT --to-destination 10.22.183.23:36336 -A KUBE-PORTALS-CONTAINER -d 10.22.183.23/32 -p tcp -m comment --comment frontend -m tcp --dport 9998 -j DNAT --to-destination 10.22.183.23:36336 -A KUBE-PORTALS-HOST -d 10.10.10.149/32 -p tcp -m comment --comment frontend -m tcp --dport 9998 -j DNAT --to-destination 10.22.183.23:36336 -A KUBE-PORTALS-HOST -d 10.22.183.23/32 -p tcp -m comment --comment frontend -m tcp --dport 9998 -j DNAT --to-destination 10.22.183.23:36336 # open up access to the internal port for the frontend service $ sudo iptables -A INPUT -i eth0 -p tcp -m state --state NEW,ESTABLISHED -m tcp  --dport ${internal_frontend_service_port} -j ACCEPT ``` Next, add a firewall rule in the Google Cloud Platform Console. Choose Compute > Compute Engine > Networks, click on the name of your mesosphere-* network, then click "New firewall rule" and allow access to TCP port 9998. ![Google Cloud Platform firewall configuration][8] Now, you can visit the guestbook in your browser! ![Kubernetes Guestbook app running on Mesos][9] </del> <ins> **NOTE:** Please be aware that there are [known issues with the current Kubernetes-Mesos implementation][7]. </ins> [1]: http://mesosphere.com/docs/tutorials/run-hadoop-on-mesos-using-installer [2]: http://mesosphere.com/docs/tutorials/run-spark-on-mesos [3]: http://mesosphere.com/docs/tutorials/run-chronos-on-mesos <del> [4]: http://cloud.google.com [5]: https://cloud.google.com/compute/ [6]: https://github.com/mesosphere/kubernetes-mesos/tree/v0.4.0/examples/guestbook [7]: https://github.com/GoogleCloudPlatform/kubernetes/blob/v0.11.0/docs/services.md#ips-and-vips [8]: mesos/k8s-firewall.png [9]: mesos/k8s-guestbook.png [10]: http://mesos.apache.org/ [11]: https://github.com/mesosphere/kubernetes-mesos/blob/master/docs/issues.md [12]: https://github.com/mesosphere/kubernetes-mesos/issues </del> <ins> [5]: http://open.mesosphere.com/getting-started/cloud/google/mesosphere/ [6]: http://mesos.apache.org/ [7]: https://github.com/mesosphere/kubernetes-mesos/blob/master/docs/issues.md [8]: https://github.com/mesosphere/kubernetes-mesos/issues [9]: ../../examples/ [10]: http://open.mesosphere.com/getting-started/cloud/google/mesosphere/#vpn-setup </ins> [![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/getting-started-guides/mesos.md?pixel)]()

Hi We are updating docs to v1beta3. v1beta1 and v1beta2 will be abandoned soon. For the examples in getting-started-guides, we tend to use hyperlinks to point to examples instead of letting them have their own individual examples (see ). However, in the mesos guide, the examples contain mesos specific setup and knowledge, so I can't simply remove the texts and point to kubernetes/examples/guestbook, which is updated to v1beta3. My suggestion is that we update the guestbook example in kubernetes-mesos, and use a hyperlink in mesos getting-started-guide to replace the example texts. Could you help to make the changes? Thanks.
cc
We're going to delete v1beta1 and v1beta2 in O(days). See PR .
cc
we've initiated an effort to upstream some components into k8s. as part of that effort, we're upgrading the guestbook examples for k8s-mesos: the getting started guide for mesos will likely have to change in order to align with the move upstream. tracking this issue here: , Brian Grant wrote:
xref , James DeFelice wrote:
Ref
cc
xref
This needs to be moved to the v1 API.
Agreed. We have changes in review right now, should be merged soon. , Brian Grant wrote:


Question: ### Setup <del> By default, the Vagrant setup will create a single master VM (called kubernetes-master) and one node (called kubernetes-node-1). Each VM will take 1 GB, so make sure you have at least 2GB to 4GB of free memory (plus appropriate free disk space). To start your local cluster, open a shell and run: </del> <ins> Setting up a cluster is as simple as running: ```sh export KUBERNETES_PROVIDER=vagrant curl -sS https://get.k8s.io | bash ``` Alternatively, you can download [Kubernetes release](https://github.com/kubernetes/kubernetes/releases) and extract the archive. To start your local cluster, open a shell and run: </ins> ```sh cd kubernetes

I cannot find such a service when . Same thing applies for . My cluster runs fine. cc:
Ah, good catch... everything is run as a container now, will update the documentation on where to get the logs.


Question: csiAttacher := attacher.(*csiAttacher) <ins> // FIXME: We need to ensure this goroutine exits in the test. </ins> go func(spec *volume.Spec, nodename string, fail bool) { attachID, err := csiAttacher.Attach(spec, types.NodeName(nodename)) if !fail && err != nil {

Which test(s) are flaking: TestAttacherWithCSIDriver Reason for failure: This is flaking rarely enough that it is not caught by our CI jobs, which currently tolerate up to 2 unit test failures per run (!). With that toleration removed in , this flake has been seen (). Reproducible with the following steps: a test binary for the affected package: the affected test using the tool (setting the timeout to a generous timeout relative to the normal test runtime of .02 seconds): failures are seen, look at the logs to see the details about the error: /sig storage cc
/assign can you help investigate this flake?
I have found the root cause. /assign
filed to address this issue.


Question: Name: "bar", }, }, <ins> InitContainers: []api.Container{ { Name: "initfoo", }, }, </ins> }, Status: api.PodStatus{ Phase: api.PodRunning,

Related issue
is there a simple example that you used to recreate the problem?
I used exactly the example below: So Pod sflask-w-init gets into pending state "Init:0/1" forever. Then you can try to run and you can see that is not working while invoking corresponding command with docker works - it displays current date every second.
thanks for the detailed test, please see the PR :)
Duh! already has a PR


Question: } func (prometheusMetricsProvider) NewLastResourceVersionMetric(name string) cache.GaugeMetric { <del> rv := prometheus.NewGauge(prometheus.GaugeOpts{ Subsystem: name, Name:      "last_resource_version", Help:      "last resource version seen for the reflectors", }) prometheus.MustRegister(rv) return rv </del> <ins> return lastResourceVersion.WithLabelValues(name) </ins> }

I'm seeing a large number of unique prometheus series showing up (due to the logging we for anonymous reflectors). This is a small cluster but I believe this may have unbounded cardinality.
we storing the resource version in the metric name?
cc
Came into the same problem, checked the code, seems we use + + as metric names, for most reflector, they use a default name provided by which is go filename. I think the proper way is use as the metric name, and add a label, like
opened a pr to fix this, ptal thanks :)


Question: if err != nil { framework.Failf("Failed to get nodes: %v", err) } <del> TestContext = Context{NodeMapper: &NodeMapper{}, VSphereInstances: vsphereInstances} </del> <ins> TestContext = Context{NodeMapper: NewNodeMapper(), VSphereInstances: vsphereInstances} </ins> // 3. Get Node to VSphere mapping err = TestContext.NodeMapper.GenerateNodeMap(vsphereInstances, *nodeList) if err != nil {

Command run: vSphere intree volume test suite occasionally failed with the following error: The test suite should always pass. We may have a chance to hit this by running ` No response binary version: 1.27.4 </details<detailsvsphere </details<details</details<details</details<details</details<details</details
/assign
/sig testing storage ?
/triage accepted


Question: CreationTimestamp: metav1.Date(2012, 1, 1, 0, 0, 0, 0, time.UTC), }, Status: v1.NodeStatus{ <del> NodeInfo: v1.NodeSystemInfo{ KubeletVersion: "v1.2.0", }, </del> Conditions: []v1.NodeCondition{ { Type:   v1.NodeReady,

As discussed on slack with we might want to clean the node controller from code that ensures backward compatibility for version 1.4 and earlier ones. /sig node
/assign
- how do we plan to devide this one ? The following order for merge should be fine I think: the tests (we can split also test case deletion if it's too long to review): logic itself + kubeletVersion on the according tests: change how we create nodecontroller: clean consts and BUILD:
Yeah, sounds good.
I'll create the PRs accordingly and assign them to you


Question: } // ListenAndServeReadOnly runs the kubelet HTTP server in read-only mode. <del> func (kl *Kubelet) ListenAndServeReadOnly(address net.IP, port uint) { server.ListenAndServeKubeletReadOnlyServer(kl, kl.resourceAnalyzer, address, port) </del> <ins> func (kl *Kubelet) ListenAndServeReadOnly(address net.IP, port uint, tp trace.TracerProvider) { server.ListenAndServeKubeletReadOnlyServer(kl, kl.resourceAnalyzer, address, port, tp) </ins> } // ListenAndServePodResources runs the kubelet podresources grpc service

Follow-up to Discussed in As part of kubelet tracing beta, we should add tracing support to the kubelet's read-only endpoint, which currently does not have tracing. It should use WithPublicEndpoint to prevent callers from influencing sampling decisions. Part of
/sig node /sig instrumentation
/triage accepted /assign
can i help you with this issue?
I don't think we'll be able to start with this until merges
Ok, I'm waiting to see if I can help you. Thank you
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale is merged now, so this can be worked on.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
Could I help with this issue?
Sure. See We pass a NoOp Tracer provider to disable tracing for the endpoint. Instead, we need to pass the tracerprovider, but make sure that the eventual uses withPublicEndpoint. Looking closer, i'm not sure its currently possible to use WithPublicEndpoint with the otelrestful package... doesn't contain that option (unlike otelhttp: ), so we may need to add that upstream first.

Another thing to keep in mind is that updating otel libraries is currently blocked by (see my attempt in )
I gave the whole thing another try. At least locally, the dependencies seem now to be able to be updated. :raccoon: Could you maybe have a look at it :)
He's on leave for the time being.
oh, can you provide some feedback or point me to the right people?
I'm back now. Feel free to assign me to PRs related to this, and I'll review
Welcome back :) Could you re-run the CI ? Its just the dependencies. The PR itself shouldn't be that much afterwards.
otel is now up-to-date enough. here:
Hi, I would like to ask about one trouble that caused by WithPublicEndpoint. Aggregated APIServer that I customized uses kubernetes's RecommendedConfig. Due to the fact that WithPublicEndpoint is enabled, it can't be tracked by the full trace for its span-link model. And I think child-parent span model is more appropriate. So, could you please give me some advice? Looking forward to your reply.
thanks. We know about this, and are tracking it in


Question: }, }, requestInfo: &request.RequestInfo{ <ins> Verb:              "create", </ins> APIGroup:          "", APIVersion:        "v1", Resource:          "pods",

I've observed, that some metrics has and labels. (create) requests are "single resource", therefor they should be classified as . Def: customresourcehandler uses , which is based on the RequstInfo only: the RequestInfo is based only on the requests: method and url example: e.g: Since this RequestInfo has Name and Namespace empty, it is classified as . In a similar way it could be explained why: There's a mismatch in scope classification between built-in and custom resources: The key difference - the check if either "name" was provided OR it is a : Where the metrics.CleanScope ONLY checks, if name is not-empty. Tested: = v1.20 There were other scope related issues <1.20 and the distinction is not obvious. <details</details<details</details<details</details<details</details<details</details
/sig api-machinery
/triage accepted


Question: // Error returns detailed information of why the pod failed to fit on each node. // A message format is "0/X nodes are available: <PreFilterMsg>. <FilterMsg>. <PostFilterMsg>." func (f *FitError) Error() string { <del> reasons := make(map[string]int) for _, status := range f.Diagnosis.NodeToStatusMap { for _, reason := range status.Reasons() { reasons[reason]++ } } </del> reasonMsg := fmt.Sprintf(NoNodeAvailableMsg+":", f.NumAllNodes) <del> // Add the messages from PreFilter plugins to reasonMsg. </del> preFilterMsg := f.Diagnosis.PreFilterMsg if preFilterMsg != "" { <ins> // PreFilter plugin returns unschedulable. // Add the messages from PreFilter plugins to reasonMsg. </ins> reasonMsg += fmt.Sprintf(SeparatorFormat, preFilterMsg) } <del> sortReasonsHistogram := func() []string { var reasonStrings []string for k, v := range reasons { reasonStrings = append(reasonStrings, fmt.Sprintf("%v %v", v, k)) </del> <ins> if preFilterMsg == "" { // the scheduling cycle went through PreFilter extension point successfully. // // When the prefilter plugin returns unschedulable, // the scheduling framework inserts the same unschedulable status to all nodes in NodeToStatusMap. // So, we shouldn't add the message from NodeToStatusMap when the PreFilter failed. // Otherwise, we will have duplicated reasons in the error message. reasons := make(map[string]int) for _, status := range f.Diagnosis.NodeToStatusMap { for _, reason := range status.Reasons() { reasons[reason]++ } } sortReasonsHistogram := func() []string { var reasonStrings []string for k, v := range reasons { reasonStrings = append(reasonStrings, fmt.Sprintf("%v %v", v, k)) } sort.Strings(reasonStrings) return reasonStrings } sortedFilterMsg := sortReasonsHistogram() if len(sortedFilterMsg) != 0 { reasonMsg += fmt.Sprintf(SeparatorFormat, strings.Join(sortedFilterMsg, ", ")) </ins> } <del> sort.Strings(reasonStrings) return reasonStrings } sortedFilterMsg := sortReasonsHistogram() if len(sortedFilterMsg) != 0 { reasonMsg += fmt.Sprintf(SeparatorFormat, strings.Join(sortedFilterMsg, ", ")) </del> } // Add the messages from PostFilter plugins to reasonMsg. <ins> // We can add this message regardless of whether the scheduling cycle fails at PreFilter or Filter // since we may run PostFilter (if enabled) in both cases. </ins> postFilterMsg := f.Diagnosis.PostFilterMsg if postFilterMsg != "" { reasonMsg += fmt.Sprintf(SeparatorFormat, postFilterMsg)

When PreFilter returns UnschedulableAndUnresolvable, we don't need to run the preemption, but currently we run. This unexpected preemption could cause errors from the preemption by trying to read non-exist PreFilter data. As pointed out (), actually we didn't run the preemption in such cases in the past versions, but introduced this bug. /kind bug /kind regression /sig scheduling /triage accepted /priority important-soon
/assign
will fix this.


Question: log.Printf("Server version: %#vn", v) } <ins> for start := time.Now(); time.Since(start) < waitTimeout; time.Sleep(5 * time.Second) { eps := getWebserverEndpoints(client) if eps.Len() >= *peerCount { break } state.Logf("%v/%v has %v endpoints, which is less than %v as expected. Waiting for all endpoints to come up.", *namespace, *service, len(eps), *peerCount) } </ins> // Do this repeatedly, in case there's some propagation delay with getting // newly started pods into the endpoints list. for i := 0; i < 15; i++ { <del> endpoints, err := client.Endpoints(*namespace).Get(*service) if err != nil { state.Logf("Unable to read the endpoints for %v/%v: %v; will try again.", *namespace, *service, err) time.Sleep(time.Duration(1+rand.Intn(10)) * time.Second) } eps := sets.String{} for _, ss := range endpoints.Subsets { for _, a := range ss.Addresses { for _, p := range ss.Ports { eps.Insert(fmt.Sprintf("http://%s:%d", a.IP, p.Port)) } } } </del> <ins> eps := getWebserverEndpoints(client) </ins> for ep := range eps { state.Logf("Attempting to contact %s", ep) contactSingle(ep, state) } time.Sleep(5 * time.Second) } } <ins> //getWebserverEndpoints returns the webserver endpoints as a set of String, each in the format like "http://{ip}:{port}" func getWebserverEndpoints(client *client.Client) sets.String { endpoints, err := client.Endpoints(*namespace).Get(*service) eps := sets.String{} if err != nil { state.Logf("Unable to read the endpoints for %v/%v: %v.", *namespace, *service, err) return eps } for _, ss := range endpoints.Subsets { for _, a := range ss.Addresses { for _, p := range ss.Ports { eps.Insert(fmt.Sprintf("http://%s:%d", a.IP, p.Port)) } } } return eps } </ins> // contactSingle dials the address 'e' and tries to POST to its /write address. func contactSingle(e string, state *State) { body, err := json.Marshal(&WritePost{

<!-- Please only use this template for submitting enhancement requests --What would you like to be : The documentation suggests that "The default pull policy is IfNotPresent" This is not the case when imagePullPolicy is defined under the spec section in a Pod definition without a specific value. Example: This will result in a Pod having imagePullPolicy as Always. Why is this needed: This is needed for clarification of the default behaviour. It is necessary for the appropriate handling of helm charts, in cases that the imagePullPolicy is empty.
/sig docs
The imagePullPolicy default value is .
In this paragraph you can see that In your reference it points out the default action for the kubelet, not the behaviour when imagePullPolicy is declared. Please do have in mind that the imagePullPolicy option for Never is mentioned only in this paragraph in the document.
If an image tag of "latest" is used, the imagePullPolicy defaults to to "Always": If no tag and no digest are specified on the image, an image tag of "latest" is used: The behavior when no image digest or tag is specified should be documented in the API docs for the and fields:
I am interested on helping this issue, but I am new to kubernete repo, it could take me sometime
I found the following which is well documented in "Best Practices for Configuration" It would be expected to include more information on the options in the main link under
I'd like to work on this one.
Very welcome, but yesterday also mentioned that he is following up on this issue, you can communicate with him before starting

cc
This has failed The two failures are different, and both seemingly totally legitimate. The first one is simply a pod never entering running state within the 5 minute timeout. Expected error: <*errors.errorString | 0xc208475150: { s: "gave up waiting for pod 'nettest-dtwk3' to be 'running' after 5m0s", } gave up waiting for pod 'nettest-dtwk3' to be 'running' after 5m0s not to have occurred The second is not being reachable: Dec 7 01:22:47.275: Unable to connect/talk to the internet: Get dial tcp 74.125.202.147:80: i/o timeout The latter could be a result of a missing MASQ rule on the node, or GCP egress problems. I don't think that this is worth trying to debug further in the context of this test for intra-pod communication. I'll check more generally whether we have a current intermittent problems with respect to pods entering running state, or reaching the internet.
I think this test may be causing builds to time out, too. See comments from me in .
For the future, I actually don't think we need the test at all in the networking test. Maybe even move it do a different test altogether.
This is clearly a bug. I'll send a PR shortly: Regarding the ping to , why did you put in in the first place?
hmmm, not sure what you mean by "in the first place" ? I did not write the test...
sorry, I thought that you'd written that part of the test. I will find out who wrote that, figure out why they put it there in the first place, and then either remove it, or rewrite it to work correctly. Clearly that part of the test needs at least some basic retry to work consistently.
ok great, no problem just wasnt sure what you meant. SGTM. Id be okay w/ deleting the ping entirely regardless of who wrote it, its a very old snippet which doesnt really add much to the overall test suite (and in fact is questionable wether it should even be an assertion at all )
somebody put it there for some reason. Better to understand why before just deleting it.

i looked, its actually rob who i work with :) i will ask him in the morning unless you find him first :) (rrati)
Regarding my alleged "clear bug" above, I was completely wrong. There's nothing wring with that code. And unfortunately, when this test goes into a presumed endless loop (see comment above) it does not produce any output (as the output is cached by Ginkgo until the test completes). So it's very difficult to debug. But in summary I agree with this test is probably never completing, and causing tests to time out. Someone will need to isolate this test in a environment where it produces output and can be debugged. I'll keep trying, but need to head home soon, so if anyone wants to pick it up overnight, go ahead.
Smoking gun:
hmm - I'm just wondering why only this particular test is hanging forever (if we agree that's the case). I don' think that some requests are hanging forever in this test, but not in any other test...
Closing in favor of which should fix this. Or at least some way to set client timeout, which and are still figuring out.
It's happening again.
Talked to offline. I am assigning this one to you for triage. Let me know if you need any help.
AFAICT the last failure was just the pod nettest-7h5jj not coming up
Looks like this should be resolved by your pr: too. :-)
Maybe. Though the webserver only deadlocks in the failure case, so the test is probably still flaky...
Looks like this test is now explicitly flaking by failing, as I expected.
Based on the , it took 15 minutes to pull the image. closing this one as dup of
New Failure:
This looks like a race between populating the endpoints list and the /status, /read methods of the service exposed through the master proxy. It looks like the webserver pod is just trying to confirm that it can reach all peers. We can do this synchronously after having observed all expected endpoints in the test (or re-write the test to use the same pattern we do in services e2e).
The test looks really racy to me. I think the following 15 retries prevents it to fail more often, but not completely.

I will submit a pr to enfore webservers to wait till alll endpoints are up before they start to contact each other.
New failure
I am making this P0
It took more than 4 minutes for a nettest pod to come up. The nettest container is waiting for all endpoints to be ready for 2 minutes plus 15 attempts (5 sec interval) to contact its peers. This happens in less than 4 minutes. Hence the first 2 nettest pods finished contacting peers and declared failure.
The node you want is: I see: that's too much silence ...
From the It looks like kubelet is slow to start the container. Due to the lower logging level in the gke, it's hard to guess what could've slowed down the goroutine...