Question: // taking responsibility to ensure mutation of in is not exposed // back to the caller. in.Spec.InitContainers = values <ins> // Call defaulters explicitly until annotations are removed for i := range in.Spec.InitContainers { c := &in.Spec.InitContainers[i] SetDefaults_Container(c) } </ins> } if err := autoConvert_v1_PodTemplateSpec_To_api_PodTemplateSpec(in, out, s); err != nil {

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): No What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): initContainers imagePullPolicy initContainers required value Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Environment: Cloud provider or hardware configuration: GCE OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: cluster/kube- Others: What happened: When trying to create either a ReplicaSet or Replication Controller whose Pod template contains an Init Container, the validation fails saying is a required value. What you expected to happen: The create should succeed and the Init Container should receive a default . How to reproduce it (as minimally and precisely as possible): Try to create either of the following: ReplicaSet Replication Controller Anything else do we need to know: If I add to the Init Container in the ReplicaSet or Replication Controller Pod template, the create succeeds. When I create an Init Container without in a StatefulSet Pod template, it succeeds. When I create an Init Container without in a bare Pod, it succeeds:
Looks like defaulting not being applied on server? That is probably due to removing defaulting from convert. Surprised we didn't catch this earlier
Some values still not being defaulted
, I'm going to create a PR if none works on it.
We decode from bytes, set defaults and then convert; but for init containers, we delay the decode from "decode phase" to "convert phase" at .

The defaults has been handled for initContainer at . /cc I'm working to find a solution for that; if any suggestion, please let me know.
Would this bug affect the deletion of jobs with an init-container? I ran into this bug and adjusted my manifest to add the pull policy to the init-container. However, I now cannot delete said job claiming a field is immutable. Example:
, not sure now. Let me check it,
Same error here with version 1.5.1, 1.5.2 The Job "grafana-import-dashboards" is invalid: [0].imagePullPolicy: Required value With this setup: kubectl apply -f monitoring Not working even if I specify a pullpolicy in the yaml. If I roll back to 1.4.7, it is working. Thank you. Greg.
any updates on a PR to address this issue? Is anyone from able to pick this issue up otherwise?
, here's the latest status: there's is a PR to generate for each Object (), it works for Object (Pod, Job) now but List; I'm working to make it generating for List.
Why would you need List to work? No client side logic should be decoding lists except as a generic submission.
The List tests are failed in . There's no for List, so no code to decode init-container annotation, e.g. of should call 's recursively.
Thanks for the update
+1
Any update on this? I still getting this issue with the last version of 1.5.3 on GKE?
ping for the comments? should we merge the PR for 1.5?
, there's a PR () that handled that issue; but we're discussing not to merge it because initContainers had been moved to fields, and fields is the recommended way to use since 1.6. But I'd like to get & 's comments on 1.5.
This is still an issue in 1.5.3 when using annotations: Without the deployment would not be accepted
ping & :).
i dont think this is fixed in 1.5 I just encountered this bug on 1.5.3
, yes, the PR is NOT merged to any branch, so the issue is still here. I'll create a PR for 1.5.
Hi Anyone help this issue.

#cat mongo- apiVersion: v1 kind: Pod metadata: labels: name: mongo-svc-a name: mongo-deployment spec: replicas: 1 template: metadata: labels: name: mongo spec: containers: image: mongo name: mongo ports: name: mongo containerPort: hostPort: volumeMounts: mountPath: "/data/db" name: mongopv volumes: name: mongopv persistentVolumeClaim: claimName: mongopvc When i want create mongo db container by using this file folowing error is occured. The Pod "mongo-deployment" is invalid:spec.containers: Required value
, it seems there're two here, please remove the first one and have a try.
Hi my kubectl binary is configured at ubuntu 16.04 machine.

NAME LABELS STATUS 10.1 Ready,SchedulingDisabled 10.2 10.1.10.2 Ready 10.3 Ready apiVersion: v1 kind: Node metadata: creationTimestamp: 2017-03-03T14:41:52Z labels: 10.1.10.1 name: 10.1.10.1 resourceVersion: "" selfLink: /api/v1/nodes/10.1.10.1 uid: -001f-11e7-8e83- spec: externalID: 10.1.10.1 status: addresses: address: 10.1.10.1 type: LegacyHostIP address: 10.1.10.1 type: InternalIP capacity: cpu: "2" memory: 4041352Ki pods: "110" conditions: lastHeartbeatTime: 2017-04-05T18:58:42Z lastTransitionTime: 2017-03-03T14:41:52Z message: kubelet has sufficient disk space available reason: KubeletHasSufficientDisk status: "False" type: OutOfDisk lastHeartbeatTime: 2017-04-05T18:58:42Z lastTransitionTime: 2017-04-05T15:27:52Z message: kubelet is posting ready status reason: KubeletReady status: "True" type: Ready nodeInfo: bootID: -6124-4ac7-ab96- containerRuntimeVersion: docker://1.12.6 kernelVersion: 4.9.16-coreos-r1 kubeProxyVersion: v1.2.4+coreos.1 kubeletVersion: v1.2.4+coreos.1 machineID: osImage: Container Linux by CoreOS 1298.7.0 (Ladybug) systemUUID: C3FD9852-2DFB-21F4-CE07-2597A746D99D When i create pv for creation pod is shows pv is pending. Please give any suggestions.....
Is there solution of this problem with version 2.7. Still facing the same. what exactly expectation of helm upgrade --install relname chartsdirectory is is expected to work on first time installation in case with CI/CD Pipelines.


Question: @when('kube-control.connected') <ins> @when('cdk-addons.configured') </ins> def send_cluster_dns_detail(kube_control): ''' Send cluster DNS info ''' # Note that the DNS server doesn't necessarily exist at this point. We know

Kubernetes version (use ): 2 Environment: Cloud provider or hardware configuration: LXD OS (e.g. from /etc/os-release): Ubuntu 16.04 Kernel (e.g. ): 4.4.0-57-generic -Ubuntu Install tools: Juju Others: What happened: The kubernetes-master charm entered a state with the following stacktrace: The 10.152.183.10 IP was already claimed by default-http-backend: What you expected to happen: default-http-backend should not deploy until after kube-dns has had a chance to reserve its dedicated IP How to reproduce it (as minimally and precisely as possible): It'll be hard to reproduce this reliably since the default-http-backend service IP is chosen randomly. However, this might occur sporadically when deploying with juju.
It looks like the juju provider always deploys kube-dns to ..*.10 and the default-http-backend already claimed it. What are other providers doing? Are they pinning dns to an IP address and not never get a collision?
I don't think this needs to be fixed for 1.6. It's a pretty rare case and only applies to deployments with Juju.
removing the 1.6 milestone as per


Question: Short: i18n.T("Unsets an individual value in a kubeconfig file"), Long:  unset_long, Run: func(cmd *cobra.Command, args []string) { <del> cmdutil.CheckErr(options.complete(cmd)) cmdutil.CheckErr(options.run()) fmt.Fprintf(out, "Property %q unset.n", options.propertyName) </del> <ins> cmdutil.CheckErr(options.complete(cmd, args)) cmdutil.CheckErr(options.run(out)) </ins> }, } return cmd } <del> func (o unsetOptions) run() error { </del> <ins> func (o unsetOptions) run(out io.Writer) error { </ins> err := o.validate() if err != nil { return err

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT Kubernetes version (use ): 1.5.3 Environment: Cloud provider or hardware configuration: minikube 0.17.1 How to reproduce it (as minimally and precisely as possible): config Note that there's only one context. the property of a non-existent context config Note that a context named foo was created just because I tried to unset a property. What you expected to happen: should give an informative message saying that context doesn't exist. Anything else we need to know: This is probably low-hanging fruit.
/sig cli
/assign i will take a look for this
GitHub didn't allow me to assign the following users: a, look, for, this, i, will, take. Note that only can be assigned. <detailsIn response to : /assign i will take a look for this Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
i can reproduce this, this is a bug, have open PR fix this. thanks
/kind bug


Question: return lcm.capacity } <ins> func TestNodeStatusWithCloudProviderNodeIP(t *testing.T) { testKubelet := newTestKubelet(t, false /* controllerAttachDetachEnabled */) defer testKubelet.Cleanup() kubelet := testKubelet.kubelet kubelet.hostname = testKubeletHostname existingNode := v1.Node{ ObjectMeta: metav1.ObjectMeta{Name: testKubeletHostname, Annotations: make(map[string]string)}, Spec:       v1.NodeSpec{}, } // TODO : is it possible to mock kubelet.validateNodeIP() to avoid relying on the host interface addresses ? addrs, err := net.InterfaceAddrs() assert.NoError(t, err) for _, addr := range addrs { var ip net.IP switch v := addr.(type) { case *net.IPNet: ip = v.IP case *net.IPAddr: ip = v.IP } if ip != nil && !ip.IsLoopback() && ip.To4() != nil { kubelet.nodeIP = ip break } } assert.NotNil(t, kubelet.nodeIP) fakeCloud := &fakecloud.FakeCloud{ Addresses: []v1.NodeAddress{ { Type:    v1.NodeExternalIP, Address: "132.143.154.163", }, { Type:    v1.NodeExternalIP, Address: kubelet.nodeIP.String(), }, { Type:    v1.NodeInternalIP, Address: "132.143.154.164", }, { Type:    v1.NodeInternalIP, Address: kubelet.nodeIP.String(), }, { Type:    v1.NodeInternalIP, Address: "132.143.154.165", }, { Type:    v1.NodeHostName, Address: testKubeletHostname, }, }, Err: nil, } kubelet.cloud = fakeCloud kubelet.setNodeAddress(&existingNode) expectedAddresses := []v1.NodeAddress{ { Type:    v1.NodeExternalIP, Address: kubelet.nodeIP.String(), }, { Type:    v1.NodeInternalIP, Address: kubelet.nodeIP.String(), }, { Type:    v1.NodeHostName, Address: testKubeletHostname, }, } assert.True(t, apiequality.Semantic.DeepEqual(expectedAddresses, existingNode.Status.Addresses), "%s", diff.ObjectDiff(expectedAddresses, existingNode.Status.Addresses)) } </ins> func TestUpdateNewNodeStatus(t *testing.T) { // generate one more than maxImagesInNodeStatus in inputImageList inputImageList, expectedImageList := generateTestingImageList(maxImagesInNodeStatus + 1)

BUG REPORT: /kind bug What happened: After upgrading the kubernetes cluster from 1.6.4 with the VSphere Cloud Provider enabled to 1.6.5/1.6.6 or 1.6.7, the cluster nodes don't have anymore InternalIPs Several issues occur at this step. For example, it breaks heapster with such an error : What you expected to happen: The node should have an InternalIP. How to reproduce it (as minimally and precisely as possible): Deploy a 1 node kubernetes cluster in a VM, and enable the vsphere cloud provider. Anything else we need to know?: with kubernetes 1.6.4 we get : Downgrading to 1.6.4 fixes the issue. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: vsphere OS (e.g. from /etc/os-release):Kernel (e.g. ):Install tools: Others: It may be related to some changes around PR kubernetes /area platform/vsphere cloudprovider
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <bre.g., for API Machinery<br(2) specifying the label manually: <bre.g., for sig/scalability<br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
/sig cluster-ops /sig network /area platform/vsphere /area cloudprovider
Performing some more tests on a mini instance outside of the production cluster, I think I've identified the issue. The configuration required to start kubelet with the option (otherwise the wrong IP/interface is used to communicate between nodes). The issue is triggered when this option is set. The cloud provider looks OK. But the issue is located here : When the is not , the loop iterates on the node addresses and returns only the first one found. Since PR kubernetes, the list always returns an ExternalIP first. I think the loop in should iterate on all the addresses to rebuild the list, so that it will include the ExternalIP and the InternalIP. Can someone confirm the analysis ?
/sig node
ping
Modifying the code like this solved the issue : which returns : I guess it can be done in a cleaner way, but it was to quickly test the fix.

CC:
Thanks for filing this issue related to vSphere cloud provider. I agree with your analysis. It seems the current function assumes that the IP address specified by user can only be InternalIP. Your fix to is a better solution than work-around in cloud provider.
Hi, what are the next steps to discuss about the PR ? Are you able to make some tests ?


Question: eps := sets.NewString() <del> for i := 0; i < maxTries; i++ { </del> <ins> for i := 0; i < tries; i++ { </ins> stdout, stderr, err := config.f.ExecShellInPodWithFullOutput(config.HostTestContainerPod.Name, cmd) if err != nil { // A failure to kubectl exec counts as a try, not a hard fail.

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug /priority failing-test /sig network What happened: From https://k8s-, seems to be broken right after they were by Example error message: and Anything else we need to know?: Is it as expected?
Oct 11 16:37:49.106: test session affinity, cost time: 1m31.742990485s This is a mistake. Sorry for my debug lines sneak in. I was just wondering why this test case doesn't run in upstream CI, so I this debug line. Now, I see because this test case only runs in . Oct 11 16:39:42.406: Expect endpoints: map[], got: map[netserver-1:{}] This check failed unexpectedly, let me try to find out why. I am working on it and will let you know soon when I have some updates.
Thanks. These two tests don't run on pull jobs as they are marked as , though https://k8s-, https://k8s- (and other CIs that include ) would have them running.
Do you mean when I raise a PR, these test cases will not be triggered? My question is how can I make sure my follow-up PR can fix this bug? Where can I access the log message when I raise a new PR?
Can I see the updated test results via directly visiting https://k8s-, https://k8s-
Yep, that is the current behavior. A hacky way to make it run on PR --- add a temporary commit to remove the word from test name so some pull jobs ( I think?) will run it. And you could view logs by clicking the button.
Nope, this CI only includes changes that are merged. It is basically a cronjob that runs tests against head.


Question: Name:      "node2normalpod", Namespace: "ns", }, <ins> DeleteOptions: &metav1.DeleteOptions{GracePeriodSeconds: &zero}, </ins> }) } } createNode2MirrorPodEviction := func(client clientset.Interface) func() error { return func() error { <ins> zero := int64(0) </ins> return client.Policy().Evictions("ns").Evict(&policy.Eviction{ TypeMeta: metav1.TypeMeta{ APIVersion: "policy/v1beta1",

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: When using the function from Python's Kuberneets API, we noticed that Pods are terminated bythe cluster without respecting their preStop hooks or terminationGracePeriodSeconds seconds. What you expected to happen: When using the eviction API, just as when using delete, Pods should be allowed to run their preStop hooks, as well as be given terminationGracePeriodSeconds before being terminated with SIGKILL. How to reproduce it (as minimally and precisely as possible): the following pod: the pod logs in another window to evict the pod. Here is a sample curl equivalent of what (1.13) is doing - happy path: Here is a sample curl equivalent of what Python's is doing (when not passed ) - broken path: The log output generated by the guinea pod on the happy path looks like this: The "Got signal 15" message shows up only after the preStop hook is finished, so ~10 seconds after the evict call is made What happens on the broken path is: We seem to be getting 2x SIGTERM, there are no messages indicating continued processing after the signal, and the signal itself is registered immediately after the eviction call is made, suggesting that preStop didn't run. Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: AWS OS (e.g. from /etc/os-release): CentOS Linux release 7.5.1804 Kernel (e.g. ): 3.10.0-862.14.4.el7.x8664 SMP Wed Sep 26 15:12:11 UTC 2018 x8664 x8664 x8664 GNU/Linux Install tools: in-house Others:
/sig api-machinery
this is because of this code: invoked from here: it is passing in the eviction delete options field as-is, meaning nil gets propagated and defaulted to mean delete immediately. when calling delete APIs normally, delete options defaults to a non-nil empty set of options:
Fixed in
patch looks legit. Thanks for the quick reaction and for the pointers!
I've a elasticsearch stateful set which have a preHook which never completes and the pod is never killed despite terminationGracePeriodSeconds being set.

Is it related ? If so can this be considered to be cherry-picked on v1.13.x release branch ?


Question: errCh.SendErrorWithCancel(err, cancel) return } <del> for i := range nodeScoreList { nodeScoreList[i].Score = nodeScoreList[i].Score * weight </del> <ins> for i, nodeScore := range nodeScoreList { // return error if score plugin returns invalid score. if nodeScore.Score > MaxNodeScore || nodeScore.Score < MinNodeScore { err := fmt.Errorf("score plugin %q returns an invalid score %q, it should in the range of [MinNodeScore, MaxNodeScore] after normalizing", pl.Name(), nodeScore.Score) errCh.SendErrorWithCancel(err, cancel) return } nodeScoreList[i].Score = nodeScore.Score * weight </ins> } })

What would you like to be : a default normalize score function in the scheduling framework. the plugin and log an error when the normalizing phase returns an invalid range of scores. Why is this needed: Per discussion in , we should enforce the range of score returned by the scoring plugin. /sig scheduling /priority important-soon /assign


Question: func DefaultServiceIPRange(passedServiceClusterIPRange net.IPNet) (net.IPNet, net.IP, error) { serviceClusterIPRange := passedServiceClusterIPRange if passedServiceClusterIPRange.IP == nil { <del> defaultNet := "10.0.0.0/24" glog.Infof("Network range for service cluster IPs is unspecified. Defaulting to %v.", defaultNet) _, defaultServiceClusterIPRange, err := net.ParseCIDR(defaultNet) if err != nil { return net.IPNet{}, net.IP{}, err } serviceClusterIPRange = *defaultServiceClusterIPRange </del> <ins> glog.Infof("Network range for service cluster IPs is unspecified. Defaulting to %v.", kubeoptions.DefaultServiceIPCIDR) serviceClusterIPRange = kubeoptions.DefaultServiceIPCIDR </ins> } if size := ipallocator.RangeSize(&serviceClusterIPRange); size < 8 { return net.IPNet{}, net.IP{}, fmt.Errorf("The service cluster IP range must be at least %d IP addresses", 8)

Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: There is currently no easy way to know what Kubernetes is using as the default CIDR for service cluster IPs. What you expected to happen: should be able to tell users that "10.0.0.0/24" is used as the default. They'd better check if their deployment environment has a conflict with this assumption. They need to customize this parameter to avoid such overlaps if any. How to reproduce it (as minimally and precisely as possible): Run , from the help, the default value for --service-cluster-ip-range is not mentioned. Anything else we need to know?: No. Environment: Kubernetes version (use ): Any Cloud provider or hardware configuration**: Any OS (e.g. from /etc/os-release): Any Kernel (e.g. ): Any Install tools: N/A Others: N/A
/sig api-machinery
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale


Question: } ps.Enabled = append(ps.Enabled, schedulerapi.Plugin{Name: pluginName, Weight: weight}) } <del> //lint:ignore SA4006 this value of pluginConfigs is never used. //lint:ignore SA4010 this result of append is never used. pluginConfigs = append(pluginConfigs, schedulerapi.PluginConfig{Name: pluginName}) </del> } }

/sig scheduling /triage accepted The method in only modifies the slice of PluginConfig internally. Instead, the method should return the appended slice so that callers can override the original. For example, here:
/assign
ref:
Hey are you still working on this, if not I'd like to try it out!
Hi I'm still working on this. Thanks for your interest about this issue.


Question: } // calculateScaleDownLimitWithBehavior returns the maximum number of pods that could be deleted for the given HPAScalingRules <del> func calculateScaleDownLimitWithBehaviors(currentReplicas int32, scaleEvents []timestampedScaleEvent, scalingRules *autoscalingv2.HPAScalingRules) int32 { </del> <ins> func calculateScaleDownLimitWithBehaviors(currentReplicas int32, scaleUpEvents, scaleDownEvents []timestampedScaleEvent, scalingRules *autoscalingv2.HPAScalingRules) int32 { </ins> var result int32 var proposed int32 var selectPolicyFn func(int32, int32) int32

We created an hpa as following: Then started a deployment with 4 replicas and made the CPU utilization about 10%, which proposed to double the replicas, and it did work: Then we changed the CPU utilization to about 1%, which would trigger scaling down and would kill 2 pods only for scaleDown behavior: Then changed the CPU utilization to about 20%, which would trigger scaling up: It did scale up, and for our expectation, it should be scaled to 620/5=24 but would be limited by scaleUp behavior as 4(1+300%)=16, but in fact, it was just scaled up to 8 It would be scaled up to 16 in last scaling actions Start an hpa with a scaleUp behavior which has little long period and a little great Percent value, then start a deployment with a container whose CPU utilization could be controlled easily. Change the CPU utilization to make it scale up within limit, scale down a little, and scale up again without limit. I think it is caused by , this function seems that it just considered about scaleUp event history when calculate scaleUp limit (so does calculate scaleDown limit). I think it is a bit unreasonable. Does this action match the design? Or it does be a bug? <details</details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig autoscaling
From what I understand, we observed the following behaviour: , there are pods. The HPA scales up new pods, so we have in total. , pods are scaled down, so we have in total. HPA calculates the desired Replicas with . Since all scaling events happened within the policies , the policy value of kicks in and should allow scaling up to . But it only scaled to .

I think that there is a bug in the following code: See the comments in the following snippets for an explanation: With a of 2, the is as follows: I will think of a possible solution to the problem and would like to contribute once I've made some progress on that :)
In fact, I knew this was caused by the calculating method of "getReplicasChangePerPeriod" and mentioned it in the "Anything else we need to know" of this issue. What I wonder now is that if it is an expected action or it is just a bug. If it quite is a bug, I hope I could contribute as well.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale


Question: clientPod, err := runVolumeTesterPod(f.ClientSet, config, "client", false, fsGroup, tests) if err != nil { framework.Failf("Failed to create client pod: %v", err) } <ins> defer func() { e2epod.DeletePodOrFail(f.ClientSet, clientPod.Namespace, clientPod.Name) e2epod.WaitForPodToDisappear(f.ClientSet, clientPod.Namespace, clientPod.Name, labels.Everything(), framework.Poll, framework.PodDeleteTimeout) }() </ins> framework.ExpectNoError(e2epod.WaitForPodRunningInNamespace(f.ClientSet, clientPod)) testVolumeContent(f, clientPod, fsGroup, fsType, tests) }

<!-- Please only use this template for submitting enhancement requests --What would you like to be : In block volume tests, wait for the Pod to be deleted instead of letting it be handled by the framework's namespace deletion Why is this needed: I think the regression caused by could have been caught if the tests check for Pod deletion because the Pod object shouldn't be deleted if TearDown fails.
/assign
This issue is not specific to block tests. We should in ideally wait for pod deletion at the end of all types of tests to confirm cleanup logic is functioning
I checked the tests under test/e2e/storage/testsuites/ where DeletePodWithWait is used to wait for the deletion.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten is this still an issue? One way to check is to make a breaking change in cleanup and see if the pull job fails.
I went through all block volume usage in e2e tests: All of them but one wait for their pods to be deleted. I'm fixing the remaining test in


Question: "STANDARD_E32_V3":        32, "STANDARD_E4-2S_V3":      8, "STANDARD_E48AS_V4":      32, <ins> "STANDARD_E48A_V4":       32, </ins> "STANDARD_E48S_V3":       32, "STANDARD_E48_V3":        32, "STANDARD_E4AS_V4":       8,

<!-- Please only use this template for submitting enhancement requests -- What would you like to be : related to Why is this needed: /assign /sig azure
here is the way about how to get MaxDataDiskCount: and then build kubelet, copy kubelet to agent node and then get MaxDataDiskCount data: There are 207 VM sizes totally.
Update: this command could get VM size list easily:


Question: <del> // +build linux /* Copyright 2017 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package cm import ( "testing" "github.com/stretchr/testify/assert" "k8s.io/api/core/v1" "k8s.io/apimachinery/pkg/api/resource" evictionapi "k8s.io/kubernetes/pkg/kubelet/eviction/api" ) func TestNodeAllocatableReservationForScheduling(t *testing.T) { memoryEvictionThreshold := resource.MustParse("100Mi") cpuMemCases := []struct { kubeReserved   v1.ResourceList systemReserved v1.ResourceList expected       v1.ResourceList capacity       v1.ResourceList hardThreshold  evictionapi.ThresholdValue }{ { kubeReserved:   getResourceList("100m", "100Mi"), systemReserved: getResourceList("50m", "50Mi"), capacity:       getResourceList("10", "10Gi"), expected:       getResourceList("150m", "150Mi"), }, { kubeReserved:   getResourceList("100m", "100Mi"), systemReserved: getResourceList("50m", "50Mi"), hardThreshold: evictionapi.ThresholdValue{ Quantity: &memoryEvictionThreshold, }, capacity: getResourceList("10", "10Gi"), expected: getResourceList("150m", "250Mi"), }, { kubeReserved:   getResourceList("100m", "100Mi"), systemReserved: getResourceList("50m", "50Mi"), capacity:       getResourceList("10", "10Gi"), hardThreshold: evictionapi.ThresholdValue{ Percentage: 0.05, }, expected: getResourceList("150m", "694157320"), }, { kubeReserved:   v1.ResourceList{}, systemReserved: v1.ResourceList{}, capacity:       getResourceList("10", "10Gi"), expected:       getResourceList("", ""), }, { kubeReserved:   getResourceList("", "100Mi"), systemReserved: getResourceList("50m", "50Mi"), capacity:       getResourceList("10", "10Gi"), expected:       getResourceList("50m", "150Mi"), }, { kubeReserved:   getResourceList("50m", "100Mi"), systemReserved: getResourceList("", "50Mi"), capacity:       getResourceList("10", "10Gi"), expected:       getResourceList("50m", "150Mi"), }, { kubeReserved:   getResourceList("", "100Mi"), systemReserved: getResourceList("", "50Mi"), capacity:       getResourceList("10", ""), expected:       getResourceList("", "150Mi"), }, } for idx, tc := range cpuMemCases { nc := NodeConfig{ NodeAllocatableConfig: NodeAllocatableConfig{ KubeReserved:   tc.kubeReserved, SystemReserved: tc.systemReserved, HardEvictionThresholds: []evictionapi.Threshold{ { Signal:   evictionapi.SignalMemoryAvailable, Operator: evictionapi.OpLessThan, Value:    tc.hardThreshold, }, }, }, } cm := &containerManagerImpl{ NodeConfig: nc, capacity:   tc.capacity, } for k, v := range cm.GetNodeAllocatableReservation() { expected, exists := tc.expected[k] assert.True(t, exists, "test case %d expected resource %q", idx+1, k) assert.Equal(t, expected.MilliValue(), v.MilliValue(), "test case %d failed for resource %q", idx+1, k) } } ephemeralStorageEvictionThreshold := resource.MustParse("100Mi") ephemeralStorageTestCases := []struct { kubeReserved  v1.ResourceList expected      v1.ResourceList capacity      v1.ResourceList hardThreshold evictionapi.ThresholdValue }{ { kubeReserved: getEphemeralStorageResourceList("100Mi"), capacity:     getEphemeralStorageResourceList("10Gi"), expected:     getEphemeralStorageResourceList("100Mi"), }, { kubeReserved: getEphemeralStorageResourceList("100Mi"), hardThreshold: evictionapi.ThresholdValue{ Quantity: &ephemeralStorageEvictionThreshold, }, capacity: getEphemeralStorageResourceList("10Gi"), expected: getEphemeralStorageResourceList("200Mi"), }, { kubeReserved: getEphemeralStorageResourceList("150Mi"), capacity:     getEphemeralStorageResourceList("10Gi"), hardThreshold: evictionapi.ThresholdValue{ Percentage: 0.05, }, expected: getEphemeralStorageResourceList("694157320"), }, { kubeReserved: v1.ResourceList{}, capacity:     getEphemeralStorageResourceList("10Gi"), expected:     getEphemeralStorageResourceList(""), }, } for idx, tc := range ephemeralStorageTestCases { nc := NodeConfig{ NodeAllocatableConfig: NodeAllocatableConfig{ KubeReserved: tc.kubeReserved, HardEvictionThresholds: []evictionapi.Threshold{ { Signal:   evictionapi.SignalNodeFsAvailable, Operator: evictionapi.OpLessThan, Value:    tc.hardThreshold, }, }, }, } cm := &containerManagerImpl{ NodeConfig: nc, capacity:   tc.capacity, } for k, v := range cm.GetNodeAllocatableReservation() { expected, exists := tc.expected[k] assert.True(t, exists, "test case %d expected resource %q", idx+1, k) assert.Equal(t, expected.MilliValue(), v.MilliValue(), "test case %d failed for resource %q", idx+1, k) } } } func TestNodeAllocatableForEnforcement(t *testing.T) { memoryEvictionThreshold := resource.MustParse("100Mi") testCases := []struct { kubeReserved   v1.ResourceList systemReserved v1.ResourceList capacity       v1.ResourceList expected       v1.ResourceList hardThreshold  evictionapi.ThresholdValue }{ { kubeReserved:   getResourceList("100m", "100Mi"), systemReserved: getResourceList("50m", "50Mi"), capacity:       getResourceList("10", "10Gi"), expected:       getResourceList("9850m", "10090Mi"), }, { kubeReserved:   getResourceList("100m", "100Mi"), systemReserved: getResourceList("50m", "50Mi"), hardThreshold: evictionapi.ThresholdValue{ Quantity: &memoryEvictionThreshold, }, capacity: getResourceList("10", "10Gi"), expected: getResourceList("9850m", "10090Mi"), }, { kubeReserved:   getResourceList("100m", "100Mi"), systemReserved: getResourceList("50m", "50Mi"), hardThreshold: evictionapi.ThresholdValue{ Percentage: 0.05, }, capacity: getResourceList("10", "10Gi"), expected: getResourceList("9850m", "10090Mi"), }, { kubeReserved:   v1.ResourceList{}, systemReserved: v1.ResourceList{}, capacity:       getResourceList("10", "10Gi"), expected:       getResourceList("10", "10Gi"), }, { kubeReserved:   getResourceList("", "100Mi"), systemReserved: getResourceList("50m", "50Mi"), capacity:       getResourceList("10", "10Gi"), expected:       getResourceList("9950m", "10090Mi"), }, { kubeReserved:   getResourceList("50m", "100Mi"), systemReserved: getResourceList("", "50Mi"), capacity:       getResourceList("10", "10Gi"), expected:       getResourceList("9950m", "10090Mi"), }, { kubeReserved:   getResourceList("", "100Mi"), systemReserved: getResourceList("", "50Mi"), capacity:       getResourceList("10", ""), expected:       getResourceList("10", ""), }, } for idx, tc := range testCases { nc := NodeConfig{ NodeAllocatableConfig: NodeAllocatableConfig{ KubeReserved:   tc.kubeReserved, SystemReserved: tc.systemReserved, HardEvictionThresholds: []evictionapi.Threshold{ { Signal:   evictionapi.SignalMemoryAvailable, Operator: evictionapi.OpLessThan, Value:    tc.hardThreshold, }, }, }, } cm := &containerManagerImpl{ NodeConfig: nc, capacity:   tc.capacity, } for k, v := range cm.getNodeAllocatableAbsolute() { expected, exists := tc.expected[k] assert.True(t, exists) assert.Equal(t, expected.MilliValue(), v.MilliValue(), "test case %d failed for resource %q", idx+1, k) } } } func TestNodeAllocatableInputValidation(t *testing.T) { memoryEvictionThreshold := resource.MustParse("100Mi") highMemoryEvictionThreshold := resource.MustParse("2Gi") cpuMemTestCases := []struct { kubeReserved         v1.ResourceList systemReserved       v1.ResourceList capacity             v1.ResourceList hardThreshold        evictionapi.ThresholdValue invalidConfiguration bool }{ { kubeReserved:   getResourceList("100m", "100Mi"), systemReserved: getResourceList("50m", "50Mi"), capacity:       getResourceList("10", "10Gi"), }, { kubeReserved:   getResourceList("100m", "100Mi"), systemReserved: getResourceList("50m", "50Mi"), hardThreshold: evictionapi.ThresholdValue{ Quantity: &memoryEvictionThreshold, }, capacity: getResourceList("10", "10Gi"), }, { kubeReserved:   getResourceList("100m", "100Mi"), systemReserved: getResourceList("50m", "50Mi"), hardThreshold: evictionapi.ThresholdValue{ Percentage: 0.05, }, capacity: getResourceList("10", "10Gi"), }, { kubeReserved:   v1.ResourceList{}, systemReserved: v1.ResourceList{}, capacity:       getResourceList("10", "10Gi"), }, { kubeReserved:   getResourceList("", "100Mi"), systemReserved: getResourceList("50m", "50Mi"), capacity:       getResourceList("10", "10Gi"), }, { kubeReserved:   getResourceList("50m", "100Mi"), systemReserved: getResourceList("", "50Mi"), capacity:       getResourceList("10", "10Gi"), }, { kubeReserved:   getResourceList("", "100Mi"), systemReserved: getResourceList("", "50Mi"), capacity:       getResourceList("10", ""), }, { kubeReserved:   getResourceList("5", "10Gi"), systemReserved: getResourceList("5", "10Gi"), hardThreshold: evictionapi.ThresholdValue{ Quantity: &highMemoryEvictionThreshold, }, capacity:             getResourceList("10", "11Gi"), invalidConfiguration: true, }, } for _, tc := range cpuMemTestCases { nc := NodeConfig{ NodeAllocatableConfig: NodeAllocatableConfig{ KubeReserved:   tc.kubeReserved, SystemReserved: tc.systemReserved, HardEvictionThresholds: []evictionapi.Threshold{ { Signal:   evictionapi.SignalMemoryAvailable, Operator: evictionapi.OpLessThan, Value:    tc.hardThreshold, }, }, }, } cm := &containerManagerImpl{ NodeConfig: nc, capacity:   tc.capacity, } err := cm.validateNodeAllocatable() if err == nil && tc.invalidConfiguration { t.Fatalf("Expected invalid node allocatable configuration") } else if err != nil && !tc.invalidConfiguration { t.Fatalf("Expected valid node allocatable configuration: %v", err) } } ephemeralStorageEvictionThreshold := resource.MustParse("100Mi") ephemeralStorageTestCases := []struct { kubeReserved         v1.ResourceList capacity             v1.ResourceList hardThreshold        evictionapi.ThresholdValue invalidConfiguration bool }{ { kubeReserved: getEphemeralStorageResourceList("100Mi"), capacity:     getEphemeralStorageResourceList("500Mi"), }, { kubeReserved: getEphemeralStorageResourceList("20Gi"), hardThreshold: evictionapi.ThresholdValue{ Quantity: &ephemeralStorageEvictionThreshold, }, capacity:             getEphemeralStorageResourceList("20Gi"), invalidConfiguration: true, }, } for _, tc := range ephemeralStorageTestCases { nc := NodeConfig{ NodeAllocatableConfig: NodeAllocatableConfig{ KubeReserved: tc.kubeReserved, HardEvictionThresholds: []evictionapi.Threshold{ { Signal:   evictionapi.SignalNodeFsAvailable, Operator: evictionapi.OpLessThan, Value:    tc.hardThreshold, }, }, }, } cm := &containerManagerImpl{ NodeConfig: nc, capacity:   tc.capacity, } err := cm.validateNodeAllocatable() if err == nil && tc.invalidConfiguration { t.Fatalf("Expected invalid node allocatable configuration") } else if err != nil && !tc.invalidConfiguration { t.Fatalf("Expected valid node allocatable configuration: %v", err) } } } // getEphemeralStorageResourceList returns a ResourceList with the // specified ephemeral storage resource values func getEphemeralStorageResourceList(storage string) v1.ResourceList { res := v1.ResourceList{} if storage != "" { res[v1.ResourceEphemeralStorage] = resource.MustParse(storage) } return res } </del>

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: After running some tests, we're seeing that the combined usage of kubelet + docker can be around 2Gi. We need to be able to account for this in scheduling to avoid crashing these services. The first step towards this is using kube-reserved and system-reserved to avoid oversubscribing the virtual memory space available on the Windows nodes with too many containers. What you expected to happen: It should reduce node allocatable memory by kube-reserved + system-reserved. How to reproduce it (as minimally and precisely as possible): This logs an error in the kubelet log such as: code: Anything else we need to know?: /milestone v1.13 /assign feiskyer /sig windows This blocks a PR where I want to set it by default to 2Gi Environment: Kubernetes version (use ): v1.11.3 (also v1.12) Cloud provider or hardware configuration: Azure OS (e.g. from /etc/os-release): Windows Server version 1803 Install tools: acs-engine
You must be a member of the github team to set the milestone. <detailsIn response to : Is this a BUG REPORT or FEATURE REQUEST?: What happened: What you expected to happen: This logs an error in the kubelet log such as: code: /milestone v1.13 /assign feiskyer Kubernetes version (use ): v1.11.3 (also v1.12) Cloud provider or hardware configuration: Azure OS (e.g. from /etc/os-release): Windows Server version 1803 Install tools: acs-engine Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/milestone v1.13
/priority important-soon


Question: var containers []container klogV := klog.V(4) klogVEnabled := klogV.Enabled() <del> for _, s := range podStatus.ContainerStatuses { </del> <ins> for _, s := range stoppedPodStatus.ContainerStatuses { </ins> if s.State == kubecontainer.ContainerStateRunning { runningContainers = append(runningContainers, s.ID.String()) }

My pod completes (success or failure). The IP temporarily is removed on one of the status updates. The IP is persistent . Apply: Watch: The initial 'succeeded' should have the IP still. On older versions: Full pod state, from 1.27: <details</detailsPrior discussion many, see above Tested on kind Linux <details</details<details</details<details</details
/sig network maybe Relevant PRs in 1.27:
cc
/priority important-soon
is about endpointslices, so no related seems related , ping
I can confirm this commit specifically is the culprit (when I build and test the parent commit it works ok): For now I don't know the reason, cc in case you have some ideas. EDIT: when commenting out these it works as before. I suspect this is because the does not have the information which is determined in
I've opened a PR with a fix. PTAL: Additionally, it turns out that the bug causes one more unnecessary request with status update.
This also relates to sig-node: /sig node We talked about this in the SIG Network meeting, looks like you are taking a look: /assign Please let us know what you find, and please let us know if you need any support!
Thank you! I think I have got to the bottom and summarized the findings in the PR description: I would love to move forward with the review :)
the question from SIG Node meeting was - what is the impact of it besides the flickering IP on status. This may help justify the cherry-pick.
do you have some additional insights about the impact?
In our usage we had to handle events on a pod informer differently to account for the fact we may read it in an invalid state if we read at the wrong time. So for us technically it hsa no impact since we worked around it, for others it could be quite severe potentionally
Thanks! Just FYI as demonstrated by experiments on you could also face this situation in case of kubelet restart. So, probably good you put this workaround in place anyway.
/triage accepted People affected by this regression likely will be affected by kubelet restart in a very precise moment. But it is a low probability event so I would advocate to fix the regression by backporting this fix instead of forcing consumers to work around this behavior.


Question: <ins> /* Copyright 2015 The Kubernetes Authors All rights reserved. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package operationmanager import ( "fmt" "sync" ) // Operation Manager is a thread-safe interface for keeping track of multiple pending async operations. type OperationManager interface { // Called when the operation with the given ID has started. // Creates a new channel with specified buffer size tracked with the specified ID. // Returns a read-only version of the newly created channel. // Returns an error if an entry with the specified ID already exists (previous entry must be removed by calling Close). Start(id string, bufferSize uint) (<-chan interface{}, error) // Called when the operation with the given ID has terminated. // Closes and removes the channel associated with ID. // Returns an error if no associated channel exists. Close(id string) error // Attempts to send msg to the channel associated with ID. // Returns an error if no associated channel exists. Send(id string, msg interface{}) error } // Returns a new instance of a channel manager. func NewOperationManager() OperationManager { return &operationManager{ chanMap: make(map[string]chan interface{}), } } type operationManager struct { sync.RWMutex chanMap map[string]chan interface{} } // Called when the operation with the given ID has started. // Creates a new channel with specified buffer size tracked with the specified ID. // Returns a read-only version of the newly created channel. // Returns an error if an entry with the specified ID already exists (previous entry must be removed by calling Close). func (cm *operationManager) Start(id string, bufferSize uint) (<-chan interface{}, error) { cm.Lock() defer cm.Unlock() if _, exists := cm.chanMap[id]; exists { return nil, fmt.Errorf("id %q already exists", id) } cm.chanMap[id] = make(chan interface{}, bufferSize) return cm.chanMap[id], nil } // Called when the operation with the given ID has terminated. // Closes and removes the channel associated with ID. // Returns an error if no associated channel exists. func (cm *operationManager) Close(id string) error { cm.Lock() defer cm.Unlock() if _, exists := cm.chanMap[id]; !exists { return fmt.Errorf("id %q not found", id) } close(cm.chanMap[id]) delete(cm.chanMap, id) return nil } // Attempts to send msg to the channel associated with ID. // Returns an error if no associated channel exists. func (cm *operationManager) Send(id string, msg interface{}) error { cm.RLock() defer cm.RUnlock() if _, exists := cm.chanMap[id]; !exists { return fmt.Errorf("id %q not found", id) } cm.chanMap[id] <- msg return nil } </ins>

The "" E2E test will sometimes fail because it fails to delete the PD used in the test after the test is finished. The error: Detailed test log: The disk that fails to delete still shows up in : My attempts to manually try to delete the PD fail with: (the work around for ) doesn't change anything here:
is the disk still mounted ? It looks like the detach call did not succeed for some reason
I'll have to get another repro and verify.
okay cool. If you get a repro check to see if the disk is still mounted. If so then the cleanup code is not being called. If not then the clean up started, detach was called but failed for some reason; look in the logs for something like "googleapi: some error about detaching". It can also just be a race condition. That is, detach was called but is taking too long; if the disk does not detach then you cannot delete it.


Question: return case res, ok := <-w.etcdIncoming: if ok { <del> if curLen := int64(len(w.etcdIncoming)); watchChannelHWM.Update(curLen) { </del> <ins> if curLen := int64(len(w.etcdIncoming)); w.incomingHWM.Update(curLen) { </ins> // Monitor if this gets backed up, and how much. <del> glog.V(2).Infof("watch: %v objects queued in channel.", curLen) </del> <ins> glog.V(1).Infof("watch: %v objects queued in incoming channel.", curLen) </ins> } w.sendResult(res) }

https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite} Previous issues for this test:
This is pretty much the same failure, that are happening in "gce-enormous" suite and we currently debugging them with The main reason of those failures is that we are terminating all watchers for a particular resource type from time to time, and some of them are actually causing a snowball effect. After more looking into it, my hypothesis is that our machinery for aggregating watchers in apiserver sometimes is just to slow. In particular, I some log in my local branch and what I'm observing is that in etcd_watcher, we are: accumulating objects in incoming channel but also accumulating objects in outgoing channel, and the accumulation there is usually higher [I will send a PR to put those logs into master - they seem to be pretty useful for debugging such issues.] So my current hypothesis is that dispatching watch events to watchers is what is too slow. We will continue debugging.
The logs that I seem to confirm my hypothesis - outgoing channel is generally more full then the incoming channel in etcd_watcher: vs I'm working on the PR that should at least improve the situation.
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.

https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
Please apply the label to this issue if it does not need to be fixed before we launch 1.4. Otherwise please advise on how to proceed with respect to 1.4.
This was a regression - I fixed it by reverting an offending PR: This seems to be fixed now.
I have 33 pods in my container (1 is substantive, the other 32 just run on log files). I get this all the time now. I'm running 1.5.4.
- to be honest, I don't undertsand what your wrote. what is "container" in "33 pods in my container"? Also, if you believe there is a bug somewhere, please open a new issue instead of adding comment to more than half-year-old one.


Question: opts := *out.(*api.ListOptions) // transform fields <ins> // TODO: queryToObject should do this. </ins> fn := func(label, value string) (newLabel, newValue string, err error) { return scope.Convertor.ConvertFieldLabel(scope.APIVersion, scope.Kind, label, value) }

Using a watch expression on endpoints can lead to unpredictable behavior that includes returning a watch stream that ignores filters (e.g. a {name} determination) and returns items of the wrong type (e.g. changes to PODS in the ENDPOINTS stream). returns This result includes updates for services "kubernetes", "kubernetes-ro", "redis-slave", "redis-master", etc. Even though the requesting URL indicated only "guestbook" updates were to be watched. This seems to only occur when no resourceVersion is specified and the most recent resourceVersion has fallen off the etcd watch window. The expected behavior when resourceVersion is not specified I would think should be either: an initial update with the latest version of the resource, followed by filtered updates. nothing, followed by filtered updates. Though I'm not sure which of these is actually the spec. I would prefer (1) since it allows you to do a resource watch without having to first do a separate GET. But (2) coupled with a preceding GET would yield the same results.
IRC: cc
The pods mentioned in the stream are references to the pods corresponding to endpoints. See for discussion behind the feature. We probably need to distinguish the resourceVersion at which an object was last modified, which is also used as an update precondition, vs. the current (global) resourceVersion needed for watch.
Does the change to use GET with watch as a query parameter enable the GET to both return the latest version and subscribe to changes, or is a separate non-watch GET still required?
/subscribe
If you pass resourceVersion=0 you get list then watch (for single or multiple).
That behavior has been there for a while - the only challenge is you don't know when the list is complete, and we felt that list then watch more clearly delineated that.
This.
Watching with resourceVersion=0 should return an error, IMO.
I think the semantics for {watch a single resource} vs. {watch all resources of the same type} should be separated. When watching a single resource I would expect the watch to have consistent semantics. That is, I would expect it to be possible to watch an item and be guaranteed either: I saw every change to the item, or I knew that a change was missed (and so could do a raw GET to re-sync).

To this end it would be ideal if watch for a single item had the following semantics: Delivered the current state, followed by a list of all changes. If any change is missed or can't be delivered then the watch is terminated. Delivers either a guarantee of all changes that occurred on that resource from (exclusive) to present. If the server cannot prove that there were no missing changes then it should return an error and terminate the watch. This would imply that: Would always return an error unless the server can prove that it can deliver all changes to the resource (e.g. when the resource hasn't been created yet.) I'm surprised that the interface wants to provide multi-resource watch. It would seem that the semantics of this watch would either be potentially expensive to deliver or "best effort" such that it is basically useless for anything other than casual human consumption. Imagine a future where the master is sharded such that even different resources of the same type are on different shards. To provide meaningful semantics to a multi-resource watch would require that resource versions be meaningful in a global sense and in a way that spans all shards. Wouldn't this be undesirable? It would require that any write at any shard would need a round-trip message to all other shards to ensure that the logical clocks remain in sync. A multi-watch would similarly have to be fanned out to all shards in the backend. Both of these would defeat the benefits of sharding the data in the first place. I could understand providing a multi-resource watch for the creation and deletion of resources of a particular type (handled potentially by a secondary index in the sharded scheme) without providing a full watch of the contents. You could then use this watch to establish resource-specific watches for the resources you needed to track the contents of. Maintaining a secondary index wouldn't have to be transactional with changes that produced or destroyed items.
resourceVersion=0 is the same thing as saying watch resourceVersion=ObjectModifiedResourceVersion-1. Perhaps it should be more explicit, but it is generally useful. Why is blind watch bad? ----- Original Message -----
Also, I don't see it mentioned above but - everything in  is ignored. Confusing, but already fixed by the deprecation of .

cc
We don't offer multi-resource watch today, but in the future you're likely to correlate very strongly with a shard based on your namespace, rather than a shard on your type. I don't think sharding by type will be common except on a few types of resources (events, namespaces, possibly endpoints). The access pattern for most clients is going to be "in namespace Foo, get type X, get type Y, get type Z, get type A". Most UIs that want to watch anything will watch on the namespace. So the access pattern is less likely to reward sharding by type.
The query: Is a multi-resource watch. It watches for all resources of type Endpoints. But do you want the interface to guarantee that all objects in the same namespace are updated with consistency? I agree that shard boundaries are almost certainly NOT type-boundaries. For instance there is clearly a consistency relationship between a Service and its Endpoints object. I would hope that it would be impossible for me to observe a change to a Service object and then to subsequently query for its Endpoints object and get a stale read (semantically speaking). But that is a query across types. Suggesting that the appropriate shard boundaries are certainly not type-boundaries.
Sorry, I was referring to "multi-resource" as "watching over multiple resource types". It is possible to emulate a watch across a small number of stable shards by creating a resource version tuple (a vector clock, essentially) - for medium sized clusters I see that as a reasonable option. It's more likely though that infrastructure components that want to "watch the world" are either themselves shardable (such that it's acceptable for world to be defined as "my shard") or an intermediate component supports them. That intermediate component might be something like a partially ordered event stream that provides a firehose to the cluster as an aggregator of the cluster changes. Today, we require that API consumers assume resource version is internally consistent across a single type in a single namespace. Controllers are allowed to watch the world until such a time as we implement a sharding solution. There is no consistency relationship across resource types - services and endpoints are independent objects with independent lifecycles. It is by design that Endpoints can be updated out of band with Services, to allow downstream components to offload them.

We relax referential rules across resource types - there is no guarantee a referenced object will be created before or after you, or that it should exist only if you exist.
I completely agree there are possible implementations that can tolerate sharded and/or inconsistent queries. And deciding that certain kinds of JOINs, particularly across resource types and/or namespaces, are always going to be inconsistent seems like a fine place to land. I was merely surprised that the API was designed to promote the appearance of consistency for these queries today. I would have expected that the API would be more conservative to reserve these degrees of freedom around consistency for future implementations without requiring a future breaking change. Offering less functionality in V1 or functionality with weaker semantics would allow weakly consistent future implementations to emerge later without breaking existing applications. As a general rule I try to avoid APIs that give the appearance of consistency if they are not intended to actually deliver consistency. I think most developers will always assume consistency unless there is a really good reason to believe otherwise. This seems like human nature since people expect systems to exhibit the same causal behavior as the real world. Take for instance Endpoints. My impression of this after seeing the API was that Endpoints was simply a VIEW (in the SQL since) that provides a convenient way to express the JOIN between Service and Pods. From that perspective I would certainly have expected there to be a causal relationship between a change in the Service and its corresponding Endpoints object. I could easily imagine myself making that mistake when writing a application. Perhaps my program changes the Service object and then wants to send a message to all of the Endpoints associated with the new Service definition. After changing the Service, my program queries Endpoints. Did it get the new set of Endpoints? There are several possible outcomes: The Endpoints is the same as it was before I changed the Service Definition: The new Service definition just happens to result in the same set of Endpoints as before. I have not waited long enough for Endpoints to be updated. How long should I wait? What indicates that I've waited long enough? The Endpoints is different: Is this the correct one? The one that corresponds to my Service definition change? Or is this the update that occurred due to a different Service definition change that happened just before mine? If so, I should wait longer for mine.

Or is this the update that occurred due to a different Service definition change that happened just after mine? If so, I should either use this one or attempt my Service definition change again. In an inconsistent and asynchronously updated world we need to provide a way to answer these questions. (e.g. perhaps Endpoints should include the the resourceVersion of the Service definition used to compute it.) All applications need to properly deal with this extra information. (e.g. need to ignore all updated Endpoints until one arrives that has a resourceVersion = their change to the Service definition.) I'd imagine it's possible to implement this and for customers to use it correctly, but I think it offers a lot of opportunity for bugs. Users might simply wait for the next update and then take the wrong action leading to a misbehaving system. If we could decide which relationships should exhibit consistency and provide it where the developer would expect it then the API is simply and easier to use and applications are more likely to be correct. If we don't offer APIs which might lead the developer to make incorrect assumptions then applications are less likely to have those bugs.
I dug into this... it's more broken than I thought it was, so I've made a fix in . Once my fix goes in, a single object acts as if you didn't pass a name: this will be fixed directly (at least for the resource types I checked). get an appropriate resource version, you will have to LIST with a fieldselector ("?fields=name%3Dfoo" or some such-- don't recall exact syntax) instead of GETting. This way we don't have to modify the API to pass a special "watch at" resource version.


Question: // RegisterPluginAsExtensionsWithWeight returns a function to register a Plugin as given extensionPoints with weight to a given registry. func RegisterPluginAsExtensionsWithWeight(pluginName string, weight int32, pluginNewFunc runtime.PluginFactory, extensions ...string) RegisterPluginFunc { <del> return func(reg *runtime.Registry, plugins *schedulerapi.Plugins, pluginConfigs []schedulerapi.PluginConfig) { </del> <ins> return func(reg *runtime.Registry, plugins *schedulerapi.Plugins) { </ins> reg.Register(pluginName, pluginNewFunc) for _, extension := range extensions { ps := getPluginSetByExtension(plugins, extension)

/sig scheduling /triage accepted The method in only modifies the slice of PluginConfig internally. Instead, the method should return the appended slice so that callers can override the original. For example, here:
/assign
ref:
Hey are you still working on this, if not I'd like to try it out!
Hi I'm still working on this. Thanks for your interest about this issue.


Question: if sctpPort != -1 { go startSCTPServer(sctpPort) } <del> startHTTPServer(httpPort) </del> <ins> addRoutes() if len(certFile) > 0 { // only start HTTPS server if a cert is provided startHTTPSServer(httpPort, certFile, privKeyFile) } else { startHTTPServer(httpPort) } </ins> } <del> func startHTTPServer(httpPort int) { </del> <ins> func addRoutes() { </ins> http.HandleFunc("/", rootHandler) http.HandleFunc("/clientip", clientIPHandler) http.HandleFunc("/echo", echoHandler)

<!-- Please only use this template for submitting enhancement requests --What would you like to be : The image used in E2E tests is used for testing a wide variety of OS-agnostic features. One of these is , which runs an HTTP server with various endpoints that can be used to validate things like request/response. It would be useful to allow for 's functionality to also support HTTPS so all of that logic can be tested for TLS-based traffic. Currently, can be run like: I'm proposing something along the lines of: The default certificate/private key being basically identical to what provides. Why is this needed: One use case is where the intent is to add HTTPS functionality to lifecycle hooks (postStart/preStop). There is currently no easy way to test this logic because the container which receives the HTTPS request (in this case, ) does not support HTTPS traffic.
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig testing


Question: } }() <del> nodePortOp := portallocator.StartOperation(rs.serviceNodePorts) defer nodePortOp.Finish() </del> if api.IsServiceIPRequested(service) { // Allocate next available. ip, err := rs.serviceIPs.AllocateNext()

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): K8S 1.5.3, and kubectl 1.5.3 Environment: Cloud provider or hardware configuration: azure OS (e.g. from /etc/os-release): xenial Kernel (e.g. ): latest xenial Install tools: Others: ACS-Engine cluster, 3 masters What happened: Nodeport is assigned to dashboard and to the nginx service that was started within a minute of starting the dashboard. The dashboard is being started as part of the addon manager. What you expected to happen: Each service should receive a separate node port. How to reproduce it (as minimally and precisely as possible): Run 3 master setup, with 3 API servers, 3 addon managers, 3 controllers (but one controller is elected leader). Run dashboard as an add-on. Create service shortly after dashboard is started an assigned nodeport. We run a Jenkin's job, and this repros about 30-50% of the time. Anything else we need to know: We instrumented the build and found the following: is initially created by the addon manager by issuing a POST to/api/v1/namespaces/kube-system/services. This is where port is , and snapshotted to etcd Immediately after the POST, another addon manager issues PATCH /api/v1/namespaces/kube-system/services/kubernetes-dashboard.

The post calls a release to , and this is snapshotted to etcd. API server receives the request for nginx and assigns to nginx service. all api servers start reporting message the node port for service nginx/namespace- was assigned to multiple services. Workaround: delete the nginx service, re-create the service. have you seen this before? This build has your latest changelist around , but from best we can observe is unrelated to the changes.
The attached files are the API Server Logs: Here is a timeline of events: 16:07:02.862 - - Start POST /api/v1/namespaces/kube-system/services 16:07:02.863 - - [Create() 16:07:03.173 - - allocated - in memory 16:07:03.255 - - Create] 16:07:03.107 - - bitmap.Snapshot (177), 16:07:03.256 - - Finish POST /api/v1/namespaces/kube-system/services 16:07:08.474 - - start patch k8s-dashboard 16:07:08.832 - - released from allocator bitmap, and snapshotted to etcd
because type=NodePort ? Why is this releasing the port? On purpose or bug? Ports are allocated randomly from 2767 values, and you're reliably reproducing this collision? Why did they not get the update to the service that released it? Nope. I'll try to repro.
type=NodePort ? Correct, the dashboard addon specifies is this releasing the port? On purpose or bug? Discussed with and we figured it is because the addon manager does not see dynamically created , in the addon yaml (because it is dynamically assigned). This points at a potential bug in the add-on manager that it should ignore the dynamically assigned values, ignoring the need for the patch. are allocated randomly from 2767 values, and you're reliably reproducing this collision? We are running Jenkins, and this repro's 30-50% of the time in the multi-master case. The times it does not repro is when the same API server handles both the PUT of the dashboard and the nginx services, because AllocateNext() chooses the next random value for Nginx. did they not get the update to the service that released it?

It is being released from the allocator bitmap in the registry, but not the service definition. My guess is that it is successfully being removed from the allocator bitmap, but then when is attempted to be removed from the service it fails because the service is of type NodePort. I need to instrument to confirm this. If once confirmed, suggest I look to see if the removal from service can happen first, before removing from the allocator bitmap (nodeport should be gone from service definition, before being removed from allocator registry). That way if removal from service fails, then allocator bitmap still keeps other services from locking it.
for 3) it is the same across multiple machines, I assume it's because the random number generator has the same constant seed. for 4) my guess is that on Service update we're doing the following: M1: Update to service-1 comes in without NodePort (this is an update to an existing service) M1: Allocator releases the port , writes to etcd. M1: Decides to re-allocate to the service-1, but hasn't yet written to etcd M2: New service service-2 is created without NodePort M2: Allocator also decides to allocate to service-2, writes it to etcd. M1: Allocator fails to update etcd At this point, both Service-1 and Service-2 have the same NodePort assigned to them.
And a clarification on the Addon bits: Addon manager has a service that looks like: But when there is a for it, it has in the response: so add-on manager sees a difference, and posts back something missing the which clears and resets and starts this cycle.
OK, I think I have found 1 bug in this path, but I can't convince myself it would cause this failure mode, still working on it.
The addon manager (the one we support, anyway - did you write your own?) uses , so this should be fine. The nodePort is the same as clusterIP in this regard. You can see it by repeatedly running on this and seeing the nodePort not change.
The dashboard service object does not lose port , it is only removed from etcd path /registry/ranges/servicenodeports. Also, I see the same thing using , in that the release is happening, but when I view the dashboard service object, it still shows holding .

Here's the instrumented logs: 21:50:20.7778 ] BEGIN PATCH /api/v1/namespaces/kube-system/services/kubernetes-dashboard 21:50:20.8228 ] [bitmap.Release(177), 21:50:20.8228 ] bitmap.Release(177)] 21:50:20.8381 ] [bitmap.Snapshot, 21:50:20.8381 ] bitmap.Snapshot] 21:50:20.8389 ] END PATCH /api/v1/namespaces/kube-system/services/kubernetes-dashboard
Can you share this patch?
ohh, maybe the same bug is triggering this! God this code is twisty. Are you game to try a patch? in , around line 435, look for change the to - just remove the , then rebuild and re-run? Tim
Yeah, its the standard addon manager. We see it continuously doing a to the API Server. (has moved to the server?)
apply generates a PATCH , Brendan Burns <:
On reflection all afternoon, I am pretty confident this is the bug. I'll send a PR, sans test probably, shortly , Tim Hockin <: We see it continuously doing a PATCH to the API Server. (has apply moved to the server?) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub , or mute the thread
, Tim Hockin <:
Thanks I'm trying the patch and testing now.
Jenkins is now passing 100% on 3 master HA. Also, logs no longer show the removal of the dashboard nodeport from registry. I confirmed this for kubectl apply as well on the dashboard file. Thanks for the fix!
Thanks for the challenging bug. It reminded me how much I hate this code :) , anhowe <:


Question: if err != nil { return nil, err } <del> if plugin == nil { // Not found but not an error return nil, nil } c, err := plugin.NewUnmounter(spec.Name(), podUID) if err == nil && c == nil { return nil, errUnsupportedVolumeType } return c, nil </del> <ins> return plugin.NewUnmounter(spec.Name(), podUID) </ins> } func (vh *volumeHost) GetCloudProvider() cloudprovider.Interface {

in there is a generic error variable defined that is used when a failure occurs and no other error is surfaced, it is a catch-all when something goes wrong. Think this should be reworded to something that is more reflective of the true issue and not so confusing for users. This error mostly comes when a builder can not be created and it is typically not true that the volume type is unsupported, but rather some other underlying issue, such as missing secrets or endpoints or etc... propose something like: or
are you working on this?
- yes, I would like to submit the PR for this

fyi When using PV/PVC for Persistent Volume volume mounts, if something goes wrong in the builder, a generic error is surfaced in the 'kubectl describe' with the following event error description *unsupported volume type": This only occurs when using the PV/PVC abstraction and not directly using a volume plugin. These type of errors are common when using volume plugins for mounts and would be useful for users to have an easy way to identify the errors rather than searching the logs for root cause. I think there are two issues: Why is the original error surfaced from the volume plugins (, , etc...) getting eaten when they return back to the I can see in the logs that 2 volume types are processed for any pod - the first is the actual underlying type (i.e. ) and this is where the actual real error is surfaced, then immediately after this the 2nd type is processed which is the general persistent claim type (i.e. ) and this when the first type error is eaten A simple work around would be to simply add a recorder event in the when there is a builder failure. This ensures the error is recorded and displayed as an event for the describer and will handle all like errors from other volume plugin types. full modifed function for reference from This will produce a better user experience when trying to see what went wrong - in this example, I am doing glusterfs PV/PVC but am missing end points which you can clearly see now in the events of the describe: rbd example: (Missing or Couldn't get secret)
I haven't had a chance to look at your proposal in detail yet, but I agree this is one of the most confusing things new users hit
another one is when gluster endpoints are missing, we get 'unsupported volume type' with your work extrapolating the permissions for NFS to automatically set supplemental groups, is there also a way to indicate more properly it's a permissions issue for it not mounting?
- the above discussion highlights the missing endpoints as the first example, it also discusses a similar rbd issue (missing secret file) but the point is, any of those type of volume plugin type of errors would be surfaced by adding the event in the
I like it! Is there a PR?


Question: "//staging/src/k8s.io/apiserver/pkg/admission/initializer:go_default_library", "//staging/src/k8s.io/apiserver/pkg/authentication/user:go_default_library", "//staging/src/k8s.io/apiserver/pkg/authorization/authorizer:go_default_library", <ins> "//staging/src/k8s.io/client-go/discovery/fake:go_default_library", "//staging/src/k8s.io/client-go/restmapper:go_default_library", "//staging/src/k8s.io/client-go/testing:go_default_library", </ins> ], )

/kind flake /sig testing /sig auth we are injecting a restMapper into this admisson controller while in its test, the restMapper somehow become empty when we remove internal clientset/informer dependency from it (). now we come up with a walkaround in which we manually init the legacy schema via a dry import so that the built-in resources can be installed. this failing test suggests that the test is not quite correct logically.
/assign Are you up for trying to figure out how to avoid requiring the internal types?
What test does this break? Links?
link here: w/o the dry import, then will fail with output:
sure. will be looking into it
OK, setting priority so that I know this isn't a 1.13 blocker. /priority important-longterm


Question: func validateClassParameters(ref *resource.ResourceClassParametersReference, fldPath *field.Path) field.ErrorList { var allErrs field.ErrorList <del> if ref != nil { if ref.Kind == "" { allErrs = append(allErrs, field.Required(fldPath.Child("kind"), "")) </del> <ins> if ref == nil { return allErrs } // group is required but the Core group is the empty value, so it can not be enforced. if ref.APIGroup != "" { for _, msg := range utilvalidation.IsDNS1123Subdomain(ref.APIGroup) { allErrs = append(allErrs, field.Invalid(fldPath.Child("apiGroup"), ref.APIGroup, msg)) </ins> } <del> if ref.Name == "" { allErrs = append(allErrs, field.Required(fldPath.Child("name"), "")) </del> <ins> } if ref.Kind == "" { allErrs = append(allErrs, field.Required(fldPath.Child("kind"), "")) } else { for _, msg := range pathvalidation.IsValidPathSegmentName(ref.Kind) { allErrs = append(allErrs, field.Invalid(fldPath.Child("kind"), ref.Kind, msg)) </ins> } <del> if ref.Namespace != "" { for _, msg := range apimachineryvalidation.ValidateNamespaceName(ref.Namespace, false) { allErrs = append(allErrs, field.Invalid(fldPath.Child("namespace"), ref.Namespace, msg)) } </del> <ins> } if ref.Name == "" { allErrs = append(allErrs, field.Required(fldPath.Child("name"), "")) } else { for _, msg := range pathvalidation.IsValidPathSegmentName(ref.Name) { allErrs = append(allErrs, field.Invalid(fldPath.Child("name"), ref.Name, msg)) } } // namespace is optional. if ref.Namespace != "" { for _, msg := range apimachineryvalidation.ValidateNamespaceName(ref.Namespace, false) { allErrs = append(allErrs, field.Invalid(fldPath.Child("namespace"), ref.Namespace, msg)) </ins> } } return allErrs

Following YAML file is accepted without error. Notice the field, it is an instead of , containing character. This can lead to an issue when resource driver generates structured parameters' in-tree object with being just without version. The DRA scheduler plugin will try to match the parameters object based on the reference here: In latest master the code appears to use an indexer, and the issue might be hidden, if applicable at all. I did not check the reproducibility in master yet. However, main question is whether the should allow value with character in general. An error is expected from the apiserver upon object creation request, reporting that APIGroup containes APIVersion. Maybe. Have any resource driver that supports structured parameters, , for instance, also puts (gpucrd.APIGroup) value into . v1.30.1 <details</details<details</details<details</details<details</details<detailsscheduler's dynamicresources plugin </details
/sig node /triage accepted
/assign


Question: return helper, err } } else { <del> client = etcd.NewClient(etcdServerList) </del> <ins> etcdClient := etcd.NewClient(etcdServerList) transport := &http.Transport{ Dial: forked.Dial, TLSClientConfig: &tls.Config{ InsecureSkipVerify: true, }, MaxIdleConnsPerHost: 500, } etcdClient.SetTransport(transport) client = etcdClient </ins> } return master.NewEtcdHelper(client, storageVersion, pathPrefix)

When evaluating results of I have noticed a very strange behavior of apiserver when running density tests on a 100 node cluster with 30 pods per node. It seems that sometimes we spent more than 40% of CPU time in net.(*netFD).dial function. My suspicion is that it's because after each watch message we terminate old connection to etcd and create a new one. With multiple watchers and many update this can be very expensive.
!
Does the etcd client not reuse HTTP connections? I thought the go http client did that by default?
Actually looking at that chart I see it stuck in write, not dial-- am I blind?
I think it is write... The number on dail is confusing though...
There is a small, but very heavy branch on the left of the chart with "dial" path".
But it shows . It does not show where is the other 22.87s.
The way I read this profile (might wrong though) is that for 28.90s dial was in the stack, but was not the "top" function for 22.87s I tried to verify that etcd client reuses HTTP connection and I think it doesn't. I simple log line here: In empty 100 node cluster it got called ~10k times in ~3.5 min. This gives 45 dials per second Do you have a theory why we create so many connections? After adding cache in this is responsible for a biggest chunk of CPU time.
I also run density test to check how correlated this is with the load.

Results below (first column is number of dials per minute, second column is time): $ sudo docker logs 2&1 uniq -c 1839 13:12 4737 13:13 1942 13:14 1184 13:15 836 13:16 949 13:17 1360 13:18 880 13:19 770 13:20 1127 13:21 682 13:22 13:23 13:24 13:25 13:26 13:27 13:28 13:29 13:30 13:31 13:32 13:33 13:34 13:35 13:36 13:37 13:38 13:39 13:40 13:41 13:42 13:43 13:44 1314 13:45 244 13:46 Performance test started at 13:23 and finished around 13:44. Correlation is quite obvious. Note also that in the peak it was generating 560 dials per second!
I think that we have a victim for it. What we've done is that we changed the Transport in go-etcd client: but setting: MaxIdleConnsPerHost: (from default 2) This speeded up the test by ~35% I think that is some exaggeration, but it definitely proves that increasing this value solves the problem.

My hypothesis why it solves the problem: have 100 opened watches( watching for the same changes (which definitely happens in 100-node cluster) and all of them are from apiserver (a single client) change that is interesting is written to etcd is responding to all of them at the same time those connections become iddle (or most of them) before the new request is send, and thus most of them are closed (we can have only 2 of them opened for a single client) the next requests within a watch we need to create new connections
Number of dials after change described by :+1: ~$ sudo docker logs 2&1 uniq -c 335 14:29 200 14:30 5 14:31 28 14:32 65 14:33 145 14:34 69 14:35 121 14:36 59 14:37 202 14:39 140 14:40 112 14:42 77 14:43 190 14:44 104 14:45 5 14:46 4 14:47 (this includes run of performance test)
Just a silly question: are we allowed to change files under Godeps/ directory? If not what is the best way to fix it?
Call instead.
is probably right. If you only use one transport, there is a maxIdelConn limitation.
thanks ! Assigning to myself - I will prepare a PR for it.
cool, this is great. max idle is probably a little high-- we don't want to use all of apiserver's available FDs for this. Does, say, 500 still get most of the benefits? (I think if we multiplexed watches that might also help a lot, but obviously it's much easier to keep more idle connections.) (If you really had needed to modify a file in the godeps/ directory, the proper way to do that is to send a PR to the downstream repo, and the update the dependency once they accept it.)
I completely agree is too much. I didn't have time to make more experiments today, but my feeling is that 500 or 1000 should also be enough. I will come up with more accurate number on Monday. Yes - but this would be much bigger change.

I don't think it's worth doing now. It seems it's ot needed. We just need to initialize the client properly. I will create a PR for it.


Question: features.ValidatingAdmissionPolicy: { Default: true, PreRelease: featuregate.Alpha}}) if err != nil { <del> // FIXME: handle error. panic("Unexpected error") </del> <ins> t.Fatalf("Unable to add feature gate: %v", err) </ins> } err = featureGate.SetFromMap(map[string]bool{string(features.ValidatingAdmissionPolicy): true}) if err != nil { <del> // FIXME: handle error. panic("Unexpected error.") </del> <ins> t.Fatalf("Unable to store flag gate: %v", err) </ins> } plug, err := NewPlugin()

/sig testing
Just wanted to know to can I help in resolving the error handling ?
in /cc /sig api-machinery
/assign you are always welcome to help if you want to try to solve this and send a PR /triage accepted
/
can anyone suggest resources for this Issue as I am new to the testing of admission policy some guidance will surely help
I think both calls can simply be replaced with and a more helpful error message. Feel free to raise a PR, I'll take care of reviewing it.
Thanks for upfront help


Question: command := exec.Command("umount", target) output, err := command.CombinedOutput() if err != nil { <ins> if err.Error() == errNoChildProcesses { if command.ProcessState.Success() { // We don't consider errNoChildProcesses an error if the process itself succeeded (see - k/k issue #103753). return nil } // Rewrite err with the actual exit error of the process. err = &exec.ExitError{ProcessState: command.ProcessState} } </ins> return fmt.Errorf("unmount failed: %vnUnmounting arguments: %snOutput: %s", err, target, string(output)) } return nil

While investigating some slowness in mounting/unmounting operations of PVs (in this case EFS volumes) for pods, we observed errors like these: fortunately it does recover quickly on Kubelet’s next retry: Looking through the Kubelet code I found out that the issue is not related to the umount calls themselves, but rather a race condition inside where we call umount. Because of the way golang’s function (called inside ) is implemented, it for the child process to finish after it is . Since there isn't any locking on the child process across those function calls, it's possible that the child process exits event before the Wait() call starts. In such case the wait call returns with a error (). This error is more likely to happen especially if the gap between the Start() and Wait() calls is more (which could for e,g happen if the goroutine had CPU cycles snatched away from it for some time in between those calls). A similar issue also exists in the . While this is a self-recovering issue (because of Kubelet retries), it does seem to increase the pod startup/cleanup latency upto a 1s. Open to suggestions for a fix. The simplest one occurring to me is to treat the error as a special one on which we don't actually return an error for the mount/unmount calls. /sig storage /sig scalability /assign
Another thing is when analyzing a distribution of such errors across different nodes, we observed a skew. The top 10% of the nodes were seeing more than 90% of these errors. And these are nodes that are running hotter (w.r.t the pod churn) compared to others: ! Given this, I'm wondering if those errors are higher on those nodes just because of statistical significance or other causes like more GC pauses happening on those nodes.
/triage accepted
^ I've opened this PR to fix the Mount/Unmount functions in the file. In general we might want to make a similar fix to other invocations of exec.Run() and exec.CombinedOutput() within our codebase. But that'll be a bigger change. Happy to open another issue for it.
Sorry for opening the PR against wrong repo, here's the right one -


Question: "encoding/json" "fmt" "io" <ins> "io/ioutil" </ins> "math/rand" "reflect" "strings" "testing" ) <ins> func TestYAMLDecoderReadBytesLength(t *testing.T) { d := `--- stuff: 1 test-foo: 1 ` testCases := []struct { bufLen    int expectLen int expectErr error }{ {len(d), len(d), nil}, {len(d) + 10, len(d), nil}, {len(d) - 10, len(d) - 10, io.ErrShortBuffer}, } for i, testCase := range testCases { r := NewDocumentDecoder(ioutil.NopCloser(bytes.NewReader([]byte(d)))) b := make([]byte, testCase.bufLen) n, err := r.Read(b) if err != testCase.expectErr || n != testCase.expectLen { t.Fatalf("%d: unexpected body: %d / %v", i, n, err) } } } </ins> func TestSplitYAMLDocument(t *testing.T) { testCases := []struct { input  string

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: In a go program, using the YAML utility package "", the YAMLDecoder Read function always returns n=0 bytes, indicating that nothing was read. What you expected to happen: The number of bytes read should be returned as specified in the How to reproduce it (as minimally and precisely as possible): See Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: minikube & AWS OS (e.g. from /etc/os-release): macOS 10.13.1 (17B1003) Kernel (e.g. ): Darwin 17.2.0 Install tools: Others:
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc


Question: "k8s.io/klog" "k8s.io/kubernetes/pkg/kubelet/cm/cpumanager/state" "k8s.io/kubernetes/pkg/kubelet/cm/topologymanager" <ins> "k8s.io/kubernetes/pkg/kubelet/config" </ins> "k8s.io/kubernetes/pkg/kubelet/status" )

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: PR , it will remove "inactive containers" while reconciling. But the cpumanager started very beginning, it will get empty activePods at it first run. This make state file no sense. What you expected to happen: We should wait a moment or notification from podmanager before starting reconcile. Some log may be before remove "inactive containers". How to reproduce it (as minimally and precisely as possible): restart kubelet Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig node
In reconcileState : You mean the cpu assignments would be wiped for the first reconciliation ?
Yes, you can test it by restarting kubelet.
Do you see any of the following log:
I see this
/assign


Question: } } else { node.Spec.ExternalID = kl.hostname <del> // If no cloud provider is defined - use the one detected by cadvisor info, err := kl.GetCachedMachineInfo() if err == nil { kl.updateCloudProviderFromMachineInfo(node, info) </del> <ins> if kl.autoDetectCloudProvider { // If no cloud provider is defined - use the one detected by cadvisor info, err := kl.GetCachedMachineInfo() if err == nil { kl.updateCloudProviderFromMachineInfo(node, info) } </ins> } } if err := kl.setNodeStatus(node); err != nil {

It appears that the kubelet will detect and use the cloud provider as detected by cadvisor thus overriding the cloud provider passed in at runtime: This will break our private cloud configuration where we do not use our cloud provider's load balancing and instead use our own load balancing tier. In my opinion, the cloud provider specified at kubelet runtime should be respected for the node. Am I reading this correctly? Any thoughts on how to handle this properly? Thank you!
If you specify any cloud provider through flags, it will be used by kubelet (see ). It's only when you don't specify any cloud provider that kubelet will detect the cloud provider from the machine info, as described by the comment.
that's exactly my situation, I want to run in our private cloud environment without any of the cloud integration. If I specify cloud-provider="" at runtime it will get overridden.
Ah I see. I guess specifying an empty string is the same as not specifying any cloud provider. This auto-detection feature was introduced by /cc the author and reviewer to see if this is an intended behavior:
Legitimate concern. I'm posting a fix. Do you consider this to be a regression or are you just getting started with k8s.
I am sorry, This was not the original intention. We can keep the auto detection with a flag, or make a flag to ask kubelet not to do it. Unless there is a way to see if the variable was assigned to that I missed. I can post a fix tomorrow if it won't exit by then.
Thank you. We're running on 1.2.4 and won't be running on 1.3 for a few months yet anyways and was just testing to look for future issues when I found this.
I assigned this issue to you since , and I can't assign this to anyway :-)
Thanks for testing our 1.3 release candidate, and reporting this regression to us. We are including the fix to 1.3.


Question: <ins> package(default_visibility = ["//visibility:public"]) licenses(["notice"]) load( "@io_bazel_rules_go//go:def.bzl", "go_library", "go_test", ) go_library( name = "go_default_library", srcs = ["checks.go"], tags = ["automanaged"], ) go_test( name = "go_default_test", srcs = ["checks_test.go"], library = ":go_default_library", tags = ["automanaged"], deps = ["//vendor:k8s.io/apimachinery/pkg/util/wait"], ) filegroup( name = "package-srcs", srcs = glob(["**"]), tags = ["automanaged"], visibility = ["//visibility:private"], ) filegroup( name = "all-srcs", srcs = [":package-srcs"], tags = ["automanaged"], ) </ins>

See: This code should make several attempts before ing. Alternatively, we could consider not listening for connections until this code has succeeded.
Feel free to correct me if I'm wrong on any of this since I've never looked at kubernetes code base before. The code for the and the both appear to make multiple attempts before returning an error which leads to a Fatal.
, I think we had waited for to start at . Do we need to wait more or make it configurable?
I would like to work on this if nobody else is.
No need to ask for permission! Just say here that you're working on it, that should be sufficient to prevent duplicate work. :)
Can I have a go at this?
I have a nearly completed solution for this already. Just working on testing/test cases. WIll hopefully sumbit a PR shortly.
Ok.
I wondered why we care about this. After all, kubelet restarts failing pods. For posterity, after talking with the reason is that the apiserver may fail so fast that the kubelet begins to back off on restarts of a failing pod. That backoff may be too aggressive. (Maybe it causes spurious alerts, or maybe it allows too much downtime.) So, let's fix the apiserver. That's fine, but clearly we're not the only user that might encounter this. You may be running a server you don't want to (or can't) modify, and you might wish you could tweak the kubelet's backoff heuristics for your pod. Alternately, maybe the kubelet could choose not to enter the backoff if apiserver exited with a different code. anything like this already in place?
That's a wonderful explanation that hadn't occurred to me. If there isn't a feature request for this already I'll work on something and reference this issue.
AFAIK, the backoff mechanism is there to protect kubelet from being overwhelmed by failed user containers. Trusting the containers with returning a specific exit code sounds like a big "no" to me. Making this configurable per pod might be ok, but it's still not good in general to allow aggressive restarts.
The referred PR () only fixes the issue for kube-apiserver. We still need a generic fix in genericapiserver (Ref ).


Question: node.Status.Conditions = append(node.Status.Conditions, *nodeOODCondition) } <del> // NOTE(aaronlevy): NodeReady condition needs to be the last in the list of node conditions. // This is due to an issue with version skewed kubelet and master components. // ref: https://github.com/kubernetes/kubernetes/issues/16961 </del> var newNodeReadyCondition api.NodeCondition var oldNodeReadyConditionStatus api.ConditionStatus if containerRuntimeUp && networkConfigured && containerRuntimeVersionRequirementMet {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: I was looking into what tests were enabled for and found that the following tests: are running 1.18. From the output of the tests (): and What you expected to happen: tests to be running against the latest code for 1.19 How to reproduce it (as minimally and precisely as possible): It looks to be caused by the setting: Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /sig release /sig windows