Question: limitations under the License. */ <ins> // containerManagerImpl implements container manager on Windows. // Only GetNodeAllocatableReservation() and GetCapacity() are implemented now. </ins> package cm import ( <del> "github.com/golang/glog" </del> <ins> "fmt" "github.com/golang/glog" </ins> "k8s.io/api/core/v1" <ins> "k8s.io/apimachinery/pkg/api/resource" utilfeature "k8s.io/apiserver/pkg/util/feature" </ins> "k8s.io/client-go/tools/record" <ins> kubefeatures "k8s.io/kubernetes/pkg/features" </ins> internalapi "k8s.io/kubernetes/pkg/kubelet/apis/cri" "k8s.io/kubernetes/pkg/kubelet/cadvisor" <ins> "k8s.io/kubernetes/pkg/kubelet/cm/cpumanager" </ins> "k8s.io/kubernetes/pkg/kubelet/config" <ins> kubecontainer "k8s.io/kubernetes/pkg/kubelet/container" "k8s.io/kubernetes/pkg/kubelet/lifecycle" </ins> "k8s.io/kubernetes/pkg/kubelet/status" <ins> "k8s.io/kubernetes/pkg/kubelet/util/pluginwatcher" schedulercache "k8s.io/kubernetes/pkg/scheduler/cache" </ins> "k8s.io/kubernetes/pkg/util/mount" ) type containerManagerImpl struct { <del> containerManagerStub </del> <ins> // Capacity of this node. capacity v1.ResourceList // Interface for cadvisor. cadvisorInterface cadvisor.Interface // Config of this node. nodeConfig NodeConfig </ins> } <del> var _ ContainerManager = &containerManagerImpl{} </del> <ins> func (cm *containerManagerImpl) Start(node *v1.Node, activePods ActivePodsFunc, sourcesReady config.SourcesReady, podStatusProvider status.PodStatusProvider, runtimeService internalapi.RuntimeService) error { glog.V(2).Infof("Starting Windows container manager") if utilfeature.DefaultFeatureGate.Enabled(kubefeatures.LocalStorageCapacityIsolation) { rootfs, err := cm.cadvisorInterface.RootFsInfo() if err != nil { return fmt.Errorf("failed to get rootfs info: %v", err) } for rName, rCap := range cadvisor.EphemeralStorageCapacityFromFsInfo(rootfs) { cm.capacity[rName] = rCap } } </ins> <del> func (cm *containerManagerImpl) Start(_ *v1.Node, _ ActivePodsFunc, _ config.SourcesReady, _ status.PodStatusProvider, _ internalapi.RuntimeService) error { glog.V(2).Infof("Starting Windows stub container manager") </del> return nil } <ins> // NewContainerManager creates windows container manager. </ins> func NewContainerManager(mountUtil mount.Interface, cadvisorInterface cadvisor.Interface, nodeConfig NodeConfig, failSwapOn bool, devicePluginEnabled bool, recorder record.EventRecorder) (ContainerManager, error) { <del> return &containerManagerImpl{}, nil </del> <ins> var capacity = v1.ResourceList{} // It is safe to invoke `MachineInfo` on cAdvisor before logically initializing cAdvisor here because // machine info is computed and cached once as part of cAdvisor object creation. // But `RootFsInfo` and `ImagesFsInfo` are not available at this moment so they will be called later during manager starts machineInfo, err := cadvisorInterface.MachineInfo() if err != nil { return nil, err } capacity = cadvisor.CapacityFromMachineInfo(machineInfo) return &containerManagerImpl{ capacity:          capacity, nodeConfig:        nodeConfig, cadvisorInterface: cadvisorInterface, }, nil } func (cm *containerManagerImpl) SystemCgroupsLimit() v1.ResourceList { return v1.ResourceList{} } func (cm *containerManagerImpl) GetNodeConfig() NodeConfig { return NodeConfig{} } func (cm *containerManagerImpl) GetMountedSubsystems() *CgroupSubsystems { return &CgroupSubsystems{} } func (cm *containerManagerImpl) GetQOSContainersInfo() QOSContainersInfo { return QOSContainersInfo{} } func (cm *containerManagerImpl) UpdateQOSCgroups() error { return nil } func (cm *containerManagerImpl) Status() Status { return Status{} } func (cm *containerManagerImpl) GetNodeAllocatableReservation() v1.ResourceList { evictionReservation := hardEvictionReservation(cm.nodeConfig.HardEvictionThresholds, cm.capacity) result := make(v1.ResourceList) for k := range cm.capacity { value := resource.NewQuantity(0, resource.DecimalSI) if cm.nodeConfig.SystemReserved != nil { value.Add(cm.nodeConfig.SystemReserved[k]) } if cm.nodeConfig.KubeReserved != nil { value.Add(cm.nodeConfig.KubeReserved[k]) } if evictionReservation != nil { value.Add(evictionReservation[k]) } if !value.IsZero() { result[k] = *value } } return result } func (cm *containerManagerImpl) GetCapacity() v1.ResourceList { return cm.capacity } func (cm *containerManagerImpl) GetPluginRegistrationHandler() pluginwatcher.PluginHandler { return nil } func (cm *containerManagerImpl) GetDevicePluginResourceCapacity() (v1.ResourceList, v1.ResourceList, []string) { return nil, nil, []string{} } func (cm *containerManagerImpl) NewPodContainerManager() PodContainerManager { return &podContainerManagerStub{} } func (cm *containerManagerImpl) GetResources(pod *v1.Pod, container *v1.Container) (*kubecontainer.RunContainerOptions, error) { return &kubecontainer.RunContainerOptions{}, nil } func (cm *containerManagerImpl) UpdatePluginResources(*schedulercache.NodeInfo, *lifecycle.PodAdmitAttributes) error { return nil } func (cm *containerManagerImpl) InternalContainerLifecycle() InternalContainerLifecycle { return &internalContainerLifecycleImpl{cpumanager.NewFakeManager()} } func (cm *containerManagerImpl) GetPodCgroupRoot() string { return "" </ins> }

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: After running some tests, we're seeing that the combined usage of kubelet + docker can be around 2Gi. We need to be able to account for this in scheduling to avoid crashing these services. The first step towards this is using kube-reserved and system-reserved to avoid oversubscribing the virtual memory space available on the Windows nodes with too many containers. What you expected to happen: It should reduce node allocatable memory by kube-reserved + system-reserved. How to reproduce it (as minimally and precisely as possible): This logs an error in the kubelet log such as: code: Anything else we need to know?: /milestone v1.13 /assign feiskyer /sig windows This blocks a PR where I want to set it by default to 2Gi Environment: Kubernetes version (use ): v1.11.3 (also v1.12) Cloud provider or hardware configuration: Azure OS (e.g. from /etc/os-release): Windows Server version 1803 Install tools: acs-engine
You must be a member of the github team to set the milestone. <detailsIn response to : Is this a BUG REPORT or FEATURE REQUEST?: What happened: What you expected to happen: This logs an error in the kubelet log such as: code: /milestone v1.13 /assign feiskyer Kubernetes version (use ): v1.11.3 (also v1.12) Cloud provider or hardware configuration: Azure OS (e.g. from /etc/os-release): Windows Server version 1803 Install tools: acs-engine Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/milestone v1.13
/priority important-soon


Question: for i := 0; i < 8; i++ { pods[i] = v1.Pod{ObjectMeta: metav1.ObjectMeta{Name: fmt.Sprintf("pod-%d", i), ResourceVersion: fmt.Sprintf("%d", i)}} } <del> switch options.ResourceVersion { case "0": </del> <ins> rvContinueLimit := func(rv, c string, l int64) metav1.ListOptions { return metav1.ListOptions{ResourceVersion: rv, Continue: c, Limit: l} } switch rvContinueLimit(options.ResourceVersion, options.Continue, options.Limit) { // initial limited list case rvContinueLimit("0", "", 4): </ins> return &v1.PodList{ListMeta: metav1.ListMeta{ResourceVersion: "10"}, Items: pods[0:4]}, nil <del> case "10": switch options.Limit { case 4: switch options.Continue { case "": return &v1.PodList{ListMeta: metav1.ListMeta{Continue: "C1", ResourceVersion: "11"}, Items: pods[0:4]}, nil case "C1": return nil, apierrs.NewResourceExpired("The resourceVersion for the provided watch is too old.") default: t.Fatalf("Unrecognized Continue: %s", options.Continue) } case 0: return &v1.PodList{ListMeta: metav1.ListMeta{ResourceVersion: "11"}, Items: pods[0:8]}, nil default: t.Fatalf("Unrecognized Limit: %d", options.Limit) } </del> <ins> // first page of the rv=10 list case rvContinueLimit("10", "", 4): return &v1.PodList{ListMeta: metav1.ListMeta{Continue: "C1", ResourceVersion: "11"}, Items: pods[0:4]}, nil // second page of the above list case rvContinueLimit("", "C1", 4): return nil, apierrs.NewResourceExpired("The resourceVersion for the provided watch is too old.") // rv=10 unlimited list case rvContinueLimit("10", "", 0): return &v1.PodList{ListMeta: metav1.ListMeta{ResourceVersion: "11"}, Items: pods[0:8]}, nil </ins> default: <del> t.Fatalf("Unrecognized ResourceVersion: %s", options.ResourceVersion) </del> <ins> err := fmt.Errorf("unexpected list options: %#v", options) t.Error(err) return nil, err </ins> } return nil, nil },

/sig scalability Debugging below done by Failed test attempt: It failed with a number of "WaitingForPods: objects timed out" errors (46 failed objects in total). Sample failing pods: 580 nodes were not ready at 18:37: nodelifecyclecontroller said that gce-scale-cluster-minion-group-2-66d8's status hasn't been updated for 41 seconds. apiserver's logs show that kubelet was doing 'PUT ' every 10 seconds. Something weird happened to watch by 'shared-informers': One watch started at 18:36:52 and finished 9ms later. The next one started at 18:40:34. So it seems that there was a gap in lease watche between 18:36:52 and 18:40:34. Between 18:36:52 and 18:40:34 there is a number of leases LIST requests in the following pattern: a single /leases?limit=500 LIST request that finishes with 200 a single /leases/continue=XXX request that finishes with 400 The kube-controller-manager is retrying every 1 second up to 18:40:34.
/assign
Looking further at this it seems that number of LIST leases by kube-controller-manager increased significantly ~week ago in large scale tests (both gce and kubemark). GCE: ! Kubemark: ! This was caused by which as a side effect enabled pagination of LIST calls when starting/restarting watchers. In some cases we're getting 400 errors from api-server for this calls which results in the symptoms described above, i.e nodes becoming not-ready and pods stuck in running but not ready state for some time. GCE: ! Kubemark: !
There are actually two issues here: that we switched from non-paginated LIST calls to paginated the 10x increase in # calls is expected, but what is worrying is that we already had ~50 LISTs in every test. Such LIST happens only when watcher restarts which shouldn't happen in a healthy cluster This will be address in need to figure out why we're sometimes getting 400 response (BAD REQUEST).

/milestone v1.17
Logs when then 400 errors occurred api-server kube-controller-manager
So what happens here is: in reflector ends with an error different than so the doesn't get reset to next list is issued with set to page is fetched successfully, the next page is requested with the got from the first page and second page fails with due to error the doesn't get cleared anywhere so we go back to 2 and the 400 error keeps repeating until we get 410 (resourceVersionTooOld, on the next etcd compaction) and the is set to takes long enough for the NodeController to mark many nodes as not ready as it doesn't get any updates from watch about node leases This was indeed introduced by but the actual bug is in the which sets both and on the subsequent pages while it should set only . I will send a PR to fix it today.
FYI
Great finding.
Does this mean we have no tests that actually check if a reflector works with paging?
I see, it's the rare situation where "Watch in reflector ends with an error different than 410" that is untested. Can we set up a unit test that exercises this scenario by wrapping etcd3 storage to allow injecting an error in a watch attempt (similar to how we wrapped storage to test stale behavior in )?
I don't think it's the case - see . It clearly seems to be that pager wasn't properly tested. The problem seems to be that we're testing it with artifical stuff, whereas in real life etcd would return "bad request" errors.
Right, I'd like to see if we can trigger the error condition in a way that the restart/paging requests go to a real storage backend.
/milestone v1.18 (v1.17 code freeze)
- this is a regression introduced in 1.17, I actually believe we should fix that in 1.17.
/milestone v1.17 as noted above, this is a release blocking bug


Question: // New creates a new Kubelet for use in main func NewMainKubelet( <del> hn string, dc dockertools.DockerInterface, ec tools.EtcdClient, rd string, ni string, ri time.Duration, </del> <ins> hostname string, dockerClient dockertools.DockerInterface, etcdClient tools.EtcdClient, rootDirectory string, networkContainerImage string, resyncInterval time.Duration, </ins> pullQPS float32, pullBurst int, minimumGCAge time.Duration, maxContainerCount int, sourcesReady SourcesReadyFn, clusterDomain string, <del> clusterDNS net.IP) *Kubelet { </del> <ins> clusterDNS net.IP) (*Kubelet, error) { if resyncInterval <= 0 { return nil, fmt.Errorf("invalid sync frequency %d", resyncInterval) } if minimumGCAge <= 0 { return nil, fmt.Errorf("invalid minimum GC age %d", minimumGCAge) } </ins> return &Kubelet{ <del> hostname:              hn, dockerClient:          dc, etcdClient:            ec, rootDirectory:         rd, resyncInterval:        ri, networkContainerImage: ni, </del> <ins> hostname:              hostname, dockerClient:          dockerClient, etcdClient:            etcdClient, rootDirectory:         rootDirectory, resyncInterval:        resyncInterval, networkContainerImage: networkContainerImage, </ins> podWorkers:            newPodWorkers(), dockerIDToRef:         map[dockertools.DockerID]*api.ObjectReference{}, <del> runner:                dockertools.NewDockerContainerCommandRunner(dc), </del> <ins> runner:                dockertools.NewDockerContainerCommandRunner(dockerClient), </ins> httpClient:            &http.Client{}, pullQPS:               pullQPS, pullBurst:             pullBurst,

Things like SyncFrequency should have a lower-bound of acceptable, and that really should be 0ns.
The main point of this is now fixed, though we could be better about input checking all over


Question: // TestRequeueByBindFailure verify Pods failed by bind plugin are // put back to the queue regardless of whether event happens or not. func TestRequeueByBindFailure(t *testing.T) { <ins> fakeBind := &firstFailBindPlugin{} </ins> registry := frameworkruntime.Registry{ <del> "firstFailBindPlugin": newFirstFailBindPlugin, </del> <ins> "firstFailBindPlugin": func(o runtime.Object, fh framework.Handle) (framework.Plugin, error) { binder, err := defaultbinder.New(nil, fh) if err != nil { return nil, err } fakeBind.defaultBinderPlugin = binder.(framework.BindPlugin) return fakeBind, nil }, </ins> } cfg := configtesting.V1ToInternalWithDefaults(t, configv1.KubeSchedulerConfiguration{ Profiles: []configv1.KubeSchedulerProfile{{ SchedulerName: pointer.String(v1.DefaultSchedulerName),

pull-kubernetes-integration: scheduler TestRequeueByBindFailure Running with stress reproduces the flake Not sure, seeing a lot of retests on pull-kubernetes-integration Bind logic in scheduler integration tests just changed 3 days ago in , not sure if that's related No response No response /sig scheduling /priority important-soon cc
I doesn't look related to the latest change you were also looking at Bind. Any ideas?
It's a little strange because the error reports but since the bind-plugin will fail for the first calling time and there's no kubelet, that means the pod status will be kept since. I created another PR to help output the error message, which maybe helpful for analysis.
This test is very straightforward: the first try to bind "pod-1", the Pod is rejected by the binding plugin. () scheduling queue puts back this "pod-1" to activeQ/backoffQ directly, and "pod-1" will soon come back to the scheduling cycle. the second try to bind "pod-1", it's successful. And, means that (1) doesn't reject "pod-1" correctly. And I don't have any idea why it can happen. The only scenario I can think of is that there is somehow another Pod created before pod-1. When is executed , is it possible that any other Pods (like system ones) are created automatically?
/triage accepted current top flake of integration
/assign I've got one possible scenario, will have a quick fix.


Question: pendingActionSet.Insert( strings.Join([]string{"delete-collection", "daemonsets", ""}, "-"), strings.Join([]string{"delete-collection", "deployments", ""}, "-"), <ins> strings.Join([]string{"delete-collection", "replicasets", ""}, "-"), </ins> strings.Join([]string{"delete-collection", "jobs", ""}, "-"), strings.Join([]string{"delete-collection", "horizontalpodautoscalers", ""}, "-"), strings.Join([]string{"delete-collection", "ingresses", ""}, "-"),

The delete command for a replicaset hangs forever. This then manifests as phantom RSs when I delete and recreate a namespace.
- delete is the gift that keeps on giving
cc
Had you already deleted the corresponding Deployment?
I am still seeing this problem. Deployments and replica sets are sticking.
From what I can tell, calling DELETE_COLLECTION on Deployments is returning: And since the controller assumes not found errors are equivalent to already deleted, it thinks the content is deleted. When I manually inspect, I still see the deployment and replica set laying around... //cc
holding on this while i debug some more...
This was a false alarm, apologies.


Question: var possibleLimitingReason, possibleLimitingMessage string if args.DesiredReplicas > args.CurrentReplicas { <del> scaleUpLimit := calculateScaleUpLimitWithScalingRules(args.CurrentReplicas, a.scaleUpEvents[args.Key], args.ScaleUpBehavior) </del> <ins> scaleUpLimit := calculateScaleUpLimitWithScalingRules(args.CurrentReplicas, a.scaleUpEvents[args.Key], a.scaleDownEvents[args.Key], args.ScaleUpBehavior) </ins> if scaleUpLimit < args.CurrentReplicas { // We shouldn't scale up further until the scaleUpEvents will be cleaned up scaleUpLimit = args.CurrentReplicas

We created an hpa as following: Then started a deployment with 4 replicas and made the CPU utilization about 10%, which proposed to double the replicas, and it did work: Then we changed the CPU utilization to about 1%, which would trigger scaling down and would kill 2 pods only for scaleDown behavior: Then changed the CPU utilization to about 20%, which would trigger scaling up: It did scale up, and for our expectation, it should be scaled to 620/5=24 but would be limited by scaleUp behavior as 4(1+300%)=16, but in fact, it was just scaled up to 8 It would be scaled up to 16 in last scaling actions Start an hpa with a scaleUp behavior which has little long period and a little great Percent value, then start a deployment with a container whose CPU utilization could be controlled easily. Change the CPU utilization to make it scale up within limit, scale down a little, and scale up again without limit. I think it is caused by , this function seems that it just considered about scaleUp event history when calculate scaleUp limit (so does calculate scaleDown limit). I think it is a bit unreasonable. Does this action match the design? Or it does be a bug? <details</details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig autoscaling
From what I understand, we observed the following behaviour: , there are pods. The HPA scales up new pods, so we have in total. , pods are scaled down, so we have in total. HPA calculates the desired Replicas with . Since all scaling events happened within the policies , the policy value of kicks in and should allow scaling up to . But it only scaled to .

I think that there is a bug in the following code: See the comments in the following snippets for an explanation: With a of 2, the is as follows: I will think of a possible solution to the problem and would like to contribute once I've made some progress on that :)
In fact, I knew this was caused by the calculating method of "getReplicasChangePerPeriod" and mentioned it in the "Anything else we need to know" of this issue. What I wonder now is that if it is an expected action or it is just a bug. If it quite is a bug, I hope I could contribute as well.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale


Question: <del> #cloud-config write_files: - path: /opt/bin/regen-apiserver-list.sh permissions: 0755 content: | #!/bin/sh m=$(echo $(etcdctl ls --recursive /corekube/apiservers | cut -d/ -f4 | sort) | tr ' ' ,) mkdir -p /run/kubelet echo "APISERVER_IPS=$m" > /run/kubelet/apiservers.env echo "FIRST_APISERVER_URL=https://${m%%,*}:6443" >> /run/kubelet/apiservers.env - path: /opt/bin/download-release.sh permissions: 0755 content: | #!/bin/bash # This temp URL is only good for the length of time specified at cluster creation time. # Afterward, it will result in a 403. OBJECT_URL="CLOUD_FILES_URL" if [ ! -s /opt/kubernetes.tar.gz ] then echo "Downloading release ($OBJECT_URL)" wget "${OBJECT_URL}" -O /opt/kubernetes.tar.gz echo "Unpacking release" rm -rf /opt/kubernetes || false tar xzf /opt/kubernetes.tar.gz -C /opt/ else echo "kubernetes release found. Skipping download." fi - path: /run/config-kubelet.sh permissions: 0755 content: | #!/bin/bash -e set -x /usr/bin/mkdir -p /var/lib/kubelet cat > /var/lib/kubelet/kubeconfig << EOF apiVersion: v1 kind: Config users: - name: kubelet user: token: KUBELET_TOKEN clusters: - name: local cluster: insecure-skip-tls-verify: true contexts: - context: cluster: local user: kubelet name: service-account-context current-context: service-account-context EOF - path: /run/config-kube-proxy.sh permissions: 0755 content: | #!/bin/bash -e set -x /usr/bin/mkdir -p /var/lib/kube-proxy cat > /var/lib/kube-proxy/kubeconfig << EOF apiVersion: v1 kind: Config users: - name: kube-proxy user: token: KUBE_PROXY_TOKEN clusters: - name: local cluster: insecure-skip-tls-verify: true contexts: - context: cluster: local user: kube-proxy name: service-account-context current-context: service-account-context EOF coreos: etcd2: discovery: https://discovery.etcd.io/DISCOVERY_ID advertise-client-urls: http://$private_ipv4:2379,http://$private_ipv4:4001 initial-advertise-peer-urls: http://$private_ipv4:2380 listen-client-urls: http://0.0.0.0:2379,http://0.0.0.0:4001 listen-peer-urls: http://$private_ipv4:2380,http://$private_ipv4:7001 flannel: ip_masq: true interface: eth2 fleet: public-ip: $private_ipv4 metadata: kubernetes_role=minion update: reboot-strategy: off units: - name: etcd2.service command: start - name: fleet.service command: start - name: flanneld.service drop-ins: - name: 50-flannel.conf content: | [Unit] Requires=etcd2.service After=etcd2.service [Service] ExecStartPre=-/usr/bin/etcdctl mk /coreos.com/network/config '{"Network":"KUBE_NETWORK", "Backend": {"Type": "host-gw"}}' command: start - name: docker.service command: start drop-ins: - name: 51-docker-mirror.conf content: | [Unit] # making sure that flanneld finished startup, otherwise containers # won't land in flannel's network... Requires=flanneld.service After=flanneld.service Restart=Always - name: download-release.service command: start content: | [Unit] Description=Downloads Kubernetes Release After=network-online.target Requires=network-online.target [Service] Type=oneshot RemainAfterExit=yes ExecStart=/usr/bin/bash /opt/bin/download-release.sh - name: kubelet.service command: start content: | [Unit] Description=Kubernetes Kubelet Documentation=https://github.com/kubernetes/kubernetes After=network-online.target Requires=network-online.target After=docker.service Requires=docker.service After=download-release.service Requires=download-release.service After=apiserver-finder.service Requires=apiserver-finder.service [Service] EnvironmentFile=/run/kubelet/apiservers.env ExecStartPre=/run/config-kubelet.sh ExecStartPre=/usr/bin/ln -sf /opt/kubernetes/server/bin/kubelet /opt/bin/kubelet ExecStart=/opt/bin/kubelet  --address=$private_ipv4  --api-servers=${FIRST_APISERVER_URL}  --cluster-dns=DNS_SERVER_IP  --cluster-domain=DNS_DOMAIN  --healthz-bind-address=$private_ipv4  --hostname-override=$private_ipv4  --logtostderr=true  --v=2 Restart=always RestartSec=5 KillMode=process - name: kube-proxy.service command: start content: | [Unit] Description=Kubernetes Proxy Documentation=https://github.com/kubernetes/kubernetes After=network-online.target Requires=network-online.target After=docker.service Requires=docker.service After=download-release.service Requires=download-release.service After=apiserver-finder.service Requires=apiserver-finder.service [Service] EnvironmentFile=/run/kubelet/apiservers.env ExecStartPre=/run/config-kube-proxy.sh ExecStartPre=/usr/bin/ln -sf /opt/kubernetes/server/bin/kube-proxy /opt/bin/kube-proxy ExecStart=/opt/bin/kube-proxy  --bind-address=$private_ipv4  --kubeconfig=/var/lib/kube-proxy/kubeconfig  --logtostderr=true  --hostname-override=$private_ipv4  --master=${FIRST_APISERVER_URL} Restart=always RestartSec=5 - name: apiserver-finder.service command: start content: | [Unit] Description=Kubernetes Apiserver finder After=network-online.target Requires=network-online.target After=etcd2.service Requires=etcd2.service [Service] ExecStartPre=/opt/bin/regen-apiserver-list.sh ExecStart=/usr/bin/etcdctl exec-watch --recursive /corekube/apiservers -- /opt/bin/regen-apiserver-list.sh Restart=always RestartSec=30 - name: cbr0.netdev command: start content: | [NetDev] Kind=bridge Name=cbr0 - name: cbr0.network command: start content: | [Match] Name=cbr0 [Network] Address=10.240.INDEX.1/24 - name: nat.service command: start content: | [Unit] Description=NAT container->outside traffic [Service] ExecStart=/usr/sbin/iptables -t nat -A POSTROUTING -o eth0 -s 10.240.INDEX.0/24 -j MASQUERADE ExecStart=/usr/sbin/iptables -t nat -A POSTROUTING -o eth1 -s 10.240.INDEX.0/24 -j MASQUERADE RemainAfterExit=yes Type=oneshot </del>

Rackspace support has not ben updated in a few months. cluster/rackspace/cloud-config/minion-cloud- is using etcd_servers for kubelet and kube-proxy, but support for them has been removed since v0.9, I think. I'll remove it in a few days if no-one makes a counter-offer to get the config working with latest release.
There's been some work on this but it is blocked while we figure out some load-balancer design stuff. , Eric Tune wrote:
Okay, I will remove the link to rackspace until it is fixed (). Leaving this issue open to remind us to delete it in, say, a month, if no progress is made.
cc
See also
I'd prefer to try to get things updated and keep support. However I don't want to continually become stale. every few months either.
Hi all, I want to use kubernetes on my existing rackspace infrastructure but I don't understand if rackspace suppor was removed or not. In Doc, under "custom cloud solution" it's linked but I wasn't able to deploy it neither with install method nor with . With i got a error (I have 16GB or RAM!). thanks in advance, Fx
It looks like no support has happened on Rackspace for a year.
You should not expect it to work, and we should delete rackspace support.
ACK.
Unfortunately I think we'll need to drop support for Rackspace as a deployment target since I can't commit to keeping the deployment scripts up to date. Rackspace support as a cloudprovider should be maintained.
I would be willing to support the Rackspace deployment target, if there is interest to keep it alive.
There would be definitely an interest from our side - sad to see that Rackspace has no interest in maintaining it
No problem, I will take over the maintenance. It will take me a bit of time to get caught up, but I will get it done as soon as possible.
Do you think it would be hard to get the of LBaaS working ?
Sorry it took me a bit to respond. there is a quick update as to where I am. I was busy with some stuff, that prevented me from working on this. But that has been resolved, and should have updates by the end of next week.

However what's causing the delays is the lack of a good API for Rackspace Cloud. I've used the gophercloud, it works but I'm not a fan. Additionally gophercloud doesn't have full API support (one of the things I'm not a fan). This would make it hard to implement LBaaS. I should have a better idea of your request sometime next week.
I have seen this api docu: - is it the same as you saw? I think it should be doable with it.


Question: <del> #!/bin/bash # Copyright 2014 The Kubernetes Authors All rights reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # #     http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # Deletes pod, deletes secret set -o errexit set -o nounset set -o pipefail ORIGIN=$(dirname "${BASH_SOURCE}") KUBE_ROOT=$(dirname "${BASH_SOURCE}")/../.. ## Delete the services ${KUBE_ROOT}/cluster/kubectl.sh delete services origin-api ${KUBE_ROOT}/cluster/kubectl.sh delete services origin-ui ## Delete the pod ${KUBE_ROOT}/cluster/kubectl.sh delete pods openshift ## Delete the secret ${KUBE_ROOT}/cluster/kubectl.sh delete secrets kubernetes-secret </del>

This PR was prematurely merged: Need to update the PR based on the actual steps that were needed to get what was demonstrated at the public Kubernetes meet-up last week.
Any chance you can get rid of the hostPort from the example?
- depending on the Kubernetes provider, yes. For Vagrant based provider, I think I still need the hostPort because I do not have routes. For the GCE provider, I should be able to get rid of it because I have routes. In both providers, there were a number of setup steps that did not make the merged PR. For those that are tracking this issue (which I know folks have pinged me on separately) needs to be configured to allow privelaged containers needs to be configured to allow pushing to an insecure registry (this is point in time) Ideally, I could use a ReplicationController in the example to load balance OpenShift, but until we can provide a pattern to run etcd as a pod, I am unable to do this and need to use the embedded etcd we end up running.
Note: I will resolve this issue in 2 phases. Vagrant (which was demonstrated) to document using GCE (which has some additional steps on firewall rules)
Turns out OpenShift Origin is too secure now ;-) This is blocked until OpenShift Origin allows you to mint certs on the client, and then supply the certs on startup as a secret to the pod. This feature is coming soon. /cc
I now have all the required pieces to get this to work. I hope to get a PR submitted to fix this issue by end of this week.


Question: // that associates usage of the specified resource based on the number of items // returned by the specified listing function. type objectCountEvaluator struct { <del> // allowCreateOnUpdate if true will ensure the evaluator tracks create // and update operations. allowCreateOnUpdate bool </del> // GroupResource that this evaluator tracks. // It is used to construct a generic object count quota name groupResource schema.GroupResource

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug /sig api-machinery What happened: The that we use to make a create validation function is given admission.Update as its operation field. In the create validation we then use the provided operation (admission.Update) , when we call the admission controllers, even though a new object is being created. A similar thing is happening with the mutating admission. This means that when an objects is created on update, the admission controllers will see this as an update request. Meaning there could be an objects in storage which never passed create admission validation and were never modified by create admission mutation. I think this is a bug. What you expected to happen: Admission controllers would handle a create-on-update exactly the same as a normal create
I think the solution to this would be to separate the static admission attributes into one for the create validation and one for the update validation. Also, the function which performs the mutating admission needs to determine whether or not it is an actual update or if the old object doesn't exist (probably by checking uid) and then call the mutating admission correctly based on which operation is being performed.
/assign I might be interested to give it a try.
How to reproduce this bug?
Instead of a reproduceable BUG, I think it's like a mismatch of conceptual design and actual implementation. If a resource allows create-on-update in its strategy e.g.: Then admission plugins would receive operations (CREATE/UPDATE) on UPDATE requests. Some problems might happen the admission plugin is written without considering the circumstances that admitting a UPDATE operation might result a object creation into the storage. Risky and confusing, WDYT?
With the current build in admission controllers, I'm not sure if there are any bugs. Maybe something with resource quota? The real problem is that there is no way to know which webhook authors will have a dependency on seeing a "create" operation for every object put into storage. If a webhook relies on creates to implement its custom logic (which should be allowed), then objects created with an update would currently be missed by the webhook.


Question: } } <ins> // acquire lock for smb mount var getSMBMountMutex = keymutex.NewHashed(0) </ins> // Mount : mounts source to target with given options. // currently only supports cifs(smb), bind mount(for disk) func (mounter *Mounter) Mount(source string, target string, fstype string, options []string) error {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: I am using Windows containers (Server Core ltsc 2019). These all have volumes mounted to three different Azure file shares under a single storage account. Multiple instances of these containers run on each node. On deploying the containers I get the following errors. Sometimes this does manage to resolve itself. MountVolume.SetUp failed for volume "log-share" : Remove-SmbGlobalMapping failed: exit status 1, output: "Remove-SmbGlobalMapping : No MSFTSmbGlobalMapping objects found with property 'RemotePath' equal to rn'storageaccounturllogs'. Verify the value of the property and retry.rnAt line:1 char:1rn+ Remove-SmbGlobalMapping -RemotePath $Env:smbremotepath -Forcern+ ~rn + CategoryInfo : ObjectNotFound: (storageaccounturllogs:String) [Remove-SmbGlobalMapping], Ci rn mJobExceptionrn + FullyQualifiedErrorId : CmdletizationQueryNotFoundRemotePath,Remove-SmbGlobalMappingrn rn" MountVolume.SetUp failed for volume "config-share" : New-SmbGlobalMapping failed: exit status 1, output: "New-SmbGlobalMapping : Generic failure rnAt line:1 char:190rn+ ... ser, Env:smbremotepath -Cred ...rn+ ~~~rn + CategoryInfo : NotSpecified: (MSFTSmbGlobalMapping:ROOT/Microsoft/...mbGlobalMapping) [New-SmbGlobalMa rn pping], CimExceptionrn + FullyQualifiedErrorId : HRESULT 0x80041001,New-SmbGlobalMappingrn rn" What you expected to happen: For the volumes to be mounted. This all worked ok in k8s 1.12.1 using Windows 1803. I have now upgraded to 1.13.4 and Windows ltsc2019.

How to reproduce it (as minimally and precisely as possible): Use aks-engine to create a 1.13.4 cluster with Windows nodes Create a storage account Create multiple file shares under the account. Add access credentials to the storage account as a secret on k8s Create a windows container with folders that will be used to mount the Azure storage account file shares. Deploy to k8s defining volume mounts and volumes in the container section of the deployment e.g. volumeMounts: name: folder1-share mountPath: "c:appfolder1" name: folder2-share mountPath: "c:appfolder2" volumes: name: folder1-share azureFile: secretName: azure-file-share-cred shareName: share1 readOnly: false name: folder2-share azureFile: secretName: azure-file-share-cred shareName: share2 readOnly: true Anything else we need to know?: This maybe related to the fix for Environment: Kubernetes version (use ): 1.13.4 Cloud provider or hardware configuration: Azure OS (e.g: ): Windows 2019 Kernel (e.g. ): 10.0..316 Install tools: aks-engine 0.13.2 Others:
/sig Azure
/sig windows
/assign
hi in your logs, the remotepath is , so is the share name? also, could you log on to that agent node, and provide , there are logs like
I think this is a race condition issue, let me file a PR to fix this, thanks for reporting


Question: func (BasicReplicationController) ParamNames() []GeneratorParam { return []GeneratorParam{ {"labels", false}, <del> {"default-name", true}, {"name", false}, </del> <ins> {"default-name", false}, {"name", true}, </ins> {"replicas", true}, {"image", true}, {"port", false},

I execute:
Appears to be related to changes made in this PR:
Looks like the change here: should have a different set of param names than
Looks like they do have a different set of names, but for the wrong default was set.


Question: - name: SchedulingMigratedInTreePVs featureGates: <del> CSIMigration: true </del> CSIMigrationAWS: true workloadTemplate: - opcode: createNodes

/sig scheduling in the job is failing. success run Due to this failure, we cannot see any scheduler metrics results from the perf-dash. master
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-kind bug /kind flake /kind failing-test
/assign /remove-kind flake
will fixed by


Question: case newer.PodRunning: *out = PodRunning case newer.PodSucceeded: <del> *out = PodTerminated </del> <ins> *out = PodSucceeded </ins> case newer.PodFailed: *out = PodTerminated case newer.PodUnknown:

This happened to me several times while debugging an e2e test. My branch was based on commit and had changes only to
docker inspect says the container exited 0
kubelet HTTP says the pod exited 0:
kubectl says it failed:
Events does not show any success or failure even. Seems like it doesn't show that for any pods, succeeded or failed.
Apiserver sees the exit code 0:
The problem might be related to there being zero values for startedAt or finishedAt. The command may have exited so quickly that docker could not record these values, and we may be somehow treating unset values for these as failure.
adding sleep 5 to my command causes docker to set the startedAt and finishedAt to reasonable values, but the pod still shows up as failed. Note that I use a different pod name each time I try to avoid that sort of obvious mistake.
So, the apiserver v1beta1 message knows that it is terminated with exit code 0. But, both kubectl and my go code in my go e2e test thing this means failed. I'm starting to suspect the conversion code.
The problem is that a v1beta1 pod can never see that a pod is failed or succeded. So it defaults to PodFailed. Grrrrrrraaaaahhhhhh!
In line 184 of
This is going to be confusing for people unless we change this.
I believe the latest fix for PodStatus should resolve the issue here. I will double confirm it before closing it.

We should do this after 1.7. xref /sig testing cc
Is this the task that needs completed - Add support to Ubuntu e2e test-infra
That is different, this is about running GPU tests as part of PR pre-submit tests. Right now we skip them because they are "[Serial]".
I can work on this issue in conjunction with . Is this issue just a matter of adding GPU tests to a different suite or are there more subtasks?
Anymore progress on this?
Thanks for the reminder. I just submitted an internal request for quota to run these tests. Once that is approved, I will add a PR in test-infra to run GPU tests on all k/k PRs. It will be non-blocking initially and then become blocking later on.
Awesome, thanks!


Question: all: push push: push-spark push-zeppelin <del> .PHONY: push push-spark push-zeppelin spark zeppelin </del> <ins> .PHONY: push push-spark push-zeppelin spark zeppelin zeppelin-build </ins> # To bump the Spark version, bump the version in base/Dockerfile, bump # the version in zeppelin/Dockerfile, bump this tag and reset to

The image by is over a gig. Investigate how to trim this down. (Cut out interpretters? Make them more modular somehow, possibly composing them once we have some solution like ?) It's not even clear there's upstream support for making this more modular, though.
cc
remove mvn tooling and all the extra steps, once upstream releases 's like the rest of apache.
I can look at that, but I think I can actually just strip the image of the dev tooling right now. Right now I think the Zeppelin deps themselves are a ridiculous portion of this, though. (I'll poke at it.)
(3) is an install of [bzr git mercurial openssh-client subversion] in the base java image, which seems wasteful to me. you'll have to switch base images to get rid of this. (2) is your install of [curl net-tools build-essential git wget unzip python python-setuptools python-dev python-numpy libfontconfig]. i expect curl is at least redundant, as is git if your using the java image. you should merge this layer w/ (1) and do an uninstall of [python-setuptools python-dev etc] & deps at the end. of course, those are less significant than (1), which is the mvn build. the long term solution here is to use the upstream binaries, when they exist. in the meantime, you should clone zeppelin into tmp, copy out the resulting build artifact and delete the clone. that should at least cut the size of this layer in half, as all the intermediate build artifacts are likely still present and in the final tarball/jar.
I can trim it down some by cutting out the build machinery (and I'll put up a PR shortly), but here's what the looks like: And expanded: (Instead of just deleting, it's actually easy to build, copy out, compose, too.)
Is it me or are all the containers rather large. For example, does spark itself need to be a gig? 1.5.1v3 6 weeks ago 989.7 MB There's some spark binary package that's 73M uncompressed:
Java runtime + baseimage + transitive closer is what is causing most of the bloat. But as we have discussed on other tickets, these images need to move out of the main repository.

I'm afraid I don't understand "transitive closer". For example I see my JDK is around 280M, and the Spark binary is 73M. Then that leaves another 636M for base + whatever. I guess I'll have to look at the imagelayers if I'm really curious.
he means all the dependencies and their dependencies. if you'd like to tackle shrinking the spark image, please open an issue for it. we've done some work on this and you can use the example investigation above to help improve image sizes.


Question: const ( DefaultRootDir = "/var/lib/kubelet" <ins> // DEPRECATED: auto detecting cloud providers goes against the initiative // for out-of-tree cloud providers as we'll now depend on cAdvisor integrations // with cloud providers instead of in the core repo. // More details here: https://github.com/kubernetes/kubernetes/issues/50986 </ins> AutoDetectCloudProvider = "auto-detect" defaultIPTablesMasqueradeBit = 14

/kind feature What happened: Continuing the discussion from , we should deprecate (or remove since its alpha) the option to set . See comment and for more details. related issues: related kubernetes-dev thread: !topic/kubernetes-dev/UAxwa2inbTA EDIT: Deprecation plan: [x] v1.8 - deprecate auto-detect cloud provider feature in kubelet [ ] v1.9 - set no cloud provider as the default in kubelet but still allow opt in for auto-detect [ ] v1.10 - completely remove the option to use
/sig cluster-lifecycle
/area cloudprovider
cc
/assign wlan0
cc - Wanted to ask you if you had any thoughts for/against this issue?
/assign andrewsykim As discussed in our meeting I'll be following up with sig-node to see if we can deprecate this as it's going against what we want for external cloud providers
FYI
Would have been nice to keep this issue open to track progress in the next releases
Indeed, should be kept open. Next task is now targeting v1.9
Thanks I edited the issue description to include the deprecation plan going forward. Open to changes on the deprecation plan if we think it's too aggressive.
Great, I think this is a good timeline. We need to make sure we communicate it very well though can you add this to the draft release notes in kubernetes/features please?
Looks like you beat me to it , gave it a review.
hehe, yes, it at the same time as everything else ;)
Can you create a PR for this soon to get this early into v1.9 please?
[MILESTONENOTIFIER] Milestone Issue Current <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Escalate to the issue owners; move out of the milestone after 1 attempt. : Adding tests, refactoring, fixing old bugs. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
i took the liberty to move this along (filed ). Hope that's ok.
Thanks appreciate it. I think there were some discussions in the mailing list thread about v1.9 being too soon.

I think it largely depends on how many users actually take advantage of the auto-detect feature. It's not many from what I've seen but we should follow up and be sure.
Any recent discussion about that? I haven't seen any activity recently but I might just be subject to email overflow. I hope we can get this into v1.9 cc
I didn't follow any discussion related to this topic, but thought this is right move to decouple cloudprovider components from the core system. If I remembered correctly, only cloudproviders for GCE, AWS and Azura depend on the feature, which can be easily override. Any concerns for v1.9?
yep, GCE/AWS/Azure -
Please look at and give your LGTM
Done. Just better document in 1.9 release notes. LGTM
Thank you!
Reopening for the removal of the option in v1.10
please move this to 1.10 milestone


Question: cadvisorInterface := &cadvisortest.Fake{ NodeName: config.NodeName, } <del> containerManager := cm.NewStubContainerManager() </del> <ins> var containerManager cm.ContainerManager if config.ExtendedResources != nil { extendedResources := v1.ResourceList{} for k, v := range config.ExtendedResources { extendedResources[v1.ResourceName(k)] = resource.MustParse(v) } containerManager = cm.NewStubContainerManagerWithDevicePluginResource(extendedResources) } else { containerManager = cm.NewStubContainerManager() } </ins> endpoint, err := fakeremote.GenerateEndpoint() if err != nil {

A kubernetes cluster is normally constructed from a variety of machine templates. Different instance groups may have different CPU, memory capacity. They may be equipped with specialized accelerators, network devices. I think we should have an extensible way for users to custom node configuration to simulate real production cluster states. Currently, nodelabel and nodetaints are supported. I am consider to accept a template with more configuration like resources so we won't grow option list in the future. Without this feature, it's hard for users to simulate heterogeneous cluster.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/sig scalability


Question: return strings.HasPrefix(media, "text/") } <ins> // checkWait returns true along with a number of seconds if the server instructed us to wait // before retrying. func checkWait(resp *http.Response) (int, bool) { if resp.StatusCode != errors.StatusTooManyRequests { return 0, false } i, ok := retryAfterSeconds(resp) return i, ok } </ins> // retryAfterSeconds returns the value of the Retry-After header and true, or 0 and false if // the header was missing or not a valid number. func retryAfterSeconds(resp *http.Response) (int, bool) {

Storing r.req / r.resp internally (for use from DoRaw and Do) is probably not the right factoring. There should be a common method doRaw that returns request and body.


Question: ### **Main Steps** #### I. Make *kubernetes* , *etcd* and *flanneld* binaries <del> First clone the kubernetes github repo, `$ git clone git@github.com:GoogleCloudPlatform/kubernetes.git` </del> <ins> First clone the kubernetes github repo, `$ git clone https://github.com/GoogleCloudPlatform/kubernetes.git` </ins> then `$ cd kubernetes/cluster/ubuntu`. Then run `$ ./build.sh`, this will download all the needed binaries into `./binaries`.

In the step 1) of docs Clone repo kubernetes via ssh would fail: Better to change it to the .
hello, I am not the owner of k8s, but I can clone vis ssh. Maybe it's your github key issue.
In any case, the docs generally tell us to use https or so I've opened a PR to change it to https. Edit: Though I also have no issue with using ssh, it clones just fine.
Hmm so I've just tested this from a computer that has not previously interacted with github (a little debian server). It fails similarly when cloning over ssh, giving the same output as reported by We should still probably switch the docs to https.


Question: finalPods := int32(math.Ceil(resourcesUsedRatio * float64(startPods))) // To breach tolerance we will create a utilization ratio difference of tolerance to usageRatioToleranceValue) <del> tc := testCase{ </del> <ins> tc1 := testCase{ </ins> minReplicas:             0, maxReplicas:             1000, initialReplicas:         startPods,

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: Resync period for HorizontalPodAutoscaler controller is set to 15 seconds in order to update HPAs every 15 seconds. Sometimes updates are skipped and period between two updates is 30 seconds. What you expected to happen: HPA should be updated every 15 seconds. How to reproduce it (as minimally and precisely as possible): Add debugs to HPA controller showing how often HPAs are processed. Create HPA. Anything else we need to know?: HPA controller uses rate RateLimitingQueue: New requests are always with 15 seconds delay: If request for the given HPA is in the queue then newly request is skipped. Requests are and removed from the queue at the same moment so there is a race condition. See following scenario: 00:00:00 - Resync adds request to queue to be executed at 00:00:15 00:00:15 - Resync adds next request to queue to be executed at 00:00:30 but it is skipped because previous request is still in the queue 00:00:15 - Request at 00:00:00 is executed 00:00:30 - Resync adds next request to queue to be executed at 00:00:45 00:00:45 - Request at 00:00:30 is executed In above scenario HPA wasn't updated between 00:00:15 and 00:00:45. Environment: Kubernetes version (use ): HEAD Cloud provider or hardware configuration: GKE <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
/sig autoscaling
/assign krzysztof-jastrzebski


Question: p.podBackoff.ClearPodBackoff(nsNameForPod(pod)) } <del> // isPodBackingOff returns whether a pod is currently undergoing backoff in the podBackoff structure </del> <ins> // isPodBackingOff returns true if a pod is still waiting for its backoff timer. // If this returns true, the pod should not be re-tried. </ins> func (p *PriorityQueue) isPodBackingOff(pod *v1.Pod) bool { boTime, exists := p.podBackoff.GetBackoffTime(nsNameForPod(pod)) if !exists {

We've noticed a strange behavior of the scheduler in GCE 5K node scalability tests. There were periods of time, lasting about 6s, when scheduler didn't schedule any new pods. It's ruining our 5s pod-startup-latency SLO. See the example below, pod density-latency-pod-2325-dgqrv created on 09:08:59, scheduler didn't schedule it until 09:09:06. An interesting pattern can be found in the around that time. Before 09:09:00 scheduler schedules ~5 pods / s. Then it goes silent for about 6s and at 09:09:06 it outputs about 30 "bound successfully" statements and continues scheduling pods with ~5 pods / s pace. We checked api-server logs for that silent period of time and it turned out that scheduler was doing something, it was constantly sending GET and PUT request for a metrics-server and heapster pod it was trying to schedule. ! We dug deeper and found that scheduler was actually trying to schedule these two pods for over 12h, below is the plot of number of calls per minute that scheduler issued to api-server for these two pods: ! As we can see there are long periods of time when scheduler issues over 1500 requests per minute. That seems strangely a lot given that the requests are only for the two pods. During the whole run scheduler issued over 400K GETs and over 200K PUTs for these two pods. ! We figured out that scheduler wasn't able to schedule these two pods because there was not enough resources in the cluster. While it's understandable that scheduler wasn't able to schedule the pods, I'd expect scheduler to not spam the system with so many retries, which, as we can see in this example, can potentially block / slow-down scheduling of other pods. I've increased the scheduler log verbosity level in , we should have more logs from the tomorrow's run. Will post them here once available.
/sig scheduling
Reiterating the mentions to trigger a notification: <detailsIn response to : /sig scheduling </details
/assign for triage
What is the version of your cluster? If you are using the HEAD, we have a back-off mechanism to the scheduler to delay rescheduling unschedulable pods.

It starts with 1 second delay and doubles it every time the pod is not schedulable up to 10 seconds.
It's from head, so apparently something doesn't work...
Yes, we're using head. The test run on Jan 21 and the cluster was built from head on the same day (commit to be more precise). As you can see in the logs it doesn't look like scheduler is using any back-off mechanism. Definitely not the one you described, it issues requests to api server much more often than 1 per second per pod. !
An update on more logs - we won't have more logs. We tried increasing scheduler's log level to v4 and it resulted in the log growing by few orders of magnitude and it was clogged with mostly useless stuff, e.g. it logged "Node gce-scale-cluster-minion-group-XXX is a potential node for preemption." over 250 000 000 times. We reverted the log level change in If more scheduler logs are needed to debug this we probably should add some new, more useful, ones.
cc/
/assign I can look into this next Tuesday
Friendly ping, any updates?
Whoops. Thanks for the ping. Log level v4 should not be needed. Although the log line you called out should probably get pushed into V9... First off, that's some really great debugging. Thanks for the detailed report. If the scheduler is getting into tight busy loops, that could in theory starve the scheduling thread, causing the observed 6 second pause in scheduling. The bug is in  A pod only gets marked for backoff if . And can increment very quickly in a high churn cluster. This would result in a pod being retried in a tight loop. One (unrelated) observation that stood out to me was this: This is being printed by the package and shows a trace with no steps logged to it, which means that we are hitting one of these four error cases in . Most likely, the PVC for those heapster & metrics-server pods are in a bad/unexpected state.
I think fixes the root problem here. If we eliminate the API server spam, the main scheduling thread hopefully will not get blocked for 6 second periods. Are you able to test with this patch?
Thanks Jonathan!

I've reserved a scale-testing slot for Thursday (Feb 21) to test the fix. Will update this issue once I have the results.
I reproduced the issue (non-schedulable heapster pod due to not enough resources) on a 5K node cluster. Then I run the load test, first on the version without the fix and the on the version with the fix. Below are the graphs of the number of scheduler requests (for the unschedulable pod) per minute. Without fix: ! With fix: ! As we can see on the graphs, the fix definitely helps, without it there were minutes were scheduler was calling api server up to 2K times per minute (2K QPM). With the fix it didn't exceed 100 QPM. But even with the fix the average QPM is ~60 (per single pod) which is 1 query per second per pod on average. is average 60 QPM (1 QPS) reasonable here or should we expect something less from the rate-limiting mechanism?
By default the scheduler rate limit is 20 qps for all scheduler operations. Default backoff delay for unschedulable pods is 1 second and goes exponentially higher every time a pod is retried, but does not exceed 10 seconds. This 10 second max backoff time also means that if a pod is not retried for 10 seconds, for example because there are many other pods in front of it in the queue, its backoff time is reset to 1 second. So the next time it is retried and is found unschedulable it will be backed off for 1 second. I am basically trying to say that 1 qps is not unreasonable. We also an optimization () that should lower qps for unschedulable pods further down. This last PR got merged only a few hours ago. With that PR merged, I would expect to see even lower qps as the scheduler will not update pod status every time it retries scheduling a pod.
Thanks Bobby! That makes sense. In that case I believe we can close this issue once merges.
/milestone v1.14 We're entering and this looks relevant to the 1.14 release. Please if I am incorrect
Thanks for fixing!


Question: labels: k8s-app: kube-dns annotations: <del> seccomp.security.alpha.kubernetes.io/pod: 'docker/default' </del> <ins> seccomp.security.alpha.kubernetes.io/pod: 'runtime/default' </ins> spec: priorityClassName: system-cluster-critical serviceAccountName: coredns

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Apply security best practices for both kube-dns and core-dns. Low hanging fruit: [x] Run as non-root & disallow privilege escalation [x] Run with the default seccomp profile (i.e. ) [x] ReadOnlyRootFilesystem [x] Avoid unnecessary HostPath volumes [x] (n/a) More advanced: [ ] Reduce dependencies in the base image () [ ] Custom seccomp profile [ ] Custom AppArmor profile [x] Drop unneeded capabilities[x] Run as non-root & disallow privilege escalation [x] Run with the default seccomp profile (i.e. ) [x] ReadOnlyRootFilesystem [x] Avoid unnecessary HostPath volumes [x] (n/a) More advanced: [ ] Reduce dependencies in the base image () [ ] Custom seccomp profile [ ] Custom AppArmor profile [x] Drop unneeded capabilities This is based on the long term issue and implemented by PR . Why is this needed: Our system pods should run using security best practices, both to enhance cluster security and serve as examples of best practices to users.
/sig auth /area security


Question: trigerWatchEventTime time.Duration initAttachErr        *storage.VolumeError finalAttachErr       *storage.VolumeError <del> sleepTime            time.Duration </del> timeout              time.Duration shouldFail           bool }{ { name:         "attach success at get", initAttached: true, <del> sleepTime:    10 * time.Millisecond, </del> timeout:      50 * time.Millisecond, shouldFail:   false, }, { name:          "attachment error ant get", initAttachErr: &storage.VolumeError{Message: "missing volume"}, <del> sleepTime:     10 * time.Millisecond, </del> timeout:       30 * time.Millisecond, shouldFail:    true, },

/kind bug /sig storage What happened: CSI unit tests occasionally end up with data race: How to reproduce it (as minimally and precisely as possible): to get CSI unit tests into a binary: the binary 500 times in parallel until it breaks: . Use different argument to really load your machine, it should not be idle! . On my Linux, I got 3 failures in ~1000 runs. puts stderr of failed runs into .
/assign
I make a Fix: PTAL


Question: t.Errorf("%s: image pull policy was changed to %s", tc.name, a) } } <ins> } // TestUpdatePod ensures that this admission controller is a no-op for update pod if no // images were changed in the new pod spec. func TestUpdatePod(t *testing.T) { namespace := "testnamespace" name := "testname" oldPod := &api.Pod{ ObjectMeta: metav1.ObjectMeta{Name: name, Namespace: namespace}, Spec: api.PodSpec{ Containers: []api.Container{ {Name: "ctr2", Image: "image", ImagePullPolicy: api.PullIfNotPresent}, }, }, } // only add new annotation pod := &api.Pod{ ObjectMeta: metav1.ObjectMeta{ Name:      name, Namespace: namespace, Annotations: map[string]string{ "test": "test", }, }, Spec: api.PodSpec{ Containers: []api.Container{ {Name: "ctr2", Image: "image", ImagePullPolicy: api.PullIfNotPresent}, }, }, } // add new label and change image podWithNewImage := &api.Pod{ ObjectMeta: metav1.ObjectMeta{ Name:      name, Namespace: namespace, Annotations: map[string]string{ "test": "test", }, }, Spec: api.PodSpec{ Containers: []api.Container{ {Name: "ctr2", Image: "image2", ImagePullPolicy: api.PullIfNotPresent}, }, }, } tests := []struct { name         string kind         string resource     string subresource  string object       runtime.Object oldObject    runtime.Object expectError  bool expectIgnore bool }{ { name:         "update IfNotPresent pod annotations", kind:         "Pod", resource:     "pods", subresource:  "finalizers", object:       pod, oldObject:    oldPod, expectIgnore: true, }, { name:        "update IfNotPresent pod image", kind:        "Pod", resource:    "pods", subresource: "finalizers", object:      podWithNewImage, oldObject:   oldPod, }, } for _, tc := range tests { handler := admissiontesting.WithReinvocationTesting(t, &AlwaysPullImages{}) err := handler.Admit(context.TODO(), admission.NewAttributesRecord(tc.object, tc.oldObject, api.Kind(tc.kind).WithVersion("version"), namespace, name, api.Resource(tc.resource).WithVersion("version"), tc.subresource, admission.Create, &metav1.UpdateOptions{}, false, nil), nil) if tc.expectError { if err == nil { t.Errorf("%s: unexpected nil error", tc.name) } continue } if tc.expectIgnore { if e, a := api.PullIfNotPresent, pod.Spec.Containers[0].ImagePullPolicy; e != a { t.Errorf("%s: image pull policy was changed to %s", tc.name, a) } continue } if err != nil { t.Errorf("%s: unexpected error: %v", tc.name, err) continue } } </ins> }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: We enabled AlwaysPullImages admission in an existing cluster, and found annotations/labels/finalizers of existing Pods whose imagePullPolicy configured as could not be updated. AlwaysPullImages admission mutated containerSpec by . And this change later got denied by Pod validation. It also caused terminating Pods with finalizers stuck forever until we removed AlwaysPullImages admission. What you expected to happen: Existing Pods are not affected by AlwaysPullImages admission controller. How to reproduce it (as minimally and precisely as possible): the kube-apiserver with AlwaysPullImages admission controller disabled a Pod with imagePullPolicy configured as in its containers AlwaysPullImages admission controller annotation to Pod Anything else we need to know?: If this is confirmed a bug, I can submit a PR to fix this. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig api-machinery
seems to be a bug. /assign
/remove-sig api-machinery /sig node auth
try to fix it in When AlwaysPullImages is disabled, you cannot update imagePullPolicy of pod. The error message is the same as yours. imagePullPolicy is immutable. When AlwaysPullImages is enabled, old pod(with imagePullPolicy=ifNotPresent) update will change imagePullPolicy as well. Workaround: recreated pod -_-
Thanks for the fixes. I still have regarding the PR. IMHO, it's OK that imagePullPolicy is not updatable. Why don't we just ignore Update in the admission, but choose to update spec instead? Any concerns?

This is another fix option. I'm not sure which is better. The pod will be still ifNotPresent after updating, even if the AlwaysPullImages is enabled. Which is expected behavior?
In my case, I would expect the admission not to touch existing Pods. Just want to make sure if there's any concerns.
I would recommend ignoring pod updates in AlwaysPullImages if the update does not change the set of image names referenced by containers in the pod. Changing images inline in a pod update is unusual, and that change would allow unrelated updates (to annotations, labels, etc) to proceed
Agree.
is updated .
This issue is fixed by


Question: conn.Close() By("Waiting for the target pod to stop running") <del> waitErr := f.WaitForPodNoLongerRunning(pod.Name) </del> <ins> if err := f.WaitForPodNoLongerRunning(pod.Name); err != nil { framework.Failf("Pod did not stop running: %v", err) } </ins> <del> By("Retrieving logs from the target pod") </del> <ins> By("Verifying logs") </ins> logOutput, err := framework.GetPodLogs(f.Client, f.Namespace.Name, pod.Name, "portforwardtester") if err != nil { <del> framework.Failf("Error retrieving logs: %v", err) } if waitErr != nil { framework.Logf("Pod log:n%s", logOutput) framework.Failf("Pod did not stop running: %v", waitErr) </del> <ins> framework.Failf("Error retrieving pod logs: %v", err) </ins> } <del> By("Verifying logs") </del> verifyLogMessage(logOutput, "Accepted client connection") verifyLogMessage(logOutput, "Expected to read 3 bytes from client, but got 0 instead") })

https://k8s- Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
Two hunks (2x 10 "x"s) are not sent back. extends the logging. At least it will tell us whether the sender gets an error or not.
https://k8s- Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
https://k8s- Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
No more insights here. Pod log doesn't show anything special.
https://k8s- Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
https://k8s- Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite} Happened on a presubmit run in .
https://k8s- Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
https://k8s- Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite} Happened on a presubmit run in .
https://k8s- Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite} Happened on a presubmit run in .
I analysed this flake instance: https://k8s- It looks like the pfpod container terminates, the kubelet realizes that and wants to remove the network interface from the net ns using CNI via kubenet.

This fails two times (somehow TeardownNetwork is called multiple times, succeeds twice, but fails after that; compare about multiple TeardownNetwork calls). These failures interrupt the kubelet's tries to kill the pod. It succeeds 30 seconds later during reconciliation. But at that point the e2e test timed out already waiting for the pod to terminate.
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
This is the real problem, not another flake: This happens roughly once per week, pretty stable.
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
Builds: kubernetes-e2e-gke-staging-parallel Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
https://k8s- Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite} Happened on a presubmit run in .
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite} Happened on a presubmit run in .
[FLAKE-PING] This flaky-test issue would love to have more attention.

Builds: kubernetes-e2e-gce-etcd3 kubernetes-e2e-gci-gce-kubenet kubernetes-e2e-gci-gce-release-1.4 kubernetes-e2e-gci-gke-multizone kubernetes-e2e-gci-gke-prod-smoke kubernetes-e2e-gci-gke-staging-parallel kubernetes-e2e-gci-gke kubernetes-e2e-gke-staging-parallel kubernetes-e2e-gke Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
Builds: kubernetes-e2e-gci-gke-staging-parallel kubernetes-e2e-gke-staging-parallel Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
Builds: kubernetes-e2e-aws kubernetes-e2e-gci-gke-prod-smoke kubernetes-e2e-gke-staging-parallel kubernetes-e2e-gke Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite} Happened on a presubmit run in .
https://k8s- Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite} Happened on a presubmit run in .
[FLAKE-PING] This flaky-test issue would love to have more attention.

https://k8s- Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite} Happened on a presubmit run in .
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
Builds: kubernetes-e2e-gci-gke-release-1.4 kubernetes-e2e-gci-gke-staging-parallel kubernetes-e2e-gke Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite} Happened on a presubmit run in .
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
any updates on this issue
No, no further insights. Feel free to invest some time here with some new ideas.

[FLAKE-PING] This flaky-test issue would love to have more attention.
Builds: kubernetes-e2e-gce-multizone kubernetes-e2e-gci-gce-kubenet kubernetes-e2e-gci-gce-proto kubernetes-e2e-gci-gke-staging-parallel Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
Builds: kubernetes-e2e-gci-gce-proto kubernetes-e2e-gci-gke-release-1.4 kubernetes-e2e-gci-gke-staging-parallel kubernetes-e2e-gke-staging-parallel Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
Builds: kubernetes-e2e-aws kubernetes-e2e-gci-gce-kubenet Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.

This needs to be triaged as a release-blocker or not for 1.5
https://k8s- Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
Builds: ci-kubernetes-e2e-aws ci-kubernetes-e2e-gci-gce-kubenet Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
or Can you help us triage this as blocking or non-blocking for 1.5. It was originally marked as non-blocking for the 1.4 release.
The initial triage done by sttts@ is still valid even though we changed something in TeardownNetwork logic in 1.5 release. I don't think this should be release blocker with the same reason provided by sttts@ for 1.4 release. But instead of asking timestclair@ to take further look, I am assigning this to cc/
I marked this as release-blocker just for triage here. Thanks!
Thanks can you take a look and retriage if needed.
We have two flakes here: -- that's what and is about, i.e. Teardown logic. happened 3 times in last 8 days, much less the days before. IMO this is not a blocker. -- this seems to happen multiple times a day when a lot of e2e tests are running. This is the actual network flake. I will try to look into one of those to understand whether there is anything non-network related we can fix.
Here is a debug PR running the flaking test 100 times: None failed until now. Moreover, 200 runs locally do not fail.
Another observation: the number of returned is always a multiple of 10. The test container sleep for 100ms after each 10 . So, this might be a hint that there is some kind of race closing the connection.

Builds: ci-kubernetes-e2e-aws ci-kubernetes-e2e-gci-gke-multizone ci-kubernetes-e2e-gci-gke-prod-parallel ci-kubernetes-e2e-gci-gke-staging-parallel ci-kubernetes-e2e-gci-gke Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
Builds: ci-kubernetes-e2e-aws ci-kubernetes-e2e-gci-gce-release-1.4 ci-kubernetes-e2e-gke-staging-parallel Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
Builds: ci-kubernetes-e2e-aws ci-kubernetes-e2e-gce-release-1.5 ci-kubernetes-e2e-gci-gce-proto ci-kubernetes-e2e-gci-gce Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
Builds: ci-kubernetes-e2e-gce-etcd3 ci-kubernetes-e2e-gci-gce-etcd3 ci-kubernetes-e2e-gci-gce-kubenet Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
https://k8s- Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
Looks like this got a lot worse on 12/3 - https://k8s- Any chance this is related to
yes, it is most probably related to . In fact, was meant to make connection closing much less aggressive on the server side. It looks like our half-open connection support using socat is far from stable. I have in flight which is blocked by broken socat. Maybe that will help eventually to fix this flake.

Builds: ci-kubernetes-e2e-gce-etcd3 ci-kubernetes-e2e-gci-gce ci-kubernetes-soak-gke-test Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
Bumping to P0 - this is happening pretty much all the time now. yes, it is most probably related to . - can we revert that PR then? Or have a fix really soon, this is the biggest trigger of failures now.
we marked both tests as last night: This shouldn't happen anymore after rebasing PRs.
Here is a first attempt for a real fix:
https://k8s- Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
We will re-enable this test via This is also supposed to include a fix. Revert if the issue reappears.
https://k8s- Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
Is it appropriate to move this to the next milestone? (and remove the non-release-blocker tag as well)
Builds: ci-kubernetes-e2e-gci-gce-release-1.4 ci-kubernetes-e2e-gci-gce-release-1.5 Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
Builds: ci-kubernetes-e2e-gci-gke-release-1.4 ci-kubernetes-e2e-gci-gke-staging-parallel Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
Closing this issue as the corresponding test has been removed from cc:


Question: return err } <del> // Do not cascade deletion for overlapping deployments. // A Deployment with this annotation will not create or manage anything, // so we can assume any matching ReplicaSets belong to another Deployment. if len(deployment.Annotations[deploymentutil.OverlapAnnotation]) > 0 { return deployments.Delete(name, nil) } </del> // Stop all replica sets belonging to this Deployment. rss, err := deploymentutil.ListReplicaSetsInternal(deployment, func(namespace string, options metav1.ListOptions) ([]*extensions.ReplicaSet, error) {

The Deployment overlap annotation () is no longer populated nor used by 1.6+ on the server side. However, in we maintained knowledge of the annotation for 1.6 clients in case they talk to pre-1.6 servers. That PR also adds code to clean up the annotation whenever it's found in 1.6+ clusters, so that pre-1.6 clients don't see it and make the wrong assumptions. The code handling this annotation should be removed when all the following conditions are met: We no longer support a 1.5.x client talking to the latest server version. We no longer support upgrading the server directly from 1.5.x to the latest version. The soonest milestone that meets both conditions according to the is v1.8. This issue should be moved to that milestone once it exists.
TODOs: [x] deployment controller: [x] deployment util: [x] deployment reaper: [x] deployment describer:
I can help on this
/assign
GitHub didn't allow me to assign the following users: wanghaoran1988. Note that only can be assigned. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
thanks, feel free to send patches


Question: // and then use the same here devsLen := int64(2) var devicePluginPod, dptemplate *v1.Pod <ins> var v1alphaPodResources *kubeletpodresourcesv1alpha1.ListPodResourcesResponse var v1PodResources *kubeletpodresourcesv1.ListPodResourcesResponse var err error </ins> ginkgo.BeforeEach(func() { ginkgo.By("Wait for node to be ready") gomega.Eventually(func() bool {

Device Plugin [Feature:DevicePluginProbe][NodeFeature:DevicePluginProbe][Serial] DevicePlugin [Serial] [Disruptive] Can schedule a pod that requires a device It appears a bit flaky on testgrid, but more recently been failing more often No response example - No response /sig node
/cc
/triage accepted /assign /priority important-soon
I ran this test locally 15 times back to back and did not run into this failure. The test passed locally. I suspect this to be a case where pod cleanup is not happening properly. In this test, we are relying on the number of PodResources and based on the error it looks like we have an addtional pod running other than the two expected pods (device plugin pod and test pod requesting device) but can't be 100% sure because we don't have enough logs to support that claim. Going to add more logs to expose output obtained from the podresource API endpoint.


Question: fi popd >/dev/null <ins> if [ "${golangci_config}" ]; then # The relative path to _output/local/bin only works if that actually is the # GOBIN. If not, then we have to make a temporary copy of the config and # replace the path with an absolute one. This could be done also # unconditionally, but the invocation that is printed below is nicer if we # don't to do it when not required. if grep -q 'path: ../_output/local/bin/' "${golangci_config}" && [ "${GOBIN}" != "${KUBE_ROOT}/_output/local/bin" ]; then kube::util::ensure-temp-dir patched_golangci_config="${KUBE_TEMP}/$(basename "${golangci_config}")" sed -e "s;path: ../_output/local/bin/;path: ${GOBIN}/;" "${golangci_config}" >"${patched_golangci_config}" golangci_config="${patched_golangci_config}" fi golangci+=(--config="${golangci_config}") fi </ins> cd "${KUBE_ROOT}" res=0

This is a subset of "make verify doesn't work under anymore" xref: In two places we assume the path for "local builds" But when dockerized it will be at should work.
I'm not sure what the cleanest solution is here. Maybe a symlink in a fixed location that we update to point to whichever we just built ...
cc
Can we set env variables as part of ? golangci-lint doesn't support a search path for plugins yet, but perhaps that can be .
Bit of a blunt instrument but we could do that. Another blunt approach would be picking some unused path under _output to add a symlink to either I guess.
It would be really nice if we could just pass searchpaths for the plugins instead of having to pass a complete path.
relative paths are resolved relative to the config file, so you can indirectly change a root path on a fixed file by using different config paths and not have to edit the file contents ... we could use two symlinks in _output to the actual config file. Still not ideal.
This is one of the few remaining verify scripts that doesnt work on mac or otherwise dockerized.
Let's fix this by making the path absolute in a temporary copy of the golangci-lint config file: Syncing .git is relevant for testing PRs, but it seems that already ensured that it is available.
That sounds reasonable. Yeah, while working on that issue I found the other handful of verify scripts that are also broken when containerized for reasons unrelated to git. This is one of them, the others are linked in


Question: dnsPolicy: Default containers: - name: fluentd-cloud-logging <del> image: gcr.io/google_containers/fluentd-gcp:1.34 </del> <ins> image: gcr.io/google_containers/fluentd-gcp:1.35 </ins> # If fluentd consumes its own logs, the following situation may happen: # fluentd fails to send a chunk to the server => writes it to the log => # tries to send this message to the server => fails to send a chunk and so on.

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): no What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): fluentd timezone timeformat Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): What happened: Log lines from docker have the wrong timestamp. Anything else do we need to know: We are building a custom Docker Image for fluentd/td-agent which is based on your Dockerfile but with some slight changes. One of which is that we set the timezone to something else than UTC. In this case fluentd will produce wrong timestamps for log lines since the string only includes a simple Z in the end. Changing the time_format to fixed the problem for us. Of course this is not really a bug in kubernetes but changing the default to the above will probably help others in the future.
Thanks for reporting! It seems to be a bug


Question: // Find the number of references to the device. refCount := 0 for i := range mps { <del> if strings.HasPrefix(mps[i].Device, deviceNamePrefix) { </del> <ins> if strings.HasPrefix(mps[i].Path, deviceNamePrefix) { </ins> refCount++ } }

During iscsi volume cleanup, kubelet reads from and retrieves a list of device paths where iscsi volume mounts on. The device path was expected to be in the form of . But currently the device path in is in the form of , thus iscsi panicked . I suspect this panic started to happen after switched to use from . seems to always resolve device path (see output below), and thus the expected iscsi configuration in the device path no longer exists.


Question: return nil, err } <del> ipList := make([]string, len(ipInfoList.Items)) for i, ip := range ipInfoList.Items { ipList[i] = ip.Subsets[0].Addresses[0].IP </del> <ins> ipList := make([]string, 0, len(ipInfoList.Items)) for _, ip := range ipInfoList.Items { if len(ip.Subsets) > 0 && len(ip.Subsets[0].Addresses) > 0 && len(ip.Subsets[0].Addresses[0].IP) > 0 { ipList = append(ipList, ip.Subsets[0].Addresses[0].IP) } </ins> } klog.V(6).Infof("Current master IPs listed in storage are %v", ipList)

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: In kubernetes 1.15, Panic happends in kube-apiserver. Subsets[0] is nil when endpoints has no subsets The Panic happens in here: And everything returns to normal when we delete the endpoints in etcd What you expected to happen: Maybe kube-apiserver should print error,not panic How to reproduce it (as minimally and precisely as possible): The host's iowait of ETCD is high Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration:NA OS (e.g: ):Kernel (e.g. ):Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig release
/remove-sig release /sig api-machinery
/assign


Question: // Confirm desired public ip resource exists var pip *network.PublicIPAddress domainNameLabel := getPublicIPDomainNameLabel(service) <del> if pip, err = az.ensurePublicIPExists(service, desiredPipName, domainNameLabel); err != nil { </del> <ins> if pip, err = az.ensurePublicIPExists(service, desiredPipName, domainNameLabel, clusterName); err != nil { </ins> return nil, err } return pip, nil

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: We deployed two Kubernetes clusters to the same Azure Resource Group. Created services of load balancer type on both clusters. One of the services never received a public IP. Upon investigation, we noticed that the first Kubernetes cluster deleted the Azure Public IP that was created for the service by the other cluster. Apparently, all public IPs in the resource group are retrieved: then public IPs which don't match this criteria: are deleted. This logic besides the name of the kubernetes service should also consider the K8s cluster where the public IP originated from to avoid stepping on each other's toes when multiple clusters share the same resource group. What you expected to happen: In case multiple clusters run in the same resource group a cluster should delete only resources created by itself. How to reproduce it (as minimally and precisely as possible): Create two Kubernetes clusters (1.14) with different cluster names to the same resource group than create kubernetes services of load balancer type in each cluster. Anything else we need to know?: Environment: Kubernetes version (use ): 1.4 Cloud provider or hardware configuration: Azure OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
thanks for reporting the issue, this is the bug we should fix for sharing resource group. /sig azure /kind bug /assign


Question: return err }) }) <ins> ginkgo.It("should get and update a ReplicationController scale", func() { rcClient := f.ClientSet.CoreV1().ReplicationControllers(ns) rcName := "e2e-rc-" + utilrand.String(5) initialRCReplicaCount := int32(1) expectedRCReplicaCount := int32(2) ginkgo.By(fmt.Sprintf("Creating ReplicationController %q", rcName)) rc := newRC(rcName, initialRCReplicaCount, map[string]string{"name": rcName}, WebserverImageName, WebserverImage, nil) _, err := rcClient.Create(context.TODO(), rc, metav1.CreateOptions{}) framework.ExpectNoError(err, "Failed to create ReplicationController: %v", err) err = wait.PollImmediate(1*time.Second, 1*time.Minute, checkReplicationControllerStatusReplicaCount(f, rcName, initialRCReplicaCount)) framework.ExpectNoError(err, "failed to confirm the quantity of ReplicationController replicas") ginkgo.By(fmt.Sprintf("Getting scale subresource for ReplicationController %q", rcName)) scale, err := rcClient.GetScale(context.TODO(), rcName, metav1.GetOptions{}) framework.ExpectNoError(err, "Failed to get scale subresource: %v", err) framework.ExpectEqual(scale.Status.Replicas, initialRCReplicaCount, "Failed to get the current replica count") ginkgo.By("Updating a scale subresource") scale.ResourceVersion = "" // indicate the scale update should be unconditional scale.Spec.Replicas = expectedRCReplicaCount _, err = rcClient.UpdateScale(context.TODO(), rcName, scale, metav1.UpdateOptions{}) framework.ExpectNoError(err, "Failed to update scale subresource: %v", err) ginkgo.By(fmt.Sprintf("Verifying replicas where modified for replication controller %q", rcName)) err = wait.PollImmediate(1*time.Second, 1*time.Minute, checkReplicationControllerStatusReplicaCount(f, rcName, expectedRCReplicaCount)) framework.ExpectNoError(err, "failed to confirm the quantity of ReplicationController replicas") }) </ins> }) func newRC(rsName string, replicas int32, rcPodLabels map[string]string, imageName string, image string, args []string) *v1.ReplicationController {

[X] Test approval issue: [X] Test PR: 29 Sept 2022 [X] Two weeks soak end date: 14 Oct 2022 [X] Test promotion PR: According to this APIsnoop query, there are still two remaining ReplicationControllerScale endpoints which are untested for conformance. readCoreV1NamespacedReplicationControllerScalereplaceCoreV1NamespacedReplicationControllerScale Using a number of existing e2e test practices a new has been created for two ReplicationControllerScale endpoints. The e2e logs for this test are listed below. This query shows which scale endpoints are hit within a short period of running this e2e test If a test with these calls gets merged, test coverage will go up by 2 point This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: artifacts="${ARTIFACTS:-"/tmp/_artifacts/$(date +%y%m%dT%H%M%S)"}" remote=${REMOTE:-"false"} remote_mode=${REMOTE_MODE:-"gce"} <del> container_runtime_endpoint=${CONTAINER_RUNTIME_ENDPOINT:-""} </del> <ins> container_runtime_endpoint=${CONTAINER_RUNTIME_ENDPOINT:-"unix:///run/containerd/containerd.sock"} </ins> image_service_endpoint=${IMAGE_SERVICE_ENDPOINT:-""} run_until_failure=${RUN_UNTIL_FAILURE:-"false"} test_args=${TEST_ARGS:-""}

$ make test-e2e-node fails with an error because of the docker deprecation. I think we have to fix the default options for helper scripts. $ make test-e2e-node succeeds in common cases by default. $ make test-e2e-node No response_ <details</details<details</details<details</details<details</details<details</details<details</details/assign
/sig node
/cc
/assign
/triage accepted /assign /priority important-soon


Question: const ( // AffinityTypeNone - no session affinity. <del> GCEAffinityTypeNone GCEAffinityType = "None" </del> <ins> GCEAffinityTypeNone GCEAffinityType = "NONE" </ins> // AffinityTypeClientIP is the Client IP based. GCEAffinityTypeClientIP GCEAffinityType = "CLIENT_IP" // AffinityTypeClientIP is the Client IP based.

Hi, I have a problem with my GCE cluster about LoadBalancer type Services It's creating a lot of unused static external IP on the google network of my project the cluster is running kubernetes v1.1.2. Tried restarting nodes, tried destroy/recreate of the entire cluster.
Seeing exactly the same issue at GKE since a few hours. First on a v1.0.7 cluster which I later upgraded to v1.1.2 to fix the problem but the issue is still there. Double checked my yaml config files but no changes from our side. Existing loadbalancers do work as before.
I dealt with this last night and a workaround was to add to the spec
Thx :+1:
We'll get this cherry picked into the two release branches once it's been verified at HEAD.