Question: return fmt.Sprintf(`{"kubernetes.io/service-name":"%s"}`, serviceName) } <del> // makeNodesHealthCheckName returns name of the health check resource used by </del> <ins> // MakeNodesHealthCheckName returns name of the health check resource used by </ins> // the GCE load balancers (l4) for performing health checks on nodes. <del> func makeNodesHealthCheckName(clusterID string) string { </del> <ins> func MakeNodesHealthCheckName(clusterID string) string { </ins> return fmt.Sprintf("k8s-%v-node", clusterID) }

A couple of gce-scale-performance runs failed due to failing to delete the network, which in turn was due to a couple of leaked firewall-rules (I manually deleted the network later to fix it): Ref: https://k8s- cc (assigning to you, feel free to reassign as apt)
I digged up a bit and seems like the firewall is created by the service-controller as part of , which should've been deleted in but wasn't. Feel free to close the bug if you think it's just a one-off failure.
Took a quick look, both and runs have these firewall rules initially but doesn't, which indicates something happen in the run. But oddly there is no build log, logs in artifacts are empty as well. Likely that run is badly broken so cluster got terminated forcibly in the middle, and resource are leaked. Will observe this suite a bit longer. /assign /unassign
I think it wasn't caused by this job. We're running this one () and on alternating days in the project. And they both create clusters (network, etc) with the same name. And I think of gce-scale-correctness leaked the firewall (and hence the network) as it runs some correctness tests around load-balancer. That likely caused this job to fail.
Ref The resource leak was likely caused by test timeout --- GCE operations took too long to finish and the cluster was terminated ahead. Because heathcheck and firewall cleanup in gce controller are coupled with targetpool deletion, deleting cluster while targetpool deletion is pending will leak the resources.
Fyi - this affected gce-scale-performance again (ref: run 35 of https://k8s-): Is there a way to make sure they get deleted even in case of a timeout? I have a dirty hack in my mind - of deleting those as part of kube-down. Maybe you have some better suggestions?
Consider the frequency this happens, I think it makes sense to implement some cleanup logic within the e2e test. Will give it a try.
Turned out we already have such cleanup logic, it is just we use an incorrect firewall name and forget to cleanup node health check resources as well. Sent to fix.
Thanks for fixing!
Sure thing, thanks for reporting!
Reopening as it seems like the issue is not fixed yet.

For e.g run-16 of https://k8s- failed due to this: This is also affecting our job, both of them being release-blocking jobs. Could you PTAL?
By any chance we could make and use different names so that it will be easier to target the leaking point? Also, in that way one leaking job will not affect the other. Or maybe we could also make different runs use different network names? The scalability job is expensive so it might be great if newer runs wouldn't be affected by the leaked resources from previous run. I haven't figured out how exactly and when the resource are leaked, but given that the operation cycle of service controller and e2e test process proceed in an asynchronous manner (plus always fail ESIPP tests ), there might be chances we will leak resource even after we put cleanup logic in e2e test. Will take a deeper look later. Also what you suggested before --- deleting those as part of kube-down --- may also be a viable solution.
I thought about this option, but it's better for them to share the same name for network and other resources (instance-prefix, etc) because that way we ensure that if one of them leaks some resources, the other will delete them during it's startup. Else we'll run out of quota in the scale-testing project (for e.g wrt routes or VMs) by trying to start another 5k-node cluster while the previous one still exists. I went to pantheon and deleted the network and that went through successfully (though fails). Wonder if the delete command in kube-down can be modified to delete all sub-resources of the network (like firewalls, etc) before deleting it?
Sounds reasonable to me. We may need yet another env to trigger it? As deleting everything correlated to that network sounds a bit dangerous.
Well, took another look, seems like the actual thing that prevented us from deleting the network was mostly firewall rules, and deleting those sounds safe enough if we are going to delete the network anyway. Will try to make a PR for this.
Thanks for fixing the issue with firewalls. However seems like the latest run of gce-scale-correctness still leaked the network (but doesn't seem due to firewall this time): ref: https://k8s-
I thought we're deleting the subnets in this case too when we're using custom subnets mode.

Yep, we are supposed to. I think the scale CI hit the same issue as described. I've sent a PR for fixing it
For the time-being I manually deleted the subnets and then the network to unblock the next scale job run. But it would be good to have this fixed soon. cc
Closing this issue as we're now deleting all remaining firewall rules in the network before deleting the network itself in kube-down. For other networking resources in general, we have issue .


Question: // some sort of solution to above problem. // kill remaining active pods wait := sync.WaitGroup{} <ins> errCh := make(chan error, int(active)) </ins> wait.Add(int(active)) for i := int32(0); i < active; i++ { go func(ix int32) { defer wait.Done() if err := jm.podControl.DeletePod(job.Namespace, activePods[ix].Name, &job); err != nil { defer utilruntime.HandleError(err) <ins> glog.V(2).Infof("Failed to delete %v, job %q/%q deadline exceeded", activePods[ix].Name, job.Namespace, job.Name) errCh <- err </ins> } }(i) } wait.Wait() <ins> select { case manageJobErr = <-errCh: if manageJobErr != nil { break } default: } </ins> // update status values accordingly failed += active active = 0

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--What keywords did you search in Kubernetes issues before filing this one? Job, resourcequota, scheduler Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-04-03T20:44:38Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-04-14T13:36:25Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"} Environment: CentOS bare metal () Cloud provider or hardware configuration: - OS (e.g. from /etc/os-release): CentOS 7 Kernel (e.g. ): Linux <server name3.10.0-514.2.2.el7.x8664 SMP Tue Dec 6 23:06:41 UTC 2016 x8664 x8664 x8664 GNU/Linux Install tools: - Others: - What happened: Having created a resourcequota (e.g. 400M) and creating enough memory-limited jobs to go over that quota (e.g. 4 x 200M limit jobs), at first, two jobs are created and executed as expected, and two jobs fail at first due to memory quota: Events: FirstSeen LastSeen Count From SubObjectPath Type Reason Message --------- -------- ----- ---- ------------- -------- ------ ------- 4s 4s 1 job-controller Warning FailedCreate Error creating: pods "memhog-y152c55u-" is forbidden: exceeded quota: memoryquota, requested: memory=200Mi, used: memory=400Mi, limited: memory=400Mi After the first two jobs have been completed, instead of starting the remaining jobs as soon as possible, the remaining two jobs stay unstarted in this state for many, many hours until they finally start and complete.

First job: Start Time: Fri, 28 Apr 2017 15:00:08 +0300 One of the remaining 2 jobs: Start Time: Sat, 29 Apr 2017 07:13:34 +0300 What you expected to happen: I'm expecting the two waiting jobs to start executing as soon as the resources they need are available, e.g. as soon as the first jobs are completed. How to reproduce it (as minimally and precisely as possible): Bring up a kubernetes cluster, then execute these commands: $ kubectl create quota memoryquota --hard=memory=400Mi (cat /dev/urandom head -n 1)' --replicas=1 --restart=OnFailure --image-pull-policy=IfNotPresent --limits=memory='2'm run- $ chmod +x run- $ for i in {1.4}; do ./run- 200 100; done job "memhog-0ju2eiuy" created job "memhog-t8qyt5sj" created job "memhog-ljaqz4ek" created job "memhog-eb8q8kr3" created Wait for around 30 seconds and the first jobs should be complete: $ kubectl get jobs NAME DESIRED SUCCESSFUL AGE memhog-0ju2eiuy 1 1 43s memhog-eb8q8kr3 1 0 42s memhog-ljaqz4ek 1 0 43s memhog-t8qyt5sj 1 1 43s $ kubectl describe jobs . However, the remaining jobs don't start at all as shown by: $ kubectl describe job memhog-ljaqz4ek Name: memhog-ljaqz4ek Namespace: test Selector: controller-uid=-2f00-11e7-8766- Labels: run=memhog-ljaqz4ek Annotations: run memhog-ljaqz4ek memhog -r100 100m --replicas=1 --restart=OnFailure --image-pull-policy=IfNotPresent --limits=memory=200Mi,cpu=100m --record=true --image=derekway.

Parallelism: 1 Completions: 1 Pods Statuses: 0 Running / 0 Succeeded / 0 Failed Pod Template: Labels: controller-uid=-2f00-11e7-8766- job-name=memhog-ljaqz4ek run=memhog-ljaqz4ek Containers: memhog-ljaqz4ek: Image: derekwaynecarr/memhog Port: Command: memhog -r100 100m Limits: cpu: 100m memory: 200Mi Environment: <noneMounts: <noneVolumes: <noneEvents: FirstSeen LastSeen Count From SubObjectPath Type Reason Message --------- -------- ----- ---- ------------- -------- ------ ------- 3m 3m 1 job-controller Warning FailedCreate Error creating: pods "memhog-ljaqz4ek-" is forbidden: exceeded quota: memoryquota, requested: memory=200Mi, used: memory=400Mi, limited: memory=400Mi Furthermore, pods related to the remaining jobs aren't even created and only the completed pods are shown: $ kubectl get pods -a NAME READY STATUS RESTARTS AGE memhog-0ju2eiuy-bz0zm 0/1 Completed 0 7m memhog-t8qyt5sj-q9msw 0/1 Completed 0 7m Anything else we need to know: I've also created a stackoverflow question about this (), and asked about this in the kubernetes slack, where I was urged to create a bug report.
I will try to reproduce. The creation of a pod will be blocked by quota. In the absence of a pod, the scheduler has nothing to schedule, so it is not a scheduling bug. It's possible the issue is related to back-off in the job controller not attempting to create pods again. The event shows that it had been 3m since the job controller attempted to create pods again for the requisite job which implies some type of back-off. None of the controllers are quota aware, so I would anticipate they should just keep trying at reasonable intervals at this point. If you describe quota after the first 2 pods are completed, do you see quota usage being replenished as expected?
I would like to fix the bug, could you assigin it to me?

if pod create failed, jobcontroller.managejob() need return error, will requeue podcreate task.
-- maybe you have more insight, but it looks to me just reading the code that when a job controller is unable to create a pod due to lack of quota, its not actually an error that forces a re-queue of the job. in addition, i cant actually see when a full resync of the job work queue happens outside of the informer sync interval which is probably really long. see: see:
- feel free to open a pr, please cc me on your solution. i would like to understand from the job folks if there should be a periodic resync interval aside from the need to requeue the job when all pods cannot be spawned (which i assume your pr will handle via exponential backoff)
-- we will want to have a fix for this downstream as well.
Yes, after the first two jobs are complete, shows that 0M out of 400M is in use. I'm also able to create new jobs, up to the memory limit so 2 at a time, that execute as expected. Any jobs over that at a time will stay in the same waiting state.
- Yes, I want handle it via exponential backoff. IMO, MaxOfRateLimiter had implemented a periodic resync, we don't need another one.
I'll make sure to cherry-pick this to openshift ( fyi).


Question: func (r *Reachability) PrintSummary(printExpected bool, printObserved bool, printComparison bool) { right, wrong, ignored, comparison := r.Summary(ignoreLoopback) if ignored > 0 { <del> framework.Logf("warning: the results of %d pod->pod cases have been ignored", ignored) </del> <ins> framework.Logf("warning: this test doesn't take into consideration hairpin traffic, i.e. traffic whose source and destination is the same pod: %d cases ignored", ignored) </ins> } framework.Logf("reachability: correct:%v, incorrect:%v, result=%tnn", right, wrong, wrong == 0) if printExpected {

What happened: What you expected to happen: How to reproduce it (as minimally and precisely as possible): Run the netpol test suite. Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/triage accepted
/sig network


Question: return nil, fmt.Errorf("netlink is not supported in this platform") } <ins> // GetAllLocalAddressesExcept is part of interface. func (h *netlinkHandle) GetAllLocalAddressesExcept(dev string) (sets.Set[string], error) { return nil, fmt.Errorf("netlink is not supported in this platform") } </ins> // Must match the one in proxier_test.go func (h *netlinkHandle) isValidForSet(ip net.IP) bool { return false

Hey, after upgrading kubernetes from 1.22 to 1.23, I'm experiencing strange service provisioning behavior using nodePort. I'm using kube-proxy (ipvs) + calico + metallb. If external IP is used by another service type (Loadbalancer - allocated by metallb), then the node owning this IP doesn't listen to NodePort on this interface. Unfortunately, I can't find any changes that could be causing this, any ideas? Kubernetes 1.23 Missing externalip1 here: Using k8s 1.22 service with nodeport caused listening on every interface, despite using external IP by another service. Kubernetes 1.22 Upgrade k8s from 1.22.15 to 1.23.15 No response <details</details<detailson-premise instances </details<details</details<detailskubeadm </details<detailsdocker </details<detailscalico v3.24.5 metallb v0.13.7 ipset v7.5, protocol version: 7 ipvsadm v1.31 2019/12/24 (compiled with popt and IPVS v1.2.1) #ipvs config: </details
/sig network
/assign
From : This is not a valid setup. A virtual IP (VIP), like the loadBalancerIP, must not be assigned to a physical interface. The reason to assign a VIP to a physical interface would be to attract traffic with L2 mechanisms, ARP for IPv4 and neighbor discovery for IPv6, but this way is flawed for several reasons. That's why you use metallb in L2 mode instead. Metallb in L2 mode answers L2 requests without_ assigning the VIP to any interface.
When you send traffic to a nodePort, the destination address should be a node address, not a loadBalancerIP.
There is however a change in the implementation (I will try to find the PR) in the way the nodePort addresses are computed. This is how i should be; On all clusterIP, loadBalancerIP and externalIPs are assigned. So the current implementation is correct. What I suspect that pre v1.23 K8s did this wrong by adding all addresses on interfaces except kube-ipvs0 (including loopback which would not work).

The reason to assign a VIP to a physical interface would be to attract traffic with L2 mechanisms, ARP for IPv4 and neighbor discovery for IPv6, but this way is flawed for several reasons. That's why you use metallb in L2 mode instead. Metallb in L2 mode answers L2 requests without assigning the VIP to any interface. These loadBalancerIP addresses are just the external IP of my nodes, which I use in metallb as an available pool for service(loadbalancer). Provider automatically assigns this address to the physical interface as a lease, I don't really have the option of a different setup at the moment. Ye, I know. In my case the public IP address pool for node and loadBalancerIP addresses are the same
So, the loadBalancerIPs assigned by metallb is actually the real addresses of your K8s nodes?
Yes, these are public IP addresses (each node has its own), which I use as an entry point for some services(loadbalancer) but I also have an external loadbalancer that targets nodePort. To eliminate SPoF, it directs traffic to the service (NodePort), which has been listening on all nodes so far, regardless of IP assignment to another service
Is this setup supported? Thanks for explaining. I understand your setup now, . I am unsure if it's supposed to work, so I have to ask. Have you tested proxy-mode=iptables? It may work, but I wouldn't be suprised if it doesn't.
I will try to test it with iptables, but I use ipvs because of specific services, so I can use the load balancing algorithm - least connection... this method won't solve my problem :(
Found the PR
Why do you need nodePorts services at all? Can't you use only loadBalancer? NodePort is mainly intended to be used by and external load-balancer, i.e. like in AWS, but when you manage your own cluster you usually don't have an external loadBalancer so use only loadBalancerIPs makes sense. We do that, and even .
Scratch that! In your case where the node addresses are the external addresses you should use NodePort only. If you use your node addresses as loadBalancerIP you will hit this PR when you upgrade to later K8s versions.

It inserts a blocking rule in the INPUT chain for loadBalancerIPs. This will block input to your node if loadBalancerIP==nodeIP.
I understand, thanks for the info. In my case it's a breaking change and I will have to migrate some services to NodePort before kubernetes upgrade on prod environment.
with proxy-mode=iptables, everything started working, node is listening on nodePort and I can get to loadBalancerIP, I cleared the rules on each node, just to be sure However this is not a solution for me, I need the lc algorithm from ipvs
Port used in the LoadBakancer service example is within the default NodePort range. I think it can be converted to a NodePort service without affecting external users. Or is just an example? Or are there other reasons for type:LoadBalancer?
/area ipvs
This is just an example, I won't have much trouble with the migration. At most, I will divide the available lease of IP addresses into smaller pools, for NodePort / Loadblancer services. I think I already know everything and there should be no more surprises. Thanks for the information and your time!
It is a little "unusal" to use the host IP as the LB IP, but it's unfortunate that we broke something that used to work. Lars, is there any path back to "working" that doesn't lose the protections in that linked PR? I could easily see iptables also breaking this..
To not exclude loadBalancerIPs (and ecternalIPs) when they are on a physical interface is not hard, but messes up the nice set-operation. The access protection in It should basically not be there if the any loadBalancerIPs (and ecternalIPs) is also on aphysical interface. Can be done, but may be messy.
/triage accepted Even though this issue is accepted, it's still under investigation. I will check the code if the old bevaivor can be restored in a fairly simple way. But there is a problem with back-port. If I make a PR now it will be in v1.26.x or v1.27 but the function was broken in v1.23 so a fix should really be back-ported all the way back.

I would prefer to document that a real address on an interface on the node can't be used as a virtual address, like loadBalancerIP or externalIP.
Just wanted to chime in and say that I've also encountered this issue... very grateful to find the discussion and looking forward to a solution. Thank you all! Also just to share, I've tried the following 3 workarounds and they all work in my case: Switching to iptables mode. It works I think because the iptables NAT rule refers to the hostname as the destination, which gets resolved to the public node IP. Disable IPv6 on my OS. This works too as it makes the code fall back to the single stack mode, which grabs the node IP also from looking up the hostname. Explicitly configure the field in the kube proxy config, that overrides the default get local addresses.
I backed down to K8s v1.22.4 and I can't get it to work unless you have the master outside the cluster (i.e. not a node), or have a setup where the external-ip you set as loadBalancerIP is not on the main K8s network, something like; <img src="" width="65%" /K8s is using the "backend" network, the "external" ips are on the "frontend" network. The reason is when you assign a loadBalancerIP, that address will be assigned to the interface, even before v1.23. So, on an in-cluster master node you will see something like; The address is the loadBalancerIP and is also the node-address of another node in the cluster, "vm-002". When the master receives "normal" packets from "vm-002" they will have src=192.168.1.2 but since that address is assigned to a local interface on the master the packets will be discarded as or spoofed packets. The result is that "vm-002" (e.g. on that node) loses contact with the master;
I discovered this when testing PR I think I can restore the pre v1.23 behavior now, but I suffer from the problem described above.
I don't think your problem is precisely the same. To disable ipv6 or set nodePortAddresses can't really be used as work-arounds for this issue.

Thank you, You're right about IPv6 - disabling it didn't really help. I forgot to remove the I put in while testing with disabling IPv6. It was the that makes the difference. That said, it really appears to work if we explicitly put in the . Maybe I'm still missing something in the code, but if I read it correctly, here is where we take out the load balancer IP: It is called from the folllowing, but only if contains any zero CIDRs. If a node port address is explicitly provided and it is NOT a zero CIDR, then will not be called. We're planning to roll out this workaround before upgrading to a future Kubernetes release with your fix. We would much appreciate it if you could shed more light why it wouldn't work. We didn't want to change to iptables mode if this second workaround really works. Thank you so much!
You are right. To explicitly set in the kube-proxy config works as a work-around. Still the loadBalancerIP (which is also an address on a node) can't be used by the main k8s network, or the master must not be a node in the cluster as described in And... This will work up to and including K8s v1.25.x, but NOT v1.26.0! I haven't checked yet but I think it's because of
Since the loadBalancerIP that is owned by a node is assigned to on other nodes you must stop those other nodes from responding to ARP. For example with:
From my perspective, this is not a big , I have the whole architecture implemented in separate networks. I think it's worth adding this information somewhere in the documentation.
Thank you so much for the response, epesically the info about v1.26. We'll check it out for sure and will report back. Our node port use case is for single-node deployments only, so luckily there are no other nodes.


Question: func (wc *watchChan) sendEvent(e *event) { if len(wc.incomingEventChan) == incomingBufSize { <del> glog.V(2).Infof("Fast watcher, slow processing. Number of buffered events: %d."+ </del> <ins> glog.Warningf("Fast watcher, slow processing. Number of buffered events: %d."+ </ins> "Probably caused by slow decoding, user not receiving fast, or other processing logic", incomingBufSize) }

https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite} Previous issues for this test:
This is pretty much the same failure, that are happening in "gce-enormous" suite and we currently debugging them with The main reason of those failures is that we are terminating all watchers for a particular resource type from time to time, and some of them are actually causing a snowball effect. After more looking into it, my hypothesis is that our machinery for aggregating watchers in apiserver sometimes is just to slow. In particular, I some log in my local branch and what I'm observing is that in etcd_watcher, we are: accumulating objects in incoming channel but also accumulating objects in outgoing channel, and the accumulation there is usually higher [I will send a PR to put those logs into master - they seem to be pretty useful for debugging such issues.] So my current hypothesis is that dispatching watch events to watchers is what is too slow. We will continue debugging.
The logs that I seem to confirm my hypothesis - outgoing channel is generally more full then the incoming channel in etcd_watcher: vs I'm working on the PR that should at least improve the situation.
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.

https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
Please apply the label to this issue if it does not need to be fixed before we launch 1.4. Otherwise please advise on how to proceed with respect to 1.4.
This was a regression - I fixed it by reverting an offending PR: This seems to be fixed now.
I have 33 pods in my container (1 is substantive, the other 32 just run on log files). I get this all the time now. I'm running 1.5.4.
- to be honest, I don't undertsand what your wrote. what is "container" in "33 pods in my container"? Also, if you believe there is a bug somewhere, please open a new issue instead of adding comment to more than half-year-old one.


Question: framework.Failf("Flunders client creation response was status %d, not 201", statusCode) } <ins> pods, err := client.CoreV1().Pods(namespace).List(metav1.ListOptions{}) framework.ExpectNoError(result.Error(), "getting pods for flunders service") </ins> // kubectl get flunders -v 9 // curl -k -v -XGET https://localhost/apis/wardle.k8s.io/v1alpha1/namespaces/default/flunders <del> contents, err := client.Discovery().RESTClient().Get().AbsPath("/apis/wardle.k8s.io/v1alpha1/namespaces/default/flunders").SetHeader("Accept", "application/json").DoRaw() </del> <ins> contents, err := restClient.Get().AbsPath("/apis/wardle.k8s.io/v1alpha1/namespaces/default/flunders").SetHeader("Accept", "application/json").DoRaw() </ins> framework.ExpectNoError(err, "attempting to get a newly created flunders resource") var flundersList samplev1alpha1.FlunderList err = json.Unmarshal(contents, &flundersList) <del> framework.ExpectNoError(err, "Error in unmarshalling %T response from server %s", contents, "/apis/wardle.k8s.io/v1alpha1") </del> <ins> validateErrorWithDebugInfo(f, err, pods, "Error in unmarshalling %T response from server %s", contents, "/apis/wardle.k8s.io/v1alpha1") </ins> if len(flundersList.Items) != 1 { framework.Failf("failed to get back the correct flunders list %v", flundersList) } // kubectl delete flunder test-flunder -v 9 // curl -k -v -XDELETE  https://35.193.112.40/apis/wardle.k8s.io/v1alpha1/namespaces/default/flunders/test-flunder <del> _, err = client.Discovery().RESTClient().Delete().AbsPath("/apis/wardle.k8s.io/v1alpha1/namespaces/default/flunders/test-flunder").DoRaw() framework.ExpectNoError(err, "attempting to delete a newly created flunders resource") </del> <ins> _, err = restClient.Delete().AbsPath("/apis/wardle.k8s.io/v1alpha1/namespaces/default/flunders/" + flunderName).DoRaw() validateErrorWithDebugInfo(f, err, pods, "attempting to delete a newly created flunders(%v) resource", flundersList.Items) </ins> // kubectl get flunders -v 9 // curl -k -v -XGET https://localhost/apis/wardle.k8s.io/v1alpha1/namespaces/default/flunders <del> contents, err = client.Discovery().RESTClient().Get().AbsPath("/apis/wardle.k8s.io/v1alpha1/namespaces/default/flunders").SetHeader("Accept", "application/json").DoRaw() </del> <ins> contents, err = restClient.Get().AbsPath("/apis/wardle.k8s.io/v1alpha1/namespaces/default/flunders").SetHeader("Accept", "application/json").DoRaw() </ins> framework.ExpectNoError(err, "confirming delete of a newly created flunders resource") err = json.Unmarshal(contents, &flundersList) <del> framework.ExpectNoError(err, "Error in unmarshalling %T response from server %s", contents, "/apis/wardle.k8s.io/v1alpha1") </del> <ins> validateErrorWithDebugInfo(f, err, pods, "Error in unmarshalling %T response from server %s", contents, "/apis/wardle.k8s.io/v1alpha1") </ins> if len(flundersList.Items) != 0 { framework.Failf("failed to get back the correct deleted flunders list %v", flundersList) } <ins> flunderName = generateFlunderName("dynamic-flunder") </ins> // Rerun the Create/List/Delete tests using the Dynamic client. resources, err := client.Discovery().ServerPreferredNamespacedResources() framework.ExpectNoError(err, "getting server preferred namespaces resources for dynamic client")

1 tests failed, 11 jobs failed, 241 builds failed. Failure stats cover 1 day time range '17 Aug 2017 22:57 UTC' to '18 Aug 2017 22:57 UTC'.
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
Seeing this failure on the e2e release-master-blocking https://k8s- Example failure https://k8s- Looks like an RBAC issue cc
this is currently blocking the alpha.3 release
looks like this new test was by in which was merged about 6 days ago and is failing consistently
This has started failing again on our GKE test suite https://k8s- https://k8s- cc
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/api-machinery: Issue will be escalated to these SIGs if needed. priority/critical-urgent: Never automatically move out of a release milestone; continually escalate to contributor and SIG through all available channels. kind/bug: Fixes a bug discovered during the current release. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details
/assign
Seems to be flaking now with the error: https://k8s-
- we need to determine (today if possible) if this is truly release blocking. If so, please add the release-blocker label. And, if not, how do we best continue work on this for 1.8.x/1.9.0?
this test has two passes and four failures on the same commit I'm seeing gke-specific authz grants in that test that are incorrect: That commit removed the wait for the authz grant to become effective (which can lead to flakes), and granted superuser permissions to all users, which is incorrect and invalidates any other authz-related tests run in parallel with this test
cc
I can add the wait back.

edit: and fix the test given admin to all authenticated users.
Actually after staring at this test for about half an hour I can't figure out what different users exist or what permissions they're being granted. ClientSet, InternalClientset, and AggregatorClient are all initialized from the same config so I don't see how one would be able to create an RBAC binding but another would fail later. any thoughts here?
I honestly think the gke specific BindClusterRole is a red-herring. It is needed so the client has permission to perform one of the setup steps. (I think it was either to create the wardler cluster role or to bind that role to the anonymous user) Once that setup step is complete we no longer need the that cluster role bound and so I don't think its related.
The gke authorizer allows the “bind” verb, so the client can create a binding to the cluster-admin. It cannot create a role directly unless it has permissions via RBAC. Since we don’t have a way to determine the username associated with , binding to all authenticated users is what was done as a workaround.
I agree that the point at which the tests are failing indicate that the previous authorization issues are not the cause.
/open
/reopen
So a lot more information to work with now but the error is still occurring. I am still looking into this.
any update on the investigation?
Friendly v1.8 release team ping. This failure still seems to be happening, is this actively being worked? Does this need to be in the v1.8 milestone?


Question: t.Errorf("expected %v call, but got %v.", item.expectedRequestCount, item.fakeNodeHandler.RequestCount) } nodes := sortedNodeNames(item.fakeNodeHandler.CreatedNodes) <del> if !reflect.DeepEqual(item.expectedCreated, nodes) { t.Errorf("expected node list %+v, got %+v", item.expectedCreated, nodes) </del> <ins> if !reflect.DeepEqual(item.expectedNameCreated, nodes) { t.Errorf("expected node list %+v, got %+v", item.expectedNameCreated, nodes) } nodeExtIDs := sortedNodeExternalIDs(item.fakeNodeHandler.CreatedNodes) if !reflect.DeepEqual(item.expectedExtIDCreated, nodeExtIDs) { t.Errorf("expected node external id list %+v, got %+v", item.expectedExtIDCreated, nodeExtIDs) </ins> } nodes = sortedNodeNames(item.fakeNodeHandler.DeletedNodes) if !reflect.DeepEqual(item.expectedDeleted, nodes) {

I see the following in the kube-controller-manager log: - This is happening to me on HEAD and is the reason why pod status is always showing unknown even when they are scheduled and running.
This is with the vagrant cloud provider. It looks like the delta between the new and old minion is the Node.Spec.ExternalID is being set on the incoming minion but is not set in the originally created minion.
I am not sure how this is not affecting other cloud providers. It looks to me that NodeController populated the node ExternalID here: It looks like the Kubelet provided value is missing the ExternalID
Actually, I am now confused on what the NodeController may or may not be doing here that is causing the status to not get updated.
It looks like the client is passing an ExternalID on the incoming request, but the original value that is in etcd does not have an ExternalID set. Not sure what changed that would have caused this issue to appear.
I think this causes problems: The memory is re-used in a for-loop by Go so referring to a pointer value in a map here will cause problems I think.
Ok, the referenced line is the problem. Sending a PR.
Well, I go back on that now. A simple way to reproduce this is to delete all the minions in your cluster, and verify the node controller is able to add them back. For me, this is causing the node controller to add them back in an unknown status, and then all updates to the node fail. The update fails because the node controller is not setting an ExternalID on the original object.
I am looking into this now. cc/ too
Related issue opened while debugging this:
Debugged through here: I can via manual fmt see that the object has an ExternalID set prior to the call Create. When I look at etcd, I end up with this persisted: Note: it is missing an ExternalID somehow ;-)
Ok, I think I see the issue Nodes returned from here: Do not have an ExternalID Which can make Nodes created from here: End up without an External ID. Having two different nodeClient.Create() paths in this code is confusing. Also we are doing bad things in maps holding pointers to memory allocated in for-loop. We need to fix that ;-)
Yeah, this is bug we should fix. Thanks for looking into this.

If you already have a potential fix, please feel free to grab the issue from me.
I think I have a PR that should fix. Testing that now. Will cc you on the PR.
Thanks for fixing - it did end up affecting AWS as well (AWS isn't in a fully working state just yet, particularly with node discovery).


Question: .vscode # This is where the result of the go build goes <del> /output/** /output /_output/** /_output </del> <ins> /output*/ /_output*/ </ins> # Emacs save files *~

There was a naive attempt to do this in , but _output it hardcoded throughout the code base so we need a more comprehensive solution.
Do we still need this ?
/sig release
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close


Question: "v1.SecurityContext": { "id": "v1.SecurityContext", "description": "SecurityContext holds security configuration that will be applied to a container. Some fields are present in both SecurityContext and PodSecurityContext.  When both are set, the values in SecurityContext take precedence.", <del> "required": [ "procMount" ], </del> "properties": { "capabilities": { "$ref": "v1.Capabilities",

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: When deploying a Manifest that is compatible with 1.11.3 against a newly provisioned 1.12.0-rc2 cluster, older manifests no longer worked. 2018-10-11T01:00:01.3680048Z error: error validating: error validating data: field [0].securityContext.procMount for v1.SecurityContext is required; if you choose to ignore these errors, turn validation off with --validate=false What you expected to happen: 11.3 compatible manifests should work against How to reproduce it (as minimally and precisely as possible): kind: DaemonSet apiVersion: extensions/v1beta1 metadata: name: traefik-ingress-controller namespace: kube-system labels: k8s-app: traefik-ingress-lb spec: template: metadata: labels: k8s-app: traefik-ingress-lb name: traefik-ingress-lb spec: serviceAccountName: traefik-ingress-controller terminationGracePeriodSeconds: 60 containers: image: traefik name: traefik-ingress-lb ports: name: http containerPort: 80 hostPort: 5001 name: admin containerPort: 8080 securityContext: capabilities: drop: ALL add: NETBINDSERVICE args: --api --kubernetes --logLevel=INFO Anything else we need to know?: Environment: Kubernetes version (use ): Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.9", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-06-29T01:07:01Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64" kubectl delivered by VSTS Build. 2018-10-11T00:59:54.7599385Z ##[section]Starting: Deploy Traefik Ingress 2018-10-11T00:59:54.7604992Z ============================================================================== 2018-10-11T00:59:54.7606132Z Task : Deploy to Kubernetes 2018-10-11T00:59:54.7606242Z Description : Deploy, configure, update your Kubernetes cluster in Azure Container Service by running kubectl commands.

2018-10-11T00:59:54.7606322Z Version : 0.1.31 2018-10-11T00:59:54.7606416Z Author : Microsoft Corporation 2018-10-11T00:59:54.7606512Z Help : 2018-10-11T00:59:54.7606590Z ============================================================================== Cloud provider or hardware configuration: ACS-Engine 0.22.4. Windows 1803.
There are no sig labels on this issue. Please add a sig label by either: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. See the . The in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
My expectation would be that this would default to "securityContext.procMount: default" - This appears to be related to . I notice that you've put this behind a feature gate. I'll admit that I'm relatively new to Kubernetes. My expectation would be that new fields would default - but is there some convention that says for features of a certain size, or of a certain complexity, we default them to on, and then force backwards compatibility via feature disable? It's not my intent to buck standard approach here.
What is it defaulting to for you now? It should default to default.
It's not defaulting at all. In the presence of this older manifest definition: securityContext: capabilities: drop: ALL add: NETBINDSERVICE args: --api --kubernetes --logLevel=INFO It's yelling at me that procMount is required. 2018-10-11T01:00:01.3680048Z error: error validating: error validating data: field [0].securityContext.procMount for v1.SecurityContext is required; if you choose to ignore these errors, turn validation off with --validate=false
ah ok I will do a fix
I'm also getting this error for the with K8s 1.12.

It's not clear from this thread what I need to do to fix this. Can anyone make a suggestion? cc


Question: }, }, options: printers.GenerateOptions{}, <del> // Columns: Name, Completions, Duration, Age expected: []metav1.TableRow{{Cells: []interface{}{"job1", "1/2", "", "0s"}}}, </del> <ins> // Columns: Name, Status, Completions, Duration, Age expected: []metav1.TableRow{{Cells: []interface{}{"job1", "Running", "1/2", "", "0s"}}}, </ins> }, // Generate table rows for Job with generate options "Wide". {

Suspend is a GA feature and it is big component of Kueue. It would be nice to display if jobs are suspended when doing . A new column that displays if the job is suspended would be helpful. Improve UX around suspend. Its a bit difficult to know why the job has not created pods unless you know about the suspend field.
/sig cli /wg batch /good-first-issue /help-wanted
This request has been marked as suitable for new contributors. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /sig cli /wg batch /good-first-issue /help-wanted Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc Just something I noticed as I was testing some job in JobSet. It would be nice to see which jobs are suspended when doing . Hope its not a controversial change.
cc who worked on
Not controversial at all IMO /triage accepted
/assign
Any suggestions on where to place this column? I'm thinking between completions and durations. Or, should this be displayed only when the output is set to wide?
I would put it in the default view as well. Probably before completions
Addressed in the PR:


Question: } mounter.csiClientGetter.driverName = csiDriverName(driverName) <del> // Save volume info in pod dir </del> dir := mounter.GetPath() <del> dataDir := filepath.Dir(dir) // dropoff /mount at end if err := os.MkdirAll(dataDir, 0750); err != nil { return nil, errors.New(log("failed to create dir %#v:  %v", dataDir, err)) } klog.V(4).Info(log("created path successfully [%s]", dataDir)) </del> mounter.MetricsProvider = NewMetricsCsi(volumeHandle, dir, csiDriverName(driverName)) <del> // persist volume info data for teardown node := string(p.host.GetNodeName()) volData := map[string]string{ volDataKey.specVolID:           spec.Name(), volDataKey.volHandle:           volumeHandle, volDataKey.driverName:          driverName, volDataKey.nodeName:            node, volDataKey.volumeLifecycleMode: string(volumeLifecycleMode), } attachID := getAttachmentName(volumeHandle, driverName, node) volData[volDataKey.attachmentID] = attachID err = saveVolumeData(dataDir, volDataFileName, volData) defer func() { // Only if there was an error and volume operation was considered // finished, we should remove the directory. if err != nil && volumetypes.IsOperationFinishedError(err) { // attempt to cleanup volume mount dir. if err = removeMountDir(p, dir); err != nil { klog.Error(log("attacher.MountDevice failed to remove mount dir after error [%s]: %v", dir, err)) } } }() if err != nil { errorMsg := log("csi.NewMounter failed to save volume info data: %v", err) klog.Error(errorMsg) return nil, errors.New(errorMsg) } </del> klog.V(4).Info(log("mounter created successfully")) return mounter, nil }

Instantiating the interface causes the creation of pod local volume directory and json data file and not - This causes a problem where is failed (and as a result never gets called) then those pod local volume directories will never be cleaned up. /sig storage cc
Looks like is the PR that introduced this change and moved data file creation calls to .
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle stale
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/repen /remove-lifecycle stale
/reopen
Reopened this issue. <detailsIn response to : /reopen </details
/remove-lifecycle stale
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue.

<detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen /remove-lifecycle rotten
Reopened this issue. <detailsIn response to : /reopen /remove-lifecycle rotten Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-contributor-experience at . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen /lifecycle frozen
Reopened this issue. <detailsIn response to : /reopen /lifecycle frozen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment.

<detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: e2enode "k8s.io/kubernetes/test/e2e/framework/node" e2epod "k8s.io/kubernetes/test/e2e/framework/pod" admissionapi "k8s.io/pod-security-admission/api" <del> utilpointer "k8s.io/utils/pointer" </del> "k8s.io/utils/ptr" )

In Aldo pointed out that: Let's use that instead of sleeping. /sig node /triage accepted /priority long-term More robust test.
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : ### What would you like to be ? Aldo pointed out that: There is. There should be a Schedulable condition set to false in the PodStatus. /triage accepted /priority long-term ### Why is this needed? Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign


Question: -gcflags "${gogcflags:-}" -asmflags "${goasmflags:-}" -ldflags "${goldflags:-}" <ins> -buildmode pie </ins> -tags "${gotags:-}" ) kube::golang::build_some_binaries "${nonstatics[@]}"

Currently is dynamically linked but is not built as a position independent executable. It would be great if it could be built with Go’s flag to enable ASLR. I’m not a security expert, and I’m not familiar with those concepts, but it seems that ASLR is a pretty common (and recommended) security mechanism. If I understand correctly, statically linked binaries (e.g. all the Kubernetes binaries except ) do not require/support PIE so this would only apply to . There was an attempt at using Go’s in but it seems that it was abandoned due to an incompatibility issue with Windows.
/sig release
cc
ASLR stands for Address Space Layout Randomization ref:
conditionally building only Linux binaries in PIE for ASLR seems acceptable. the problem is k8s is using a bash library for building which is not very flexible. also not clear how this would work for bazel builds.
I know at least some distributions which build go binaries per default with . do you see any reason to not turn it on for Linux per default?
i think it should be turned on. my comment was that our build infra is a bit entangled at places. /priority backlog the label seems appropriate.
Whoa! timely - https://go- also https://go-
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
I will work on this issue.
/sig security
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at .

/lifecycle stale
/remove-lifecycle stale
there are outstanding questions about this PTAL, I can lend some pointers to the build but I'd like more input on if this is desirable / worthwhile. Some notes: buildmode=pie does not seem to be supported for all platforms even on linux, which seems problematic given that we compile for many platforms. Custom builds can of course set this without the Kubernetes project choosing to do so, see .
IMO, there's value in building as PIE on the platforms where that is possible, but this is a low priority / nice-to-have. shows one way that building a PIE to enable ASLR can make attacks harder even though Go is memory-safe. It could also be a mitigation against a hypothetical memory-unsafety bug in the Go runtime.
not sure if it moves the discussion forward but ASLR is one mechanism to address NIST 800-53 control requirements for memory integrity/protection (ie SI-16). CIS mapped this in one of their . Note that is an older mapping and rev 5 is coming soon which keep this in as a requirement. I'm an expert enough not to call myself an expert in anything but I can argue it both ways but ultimately it comes down to this - we do a lot in all layers of a system for security and sometimes those help compliance, similarly we do a lot in all layers of a system for compliance that in no way help security but pay the bills. You can judge which is the greater good (or evil).
ROP mitigation is probably the most valuable aspect of this since Kubelet is always running in memory. Its binary may be a rich source of gadgets, of which their offsets in the binary need to be well known to an attacker.
cc to take a look
Started to follow-up on this in


Question: return attachment.Status.AttachmentMetadata, nil } <del> func unregisterDriver(driverName string) { </del> <ins> func unregisterDriver(driverName string) error { </ins> func() { csiDrivers.Lock() defer csiDrivers.Unlock()

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: When the Kubelet plugin watcher invokes method, right now the CSI code does nothing (method is not implemented). What you expected to happen: When the Kubelet plugin watcher no longer detects the socket file for a given driver, it invokes the method on the CSI registration handler, the CSI code should then clean up plugin information and remove the corresponding CSIDriver object. How to reproduce it (as minimally and precisely as possible): Delete the socket file of a CSI plugin exposed for registration, nothing happens. /sig storage
/assign


Question: <del> reviewers: - smarterclayton - liggitt - davidopp - eparis - resouer </del>

/kind bug /sig release Such as:
Cool issue. /assign
I see the code in was pushed by you. I don't understand what Intercept() does. Do you think there is an Infinite recursive call?
Chaos client isn’t used anywhere, and we need to remove it soon.
Ok, I can help remove it.
But there are also one place using it. How to deal with it? Delete it directly? Or something we need to do?
Has never been anything other than experimental and was not used in practice. We can remove it and it’s flag from kubelet as it was experimental. are we deprecating experimental flags for one release or can we just remove?


Question: Spec: api.ServiceSpec{ Type:     api.ServiceType(apiserverServiceType), Selector: apiserverSvcSelector, <del> Ports: []api.ServicePort{ { Name:       "https", Protocol:   "TCP", Port:       443, TargetPort: intstr.FromString(apiServerSecurePortName), }, }, </del> <ins> Ports:    []api.ServicePort{port}, </ins> }, }

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): Is this a BUG REPORT or FEATURE REQUEST? (choose one): FEATURE REQUEST <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): 1.6.2 Environment: Cloud provider or hardware configuration: AWS using Juju OS (e.g. from /etc/os-release): Ubuntu Xenial Kernel (e.g. ): 4.4.0-59-generic Install tools: Juju Others: What happened: kubefed init entered a infinite loop health checking the federation api service What you expected to happen: kubefed to exit with a success code How to reproduce it (as minimally and precisely as possible): Anything else we need to know: When doing a kubefed init on a cluster setup using VMs on aws (using Juju in my case) the worker nodes have two IPs: one that is visible to the internet and one internal visible only to VMs within aws. In my case for example the VM on aws hosting the kubernetes-worker node had external IP in the 54.174.73.1/24 subnet and it also had an internal IP in the 172.31.60.1/24 subnet. Using a client machine outside aws to bootstrap federation: I got the kubefed client stuck on doing a health check against the clouds internal subnet https://172.31.60.XXX:/healthz . To address this I can use the flag and pass the external IP of any kubernetes-worker node.

With this flag kubeconfig file gets updated with the right endpoint for the federation-apiserver. However, apart from the I should also be able to specify the desired port the service will be listening on. I have to make sure that port is opened on the aws routing/firewall prior to issuing . In the example above, kubefed init will try to reach https://54.174.73.XXX:/healthz, port is randomly assigned and is not reachable. I should be able to first open the port on aws and then do a with and port pinned to .
/cc
/assign
is there a reason why you are not using the default load balanced service on AWS?
I am using Juju to stay cloud agnostic. AWS just happened to be where I deployed this time. Juju just grabs machines (virtual or physical) from wherever you have resources from. In effect your cluster is more like an on-prem than anything else. How would the default load balanced service work?
got it.


Question: return listener, nil } <ins> func (c *Cloud) getSubnetCidrs(subnetIDs []string) ([]string, error) { request := &ec2.DescribeSubnetsInput{} for _, subnetID := range subnetIDs { request.SubnetIds = append(request.SubnetIds, aws.String(subnetID)) } subnets, err := c.ec2.DescribeSubnets(request) if err != nil { return nil, fmt.Errorf("error querying Subnet for ELB: %q", err) } if len(subnets) != len(subnetIDs) { return nil, fmt.Errorf("error querying Subnet for ELB, got %d subnets for %v", len(subnets), subnetIDs) } cidrs := make([]string, 0, len(subnets)) for _, subnet := range subnets { cidrs = append(cidrs, aws.StringValue(subnet.CidrBlock)) } return cidrs, nil } </ins> // EnsureLoadBalancer implements LoadBalancer.EnsureLoadBalancer func (c *Cloud) EnsureLoadBalancer(ctx context.Context, clusterName string, apiService *v1.Service, nodes []*v1.Node) (*v1.LoadBalancerStatus, error) { annotations := apiService.Annotations

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: For each NLB created, K8s creates an ingress rule on the node group's security group for each port exposed for each CIDR attached to the VPC to allow health checks to succeed. AWS limits the number of rules per security group to 60 by default. This quota is adjustable but multiplied by the quota for security groups per network interface cannot exceed 1000. I have a VPC with 16 CIDR blocks attached to it and a quota for rules per security group set to 80. Each NLB created adds 18 new rules (16 for Health Check, 1 for Client, and 1 for MTU) to the node group's security groups. This high number of unnecessary rules created makes it impossible to create more than 4 NLBs per node group (if each one has a single port exposed). What you expected to happen: For the health check to succeed, the NLB must be able to ping the node port for the service on the cluster's node group. So the node group's security group just needs an inbound rule for each ENI of the NLB, making the number of rules necessary on the cluster's node group directly proportional to the number of AZs the NLB is configured. How to reproduce it (as minimally and precisely as possible): Create a VPC on AWS with 8 CIDRs attached to it; Create a K8s Cluster; Create a Network Load Balancer with 6 ports exposed; If the default quota of rules per security group has not been changed, the NLB creation will fail with RulesPerSecurityGroupLimitExceeded. Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: OS (e.g: ):Kernel (e.g. ):Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig cloud-provider
/area provider/aws


Question: }, Volumes: []v1.Volume{ { <del> Name: logDirVolumeName, </del> <ins> Name: "logdir", </ins> VolumeSource: v1.VolumeSource{ HostPath: &v1.HostPathVolumeSource{ <del> Path: expectedlogFile, </del> <ins> Path: expectedLogPath, </ins> Type: hostPathType, }, },

The should be updated to check CRI log paths.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
May I try to help with this?
GitHub didn't allow me to assign the following users: fatsheep9146. Note that only and repo collaborators can be assigned. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: ... ``` <del> Then we will update the configmap. First check your kubectl version. </del> <ins> Then we will update the configmap by changing `index.html` to `index2.html`. </ins> ```sh <del> $ kubectl version Client Version: version.Info{Major:"1", Minor:"3", GitVersion:"v1.3.4", GitCommit:"dd6b458ef8dbf24aff55795baa68f83383c9b3a9", GitTreeState:"clean", BuildDate:"2016-08-01T16:45:16Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"3", GitVersion:"v1.3.5", GitCommit:"b0deb2eb8f4037421077f77cb163dbb4c0a2a9f5", GitTreeState:"clean", BuildDate:"2016-08-11T20:21:58Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"} ``` If you are using 1.5 or higher: Edit file `default.conf`: change `index index.html;` in line 8 to `index index2.html;`. ```sh $ kubectl replace configmap nginxconfigmap --from-file=default.conf configmap "nginxconfigmap" replaced ``` If you are using 1.4 or lower: Retrieve configmap nginxconfigmap. ```sh $ kubectl get configmap nginxconfigmap -o yaml > examples/https-nginx/nginxcm.yaml ``` Edit file `nginxcm.yaml`: change `index index.html;` to `index index2.html;`. Apply the change. ```sh $ kubectl apply -f examples/https-nginx/nginxcm.yaml </del> <ins> kubectl create configmap nginxconfigmap --from-file=examples/https-nginx/default.conf -o yaml --dry-run | sed 's/index.html/index2.html/g' | kubectl apply -f - </ins> configmap "nginxconfigmap" configured ```

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT on sample file I was using the /examples/https-nginx sample application for some small testing purposes and was surprised by the relative high load it kept having in the nginx pods. After some inspection I found out that kubernetes/examples/https-nginx/auto-reload- is basically running a busy loop. the command doesn't filter out some of the events it should filter out. At least on my system with just this sample config I always get a OPEN on basically immediately so I would recommend adding as arguments to the inotifywait command
I can take this.
I already have a PR in flight for this.
OK thanks for info


Question: Type=oneshot RemainAfterExit=yes ExecStartPre=/usr/bin/mkdir -p /opt/cni <del> ExecStartPre=/usr/bin/curl --fail --silent --show-error --location --create-dirs --output /opt/downloads/cni.tar.gz https://storage.googleapis.com/kubernetes-release/network-plugins/cni-8a936732094c0941e1543ef5d292a1f4fffa1ac5.tar.gz </del> <ins> ExecStartPre=/usr/bin/curl --fail --silent --show-error --location --create-dirs --output /opt/downloads/cni.tar.gz https://storage.googleapis.com/kubernetes-release/network-plugins/cni-9d5e6e60e79491207834ae8439e80c943db65a69.tar.gz </ins> ExecStart=/usr/bin/tar xf /opt/downloads/cni.tar.gz -C /opt/cni/ - name: kubernetes-install-rkt.service

We should update cni binaries to containernetworking/cni and release binaries for all arches. I do not have push access to the repo, so I guess some googler have to do it.
Thanks. I have this on my plate.


Question: configs[VolumeISCSIServer] = Config{e2eRegistry, "volume/iscsi", "2.0"} configs[VolumeGlusterServer] = Config{e2eRegistry, "volume/gluster", "1.0"} configs[VolumeRBDServer] = Config{e2eRegistry, "volume/rbd", "1.0.1"} <del> configs[WindowsNanoServer] = Config{e2eRegistry, "windows-nanoserver", "v1"} </del> return configs }

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: jobs primarely, like: Which test(s) are failing: [] Container Runtime blackbox test when running a container with a new image should be able to pull image from [NodeConformance] Since when has it been failing: Since this commit merged: Testgrid link: Reason for failure: The test should always pull the image from (), it is not configurable. Anything else we need to know: /sig testing /sig windows /area conformance This issue was fixed once before:


Question: streamingConfig := getStreamingConfig(kubeCfg, kubeDeps) // Use the new CRI shim for docker. ds, err := dockershim.NewDockerService(klet.dockerClient, kubeCfg.SeccompProfileRoot, kubeCfg.PodInfraContainerImage, <del> streamingConfig, &pluginSettings, kubeCfg.RuntimeCgroups, kubeCfg.CgroupDriver) </del> <ins> streamingConfig, &pluginSettings, kubeCfg.RuntimeCgroups, kubeCfg.CgroupDriver, dockerExecHandler) </ins> if err != nil { return nil, err }

Either deprecate the nsenter exec handler, or add nsenter handling to the dockershim (after goes in). /cc
I vote for deprecating the nsenter exec handler for docker. has been the default exec handler since kubernetes pre-1.0. Is there any reason (or use case) for keeping the pluggable exec handler? Another question is that whether we can drop exec in one release, or we need to provide support in CRI for at lease one more release. /cc
+1 for docker exec.
I suggest deprecating docker nsenter exec in a long term, but to achieve that, we need to follow the deprecation policy. The first thing we need to address is marking this flag (--docker-exec-handler, default: native) being deprecated:
Also there are several issues raised before which are related to the issue of docker exec in various versions. The engineer even suggested to switch the default mechanism is docker nsenter exec. One of the examples is
Now let's narrow down to the CRI scope, and below is my 2 cents: 1) Can we mark the flag deprecated in 1.6? 2) Drop the support in CRI in 1.6? Inform the kubernetes users with this decision along CRI announcement being scheduled sometimes in February? 3) Drop the support completely from Kubernetes unless there are real use cases coming up.
We'll have to tell the users that they need to disable CRI in order to use the nsenter exec plugin. could you give a quick estimate about the amount of work required to add support for nsenter exec in dockershim? Thanks.
Talked to offline. I'll Add support nsenter as exec in dockershim Mark the flag as deprecated.


Question: return cs } <del> func TestStartResourceQuotaController_DiscoveryError(t *testing.T) { </del> <ins> func possibleDiscoveryResource() []*metav1.APIResourceList { return []*metav1.APIResourceList{ { GroupVersion: "create/v1", APIResources: []metav1.APIResource{ { Name:       "jobs", Verbs:      []string{"create", "list", "watch", "delete"}, ShortNames: []string{"jz"}, Categories: []string{"all"}, }, }, }, } } type controllerInitFunc func(ControllerContext) (http.Handler, bool, error) func TestController_DiscoveryError(t *testing.T) { controllerInitFuncMap := map[string]controllerInitFunc{ "ResourceQuotaController":    startResourceQuotaController, "GarbageCollectorController": startGarbageCollectorController, } </ins> tcs := map[string]struct { discoveryError    error expectedErr       bool

In we observed a crashlooping controller cause a cluster cold boot failure, as controller-manager wasn't able to stay up long enough to run the deployment that would have fixed the problem. We should audit everything that runs in kube-controller-manager and ensure that they all fail gracefully enough to prevent crashloops, or make them individually restart in that case, instead of the process. Longer term, we should consider classifying the controllers by run level or bootstrap order (), and splitting them into separate binaries or changing the start-up order in kcm to make sure this sort of thing doesn't happen in the future.
Same issue with metrics-server, using minikube. Also reproduced on our test cluster. Steps to reproduce: Set up localkube Install metrics-server Stop minikube: sudo minikube stop Start minikube: sudo -E minikube start --vm-driver=none Expected: Kubernetes is running Actual: Kubernetes is dead Logs: Workaround: To fix the system, we must delete the metrics-server: Now kubernetes will start normally. After that we can reinstall the metrics server, and it will be ok.
Related:
This will be a hard problem to fix permanently. In all likelihood, we should stop allowing Fatalf() in kubernetes/kubernetes, but as you can see we do this to ourselves right here...
/kind bug
I also think this bug would benefit from an e2e test where we misconfigure things in a variety of ways and verify that KCM still comes up and stays up: [ ] misconfigured apiservice [ ] broken service attached to that apiservice [ ] RBAC rules that disallow things. maybe everything! [ ] webhooks that disallow everything [ ] yr example here
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: t.Errorf("Didn't fail with non-existent service") } <del> loadBalancer.NewService(service, api.ServiceAffinityClientIP, 0) </del> <ins> // Call OnUpdate() before NewService() </ins> endpoints := make([]api.Endpoints, 1) endpoints[0] = api.Endpoints{ ObjectMeta: api.ObjectMeta{Name: service.Name, Namespace: service.Namespace}, Subsets: []api.EndpointSubset{ <del> { Addresses: []api.EndpointAddress{{IP: "endpoint"}}, Ports:     []api.EndpointPort{{Port: 1}, {Port: 2}, {Port: 3}}, }, </del> <ins> {Addresses: []api.EndpointAddress{{IP: "endpoint1"}}, Ports: []api.EndpointPort{{Port: 1}}}, {Addresses: []api.EndpointAddress{{IP: "endpoint2"}}, Ports: []api.EndpointPort{{Port: 2}}}, </ins> }, } loadBalancer.OnUpdate(endpoints) <del> shuffledEndpoints := loadBalancer.services[service].endpoints expectEndpoint(t, loadBalancer, service, shuffledEndpoints[0], client1) expectEndpoint(t, loadBalancer, service, shuffledEndpoints[0], client1) expectEndpoint(t, loadBalancer, service, shuffledEndpoints[1], client2) expectEndpoint(t, loadBalancer, service, shuffledEndpoints[1], client2) expectEndpoint(t, loadBalancer, service, shuffledEndpoints[2], client3) expectEndpoint(t, loadBalancer, service, shuffledEndpoints[2], client3) expectEndpoint(t, loadBalancer, service, shuffledEndpoints[0], client1) expectEndpoint(t, loadBalancer, service, shuffledEndpoints[0], client1) } </del> <ins> loadBalancer.NewService(service, api.ServiceAffinityClientIP, 0) </ins> <del> func TestStickyLoadBalanaceWorksWithMultipleEndpointsStickyNone(t *testing.T) { </del> client1 := &net.TCPAddr{IP: net.IPv4(127, 0, 0, 1), Port: 0} client2 := &net.TCPAddr{IP: net.IPv4(127, 0, 0, 2), Port: 0} client3 := &net.TCPAddr{IP: net.IPv4(127, 0, 0, 3), Port: 0} <del> loadBalancer := NewLoadBalancerRR() service := ServicePortName{types.NamespacedName{"testnamespace", "foo"}, ""} endpoint, err := loadBalancer.NextEndpoint(service, nil) if err == nil || len(endpoint) != 0 { t.Errorf("Didn't fail with non-existent service") </del> <ins> ep1, err := loadBalancer.NextEndpoint(service, client1) if err != nil { t.Errorf("Didn't find a service for %s: %v", service, err) </ins> } <ins> expectEndpoint(t, loadBalancer, service, ep1, client1) expectEndpoint(t, loadBalancer, service, ep1, client1) expectEndpoint(t, loadBalancer, service, ep1, client1) </ins> <del> loadBalancer.NewService(service, api.ServiceAffinityNone, 0) endpoints := make([]api.Endpoints, 1) endpoints[0] = api.Endpoints{ ObjectMeta: api.ObjectMeta{Name: service.Name, Namespace: service.Namespace}, Subsets: []api.EndpointSubset{ { Addresses: []api.EndpointAddress{{IP: "endpoint"}}, Ports:     []api.EndpointPort{{Port: 1}, {Port: 2}, {Port: 3}}, }, }, </del> <ins> ep2, err := loadBalancer.NextEndpoint(service, client2) if err != nil { t.Errorf("Didn't find a service for %s: %v", service, err) </ins> } <del> loadBalancer.OnUpdate(endpoints) </del> <ins> expectEndpoint(t, loadBalancer, service, ep2, client2) expectEndpoint(t, loadBalancer, service, ep2, client2) expectEndpoint(t, loadBalancer, service, ep2, client2) </ins> <del> shuffledEndpoints := loadBalancer.services[service].endpoints expectEndpoint(t, loadBalancer, service, shuffledEndpoints[0], client1) expectEndpoint(t, loadBalancer, service, shuffledEndpoints[1], client1) expectEndpoint(t, loadBalancer, service, shuffledEndpoints[2], client2) expectEndpoint(t, loadBalancer, service, shuffledEndpoints[0], client2) expectEndpoint(t, loadBalancer, service, shuffledEndpoints[1], client3) expectEndpoint(t, loadBalancer, service, shuffledEndpoints[2], client3) expectEndpoint(t, loadBalancer, service, shuffledEndpoints[0], client1) expectEndpoint(t, loadBalancer, service, shuffledEndpoints[1], client1) </del> <ins> ep3, err := loadBalancer.NextEndpoint(service, client3) if err != nil { t.Errorf("Didn't find a service for %s: %v", service, err) } expectEndpoint(t, loadBalancer, service, ep3, client3) expectEndpoint(t, loadBalancer, service, ep3, client3) expectEndpoint(t, loadBalancer, service, ep3, client3) expectEndpoint(t, loadBalancer, service, ep1, client1) expectEndpoint(t, loadBalancer, service, ep2, client2) expectEndpoint(t, loadBalancer, service, ep3, client3) expectEndpoint(t, loadBalancer, service, ep1, client1) expectEndpoint(t, loadBalancer, service, ep2, client2) expectEndpoint(t, loadBalancer, service, ep3, client3) </ins> } func TestStickyLoadBalanaceWorksWithMultipleEndpointsRemoveOne(t *testing.T) {

!topic/google-containers/pjx13NWEFcg The bug seems to be in kube-proxy.


Question: return maximumAllowedReplicas, possibleLimitingReason, possibleLimitingMessage } } else if args.DesiredReplicas < args.CurrentReplicas { <del> scaleDownLimit := calculateScaleDownLimitWithBehaviors(args.CurrentReplicas, a.scaleDownEvents[args.Key], args.ScaleDownBehavior) </del> <ins> scaleDownLimit := calculateScaleDownLimitWithBehaviors(args.CurrentReplicas, a.scaleUpEvents[args.Key], a.scaleDownEvents[args.Key], args.ScaleDownBehavior) </ins> if scaleDownLimit > args.CurrentReplicas { // We shouldn't scale down further until the scaleDownEvents will be cleaned up scaleDownLimit = args.CurrentReplicas

We created an hpa as following: Then started a deployment with 4 replicas and made the CPU utilization about 10%, which proposed to double the replicas, and it did work: Then we changed the CPU utilization to about 1%, which would trigger scaling down and would kill 2 pods only for scaleDown behavior: Then changed the CPU utilization to about 20%, which would trigger scaling up: It did scale up, and for our expectation, it should be scaled to 620/5=24 but would be limited by scaleUp behavior as 4(1+300%)=16, but in fact, it was just scaled up to 8 It would be scaled up to 16 in last scaling actions Start an hpa with a scaleUp behavior which has little long period and a little great Percent value, then start a deployment with a container whose CPU utilization could be controlled easily. Change the CPU utilization to make it scale up within limit, scale down a little, and scale up again without limit. I think it is caused by , this function seems that it just considered about scaleUp event history when calculate scaleUp limit (so does calculate scaleDown limit). I think it is a bit unreasonable. Does this action match the design? Or it does be a bug? <details</details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig autoscaling
From what I understand, we observed the following behaviour: , there are pods. The HPA scales up new pods, so we have in total. , pods are scaled down, so we have in total. HPA calculates the desired Replicas with . Since all scaling events happened within the policies , the policy value of kicks in and should allow scaling up to . But it only scaled to .

I think that there is a bug in the following code: See the comments in the following snippets for an explanation: With a of 2, the is as follows: I will think of a possible solution to the problem and would like to contribute once I've made some progress on that :)
In fact, I knew this was caused by the calculating method of "getReplicasChangePerPeriod" and mentioned it in the "Anything else we need to know" of this issue. What I wonder now is that if it is an expected action or it is just a bug. If it quite is a bug, I hope I could contribute as well.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale


Question: --enable-hostpath-provisioner="${ENABLE_HOSTPATH_PROVISIONER}"  --pvclaimbinder-sync-period="${CLAIM_BINDER_SYNC_PERIOD}"  --feature-gates="${FEATURE_GATES}"  <ins> --emulated-version="${EMULATED_VERSION}"  </ins> "${cloud_config_arg[@]}"  --authentication-kubeconfig "${CERT_DIR}"/controller.kubeconfig  --authorization-kubeconfig "${CERT_DIR}"/controller.kubeconfig

Currently the local-up- script does not have support for the flag. This flag was recently here as a part of For development, testing, and full plumbing through of the feature the flag should be to local-up- to allow for usage of the feature when setting up a cluster via local-up-
/sig apimachinery
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /sig apimachinery Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig api-machinery
/sig testing /triage accepted /assign


Question: case *corev1.Pod: // if allContainers is true, then we're going to locate all containers and then iterate through them. At that point, "allContainers" is false if !allContainers { <ins> currOpts := new(corev1.PodLogOptions) if opts != nil { opts.DeepCopyInto(currOpts) } </ins> // in case the "kubectl.kubernetes.io/default-container" annotation is present, we preset the opts.Containers to default to selected // container. This gives users ability to preselect the most interesting container in pod. <del> if annotations := t.GetAnnotations(); annotations != nil && len(opts.Container) == 0 { var containerName string </del> <ins> if annotations := t.GetAnnotations(); annotations != nil && currOpts.Container == "" { var defaultContainer string </ins> if len(annotations[podcmd.DefaultContainerAnnotationName]) > 0 { <del> containerName = annotations[podcmd.DefaultContainerAnnotationName] </del> <ins> defaultContainer = annotations[podcmd.DefaultContainerAnnotationName] </ins> } else if len(annotations[defaultLogsContainerAnnotationName]) > 0 { // Only log deprecation if we have only the old annotation. This allows users to // set both to support multiple versions of kubectl; if they are setting both // they must already know it is deprecated, so we don't need to add noisy // warnings. <del> containerName = annotations[defaultLogsContainerAnnotationName] </del> <ins> defaultContainer = annotations[defaultLogsContainerAnnotationName] </ins> fmt.Fprintf(os.Stderr, "Using deprecated annotation `kubectl.kubernetes.io/default-logs-container` in pod/%v. Please use `kubectl.kubernetes.io/default-container` insteadn", t.Name) } <del> if len(containerName) > 0 { if exists, _ := podcmd.FindContainerByName(t, containerName); exists != nil { opts.Container = containerName </del> <ins> if len(defaultContainer) > 0 { if exists, _ := podcmd.FindContainerByName(t, defaultContainer); exists == nil { fmt.Fprintf(os.Stderr, "Default container name %q not found in pod %sn", defaultContainer, t.Name) </ins> } else { <del> fmt.Fprintf(os.Stderr, "Default container name %q not found in a podn", containerName) </del> <ins> currOpts.Container = defaultContainer </ins> } } } <del> var containerName string if opts == nil || len(opts.Container) == 0 { </del> <ins> if currOpts.Container == "" { </ins> // We don't know container name. In this case we expect only one container to be present in the pod (ignoring InitContainers). // If there is more than one container, we should return an error showing all container names. if len(t.Spec.Containers) != 1 {

/kind bug The logs of pods with label will be output expected without errors. create the following resources: kubectl logs -l Kubernetes version (use ): latest master.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cli
I am able to pull logs from multiple containers in my test setup, this doesn't seem to be an issue with Could you check if all the containers in the specified pod are healthy?
yes, all of these pods' containers are healthy. I the section.
This is working as expected, was not able to find the default container which in both deployments you explicitly said is but only the first deployment has container, the second deployment has . /close
Closing this issue. <detailsIn response to : This is working as expected, was not able to find the default container which in both deployments you explicitly said is but only the first deployment has container, the second deployment has . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: return nil } <del> ret, err := podClient.Create(pod) if err != nil { framework.ExpectNoError(err, "Failed to create %q pod: %v", pod.GetName(), err) } </del> <ins> // Creating the pod can fail initially while the service // account's secret isn't provisioned yet ('No API token found // for service account "csi-service-account", retry after the // token is automatically created and added to the service // account', see https://github.com/kubernetes/kubernetes/issues/68776). // We could use a DaemonSet, but then the name of the csi-pod changes // during each test run. It's simpler to just try for a while here. podClient := f.PodClient() ret := podClient.CreateEventually(pod) </ins> // Wait for pod to come up framework.ExpectNoError(framework.WaitForPodRunningInNamespace(client, ret))

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: The code creates a pod directly, without waiting for the service account secret to be available. Depending on timing, this can fail with a error. What you expected to happen: It should use a DaemonSet instead, then the daemon set controller will ensure that the pod gets created eventually. [Edit: when implementing the fix, I found it easier and less intrusive to just retry pod creation.] How to reproduce it (as minimally and precisely as possible): It's not possible to trigger the failing race condition reliably. I've seen it a few times on my own cluster. Anything else we need to know?: Originally discussed in and in the CSI working group meeting on Monday, September 17th ().
/sig storage


Question: "STANDARD_D3":            16, "STANDARD_D32AS_V4":      32, "STANDARD_D32A_V4":       32, <ins> "STANDARD_D32DS_V4":      32, "STANDARD_D32D_V4":       32, </ins> "STANDARD_D32S_V3":       32, "STANDARD_D32_V3":        32, "STANDARD_D3_V2":         16,

<!-- Please only use this template for submitting enhancement requests -- What would you like to be : related to Why is this needed: /assign /sig azure
here is the way about how to get MaxDataDiskCount: and then build kubelet, copy kubelet to agent node and then get MaxDataDiskCount data: There are 207 VM sizes totally.
Update: this command could get VM size list easily:


Question: // wait for node to finish processing (should terminate and not time out) } } <ins> func TestNodeUpdateRetryTimeout(t *testing.T) { for _, tc := range []struct { count int want  time.Duration }{ {count: 0, want: 250 * time.Millisecond}, {count: 1, want: 500 * time.Millisecond}, {count: 2, want: 1000 * time.Millisecond}, {count: 3, want: 2000 * time.Millisecond}, {count: 50, want: 5000 * time.Millisecond}, } { t.Run(fmt.Sprintf("count %d", tc.count), func(t *testing.T) { if got := nodeUpdateRetryTimeout(tc.count); got != tc.want { t.Errorf("nodeUpdateRetryTimeout(tc.count) = %v; want %v", got, tc.want) } }) } } </ins>

/kind bug /sig network What happened: Networking tight loop causing GCP Cloud rate limiting to kick in. Causing tests to fail. Suspected call stack: Looks like it could be the root cause of these test failures here: https://k8s- These storage tests are hitting rate limit on the cloud api when trying to detach/delete disks. However, we have exponential backoff for all these functions. But the rate limiting from an external source is affecting the tests. See logs snippet: Note: These logs span exactly 1 minute. Try grepping "rateLimit" or "CIDRNotAvailable" -also note that the CIDRNotAvailable error messages are probably not verbose enough as they do not pinpoint what the actual error is, you can only associate them by correlation in timing and then looking at the code. Shorter snippet from above log file to demonstrate: since this is running on hundreds of nodes hitting this API call should involve some sort of exponential backoff as well as jitter. /cc /assign
/assign
/assign
/sig cloud-provider /sig gcp /milestone v1.12 (since is marked as v1.12)
may we have a priority on this since Code Freeze is next week? :)
Quick question, are we using the default API rate limits quota for scalability projects?
I will defer to for priority because he probably has the most context on how serious this is and what it could affect
Which rate limit are you referring to? (I don't remember us changing it any time) Btw - this is the project being used for that job -
I put up a patch for adding exponential retry
Gentle reminder that for this to be considered for 1.12 past Code Freeze, which is next week, it must have a status/critical-urgent label. I see this is being actively worked on


Question: <ins> /* Copyright 2017 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package v1beta2 import ( "fmt" apps "k8s.io/api/apps/v1beta2" "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/labels" ) // DaemonSetListerExpansion allows custom methods to be added to // DaemonSetLister. type DaemonSetListerExpansion interface { GetPodDaemonSets(pod *v1.Pod) ([]*apps.DaemonSet, error) GetHistoryDaemonSets(history *apps.ControllerRevision) ([]*apps.DaemonSet, error) } // DaemonSetNamespaceListerExpansion allows custom methods to be added to // DaemonSetNamespaceLister. type DaemonSetNamespaceListerExpansion interface{} // GetPodDaemonSets returns a list of DaemonSets that potentially match a pod. // Only the one specified in the Pod's ControllerRef will actually manage it. // Returns an error only if no matching DaemonSets are found. func (s *daemonSetLister) GetPodDaemonSets(pod *v1.Pod) ([]*apps.DaemonSet, error) { var selector labels.Selector var daemonSet *apps.DaemonSet if len(pod.Labels) == 0 { return nil, fmt.Errorf("no daemon sets found for pod %v because it has no labels", pod.Name) } list, err := s.DaemonSets(pod.Namespace).List(labels.Everything()) if err != nil { return nil, err } var daemonSets []*apps.DaemonSet for i := range list { daemonSet = list[i] if daemonSet.Namespace != pod.Namespace { continue } selector, err = metav1.LabelSelectorAsSelector(daemonSet.Spec.Selector) if err != nil { // this should not happen if the DaemonSet passed validation return nil, err } // If a daemonSet with a nil or empty selector creeps in, it should match nothing, not everything. if selector.Empty() || !selector.Matches(labels.Set(pod.Labels)) { continue } daemonSets = append(daemonSets, daemonSet) } if len(daemonSets) == 0 { return nil, fmt.Errorf("could not find daemon set for pod %s in namespace %s with labels: %v", pod.Name, pod.Namespace, pod.Labels) } return daemonSets, nil } // GetHistoryDaemonSets returns a list of DaemonSets that potentially // match a ControllerRevision. Only the one specified in the ControllerRevision's ControllerRef // will actually manage it. // Returns an error only if no matching DaemonSets are found. func (s *daemonSetLister) GetHistoryDaemonSets(history *apps.ControllerRevision) ([]*apps.DaemonSet, error) { if len(history.Labels) == 0 { return nil, fmt.Errorf("no DaemonSet found for ControllerRevision %s because it has no labels", history.Name) } list, err := s.DaemonSets(history.Namespace).List(labels.Everything()) if err != nil { return nil, err } var daemonSets []*apps.DaemonSet for _, ds := range list { selector, err := metav1.LabelSelectorAsSelector(ds.Spec.Selector) if err != nil { return nil, fmt.Errorf("invalid label selector: %v", err) } // If a DaemonSet with a nil or empty selector creeps in, it should match nothing, not everything. if selector.Empty() || !selector.Matches(labels.Set(history.Labels)) { continue } daemonSets = append(daemonSets, ds) } if len(daemonSets) == 0 { return nil, fmt.Errorf("could not find DaemonSets for ControllerRevision %s in namespace %s with labels: %v", history.Name, history.Namespace, history.Labels) } return daemonSets, nil } </ins>

As a prerequisite of moving the , , , and controllers to use the apps/v1beta2 APIs, we need to add list expansions for , , , and to .
/assign
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig apps


Question: "port": 10001, "labels": { "name": "redisslave" <del> } </del> <ins> }, </ins> "selector": { "name": "redisslave" }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: not fixed the SMB mount issue totally, when there is a SMB mount with same remote path, original SMB mapping would be removed and SMB mapping re-established, that would cause temporally data inaccessible with the original container. should change to should use to check whether original smb mapping is valid, here is an example showing that after SMB server password changed, could validate whether connection is ok: the main diff on smb mount between linux and windows is that on LInux, every smb mount is independent, while on Windows, one remote smb server address(e.g. ) can only have one SMB-GlobalMapping. So on Windows, there is a difficulty: in some cases, we should skip create SMB-GlobalMapping since it already exists in other cases, original password is incorrect or expired, user wants to delete SMB-GlobalMapping, and then create a new SMB-GlobalMapping (fixed by ) On Windows, there is no way to check whether original SMB-GlobalMapping is valid or not, that was the answer about 1 years ago from windows team, will update this thread(also about the max SMB-GlobalMapping limit) if there is any new update recently. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /sig storage /assign

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: ci-kubernetes-e2e-aks-engine-azure-master-windows-2004 Which test(s) are failing: [] Container Runtime blackbox test when running a container with a new image should be able to pull from private registry with secret [NodeConformance] Since when has it been failing: since enabling the Windows 2004 SAC release tests Testgrid link: Reason for failure: The manifest for is missing the new windows server version 2004. This has occurred before: I looks like we need someone with appropriate permissions to update the manifests once the images are ready. Anything else we need to know: There was some work done in to automate some of this. Is there something missing? There are a few other tests that are failing in this job but should be covered by updating /cc /sig windows
The test passed 2x consecutively. We can close this now. /close
Closing this issue. <detailsIn response to : The test passed 2x consecutively. We can close this now. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details