Question: // may be synced to a different version and they will never converge. // TODO: We should use etcd progress-notify feature to avoid this behavior but until this is // done we simply try to relist from now to avoid continuous errors on relists. <del> for i := 1; i <= 2; i++ { </del> <ins> for i := 1; i <= 3; i++ { </ins> // relist twice to cover the two variants of TooLargeResourceVersion api errors stopCh = make(chan struct{}) if err := r.ListAndWatch(stopCh); err != nil {

What happened: Using client-go v1.18.8 to watch resources on API server versions 1.17.x and 1.18.0 - 1.18.5 After the connection to the server is temporarily interrupted the client never recovers from error See the earlier issue for steps to reproduce: This was assumed fixed by But it appears the fix for the client was (probably inadvertently) implemented in a non-backward-compatible fashion and only solves the issue when the server is running v1.18.6 or later as well.
/sig api-machinery
Cc
PR
We generally recommend having apiservers be updated first before all other components. But yeah - it's true that this doesn't work for clients being more fresh - let me take a look at your PR.
/cc
This will be addressed by (and the cherrypicking it back).
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
This has been fixed by and corresponding cherrypicks.

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: I have disconnected a node from network for a few minutes. After reconnecting, I keep receiving such error messages from kubelet on the node, even after 15 minutes in reconnected state: What you expected to happen: I expect that after network recovery kubelet reconnects to the Apiserver as before, and after a recovery such timeouts do not occur. How to reproduce it (as minimally and precisely as possible): Just have a node connected to the cluster. Then, disconnect it from the network for 3-4 minutes, then reconnect. Then observe kubelet's logs. Anything else we need to know?: I have strict tcp keepalive settings in place on master and worker nodes, but this should not be the cause. Restarting kubelet solves the issue, the error messages disappear. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: bare metal OS (e.g: ):Kernel (e.g. ):Install tools: kubeadm Network plugin and version (if this is a network-related bug): kube-router Others:
/sig api-machinery
/sig api-machinery
It seems that after some more time, kubelet recovers. Sometimes it takes one hour, sometimes two. Recovery is successful when I receive similar logs: What is the reason that it takes so long for it to recover?
The root cause is , deduplicating against that issue
Debugging, tracing the code, I am not really sure that this is the same issue as the other mentioned here. Error messages show that the client is requesting higher resource versions that is available at the server: , the former is greater by 989 than the latter. I dont know from where this resource version comes from. I've made a very stupid patch on my kubelet/client-go: I am pretty sure that this is not the best solution, but, it seems that it forces a full list every time ListAndWatch is called.
You're correct, sorry about that... I misread the timeout message. The error it is getting is coming from the server, which means kubelet -apiserver connectivity is fine.

This message comes from the watch cache, which means the watch cache in the API server is behind where the kubelet's informer was, which is pretty surprising. A few questions: Is there a single apiserver or multi-apiserver cluster? Is there a single etcd member or multiple etcd members? Is there an API server running on the node that was detached/reattached?
cc does this ring any bells?
cc for eyes on watch cache
I don't think I've hit this problem specifically. Since this could happen due to an etcd partition or due to a partition between a apiserver and etcd, the answers questions asked in seem like the appropriate next things to figure out.
Yeah - I agree with Joe. It seems like some network partition issue of the apiserver the node was connected to.
I will be able to answer the questions in detail later, but in short: there are 3 masters set up with kubeadm, haproxy load-balances the queries to the apiservers. The only thing to reproduce the issue is to disconnect the node from the network for a few minutes. Masters or etcd members are not touched, and anyway the cluster seems to be healthy.
Today, it occured on a diffrent cluster. Setup is similar: stacked etcd on master nodes, haproxy load-balances traffic between them. Just was playing with haproxy servers and the HA setup, and after a few restarts of haproxy, one of the nodes entered this state. So during a haproxy restart, all kubelets' connections to apiservers are terminated, but rarely, one of them enters this unhealthy state.
Multi apiserver cluster Stacked etcd members are present. I think In my setup it is not relevant, as all the kubelets are talking to the apiserver through the load-balancer, which picks one of the available apiservers. I am doing a rolling restarts of the nodes, and there were at least one occurence, when even one of the master nodes got stucked in this state. How can I help debugging/fixing this?
How can we proceed with this issue?

got same problem in a old cluster, ( up since 3+years) after updating to kube 1.18.3 , one node kubelet start logging those errors ( ~10 errors/minute with various resources) I've now just restarted kubelet and now seems working again.. Update: while testing I've tried to connect the kubelet of that node to each single master nodes to check if there were some kind of network issue / partitioning.. none of the master servers returned errors ( the kubelet worked fine with all master nodes bypassing the nginx proxy that manage masters HA
I think this problem is getting serious. I am not 100% sure, but today, during node restarts (including masters) coredns was migrated multiple times, and unfortunately, kube-proxy pods had not received the updated situation, thus, dns resolution was not working inside the cluster. After detecting this, a simple restart of all kube-proxy pods resolved our problem. There were no network issues during this, so I suspect we've hit the same bug, now in kube-proxy.
I am trying to dig this. Now, I've written a little go prog which uses the same reflector what kubelet uses. It just starts a watch for CSIDrivers, and I've made client-go to print the last resourceVersion. Starting the program multiple times (i.e. connecting to different masters) produces the following: Right now I dont exactly know what url is being fetched and what arguments are passed in, I still have to figure that out. But now it seems, that different masters return different resouceVersions. Etcd does not report any problems/issues.
Any ideas, suggestions where to look further?
Today I have re-initialized 2 of my 3 master nodes to make sure etcd replicas are not corrupted. Just after the nodes have been joined, the same behavior could be observed: connecting to different masters resulted in different resourceVersions.
Now, it can be seen, that querying different apiservers again return different resourceVersions: 192.168.8.60: is a haproxy load-balancing traffic between 3 masters.
please let me know where to go further. My cluster runs on PI boards, perhaps, not the fastest ones.

But if this issue depends on slow hardware, than it can pop up anytime on fast hardware as well.As I wrote, today I reinitialized 2 of my 3 masters to make sure no partitioning occured. Also, please have a look at my comments. Or, if you have a clue that my setup is broken, please let me know!
It seems that it happens to resources where no objects exist at all. Can it be?
Ahh - the fact tht it happens for types with no objects actually made me know what is happening here. Basically what is happening is that when kube-apiserver is starting, watchcache is initialized from "now" (i.e. it will be synced to resource version that was globally last in etcd at the time when it was initialized). However, later it's updated only on changes of resources of those types. So when no objects of that type is being /modified/deleted it will stay the same. So it's kind-of "working-as-implemented" from the kube-apiserver perspective. That can be improved - I'm actually already looking into reusing etcd progress-notify feature to get rid of this behavior ( & ) - I would like to have alpha for that in 1.20 and beta in 1.21 (planning to write down KEP once I'm back from my soon-starting vacation), but that shouldn't be the problem by itself. And it's not what is really biting you. The real problem is that there is a bug in reflector in: It handled the "too old RV" error, but it doesn't handle "too large RV" correctly. What it should be doing instead is to: trigger relist from etcd (to get RV=now") start watching from that RV Watch will work fine because watch will simply wait until events with larger RV appear. I will send out a PR fixing that bug (will try to do it later today).
is out for review
This has been fixed/mitigated in 1.18. Going to backport to 1.18 now. It seems 1.18 is the first version affected (the offending changes were reverted in 1.17 months ago).
This has been mitigated in 1.18 and head. The more proper fix is proposed in Closing this one.

Hello, as i saw, i have the same issue in 1.19.3 version: Our setup is 3 etcd, 2 apiserver bound on those 3 etcd and a haproxy in front of apiservers
If that happens once - that can definitely happen (and that's fine). The bug was that the components were stuck with those errors.
18.0 get same error . restart docker and kubelet return OK. so , should I update the 1.18.0 to 1.18.6+ ?


Question: beta.kubernetes.io/os: linux containers: - name: coredns <del> image: k8s.gcr.io/coredns:1.3.1 </del> <ins> image: k8s.gcr.io/coredns:1.6.2 </ins> imagePullPolicy: IfNotPresent resources: limits:

Which jobs are failing: Multiple jobs on . Which test(s) are failing: on the following jobs: Since when has it been failing: Since 5/17. Testgrid links: See above. Reason for failure: All the failures mentioned above seem to have to have the same root cause, namely failure while waiting for CoreDNS to be ready. Example: /sig testing /priority critical-urgent /kind failing test /milestone v1.15 /cc
Some of this (possible all of it) is due to , which is dependent on being merged.
/sig cluster-lifecycle
Not the gce jobs thoght. Only the kinder job. So the gce jobs are failing with a problem outside of kubeadm/kinder but possibly still related to the corefile plugin change
/assign (At kubecon this week)
The gce failures appear to show the same behavior, but with kube-up instead of kubeadm. I.e. it appears that kube-up may also be retaining old configs when upgrading/downgrading. If thats true, then we may need to integrate the migration tools into kube-up as well.
I have opened which aims to fix all the failing tests except , which will be fixed via
Great job so far and Would it be possible to ask for the open PRs to be merged and/or a solution found soon? We have these tests failing for quite some time and code freeze coming up in the next days. Thank you in advance for your efforts!
thanks for the heads up. let me know if you don't have the time for the refactor that i requested and i can try to take over there. the alternative is to rollback the coredns version which i assume is not ideal, due to a variety of fixes in the new version.
is there any more work planned for this issue? I see that some of the kinder jobs are still failing after merged:
merging and will resolve and close this issue.
Thank you for the update !
updates: kinder tests were fixed via
/unassign i was here for: it's a bit flaky (which is due to the tooling), but green.
Since and we're moved to 1.16, and we're waiting for to merge, can we move this issue to 1.16?
? is that the right PR?

Good catch, I meant
The GCE test failures were fixed by and /close
Closing this issue. <detailsIn response to : The GCE test failures were fixed by and /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: } } <del> // Addresses 5.5.5.5 and 6.6.6.6 should not be bound any more </del> <ins> // Addresses 5.5.5.5 and 6.6.6.6 should not be bound any more, but the ipv6 addresses should remain </ins> remainingAddrs, _ := fp.netlinkHandle.ListBindAddress(DefaultDummyDevice) <del> if len(remainingAddrs) != 4 { t.Errorf("Expected number of remaining bound addrs after cleanup to be %v. Got %v", 4, len(remainingAddrs)) </del> <ins> if len(remainingAddrs) != 6 { t.Errorf("Expected number of remaining bound addrs after cleanup to be %v. Got %v", 6, len(remainingAddrs)) </ins> } <del> // check that address "1.1.1.1", "2.2.2.2", "3.3.3.3", "4.4.4.4" are still bound </del> <ins> // check that address "1.1.1.1", "2.2.2.2", "3.3.3.3", "4.4.4.4" are bound, ignore ipv6 addresses </ins> remainingAddrsMap := make(map[string]bool) <del> for i := range remainingAddrs { remainingAddrsMap[remainingAddrs[i]] = true </del> <ins> for _, a := range remainingAddrs { if net.ParseIP(a).To4() == nil { continue } remainingAddrsMap[a] = true } if !reflect.DeepEqual(activeBindAddrs, remainingAddrsMap) { t.Errorf("Expected remainingAddrsMap %v, got %v", activeBindAddrs, remainingAddrsMap) } } func TestCleanLegacyService6(t *testing.T) { ipt := iptablestest.NewFake() ipvs := ipvstest.NewFake() ipset := ipsettest.NewFake(testIPSetVersion) fp := NewFakeProxier(ipt, ipvs, ipset, nil, []string{"3000::/64", "4000::/64"}) fp.nodeIP = net.ParseIP("::1") // All ipvs services that were processed in the latest sync loop. activeServices := map[string]bool{"ipvs0": true, "ipvs1": true} // All ipvs services in the system. currentServices := map[string]*utilipvs.VirtualServer{ // Created by kube-proxy. "ipvs0": { Address:   net.ParseIP("1000::1"), Protocol:  string(v1.ProtocolUDP), Port:      53, Scheduler: "rr", Flags:     utilipvs.FlagHashed, }, // Created by kube-proxy. "ipvs1": { Address:   net.ParseIP("1000::2"), Protocol:  string(v1.ProtocolUDP), Port:      54, Scheduler: "rr", Flags:     utilipvs.FlagHashed, }, // Created by an external party. "ipvs2": { Address:   net.ParseIP("3000::1"), Protocol:  string(v1.ProtocolUDP), Port:      55, Scheduler: "rr", Flags:     utilipvs.FlagHashed, }, // Created by an external party. "ipvs3": { Address:   net.ParseIP("4000::1"), Protocol:  string(v1.ProtocolUDP), Port:      56, Scheduler: "rr", Flags:     utilipvs.FlagHashed, }, // Created by an external party. "ipvs4": { Address:   net.ParseIP("5000::1"), Protocol:  string(v1.ProtocolUDP), Port:      57, Scheduler: "rr", Flags:     utilipvs.FlagHashed, }, // Created by kube-proxy, but now stale. "ipvs5": { Address:   net.ParseIP("1000::6"), Protocol:  string(v1.ProtocolUDP), Port:      58, Scheduler: "rr", Flags:     utilipvs.FlagHashed, }, } for v := range currentServices { fp.ipvs.AddVirtualServer(currentServices[v]) } fp.netlinkHandle.EnsureDummyDevice(DefaultDummyDevice) activeBindAddrs := map[string]bool{"1000::1": true, "1000::2": true, "3000::1": true, "4000::1": true} // This is ipv6-only so ipv4 addresses should be ignored currentBindAddrs := []string{"1000::1", "1000::2", "3000::1", "4000::1", "5000::1", "1000::6", "1.1.1.1", "2.2.2.2"} for i := range currentBindAddrs { fp.netlinkHandle.EnsureAddressBind(currentBindAddrs[i], DefaultDummyDevice) } fp.cleanLegacyService(activeServices, currentServices, map[string]bool{"5000::1": true, "1000::6": true}) // ipvs4 and ipvs5 should have been cleaned. remainingVirtualServers, _ := fp.ipvs.GetVirtualServers() if len(remainingVirtualServers) != 4 { t.Errorf("Expected number of remaining IPVS services after cleanup to be %v. Got %v", 4, len(remainingVirtualServers)) } for _, vs := range remainingVirtualServers { // Checking that ipvs4 and ipvs5 were removed. if vs.Port == 57 { t.Errorf("Expected ipvs4 to be removed after cleanup. It still remains") } if vs.Port == 58 { t.Errorf("Expected ipvs5 to be removed after cleanup. It still remains") } } // Addresses 5000::1 and 1000::6 should not be bound any more, but the ipv4 addresses should remain remainingAddrs, _ := fp.netlinkHandle.ListBindAddress(DefaultDummyDevice) if len(remainingAddrs) != 6 { t.Errorf("Expected number of remaining bound addrs after cleanup to be %v. Got %v", 6, len(remainingAddrs)) } // check that address "1000::1", "1000::2", "3000::1", "4000::1" are still bound, ignore ipv4 addresses remainingAddrsMap := make(map[string]bool) for _, a := range remainingAddrs { if net.ParseIP(a).To4() != nil { continue } remainingAddrsMap[a] = true </ins> } if !reflect.DeepEqual(activeBindAddrs, remainingAddrsMap) { t.Errorf("Expected remainingAddrsMap %v, got %v", activeBindAddrs, remainingAddrsMap)

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug /sig network /area ipvs /area ipv6 What happened: with IPVS mode creates a interface for binding service IPs. kube-proxy periodically addresses it doesn't know about from the interface. It tries to unbind the IPv6 address over and over. The error is a little vague, but see below. The IP parsing or address deletion code may be a suspect. What you expected to happen: Either ignore the IPv6 address on the interface (its not hurting anything) or remove it correctly. How to reproduce it (as minimally and precisely as possible): Its perfectly natural for a new interface to have an inet6 address by default. If for some reason your interface doesn't have a similar address, you can add one to repro. Anything else we need to know?: Environment: Kubernetes version (use ): v1.12.1 Cloud provider or hardware configuration: AWS, GCP, DigitalOcean, bare-metal, Azure OS (e.g. from /etc/os-release): Container Linux 1855.4.0 Kernel (e.g. ): 4.14.67-coreos Install tools: Typhoon
The error message from is a little vague. Hop on a node to help out with its deletion. This gives a clue about where goes wrong. With IPv4 addresses, leaving off the is permitted (though with a scary warning). Having the wrong prefix size produces the same error. With more verbose logging, we see the mask was stripped before calling 's AddrDel, so the bug lies in kube-proxy's construction of . The mask is dropped . should be (, net.IPNet). However, it looks like much of kube-proxy ipvs deals in string IPs (e.g. 1.2.3.4, 1002:ab8::2:1) so propagating a or adding a mask could be a large change. One might also point to called as the source - it assumes a mask for IPv4 and a mask for IPv6.

Neither is neccessarily right for arbitrary addresses that might end up bound on a interface, but for IPv4 happens to work for Kubernetes IPv4 service IPs. for the IPv6 address does not. We could construct a netlink.IPNet without the helper, but by the time we reach , kube-proxy has already thrown away the mask information.
Any relation with this you think? Doesn't seem so since this is in and the other is ipvs targets...
Hm, I think adapting the sync'er for IPv6 might be a different fix. I was working on a fix for the IPv6 cleanup on the interface, lemme see if I can get that to a state it can be posted.
I can only reproduce this fault if I do; This will automatically assign a link-local generated address to the interface and the problem occurs. But the should not be up. By default if I start v1.12.0 it is down. Is some program, e.g network-manager, brings the interface up? Anyway I think it is possible to suspress auto-generated link-local ipv6 addresses. I will try to find the setting.
After creation; should fix this problem. I will try to get time to do a PR.


Question: var _ volume.DeviceUnmounter = &csiAttacher{} func (c *csiAttacher) Detach(volumeName string, nodeName types.NodeName) error { <ins> _, ok := c.plugin.host.(volume.KubeletVolumeHost) if ok { return errors.New("detaching volumes from the kubelet is not supported") } </ins> var attachID string var volID string

The Kubelet panics when: 's started with the option; a CSI volume is attached to a Node. From the Kubelet logs: Either document that this is not supported (and stop the panic'ing) or support and test this use case. kubelet with the option set. a CSI PVC and use it in a pod. No response <details</details<detailsany </details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage
/assign


Question: args := []string{"-t", string(table)} klog.V(4).Infof("running %s %v", iptablesSaveCmd, args) cmd := runner.exec.Command(iptablesSaveCmd, args...) <del> // Since CombinedOutput() doesn't support redirecting it to a buffer, // we need to workaround it by redirecting stdout and stderr to buffer // and explicitly calling Run() [CombinedOutput() underneath itself // creates a new buffer, redirects stdout and stderr to it and also // calls Run()]. </del> cmd.SetStdout(buffer) <del> cmd.SetStderr(buffer) return cmd.Run() </del> <ins> stderrBuffer := bytes.NewBuffer(nil) cmd.SetStderr(stderrBuffer) err := cmd.Run() if err != nil { stderrBuffer.WriteTo(buffer) // ignore error, since we need to return the original error } return err </ins> } // Restore is part of Interface.

On some systems, iptables-save outputs the following warning to stderr: Note that the error message includes a newline. Because kube-proxy captures both stdout and stderr when calling , the newline might cause rules to be garbled. For example, the output might contain the following line: The parser treats this as a chain line, and panics, as predicted in
/sig network
FYI
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/assign
Just FYI here's another example of garbled output:
Anyway, fix is in Please review.
/assign
/remove-triage unresolved


Question: // Defaults to using only the latest image. Acceptable values are [0, # of images that match the regex). // If the number of existing previous images is lesser than what is desired, the test will use that is available. PreviousImages int `json:"previous_images,omitempty"` <ins> // ImageFamily is the image family to use. The latest image from the image family will be used. ImageFamily string `json:"image_family,omitempty"` </ins> Machine   string    `json:"machine,omitempty"` Resources Resources `json:"resources,omitempty"`

We should support image family in node e2e image config. It makes image validation much easier on GCE.
Cool idea :) Adding a link to for anyone like me who hadn't heard of them before :)


Question: } // DeleteTCPLoadBalancer is an implementation of TCPLoadBalancer.DeleteTCPLoadBalancer. <del> func (gce *GCECloud) DeleteTCPLoadBalancer(name, region string) error { </del> <ins> func (gce *GCECloud) DeleteTCPLoadBalancer(origName, region string) error { name := normalizeName(origName) </ins> op, err := gce.service.ForwardingRules.Delete(gce.projectID, region, name).Do() if err != nil && isHTTPErrorCode(err, http.StatusNotFound) { glog.Infof("Forwarding rule %s already deleted. Continuing to delete target pool.", name)

GCE has a length limit of 63 characters for its load balancer components (target pools and forwarding rules), and AWS has a length limit of 32 characters. Our current load balancer name construction of clustername-namespace-servicename can exceed both limits, especially AWS's. We should use a shorter name construction method less likely to hit these limits.
Thanks for filing this Alex. My suggestion would be to use a simple UID, as per utils.NewUID(). We'll need to store the mapping from {clustername, namespace, servicename} -UID somewhere stable like etcd. Using anything based directly on {cluster, namespace, name}, for example a stable hash of each, has the problem that the aforementioned tuple is not unique over time. This raises the possibility of e.g. the following scenario: creates mycluster/mynamespace/myservicename, which creates external load balancer named stablehash(mycluster,mynamespace,myservicename) deletes mycluster/mynamespace/myservicename, but at the time MyCloudProvider is down, so kubernetes can't delete ELB named stablehash(mycluster,mynamespace,myservicename). Kubernetes keeps trying in the background to delete the ELB. the mean time user creates a new service named the same as her old one mycluster/mynamespace/myservicename. Kubernetes tries to create an associated ELB named stablehash(mycluster,mynamespace,myservicename), but this fails, because the ELB exists already due to 2 above. ... etc. Badness ensues.
Assigning to Chao Xu, as this makes for a good starter bug.
Hmmmm.... can't seem to assign to caesarxuchao yet. Not sure why.
He has to have write privileges on the repo before issues/PRs can be assigned to him.
But we have many contributors with issues assigned to them that are not in the kubernetes-write group (that only has 48 members). I must be missing something. How do I grant him write privileges on the repo?
We do? That's not how I understand it to work: If you want to add him, go .
has been because the load balancer name changes if the service's UID changes (e.g. if it's deleted and recreated).

Think we can change this? A hash of cluster name, namespace, and service name should get around it, but there may be better ideas as well.
Sure. I'm not confident that I can fix this quickly. If it's urgent, maybe you can assign some one more experienced to fix it? One question: I guess the ELB's name should stay unchanged as long as the "zone+namespace+service name" doesn't change, am I right?
I think I'm the guilty party here. I was unaware that when we create a new service with the same name as a previously existing service, that it needed to keep the same load balancer name and IP. This seems broken to me, but I may be missing the larger context. We can quite safely roll back the change while we decide on the best alternative solution. Q , Chao Xu wrote:
I think the real breakage is that we don't have any way to claim an LB-IP at a larger lifespan than a single service. Alex, can you go back to OP and get a sense of what's happening there to trigger this? , Quinton Hoole wrote:
Will do. Even if the name is kept consistent as it was before, with the way everything currently works the IP won't be consistent.
yeah, I'm not positive that the name has to be consistent. What we need is a way to claim the same public IP address, and we already have that (at least for GCE) Assigning to me, and I'll validate the length requirements, make sure public ip assignment works for AWS and close out this issue.
Note there are two separate things here: (1) a purely-name-based construction can exceed length limits given a single service and (2) some form of across-service consistency is apparently needed. (1) is trivially solvable by either always using a hash or alternatively using a two-stage process where you generate a desired name and only hashify if it's longer than the length limits by truncating it and appending a hash of said name.
AWS's load balancer hasn't been merged yet, so there's not much to check out there. I'm going to be reducing the churn of load balancers later today, which should help prevent IP address changes for services that are modified.

I think we need to be able to figure out which Service an LB is attached to from the Service, so that LBs can't get orphaned. regarding being able to claim an LB IP. I don't think that is covered in revamp... , Brendan Burns wrote:
I think that we need to roll back and think this issue through a bit better. See comment on that PR.


Question: } func (r *rbdDriver) CreateDriver() { <del> // Reset config. It might have been modified by a previous CreateVolume call. r.driverInfo.Config.ServerConfig = nil </del> } func (r *rbdDriver) CleanupDriver() {

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: gce/gke multizone/regional jobs Which test(s) are failing: Only PD tests Since when has it been failing: Around 12/21 with the 16:49 run Testgrid link: https://k8s- Reason for failure: TBD Anything else we need to know: <!-- DO NOT EDIT BELOW THIS LINE --/kind failing-test
The runs started flaking after merged. cc Here's one failing run: Pod pod-subpath-test-gcepd-dynamicpv-pfsm failed to schedule because of conflicting node selectors: We shouldn't need to set the node selector for PD tests using PVs.
The PR broke the subpath test because the modification to the config struct embedded in the driver info now persists across tests: A quick fix would be to reset that field in CreateDriver. The long-term fix is the change discussed in My preference is to do the quick fix now, then either merge or close PRs in this order: (needs a decision about logging of tests that aren't getting defined, see ) a future PR for What I'd like to avoid is having to rebase PR on top of the long-term solution for issue - that'll be lots of code conflicts.
/assign PR is here: PR


Question: // MakeRegexpArray splits a comma separated list of regexps into an array of Regexp objects. func MakeRegexpArray(str string) ([]*regexp.Regexp, error) { <ins> if str == "" { return []*regexp.Regexp{}, nil } </ins> parts := strings.Split(str, ",") result := make([]*regexp.Regexp, len(parts)) for ix := range parts {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --- Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig cli
/triage accepted


Question: t.Logf("Running test: %v", tc.name) gotTop, err := translateAllowedTopologies(tc.topology, GCEPDTopologyKey) if err != nil { <del> t.Errorf("Unexpected error: %w", err) </del> <ins> t.Errorf("Unexpected error: %v", err) </ins> } if !reflect.DeepEqual(gotTop, tc.expectedToplogy) {

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Following kubernetes tests are failing make test make test Sample failure error trace Full Test report can be found Incorrect format specifiers used in Errorf in test logging
This we have started seeing with tip golang(1.18) version
/sig storage /sig apps
/triage accepted
/assign


Question: } func TestEnsureLoadBalancerDeletesExistingInternalLb(t *testing.T) { <del> t.Parallel() </del> vals := DefaultTestClusterValues() gce, err := fakeGCECloud(vals) require.NoError(t, err)

Since merged on Friday night, my PR () has failed several retests of pull-kubernetes-bazel-test. My PR is unrelated to the load balancer so I suspect that introduced a flake. I see several runs of that test against master that aren't failing, so clearly the test works sometimes (and the original PR passed it several times to before it merged). Perhaps the test harness sometimes runs problem tests in parallel and other times it doesn't. Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: My PR had one good run of pull-kubernetes-bazel-test before merged, but has failed several times since with no code changes to my PR. What you expected to happen: The tests should consistently pass unless related problem code is introduced.
/cc /cc /sig gcp /kind flake
My PR started failing recently too. See https://k8s-
Hey, thanks for identifying the issue. contains tests for - but the race condition was caused when tests with modified a singleton variable in a mock. The short-term fix is removing the calls, but the long-term fix is to not use a singleton. Fix is .


Question: "strings" "time" <del> "context" </del> "k8s.io/klog" api "k8s.io/api/core/v1"

What happened: Upgraded from 1.16.6 to 1.17.2 and will no longer start on control-plane hosts. It fails with: What you expected to happen: to start, static pods to launch, to connect to in the static pod How to reproduce it (as minimally and precisely as possible): Have the , , , etc in the static pod manifest folder. Try to start the with the container images not preloaded on the host. Just in case it is needed: is used as the backend is run using Anything else we need to know?: This looks to be two things interacting poorly: 1- It takes a long time for static pods to be created for the first time when the can't talk to yet. 2- The new CSI migration code force terminates causing these log lines to appear in : It looks like introduced this new fatal error. Environment: Kubernetes version (use ): 1.17.2 Cloud provider or hardware configuration: Bare-metal OS (e.g: ): Fedora 31..1 (CoreOS preview)" Kernel (e.g. ): 5.3.12-300.fc31.x86_64 Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig cluster-lifecycle
Actually, probably: /sig node
The referenced PR was in 1.14, so that interaction is not new. What is likely new is the size increase of hyperkube in 1.17 (xref ). When not pre-pulled, this can increase the time it takes to start hyperkube for the first time. Does the image pull make progress as the kubelet restarts/retries?
It might make sense for the initializeCSINode() method to: first wait to successfully talk to the kube apiserver at all, perhaps indefinitely after successfully talking to the kube apiserver, then try to register the CSINode object with backoff, and fail if it cannot do that successfully
/sig storage /remove-sig cluster-lifecycle
It doesn't seem to. Once the connection to containerd is closed, it cancels the context. Otherwise I would expect this to succeed eventually if the pull happened in the background. That sounds reasonable to me assuming it isn't blocking the static pod creation code. Do you happen to know why it takes a very long time for the static pod creation code to be called?

I would expect the pods to be created very soon after the kubelet is started.
Pulling the image and starting the processes for those pods happens immediately (from the containerd logs, you can see the image pull happening... it just doesn't complete in time before the independent kubelet logic to register the CSI node info times out).
Ah, yes. I just upgraded a couple of days ago so I didn't check whether I still saw the 2 minute delay in creating the "pod sandboxes" (this is before the pull even happens). It now is only a 23 second delay: containerd: kubelet:
ack, I like the idea Will work on a fix today and cherry-pick it as necessary


Question: } for _, disk := range disks { <del> if disk.Lun != nil && (disk.Name != nil && diskName != "" && *disk.Name == diskName) || (disk.Vhd != nil && disk.Vhd.URI != nil && diskURI != "" && *disk.Vhd.URI == diskURI) || (disk.ManagedDisk != nil && *disk.ManagedDisk.ID == diskURI) { </del> <ins> if disk.Lun != nil && (disk.Name != nil && diskName != "" && strings.EqualFold(*disk.Name, diskName)) || (disk.Vhd != nil && disk.Vhd.URI != nil && diskURI != "" && strings.EqualFold(*disk.Vhd.URI, diskURI)) || (disk.ManagedDisk != nil && strings.EqualFold(*disk.ManagedDisk.ID, diskURI)) { </ins> // found the disk klog.V(2).Infof("azureDisk - find disk: lun %d name %q uri %q", *disk.Lun, diskName, diskURI) return *disk.Lun, nil

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Pods with PVs could not be moved from one node to another. According to the controller-manager logs, K8s tried to detach to Azure disks from the same node. One of the detach operation waited for this lock (). While it waited for the lock, the controller-manager pod was restarted (for whatever reason, not related to this bug, lets assume an operator restarted the pod). After the restart, K8s only tried to attach the disk to the new node but is not trying to detach it first. What you expected to happen: After a restart of the controller-manager, it should continue unfinished detach operations How to reproduce it (as minimally and precisely as possible): 2 Pods with PVs running on one node Move both pods to another node restart the controller-manager shortly after the move Anything else we need to know?: Environment: Kubernetes version (use ): 1.15.1 Cloud provider or hardware configuration: Azure OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig azure
/sig cloud-provider
/area provider/azure that's expected behavior, k8s controller won't continue detach volume if controller manager restarted, user needs to detach that disk volume manually.
Could you please elaborate why this behaviour is expected. I would assume that the controller manager tries to reach the desired state after a restart? Then it would need to detect that the volume is still attached to the wrong node and should try to detach it once more, before it continuing to attach the (still attached) volume? This leads to many ... Why is it required to resolve this situation by manual interaction?
I can't imagine that "manual intervention" is "desired behaviour" (yes, you wrote "expected", but that doesn't matter, if you make no proposal, how to improve this). If this isn't fixed, we can't use Azure, period. We can't have clusters that need to be babysitted / you cannot expect people running workloads on Azure clusters and babysit them and manually fixing what should/can fix itself.

Level-triggered behaviour and resilience is what made Kubernetes so successful... please bring that also to Azure's KCM/CCM.
do you know why controller manager is restarted, it's not a common scenario, we cannot guarantee disk attach/detach functionality works if controller manager restarted at that time.
related to
I think there can be many reasons why the controller-manager is restarted during running detach or attach operations e. g. the hosting machine turn into error state, if you run the controller-manager in k8s -vertical pod autoscaling,... etc. I understand you can't guarantee that attach/detach works when the controller-manager is restarted, but I don't understand why the controller-manager can't recover the situation autonomously after a restart. After restart it tries to attach the volume of the rescheduled pod and this does not work because the volume is still attached to another machine. This is indicated by a error from the arm and the controller-manager could react to it properly, right?
This is an upstream issue, and there is a discussion in , let's check whether we can fix it in cloud provider driver in that thread. Thanks.
I will try to fix this issue in azure disk driver, thanks all for the discussion. details: /assign
Thank you very much It means a lot to us, that there are no manual operations. I wonder how you guys deal with it in AKS, since Azure uses the same controllers, right? Anyway, it helps tremendously, that the clusters are self-healing.
AKS use same kube-controller-manager, while it won't be restarted easily.
OK, but how? I understand that AKS only supports 100 nodes. So, I guess you have some upper limit and just hope, nobody stresses it, because if this happens, you it either restarts or you need to scale it up and then it gets restarted as well. Also, regular maintenance, low-level as well as Kubernetes updates, all must impact it. Considering the probably thousands over thousands of clusters (I don't know, but I expect you run many), any such problem, even if unlikely by itself, must be in your interest to fix, simply because the number of clusters increases the likelihood that it happens, I would think. Anyway, thank you for making this change
Yes, that's reasonable, I will try to fix it as my top priority.


Question: klog.Fatalf("%v", err) } lastScheduledCount := len(podsScheduled) <ins> ticker := time.NewTicker(throughputSampleFrequency) defer ticker.Stop() </ins> for { select { case <-ctx.Done(): return <del> // TODO(#94665): use time.Ticker instead case <-time.After(throughputSampleFrequency): </del> <ins> case <-ticker.C: </ins> podsScheduled, err := getScheduledPods(tc.podInformer, tc.namespaces...) if err != nil { klog.Fatalf("%v", err)

/sig scheduling /sig testing /assign
/kind cleanup
May I give a try?
please go ahead. Un-assigning myself.
/assign What should I do just replace with ?
please see the go docs for and the example they have there: They do a better job than I possibly can :)


Question: ctx, cancel := context.WithCancel(context.Background()) defer cancel() // 3. Populate vcToZoneHostsMap and vcToHostDatastoresMap <del> for _, nodeInfo := range nameToNodeInfo { </del> <ins> for _, nodeInfo := range nm.nameToNodeInfo { </ins> vc := nodeInfo.VSphere.Config.Hostname host := nodeInfo.HostSystemRef.Value for _, zone := range nodeInfo.Zones {

Command run: vSphere intree volume test suite occasionally failed with the following error: The test suite should always pass. We may have a chance to hit this by running ` No response binary version: 1.27.4 </details<detailsvsphere </details<details</details<details</details<details</details<details</details
/assign
/sig testing storage ?
/triage accepted


Question: INDIRECT=$$(go list -e                                 -f '{{range .Deps}}{{.}}{{"n"}}{{end}}'           ./cmd/libs/go2idl/conversion-gen                   <del> | grep "^$(PRJ_SRC_PATH)"                          </del> <ins> | grep --color=never "^$(PRJ_SRC_PATH)"            </ins> | sed 's|^$(PRJ_SRC_PATH)|./|'                     | xargs go list -e -f '{{.Dir}} {{.Dir}}/*.go');   echo $$DIRECT $$INDIRECT

Using the grep from OSX, fails when GREPOPTIONS/GREPCOLOR is set as shown below (). Workaround : unset GREPOPTIONS/GREPCOLOR Note that using GNU grep using brew install will show that GREPOPTIONS shows immediately that GREP_OPTIONS may be interfering with make ()
oh, here's badly manged conversion-
Will look, thanks. For make-related bugs, please @ me directly.
Will do. thanks
This breaks building kubernetes inside Alpine Linux:
I'm pressing more and more on the idea that the only supported build is the dockerized build. Trying to accomodate every random environment is just too taxing. , Marc Villacorta wrote:
I agree with We can support the stuff in to run on mac to get you dockerized but everything else is only supported in there. I've worked around a few BSD/OSX things in but it isn't fun. We should try to keep that stuff isolated.
I agree, with the usual caveat that we need to be doing all of the build in one container with as little as possible happening outside. OS X tends to have old and broken versions of important programs, which is terrible. We discussed setting up CI for it but it's not going to happen in the near term. Alpine/Busybox purposefully omit features from their versions for good reasons, but it's still more work to support than it's worth.
Ok, I will re-think my workflow.
How did you find yourself building in Alpine, in I may ask? , Marc Villacorta wrote:
nothing is forcing me to use Alpine I just like the distro. I run stuff on CoreOS and my aim is to solve the build-deliver-run pipeline with a single small container. By the way, I have the problem installing the GNU package :-)


Question: --> **What type of PR is this?** <del> > Uncomment only one, leave it on its own line: </del> <ins> > Uncomment only one ` /kind <>` line, hit enter to put that in a new line, and remove leading whitespaces from that line: </ins> > > /kind api-change > /kind bug

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: The PR template is a little unclear to new contributors. EG: If I understand what "uncomment" is, and leave it on it's own line I get: which has two problems: I probably left the space after the (so the bot regex won't match) I probably didn't put a newline before the uncommented line, which is necessary for it to not just continue to be a comment block: What type of PR is this? /kind cleanup vs What type of PR is this? /kind cleanup Another issue is in this bit: release-note release-note NONE If you're not an experienced contributor, it's not very clear where you should write "NONE" and what kind of quoting you should use, etc. What you expected to happen: I hope for the PR template to be a little more obvious and easy to use How to reproduce it (as minimally and precisely as possible): create a new PR Anything else we need to know?: /sig contributor-experience cc Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
/remove-kind cleanup
/assign
GitHub didn't allow me to assign the following users: paralika. Note that only and repo collaborators can be assigned. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I would like to work on this issue.
was an improvement but many of the original observations still stand. Here's what I found looking at 75 latest PRs (~67 human-written ones if you discount 8 "Automated cherry pick"): The instruction to "uncomment only one /kind line" seems to suggest keeping all the other options present with prefix, but few people do that! I found just 7, of which: 2 PRs retained that whole section of the template, not picking a kind at all.

1 of these still got kind label magically set by the bot (possibly from linked issue?) I confirmed (tip: ⋯ -Quote reply lets you see the markdown source) these were NOT people that removed but forgot newline, getting bitten by markdown "lazy continuation" rule — they retained on all lines. 3 PRs uncommented 1 line. 2 PRs uncommented multiple lines, resulting in multile kind/... labels! Example: I don't know if you'll want to encourage "Uncomment one or more", or want to keep it a "hidden feature" for experienced users? However, from the (secondary) angle "template teaches people how the project/infrastructure works", the "only one" wording is slightly misleading — it obscures the fact /kind just sets a label, similar to /area, /sig etc. (and from what I saw, multiple areas or SIGs are common?) One hypothesis is people feel leaving the rest is noise that detracts from writing & reading the description? Weak evidence that's not the only motivation: a few (didn't count, I'd say ~3) retained the instructions line, which is clearly noise, e.g.: I don't think we can measure this, but see below. Found 3 of these, these are actual failures of the process! ~5% of people took the effort to choose a kind but didn't understand the "uncomment" instruction :cryingcatface: (bot assigned kind/cleanup anyway, not sure how)
It seems delete-all-irrelevant mechanic is better than "uncomment" mechanic. It avoids markdown pitfall, the end result has less noise, and most people choose it anyway. What if the template had them without prefixes? This would prevent mistakes of the last kind, where prevented bot from seeing contributor's choice! A downside is that when somebody is lazy / misses this step, they'll create PRs with ALL kind/ labels :worried: Is the bot smart about /commands inside fenced blocks? Or HTML comments? If yes, could use them as a signal that you made a conscious choice: I've also seen projects that use checklists in templates: downsides: Would require bot support to understand it.

It's slightly worse on the "template teaches people how the project/infrastructure works" angle — you won't learn from that these are regular /commands you can just write wherever. Misses on benefits of "delete" mechanic. People will likely delete anyway, with same mild confusion about two-ways-to-do-it. Well, this was too many words to suggest such small tweak :laughing:, but: exactly what to do, rather than vague "uncomment". Not sure if this will reduce mistakes of the last kind. space between avoids risk of leaving space mentioned in OP, without having to explain it. removing the rest =less worries, no markdown pitfall.
OK, that answers the question about bot & fenced blocks :grin: ^^^^^ /remove-kind api-change /remove-kind cleanup /remove-kind documentation /remove-kind failing-test /remove-kind feature
/remove-kind design /remove-kind flake


Question: if [[ -x ./hack/jenkins/upload-to-gcs.sh ]]; then ./hack/jenkins/upload-to-gcs.sh else <del> curl -fsS --retry 3 "https://raw.githubusercontent.com/kubernetes/kubernetes/master/hack/jenkins/upload-to-gcs.sh" | bash - </del> <ins> curl -fsS --retry 3 --keepalive-time 2 "https://raw.githubusercontent.com/kubernetes/kubernetes/master/hack/jenkins/upload-to-gcs.sh" | bash - </ins> fi

http://pr- Seems like a flake? Could probably retry this, but not sure whether it's worth it.
we most definitely want to retry! Will you send a PR?
I just hit this in PR
We should close this, should be fixed by:
that didn't actually solve the problem. Per the docs, only works on "transient" errors, which the manpage defines as We might want to try passing , which some googling suggests might help. The default is 60s, which may be too long.
Please reactive this if you see it after June 15


Question: return *s } <del> // convertStatusToAPIStatus creates an api PodStatus for the given pod from // the given internal pod status.  It is purely transformative and does not // alter the kubelet state at all. func (kl *Kubelet) convertStatusToAPIStatus(pod *v1.Pod, podStatus *kubecontainer.PodStatus) *v1.PodStatus { var apiPodStatus v1.PodStatus // The runtime pod status may have an arbitrary number of IPs, in an arbitrary // order. Pick out the first returned IP of the same IP family as the node IP // first, followed by the first IP of the opposite IP family (if any). podIPs := make([]v1.PodIP, 0, len(podStatus.IPs)) </del> <ins> // sortPodIPs return the PodIPs sorted and truncated by the cluster IP family preference. // The runtime pod status may have an arbitrary number of IPs, in an arbitrary order. // PodIPs are obtained by: func (m *kubeGenericRuntimeManager) determinePodSandboxIPs() // Pick out the first returned IP of the same IP family as the node IP // first, followed by the first IP of the opposite IP family (if any) // and use them for the Pod.Status.PodIPs and the Downward API environment variables func (kl *Kubelet) sortPodIPs(podIPs []string) []string { ips := make([]string, 0, 2) </ins> var validPrimaryIP, validSecondaryIP func(ip string) bool if len(kl.nodeIPs) == 0 || utilnet.IsIPv4(kl.nodeIPs[0]) { validPrimaryIP = utilnet.IsIPv4String

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --In a dual-stack cluster with IPv4 as default family (first kubelet IP), a container is created with a env var set to . When the container is running, properly contains the IPv4 But at runtime, this env var (used in container args) is expanded to the pod IPv6 instead of the IPv4. When using , it should always return the value actually exposed as and not something else. Kubernetes version:Cloud provider or hardware configuration: Bare metal cluster OS (e.g: ): Ubuntu 20.04 Kernel (e.g. ): Linux 5.8.0-59-generic 66~20.04.1-Ubuntu SMP Thu Jun 17 11:14:10 UTC 2021 x8664 x8664 x86_64 GNU/Linux Network plugin and version (if this is a network-related bug): cilium 1.10 Others: CRI-O 1.21 + crun.
Note: I have a configuration that matches the ones that had IP ordering issue before was fixed. So it may be related.
/sig network
Just a guess, but I wonder if it may be caused by
That's it. The fix from properly sort IP when computing the Pod status, but the sorted IPs are then replaced by computed IPs as this log shows.
/triage accepted Thanks for the detailed report
/assign


Question: "google.golang.org/grpc" <ins> "k8s.io/apimachinery/pkg/util/wait" </ins> "k8s.io/klog/v2" pluginapi "k8s.io/kubelet/pkg/apis/deviceplugin/v1beta1" watcherapi "k8s.io/kubelet/pkg/apis/pluginregistration/v1"

Which jobs are flaking: pull-kubernetes-bazel-test Which test(s) are flaking: TestDevicePluginReRegistrationProbeMode Testgrid link: Reason for failure: Anything else we need to know: Reproduce with /sig node
/assign
See
not sure this is actually fixed, I see a failure from today


Question: cleanupPods(cs, t, []*v1.Pod{pod}) } } <ins> // TestPreemptWithPermitPlugin tests preempt with permit plugins. func TestPreemptWithPermitPlugin(t *testing.T) { // Create a plugin registry for testing. Register only a permit plugin. registry := framework.Registry{permitPluginName: NewPermitPlugin} // Setup initial permit plugin for testing. plugins := &schedulerconfig.Plugins{ Permit: &schedulerconfig.PluginSet{ Enabled: []schedulerconfig.Plugin{ { Name: permitPluginName, }, }, }, } // Set permit plugin config for testing pluginConfig := []schedulerconfig.PluginConfig{ { Name: permitPluginName, Args: runtime.Unknown{}, }, } // Create the master and the scheduler with the test plugin set. context := initTestSchedulerWithOptions(t, initTestMaster(t, "preempt-with-permit-plugin", nil), false, nil, registry, plugins, pluginConfig, false, time.Second) defer cleanupTest(t, context) cs := context.clientSet // Add one node. nodeRes := &v1.ResourceList{ v1.ResourcePods:   *resource.NewQuantity(32, resource.DecimalSI), v1.ResourceCPU:    *resource.NewMilliQuantity(500, resource.DecimalSI), v1.ResourceMemory: *resource.NewQuantity(500, resource.DecimalSI), } _, err := createNodes(cs, "test-node", nodeRes, 1) if err != nil { t.Fatalf("Cannot create nodes: %v", err) } perPlugin.failPermit = false perPlugin.rejectPermit = false perPlugin.timeoutPermit = false perPlugin.waitAndRejectPermit = false perPlugin.waitAndAllowPermit = true perPlugin.allowPermit = true lowPriority, highPriority := int32(100), int32(300) resourceRequest := v1.ResourceRequirements{Requests: v1.ResourceList{ v1.ResourceCPU:    *resource.NewMilliQuantity(400, resource.DecimalSI), v1.ResourceMemory: *resource.NewQuantity(400, resource.DecimalSI)}, } // Create two pods. waitingPod, err := createPausePod(cs, initPausePod(cs, &pausePodConfig{Name: "waiting-pod", Namespace: context.ns.Name, Priority: &lowPriority, Resources: &resourceRequest})) if err != nil { t.Errorf("Error while creating the waiting pod: %v", err) } // Wait until the waiting-pod is actually waiting, then create a preemptor pod to preempt it. wait.Poll(10*time.Millisecond, 30*time.Second, func() (bool, error) { w := false perPlugin.fh.IterateOverWaitingPods(func(wp framework.WaitingPod) { w = true }) return w, nil }) preemptorPod, err := createPausePod(cs, initPausePod(cs, &pausePodConfig{Name: "preemptor-pod", Namespace: context.ns.Name, Priority: &highPriority, Resources: &resourceRequest})) if err != nil { t.Errorf("Error while creating the preemptor pod: %v", err) } if err = waitForPodToSchedule(cs, preemptorPod); err != nil { t.Errorf("Expected the preemptor pod to be scheduled. error: %v", err) } if _, err := getPod(cs, waitingPod.Name, waitingPod.Namespace); err == nil { t.Error("Expected the waiting pod to get preempted and deleted") } if perPlugin.numPermitCalled == 0 { t.Errorf("Expected the permit plugin to be called.") } perPlugin.reset() cleanupPods(cs, t, []*v1.Pod{waitingPod, preemptorPod}) } </ins>

<!-- Please only use this template for submitting enhancement requests --What would you like to be : When preempting a pod that is a that permit plugins are waiting for , we need send a message(actually send a ) to the , otherwise permit plugins might wait until time out, and do not know the pod has been preempted. Why is this needed: /sig scheduling /kind cleanup /assign
cc
Interesting interaction between the permit and preempt, I will take a look at the PR you sent, thanks!


Question: return nodes == 1 }, time.Minute, time.Second).Should(gomega.BeTrue()) <ins> v1alphaPodResources, err = getV1alpha1NodeDevices() framework.ExpectNoError(err, "should get node local podresources by accessing the (v1alpha) podresources API endpoint") v1PodResources, err = getV1NodeDevices() framework.ExpectNoError(err, "should get node local podresources by accessing the (v1) podresources API endpoint") // Before we run the device plugin test, we need to ensure // that the cluster is in a clean state and there are no // pods running on this node. gomega.Expect(v1alphaPodResources.PodResources).To(gomega.BeEmpty(), "should have no pod resources") gomega.Expect(v1PodResources.PodResources).To(gomega.BeEmpty(), "should have no pod resources") </ins> ginkgo.By("Scheduling a sample device plugin pod") data, err := e2etestfiles.Read(SampleDevicePluginDSYAML) if err != nil {

Device Plugin [Feature:DevicePluginProbe][NodeFeature:DevicePluginProbe][Serial] DevicePlugin [Serial] [Disruptive] Can schedule a pod that requires a device It appears a bit flaky on testgrid, but more recently been failing more often No response example - No response /sig node
/cc
/triage accepted /assign /priority important-soon
I ran this test locally 15 times back to back and did not run into this failure. The test passed locally. I suspect this to be a case where pod cleanup is not happening properly. In this test, we are relying on the number of PodResources and based on the error it looks like we have an addtional pod running other than the two expected pods (device plugin pod and test pod requesting device) but can't be 100% sure because we don't have enough logs to support that claim. Going to add more logs to expose output obtained from the podresource API endpoint.


Question: func (rc *ResourceConsumer) makeConsumeCustomMetric() { defer GinkgoRecover() <ins> rc.stopWaitGroup.Add(1) defer rc.stopWaitGroup.Done() </ins> sleepTime := time.Duration(0) delta := 0 for {

https://k8s-
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
might be a culprit, but maybe not
This could be related to how the test initializes the framework or cleans it up, maybe.
Another failure in the same run, not just this one test:
I'm not positive but the stack trace might be wrong due to inlining optimizations. rc.framework == nil was my first guess, but that would fail earlier in that method.
Framework AfterEach runs namespace deletion, which may mean that .Namespace is nil (if inlining is indeed present).
I am looking into this right now.
It's unrelated. I reverted from the current master HEAD locally (via ), then did and finally and it did not help. The test still fails with the same message "Expected <*v1.Initializers | 0x0: nil not to be nil" (location: )
Note that I have executed one of the initializers e2e test (not autoscaling), because it is one of the tests reported by https://k8s- as panicked.
Looking a bit further I found out that commit the test above as already failing.
I have just realized that I have discovered yesterday a race condition in ResourceConsumer implementation, and might actually make it worse. Let me quickly make a PR out of a fix.
Let me try to recap what I have found out so far: The test that is being referred to at the top of this issue () fails "out of the box" and probably deserves a separate issue to track it Other failing tests that look like they have a similar failure pattern (NPE) fail because of trying to send a "consume CPU" request to a non-existent pod (unrelated to above, as far as I can tell) PR might cause the effect above (and probably does) because it actually deletes the pods that should have been deleted However, the root cause is that there is a race condition inside ResourceConsumer that makes "consume CPU" requests being sometimes sent already after "CleanUp" got called (fix prepared in PR )
Initializer was fixed in a different PR, depending on what cluster you test on.

, Andrzej Wasylkowski < :
Yeah, confirmed initializers test is now passing on GCE and disabled on GKE (which doesn't enable initializers configuration group yet) , Clayton Coleman <: The test that is being referred to at the top of this issue () fails "out of the box" and probably deserves a separate issue to track it Other failing tests that look like they have a similar failure pattern (NPE) fail because of trying to send a "consume CPU" request to a non-existent pod (unrelated to above, as far as I can tell) PR cause the effect above (and probably does) because it actually deletes the pods that should have been deleted However, the root cause is that there is a race condition inside ResourceConsumer that makes "consume CPU" requests being sometimes sent already after "CleanUp" got called (fix prepared in PR ) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub , or mute the thread
Clayton, can you assign this to me? I cannot do this myself.


Question: // the provided timer until the provided context is cancelled, the condition returns // true, or the condition returns an error. If sliding is true, the period is computed // after condition runs. If it is false then period includes the runtime for condition. <del> // If immediate is false the first delay happens before any call to condition. The // returned error is the error returned by the last condition or the context error if // the context was terminated. </del> <ins> // If immediate is false the first delay happens before any call to condition, if // immediate is true the condition will be invoked before waiting and guarantees that // the condition is invoked at least once, regardless of whether the context has been // cancelled. The returned error is the error returned by the last condition or the // context error if the context was terminated. </ins> // // This is the common loop construct for all polling in the wait package. func loopConditionUntilContext(ctx context.Context, t Timer, immediate, sliding bool, condition ConditionWithContextFunc) error {

Calling wait.PollUntilContextTimeout with immediate=true does not invoke the passed condition function if context has already been cancelled. wait.PollUntilContextTimeout with immediate=true should invoke the passed condition function at least once even if context has already been cancelled as wait.PollUntilContextCancel documentation says: Note that wait.PollUntilContextTimeout wraps wait.PollUntilContextCancel, so the above should be applied to wait.PollUntilContextTimeout as well. This program panics because context ctx.Err is checked before invoking the passed function. The deprecated wait.PollImmediateWithContext does not have this kind of issue. If wait.PollUntilContextTimeout is the replacement of wait.PollImmediateWithContext, it should exhibit the same behavior. k8s v1.27.2 <details</details<details</details<details</details<details</details<details</details<details</details
/sig api-machinery
TestloopConditionUntilContextsemantic also has an issue that immediate and sliding are not tested as expected. immediate and sliding are loop variable, but they are not used in the test. The ones that are used are in testcase struct which are always false.
/assign
ok, it seems this would do it
/assign
/triage accepted


Question: func (handler *fakeIOHandler) ReadDir(dirname string) ([]os.FileInfo, error) { switch dirname { case "/dev/disk/by-path/": <del> f := &fakeFileInfo{ </del> <ins> f1 := &fakeFileInfo{ </ins> name: "pci-0000:41:00.0-fc-0x500a0981891b8dc5-lun-0", } <del> return []os.FileInfo{f}, nil </del> <ins> f2 := &fakeFileInfo{ name: "fc-0x5005076810213b32-lun-2", } f3 := &fakeFileInfo{ name: "abc-0000:41:00.0-fc-0x5005076810213404-lun-0", } return []os.FileInfo{f1, f2, f3}, nil </ins> case "/sys/block/": f := &fakeFileInfo{ name: "dm-1",

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Currently, we have logic here to look for the disk names with but this fails when we have devices which are starting with e.g: on ppc64le architecture, NPIV based fc disks are discovered as follows: and here is the udev information for one of the disk: Because of the existing logic k8s fails to detect the disk because it does not start with instead starts with , so pod creation fails with the following error: What you expected to happen: FC disk should get attached properly to the pod How to reproduce it (as minimally and precisely as possible): sample pod yaml on a machine with path names starts with Anything else we need to know?: Environment:
/sig storage
/triage accepted


Question: func TestPathBuilderWithMultiple(t *testing.T) { b := NewBuilder(latest.RESTMapper, api.Scheme, fakeClient()). FilenameParam(false, "../../../examples/guestbook/redis-master-controller.yaml"). <del> FilenameParam(false, "../../../examples/guestbook/redis-master-controller.yaml"). </del> <ins> FilenameParam(false, "../../../examples/pod"). </ins> NamespaceParam("test").DefaultNamespace() test := &testVisitor{}

It works when I pipe but not with filename.
broke in this commit cc
Did you try to add an extension .json to the file? We filter for file extensions
ahhh ok. so working as intended. it used to autodetect :( ... thanks
That was unintended. The file extensions should only be used for deciding which files in a directory to process. For a single file, any extension should be allowed. We don't actually infer the content type from the extensions. For stdin and http sources, for instance, there are no extensions. cc
Fix is pretty straightforward (ignore file extensions when dealing with single file); I have it working with manual test, just adding unit test. Will send shortly.
Thanks, but this is v1.0-post, so no rush


Question: podLimits := sum(containerLimits) for k, v := range limit.Min { if err := minConstraint(limitType, k, v, podRequests, podLimits); err != nil { <del> return err </del> <ins> errs = append(errs, err) </ins> } } for k, v := range limit.Max { if err := maxConstraint(limitType, k, v, podRequests, podLimits); err != nil { <del> return err </del> <ins> errs = append(errs, err) </ins> } } for k, v := range limit.MaxLimitRequestRatio { if err := limitRequestRatioConstraint(limitType, k, v, podRequests, podLimits); err != nil { <del> return err </del> <ins> errs = append(errs, err) </ins> } } } } <del> return nil </del> <ins> return errors.NewAggregate(errs) </ins> }

When a resource violates the limitrange, both for cpu and memory, only one error will be returned at a time. That means when a user is creating such a resource, first it will fail and return something like , them user may edit the cpu usage and create again, then fail again and get . It seem admission controls want to reject the request as fast as possible, if error occurs. But in this case, it may give users a bad experience. Would it be better to aggregate all errors and return them at one time. Also, the error message may cause misunderstanding. , it means the resource requested is 3, it may also cause a confusion that it is limited to 3.
Closing since it's fixed


Question: origStateIsCurrent = true continue } <del> return decode(s.codec, s.versioner, origState.data, out, origState.rev) </del> <ins> if len(txnResp.Responses) == 0 || txnResp.Responses[0].GetResponseDeleteRange() == nil { return errors.New(fmt.Sprintf("invalid DeleteRange response: %v", txnResp.Responses)) } deleteResp := txnResp.Responses[0].GetResponseDeleteRange() if deleteResp.Header == nil { return errors.New("invalid DeleteRange response - nil header") } return decode(s.codec, s.versioner, origState.data, out, deleteResp.Header.Revision) </ins> } }

While working on generalizing tests to somehow better define spec for storage interface (and share them across all implementation of storage.interface - in particular etcd3 and watchcache), I found an unexpected inconsistency in Delete operation in etcd3. What is happening is that it effectively returns the state of the object from just before the deletion: However, this is inconsistent with how our watch is working, which is returning the state of the object from just before the deletion, but it sets the RV to the moment of deletion (which Delete call isn't doing). This sounds like a bug to me, but given this is a long-standing behavior we may need to take additional care when fixing this. /kind bug /sig api-machinery /priority important-soon /cc
I opened as a fix for it.
/triage accepted


Question: } else if err != nil { t.Errorf("%d: expected nil error, got %v", i, err) } <del> if !reflect.DeepEqual(s, test.expected) { t.Errorf("%d: expected %+v, got %+v", i, test.expected, s) </del> <ins> if !reflect.DeepEqual(*v.value, test.expected) { t.Errorf("%d: expected %+v, got %+v", i, test.expected, *v.value) </ins> } if v.changed != test.changed { t.Errorf("%d: expected %t got %t", i, test.changed, v.changed)

I ran and it panicked: no panic The panic was introduced by . I submitted as an attempt to fix this, but review led to the realization that there were further bugs and I don't know what the right fix is cc
/assign
/sig testing /sig cloud-provider
/triage accepted


Question: "apiVersion": "v1beta1", "id": "service-${version}-test", "port": 80, <del> "protocol": "TCP", "selector": { "version": "old" } </del> <ins> "protocol": "TCP" </ins> } __EOF__ # Post-condition:redis-master-service service is running kube::test::get_object_assert services '{{range.items}}{{.id}}:{{end}}' 'kubernetes:kubernetes-ro:redis-master:service-.*-test:' <del> ### Update selector of service-${version}-test # Pre-condition: selector.version == old kube::test::get_object_assert "service service-${version}-test" "{{range.selector}}{{.}}{{end}}" 'old' </del> # Command <del> kubectl update service "${kube_flags[@]}" service-${version}-test --patch="{"selector":{"version":"${version}"},"apiVersion":"${version}"}" </del> <ins> kubectl update service "${kube_flags[@]}" service-${version}-test --patch="{"selector":{"my":"test-label"},"apiVersion":"v1beta1"}" </ins> # Post-condition: selector.version == ${version} # This test works only in v1beta1 and v1beta2 # https://github.com/GoogleCloudPlatform/kubernetes/issues/4771 <del> if [[ "$version" != "" ]] && [[ "$version" != "v1beta3" ]]; then kube::test::get_object_assert "service service-${version}-test" "{{range.selector}}{{.}}{{end}}" "${version}" fi </del> <ins> kube::test::get_object_assert "service service-${version}-test" "{{range.selector}}{{.}}{{end}}" "test-label" </ins> ### Identity kubectl get service "${kube_flags[@]}" service-${version}-test -o json | kubectl update "${kube_flags[@]}" -f -

contains a list of commands to test. Those commands are run in a loop of API version. Among those commands, there is the following one: This command is supposed to update the version selector of the service. The command that is just after shows: The on the first and last lines indicate that the update failed.
I can look at this.
For the case where ${version}="" the patch is which is invalid as there is no apiversion. For the case where ${version}="v1beta3" the patch is which is a valid patch but has been moved to in the v1beta3 api, so the patch doesn't actually change the service object. Test is fixed with attached PR. The error for the first is easily caught by validating a patch has an api version. The second is less easy as json.Unmarshal() doesn't error when there are fields in the json that are not present in the struct. The test is fixed by treating the patch json how we treat the service json: keep the apiVersion invariant over the loops so we don't have to worry about changes in the api schema.


Question: glog.Errorf("Unable to convert pod manifest: %v", err) } <del> events, _ := d.Events(namespace).Search(pod) </del> <ins> var events *api.EventList if ref, err := api.GetReference(pod); err != nil { glog.Errorf("Unable to construct reference to '%#v': %v", pod, err) } else { ref.Kind = "" // Find BoundPod objects, too! events, _ = d.Events(namespace).Search(ref) } </ins> return tabbedString(func(out io.Writer) error { fmt.Fprintf(out, "Name:t%sn", pod.Name)

We introduced UID / UUID to provide time-based uniqueness, but UID / UUID are not user friendly. We are going to have this kind of issues in general at client side.
Yeah, describe does a "get" first, so it has the UID available.
Pretty sure this is a feature until we have a way to lookup dead pods. On Nov 21, 2014 2:47 PM, "Daniel Smith" wrote:


Question: return false, nil } if err != nil { <del> return false, err </del> <ins> // This could be a connection error so we want to retry, but log the error. glog.Errorf("Error when reading pod %q: %v", podID, err) return false, nil </ins> } if pod.Status.Phase != api.PodRunning { <del> return false, errors.New(fmt.Sprintf("Pod status is %q", pod.Status.Phase)) </del> <ins> return false, nil </ins> } return true, nil }

The bug is around line 261 in "" It seems that the test is expecting a file just written to have appeared immediately, and that doesn't seem to actually occur. Here's the chunk of code I'm talking about: When run on some systems (it seems to depend on something like filesystem, exact arch, whether you're in boot2docker, etc.) this eventually outputs the following error: F0324 19:38:20. 696 ] static-pod-from-manifest FAILED: mirror pod has not been created or is not running: Pod status is "Unknown" !!! Error in hack/test- '"1"' exited with status 255 Call stack: When I add "time.Sleep(30 * time.Second)" it works, on at least one system that it did not work on previously. If a sleep fixes it, it must be a race. I'm not sure of the best solution, the sleep works, but is of course absolutely horrid. thanks!
Hi thanks for reporting the bug! The test was supposed to poll every second until the 2 minutes timeout, so the extra sleep should not be needed. There was a bug causing problems, and it was fixed by yesterday. Are you running HEAD? If not, could you try running again with HEAD? Thanks!
Actually I think there is a bug in podRunning method. It never waits for PodRunning phase - it returns error on the first check. I'll send PR to fix this.
Indeed... my hostname is "slate", all lower-case, and this is on linux. I should've mentioned that. But, it looks like you already caught the real race. Thanks for the fix!


Question: metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/util/intstr" "k8s.io/kubernetes/pkg/api/v1" <ins> appsv1beta1 "k8s.io/kubernetes/pkg/apis/apps/v1beta1" </ins> ) // describes the attributes of a scale subresource

Here are the hash label keys of all controllers: Deployment: DaemonSet: (new in 1.7) StatefulSet: (new in 1.7) Had a brief discussion with and about this, it should be consistent. We can use for DaemonSet and StatefulSet, and keep Deployment as-is. Deployment hash label key should be different from other two because Deployment pods hash labels is used by selector (unlike the other two controllers). We probably should have used hash annotations instead of hash labels for DaemonSet and StatefulSet, since we don't use hash labels to select their history or pods. Hash labels are chosen for consistency. This is fine since we don't modify hash label values.
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
Actually, controller revisions created by statefulset, daemonset has different label: // controller revision created by statefulset label: "" // controller revision created by daemonset label: controller-revision-hash: "" should be consistent?


Question: if len(conflictTaints) > 0 { return fmt.Errorf("can not both modify and remove the following taint(s) in the same command: %s", strings.Join(conflictTaints, ", ")) } <del> return nil </del> <ins> return o.validateFlags() </ins> } // RunTaint does the work

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): No What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): taint, selector Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-03-28T16:36:33Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-04-03T20:33:27Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"} Environment: Cloud provider or hardware configuration: GKE OS (e.g. from /etc/os-release): MacOS client, linux server Kernel (e.g. ): Install tools:gcloud Others: What happened: I tried to kubectl taint some nodes using a label selector: kubectl taint nodes -l dedicated=foo dedicated=foo:PreferNoSchedule and got the error: error: at least one resource name must be specified since 'all' parameter is not set What you expected to happen: Those nodes would be tainted with dedicated=foo:PreferNoSchedule. Trying to use --all along with the selector is also not supported.

How to reproduce it (as minimally and precisely as possible): kubectl taint nodes -l <any label selectordedicated=foo:PreferNoSchedule Anything else we need to know:
can you post the output of kubectl taint nodes -l myLabel=X dedicated=foo:PreferNoSchedule (((And first of course Make sure that myLabel is on the cluster ....)))
Would you like me to explicitly use the label "myLabel"? Otherwise that's the command and output I posted in "What happened" above. (I had set a label dedicated=foo ahead of time on two nodes.)
Ah ok so the taint k/v is the exact same as the label k/v, that makes the example tricky to parse.... but I agree .... this should work .. I can try to reproduce this....
Yup, the same k/v. (I wanted both labels and taints to determine this set of nodes, and it seemed like it would be confusing to have different k/v's for each. I kinda thought this would be the common case...)
Ah , yeah i see it to ... Even though clearly, the selector exists.
Is the taints and tolerations expert on our team. ravi can you look at ? this logic might be shadowing the label selector, such that its impossible to use labels to select the nodes you're trying to taint.
cc. m i definitely think this is a bug ... ? Looks like the taints maybe are shadowing the label selectors above in
/assign
GitHub didn't allow me to assign the following users: ravisantoshgudimetla. Note that only can be assigned.. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. I understand the commands that are listed . </details
/assign
It seems the line is the culprit. Here we are checking for resources if they are 2, the resources allowed are node,nodes,no so if it needs it has to be 1 and this check is already available So, I guess its safe to remove that. I tested that functionality after removing this and it fixed the issue. I will submit a PR for this unless someone has objections.


Question: preStopWaitTimeout   = 30 * time.Second ) ginkgo.Context("when create a pod with lifecycle hook", func() { <del> var targetIP, targetURL string </del> <ins> var targetIP, targetURL, targetNode string </ins> ports := []v1.ContainerPort{ { ContainerPort: 8080,

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --k8s-e2e-ltsc2019-containerd-flannel-sdnoverlay-master [] Container Lifecycle Hook when create a pod with lifecycle hook should execute prestart http hook properly [NodeConformance] [Conformance] [] Container Lifecycle Hook when create a pod with lifecycle hook should execute poststart http hook properly [NodeConformance] [Conformance] Note: the tests have been excluded until they are fixed. In the case of multinode clusters, the http server pod and the test cluster can spawn on different nodes, which can be problematic for poststart / prestop hooks, as they are executed by the kubelet itself, and the cross-node lifecycle hook might fail (according to the [Kubernetes network model], it is not mandatory for kubelet to be able to access pods on a different node). However, that is not always the case, different Networking setups allow cross-node communication between kubelet and pods, which means that the above tests are passing properly (for example: ), which begs the question if this behaviour should be universal or not, especially since the tests are marked as , which would imply that it should work in all cases. [Kubernetes network model]: links to appreciated links to specific failures in spyglass appreciated Slack: /sig testing /sig networking /sig windows <!-- Please see the deflaking doc () for more guidance! --
/sig network /sig node /area conformance /assign /priority critical-urgent /cc This is important since affects conformance and defines the networking behaviour for the poststart/ prestop hooks
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /sig network /sig node /sig conformance /assign /priority critical-urgent /cc Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
i'd personally let folks have first dibs on what we need to do here. Though from the name of the test itself, it seems like the test is NOT a sig-network test :)
Indeed, it does not have any tag in its name, but the tests rely on networking and on how CNIs work.

As I've stated above, it's not a problem for flannel sdnbridge, but it is for flannel sdnoverlay; other CNIs might have a similar issue too.
because the network is not a thing :) let´s tag the sig owner then /sig node I think that the key here is to decide if kubelet should be able to access pod in other nodes, and based on the link Claudiu is part of the Kubernetes networking model. I saw users that doesn't want kubelet to access even the pods on same host, so I think that is legit to say kubelet MAY reach pods in other nodes but not MUST ... but I'm not familiar with the to fully understand the context. If kubelet "may" reach pods in the other nodes, the test must only test lifecycle hooks for pods in the same node as the kubelet under test, right?
This is correct. The model also says: "For those platforms that support Pods running in the host network (e.g. Linux): pods in the host network of a node can communicate with all pods on all nodes without NAT" I believe that Windows host network is not supported, right?
Right, and there are specific tests that checks that exact scenario you've mentioned: However, the flaky tests I've mentioned in the issue require kubelet to be able to dial-in into a webserver pod for a different pod's poststart / prestop HTTP lifecycle hooks. There are couple of things regarding this: of all, in order to ensure kubelet would be able to run the HTTP lifecycle hooks, we'd have to make sure both pods are on the same node, as that is a supported scenario, according to the Kubernetes Network Model. I have sent a PR for this above. Alternatively, a Cluster IP could probably work as well, but IMO, ensuring that both pods end up on the same node is simpler and easier to clean up. , it's interesting that those HTTP lifecycle hook tests work on different networking setups, without any pods having HostNetwork enabled, even though you've mentioned they should (e.g.: Linux / Windows with flannel l2bridge, and even Linux with flannel overlay). Now, my question is: is this behaviour a feature, or a bug, especially since the tests are marked as `?

oh, I missed that part, so in Linux kubelet must be able to communicate with pods on all nodes (becuase it is running in the hostnetwork) If that is correct, it seems that one solution is to keep the test for Linux and tag it as LinuxOnly, and a new test for WindowsOnly that only spawn the pods under test in the same node EDIT Discussed in sig-net meeting , no clear answer... can be kubelet considered as a hostnetwork pod?
For the purposes of testing, I don't mind forcing the pods onto the same node., but let's document the heck out of that at the code-site?
correct but it will be supported once HostProcess containers are merged in 1.22: Only in Overlay networks (ie Flannel where these tests are flaking) does the pod have to be on the same node. It doesn't require host network mode and these tests are . I would be more inclined to disable this test for Windows Overlay test setups and have a sig-windows test for the overlay functionality. I would also question if the lifecycle hooks only running on the same node is useful and if it should just be documented that it doesn't work in the Windows Overlay setup.
is looking at test PRs from SIG Node. I will let sig-net triage this issue
Next steps on this?
/triage accepted
We still haven't merged this: I also had this, but it has been decided to be blocked:
requires conformance approval, at this point I think that has to be postponed to 1.23 right, since that behavior is not required, we shouldn't test it, because is legit to fail and having a test for something that can fail can create confusion :smile:


Question: for _, service := range services { activeServices.Insert(service.ID) info, exists := proxier.getServiceInfo(service.ID) <del> if exists && info.port == service.Port { </del> <ins> if exists && info.active && info.port == service.Port { </ins> continue } <del> if exists { </del> <ins> if exists && info.port != service.Port { </ins> proxier.StopProxy(service.ID) } glog.Infof("Adding a new service %s on port %d", service.ID, service.Port)

if you delete a service and restart it the port is no longer accessible To reproduce: 1) start a pod 2) start a service All works fine port is assigned - traffic flows 3) delete service 4) start service the port is now not assigned Here are sample configs: pod: Service:
I thought this was a known issue, but I could not find another issue on the same topic. Note that all containers started after the first instance of the service was created will have environment variables referring to that instance and won't see environment variables referring to the new instance until they restart. This is a fundamental shortcoming of the current approach.
I understand what you just replied but that is not the issue here. The problem is that when you delete a service , no other service can resuse that port successfully. The port is removed from a netstat listing. When a new service tries to use that port it gets allocated but you can not communicate to the pod through it. It is hosed until I re-install the cluster. (I have not tried rebooting the cluster) BTW this is a bug not a design issue.
Ah. The previous issue I was thinking of was . It sounds like the issue hasn't been completely resolved.
I will take a look at this today.
Sorry for the problem, has the fix. --brendan
Thanks! On Aug 19, 2014, at 2:26 PM, brendandburns wrote:


Question: w.Write([]byte(output)) } <ins> // Handler that returns the number of request headers in the body headerCounterHandler := func(w http.ResponseWriter, r *http.Request) { w.WriteHeader(200) w.Write([]byte(strconv.Itoa(len(r.Header)))) } // Handler that returns the keys of request headers in the body headerKeysNamesHandler := func(w http.ResponseWriter, r *http.Request) { w.WriteHeader(200) keys := make([]string, 0, len(r.Header)) for k := range r.Header { keys = append(keys, k) } sort.Strings(keys) w.Write([]byte(strings.Join(keys, "n"))) } </ins> redirectHandler := func(s int, bad bool) func(w http.ResponseWriter, r *http.Request) { return func(w http.ResponseWriter, r *http.Request) { if r.URL.Path == "/" {

<!-- Please only use this template for submitting enhancement requests --What would you like to be : I would like that besides User-agent we the Accept header as well with the "acept all" default. Just like curl does. Why is this needed: Because it was my expectation that the probe would work similarly to a curl but is not the case. I think more folks can hit this situation and waste some time to finally sort it out If this is not something you would like to add I would then like the docs to mention this specifically. _ PS: I would love to propose the improvement my self
/sig node network
/assign
Is the goal simply to avoid tripping over servers that look for that header? I don't think this presents any hazards - is that right? If so, it seems like a simple PR if you want to tackle it. /triage accepted /kind feature
cool! will look into it! thanks :)
Hello sorry but be bugging you, but I have opened the PR but I had 0 response since then. I think I have met all the requirements, so I was wondering (as this is my first PR in k8s) if you could help me a bit. Thanks anyway.


Question: } return shuffled } <ins> // Int64Slice attaches the methods of Interface to []int64, // sorting in increasing order. type Int64Slice []int64 func (p Int64Slice) Len() int           { return len(p) } func (p Int64Slice) Less(i, j int) bool { return p[i] < p[j] } func (p Int64Slice) Swap(i, j int)      { p[i], p[j] = p[j], p[i] } // Sorts []int64 in increasing order func SortInts64(a []int64) { sort.Sort(Int64Slice(a)) } </ins>

Ref and should sort revisions correctly (10 should be after 9, not 1).
I can help with this bug. Looks like trivial fix in PrintRolloutHistory
Thanks
are you still taking care of this one ? I am searching for some easy to do PR to just get familiar with the process and put my first PR in.
I created pr some time ago to close this issue
great thanks :-)
you can look for help-wanted issues or check and
thanks


Question: // and should assume that any attribute may be empty, invalid, or under // defined. type StatusDetails struct { <del> // The ID attribute of the resource associated with the status StatusReason // (when there is a single ID which can be described). ID string `json:"id,omitempty" description:"the ID attribute of the resource associated with the status StatusReason (when there is a single ID which can be described)"` </del> <ins> // The name attribute of the resource associated with the status StatusReason // (when there is a single name which can be described). Name string `json:"name,omitempty" description:"the name attribute of the resource associated with the status StatusReason (when there is a single name which can be described)"` </ins> // The kind attribute of the resource associated with the status StatusReason. // On some operations may differ from the requested resource Kind. Kind string `json:"kind,omitempty" description:"the kind attribute of the resource associated with the status StatusReason; on some operations may differ from the requested resource Kind"`

When accessing a non existing entity (example - namespace), the message specifies the requested field as 'id', but it should be 'name'. Example (see the details section) { "kind": "Status", "apiVersion": "v1beta3", "metadata": {}, "status": "Failure", "message": "namespaces "staging" not found", "reason": "NotFound", "details": { "id": "staging", "kind": "namespaces" }, "code": 404 }
Yes. We should rename "id" to "name" in StatusDetails: Not sure if we can change v1beta3 API now. cc
You can add a new field and fill in both, and MAYBE remove ID in v1 , Nikhil Jindal wrote:
I suggest we fix it at most in v1 only. This is an annoying wart, but we could just fix the description string for now.
If trivial, we should fix v1.


Question: } func (cc *cadvisorClient) exportHTTP(port uint) error { <ins> // Register the handlers regardless as this registers the prometheus // collector properly. </ins> mux := http.NewServeMux() err := cadvisorHttp.RegisterHandlers(mux, cc, "", "", "", "", "/metrics") if err != nil { return err } <del> serv := &http.Server{ Addr:    fmt.Sprintf(":%d", port), Handler: mux, } // TODO(vmarmol): Remove this when the cAdvisor port is once again free. // If export failed, retry in the background until we are able to bind. // This allows an existing cAdvisor to be killed before this one registers. go func() { defer util.HandleCrash() err := serv.ListenAndServe() for err != nil { glog.Infof("Failed to register cAdvisor on port %d, retrying. Error: %v", port, err) time.Sleep(time.Minute) err = serv.ListenAndServe() </del> <ins> // Only start the http server if port > 0 if port > 0 { serv := &http.Server{ Addr:    fmt.Sprintf(":%d", port), Handler: mux, </ins> } <del> }() </del> <ins> // TODO(vmarmol): Remove this when the cAdvisor port is once again free. // If export failed, retry in the background until we are able to bind. // This allows an existing cAdvisor to be killed before this one registers. go func() { defer util.HandleCrash() err := serv.ListenAndServe() for err != nil { glog.Infof("Failed to register cAdvisor on port %d, retrying. Error: %v", port, err) time.Sleep(time.Minute) err = serv.ListenAndServe() } }() } </ins> return nil }

If is passed to the kubelet, container metrics are not included in the Prometheus metrics endpoint.


Question: if ! gcloud compute instances add-metadata "${instance}"  --project "${PROJECT}"  --zone "${ZONE}"  <del> --metadata-from-file $(IFS=, ; echo "${kvs[*]}"); then </del> <ins> --metadata-from-file "$(join_csv ${kvs[@]})"; then </ins> if (( attempt > 5 )); then echo -e "${color_red}Failed to add instance metadata in ${instance} ${color_norm}" exit 2

This happens over and over. Scary messages. WARNING: Lists should be separated by commas, try "--allow=tcp,udp,icmp,esp,ah,sctp". If you intend to use udp, icmp, esp, ah, sctp for positional arguments, put them before the flag. WARNING: Lists should be separated by commas, try "--scopes=storage-ro,compute-rw". If you intend to use compute-rw for positional arguments, put them before the flag. WARNING: Lists should be separated by commas, try "--metadata-from-file=startup-script=/usr/local/google/home/thockin/src/kubernetes/hack/e2e-internal/../../cluster/../cluster/gce/../../cluster/../cluster/gce/configure-,kube-env=/tmp/kubernetes.MURmzX/master-kube-". If you intend to use kube-env=/tmp/kubernetes.MURmzX/master-kube- for positional arguments, put them before the flag. WARNING: Lists should be separated by commas, try "--disk=name=e2e-test-thockin-master-pd,device-name=master-pd,mode=rw,boot=no,auto-delete=no". If you intend to use device-name=master-pd, mode=rw, boot=no, auto-delete=no for positional arguments, put them before the flag.
That's a new one. gcloud started getting a little more anal here. Exciting. cc
They changed list argument syntax a while back, but the warnings are new. Time to update startup scripts I guess.


Question: package azure import ( <ins> "errors" </ins> "fmt" <ins> "regexp" </ins> "strings" "k8s.io/kubernetes/pkg/api/v1"

This is required to make Cloud Controller Manager operational on the various clouds that Kubernetes runs on. We have already PRs for the following clouds: GCE - AWS - Azure - OpenStack - The following clouds still need to be addresses: CloudStack Mesos OVirt Photon Rackspace vSphere If anyone is interested in contributing to this effort, please ping me directly (on kubernetes slack or through email)
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
/sig cluster-lifecycle
cc:
I will take care of the vSphere and Photon.
I will take care of the Rackspace.
Merging for OpenStack
Reposting I am trying to use my custom out-of-tree cloud provider with Kubernetes. But I noticed that kubelet calls apis like with empty for cloud provider. Digging in the code for 1.7 branch, I see that is null for cloud-providers. As a result returns empty string. As a result, ByID() methods don't work for cloud providers since is not set.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close