Question: } }) <del> // This test verifies if service load balancer cleanup finalizer is properly // handled during service lifecycle. // 1. Create service with type=LoadBalancer. Finalizer should be added. // 2. Update service to type=ClusterIP. Finalizer should be removed. // 3. Update service to type=LoadBalancer. Finalizer should be added. // 4. Delete service with type=LoadBalancer. Finalizer should be removed. ginkgo.It("should handle load balancer cleanup finalizer for service [Slow]", func() { jig := e2eservice.NewTestJig(cs, f.Namespace.Name, "lb-finalizer") ginkgo.By("Create load balancer service") svc, err := jig.CreateTCPService(func(svc *v1.Service) { svc.Spec.Type = v1.ServiceTypeLoadBalancer }) framework.ExpectNoError(err) defer func() { ginkgo.By("Check that service can be deleted with finalizer") e2eservice.WaitForServiceDeletedWithFinalizer(cs, svc.Namespace, svc.Name) }() ginkgo.By("Wait for load balancer to serve traffic") svc, err = jig.WaitForLoadBalancer(e2eservice.GetServiceLoadBalancerCreationTimeout(cs)) framework.ExpectNoError(err) ginkgo.By("Check if finalizer presents on service with type=LoadBalancer") e2eservice.WaitForServiceUpdatedWithFinalizer(cs, svc.Namespace, svc.Name, true) ginkgo.By("Check if finalizer is removed on service after changed to type=ClusterIP") err = jig.ChangeServiceType(v1.ServiceTypeClusterIP, e2eservice.GetServiceLoadBalancerCreationTimeout(cs)) framework.ExpectNoError(err) e2eservice.WaitForServiceUpdatedWithFinalizer(cs, svc.Namespace, svc.Name, false) ginkgo.By("Check if finalizer is added back to service after changed to type=LoadBalancer") err = jig.ChangeServiceType(v1.ServiceTypeLoadBalancer, e2eservice.GetServiceLoadBalancerCreationTimeout(cs)) framework.ExpectNoError(err) e2eservice.WaitForServiceUpdatedWithFinalizer(cs, svc.Namespace, svc.Name, true) }) </del> /* Release: v1.18 Testname: Find Kubernetes Service in default Namespace

Which jobs are flaking: gce-master-scale-correctness Which test(s) are flaking: Services should be able to change the type and ports of a service [Slow] Testgrid link: https://k8s- Reason for failure: Test timeouts in ~half of the runs since it was enabled here: /sig scalability /sig network
/cc /cc
The context is - the problems are strictly related to scale - 5k-node jobs are timing out. I very briefly skimmed through the test and didn't see anything obviously suspicious (apart from the fact there are gazillions long timeouts there, which alltogether sum up long hours). That said - it seems to me that there are effectively two separate tests inside it: one for TCP one for UDP They don't seem to share anything with each other. - would it make sense to split this test two effectively two separate tests: one for TCP and one for UDP. (I guess it would also allow us to share some code there).
[And potentially it could help with debugging the problem if it is strictly related to only one of those cases...]
I wanted to focus on this past week but I didn´t have time, this is my list of more expensive times and approx average duration, all of the sig-network ones use LoadBalancers. I wanted to time the test to understand if there are places where can save time ... I don´t have much context on those tests, maybe Tim can help us more, but this comment makes me think that the use of both protocols is intended
/triage accepted
/triage accepted It's been a very long time since I touched this, and I recall holding my nose on it in the first place. Splitting TCP from UDP seems reasonable if we can run them in parallel. we'll want to share the flow, and parameterize some shared funcs. I don't know that this will prevent test timeouts, though.
We could use a volunteer to go poke at this, and see how hard it will really be.
You mean "forcing them to run in parallel"? Or "allowing them to run in parallel"? The later doesn't require any additiona work. The former I think mean that it will still have to be a single test (just the code will be organized better) and that won't really help with debugging (nor preventing timeouts).

this was discussed yesterday in sig-net meeting and my understanding was to split current test in 2 different so the test can run in parallel, if you want to do so and one protocol doesn´t have to wait for the other. I think that is what you meant here
As long as we CAN parallelize. Creating LBs can be sloooooow. :P Antonio Ojea <:
Reopening - I don't think just splitting will address the problem. It may help us with debugging though...
With the split we moved from 60 mins to run the test, to run 2 tests in [30-40] mins each https://k8s- Seems a win :smile:
Awesome - that's something I was silently counting for TBH :) [I doubt it will fix flakiness, but maybe it will make it easier to debug...]
Actually - I think we're getting close thanks to the two changes ( + this one). Here is the run: This particular one filed on UDP (not sure if it's just UDP-related, or just manifested with UDP now). And we have a clear failure now: In my PR I decreased the timeout from 2h to 45m, as it seemed to me that in the past it should be enough. But it failed on 45m timeout here. So I think the next question now is: is it that slow - that we need more than 45m or something isn't working as expected (either in the k8s itself or in the test) But I think we have a clear single place to look at now.
Actually - the full stacktrace: So it's this line that failed: What is interesting is that something was changing during those 45 minutes:
Note that it's 4th time we actually wait for LB in this test. The failure happens after the pod were scaled down to 0 and then to 1. So that makes me think, that's it's actually not LB-related, but rather something internally in k8s, becuase L4 LB doesn't depend on pods - it targets nodes...
If it is always UDP, I start to see ghosts between GCE and UDP ... we only have KIND to compare against, or do we have some other non gce enviroment?

I mean, there are other tests failing in UDP , and show much more failures in GCE than in kind
- this does not look "normal". Can we give some prio to helping track this flake?
hmm interesting. we will investigate.
/assign
- any progress on this?
I haven't had a chance to look into this yet. I will prioritize it this week.
Digging through the logs I believe this was a bug in the test code that was fixed by . In the test, the service port changes around 12:33. However due to the bug, the test continues to use the old service port as seen in the test output: The port should be 81. I believe the following sequence occurs from this point: 1) The reachability test fails in the beginning because the service has changed the port and the load balancer has not been updated yet resulting in the error. 1) Around 12:38, the service controller is syncing the service and notices that the service port has changed and starts to delete the forwarding rule causing the error to change to . 1) Once the forwarding rule is recreated, the error goes back to being because the forwarding rule exists but the wrong port is used. Looking at , the test runs after the one linked have all passed. Is there anything else that needs to be investigated as part of this issue?
OK - the test was quite flaky in the past but indeed it didn't flake for the last 3 weeks (neither this nor TCP one). I'm fine with closing this one - we can open a new one if this happens again. - thanks for debugging. - it seems that the splitting of the test actually made things waay better. Which is great.


Question: // KindFor fulfills meta.RESTMapper func (e shortcutExpander) KindFor(resource schema.GroupVersionResource) (schema.GroupVersionKind, error) { <del> return e.RESTMapper.KindFor(e.expandResourceShortcut(resource)) </del> <ins> // expandResourceShortcut works with current API resources as read from discovery cache. // In case of new CRDs this means we potentially don't have current state of discovery. // In the current wiring in k8s.io/cli-runtime/pkg/genericclioptions/config_flags.go#toRESTMapper, // we are using DeferredDiscoveryRESTMapper which on KindFor failure will clear the // cache and fetch all data from a cluster (see vendor/k8s.io/client-go/restmapper/discovery.go#KindFor). // Thus another call to expandResourceShortcut, after a NoMatchError should successfully // read Kind to the user or an error. gvk, err := e.RESTMapper.KindFor(e.expandResourceShortcut(resource)) if meta.IsNoMatchError(err) { return e.RESTMapper.KindFor(e.expandResourceShortcut(resource)) } return gvk, err </ins> } // KindsFor fulfills meta.RESTMapper

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: When I created a crd e.g.: Then I used . I found that the first time the result will be with an error: And as for the second time it will be normal: And it can be stably reproduced every time I created a new CRD. What you expected to happen: kubectl returns the first time I requested the new CRD instead of an error. Anything else we need to know?: I also found that requesting short name e.g. will be the same. It returns an error the first time but becomes normal the second time. Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: /kind bug
/area custom-resources
Confirmed that could show up even the crd has status /assign
/sig api-machinery Haven't looked into the problem yet but could be related:
Looks like this happens because cache file in .kube/http-cache/ or .kube/cache/ is not synced with kube-apiserver. delete all files under .kube/http-cache/ and .kube/cache/ will cause a sync and the error will not show up. This looks like a cli issue.
fyi
Tried to fix in cli:
/cc
/assign
this seems similar to
Even though this bug looks small, it might involve larger changes both in fixing and refactoring.. A typical corner case it is /shrug
Can you put this into words? What's the actual issue?
Even though this bug looks small, it might involve larger changes both in fixing and refactoring.. A typical corner case it is Can you put this into words? What's the actual issue? SRY, I should have put it clearer.. I just mean that this issue is sort of fundamental and obviously has been existing for a long time.
/remove-shrug
/unshrug looks like an important problem we want to solve (?)
¯()/¯ <detailsIn response to : /unshrug Instructions for interacting with me using PR comments are available .

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen /remove-lifecycle rotten ouch..
Reopened this issue. <detailsIn response to : /reopen /remove-lifecycle rotten Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc
encountered same issue, as suggested the error will not show up.
any progress?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with .

Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Here is a little more info, and here is how I am able to reproduce it (kubectl version 1.18.0 and cluster version 1.19.0-beta.2): A couple interesting things i noted... If I deleted my and re-ran it, I could not reproduce it. It is only the first time the short name was EVER used that this occurs. I only tried from one client machine, so I'm not sure this is cluster-specific or client-specific. If I used a different short name, like 'fo2' and re-ran the example, then it reproduces the problem. You can also run after creating the CRD and it has the same effect as deleting your . EDIT TO ADD: This doesn't seem to be a timing thing... if I sleep for 10 seconds after creating the CRD, it still has the problem.
The root of the problem is that the is run before the discovery. So in this particular case this line continues to b/c read cached data and never reached out to server. Whereas discovery mapper in will get a not found error and will proceed invalidating cache and re-fetching discovery data and return that back. In the subsequent call the shortcut expander is already aware of the new shortcut so it works just fine.

We'd need to teach shortcut expander to re-fetch discovery data, but I'm not sure that's desirable, or we could at least warn a user about this situation.
See for possible approach.
Any new progress on this ?
lists a candidate solution. This bug is causing steadily failing tests. I'll talk to a few people to see if this can get any traction. do you know who can get this moving? It sounds like you have identified the problem and have proposed a solution.


Question: ServeLogs(w http.ResponseWriter, req *http.Request) } <ins> // NewServer initializes and configures a kubelet.Server object to handle HTTP requests func NewServer(host HostInterface, updates chan<- interface{}) Server { server := Server{ host:    host, updates: updates, mux:     http.NewServeMux(), } server.InstallDefaultHandlers() return server } // InstallDefaultHandlers registers the set of supported HTTP request patterns with the mux func (s *Server) InstallDefaultHandlers() { s.mux.HandleFunc("/healthz", s.handleHealth) s.mux.HandleFunc("/container", s.handleContainer) s.mux.HandleFunc("/containers", s.handleContainers) s.mux.HandleFunc("/podInfo", s.handlePodInfo) s.mux.HandleFunc("/stats/", s.handleStats) s.mux.HandleFunc("/logs/", s.handleLogs) s.mux.HandleFunc("/spec/", s.handleSpec) } // error serializes an error object into an HTTP response </ins> func (s *Server) error(w http.ResponseWriter, err error) { http.Error(w, fmt.Sprintf("Internal Error: %v", err), http.StatusInternalServerError) } <del> func (s *Server) ServeHTTP(w http.ResponseWriter, req *http.Request) { defer httplog.MakeLogged(req, &w).StacktraceWhen( httplog.StatusIsNot( http.StatusOK, http.StatusNotFound, ), ).Log() </del> <ins> // handleHealth handles health checking requests against the Kubelet func (s *Server) handleHealth(w http.ResponseWriter, req *http.Request) { } // handleContainer handles container requests against the Kubelet func (s *Server) handleContainer(w http.ResponseWriter, req *http.Request) { defer req.Body.Close() data, err := ioutil.ReadAll(req.Body) if err != nil { s.error(w, err) return } // This is to provide backward compatibility. It only supports a single manifest var pod Pod err = yaml.Unmarshal(data, &pod.Manifest) if err != nil { s.error(w, err) return } //TODO: sha1 of manifest? pod.Name = "1" s.updates <- PodUpdate{[]Pod{pod}, SET} } // handleContainers handles containers requests against the Kubelet func (s *Server) handleContainers(w http.ResponseWriter, req *http.Request) { defer req.Body.Close() data, err := ioutil.ReadAll(req.Body) if err != nil { s.error(w, err) return } var manifests []api.ContainerManifest err = yaml.Unmarshal(data, &manifests) if err != nil { s.error(w, err) return } pods := make([]Pod, len(manifests)) for i := range manifests { pods[i].Name = fmt.Sprintf("%d", i+1) pods[i].Manifest = manifests[i] } s.updates <- PodUpdate{pods, SET} } // handlePodInfo handles podInfo requests against the Kubelet func (s *Server) handlePodInfo(w http.ResponseWriter, req *http.Request) { </ins> u, err := url.ParseRequestURI(req.RequestURI) if err != nil { s.error(w, err) return } <del> // TODO: use an http.ServeMux instead of a switch. switch { case u.Path == "/container" || u.Path == "/containers": defer req.Body.Close() data, err := ioutil.ReadAll(req.Body) if err != nil { s.error(w, err) return } if u.Path == "/container" { // This is to provide backward compatibility. It only supports a single manifest var pod Pod err = yaml.Unmarshal(data, &pod.Manifest) if err != nil { s.error(w, err) return } //TODO: sha1 of manifest? pod.Name = "1" s.updates <- PodUpdate{[]Pod{pod}, SET} } else if u.Path == "/containers" { var manifests []api.ContainerManifest err = yaml.Unmarshal(data, &manifests) if err != nil { s.error(w, err) return } pods := make([]Pod, len(manifests)) for i := range manifests { pods[i].Name = fmt.Sprintf("%d", i+1) pods[i].Manifest = manifests[i] } s.updates <- PodUpdate{pods, SET} } case u.Path == "/podInfo": podID := u.Query().Get("podID") if len(podID) == 0 { w.WriteHeader(http.StatusBadRequest) http.Error(w, "Missing 'podID=' query entry.", http.StatusBadRequest) return } // TODO: backwards compatibility with existing API, needs API change podFullName := GetPodFullName(&Pod{Name: podID, Namespace: "etcd"}) info, err := s.host.GetPodInfo(podFullName) if err == ErrNoContainersInPod { http.Error(w, "Pod does not exist", http.StatusNotFound) return } if err != nil { s.error(w, err) return } data, err := json.Marshal(info) if err != nil { s.error(w, err) return } w.WriteHeader(http.StatusOK) w.Header().Add("Content-type", "application/json") w.Write(data) case strings.HasPrefix(u.Path, "/stats"): s.serveStats(w, req) case strings.HasPrefix(u.Path, "/spec"): info, err := s.host.GetMachineInfo() if err != nil { s.error(w, err) return } data, err := json.Marshal(info) if err != nil { s.error(w, err) return } w.Header().Add("Content-type", "application/json") w.Write(data) case strings.HasPrefix(u.Path, "/logs/"): s.host.ServeLogs(w, req) default: if s.handler != nil { s.handler.ServeHTTP(w, req) } </del> <ins> podID := u.Query().Get("podID") if len(podID) == 0 { w.WriteHeader(http.StatusBadRequest) http.Error(w, "Missing 'podID=' query entry.", http.StatusBadRequest) return } // TODO: backwards compatibility with existing API, needs API change podFullName := GetPodFullName(&Pod{Name: podID, Namespace: "etcd"}) info, err := s.host.GetPodInfo(podFullName) if err == ErrNoContainersInPod { http.Error(w, "Pod does not exist", http.StatusNotFound) return </ins> } <ins> if err != nil { s.error(w, err) return } data, err := json.Marshal(info) if err != nil { s.error(w, err) return } w.WriteHeader(http.StatusOK) w.Header().Add("Content-type", "application/json") w.Write(data) } // handleStats handles stats requests against the Kubelet func (s *Server) handleStats(w http.ResponseWriter, req *http.Request) { s.serveStats(w, req) } // handleLogs handles logs requests against the Kubelet func (s *Server) handleLogs(w http.ResponseWriter, req *http.Request) { s.host.ServeLogs(w, req) } // handleSpec handles spec requests against the Kubelet func (s *Server) handleSpec(w http.ResponseWriter, req *http.Request) { info, err := s.host.GetMachineInfo() if err != nil { s.error(w, err) return } data, err := json.Marshal(info) if err != nil { s.error(w, err) return } w.Header().Add("Content-type", "application/json") w.Write(data) } // ServeHTTP responds to HTTP requests on the Kubelet func (s *Server) ServeHTTP(w http.ResponseWriter, req *http.Request) { defer httplog.MakeLogged(req, &w).StacktraceWhen( httplog.StatusIsNot( http.StatusOK, http.StatusNotFound, ), ).Log() s.mux.ServeHTTP(w, req) </ins> } <ins> // serveStats implements stats logic </ins> func (s *Server) serveStats(w http.ResponseWriter, req *http.Request) { // /stats/<podfullname>/<containerName> components := strings.Split(strings.TrimPrefix(path.Clean(req.URL.Path), "/"), "/")

The Kubelet REST end-point has turned into a complicated switch-case code block that is harder to maintain overtime. The Kubelet should move to a http.ServeMux pattern and register handler functions to a pattern.
I am working on a PR to submit in next couple of days that will make this change, so feel free to assign to me if possible.
Tagging because I asked for this in one of his PRs
This one is also under my radar. If no one had time, feel free to assign it to me too.
I have a branch in progress. Feel free to offer feedback on it when I submit. Sent from my iPhone


Question: // Open security group ingress rules on the instances so that the load balancer can talk to them // Will also remove any security groups ingress rules for the load balancer that are _not_ needed for allInstances <del> func (c *Cloud) updateInstanceSecurityGroupsForLoadBalancer(lb *elb.LoadBalancerDescription, instances map[InstanceID]*ec2.Instance) error { </del> <ins> func (c *Cloud) updateInstanceSecurityGroupsForLoadBalancer(lb *elb.LoadBalancerDescription, instances map[InstanceID]*ec2.Instance, annotations map[string]string) error { </ins> if c.cfg.Global.DisableSecurityGroupIngress { return nil } // Determine the load balancer security group id <del> loadBalancerSecurityGroupID := "" for _, securityGroup := range lb.SecurityGroups { if aws.StringValue(securityGroup) == "" { continue } if loadBalancerSecurityGroupID != "" { // We create LBs with one SG klog.Warningf("Multiple security groups for load balancer: %q", aws.StringValue(lb.LoadBalancerName)) } loadBalancerSecurityGroupID = *securityGroup } if loadBalancerSecurityGroupID == "" { </del> <ins> lbSecurityGroupIDs := aws.StringValueSlice(lb.SecurityGroups) if len(lbSecurityGroupIDs) == 0 { </ins> return fmt.Errorf("could not determine security group for load balancer: %s", aws.StringValue(lb.LoadBalancerName)) } <ins> c.sortELBSecurityGroupList(lbSecurityGroupIDs, annotations) loadBalancerSecurityGroupID := lbSecurityGroupIDs[0] </ins> // Get the actual list of groups that allow ingress from the load-balancer var actualGroups []*ec2.SecurityGroup

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When multiple securityGroup are specified for AWS ELB type service, a random one will be chosen to grant inbound access to worker node. What you expected to happen: Always first LoadBalancer securityGroup be chosen to access worker nodes. How to reproduce it (as minimally and precisely as possible): use or on aws elb. Anything else we need to know?: Environment: Kubernetes version (use ): <=1.18 Cloud provider or hardware configuration: aws OS (e.g: ): linux Kernel (e.g. ): linux Install tools: kops/eks Network plugin and version (if this is a network-related bug): Others:
/sig cloud-provider These SIGs are my best guesses for this issue. Please comment if I am incorrect about one. I am a bot run by vllry.

What happened: We were running a service with LoadBalancer type with AWS cloud provider and after deleting this service, the cleanup process on AWS removed a wrong ingress input rule which provoked that all traffic between nodes was unable to reach out the destination. The service we were running had a configuration similar to: After applying this manfiest on the cluster, the cloud provider created the associated load balancer with two security groups: sg- (due to extra security group annotation) sg- (a new SG created by the AWS cloud provider with the rules defined on the ) Cloud provider also created a new input rule on the nodes security group which allows all traffic from all ports from (which is the sg created for the newly created load balancer in our example). One particularity of our setup is that the security group we as was the nodes security group. The input rules of this security group after the creation of this service was similar to: Everything worked fine with this setup until we deleted this service and the cloud provider started the deletion process of the LoadBalancer on AWS. We've been debugging the deletion code and we found that the first thing it does is cleaning up the security group. To do it, it first removes the input rule from the nodes security group which allows traffic from the load balancer security group (sg-, on this example) and then, as it's not referenced, it removes the security group. The expected output of this process, would be that nodes security groups would be: But, in our case, it ended up as: which dropped all the traffic between nodes (as it didn't allow any traffic between them). The cleanup process didn't properly remove the input rule from the node security group (the one which allows all traffic from source sg-) and it also did not remove the sg- security group. In our case, as we were using the nodes security group on the extra security groups, this bug provoked a network partition between nodes, but we think that if is used, the cleanup process may be broken and have unexpected side-effects like those we observed. We've been debugging and we found that the process which removes the input rule from the nodes security groups starts here: The first thing this function does is this: : When extra security groups are used, will contain the security group created for the LoadBalancer and all the extra security groups.

We also found that AWS returns this SecurityGroups list in distinct orders after changing the LoadBalancer configuration. That code is always assuming that latest security group on the list is the , which may not be true when extra security groups are used. On the cases where the load balancer security group is not returned on the latest position, the cleanup process does not remove the security group and the input rule from the nodes security group is not removed as expected. We've seen that master branch still contains this code and we think this may be hapenning on newer versions: ), so we think that newer versions may be affected by this bug (we didn't try it). In our case, we saw several times this log during the removal process: Additionally, the code tried to remove the security group (but being unable to do so), after wrongly revoking the rule we have talked about before. What you expected to happen: The cleanup process should properly detect the load balancer security group, remove the input rule from nodes security group and remove the load balancer security group, regardless of the order in which AWS returns the security group list on . How to reproduce it (as minimally and precisely as possible): We did not find any consistent process to create the scenario, as it happens randomly (we are not sure if due to AWS or any kind of race condition on AWS cloud provider). Setup: Create a Service with running controller manager with AWS cloud provider with tha annotation containing one or more security groups (as the one the section) Wait until load balancer is created Run Check if does not have the newly created security group for the balancer in the latest position. Repeat until the balancer does not have the balancer security group on the latest position Trigger the bug: Result:The log will appear multiple times on controller-manager logs Load balancer associated security group is not removed Nodes security group input rule allowing all traffic from load balancer security group is not removed If the latest security group on the list was the nodes security group, the input rule allowing all traffic on all ports from all the other nodes is removed and no traffic is allowed between nodes on the cluster.

Environment: Kubernetes version (use ): version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.10", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-05-01T04:05:01Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: aws OS (e.g: ):Kernel (e.g. ): Linux ip-10-2-13-67 4.9.0-7-amd64 SMP Debian 4.9.110-3+deb9u2 (2018-08-13) x8664 GNU/Linux Install tools: kops
/sig cloud-provider
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/area provider/aws /lifecycle frozen
Working on a repro for this issue (on 1.11 first, then I'll try on master) and I've not quite observed the same behavior yet. Depending on the order of the security groups on the load balancer, I only see the ingress rule for the new LB security group to the node security group if the new security group comes last. Otherwise, if the NODE security group comes last, the ingress rule is not , and I see in the logs. Even without a full repro yet, I find it troubling that behavior is dependent on the order of security groups attached to the load balancer. One question I have related to your setup, can you describe the reason why you're adding the node security group as an ? I'd like to understand your use case a little better.
Hi Nicholas. I have a working patch for this specific issue: This bug was the subject of a talk for a local meetup I did recently.

I also published a personal blog post talking about it: There you can see more details about how it can be reproduced easily. If you think the patch is valid, I'll try to add a few test cases for it and then propose the change upstream. Thanks!
It has been a while since we got this error and I'm not sure if we saw this error. What I experienced is that the order is changed by Kubernetes or by some call to AWS. It took me not more than 3 attempts to get this behavior. There's something I can do to help you reproduce the error? Do you need more details of our setup? This was a misconfiguration on our side. When we noticed that this setup was not correct we removed this security group from , which removed the mentioned security group rule and took down the Kubernetes cluster (no communication between nodes was allowed).


Question: return info } <ins> func (network hnsNetworkInfo) findRemoteSubnetProviderAddress(ip string) string { var providerAddress string for _, rs := range network.remoteSubnets { _, ipNet, err := net.ParseCIDR(rs.destinationPrefix) if err != nil { klog.Fatalf("%v", err) } if ipNet.Contains(net.ParseIP(ip)) { providerAddress = rs.providerAddress } if ip == rs.providerAddress { providerAddress = rs.providerAddress } } return providerAddress } </ins> type endpointsChange struct { previous proxyEndpointsMap current  proxyEndpointsMap

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: In Overlay mode, When 2 service has the same backend node IP, the LB was not programmed correctly causing datapath issues. What you expected to happen: Service data path to work properly How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig network /sig windows These SIGs are my best guesses for this issue. Please comment if I am incorrect about one. I am a bot run by vllry.
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.


Question: nc.deliverNamespace(namespace, nc.namespaceReviewDelay, false) } <del> func (nc *NamespaceController) delete(namespace *api_v1.Namespace) { </del> <ins> // delete  deletes the given namespace or returns error if the deletion was not complete. func (nc *NamespaceController) delete(namespace *api_v1.Namespace) error { </ins> // Set Terminating status. updatedNamespace := &api_v1.Namespace{ ObjectMeta: namespace.ObjectMeta,

In PR the specific resource types are handled explicitly. Need to delete them in more generic way, in a similar way that namespace controller does.
- switching this to 1.6 as it's too late for 1.5. ok? (please switch it right back if you disagree)
did fix this?
Please close or move to next-candidate with work remaining to be done.
Yes fixed by


Question: return fields[len(fields)-1] } <del> func determineRequestedIP(svc *v1.Service, fwdRule *compute.ForwardingRule) string { </del> <ins> // ilbIPToUse determines which IP address needs to be used in the ForwardingRule. If an IP has been // specified by the user, that is used. If there is an existing ForwardingRule, the ip address from // that is reused. In case a subnetwork change is requested, the existing ForwardingRule IP is ignored. func ilbIPToUse(svc *v1.Service, fwdRule *compute.ForwardingRule, requestedSubnet string) string { </ins> if svc.Spec.LoadBalancerIP != "" { return svc.Spec.LoadBalancerIP } <del> if fwdRule != nil { return fwdRule.IPAddress </del> <ins> if fwdRule == nil { return "" </ins> } <del> return "" </del> <ins> if requestedSubnet != fwdRule.Subnetwork { // reset ip address since subnet is being changed. return "" } return fwdRule.IPAddress </ins> } func getILBOptions(svc *v1.Service) ILBOptions { <del> return ILBOptions{AllowGlobalAccess: GetLoadBalancerAnnotationAllowGlobalAccess(svc)} </del> <ins> return ILBOptions{AllowGlobalAccess: GetLoadBalancerAnnotationAllowGlobalAccess(svc), SubnetName: GetLoadBalancerAnnotationSubnet(svc), } </ins> } // forwardingRuleComposite is a composite type encapsulating both the GA and Beta ForwardingRules.

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: ILB ip addresses come from the subnet that the K8s cluster uses. There is no ability to specify a different subnet. What you expected to happen: ILB ip addresses should be allocated from a different subnet if specified. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: GCE OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/area platform/gce /assign
/sig network


Question: newPath := field.NewPath("KubeProxyConfiguration") successCases := []kubeproxyconfig.ProxyMode{ kubeproxyconfig.ProxyModeUserspace, <del> kubeproxyconfig.ProxyModeIPTables, kubeproxyconfig.ProxyModeIPVS, </del> kubeproxyconfig.ProxyMode(""), } <ins> if runtime.GOOS == "windows" { successCases = append(successCases, kubeproxyconfig.ProxyModeKernelspace) } else { successCases = append(successCases, kubeproxyconfig.ProxyModeIPTables, kubeproxyconfig.ProxyModeIPVS) } </ins> for _, successCase := range successCases { if errs := validateProxyMode(successCase, newPath.Child("ProxyMode")); len(errs) != 0 { t.Errorf("expected success: %v", errs)

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: When running , and passing the flag which was introduced in , the following error occurs: What you expected to happen: The proxy should work as it used to, supporting this mode. How to reproduce it (as minimally and precisely as possible): ./kube- --proxy-mode=kernelspace Anything else we need to know?: This was introduced after the validation was refactored in which regressed validation for . IPVS validation (and thus support) was later , but we need another mode for Windows. Environment: Kubernetes version (use ): v1.10.0-alpha.0.684+ Cloud provider or hardware configuration: n/a OS (e.g. from ): Windows Server Core 1709 Kernel (e.g. ): Install tools: Others:
/sig windows /sig network
Tentatively adding milestone so we can track getting consensus on release-blocking status.
/priority critical-urgent
[MILESTONENOTIFIER] Milestone Issue Needs Approval Action required: This issue must have the label applied by a SIG maintainer. <details<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details


Question: ctx, cancel := getContextWithCancel() defer cancel() klog.V(2).Infof("azureDisk - update(%s): vm(%s) - detach disk(%s)", nodeResourceGroup, nodeName, diskName) <del> if _, err := ss.VirtualMachineScaleSetVMsClient.Update(ctx, nodeResourceGroup, ssName, instanceID, vm); err != nil { </del> <ins> _, err = ss.VirtualMachineScaleSetVMsClient.Update(ctx, nodeResourceGroup, ssName, instanceID, vm) if err != nil { </ins> klog.Errorf("azureDisk - detach disk(%s) from %s failed, err: %v", diskName, nodeName, err) } else { klog.V(2).Infof("azureDisk - detach disk(%s) succeeded", diskName)

What happened: After running a rolling-update on a StatefulSet that uses Azure Disks, part of the replicas remain in a state forever, exhibiting the following events: Restarting the occasionally unblocks the situation: What you expected to happen: Attach/detach operations succeed consistently. How to reproduce it (as minimally and precisely as possible): Here is a copy of the manifest of the affected StatefulSet: <details<summarysolr-</summary</details</brAnd the default StorageClass: <details</summary</details</brExtra information: Two other kind of Pod events that are regularly associated with the error above: kubelet logs of a node where attaching the volume is failing kube-controller-manager logs in an environment where attaching the volume is failing Environment: Kubernetes version (use ): v1.10.10 Cloud provider or hardware configuration: Azure (westeurope) OS (e.g. from /etc/os-release): 16.04.4 LTS Kernel (e.g. ): 4.13.0-1018-azure Install tools: ACS-Engine Others: /kind bug /sig storage /sig azure this might be of interest for you, since you were involved in , which was supposed to fix attach/detach issues in Azure.
/assign
fix the attach disk issue in kubelet, while in your case, you restarted the controller manager, and then it works. It looks like disk volume is never attached to vm successfully before you restart controller manager.
Indeed. For completeness I updated the description and 2 more type of events that are regularly associated to this issue.
could you check activity logs in that resource group, the time should be before you restart controller manager, you could check whether there is any disk related issue. Since you have already restarted controller manager, we cannot get all logs before your restart.
I see a bunch of the following failures in the activity log, but the errors stopped 4h before I gathered the logs and restarted the controller-manager: <details<summaryactivity log</summary</details</brI managed to grab the kube-controller-manager logs from an environment where the situation is currently happening. See the updated description.
Hi would you add my slack: andyzhang workspace?

I think your issue is here, you are using strategy According to , if statefulset replica is using azure disk pvc, is not the right way, pls change the strategy to 'Recreate' Update: may be not related to that issue, still in investigation.
In our deployment every Pod uses a distinct Azure Disk, so there is not multi-attachment. I'm still waiting for feedback from customers, hoping this happens again today.
I have tried in my testing env with v1.10.10 on vmss nodes by , there could be error like following after disk attached to that VM, the disk detect action in vmss node seems slow sometimes, while in the end it could be recovered:
thanks for looking into it. In our case the disk never gets attached (1d in ContainerCreating state).
Hi I found an regression issue of , will send a PR soon. Thanks for reporting.


Question: // Add implements port mappings. // id should be a unique identifier for a pod, e.g. podSandboxID. // podPortMapping is the associated port mapping information for the pod. <del> // natInterfaceName is the interface that localhost used to talk to the given pod. </del> <ins> // natInterfaceName is the interface that localhost uses to talk to the given pod, if known. </ins> Add(id string, podPortMapping *PodPortMapping, natInterfaceName string) error // Remove cleans up matching port mappings // Remove must be able to clean up port mappings without pod IP

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: In Ubuntu 18.04 DNS queries are being served via daemon which is listening on and serves queries only from . The only line in is , and this file is managed by systemd. Once any pod with notation appears on the host, fires the iptables rule in chain of table: -A POSTROUTING -s 127.0.0.0/8 -o lo -m comment --comment "SNAT for localhost access to hostports" -j MASQUERADE See commit and lines for details. Therefore all queries to goes not from , but from interface with default route due to masquerading, and rejects all of these requests with systemd-resolved[]: Got packet on unexpected IP range, refusing. How to reproduce it (as minimally and precisely as possible): Just install Kubernetes with CNI on Ubuntu-18.04 (with kubeadm, kubespray, chef-kubernetes, whatever) and run any pod with . Voila! You just lost DNS resolving. Environment: Kubernetes version (use ): Cloud provider or hardware configuration: Digital Ocean (but reproduceable on any configuration) OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: (but reproduceable with any install tool) Others: CNI: 0.6.0 CNI plugins: 0.7.1 CNI daemon: weave /sig network
/cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Hi, is anyone managed to figure a workaround for this?
Don't use Ubuntu 18.04?
Please if you have nothing useful to add, just move along. The workaround that I have settled on, is to run the following command on your Ubuntu 18.04 system during your bootstrapping: This ^^ solves the issue for me.
Can you explain what that means? Thank you!

Adding an SNAT rule above the rule generated by kubelet fixes it (replace the with whatever the slot is above the offending rule for you): I'm going to look at upstreaming this fix.
Thanks for your solution! Is it possible to fix on systemd side?
reopen if needed
but it is still broken, why did you close issue?
I have the same issue on Kubernetes version 1.14.2 on Ubuntu 18.04
Fixed in
Bug still present in kubernetes 1.16.0
I don't know if people still encounter this issue. For kops users, the solution I've found is to add this to cluster spec: This will actually tell kubelets to use the systemd insted of the default one in . For the users not using kops, you can just set in the kublet options.
I'm still having it on Ubuntu 19.10 running latest 1.6.2 minikube (aka k8s v1.17.0) but with "cri-o" as container engine which enables CNI by default. Since cri-o is very slow with releasing .deb packages - it is 1.15 which may be still impacted by this issue. Adding to solves issue - but it does not solve issue described in here. Temporarily I solve it by running: after first DaemonSet starts first Pod.
Update: After updating cri-o from 1.15 to 1.16 the problem vanished.


Question: ### Avoiding Collisions in IP allocation to Services Kubernetes 1.24 introduced a new opt-in feature that allows you to <del> [soft-reserve a range for static IP address assignments](/docs/concepts/services-networking/service/#service-ip-static-sub-range) </del> <ins> [soft-reserve a range for static IP address assignments](https://kubernetes.io/docs/concepts/services-networking/service/#service-ip-static-sub-range) </ins> to Services. With the manual enablement of this feature, the cluster will prefer automatic assignment from the pool of Service IP addresses thereby reducing the risk of collision.

has some broken links. These are relative links that work OK in the release blog article but need to be absolute hyperlinks in the release notes. Links in release notes work. Visit Click on all the links. Specifically, “this guide” in (there are some more, this is one example) No response v1.24.0 n/a n/a n/a n/a n/a
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig release and IMO /priority important-soon
Hi Tim, I can help fix this /assign


Question: Args: []string{ "/bin/sh", "-c", <del> fmt.Sprintf("for i in $(seq 1 5); do curl -s -m 8 %s.%s:%d && exit 0 || sleep 1; done; exit 1", </del> <ins> fmt.Sprintf("for i in $(seq 1 5); do nc -vz -w 8 %s.%s %d && exit 0 || sleep 1; done; exit 1", </ins> targetService.Name, targetService.Namespace, targetPort), }, },

What happened: I wanted to run the e2e network policy tests with: and I noticed that all tests are failing (I actually don't know why nobody has noticed this before). Looking into the logs I saw the following errors: which are correct because the tests using as "client" which doesn't contain . What you expected to happen: The e2e tests for network policies should actually pass. How to reproduce it (as minimally and precisely as possible): Just run: Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: kind OS (e.g: ): Kernel (e.g. ): Install tools: Others:
/sig testing
With the current fix two tests are still failing: I will take a look and fix these failing tests too.
Seems to be an implementation error on the network plugin side (tested with weave latest and calico 3.6). I will investigate some time in the implementation details on the calico side. But this error/failures are no blocker for the PR from my perspective.

As far as I know the busybox image doesn't contain , only so I don't understand how the PR that changed from to has passed the CI What happened: What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: /kind bug cc
/sig network
NetworkPolicy tests are a and so don't get run by default. But I imagine at least some of the CI test environments implement NetworkPolicy (e2e-gce?) so probably we should fix that. So 2 fixes: Make the tests pass again; probably requires changing the test image from busybox to something else. I bet there are other tests in that they switched from wget to curl where they did remember to change the image (because the tests would have failed otherwise) so this should just be a matter of seeing what image they used there, and use it here too. Make at least one of the CI tests test NetworkPolicy /good-first-issue (at least the first part anyway)
/remove-good-first-issue /remove-help (after someone asked me about this, I realized that trying to run tests that don't get run by CI is kind of complicated...)


Question: volumeSpec.Name(), uniquePodName, err) <ins> allVolumesAdded = false </ins> } glog.V(10).Infof(

Deployment/Service manifest: logs from events: In aws, the volume is attached, On the box: In only one is mounted: Where can I debug this further?
Take a look at the on your master for events related to your volume . That will show you when the volume was attached/detached. It appears that your volume attached without issue so there should be nothing unusual here. Then take a look at on the node that the volume is supposed to be attached to. Looks for the UID of your pod (). I suspect one of your mounts is failing. It looks like you are using hyperkube so it might be a mount namespace propagation issue.
I believe this happens when the RC/Pod is being created shortly after the PV or PVC. The latter two appear to need several seconds to be fully available (even if the status is already set to "Bound"). I ran into the same issue with a NFS PV. When I delete and create PV+PVC+SVC+RC all in one go, the associated Pod will be in ContainerCreating state for several seconds (10,<60), then log the warning event and afterwards be created without any problems or further delays. When I first create PV+PVC, wait for several seconds (30) and only then create SVC+RC, the pod will be created immediately and no warning event will be logged. One oddity I noticed is that the PVC will have capacity 0 and empty access modes (as shown by ), even though it was setup to request ROX for 1Gi and the PV provides 1Gi with ROX. This might be a different bug, though.
Thanks for digging into it The capacity 0 issue () was fixed by PR and is in v1.3.5. This is more or less expected behavior. The mount logic has exponential backoff so if it rapidly encounters PVC/PV does not exist, it'll back off the retries resulting in a longer mount time. We can consider optimizing this particular scenario. But I'll lower the priority on this for now and remove the milestone.
Maybe the "Bound" status could be delayed, so the RC would be held back for a while and be scheduled only when the PVC is actually ready?
I don't think the RC actually waits for PVC to be bound before scheduling.

I have also this problem using deployment with HELM, I deploy all my pv,pvc,deploy,svc in one Charts, and my pod who requiere a volume from the PVC get scheduled and stay in ContainerCreating mode. I need to wait minutes or delete pods for they correctly get rescheduled using the volume. Here is the log when the pod get automaticly deleted (after minutes) : It would be awesome if the deployment or RC wait for the PVC to be bound before schedule !
may slightly improve this situation as the maximum backoff time of 2 minutes will not fire anymore.
Note that the same error is thrown when node-size == C5 or M5 for kubectl and with kops . Solution is to set node as t2 machines. Due to it seems.
Reading from the new solution is:


Question: // This is a temporary workaround to get stats for cri-o from cadvisor // and should be removed. // Related to https://github.com/kubernetes/kubernetes/issues/51798 <del> if i.runtimeEndpoint == "/var/run/crio.sock" { </del> <ins> if i.runtimeEndpoint == CrioSocket { </ins> return cadvisorfs.LabelCrioImages, nil } }

BUG REPORT While configuring the kubelet with: And having the sock file on that location (default location from cri), the following error occurs: Later line: Uses the correct socket. Result: Disruptive in understanding other failures. Containers still start. Environment: Kubernetes version: 1.8.0-5, 1.9.0-beta.2 Cloud provider: gce OS: Ubuntu 16.04 Kernel (e.g. ): Linux kubeworker01 4.10.0-42-generic ~16.04.1-Ubuntu SMP Mon Dec 4 15:57:59 UTC 2017 x8664 x8664 x8664 GNU/Linux Install tools: The hard way Others: Full kubelet config (as of now): Internal log reference: syslog.v6.3
/area [kubelet] /sig [node]
/area kubelet /sig node /assign
This is because uses , while is using as default. We should make them consistent. Fixed in . PTAL.
What are the consequences? Is it more than cosmetical? Does it affect the usage of CRI-O in 1.9.0 or how does it affect users?
/assign Is this relevant for CRI-O in 1.9.0?
It (at this moment) mostly looks cosmetical, however when I add a cloud provider (--cloud-provider=gce in this case) to the kubelet config, the whole thing breaks consistently on PLEG. I am trying to figure out if there is a relation between that and the
Can you weigh in on whether this is a release-blocker for 1.9.0?
Tentatively adding milestone so we can track getting a judgement from sig-node.
-- i think this is fixed in the packaging, let me close with and
fixed in packaging yeah, from 1.8 we moved to "" so you need to start kubelet with that socket path
Discussed at sig-node earlier today, and this is not a release blocker. Also like mentioned above, the issue was fixed in the packaging.
There is a reference that needs to be fixed for the new socket location. The referenced pr is required to ensure stats summary endpoint understands the right location, and this keeps it consistent with cadvisor after its rebase in 1.9.

/priority critical-urgent /kind bug /sig node /status approved-for-milestone
/status in-progress
[MILESTONENOTIFIER] Milestone Issue Current Note: This issue is marked as , and must be updated every 1 day during code freeze. Example update: <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
Hi, I am hitting the same error in this thread on my Openshift4.2 cluster which has configured CRIO on the kubelet. I would like to hit to get inventory of the containers. Any body has any insights why I am hitting this error when I make curl calls from the k8s node which has configured for the CRIO? [root /]# curl -v --unix-socket http://localhost/info Trying Immediate connect fail for No such file or directory Closing connection 0 curl: (7) Couldn't connect to server [root /]# sshadmin kubectl version Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2020-01-14T21:04:32Z", GoVersion:"go1.13.5", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"14+", GitVersion:"v1.14.6+", GitCommit:"", GitTreeState:"clean", BuildDate:"2020-01-17T17:43:42Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"


Question: t.Errorf("Expected %#v, Got %#v", 2, lastIndex) } update := (<-ch).(kubelet.PodUpdate) <del> expected := CreatePodUpdate(kubelet.SET, kubelet.Pod{Name: "foo", Manifest: api.ContainerManifest{ID: "foo"}}) </del> <ins> expected := CreatePodUpdate(kubelet.SET, kubelet.Pod{Name: "foo", Manifest: manifest}) </ins> if !reflect.DeepEqual(expected, update) { t.Errorf("Expected %#v, Got %#v", expected, update) } <ins> for i := range update.Pods { if errs := kubelet.ValidatePod(&update.Pods[i]); len(errs) != 0 { t.Errorf("Expected no validation errors on %#v, Got %#v", update.Pods[i], errs) } } </ins> } func TestWatchEtcd(t *testing.T) {

The hash function generates names with upper-case letters, but the validation only allows lower-case. First, we clearly need a test for this. Second, we need to fix it. Should we allow upper-case characters (which the RFC does allow for DNS labels) or should we fix the hash? Allowing upper-case seems attractive, except that the RFC also says that case doesn't matter, but should be preserved "wherever possible". So we really have three choices, I see. a) only allow lower-case b) allow upper-case and always use caseless comparisons everywhere c) allow upper-case, but normalize to lower-case since it was your hash code
Re: test - sounds like a config source -config integration test, which I'll add. Re: hash - would argue we should fix the hash to generate lowercase for now. I think, but will double check, that the collision space is still improbably low for 63 lowercase chars. I can do that shortly.
And a validate test on all sources.
Lowercase would require base32, rather than base64, no? On Aug 3, 2014 8:20 AM, "Clayton Coleman" wrote:
Yeah, although I don't believe there's a difference to the outcome between base32 and base64(lowercase) given an appropriate hash function.
Well, base32 is reversible and a modified base64is not :) On Aug 3, 2014 9:00 AM, "Clayton Coleman" wrote:
Remind me again why reversibility is important? We discussed it in another issue but I can't find the context.
I'm going to change the generated name to which even with 2^8 files per host and a thousand hosts should be well below 0.% chance of collision
Its only important if we want to use it. We don't AFAIK. On Aug 3, 2014 9:34 AM, "Clayton Coleman" wrote:
Fix makes them unique, somewhat traceable (first 15 safe chars of the name), and generates base32 output.


Question: `, }, { <ins> name: "nil node metrics", summary: &statsapi.Summary{ Node: statsapi.NodeStats{ CPU: &statsapi.CPUStats{ Time:                 testTime, UsageCoreNanoSeconds: nil, }, Memory: &statsapi.MemoryStats{ Time:            testTime, WorkingSetBytes: nil, }, }, }, summaryErr: nil, expectedMetrics: ` # HELP scrape_error [ALPHA] 1 if there was an error while getting container metrics, 0 otherwise # TYPE scrape_error gauge scrape_error 0 `, }, { </ins> name: "arbitrary container metrics for different container, pods and namespaces", summary: &statsapi.Summary{ Pods: []statsapi.PodStats{

kubelet panic when accessing metrics/resource endpoint kubelet log kubelet works fine access metrics/resource endpoint of kubelet No response <details</details<details</details<details</details<details</details<detailsINFO[2022-07-14 05:36:21.474173892Z] Starting CRI-O, version: 1.23.0, git: (dirty) Version: 1.23.0 GitCommit: GitTreeState: dirty BuildDate: 1980-01-01T00:00:00Z GoVersion: go1.17.4 Compiler: gc Platform: linux/amd64 Linkmode: static BuildTags: apparmor, excludegraphdriverdevicemapper, seccomp, selinux SeccompEnabled: true AppArmorEnabled: true </details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/sig node


Question: "strings" "k8s.io/apimachinery/pkg/api/meta" <ins> "k8s.io/apimachinery/pkg/util/sets" </ins> "k8s.io/apimachinery/pkg/util/validation" ) // FormatMap formats map[string]string to a string. func FormatMap(m map[string]string) (fmtStr string) { <del> for key, value := range m { fmtStr += fmt.Sprintf("%v=%qn", key, value) </del> <ins> // output with keys in sorted order to provide stable output keys := sets.NewString() for key := range m { keys.Insert(key) } for _, key := range keys.List() { fmtStr += fmt.Sprintf("%v=%qn", key, m[key]) </ins> } fmtStr = strings.TrimSuffix(fmtStr, "n")

/kind bug What happened: When using the downward api to render a file via a volume mount from and annotation that has newlines in it, it appears that the file is rendered with the content sorted by line. A concrete example: Expected is that would contain But it does not The above is obviously contrived, this is creating a problem for my team when we are trying to use the downward api to render a yaml file (which obviously has newlines) Environment: Kubernetes version (use ): 1.9.7 Cloud provider or hardware configuration: aws OS (e.g. from /etc/os-release): centos
/sig apps Not sure about the sig here
/sig node /sig storage
Just tested it on 1.10.5 and this bug still happens.
thanks!


Question: } allErrs = append(allErrs, validateIPVSTimeout(config, fldPath)...) <del> allErrs = append(allErrs, validateIPVSSchedulerMethod(kubeproxyconfig.IPVSSchedulerMethod(config.Scheduler), fldPath.Child("Scheduler"))...) </del> allErrs = append(allErrs, validateIPVSExcludeCIDRs(config.ExcludeCIDRs, fldPath.Child("ExcludeCidrs"))...) return allErrs

The ipvs mode limit the scheduler algorithm to : now in new kernel have this new algorithm : the are specialy interesting... that be in kernel 4.18...
/sig network /area ipvs
/assign
This seems non-controversial on its face, but a) how do we know whether the kernel supports it or not; and b) if every node makes independent decisions, doesn't this basically become random? /assign uablrek
a) It should be the user's responsibility to ensure that the configured ipvs sheduler is available on all nodes This sounds rude, but please read for some rationale (and ). Exception may be the which is the default, but an installation with but without the sheduler is extremely unlikely. If that relaxing of the sheduler pre-test is acceptable, I would say that any current and future ipvs sheduler shall be allowed. Just treat them as a string. But it should be documented in the release-notes and some place else, but I don't know where.
b) For all shedulers that make decisions based on local state, yes. (that includes btw) Well, not really random ... For instance if is used and a new endpoint is addred, then all ipvs instances will get an "empty" target and will select it for new connections and may overload the newcomer, or it will become "statistically correct" over time. But IMHO users should be allowed to try. Some sheduler may be very useful in their environment. That will (already has) of course spawn some issues when it doesnt work as expected (for the reason in b) for instance), but allowing all shedulers will not change that.
So, my proposal is: Drop the scheduler module check Treat the configured scheduler as a string in the ipvs proxier Document
The module check can be replaced with a function check, e.g. insert/delete an ipvs dummy rule.
people is trying to reduce privileges on containers these days, does it requires more privileges than today or is the same?
The same. Kube-proxy configures ipvs already. This would only add an extra probing insert/delete on start. Even on first start (if ipvs is empty) with some care.
Could even be less privileges if "modprobe" requires any privileges (now or in the future)
What is your use-case?

I have wanted to check the sceduler in ipvs, but havn't got time yet. The load-balancer is stateless, but is not (well, at least not normally). A problem Maglev tries to solve is if the number of Application Servers (AS) (endpoints in K8s) changes, the updated hash buckets must be as few as possible. But if the load-balancer is stateful and stores the connections (like ipvs or the Linux conntrack used in proxy-mode=iptables), no existing connections are affected, no matter what hash algorithm is used for new connections. So, for both proxy-mode ipvs and iptables existing connection are not affected by adding or removing endpoints. But... if external traffic is routed to different nodes, e.g. a GW rehashes ECMP targets after a cluster scaling, then traffic is affected. This is easily tested: Setup a service of type LoadBalancer Route external traffic to the loadBalancerIP to node A Setup a number of TCP connections Re-route external traffic to the loadBalancerIP to node B All connections are lost (I have tested). But if the load-balancer would have been a stateless Maglev no connections would be lost. Google Maglev stores connections but only as a cache and getting the best of stateful, existing connections never lost on AS scaling, and stateless, the load-balancing tier can be scaled with minimum impact. This can be the case in ipvs , but is it?
Lars: proposal SGTM. IPVS impl is pretty much your call to make :) On Thu, Dec 22, 2022, 1:37 AM Lars Ekman
I have started working on a PR to remove the module checks. You are the only active reviewer and I will try my best to divide it into small reviewable commits as you wrote in :smile:
PR to remove the module checks I also intend to remove the constraint on ipvs schedulers, which will fix this issue. If you can create a VS with your configured scheduler you can use it, whatever that scheduler may be.
Looks like it is. Please read I have tested "mh" in kube-proxy (with a patch) and first problem is that is hashes on address only by default, so a traffic generator using the same source all the time will end up on the same endpoint.

This can be configured but will require special treatment in kube-proxy.
Should we close this now that is in?
/triage accepted
No, just removes the module ckeck. The validator still restricts the usable schedulers
to add afinity without using states. is the finality of maglev to avoid perf on managing state... and is more robust on target lost than other consistant hashing method... is normal is what you whant when you use maglev... to have afinity on ip src... but without need of storing states... is afinity by consistant hashing... cilium use this algorithm too....
the google research paper on the algorythm:
If you can build and use an own please try PR I have tested and "mh" can be configured, and with many src-addresses load-balancing works. However if external ingress traffic is re-routed from one K8s node to another, all connections are lost, same as "rr".
with ipvs the node who receive the traffic is considered a director, if you LB to other node... this node must have default route to the director because of nat masquerading of ipvs... when you use MH/RR with Direct Server Return avoid that problem (no more nat masquerading)... the node respond to the client directly (no need to return back to the first node who receive the traffic). but you need to tunnel the traffic from node who receive the request to the node you forward the traffic (cloud flare use fou protocol for that)... i dont know if kubeproxy already support that sort of configuration... cilium for LB DSR + Maglev use tcp option field to store the Virtual IP that receive the traffic in the packet and nat the destination with the pod ip before sending to the node who host the node (that work in full routing mode), and the node respond to the client directly in replacing thaire ip with the Virtual IP from tcp option field.... in that way no state needed... no tunneling... no complex network configuration.... some has been done on kube-router project: and i think you must use in conjonction with MH/RR to use Direct Server Return if i understand whell... and good exemple of the use case : that use ipvs (use ipip tunnel) to do that....
Regarding "mh" algorithm, I submitted PR a few weeks ago which adds the new load distribution method to kube-proxy.

As this commit supports 'mh" with hashing on srcIP and srcPort by default, connections from the same srcIP with different srcPorts will be distributed equally to all EndPoints. If users want to have an affinity on srcIP, they can use "mh" with "Session Affinity" option enabled. The PR is targeted for the main use-cases below: Connection failover from a worker node to another. Scale-in/out worker nodes Rerouting from GWs or external LoadBalancer changes receiving worker nodes of existing traffic flows. I would need to rebase the commit after your fix for this issue PR . PR is now waiting triage state, please verify if it is reasonable to add this new algorithm to kube-proxy.
No, and it probably never will. The proper (only) way to get this into is to write a (KEP). To :
I would also like to see a demo or e2e test working with this setup, it doesn't seem obvious from the comments that this will work out of the box ... The project Kubernetes, as Lars correctly mentions , use KEPs to discuss and document feature, I agree with him and I think this will require to write a KEP
What I meant was your use-case in K8s. Having worked most of 2021 with the Maglev hash algorithm when developing the I know what it can do, but I also know K8s and I think I can say right now; To allow the "mh" in kube-proxy=ipvs will not give you what you hope for.
And a side-remark on DSR. Cilium can do this because they modify packets with eBPF. The traditional way of doing DSR is to actually assign the Virtual IP address (VIP) to an interface in the server (the POD in the K8s case). This must have support in the CNI-plugin, which I guess is the case for as it handles load-balancing and is a CNI-plugin. To add generic support for DSR in K8s is not simple.
I'm new to this community. Thanks for your suggestion about the process of adding a new feature to kube-proxy. I'll prepare a KEP for the "mh" load distribution.
But in any case I would like PR to be merged. For sanity reasons if not others.

The for scheduler=mh can/should be in a separate issue+PR. This will allow users to test "mh" in kube-proxy, because; Nobody will take my word for it :smile:
I don't want to sound discouraging but IMO you should not spend too much time on a KEP before presenting it to get some early feedback. I can't see how "mh" can provide the statelessness unless DSR is used, and that may be very hard to get into the . So, I suggest you describe a use-case in the KEP, and what it takes to support it, and present it on a bi-weekly sig/network meeting. And don't worry, people are nice there :smile:
Perhaps DSR is not needed to use with "mh" since I cannot really see any problem with nat masquerading of ipvs if subsequence packets are sent to the second node. With net/ipv4/vs/sloppy_* enabled, ipvs on the second node accepts the packets as an existing connection and then keeps forwarding them to the original endpoint thanks to the mh algorithm. IPVS of course performs snat and "remember" the client IP of the packets as if they are sent to the first node. When receiving replies from the endpoint, ipvs on the second node will be able to forward the replies to the client with the cached client IP in its NAT table. Please correct me if I'm missing something in this regard.
So even if the kube-proxy forwards packets to the same POD (assuming the same mh configuration in node A and B) the receiving POD will reject with a TCP-RST packet.
Since you have written PR you should be able to test this with you own-built kube-proxy. For me all connections are teminated when re-directing ingress traffic from one K8s node to another.
Thanks for your clarification. Now I could see this problem is on the pod side, not ipvs. It seems that having DSR does not help to solve the problem. Though, I suppose this issue will not happen with the following scenarios: UDP traffic Traffic is sent to Cluster-IP instead of NodeIP
With DSR the src=VIP all the way, so it should be OK, even if masq (DNAT) is used.

UDP same thing, unless your UDP server has no session and doesn't care about addresses, just the server port. This is however not the usual case. Traffic sent to the Cluster-IP will be have src=pod, yes, but it will always be hadled by the local kube-proxy, so the use-case to move traffic to another node doesn't exist for clusterIP.
Me neither :) Agree - you will have the best experience if do this iteratively. Start with an email to the SIG list. Then a doc discussing the peoblem and high-level solution details. THEN a KEP makes sense.
cilium assign vip also but more higher in the kernel stack with ebpf... in that way is more simple to implemente than in ipvs... juste simple packet manipulation (plus the fact that is working only in full routed mode)... ipvs can do that, helped with some config on each node (configuring tunnel for dsr)... for mh usage without dsr we need to add default route on other node to the actual dedicated kubeproxy as l4 director (one node can be designated as director and move to another node when the first one die). is generaly more for loadbalancer/external ip usage....
Is this an acceptable way? merged. This will allow "mh" to be configured as a ipvs scheduler, but only with L3 hashing (no ports). a new issue like "Ipvs scheduler=mh only hashes on addresses, not ports", and implement it in a new (small) PR with FlagSourceHash as have already done in this issue as "done" since "mh" can be configured and as it's implemented in another way. the work towards a KEP to make "mh" work as you want it to work. Including a K8s use-case, DSR (if needed in some form), setting of sysctl's, etc. I leave item 4. to you and . Item 1-3 will enable "mh" but because of NAT, connections are still lost when traffic is re-directed from one node to another (this can be tested without too much effort. I have done it).
Agree with the approach. fixes the raised issue that it enables all distribution methods supported by ipvs in kube-proxy (including "mh" one).

A separate issue for enabling source port hash when using 'mh' sounds reasonable. This is not only applicable for "mh", but also for the "sh" method which kube-proxy is currently supporting as stated in k8s documents. Agree. Sure. I'll need to verify the issue further and check if any easy way possible to avoid the NAT issue on different nodes. IMHO, replacing VIP (virtual IP) with NodeIP seems to be the first option to overcome the issue, but we need to verify its feasibility and impact in k8s.
To allow the "mh" in kube-proxy=ipvs will not give you what you hope for. Nobody will take my word for it :smile: Ok, ok, please let me rephrase: To just allow "mh" in the current ipvs proxier will not give you what you hope for. So, is not enough.
/kind feature


Question: import ( "fmt" <ins> autoscalingv1 "k8s.io/kubernetes/pkg/apis/autoscaling/v1" </ins> "k8s.io/kubernetes/test/e2e/common" "k8s.io/kubernetes/test/e2e/framework"

Currently the time it takes for HPA and CA to scale up after load increase is not measured at all, and this can lead to regressions that will be missed. This issue is about adding an e2e test that will prevent such regressions.
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list .
Filip, can you reopen? I cannot do this myself, and this issue is clearly not closed yet.
Thanks Wojtek!


Question: func SetFeatureGateDuringTest(tb testing.TB, gate featuregate.FeatureGate, f featuregate.Feature, value bool) func() { originalValue := gate.Enabled(f) <ins> // Specially handle AllAlpha and AllBeta var cleanups []func() if f == "AllAlpha" || f == "AllBeta" { // Iterate over individual gates so their individual values get restored for k, v := range gate.(featuregate.MutableFeatureGate).GetAll() { if k == "AllAlpha" || k == "AllBeta" { continue } if (f == "AllAlpha" && v.PreRelease == featuregate.Alpha) || (f == "AllBeta" && v.PreRelease == featuregate.Beta) { cleanups = append(cleanups, SetFeatureGateDuringTest(tb, gate, k, value)) } } } </ins> if err := gate.(featuregate.MutableFeatureGate).Set(fmt.Sprintf("%s=%v", f, value)); err != nil { tb.Errorf("error setting %s=%v: %v", f, value, err) }

PR was reverted by because it (erroneously) relied on OpenAPIV3 feature to be enabled, but would non-deterministically leave the feature disabled after running. I tracked it down to the fact that this test disables all or features in a loop which iterates over a , which are known to have unstable iteration order: When key is processed before key , then would disable before s default value is to be saved. Normally, ignores features that have been explicitly set, so it only affects those which fall back on the value. But this issue does not just affect but any non-GA feature. I suggest the loop in be made to skip the special feature keys such as . /cc /cc The value before every feature gate should be restored after running . This test always fails on my machine (just pasted it into ): No response <details</details<details</details<details</details<details</details<details</details<details</details
/assign
Wow, good catch, sorry to make you track down what turned out to be a bug in our test
/sig auth
/area test
/triage accepted


Question: // ForEach calls the provided function for each allocated IP. func (r *Range) ForEach(fn func(net.IP)) { r.alloc.ForEach(func(offset int) { <del> ip, _ := utilnet.GetIndexedIP(r.net, offset+1) // +1 because Range doesn't store IP 0 </del> <ins> ip, _ := GetIndexedIP(r.net, offset+1) // +1 because Range doesn't store IP 0 </ins> fn(ip) }) }

What happened: All the services are asking to be recreated because the service CIDR changed after upgrading from 1.16.3 ro 1.17.0 What you expected to happen: Smooth migration from to , which would not break service CIDR. How to reproduce it (as minimally and precisely as possible): Do an upgrade from version 1.16.3 to 1.17.0 Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: OS (e.g: ):Kernel (e.g. ):Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig node
do you mean that the service cidr changed without any intervention after the upgrade? EDIT 96.0.0/12 is the default service cidr when using kubeadm --service-cidr string Default: "10.96.0.0/12" EDIT if not using kubeadm the default is May it be possible you were using another service cidr before the upgrade and you didn't set it during the upgrade, so it fall back to the default?
Can you provide your kube-apiserver invocation?
xref which warnings if no explicit CIDR flag was passed but should not have changed behavior otherwise
Where do you see that? The default is specified here as 10.0.0.0/24 and did not change between 1.16.0 and 1.17.0:
I was assuming it was using kubeadm based on the log message 96.0.0/12 is the default service CIDR for kubeadm
ah, ok. If they are using kubeadm, that default also did not change between 1.16.0 and 1.17.0:
a few questions for you: how did you install your 1.16.x cluster? did you customize the service cidr in that deployment? how did you upgrade your 1.16.x cluster to 1.17.x? did you apply the same service cidr customization in that deployment?
16.x was a straight and normal install no change on service CIRD, and we have never touched this CIDR configuration. And kubeadm is supposed to be used during the upgrade from 1.16.x to 1.17.x.

We just followed the describe process: The logs system started showing the shared message.
can you reproduce?
can you paste ? That will show you kubeadm config
created a single node cluster with v1.16.4 and a custom service cidr: applied upgrade to v1.17.0: i do not see a change in the custom service cidr.
if you never touched the service CIDR on the initial install it should have been which is the kubeadm default. your config map shows . how did you adapt the IP to work with the kubeadm default service cidr () if you initially deployed without modifying said cidr? to me this suggests that a change was required at some point.
The only change is the update from 1.16.3 to 1.17.0, done yesterday. Please find below a snapshot from our logs: <img width="1635" alt="Screen Shot 2019-12-21 at 22 22 29" src="https://user-"
my point is that if you did not apply a custom service cidr on "kubeadm init", then this cidr was the default out of kubeadm and after upgrade the service cidr was preserved as your config map shows. what was the elasticsearch/logging service IP before upgrade? EDIT: also do you have the YAML that was used to create this service?
101.249.43 belongs to the service rubnet range at minimum the error message is wrong /kind bug
are those errors happening constantly or only happened during the upgrade? are you able to create new services within that range? let's say, with a service IP of 10.101.249.45 Also, can you paste your current service-cluster-cidr configured in the controller-manager and apiserver? i.e.
this is a bug in core: this call in the repair controller returns for an IP that is actually in the CIDR range: also indicates a missing unit test case. latest commits: this might have regressed with the dual-stack changes? cc /remove-triage support needs-information /sig network /remove-sig cluster-lifecycle
/priority important-longterm
I was able to reproduce the same behavior again, with a fresh install k8s release 1.16.4.

Just After installing metrics server, we could notice a shift in the service CIDR, but nothing in the logs.
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/cc
so the problem here is that an IPv4 CIDR such as translates to addresses. but this line here clamps the maximum to 2^16 () so for an address like the logic in the ip allocator fails in the function: as the offset ends up bigger than the maximum. it feels like the maximums for IPv4 and IPv6 should be different and larger. otherwise for CIDRs in the notation : can give incorrect results for IPv4 can give incorrect results for IPv6 cc who touched related logic.
I think that was touched in … was this bug preexisting or did that change behavior?
before PR () i do not see a clamp for IPv4: only one for IPv6 at 16bits.
We should: revert add unit tests for the missed cases get that into 1.17.1 reintroduce the reliance on the function carefully
What would be the safest path to recreate all the services whilst waiting a final solution ? Especially for production K8S.
to mitigate the error in 1.17.0 you need to modify the serivce CIDR to be , instead of . modify the ConfigMap CIDR to have modify the kubeadm static pod manifest for the api server () to have edit problematic Service objects and give them an IP in the range if needed.
Thank you for those instructions, I'm affected by this, does this have anything to do with a possibility of ClusterIP's not properly responding or does it only print out alerts?
hello, can you test if the ClusterIP is not responding? i don't think we got a confirmation if this is only an alert. my guess is that it is just that, but there might be side effects.
i see you logged i will comment there.
How do you "recreate" a service? Literally delete and create again? Or is there a command which would cause it to re-create itself without having to be deleted?


Question: defer func() { podClient.Delete(podName, nil) <del> err := waitForPodNotFoundInNamespace(client, podName, injectPod.Namespace, PodDeleteTimeout) ExpectNoError(err) </del> }() injectPod, err := podClient.Create(injectPod)

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: Which test(s) are failing: [sig-storage] In-tree Volumes [Driver: local][LocalVolumeType: block] [Testpattern: Pre-provisioned PV (default fs)] volumes should be mountable Since when has it been failing: Testgrid link: Reason for failure: Volume of previous pod is not unmounted before the second pod is trying to mount its volume. Anything else we need to know:
/sig storage /assign
The problem is when client is deleting first pod, it is deleted immediately. This is because when pod status phase is , . But pod is set to before its volumes are unmounted and detached. pod object may be deleted from apiserver before its volumes are unmounted, so it may have conflicts in setting fsGroup for second pod `in-tree-client. If we want to wait for the pod to be deleted by kubelet instead of apiserver, we need to make it a long-running pod. Alternatively, we can make the two pods use the same fsgroup . cc What do you think?
cc is this expected? I thought kubelet is supposed to delete the pod object only after the volumes are unmounted. This also impacts the StorageProtection feature, which assumes that if the Pod object is deleted, then the volume has been unmounted. For a storage backend like nfs, it may result in unmount hanging. cc to fix the flakiness, the simplest thing may be to make the two pods use the same fsgroup. I don't see a strong use case for two pods using different fsgroups.
That is only during graceful deletion. If a pod is Succeeded, do we begin unmounting volumes? Or do we wait until the deletion timestamp is set? Would it be good enough to start unmounting when the pod is failed/succeeded? If we want to be 100% sure this isn't a problem, we would need to always do graceful deletion, even when the pod is failed/succeeded.
Yes, we do start unmounting when pod is . The API is setting grace period to 0 in the case of pod success or failed. Maybe that needs to be removed if we want to use Pod deletion as a signal that volumes are unmounted.

To always do graceful deletion even when the pod is failed/succeeded is simple, then we can assume that normally when pod objects are deleted (except forcefully), volumes are already unmounted by kubelet. But , and it helps to improve performance in deleting jobs in batch. For this flaky test only, what I need is to make sure volumes are unmounted before the second pod is going to mount them, but local volumes are not reported in or other fields of node status, I can't make sure of this by checking node status either. Possible workarounds: make the first a long-running pod (too hacky) wait for a while before starting the second pod (short period may be flaky too, long period increases test time) use same fsGroup for both pods (do not test that fsGroup can be changed) I'm going to use same fsGroup for both pods until we have a reliable way to detect volumes are unmounted or not.


Question: return watcher.Stop } <del> // StartRecordingToSink starts sending events received from the specified eventBroadcaster to the given sink. func (e *eventBroadcasterImpl) StartRecordingToSink(stopCh <-chan struct{}) { go wait.Until(func() { e.refreshExistingEventSeries() }, refreshTime, stopCh) go wait.Until(func() { e.finishSeries() }, finishTime, stopCh) </del> <ins> func (e *eventBroadcasterImpl) startRecordingEvents(stopCh <-chan struct{}) { </ins> eventHandler := func(obj runtime.Object) { event, ok := obj.(*eventsv1.Event) if !ok {

Which jobs are flaking: periodic-kubernetes-bazel-test-master (xref ) Which test(s) are flaking: TestEventSeriesf Testgrid link: Reason for failure: Timeout: Anything else we need to know: To reproduce with stress: /cc
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
filed to address this issue.
contains the correct analysis of flakes (thanks a lot!), I commented on way of fixing it (which I'm not convinced about).


Question: // see an update podUpdate := CreatePodUpdate(kubetypes.ADD, TestSource, CreateValidPod("foo", "")) channel <- podUpdate <ins> expectPodUpdate(t, ch, CreatePodUpdate(kubetypes.ADD, TestSource, CreateValidPod("foo", ""))) </ins> config.Sync() <del> expectPodUpdate(t, ch, CreatePodUpdate(kubetypes.SET, kubetypes.AllSource)) </del> <ins> expectPodUpdate(t, ch, CreatePodUpdate(kubetypes.SET, kubetypes.AllSource, CreateValidPod("foo", ""))) </ins> } func TestNewPodAddedDefaultNamespace(t *testing.T) {

To read a pod manifest file from disk (or from the http source), kubelet execute the following code to Decode a on-disk v1 pod manifest to an internal pod object (the defaults are applied in this process). Validate the internal pod. Convert the internal pod back to a v1 pod. The downstream pod cache (which gets v1 pods from different sources) performs similar operations: Convert the v1 pod to an internal pod. Validate the internal pod. One of two conversion/validation should be removed, and some of them should be simplified.
/cc for suggestions on defaulting and conversion.
If no one is working on this, I will take it.


Question: func (s *APIServer) AddFlags(fs *pflag.FlagSet) { // Note: the weird ""+ in below lines seems to be the only way to get gofmt to // arrange these text blocks sensibly. Grrr. <del> fs.IntVar(&s.Port, "port", s.Port, ""+ </del> <ins> fs.IntVar(&s.WideOpenPort, "port", s.WideOpenPort, ""+ </ins> "The port to listen on. Default 8080. It is assumed that firewall rules are "+ "set up such that this port is not reachable from outside of the cluster. It is "+ "further assumed that port 443 on the cluster's public address is proxied to this "+

Trying to hit the read-write kubernetes service from inside a cluster does not work. tailing the kube-proxy logs shows: And sure enough, kube-apiserver is not listening on the NIC on that port. 8080 is only for the localhost HTTP communications between master components (and nginx redirect after HTTPS termination). The "kubernetes" master service needs to get created with Port set to 443.
+1
This needs to be port 6443 the auth_port On Feb 6, 2015 10:14 AM, "Paulo Pires" wrote:
The code says 8443, but then a flag changes to 6443. Will fix. , Eric Tune wrote:
Holy macaroni this is a mess. , Tim Hockin wrote:


Question: kube::util::wait_for_url "http://${API_HOST}:${API_PORT}/api/v1beta1/pods" "apiserver: " CTLRMGR_LOG=/tmp/kube-controller-manager.log <del> "${GO_OUT}/kube-controller-manager"  </del> <ins> sudo "${GO_OUT}/kube-controller-manager"  </ins> -v=${LOG_LEVEL}  --machines="127.0.0.1"  --master="${API_HOST}:${API_PORT}" >"${CTLRMGR_LOG}" 2>&1 & CTLRMGR_PID=$! KUBELET_LOG=/tmp/kubelet.log <del> "${GO_OUT}/kubelet"  </del> <ins> sudo "${GO_OUT}/kubelet"  </ins> -v=${LOG_LEVEL}  --etcd_servers="http://127.0.0.1:4001"  --hostname_override="127.0.0.1"

/kind failing-test <!-- If this is a flake, please add: /kind flake --<!-- Please assign a SIG using: /sig SIG-NAME --
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
looks like started a few days ago ! /sig cluster-lifecycle
cc
fix was sent earlier
/sig cluster-lifecycle
thanks :) i saw it yesterday and forgot to raise the issue. i should have checked the PR queue :)


Question: } func TestDescribeService(t *testing.T) { <del> fake := fake.NewSimpleClientset(&api.Service{ ObjectMeta: metav1.ObjectMeta{ Name:      "bar", Namespace: "foo", </del> <ins> testCases := []struct { service *api.Service expect  []string }{ { service: &api.Service{ ObjectMeta: metav1.ObjectMeta{ Name:      "bar", Namespace: "foo", }, Spec: api.ServiceSpec{ Type: api.ServiceTypeLoadBalancer, Ports: []api.ServicePort{{ Name:       "port-tcp", Port:       8080, Protocol:   api.ProtocolTCP, TargetPort: intstr.FromInt(9527), NodePort:   31111, }}, Selector:              map[string]string{"blah": "heh"}, ClusterIP:             "1.2.3.4", LoadBalancerIP:        "5.6.7.8", SessionAffinity:       "None", ExternalTrafficPolicy: "Local", HealthCheckNodePort:   32222, }, }, expect: []string{ "Name", "bar", "Namespace", "foo", "Selector", "blah=heh", "Type", "LoadBalancer", "IP", "1.2.3.4", "Port", "port-tcp", "8080/TCP", "TargetPort", "9527/TCP", "NodePort", "port-tcp", "31111/TCP", "Session Affinity", "None", "External Traffic Policy", "Local", "HealthCheck NodePort", "32222", }, </ins> }, <del> Spec: api.ServiceSpec{ Type: api.ServiceTypeLoadBalancer, Ports: []api.ServicePort{{ Name:       "port-tcp", Port:       8080, Protocol:   api.ProtocolTCP, TargetPort: intstr.FromInt(9527), NodePort:   31111, }}, Selector:              map[string]string{"blah": "heh"}, ClusterIP:             "1.2.3.4", LoadBalancerIP:        "5.6.7.8", SessionAffinity:       "None", ExternalTrafficPolicy: "Local", HealthCheckNodePort:   32222, </del> <ins> { service: &api.Service{ ObjectMeta: metav1.ObjectMeta{ Name:      "bar", Namespace: "foo", }, Spec: api.ServiceSpec{ Type: api.ServiceTypeLoadBalancer, Ports: []api.ServicePort{{ Name:       "port-tcp", Port:       8080, Protocol:   api.ProtocolTCP, TargetPort: intstr.FromString("targetPort"), NodePort:   31111, }}, Selector:              map[string]string{"blah": "heh"}, ClusterIP:             "1.2.3.4", LoadBalancerIP:        "5.6.7.8", SessionAffinity:       "None", ExternalTrafficPolicy: "Local", HealthCheckNodePort:   32222, }, }, expect: []string{ "Name", "bar", "Namespace", "foo", "Selector", "blah=heh", "Type", "LoadBalancer", "IP", "1.2.3.4", "Port", "port-tcp", "8080/TCP", "TargetPort", "targetPort/TCP", "NodePort", "port-tcp", "31111/TCP", "Session Affinity", "None", "External Traffic Policy", "Local", "HealthCheck NodePort", "32222", }, </ins> }, <del> }) expectedElements := []string{ "Name", "bar", "Namespace", "foo", "Selector", "blah=heh", "Type", "LoadBalancer", "IP", "1.2.3.4", "Port", "port-tcp", "8080/TCP", "TargetPort", "9527/TCP", "NodePort", "port-tcp", "31111/TCP", "Session Affinity", "None", "External Traffic Policy", "Local", "HealthCheck NodePort", "32222", } c := &describeClient{T: t, Namespace: "foo", Interface: fake} d := ServiceDescriber{c} out, err := d.Describe("foo", "bar", printers.DescriberSettings{ShowEvents: true}) if err != nil { t.Errorf("unexpected error: %v", err) </del> } <del> for _, expected := range expectedElements { if !strings.Contains(out, expected) { t.Errorf("expected to find %q in output: %q", expected, out) </del> <ins> for _, testCase := range testCases { fake := fake.NewSimpleClientset(testCase.service) c := &describeClient{T: t, Namespace: "foo", Interface: fake} d := ServiceDescriber{c} out, err := d.Describe("foo", "bar", printers.DescriberSettings{ShowEvents: true}) if err != nil { t.Errorf("unexpected error: %v", err) } for _, expected := range testCase.expect { if !strings.Contains(out, expected) { t.Errorf("expected to find %q in output: %q", expected, out) } </ins> } } }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: when using , the TargetPort is wrong What you expected to happen: if the targetPort is string, show the name correctly. Environment: Kubernetes version (use ): Cloud provider or hardware configuration**: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
could you paste more details about the Kubernetes Version and the yaml you used to create the service? I tried to reproduce this error but it worked fine in my environment.
use the master branch. yaml output
I'm not sure if the targetPort could be set to string type, looks like the targetPort here makes no sense of port1 for it doesn't point to an actual port. I think we have to validate this value rather than displaying the string. WDYT?
It can be a string, you can see this
thanks for your ref, I will try to fix this and pls let me know if any new progress :)
I have commit a pr
ok, thanks~


Question: csitranslationplugins.AzureFileInTreePluginName: func() bool { return utilfeature.DefaultFeatureGate.Enabled(features.CSIMigrationAzureFile) }, <ins> csitranslationplugins.VSphereInTreePluginName: func() bool { return true }, </ins> csitranslationplugins.PortworxVolumePluginName: func() bool { return utilfeature.DefaultFeatureGate.Enabled(features.CSIMigrationPortworx) },

A pod that uses in-tree vSphere volume never starts in Kubernetes 1.29.0. Such pod should start as usual. See above Feature gate removal deleted that vSphere volumes are migrated to CSI. So the CSI volume plugin ignores in-tree vSphere volumes and does not translate them to CSI. <details</details<details</details<details</details<details</details<details</details<details</details
/sig storage /priority important-soon /triage accepted /assign
cc have you ever tested with CSI migration?
/assign
I already have a PR: