Question: serviceQ := ServicePortName{types.NamespacedName{"testnamespace", "echo"}, "q"} serviceX := ServicePortName{types.NamespacedName{"testnamespace", "echo"}, "x"} <del> p := CreateProxier(lb, net.ParseIP("0.0.0.0"), &fakeIptables{}, net.ParseIP("127.0.0.1")) </del> <ins> p := createProxier(lb, net.ParseIP("0.0.0.0"), &fakeIptables{}, net.ParseIP("127.0.0.1")) </ins> waitForNumProxyLoops(t, p, 0) p.OnUpdate([]api.Service{{

If another process is altering iptables rules at the time k8s attempts to add iptables rules it will fail with a locking error. k8s should use -w and retry after a delay. Below, an external process (openshift-sdn-node) modifies iptables at roughly the same time and k8s fails without retrying. Related to
Sorry, I misunderstood the -w flag, with it retry logic wouldn't be necessary as it'd wait. Perhaps simply retrying is a better solution as -w flag may not exist everywhere we'd like.
Looks like -w was to iptables 1.4.20(August 2013).
Yeah, we have to be pretty back-compatible with iptables. We always retry after a few seconds - is there really a bug here? , Scott Dodson wrote:
It definitely seems like there is some sort of a problem. We see this randomly on system startup. In our environment OpenShift is launching kubernetes around the same time other systemd units are modifying the firewall config. If one of these others units has the lock kubernetes will report the error Scott mentions and then will never create the required nat'ing for services. Here's how to reproduce this: At this point if you run you will see the exact same error kubernetes reports. It's probably not obvious why you have to set the breakpoint at 257 but if you set the breakpoint at 260 and then step through the code it appears that the compiler has optimized lines 263 and 257 to be the same. In any case running will verify if you have the lock or not. Once you've verified you have the lock just restart OpenShift (running kubernetes v0.14.1-582-gb12d75d): At this point you can press in gdb to continue the iptables invocation and release the lock. Afterwards I will see output from cadvisor every once in a while, however the will show the firewall is still blank: I initially restarted Kubernetes with a blank firewall at 11:21 and yet at 11:38 I still have nothing:
I probably should have made it more clear, I removed the lock by pressing 'c' in gdb to continue. That's when I started waiting. I'll update my comment to make that more clear.
So something for a holding the lock?

I am out of your office for the rest of this week, but maybe you can figure out - is there a stale process holding a lock? What kind of lock is it? Flock or semaphore or ... ? On Apr 29, 2015 8:44 AM, "Brenton Leanhardt" wrote:
What we're saying is that on startup it's fairly common for multiple systemd units to make firewall additions. If one of them grabs the lock (rarely for more than a few seconds) it will cause kubernetes to fail to initialize iptables and it will never fix itself. The lock only need be grabbed the microsecond kubernetes tries to initialize iptables for this bug to present itself. We know exactly which program in our environment was concurrently making iptables updates. The problem is that kubernetes doesn't appear to be retrying if it fails to initialize iptables.
On RHEL the lock is the xtables unix socket lock in iptables to allow multiple programs to update iptables concurrently. Other platforms may use a flock since that is the default upstream. To solve this kubernetes needs one more more of the following: pass the flag to as scott and the logs mention (possibly only available on some platforms). If kubernetes desides to wait for the lock it will need a timeout. retry the firewall initialization quickly and maybe backoff if it still can't get the lock It's probably worth noting that if iptables on a platform doesn't have iptables lock support concurrency is likely broken already in other ways for that platform.
It does retry all up tables ops. Unless it somehow gets pathologically phase locked with your agent, or there is some deeper bug. We have e2e tests that prove you can delete tables and it will recreate them. Are you saying that it never retries (did you set -v=4?) or that every retry fails? On RHEL the lock is the xtables unix socket lock in iptables to allow multiple programs to update iptables concurrently. Other platforms may use a flock since that is the default upstream. To solve this kubernetes needs one more more of the following: pass the -w flag to iptables as scott and the logs mention (possibly only available on some platforms). If kubernetes desides to wait for the lock it will need a timeout.

retry the firewall initialization quickly and maybe backoff if it still can't get the lock It's probably worth noting that if iptables on a platform doesn't have iptables lock support concurrency is likely broken already in other ways for that platform. — Reply to this email directly or view it on GitHub .
I can grab the lock for just the second that kubernetes needs it and then let it go. The kube-proxy nat'ing will never be rebuilt without intervention (restarting kubernetes). That seems like a bug to me and is what my original comment reproduces. In addition, once the environment is in this state it won't even pick up new service nat rules. Other nodes in my environment will pick them up but this one is dead. I highly suspect it has to do with the fact that iptables initialization is failing. We test all the time that once a system is up and running you can clean iptables and it will come back. This is a slightly different case.
This seems like a v1 blocker - we see this in every realistic environment we put the Kube-proxy in. Disagreement?
I still don't understand this issue. It is demonstrably true that the kube-proxy retries all of the iptables logic periodically. It's possible (given the mention of "won't even pick up new service nat rules") that this locking tickles a bug in kube-proxy, but I am having trouble reproducing it. Can you run your kube-proxy with -v=4, trigger the bug, and then take a look at the logs? If this is real, yeah it's a 1.0 blocker.
The problem is in the initialization process. If you hold the lock while the kube-proxy tries to initialize itself then it will be wedged forever. What you're showing is the kube-proxy can repair the iptables rules after the initialization code has run. The logs I pasted earlier show that after the initialization fails at it will never try again. I tend to run at loglevel 4 by default but I'll try to find time today to run through this again.
I have a fix coming - can you test the PR before commit?
I ran the exact same test as mentioned in my original comment.

While the debug lines have changed the end result is the same: grab the lock (at this point even will fail) start the kube-proxy release the lock (at this point works) wait.. try creating another service wait.. The rules never come back. Here are the logs:
Sure, I'll test the PR. I'm betting it will apply fairly cleanly on to my OpenShift codebase. If not I'll launch Kubernetes directly.
Many of those log lines are not part of upstream. In fact, what upstream has is a glog.Fatalf() which will kill kube-proxy, and when it restarts it will try again. Did you guys break it by not actually exiting?
Looks like we did. We need to put it into a restart loop. ----- Original Message -----
is teh fix, though I assert we should not need it, and if you need it this fix may not help you anyway. I am still OK to commit this.
Although NewProxier sucks because it doesn't return a usable error. I'll take the todo to clean up that part. ----- Original Message -----
hey, it logs its error - nothing wrong with that :) , Clayton Coleman wrote:
:)
Opened and will move the rest back to openshift.
I restarted docker and it worked then


Question: // assert that unwanted pods were killed fakeRuntime.AssertKilledPods([]string{"12345678"}) <ins> // simulate Runtime.KillPod fakeRuntime.PodList = nil </ins> kubelet.HandlePodCleanups() kubelet.HandlePodCleanups() kubelet.HandlePodCleanups()

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Right after calling , returns false even the pod is not killed yet. What you expected to happen: In kubelet, after is complete, the pod should be pending termination or be killed. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Related to Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node /assign


Question: klog.V(2).Infof("GetDiskLun returned: %v. Initiating attaching volume %q to node %q.", err, volumeSource.DataDiskURI, nodeName) isManagedDisk := (*volumeSource.Kind == v1.AzureManagedDisk) <del> err = diskController.AttachDisk(isManagedDisk, volumeSource.DiskName, volumeSource.DataDiskURI, nodeName, compute.CachingTypes(*volumeSource.CachingMode)) </del> <ins> lun, err = diskController.AttachDisk(isManagedDisk, volumeSource.DiskName, volumeSource.DataDiskURI, nodeName, compute.CachingTypes(*volumeSource.CachingMode)) </ins> if err == nil { klog.V(2).Infof("Attach operation successful: volume %q attached to node %q.", volumeSource.DataDiskURI, nodeName) } else {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /sig azure /assign


Question: "k8s.io/klog/v2" ) <del> // NewDeltaFIFO returns a Queue which can be used to process changes to items. // // keyFunc is used to figure out what key an object should have. (It is // exposed in the returned DeltaFIFO's KeyOf() method, with additional handling // around deleted objects and queue state). // // 'knownObjects' may be supplied to modify the behavior of Delete, // Replace, and Resync.  It may be nil if you do not need those // modifications. // // TODO: consider merging keyLister with this object, tracking a list of //       "known" keys when Pop() is called. Have to think about how that //       affects error retrying. // NOTE: It is possible to misuse this and cause a race when using an //       external known object source. //       Whether there is a potential race depends on how the consumer //       modifies knownObjects. In Pop(), process function is called under //       lock, so it is safe to update data structures in it that need to be //       in sync with the queue (e.g. knownObjects). // //       Example: //       In case of sharedIndexInformer being a consumer //       (https://github.com/kubernetes/kubernetes/blob/0cdd940f/staging/ //       src/k8s.io/client-go/tools/cache/shared_informer.go#L192), //       there is no race as knownObjects (s.indexer) is modified safely //       under DeltaFIFO's lock. The only exceptions are GetStore() and //       GetIndexer() methods, which expose ways to modify the underlying //       storage. Currently these two methods are used for creating Lister //       and internal tests. // // Also see the comment on DeltaFIFO. // // Warning: This constructs a DeltaFIFO that does not differentiate between // events caused by a call to Replace (e.g., from a relist, which may // contain object updates), and synthetic events caused by a periodic resync // (which just emit the existing object). See https://issue.k8s.io/86015 for details. // // Use `NewDeltaFIFOWithOptions(DeltaFIFOOptions{..., EmitDeltaTypeReplaced: true})` // instead to receive a `Replaced` event depending on the type. // // Deprecated: Equivalent to NewDeltaFIFOWithOptions(DeltaFIFOOptions{KeyFunction: keyFunc, KnownObjects: knownObjects}) func NewDeltaFIFO(keyFunc KeyFunc, knownObjects KeyListerGetter) *DeltaFIFO { return NewDeltaFIFOWithOptions(DeltaFIFOOptions{ KeyFunction:  keyFunc, KnownObjects: knownObjects, }) } </del> // DeltaFIFOOptions is the configuration parameters for DeltaFIFO. All are // optional. type DeltaFIFOOptions struct {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: While reviewing for I noticed that introduced a second type of Delta that is specific to DeltaFIFO but did not update the relevant documentation at , which was introduced by concurrent PR . What you expected to happen: I expected the documentation on DeltaFIFO to be current. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
BTW, I think it would be more reader-friendly if the definitions of Deltas and related types were closer to their mentions near the top of the file rather than near the end.
/help /triage accepted
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help /triage accepted </details
Hi, is someone working on this? If no, can I pick it up? Thanks.
: go ahead, and thanks!
Thanks! /assign


Question: docker build -t in-cluster . If you are not using Minikube, you should build this image and push it to a registry <del> that your Kubernetes cluster can pull from. If you have RBAC enabled, use the following </del> <ins> that your Kubernetes cluster can pull from. If you have RBAC enabled on your cluster, use the following </ins> snippet to create role binding which will grant the default service account view permissions.

What would you like to be : Clarify the example in . Specifically, this paragraph: Why is this needed: When read in the context of the rest of the document, because of the way it is written, it suggests that the following step is unnecessary if you are using Minikube. In fact, it is necessary with the latest version of Minikube, and when this is not done, the command hangs with no output.
/sig docs


Question: It("should be submitted and removed", func() { podClient := c.Pods(api.NamespaceDefault) <del> By("loading the pod json") pod := loadPodOrDie(assetPath("api", "examples", "pod.json")) </del> <ins> By("creating the pod") name := "pod-update-" + string(util.NewUUID()) </ins> value := strconv.Itoa(time.Now().Nanosecond()) <del> pod.Name = pod.Name + "-" + randomSuffix() pod.Labels["time"] = value pod.Spec.Containers[0].Ports[0].HostPort = 0 </del> <ins> pod := &api.Pod{ ObjectMeta: api.ObjectMeta{ Name: name, Labels: map[string]string{ "name": "foo", "time": value, }, }, Spec: api.PodSpec{ Containers: []api.Container{ { Name:  "nginx", Image: "dockerfile/nginx", Ports: []api.Port{{ContainerPort: 80}}, LivenessProbe: &api.Probe{ Handler: api.Handler{ HTTPGet: &api.HTTPGetAction{ Path: "/index.html", Port: util.NewIntOrStringFromInt(8080), }, }, InitialDelaySeconds: 30, }, }, }, }, } </ins> By("submitting the pod to kubernetes") _, err := podClient.Create(pod)

rebase raced with change and missed a line in commit , leaving that test referencing the old file.


Question: assert.True(t, calledFunctionCount > 2, "expect more than two PodContainerManager calls") } <ins> func TestDispatchWorkOfCompletedPod(t *testing.T) { testKubelet := newTestKubelet(t, false /* controllerAttachDetachEnabled */) defer testKubelet.Cleanup() kubelet := testKubelet.kubelet kubelet.podWorkers = &fakePodWorkers{ syncPodFn: func(options syncPodOptions) error { return fmt.Errorf("should ignore completed pod %q", options.pod.Name) }, cache: kubelet.podCache, t:     t, } now := metav1.NewTime(time.Now()) pods := []*v1.Pod{ { ObjectMeta: metav1.ObjectMeta{ UID:         "1", Name:        "completed-pod1", Namespace:   "ns", Annotations: make(map[string]string), }, Status: v1.PodStatus{ Phase: v1.PodFailed, ContainerStatuses: []v1.ContainerStatus{ { State: v1.ContainerState{ Terminated: &v1.ContainerStateTerminated{}, }, }, }, }, }, { ObjectMeta: metav1.ObjectMeta{ UID:         "2", Name:        "completed-pod2", Namespace:   "ns", Annotations: make(map[string]string), }, Status: v1.PodStatus{ Phase: v1.PodSucceeded, ContainerStatuses: []v1.ContainerStatus{ { State: v1.ContainerState{ Terminated: &v1.ContainerStateTerminated{}, }, }, }, }, }, { ObjectMeta: metav1.ObjectMeta{ UID:               "3", Name:              "completed-pod3", Namespace:         "ns", Annotations:       make(map[string]string), DeletionTimestamp: &now, }, Status: v1.PodStatus{ ContainerStatuses: []v1.ContainerStatus{ { State: v1.ContainerState{ Terminated: &v1.ContainerStateTerminated{}, }, }, }, }, }, } for _, pod := range pods { kubelet.dispatchWork(pod, kubetypes.SyncPodSync, nil, time.Now()) } } func TestDispatchWorkOfActivePod(t *testing.T) { testKubelet := newTestKubelet(t, false /* controllerAttachDetachEnabled */) defer testKubelet.Cleanup() kubelet := testKubelet.kubelet var got bool kubelet.podWorkers = &fakePodWorkers{ syncPodFn: func(options syncPodOptions) error { got = true return nil }, cache: kubelet.podCache, t:     t, } pods := []*v1.Pod{ { ObjectMeta: metav1.ObjectMeta{ UID:         "1", Name:        "active-pod1", Namespace:   "ns", Annotations: make(map[string]string), }, Status: v1.PodStatus{ Phase: v1.PodRunning, }, }, { ObjectMeta: metav1.ObjectMeta{ UID:         "2", Name:        "active-pod2", Namespace:   "ns", Annotations: make(map[string]string), }, Status: v1.PodStatus{ Phase: v1.PodFailed, ContainerStatuses: []v1.ContainerStatus{ { State: v1.ContainerState{ Running: &v1.ContainerStateRunning{}, }, }, }, }, }, } for _, pod := range pods { kubelet.dispatchWork(pod, kubetypes.SyncPodSync, nil, time.Now()) if !got { t.Errorf("Should not skip active pod %q", pod.Name) } got = false } } </ins> func TestSyncPodsDeletesWhenSourcesAreReady(t *testing.T) { ready := false

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When pod exceeds we find that, The pod is correctly states with the reason but as you can see above the container state is still reported as What you expected to happen: Container state should be not because pod has exceeded How to reproduce it (as minimally and precisely as possible): Create the pod described above and wait till it exceeds listed . Then get the container status using, Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
I would like to address this :) /assign
thanks.


Question: DryRunStrategy cmdutil.DryRunStrategy // OnPodDeletedOrEvicted is called when a pod is evicted/deleted; for printing progress output <ins> // Deprecated: use OnPodDeletionOrEvictionFinished instead </ins> OnPodDeletedOrEvicted func(pod *corev1.Pod, usingEviction bool) <ins> // OnPodDeletionOrEvictionFinished is called when a pod is eviction/deletetion is failed; for printing progress output OnPodDeletionOrEvictionFinished func(pod *corev1.Pod, usingEviction bool, err error) // OnPodDeletionOrEvictionStarted is called when a pod eviction/deletion is started; for printing progress output OnPodDeletionOrEvictionStarted func(pod *corev1.Pod, usingEviction bool) </ins> } type waitForDeleteParams struct {

kubedrain has "onPodDeletedOrEvicted" callback that can be used in custom controllers. It would be very helpful if we add more callbacks that can give more granular control over monitoring, like "onPodDeletionOrEvicitionStarted" and "onPodDeletionOrEvictionFailed". It would help monitor the draining of nodes on a more granular level. Sometimes while draining, if the pod is stuck during the eviction or deletion of node, there is not much information that custom controllers can get from kubedrain, with the above-mentioned callbacks custom controllers will know which pod is evicting or if it failed during the eviction process.
/sig node
What is the custom controllers in your scenario? Can you elaborate on what you want to achieve with more details. Is this is kubectl improvement, the best sig to involve is sig cli. /sig cli
/remove-sig node please re-apply the sig node if this will end up being a request to sig node
In our case we have in CAPI (). Sometimes while draining, if the pod or volume is stuck during the eviction, there is not much information that we get from kubedrain.Helper. It would be much easier for the controller to report these kinds of issues with more granular callbacks like "onPodDeletionOrEvicitionStarted" and "onPodDeletionOrEvictionFailed" on kubedrain.Helper.
/assign Assigning this to myself, I have discussed this issue in sig-cli community meeting and they have given green light for this improvement.
/triage accepted /priority backlog


Question: info.Refresh(patchedObject, true) <del> visitedUids.Insert(string(metadata.GetUID())) </del> if string(patchBytes) == "{}" && !printObject { count++

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: The dry run output listed all the resources that would be configured: deployments, HPAs, services, and an ingress: This was expected. However, it then listed all resources again (except for the HPAs) as being pruned: The config was then applied with: As expected, no resources were actually pruned. What you expected to happen: should actually display which resources are going to be pruned. How to reproduce it (as minimally and precisely as possible): a YAML file to with it: run + prune: should be listed as both configured and pruned it again with pruning: that the deployment is configured but not pruned, even though the dry run said it would be. Anything else we need to know?: Was trying to write a devops script that gave a warning before any resources would be pruned. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: GKE OS (e.g. from /etc/os-release): (Lubuntu 18.04) Kernel (e.g. ):Install tools: Others:
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I can reproduce it, and i will try to fix it.


Question: }) <del> It("should be able to discover a non-local federated service", func() { </del> <ins> PIt("should be able to discover a non-local federated service", func() { </ins> fedframework.SkipUnlessFederated(f.ClientSet) nsName := f.FederationNamespace.Name

This has surfaced in Ingress and Service e2e tests that manipulate DNS entries. The service controller queries the managed dns entry, thinks that the CNAME doesn't exist and adds a new one. However, clouddns says (through a 409) that the entry already exists. Error message aside, the control flow within the service controller's endpoint_helper prevents a service from being deleted. It's unclear at this point whether there's a race condition. I doubt it given the amount of logging that happens showing that there is no concurrent CNAME entry being created. cc Have you seen this in your refactoring?
Hi I used CoreDNS for most of my testing for service controller refactoring and i did not notice these errors. my local test environment is slightly different than the CI environment. i would do more tests on this issue using CoreDNS and would confirm, if it is an issue also in coredns or just google-clouddns.
If you're not still working on this, please unassign yourself and we'll try to find someone else. Thanks.
This issue was labelled only for sig/multicluster and is thus moved over to kubernetes/federation. If this does not seem to be right, please reopen this and notify us /close


Question: framework.ExpectNoError(err) framework.ExpectEqual(updatedStatus.Status.LoadBalancer, statusToUpdate.Status.LoadBalancer, fmt.Sprintf("updated object expected to have updated loadbalancer status %#v, got %#v", statusToUpdate.Status.LoadBalancer, updatedStatus.Status.LoadBalancer)) <ins> ginkgo.By("get /status") ingResource := schema.GroupVersionResource{Group: "networking.k8s.io", Version: ingVersion, Resource: "ingresses"} gottenStatus, err := f.DynamicClient.Resource(ingResource).Namespace(ns).Get(context.TODO(), createdIngress.Name, metav1.GetOptions{}, "status") framework.ExpectNoError(err) statusUID, _, err := unstructured.NestedFieldCopy(gottenStatus.Object, "metadata", "uid") framework.ExpectNoError(err) framework.ExpectEqual(string(createdIngress.UID), statusUID, fmt.Sprintf("createdIngress.UID: %v expected to match statusUID: %v ", createdIngress.UID, statusUID)) </ins> // Ingress resource delete operations ginkgo.By("deleting") err = ingClient.Delete(context.TODO(), createdIngress.Name, metav1.DeleteOptions{})

[ ] two weeks soak start date : testgrid-link [ ] two weeks soak end date : [ ] test promotion pr : kubernetes/kubernetes#? It was picked up in APISnoop that an untested endpoint was promoted from Beta to GA. PR ingress: promote CRUD API tests for v1 to conformance covered the Ingress life cycle tests, but readNetworkingV1NamespacedIngressStatus was missed. According to this APIsnoop readNetworkingV1NamespacedIngressStatus endpoint is untested. ginkgo.By("get /status") ingResource := schema.GroupVersionResource{Group: "", Version: ingVersion, Resource: "ingresses"} gottenStatus, err := f.DynamicClient.Resource(ingResource).Namespace(ns).Get(context.TODO(), createdIngress.Name, metav1.GetOptions{}, "status") framework.ExpectNoError(err) statusUID, _, err := unstructured.NestedFieldCopy(gottenStatus.Object, "metadata", "uid") framework.ExpectNoError(err) framework.ExpectEqual(string(createdIngress.UID), statusUID, fmt.Sprintf("createdIngress.UID: %v expected to match statusUID: %v ", createdIngress.UID, statusUID)) ''' List endpoints hit by the test: readNetworkingV1NamespacedIngressStatus endpoint If this test get merged, test coverage will go up by 1 point This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
Duplicate of
/reopen Merged /assign Ready for creation of Promotion PR
Reopened this issue. <detailsIn response to : /reopen Merged /assign Ready for creation of Promotion PR Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Code change was a bug fix in a Merged conformance test. The correction should not require Promotion. /close
Closing this issue. <detailsIn response to : Code change was a bug fix in a Merged conformance test. The correction should not require Promotion. /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: var ( vmssNameSeparator = "_" <ins> vmssKey                 = "k8svmssKey" </ins> vmssVirtualMachinesKey  = "k8svmssVirtualMachinesKey" availabilitySetNodesKey = "k8sAvailabilitySetNodesKey" availabilitySetNodesCacheTTL = 15 * time.Minute <ins> vmssTTL                      = 10 * time.Minute </ins> vmssVirtualMachinesTTL       = 10 * time.Minute )

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Recently we upgraded our cluster from 1.14.4 to 1.14.7. Then we started to observe tons of following errors: I guess the cause may be related to this line which was introduced recently: As we have hundreds of services with LoadBalancer type, and we also enable cluster-autoscaler, the nodes list change very often. For each service, there will be one call to GetScaleSet. The quota will be exhausted quickly. When fails, new nodes created by cluster-autoscaler will lose connectivity to load balancer. What you expected to happen: Kubernetes should not hit VMSS throttling threshold. How to reproduce it (as minimally and precisely as possible): a cluster with aks-engine. Use VMSS. 200 services with LoadBalancer type. up VMSS to 50 nodes down VMSS to 60 nodes scaling up and down VMSS Anything else we need to know?: Environment: Kubernetes version (use ): 1.14.7 Cloud provider or hardware configuration: Azure, VMSS OS (e.g: ): Ubuntu Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig cloud-provider /area provider/azure
I create a PR to mitigate the issue.
do we have a rough num about how this PR would mitigate this issue? looks like we need to cherry-pick this PR to old releases. it's better if we have a rough estimation.
The revised version of code calls instead of azure API. The cache refreshes itself every 10 min. I believe this will solve the throttling issue.
I believe we're being affected by this. Apart from getting the - when creating our to use a pre-configured Azure Public IP, the ingress appears to be getting a different IP each time
Thank you all!
The PR is merged. Does your issue remain?

pls cherry-pick that fix to old release, thanks
Cherry picked to release 1.14-1.16
I’ll retry today and let you know Wysłane z iPhone'a
Issue returned in K8S 1.15.4 Warning UpdateLoadBalancerFailed service/nginx-ingress-controller Error updating load balancer with new hosts map[<nodes names]: [EnsureHostInPool(ingress/nginx-ingress-controller): backendPoolID(/subscriptions/<subscription/resourceGroups/<rg/providers/Microsoft.Network/loadBalancers/<rg/backendAddressPools/<rg) - failed to ensure host in pool: "azure - cloud provider rate limited(read) for operation:VMSSGet", EnsureHostInPool(ingress/nginx-ingress-controller):


Question: github.com/quobyte/api # github.com/robfig/cron v1.1.0 => github.com/robfig/cron v1.1.0 github.com/robfig/cron <del> # github.com/rubiojr/go-vhd v0.0.0-20160810183302-0bfd3b39853c => github.com/rubiojr/go-vhd v0.0.0-20160810183302-0bfd3b39853c </del> <ins> # github.com/rubiojr/go-vhd v0.0.0-20200706105327-02e210299021 => github.com/rubiojr/go-vhd v0.0.0-20200706105327-02e210299021 </ins> github.com/rubiojr/go-vhd/vhd # github.com/russross/blackfriday v1.5.2 => github.com/russross/blackfriday v1.5.2 github.com/russross/blackfriday

I ran an OSS license scan of a codebase that has this project as a dependency and came across an interesting hit. This go-vhd dependency specified here: has a snippet of code from StackOverflow which is license under creative commons sharealike. Here's the code in question: I'm not a license expert, so I don't know if there are any real issues/conflicts here or not, but wanted to share to see what you all think. Here's SO's license info for reference: My understanding is that the potential issue would be with this clause: which could potentially require go-vhd and projects that depend on it to use the Creative Commons license as well.
Thanks for the report! can you please take a look if this is concerning? If so, we'll check if we can avoid this dependency (looks unlikely unless we duplicate what it's ) or we'll follow up upstream to update the code. /cc OWNERS for this repo
CC-BY-SA can be problematic in Apache-2.0 projects like k8s because of the ShareAlike aspect as noted. I'd recommend working with upstream to rewrite just that one function particularly since it looks like it's fairly limited.
let's please move this issue to kubernetes/kubernetes?
/provider azure
/sig release /area code-organization
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig release /area code-organization Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
cc /assign
/area code-organization
/assign
/milestone v1.19 /priority important-soon
Not a huge difference, but this is likely actually licensed under , not 4.0. See
the k/k code actually does not reference , since that function is in , do you mean we should remove from k/k totally?
Yes, we would either need to remove from k/k or ask the maintainers of to update . This means that (and k/k if we end up using it) would need to be licensed until CC-BY-SA.
Would this ultimately be backported to older versions of k8s or would you just do it in 1.19?

thanks, Let me remove the code in v1.19, actually that part of functionality is deprecated in Azure cloud provider already. My initial thought is only do this in v1.19
/kind bug /priority important-soon /sig cloud-provider /area provider/azure
Since it isn't trivial to remove as a dependency entirely (), I've created upstream to track updating the code there.


Question: tags = ["automanaged"], deps = [ "//pkg/probe:go_default_library", <ins> "//pkg/version:go_default_library", </ins> "//vendor/github.com/golang/glog:go_default_library", "//vendor/k8s.io/apimachinery/pkg/util/net:go_default_library", ],

Grabbed tcpdump of a readiness probe - the src ip was the bridge address (10.244.1.1), but apart from this, there is no identifying information in the HTTP request that this is a Kubernetes daemon requesting readiness checks - this might help QoS/priority assignment by the container application.
IIUC, you mean we should add a better when kubernetes daemon request a readiness check to identify this request comes from kubernetes daemon?
Yes, we should override the default net/http user-agent with our own. Better usability for users and we have no future upgrade hazards if users start hardcoding and expecting this default User-Agent string in their analytic tools.
Any news about this one? It would be nice! I've this little hack in my nginx configuration (in order to not be spamed by health check in logs) :)
This can probably be closed now that probes have support.


Question: import ( "fmt" <del> "reflect" </del> "time" apps "k8s.io/api/apps/v1beta1" "k8s.io/api/core/v1" "k8s.io/api/extensions/v1beta1" policy "k8s.io/api/policy/v1beta1" <ins> apiequality "k8s.io/apimachinery/pkg/api/equality" </ins> "k8s.io/apimachinery/pkg/api/errors" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/types"

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: There are too many pdb update operations compared with other resources. ! That is, there are many "Put" operations for pdb periodically in kube-apiserver log. In my view, the amount of pdb operations is connected with disruption controller's resyncPeriod and pdb's quantity in cluster. For example: disruption-controlle's default resyncPeriod is 30s, and there are 100 pdbs in cluster. We can find that the update operation times per minute is about 200. The data is reasonable, explanation is as follows: According to disruption-controlle's design, there is one resync every 30s. One resync means to call updatePdbStatus function for every pdb, so there are 200 times updatePdbStatus. Function updatePdbStatus judges whether the pdb is changed. If changed, call apiserver to update the pdb' status. When no eviction happens, disruptedPods is empty. In this situation, disruptedPods is an empty map, but pdb.Status.DisruptedPods is nil, empty map is not equal to nil. So reflect.DeepEqual(pdb.Status.DisruptedPods, disruptedPods) returns false. The pdb is thought to has be changed, then apiserver is called to update the pdb' status. In fact, there is no change for the pdb when disruptedPods is an empty map and pdb.Status.DisruptedPods is nil. What you expected to happen: There shouldn't be so many pdb update operations. That is, when pdb has no change, apiserver shouldn't be called to update its status. How to reproduce it (as minimally and precisely as possible): Create some pdbs in cluster, you will see many "Put" operations for pdb periodically in kube-apiserver log. Anything else we need to know?: Environment: kubernetes v1.13.2 Fix suggestions: The bug is caused by the mixed usage of empty and nil for disruptedPods, the type of which is map[string]metav1.Time.

There are two fix suggestions for it: one of empty map and nil for disruptedPods, use it consistently and refuse to mix empty map and nil. For example, when disruptedPods is empty or nil, save it as an empty map in etcd. Then every client gets pdb from etcd, the pdb.Status.DisruptedPods is an empty map. get pdb for etcd, assign an empty map to it if pdb.Status.DisruptedPods is nil. Maybe the mixed usage of empty and nil for map is a common problem in k8s design, not just for disruptedPods. I will propose a work around solution for the problem. If the solution has any problem, please let me know and I can modify the code.
/sig apps
/assign Morten can you PTAL and the open PR. I reviewed as well.
ref


Question: return nil }, }, <ins> { name: "scheduled pod uses read-write-once-pod pvc", init: func(cs kubernetes.Interface, ns string) error { _, err := createNode(cs, st.MakeNode().Name("node").Obj()) if err != nil { return fmt.Errorf("cannot create node: %v", err) } storage := v1.ResourceRequirements{Requests: v1.ResourceList{v1.ResourceStorage: resource.MustParse("1Mi")}} volType := v1.HostPathDirectoryOrCreate pv, err := testutils.CreatePV(cs, st.MakePersistentVolume(). Name("pv-with-read-write-once-pod"). AccessModes([]v1.PersistentVolumeAccessMode{v1.ReadWriteOncePod}). Capacity(storage.Requests). HostPathVolumeSource(&v1.HostPathVolumeSource{Path: "/mnt", Type: &volType}). Obj()) if err != nil { return fmt.Errorf("cannot create pv: %v", err) } pvc, err := testutils.CreatePVC(cs, st.MakePersistentVolumeClaim(). Name("pvc-with-read-write-once-pod"). Namespace(ns). // Annotation and volume name required for PVC to be considered bound. Annotation(volume.AnnBindCompleted, "true"). VolumeName(pv.Name). AccessModes([]v1.PersistentVolumeAccessMode{v1.ReadWriteOncePod}). Resources(storage). Obj()) if err != nil { return fmt.Errorf("cannot create pvc: %v", err) } pod := initPausePod(&testutils.PausePodConfig{ Name:      "pod-to-be-deleted", Namespace: ns, Volumes: []v1.Volume{{ Name: "volume", VolumeSource: v1.VolumeSource{ PersistentVolumeClaim: &v1.PersistentVolumeClaimVolumeSource{ ClaimName: pvc.Name, }, }, }}, }) if _, err := createPausePod(cs, pod); err != nil { return fmt.Errorf("cannot create pod: %v", err) } return nil }, pod: &testutils.PausePodConfig{ Name: "pod-to-take-over-pvc", Volumes: []v1.Volume{{ Name: "volume", VolumeSource: v1.VolumeSource{ PersistentVolumeClaim: &v1.PersistentVolumeClaimVolumeSource{ ClaimName: "pvc-with-read-write-once-pod", }, }, }}, }, update: func(cs kubernetes.Interface, ns string) error { return deletePod(cs, "pod-to-be-deleted", ns) }, enableReadWriteOncePod: true, }, </ins> } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { <ins> defer featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, features.ReadWriteOncePod, tt.enableReadWriteOncePod)() </ins> testCtx := initTest(t, "scheduler-informer") defer testutils.CleanupTest(t, testCtx)

We need an integration test for the scheduler enforcement of the alpha feature. This is required for beta graduation; we need to collect signal on the current behavior before we can proceed. See this comment for context:
/sig storage /sig scheduler
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /sig storage /sig scheduler Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scheduling


Question: " No preemption is performed.") } else { preemptionStartTime := time.Now() <del> sched.preempt(pod, fitError) </del> <ins> sched.preempt(fwk, pod, fitError) </ins> metrics.PreemptionAttempts.Inc() metrics.SchedulingAlgorithmPremptionEvaluationDuration.Observe(metrics.SinceInSeconds(preemptionStartTime)) metrics.DeprecatedSchedulingAlgorithmPremptionEvaluationDuration.Observe(metrics.SinceInMicroseconds(preemptionStartTime))

<!-- Please only use this template for submitting enhancement requests --What would you like to be : When preempting a pod that is a that permit plugins are waiting for , we need send a message(actually send a ) to the , otherwise permit plugins might wait until time out, and do not know the pod has been preempted. Why is this needed: /sig scheduling /kind cleanup /assign
cc
Interesting interaction between the permit and preempt, I will take a look at the PR you sent, thanks!


Question: By("deleting host1Pod") framework.ExpectNoError(podClient.Delete(host1Pod.Name, api.NewDeleteOptions(0)), "Failed to delete host1Pod") <ins> By("Test completed successfully, waiting for PD to safely detach") waitForPDDetach(diskName, host0Name) waitForPDDetach(diskName, host1Name) </ins> return })

PD E2E tests have very aggressive clean up routine that does not wait for volumes to safely detached. After the test completes, the test cleanup routine unilaterally detaches the PD (instead of waiting for k8s to do it cleanly). Although this does not cause the test to fail, it results in errors in the kernel logs: We should modify these tests to wait for k8s to safely detach the disk.


Question: - source_hash: {{ etcd_tar_hash }} - archive_format: tar - if_missing: /usr/local/src/etcd-{{ etcd_version }}-linux-amd64 <ins> {% if grains['saltversioninfo'] <= (2014, 7, 0, 0) %} </ins> - tar_options: xz <ins> {% endif %} </ins> file.directory: - name: /usr/local/src/etcd-{{ etcd_version }}-linux-amd64 - user: root

I now see this in /var/log/salt/minion I think this is the problem: (They just pushed a new version of SaltStack, 2014.7.1 which includes the issue) I think it's an easy fix; we just change the options from "xz" to "z". PR on its way...
Ugh... even after changing xz -z, that exposes another bug where salt messed up quoting: If we don't pass tar_options at all, there is yet another code path, which isn't broken but uses the built-in Python tarfile library to extract the tar file. I think that's our only option, unless we want to revert to the old saltstack until the new one becomes stable.
Appears fixed.


Question: for _, ipInfo := range apiPodStatus.PodIPs { podStatus.IPs = append(podStatus.IPs, ipInfo.IP) } <ins> if len(podStatus.IPs) == 0 && len(apiPodStatus.PodIP) > 0 { podStatus.IPs = []string{apiPodStatus.PodIP} } </ins> // Record the time it takes for the pod to become running. existingStatus, ok := kl.statusManager.GetPodStatus(pod.UID) if !ok || existingStatus.Phase == v1.PodPending && apiPodStatus.Phase == v1.PodRunning &&

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: is unset on a networking daemonset pod used by (read via the downward api). Previously this was reliably set to match the hostIP. What you expected to happen: This should be the hostIP (which is set). How to reproduce it (as minimally and precisely as possible): We know the kind CI tests cover this ,%20master%20(dev)%20%5Bnon-serial%5D&show-stale-tests=&width=5 Anything else we need to know?: This is also tracked in , we observed this change in Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig network /cc /priority important-soon
/assign


Question: label := labels.SelectorFromSet(labels.Set(map[string]string{"name": name})) pods, err := podsCreated(f.Client, f.Namespace.Name, name, replicas) <ins> Expect(err).NotTo(HaveOccurred()) </ins> By("Ensuring each pod is running")

This might be fixed, haven't reproduced on latest upstream source. But in any case, I thought I'd create the issue. bug = i assume any failure in e2es shouldnt be b/c of a panic. ... I noticed a failure on Openshift today, which I think alludes to a possible issue in replica sets.
cc
cc
Yup, looks like a bug, and it looks like it's still there. The isn't checked, so can be nil, which will cause the panic on L102. cc
i think the iterators should be declarative here i.e. ? thats a better fix .... (update : nvm, i think its fine as is, just add a nil check: Because, the declarative won't work for the entire test, since the function for checking pod name responses requires a full )
Cc
Thanks, Jay. Even with this change, the test would have failed, so we should still debug the cause.
It seems we don't check err here: if return an error, will be nil.
Yup.... that is fixed in the PR, right?


Question: qs.obsPair.RequestsWaiting.Add(1) } <del> // dispatchAsMuchAsPossibleLocked runs a loop, as long as there // are non-empty queues and the number currently executing is less than the // assured concurrency value.  The body of the loop uses the fair queuing // technique to pick a queue, dequeue the request at the head of that // queue, increment the count of the number executing, and send true // to the request's channel. </del> <ins> // dispatchAsMuchAsPossibleLocked does as many dispatches as possible now. </ins> func (qs *queueSet) dispatchAsMuchAsPossibleLocked() { <del> for qs.totRequestsWaiting != 0 && qs.totSeatsInUse < qs.dCfg.ConcurrencyLimit { ok := qs.dispatchLocked() if !ok { break } </del> <ins> for qs.totRequestsWaiting != 0 && qs.totSeatsInUse < qs.dCfg.ConcurrencyLimit && qs.dispatchLocked() { </ins> } }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via -- Kubernetes version (use ):Cloud provider or hardware configuration: OS (e.g: ):Kernel (e.g. ):Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig api-machinery
/triage accepted
adds a defensive check so we can avoid a panic. I was looking into why can be . This is the call chain, from the stack trace: makes sure there is at least one request waiting - acquires a lock on the queueset in the beginning. So should have at least one request waiting in a queue and that queue should be selected. could this be an issue with the locking somewhere, any thoughts
/cc
is the client version, not the server. In that commit, line 725 of is a comment.
check the first link at the top of the issue description -
Something is violating locking or counting discipline. Perhaps some cheap sanity checks on counts (like, never negative) would at least confirm that this is where the problem lies. With locking or counting being done wrong, a defense in one place will just make symptoms appear elsewhere. We need to find the problem. The needed discipline is very mechanical. A careful read of all the code should find any violations.
Also, we could make queueSet::UpdateObservations check that the queueSet has the right sums over its queues (under the queueSet lock, of course). I think it is called infrequently enough for that to be tolerable. The point would be to reveal something to us developers. We could even make it do repair if an error is found; this would not really solve the problem, of course, but might make a helpful reduction in the severity of the symptoms.
Also, some comments are outdated, incomplete, or missing. Some comments on the fields in reference a now-non-existent field . .. reads a bit odd, there is no "array" in sight. has no comment. The presence of "without ..." suggests that it is adding information, but as far as I can tell it is not. I would feel more comfortable with an explicit statement about concurrency and that returned .

Perhaps something like updating the comment on to say something like the following. .. should also describe the significance of the parameters. Since it is the same as for , a reference to that would suffice. Checking that locking discipline is followed would be easier if each non-Locked method of queueSet had a comment that explicitly says whether the method is lock-oblivious or an entry method. This would make it a local check to avoid the case of an entry method calling another entry method (which would cause a deadlock, not the symptom in this Issue). The function comment on uses the wrong capitalization for the function name. s/modify/modifying/
.. continued issues with comments. s/returns/returned/ The fields of should be re-ordered to start with immutable ones then have the mutable ones, and have a comment about locking. would be better named . The values of type should have comments defining their meaning. Note that both and can involve a timeout, and the comments/log-messages in say simply "time out", but they are different timeouts --- one is the timeout defined in APF and the other is the timeout/cancel from the request's context. should be named . Why are the and methods of public? The method of should be named .
and do not take into account the possibility that the decision was already set to . This obliviousness can lead to confusion about what is in and mis-counting. I.e., cause the symptom in this Issue.
I will propose a fix.
Thanks for your investigation. I still have some confusions need your clarification. the above issue occur under what conditions? As far as I know, will be set to just at the stage of for timeout, and this occurs at the , but will set decision prior to this and decision can only be set once. the can only be set once, is this parameter a bit redundant? two identical requests arrive at the same apiserver almost at the same time, what will happen and how to deal with it?
given particular request X might not get dispatched during its ; request X might have to wait a while if there are not enough seats available at the moment. The goroutine that calls for X then proceeds to call , which calls --- and that calls the method of X's . If X's context times out or is cancelled before X is dispatched or rejected by APF logic then X's gets set to .

hope my answer to 1. also answers this. am not sure I understand what potential problem you are thinking about here. Each request is a distinct , even if the client and request message are identical. In turn, each request gets a distinct .
Thank you for your clarification. Aout question 2, I mean the has been set to , the setting of will no longer work. maybe nil is better.
The attempt to set a request's decision to is made at dispatch time. This might happen after the request's context has been canceled or timed out and the method has thus already set the decision to .


Question: return ioutil.NopCloser(bytes.NewReader([]byte(body))) } <ins> // TODO(jlowdermilk): refactor the Factory so we can test client versions properly, // with different client/server version skew scenarios. </ins> // Verify that resource.RESTClients constructed from a factory respect mapping.APIVersion <del> func TestClientVersions(t *testing.T) { f := cmdutil.NewFactory(nil) version := testapi.Version() mapping := &meta.RESTMapping{ APIVersion: version, } c, err := f.RESTClient(mapping) if err != nil { t.Errorf("unexpected error: %v", err) } client := c.(*client.RESTClient) if client.APIVersion() != version { t.Errorf("unexpected Client APIVersion: %s %v", client.APIVersion, client) } } </del> <ins> //func TestClientVersions(t *testing.T) { //	f := cmdutil.NewFactory(nil) // //	version := testapi.Version() //	mapping := &meta.RESTMapping{ //		APIVersion: version, //	} //	c, err := f.RESTClient(mapping) //	if err != nil { //		t.Errorf("unexpected error: %v", err) //	} //	client := c.(*client.RESTClient) //	if client.APIVersion() != version { //		t.Errorf("unexpected Client APIVersion: %s %v", client.APIVersion, client) //	} //} </ins> func ExamplePrintReplicationController() { f, tf, codec := NewAPIFactory()

This seems like a bug to me No output, and command exits success... wat. Have to explicitly pass to see expected output. If the client version is greater the server version, it should be making requests based on the server's newest accepted apiversion.
cc


Question: t.Errorf("Expected %v, but got %v", e, a) } <ins> // Test a scheme + name + port (using second ip). location, _, err = redirector.ResourceLocation(ctx, "https:foo-second-ip:p") if err != nil { t.Errorf("Unexpected error: %v", err) } if location == nil { t.Errorf("Unexpected nil: %v", location) } if e, a := "https://[2001:db7::]:93", location.String(); e != a { t.Errorf("Expected %v, but got %v", e, a) } </ins> // Test a non-existent name + port. location, _, err = redirector.ResourceLocation(ctx, "foo:q") if err == nil { t.Errorf("Unexpected nil error") } <ins> // Test a non-existent name + port (using second ip). location, _, err = redirector.ResourceLocation(ctx, "foo-second-ip:q") if err == nil { t.Errorf("Unexpected nil error") } </ins> // Test error path if _, _, err = redirector.ResourceLocation(ctx, "bar"); err == nil { t.Errorf("unexpected nil error")

Hi, I have a issue with a dualstack cluster for my Kubernetes Dashboard deployment. I opened a ticket here (), but I am informed that it is not related to Kubernetes Dashboard. What is the problem? Errors in kube-api: The service is created with the correct ipFamily (=IPv6). My endpoint is fully in IPv6. Indeed, my pod has one IP in IPv4 and another in IPv6. Do you have also the issue? Kubernetes version: 1.18 OS: RHEL7
/sig network /sig api-machinery
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
I think the following should fix the error you saw:
/remove-sig api-machinery


Question: errs = append(errs, "must contain only alpha-numeric characters (a-z, 0-9), and hyphens (-)") } if !portNameOneLetterRegexp.MatchString(port) { <del> errs = append(errs, "must contain at least one letter or number (a-z, 0-9)") </del> <ins> errs = append(errs, "must contain at least one letter (a-z)") </ins> } if strings.Contains(port, "--") { errs = append(errs, "must not contain consecutive hyphens")

The only support at least one letterbut the error message is at least one letter or number. It will be error if portname only numbers, ex: '1-2' or '1' or '123'. Fix the error message or regex syntax No response at least 1.19+ None <details</details<details</details<details</details<details</details
/sig api-machinery
I would like to take up this issue. /assign
/triage accepted
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/remove-lifecycle rotten


Question: } } <ins> // Install the cni plugin. Note that /opt/cni does not get cleaned up after // the test completes. if _, err := RunSshCommand("ssh", GetHostnameOrIp(host), "--", "sh", "-c", getSshCommand(" ; ", "sudo mkdir -p /opt/cni", fmt.Sprintf("sudo wget -O - %s | sudo tar -xz -C /opt/cni", CNI_URL))); err != nil { // Exit failure with the error return "", false, err } </ins> // Create the temp staging directory glog.Infof("Staging test binaries on %s", host) tmp := fmt.Sprintf("/tmp/gcloud-e2e-%d", rand.Int31())

Doesn't seem to haven been caused by my change:
/cc
The test only failed on coreos, and the is full of error messages. Plus the log is almost unreadable because of the extremely high verbosity. I filed for that. shows that the node has been rebooted a couple of times. And the kernel log shows This looks like . The hairpin mode is set to hairpin-veth. /cc
Yeah don't run with hairpin-mode I guess. is coreos using kubenet?
Node problem detector should detect this issue and report KernelDeadLock condition.
Yeah, but this is node e2e test, we don't have node problem detector running on node e2e cluster.
Not sure how to assign coreos issues... Assigning to people to reassign based on thread activity. Please reassign correctly!
ping!
/cc
This happens in node e2e test right? I believe node e2e test is NOT using kubenet.
Running node e2e with kubenet+hairpin-veth could prevent this.
Sorry, this is lost in github notifications.. Feel free to ping me on slack if I am not responding in a day next time :)
To run kubenet on a node, the framework needs to download and unpack the cni plugin in kubelet needs to run with and a fixed pod CIDR (since there is no node controller) I did those in , but it failed on the trusty images because is not installed. We need to either install this in the node e2e framework, or bake it in to the image. what's the best practice for installing packages on the images?
I believe updating is the best way to bake it into the image, and I also think that should be preferred over installing it each time. You could do the same with CNI. Thanks for working on this!
I updated the PR to install nsenter on the trusty images. As for cni, I still keep it in the node e2e framework to install it every time because 1) it only downloads from GCS and untar a file of ~14MB, 2) the tarball is maintained and updated by It'd be an extra burden to update the images just for cni updates.
OK with the plan.

Seeing the error message again after switching to kubenet+hairpin-veth do you want to take a look at the logs?
let me take a look
urrrh... we hit it again. Switching back to
There's a difference in the above kernel log; the device is not a interface but rather an interface (presumably the one within a container's net ns since there is no eth0 on the host). I think all the prior instances of this were veths on the host.
True. This one is slightly different. I think deadlock on interface also showed up before.
Haven't seen any new occurrence since landed. We'll know in a day or so whether this problem has been fixed.
AFAIK, there are no new failures after landed. It should be safe to let this issue stay closed.
unfortunately I hit the bug with hairpinmode sets to promiscuous-bridge kubelet v1.3.0, linux kernel 4.6.3 cbr0 has already enter the promiscuous mode, all veth hairpinmode = 1 start with parameter
can you ? what distro are you running?
also are you using a network-plugin, overlay (like flannel), which cloudprovider?
Are the SELinux warnings normal ?
Maybe a little out of topic. Are you running K8s on GCE? Do you have the node problem detector running in your cluster?
all veths hairpin_mode = 1 my distro is coreos 1068.6.0 I don't use any network-plugins, and no cloudprovider support here, controller-manager start with kubelet setup after address allocated by controller-manager, all pods connected each other with static routes which controlled by ospfd running inside docker. I can confirm the selinux warnings not related to the bug.
not running on GCE. no detector running inside.
Alright, just want to verify whether the NPD works as expected.
here is the logs of kubelet startup.
That's the problem, the reason we ran into this to begin with is because of hairpin mode. Apparently it is still getting set, or isn't getting erased in your environment. I don't think we erase existing values.
that's wired.

I checked all my nodes, each veth hairpin_mode is 1. it seems it was set to 1 regardless of --configure-cbr0.
there is no existing veth before kubelet create bridge , docker could not start without cbr0 which created by kubelet.
hairpin_mode is always enabled if the device is a bridge interface,
Trust me it's not on my (and our other e2e test) clusters, or we'd know because tests would fail, and teh default setup is docker running on cbr0 and using veths. In 1.3 we started using kubenet network plugin by default, but hairpin flag wasn't set since 1.2. I can trace the code path down when I have some time, but I suspect it's something to do with the way coreos is setup.
I trust you, I promise. I did some tests. None of results told me the flag is set by kubelet. I could not figure out what was happend to these veths. finally, I duplicate to , the only difference is hairepinEnable default to "0", and invoke the hairpinunset.SetUpContainerPid when false. It works! But, if I create veth by hand, the default hairpin_mode is 0 , oh my god. It will not happen in my test environment, only in production env
will enable hairpin_mode That's my fault, I should not set it to false because kubernetes do not use docker's network model.


Question: "type": "array" }, "operations": { <del> "description": "Operations is the operations the admission hook cares about - CREATE, UPDATE, or * for all operations. If '*' is present, the length of the slice must be one. Required.", </del> <ins> "description": "Operations is the operations the admission hook cares about - CREATE, UPDATE, DELETE, CONNECT or * for all of those operations and any future admission operations that are added. If '*' is present, the length of the slice must be one. Required.", </ins> "items": { "type": "string" },

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Working on other doc bugs, I went looking for what calls admission plugins. I found four call sites for validating plugins, documented in . They correspond to the verbs CONNECT, CREATE, DELETE, and UPDATE. But only two of those verbs are documented at What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
Looks like docs are correct in and
The API doc can be updated to call out all four operations and clarify that "all" means "all four admission operations"
/good-first-issue
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /good-first-issue </details
/assign


Question: <ins> # Copyright 2016 The Kubernetes Authors. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # #     http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # kube-startup.ps1 is used to run kubelet and kubeproxy as a process. It uses nssm (https://nssm.cc/) process manager to register kubelet and kube-proxy process, # The processes can be viewed using TaskManager(Taskmgr.exe). # Please note that this startup script does not start the API server. Kubernetes control plane currently runs on Linux # and only Kubelet and Kube-Proxy can be run on Windows param ( [Parameter(Mandatory=$true)][string]$ContainerNetwork, [string]$InterfaceForServiceIP = "vEthernet (HNS Internal NIC)", [string]$LogDirectory = "C:temp", [Parameter(Mandatory=$true)][string]$Hostname, [Parameter(Mandatory=$true)][string]$APIServer, [string]$InfraContainerImage = "apprenda/pause", [string]$ClusterDNS = "10.0.0.10", [string]$KubeletExePath = ".kubelet.exe", [string]$KubeProxyExePath = ".kube-proxy.exe" ) $kubeletDirectory = (Get-Item $KubeletExePath).Directory.FullName $kubeproxyDirectory = (Get-Item $KubeProxyExePath).Directory.FullName # Assemble the Kubelet executable arguments $kubeletArgs = @("--hostname-override=$Hostname","--pod-infra-container-image=$InfraContainerImage","--resolv-conf=""""","--api-servers=$APIServer","--cluster-dns=$ClusterDNS") # Assemble the kube-proxy executable arguments $kubeproxyArgs = @("--hostname-override=$Hostname","--proxy-mode=userspace","--bind-address=$Hostname","--master=$APIServer") # Setup kubelet service nssm install kubelet "C:WindowsSystem32WindowsPowerShellv1.0powershell.exe" nssm set kubelet Application "$KubeletExePath" nssm set kubelet AppDirectory "$kubeletDirectory" nssm set kubelet AppParameters $kubeletArgs nssm set kubelet DisplayName kubelet nssm set kubelet Description kubelet nssm set kubelet Start SERVICE_AUTO_START nssm set kubelet ObjectName LocalSystem nssm set kubelet Type SERVICE_WIN32_OWN_PROCESS # Delay restart if application runs for less than 1500 ms nssm set kubelet AppThrottle 1500 nssm set kubelet AppStdout "$LogDirectorykubelet.log" nssm set kubelet AppStderr "$LogDirectorykubelet.err.log" nssm set kubelet AppStdoutCreationDisposition 4 nssm set kubelet AppStderrCreationDisposition 4 nssm set kubelet AppRotateFiles 1 nssm set kubelet AppRotateOnline 1 # Rotate Logs Every 24 hours or 1 gb nssm set kubelet AppRotateSeconds 86400 nssm set kubelet AppRotateBytes 1073741824 nssm set kubelet AppEnvironmentExtra CONTAINER_NETWORK=$ContainerNetwork # Setup kube-proxy service nssm install kube-proxy "C:WindowsSystem32WindowsPowerShellv1.0powershell.exe" nssm set kube-proxy Application "$KubeProxyExePath" nssm set kube-proxy AppDirectory "$kubeproxyDirectory" nssm set kube-proxy AppParameters $kubeproxyArgs nssm set kube-proxy DisplayName kube-proxy nssm set kube-proxy Description kube-proxy nssm set kube-proxy Start SERVICE_AUTO_START nssm set kube-proxy ObjectName LocalSystem nssm set kube-proxy Type SERVICE_WIN32_OWN_PROCESS # Delay restart if application runs for less than 1500 ms nssm set kube-proxy AppThrottle 1500 nssm set kube-proxy AppStdout "$LogDirectorykube-proxy.log" nssm set kube-proxy AppStderr "$LogDirectorykube-proxy.err.log" nssm set kube-proxy AppStdoutCreationDisposition 4 nssm set kube-proxy AppStderrCreationDisposition 4 nssm set kube-proxy AppRotateFiles 1 nssm set kube-proxy AppRotateOnline 1 # Rotate Logs Every 24 hours or 1 gb nssm set kube-proxy AppRotateSeconds 86400 nssm set kube-proxy AppRotateBytes 1073741824 nssm set kube-proxy AppEnvironmentExtra INTERFACE_TO_ADD_SERVICE_IP=$InterfaceForServiceIP # Start kubelet and kube-proxy Services echo "Starting kubelet" Start-Service kubelet echo "Starting kube-proxy" Start-Service kube-proxy </ins>

Currently kubelet and proxy for Windows is an and we manually set up the environment variables and run the . Create a startup script which, Sets up Environment variable by accepting user input and have appropriate default values Runs validation Run kubelet and proxy in the background
Changes are merged


Question: framework.ExpectNoError(err) for _, node := range nodeList.Items { <del> framework.Logf("nLogging pods the kubelet thinks is on node %v before test", node.Name) printAllKubeletPods(cs, node.Name) </del> <ins> framework.Logf("nLogging pods the apiserver thinks is on node %v before test", node.Name) printAllPodsOnNode(cs, node.Name) </ins> } })

What happened: In reviewing a test for promotion to conformance I noticed the kubelet API was being accessed via the API server's node proxy endpoint (ref: ). This seemed off, so I reviewed . It doesn't explicitly call out kubelet API access as allowed or disallowed. . and agreed kubelet API access seems like a bad idea. I'm now surveying what we have that does_ assume kubelet API access. It's definitely used by parts of e2e framework and some Conformance tests. I'm trying to verify how critical its use is. It appears that much of it is related to debugging or verifying whether kubelets and the control plane agree on who is running what. What you expected to happen: Find a blanket policy that bans kubelet API access for Conformance tests, whether directly, or indirectly through the API server's node proxy endpoint, at any time. Alternatively, find a policy that says kubelet API access may be allowed, but only for debugging or failure purposes (must not be on the critical path to make the test pass) Alternatively, find a policy that says kubelet API access is allowed, but only indirectly through the API server, and evaluated against the same constraints we use for control plane access (GA, non-optional) If we want to say kubelet API access is banned, we can use this as a tracking issue to demote whichever tests rely on it, and come up with a plan to verify the same behavior without kubelet API access. /area conformance /sig architecture /priority important-soon /milestone v1.16 /assign
Look for anything that ends up calling calls this in BeforeEach, which will happen before some Conformance tests (if this fails, the tests will fail) Conformance test requires kubelet API access to pass may end up calling it if the test fails, via: Some Conformance tests may end up calling this if the test fails specifically Other areas this is used: only if the test fails required for a pre-stop hook test to pass required to make sure the readonly kubelet apis aren't exposed via api server proxy resource usage tracking tests require kubelet api access to pass tests call this in
Looking for doesn't turn up any additional conformance tests, and I don't feel compelled to link the two other tests that use it as they are unlikely to ever be eligible.

The examples pointed out by on slack, the kubelet's endpoints is required to be available for these to pass
Hello! I'm the bug triage lead for the 1.16 release cycle and considering this issue is tagged for 1.16, I'd like to check its status. The code freeze is starting on August 29th EOD PST (about 5 days from now) and once the code freeze starts, only release-blocking changes will be considered to be merged. Is this issue still planned to be fixed for the 1.16 release?
are there further changes for this targeted at 1.16?
Any update on this?
/milestone v1.17
Bug Triage Release Team checking in. Code freeze for the 1.17 release is on November 14. Will this make it in? Thanks!
/assign
hi is this still targetting for 1.17 or can we move to 1.18? Code freeze is Nov 14th. thanks!
/milestone v1.18
So we've updated the documented policy. My suggestion to close this out is to drop the [Conformance] tag from the tests that currently require the kubelet API to pass, and open issues that they need to be rewritten. Ignore whether the API is needed for diagnostic info in failure modes. SchedulerPredicates [Serial] calls this in BeforeEach, which will happen before some Conformance tests (if this fails, the tests will fail) Conformance test Pods should be submitted and removed [NodeConformance] requires kubelet API access to pass
i'd support this ^^
Hello , Bug Triage team here for the 1.18 release. This is a friendly reminder that code freeze is scheduled for 5 March. Is this issue still intended for milestone 1.18?
Hello There's been no movement on this issue this year. Should this get pushed to the next milestone?
yes, please John T Skarbek <:
/milestone v1.19
/cc


Question: <ins> #!/bin/bash # Copyright 2014 Google Inc. All rights reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # #     http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # Bring up a Kubernetes cluster. # # If the full release name (gs://<bucket>/<release>) is passed in then we take # that directly.  If not then we assume we are doing development stuff and take # the defaults in the release config. set -o errexit set -o nounset set -o pipefail KUBE_ROOT=$(dirname "${BASH_SOURCE}")/.. KUBE_CONFIG_FILE="config-default.sh" ${KUBE_ROOT}/hack/ginkgo-e2e.sh --ginkgo.focus=Networking </ins>  No newline at end of file

Compare requestdurationseconds, requestterminationstotal, requestabortstotal API server metrics between the two runs. The acceptable delta should be less than 20%. xref:
/sig auth /triage accepted
/assign ritazh /milestone v1.29


Question: "github.com/stretchr/testify/assert" v1 "k8s.io/api/core/v1" <ins> metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/utils/ptr" </ins> runtimeapi "k8s.io/cri-api/pkg/apis/runtime/v1" _ "k8s.io/kubernetes/pkg/apis/core/install"

When the node network is not yet connected, start kubelet and create static pod like this The created container file does not contain the configuration of hostAliases. The created container file should contain the configuration of hostAliases. kubelet service. static pod with configuration of hostAliases. network, like . kubelet service. created container file does not contain the configuration of hostAliases. No response <details</details<details</details<details</details<detailskubeadm </details<detailscontainerd 1.6.4 </details<details</details
/sig node
23 is EOL please check if this is happening in a supported release, latest supported is 1.25
You might want to set up a dependency on network from kubelet (assuming you're using systemd), so that it doesn't start until network is ready. Also, could you reproduce this on a later version (eg. 1.27)? /triage needs-information
The kubelet systemd configuration file already declares a network-online target dependency, but it doesn't seem to work. Looks like additional dependencies are required such as NetworkManager-wait-online.service. I think the best way is that kubelet is compatible with this scenario. Also, I also tested on 1.27 and the problem is still reproducible.
Is there any progress in dealing with this issue?
I haven't been working on this, not sure if anyone is actively looking into it.
did you try adding this dependency? Did it work?
NetworkManager-wait-online.service only waits for 60s. If the network is still abnormal after 60s, the problem will still occur.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/remove-lifecycle rotten
The Kubernetes project currently lacks enough contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue with Mark this issue as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close not-planned [1]:
Closing this issue, marking it as "Not Planned". <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue with Mark this issue as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen /remove-lifecycle rotten
Reopened this issue. <detailsIn response to : /reopen /remove-lifecycle rotten Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig network The kubelet wait for the PodIPs to be present to mount the etc hosts files however, it does not check if the pod is host network. If the Pod is using hostNetwork, then it seems the PodIPs are ignored so it seems legit to mount the aliases independently of PodIPs if the pod is using host network /assign /sig network /triage accepted
/remove-sig node


Question: ## Code reviews <del> All changes must be code reviewed.  For non-maintainers this is obvious, since you can't commit anyway.  But even for maintainers, we want all changes to get at least one review, preferably from someone who knows the areas the change touches.  For non-trivial changes we may want two reviewers.  The primary reviewer will make this decision and nominate a second reviewer, if needed.  Except for trivial changes, PRs should sit for at least a 2 hours to allow for wider review. </del> <ins> All changes must be code reviewed.  For non-maintainers this is obvious, since you can't commit anyway.  But even for maintainers, we want all changes to get at least one review, preferably from someone who knows the areas the change touches.  For non-trivial changes we may want two reviewers.  The primary reviewer will make this decision and nominate a second reviewer, if needed.  Except for trivial changes, PRs should sit for at least 2 hours to allow for wider review. </ins> Most PRs will find reviewers organically.  If a maintainer intends to be the primary reviewer of a PR they should set themselves as the assignee on GitHub and say so in a reply to the PR.  Only the primary reviewer of a change should actually do the merge, except in rare cases (e.g. they are unavailable in a reasonable timeframe).

Files storing latest versions for minors are not updated since infix was to the version string. Example Similar issues can be found in 1.25 and 1.24 builds (last 1.23 build infix so it has not happened there yet): I expect the files to be overwritten with newer version of the build. Manually run build job for one of the minors. No response Not related. Not related. No response No response No response No response
I believe this may be also responsible for breaking kubemark tests: since they fail due to version mismatch
/sig release
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /priority high Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Thanks for raising the issue! /cc who is working on before
I assume this will be fixed automatically with the next patches, because then we would compare with . Anyway, I think we should fix that intermediately by manually updating the version markers to match their correct version. I did that now for the bucket
We had to update the versions markers again, because those were pointing to two commits earlier and didn't include the Go updates. The latest version markers are: v1.24.12-35+ v1.25.8-37+ v1.26.3-41+
latest-1.27 has been affected by this as well. I updated it manually to v1.27.0-9+. See this thread for more details:
/triage accepted
This should be fixed now, closing it. Feel free to reopen if we missed anything.
This issue is still present, see this Slack thread for more details: /reopen
Reopened this issue. <detailsIn response to : This issue is still present, see this Slack thread for more details: /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
We have a similar situation with latest-1.28 which is currently pointing to: . This is one commit before the release commit: ! It's mostly like that the release step didn't update version markers, we should look into this.

As of now, I'm going to manually update this version marker to unblock 1.28.1 patch release. I'll also take a look into this. /assign
The PR that fixed this issue got reverted so this is still relevant: /reopen
Reopened this issue. <detailsIn response to : The PR that fixed this issue got reverted so this is still relevant: /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
We're running into this issue again with stable-1.29 marker. It's currently pointing to which is It should instead point to I'm going to manually change it following the same procedure as the last time. Version marker updated to
This issue is labeled with but has not been updated in over 90 days, and should be re-triaged. Important-soon issues must be staffed and worked on either currently, or very soon, ideally in time for the next release. You can: Confirm that this issue is still relevant with (org members only) Deprioritize it with or Close this issue with For more details on the triage process, see /remove-triage accepted
/triage accepted
Same issue for 1.30, where I had to update the version marker manually again.


Question: // BlockStorageOpts is used to talk to Cinder service type BlockStorageOpts struct { <del> BSVersion       string `gcfg:"bs-version"`        // overrides autodetection. v1 or v2. Defaults to auto TrustDevicePath bool   `gcfg:"trust-device-path"` // See Issue #33128 IgnoreVolumeAZ  bool   `gcfg:"ignore-volume-az"` </del> <ins> BSVersion             string `gcfg:"bs-version"`        // overrides autodetection. v1 or v2. Defaults to auto TrustDevicePath       bool   `gcfg:"trust-device-path"` // See Issue #33128 IgnoreVolumeAZ        bool   `gcfg:"ignore-volume-az"` NodeVolumeAttachLimit int    `gcfg:"node-volume-attach-limit"` // override volume attach limit for Cinder. Default is : 256 </ins> } // RouterOpts is used for Neutron routes

We merged to allow configurable volume limits for Cinder. But proposed adding such a limit to as well, among other things this will enable us to support node specific limits more easily - /sig openstack /sig storage /sig scheduling
/assign
cc
/kind feature
This will require updating in-tree Openstack cloudprovider code. are we onboard with that?
What would be the extent of the modification? It must not break current compatibility, and the in-tree provider is scheduled for removal this year. I'm ok with it as long as existing users aren't broken by it.
it should be a change into which would add a new configurable field. It would be backward compatible but would allow admins to configure different limits on a node if they must.
/milestone v1.14 We're entering and this looks relevant to the 1.14 release. Please if I am incorrect


Question: <ins> #!/usr/bin/env bash # Copyright 2021 The Kubernetes Authors. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # #     http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. set -o errexit set -o nounset set -o pipefail KUBE_ROOT=$(dirname "${BASH_SOURCE[0]}")/.. cd "${KUBE_ROOT}" # generate json spec -> yaml test/conformance/gen-conformance-yaml.sh # diff generated and checked-in if diff -u test/conformance/testdata/conformance.yaml _output/conformance.yaml; then echo PASS exit 0 fi echo 'See instructions in test/conformance/README.md' exit 1 </ins>

The currently provided command for generating requires bazel, the command should be updated to one which does not require bazel Currently: Suggested: /sig testing /sig architecture /area conformance
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/assign /remove-lifecycle rotten
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle rotten
/remove-lifecycle rotten /help
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /remove-lifecycle rotten /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.

</details
We're going to need to update this to verify under as well instead of under the bazel "unit tests", ref: I'll work on this.
done in
This is wonderful to see. Thank you


Question: }) } <ins> // waitForPodNotFoundInNamespace returns an error if it takes too long for the pod to fully terminate. // Unlike `waitForPodTerminatedInNamespace`, the pod's Phase and Reason are ignored. If the pod Get // api returns IsNotFound then the wait stops and nil is returned. If the Get api returns an error other // than "not found" then that error is returned and the wait stops. func waitForPodNotFoundInNamespace(c clientset.Interface, podName, ns string, timeout time.Duration) error { return wait.PollImmediate(Poll, timeout, func() (bool, error) { _, err := c.CoreV1().Pods(ns).Get(podName, metav1.GetOptions{}) if apierrs.IsNotFound(err) { return true, nil // done } if err != nil { return true, err // stop wait with error } return false, nil }) } </ins> // waitForPodSuccessInNamespaceTimeout returns nil if the pod reached state success, or an error if it reached failure or ran too long. func waitForPodSuccessInNamespaceTimeout(c clientset.Interface, podName string, namespace string, timeout time.Duration) error { return WaitForPodCondition(c, namespace, podName, "success or failure", timeout, func(pod *v1.Pod) (bool, error) {

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: See One /sig storage
/assign
[Slow] test suite was run. Glusterfs io stream test failed due to ns not being deleted, which was due to 1 pod remaining. The glusterfs-server pod was deleted. It appears that the glusterfs-client pod was not deleted even though returned no error. In cases where this test succeeds (which is most of the time) returns which is seen in the log as INFO: Ignore "not found" error above. Pod "gluster-io-client" successfully deleted. Sometimes when the plugin io streaming tests pass you'll see logged for the client pod. However, sometimes a client pod is deleted but the above log entry is not triggered. Eg in you'll see the nfs-client pod is deleted (the ns was able to be deleted) yet the above log msg stating to ignore the "not found" error was not generated. (scroll down to the line "I0724 20:27:20.364] Jul 24 20:27:00.559: INFO: Waiting for pod nfs-io-client in namespace 'e2e-tests-volume-io-pmdks' status to be 'terminated due to deadline exceeded'(found phase: "Running", readiness: true) (29.883170178s elapsed) I0724 20:27:20.365] [AfterEach] [sig-storage] NFS"). My guess is that in this case the plugin's client pod did not terminate in 30s -- WHY NOT THE FULL 5m? -- but did terminate in time for the ns to be deleted. And sometimes the client pod cannot be deleted in 30s (again WHY NOT THE FULL 5m?) and is still present when the ns is deleted. Note: if we cannot detect that the client pod was not actually deleted and then we delete the server pod, there is no chance that the ns can be deleted. Once the server pod is gone the client pod will not delete.
It looks like the client pod failed to terminate:
30s may not always be enough time for the gluster client pod to terminate?

also, why is it so common to see ~29 seconds for the client pods (nfs, gluster) to supposedly terminate. The wait should be for up to 5m yet I never see = 30s wait times... yet, we know that sometimes the gluster client pod is not terminated. Seems like maybe a bug in this area?
can you take a look? My findings so far are this test fails about 20% of the time. The gluster client pod's Status is set to Failed and therefore the pod won't be deleted. There is e2e code that waits for the pod to be deleted but escapes if the pod has "Failed" and if the passed-in reason matches the pod's Status.Reason. In the gluster pod test failures the pod's Status.Reason == "". This causes the test to incorrectly assume that the pod was deleted when it wasn't. This problem manifests itself when the namespace is deleted and that step fails. It seems like the Failed pod's Status.Reason should not be blank. cc
It looks like gluster unmount failed:
But in the log I looked at today (https://k8s-) I see the pod's Status=Running. Then the pod is deleted and the status remains Running for ~30s then changes to Failed. Even though the pod is Failed its Reason remains "". So 1) I wonder why the pod is marked as Running if its mount failed, and 2) once the pod is Failed why is Reason still blank?
It was the unmount that failed, not the mount.
ah, ok thanks, I missed that. Still need to know why Pod.Status.Reason is blank...
Below are some records from a related to the glusterfs-client pod, gluster-io-cliente2e-tests-volume-io-6h140(-778e-11e7-afad-)_ and the "e2e-tests-volume-io-6h140" ns:
I think the issue is that the test is deleting the gluster endpoints before deleting the pod.
Nm, the test does seem to be cleaning up in the correct order: client pod, then server pod and endpoints. So the issue seems to be that during Pod termination, it goes to Failed state.

Before, I thought it was because the unmount failed, but based on the timestamps of the logs, it seems like it fails before we start unmounting.
Just spoke to and he said that it's normal for a Pod to go to Failed state during the termination sequence. In between Failed state and Pod object deletion is when we unmount the volumes. So I think the issue here is that the WaitForPodTerminated function needs to wait only for the Pod object to be deleted, and not return early if Pod state is Failed.
do you have time to look at this? The test is still flaking
yes I can take a look. Do you know if there has been a pr started to address this issue? BTW, I will be gone most of tomorrow but will start tomorrow night.


Question: authenticator := union.New(authenticators...) <del> authenticator = group.NewGroupAdder(authenticator, []string{user.AllAuthenticated}) </del> <ins> authenticator = group.NewAuthenticatedGroupAdder(authenticator) </ins> if config.Anonymous { // If the authenticator chain returns an error, return an error (don't consider a bad bearer token anonymous).

When an auth proxy tries to proxy an anonymous request, the user.Info ends up with the group system:authenticated, which is wrong.


Question: // Get exist taints of node. nodeTaints := taintutils.TaintSetFilter(node.Spec.Taints, func(t *v1.Taint) bool { <ins> // only NoSchedule taints are candidates to be compared with "taints" later if t.Effect != v1.TaintEffectNoSchedule { return false } </ins> // Find unschedulable taint of node. if t.Key == algorithm.TaintNodeUnschedulable { return true

In autoscaling tests, we verify that when a node suddenly disappears from the cluster, Cluster Autoscaler will add another one to replace it. With TaintNodesByCondition (enabled by default in ), this is now failing: https://k8s- Every time a node disappears, a subset of pods become NodeLost and aren't rescheduled on a new node. /cc /sig node
I can work on this.
just got merged. Let's check result of autoscaling testings tomorrow: https://k8s-
latest autoscaling testing is green. I think we can close this issue.
Thanks for fixing this!