Question: }) } } <ins> func TestInitPolicyFromFile(t *testing.T) { dir, err := ioutil.TempDir(os.TempDir(), "policy") if err != nil { t.Errorf("unexpected error: %v", err) } defer os.RemoveAll(dir) for i, test := range []struct { policy               string expectedPredicates   sets.String expectedPrioritizers sets.String }{ // Test json format policy file { policy: `{ "kind" : "Policy", "apiVersion" : "v1", "predicates" : [ {"name" : "PredicateOne"}, {"name" : "PredicateTwo"} ], "priorities" : [ {"name" : "PriorityOne", "weight" : 1}, {"name" : "PriorityTwo", "weight" : 5} ] }`, expectedPredicates: sets.NewString( "PredicateOne", "PredicateTwo", ), expectedPrioritizers: sets.NewString( "PriorityOne", "PriorityTwo", ), }, // Test yaml format policy file { policy: `apiVersion: v1 kind: Policy predicates: - name: PredicateOne - name: PredicateTwo priorities: - name: PriorityOne weight: 1 - name: PriorityTwo weight: 5 `, expectedPredicates: sets.NewString( "PredicateOne", "PredicateTwo", ), expectedPrioritizers: sets.NewString( "PriorityOne", "PriorityTwo", ), }, } { file := fmt.Sprintf("scheduler-policy-config-file-%d", i) fullPath := path.Join(dir, file) if err := ioutil.WriteFile(fullPath, []byte(test.policy), 0644); err != nil { t.Fatalf("Failed writing a policy config file: %v", err) } policy := &schedulerapi.Policy{} if err := initPolicyFromFile(fullPath, policy); err != nil { t.Fatalf("Failed writing a policy config file: %v", err) } // Verify that the policy is initialized correctly. schedPredicates := sets.NewString() for _, p := range policy.Predicates { schedPredicates.Insert(p.Name) } schedPrioritizers := sets.NewString() for _, p := range policy.Priorities { schedPrioritizers.Insert(p.Name) } if !schedPredicates.Equal(test.expectedPredicates) { t.Errorf("Expected predicates %v, got %v", test.expectedPredicates, schedPredicates) } if !schedPrioritizers.Equal(test.expectedPrioritizers) { t.Errorf("Expected priority functions %v, got %v", test.expectedPrioritizers, schedPrioritizers) } } } </ins>

Kubernetes scheduler enables you to overwrite default Predicates/Priorities as well as configuring extenders in a "scheduler policy". In latest scheduler componentconfig API (v1alpha1), it can be a file or a configmap: However, if you defines a scheduler config file and specify a YAML policy file: It doesn't work. It's because only JSON is supported: It'd nice to enhance it with an UniversalSerilizer so as to support both JSON and YAML. /sig scheduling /help /priority important-longterm
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : Kubernetes scheduler enables you to overwrite default Predicates/Priorities as well as configuring extenders in a "scheduler policy". /help /priority important-longterm </details
/assign


Question: } } <del> // Proxy the connection. wg := &sync.WaitGroup{} wg.Add(2) </del> <ins> // Proxy the connection. This is bidirectional, so we need a goroutine // to copy in each direction. Once one side of the connection exits, we // exit the function which performs cleanup and in the process closes // the other half of the connection in the defer. writerComplete := make(chan struct{}) readerComplete := make(chan struct{}) </ins> go func() { var writer io.WriteCloser

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: When opening port-forward requests to a pod, kubelet memory usage increases without releasing it again when the port-forward is stopped. What you expected to happen: Kubelet should cleanup the port-forward when it is closed How to reproduce it (as minimally and precisely as possible): Using the following script I can observe an increase in Kubelet memory usage on the node where the pod runs: 300 connections allocates roughly 30mb by kubelet We first noticed this on the node where Tiller (Helm) runs. The node regularly runs out of memory because of the Kubelet memory usage. see also We don't experience the helm issue on all clusters, but I can reproduce the memory leak with the above script on all clusters. Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: AWS OS (e.g. from /etc/os-release): Debian jessie Kernel (e.g. ): Install tools: Kops 1.8.0 Others:
/sig node In my testing (v1.8.0), I observed create a TCP connection between apiserver and kubelet for each port-forward (and one between apiserver and kubectl). This apiserver<-kubelet TCP connection was not torn down when the corresponding command was killed. I have not tried against a newer apiserver/kubelet, but if someone has one handy, it's easy to just compare output on the apiserver or kubelet host before and after running several commands. Be aware that the CLOSE_WAIT connections can take minutes to expire after the kubectl<-apiserver commands exit, and this is normal and fine - we're interested in the ones that stay in state ESTABLISHED.
I'm not aware of any changes in 1.9 or 1.10 that would have fixed this. Not that we shouldn't fix this issue, but something like would be a good backstop to leaking connections like this
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with .

Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale behaviour is still observable for us on 1.9
On v1.11 I see plenty of and connections that hang around after many minutes.
I some logging to (among many other places) After closing a port-forward from kubectl, I see immediately the reader close on the apiserver, and the writer close on the kubelet. I do not see the other side of the connections close. Maybe 1 on each side when the pod ends. Is there a flag to enable pprof so we can dump some goroutine stacktraces? /sig apimachinery
kubelet stacktrace, the important parts
/sig api-machinery
apiserver stacktrace, the important parts
Lastly, if you restart on one side, the other side does clean up. At least if it's the kubelet. I restarted the kubelet, and apiserver shows Which is the ~600 waiting goroutines.
Based on some conversation with I will look into the approach for this issue. That issue () seems like a similar problem.


Question: <ins> // +build windows /* Copyright 2019 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package mount import ( "os" "syscall" "k8s.io/klog" ) // following failure codes are from https://docs.microsoft.com/en-us/windows/desktop/debug/system-error-codes--1300-1699- // ERROR_BAD_NETPATH                 = 53 // ERROR_NETWORK_BUSY                = 54 // ERROR_UNEXP_NET_ERR               = 59 // ERROR_NETNAME_DELETED             = 64 // ERROR_NETWORK_ACCESS_DENIED       = 65 // ERROR_BAD_DEV_TYPE                = 66 // ERROR_BAD_NET_NAME                = 67 // ERROR_SESSION_CREDENTIAL_CONFLICT = 1219 // ERROR_LOGON_FAILURE               = 1326 var errorNoList = [...]int{53, 54, 59, 64, 65, 66, 67, 1219, 1326} // IsCorruptedMnt return true if err is about corrupted mount point func IsCorruptedMnt(err error) bool { if err == nil { return false } var underlyingError error switch pe := err.(type) { case nil: return false case *os.PathError: underlyingError = pe.Err case *os.LinkError: underlyingError = pe.Err case *os.SyscallError: underlyingError = pe.Err } if ee, ok := underlyingError.(syscall.Errno); ok { for _, errno := range errorNoList { if int(ee) == errno { klog.Warningf("IsCorruptedMnt failed with error: %v, error code: %v", err, errno) return true } } } return false } </ins>

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: pod with smb(cifs) mount could not be terminated on Windows when that smb account is invalid, the pod will be stuck forever What you expected to happen: How to reproduce it (as minimally and precisely as possible): a pod with azure file mount on Windows the storage account of azure file invalid the pod with azure file mount could not be terminated forever Anything else we need to know?: similar to case: Environment: Kubernetes version (use ): v1.12.6 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Others: /assign /sig windows


Question: # By now, the contents should be normalized and stripped of any # auto-managed content. <del> if diff -Bw >/dev/null <(echo "${original}") <(echo "${generated}"); then </del> <ins> if diff -B >/dev/null <(echo "${original}") <(echo "${generated}"); then </ins> # actual contents same, overwrite generated with original. cp "${OUTPUT}/${file}" "${OUTPUT_TMP}/${file}" fi

Hi, there is a minor bug in the script or at least some inconsistency. When I run the update script white spaces are ignores with the option but in the white spaces are not ignored. So for consistency we should either ignore white spaces at all or we should diff white spaces (what I would prefer). I wanted to discuss with you the different options before providing a pull request because it's very confusing when you run the update script and afterwards the verify script tells you there are diffs. Thanks!
cc


Question: return ContextOverrideFlags{ ClusterName:  FlagInfo{prefix + FlagClusterName, "", "", "The name of the kubeconfig cluster to use"}, AuthInfoName: FlagInfo{prefix + FlagAuthInfoName, "", "", "The name of the kubeconfig user to use"}, <del> Namespace:    FlagInfo{prefix + FlagNamespace, "", "", "If present, the namespace scope for this CLI request"}, </del> <ins> Namespace:    FlagInfo{prefix + FlagNamespace, "n", "", "If present, the namespace scope for this CLI request"}, </ins> } }

Minor improvement in UX maybe?
We have semi-ambiguous syntax because 'get pod/foo' is also valid. We should have define syntax for this early on. On Apr 8, 2016 7:13 PM, "Prashanth B" wrote:
+1 for the UX improvement.
get pod/foo != get pod ns/foo ?
.... LOL
This would make SO much easier to use :+1:
I dont disagree, but it is somewhat complicated to transition now. Maybe something like 'kind/ns:name' which would be less ambiguous and strictly compatible (ns:name is not a valid name, nor is it confusing wrt kind/name). Need to define precedence of this with --namespace flag and in-file namespace yaml. I also don't see it as a huge win, personally. It would have been nice from day1 but relatively minor now, IMO. But this is a decision for UX team - if someone sent a patch implementing something like it, or at least a proposal that worked through all the details, it would be a better starting place. On Apr 10, 2016 12:33 PM, "Lucas Käldström" wrote:
This is more like P0 to me. Somewhere else I complained about having to type too much and the kind/name syntax. My vote if for kind/ns:name, too.
cc
What is the scenario you're trying to simplify? You often need to use resources from multiple namespaces on the same command line?
cc
ns:name is not a valid name for namespaces and our core resources, but it can be a valid name for extension resources in the Kube ecosystem, like users, roles, authorizations. We'd have to add more slashes
Attaching the namespace to the "name" part won't work well because then you have to chose whether is "list all resources in namespace token" or "get the resource named token in the current namespace". To allow both cases, you must attach the information as a discretely tokenized part or as part of the "resource" part. I think that parses unambiguously since resource can't have dots groups can't have colons and namespaces can't have slashes.
I can assign short names to namespaces and end up with a small kubectl get line, --namespace= just feels unnecessary. I guess it would feel more natural if I could also do .

This isn't a huge deal for me personally because I just bash alias my way out of typing.
In my case, I work with multiple namespaces, not on the same command line (I use --all-namespaces frequently, too). I guess I could create bash aliases, but I also tend to reinstall the machines I run kubectl from on a regular basis.
--all-namespaces is so long anyway that it makes me cringe. , Rudi C wrote:
What api group syntax did we settle on? I don't see it documented anywhere. cc
With Ubernetes, we'll also have multi-cluster scenarios coming.
Syntax is: <resource[[.<version].<group] , Brian Grant wrote:
I think either a short option, like or could be a solution or suffix the namespace using " .
was hoping to save for indicating a for the day when we'd be able to retrieve a historical version of an object via the API. Does not alias to like openshift?
No it does not.
would be an improvement for me: it's noticeably shorter than . Does introduce ambiguities with other flags? or introduce some other issue? or is the hope that there's a nicer unambiguous solution (e.g., something similar to the original proposal)?
I don't know of any conflicts. I'm in favor of the short flag because it gets used all the time.
I am fine with , really appreciate if you could help with this.
We originally had -n in Kube but took it out while we sorted out the other config. We went and it in OpenShift because it was used all the time, but forgot to add it in in kubectl. I think it's time to bring it back to kubectl. Other folks who care about this in kubectl should v oice objections in the next day or so, otherwise I think we have consensus to add as the alias for . Can someone scan over the flags in use today in kubectl and verify that no one has introduced a ? , TonyAdo wrote:
Already did, just client generation. has opened to add this. I've got a minor comments unrelated to the code and tagged to give him a chance to object.
I'm also in favor for adding as a shorthand for


Question: <del> 1.0 </del> <ins> 1.1 </ins>

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: BASEIMAGE for resource-consumer was ubuntu before and there is a ask for migrating to debian via - What you expected to happen: Use debian-base as BASEIMAGE for resource-consumer test image How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration**: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/assign
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig testing


Question: <del> 1.0 </del> <ins> 1.1 </ins>

/kind bug /sig release when we use , we must make sure xxx is not nil. So before we use it, we should check err value first. Such as:
I think even if the file is nil , maybe It doesn't matter
Sometimes it does matter, while can't handle nil correctly. Maybe it is just a good practise.
Although File.Close() can handle cases like this, serverConn.Close() will cause panic to happen. Anyway, these are bugs that exist. /assign


Question: <del> 1.0 </del> <ins> 1.1 </ins>

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: N/A Which test(s) are failing: should support configurable pod Since when has it been failing: N/A Testgrid link: N/A Reason for failure: Since ` was used instead. Anything else we need to know: /sig testing /sig windows


Question: "containers": [{ "name": "master", "image": "dockerfile/redis", <ins> "cpu": 100, </ins> "ports": [{ "containerPort": 6379, "hostPort": 6379

HPA tests are currently failing in gke-large cluster suite: I did some debugging and the problem seems to be related to pod-nanny. Looking into logs from the last run: was first scheduled on the large node dedicated to it pod-nany updated heapster deployment then heapster was scheduled on the normal node (when it actually shouldn't fit) that point it was never updated by the pod nanny. that point, heapster started crashlooping (OOMing) [which in fact was expected in too small machine] So it's pretty clear that that pod-nanny changed the amount of resources to something smaller, which in fact it shouldn't.


Question: // volume types which support fsGroup based ownership(and permissions). // It will have no effect on ephemeral volume types such as: secret, configmaps // and emptydir. <del> // Valid values are "OnRootMismatch" and "Always". If not specified defaults to "Always". </del> <ins> // Valid values are "OnRootMismatch" and "Always". If not specified, "Always" is used. </ins> // +optional optional string fsGroupChangePolicy = 9;

We need to move this field to beta in 1.20. /sig storage
/triage accepted
According to , is working on a better design on this.
Nope. The current design is the design we are going to use to handle fsgroup. we discussed this at length in sig-storage call.
Is it okay if I make this change?


Question: By("creating PD2") disk2Name, err := createPDWithRetry() framework.ExpectNoError(err, "Error creating PD2") <del> host0Pod := testPDPod([]string{disk1Name, disk2Name}, host0Name, false /* readOnly */, 1 /* numContainers */) </del> <ins> var host0Pod *api.Pod </ins> defer func() { By("cleaning up PD-RW test environment") // Teardown pods, PD. Ignore errors. // Teardown should do nothing unless test failed. <del> podClient.Delete(host0Pod.Name, api.NewDeleteOptions(0)) </del> <ins> if host0Pod != nil { podClient.Delete(host0Pod.Name, api.NewDeleteOptions(0)) } </ins> detachAndDeletePDs(disk1Name, []string{host0Name}) detachAndDeletePDs(disk2Name, []string{host0Name}) }()

PR moved some PD tests to flaky to resolve issue (and duplicate ). These tests started failing due to PR . is working on a patch to de-flake the tests. This issue is to track the fix and should be closed when: PD tests have been deflaked (verify by running the "" e2e tests back to back for a couple of hours to verify the issue is fixed). flaky label has been removed from the "" tests that were marked flaky in CC
how to watch the submit queue and find if the flake is gone?
I don't know. is there any way can verify whether test is no longer flaky?
I'm not sure what the publicly accessible link for test history is. I took a look for you: Pod Disks should schedule a pod w/ a RW PD shared between multiple containers, write to PD, delete pod, verify contents, and repeat in rapid succession Failed 3 times in the last 30 runs. Stability: 90 % Pod Disks should schedule a pod w/two RW PDs both mounted to one container, write to PD, verify contents, delete pod, recreate pod, verify contents, and repeat in rapid succession Failed 1 times in the last 30 runs. Stability: 96 % I have a PR pending that enables the new attach/detach controller and I was able to run the tests back to back overnight without any failures. So once that gets merged, we can take a look at moving these tests back.
Something in PD attach/detach affects also 'Dynamic provisioning' e2e test, a PD is still attached to a node ~15 minutes after appropriate pod was deleted and thus deleter can't delete it. From controller log:
please take a look at https://k8s-
thanks so at least one of the flake has a different error than "container not found" In one, the error is
^^ the above seems like a possible bug in exec
^
The error to me indicates that the original container is no longer around as exec can't find the ipc namespace fd. Are we sure the container that we are trying to join is still around?
my memory is rusty but did you have an idea to randomize the container id in this pd test?
see the info here:
Close we will track this with


Question: return nil } <ins> func (cm *unsupportedContainerManager) GetCapacity() v1.ResourceList { return nil } </ins> func (cm *unsupportedContainerManager) NewPodContainerManager() PodContainerManager { return &unsupportedPodContainerManager{} }

It was introduced by cc/
cc/
Only in 1.7 or also 1.8?
PR sent.


Question: dnsPolicy: Default containers: - name: metadata-proxy <del> image: gcr.io/google-containers/metadata-proxy:0.1.1 </del> <ins> image: gcr.io/google-containers/metadata-proxy:0.1.2 </ins> imagePullPolicy: Always securityContext: privileged: true

A number of addon images have CVEs that have been fixed in the upstream base images. We should update these images before the final 1.7 release. cc
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
/cc
cc
The new image appears to have removed . I'm not sure if this was expected, but it caused at least fluentd to stop working on COS (). This might also affect kubeproxy, which uses (a derivative of ).
also the metadata-proxy, prometheus-to-sd, the fluentd-event-exporter, and ip-masq-agent.
anything more to do for this issue?
yes, I need to get merged and update those images again. I also need to bump a few other images (dns and ip-masq-agent).
should be the last update needed.


Question: } if mountEtcHostsFile { hostAliases := pod.Spec.HostAliases <del> hostsMount, err := makeHostsMount(podDir, podIP, hostName, hostDomain, hostAliases) </del> <ins> hostsMount, err := makeHostsMount(podDir, podIP, hostName, hostDomain, hostAliases, pod.Spec.HostNetwork) </ins> if err != nil { return nil, err }

Currently, kubelet mounts hosts file for pods. For hostNetwork pods, the behavior is different and instead, kubelet relies on docker (or whatever container runtime) to mount the host's hosts file. We should have kubelet handle hosts file for hostNetwork pods. This would have several benefits: control over hosts file in case the container runtime does something consistent behavior between hostNetwork and non-hostNetwork pods consistent behavior between different container runtimes -managed hosts file enables features like to be used for hostNetwork pods Implementation would require: kubelet's or its caller output different content based on hostNetwork. i. If it's not hostNetwork, use existing behavior. ii. If it's using hostNetwork, read the content from the host's hosts file. hostNetwork constraint in deciding whether to mount the file Open Questions: do we treat changes to the host's hosts file? should the file's content be static between container restarts of the same Pod? or should every container restart pick up the latest content of the hosts file? the former seems better for consistency, but letter might be more in line with other features that use host resources like host volumes. condition to manage hosts file also excludes windows even though windows also the use of hosts file and the same format. seems like this should be supported also. windows-support ships, should it have validation against windows platform? the tricky bit is the client, API, and minions' kubelet may be running on different platforms, and currently, there's no precedence to platform validation. /kind feature /sig node


Question: framework.Logf("Observed event: %+v", event.Object) return false, nil }) <ins> if err != nil { p, _ := f.ClientSet.CoreV1().Pods(testNamespaceName).Get(context.TODO(), testPodName, metav1.GetOptions{}) framework.Logf("Pod: %+v", p) } </ins> framework.ExpectNoError(err, "failed to see Pod %v in namespace %v running", testPod.ObjectMeta.Name, testNamespaceName) ginkgo.By("patching the Pod with a new Label and updated data")

ci-kubernetes-kind-e2e-parallel ci-kubernetes-e2enode-ubuntu2-k8sstable1-gkespec Pods should run through the lifecycle of Pods and PodStatus failed to see Pod pod-test in namespace Triage: /sig node
/assign /triage accepted
GitHub didn't allow me to assign the following users: fromanirh. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign /triage accepted Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/milestone v1.22
/unassign /assign
I cannot reproduce (despite trying hard!) on my PC... I wonder if that flake wasn't greatly improved by: Now it looks like it fails rarely:
/area node-e2e
/priority important-longterm
/remove-kind flake /kind failing-test Appears to only be failing on the node-kubelet-orphans job: /priority important-soon /remove-priority important-longterm Next steps: check Prow config for this job, doesn't appear to be failing on any other jobs.
Now let's wait 12h to have the next run of kubelet-orphans /reopen
Reopened this issue. <detailsIn response to : Now let's wait 12h to have the next run of kubelet-orphans /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign /unassign MemoryManager tests probably shouldn't be running on these jobs, if we adjust the job configs this should fix the failures.
I updated the orphans job config to exclude -
/milestone v1.23
update: will keep looking into it
I think we can close it now, the orphans job is looking green now.
Feel free to re-open if you will encounter issues with the reported test under a different job.
/close
Closing this issue. <detailsIn response to : /close Instructions for interacting with me using PR comments are available .


Question: }, } <ins> node60040900 := v1.NodeStatus{ Images: []v1.ContainerImage{ { Names: []string{ "gcr.io/600:latest", }, SizeBytes: int64(600 * mb), }, { Names: []string{ "gcr.io/40:latest", }, SizeBytes: int64(40 * mb), }, { Names: []string{ "gcr.io/900:latest", }, SizeBytes: int64(900 * mb), }, }, } node300600900 := v1.NodeStatus{ Images: []v1.ContainerImage{ { Names: []string{ "gcr.io/300:latest", }, SizeBytes: int64(300 * mb), }, { Names: []string{ "gcr.io/600:latest", }, SizeBytes: int64(600 * mb), }, { Names: []string{ "gcr.io/900:latest", }, SizeBytes: int64(900 * mb), }, }, } node400030 := v1.NodeStatus{ Images: []v1.ContainerImage{ { Names: []string{ "gcr.io/4000:latest", }, SizeBytes: int64(4000 * mb), }, { Names: []string{ "gcr.io/30:latest", }, SizeBytes: int64(30 * mb), }, }, } node203040 := v1.NodeStatus{ Images: []v1.ContainerImage{ { Names: []string{ "gcr.io/20:latest", }, SizeBytes: int64(20 * mb), }, { Names: []string{ "gcr.io/30:latest", }, SizeBytes: int64(30 * mb), }, { Names: []string{ "gcr.io/40:latest", }, SizeBytes: int64(40 * mb), }, }, } </ins> nodeWithNoImages := v1.NodeStatus{} tests := []struct {

<!-- Please only use this template for submitting enhancement requests --What would you like to be : redefine the formula, the score should be capped per images instead of pod. Why is this needed: For the pod that going to be scheduled, calculate the score based on the below steps, Scaled score base on the image size and it's spread over the cluster, image[i].nodes: number of nodes that already have image ! Capped score based on the assumption 90% image size on dockerhub falls into the range [23M ~ 1000M] lowerbound: 23M upperbound:1000M ! Final scores returned for scheduling, maxscore: 100 ! The second step is not well guarded, the assumption is based on the image size while the pod may hold multiple images, it is not surprise to see that total image size of the pods is larger than 1000M, while each of might be only around 300M. In this below case, following the above formula, NODE1 and NODE2 willl get the same scores(100), since both of them are capped by the max score (100), the ideal result is score of NODE2 should be larger than NODE1 (all requested images are ready on NODE2) and is preferred, the larger the sample1 here the more scores should be expected for NODE2. !
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig scheduler Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scheduling
cc
Thanks a lot for summarizing the problem. I agree that the formula needs improvement. ~I think doing the capping for each image solves the original problem and what said in There are two options: only the total from to for each container. We have to ask ourselves how much signal each formula gives and depending on what. Think about different scenarios: a pod with a lot of very small containers, a pod with a few big containers, etc.
/assign
Here is my thoughts, step1: calculate image level priority, for each image (should be same as exsting func ) step2: cap the each image's score with thresholds, minThreshold and maxThreshold.

(should be same as existing func except that it's per image instead of pod) step3: score normalization per image. step4: calculate pod level priority, and return the final score, range (0~100) step1 and step2 primarily update the exiting code to make it is image level priorities instead of pod level. Based on the above steps, consider the below two cases, a pod with four images, case1: All images are ready on node2 node1 (0, 80, 90, 60) - final score is: (80+90+60) / 4 = 57 node2 (90, 80, 90, 60) - final score is: (90+80+90+60) / 4 = 80 √ case2: each node has three images ready but image1 is larger than image2. node1 (0, 80, 70, 60) - final score is: (80+70+60) / 4 = 52 node2 (90, 0, 70, 60) - final score is: (90 +70+60) / 4 = 55 √ thoughts? cc
Thanks can you update the the examples with the score that we would have gotten with the current function?
For reference, see this analysis for a different plugin
Also it's worth to see the difference with just capping per pod, instead of per container.
can you also describe the image sizes you used? the example in the original post assumed three images, while your analysis is assuming four.
agree with and the explanation in looks good but just to be sure it would be nice to see the math worked out with image sizes, and compared to the current formula
Sorry for the confusion, let me switch back to the original three images, update the image size a little bit to make it easier to understand, and also fit to the case of and . Here is the pod, ! and three nodes in the cluster, ! Let's see the difference between two approaches, capping per pod, step1: ! NODE1: scores = 30 2/3 + 1500 2/3 = 1020 NODE2: scores = 30 2/3 + 600 2/3 + 1500 2/3 = 1420 step2: !

NODE1: 1020 1000 (maxThreshold), cappedscore = 1000 NODE2: 1420 1000 (maxThreshold), cappedscore = 1000 step3: ! upperbound = 100 / lowerbound = 23 / maxscore = 100 NOTE1: score = 100 (1000 - 23) / (1000 - 23) = 100 NODE2: score = 100 (1000 - 23) / (1000 - 23) = 100 potentially, NODE1 will be chose for scheduling. capping per images, step1: calculate image level priority, NODE1: image1: score = 30 2/3 = 20 image2: score = 0 image3: score = 1500 2/3 = 1000 NODE2: image1: score = 30 2/3 = 20 image2: score = 600 2/3 = 400 image3: score = 1500 2/3 = 1000 step2/3: cap each image's score with thresholds and calculate the image score, ! NODE1: image1: score = 100 (23 - 23) / ( 1000 - 23) = 0 image2: score = 0 image3: score = 100 (1000 - 23) / (1000 - 23) = 100 NODE2: image1: score = 100 (23 - 23) / ( 1000 - 23) = 0 image2: score = 100 (400 - 23) / (1000 - 23) = 35 image3: score = 100 * (1000 - 23) / (1000 - 23) = 100 step4: final score NODE1: finalscore = 100 / 3 = 33 NODE2: final_score = 135 / 3 = 45 NODE2 will be chosen for scheduling.
Computing the score per image and normalizing by the number of images make sense to me at first glance, but I would like to see more examples.

It would be much easier to digest the analysis if it was summarized in tabular form comparing three scoring functions: (1) current one, (2) one in and (3) multiple big images, multiple small images (total less than the threshold) and a mix. Also, compare cases where the spread is high vs low.
Yes, we don't need to see the individual steps, but a good set of scenarios with final scores. I didn't mean just the old function. We know it's very ineffective. What I meant is capping per pod using the strategy you used in
Did some comparison here, spread has been included based on the layout of each node, it's not easy for me to pull this factor out to focus on this factor only. I will design more testcases when the patch is ready for review which I hope could be more clear. column 1: pod with three images column 2: three nodes' images layout column 3 / 4 / 5: score of each node for different approaches. pod cap and normalizing per image
[10, 20, 30] & small [0,0,0] [800, 1200, 1500] & big [33, 59, 67] [30, 600, 1500] & mix [33, 45, 0]
Normalizing by the number of images gives each image the same "weight". Another variation of the third option is to try and multiply each image score with its size divided by the total size of all images, can you add that as another option?
I don't understand that suggestion. The score of an image is already proportional to its size. I would say the difference between the two suggestions is: Capping per pod cares about total downloaded size Capping per image gives roughly the same importance to each image. I think capping per pod makes more sense. However, there is some information we are losing in both suggestions, as seen in the first case. Even if a lot of small images need to be downloaded, we are giving a 0 score to all nodes. I think a cap from makes more sense, as suggested in the initial PR.
It helps giving more importance to larger images.
If the statistics are accurate with regards to image size distribution, then having many small images all scoring zero isn't common.

For the more common cases (sizes between 32-1000), it seems that both approaches will likely give similar results, so I am fine with both approaches.
That would make the score proportional to the square of the size. Just to play safe (ie, not make the score too strong in a way that could affect spreading or resource allocation) I suggest we go with capping per pod. And then we can re-evaluate next release.
Speaking of statistics. They might have changed since this plugin was first developed. Is there a reliable source?
True for uncapped images, but everything we are doing is just heuristics, and the suggestion is just for the sake of exploring options, not advocating a specific one. Playing it safe sounds good to me.
This approach needs the smallest change and the minimum impact.
+1, this is not common case we can see that a pod with lots of small images (all < 23M), but a corner case that might be improved. No perfect solution so far, but either one is better than the original one IMHO.
/cc , Do you have any comments so far?
btw, has been updated to the second approach (capping and normalization per image).
I would go with this approach as well, it preserves the behavior for multiple small images, and improve it for multiple big ones.
seems like a simple enough compromise for now, and running through some of the examples with that formula it does get more of the specificity you're looking for with large images without normalizing all smaller images right to 0 (so we keep some of the detail there too). I would like to see some updated statistics on the thresholds, and while 90% of the images might fall between them I'm wondering if that's a totally normal distribution. suggestion in got me (tangentially) thinking if we could use thresholds that are more relevant to the pod in question, rather than all of Dockerhub. Because as it is now hypothetical pods (and clusters) which have all small images don't get much benefit from ImageLocality. But I recognize that could be intentional since download resources for small images is ultimately negligible.
I also believe that scoring of small images by zero is intentional, but seems like making lots of small images in pod accountable for the scheduling is slightly better.
It looks like we have consensus. Can you update the PR?


Question: for k, v := range errorCases { _, errs := validateVolumes(v.V) if len(errs) == 0 { <del> t.Errorf("expected failure for %s", k) </del> <ins> t.Errorf("expected failure %s for %v", k, v.V) </ins> continue } for i := range errs {

Given the following configuration: No volume will be mounted inside the resulting container: If I modify the volumes element to look like this: I get a volume mounted on /srv: I see the same behavior using either or .
I suspect I broke this default behavior. Hopefully someone else can provide a fix? I'll fix if it's still pending in a day or two. (In the middle of other things)
I will look into this.


Question: # Tests which are not able to be run in parallel. GCE_PARALLEL_SKIP_TESTS=( <del> "Etcd" "NetworkingNew" </del> "NodessNetwork" <del> "NodessResize" </del> "MaxPods" "Resourcesusagesofssystemscontainers" "SchedulerPredicates" <del> "Services.*restarting" </del> "resourcesusagestracking" <ins> "${DISRUPTIVE_TESTS[@]}" </ins> ) # Tests which are known to be flaky when run in parallel.

We should be testing the reliability of a long-running cluster. These tests kill/restart components or nodes in the cluster, defeating the purpose of a soak cluster.
Example tests: NodessResize Services.restarting DaemonRestart NodessResize Reboot Etcd failure
Agreed. Some of them already seem to be skipped: GCESOAKCONTINUOUSSKIPTESTS=( "Density.30spods" "Elasticsearch" "Etcd.SIGKILL" "externalsloadsbalancer" "identicallysnamedsservices" "networkspartition" "Services.Typesgoessfrom" ) Q
FYI
It'd be hard to prevent regression as more tests are . It might be better if we explicitly define a set of DISRUPTIVETESTS, and include them wherever needed, e.g., GCEPARALLELSKIPTESTS and GCESOAKCONTINUOUSSKIPTESTS. That way, people who are adding tests will be aware of the category. WDYT?
Agreed, that seems like a good idea. Want to send in a PR? Or at least open an issue?


Question: port("foo", v1.ProtocolUDP, 555, 5555, ""), // Duplicated, should be ignored. port("foo", v1.ProtocolUDP, 888, 8888, ""), <del> // Duplicated, should be ignored. port("", v1.ProtocolTCP, 80, 8888, "127.0.0.1"), </del> // Duplicated with different address family, shouldn't be ignored port("", v1.ProtocolTCP, 80, 8080, "::"), // No address family specified

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: A container ports spec such as: Results in this error: From . Because of this, the port mappings are not setup correctly on the host when using a CNI-compliant portmap solution. What you expected to happen: Auto-generated port names should never result in duplicates How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: /sig node Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.7", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-01-16T21:59:57Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.11", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-04-05T17:16:46Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: VirtualBox OS (e.g. from /etc/os-release): Ubuntu Kernel (e.g. ): Linux node2 4.4.0-104-generic -Ubuntu SMP Mon Dec 11 12:16:42 UTC 2017 x8664 x8664 x86_64 GNU/Linux Install tools: kubeadm Others:
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with .

Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
/assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
I see that the problem is legit, but is this setup frequently used? why not using services instead of hostports? I mean, it will complicate more the solution but will provide a good abstraction and remove the dependency on the node.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
Why is this still an issue?
Issues go stale after 90d of inactivity.


Question: "math" "strings" <ins> corev1 "k8s.io/api/core/v1" </ins> "k8s.io/apimachinery/pkg/api/meta" "k8s.io/apimachinery/pkg/api/validation" <ins> metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" </ins> "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" "k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/runtime/schema" <ins> "k8s.io/apimachinery/pkg/util/sets" apimachineryvalidation "k8s.io/apimachinery/pkg/util/validation" </ins> "k8s.io/apimachinery/pkg/util/validation/field" "k8s.io/apiextensions-apiserver/pkg/apis/apiextensions"

A client is allowed to add custom finalizers to Custom Resources with a finalizer name that's not fully qualified (e.g. or ). However they're prevented from doing the same on builtin resource types, returning the following error: Invalid value: "": name is neither a standard finalizer name nor is it fully qualified Per , custom finalizer names are expected to be in format "consist of a domain name, a forward slash and the name of the finalizer" (e.g. ). Therefore, I expected the enforcement to be the same for CRs and builtin types. a custom resource on the cluster it to add the edit does not fail the same on a builtin resource (e.g. Namespace or Pod), observe it fails No response Reproducible at least starting from 1.21 and still on latest versions. <details</details<detailsN/A kind cluster </details<detailsN/A - kind cluster accessed via macOS </details<details</details<details</details<details</details
/area api /cc /sig api-machinery
/assign /triage accepted
There is no validation of finalizer in and . Please unassign me if you guys have started to work on it. /assign
it is called from here on custom resources:
lacks validation for finalizer name. I have raised a pr for this. PTAL.


Question: docs/user-guide/kubectl/kubectl_describe.md docs/user-guide/kubectl/kubectl_drain.md docs/user-guide/kubectl/kubectl_edit.md <del> docs/user-guide/kubectl/kubectl_edit_configmap.md </del> docs/user-guide/kubectl/kubectl_exec.md docs/user-guide/kubectl/kubectl_explain.md docs/user-guide/kubectl/kubectl_expose.md

kubectl edit is not working on HEAD. It's working on 1.5.


Question: framework.Logf("Collection of services has been deleted") }) <ins> ginkgo.It("should serve endpoints on same port and different protocols", func(ctx context.Context) { serviceName := "multiprotocol-test" testLabels := map[string]string{"app": "multiport"} ns := f.Namespace.Name containerPort := 80 svcTCPport := v1.ServicePort{ Name:       "tcp-port", Port:       80, TargetPort: intstr.FromInt(containerPort), Protocol:   v1.ProtocolTCP, } svcUDPport := v1.ServicePort{ Name:       "udp-port", Port:       80, TargetPort: intstr.FromInt(containerPort), Protocol:   v1.ProtocolUDP, } ginkgo.By("creating service " + serviceName + " in namespace " + ns) testService := v1.Service{ ObjectMeta: metav1.ObjectMeta{ Name:   serviceName, Labels: testLabels, }, Spec: v1.ServiceSpec{ Type:     v1.ServiceTypeClusterIP, Selector: testLabels, Ports:    []v1.ServicePort{svcTCPport, svcUDPport}, }, } service, err := cs.CoreV1().Services(ns).Create(ctx, &testService, metav1.CreateOptions{}) framework.ExpectNoError(err, "failed to create Service") containerPorts := []v1.ContainerPort{{ Name:          svcTCPport.Name, ContainerPort: int32(containerPort), Protocol:      v1.ProtocolTCP, }, { Name:          svcUDPport.Name, ContainerPort: int32(containerPort), Protocol:      v1.ProtocolUDP, }} podname1 := "pod1" ginkgo.By("creating pod " + podname1 + " in namespace " + ns) createPodOrFail(ctx, f, ns, podname1, testLabels, containerPorts, "netexec", "--http-port", strconv.Itoa(containerPort), "--udp-port", strconv.Itoa(containerPort)) validateEndpointsPortsWithProtocolsOrFail(cs, ns, serviceName, fullPortsByPodName{podname1: containerPorts}) ginkgo.By("Checking if the Service forwards traffic to the TCP and UDP port") execPod := e2epod.CreateExecPodOrFail(ctx, cs, ns, "execpod", nil) err = testEndpointReachability(ctx, service.Spec.ClusterIP, 80, v1.ProtocolTCP, execPod, 30*time.Second) if err != nil { framework.Failf("Failed to connect to Service TCP port: %v", err) } err = testEndpointReachability(ctx, service.Spec.ClusterIP, 80, v1.ProtocolUDP, execPod, 30*time.Second) if err != nil { framework.Failf("Failed to connect to Service UDP port: %v", err) } ginkgo.By("Checking if the Service forwards traffic to TCP only") service, err = cs.CoreV1().Services(ns).Get(ctx, serviceName, metav1.GetOptions{}) if err != nil { framework.Failf("failed to get Service %q: %v", serviceName, err) } service.Spec.Ports = []v1.ServicePort{svcTCPport} _, err = cs.CoreV1().Services(ns).Update(ctx, service, metav1.UpdateOptions{}) if err != nil { framework.Failf("failed to get Service %q: %v", serviceName, err) } // test reachability err = testEndpointReachability(ctx, service.Spec.ClusterIP, 80, v1.ProtocolTCP, execPod, 30*time.Second) if err != nil { framework.Failf("Failed to connect to Service TCP port: %v", err) } // take into account the NetworkProgrammingLatency // testEndpointReachability tries 3 times every 3 second // we retry again during 30 seconds to check if the port stops forwarding gomega.Eventually(ctx, func() error { return testEndpointReachability(ctx, service.Spec.ClusterIP, 80, v1.ProtocolUDP, execPod, 6*time.Second) }).WithTimeout(30 * time.Second).WithPolling(5 * time.Second).ShouldNot(gomega.BeNil()) ginkgo.By("Checking if the Service forwards traffic to UDP only") service, err = cs.CoreV1().Services(ns).Get(ctx, serviceName, metav1.GetOptions{}) if err != nil { framework.Failf("failed to get Service %q: %v", serviceName, err) } service.Spec.Ports = []v1.ServicePort{svcUDPport} _, err = cs.CoreV1().Services(ns).Update(ctx, service, metav1.UpdateOptions{}) if err != nil { framework.Failf("failed to update Service %q: %v", serviceName, err) } // test reachability err = testEndpointReachability(ctx, service.Spec.ClusterIP, 80, v1.ProtocolUDP, execPod, 30*time.Second) if err != nil { framework.Failf("Failed to connect to Service UDP port: %v", err) } // take into account the NetworkProgrammingLatency // testEndpointReachability tries 3 times every 3 second // we retry again during 30 seconds to check if the port stops forwarding gomega.Eventually(ctx, func() error { return testEndpointReachability(ctx, service.Spec.ClusterIP, 80, v1.ProtocolTCP, execPod, 6*time.Second) }).WithTimeout(30 * time.Second).WithPolling(5 * time.Second).ShouldNot(gomega.BeNil()) }) </ins> /* Release: v1.26 Testname: Service, same ports with different protocols on a Load Balancer Service

see A pod listens on port N, on both TCP and UDP. A service forwards to port N, only TCP. Accessing the service via UDP should fail.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
cc
/assign this was caught with hostPort , but we should have one for Services


Question: var r []ginkgo.Reporter if *reportDir != "" { r = append(r, reporters.NewJUnitReporter(path.Join(*reportDir, fmt.Sprintf("junit_%02d.xml", config.GinkgoConfig.ParallelNode)))) <ins> failReport := &failReporter{} r = append(r, failReport) defer func() { if failReport.failed { coreDump(*reportDir) } }() </ins> } ginkgo.RunSpecsWithDefaultAndCustomReporters(t, "Kubernetes e2e suite", r) }

We need to be able to diagnose e2e test failures after the relevant test cluster has been shut down. Right now this almost works, but not quite. Fluentd can currently only be configured to send log files to one place, for example: Cloud Logging, or (running as a pod in the e2e test cluster, and saved to local disk on the node). With the current configuration in , logs typically go to ElasticSearch (and there is an e2e test to check that that works correctly), and tears down the e2e test cluster after each test run, so the logs are deleted. The proposed solution is to have fluentd configurable to send to multiple log collector backends, and configure it as such to both Google Cloud Logging and ElasticSearch for CI e2e test runs to facilitate easier debugging. Satnam6502@ has offered to sort this out.
Originally it was possible to send logs to both Elasticsearch and Cloud Logging but at some point this got made into a binary choice of one or the other (by or ?). I propose to undo this unless there are strong objections. : what do you think?
I didn't turn it into a binary choice. :) , Satnam Singh wrote:
Yeah, pretty sure it was Joe. I am sure it was a wide decision at the time.
Would be nice to have these at maximum verbosity because when we need them we need them all, IIRC we currently only get V(2) on a kube-up cluster
I see this is marked as a post v1.0 milestone? If we don't want this urgently then I will un-assign myself for now.
There is a valid argument that we would not block the 1.0 release for lack of these persisted test logs, and hence the v1.0 milestone was tentatively removed. There is a counter argument that we stand little chance of being able to debug the existing and still-to-be-found flaky test failures in the absence of log files, and that one or more of those bugs, if not fixed, will block the launch anyway. I'd say that if there is nothing on the v1.0 milestone that you can sensibly work on, this one is fair game.

I would find it useful, and I suppose this should be on par with writing tests (of which we have a few 1.0: )
I'm not sure we'll be able to convince folks to move this to 1.0, but feel free to work on this, as it seems very valuable.
I thought about changing our setup to allow both Elasticsearch and Cloud Logging to be targeted. I've been a little hesitant because this could lead to a breakage (esp. for GKE) and we are getting close to 1.0 ...
/cc who might have memories about a GKE breakage caused by changes in env vars used for logging.
I'd love to see this fixed soon to help debug flakes, but yes, it'd be nice to at least be cc'ed on env var changes.
Try again to have it to the high priority list, as this is making debugging the system very difficult. cc Who expressed interest.
How do we access those logs? I tried to find them for gce-scalability jenkins suite, but didn't find it.


Question: valid2 := map[string]string{ "Corefile": fmt.Sprintf(`:53 { health <ins> ready </ins> kubernetes %v in-addr.arpa ip6.arpa { pods insecure upstream

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: ci-kubernetes-e2e-gci-gce-serial Which test(s) are failing: [sig-network] DNS configMap nameserver [IPv4] Change stubDomain should be able to change stubDomain configuration [Slow][Serial] [sig-network] DNS configMap nameserver [IPv4] Forward external name lookup should forward externalname lookup to upstream nameserver [sig-network] DNS configMap nameserver [IPv4] Forward PTR lookup should forward PTR records lookup to upstream nameserver [Slow][Serial] Since when has it been failing: These tests have been failing since 5/18. Testgrid link: Reason for failure: All of these have this failure message in common: The failures coincide with for which the possible fix is . However, we are filing these failures separate from to see if there are any additional steps that need to be done for these series of tests. /cc /priority critical-urgent /kind failing test /milestone v1.15 /cc
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/assign
/remove-triage unresolved
Tests have been green since 6/1


Question: "k8s.io/apimachinery/pkg/api/resource" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" v1helper "k8s.io/kubernetes/pkg/apis/core/v1/helper" <ins> "k8s.io/kubernetes/pkg/kubelet/types" </ins> schedulerframework "k8s.io/kubernetes/pkg/scheduler/framework" "k8s.io/kubernetes/pkg/scheduler/framework/plugins/nodename" "k8s.io/kubernetes/pkg/scheduler/framework/plugins/nodeports" <ins> "k8s.io/kubernetes/pkg/scheduler/framework/plugins/tainttoleration" </ins> ) var (

The method doesn't check taints and tolerations. This is problematic in 2 scenarios: If the users bypass scheduler by setting a value in If the node restarts. The node would try to restart scheduled pods once it comes back up, even if it's not ready (has a not-ready taint). Kubelet to check taints and tolerations. a taint in a node. a Pod with set to the Node above, without a matching toleration. /sig node /sig scheduling
this reminds me of your kubelet sync PR ...
I don't think this is a syncing issue. The logic doesn't check taints at all.
scenario 1 is pretty bad, i will let the #sig-node slack know about this.
It doesn't sound like a syncing issue, but we seem to have multiple issues with pod admission/management on restart currently
I'm aware of that issue. This should be unrelated.
I would like to address this. /assign
I think it would be great to proceed after merging ref:
Please add me as reviewer
/triage accepted
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale PR in review:
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.


Question: OpenAPI         openapi.Resources Force           bool ServerSideApply bool <ins> FieldManager    string </ins> ForceConflicts  bool genericclioptions.IOStreams }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: fails. What you expected to happen: I would expect the PatchOptions to not include FieldManager. The following fix will eliminate the problem. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): 1.16 and 1.17 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig cli
cc
/wg apply
cc
I agree, we clearly missed that. Would you mind sending the fix in the form of a PR? Also please include a test, because that shouldn't have happened if I had tested my code properly :-(
Clearly we should also cherry-pick that into 1.17. I'll send the PR on Monday if you prefer! Thanks.
Ok, I will send a PR. However, I can't include a test immediately because I don't understand how the test works.
Thanks, I’ll try to show you how to get started!
Any update we clearly need this promptly since code-freeze is coming! Thanks.
I'll send a PR on Monday if you haven't! Thanks.
Sorry, I haven't started working on this test. Thanks.


Question: ginkgo.Context("when running a container with a new image", func() { <del> // Images used for ConformanceContainer are not added into NodeImageWhiteList, because this test is </del> <ins> // Images used for ConformanceContainer are not added into NodePrePullImageList, because this test is </ins> // testing image pulling, these images don't need to be prepulled. The ImagePullPolicy <del> // is v1.PullAlways, so it won't be blocked by framework image white list check. </del> <ins> // is v1.PullAlways, so it won't be blocked by framework image pre-pull list check. </ins> imagePullTest := func(image string, hasSecret bool, expectedPhase v1.PodPhase, expectedPullStatus bool, windowsImage bool) { command := []string{"/bin/sh", "-c", "while true; do sleep 1; done"} if windowsImage {

<!-- Please only use this template for submitting enhancement requests --What would you like to be : We should come up with a better name for : Why is this needed: From /sig testing /priority important-soon /cleanup /help /good-first-issue
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : <!-- Please only use this template for submitting enhancement requests --We should come up with a better name for : From /priority important-soon /cleanup /help /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I'd like to help out with this. /assign
mind. It seems like you have a PR for it already. /unassign
And I misread the changes you made in . I'm back on this one. /assign
I'm going to rename this to . Please let me know if there are any objections or if there are things that I should be considering.
please feel free to take it. I'm only changing ImageWhileList withint
ultimately was changed to NodePrePullImageList because that's what it's actually used for, not for any sort of filtering.


Question: github.com/mitchellh/mapstructure v0.0.0-20160808181253-ca63d7c062ee/go.mod h1:FVVH3fgwuzCH5S8UJGiWEs2h04kUh9fWfEaFds41c1Y= github.com/mitchellh/mapstructure v1.1.2/go.mod h1:FVVH3fgwuzCH5S8UJGiWEs2h04kUh9fWfEaFds41c1Y= github.com/mitchellh/mapstructure v1.4.1/go.mod h1:bFUtVrKA4DC2yAKiSyO/QUcy7e+RRV2QTWOzhPopBRo= <del> github.com/moby/spdystream v0.2.0 h1:cjW1zVyyoiM0T7b6UoySUFqzXMoqRckQtXwGPiBhOM8= </del> github.com/moby/spdystream v0.2.0/go.mod h1:f7i0iNDQJ059oMTcWxx8MA/zKFIuD/lY+0GqbN2Wy8c= github.com/moby/term v0.0.0-20210610120745-9d4ed1856297/go.mod h1:vgPCkQMyxTZ7IDy8SXRufE172gr8+K/JE/7hHFxHW3A= github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=

What happened: upgrade requests to aggregated APIs (like websocket-based watch requests) on a kube-apiserver using --ssh-user hang What you expected to happen: Websocket-based watch requests work properly How to reproduce it (as minimally and precisely as possible): a kube-apiserver configured to use a node dialer to reach services/pods (e.g. with ) an aggregated API server (like the sample-apiserver) on a node other than the kube-apiserver a websocket-based watch against the aggregated API Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
/sig api-machinery /cc lavalamp
The code in question is here: handlingInfo.proxyRoundTripper is built here: and inherits the proxyDialerFn built in from the following construction chain: However, drops the custom dialer contained inside and from the returned round tripper. That means that upgrade requests fall back to use the default dialer, which cannot reach the aggregated API server
cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/assign would be good to resolve with the apiserver proxy work


Question: The IP address in the `EXTERNAL_IP` column is the one that is available on the public internet.  The `CLUSTER_IP` is only available inside your cluster/private cloud network. <ins> Note that on AWS, type `LoadBalancer` creates an ELB, which uses a (long) hostname, not an IP.  It's too long to fit in the standard `kubectl get svc` output, in fact, so you'll need to do `kubectl describe service nginxsvc` to see it.  You'll see something like this: ``` > kubectl describe service nginxsvc ... LoadBalancer Ingress:   a320587ffd19711e5a37606cf4a74574-1142138393.us-east-1.elb.amazonaws.com ... ``` </ins> ## What's next? [Learn about more Kubernetes features that will help you run containers reliably in production.](production-pods.md)

GCP uses IP addresses, which is shown on an extra line. AWS does not give an IP address. We should show it. GCP: AWS: I'd like to try to do this work.
cc
AWS uses hostnames for load balancers which is why they aren't showing up. The hostname is also pretty long, and I know we've had problems with the width of the kubectl output before. That said, I seem to recall a "--wide" option, and I think you can maybe choose which columns appear? I think it is a useful suggestion, and I think your biggest battle will be to keep the number of columns such that it doesn't normally wrap. Just so you know, I normally do to see the AWS ELB hostname.
Duping this into , we also have an lgtm'd pr out


Question: printenv | sort echo "--------------------------------------------------------------------------------" <del> if [[ "${E2E_UP,,}" == "true" ]]; then </del> <ins> # We get the Kubernetes tarballs on either cluster creation or when we want to # replace existing ones in a multi-step job (e.g. a cluster upgrade). if [[ "${E2E_UP,,}" == "true" || "${JENKINS_FORCE_GET_TARS:-}" =~ ^[yY]$ ]]; then </ins> if [[ ${KUBE_RUN_FROM_OUTPUT:-} =~ ^[yY]$ ]]; then echo "Found KUBE_RUN_FROM_OUTPUT=y; will use binaries from _output" cp _output/release-tars/kubernetes*.tar.gz . else echo "Pulling binaries from GCS" <del> if [[ $(find . | wc -l) != 1 ]]; then </del> <ins> # In a multi-step job, clean up just the kubernetes build files. # Otherwise, we want a completely empty directory. if [[ "${JENKINS_FORCE_GET_TARS:-}" =~ ^[yY]$ ]]; then rm -rf kubernetes* elif [[ $(find . | wc -l) != 1 ]]; then </ins> echo $PWD not empty, bailing! exit 1 fi

The Jenkins job for GKE upgrade, step 3 (e2e) has been failing since the 4:40pm (PST) run yesterday because of the following DNS problems: This is after the cluster is upgraded.
+cc for triage / suggestions; mentioned as a possibility.
This was almost certainly broken by which was merged at 3:35pm yesterday.
/cc
cc
Yup, my money would be on . Should not be too difficult to track down the bug, given that GCE is working just fine. I'd advise against rolling back if feasible, and rather debug the GKE end of the problem.
Staring at the test it is not clear if anything is wrong with it -- perhaps we need to look at the upgrade code.
I believe that many of the e2e tests begin to fail if DNS isn't working, which to me implies that this is a problem with the test since the failures are isolated to just these two tests. As a side note, I think that this failure is actually correct in the sense that it is catching a legitimate behavioral change between the latest release (0.20.2) and head -- if someone was running code that depended upon the DNS pods to be running in the default namespace and we moved them away then their code would break. In this case, the break was intentional, but it's a behavioral change none the less.
also reported the test failure after upgrade test due to no node is ready in test cluster. We checked it, and found cbr0 is not configured because master never allocate ip range to node, and docker refuse to start.
The code at head selects DNS pods from the system namespace. The code from the 0.20.2 release selects DNS pods from the default namespace. Right now this test is running the e2e test code from the latest release against a cluster that has been upgraded to HEAD.
Thanks for the debugging, all—as Robby said, this catches a behavioral change, which is good. To have the tests pass, we'll want to run the updated tests after an upgrade (this is my next task). I'll mark this as release-note and close it.


Question: // TODO: add tests that cover deployment.Spec.MinReadySeconds once we solved clock-skew issues // See https://github.com/kubernetes/kubernetes/issues/29229 <del> ginkgo.It("should run the lifecycle of a Deployment", func() { </del> <ins> /* Release: v1.20 Testname: Deployment, completes the lifecycle of a Deployment Description: When a Deployment is created it MUST succeed with the required number of replicas. It MUST succeed when the Deployment is patched. When scaling the deployment is MUST succeed. When fetching and patching the DeploymentStatus it MUST succeed. It MUST succeed when deleting the Deployment. */ framework.ConformanceIt("should run the lifecycle of a Deployment", func() { </ins> zero := int64(0) deploymentResource := schema.GroupVersionResource{Group: "apps", Version: "v1", Resource: "deployments"} testNamespaceName := f.Namespace.Name

[x] Original approval issue: [x] Test pr: [x] Two weeks soak end date: 19 November 2020 This issue is created to allow edit by /wip /hold According to this APIsnoop query, there are still some remaining Deployment endpoints which are untested. a Deployment with a static label the Deployment with a new Label and updated data the Deployment to ensure it's patched all Deployments in all Namespaces find the Deployment(1) ensure that the Deployment is found and is patched Namespaced Deployment(1) via a Collection with a LabelSelector Discover useragents: List endpoints hit by the test: Display endpoint coverage change: If a test with these calls gets merged, test coverage will go up by 5 points* This test is also created with the goal of conformance promotion.
/sig testing /sig architecture /area conformance
BobyMcbobs commented in original Issue: Unsure if the endpoint replaceAppsV1NamespacedDeploymentStatus is able to be hit. When this endpoint is attempted to be updated the data after retrieving the update resource doesn't reflect the values in the update
/assign
When using the appv1.Deployment structure instead of maps of interfaces to build the structure in a patch of the Deployment, this error comes up. Still in progress yment-static":"true"}: does not match template , spec.selector: Invalid value: "null": field is immutable]
replaceAppsV1NamespacedDeploymentStatus to test. Increase cover by 1 Endpoint
/reopen PR reverted by due to flaky test
Reopened this issue. <detailsIn response to : /reopen PR reverted by due to flaky test Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/reopen Soaking, ready for promotion 19 November 2020
Reopened this issue. <detailsIn response to : /reopen Soaking, ready for promotion 19 November 2020 Instructions for interacting with me using PR comments are available .

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-lifecycle stale
/triage accepted
/merged


Question: // We used to make the call every 1sec (1 QPS), the goal here is to achieve ~98% traffic reduction when // API server is not healthy. With these parameters, backoff will stop at [30,60) sec interval which is // 0.22 QPS. If we don't backoff for 2min, assume API server is healthy and we reset the backoff. <del> backoffManager:         wait.NewExponentialBackoffManager(800*time.Millisecond, 30*time.Second, 2*time.Minute, 2.0, 1.0, reflectorClock), initConnBackoffManager: wait.NewExponentialBackoffManager(800*time.Millisecond, 30*time.Second, 2*time.Minute, 2.0, 1.0, reflectorClock), clock:                  reflectorClock, watchErrorHandler:      WatchErrorHandler(DefaultWatchErrorHandler), expectedType:           reflect.TypeOf(expectedType), </del> <ins> backoffManager:    wait.NewExponentialBackoffManager(800*time.Millisecond, 30*time.Second, 2*time.Minute, 2.0, 1.0, reflectorClock), clock:             reflectorClock, watchErrorHandler: WatchErrorHandler(DefaultWatchErrorHandler), expectedType:      reflect.TypeOf(expectedType), </ins> } if r.name == "" {

Currently, reflector uses two backoff managers: One is used for initial list call and other for watch call. Imagine following scenaraion when k8s apiserver is under very heavy load: the beginning, reflector spams k8s apiserver (with exponential backoff) with calls until one of them succeeds (due to APF throtling and receiving ) tries to run request and uses separate backoffManager - so again spams k8s apiserver with requests Watch does not succeed during Etcd compaction window (2.5 - 5 minutes usually) at some point it will receive and go back to point 1, spamming k8s apiserver again with List calls (because first backoffManager for List got reseted after 2 minutes of retries for Watch requests) I propose we combine these two backoff managers into one. If we combine these two backoffManagers into one, it will work like this: At the beginning, reflector spams k8s apiserver (still with exponential backoff) with calls until one of them succeeds Tries to make first watch call instantly (as there is no backoff on this path) If it fails, it will makre retries every (30, 60)s If watch request fails all the times during Etcd compaction window it will go back to List call, but already being throttled with (30, 60)s window. /sig scalability
/cc
/assign
I think the proposed solution makes sense to me. Probably before we start, we should to understand why we ended up with two back off managers by looking into PRs touching them. Maybe the code authors had some specific case in mind.
for thoughts too I agree it makes sense, but I don't know the historical context here.
/sig api-machinery
/triage accepted


Question: # use local dev binaries. # # Assumed vars: <ins> #   KUBE_VERSION </ins> #   KUBE_VERSION_REGEX #   KUBE_CI_VERSION_REGEX # Vars set: <del> #   KUBE_TAR_URL </del> #   KUBE_TAR_HASH #   SERVER_BINARY_TAR_URL #   SERVER_BINARY_TAR_HASH

I suspect we're going through an if/else clause where KUBETARURL isn't defined, and not defaulting it, though the function header seems to indicate it should be set in the callee.
Will take a look.
Is this in Jenkins, or on a local run? The Jenkins push job is borked as well, but it looks like it's for a different reason: Which I'm taking care of in .
Jobs are generating incredibly large log files (300 Megs?) with lots of repeated lines like this: Failure trying to curl release .sha1 /jenkins-master-data/jobs/kubernetes-push-gce-step1-deploy/workspace/kubernetes/hack/e2e-internal/../../cluster/../cluster/../c line 283: KUBETARURL: unbound variable Failure trying to curl release .sha1 /jenkins-master-data/jobs/kubernetes-push-gce-step1-deploy/workspace/kubernetes/hack/e2e-internal/../../cluster/../cluster/../c line 283: KUBETARURL: unbound variable Failure trying to curl release .sha1 I think this needs to be resolved, since it already caused an out-of-disk in Jenkins.
Okiedokie. Yes, this was my fault. silently expects and to be set in . I the extra expectation that be set, but didn't think about how would then also have to set that. Working on a fix.
(For anyone who cares, culprit is .)


Question: existingPIPs  []network.PublicIPAddress expectedPIP   *network.PublicIPAddress expectedID    string <ins> expectedDNS   string </ins> expectedError bool }{ {

Is this a BUG REPORT or FEATURE REQUEST?: What happened: Azure will not update the DNS label of the public IP, when update the service only changing the annotation of the DNS label from empty. annotations: -annotations: "to-update" What you expected to happen: It should update the dns label of the assigned public IP How to reproduce it (as minimally and precisely as possible): It probably caused when applying function 'getPublicIPAddress' in This function returns a bool 'existsPip' to judge the public ip existence. When only the annotations changed, the existsPip still returns true. Then, the reference in line 553 and line 446 step into this bug. pip, existsPip, err := az.getPublicIPAddress(pipResourceGroup, pipName) if err != nil { return false, err } if !existsPip { return true, nil } Firstly, in line 554, it will make function 'isFrontendIPChanged' return false, and then will not run 'ensurePublicIP' in line 691. Secondly, even if 'ensurePublicIP' in line 691 runs, it will detect the public ip exists and make no update to the DNS label in line 446_. Environment: Kubernetes version (use ): Cloud provider or hardware configuration: Azure
/sig azure
/assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
-- what's the status on this?
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity.

Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/open I will take a look at this
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Still not fixed. /reopen /kind bug /priority important-soon /milestone v1.16
Reopened this issue. <detailsIn response to : Still not fixed. /kind bug /priority important-soon /milestone v1.16 Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cloud-provider
i'm working on it.
/assign


Question: package gce import ( <ins> "context" </ins> "fmt" "io" "net/http"

/kind bug What happened: Continuation of . The fix to that bug only takes the first subnetwork in the correct region. However, if the user manual subnets in the region that the cluster exists, it's possible the cloudprovider could choose the wrong subnet. What you expected to happen: If no subnet is specified for a non-legacy network, always take the subnet which was automatically created. /assign /sig network /area platform/gce


Question: "http://metadata.google.internal", "http://169.254.169.254/", "http://metadata.google.internal/", <del> "http://metadata.google.internal/0.1", "http://metadata.google.internal/0.1/", </del> "http://metadata.google.internal/computeMetadata", "http://metadata.google.internal/computeMetadata/v1", // Allowed API versions.

Warning: The v0.1 and v1beta1 metadata server endpoints are deprecated and scheduled for shutdown on September 30, 2020. This test (which runs in most e2e jobs that run on GCE): Schedules a container that runs this: Resulting in a number of legacy metadata endpoints getting hit. To see if this test is going to start failing across all release branches on Sep 30th, we could try disabling the endpoints ahead of time via: In fact and support for expecting different results based on whether is set. My concern is the test expects 200 for some root endpoints, and 403 for disabled endpoints. However the e-mail I received as a GCP customer tells me any legacy endpoints may return 404. I'm inclined to suggest we remove the legacy endpoints from the check, bump the image version, and cherry-pick use of the new image version back to all release-branches. WDYT since you were involved with the earlier PRs /sig auth /sig cloud-provider /provider gcp /sig testing /priority important-soon (I'm tempted to critical-urgent this but it's not drop-everything yet) /milestone v1.20
/kind cleanup
Heyho I wanted to check how it is going? The code freeze is starting 12.11.2020 (about 3-4 weeks from now) and while there is still plenty of time, we want to ensure that each relevant issue has a chance to be considered. As the issue is tagged for 1.20, is it still planned for this release?
Bug Triage here, we're pretty late into code freeze and it doesn't look like there has been much activity recently. I am going to clear the milestone on this issue. /milestone clear
any update here?
Try to cleanup the legacy endpoints in
/milestone v1.21
/reopen This needs to stay open until the new image has been verified to work and has been promoted (ref: )
Reopened this issue. <detailsIn response to : /reopen This needs to stay open until the new image has been verified to work and has been promoted (ref: ) Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage accepted /assign


Question: // NewNamespaceOptions provides an instance of NamespaceOptions with default values func NewNamespaceOptions(streams genericclioptions.IOStreams) *NamespaceOptions { return &NamespaceOptions{ <del> configFlags: genericclioptions.NewConfigFlags(), </del> <ins> configFlags: genericclioptions.NewConfigFlags(true), </ins> IOStreams: streams, }

/kind bug What happened: takes ~11 seconds when there are thousands of CRDs and is around 28KiB. logs the following message once per resource: Removing unnecessary entries from reduces the runtime by around ten seconds. What you expected to happen: only parses once, and therefore doesn't run unnecessarily slowly. How to reproduce it (as minimally and precisely as possible): shows the config being loaded multiple times (assuming you have multiple pods in the current cluster). Alternatively, add 10-15 clusters to and create ~3000 of a cheap resource (eg custom resources or empty config maps). Run on this resource and observe that it is unnecessarily slow. Anything else we need to know?: I've created a CPU profile using the latest CI build of kubectl, which shows ~12 seconds being spent in . Environment: Kubernetes version (use ): v1.12.1 (also Cloud provider or hardware configuration: GKE or minikube --vm-driver=none OS (e.g. from /etc/os-release): Debian Buster-like Kernel (e.g. ): 4.17.0
/sig cli
/assign
Same problem here. I have less resources but more contexts in my kubeconfig file. With 90 pods and ~300 contexts in the kubeconfig it also takes ~10 seconds for a simple kubectl get pods Note: I tried it with kubectl: 1.11.0, 1.12. With kubectl 1.10.5 the kubeconfig was loaded only once.
I'm unable to reproduce this as of the latest master. My kubeconfig is only loaded 4 times, no matter how many resources are loaded.
When I try a kubectl v1.13.0-alpha.3 against unfortunately a 1.9.11 Kubernetes Cluster I still have the problem: `
Maybe the bug is Linux-only? Presumably the kubeconfig location is different on different platforms, so maybe it has a different code path. I can repro on latest master:
I filed to fixed this. You can try to help validate it. Thanks.
Works for me, thanks!


Question: import ( "context" <del> "k8s.io/api/core/v1" </del> <ins> v1 "k8s.io/api/core/v1" </ins> clientset "k8s.io/client-go/kubernetes" "k8s.io/kubernetes/test/e2e/framework" e2eskipper "k8s.io/kubernetes/test/e2e/framework/skipper"

<!-- Please only use this template for submitting enhancement requests --What would you like to be : In block volume tests, wait for the Pod to be deleted instead of letting it be handled by the framework's namespace deletion Why is this needed: I think the regression caused by could have been caught if the tests check for Pod deletion because the Pod object shouldn't be deleted if TearDown fails.
/assign
This issue is not specific to block tests. We should in ideally wait for pod deletion at the end of all types of tests to confirm cleanup logic is functioning
I checked the tests under test/e2e/storage/testsuites/ where DeletePodWithWait is used to wait for the deletion.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten is this still an issue? One way to check is to make a breaking change in cleanup and see if the pull job fails.
I went through all block volume usage in e2e tests: All of them but one wait for their pods to be deleted. I'm fixing the remaining test in


Question: [SERVICE] Flush         5 Grace         120 <del> Log_Level     debug </del> <ins> Log_Level     info </ins> Log_File      /var/log/fluentbit.log Daemon        off Parsers_File  parsers.conf

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --The default fluent-bit configuration for GCE Windows nodes produces unparsed JSON logs and does not forward container logs. Container log output is produced. fluent-bit on a given windows image. a GCE Windows e2e cluster and view the stackdriver output for pods. Kubernetes version (use ): 1.20+ Cloud provider or hardware configuration: GKE/GCE OS (e.g: ): Windows Kernel (e.g. ): Any version Install tools: fluent-bit Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig windows


Question: manager.podStoreSynced = alwaysReady var testJob experimental.Job <del> received := make(chan string) </del> <ins> received := make(chan struct{}) </ins> // The update sent through the fakeWatcher should make its way into the workqueue, // and eventually into the syncHandler.

Ran a test executable with more than 20,000 times and no failures. Any hints as to how to get it to show up? I'm running 100 jobs at a time, so my system is stressed already.
I to am unable to reproduce this. I get similar errors if I reduce the timeout at the top of the file. I spent a bit looking at how to rewrite the tests without the timeout, but nothing jumped out at me. Looking at it though, it's really just a 500ms timeout for the test to complete. I'm just going to remove them. Go has a default timeout of 10 minutes. If it's going to pass, it will pass then. If not, Go will give you a nice traceback where it deadlocked.


Question: } for i := range list.Items { ep := &list.Items[i] <ins> if _, ok := ep.Annotations[resourcelock.LeaderElectionRecordAnnotationKey]; ok { // when there are multiple controller-manager instances, // we observe that it will delete leader-election endpoints after 5min // and cause re-election // so skip the delete here // as leader-election only have endpoints without service continue } </ins> key, err := keyFunc(ep) if err != nil { utilruntime.HandleError(fmt.Errorf("Unable to get key for endpoint %#v", ep))

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG OR Do i miss something important? Kubernetes version (use ): k8s 1.3.5. still exists in 1.6.2 What happened: HA of kube-controller-manager and kube-scheduler is achieved by create endpoint and update it periodically. However, endpoint controller deletes endpoints without corresponding services which is implemented in checkLeftoverEndpoints func: That will result in leader transition. What you expected to happen: checkLeftoverEndpoints func shouldn't delete endpoints used by LeaderElection. How to reproduce it (as minimally and precisely as possible): turn on LeaderElection for kube-controller-manager and kube-scheduler. After five minutes, endpoint controller will delete orphan endpoints. Anything else we need to know: we can add some logic in checkLeftoverEndpoints, for example, not add endpoint in kube-system namespace to queue.
Fixed by
Thanks


Question: createVolume := "CreateVolume" deleteVolume := "DeleteVolume" <del> publishVolume := "NodePublishVolume" </del> <ins> // publishVolume := "NodePublishVolume" </ins> unpublishVolume := "NodeUnpublishVolume" <del> stageVolume := "NodeStageVolume" </del> <ins> // stageVolume := "NodeStageVolume" </ins> unstageVolume := "NodeUnstageVolume" // These calls are assumed to occur in this order for <del> // each test run. </del> <ins> // each test run. NodeStageVolume and // NodePublishVolume should also be deterministic and // only get called once, but sometimes kubelet calls // both multiple times, which breaks this test // (https://github.com/kubernetes/kubernetes/issues/90250). // Therefore they are temporarily commented out until // that issue is resolved. </ins> deterministicCalls := []string{ createVolume, <del> stageVolume, publishVolume, </del> <ins> // stageVolume, // publishVolume, </ins> unpublishVolume, unstageVolume, deleteVolume,

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: Which test(s) are flaking: CSI mock volume storage capacity exhausted, late binding, with topology Testgrid link: Reason for failure: Anything else we need to know: links to appreciated links to specific failures in spyglass appreciated
/assign
According to , there really are extra calls to and that shouldn't be there: The current theory is that this might be related to kubelet assuming that a CSI driver does real mounts, something that the mock driver (and potentially other) CSI drivers don't do. A fix for that is pending in That this test runs into this randomly is caused by the test doing more strict checking of CSI calls than other tests. The flaky test failures could be avoided by relaxing that checking, but that won't fix the underlying issue.
I'm not sure I would pin it on the mount check yet. The broader question is why is MountVolume() getting called twice after the first one returns success.
is it due that, the timeout of CSI transaction or call is very low where CSI driver takes more time than this timeout?