Question: Exec: &api.ExecAction{Command: []string{"touch", file}}, }, } <del> testPodWithHook(podWithHook) </del> <ins> testPodWithExecHook(podWithHook) </ins> }) It("should execute prestop exec hook properly [Conformance]", func() { <del> podWithHook := getLifecycleHookTestPod("pod-with-prestop-exec-hook", </del> <ins> podWithHook := getExecHookTestPod("pod-with-prestop-exec-hook", </ins> // Block forever []string{"tail", "-f", "/dev/null"}, )

https://k8s- Failed: [] Container Lifecycle Hook when create a pod with lifecycle hook when it is http hook should execute poststart http hook properly {E2eNode Suite}
https://k8s- Failed: [] Container Lifecycle Hook when create a pod with lifecycle hook when it is http hook should execute poststart http hook properly {E2eNode Suite}
https://k8s- Failed: [] Container Lifecycle Hook when create a pod with lifecycle hook when it is http hook should execute poststart http hook properly {E2eNode Suite}
https://k8s- Failed: [] Container Lifecycle Hook when create a pod with lifecycle hook when it is http hook should execute poststart http hook properly {E2eNode Suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] Container Lifecycle Hook when create a pod with lifecycle hook when it is http hook should execute poststart http hook properly {E2eNode Suite} Happened on a presubmit run in .
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] Container Lifecycle Hook when create a pod with lifecycle hook when it is http hook should execute poststart http hook properly {E2eNode Suite} Happened on a presubmit run in .
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] Container Lifecycle Hook when create a pod with lifecycle hook when it is http hook should execute poststart http hook properly {E2eNode Suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.

[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] Container Lifecycle Hook when create a pod with lifecycle hook when it is http hook should execute poststart http hook properly {E2eNode Suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] Container Lifecycle Hook when create a pod with lifecycle hook when it is http hook should execute poststart http hook properly {E2eNode Suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
the for this issue looks pretty good. We could probably close this issue.
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] Container Lifecycle Hook when create a pod with lifecycle hook when it is http hook should execute poststart http hook properly {E2eNode Suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
Looks like more recent flakes:
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
Adding labels for team node as is failure is part of the node e2e tests. All issues must be labeled either release blocker or non release blocking by end of day 18 November 2016 PST. Given the test flakes very infrequently and doesn't seem to be associated with a particular code change Could either or someone from please advise as to whether you believe this issue is actually a release blocker and I'll update the issue. Any updates from you would also be greatly appreciated.Thanks! cc:
The test was introduced by which causes some flakiness: Re-assign this to Lantao, I think the issue should be in the test itself, and non-resource-blocker. But I temporarily marked it as release blocker after your triage. Thanks!

This is a issue in the test itself. I've sent a fix for this
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.


Question: # Network Policy plugin specific settings. NETWORK_POLICY_PROVIDER="${NETWORK_POLICY_PROVIDER:-none}" # calico <ins> # Should the kubelet configure egress masquerade (old way) or let a daemonset do it? NON_MASQUERADE_CIDR="0.0.0.0/0" </ins> # How should the kubelet configure hairpin mode? HAIRPIN_MODE="${HAIRPIN_MODE:-promiscuous-bridge}" # promiscuous-bridge, hairpin-veth, none

https://k8s- is failing to run the follow up e2es because the ip-masq-agent daemonset is scheduling pods that get stuck in a pending state on 1.5 nodes. This short-circuits the logic for the skewed e2es, and they never run. from https://k8s-
So I see that in 1.7.0 we start setting the non-masq-cidr to 0/0 gce/config-"0.0.0.0/0" We set the addon if non-masq-cidr is 0/0 if [[ "${NONMASQUERADECIDR:-}" == "0.0.0.0/0" ]]; then setup-addon-manifests "addons" "ip-masq-agent" fi Also the agent cannot run on nodes prior to 1.7 so this seems to be correct. Once the nodes are upgraded to 1.7 then the ip-masq agent should be running and should get out of the pending state. I think this is just a general problem with the way addons are handled. I may want to enable a new addon when a master is upgraded to 1.7 by default. That addon may be a DaemonSet that wants to run on nodes meeting a certain criteria. That criteria may not be met in 1.5 but may be met when they are updated to 1.7. A few things that come to mind to fix this: -Fix the test's expectations to allow this until the nodes are updated to 1.7. -rollback the PR that made this the default: -set NONMASQUERADECIDR=10.0.0.0/8 in the test properties so that the ip-masq-agent doesn't get .
It seems like the most conservative way forward is to revert the PR (). Customers who want to use the feature can set the variable () during to be able to get the feature. What do you guys think?
I think I asked the similar question related to ip-masq-agent, but my question was dropped. At this stage, I think reverting the preferred fix.

ip-masq-agent was a daemonset introduced in 1.7 which I questioned at The problem is that after master is upgraded to 1.7, addons is trying to schedule ip-masq-agent daemonset to every node no matter which Kubelet version the node is on. The problem is that ip-masq-agent can only run on node with 1.7 release.
Even if we fix the test it doesn't fix GKE or any other environment where the master is upgraded to 1.7 and the nodes are < 1.7. I think the real problem here is that there is no distinction between staging an addon and enabling an addon. Essentially staging = running. So we end up adding labels to nodes as a way of keeping the DaemonSet disabled until those labels are to nodes. While that works fine for addons that are DaemonDets, it doesn't work for deployments and replica sets. I feel like this is going to happen again regardless of what we do here with ip-masq-agent. It also creates a secondary step. Who applies those addons? What if they forget? what if it's deleted?
Fixed by (the formatting wasn't quite right in the IR to auto-close this issue).
got reverted as it broke tests.


Question: s.StartTLS() defer s.Close() <del> client, err := newGOAWAYClient(nil) </del> <ins> // create the http client tlsConfig := &tls.Config{ InsecureSkipVerify: true, NextProtos:         []string{http2.NextProtoTLS}, } tr := &http.Transport{ TLSHandshakeTimeout: 10 * time.Second, TLSClientConfig:     tlsConfig, MaxIdleConnsPerHost: 25, } if err := http2.ConfigureTransport(tr); err != nil { t.Fatalf("failed to configure http transport, err: %v", err) } client := &http.Client{ Transport: tr, } </ins> if err != nil { t.Fatalf("failed to set-up client, err: %v", err) }

Before merged, TestClientReceivedGOAWAY failed with data race on . However, as far as I know, it's not the root cause of TestClientReceivedGOAWAY flake, because dial a new connection only when client send a new request, and in this case, requests are sent one by one. In other words, if data race on , then the additional unexpected connection set up, which means the test case or the configuration has problem. After merged, TestClientReceivedGOAWAY still failed in many jobs, such as , , which may be a good proof of my suspect. And I got a failed job with data race issue that read at (main testing goroutine) and write at , which told us the http client try to set-up a new connection even when we had already finished all requests(may caused by goroutine leak/block in http client). I'm still debug the root cause of TestClientReceivedGOAWAY flake, for now, I doubt that I used an incorrect http2 server/client configuration( or SETTINGSMAXCONCURRENT_STREAMS limited), or the dial function ignored context.Context which will cause more than one goroutine dial the connection when the first canceled(not found any where will cancel the requests). I still can not reproduce this failed case and it's hard to find the root cause. Couple of days may be needed to find the root cause, and I will post a PR to skip TestClientReceivedGOAWAY before I solved the problem. Sorry for any inconvenient.
/assign /sig testing
cc
Thank you for working on this!! How often does it flake? I'm just wondering if we should disable it in the mean time to avoid angry contributors.
On my PC, about 1/1000. I will post a PR to disable it for now.
Thank you so much! That's not a bad rate at all, but we run a lot of tests. This is super helpful!
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or .

/lifecycle stale
/lifecycle stale I'm still working on it, a new golang version(1.16 or 1.15.7+) release required
/sig api-machinery /remove-sig testing AFAICT this is specific to these tests, not the test framework or CI infrastructure.
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
the tests re-enabled in started flaking again immediately Example failure: /milestone v1.21 the tests should be deflaked or disabled for 1.21
/cc
/triage accepted
+1 in recent PR tests
Sorry for my slow response. I'm working on it, looks like I need to debug golang HTTP client again :-P
Shall we revert in the meantime?
No, has already fixed by golang 1.15.8 I think. This one is caused by another problem.
I will disable TestClientReceivedGOAWAY case again if I can not find the root cause this night.
Can not reproduce on my PC, disable this test case before I figure out the root cause:
I run this test case with this command and can not reproduce. Does any thing different with command ?
moving out of the milestone and lowering priority becuase merged, leaving open because the fundamental issue is unresolved /milestone clear
will do something comparable under make. There's a bunch of stuff in there, generally it's not relevant for a unit test bug like this but the answer to what exactly it does is not easily concise. Note that CI may have more load than your local machine running it once, and also currently runs as root.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/lifecycle active /assign ref:


Question: package etcd import ( <ins> "fmt" </ins> "github.com/GoogleCloudPlatform/kubernetes/pkg/api" kubeerr "github.com/GoogleCloudPlatform/kubernetes/pkg/api/errors" etcderr "github.com/GoogleCloudPlatform/kubernetes/pkg/api/errors/etcd"

The use of AtomicUpdate by genericetcd doesn't handle conflicting writes by rejecting that to a user. It should. Eventually, we may want to hide write conflicts to /pods when status has changed, and to spec when /pods/status changes, but that requires us to have two version counters and that's not worth the effort. Fix incoming.
Also we need a unit/integration test for this.


Question: echo "VAGRANT_DEFAULT_PROVIDER='${VAGRANT_DEFAULT_PROVIDER:-}'" echo "KUBELET_TOKEN='${KUBELET_TOKEN:-}'" echo "KUBE_PROXY_TOKEN='${KUBE_PROXY_TOKEN:-}'" <ins> echo "MASTER_EXTRA_SANS='${MASTER_EXTRA_SANS:-}'" </ins> awk '!/^#/' "${KUBE_ROOT}/cluster/vagrant/provision-network.sh" awk '!/^#/' "${KUBE_ROOT}/cluster/vagrant/provision-minion.sh" ) > "${KUBE_TEMP}/minion-start-${i}.sh"

On latest Kubernetes version Release 0.20.1 Candidate I get the following when you setup cluster with "./cluster/kube-". This is on Mac OSX with VirtualBox, Vagrant Any ideas, thanks .... ==minion-3:  INFO: Running daemons_running() ==minion-3:  INFO: Salt installed! Wrote config for vagrant to Each machine instance has been created/updated. Now waiting for the Salt provisioning process to complete on each machine. This can take some time based on your network, disk, and cpu speed. It is possible for an error to occur during Salt provision of cluster and this could loop forever. Validating master Validating minion-1 Validating minion-2 Validating minion-3 ..... Waiting for each minion to be registered with cloud provider error: couldn't read version from server: Get https://10.245.1.2/api: dial tcp 10.245.1.2:443: connection refused kubernetes user1$ get nodes error: couldn't read version from server: Get https://10.245.1.2/api: dial tcp 10.245.1.2:443: connection refused
the same for the first time when i ran v0.20.1 (the published one)
here is the installation log on my core i5-5200U ubuntu 14.04 3.16 kernel this is that console says
See my comment on
I think is probably right. There's a 0.20.2 brewing already (and we should have it out later today), so hang tight.
I just tried this also with Release 0.20.2 Candidate on Ubuntu 14.04 with VirtualBox, Vagrant and I am seeing:
- Is there someone on your team that could take a look?
Honestly no, everyone is busy, and working on either cluster or network related issue. But I talked to briefly, he kindly agree to take a look. The cloud provider here is Vagrant, if you could reproduce the issue, that would be huge help. Feel free to re-assign it to me if you are blocked. Thanks!
cc/ since he looked into vagrant issue lately. cc/ on Vagrant.

+1, 0.20.2 still has this issue
+1
The fix for this issue is this PR:
Assigning issue to me since I am focused today on closing out remaining Vagrant setup issues.
Sigh, I'm sorry. So busy working through unassigned PRs I forgot to work through my own queue.
I was running into the same issue and cherry picked your changes in , but still have the same issue. Prior to those changes the kubelet never started on the master. With the changes it does start, but I still get: It looks like also needed before vagrant works correctly.
Yep, I picked and then merged and it started to work again... just my 2 cents
this error is gone in 0.21.0 rc ()
I am using Mac OSX 10.10.5 / VMWare Fusion / bento/fedora-21/ Vagrant 1.7.4 / and built kubernetes from master (GitVersion:"v1.1.0-alpha.1.687+") and I am seeing this issue when I run ./cluster/kube- Each machine instance has been created/updated. Now waiting for the Salt provisioning process to complete on each machine. This can take some time based on your network, disk, and cpu speed. It is possible for an error to occur during Salt provision of cluster and this could loop forever. Validating master Connection to 192.168.100.177 closed. ..Validating minion-1 ............. Waiting for each minion to be registered with cloud provider error: couldn't read version from server: Get https://10.245.1.2/api: dial tcp 10.245.1.2:443: connection refused
I'm running into a similar issue with a Kubernetes cluster on AWS. I set it up last week and now can't connect to the master node: oamasood kubectl cluster-info error: couldn't read version from server: Get https://11.11.111.111/api: dial tcp 11.11.111.111:443: connection refused Any help would be appreciated.
I find that if you wait for the master and minion to calm down on activity after a bit (~5 minutes) the kubectl commands work, however the problem is I don't see any nodes running.

I have re-installed virtual box, made sure my .kube directory is clean and I am now pulling just the stock vagrant box cube fedora 21 and kubernetes 1.0.4 release. I do not see any docker processes running on my minion. Other than that it seems that the cluster is running: $ get cs NAME STATUS MESSAGE ERROR controller-manager Healthy ok nil scheduler Healthy ok nil etcd-0 Healthy {"health": "true"} nil $ cluster-info Kubernetes master is running at https://10.245.1.2 KubeDNS is running at https://10.245.1.2/api/v1/proxy/namespaces/kube-system/services/kube-dns KubeUI is running at https://10.245.1.2/api/v1/proxy/namespaces/kube-system/services/kube-ui yggdrasil:kubernetes jonathantronson$ vagrant ssh master Last login: Wed Sep 16 23:49:51 2015 from 10.0.2.2 [vagrant ~]$ sudo docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES "/bin/sh -c '/usr/loc" 50 minutes ago Up 50 minutes k8skube-apiserver.2abef58bkube-apiserver-kubernetes-masterkube-system496e220b4704d4b390883b4b90aa66354e41e61f "/bin/sh -c '/usr/loc" 50 minutes ago Up 50 minutes k8setcd-container.fa2ab1d9etcd-server-kubernetes-masterkube-system6ef1b07678064a7edcdbec6f0570e91759d753f9 "/bin/sh -c '/usr/loc" 53 minutes ago Up 53 minutes k8skube-scheduler.ea78c200kube-scheduler-kubernetes-masterkube-system3d1ced0542467b566546021cb74d9bdf9550be6a "/bin/sh -c '/usr/loc" 53 minutes ago Up 53 minutes k8skube-controller-manager.fd0f0b82kube-controller-manager-kubernetes-masterkube-system2f59d05ecb2212d0379dc35dbe40d5216292ac06 "/pause" 54 minutes ago Up 54 minutes k8sPOD.

e4cc795kube-scheduler-kubernetes-masterkube-system3d1ced0542467b566546021cb74d9bdf62e6c529 "/pause" 54 minutes ago Up 54 minutes k8sPOD.e4cc795kube-controller-manager-kubernetes-masterkube-system2f59d05ecb2212d0379dc35dbe40d52109062f04 "/pause" 54 minutes ago Up 54 minutes k8sPOD.e4cc795etcd-server-kubernetes-masterkube-system6ef1b07678064a7edcdbec6f0570e9173d73246b "/pause" 54 minutes ago Up 54 minutes k8sPOD.e4cc795kube-apiserver-kubernetes-masterkube-system496e220b4704d4b390883b4b90aa6635c1bfe2c3 [vagrant ~]$ sudo docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
The only errors in my minion salt log are: 2015-09-16 23:40:22,049 [][ERROR ][2668] Command 'systemctl status docker.service' failed with return code: 3 2015-09-16 23:40:22,049 [][ERROR ][2668] output: * docker.service - Docker Application Container Engine Loaded: loaded (/usr/lib/systemd/system/docker.service; disabled) Active: inactive (dead) Docs:
Hi, I am running the latest version of kubernetes tar ball (v1.1.3) and facing issues with ./kube- I was unable to fix the issue. Any help would be appreciated. cluster "vagrant" set. user "vagrant" set. context "vagrant" set. switched to context "vagrant". Wrote config for vagrant to Each machine instance has been created/updated. Now waiting for the Salt provisioning process to complete on each machine. This can take some time based on your network, disk, and cpu speed. It is possible for an error to occur during Salt provision of cluster and this could loop forever. Validating master Connection to 127.0.0.1 closed.

Validating minion-1 Waiting for each minion to be registered with cloud provider error: couldn't read version from server: Get https://10.245.1.2/api: dial tcp 10.245.1.2:443: connection refused Thank You
Mac 10.11.2 installing kubernetes 1.1.3 got same issue: I think the fix didn't catch the root cause.
same issues on 1.1.7 cluster "vagrant" set. user "vagrant" set. context "vagrant" set. switched to context "vagrant". Wrote config for vagrant to Each machine instance has been created/updated. Now waiting for the Salt provisioning process to complete on each machine. This can take some time based on your network, disk, and cpu speed. It is possible for an error to occur during Salt provision of cluster and this could loop forever. Validating master ^ to 127.0.0.1 closed. Validating minion-1 ...^ Waiting for each minion to be registered with cloud provider error: couldn't read version from server: Get https://10.245.1.2/api: dial tcp 10.245.1.2:443: connection refused
when I rerun the cluster/kube-, the error still exist, and can't get nodes/services etc status. The first time, it works. weird.
can u help to take a look this issues? thanks
- can you open a new issue and describe what host environment you are using? Are you testing from master or a branch? cc me on the issue.
I used released version 1.1.7. btw, I tried several times. even at the last step failed. I can run get pods/services etcs. seems during verify stage. it will search local kubectl files? I am not sure. as on my Mac , there have some mismatch version kubectl file.
I experience the very same issue on AWS and, usually, I can't connect to my cluster anymore the day after I had set it up - which is a huge blocker. Here's the output from :
I expreience the same issue on Ubuntu.
The issue disappeared for me with the following version:
I encounter the same problem,can you give me a detail configuration of kuberenetes?

ubuntu /opt/bin/kubectl version Client Version: version.Info{Major:"1", Minor:"1", GitVersion:"v1.1.8", GitCommit:"", GitTreeState:"clean"} error: couldn't read version from server: Get http://localhost:8080/api: dial tcp 127.0.0.1:8080: connection refused


Question: func (ts *fakeTokenSource) Token() (*azureToken, error) { return &azureToken{ token:       newFackeAzureToken(ts.accessToken, ts.expiresOn), <ins> environment: "testenv", </ins> clientID:    "fake", tenantID:    "fake", apiserverID: "fake",

What happened: AAD support in kubectl does not work for sovereign cloud The 'environment' parameter was introduced to support sovereign cloud, but it works in first round. The 'environment' value is not persisted, so follow-up requests will fail. What you expected to happen: 'environment' parameter persisted in kubectl config file. How to reproduce it (as minimally and precisely as possible): Use kubectl to connect to Azure sovereign cloud, make the token expire, and do auth. Anything else we need to know?: Environment: Kubernetes version (use ): 1.11 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
There are no sig labels on this issue. Please add a sig label by either: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. See the . The in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: .:53 { errors health <ins> ready </ins> kubernetes __PILLAR__DNS__DOMAIN__ in-addr.arpa ip6.arpa { pods insecure <del> upstream </del> fallthrough in-addr.arpa ip6.arpa ttl 30 }

Which jobs are failing: Multiple jobs on . Which test(s) are failing: on the following jobs: Since when has it been failing: Since 5/17. Testgrid links: See above. Reason for failure: All the failures mentioned above seem to have to have the same root cause, namely failure while waiting for CoreDNS to be ready. Example: /sig testing /priority critical-urgent /kind failing test /milestone v1.15 /cc
Some of this (possible all of it) is due to , which is dependent on being merged.
/sig cluster-lifecycle
Not the gce jobs thoght. Only the kinder job. So the gce jobs are failing with a problem outside of kubeadm/kinder but possibly still related to the corefile plugin change
/assign (At kubecon this week)
The gce failures appear to show the same behavior, but with kube-up instead of kubeadm. I.e. it appears that kube-up may also be retaining old configs when upgrading/downgrading. If thats true, then we may need to integrate the migration tools into kube-up as well.
I have opened which aims to fix all the failing tests except , which will be fixed via
Great job so far and Would it be possible to ask for the open PRs to be merged and/or a solution found soon? We have these tests failing for quite some time and code freeze coming up in the next days. Thank you in advance for your efforts!
thanks for the heads up. let me know if you don't have the time for the refactor that i requested and i can try to take over there. the alternative is to rollback the coredns version which i assume is not ideal, due to a variety of fixes in the new version.
is there any more work planned for this issue? I see that some of the kinder jobs are still failing after merged:
merging and will resolve and close this issue.
Thank you for the update !
updates: kinder tests were fixed via
/unassign i was here for: it's a bit flaky (which is due to the tooling), but green.
Since and we're moved to 1.16, and we're waiting for to merge, can we move this issue to 1.16?
? is that the right PR?

Good catch, I meant
The GCE test failures were fixed by and /close
Closing this issue. <detailsIn response to : The GCE test failures were fixed by and /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: if err := autoConvert_v1_NodeSpec_To_core_NodeSpec(in, out, s); err != nil { return err } <del> // If both fields (v1.PodCIDRs and v1.PodCIDR) are provided, then test v1.PodCIDR == v1.PodCIDRs[0] </del> <ins> // If both fields (v1.PodCIDRs and v1.PodCIDR) are provided and differ, then PodCIDR is authoritative for compatibility with older clients </ins> if (len(in.PodCIDR) > 0 && len(in.PodCIDRs) > 0) && (in.PodCIDR != in.PodCIDRs[0]) { <del> return fmt.Errorf("conversion Error: v1.PodCIDR(%v) != v1.CIDRs[0](%v)", in.PodCIDR, in.PodCIDRs[0]) </del> <ins> out.PodCIDRs = []string{in.PodCIDR} </ins> } // at the this point, autoConvert copied v1.PodCIDRs -> core.PodCIDRs

What happened: The cluster master in a Kubernetes v1.14.10 cluster was upgraded to V1.16.7. The process of upgrading nodes is under way and has been spread over a week so far. The cluster owner reported seeing a high percentage of pod status PATCH failures. The apiserver logs show errors like: The errors are all from the v1.14 nodes and have all been for pods associated with DaemonSets. We don't yet know why pods' IPs are changing, but it appears that pod status patches that include a changed result in an object where contains the old IP, and has the new IP. We believe this is because was introduced in Kubernetes v1.15 and the patch request from a v1.14 node results in both and being present in the result. If you aren't monitoring for request failures it may not obvious that any error is occurring: The pods continue to show their last successfully reported state. Deleting such pods, however, results in the pod being stuck in a state. We are only seeing this for DaemonSet pods - I suspect because whatever resulted in pod IPs changing also causes non-DaemonSet pods to be deleted and recreated. We have a number of cluster still on v1.14 that we expect to upgrade to v1.15 or v1.16. What you expected to happen: Kubernetes v1.14 nodes should be able to update pod status during the transition from v1.14 to v1.16 even if pod IPs change during that time. How to reproduce it (as minimally and precisely as possible): We have recreated this in the IBM Cloud Kubernetes Service by "reloading" nodes. The reload process leaves the Kubernetes node object in place, and essentially reloads the image with the proper kubeconfig to rejoin the cluster. The cluster I was debugging does not appear to have had nodes "reloaded" - something else is triggering the pod IP changes. Create a v1.14 cluster Deploy a DaemonSet (anything with pod networking should suffice) Upgrade cluster master to v1.16 Reload a v1.14 node. One the node is back up, shows the pod's network IP has changed. Kubelet and apiserver logs show the errors describe earlier. Anything else we need to know?

: Environment: Kubernetes version (use ): v1.14.x (nodes) and v1.16.7 (master) Cloud provider or hardware configuration: IBM Cloud Kubernetes Service OS (e.g: ): 18.04.4 LTS Kernel (e.g. ): 4.15.0-76-generic Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig api-machinery
/sig network
cc
/priority critical-urgent if it's possible for podIPs to change and the server defaults to , then the API server has to treat the singular pod IP as authoritative if there's a mismatch or the patch case breaks
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/remove-sig api-machinery


Question: } func getPhysicallyInstalledSystemMemoryBytes() (uint64, error) { <del> var physicalMemoryKiloBytes uint64 if ok := getPhysicallyInstalledSystemMemory(&physicalMemoryKiloBytes); !ok { </del> <ins> // We use GlobalMemoryStatusEx instead of GetPhysicallyInstalledSystemMemory // on Windows node for the following reasons: // 1. GetPhysicallyInstalledSystemMemory retrieves the amount of physically // installed RAM from the computer's SMBIOS firmware tables. // https://msdn.microsoft.com/en-us/library/windows/desktop/cc300158(v=vs.85).aspx // On some VM, it is unable to read data from SMBIOS and fails with ERROR_INVALID_DATA. // 2. On Linux node, total physical memory is read from MemTotal in /proc/meminfo. // GlobalMemoryStatusEx returns the amount of physical memory that is available // for the operating system to use. The amount returned by GlobalMemoryStatusEx // is closer in parity with Linux // https://www.kernel.org/doc/Documentation/filesystems/proc.txt var statex MemoryStatusEx statex.Length = uint32(unsafe.Sizeof(statex)) ret, _, _ := procGlobalMemoryStatusEx.Call(uintptr(unsafe.Pointer(&statex))) if ret == 0 { </ins> return 0, errors.New("unable to read physical memory") } <del> return physicalMemoryKiloBytes * 1024, nil // convert kilobytes to bytes } func getPhysicallyInstalledSystemMemory(totalMemoryInKilobytes *uint64) bool { ret, _, _ := syscall.Syscall(procGetPhysicallyInstalledSystemMemory.Addr(), 1, uintptr(unsafe.Pointer(totalMemoryInKilobytes)), 0, 0) return ret != 0 </del> <ins> return statex.TotalPhys, nil </ins> }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Windows uses GetPhysicallyInstalledSystemMemory to get physical memory installed on Windows machine and reports it as part of node metrics to heapster, etc. On VMWare Fusion, GetPhysicallyInstalledSystemMemory fails to read data from SMBIOS firmware tables and returns error code 0x0d. That causes on Windows to bail out. It is also reported on Virtualbox. What you expected to happen: Windows should be able read to physical memory on the system and shouldn't exit. How to reproduce it (as minimally and precisely as possible): Run on VMWare Fusion or Virtualbox . Anything else we need to know?: Environment: Kubernetes version (use ): latest Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Windows Kernel (e.g. ): Install tools: Others: VMWare Fusion or Virtualbox
/sig windows /sig node
this will likely get fixed after 1.9 is released
Sure, I am fixing it. Unfortunately it missed the 1.9 train.
Hi, any idea on when this might get fixed? Thanks!
It is fixed in Got test failure in pull-kubernetes-e2e-kops-aws yesterday but not before. I will look into it shortly.
thanks for the swift reply! I need to learn some new (for me) github skills to get that PR.
i just approved the PR again. we should be good to go. can you please check


Question: klog.V(5).InfoS("Restoring iptables", "rules", string(proxier.iptablesData.Bytes())) err = proxier.iptables.RestoreAll(proxier.iptablesData.Bytes(), utiliptables.NoFlushTables, utiliptables.RestoreCounters) if err != nil { <del> klog.ErrorS(err, "Failed to execute iptables-restore", "rules", string(proxier.iptablesData.Bytes())) </del> <ins> if pErr, ok := err.(utiliptables.ParseError); ok { lines := utiliptables.ExtractLines(proxier.iptablesData.Bytes(), pErr.Line(), 3) klog.ErrorS(pErr, "Failed to execute iptables-restore", "rules", lines) } else { klog.ErrorS(err, "Failed to execute iptables-restore", "rules", string(proxier.iptablesData.Bytes())) } </ins> metrics.IptablesRestoreFailuresTotal.Inc() // Revert new local ports. utilproxy.RevertPorts(replacementPortsMap, proxier.portsMap)

my k8s cluster version is 1.17.x this cluster has 10 master only 2 nodes get problem. it make my svc sometime can usesometime can timeout what can i do can resolve this problem ? this 2 node's svc is nodeportType
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
you can increase kube-proxy verbosity adding flag and you should get more information about the cause of the error Kubernetes 1.17 is EOL, so you should upgrade to 1.19 at least
/sig network
hello bromy cluster is use kubeadm init .so what should i do .i know this is use configmap but in this i can not found how edit this configmap is availableabout upgrade this cluster .this time is impossible
about stack. i only get this log.
To increase the verbosity on kube-proxy on a kubeadm deployment you can do and wait until pods are restarted. Despite that, if there is a bug, it will not be backported to 1.17, so you have to go to a supported version first
I swear we code to dump the ruleset when this happens, but I don't see it any more.
I'm going to re-purpose this as a feature :)
Hi, I would like to pick this up. Can someone give me some code pointers on where to start?
Let me know if I can help in anyway.
/assign
I'd start with - It's worth thinking about the cleanest solution here, but maybe something like: When you get an error from exec, parse the error for and if it matches, return a . Then let the callers check for a by interface assertion, extract the Line(), extract their input's data at -3 and +3 from that line and log it. E.g. You'll proably want a helper func to do that, which could live in that same iptables package. And a test.
/assign
hi everyone! Is anyone doing this?

I am willing to do this
at least I didn't see any PR from , in case of collision please sync between both to avoid duplicate efforts
hi did you work on it?
Hi apologies I was busy with some other work. You must please continue.
thanks
/assign
I'm working on this now, but I'm stuck in here, I got the line, and then how I extract at -3 and +3 from that line?
you may need to write a helper function to extract them from the logs. my 2 cents.. here is my version for reference. you could add some tests, to cover the boundary conditions. to play around :)


Question: attribute.Stringer("client", &lazyClientIP{req: req}), attribute.String("protocol", req.Proto), attribute.Stringer("resource", &lazyResource{req: req}), <ins> attribute.Stringer("scope", &lazyScope{req: req}), </ins> attribute.String("url", req.URL.Path), attribute.Stringer("user-agent", &lazyTruncatedUserAgent{req: req}), attribute.Stringer("verb", &lazyVerb{req: req}),

One of the use-cases for APIServer tracing would be to try and debug SLO-violating requests served by the API Server. Right now, it adds just some basic attributes: For API Server , SLO-related metrics have the following labels: "verb", "dryrun", "group", "version", "resource", "subresource", "scope", "component", "code" We should add those labels to at least one of the spans created by API Server requests. It isn't a graduation requirement for , but would make the feature more helpful.
/sig instrumentation
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
We might be able to do something using RequestInfo, similar to how the dropped request metric is recorded:


Question: <ins> /* Copyright 2014 Google Inc. All rights reserved. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package registry import ( "github.com/GoogleCloudPlatform/kubernetes/pkg/api" "github.com/GoogleCloudPlatform/kubernetes/pkg/labels" ) type MockPodRegistry struct { err  error pod  *api.Pod pods []api.Pod } func MakeMockPodRegistry(pods []api.Pod) *MockPodRegistry { return &MockPodRegistry{ pods: pods, } } func (registry *MockPodRegistry) ListPods(query labels.Query) ([]api.Pod, error) { if registry.err != nil { return registry.pods, registry.err } var filtered []api.Pod for _, pod := range registry.pods { if query.Matches(labels.Set(pod.Labels)) { filtered = append(filtered, pod) } } return filtered, nil } func (registry *MockPodRegistry) GetPod(podId string) (*api.Pod, error) { return registry.pod, registry.err } func (registry *MockPodRegistry) CreatePod(machine string, pod api.Pod) error { return registry.err } func (registry *MockPodRegistry) UpdatePod(pod api.Pod) error { return registry.err } func (registry *MockPodRegistry) DeletePod(podId string) error { return registry.err } </ins>

Which jobs are failing: cip-postsubmit-push-to-staging (cip-postsubmit-push-to-staging) Since when has it been failing: This is a postsubmit of repository. Testgrid link: Reason for failure: Anything else we need to know: /cc /priority critical-urgent /milestone v1.18
So this is the "create a staging image of the promoter upon a merge to its master branch" job so that I don't have to do it manually myself. As for the failure, I am probably doing something wrong but from the logs it's hard to tell what I'm doing wrong. I don't think this is critical as it is not blocking the official promoter or anything.
I think the error could be that the service account for doesn't have creds to push to the buckets. I'll post another update later.
Thanks for the quick response. So, since this job is not critical, can we removed it from testgrid or is there a good reason to keep it there and just ignore the alerts?
I should be able to fix it this week; if that deadline passes then we can move it to another testgrid dashboard.
/sig release
/reopen Let's observe the job first before closing.
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Do we have to rerun the job to make sure that it's fixed or wait for the next PR to be merge?
The behavior I have observed is that the job will re-run for each PR merge. I am now seeing that the latest build is failing due to a MemoryError when joining the heaviest Docker layer that contains gcloud: I am going to change the machine settings in another PR; stay tuned.
Looks like the last couple PRs did the trick. It's green now!
Thanks ! /close
Closing this issue. <detailsIn response to : Thanks ! Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: } return false } <ins> //GetQuotaServiceType returns ServiceType if the service type is eligible to track against a quota, nor return "" func GetQuotaServiceType(service *api.Service) api.ServiceType { switch service.Spec.Type { case api.ServiceTypeNodePort: return api.ServiceTypeNodePort case api.ServiceTypeLoadBalancer: return api.ServiceTypeLoadBalancer } return api.ServiceType("") } </ins>

master-informing: capz-windows-containerd-master capz-e2e.[It] Conformance Tests conformance-tests ! 2023-03-10 04:56:51 +0000 UTC No response /sig testing
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : ### Which jobs are failing? ### Since when has it been failing? /sig testing Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig windows /triage accepted
no major change between passing test on 3/9 17:55 PST and failing test on 3/9 20:56 PST
containerd logs have error about failing to unpack azure-cloud-manager on both nodes:
kubelet logs confirm this:
looks to be related to bumping capz to 1.8 which uses the cloud-provider oot image: caused by upgrading to 1.8 capz in combination with using OOT
This looks to be affecting our release jobs as well:
looks like the image being built is linux only () and is being pulled by windows. When setting up the snapshot in containerd it is failing (as it is missing files required by windows)
For the passing test the image is multi-arch manifest, so the windows image pulls properly (azure-cloud-node-manager:)
There is a missing env in the see


Question: return "" } <ins> func commaSeparatedHeaderValues(header []string) []string { var parsedClientProtocols []string for i := range header { for _, clientProtocol := range strings.Split(header[i], ",") { if proto := strings.Trim(clientProtocol, " "); len(proto) > 0 { parsedClientProtocols = append(parsedClientProtocols, proto) } } } return parsedClientProtocols } </ins> // Handshake performs a subprotocol negotiation. If the client did request a // subprotocol, Handshake will select the first common value found in // serverProtocols. If a match is found, Handshake adds a response header

What happened: I'm trying to establish a SPDY connection to exec into a pod. I sent this header: X-Stream-Protocol-Version: , , , The request failed with Forbidden - unable to upgrade: unable to negotiate protocol: client supports [, , , ], server accepts [ ] It appears that the server is not parsing the list out of the HTTP header and is treating it as a single value. What you expected to happen: Protocol negotiation should succeed. RFC2616 says: I can't simply send a multiple headers, because the HTTP library I'm using has no mechanism for sending separate headers as opposed to combining them. Kubernetes should parse the header whether it's given as separate headers or a list. Anything else we need to know?: I'm having to use SPDY because the web sockets implementation of exec . Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2020-02-11T18:14:22Z", GoVersion:"go1.13.6", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-05-16T16:14:56Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: Azure VMOS (e.g: ): NAME="Ubuntu" VERSION="16.04.6 LTS (Xenial Xerus)" ID=ubuntu IDLIKE=debian PRETTYNAME="Ubuntu 16.04.6 LTS" VERSIONID="16.04" HOMEURL="" SUPPORTURL="" BUGREPORTURL="" VERSIONCODENAME=xenial UBUNTUCODENAME=xenial Kernel (e.g. ): Linux AdamArisDev 4.15.0-1063-azure -Ubuntu SMP Fri Nov 8 09:30:20 UTC 2019 x8664 x8664 x8664 GNU/LinuxOthers: My cluster is running in minikube v1.1.0.

/sig api-machinery
Thinking of the following change:
Since none of the accepted protocols include a comma, we should probably just iterate over all the headers, split by , trim each resulting item, omitting empty strings when computing clientProtocols, regardless of the number of headers.
Iterate over
Thanks for the quick update. I don't know what the expected HTTP behavior is when you use multiple headers and also have lists in each header, but if the same rule applies that multiple headers should be treated the same as one header with the items separated by commas, then presumably: Foo: a, b Foo: c, d should be the same as Foo: a, b, c, d in which case suggestion of basically moving the split into the inner loop would be more correct. In any case, I suppose a bug fix of this nature would only affect newer versions of Kubernetes? (I don't know if you do bug-fix-only patches to older versions.) I guess as a workaround I could try to catch the Forbidden error, look for the "unable to negotiate protocol" text, and switch to using only a single header value for older Kubernetes versions. Or is there a better approach?
That said, presumably this could also be handled in a lower layer - in the HTTP layer where headers are parsed - which might fix similar bugs with other headers that might exist elsewhere. (I think the only general exception to the equivalency rule is the Set-Cookie header because in practice cookies can contain commas but aren't quoted.)
That's correct. Given all the currently supported protocols are available in all supported kubernetes versions (the most recent protocols were in in v1.2.0 and in in v1.4.0), I'd probably just pick the specific protocol you want. If you wanted to detect a protocol failure, the X-Accepted-Stream-Protocol-Versions header is sent on a forbidden response due to protocol negotiation failure.
That's good to know. I can just stick to v4, then, since I don't expect to need anything that old. :-) Thanks.
/reopen I agree, should this fix be more general? Some other http headers have also the comma problems.
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.

</details
I would not recommend that, as not all headers value handling can safely assume commas are value delimiters.
I would keep this issue scoped to this particular header
/assign (since you are already working on this one, thanks!)
/close fixed in
Closing this issue. <detailsIn response to : /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: return nil } <ins> // Wait blocks until the cacher is Ready or Stopped, it returns an error if Stopped. func (c *Cacher) Wait(ctx context.Context) error { return c.ready.wait(ctx) } </ins> // errWatcher implements watch.Interface to return a single error type errWatcher struct { result chan watch.Event

pull-kubernetes-unit TestStoreListResourceVersion in seems like July 7th 2024: ! Don't know, here is the log: No response /sig
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig api-machinery
See also
/sig etcd
/assign /cc
Bisected this to cc
I'm not sure if this is the proper way to fix it, but setting a limit in the ListOptions deflakes the test. The flakiness comes from the fact that the cache is sometimes not initialized in time which causes it to return a "too many requests". Perhaps in tests we could wait till the cache is ready in order to proceed?
Right, setting a limit when a resource version is set would delegate the list to etcd when cache is not initialized based on the KEP description. It is due to this type of request is not that costly to etcd. Looks like it is exactly how the wait cache ready is in See below So I think we should implement this same thing in when initializing the .
Agree - waiting for cacher to be initialized when creating it here is a better fix for it:
I thought about this as well, but waiting on is not possible outside of the cacher module, since it is a private field. I'm thinking to either add an option to cacherstorage.Config and make wait or add a new public method to . Do you guys have a better idea? Thanks.
Exposing Wait(context) method sounds fine
I have pushed the changes we discussed. Thank you all for your guidance.


Question: // Put one job and one pod into the store testJob := newJob(2, 2) manager.jobStore.Store.Add(testJob) <del> received := make(chan string) </del> <ins> received := make(chan struct{}) </ins> // The pod update sent through the fakeWatcher should figure out the managing job and // send it into the syncHandler. manager.syncHandler = func(key string) error {

Ran a test executable with more than 20,000 times and no failures. Any hints as to how to get it to show up? I'm running 100 jobs at a time, so my system is stressed already.
I to am unable to reproduce this. I get similar errors if I reduce the timeout at the top of the file. I spent a bit looking at how to rewrite the tests without the timeout, but nothing jumped out at me. Looking at it though, it's really just a 500ms timeout for the test to complete. I'm just going to remove them. Go has a default timeout of 10 minutes. If it's going to pass, it will pass then. If not, Go will give you a nice traceback where it deadlocked.


Question: # - iproute2: includes ss used in NodePort tests # from iperf image # install necessary packages: iperf, bash <del> RUN apk --update add bind-tools curl netcat-openbsd iproute2 iperf bash && rm -rf /var/cache/apk/*  </del> <ins> RUN retry () { i=0; while [ $i -lt 9 ]; do "$@" && return || sleep 30; i="${i+1}"; done; "$@"; }  && retry apk --update add bind-tools curl netcat-openbsd iproute2 iperf bash && rm -rf /var/cache/apk/*  </ins> && ln -s /usr/bin/iperf /usr/local/bin/iperf  && ls -altrh /usr/local/bin/iperf

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --post-kubernetes-push-e2e-agnhost-test-images Image build unhappy on s390x arch: Building image for agnhost OS/ARCH: linux/s390x. [.] [ 3/10] RUN apk --update add bind-tools curl netcat-openbsd iproute2 ipe. 0.165 fetch http://dl- 0.725 fetch http://dl- 1.719 (1/35) Installing ncurses-terminfo-base (6.2p20200523-r0) 1.758 (2/35) Installing ncurses-libs (6.2p20200523-r0) 1.811 (3/35) Installing readline (8.0.4-r0) 1.851 (4/35) Installing bash (5.0.17-r0) 1.939 Executing bash-5.0.17- 1.945 ERROR: bash-5.0.17- script exited with error 1 1.945 (5/35) Installing fstrm (0.6.0-r1) 1.987 (6/35) Installing libgcc (9.3.0-r2) 2.013 (7/35) Installing krb5-conf (1.0-r2) 2.053 (8/35) Installing libcomerr (1.45.6-r0) 2.093 (9/35) Installing keyutils-libs (1.6.1-r1) 2.133 (10/35) Installing libverto (0.3.1-r1) 2.174 (11/35) Installing krb5-libs (1.18.3-r0) 2.334 (12/35) Installing json-c (0.14-r1) 2.376 (13/35) Installing libstdc++ (9.3.0-r2) 2.493 (14/35) Installing libprotobuf (3.12.2-r0) 2.667 (15/35) Installing libprotoc (3.12.2-r0) 2.807 (16/35) Installing protobuf-c (1.3.3-r1) 2.854 (17/35) Installing libuv (1.38.

1-r0) 2.906 (18/35) Installing xz-libs (5.2.5-r0) 2.954 (19/35) Installing libxml2 (2.9.10-r5) 3.071 (20/35) Installing bind-libs (9.16.15-r0) 3.287 (21/35) Installing bind-tools (9.16.15-r0) 3.363 (22/35) Installing ca-certificates (-r4) 3.426 (23/35) Installing nghttp2-libs (1.41.0-r0) 3.456 (24/35) Installing libcurl (7.76.1-r0) 3.510 (25/35) Installing curl (7.76.1-r0) 3.561 (26/35) Installing iperf (2.0.13-r2) 3.607 (27/35) Installing libbz2 (1.0.8-r1) 3.649 (28/35) Installing fts (1.2.7-r1) 3.691 (29/35) Installing libelf (0.179-r0) 3.774 (30/35) Installing libmnl (1.0.4-r0) 3.814 (31/35) Installing libnftnl-libs (1.1.6-r0) 3.865 (32/35) Installing iptables (1.8.4-r2) 4.387 (33/35) Installing iproute2 (5.6.0-r0) 4.535 Executing iproute2-5.6.0- 4.540 ERROR: iproute2-5.6.0- script exited with error 1 4.540 (34/35) Installing libbsd (0.10.0-r0) 4.584 (35/35) Installing netcat-openbsd (1.130-r1) 4.627 Executing busybox-1.31.1-r20.trigger 4.632 ERROR: busybox-1.31.1-r20.trigger: script exited with error 1 4.632 Executing ca-certificates--r4.trigger 4.686 2 errors;

29 MiB in 49 packages ERROR: executor failed running [/dev/.buildkitqemuemulator /bin/sh -c apk --update add bind-tools curl netcat-openbsd iproute2 iperf bash && rm -rf /var/cache/apk/ && ln -s /usr/bin/iperf /usr/local/bin/iperf && ls -altrh /usr/local/bin/iperf]: exit code: 2 failed to solve: rpc error: code = Unknown desc = executor failed running [/dev/.buildkitqemu_emulator /bin/sh -c apk --update add bind-tools curl netcat-openbsd iproute2 iperf bash && rm -rf /var/cache/apk/ && ln -s /usr/bin/iperf /usr/local/bin/iperf && ls -altrh /usr/local/bin/iperf]: exit code: 2 make: ** [Makefile:43: all-build-and-push] Error 1 ERROR ERROR: build step 0 "" failed: step exited with non-zero status: 2 ERROR: () build -78ae-4086-bc26- completed with status "FAILURE"
2021/04/30 16:58:31 Failed to run some build jobs: [error running [gcloud builds submit --verbosity info --config --substitutions PULLBASEREF=master,WHAT=agnhost,GITTAG=v20210430-v1.22.0-alpha.1-58-gf039f94a6fe --project k8s-staging-e2e-test-images --gcs-log-dir gs://k8s-staging-e2e-test-images-gcb/logs --gcs-source-staging-dir gs://k8s-staging-e2e-test-images-gcb/source gs://k8s-staging-e2e-test-images-gcb/source/-1c8a-4141-9bbb-]: exit status 1] Need this fixed to finish image version dance to fix conformance tests for 1.21.1:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available .

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This has failed 3 times consecutively now. All because s390x issues ... I've manually re-triggered it the fourth time. /sig testing but also /sig release because this is a major problem for continuing to ship this platform. cc We've seen this before with flakes across other image builds for this platform.
looks like the latest run passed -
there's still underlying flakiness i need to get to the bottom of
Also what latest run? Were new changes made without bumping the version? Or did someone else manually trigger it yet again?
I just saw a green run at and the image was pullable, so I thought it had succeeded
I think someone must have retried it again manually. Postsubmits only run on merge, and we've had nothing else merge
Guirish tried to see whether our local s390x environment has the same problem too, he'll update us here.
the issue seams to be intermittent caused while pulling respective binaries from Alpine repo. I tried to reproduce the issue locally, I am observing similar build failures while cross building the image for other ARCH too, so the issue is not specific to s390x and it is to do with alpine repo.
thanks i've filed a PR that may help mitigate please review!
looks like a long pending problem
Interesting. FWIW 4/4 consecutive failures appeared to be s390x in CI. We've had previous flakes relating to symlinks affecting this platform only.


Question: existingContainers, _ := kl.ListContainers() log.Printf("Existing:n%#v Desired: %#v", existingContainers, desired) for _, container := range existingContainers { <del> // This is slightly hacky, but we ignore containers that lack '--' in their name // to allow users to manually spin up their own containers if they want. if !strings.Contains(container, "--") { </del> <ins> // Skip containers that we didn't create to allow users to manually // spin up their own containers if they want. if !strings.HasPrefix(container, "/"+containerNamePrefix+"--") { </ins> continue } if !desired[container] {

release-master-blocking Conformance-GCE-master-kubetest2Kubernetes e2e suite.[It] [sig-network] Services should be able to change the type from ExternalName to ClusterIP [Conformance] ci-kubernetes-gce-conformance-latest-kubetest2.Overall kubetest2.Test 07-11 15:51 EEST No response /sig cloud-provider cc
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I remember some timeout has changed with ginkgo2 and something moved from 24hours to 1 hours, or something like that, is it possible that this is the problem here, in this job?
The Grinko timeout change could also be the root problem with this job:
hmm, we have to revisit the flags
hmm, this seems correct
that was a good observations, seems that job is failing because gathering the logs takes too much time !
there is no timeout set in the kubetest2 command here (not sure if this would be the place to set a timeout)
it seems kubetest has to be migrated to ginkgov2 it should use this as basis I'm on vacation this week, can somebody take over and open the issue and patch kubetest2?
looks like this file needs to get updated: add line 72:
oh, you already commented this :D
I will have a look later
Related: umbrella issue:
last testgrid run passed :)


Question: if o == nil { return } <del> fs.StringVar(&o.ServiceCIDR, "service-cluster-ip-range", o.ServiceCIDR, "CIDR Range for Services in cluster. Requires --allocate-node-cidrs to be true") </del> <ins> fs.StringVar(&o.ServiceCIDR, "service-cluster-ip-range", o.ServiceCIDR, "CIDR Range for Services in cluster. Only used when --allocate-node-cidrs=true; if false, this option will be ignored.") </ins> fs.Int32Var(&o.NodeCIDRMaskSize, "node-cidr-mask-size", o.NodeCIDRMaskSize, "Mask size for node cidr in cluster. Default is 24 for IPv4 and 64 for IPv6.") fs.Int32Var(&o.NodeCIDRMaskSizeIPv4, "node-cidr-mask-size-ipv4", o.NodeCIDRMaskSizeIPv4, "Mask size for IPv4 node cidr in dual-stack cluster. Default is 24.") fs.Int32Var(&o.NodeCIDRMaskSizeIPv6, "node-cidr-mask-size-ipv6", o.NodeCIDRMaskSizeIPv6, "Mask size for IPv6 node cidr in dual-stack cluster. Default is 64.")

In on --cluster-cidr flag is written that it requires --allocate-node-cidrs to be true. Bu actual, we can set non empty --cluster-cidr and disable node cidr allocation (set --allocate-node-cidrs to false). I expect to see in documentation of --allocate-node-cidrs flag note that for this node pod cidr allocation functionality is required to set the --cluster-cidr flag. Also in documentation on --cluster-cidr flag I expect removing mention about that it requires --allocate-node-cidrs flag to be true. Run controller manager with 2 flags: --allocate-node-cidrs=false --cluster-cidr=172.20.0.0/16 We are using this functionality to change the expand cluster CIDR subnet. As I see, we can temporary disable nodeIPAM controller work by setting --allocate-node-cidrs=false. I found a with the code that ignores cluster cidr parsing if allocate-node-cidrs flag is false. kubernetes version 1.23.17 bare metal No response No response No response No response
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig docs
if you disable the value of is not used anyhwere, is it? I do not understand exactly what do you mean here, can you elaborate in your use case?
/assign
We have a cluster which was initialized with and . Our cluster grew to 65 nodes. In kube-controlle-manager logs we observed errors like: This error appears because the nodeIPAM controller is enabled and it tries to allocate a podCIDR subnet with /24 mask for each node. have 64 subnets - so for 65'th node there is now free subnet. We want to expand cluster CIDR subnet. For this action, while maintenance, we want to disable kube-controller-manager's nodeIPAM controller. Actually we can do this by changing only one flag in kube-controller-manager . The flag can remain unchanged.

Yes it is. The main idea of the issue is to change the affirmation in . Would be more transparent if flag will require flag to be set. Bot not vice versa. From the docs It is not clear that you can disable allocation of node CIDR and leave the flag cluster-cidr. I try to thing back side: If the flags and are 100% connected and there are used only by nodeIPAM, why do we need this separate flags. As I understand: configures the CIDR parameter for nodeIPAM controller. enables the controller. So if I am right in previous affirmations, the flag actually does not require to be true. NodeIPAM controller requires and with a subnet set.
/cc as we talked about this today
The Kubernetes project currently lacks enough contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/remove-lifecycle rotten
The Kubernetes project currently lacks enough contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at .

/lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/remove-lifecycle rotten
/assign


Question: import ( "fmt" <ins> "strings" </ins> "testing" "github.com/Azure/azure-sdk-for-go/services/compute/mgmt/2019-12-01/compute"

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Azure cloud provider gets the availability zone from IMDS directly and the zone is usually in lower cases in most regions. But there're chances this is not true in some regions, hence customer's workloads depend on availability zone may not work. What you expected to happen: The availability zone is always in lower cases. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /kind bug /sig cloud-provider /area provider/azure
/assign
Fixed by .


Question: export KUBE_ROOT=$(dirname "${BASH_SOURCE}")/.. source "${KUBE_ROOT}/hack/lib/init.sh" <del> kube::util::ensure-gnu-sed </del> # Remove generated files prior to running kazel. # TODO(spxtr): Remove this line once Bazel is the only way to build. rm -f "${KUBE_ROOT}/pkg/generated/openapi/zz_generated.openapi.go"

I am trying to update aws-sdk-go dep version, following the guide in I checkout to the master branch, and I use these commands: But all the "BUILD" files in the "vendor" folder are deleted. What did i do wrongWhat are the right steps? When I use the same commands in the "release-1.9" branch, all these files are not changed.
/sig contributor-experience
I came across this recently as well. same for me. /kind bug
Looks like is the culprit? /cc
you didn't do anything wrong. :) Updating bazel files again will solve the problem for now: But we should either fix the script or update the instructions.
This should run already as a part of godep- Is that step producing any errors?
There are not any error message. Here is the output:
yeah, I can reproduce this - running yields lots of missing files. still investigating, but I'm guessing the files are being generated in the generated gopath (from , rather than the repo root.
... though that shouldn't matter, since it's just a symlink. hm.
I think I found it.. I think it has to do with the fact that the init script uses to strip out symlinks, which doesn't play nice with . I'm testing a fix locally.. if it works, I'll push a PR. I don't know if removing the flag will break other things tho.
Nope. That didn't work. Next test I'm running is a revert of .
after running , if I then run , it still doesn't regenerate the files, which makes me think doesn't like running in the artificial gopath.
oh, I think might full resolve either the or its working directory, which causes problems since we have an artificial gopath. cc if he's seen this before.
seems to need to run from the real (or at least a subdir of the real ). seems to need to run from a valid - i.e. must be in . If I to the real in , works properly, but then explodes.
Just a straight revert of that last change didn't work either. I wonder how long this has been broken for..
When Gazelle starts, it locates the repository root (if is not given, it's the nearest parent directory containing a file named ), then evaluates symlinks in that path.

When Gazelle walks the directory tree, it only follows symlinks to directories that point outside the repository. It checks that by evaluating symlinks, then checking whether the resolved path has the resolved repository root as a prefix. This is all in service of avoiding symlink cycles and preventing the same directory from being visited on more than one path (which would cause incorrect or inconsistent attributes).
I'd guess since .
So I think passing to Gazelle would fix it. Then it won't go looking outside the GOPATH for the WORKSPACE file.
alone doesn't seem to be helping, at least not if I set it to the real repo root. and the files under are still missing. What does seem to help is if I to the real repo root before calling .
note also that it seems that gazelle isn't walking any of the tree when run under ; it's not just that's affected.
Easiest way to reproduce: out kubernetes/kubernetes some file in the tree (e.g. ) that the file you removed wasn't regenerated that the file was now regenerated
yeah, gazelle really doesn't like the symlink. super hacky fix that is probably not a good idea here:
Copying my comment from slack:
Please try again with a fresh master. Fix was merged in last night.
It's ok now!


Question: diff = 0 } jm.expectations.ExpectCreations(jobKey, int(diff)) <ins> errCh = make(chan error, diff) </ins> glog.V(4).Infof("Too few pods running job %q, need %d, creating %d", jobKey, wantActive, diff) active += diff

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--What keywords did you search in Kubernetes issues before filing this one? Job, resourcequota, scheduler Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-04-03T20:44:38Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-04-14T13:36:25Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"} Environment: CentOS bare metal () Cloud provider or hardware configuration: - OS (e.g. from /etc/os-release): CentOS 7 Kernel (e.g. ): Linux <server name3.10.0-514.2.2.el7.x8664 SMP Tue Dec 6 23:06:41 UTC 2016 x8664 x8664 x8664 GNU/Linux Install tools: - Others: - What happened: Having created a resourcequota (e.g. 400M) and creating enough memory-limited jobs to go over that quota (e.g. 4 x 200M limit jobs), at first, two jobs are created and executed as expected, and two jobs fail at first due to memory quota: Events: FirstSeen LastSeen Count From SubObjectPath Type Reason Message --------- -------- ----- ---- ------------- -------- ------ ------- 4s 4s 1 job-controller Warning FailedCreate Error creating: pods "memhog-y152c55u-" is forbidden: exceeded quota: memoryquota, requested: memory=200Mi, used: memory=400Mi, limited: memory=400Mi After the first two jobs have been completed, instead of starting the remaining jobs as soon as possible, the remaining two jobs stay unstarted in this state for many, many hours until they finally start and complete.

First job: Start Time: Fri, 28 Apr 2017 15:00:08 +0300 One of the remaining 2 jobs: Start Time: Sat, 29 Apr 2017 07:13:34 +0300 What you expected to happen: I'm expecting the two waiting jobs to start executing as soon as the resources they need are available, e.g. as soon as the first jobs are completed. How to reproduce it (as minimally and precisely as possible): Bring up a kubernetes cluster, then execute these commands: $ kubectl create quota memoryquota --hard=memory=400Mi (cat /dev/urandom head -n 1)' --replicas=1 --restart=OnFailure --image-pull-policy=IfNotPresent --limits=memory='2'm run- $ chmod +x run- $ for i in {1.4}; do ./run- 200 100; done job "memhog-0ju2eiuy" created job "memhog-t8qyt5sj" created job "memhog-ljaqz4ek" created job "memhog-eb8q8kr3" created Wait for around 30 seconds and the first jobs should be complete: $ kubectl get jobs NAME DESIRED SUCCESSFUL AGE memhog-0ju2eiuy 1 1 43s memhog-eb8q8kr3 1 0 42s memhog-ljaqz4ek 1 0 43s memhog-t8qyt5sj 1 1 43s $ kubectl describe jobs . However, the remaining jobs don't start at all as shown by: $ kubectl describe job memhog-ljaqz4ek Name: memhog-ljaqz4ek Namespace: test Selector: controller-uid=-2f00-11e7-8766- Labels: run=memhog-ljaqz4ek Annotations: run memhog-ljaqz4ek memhog -r100 100m --replicas=1 --restart=OnFailure --image-pull-policy=IfNotPresent --limits=memory=200Mi,cpu=100m --record=true --image=derekway.

Parallelism: 1 Completions: 1 Pods Statuses: 0 Running / 0 Succeeded / 0 Failed Pod Template: Labels: controller-uid=-2f00-11e7-8766- job-name=memhog-ljaqz4ek run=memhog-ljaqz4ek Containers: memhog-ljaqz4ek: Image: derekwaynecarr/memhog Port: Command: memhog -r100 100m Limits: cpu: 100m memory: 200Mi Environment: <noneMounts: <noneVolumes: <noneEvents: FirstSeen LastSeen Count From SubObjectPath Type Reason Message --------- -------- ----- ---- ------------- -------- ------ ------- 3m 3m 1 job-controller Warning FailedCreate Error creating: pods "memhog-ljaqz4ek-" is forbidden: exceeded quota: memoryquota, requested: memory=200Mi, used: memory=400Mi, limited: memory=400Mi Furthermore, pods related to the remaining jobs aren't even created and only the completed pods are shown: $ kubectl get pods -a NAME READY STATUS RESTARTS AGE memhog-0ju2eiuy-bz0zm 0/1 Completed 0 7m memhog-t8qyt5sj-q9msw 0/1 Completed 0 7m Anything else we need to know: I've also created a stackoverflow question about this (), and asked about this in the kubernetes slack, where I was urged to create a bug report.
I will try to reproduce. The creation of a pod will be blocked by quota. In the absence of a pod, the scheduler has nothing to schedule, so it is not a scheduling bug. It's possible the issue is related to back-off in the job controller not attempting to create pods again. The event shows that it had been 3m since the job controller attempted to create pods again for the requisite job which implies some type of back-off. None of the controllers are quota aware, so I would anticipate they should just keep trying at reasonable intervals at this point. If you describe quota after the first 2 pods are completed, do you see quota usage being replenished as expected?
I would like to fix the bug, could you assigin it to me?

if pod create failed, jobcontroller.managejob() need return error, will requeue podcreate task.
-- maybe you have more insight, but it looks to me just reading the code that when a job controller is unable to create a pod due to lack of quota, its not actually an error that forces a re-queue of the job. in addition, i cant actually see when a full resync of the job work queue happens outside of the informer sync interval which is probably really long. see: see:
- feel free to open a pr, please cc me on your solution. i would like to understand from the job folks if there should be a periodic resync interval aside from the need to requeue the job when all pods cannot be spawned (which i assume your pr will handle via exponential backoff)
-- we will want to have a fix for this downstream as well.
Yes, after the first two jobs are complete, shows that 0M out of 400M is in use. I'm also able to create new jobs, up to the memory limit so 2 at a time, that execute as expected. Any jobs over that at a time will stay in the same waiting state.
- Yes, I want handle it via exponential backoff. IMO, MaxOfRateLimiter had implemented a periodic resync, we don't need another one.
I'll make sure to cherry-pick this to openshift ( fyi).


Question: ret=1 fi <del> if ! _out="$(diff -Naupr -x "BUILD" -x "zz_generated.openapi.go" -x "AUTHORS*" -x "CONTRIBUTORS*" vendor "${_kubetmp}/vendor")"; then </del> <ins> if ! _out="$(diff -Naupr -x "BUILD" -x "AUTHORS*" -x "CONTRIBUTORS*" vendor "${_kubetmp}/vendor")"; then </ins> echo "Your vendored results are different:" >&2 echo "${_out}" >&2 echo "Godeps Verify failed." >&2

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: I'm trying to vendor , which due to (recently ?) code in depends on The problem now is that the content of is incomplete: apiextensions- calls generatedopenapi.GetOpenAPIDefinitions, but apiextensions-apiserver/pkg/generated/openapi/ doesn't contain the generated code. What you expected to happen: I expect to compile when fetching it with "go get" or vendoring it with tools like "dep". How to reproduce it (as minimally and precisely as possible): Environment: Kubernetes version (use ): master
/assign
/sig release
/assign
sent a fix in


Question: "watch":    true, } <ins> // Constant for the retry-after interval on rate limiting. // TODO: maybe make this dynamic? or user-adjustable? const RetryAfter = "1" </ins> // IsReadOnlyReq() is true for any (or at least many) request which has no observable // side effects on state of apiserver (though there may be internal side effects like // caching and logging).

Related with issue , the API server should have a cap on the number of outstanding operations/requests it's handling, and return an error otherwise. A simple semaphore is an idea for this.
+1 !
The good news/bad news on this is that there is a rate limiter in the API server (), but the defaults for it seem too high: 10 qps average, 200 burst (). We could start by lowering the burst level to something smaller, like 30. It may be interesting to also differentiate between GET and PUT/POST requests.
I think that we need limits on the number of outstanding requests, not the number of QPS.
-An max outstanding would be useful. But it may force us to have separate buckets for different types of requests, i.e. a small number of long running requests causing API server to reject short requests. -Heard that you might be looking into adding support for flow control (or something like it?). We may need it soon to help avoid bugs like runaway node status updates (see issue ).
I had not - the things we were looking at were the bastion changes to apiserver which will break logs and proxying into their own rest resources so they can be granularly controlled. Being able to timebox and classify requests differently would be good. ----- Original Message -----
- Ah, np, I probably misheard.
What are our plans for handling this client side? Do we want clients to handle "you're being throttled" error themselves, or do we plan to handle it in the library? I'm asking because currently kubelet does not have a 'short' retry loop, hence it tries to report NodeStatus every X (2) seconds, whether it succeed or not. Neither we have a sidechannel for heartbeat messages. This would mean, that if API server will be under heavy load, and with not-negligible probability NodeStatus reporting will be throttled, we may incorrectly mark a Node as unreachable. Currently it's not a huge problem, because X << Y, where Y is the time we wait before we decide a Node is unreachable. But because of the effort somewhere close to it'll probably get worse.
The client library should have options for retry on client.Config which include the ability to disable the behavior. Automatic retry is probably an "opt-out" behavior.

----- Original Message -----
says there are people working on rate limiting client connections to API server.
I have in the past adjusted the rate limiter to support the creation of large clusters. I also the logic. If you adjust the rate limiter settings, please make sure it is still possible to create large clusters.
I'm closing this as fixed, given the original subject. If we want to expand client-side support for retry, please open a dedicated issue. (Also note that QPS based throttling was to the client recently too)


Question: // Tests network performance using iperf or other containers. import ( <ins> "context" </ins> "fmt" <ins> discoveryv1beta1 "k8s.io/api/discovery/v1beta1" </ins> "time" "github.com/onsi/ginkgo" <ins> appsv1 "k8s.io/api/apps/v1" </ins> v1 "k8s.io/api/core/v1" <ins> metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/util/wait" clientset "k8s.io/client-go/kubernetes" </ins> "k8s.io/kubernetes/test/e2e/framework" <ins> e2edeployment "k8s.io/kubernetes/test/e2e/framework/deployment" </ins> e2enode "k8s.io/kubernetes/test/e2e/framework/node" <del> e2eservice "k8s.io/kubernetes/test/e2e/framework/service" </del> imageutils "k8s.io/kubernetes/test/utils/image" ) const ( <del> // empirically derived as a baseline for expectations from running this test using kube-up.sh. gceBandwidthBitsEstimate = int64(30000000000) // on 4 node clusters, we found this test passes very quickly, generally in less then 100 seconds. smallClusterTimeout = 200 * time.Second </del> <ins> // use this timeout for larger clusters largeClusterTimeout = 400 * time.Second // iperf2BaselineBandwidthMegabytesPerSecond sets a baseline for iperf2 bandwidth of 90 MB/s iperf2BaselineBandwidthMegabytesPerSecond = 90 // iperf2Port selects an arbitrary, unique port to run iperf2's client and server on iperf2Port = 6789 // labelKey is used as a key for selectors labelKey = "app" // clientLabelValue is used as a value for iperf2 client selectors clientLabelValue = "iperf2-client" // serverLabelValue is used as a value for iperf2 server selectors serverLabelValue = "iperf2-server" // serverServiceName defines the service name used for the iperf2 server serverServiceName = "iperf2-server" </ins> ) <del> // Declared as Flakey since it has not been proven to run in parallel on small nodes or slow networks in CI var _ = SIGDescribe("Networking IPerf [Experimental] [Slow] [Feature:Networking-Performance]", func() { </del> <ins> func iperf2ServerDeployment(client clientset.Interface, namespace string, isIPV6 bool) (*appsv1.Deployment, error) { framework.Logf("deploying iperf2 server") one := int64(1) replicas := int32(1) labels := map[string]string{labelKey: serverLabelValue} args := []string{ "-s", "-p", fmt.Sprintf("%d", iperf2Port), } if isIPV6 { args = append(args, "-V") } deploymentSpec := &appsv1.Deployment{ ObjectMeta: metav1.ObjectMeta{ Name:   "iperf2-server-deployment", Labels: labels, }, Spec: appsv1.DeploymentSpec{ Replicas: &replicas, Selector: &metav1.LabelSelector{ MatchLabels: labels, }, Template: v1.PodTemplateSpec{ ObjectMeta: metav1.ObjectMeta{ Labels: labels, }, Spec: v1.PodSpec{ TerminationGracePeriodSeconds: &one, Containers: []v1.Container{ { Name:    "iperf2-server", Image:   imageutils.GetE2EImage(imageutils.Agnhost), Command: []string{"iperf"}, Args:    args, Ports: []v1.ContainerPort{ { ContainerPort: iperf2Port, Protocol:      v1.ProtocolTCP, }, }, }, }, }, }, }, } </ins> <del> f := framework.NewDefaultFramework("network-perf") </del> <ins> deployment, err := client.AppsV1().Deployments(namespace).Create(context.TODO(), deploymentSpec, metav1.CreateOptions{}) if err != nil { return nil, fmt.Errorf("deployment %q Create API error: %v", deploymentSpec.Name, err) } framework.Logf("Waiting for deployment %q to complete", deploymentSpec.Name) err = e2edeployment.WaitForDeploymentComplete(client, deployment) if err != nil { return nil, fmt.Errorf("deployment %q failed to complete: %v", deploymentSpec.Name, err) } </ins> <del> // A few simple bandwidth tests which are capped by nodes. // TODO replace the 1 with the scale option implementation // TODO: Make this a function parameter, once we distribute iperf endpoints, possibly via session affinity. numClient := 1 numServer := 1 maxBandwidthBits := gceBandwidthBitsEstimate </del> <ins> return deployment, nil } </ins> <del> familyStr := "" if framework.TestContext.ClusterIsIPv6() { familyStr = "-V " </del> <ins> func iperf2ServerService(client clientset.Interface, namespace string) (*v1.Service, error) { service := &v1.Service{ ObjectMeta: metav1.ObjectMeta{Name: serverServiceName}, Spec: v1.ServiceSpec{ Selector: map[string]string{ labelKey: serverLabelValue, }, Ports: []v1.ServicePort{ {Protocol: v1.ProtocolTCP, Port: iperf2Port}, }, }, </ins> } <ins> return client.CoreV1().Services(namespace).Create(context.TODO(), service, metav1.CreateOptions{}) } </ins> <del> ginkgo.It(fmt.Sprintf("should transfer ~ 1GB onto the service endpoint %v servers (maximum of %v clients)", numServer, numClient), func() { nodes, err := e2enode.GetReadySchedulableNodes(f.ClientSet) framework.ExpectNoError(err) totalPods := len(nodes.Items) // for a single service, we expect to divide bandwidth between the network.  Very crude estimate. expectedBandwidth := int(float64(maxBandwidthBits) / float64(totalPods)) appName := "iperf-e2e" _, err = e2eservice.CreateServiceForSimpleAppWithPods( f.ClientSet, 8001, 8002, f.Namespace.Name, appName, func(n v1.Node) v1.PodSpec { return v1.PodSpec{ Containers: []v1.Container{{ Name:    "iperf-server", Image:   imageutils.GetE2EImage(imageutils.Agnhost), Command: []string{"/bin/sh"}, Args: []string{ "-c", "/usr/local/bin/iperf " + familyStr + "-s -p 8001 ", }, Ports: []v1.ContainerPort{{ContainerPort: 8001}}, }}, NodeName:      n.Name, RestartPolicy: v1.RestartPolicyOnFailure, } </del> <ins> func iperf2ClientDaemonSet(client clientset.Interface, namespace string) (*appsv1.DaemonSet, error) { one := int64(1) labels := map[string]string{labelKey: clientLabelValue} spec := &appsv1.DaemonSet{ ObjectMeta: metav1.ObjectMeta{ Name:   "iperf2-clients", Labels: labels, }, Spec: appsv1.DaemonSetSpec{ Selector: &metav1.LabelSelector{ MatchLabels: labels, </ins> }, <del> // this will be used to generate the -service name which all iperf clients point at. numServer, // Generally should be 1 server unless we do affinity or use a version of iperf that supports LB true,      // Make sure we wait, otherwise all the clients will die and need to restart. ) if err != nil { framework.Failf("Fatal error waiting for iperf server endpoint : %v", err) } iperfClientPodLabels := e2enode.CreatePodsPerNodeForSimpleApp( f.ClientSet, f.Namespace.Name, "iperf-e2e-cli", func(n v1.Node) v1.PodSpec { return v1.PodSpec{ </del> <ins> Template: v1.PodTemplateSpec{ ObjectMeta: metav1.ObjectMeta{ Labels: labels, }, Spec: v1.PodSpec{ </ins> Containers: []v1.Container{ { <del> Name:    "iperf-client", </del> <ins> Name:    "iperf2-client", </ins> Image:   imageutils.GetE2EImage(imageutils.Agnhost), <del> Command: []string{"/bin/sh"}, Args: []string{ "-c", "/usr/local/bin/iperf " + familyStr + "-c service-for-" + appName + " -p 8002 --reportstyle C && sleep 5", }, </del> <ins> Command: []string{"/agnhost"}, Args:    []string{"pause"}, </ins> }, }, <del> RestartPolicy: v1.RestartPolicyOnFailure, // let them successfully die. } </del> <ins> TerminationGracePeriodSeconds: &one, }, </ins> }, <del> numClient, ) expectedCli := numClient if len(nodes.Items) < expectedCli { expectedCli = len(nodes.Items) </del> <ins> }, Status: appsv1.DaemonSetStatus{}, } ds, err := client.AppsV1().DaemonSets(namespace).Create(context.TODO(), spec, metav1.CreateOptions{}) if err != nil { return nil, fmt.Errorf("daemonset %s Create API error: %v", spec.Name, err) } return ds, nil } // Test summary: //   This test uses iperf2 to obtain bandwidth data between nodes in the cluster, providing a coarse measure //   of the health of the cluster network.  The test runs two sets of pods: //     1. an iperf2 server on a single node //     2. a daemonset of iperf2 clients //   The test then iterates through the clients, one by one, running iperf2 from each of them to transfer //   data to the server and back for ten seconds, after which the results are collected and parsed. //   Thus, if your cluster has 10 nodes, then 10 test runs are performed. //     Note: a more complete test could run this scenario with a daemonset of servers as well; however, this //     would require n^2 tests, n^2 time, and n^2 network resources which quickly become prohibitively large //     as the cluster size increases. //   Finally, after collecting all data, the results are analyzed and tabulated. var _ = SIGDescribe("Networking IPerf2 [Feature:Networking-Performance]", func() { // this test runs iperf2: one pod as a server, and a daemonset of clients f := framework.NewDefaultFramework("network-perf") ginkgo.It(fmt.Sprintf("should run iperf2"), func() { readySchedulableNodes, err := e2enode.GetReadySchedulableNodes(f.ClientSet) framework.ExpectNoError(err) familyStr := "" if framework.TestContext.ClusterIsIPv6() { familyStr = "-V " </ins> } <del> framework.Logf("Reading all perf results to stdout.") framework.Logf("date,cli,cliPort,server,serverPort,id,interval,transferBits,bandwidthBits") </del> <ins> serverPodsListOptions := metav1.ListOptions{ LabelSelector: fmt.Sprintf("%s=%s", labelKey, serverLabelValue), } </ins> <del> // Extra 1/10 second per client. iperfTimeout := smallClusterTimeout + (time.Duration(expectedCli/10) * time.Second) iperfResults := &IPerfResults{} </del> <ins> // Step 1: set up iperf2 server -- a single pod on any node _, err = iperf2ServerDeployment(f.ClientSet, f.Namespace.Name, framework.TestContext.ClusterIsIPv6()) framework.ExpectNoError(err, "deploy iperf2 server deployment") </ins> <del> iperfClusterVerification := f.NewClusterVerification( f.Namespace, framework.PodStateVerification{ Selectors:   iperfClientPodLabels, ValidPhases: []v1.PodPhase{v1.PodSucceeded}, }, ) pods, err2 := iperfClusterVerification.WaitFor(expectedCli, iperfTimeout) if err2 != nil { framework.Failf("Error in wait...") } else if len(pods) < expectedCli { framework.Failf("IPerf restuls : Only got %v out of %v, after waiting %v", len(pods), expectedCli, iperfTimeout) } else { // For each builds up a collection of IPerfRecords iperfClusterVerification.ForEach( func(p v1.Pod) { resultS, err := framework.LookForStringInLog(f.Namespace.Name, p.Name, "iperf-client", "0-", 1*time.Second) if err == nil { framework.Logf(resultS) iperfResults.Add(NewIPerf(resultS)) } else { framework.Failf("Unexpected error, %v when running forEach on the pods.", err) } }) </del> <ins> _, err = iperf2ServerService(f.ClientSet, f.Namespace.Name) framework.ExpectNoError(err, "deploy iperf2 server service") // Step 2: set up iperf2 client daemonset //   initially, the clients don't do anything -- they simply pause until they're called _, err = iperf2ClientDaemonSet(f.ClientSet, f.Namespace.Name) framework.ExpectNoError(err, "deploy iperf2 client daemonset") // Make sure the server is ready to go framework.Logf("waiting for iperf2 server endpoints") err = wait.Poll(2*time.Second, largeClusterTimeout, func() (done bool, err error) { listOptions := metav1.ListOptions{LabelSelector: fmt.Sprintf("%s=%s", discoveryv1beta1.LabelServiceName, serverServiceName)} esList, err := f.ClientSet.DiscoveryV1beta1().EndpointSlices(f.Namespace.Name).List(context.TODO(), listOptions) framework.ExpectNoError(err, "Error fetching EndpointSlice for Service %s/%s", f.Namespace.Name, serverServiceName) if len(esList.Items) == 0 { framework.Logf("EndpointSlice for Service %s/%s not found", f.Namespace.Name, serverServiceName) return false, nil } return true, nil }) framework.ExpectNoError(err, "unable to wait for endpoints for the iperf service") framework.Logf("found iperf2 server endpoints") clientPodsListOptions := metav1.ListOptions{ LabelSelector: fmt.Sprintf("%s=%s", labelKey, clientLabelValue), } framework.Logf("waiting for client pods to be running") var clientPodList *v1.PodList err = wait.Poll(2*time.Second, largeClusterTimeout, func() (done bool, err error) { clientPodList, err = f.ClientSet.CoreV1().Pods(f.Namespace.Name).List(context.TODO(), clientPodsListOptions) if err != nil { return false, err } if len(clientPodList.Items) < len(readySchedulableNodes.Items) { return false, nil } for _, pod := range clientPodList.Items { if pod.Status.Phase != v1.PodRunning { return false, nil } } return true, nil }) framework.ExpectNoError(err, "unable to wait for client pods to come up") framework.Logf("all client pods are ready: %d pods", len(clientPodList.Items)) // Get a reference to the server pod for later serverPodList, err := f.ClientSet.CoreV1().Pods(f.Namespace.Name).List(context.TODO(), serverPodsListOptions) framework.ExpectNoError(err) if len(serverPodList.Items) != 1 { framework.Failf("expected 1 server pod, found %d", len(serverPodList.Items)) </ins> } <del> fmt.Println("[begin] Node,Bandwidth CSV") fmt.Println(iperfResults.ToTSV()) fmt.Println("[end] Node,Bandwidth CSV") </del> <ins> serverPod := serverPodList.Items[0] framework.Logf("server pod phase %s", serverPod.Status.Phase) for i, condition := range serverPod.Status.Conditions { framework.Logf("server pod condition %d: %+v", i, condition) } for i, cont := range serverPod.Status.ContainerStatuses { framework.Logf("server pod container status %d: %+v", i, cont) } framework.Logf("found %d matching client pods", len(clientPodList.Items)) nodeResults := &IPerf2NodeToNodeCSVResults{ ServerNode: serverPod.Spec.NodeName, Results:    map[string]*IPerf2EnhancedCSVResults{}, } // Step 3: iterate through the client pods one by one, running iperf2 in client mode to transfer //   data to the server and back and measure bandwidth for _, pod := range clientPodList.Items { podName := pod.Name nodeName := pod.Spec.NodeName iperfVersion := f.ExecShellInPod(podName, "iperf -v || true") framework.Logf("iperf version: %s", iperfVersion) </ins> <del> for ipClient, bandwidth := range iperfResults.BandwidthMap { framework.Logf("%v had bandwidth %v.  Ratio to expected (%v) was %f", ipClient, bandwidth, expectedBandwidth, float64(bandwidth)/float64(expectedBandwidth)) </del> <ins> for try := 0; ; try++ { /* iperf2 command parameters: *  -e: use enhanced reporting giving more tcp/udp and traffic information *  -p %d: server port to connect to *  --reportstyle C: report as Comma-Separated Values *  -i 1: seconds between periodic bandwidth reports *  -c %s: run in client mode, connecting to <host> */ command := fmt.Sprintf(`iperf %s -e -p %d --reportstyle C -i 1 -c %s && sleep 5`, familyStr, iperf2Port, serverServiceName) framework.Logf("attempting to run command '%s' in client pod %s (node %s)", command, podName, nodeName) output := f.ExecShellInPod(podName, command) framework.Logf("output from exec on client pod %s (node %s): n%sn", podName, nodeName, output) results, err := ParseIPerf2EnhancedResultsFromCSV(output) if err == nil { nodeResults.Results[nodeName] = results break } else if try == 2 { framework.ExpectNoError(err, "unable to parse iperf2 output from client pod %s (node %s)", pod.Name, nodeName) } else { framework.Logf("Retrying: IPerf run failed: %+v", err) } } } // Step 4: after collecting all the client<->server data, compile and present the results /* Example output: Dec 22 07:52:41.102: INFO:                                From                                 To    Bandwidth (MB/s) Dec 22 07:52:41.102: INFO:              three-node-ipv6-worker            three-node-ipv6-worker2                2381 Dec 22 07:52:41.102: INFO:             three-node-ipv6-worker2            three-node-ipv6-worker2                2214 Dec 22 07:52:41.102: INFO:             three-node-ipv6-worker3            three-node-ipv6-worker2                3123 */ framework.Logf("%35s%35s%20s", "From", "To", "Bandwidth (MB/s)") for nodeFrom, results := range nodeResults.Results { framework.Logf("%35s%35s%20d", nodeFrom, nodeResults.ServerNode, results.Total.bandwidthMB()) } for clientNode, results := range nodeResults.Results { megabytesPerSecond := results.Total.bandwidthMB() if megabytesPerSecond < iperf2BaselineBandwidthMegabytesPerSecond { framework.Failf("iperf2 MB/s received below baseline of %d for client %s to server %s: %d", iperf2BaselineBandwidthMegabytesPerSecond, clientNode, nodeResults.ServerNode, megabytesPerSecond) } </ins> } }) })

What happened: The iperf tests were written a while ago but i think havent been updated. They measure bandwidth over multiple pods, but they probably should also have a test which reports individual pod's bandwidth between one pod over a SEP. They should pretty print a table so its easy to see exactly what is going on with the bandwidth They should use iperf3 (need to check that) Maybe we should have permutations for udp and tcp? The current output is something like: What you expected to happen: That running the tests would result in a table such as those that you can get from iperf3, i.e. How to reproduce it: run the e2es on any cluster, Environment: Kubernetes version (use ): master, any version
/sig network
/assign
/assign
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/remove-triage unresolved
I've taken a first pass at this , working on resolving a couple issues then we can circle back to determine how else it could be improved!
Cool c/p the new test output so we can compare when u get a chance
ah yess. much better then what we have now.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: url.Host += ":80" } } <del> headers, err := headersForConfig(config) </del> <ins> headers, err := headersForConfig(config, url) </ins> if err != nil { return nil, fmt.Errorf("failed to load http headers: %v", err) }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: When I did e2e conformance test with log level 5, it completely passed! When I did e2e conformance test with log level 8, it just failed. The only thing I did is just leveling up from log level 5 to 8. But, I've got this error message. What you expected to happen: test would be completely passed and just get more verbose log Environment: local cluster <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
/sig testing
Nice report. I confirmed this issue happens on my local environment also like: