Question: # which in turn restarts docker. /etc/init.d/docker stop <ins> # Make sure docker gracefully terminated before start again while pidof docker > /dev/null; do echo "waiting clean shutdown" sleep 10 done </ins> /etc/init.d/docker start <del> echo "waiting a minute for startup" sleep 60 </del> <ins> echo "waiting 30s for startup" sleep 30 </ins> while true; do if ! sudo timeout 10 docker version > /dev/null; then

I've seen a bunch of different failure, with number of tests failing due to: The example failure is here: [but there were 5+ more today in different places & suites]
I guess we need a more specific diagnosis. to diagnose/delegate.
Yes, I checked the instance posted above yesterday, and nothing is really outstanding. Meanwhile, I couldn't find other instances you mentioned since there is no link.
- what do you mean by "nothing is really outstanding". This shouldn't happen... Regarding other failures - I can probably paste more tomorrow.
Ahh, I should make my comment more clear here. I didn't mean there is no issue. From the log posted above, I saw after the test finished, test checked all nodes ready or not, and one node is not ready thus the test is marked not ready, and the node is not self-healing within a reasonable time, which is not acceptable. But from all available information, nothing is outstanding to me to figure out the root cause.
I managed finding one node not ready after test, and checked Looks like kubelet is failed the healthcheck, and killed by supervisord: By checking kubelet log, I noticed that after last SYNCLOOP, kubelet doesn't nothing except responding /healthz, and after 5m, it is killed by supervisord: 5m here makes me think kubelet might stuck at SyncPods because docker is not responding within a reasonable time, Kubelet failed healthz after 5ms. Unfortunately, supervisord omit all health failure reasons, I cannot prove my theory here. (1) I will file a separate issue to fix supervisord scripts to collect the failure reason, and collect those logs for our jenkins. Another found I have is that right before this, supervisord also killed docker daemon: recently changed docker version to docker ps so that we could reliably detect docker health, especially for But caveat is that docker ps might take long time occasionally, and supervisord might kill docker daemon unnecessarily, which might augment this issue here. (2) has a pending pr () to revert that change back, so that we can rule out docker healthcheck issue here. Also we need to have docker log available for debugging.

Above two issues make me think we might hit docker 1.9.X performance issue mentioned at: , but our performance stress test couldn't reproduce it: We need to verify this. (3) Also reminded me that recently we saw kubelet stuck at syncloop in but with docker 1.8.3 then. In this case, we might run into the hairpin mode bug. Anyway, my pending pr () should help on this case. cc/
fwiw, I am trying to reproduce and I keep running into this. So this one seems easy to reproduce manually if you think that helps.
we have two NotReady nodes (as of 10 am CET), but there were 210 NotReady ones during cluster validation phase. I disabled enormous cluster test, so you can debug.
- you can take a look into machines from:
The problematic run we are talking about is this one: where only 789/1000 of nodes get ready within timeout.
and I filed a separate issue for the enormous cluster failure: Basically it is caused by crashlooping etcd-container due to oom killing.
- thanks for investigation!
Do you have more incidents of NotReady nodes after the initial cluster validation? I am looking for those failures :-)
I checked nikhiljindal's cluster mentioned: It is a dup issue of , the kernel deadlock issue triggered by hairpin mode: I keep this one open to make sure the situation of the node is becoming NotReady is not caused by docker 1.9.1.
I think http://pr- might be another example of this?
I believe the failure you mentioned at is also hairpin mode related issue. Unfortunately there is no logs from worker nodes for that jenkins build.
Recent e2es have failed with this symptom.
can't tell without system logs which we don't seem to collect when build times out. there were quite a few cases of node not ready in , docker was crashlooping. And at least the first few weren't the hung task kernel bug from I'm collecting in () because I want to know if it was some corrupt docker-local-db state type thing.

- another instance:
And one more:
IMO this is blocked on the lack of logs when a build times out, just like (and they're probably the same root cause)
A little bit summary here: docker used to kill pre-running docker containers at startup stage, but recent releases, they move this logic to shutdown stage, which is good. But docker 1.9.1, since the rearchitect of docker daemon including separate docker network, persistent network & storage state requires much long time. Meanwhile, our supervisor babysitter script does respect such change in docker without waiting for docker gracefully terminated. In this case, we might run into the issue of supervisor keep shutdown the docker daemon, and immediately start it, but start failed due to previous process is not terminated. Then supervisor repeat above procedure. I am going to send a fix shortly.


Question: type HTTPIngressPath struct { // Path is matched against the path of an incoming request. Currently it can // contain characters disallowed from the conventional "path" part of a URL <del> // as defined by RFC 3986. Paths must begin with a '/'. When unspecified, // all paths from incoming requests are matched. </del> <ins> // as defined by RFC 3986. Paths must begin with a '/' and must be present // when using PathType with value "Exact" or "Prefix". </ins> // +optional Path string `json:"path,omitempty" protobuf:"bytes,1,opt,name=path"`

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --When I tried to use the default path I got . No error. According to the path is not needed. "When unspecified, all paths from incoming requests are matched." Create a ingress like so: Kubernetes version (use ):Cloud provider or hardware configuration: AWS OS (e.g: ): N/A Kernel (e.g. ): N/A Install tools: brew Network plugin and version (if this is a network-related bug): N/A Others:
/sig network /area ingress It seems to be a problem of ingress and I suggest creating an issue in I find a similar one which is a little different there
We use - would this be a problem with the aws repo then?
/assign
So digging a bit here, the API docs needs to be improved. By the , is required if the PathPrefix is Exact or Prefix. To solve your issue specifically, you have two ways: Add that accepts ingress without paths Add and change PathPrefix to Prefix, so this becomes a catchAll I'll open a PR to fix the comment/docs. Thanks!
Thanks! In my case because I'm using ALB controller I had to use with prefix to get it working - see
/triage accepted PR will close this ;)


Question: } // Return the first match. <del> backingFilePath := strings.TrimSpace(string(data)) </del> <ins> backingFilePath := cleanBackingFilePath(string(data)) </ins> if backingFilePath == path || backingFilePath == realPath { return fmt.Sprintf("/dev/%s", filepath.Base(device)), nil }

A pod uses a PVC with . The block volume is backed by a block device on the node (which is created by some CSI storage providers). If the underlying block device corresponding to the volume is accidentally disconnected from the node (due to network, hardware problem, etc...), the user cannot delete the pod. The pod would be stuck in terminating forever since Kubelet refuses to unmount the block volume. This prevents the user from cleaning up the pod and spinning up a new replacement pod thus leading to a long service degradation. For example, if the pod is part of a statefulset, the replacement pod cannot come up due to the old pod being stuck terminating. Kubelet should be able to finish terminating a pod when the corresponding block device is already removed from the node. a CSI storage provider that supports . For example, a statefulset that has volumeClaimTemplates with . For example: <details close<summaryclick to open yaml manifest</summary</detailsa network/hardware problem in which the backing block device is removed from the host unexpectedly. In our example, we delete a Longhorn engine instance manager which leads to the underlying block device being removed from the host. that the pod cannot access the block volume, try to delete the pod. pod is stuck in terminating state forever. Logs from kubelet: <details close<summaryClick to open logs</summary<br</detailsPlease pay attention to the log . Kubelet is refusing to terminate the pod because the underlying block device is removed from the host. If we run on the host, we can see that the backing file of the corresponding loop device is marked as : In our humble opinion, we think that Kubelet should process to remove the loop device corresponding to the deleted block device so that pod can finish terminating as detailed in this PR: We have tested that PR and can confirm that it fixes the issue. We have prepared a PR at : We have tested that PR and can confirm that it fixes the issue. <details</details<detailsRancher RKE2 </details<details</details<detailsRancher RKE2 </details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance.

The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage
Ref:
/cc
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/remove-lifecycle rotten
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle rotten
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
kubelet log 13:31:57 wanguo-72 kubelet[1131]: E0428 13:31:57. 1131 ] Operation for "{^-54fa-4068-bbad- podName:-9521-4aed-9bfe- nodeName:}" failed. No retries permitted until 2023-04-28 13:33:59. +0800 CST m=+8838. (durationBeforeRetry 2m2s). Error: UnmapVolume.UnmapBlockVolume failed for volume "pvc--a4e4-48a7-9057-" (UniqueName: "^-54fa-4068-bbad-") pod "-9521-4aed-9bfe-" (UID: "-9521-4aed-9bfe-") : blkUtil.DetachFileDevice failed. globalUnmapPath:, podUID: -9521-4aed-9bfe-, bindMount: true: failed to unmap device from map path. mapPath is empty node poweroff restart , pod use block mode volume, and pod stuck in terminating
so it is a new issue, and I encounter it as well with XSKY CSI after node restart. My kubernetes version is 1.25.8
which Kubernetes version is it?
Hi, I also see the issue reported by I'm using v1.25.10. I assume it happens when using containerized kubelet. In my environment, losetup reports weird backing file paths. This can be reproduced as follows. Until this moment, the correct path is visible from the host. But, after terminating the container, I get the wrong results. I consider this behavior causes failing unmap. Any thoughts?


Question: // charge for storage if request, found := pvc.Spec.Resources.Requests[corev1.ResourceStorage]; found { <ins> roundedRequest := request.DeepCopy() if !roundedRequest.RoundUp(0) { // Ensure storage requests are counted as whole byte values, to pass resourcequota validation. // See http://issue.k8s.io/94313 request = roundedRequest } </ins> result[corev1.ResourceRequestsStorage] = request // charge usage to the storage class (if present) if len(storageClassRef) > 0 {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When a pvc was specified with a quantity including the suffix : attempts to include that quantity in the resourcequota's storage resulted in the following error, which blocked all other updates to the resourcequota: What you expected to happen: Updates to all other quantities in the resourcequota should not have been blocked by the error above In general, as a suffix (milli) doesn't really make sense when used in the context of memory, and causes issues like the above. Would it be possible to deny the usage of this suffix at admission time for things like pvcs? (Our mitigation right now is to add a rule that does this in Open Policy Agent). How to reproduce it (as minimally and precisely as possible): Create a resourcequota object with limits on pvcs and the a storage type, e.g. Then create a PVC with the following spec: The resource quota controller should start producing the error log above. Anything else we need to know?: Environment: Kubernetes version (use ): v1.17.3 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig api-machinery
/remove-sig api-machinery /sig storage
i guess we should leave api-machinery! /sig api-machinery
/assign I think we should validate the value of field.
I've filed to address this issue.
looks like it is not easy to tighten the validation and that requires careful consideration xref:
/assign
"Updates to all other quantities in the resourcequota should not have been blocked by the error above" should be fixed separately
Opened to round storage requests up to the nearest byte in the quota evaluator.


Question: // with cluster e2e and launch kubelet outside of the test for both regular node e2e and // node conformance test. // TODO(random-liu): Switch to use standard node bootstrap script. <del> func launchKubelet(host, workspace, results, testArgs string) error { </del> <ins> func launchKubelet(host, workspace, results, testArgs, bearerToken string) error { </ins> podManifestPath := getPodPath(workspace) if output, err := SSH(host, "mkdir", podManifestPath); err != nil { return fmt.Errorf("failed to create kubelet pod manifest path %q: error - %v output - %q", podManifestPath, err, output) } startKubeletCmd := fmt.Sprintf("./%s --run-kubelet-mode --logtostderr --node-name=%s"+ <ins> " --bearer-token=%s"+ </ins> " --report-dir=%s %s --kubelet-flags=--pod-manifest-path=%s > %s 2>&1", <del> conformanceTestBinary, host, results, testArgs, podManifestPath, filepath.Join(results, kubeletLauncherLog)) </del> <ins> conformanceTestBinary, host, bearerToken, results, testArgs, podManifestPath, filepath.Join(results, kubeletLauncherLog)) </ins> var cmd []string systemd, err := isSystemd(host) if err != nil {

Which jobs are failing: Which test(s) are failing: The tests are failing in the . Since when has it been failing: 10/23/2020 Testgrid link: Reason for failure: It started failing at commit , but it's unknown if it's related to that. Anything else we need to know: We discussed in the Kubernetes SIG-Node CI subgroup and will start taking a look at it.
/assign
/sig node
Investigation notes: between successful and unsuccessful runs and found no notable differences. , and had a couple of failed health checks on initialization, which doesn't happen in the successful test runs, though there's only 2 seconds between starting readiness checks and an eventual successful check. config the same between runs looking through the kubelet logs for and , but I haven't found anything interesting yet. kubelet- failed health checks The below is a list of errors that occur in runs that succeeded and failed, which may be due to startup or specific testing scenarios. Bunch of these for numerous containers:
Both new and old test runs have a bunch of logs like this, implying that it's been having issues for a while: The failed one has this additional line: However, the doesn't have the same issues. I don't see the connection timeouts at all. I also don't see anything else in the kubelet logs that appears different between successful and failed runs. Unfortunately, I don't know enough about the infrastructure setup, but here are my current theories: about the infrastructure has gotten progressively more overloaded and now it's falling over. The tests run on 8 nodes versus 1, so they have more capacity there. Not totally buying this because why did it go from mostly successful to 100% failures rather than a mix for a bit. changed in the GCE configuration. Not sure how that is managed.
Diff there suggests three merges from k/k Only merge in test-infra is an image bump <img width="215" alt="image" src="https://user-"Interesting that a similar ci version and pull version of the job passes and the one ci version does not. pull: passing ci: failing ci: The failing ones do not have a node_e2e , but I don't see any of the expected log messages for failure of the services to start.
It is interesting that one passes and one does not.

Primary differences are the number of nodes, resource settings, and the fact that the conformance tests use Docker.
In addition to the services log missing, the overall system run logs are also missing. May be related to any decision we make for
Would we be able to kick off a Prow job with custom configuration? In other words, to rule in or rule out something in the image change, can we kick off a job with the latest everything else but use the older image? In reference to the decision on , are you suggesting we pause investigation and see if we decide to just move or change the conformance tests?
/triage accepted /priority critical-urgent
Catching up on this. qq: has anyone vetted ? Ise that pr went in around the time this job began to fail The beforesuite (AI: we should list out what it does), fails to communicate with nodes the mention of 6443 ports looks interesting here.
Good find. Definitely looks suspicious. I'll dig into that.
From : This comes from , which has some node-conformance specific test setup, so maybe something interesting there.
quick interruption for instructions on how to reproduce this one: I think this may work Focus and skip The FOCUS and SKIP arguments im getting from Image config file The job also uses the following config file Make sure to copy it and name it (the argument to IMAGECONFIG_FILE above). Additional arguments The test args to this job are I think these are the defaults already but just in case may be actually useful to add the following to the above command Miscellaneous If you run into trouble with the above commands try going through The info here came from
You'll also need to use the flag to make sure it runs in Docker, I believe.
ah yeah, good catch! Thank you. Don't know about your availability, but if you are all busy ima allocate sometime later today to dive deeper into the info we have here
I'm going to try to spend time of this in the afternoon (PDT). Specifically, I was going to try to verify that the logs from starting up the test suite look as expected and possibly try to run the case to reproduce.

My first suspicion was to have the server running in another port than 6443 (which Kubelet has configured), this is not the case: Only o.SecureServing.BindAddress is being set explicitly, but the port comes from o.NewServerRunOptions().
I think it's related to the movement of this line. I couldn't figure out how to link to the PR so I made a comment.
as you notice before there's no , and we should be seeing Before:
I can repro the issue now, and I think the line was moved upward to generate the token before starting the kubelet. Moving it up out of the if-statement may help partially, but in the conformance test the kubelet is manually started ahead of this code, so it may not matter.
Make sense. Do you mean moving this out of the conformance check?
Haha, yea, that message is deceptive... I think I'll make a PR to at least let it be more correct and give us more info.
for alerts to subscribers, PR created. Please comment.
looking green!
The output looks very strange, I have no idea why. Output has double-underscore-number for the different runs, instead of the image used. [__08] vs [cos-lts]
Ya, I noticed that. It's weird. Not sure how it's set, but you can see that the junit config names have a similar issue: -junitcos- -
Ok, found the root of the problem. Here's the command for the master suite: Note , which is not present in the conformance tests. but not sure how it eventually gets here.


Question: if cfg.Global.Datacenter != "us-west" { t.Errorf("incorrect datacenter: %s", cfg.Global.Datacenter) } <ins> if cfg.Global.VMUUID != "1234" { t.Errorf("incorrect vm-uuid: %s", cfg.Global.VMUUID) } </ins> } func TestNewVSphere(t *testing.T) {

What keywords did you search in Kubernetes issues before filing this one?: VMware, vSphere, IP, UUID BUG REPORT. Kubernetes version: Environment: Cloud provider or hardware configuration: VMware vSphere 6.5.0 OS : Debian GNU/Linux 8.7 (jessie) Kernel : Install tools: Kargo What happened: Some containers were launched on a node with a name containing the hostname of a different node and the does not retrieve the right node name : Here is retrieved instead of . When using the VSphere Cloud provider, the node name is retrieved from the function in Since this PR (), the node data are retrieved based on the IP address instead of the UUID. The function gets all the IP address and check if one of the IP on any interface matches a VM on vSphere. The problem is that the docker0 interface has the same IP on all the nodes (usually 172.17.0.1) and this IP is referenced into VMware. As this IP may be the first to appear in the it will match the first VM that vSphere gives with the function and this can be any VM running Docker on vSphere. Thus, the provider has a big chance to not get the informations for the VM it is running on, leading to an inconsistent state in the cluster (wrong pod name for example). Anything else we need to know: An idea would be to first retrieve the VM name with the function and if no results use the IP address (and add an exclusion to not use the interface). What do you think ? If you think this might be the right way to do, we can try to propose the PR. Thanks for your feedback.
Wow, this very much is a bad, bad bug. It was introduced by one of my colleagues here at HPE. It's made worse if you run multiple Kubernetes clusters in the same vSphere Datacenter. I'll send a PR upstream tomorrow to resolve this. I think it's best to go back to using dmidecode, but maybe someone from VMware ( or ?) has another suggestion? This would still require root access, but I can't think of a better way to get the machine's ID.
Related issue Will dig in more. For now instead of making it a value determined runtime the deployment framework should have root privileges and can populate it as part of the cloud config post VM creation.
is my colleague.

A little more info and a few ideas. I would also support going back to ` (still requires root though). We could also have the following logic: Get UUID and use it to get the VM Name with FindByUUID if possible, If not, get the hostname and use FindByDNSName, If still nothing is found, keep the current logic, excluding docker0 from the interfaces we examine. However, I agree with that getting this from the cloud config may be the best and most reliable solution.
I'm working on a patch right now that will do the following: Add a VMUUID field to the cloud config If VMUUID is not set, will use Error out if neither one is available
PR is - if someone would like to review :)


Question: $PARSERS_CONFIG | Out-File -FilePath $fluentbit_parser_file -Encoding ASCII # Create directory for all the log position files. <del> New-Item -Type Directory -Path "/var/run/google-fluentbit/pos-files/" </del> <ins> New-Item -Type Directory -Path "/var/run/google-fluentbit/pos-files/" -Force | Out-Null </ins> Log-Output "Wrote logging config to $fluentbit_parser_file" }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --The default fluent-bit configuration for GCE Windows nodes produces unparsed JSON logs and does not forward container logs. Container log output is produced. fluent-bit on a given windows image. a GCE Windows e2e cluster and view the stackdriver output for pods. Kubernetes version (use ): 1.20+ Cloud provider or hardware configuration: GKE/GCE OS (e.g: ): Windows Kernel (e.g. ): Any version Install tools: fluent-bit Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig windows


Question: cName6 = "container6-name" cName7 = "container7-name" cName8 = "container8-name" <ins> cName9 = "container9-name" </ins> ) func TestCRIListPodStats(t *testing.T) {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: CPU.Time is set to cs.StartTime but ps.Memory.Time is set to timestamp when collecting the memoryusage. What you expected to happen: CPU.Time is set to cs.CPU.Time How to reproduce it (as minimally and precisely as possible): Check the code Anything else we need to know?: Environment: Kubernetes version (use ): master branch when creating this issue. Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/assign
/sig kubelet
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig kubelet Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
I am happy to take over this. /assign


Question: __custom_func() { case ${last_command} in kubectl_get | kubectl_describe | kubectl_delete | kubectl_label | kubectl_stop) <del> __kubectl_get_resource </del> <ins> __kubectl_get_resource return ;; kubectl_logs) __kubectl_require_pod_and_container return ;; kubectl_exec) __kubectl_get_resource_pod </ins> return ;; <del> kubectl_logs) __kubectl_require_pod_and_container return ;; </del> *) ;; esac

TODOs: Last updated on Sept 2 by Janet [x] kubectl label - suggest a resource type [x] kubectl get and delete also use kubectl.NewHumanReadablePrinter to get the resources back, thus should be fixed as well. [x] kubectl exec - suggest the pod name [x] kubectl rolling-update - suggest the rc name [x] check that is doing the right thing [x] Distribute with kubernetes binaries [x] Put information about kubectl completion in user guide [x] docs/getting-started- Recently some users reported that there is no completion for the kubectl command. But there is. Unfortunately it is not documented how to install it. I found that all needs to be done is to: (or maybe just copy this file into /etc/bash_completion.d/) This information should be put into the user guide (maybe it is there - I just can't find it) I think it should be included for instance here: docs/getting-started- There are also some minor issues: does not suggest the rc name does not suggest the pod name does not suggest a resource type (maybe this is ok because we can give --overwrite here)
It looks like the bash-completion file is not distributed with kubernetes binaries:
:+1: This should definitely be provided with the binaries.
Good suggestions. Cc
, please cofirm that the bash completion file is distributed with k8s binaries: namely contrib/completions/bash/kubectl For me it looks like it isn't.
cc
I confirmed and it isn't.
TODOs: [x] kubectl label - suggest a resource type [x] kubectl get and delete also use kubectl.NewHumanReadablePrinter to get the resources back, thus should be fixed as well. [x] kubectl exec - suggest the pod name [x] kubectl rolling-update - suggest the rc name [x] check that is doing the right thing [x] Distribute with kubernetes binaries [x] Put information about kubectl completion in user guide [x] docs/getting-started- Last updated on Sept 2
Maybe add: check that is doing the right thing....
Thanks for the suggestion. Checklist updated.
Depending on where you want it in the tarball, is a good place to start. You can use and then iterate on until you're comfortable with the changes.

(If I'm making packaging changes, I'll frequently start by dumping the sorted output of that command into a file, then later diff against it to make sure I got exactly the change I wanted.)
Right now both and brings up all containers that has. Since is optional (the without -c way is supported as legacy behavior), this is expected. I'm not sure how to force the bash completion to print instead of container names when since flag isn't set as required in code.


Question: log_level_logfile: debug EOF <del> hostnamef=$(hostname -f) </del> <ins> hostnamef=$(uname -n) </ins> apt-get install -y ipcalc netmask=$(ipcalc $MINION_IP_RANGE | grep Netmask | awk '{ print $2 }') network=$(ipcalc $MINION_IP_RANGE | grep Address | awk '{ print $2 }')

Actually appending fqdn suffix already causes issue for non-resolvable node names based on today: Today when NodeController "registers" nodes in SyncCloud by getting bare instance names () from GCE, and recreating the fqdn and append it to instance names().


Question: }, { "ImportPath": "github.com/opencontainers/runc/libcontainer", <del> "Comment": "v1.0.0-rc2-49-gd223e2a", </del> <ins> "Comment": "v1.0.0-rc2-49-gd223e2ad", </ins> "Rev": "d223e2adae83f62d58448a799a5da05730228089" }, { "ImportPath": "github.com/opencontainers/runc/libcontainer/apparmor", <del> "Comment": "v1.0.0-rc2-49-gd223e2a", </del> <ins> "Comment": "v1.0.0-rc2-49-gd223e2ad", </ins> "Rev": "d223e2adae83f62d58448a799a5da05730228089" }, { "ImportPath": "github.com/opencontainers/runc/libcontainer/cgroups", <del> "Comment": "v1.0.0-rc2-49-gd223e2a", </del> <ins> "Comment": "v1.0.0-rc2-49-gd223e2ad", </ins> "Rev": "d223e2adae83f62d58448a799a5da05730228089" }, { "ImportPath": "github.com/opencontainers/runc/libcontainer/cgroups/fs", <del> "Comment": "v1.0.0-rc2-49-gd223e2a", </del> <ins> "Comment": "v1.0.0-rc2-49-gd223e2ad", </ins> "Rev": "d223e2adae83f62d58448a799a5da05730228089" }, { "ImportPath": "github.com/opencontainers/runc/libcontainer/cgroups/systemd", <del> "Comment": "v1.0.0-rc2-49-gd223e2a", </del> <ins> "Comment": "v1.0.0-rc2-49-gd223e2ad", </ins> "Rev": "d223e2adae83f62d58448a799a5da05730228089" }, { "ImportPath": "github.com/opencontainers/runc/libcontainer/configs", <del> "Comment": "v1.0.0-rc2-49-gd223e2a", </del> <ins> "Comment": "v1.0.0-rc2-49-gd223e2ad", </ins> "Rev": "d223e2adae83f62d58448a799a5da05730228089" }, { "ImportPath": "github.com/opencontainers/runc/libcontainer/configs/validate", <del> "Comment": "v1.0.0-rc2-49-gd223e2a", </del> <ins> "Comment": "v1.0.0-rc2-49-gd223e2ad", </ins> "Rev": "d223e2adae83f62d58448a799a5da05730228089" }, { "ImportPath": "github.com/opencontainers/runc/libcontainer/criurpc", <del> "Comment": "v1.0.0-rc2-49-gd223e2a", </del> <ins> "Comment": "v1.0.0-rc2-49-gd223e2ad", </ins> "Rev": "d223e2adae83f62d58448a799a5da05730228089" }, { "ImportPath": "github.com/opencontainers/runc/libcontainer/keys", <del> "Comment": "v1.0.0-rc2-49-gd223e2a", </del> <ins> "Comment": "v1.0.0-rc2-49-gd223e2ad", </ins> "Rev": "d223e2adae83f62d58448a799a5da05730228089" }, { "ImportPath": "github.com/opencontainers/runc/libcontainer/label", <del> "Comment": "v1.0.0-rc2-49-gd223e2a", </del> <ins> "Comment": "v1.0.0-rc2-49-gd223e2ad", </ins> "Rev": "d223e2adae83f62d58448a799a5da05730228089" }, { "ImportPath": "github.com/opencontainers/runc/libcontainer/seccomp", <del> "Comment": "v1.0.0-rc2-49-gd223e2a", </del> <ins> "Comment": "v1.0.0-rc2-49-gd223e2ad", </ins> "Rev": "d223e2adae83f62d58448a799a5da05730228089" }, { "ImportPath": "github.com/opencontainers/runc/libcontainer/selinux", <del> "Comment": "v1.0.0-rc2-49-gd223e2a", </del> <ins> "Comment": "v1.0.0-rc2-49-gd223e2ad", </ins> "Rev": "d223e2adae83f62d58448a799a5da05730228089" }, { "ImportPath": "github.com/opencontainers/runc/libcontainer/stacktrace", <del> "Comment": "v1.0.0-rc2-49-gd223e2a", </del> <ins> "Comment": "v1.0.0-rc2-49-gd223e2ad", </ins> "Rev": "d223e2adae83f62d58448a799a5da05730228089" }, { "ImportPath": "github.com/opencontainers/runc/libcontainer/system", <del> "Comment": "v1.0.0-rc2-49-gd223e2a", </del> <ins> "Comment": "v1.0.0-rc2-49-gd223e2ad", </ins> "Rev": "d223e2adae83f62d58448a799a5da05730228089" }, { "ImportPath": "github.com/opencontainers/runc/libcontainer/user", <del> "Comment": "v1.0.0-rc2-49-gd223e2a", </del> <ins> "Comment": "v1.0.0-rc2-49-gd223e2ad", </ins> "Rev": "d223e2adae83f62d58448a799a5da05730228089" }, { "ImportPath": "github.com/opencontainers/runc/libcontainer/utils", <del> "Comment": "v1.0.0-rc2-49-gd223e2a", </del> <ins> "Comment": "v1.0.0-rc2-49-gd223e2ad", </ins> "Rev": "d223e2adae83f62d58448a799a5da05730228089" }, {

Is this a BUG REPORT or FEATURE REQUEST? (choose one): bug report Kubernetes version (use ): Environment: Cloud provider or hardware configuration: vsphere OS (e.g. from /etc/os-release):Kernel (e.g. ): 4.9.24-coreos Install tools: Others: What happened: After bringing a cluster up, there are no container metrics in the kubelet at /metrics What you expected to happen: I expect to see the metrics, but there are none How to reproduce it (as minimally and precisely as possible): Anything else we need to know: If I try and access the metrics in cAdvisor directly on port 4194 instead of on port then I see the following type of error for all of the metrics:
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
/sig node
This appears to be a v1.7 issue. If I run 1.6.6 I get these metrics. Seems like something has happened in 1.7. I'm running the 1.7 beta because I need nodePort for cni, so now I'm stuck.
Can you triage the issue since?
cc/
Can you help verifying is this is a new regression introduced in 1.7 cadvisor through node-e2e? Thanks!
I've tried this on both cos and cvm in 1.7 cluster, and no container metrics show up. However, I created a 1.6.4 GKE cluster, and container metrics are there. This is a regression in 1.7. Will figure out what is wrong here.
Here is a corresponding cadvisor issue
Even the user still can get the container metrics / stats from kubelet /summary api (recommended way), but since this is a regression introduced in 1.7 release, and we aren't dropping prometheus support, let's fix this for 1.7. Thanks!
Fixed in Will close this issue after building new cadvisor release and cherrypick into 1.7
Thanks for the quick find everyone! I'll be happy to give this a go on the next release!

I'm still facing this same issue. My kubernetes cluster is deployed with kubeadm on Centos 7. I run a k8S v1.8.1 cluster. When I try to fetch containers metrics this way: http://localhost:8001/api/v1/proxy/nodes/node1:0/metrics http://localhost:8001/api/v1/proxy/nodes/node1:/metrics http://localhost:8001/api/v1/proxy/nodes/node1:/metrics I find no metrics. kubelet is started with . If I start kubelet with . metrics are available through .


Question: #   CLUSTER_NAME # Vars set: #   NODE_INSTANCE_GROUPS <ins> #   ALL_INSTANCE_GROUP_URLS </ins> function detect-node-instance-groups { echo "... in gke:detect-node-instance-groups()" >&2 local urls=$("${GCLOUD}" ${CMD_GROUP:-} container clusters describe  --project="${PROJECT}" --zone="${ZONE}"  --format='value(instanceGroupUrls)' "${CLUSTER_NAME}") urls=(${urls//;/ }) <ins> ALL_INSTANCE_GROUP_URLS=${urls[*]} </ins> NODE_INSTANCE_GROUPS=() for url in "${urls[@]:-}"; do <del> NODE_INSTANCE_GROUPS+=("${url##*/}") </del> <ins> local igm_zone=$(expr match ${url} '.*/zones/([a-z0-9-]*)/') if [[ "${igm_zone}" == "${ZONE}" ]]; then NODE_INSTANCE_GROUPS+=("${url##*/}") fi </ins> done }

The test job was passing consistently until midday on 6/18, e.g. Then it started failing consistently, e.g: ... with this error: 12:31:41 ... in gke:detect-node-instance-groups() 12:31:42 ERROR: (gcloud.compute.instance-) Some requests did not succeed: 12:31:42 - The resource 'projects/k8s-jkns-e2e-gke-multizone/zones/us-central1-f/instanceGroupManagers/gke-jenkins-e2e-default-pool--grp' was not found 12:31:42 Assigning to and to triage and assign someone to fix.
cc:
Almost certainly caused by . I'm not going to bother rushing in a rollback today since it's not a critical build, but could you fix this up tomorrow
Yeah, that's me. The change to support multiple MIGs in a single zone doesn't quite work. Didn't know of this test variation. Just to confirm, to run this test variation, the additional env vars I should set are (aside from the standard ones): export KUBERNETESPROVIDER=gke export ADDITIONALZONES=us-central1-a,us-central1-b Any other?
-- yet another instance of a PR passing all presubmit tests and breaking other jenkins jobs after being merged.
Can you tell me a bit more how this test is setup? I tried to do a "kube-" with the additional zones and it worked for me. My settings: export KUBERNETESPROVIDER=gke export ZONE=us-central1-f export CLOUDSDKBUCKET=gs://cloud-sdk-testing/ci/staging export CLOUDSDKAPIENDPOINTOVERRIDESCONTAINER=https://test- export ADDITIONAL_ZONES=us-central1-a,us-central1-b
Have you read ?
Oh, sorry, wrong link.
Take a look at and let me know if you need any more info.
The above PR was the one for GCE. The following is the one did for GKE. There are some relatively minor differences:
Hum, I'll have to dig a bit deeper. I tried the exact combination of env vars and my "kube-up" was successful (with the correct number of nodes across the zones).

Curious...
Any ETA, I'm getting tired of all the test failure emails :p
I think I found the problem. I have a workaround that'll make it work for now. I'm hoping sometime this afternoon.
The PR merged and gke-multizone Jenkins has gone green. Sorry for the trouble. There's still a larger issue of multi-zone handling in the test runner itself, but I think that's for another day.


Question: svc = jig.WaitForLoadBalancerOrFail(namespace, serviceName, loadBalancerCreateTimeout) jig.SanityCheckService(svc, v1.ServiceTypeLoadBalancer) <ins> // timeout when we haven't just created the load balancer normalReachabilityTimeout := 2 * time.Minute </ins> By("check reachability from different sources") svcIP := framework.GetIngressPoint(&svc.Status.LoadBalancer.Ingress[0]) <del> framework.CheckReachabilityFromPod(true, namespace, acceptPodName, svcIP) framework.CheckReachabilityFromPod(false, namespace, dropPodName, svcIP) </del> <ins> // Wait longer as this is our first request after creation.  We can't check using a separate method, // because the LB should only be reachable from the "accept" pod framework.CheckReachabilityFromPod(true, loadBalancerLagTimeout, namespace, acceptPodName, svcIP) framework.CheckReachabilityFromPod(false, normalReachabilityTimeout, namespace, dropPodName, svcIP) </ins> By("Update service LoadBalancerSourceRange and check reachability") jig.UpdateServiceOrFail(svc.Namespace, svc.Name, func(svc *v1.Service) { // only allow access from dropPod svc.Spec.LoadBalancerSourceRanges = []string{dropPod.Status.PodIP + "/32"} }) <del> framework.CheckReachabilityFromPod(false, namespace, acceptPodName, svcIP) framework.CheckReachabilityFromPod(true, namespace, dropPodName, svcIP) </del> <ins> framework.CheckReachabilityFromPod(false, normalReachabilityTimeout, namespace, acceptPodName, svcIP) framework.CheckReachabilityFromPod(true, normalReachabilityTimeout, namespace, dropPodName, svcIP) </ins> By("Delete LoadBalancerSourceRange field and check reachability") jig.UpdateServiceOrFail(svc.Namespace, svc.Name, func(svc *v1.Service) { svc.Spec.LoadBalancerSourceRanges = nil }) <del> framework.CheckReachabilityFromPod(true, namespace, acceptPodName, svcIP) framework.CheckReachabilityFromPod(true, namespace, dropPodName, svcIP) </del> <ins> framework.CheckReachabilityFromPod(true, normalReachabilityTimeout, namespace, acceptPodName, svcIP) framework.CheckReachabilityFromPod(true, normalReachabilityTimeout, namespace, dropPodName, svcIP) </ins> }) })

Sub-issue for For AWS ELB, when an ELB is first created, is made it is very normal that it takes 2 minutes to be created, but the DNS name is reported fairly quickly.


Question: package anonymous import ( <ins> "net/http" </ins> "testing" "k8s.io/apimachinery/pkg/util/sets"

Semantics are going to vary. We need to cross off these authenticators: [x] if ctx has an audience limitation, verify the token audience and echo in the response [x] take implicit audiences at WebhookTokenAuthenticator construction time if ctx is audience-limited, add into the tokenreview object if the tokenreview returns with an audience set that matches the ctx, copy into the response and return success if the tokenreview returns without an audience set, ensure the ctx audiences intersect with the implicit audiences taken at WebhookTokenAuthenticator construction time, and pick one to add into the response [x] echo audience in response if present in ctx [x] echo audience in response if present in ctx [x] check that the the order of (audience in the context) ∩ ( (implicit api audiences) ∪ (clientids) ) 0 Take implicit audiences at Authenticator construction time, and if ctx is audience-limited, ensure it intersects with the implicit audiences, and echo an audience in the response. [x] [x] [x] [x] [x] [x] Do nothing? [x] [x] [x] [x] [x] [x] [x] note: we already keyed the cache with the audience. [x]
There are no sig labels on this issue. Please add a sig label by either: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. See the . The in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/close
Closing this issue. <detailsIn response to : /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: }, }, Requested: &Resource{ <del> MilliCPU:         1300, Memory:           1000, </del> <ins> MilliCPU:         2300, Memory:           209716700, //1500 + 200MB in initContainers </ins> EphemeralStorage: 0, AllowedPodNumber: 0, ScalarResources:  map[v1.ResourceName]int64(nil), }, NonZeroRequested: &Resource{ <del> MilliCPU:         1300, Memory:           209716200, //200MB + 1000 specified in requests/overhead </del> <ins> MilliCPU:         2300, Memory:           419431900, //200MB(initContainers) + 200MB(default memory value) + 1500 specified in requests/overhead </ins> EphemeralStorage: 0, AllowedPodNumber: 0, ScalarResources:  map[v1.ResourceName]int64(nil),

The scheduler does not properly calculate pod requests for a resource as - it ignores init containers completely in . This would allow a workload to get "free" scheduling above the limit of containers (mitigated by quota, which calculates this properly) and so the scheduler would make non-optimal fits. Both kubelet and scheduler enforce the , so you can't actually schedule, but you sometimes get steered to the wrong node if you have more init container resource usage than containers. in should use and the similar helpers instead of writing its own method to calculate the request for resources. /sig scheduling
/priority important-soon
Let me follow this issue. /assign
fyi
After I checked the implementation of function, I think we just need to use it to instead of the function in the scheduler.
/priority critical-urgent
Is the pod phase not considered? When the pod phase is running, the requested resource is actually considered as . This change seems to reduce the resource utilization of the node.
Actually, I don't think we need to consider the pod.Phase. The resource request & limit are mainly used in the scheduling process. The scenario you said is not belong to the responsibility of kube-scheduler
Correct. Setting appropriate requests/limits that maintain good node usage is users responsibility. Considering the max(initContainers, sum(containers)) is the documented behavior.
I understand. Thank you for your comments!


Question: IfName:      network.DefaultInterfaceName, } } <ins> func (plugin *kubenetNetworkPlugin) addContainerToNetwork(id kubecontainer.ContainerID, rt *libcni.RuntimeConf) error { plugin.mu.Lock() defer plugin.mu.Unlock() glog.V(3).Infof("Calling cni plugins to add container to network with cni runtime: %+v", rt) res, err := plugin.cniConfig.AddNetwork(plugin.netConfig, rt) if err != nil { return fmt.Errorf("Error adding container to network: %v", err) } if res.IP4 == nil || res.IP4.IP.String() == "" { return fmt.Errorf("CNI plugin reported no IPv4 address for container %v.", id) } plugin.podCIDRs[id] = res.IP4.IP.String() return nil } func (plugin *kubenetNetworkPlugin) delContainerFromNetwork(id kubecontainer.ContainerID, rt *libcni.RuntimeConf) error { plugin.mu.Lock() defer plugin.mu.Unlock() glog.V(3).Infof("Calling cni plugins to remove container from network with cni runtime: %+v", rt) if err := plugin.cniConfig.DelNetwork(plugin.netConfig, rt); err != nil { return fmt.Errorf("Error removing container from network: %v", err) } delete(plugin.podCIDRs, id) return nil } </ins>

I gathered e2e test results of suite for last 100 runs. The following is the list of flaky tests. Some might not be related to . I will start with the service tests. Any help would be appreciated. Detail list: Jenkins project:
cc/ Cluster team is working on to make kubenet default. Once we switch this to production, we don't need to restart docker to configure cbr0.
After a bunch of PRs got merged, the kubenet suite is much more stable. One more known flake can be fixed by I will post new flake tests here.
Just bump into a new failure: One of the nodes () got panic: Looks like it crashed at shaper reset.: Can you take a look?


Question: } scaleTest := &HPAScaleTest{ initPods:                    5, <del> totalInitialCPUUsage:        375, perPodCPURequest:            500, </del> <ins> totalInitialCPUUsage:        650, perPodCPURequest:            1000, </ins> targetCPUUtilizationPercent: 30, minPods:                     1, maxPods:                     5,

Test: https://k8s- Example: https://k8s- The HPA tests for gce-serial are flaking rather regularly; they fail about 2/3 of the time for the last week, and are often the cause of a failed test run. Can we de-flake these? Over multiple test runs the problem seems to be the number of replicas jumping up to 4 unexpectedly: Is the limit not getting set correctly here? Is this an actual bug? /sig autoscaling /priority important-soon /kind failing-test /kind flake
cc:
Recent failures fall into three categories: not starting (, , , , , , ). They fail with error message: increasing unexpectedly (, , ). They fail with error message: waiting for scale down (). This one failed with error message: I suspect 2. and 3. may be caused by test using correct amount of CPU when averaged over minute-long window but spiky (for example using 100% cpu for some time then 0% cpu for rest of the minute). I saw similar behavior when validating recent changes manually. This would result in HPA: behaving as the test expects with before recent changes (because it ignored any spikes during scale*forbidden windows, longer sample windows masking the spike), HPA (correctly) increasing size if spikes align after those changes (failures in group 2.), HPA (correctly) not decreasing size if spikes align after those changes (failure in group 3.), I'll take a look at code of those tests and try to responsible for most of the flakes but I don't have any ideas for a cause yet.
/remove-kind failing-test
I was busy this week but I think I'll be able to prepare a fix for this tomorrow.
/milestone v1.13 /kind failing-test
this tests is failing consistently in
On Friday I verified that CPU usage generated by resource consumer stays really close to the target value but oscillates slightly. To fix this I: Increase CPU usage in the test (resource consumer is implemented in a way that makes me think that deviation fro target are of a fixed size so higher target would make deviation a smaller percent of the target). Lower generated load (the test right now is targeting border between recommendation of 3 instances and 4 instances, I will change the load to something between 2 and 3 instances).

I'm checking if this helps the test.
It seems to have helped. I've sent a PR with those changes for review.
Re-opening until we have at least 3 green runs: https://k8s- /reopen
Reopening this issue. <detailsIn response to : Re-opening because we just had another flake last night: Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
But it doesn't work on test grid, the test now fails with: I'll look into this.
Yes, and it's now causing another test to fail as well: https://k8s- Escalating to critical-urgent. This test has been failing for 3 weeks now, and is potentially blocking visibility into other failures. /priority critical-urgent /remove-priority important-soon
This test had a couple of successful runs yesterday but is now flaking/failing in master-blocking: https://k8s- It seems to be passing in master-upgrade. Any chance we could prioritize work on it? It's preventing us from seeing whether fixes for other test failures are working.
There was only 1 flake since merged on Thursday (and ~15 green runs). was an unexpected scale up happening during stabilization. I'll check if lowering CPU utilization a bit lower deflakes the test more but I think it's ok to close this issue now.
Lowering CPU utilization didn't help.
Test jobs where these tests have flaked or failed: https://k8s- (today) https://k8s- (last 3 days ago) https://k8s- (yesterday) https://k8s- (yesterday) https://k8s- (today) https://k8s- (yesterday) ... basically, everywhere the tests run, they're flaky as recently as today. More on Triage: You can see here that across all jobs the HPA tests are flaky, with a "worst flake" perioud of 10/23-10/25, but by no means cleared up.
SIG-Autoscaling's boards are also showing flakiness: https://k8s-, although that looks better than the sig-release test boards. Any idea why?

Thanks for the links I've been looking only at one of those dashboards. I've sent that should reduce flakiness of our tests.
Thanks!
The PR merged. Now let's wait and see if it solves the problem.
The test named in the tile of this issue<sup1</supgot better. Last failures at the dashboards you listed: , (before merged), error before taht was on 10-25), , , . So it looks like the test didn't flake in the last ~2 days. Another two tests<sup2</supare flaking , I'll send a PR that should fix it. I think those flakes are result of behavior of HPA intentionally changing<sup3</supso the PR will just relax expectations of the test. <b1</b[sig-autoscaling] [HPA] Horizontal pod autoscaling (scale resource: CPU) [sig-autoscaling] [Serial] [Slow] ReplicationController Should scale from 5 pods to 3 pods and from 3 to 1 and verify decision stability <b2</b[sig-autoscaling] [HPA] Horizontal pod autoscaling (scale resource: Custom Metrics from Stackdriver) should scale down with External Metric with target value from Stackdriver [Feature:CustomMetricsAutoscaling] and [sig-autoscaling] [HPA] Horizontal pod autoscaling (scale resource: Custom Metrics from Stackdriver) should scale down with External Metric with target average value from Stackdriver [Feature:CustomMetricsAutoscaling] <b3</bHPA no longer delays scale ups based on metrics other than CPU. Flakes are when HPA scales deployment to 2 or 3 so test doesn't observe it having size 1 and times out waiting for that size.
I misread titles of custom metric tests as tests for scaling up somehow. Those flaking tests are tests for scaling down and scale downs sometimes don't happen. I'm looking into why.
While this test has gotten better, it's still flaking some of the time: https://k8s-
I looked at "Latest failures" in the two top categories according to (all the other categories have almost no recent flakes). All of the failures log as if they were not affected by : instead of .

Tests results also say they are running version 1.11... or 1.12... (or more rarely 1.9... or 1.10... but I'm ignoring those for now - they are not related to any changes I made). So it looks like I need to cherry pick the PR into 1.11 and 1.12 branches.
... when I look at git branches it looks like my PR is there. There was with logs that looking like affected it. I'll wait to see if there are more failures in test runs including that change. In case there are I'll add some more logging (I have no ideas for cause of the one failure I saw so far).
/assign
GitHub didn't allow me to assign the following users: jbartosik. Note that only and repo collaborators can be assigned. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
any further update on this? I see it turning into more of a failure than a flake and I see another test failing as well in latest runs
We are due to cut Beta for 1.13 today and would like to understand if this should be a blocker or not.
Also are there cherrypick PRs to backport your change to 1.11 and 1.12 branches?
Potentially related:
6 (consecutive) most recent failures of test do not have judging by their logs. That PR would change them from failures (unexpected scale from 3 to 4 instances during stabilisation) to success (expected 3 to 4 instances during stabilisation). One of the 2 recent failures of looks wrong. One failure was scaling up immediately to 5 instances (wrong). The other was scale up to 4 instances (ok, I'll send a PR relaxing that condition tomorrow). I didn't send PRs cherrypicking test changes to 1.12 (because when I checked it looked like the change was there) not to 1.11 (my changes aren't there so old tests should keep working). merged and it ads more logging, which will help me debug this. Failures in the look like the PR relaxing stability requirement should have helped there too.

the , which has the 6 (consecutive) most recent failures , upgrades the master alone to 1.13(master) and and keep the nodes in old version and then run the old test suite. The latest failing runs show that the tests are run from 1.11.5 branch Doesn't this mean your changes need to be backported to 1.11 branch? That is the reason why you dont see you change in those tests logs
FWIW, the same test also flakes in job which upgrade both master and nodes to 1.13 and runs 1.11 tests.
this failure is currently blocking Beta for 1..13. Can you please investigate these and let us know the path forward.
Fixes done by are all test fixes, not feature fixes. So the problem should be fixed once cherry-pick gets to 1.11. That being said - why are we running 1.11 e2e tests against 1.13 cluster? That assumes full compatibility of both API and implementation. We can never remove a feature without breaking this suite (the tests for the feature will still exist on older branches after all). And the tests are often tweaked to the specifics of implementation. Whenever we make any change to autoscaling algorithm tests need to be updated accordingly. Making tests on 1.11 work well with 1.13 risks they will stop working well with 1.11. Shouldn't the upgrade job create 1.11 cluster, upgrade it to 1.13 and run 1.13 e2e tests on that cluster?
I see these 2 pass in the latest run of https://k8s- Over to to close when he sees fit
We have other flakes on that job now, but HPA is no longer flaking. Thanks so much for the dilligence in getting these flakes cleared up, /close
Closing this issue. <detailsIn response to : We have other flakes on that job now, but HPA is no longer flaking. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: } pod.ResourceVersion = "" <ins> watching, err := h.Watch(context.TODO(), key, "0", storage.Everything) if err != nil { t.Fatalf("Unexpected error: %v", err) } // The create trigger ADDED event when watching from 0 event := <-watching.ResultChan() watching.Stop() if event.Type != watch.Added { t.Errorf("Unexpected event %#v", event) } </ins> // check for concatenation on watch event with CAS updateFn := func(input runtime.Object, res storage.ResponseMeta) (runtime.Object, *uint64, error) { pod := input.(*api.Pod)

Currently, Watch request with resourceVersion=0 results in: returning a single "watch event" for every object existing "now" starts watching from that point This is pretty misleading (e.g. it's different that what we have in etcd, where there is no first step from above). In the v2 api, we should probably eliminate the first stop (or even disallow watch requests with resourceVersion = 0).
/cc
Issues go stale after 30d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
/remove-lifecycle rotten /lifecycle frozen
What do you mean by v2 API? Are we ever going to have a v2 of the machiery and semantics?
In 2015 I thought it's not ridiculous. I'm not sure I believe in it now. Moreover, with we will actually be taking benefits of semantic like this, so we should close this.


Question: <ins> // +build !windows /* Copyright 2020 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package kuberuntime import ( "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "github.com/stretchr/testify/assert" "testing" ) func TestVerifyRunAsNonRoot(t *testing.T) { pod := &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ UID:       "12345678", Name:      "bar", Namespace: "new", }, Spec: v1.PodSpec{ Containers: []v1.Container{ { Name:            "foo", Image:           "busybox", ImagePullPolicy: v1.PullIfNotPresent, Command:         []string{"testCommand"}, WorkingDir:      "testWorkingDir", }, }, }, } rootUser := int64(0) anyUser := int64(1000) runAsNonRootTrue := true runAsNonRootFalse := false for _, test := range []struct { desc     string sc       *v1.SecurityContext uid      *int64 username string fail     bool }{ { desc: "Pass if SecurityContext is not set", sc:   nil, uid:  &rootUser, fail: false, }, { desc: "Pass if RunAsNonRoot is not set", sc: &v1.SecurityContext{ RunAsUser: &rootUser, }, uid:  &rootUser, fail: false, }, { desc: "Pass if RunAsNonRoot is false (image user is root)", sc: &v1.SecurityContext{ RunAsNonRoot: &runAsNonRootFalse, }, uid:  &rootUser, fail: false, }, { desc: "Pass if RunAsNonRoot is false (RunAsUser is root)", sc: &v1.SecurityContext{ RunAsNonRoot: &runAsNonRootFalse, RunAsUser:    &rootUser, }, uid:  &rootUser, fail: false, }, { desc: "Fail if container's RunAsUser is root and RunAsNonRoot is true", sc: &v1.SecurityContext{ RunAsNonRoot: &runAsNonRootTrue, RunAsUser:    &rootUser, }, uid:  &rootUser, fail: true, }, { desc: "Fail if image's user is root and RunAsNonRoot is true", sc: &v1.SecurityContext{ RunAsNonRoot: &runAsNonRootTrue, }, uid:  &rootUser, fail: true, }, { desc: "Fail if image's username is set and RunAsNonRoot is true", sc: &v1.SecurityContext{ RunAsNonRoot: &runAsNonRootTrue, }, username: "test", fail:     true, }, { desc: "Pass if image's user is non-root and RunAsNonRoot is true", sc: &v1.SecurityContext{ RunAsNonRoot: &runAsNonRootTrue, }, uid:  &anyUser, fail: false, }, { desc: "Pass if container's user and image's user aren't set and RunAsNonRoot is true", sc: &v1.SecurityContext{ RunAsNonRoot: &runAsNonRootTrue, }, fail: false, }, } { pod.Spec.Containers[0].SecurityContext = test.sc err := verifyRunAsNonRoot(pod, &pod.Spec.Containers[0], test.uid, test.username) if test.fail { assert.Error(t, err, test.desc) } else { assert.NoError(t, err, test.desc) } } } </ins>

Hello, What happened: When we apply PodSecurityPolicy with rule: WIndows containers are still blocked instead we put: the pod is still blocked with logs: What you expected to happen: the Container should start How to reproduce it (as minimally and precisely as possible): Create Psp disableling Root Users: Create a pod Windows with runAsUserName: Anything else we need to know?: Environment: Kubernetes version (use ): v1.18.3 Cloud provider or hardware configuration: OS (e.g: ): CentOS Linux 7 (Core) / windows server 2019 version 1809 Kernel (e.g. ): 3.10.0-514.10.2.el7.x86_64 Others: /sig windows
/sig windows
/assign I will test it and then post the test results
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale


Question: return client } <ins> func (t *DynamicControllerClientBuilder) DiscoveryClient(name string) (discovery.DiscoveryInterface, error) { clientConfig, err := t.Config(name) if err != nil { return nil, err } // Discovery makes a lot of requests infrequently.  This allows the burst to succeed and refill to happen // in just a few seconds. clientConfig.Burst = 200 clientConfig.QPS = 20 return clientset.NewForConfig(clientConfig) } func (t *DynamicControllerClientBuilder) DiscoveryClientOrDie(name string) discovery.DiscoveryInterface { client, err := t.DiscoveryClient(name) if err != nil { klog.Fatal(err) } return client } </ins> type tokenSourceImpl struct { namespace          string serviceAccountName string

Ran kube-controller-manager with and observed this in the log every 30 seconds: This is also happening in 1.20, 1.19, and 1.18 The GC and quota clients look up discovery information every 30 seconds. They have a QPS of 20 requests per second and a burst of 30 requests. Default configuration of kube-controller-manager runs without constant throttling warnings. /sig api-machinery
/assign /triage accepted


Question: for i, pod := range pods { p, err := params.getPodFn(pod.Namespace, pod.Name) if apierrors.IsNotFound(err) || (p != nil && p.ObjectMeta.UID != pod.ObjectMeta.UID) { <del> if params.onDoneFn != nil { </del> <ins> if params.onFinishFn != nil { params.onFinishFn(&pod, params.usingEviction, nil) } else if params.onDoneFn != nil { </ins> params.onDoneFn(&pod, params.usingEviction) } continue } else if err != nil { <ins> if params.onFinishFn != nil { params.onFinishFn(&pod, params.usingEviction, err) } </ins> return false, err } else { if shouldSkipPod(*p, params.skipWaitForDeleteTimeoutSeconds) {

kubedrain has "onPodDeletedOrEvicted" callback that can be used in custom controllers. It would be very helpful if we add more callbacks that can give more granular control over monitoring, like "onPodDeletionOrEvicitionStarted" and "onPodDeletionOrEvictionFailed". It would help monitor the draining of nodes on a more granular level. Sometimes while draining, if the pod is stuck during the eviction or deletion of node, there is not much information that custom controllers can get from kubedrain, with the above-mentioned callbacks custom controllers will know which pod is evicting or if it failed during the eviction process.
/sig node
What is the custom controllers in your scenario? Can you elaborate on what you want to achieve with more details. Is this is kubectl improvement, the best sig to involve is sig cli. /sig cli
/remove-sig node please re-apply the sig node if this will end up being a request to sig node
In our case we have in CAPI (). Sometimes while draining, if the pod or volume is stuck during the eviction, there is not much information that we get from kubedrain.Helper. It would be much easier for the controller to report these kinds of issues with more granular callbacks like "onPodDeletionOrEvicitionStarted" and "onPodDeletionOrEvictionFailed" on kubedrain.Helper.
/assign Assigning this to myself, I have discussed this issue in sig-cli community meeting and they have given green light for this improvement.
/triage accepted /priority backlog


Question: {Host: "nodeC", Score: 0}, }, }, <ins> { test: "Default behaviour No taints and tolerations, lands on node with no taints", //pod without tolerations pod: podWithTolerations([]v1.Toleration{}), nodes: []*v1.Node{ //Node without taints nodeWithTaints("nodeA", []v1.Taint{}), nodeWithTaints("nodeB", []v1.Taint{ { Key:    "cpu-type", Value:  "arm64", Effect: v1.TaintEffectPreferNoSchedule, }, }), }, expectedList: []schedulerapi.HostPriority{ {Host: "nodeA", Score: 10}, {Host: "nodeB", Score: 0}, }, }, </ins> } for _, test := range tests { nodeNameToInfo := schedulercache.CreateNodeNameToInfoMap(nil, test.nodes)

Kubernetes version (use kubectl version): master What happened: Unnecessary always nil return value for function getTolerationListFromPod. What you expected to happen: We need to remove error return value for getTolerationListFromPod function.
/cc
cc


Question: diskID := "" err = kwait.ExponentialBackoff(defaultBackOff, func() (bool, error) { <del> provisionState, id, err := c.getDisk(diskName) </del> <ins> provisionState, id, err := c.getDisk(resourceGroup, diskName) </ins> diskID = id // We are waiting for provisioningState==Succeeded // We don't want to hand-off managed disks to k8s while they are

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: there will be azure disk creation issue when specifying an external resource group, after azure disk creation succeeded, it fails to get azure disk state What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): v1.11 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: /assign /azure
/sig azure
/kind bug


Question: // Run start a goroutine to try to delete rs in the graceful delete rsList with an interval 1 minute func (m *GracefulTerminationManager) Run() { <del> // before start, add leftover in delete rs to graceful delete rsList vss, err := m.ipvs.GetVirtualServers() if err != nil { klog.Errorf("IPVS graceful delete manager failed to get IPVS virtualserver") } for _, vs := range vss { rss, err := m.ipvs.GetRealServers(vs) if err != nil { klog.Errorf("IPVS graceful delete manager failed to get %v realserver", vs) continue } for _, rs := range rss { m.GracefulDeleteRS(vs, rs) } } </del> go wait.Until(m.tryDeleteRs, rsCheckDeleteInterval, wait.NeverStop) }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: After upgrading to kube-proxy , the ClusterIP will be disconnected when restarting kube-proxy. When a service uses ClusterIP to connect to other services, it will be affected. As shown in the figure, during the kube-proxy restart, the response time is obviously slower. ! When back to kube-proxy , no this problem. this is the log, attention the time, kube-proxy restart at , see the picture above, response time at the time is normal. when use kube-proxy , ClusterIP of will also be disconnected, name resolve will failed ! What you expected to happen: ClusterIP won't lost connection when kube-proxy restart How to reproduce it (as minimally and precisely as possible): Use kube-proxy Anything else we need to know?: Add flag , same result ! Environment: Kubernetes version (use ): 1.12.5 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): CentOS 7 Kernel (e.g. ): 3.10 Install tools: manual Others:
/area ipvs
Reiterating the mentions to trigger a notification: <detailsIn response to : /area ipvs Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
perhaps it was introduced by graceful termination
I'll have a look this week (probably not today) It could be caused by the VS being cleaned up on startup (not likely) or the clusterIP being unbound and rebound
Is there any progress on this issue?
I've started looking into this Just to understand well: you notice an increased latency for new connections to a clusterIP during the restart phase? Things are fine during the deletion period ? (and while kube-proxy is not running?)
It seems you mean that for a short while the clusterIP is not even working (given what you mentioned on coredns). Is this correct?

Yes, seems ipvs rule will be recreated when restart kube-proxy
I think I have reproduced this (using wrk2 and deleting the kube-proxy process while it's running) I wonder if it is related to this part of the code that is run at start-up: It will delete all RS that have no connections and set the weight of the other to 0, which means we won't be able to establish connections until syncProxyRules runs I don't know the reason for doing this at initialization. / you probably have more context. I can still create a build to validate this hypothesis
: I tested my idea (not deleting RS at start-up) and initial tests look promising: max latency does not change during a restart of the kube-proxy process (in my very simple wrk test) VS and RS reconciliation still work fine without the deletion (very limited testing there: I deleted a service and scaled down a deployment backing another while kube-proxy was not running and at restart everything was immediately resynced) Here is the diff against the branch: If you can try on a test cluster, I have a public test image here: Again, I don't know the reason for deleting all RS at start-up in the first place, so be careful (and let me know of you notice anything weird).
I had compiled kube-proxy use the new code in , and tested with , When restart kube-proxy, all DNS queries have succeeded while original kube-proxy will have some failed query. Thanks!
this looks really promising I'm creating a PR. If there is any doubt on the impacts we can discuss there
Turns out the iptable proxier has similar issues (for different reasons):


Question: expectedAPIConfig: func() *serverstore.ResourceConfig { config := newFakeAPIResourceConfigSource() config.EnableVersions(scheme.PrioritizedVersionsAllGroups()...) <del> config.EnableResources( extensionsapiv1beta1.SchemeGroupVersion.WithResource("deployments"), extensionsapiv1beta1.SchemeGroupVersion.WithResource("replicasets"), extensionsapiv1beta1.SchemeGroupVersion.WithResource("daemonsets"), ) </del> return config }, err: false,

https://k8s- sample failure: (also affecting other CI jobs) cc /sig gcp
/assign We're interested in disabling the job as blocking entirely. We're of the opinion this shouldn't have bee allowed to block the community in the first place. I swear there's a historical issue around tightening the regex somewhere.
... wrong button sorry about that
possibly caused by erroring on invalid values that were previously silently skipped opened to downgrade that error to a warning for a release
Well, I mentioned the idea of making the job optional in
PR out to drop the runifchanged regex from pull-kubernetes-e2e-gke
Confirmed turns the test green
Can call this closed when merges
/pony hooray
Couldn't find a pony matching that query. <detailsIn response to : /pony hooray Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: "github.com/golang/glog" <ins> "k8s.io/api/core/v1" </ins> "k8s.io/apimachinery/pkg/api/errors" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/types"

/kind bug /sig network What happened: Networking tight loop causing GCP Cloud rate limiting to kick in. Causing tests to fail. Suspected call stack: Looks like it could be the root cause of these test failures here: https://k8s- These storage tests are hitting rate limit on the cloud api when trying to detach/delete disks. However, we have exponential backoff for all these functions. But the rate limiting from an external source is affecting the tests. See logs snippet: Note: These logs span exactly 1 minute. Try grepping "rateLimit" or "CIDRNotAvailable" -also note that the CIDRNotAvailable error messages are probably not verbose enough as they do not pinpoint what the actual error is, you can only associate them by correlation in timing and then looking at the code. Shorter snippet from above log file to demonstrate: since this is running on hundreds of nodes hitting this API call should involve some sort of exponential backoff as well as jitter. /cc /assign
/assign
/assign
/sig cloud-provider /sig gcp /milestone v1.12 (since is marked as v1.12)
may we have a priority on this since Code Freeze is next week? :)
Quick question, are we using the default API rate limits quota for scalability projects?
I will defer to for priority because he probably has the most context on how serious this is and what it could affect
Which rate limit are you referring to? (I don't remember us changing it any time) Btw - this is the project being used for that job -
I put up a patch for adding exponential retry
Gentle reminder that for this to be considered for 1.12 past Code Freeze, which is next week, it must have a status/critical-urgent label. I see this is being actively worked on


Question: "k8s.io/apimachinery/pkg/runtime/schema" "k8s.io/apimachinery/pkg/util/validation/field" genericapirequest "k8s.io/apiserver/pkg/endpoints/request" <del> "k8s.io/apiserver/pkg/features" </del> "k8s.io/apiserver/pkg/storage/names" <del> utilfeature "k8s.io/apiserver/pkg/util/feature" </del> "k8s.io/kubernetes/pkg/api/legacyscheme" "k8s.io/kubernetes/pkg/apis/flowcontrol" "k8s.io/kubernetes/pkg/apis/flowcontrol/validation"

In 1.30, Priority and Fairness should allow a zero value, either via v1 or v1beta3 (with the roundtrip annotation) in create and update requests See:
/sig api-machinery /cc /assign
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /triage accpted Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage accepted


Question: var ( // TODO: Deprecate gitMajor and gitMinor, use only gitVersion instead. gitMajor     string = "0"              // major version, always numeric <del> gitMinor     string = "2+"             // minor version, numeric possibly followed by "+" gitVersion   string = "v0.2-dev"       // version from git, output of $(git describe) </del> <ins> gitMinor     string = "3+"             // minor version, numeric possibly followed by "+" gitVersion   string = "v0.3-dev"       // version from git, output of $(git describe) </ins> gitCommit    string = ""               // sha1 from git, output of $(git rev-parse HEAD) gitTreeState string = "not a git tree" // state of git tree, either "clean" or "dirty" )

found this in while finding beta API dependencies for 1.22. This will block from running at all in 1.22 I think this is for this deployment: , and in ingress-gce I found
/sig network /triage accepted I pinged some GCE sig-network folks in the PR
given that uses extensions/v1beta1 Deployments, that particular manifest seems unused
once we land the fix in , anyone know how to update the image we use in e2e tests?
/assign The update to use v1 APIs is merged to in What version should be tagged we can pick up in Does need to be picked back to one or more release branches?
It would be helpful to get a version containing the fix tagged, so the manifest can be updated in to get e2e signal even before opens for 1.22.
Ack. We are planning to cut a release soon. Once the release is cut, I will update the k/k to use that
what's the ETA for cutting ingress-gce release? I think it might be related to failing pull-kubernetes-e2e-gce-100-performance run in


Question: # keep request = limit to keep this container in guaranteed class limits: cpu: 10m <del> memory: 20Mi </del> <ins> memory: 50Mi </ins> requests: cpu: 10m <del> memory: 20Mi </del> <ins> # Note that this container shouldn't really need 50Mi of memory. The # limits are set higher than expected pending investigation on #29688. # The extra memory was stolen from the kubedns container to keep the # net memory requested by the pod constant. memory: 50Mi </ins> args: - -cmd=nslookup kubernetes.default.svc.$DNS_DOMAIN 127.0.0.1 >/dev/null && nslookup kubernetes.default.svc.$DNS_DOMAIN 127.0.0.1:10053 >/dev/null - -port=8080

Observed weird memory spikes for both the dns healthz and kube-dns pods in production. Kube-dns didn't oom because there's enough headroom, but healthz did. I can't explain the spikes yet. I tried a bunch of different repros: no , bad nslookup command, sharing the node with cpu hogs etc, but was unable to trigger it. The memory grows, but stays under 10Mi. As a short term mitigation, we stole some ram from kube-dns, because it was always comfortably within limits. Related b/
can be closed -- healthz has been removed
-- close issue?


Question: "context" "encoding/json" "fmt" <ins> "path/filepath" </ins> "reflect" "strings" "testing"

Improve the to ensure that the expected storage version for a resource is one understood by previous releases. We have API testdata from prior versions available in When checking each group/version/kind in the test: accumulate all the versions of the group/kind available in previous (non-HEAD) releases if there is a beta or stable version of the group/kind available in previous releases, then the storage version must be in the set of previously available versions It is too easy to accidentally change the storage version prematurely and edit the test/integration/etcd test to match the bug cc
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: }, nil } <ins> // parseACRLoginServerFromImage takes image as parameter and returns login server of it. // Parameter `image` is expected in following format: foo.azurecr.io/bar/imageName:version // If the provided image is not an acr image, this function will return an empty string. func parseACRLoginServerFromImage(image string) string { match := acrRE.FindAllString(image, -1) if len(match) == 1 { return match[0] } return "" } </ins> func (a *acrProvider) LazyProvide(image string) *credentialprovider.DockerConfigEntry { return nil }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: PR() fixed pull image error from a cross-subscription azure container registry by service principal, while for MSI, this issue is not fixed. Need to find a way how to fix this issue for MSI, related code is here: Here the difficulty is could not list all ACR since it's based on one subscriptionID: So if we could not get in a different subscription, how could we set the . What you expected to happen: How to reproduce it (as minimally and precisely as possible): error msg would be like following when pull a repository from an ACR in another subscription: Anything else we need to know?: Environment: Kubernetes version (use ): v.9 - v1.12 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: /sig azure /assign
By further investigation, I think we should implement LazyProvider for azure acrProvider: Related code:
Update: It's hard to fix this since in the beginning we cannot list ACR in another subscription, will always list acr under current subscription, and without target loginServer address of ACR, we cannot get : And in func, we cannot get target repo, it's not passed on
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
-- is this still an issue? If not, let's it out.
it's still an issue, and currently it should be low priority, let's keep it
Rotten issues close after 30d of inactivity.

Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-lifecycle rotten /reopen
Reopened this issue. <detailsIn response to : /remove-lifecycle rotten /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: "path/filepath" "testing" <ins> "github.com/peterbourgon/diskv" </ins> "github.com/stretchr/testify/assert" )

We use a lot of CRDs (frequently more than 1,000) in the project. This has historically caused a bunch of problems with client-side API discovery, among other things. At the time of writing the latest releases of kubectl use a discovery client rate limiter with a generous QPS and burst (50 qps, 300 qps burst), so client-side rate limiting doesn't become a problem until the API server is serving more than 300 API groups. Despite rate-limiting no longer coming into effect, we noticed that discovery was very slow - taking 10-20 seconds to complete. After having a few folks at run a little test (see and following comments) we noticed that discovery specifically seems slow on MacOS - it's dramatically faster on Linux. Here's discovery happening on Linux with 1,577 CRDs across 343 API groups in around a second: ` Here's the same test run from MacOS against the exact same EKS cluster, taking 12+ seconds. I expected kubectl discovery on MacOS to be as fast as it was on Linux. an EKS cluster. ~1,500 CRDs across ~350 API groups. the discovery cache in how long the first command you run against the EKS cluster takes to complete Refer to and subsequent comments for more examples of folks reproducing this. <detailsNote that the version of kubectl was the same on Mac and Linux for my tests. </details<detailsAWS - EKS </details<detailsMacOS 12.4, M1 Max Linux as below </details<detailsN/A </details<detailsN/A </details<detailsN/A </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cli
will show you how many milliseconds was the GET request taken. There are 37 GET requests after was removed. Is there any difference?
I'll add more details ASAP, but we're confident the slow-down is not in the network (i.e. the many GET requests).

Initial profiling indicates it's something to do with writing what is fetched to the discovery cache on disk.
Watch out for anything where writes trigger : there was an issue on macOS where the only way for an app to be confident that a write had completed was to trigger a whole-filesystem sync. Libraries that like data integrity opted for the slower-but-effective approach. I've stopped using macOS so this hint is from a while back, but it might relevant.
Thanks for the tip The issue definitely seems to be in the . When I apply the diff below to switch over to a the slowness disappears and MacOS performance matches that of Linux - about a second for to perform discovery:
It seems to be a problem that would occur only once when establishing the . Should we add it to known issues? Some explain about fsync in MacOs in
Hi, I have no strong opinion here.
You're right, and this happens every 6 hours in the latest versions of kubectl when the cache is invalidated. It's still not an ideal user experience though - you can imagine kubectl 'hanging' for 10-20 seconds the first time you used it each day.
Also note that this should fix a lot of that problem:
has anyone been able to profile kubectl to see what it's doing?
Also note that your CPU is mostly doing nothing, so it's most likely stuck on IO
I think is correct - my bet is we're blocking on , which per uses on MacOS. Per : A mutex profile shows a lot of time waiting for inside , which you can see sets the argument to . I believe this means we're doing an for every HTTP response body we encounter while doing discovery per .
I've confirmed the slowdown is coming from the . Using the following diff: I see MacOS is down to around one second, similar to Linux:
How do folks feel about not explicitly calling for each file in the HTTP cache? Given that it is indeed only a cache I lean toward optimizing for write performance rather than data safety. If we did want to stop calling it seems like the best option would be to update the package of that we use today to make whether it calls with configurable. I'm skeptical we could get this change merged upstream though - the latest change to was three years ago. Another option might be to switch the the package.

I just took a brief shot at this, but it seems like it would require some refactoring to the to handle calling on the underlying database.
I don't care a ton if we drop a cache file coherently. I care a lot if a write fails in a way that leaves a partially written file in a way that pollutes the cache and makes the user have to delete the file from the cache manually. I don't know which of those we're opening ourselves up to by writing without sync. Can we find out?
Presuming we do find that not fsyncing can result in leaving the cache in a broken state (as opposed to a partial state) would falling back to invaliding the cache and recreating it be acceptable?
Little more insight at and Notably the second link mentions:
I dug into this a bit - dumping state here before I sign off for the day. First, note that the discovery client uses two layers of cache - a "discovery cache" and an "HTTP cache". The former appears to be backed by the latter. My -fu is not strong enough to figure out why we do this. The best I could find is , which refers to "restoring the old cached client behavior" and appears to (re?)introduce the caching HTTP transport. Perhaps or know why this is? Only the HTTP cache is explicitly calling . The discovery cache just writes the files and calls , so it's already subject to the risks of not using . Per that risk is that despite the write having been flushed from the OS to the disk controller (e.g. by successfully calling ): My understanding of the and code that powers the HTTP cache is: If a cache file read returns an error we just act as if the value isn't in cache. If a cache file read returns garbled data that can't parse as an http response would fall back to its underlying (i.e. make a call to the API server) and will happily cache over the top of an existing (potentially garbled) file. The caches will be invalidated after six hours. So the failure mode that wouldn't be covered if we stopped calling is that if power were cut to the drive before its controller could sync down to persistent storage and_ this resulted in a file that was garbled just right such that it appeared to be a valid HTTP response but contained an invalid body (i.e.

not real discovery data) someone would have to either manually remove their cache, or wait six hours for the cache to be automatically invalidated. It seems like we could probably make the discovery client force an uncached request through the caching HTTP transport if it encountered what appeared to be cached, invalid discovery data, but I do wonder how far we really need to go to ensure that this cache is never corrupted.
For either of these caches, there are techniques (checksums and the like) where we can allow the app to see whether or not it is looking at a coherent cache document or something that might not be correct. Whenever a cache entry doesn't look valid, the code can asset that the date is current and the entry checksum is OK. If the entry turns out to be invalid or stale, remove it and then act as if we got a read error. That should cover any kind of unsynchronised write failure, I think. For locking the cache we also don't need to ensure sync-to-disk, so long as all running processes see a consistent view of locking.
Agreed. My question is whether we need to add such techniques to this particular cache (or caches) before we'll feel comfortable removing the fsync. Removing the fsync from the HTTP cache would presumably bring it back down to the same level of guarantee as the discovery cache. My guess would be that the fact we were ever explicitly fsyncing each cache value in the HTTP cache was a happy accident rather than something that was explicitly a desirable quality for the HTTP cache.
It seemed like writing our own httpcache cache backend was the best way to go here, so I went ahead and one that uses checksums rather than syncs per the above PR.


Question: } return &objectCountEvaluator{ <del> allowCreateOnUpdate: allowCreateOnUpdate, </del> groupResource:       groupResource, listFuncByNamespace: listFuncByNamespace, resourceNames:       resourceNames,

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug /sig api-machinery What happened: The that we use to make a create validation function is given admission.Update as its operation field. In the create validation we then use the provided operation (admission.Update) , when we call the admission controllers, even though a new object is being created. A similar thing is happening with the mutating admission. This means that when an objects is created on update, the admission controllers will see this as an update request. Meaning there could be an objects in storage which never passed create admission validation and were never modified by create admission mutation. I think this is a bug. What you expected to happen: Admission controllers would handle a create-on-update exactly the same as a normal create
I think the solution to this would be to separate the static admission attributes into one for the create validation and one for the update validation. Also, the function which performs the mutating admission needs to determine whether or not it is an actual update or if the old object doesn't exist (probably by checking uid) and then call the mutating admission correctly based on which operation is being performed.
/assign I might be interested to give it a try.
How to reproduce this bug?
Instead of a reproduceable BUG, I think it's like a mismatch of conceptual design and actual implementation. If a resource allows create-on-update in its strategy e.g.: Then admission plugins would receive operations (CREATE/UPDATE) on UPDATE requests. Some problems might happen the admission plugin is written without considering the circumstances that admitting a UPDATE operation might result a object creation into the storage. Risky and confusing, WDYT?
With the current build in admission controllers, I'm not sure if there are any bugs. Maybe something with resource quota? The real problem is that there is no way to know which webhook authors will have a dependency on seeing a "create" operation for every object put into storage. If a webhook relies on creates to implement its custom logic (which should be allowed), then objects created with an update would currently be missed by the webhook.