Question: cidrUpdateRetries = 3 // updateRetryTimeout is the time to wait before requeing a failed node for retry <del> updateRetryTimeout = 100 * time.Millisecond </del> <ins> updateRetryTimeout = 250 * time.Millisecond // maxUpdateRetryTimeout is the maximum amount of time between timeouts. maxUpdateRetryTimeout = 5 * time.Second </ins> // updateMaxRetries is the max retries for a failed node updateMaxRetries = 10

/kind bug /sig network What happened: Networking tight loop causing GCP Cloud rate limiting to kick in. Causing tests to fail. Suspected call stack: Looks like it could be the root cause of these test failures here: https://k8s- These storage tests are hitting rate limit on the cloud api when trying to detach/delete disks. However, we have exponential backoff for all these functions. But the rate limiting from an external source is affecting the tests. See logs snippet: Note: These logs span exactly 1 minute. Try grepping "rateLimit" or "CIDRNotAvailable" -also note that the CIDRNotAvailable error messages are probably not verbose enough as they do not pinpoint what the actual error is, you can only associate them by correlation in timing and then looking at the code. Shorter snippet from above log file to demonstrate: since this is running on hundreds of nodes hitting this API call should involve some sort of exponential backoff as well as jitter. /cc /assign
/assign
/assign
/sig cloud-provider /sig gcp /milestone v1.12 (since is marked as v1.12)
may we have a priority on this since Code Freeze is next week? :)
Quick question, are we using the default API rate limits quota for scalability projects?
I will defer to for priority because he probably has the most context on how serious this is and what it could affect
Which rate limit are you referring to? (I don't remember us changing it any time) Btw - this is the project being used for that job -
I put up a patch for adding exponential retry
Gentle reminder that for this to be considered for 1.12 past Code Freeze, which is next week, it must have a status/critical-urgent label. I see this is being actively worked on


Question: // Returns a cache.ListWatch that gets all changes to services. func createServiceLW(kubeClient *kclient.Client) *kcache.ListWatch { <del> return kcache.NewListWatchFromClient(kubeClient, "services", kapi.NamespaceAll, kSelector.Everything()) </del> <ins> return kcache.NewListWatchFromClient(kubeClient, "services", kapi.NamespaceAll, kselector.Everything()) </ins> } // Returns a cache.ListWatch that gets all changes to endpoints. func createEndpointsLW(kubeClient *kclient.Client) *kcache.ListWatch { <del> return kcache.NewListWatchFromClient(kubeClient, "endpoints", kapi.NamespaceAll, kSelector.Everything()) </del> <ins> return kcache.NewListWatchFromClient(kubeClient, "endpoints", kapi.NamespaceAll, kselector.Everything()) </ins> } // Returns a cache.ListWatch that gets all changes to pods. func createEndpointsPodLW(kubeClient *kclient.Client) *kcache.ListWatch { <del> return kcache.NewListWatchFromClient(kubeClient, "pods", kapi.NamespaceAll, kSelector.Everything()) </del> <ins> return kcache.NewListWatchFromClient(kubeClient, "pods", kapi.NamespaceAll, kselector.Everything()) </ins> } func (ks *kube2sky) newService(obj interface{}) {

Forked from pointed out that all package names should be lower case, no under_scores or mixedCaps. The same rule applies to renaming imported packages. () generated a list of mixedCaps package names in our codebase: I will upload a PR later to rename kubeletTypes to kubetypes, and the rest is up for grabs.
sorry i have not saw this issue as i open a similar one(), do you mind i participate in this issue? i would happy to do some work for this. best regards.
of course, fixes are welcome. Thanks for picking up on this! We'll try to expedite the review and merge process of such PRs to avoid a lot of churns, but chances are that some rebases are needed :)
ok, i will send some fix sonner, and tag you for review. thanks again.
Not done:


Question: // any fields that are currently set. // TODO(madhusudancs): Simplify the logic for setting node conditions and // refactor the node status condition code out to a different file. <del> func (kl *Kubelet) setNodeStatus(node *v1.Node) error { </del> <ins> func (kl *Kubelet) setNodeStatus(node *v1.Node) { </ins> for _, f := range kl.setNodeStatusFuncs { if err := f(node); err != nil { <del> return err </del> <ins> glog.Warningf("Failed to set some node status fields: %s", err) </ins> } } <del> return nil </del> } // defaultNodeStatusFuncs is a factory that generates the default set of

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): Is this a BUG REPORT or FEATURE REQUEST? (choose one): <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): 1.3.8 custom build - the code seems to still be in place in master. Environment: Cloud provider or hardware configuration: OpenStack OS (e.g. from /etc/os-release): VERSION="14.04.5 LTS, Trusty Tahr" Kernel (e.g. ): 4.4.0-38-generic ~14.04.1-Ubuntu SMP Tue Sep 6 17:20:43 UTC 2016 x8664 x8664 x86_64 GNU/Linux Install tools: Custom Others: What happened: When the OpenStack Nova API is not accessible Nodes are marked as "NotReady" The OpenStack cloud providers seems to call Nova on a schedule to get information. If the API is not contactable, kubelet fails to call back to the masters, and gets marked as NotReady. Masters also call the Nova API, and when it fails, it can mark all nodes as NotReady. What you expected to happen: Nothing - the cluster should still work, regardless of Nova's API How to reproduce it (as minimally and precisely as possible): Install Kubernetes using the OpenStack provider. On the OpenStack controller stop the nova-api process.


Question: streamingServer    streaming.Server networkPlugin      network.NetworkPlugin containerManager   cm.ContainerManager <ins> // cgroup driver used by Docker runtime. cgroupDriver string </ins> } // Version returns the runtime name, runtime version and runtime API version

Kubernetes version (use ): v1.5.0-alpha.1.2189+-dirty docker: 1.12.3 Environment: fedora24 docker-engine-1.12.3-1.fc24.x86_64 What happened: if docker Cgroup Driver=cgroupfs, and set --cgroup-driver=systemd in kubelet, then enable --experimental-cgroups-per-qos=true; The container cgroup some is not correct. How to reproduce it (as minimally and precisely as possible): Anything else do we need to know: 1) docker Cgroup Driver=cgroupfs and --cgroup-driver=cgroupfs in kubelet works well 2) docker Cgroup Driver=systemd and --cgroup-driver=systemd in kubelet works well
help check, thanks.
In Kubernetes 1.6, we can work to detect this case and return an error. IMHO its a configuration error to have docker running with a cgroupfs driver paired with the kubelet running the systemd cgroup driver. /cc
We can also document the compatibility matrix for now.


Question: const ( aggregatorServicePort = 7443 <ins> apiServiceRetryPeriod  = 1 * time.Second apiServiceRetryTimeout = 1 * time.Minute </ins> ) var _ = SIGDescribe("Aggregator", func() {

[X] Test approval issue: [X] Test PR: 7 July 2022 [x] Two weeks soak end date: 21 Jul 2022 [x] Test promotion According to this APIsnoop query, there are still some remaining APIService endpoints which are untested. replaceApiregistrationV1APIServiceStatuspatchApiregistrationV1APIServiceStatusreplaceApiregistrationV1APIServicedeleteApiregistrationV1CollectionAPIService Using a number of existing e2e test practices a new has been created for 4 APIService endpoints. The e2e logs for this test are listed below. This query shows the following endpoints are hit within a short period of running this e2e test If a test with these calls gets merged, test coverage will go up by 4 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: return false, errors.Errorf("%s is not a known certificate", name) } <del> return handler.readwriter.Exists(), nil </del> <ins> return handler.readwriter.Exists() </ins> } // GetCertificateExpirationInfo returns certificate expiration info.

When the command is run with superuser privileges, it prints the requested information. Print a warning that files could not be read due to insufficient permission, or that information was missing (e.g., when the /etc/kubernetes directory is not initialized). When the directory is readable, it works as expected: where OS and kubeadm version detail was: Issue the command as a normal user when the directory /etc/kubernetes/pki exists but not readable. No response <details</details<detailsBare metal </details<details</details<detailsapt-get install kubeadm (source repo: ) </details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cluster-lifecycle


Question: return 100 * time.Millisecond } <del> idleSuccess := make(chan struct{}) fw.fakeKubelet.portForwardFunc = func(name string, uid types.UID, port uint16, stream io.ReadWriteCloser) error { select { case <-idleSuccess: case <-time.After(150 * time.Millisecond): t.Fatalf("execFunc timed out waiting for idle timeout") } return nil } </del> podNamespace := "other" podName := "foo"

this is you
Hmm, I had upped waiting from 100ms to 500ms and that seemed to fix it. Do you have any better ideas?
Don't wait? :)
If the only reason you would timeout would be the code is broken, that's why the test runs have -timeout set.
Then the test fails immediately. It's waiting for the net.Listen calls to finish so it can connect to the specified local ports.
*finish enough so that socket connections work
So I can remove the timeout gating in my tests?
Yes ----- Original Message -----
Just failed again: https://travis-
Seems to be failing frequently now.
If this is test is failing again, my gut says it's not the same root cause. The original failure was a timeout issue. This new one is connection reset by peer. I've got the test running in a loop waiting for a failure but haven't seen one yet. is this still failing frequently?
No. It had failed twice in ~10 builds when I reopened this issue, but hasn't failed since. Reclosing. Will file a new issue if it recurs.


Question: "net" "os" "path/filepath" <ins> "strconv" </ins> "strings" "k8s.io/api/core/v1"

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: For IPv6, the node CIDR size is hard coded to /64. This causes an error bringing up the kube-controller, as the size must be for a smaller subnet (larger value) that what is used for the pod subnet, which is specified for some CNI plugins (e.g. Calico). In addition, this is a catch-22, as the pod subnet prefix cannot be for a larger network (smaller value) than /66, so we cannot set it to /63, for example. What you expected to happen: The node CIDR size should be computed, based on the pod subnet specified, so that it can meet the requirements. How to reproduce it (as minimally and precisely as possible): Bring up a cluster using KubeAdm, specifying IPv6 and choose any pod subnet prefix desired. Anything else we need to know?: Note, there is a second requirement, that the difference between pod subnet size and node CIDR size be 16 or less (issue ). I'll be submitting a proposed PR to address this. Environment: Kubernetes version (use ): latest on master, using 1.9.2 in my case Cloud provider or hardware configuration: bare metal, single node OS (e.g. from /etc/os-release): Ubuntu 16.04 Kernel (e.g. ): Linux bxb-c2-77 4.4.0-109-generic -Ubuntu SMP Tue Jan 9 19:52:39 UTC 2018 x8664 x8664 x86_64 GNU/Linux Install tools: kubeadm Others: /area ipv6 /sig network
/assign pmichali


Question: inkscape:pageopacity="0.0" inkscape:pageshadow="2" inkscape:zoom="0.75130096" <del> inkscape:cx="671.4547" inkscape:cy="483.30875" </del> <ins> inkscape:cx="423.88414" inkscape:cy="492.66426" </ins> inkscape:document-units="px" inkscape:current-layer="layer1" showgrid="false" inkscape:window-width="1519" inkscape:window-height="848" <del> inkscape:window-x="126" </del> <ins> inkscape:window-x="73" </ins> inkscape:window-y="24" inkscape:window-maximized="0" inkscape:snap-global="false" />

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --AWS EBS CSI driver returns topology segments It should merge the topology segments Enabling , and then describe the I have found in the docs which indicates that the duplication is by intention Also I found this and pull
/assign
/sig storage /tirage accepted /assign
It's caused by the migration from CSI to in-tree PV. AWS EBS CSI driver provides PV with: CSI translation changes -and we end up with two same items in the array. It makes sense to squash these duplicates.
cc what do you think?
Why is aws csi driver provides PV with Seems to me this is not needed.
I believe aws csi driver recently changed to using kubernetes topology labels instead of vendor-specific topology labels. Note that the elements of a matchExpression are together. I'm not sure if that's the correct approach if the intention is to eventually get rid of the vendor-specific topology label. cc
I think the topology label is defined by the csi driver? Is it okay to get rid of the vendor specific label?
The intention was to put both labels on all nodes to maintain backwards compatibility with existing PVs. but provisioned PVs should only have one or the other, putting both labels on the PV is a bug: Now that we've released the bug and some PVs have both labels, it will be hard to remove the vendor specific label.
thanks for your follow up. After the provisioned PV is assigned to a node, removing the duplicate label would still be an issue ?
It depends on which label is going to be removed. I'd suggest to remove the in-tree one (), because we can be sure that nodes have the other label from registering the CSI driver.
+1
/triage accepted /assign
fixed by Closing


Question: } } } <ins> func TestSafeSort(t *testing.T) { tests := []struct { name   string in     []string inCopy []string want   []string }{ { name:   "nil strings", in:     nil, inCopy: nil, want:   nil, }, { name:   "ordered strings", in:     []string{"bar", "foo"}, inCopy: []string{"bar", "foo"}, want:   []string{"bar", "foo"}, }, { name:   "unordered strings", in:     []string{"foo", "bar"}, inCopy: []string{"foo", "bar"}, want:   []string{"bar", "foo"}, }, { name:   "duplicated strings", in:     []string{"foo", "bar", "foo", "bar"}, inCopy: []string{"foo", "bar", "foo", "bar"}, want:   []string{"bar", "bar", "foo", "foo"}, }, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { if got := safeSort(tt.in); !reflect.DeepEqual(got, tt.want) { t.Errorf("safeSort() = %v, want %v", got, tt.want) } if !reflect.DeepEqual(tt.in, tt.inCopy) { t.Errorf("after safeSort(), input = %v, want %v", tt.in, tt.inCopy) } }) } } </ins>

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: After creating a DaemonSet that uses node affinity to request only creating pods on nodes with a label matching a set of candidate values, pods get created on the appropriate nodes—and, fortunately, only on the appropriate nodes—but the pods get terminated and deleted quickly before they can start running. The cycle takes about ten seconds from pod creation through completion of deletion; the pod appears to be deleted within one second after its creation, but it takes about ten seconds for the deletion to complete. If I vary the DaemonSet's scheduling predicates, I find the following work to get pods running successfully: Use to choose a specific node. Use with an operator of "Exists" or "NotIn" to choose some nodes. Use with an operator of "In," but with only one value to choose some nodes. Use with a label that is not applied by the cloud provider. If I come up with my own label such as "special-hardware" and use kubectl label node to apply it, and use that label as a match expression key, it seems to work fine. It's the dynamically applied labels like "" and "" that trigger this problem. Without the above concessions, if I use with an operator of "In," but with more than one value, the pods don't start correctly. Note that this behavior is a regression from Kubernetes version 1.10.4. The same configuration works as intended in a cluster running that earlier version. What you expected to happen: After creating the DaemonSet, pods would start successfully on the nodes with label values matching one of the DaemonSet's candidates values. How to reproduce it (as minimally and precisely as possible): a DaemonSet that uses node affinity to request only creating pods on nodes with a label matching a set of candidate values. that at least one node has a label with a value that matches a member of that candidate set. kubectl get pods -o wide to observe that there are pods created on behalf of the DaemonSet on nodes that match the predicate, but they have status "Terminating." By watching the pods, you can see new pods arrive with status "Pending," then "ContainerCreating," then "Terminating," which they'll retain until deletion completes and a replacement arrives. a similar pod directly, without a supervising DaemonSet.

If the "" label is inconvenient within your cluster, instead consider using "" and a few hostnames as the candidate values. that one pod starts running successfully on one of the nodes that match that the predicate. that lone pod (for clarity), and vary the predicate on the DaemonSet to see which changes allow pods to start running successfully. Try each of these individually: Remove one of the values in the sequence. That is, leave only one candidate value. Change the operator to "NotIn," and adjust the values to select some subset of the nodes. Alternately, try "Exists" with no values. Remove the node affinity stanza's and add a node selector in its place: Anything else we need to know?: Here is the DaemonSet object as captured via kubectl get daemonset -o yaml: Here is one of the pods created on behalf of the DaemonSet: Here is one of the nodes on which pods like these should run: There some preceding discussion in the "sig-node" channel of the "Kubernetes" Slack team . Possibly related issues: , Possibly related PRs: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: AWS EC2 (g3.8xlarge instance, but other instance types exhibit the same behavior) OS (e.g. from /etc/os-release):Kernel (e.g. ):Install tools: kubeadm init, kubeadm join
/sig node /sig apps
To work around this problem, I found that I can use multiple match expressions, with one candidate value in each. This works because sibling match expressions are a disjunction.
if the kubelet is rejecting the pod, then this belongs to sig-node it might be reusing logic from the scheduler library, so looping in sig-scheduling for reference: /sig scheduling
I can reproduce it on kubeadm v1.11.0. Let me try to give a fix. /assign
GitHub didn't allow me to assign the following users: Huang-Wei. Note that only and repo collaborators can be assigned. <detailsIn response to : I can reproduce it on kubeadm v1.11.0. Let me try to give a fix. </details
I've found the root cause, and should be able to send out a PR next week.


Question: // Right now we make about ten retry attempts if we get a Retry-After response. // TODO: Change to a timeout based approach. <ins> maxRetries := 10 </ins> retries := 0 for { <del> if r.err != nil { return nil, r.err } // TODO: added to catch programmer errors (invoking operations with an object with an empty namespace) if (r.verb == "GET" || r.verb == "PUT" || r.verb == "DELETE") && r.namespaceSet && len(r.resourceName) > 0 && len(r.namespace) == 0 { return nil, fmt.Errorf("an empty namespace may not be set when a resource name is provided") } if (r.verb == "POST") && r.namespaceSet && len(r.namespace) == 0 { return nil, fmt.Errorf("an empty namespace may not be set during creation") } var err error r.req, err = http.NewRequest(r.verb, r.finalURL(), r.body) </del> <ins> url := r.finalURL() req, err := http.NewRequest(r.verb, r.finalURL(), r.body) </ins> if err != nil { <del> return nil, err </del> <ins> return err </ins> } <del> r.req.Header = r.headers r.resp, err = client.Do(r.req) </del> <ins> req.Header = r.headers resp, err := client.Do(req) </ins> if err != nil { <del> return nil, err </del> <ins> return err </ins> } <del> defer r.resp.Body.Close() // Check to see if we got a 429 Too Many Requests response code. if r.resp.StatusCode == errors.StatusTooManyRequests { if retries < 10 { retries++ if waitFor := r.resp.Header.Get("Retry-After"); waitFor != "" { delay, err := strconv.Atoi(waitFor) if err == nil { glog.V(4).Infof("Got a Retry-After %s response for attempt %d to %v", waitFor, retries, r.finalURL()) time.Sleep(time.Duration(delay) * time.Second) continue } } </del> <ins> done := func() bool { // ensure the response body is closed before we reconnect, so that we reuse the same // TCP connection defer resp.Body.Close() retries++ if seconds, wait := checkWait(resp); wait && retries < maxRetries { glog.V(4).Infof("Got a Retry-After %s response for attempt %d to %v", seconds, retries, url) time.Sleep(time.Duration(seconds) * time.Second) return false </ins> } <ins> fn(req, resp) return true }() if done { return nil </ins> } <del> body, err := ioutil.ReadAll(r.resp.Body) if err != nil { return nil, err } return body, err </del> } }

Storing r.req / r.resp internally (for use from DoRaw and Do) is probably not the right factoring. There should be a common method doRaw that returns request and body.


Question: "//pkg/security/podsecuritypolicy/util:go_default_library", "//plugin/pkg/admission/serviceaccount:go_default_library", "//staging/src/k8s.io/api/apps/v1:go_default_library", <ins> "//staging/src/k8s.io/api/authentication/v1:go_default_library", </ins> "//staging/src/k8s.io/api/batch/v1:go_default_library", "//staging/src/k8s.io/api/certificates/v1beta1:go_default_library", "//staging/src/k8s.io/api/core/v1:go_default_library",

https://k8s- /sig auth /kind failing-test
Adding a priority to this: /priority important-soon Please raise the priority if you consider this a release blocker. Given the test job, it normally wouldn't be.
It's a test issue. I'm hoping to fix this before 1.13 but I don't consider it a release blocker.


Question: OUT_DIR?=_output HYPERKUBE_BIN?=$(OUT_DIR)/dockerized/bin/linux/$(ARCH)/hyperkube <del> BASEIMAGE=k8s.gcr.io/debian-hyperkube-base-$(ARCH):0.8 </del> <ins> BASEIMAGE=k8s.gcr.io/debian-hyperkube-base-$(ARCH):0.10 </ins> TEMP_DIR:=$(shell mktemp -d -t hyperkubeXXXXXX) all: build

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Cloud providers are using the command-line tool to retrieve status and format volume mounts. is missing from the hyperkube image resulting in missing volumes. Issues: What you expected to happen: Volume mounts on cloud providers are expected to be formatted and mounted. How to reproduce it (as minimally and precisely as possible): GCE and Openstack volume mounts are effected. Anything else we need to know?: Environment: Kubernetes version (use ): 1.9, 1.10+ Cloud provider or hardware configuration: GCE and Openstack OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/cc
/sig storage

Kubernetes version (use ): Environment: Cloud provider or hardware configuration: GKE OS (e.g. from /etc/os-release): container-os Kernel (e.g. ): Linux gke-wordpress-cluster-default-pool--m764 4.4.21+ SMP Fri Feb 17 15:34:45 PST 2017 x86_64 Intel(R) Xeon(R) CPU @ 2.60GHz GenuineIntel GNU/Linux Install tools: Others: What happened: ` I was able to bring up wordpress okay the first time, except GKE wasn't creating loadbalancer ips due to a quota issue, which I resolved, note at this point the mysql pod was up and had attached to it's volume. Upon deleting the wordpress deployment and creating it again I started getting the above errors. I deleted the mysql pod as well and brought it up again to see that it had the same issue. The volumes are backed by a gluster cluster on GCE. Looking at the brick logs on one of the gluster nodes I see I've tried restarting kubelet on the node. I can't find the kubelet log file on any of gke nodes. And I don't know how to get the kube-controller and apiserver logs from the master (GKE). I suspect it's a failing of the glusterfs client on the gke nodes? What you expected to happen: Deployment to mount the volumes successfully. How to reproduce it (as minimally and precisely as possible): Not sure how, but I've run into this intermittently Anything else we need to know: The volumes are backed by gluster
/cc
If you delete pod and create pods soon, it is possible that you could see some errors like "timeout expired waiting for volumes to attach/mount" temporally. Could you wait for a few more minutes to see whether the pods could be started correctly?
I actually tried that, deleted the pods, then waited 5 minutes and recreated them with the same error. Maybe it wasn't long enough
Can you post output of ? It may have some other events for failure..
I deleted the cluster this afternoon, but I checked constantly and those were the only errors, Timeout on mount and a failed to sync error
If you use level 4 log, it should also give the detailed messages of failed mount so that we can figure out why it failed.

Can you email me your project name, cluster name, and zone information? I can take a look at the master logs to see if there's anything useful there.
This is starting to happen more frequently..
could you provide more details about in what situation it happened frequently? Kubelet log could give us more information. If you use GKE cluster, run "journctlctl -u kubelet /tmp/log" to get the log. You could also share with us the log if it is ok.
This seems to be the relevant part of the log I re-created that and my pods sprang back up. Weird thing is I had created that, along with the pvcs and pvs but seems it's somehow getting deleted. There's almost certainly certain I'm doing wrong but I'm not sure what. ]
HI Jonathan, can you clarify what you had to recreate?
The glusterfs endpoints , Michelle Au <:
I have a very similar problem running OpenShift 3.4 in AWS EC2 doing the same exact thing. Brand new OSE install [root master]# oc version oc v3.4.1.12 kubernetes v1.4.0+ features: Basic-Auth GSSAPI Kerberos SPNEGO Server https://172.31.40.126:8443 openshift v3.4.1.12 kubernetes v1.4.0+ POD in question is the docker-registry (kind of important for OSE to work) I've followed this to see if it would fix the issue: But no luck or change. Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod "docker-registry-3-afbbt"/"default". list of unattached/unmounted volumes=[registry-storage] then: Unable to mount volumes for pod "docker-registry-3-afbbt_default(-1a45-11e7-abf8-)": timeout expired waiting for volumes to attach/mount for pod "docker-registry-3-afbbt"/"default". list of unattached/unmounted volumes=[registry-storage] I have no idea what log(s) to look at, this was of no help when I deployed a new docker-registry: oc logs -f dc/docker-registry Any help or guidance would be greatly appreciated as this is a blocker issue for me.

thanks and best wishes,
are you using glusterfs for registry-storage volume or other type of volume? Could you share us your kubelet log if possible over slack Thanks!
Hi Jing Xu, Would you know where this log would be? Or is the name kublet<somethingso I may find it for you? I just requested to join the slack channel, as soon as I'm signed in I will share the log. Probably tomorrow as I'm on the east coast USA Thank you! Joel
For you are using GKE with GCI image, you can use "journalctl -u kubelet " to get the log. Otherwise, it should be in " , shepp67 <:
I also experienced this. Kubernetes version:Cloud provider or hardware configuration: OpenStack OS (e.g. from /etc/os-release): Ubuntu 16.04.1 LTS Kernel (e.g. ): Install tools: custom That kubernetes worker vm actually had volumes attached. I saw it on OpenStack dashboard. But: and actually: Besides, this is a new worker vm. I have other worker vms already running, using cinder volumes and there were no such errors. I use Helm to deploy this Pod. I have deployed it earlier and removed it. But it was 2 days ago. I found the solution here: . On that worker vm I run: Edit: or upgrade Kubernetes to = 1.6.0-beta.3 I don't know if this is the solution to issue.
I'm seeing this same error on all my manually launched private Ubuntu based clusters (6), running 1.5.3 and 1.6.0. In my case the messages reference only stopped pods launched by CronJob Jobs. The messages indicate issues mounting every volume in the pod including PVs and node-local volumes. These pods, and future ones launched by these CronJobs, are fully successful when they run. It is only after they run to successful completion and exit that these messages show up. Glancing at the kubelet code, this appears to be coming out of some housekeeping/synch loop.
I've identified a bug and am working on a fix for when "Unable to mount volumes for pod..." + "Error syncing pod" happen indefinitely BUT the pod is actually running just fine, as observed by .

For others, if you see this message but your pod is Running, you can safely ignore it. It's caused by a small limitation of the reconstruction of state (what volumes are mounted) so there is a chance you will see this only if you restart kubelet while your pods had volumes mounted. correction: you say they even Completed, they weren't just Running fine. So yeah, it's just misleading logging and not really an indication of anything breaking
Because the error message is very generic, I think there might be different problems mentioned in this thread. 's issue could be fixed by 's PR are you still have any issues with mount? could you also let us know whether you still have the issue? Thanks! Jing
Hi my issue was related to which helped me sort out. I was creating the gluster endpoints without a service and turns out the kube-controller deletes all such endpoints that are not attached to a service at regular intervals. So after every so often when a pod went down and tried to come up again and mount its respective pv it failed because it couldn't connect to gluster. The fix is to create a service for the gluster endpoints.
I also just ran into such timeout issue: After running a load test against my postgres backed web application, the database didn't respond anymore for still unknown reasons. According to kubernetes it was up and running The log showed a bunch of these messages. Not sure yet why but possibly it couldn't access the volume anymore: Short after that, the pod entered "error" state: The event log showed: This error disappeared after a few minutes (not sure how long exactly, but it took a while), but then I got the same error as described here: I couldn't figure out anything more on my own. Nothing in the syslog, nothing more than this in the kubelet log. Finally I rebooted the machine which fixed the issue.
cc do you know what error of the network not ready? It might be helpful if you could share the kubelet log with us to debug. Thanks!
AFAIK, the "network not ready error" is benign, kubelet will retry after this error () cc
I hope I didn't miss some secrets in there, quite a talky log. But here it is: The database didn't respond anymore because it ran OOM and was killed, as also shown in the log.

Although it appears like it was OOM killed after it became unresponsive in the log.. Either way, a bunch of other things got killed too, so granted - "not ideal" situation :). Yet it shouldn't drop the volume and should re-attach it.
I think you are hitting the issue mentioned above. He has a PR to fix it. I will make sure it can merge soon. Thanks!
Great, thanks! Do you know roughly when this will be available in GCE?
you mean GKE or GCE? For GKE it will take a little bit longer. On GCE, when the PR gets merged, the fix should be available when the next release is out (I think should be soon)
Running on GKE. I'd rather not launch my product with this bug around. Is there a workaround for it or something I can monitor for? Or maybe it could get hot fixed on GKE? If there is no workaround, this effectively renders GKE useless for stateful stuff.
It's hard to say if volumes are actually being dropped/unmounted or if the FailedMount errors are wrong & misleading, so I wouldn't go so far as to say it's broken because of some volume issues. It could be something else entirely. If nothing else, the PR jing's referring to will clean up the event noise and help us debug the root cause. In the meantime, I wouldn't trust the FailedMount events to be accurate, so if your applications are unresponsive, there may be other causes worth considering
Well, I only can say that my volumes were dropped and it didn't recover on it's own (only after reboot). Dropped a seen by kubernetes, dunno if the volumes were still attached but they weren't usable by my scheduled database. To be clear: I'm not "complaining" and this was just test/R&D, but it would this is the scariest kind of issue for me to happen in prod, given that it's a managed service where I can't find a workaround on my own. Rebooting fixed it in my case, but that's a big hammer..
You don't need to reboot the node. Delete the pod and restart it should work for this issue. Is this workaround ok for your case? , discordianfish <:
Good to know, thanks!

Will try this next time.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Hi, I'm facing the same issue with k8s 1.9, installed by kops on AWS. volume created successfully on AWS and attached to the right node, but the pod is failed to start.
/remove-lifecycle stale
could you please provide more details about your issue? What are the steps to reproduce it? Thanks!
yeah can you open a new bug for EBS and cc me with details? I was hoping we fixed most of EBS problems in 1.9 - I will be more than happy to walk through and find what is causing volume to not mount. As I understand this is different from typical EBS problems where volume fails to attach. In your case, volume is being attached to right node but it is not being seen by kubelet.
I found the issue, and it was the fact that i used c5 instances with nvme, this is why the volume never been attached as k8s looking for /dev/xvdXX and with nvme that path is different. I did notice that in 1.9 k8s should support nvme natively but it seems that it's still not stable. thanks
oh weird. You should open a bug about NVMe support then. I was thinking it would just work.
cc Is there an issue related to nvme support? please let us know any issues you still have with 1.9. Thanks!
This is nvme on AWS, not related to the work is doing on gce.
Thanks Definitely related to NVMe support. Ran into this trying to mount a gp2 EBS volume on a node which was an m5 EC2 instance.
I think I am seeing like this with , but unsure of "why" at the moment. It would seem works perfectly fine.
did you guys had a chance to open a BUG for NVMe support on EBS? If you do - please paste output of and PersistentVolume spec.

We can't fix it unless we know bit more. :(
sorry, we worked around this by reverting back to m4 class instances. It should be simple enough to reproduce however... create a k8s cluster w/ m5 EBS backed nodes and create a simple pod w/ a persistent volume. You'll notice that the pod will never launch and you should be able to debug further. If I do run into this again I'll be sure to get some more data for you.
I too have the error, error syncing pod. kubectl describe pod kafka1-kafka-0 .... .... .... Warning FailedSync 5m (x141 over 35m) kubelet, 10.186.. Error syncing pod Warning BackOff 57s (x164 over 35m) kubelet, 10.186.. Back-off restarting failed container kubectl logs kafka1-kafka-0 -p export KAFKABROKERID=kafka1-kafka-0 export KAFKAADVERTISEDLISTENERS=PLAINTEXT://172.30.*.:9092 /etc/confluent/docker/run sh: 3: /etc/confluent/docker/run: not found


Question: // to be something else, see if the interval will be reconciled. It("should reconcile LB health check interval [Slow][Serial]", func() { const gceHcCheckIntervalSeconds = int64(8) <del> // This test is for clusters on GCE/GKE. framework.SkipUnlessProviderIs("gce", "gke") </del> <ins> // This test is for clusters on GCE. // (It restarts kube-controller-manager, which we don't support on GKE) framework.SkipUnlessProviderIs("gce") </ins> clusterID, err := gce.GetClusterID(cs) if err != nil { framework.Failf("framework.GetClusterID(cs) = _, %v; want nil", err)

is failing with "framework.RestartControllerManager() = unsupported provider: gke; want nil" /kind failing-test
/sig network
Introduced in , should be fixed by


Question: roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole <ins> name: system:node subjects: [] --- # This is required so that new clusters still have bootstrap permissions apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: kubelet-bootstrap labels: kubernetes.io/cluster-service: "true" addonmanager.kubernetes.io/mode: Reconcile roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole </ins> name: system:node-bootstrapper subjects: - apiGroup: rbac.authorization.k8s.io

/priority critical-urgent /priority failing-test /kind bug /status approved-for-milestone /area platform/gke owns the test This test has been failing since at least 2017-11-30 for the following job: https://k8s- This job is on the , and prevents us from cutting v1.9.0 (kubernetes/sig-release). Is there work ongoing to bring this test back to green? /assign Pulling this out of into its own issue. It might be GKE specific? But the GCE downgrade jobs are failing right now, so I don't have enough data to say for sure
Copying latest findings here
Agree, seems like a problem with addon manager /cc
At this point it seems somehow vague to tell which exact component is causing the issue. Probably worth to solve the fundamental downgrade issue () first.
Digging in a bit ... Looks like the test installs a Log Provider using this: and fails here as it finds 2 agents instead of just the one that it installs: Probably because i am guessing fluentd was switched on in this environment? This seems to have happened around 11/30, ring any bells to anyone?
Test doesn't install anything, there's a valid error in addon manager, which results in two addon DaemonSets running after the downgrade Flunetd is there since forever, as DaemonSet for 3-4 releases already, nothing has changed in it, except for the version, which should be handled adequately by addon manager, but it's not
Ack thanks!
Another clue a better one i hope, grepping through the UUID(s) for the fluentd containers - spotted the following and trying to look for why it failed, saw the following These logs are from:
/sig network
This log line usually shows up when kubelet restarts. It should recover after a while.
Can someone from sig-node investigate why kubelet was restarted and if it's normal?
[MILESTONENOTIFIER] Milestone Issue Needs Attention Action required: During code freeze, issues in the milestone should be in progress. If this issue is not being actively worked on, please remove it from the milestone. If it is being worked on, please add the label so it can be tracked with other in-flight issues. Note: This issue is marked as , and must be updated every 1 day during code freeze.

Example update: <details<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
will take a look at kubelet restarts to see if that's normal
I think I got the root cause. Starting from addon-manager's log of the current run (https://k8s-): So clearly something changed for ClusterRoleBinding "kubelet-cluster-admin" between 1.8 and 1.9, upon the downgrade, addonmanager tried to apply the change to an immutable field and failed. Due to some implementation details in , when any error occurred during create/update, the prune operation will not be performed. Hence fluentd-gcp-v2.0.9 daemonset was created but fluentd-gcp-v2.0.10 daemonset was not pruned. Also due to some implementation details in addonmanager, fluentd daemonset and fluentd confimap are managed in seperate label groups, which explains why fluentd-gcp-config-v1.2.3 was pruned successfully. might know about the ClusterRoleBinding "kubelet-cluster-admin" change?
cc who last touched these files:
Seems like ClusterRoleBinding kubelet-cluster-admin does change a bit in the legacy-kubelet-user-disable case:
It seems related to In this case, we create a new cluster, which gets the new binding. Then we try to downgrade, which ends up attempting to switch back to the old binding. I wonder if we can cherry-pick something to 1.8 to make it keep the new binding upon downgrade?
cc and who reviewed the above PR.
From addonmanager perspective, one way to achieve this is by changing the Reconcile mode to EnsureExist mode.

8 created kubelet-cluster-admin binding to system:node role in ReconcileMode If we want 1.9 to tolerate that and not remove it, we should leave the kubelet-cluster-admin binding to system:node role with no subjects and EnsureExists mode Separately in 1.9 we can create a new kubelet-bootstrapper binding to system:node-bootstrapper for the kubelet subject with reconcile mode
(Which is what it looks like did… the later change of what role it bound to was the breaking change)
should have left the existing binding alone and created a new binding for the new role
Just to make sure we're on the same page: We didn't see any problem in 1.8-1.9 upgrade tests. This only shows up in the downgrade test where we start with a fresh 1.9 cluster and downgrade to 1.8. It sounds like your plan will help with both directions, but just want to be clear. Can you work on a PR or do I need to find someone?
Nice work So we don't really capture the addon manager logs do we? Looks like you had to pick one up from the running system, is that right?
Yeah, unfortunately addon manager logs on GKE CIs is not publicly visible. There may be ways to retrieve the same logs for history runs, just that getting it from a running cluster seems easiest to me.
Quickly looked at Kubelet and netconfig is not ready error message should be red herring. Right after that, all nodes are ready with the following logging: Will look more.
Ahh, after refreshing the issue, and found that already identified the root cause. Nice work and thanks!
The add-on manager would still have hit issues trying to reapply the changed rolebinding, but the 1.8 binding granted a superset of permissions, so we likely didn't notice. Correct opened
Thanks at least my investigation triggered a deeper look which seems to have helped the cause. yay!
Reopening until we confirm the fix worked: https://k8s-
Looks like we are green - https://k8s-


Question: // Check whether "STAGE_UNSTAGE_VOLUME" is set stageUnstageSet, err := hasStageUnstageCapability(ctx, csi) if err != nil { <del> glog.Error(log("attacher.MountDevice failed to check STAGE_UNSTAGE_VOLUME: %v", err)) </del> return err } if !stageUnstageSet { glog.Infof(log("attacher.MountDevice STAGE_UNSTAGE_VOLUME capability not set. Skipping MountDevice...")) <ins> // defer does *not* remove the metadata file and it's correct - UnmountDevice needs it there. </ins> return nil } // Start MountDevice if deviceMountPath == "" { <del> return fmt.Errorf("attacher.MountDevice failed, deviceMountPath is empty") </del> <ins> err = fmt.Errorf("attacher.MountDevice failed, deviceMountPath is empty") return err </ins> } nodeName := string(c.plugin.host.GetNodeName())

/kind bug /sig storage What happened: When using a CSI driver that does not support NODESTAGE, I can see orphan json files in that are never cleaned up. What you expected to happen: No orphan file /assign


Question: } genericPluginInitializer := initializer.New(nil, nil, fakeAuthorizer{}, nil) <del> pluginInitializer := kubeadmission.NewPluginInitializer(nil, testrestmapper.TestOnlyStaticRESTMapper(legacyscheme.Scheme), nil) </del> <ins> fakeDiscoveryClient := &fakediscovery.FakeDiscovery{Fake: &coretesting.Fake{}} fakeDiscoveryClient.Resources = []*metav1.APIResourceList{ { GroupVersion: corev1.SchemeGroupVersion.String(), APIResources: []metav1.APIResource{ {Name: "nodes", Namespaced: false, Kind: "Node"}, {Name: "pods", Namespaced: true, Kind: "Pod"}, {Name: "replicationcontrollers", Namespaced: true, Kind: "ReplicationController"}, }, }, { GroupVersion: extensionv1beta1.SchemeGroupVersion.String(), APIResources: []metav1.APIResource{ {Name: "daemonsets", Namespaced: true, Kind: "DaemonSet"}, }, }, } restMapperRes, err := restmapper.GetAPIGroupResources(fakeDiscoveryClient) if err != nil { return nil, fmt.Errorf("unexpected error while constructing resource list from fake discovery client: %v", err) } restMapper := restmapper.NewDiscoveryRESTMapper(restMapperRes) pluginInitializer := kubeadmission.NewPluginInitializer(nil, restMapper, nil) </ins> initializersChain := admission.PluginInitializers{} initializersChain = append(initializersChain, genericPluginInitializer) initializersChain = append(initializersChain, pluginInitializer)

/kind flake /sig testing /sig auth we are injecting a restMapper into this admisson controller while in its test, the restMapper somehow become empty when we remove internal clientset/informer dependency from it (). now we come up with a walkaround in which we manually init the legacy schema via a dry import so that the built-in resources can be installed. this failing test suggests that the test is not quite correct logically.
/assign Are you up for trying to figure out how to avoid requiring the internal types?
What test does this break? Links?
link here: w/o the dry import, then will fail with output:
sure. will be looking into it
OK, setting priority so that I know this isn't a 1.13 blocker. /priority important-longterm


Question: kubecontainer "k8s.io/kubernetes/pkg/kubelet/container" kubelettypes "k8s.io/kubernetes/pkg/kubelet/types" utilnode "k8s.io/kubernetes/pkg/util/node" <ins> "k8s.io/kubernetes/pkg/volume/csi" </ins> ) // getRootDir returns the full path to the directory under which kubelet can

In , we check mount on first level directory in pod volume directories. It is true for many volume plugins, but not for CSI plugin which . cc
/assign
Pardon my ignorance of kubelet source code, but this looks very much similar to an issue we have been running to many times recently, when developing a CSI plugin. The problem is that when CSI volume publish is called and the mount point is not propagated from the CSI plugin pod (either missing bidirectional flag in yaml or not supported by underlaying k8s cluster implementation), the kubelet nevertheless starts a pod using the volume. The volume is not there, so the app writes to root filesystem in fact, leading to all other kinds of problems - i.e. losing data, filling up root fs. IMHO what kubelet should do is to check that the mounted filesystem is indeed visible after publish call and if not it should fail the volume attach request. If this ticket was fixed, would it solve the issue we are seeing or should I open a new ticket? thanks
I think we can potentially add that check but we need to make sure first it can work with ephemeral CSI drivers. Ephemeral drivers would have to "stage" their directories somewhere, like /var/lib/kubelet/plugins/<csi driver/, and then bind mount it to the publish path. cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Hmm... That is a good question. Does csi require a mount to be made or only that the specified directory be created? Will have to go read through the spec...
/remove-lifecycle stale
The only place in the spec I could find was this comment: // For volumes with an access type of mount, the SP SHALL place the // mounted directory at target_path. So I think having Kubelet enforce that check would be ok with the spec, and a good thing to do.
cc CSI spec says that the target path has to be a mount. Shall we go ahead and enforce it then? Are we going to break drivers that may not be doing this?

Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
The issue mentioned here involves two functions . Now with PR , the issue in CheckVolumeExistenceOperation is solved by checking mount path instead of volume path. The only issue here is for getMountedVolumePathListFromDisk which is used during pod deletion. This function was introduced by PR . Now I am questioning whether we need to check the mount point or just need to check pod dirs exist before deleting the pod. Firstly, for some plugins, there might not be mount point. Secondly, without checking mount point, it means that pod cannot be deleted as long as there are pod volume dirs. In normal case, volume dirs should be cleanedup during volume teardown. For cases that kubelet restarts, we have reconciler try to clean up volume mounts if possible. So we can propose a couple of solutions here mount point check. It solves the problem, the downside is that there might be some rare case that reconciler could not clean up dirs after kubelet restarts. check mount point, and also check the next level "/mount" dir. This might still cause problem for some plugins that do not have mounts. cc
Discussed offline with For proposal 1., pod will not be deleted from api server if there are still volume dirs exist. So to make sure it works well, we need to have reliable way to clean up dirs. Normally when kubelet restarts, reconciler will either add it into actual state (if reconstruction works) or try to clean it up directly once unless volume is still in desired state. So if reconstruction does not work, and it fails to clean up during reconciler sync, there is no other chance to clean up the dirs.

Or if volume was in desired state and removed quickly afterwords, the dir might not get the chance to be cleaned up at all. We are thinking to improve this part, we could use mark volume "uncertain" in actual state feature. During reconstruction, we can remove the check for mounted points (since the mount might be gone during node reboots or driver does not have moutns at all). Reconciler could mark those volumes as uncertain (even if it was in desired state) to make sure those volume got cleaned up or remount whenever needed with retries. With this change, I think it is safe to remove the mount point check for pod deletion too.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: // Number must be in the range 1 to 65535. Name must be an IANA_SVC_NAME. // If this is a string, it will be looked up as a named port in the // target Pod's container ports. If this is not specified, the value <del> // of Port is used (an identity map). // Defaults to the service port. </del> <ins> // of the 'port' field is used (an identity map). // This field is ignored for services with clusterIP=None, and should be // omitted or set equal to the 'port' field. </ins> // More info: http://releases.k8s.io/HEAD/docs/user-guide/services.md#defining-a-service TargetPort intstr.IntOrString `json:"targetPort,omitempty"`

In 1.0 and 1.1, headless services that specified a mismatched port/targetPort were tolerated (the targetPort was ignored). tightened validation to reject these services. That causes problems when upgrading from currently-working clusters.
2 candidate to avoid breaking backwards compatibility
oh hell. Yeah. I don't think this is worth a breaking change. We should either roll that back ASAP or change it to nuke any user-provided value and replace it with the value. I think the latter is potentially dangerous and also not worth the risk, so I think we should rollback and just update the docs for targetPort. objections?
Who has time to do this? It doesn't look like an auto-rollback is doable.
Already opened a PR in
Yes, I agree this isn't worth a breaking change, But could we give an warning log that targetPort isn't equal to port and would be ignored?
There's no facility in validation for warning. I don't think spamming the server log in response to API requests is that helpful here
Yes, that sounds reasonable. I just think that targetPort was ignored is a little unfriendly to user.


Question: return strings.HasPrefix(*fip.Name, baseName) } <del> func (az *Cloud) getFrontendIPConfigName(service *v1.Service, subnetName *string) string { </del> <ins> func (az *Cloud) getFrontendIPConfigName(service *v1.Service) string { </ins> baseName := az.GetLoadBalancerName(context.TODO(), "", service) <ins> subnetName := subnet(service) </ins> if subnetName != nil { <del> return fmt.Sprintf("%s-%s", baseName, *subnetName) </del> <ins> ipcName := fmt.Sprintf("%s-%s", baseName, *subnetName) // Azure lb front end configuration name must not exceed 80 characters if len(ipcName) > frontendIPConfigNameMaxLength { ipcName = ipcName[:frontendIPConfigNameMaxLength] } return ipcName </ins> } return baseName }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When the specified internal LB uses a subnet with longer name, the LB configuration will fail with following error: What you expected to happen: When constructing the frontend IP configuration name and the load balancing rules name, the subnet name should be truncated if too long. How to reproduce it (as minimally and precisely as possible): Use a subnet that has long name (70 charactors) and create ILB service. Anything else we need to know?: Environment: Kubernetes version (use ): 1.15 Cloud provider or hardware configuration: azure OS (e.g: ): Ubuntu 16.04 Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig cloud-provider /area provider/azure /assign


Question: } // podPreviouslyProcessed returns true if the volumes for this pod have already <del> // been processed by the populator </del> <ins> // been processed/reprocessed by the populator. Otherwise, the volumes for this pod need to // be reprocessed. </ins> func (dswp *desiredStateOfWorldPopulator) podPreviouslyProcessed( podName volumetypes.UniquePodName) bool { dswp.pods.RLock() defer dswp.pods.RUnlock() <del> _, exists := dswp.pods.processedPods[podName] return exists </del> <ins> return dswp.pods.processedPods[podName] } // markPodProcessingFailed marks the specified pod from processedPods as false to indicate that it failed processing func (dswp *desiredStateOfWorldPopulator) markPodProcessingFailed( podName volumetypes.UniquePodName) { dswp.pods.Lock() dswp.pods.processedPods[podName] = false dswp.pods.Unlock() } // podHasBeenSeenOnce returns true if the pod has been seen by the popoulator // at least once. func (dswp *desiredStateOfWorldPopulator) podHasBeenSeenOnce( podName volumetypes.UniquePodName) bool { dswp.pods.RLock() _, exist := dswp.pods.processedPods[podName] dswp.pods.RUnlock() return exist </ins> } // markPodProcessed records that the volumes for the specified pod have been

It happens during the "delete" phase in the load test. ! In perf-dash we can see that in total there are ~20M requests like that during the load test. ! It started in Sep 2019 (sic!), so probably when we enabled PVs in our load tests. We haven't detected it so far as, strangely, it doesn't seem to be impacting our scalability SLIs. We should investigate this. /sig scalability
I checked and we create 35 PersistentVolumeClaims per 100 nodes. So in case of 5K node test there will be 1750 PVCs in the cluster. This is close to the 2K of GET PVC requests we see.
Let's analyze the last run of 5K node test in more depths - Grafana: ! As I wrote already somewhere here, it happens during the delete phase. We reach the peak of ~2K QPS around 2:30 AM UTC in that run. Let's check the logs (api-server, kubelets) around that time.
Looking at the api-server logs one can easily tell that something is wrong. It's swamped by GET persistentvolumeclaim requests:
Restricting the logs to a single stateful set on a single node (single client ip) - it also doesn't look right. Total such requests from this IP address: So it looks like a single kubelet issued almost 20K GET requests for this PVC within ~75min, which gives us ~4 QPS from a single kubelet...
From the scheduler logs it looks like the statefulset pod for this PVC was bound on the node Let's check the kubelet logs there
Kubelet logs: Bingo, around the time when the issue happens, it's full of the following logs:
Looks like the errors disappeared exactly when the pod was deleted So, what happens here is partially caused by our test as it may happen that we'll first delete PVC and won't delete the corresponding pod for the next O(1h). Nevertheless, the code in the should be fixed so it doesn't cause kubelet to send ~4QPS of GET PVC requests in such case.
/sig node /sig storage
There is a pending discussion about ways to fix it in
Thanks for fixing that! I know how much work was it and I really appreciate!


Question: REGISTRY?=gcr.io/google_containers ARCH?=amd64 TEMP_DIR:=$(shell mktemp -d) <del> CNI_RELEASE=9d5e6e60e79491207834ae8439e80c943db65a69 </del> <ins> CNI_RELEASE=07a8a28637e97b22eb8dfe710eeae1344f69d16e </ins> UNAME_S:=$(shell uname -s) ifeq ($(UNAME_S),Darwin)

I have a container that got the broadcast address of podCIDR subnet. My CNI conf is: And the container in question has which is the broadcast address for the subnet. So it can't figure out the MAC of cbr0 because all frames are getting sent to (or at least, that's what seems to indicate). Eg: tcpdump alonside a wget on vs a similar normal container
Looks like a bug in ipam. Will try to put together a fix.
I stress tested your fix overnight with the networking suite I'm writing (the one that found the bug), didn't hit the failure in 100 runs, we should get it in asap
Great. It would be better to pull from upstream once the fix is merged. Otherwise, we have to fork it like we did in 1.3.
Marking this p0 flake so it shows up in release queries
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.


Question: scope.err(err, w, req) return } <del> requestInfo, _ := request.RequestInfoFrom(ctx) metrics.RecordLongRunning(req, requestInfo, metrics.APIServerComponent, func() { defer watcher.Stop() handler.ServeHTTP(w, req) }) </del> <ins> // Invalidate cancel() to defer until serve() is complete. deferredCancel := cancel cancel = func() {} serve := func() { defer deferredCancel() requestInfo, _ := request.RequestInfoFrom(ctx) metrics.RecordLongRunning(req, requestInfo, metrics.APIServerComponent, func() { defer watcher.Stop() handler.ServeHTTP(w, req) }) } // Run watch serving in a separate goroutine to allow freeing current stack memory t := genericfilters.TaskFrom(req.Context()) if t != nil { t.Func = serve } else { serve() } </ins> return }

Optimize the memory usage of goroutine stack for serving watch requests. According to the of handling a watch request, the stack memory is 0x3768, which is very close to 16 KiB and Golang runtime will double it to 32 KiB once exceeded. In fact, it does exceed if we print the traceback deeper and take the fact that extra stack memory reserved by Golang itself into consideration. A naive optimization is to run as much as preparation handlers in a separate goroutine and then move the actual serving to another goroutine, so the former routine can free the stack memory earlier and make it more memory efficient. Given the extensive use of watch requests, 32 KiB per watch could add up to a very high amount of memory overhead for . A typical watch request commonly lasts for 30s or longer, which means this memory will be active for quite long. I see a great improvement in memory efficiency to do this optimization.
/cc
/sig api-machinery
/cc if you would like to take a look when you have time :) Thank you /triage accepted


Question: "sort" "strings" <del> "github.com/onsi/ginkgo" </del> <ins> e2elog "k8s.io/kubernetes/test/e2e/framework/log" </ins> ) var filesources []FileSource

/help /kind cleanup /area e2e-testing-framework mini guide: make sure you have signed the CNCF clone kubernetes;/kubernetes search for ginkgo.Fail calls under test/e2e/framework e.g. replace them with this that was in this PR: send a PR for kubernetes/kubernetes
/remove-kind feature /area e2e-test-framework
/sig testing
I will take this.
thank you! Please feel free to ping #testing-commons in the Kubernetes slack if you have any questions :smiley: /cc
/remove-help
/assign


Question: -e MIRROR_TAGS_CACHE_TTL=1800  quay.io/devops/docker-registry:latest - name: docker.service <ins> content: | [Unit] Description=Docker Application Container Engine Documentation=http://docs.docker.com After=docker.socket early-docker.target network.target Requires=docker.socket early-docker.target [Service] Environment=TMPDIR=/var/tmp EnvironmentFile=-/run/flannel_docker_opts.env EnvironmentFile=/etc/network-environment MountFlags=slave LimitNOFILE=1048576 LimitNPROC=1048576 ExecStart=/usr/lib/coreos/dockerd --daemon --host=fd:// --registry-mirror=http://${DEFAULT_IPV4}:5000 $DOCKER_OPT_BIP $DOCKER_OPT_MTU $DOCKER_OPT_IPMASQ [Install] WantedBy=multi-user.target </ins> drop-ins: - name: 51-docker-mirror.conf content: |

Minor issues with 1) The examples rely on cloud vendor specific variables ($private_ipv4 ) 2) The wget command for kube-register always re-downloads


Question: return err } <del> _, err = plugin.defaultNetwork.addToNetwork(name, namespace, id, netnsPath) </del> <ins> _, err = plugin.getDefaultNetwork().addToNetwork(name, namespace, id, netnsPath) </ins> if err != nil { glog.Errorf("Error while adding to cni network: %s", err) return err

Host network pods should be able to be created before the container network exists. Pod syncs for pods on the container network should fail until the container network is configured. cc
ref:
I think this is a thing we should make working before v1.4, so adding the milestone. Complain if you disagree. Also, does volunteer to make a PR? :)
Going to make a first pass at this in
Great!
/cc


Question: } # k8s <del> KUBE_VERSION=${KUBE_VERSION:-"1.1.7"} </del> <ins> KUBE_VERSION=${KUBE_VERSION:-"1.1.8"} </ins> echo "Prepare kubernetes ${KUBE_VERSION} release ..." grep -q "^${KUBE_VERSION}$" binaries/.kubernetes 2>/dev/null || { curl -L https://github.com/kubernetes/kubernetes/releases/download/v${KUBE_VERSION}/kubernetes.tar.gz -o kubernetes.tar.gz

The default Kubernetes release number in cluster/ubuntu/download- is 1.1.7 but 1.1.8 is out now.
Resolved by PR


Question: } func (om *fakeObjectManager) UpdatePod(pod *v1.Pod) error { <del> defer om.updatePodTracker.inc() if om.updatePodTracker.errorReady() { defer om.updatePodTracker.reset() return om.updatePodTracker.getErr() </del> <ins> defer om.updatePodTracker.trackParallelRequests() if err := om.updatePodTracker.incWithOptionalError(); err != nil { return err </ins> } return om.podsIndexer.Update(pod) } func (om *fakeObjectManager) DeletePod(pod *v1.Pod) error { <del> defer om.deletePodTracker.inc() if om.deletePodTracker.errorReady() { defer om.deletePodTracker.reset() return om.deletePodTracker.getErr() </del> <ins> defer om.deletePodTracker.trackParallelRequests() if err := om.deletePodTracker.incWithOptionalError(); err != nil { return err </ins> } if key, err := controller.KeyFunc(pod); err != nil { return err

https://prow.ppc64le- TestStatefulSetControl/CreatePodFailure/Burst/ScaleDownOnly/StatefulSetAutoDeletePVCEnabled of Have started observing more flakes recently since last week. No response_ Able to reproduce using stress tool. /sig testing /sig apps
/triage accepted /assign
opened a PR with a fix:


Question: <ins> /* Copyright 2017 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package v1alpha1 import "k8s.io/apimachinery/pkg/conversion" // Convert_Slice_string_To_v1alpha1_IncludeObjectPolicy allows converting a URL query parameter value func Convert_Slice_string_To_v1alpha1_IncludeObjectPolicy(input *[]string, out *IncludeObjectPolicy, s conversion.Scope) error { if len(*input) > 0 { *out = IncludeObjectPolicy((*input)[0]) } return nil } </ins>

Making an API request for table output and specifying the parameter causes a server error in parameter conversion, and an inaccurate error message in response: server log: E0804 09:41:14. ] apiserver received an error that is not an metav1.Status: converting (url.Values).[]string to (v1alpha1.TableOptions).v1alpha1.IncludeObjectPolicy: couldn't copy '[]string' into 'v1alpha1.IncludeObjectPolicy'; didn't understand types
Ugh, typed strings.


Question: exec, syncPeriod, minSyncPeriod, filterCIDRs(true, excludeCIDRs), strictARP, tcpTimeout, tcpFinTimeout, udpTimeout, masqueradeAll, masqueradeBit, localDetectors[1], hostname, nodeIP[1], <del> nil, nil, scheduler, ipFamilyMap[v1.IPv6Protocol], kernelHandler) </del> <ins> recorder, healthzServer, scheduler, ipFamilyMap[v1.IPv6Protocol], kernelHandler) </ins> if err != nil { return nil, fmt.Errorf("unable to create ipv6 proxier: %v", err) }

! it seems all the job is timing out 05-04 ,%20ipvs,%20IPv6,%20master No response No response /sig network
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/area ipvs
possible pre-submit or infra related issue? I saw this event/warning in most of the cases.
where do you see that error? The jobs are timing out and started to fail at one point in time, the ipvs ipv4 jobs are not failing, usually that indicates there is some trigger, that may be a change in kubernetes/test-infra or in kubernetes/kubernetes, as first check we should verify the changes that went in between the last good and the first bad run
In the events tab of Job Pod Info section
They're configured to run in the same host cluster so the same things should be available. ci-kubernetes-kind-network-ipvs-dual and ci-kubernetes-kind-network-ipvs-ipv6
the following test times out for IPVS-IPv6 on local kind cluster after 1 hr. The same test for IPv4 passes in 11secs.
I stand corrected then, why does it times out? are there some very large timeouts?
looking into it
you can run that test alone with verbosity using and see in which part is blocked
will probably fix the job timeouts
Logs for the failing job just before it crashed.
for reference , talking with Daman offline it seems that the tests that curl inside a pod timeout because the request hangs forever. It only happens on IPVS with IPv6
We talked about this on the SIG Network call today, suggested we assign you /assign Let us know if you need help moving it forward! :vulcan_salute:
Is there any reason why we pass nil instead of healthzServer while instantiating the IPVS IPv6 proxier?

<brThis results in a nil pointer exception on health check node port servers and the read lock is never released: <brThe write lock in the SyncServices call will never be acquired since the read lock is not released <brthis will ultimately block the reconciliation loop, and kube-proxy hangs.
it should not, commented in the PR ok, in golang the http server recovers from panics on handlers, that explains why the whole proxy does not goes down. that second unlock is wrong, it has to be done before calling can you. please submit a fix for that too?
yeah, the proxy goes down because the HTTP handler panicked before releasing the read lock. Sure, will submit a fix for that too.
the job is working fine now - ,%20ipvs,%20IPv6,%20master&show-stale-tests= All greens expect [It] [sig-network] Services should fail health check node port if there are only terminating endpoints This test case will work after is merged.
Thanks Daman, you did a great job narrowing this down and fixing it
Yes, thanks for the quick turnaround it's very appreciated!! :vulcan_salute:


Question: if c.ReadWritePort == 0 { c.ReadWritePort = 443 } <del> if c.PublicAddress == "" { </del> <ins> for c.PublicAddress == "" { </ins> // Find and use the first non-loopback address. // TODO: potentially it'd be useful to skip the docker interface if it // somehow is first in the list.

I'm not using publicaddressoverride. At boot I'm trying to launch using: /usr/bin/kube-apiserver --logtostderr=true --v=0 --etcdservers=http://10.13.137.165:4001 --address=0.0.0.0 --port=8080 --kubeletport= --allowprivileged=true --portalnet=10.254.0.0/16 But this gets launched before eth0 has an ip address. This used to work. Now apiserver just refused to start (complaining about 127.0.0.1 being invalid). I think it needs to just keep trying until it gets an address and can get to work....
Adding --publicaddressoverride= as suggested doesn't help.... Nov 05 12:43:55 localhost kube-apiserver[713]: F1105 12:43:55. ] listen tcp 10.13.137.165:7080: bind: cannot assign requested address Which is a fatal error. I can't bind to it. It isn't up yet.
Bisect says either commit or
If publicaddressoverride is specified, I can put the read-only server bind in a goroutine and retry periodically. If it's not, though, since we can't change the public address once started, we'd have to delay construction until a suitable address shows up. That's pretty ugly; is it an important case to support?
I personally think delaying construction is fine. Being able to launch the apiserver on any node from the init system and have it just work 99% of the time seems like a good thing....
OK. Fix coming in 3, 2, 1...
See if does the trick for you.
commented there. it fixed it.


Question: if !isCreated(replicas[i]) { return ssc.podControl.CreateStatefulPod(set, replicas[i]) } <ins> // If we find a Pod that is currently terminating, we must wait until graceful deletion // completes before we continue to make  progress. if isTerminating(replicas[i]) { glog.V(2).Infof("StatefulSet %s is waiting for Pod %s to Terminate", set.Name, replicas[i].Name) return nil } </ins> // If we have a Pod that has been created but is not running and ready we can not make progress. // We must ensure that all for each Pod, when we create it, all of its predecessors, with respect to its // ordinal, are Running and Ready.

/assign flakes in PR jobs more than once per day The can filter down to searching for results for a single test:
This test continues to flake PRs at least once per day:
https://k8s-
looking at it


Question: if err := jm.podControl.DeletePod(job.Namespace, activePods[ix].Name, job); err != nil { defer utilruntime.HandleError(err) // Decrement the expected number of deletes because the informer won't observe this deletion <ins> glog.V(2).Infof("Failed to delete %v, decrementing expectations for job %q/%q", activePods[ix].Name, job.Namespace, job.Name) </ins> jm.expectations.DeletionObserved(jobKey) activeLock.Lock() active++ activeLock.Unlock() <ins> errCh <- err </ins> } }(i) }

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--What keywords did you search in Kubernetes issues before filing this one? Job, resourcequota, scheduler Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-04-03T20:44:38Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-04-14T13:36:25Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"} Environment: CentOS bare metal () Cloud provider or hardware configuration: - OS (e.g. from /etc/os-release): CentOS 7 Kernel (e.g. ): Linux <server name3.10.0-514.2.2.el7.x8664 SMP Tue Dec 6 23:06:41 UTC 2016 x8664 x8664 x8664 GNU/Linux Install tools: - Others: - What happened: Having created a resourcequota (e.g. 400M) and creating enough memory-limited jobs to go over that quota (e.g. 4 x 200M limit jobs), at first, two jobs are created and executed as expected, and two jobs fail at first due to memory quota: Events: FirstSeen LastSeen Count From SubObjectPath Type Reason Message --------- -------- ----- ---- ------------- -------- ------ ------- 4s 4s 1 job-controller Warning FailedCreate Error creating: pods "memhog-y152c55u-" is forbidden: exceeded quota: memoryquota, requested: memory=200Mi, used: memory=400Mi, limited: memory=400Mi After the first two jobs have been completed, instead of starting the remaining jobs as soon as possible, the remaining two jobs stay unstarted in this state for many, many hours until they finally start and complete.

First job: Start Time: Fri, 28 Apr 2017 15:00:08 +0300 One of the remaining 2 jobs: Start Time: Sat, 29 Apr 2017 07:13:34 +0300 What you expected to happen: I'm expecting the two waiting jobs to start executing as soon as the resources they need are available, e.g. as soon as the first jobs are completed. How to reproduce it (as minimally and precisely as possible): Bring up a kubernetes cluster, then execute these commands: $ kubectl create quota memoryquota --hard=memory=400Mi (cat /dev/urandom head -n 1)' --replicas=1 --restart=OnFailure --image-pull-policy=IfNotPresent --limits=memory='2'm run- $ chmod +x run- $ for i in {1.4}; do ./run- 200 100; done job "memhog-0ju2eiuy" created job "memhog-t8qyt5sj" created job "memhog-ljaqz4ek" created job "memhog-eb8q8kr3" created Wait for around 30 seconds and the first jobs should be complete: $ kubectl get jobs NAME DESIRED SUCCESSFUL AGE memhog-0ju2eiuy 1 1 43s memhog-eb8q8kr3 1 0 42s memhog-ljaqz4ek 1 0 43s memhog-t8qyt5sj 1 1 43s $ kubectl describe jobs . However, the remaining jobs don't start at all as shown by: $ kubectl describe job memhog-ljaqz4ek Name: memhog-ljaqz4ek Namespace: test Selector: controller-uid=-2f00-11e7-8766- Labels: run=memhog-ljaqz4ek Annotations: run memhog-ljaqz4ek memhog -r100 100m --replicas=1 --restart=OnFailure --image-pull-policy=IfNotPresent --limits=memory=200Mi,cpu=100m --record=true --image=derekway.

Parallelism: 1 Completions: 1 Pods Statuses: 0 Running / 0 Succeeded / 0 Failed Pod Template: Labels: controller-uid=-2f00-11e7-8766- job-name=memhog-ljaqz4ek run=memhog-ljaqz4ek Containers: memhog-ljaqz4ek: Image: derekwaynecarr/memhog Port: Command: memhog -r100 100m Limits: cpu: 100m memory: 200Mi Environment: <noneMounts: <noneVolumes: <noneEvents: FirstSeen LastSeen Count From SubObjectPath Type Reason Message --------- -------- ----- ---- ------------- -------- ------ ------- 3m 3m 1 job-controller Warning FailedCreate Error creating: pods "memhog-ljaqz4ek-" is forbidden: exceeded quota: memoryquota, requested: memory=200Mi, used: memory=400Mi, limited: memory=400Mi Furthermore, pods related to the remaining jobs aren't even created and only the completed pods are shown: $ kubectl get pods -a NAME READY STATUS RESTARTS AGE memhog-0ju2eiuy-bz0zm 0/1 Completed 0 7m memhog-t8qyt5sj-q9msw 0/1 Completed 0 7m Anything else we need to know: I've also created a stackoverflow question about this (), and asked about this in the kubernetes slack, where I was urged to create a bug report.
I will try to reproduce. The creation of a pod will be blocked by quota. In the absence of a pod, the scheduler has nothing to schedule, so it is not a scheduling bug. It's possible the issue is related to back-off in the job controller not attempting to create pods again. The event shows that it had been 3m since the job controller attempted to create pods again for the requisite job which implies some type of back-off. None of the controllers are quota aware, so I would anticipate they should just keep trying at reasonable intervals at this point. If you describe quota after the first 2 pods are completed, do you see quota usage being replenished as expected?
I would like to fix the bug, could you assigin it to me?

if pod create failed, jobcontroller.managejob() need return error, will requeue podcreate task.
-- maybe you have more insight, but it looks to me just reading the code that when a job controller is unable to create a pod due to lack of quota, its not actually an error that forces a re-queue of the job. in addition, i cant actually see when a full resync of the job work queue happens outside of the informer sync interval which is probably really long. see: see:
- feel free to open a pr, please cc me on your solution. i would like to understand from the job folks if there should be a periodic resync interval aside from the need to requeue the job when all pods cannot be spawned (which i assume your pr will handle via exponential backoff)
-- we will want to have a fix for this downstream as well.
Yes, after the first two jobs are complete, shows that 0M out of 400M is in use. I'm also able to create new jobs, up to the memory limit so 2 at a time, that execute as expected. Any jobs over that at a time will stay in the same waiting state.
- Yes, I want handle it via exponential backoff. IMO, MaxOfRateLimiter had implemented a periodic resync, we don't need another one.
I'll make sure to cherry-pick this to openshift ( fyi).


Question: } } <del> func verifyIfKMSTransformersSwapped(t *testing.T, wantPrefix, wantPrefixForEncryptAll string, test *transformTest) { </del> <ins> func verifyIfKMSTransformersSwapped(t *testing.T, wantPrefix string, test *transformTest) { </ins> t.Helper() var swapErr error

transformation expandless | 14m1s Failed; === RUN TestEncryptionConfigHotReload TestEncryptionConfigHotReload When I see this flake, 10-09 is the first one.(But the job history in the testgrid only started from 10-03) some PR that may be related: /sig auth /cc
/cc
/triage accepted
/cc failed two times in recent 3 runs:
/reopen as of this is still .
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: // The context and/or DRAPlugin.Stop can be used to stop all background activity. // Stop also blocks. A logger can be stored in the context to add values or // a name to all log entries. <ins> // // If the plugin will be used to publish resources, [KubeClient] and [NodeName] // options are mandatory. </ins> func Start(ctx context.Context, nodeServer interface{}, opts ...Option) (result DRAPlugin, finalErr error) { logger := klog.FromContext(ctx) o := options{

When calling draplugin.Start without supplying kubeclient no error is reported, while the kubeclient is vital for further announcement of resourceSlice. An error needs to be returned when nodeName or kubeclient parameters are not set. Do not supply kubeclient parameter when creating draplugin, for isntance here: No response master <details</details<details</details<details</details<details</details<details</details
/sig node /triage accepted /project 95


Question: } // MergeSpecsV3 merges OpenAPI v3 specs for CRDs <del> // For V3, the static spec is never merged with the individual CRD specs so no conflict resolution is necessary </del> <ins> // Conflicts belonging to the meta.v1 or autoscaling.v1 group versions are skipped as all CRDs reference those types // Other conflicts will result in an error </ins> func MergeSpecsV3(crdSpecs ...*spec3.OpenAPI) (*spec3.OpenAPI, error) { <del> // create shallow copy of staticSpec, but replace paths and definitions because we modify them. </del> crdSpec := &spec3.OpenAPI{} if len(crdSpecs) > 0 { crdSpec.Version = crdSpecs[0].Version crdSpec.Info = crdSpecs[0].Info } for _, s := range crdSpecs { <del> // merge specs without checking conflicts, since the naming controller prevents // conflicts between user-defined CRDs mergeSpecV3(crdSpec, s) </del> <ins> err := mergeSpecV3(crdSpec, s) if err != nil { return nil, err } </ins> } return crdSpec, nil } // mergeSpecV3 copies paths and definitions from source to dest, mutating dest, but not source. <del> // We assume that conflicts do not matter. func mergeSpecV3(dest, source *spec3.OpenAPI) { </del> <ins> // Conflicts belonging to the meta.v1 or autoscaling.v1 group versions are skipped as all CRDs reference those types // Other conflicts will result in an error func mergeSpecV3(dest, source *spec3.OpenAPI) error { </ins> if source == nil || source.Paths == nil { <del> return </del> <ins> return nil </ins> } if dest.Paths == nil { dest.Paths = &spec3.Paths{}

Related to Seeing the following errors in kube-apiserver logs when CRDs are applied: No errors in logs Apply any CRD manifest No response <details</details<details</details<details</details<detailsRancher RKE2 </details<detailscontainerd 1.6.4 </details<detailscanal </details
/sig api-machinery
/assign
Did this cause any specific trouble to your cluster I'm curious what the impact/severity of this is.
Everything seems to work fine, but there's a bunch of log spew every time a CRD manifest is loaded. In K3s we mix all the control-plane component logs together in a single pair of stdout/stderr streams so it's quite noisy, compared to previous releases. The fact that these come out at WARN level with a scary prefix makes them look pretty serious, and I expect we will get a fair bit of users asking about it once we release.
/triage accepted


Question: FilteredNodesStatuses: filteredNodesStatuses, } } <del> trace.Step("Computing predicates done") </del> <ins> trace.Step("Running postfilter plugins done") </ins> metrics.SchedulingAlgorithmPredicateEvaluationDuration.Observe(metrics.SinceInSeconds(startPredicateEvalTime)) metrics.DeprecatedSchedulingAlgorithmPredicateEvaluationDuration.Observe(metrics.SinceInMicroseconds(startPredicateEvalTime)) metrics.SchedulingLatency.WithLabelValues(metrics.PredicateEvaluation).Observe(metrics.SinceInSeconds(startPredicateEvalTime))

<!-- Please only use this template for submitting enhancement requests --What would you like to be : We have enabled in genericscheduler. However, the steps being traced seem outdated. In particular, the "snapshot" is expensive and should be traced. Other steps can be traced as well wherever appropriate, e.g., PreFilter. Why is this needed: Improve granularity of tracing. /sig scheduling
/assign


Question: } } <del> func (o *EditOptions) visitToPatch( originalInfos []*resource.Info, patchVisitor resource.Visitor, results *editResults, ) error { </del> <ins> func (o *EditOptions) visitToPatch(originalInfos []*resource.Info, patchVisitor resource.Visitor, results *editResults) error { </ins> err := patchVisitor.Visit(func(info *resource.Info, incomingErr error) error { editObjUID, err := meta.NewAccessor().UID(info.Object) if err != nil {

Try running and adding an ingress object, or an empty list, then exit the editor – it will panic like this: I've also tried , and for the sake of experiment tried adding a Pod, then it failed rather gracefully with this message: But when I tried to add a DaemonSet, it paniced. I am not sure how we should fix, perhaps editing of lists isn't something a normal user is expected to do... Let's discuss.
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig cli
So I wanted to test how it would behave if I take an item out of my list, so I've create and ingress object and tried . To my surprise I've found only one ingress object without a list. When I a second object, gave me a list to edit. So I must say that flattening of the list in case there is only one object was a bit surprising. Now that I have a list of two items, I've decided to try deleting one by removing it from the list, and it turns that's not possible.
It feels like the problem is complicated, and I'd rather see disallowed unless users passed a name of a specific resource. If we belive some folks rely on to work when there is only one resource to edit, maybe we can keep that... If we cannot disallow this, I'd rather see it fixed in a way where removing items results in deletion and editing a single item list is actually about editing a list. At the very least, of course, we should fix the panic, but I still want to hear what people think about this before I open a PR (presumably fixing that panic should be trivial).
I'm working on this. Already submitted a PR in . PTAL.
I think at the very beginning, at least v1.5.0 the logic is already like this, I'm not sure why this bothering you I use this lot of time, I'd prefer keep it.

also cc
I don't think edit should open with an empty list. should open a list if there are multiple items, a single item if there is only one, and error if there are none of that type. should open a single item if it is found, and error if it is not found
that sounds good to me. I was just a bit surprised, that's all.


Question: }, expected: "rule foo.com=othersvc:8080 is invalid and should be in format host/path=svcname:svcport[,tls[=secret]]", }, <ins> "valid catch all rule": { rules: []string{ "/path/subpath*=svc:redis,tls=blo", }, expected: "", }, </ins> } for name, tc := range tests {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: kubectl create ingress panics because there is no cmdutil.AddDryRunFlag What you expected to happen: Command to work How to reproduce it (as minimally and precisely as possible): Compile the upstream, run ` Anything else we need to know?: Environment: Kubernetes version (use ): upstream/dev /sig cli /kind bug /priority critical-urgent
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: <ins> #!/bin/bash # Copyright 2014 Google Inc. All rights reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # #     http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # Launches an nginx container and verifies it can be reached. Assumes that # we're being called by hack/e2e-test.sh (we use some env vars it sets up). # Exit on error set -e source "${KUBE_REPO_ROOT}/cluster/kube-env.sh" source "${KUBE_REPO_ROOT}/cluster/$KUBERNETES_PROVIDER/util.sh" function validate() { POD_ID_LIST=$($CLOUDCFG '-template={{range.Items}}{{.ID}} {{end}}' -l name=$controller list pods) # Container turn up on a clean cluster can take a while for the docker image pull. ALL_RUNNING=0 while [ $ALL_RUNNING -ne 1 ]; do echo "Waiting for all containers in pod to come up." sleep 5 ALL_RUNNING=1 for id in $POD_ID_LIST; do CURRENT_STATUS=$($CLOUDCFG -template '{{and .CurrentState.Info.datacontroller.State.Running .CurrentState.Info.net.State.Running}}' get pods/$id) if [ "$CURRENT_STATUS" != "true" ]; then ALL_RUNNING=0 fi done done ids=($POD_ID_LIST) if [ ${#ids[@]} -ne $1 ]; then echo "Unexpected number of pods: ${#ids[@]}" exit 1 fi } controller=dataController # Launch a container $CLOUDCFG -p 8080:80 run brendanburns/data 2 $controller function teardown() { echo "Cleaning up test artifacts" $CLOUDCFG stop $controller $CLOUDCFG rm $controller } trap "teardown" EXIT validate 2 $CLOUDCFG resize $controller 1 validate 1 $CLOUDCFG resize $controller 2 validate 2 # TODO: test rolling update here, but to do so, we need to make the update blocking # $CLOUDCFG -u=20s rollingupdate $controller # # Wait for the replica controller to recreate # sleep 10 # # validate 2 exit 0 </ins>

We have an e2e test that runs the guestbook example. We should also have one that runs through the other example.


Question: // being upgraded, then the container runtime must also support the initially // selected version or the redial is expected to fail, which requires a restart // of kubelet. <del> func (r *remoteRuntimeService) determineAPIVersion(conn *grpc.ClientConn) error { </del> <ins> func (r *remoteRuntimeService) determineAPIVersion(conn *grpc.ClientConn, endpoint string) error { </ins> ctx, cancel := getContextWithTimeout(r.timeout) defer cancel()

When installing the latest component, got the error: Then the flag , got error: It is confusing to see these error msgs at the first glance, need to make it more clearly. Add the required field validation, and add more supportive error message. Download the latest v1.25.0 binary, and install it with systemd service or just manually start the binary, will reproduce these errors. <details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
kubelet 1.24+ is not working with the docker, unless using Are you using it? If not, try switching to other available container runtimes. Please re-open if you have further questions /close
Closing this issue. <detailsIn response to : kubelet 1.24+ is not working with the docker, unless using Are you using it? If not, try switching to other available container runtimes. Please re-open if you have further questions Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Yes, I got to know the , so need to add runtime validation by the fix PR: PTAL thanks.


Question: podCount            = 100 podCreationInterval = 100 * time.Millisecond recoverTimeout      = 5 * time.Minute <del> startTimeout        = 3 * time.Minute </del> <ins> startTimeout        = 5 * time.Minute </ins> // restartCount is chosen so even with minPods we exhaust the default // allocation of a /24. minPods      = 50

node-kubelet-serial Kubelet should correctly account for terminated pods after restart Since test was created, flakes 10-20% of the time. Not entirely clear why, I haven't looked. No response /assign /sig node
Introduced by , maybe we should use to replace See the log below, a pod becomes Succeed may exhaust tens of seconds
/triage accepted
Yeah I only tested this locally and not in CI, it seems that in CI the pods take much longer. Timings just need to be tweaked.
/reopen still a number of flakes since the attached PR merged, I don't think that fixed it.
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
from this link, I see all suites passed, which flakes do you mean?
I see this failure in
so this problem should have been fixed by
From gubernator: In all cases, it's timing out. I'll try upping the startTimeout.
/reopen still flaking
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Luckily the kubelet log is clear as to what's going on. From (flake ) The test should work fine if we drop the CPU request a bit, let me tweak it to make up for the possible extra 100m scheduled. I'm not sure what's causing that (aren't these tests serial? what the heck is getting scheduled) but we should be resilient to it. Also, needs to be fixed. This test will no longer detect the regression it was designed to show after that change. I will revert it and also fix the resource requirements and that should fix this test.
/milestone v1.23 need to fix this before test freeze


Question: EtcdUpgradeVersion       string UpgradeImage             string GCEUpgradeScript         string <del> PrometheusPushGateway    string </del> ContainerRuntime         string ContainerRuntimeEndpoint string ImageServiceEndpoint     string

TLDR: removed a feature but left documentation and the flag intact. Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-03-28T16:36:33Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-01-12T04:52:34Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"} Environment: Cloud provider or hardware configuration: irrelevant OS (e.g. from /etc/os-release): irrelevant Kernel (e.g. ): irrelevant Install tools: kops Others: What happened: I am trying to set up periodic conformance tests on our cluster to make sure it is working as expected. Prometheus will be used for monitoring. After ing kubernetes I executed a command along the lines of: GINKGOPARALLEL=y KUBERNETESPROVIDER=local KUBEMASTER=local --host="" --test.v=true --"[(SlowSerial)]" --"[Conformance]" --prom-push-gateway=prometheus-push- The push gateway does never receive the metrics, because the feature was removed. What you expected to happen: The tests send the metrics to the push gateway. How to reproduce it (as minimally and precisely as possible): Search for the "PrometheusPushGateway" flag in this repo. It is used nowhere. Anything else we need to know: The feature was removed in but the flags were left in place and also in the documentation. It is frustrating to see that this flag is just a black hole. A solution to this would be to either reinstate the feature or remove the flag from docs and flags.
Blerg. pointed this out back in the PR that remove the functionality, let's follow through and remove the flags. Easy PR if someone wants it