Question: // RunIfContainerRuntimeIs runs if the container runtime is included in the runtimes. func RunIfContainerRuntimeIs(runtimes ...string) { for _, containerRuntime := range runtimes { <del> if containerRuntime == TestContext.ContainerRuntime { </del> <ins> if containerRuntime == framework.TestContext.ContainerRuntime { </ins> return } } <del> skipInternalf(1, "Skipped because container runtime %q is not in %s", TestContext.ContainerRuntime, runtimes) </del> <ins> skipInternalf(1, "Skipped because container runtime %q is not in %s", framework.TestContext.ContainerRuntime, runtimes) </ins> } // RunIfSystemSpecNameIs runs if the system spec name is included in the names. func RunIfSystemSpecNameIs(names ...string) { for _, name := range names { <del> if name == TestContext.SystemSpecName { </del> <ins> if name == framework.TestContext.SystemSpecName { </ins> return } } <del> skipInternalf(1, "Skipped because system spec name %q is not in %v", TestContext.SystemSpecName, names) </del> <ins> skipInternalf(1, "Skipped because system spec name %q is not in %v", framework.TestContext.SystemSpecName, names) </ins> }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Flexvolume e2e tests are skipped in https://k8s-, so as far as I can tell we don't have Flexvolume test coverage. They are skipped because they require SSH access to drop the Flexvolume plugin onto the host, but these test clusters do not have SSH access. We should change all Flexvolume tests to use the DaemonSet installation mechanism instead which does not require SSH access. See and Flexvolume tests I'm aware of are below, but there could be more: Looking for help with this bug! /sig storage /priority important-soon /help
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : <!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! --What happened: Flexvolume e2e tests are skipped in https://k8s-, so as far as I can tell we don't have Flexvolume test coverage. /priority important-soon /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hi, Can I work on this issue? I think i know how to handle it.
are you still working on this? I'm interested in helping out as well.
/assign


Question: kube::etcd::install() { ( <ins> local os </ins> cd "${KUBE_ROOT}/third_party" <del> if [[ $(readlink etcd) == etcd-v${ETCD_VERSION}-* ]]; then </del> <ins> os=$(uname | tr "[:upper:]" "[:lower:]") if [[ $(readlink etcd) == etcd-v${ETCD_VERSION}-${os}-* ]]; then </ins> return  # already installed fi <del> if [[ $(uname) == "Darwin" ]]; then </del> <ins> if [[ ${os} == "darwin" ]]; then </ins> download_file="etcd-v${ETCD_VERSION}-darwin-amd64.zip" url="https://github.com/coreos/etcd/releases/download/v${ETCD_VERSION}/${download_file}" kube::util::download_file "${url}" "${download_file}"

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: During k8s cluster launch, is checked and if any etcd binary is found regardless of current OS Distribution, expected etcd binary is not downloaded and etcd soft link is not updated to point desired etcd binary, so cluster creation fails. What you expected to happen: Along with expected etcd version, OS Distribution should also be checked for successfully and efficiently installing etcd during setting up local cluster. How to reproduce it (as minimally and precisely as possible): Checkout k8s codbase on MacOS and launch cluster. working k8s codebase under virtual machine to launch cluster in VM. (Cluster creation would fail, prompting error that etcd needs to be installed and update in {KUBEROOT}/thirdparty/etcd-v{ETCDVERSION}-linux-* Anything else we need to know?: It's an optimization on managing thirdparty etcd binary for local cluster setup. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
do you have a patch that you are working on? we don't really support local-up-cluster on macos yet. /sig testing
is fixing this issue. Though I use Mac as my workstation but sometimes need local-cluster to test codebase so I mapped (softlinked) my GOPATH in virtual machine keeping separate bin directory. Everything is running perfectly fine after making changes mentioned in this PR. So thought to submit small optimization on etcd installation logic.
/close


Question: sharedConcurrencyMetricsName      = "apiserver_flowcontrol_request_concurrency_limit" dispatchedRequestCountMetricsName = "apiserver_flowcontrol_dispatched_requests_total" rejectedRequestCountMetricsName   = "apiserver_flowcontrol_rejected_requests_total" <del> labelPriorityLevel                = "priorityLevel" </del> <ins> labelPriorityLevel                = "priority_level" </ins> timeout                           = time.Second * 10 )

<!-- Please only use this template for submitting enhancement requests --What would you like to be : API priority and fairness metrics have labels such as "priorityLevel" and "flowSchema". These should be "prioritylevel" and "flowschema" respectively. These metrics have been to an exception list: Why is this needed: snake_case is consistent with other metrics. /sig api-machinery /kind cleanup /cc Do we want to do this before 1.20 beta?
yes, lets do that.
/triage accepted Yes, we do not want to go to beta with known wrongness in the metrics.


Question: MaxDisruptedPodSize = 2000 ) <ins> // EvictionsRetry is the retry for a conflict where multiple clients // are making changes to the same resource. var EvictionsRetry = wait.Backoff{ Steps:    20, Duration: 500 * time.Millisecond, Factor:   1.0, Jitter:   0.1, } </ins> func newEvictionStorage(store *registry.Store, podDisruptionBudgetClient policyclient.PodDisruptionBudgetsGetter) *EvictionREST { return &EvictionREST{store: store, podDisruptionBudgetClient: podDisruptionBudgetClient} }

What happened: When I try to test after implementing the 1st approach according to , I found pod disruption budgets cannot handle concurrent requests. The error message is: It is an Optimistic Lock error message. It means PDB can't handle concurrent requests. How to reproduce it (as minimally and precisely as possible): 1) Create a deployment with 6 replicas and schedule them to the same node, say node A. You can leave only one node scheduling enabled, and then create the deployment. 2) Create the PDB that match the deployment's label 3) Uncordon another node, say node B 4) Drain node A using concurrent POST eviction requests. Ref: I'm not sure if this behavior is OK for PDB. I think I can make work by waiting for a while if current pod label(s) match previous pod label(s).
Oh, that's interesting. I think the problem is that this function doesn't retry if UpdateStatus() fails due to version conflict. I think the solution is to modify the caller, i.e. this function so that instead of just it instead does something like This will loop re-doing checkAndDecrement() and getPodDisruptionBudgets() until checkAndDecrement() succeeds, or fails with a non-conflict error. Does that make sense? cc/ for sanity check
Or use PATCH?
Yes, PATCH should work also. I haven't written code that uses PATCH, so I can't advise exactly the code to write here (there are examples by grepping for patch, but I'm not sure exactly what is the right way to do it from inside of API server code, which is where we are here). can you do a PR to implement this, either using the approach I described above or using PATCH? I think this is important to get into 1.5.
I think PATCH may NOT be safe here. For example: 1) current pdb 2) creating eviction A: get 3) creating eviction B: get 4) creating eviction A: PATCH pdb 5) creating eviction B: PATCH pdb 6) pdb's . It's wrong. Retry loop make more sense to me.
Good point! You're right. Retry loop should be OK.
I can help to create a PR using retry loop.

Could you please triage this for 1.5 and apply blocker or non-blocker labels as appropriate along with priority.
Done.
who is actually working on this issue? Could one of you set yourselves as an assignee for this issue? Thanks! cc:
retry logic with PR needs reviews/lgtm


Question: ./cluster/kube-up.sh ``` <del> Vagrant will provision each machine in the cluster with all the necessary components to run Kubernetes.  The initial setup can take a few minutes to complete on each machine. By default, each VM in the cluster is running Fedora, and all of the Kubernetes services are installed into systemd. </del> <ins> By default, each VM in the cluster is running Fedora. </ins> To access the master or any node:

I cannot find such a service when . Same thing applies for . My cluster runs fine. cc:
Ah, good catch... everything is run as a container now, will update the documentation on where to get the logs.


Question: var map_RuleWithOperations = map[string]string{ "":           "RuleWithOperations is a tuple of Operations and Resources. It is recommended to make sure that all the tuple expansions are valid.", <del> "operations": "Operations is the operations the admission hook cares about - CREATE, UPDATE, or * for all operations. If '*' is present, the length of the slice must be one. Required.", </del> <ins> "operations": "Operations is the operations the admission hook cares about - CREATE, UPDATE, DELETE, CONNECT or * for all of those operations and any future admission operations that are added. If '*' is present, the length of the slice must be one. Required.", </ins> } func (RuleWithOperations) SwaggerDoc() map[string]string {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Working on other doc bugs, I went looking for what calls admission plugins. I found four call sites for validating plugins, documented in . They correspond to the verbs CONNECT, CREATE, DELETE, and UPDATE. But only two of those verbs are documented at What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
Looks like docs are correct in and
The API doc can be updated to call out all four operations and clarify that "all" means "all four admission operations"
/good-first-issue
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /good-first-issue </details
/assign


Question: "Maximum number of NAT connections to track (0 to leave as-is). This overrides conntrack-max-per-core and conntrack-min.") fs.MarkDeprecated("conntrack-max", "This feature will be removed in a later release.") <del> fs.BoolVar(&o.CleanupAndExit, "cleanup-iptables", o.CleanupAndExit, "If true cleanup iptables and ipvs rules and exit.") fs.MarkDeprecated("cleanup-iptables", "This flag is replaced by --cleanup.") </del> fs.StringVar(&o.config.ResourceContainer, "resource-container", o.config.ResourceContainer, "Absolute name of the resource-only container to create and run the Kube-proxy in (Default: /kube-proxy).") fs.MarkDeprecated("resource-container", "This feature will be removed in a later release.") }

The flag has been deprecated for many releases and can be removed. The depreciation guide specifies that admin-only CLI flags can be removed after 6 months: /kind cleanup /sig network /priority backlog
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/remove-triage unresolved
Thanks In future, please comment/check when taking on an issue, I didn't realize someone was going to do this.
Sorry, I'll comment on the issue and check next time.
That's okay, I only wasted a few minutes on it before I noticed the PR.


Question: func (t *TableSorter) Swap(i, j int) { t.obj.Rows[i], t.obj.Rows[j] = t.obj.Rows[j], t.obj.Rows[i] <ins> t.parsedRows[i], t.parsedRows[j] = t.parsedRows[j], t.parsedRows[i] </ins> } func (t *TableSorter) Less(i, j int) bool { iValues := t.parsedRows[i] jValues := t.parsedRows[j] <del> if len(iValues) == 0 || len(iValues[0]) == 0 || len(jValues) == 0 || len(jValues[0]) == 0 { klog.Fatalf("couldn't find any field with path %q in the list of objects", t.field) </del> <ins> if len(iValues) == 0 || len(iValues[0]) == 0 { return true } if len(jValues) == 0 || len(jValues[0]) == 0 { return false </ins> } iField := iValues[0][0]

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: seems to have regressed in v1.13.0 and no longer sorts by creation timestamp. What you expected to happen: should sort by creation timestamp. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: N/A Environment: Kubernetes version (use ): <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
/sig cli
/assign
bisected to () /assign
opened to fix
/assign


Question: // NestedObjectDecoder is an optional interface that objects may implement to be given // an opportunity to decode any nested Objects / RawExtensions during serialization. <ins> // It is possible for DecodeNestedObjects to return a non-nil error but for the decoding // to have succeeded in the case of strict decoding errors (e.g. unknown/duplicate fields). // As such it is important for callers of DecodeNestedObjects to check to confirm whether // an error is a runtime.StrictDecodingError before short circuiting. // Similarly, implementations of DecodeNestedObjects should ensure that a runtime.StrictDecodingError // is only returned when the rest of decoding has succeeded. </ins> type NestedObjectDecoder interface { DecodeNestedObjects(d Decoder) error }

With the addition of server side validation in 1.23, it is now possible for a call to successfully decode it’s data and return a non-nil error when there are strict decoding errors (e.g. duplicate/unknown fields). This means that callers of can distinguish between fatal errors (that cause decoding to fail) and strict decoding errors (that still allow the decode to succeed) by inspecting the target object (to see if it was successfully decoded into) and the error itself (to see if it is a ) With the NestedObjectDecoder however, it only returns an error from giving no indication whether a strict decoding error it has returned comes from an internal decode step that short circuited (by not accounting for strict decoding errors) or actually did successfully decode, but also encountered strict decoding errors. We propose taking a few actions to minimize the possibility of erroneous short-circuiting from Document how all nested decoders should only return strict decoding errors if all decoding is successful or return only non-strict decoding (ie. fatal errors). Document this for all known in-tree and out-of-tree . Update in-tree nested decoders to do the above mentioned behavior (which is just the kube-scheduler) Long-term solutions to better guarantee that no erroneous short-circuiting is taking place could include deprecating the current interface in favor of one that returns both the target object that was decoded into along with an error return a special error type specific to indicating that all internal decoding was successful.
/sig api-machinery
/triage accepted


Question: } // Unmount runs umount(8) in the host's mount namespace. <del> func (*NsenterMounter) Unmount(target string) error { </del> <ins> func (n *NsenterMounter) Unmount(target string) error { </ins> args := []string{ "--mount=/rootfs/proc/1/ns/mnt", <del> "/usr/bin/umount", </del> <ins> "--", n.absHostPath("umount"), </ins> target, }

What happened: Windows documentation claims: V1.VolumeMount.mountPropagation - only MountPropagationHostToContainer is available. However, the internals suggest otherwise; is always returned on Windows: Either the docs should be corrected, or the Kubelet should be updated, or we should have a comment on why this does in fact enable MountPropagationHostToContainer on Windows. /cc /sig windows
/cc
/cc
I think the doc is not accurate. The doc claims the same thing. For MountPropagationHostToContainer option, we can test it whether it is working or not.
i created two PRs to address the documentation gap. we will also test this and get back to this thread.
For 1.15 we may need to update Kubelet code to allow HostToContainer.


Question: R: &etcd.Response{ EtcdIndex: 10, Node: &etcd.Node{ <ins> Dir: true, </ins> Nodes: []*etcd.Node{ { <ins> Key:           "/foo", </ins> Value:         `{"id":"foo","kind":"Pod","apiVersion":"v1beta1"}`, <ins> Dir:           false, </ins> ModifiedIndex: 1, }, { <ins> Key:           "/bar", </ins> Value:         `{"id":"bar","kind":"Pod","apiVersion":"v1beta1"}`, <ins> Dir:           false, </ins> ModifiedIndex: 2, }, { <ins> Key:           "/baz", </ins> Value:         `{"id":"baz","kind":"Pod","apiVersion":"v1beta1"}`, <ins> Dir:           false, </ins> ModifiedIndex: 3, }, },

kubectl returns resources in a random order, which makes it hard to look for things when users have a lot resources. Returning resources in deterministic order should be a helpful and trivial change (hopefully). We have an issue about pagination , but that's a larger and separate issue.
We should be cause to not do this on the * namespace - I think thats a special case. No disagreement for other list calls
What is the default etcd response order? Last created or last modified? Sent from my iPhone
I think the order is by key if sort param is given. If not given, the order is by last created (maybe, but we shouldn't trust it). It seems we are not trying to get sorted result from etcd.
Today, I think it's random order in the child key space in etcd (because the node structure uses a map), so you get golang map iteration order without sort. I don't know if sort on server is any cheaper than sort on client today, so we may want to do this in rest / registry rather than by passing etcd.
Apart from performance, sorting in clients means we can also take control of how responses are sorted, so for * namespace, we can sort by namespace, then by resource ID. , Clayton Coleman wrote: Master of Electrical and Computer Engineering Carnegie Mellon University
We can always hint sort to the registry storage layer in the future, so sorting on the client now won't kill us.
Just took a closer look at our rest/registry code to implement the sort (in client), but implementation-wise, it seems much easier to pass sort to etcd (one line change). Also, namespace is a prefix on resource, so sort across all namespace with etcd key will essentially sort resources by namespace then by Name. If we are to implement sort on client side, we can either: sort package: have each list object, ie.e PodList, ControllerList, implement sort interface. (We won't do this). have ListMeta implement sort interface, if there is a way to find enclosing structs (so we can use items field) have api.Object implement sort interface, verify object is list before sorting. sort algorithm ourself, and do it in rest. Is there other golang or kubernetes idom that I haven't been aware of that can be applied to this use case?

----- Original Message ----- I think sorting it on etcd is fine for now - that's the simplest possible thing that works.
Is there any material perf impact in etcd to enable it? Sent from my iPhone
Probably not material. Can always switch away if necessary
Famous last words :-) Sent from my iPhone
Got my hat ready


Question: containerStatus = append(containerStatus, s) } podStatus := v1.PodStatus{ <del> InitContainerStatuses: containerStatus[:24], ContainerStatuses:     containerStatus[24:], </del> <ins> InitContainerStatuses:      containerStatus[:16], ContainerStatuses:          containerStatus[16:32], EphemeralContainerStatuses: containerStatus[32:], </ins> } result := normalizeStatus(&pod, &podStatus) count := 0

is not normalized in in the status manager: This does not seem to cause any user facing problem because is sorted before passed to the status manager and timestamps are normalized at marshaling to create a patch: However, an unexpected behavior is caused internally because timestamps are not normalized. If the log verbosity is 3 or larger, the following messages are logged periodically in kubelet when an ephemeral container exists: This message is emitted here: After returns , is called. This does not look so harmful because gets eventually: API is called once unnecessarily in , though: It might be better to normalize ephemeral container statuses, at least timestamps, in the status manager to avoid this unexpected behavior. Set the log verbosity of kubelet to 3 or larger. Then, create an ephemeral container like . No response <details</details<detailsNone </details<details</details<details</details<details</details<details</details
/sig node
This is a bug. Can you submit a PR to fix it? If not, I'd be happy to work on it
If you are interested, please feel free to assign yourself.
thx
/assign
/triage accepted /priority important-longterm


Question: &HorizontalPodAutoscaler{}, &HorizontalPodAutoscalerList{}, &api.ListOptions{}, <ins> &api.DeleteOptions{}, </ins> ) return nil }

Kubernetes version: HEAD What happened: Using interface to delete Job throws an error . What you expected to happen: Job should be deleted. We've found that issue after dependency update on dashboard: We are using to delete resources. It works fine for all other resources. I've investigated that and it appears to be the core issue. Other groups have registered as a known type. however batch is missing it After adding I was able to successfully delete Job using our code. I can quickly create PR for that. Just need a confirmation that this is indeed a bug. cc
+1 apis/autoscaling is similar
I thought we only needed to register it in the external versions (). I'll double check. cc
Registering api.DeleteOptions in batch looks fine. In the future, I think it's more proper to register DeleteOptions and ListOptions as unversioned, like how we treat .
DeleteOptions and ListOptions are expected to be in their group. They have to be in internal version and external version. Most of this code will be fixed in 1.6 (we'll standardize registering the "common" things). , Chao Xu wrote:
I've created PR to fix this issue. Could anyone check that?


Question: if err != nil && err != ErrNotFound { return nil, fmt.Errorf("Error getting floating ip for port %s: %v", portID, err) } <del> if floatIP == nil && lbaas.opts.FloatingNetworkId != "" { </del> <ins> if floatIP == nil && floatingPool != "" { </ins> glog.V(4).Infof("Creating floating ip for loadbalancer %s port %s", loadbalancer.ID, portID) floatIPOpts := floatingips.CreateOpts{ <del> FloatingNetworkID: lbaas.opts.FloatingNetworkId, </del> <ins> FloatingNetworkID: floatingPool, </ins> PortID:            portID, } floatIP, err = floatingips.Create(lbaas.network, floatIPOpts).Extract()

Is this a BUG REPORT or FEATURE REQUEST?: Feature request /kind feature What happened: Currently we have kubernetes cluster installed, in node machines we have one subnet and two routers attached. One router for external network and one router for internal network. However, when we create type: loadbalancer services in kubernetes, we have to hardcode floatingnetwork id to only one of those (external or internal). What you expected to happen: I except that I could define from which floatingnetwork loadbalancer is taking its floating ip address in each service. One possible solution could be that if floatingnetworkid is defined like currently - that networkid is default one. Then we could have possibility to override that in each service configuration. Like if we do not want use default floating-pool we could change it: I do not know does this even work in openstack, it might need some manual route settings in lbaas. Environment: Kubernetes version (use ): 1.6 Cloud provider or hardware configuration**: openstack OS (e.g. from /etc/os-release): centos cc
Exposing first to external network so the external floatingip is working. Then update floating-networkid in kubernetes cloud-config in all nodes+masters and restart all machines. Exposing to internal network However, this last curl did not work from outside openstack. There is some route issue with it, if I try that outside openstack the result is: tcpdump from node the external one which works 192.168.1.12 is lbaas fixed ip: tcpdump from node the internal one which does not work 192.168.1.13 is lbaas fixed ip: This might mean that lbaas is using default router (which is external) to answer back to client, so the route in lbaas should be modified. No idea is that possible.
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list .

The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig openstack /sig network
I have defined host route to subnet, which is 10.222.0.0/16,192.168.1.4 this row basically means that all connections toward 10.222.0.0/16 should use 192.168.1.4 router interface, which is internal router. However, openstack lbaas is ignoring these host routes so before that is fixed there is no sense to code this feature to kubernetes. edit. looks like it is fixed that fix is not to our openstack yet. After that fix this is this my idea should work. Now the question is how we should implement this multiple floatingnetworkids to kubernetes. I can make PR but first I would like to discuss how it should be done.
Sounds good. cc
the problem what I am thinking currently is that I would like to have possibility to specify floating-pool in each service. Some service people may want expose service to external network and some service to internal network. However, the service api spec is quite hardcoded or can I add variable there? I am planning to add variable like there.


Question: * GET /watch/<resourceNamePlural> - Receive a stream of JSON objects corresponding to changes made to any resource of the given kind over time. * GET /watch/<resourceNamePlural>/<name> - Receive a stream of JSON objects corresponding to changes made to the named resource of the given kind over time. * GET /redirect/<resourceNamePlural>/<name> - If the named resource can be described by a URL, return an HTTP redirect to that URL instead of a JSON response. For example, a service exposes a port and IP address and a client could invoke the redirect verb to receive an HTTP 307 redirection to that port and IP. <ins> * GET /proxy/<resourceNamePlural>/<name>/{subpath:*} - Proxy GET request to the named resource of the given kind. Can be used for example, to access log files on pods. </ins> Support of additional verbs is not required for all object types. <del> TODO: document proxy </del> TODO: more documentation of Watch

The schema understood by the Kubernetes configuration file is not described any where. The current practice mostly is copy/paste existing samples (for example ) and edit them. There is a need to define a comprehensive list of elements, and their documentation, so that other examples can leverage them.
Configuration files specify exactly our API objects. See:
This is exactly what I was looking for but not easily googlable, or documented clearly.
Could you please put a link to the swagger UI here?
The only mention in the public docs is here: which links directly to the struct definition on github. It appears that the Swagger docs have been deprecated, and I'm unable to find any documentation for the KubeletConfiguration resource kind in the new docs. It's referred to in at least one place, but is still not actually covered:


Question: "x-kubernetes-patch-strategy": "merge" }, "generateName": { <del> "description": "GenerateName is an optional prefix, used by the server, to generate a unique name ONLY IF the Name field has not been provided. If this field is used, the name returned to the client will be different than the name passed. This value will also be combined with a unique suffix. The provided value has the same validation rules as the Name field, and may be truncated by the length of the suffix required to make the value unique on the server.nnIf this field is specified and the generated name exists, the server will NOT return a 409 - instead, it will either return 201 Created or 500 with Reason ServerTimeout indicating a unique name could not be found in the time allotted, and the client should retry (optionally after the time indicated in the Retry-After header).nnApplied only if Name is not specified. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#idempotency", </del> <ins> "description": "GenerateName is an optional prefix, used by the server, to generate a unique name ONLY IF the Name field has not been provided. If this field is used, the name returned to the client will be different than the name passed. This value will also be combined with a unique suffix. The provided value has the same validation rules as the Name field, and may be truncated by the length of the suffix required to make the value unique on the server.nnIf this field is specified and the generated name exists, the server will return a 409.nnApplied only if Name is not specified. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#idempotency", </ins> "type": "string" }, "generation": {

The error code of is 409. Is this an expected behavior? The doc seems to say no: NOT return a 409 - instead, it will either return 201 Created or 500 with Reason ServerTimeout indicating a unique name could not be found in the time allotted, and the client should retry (optionally after the time indicated in the Retry-After header). seems to create which error code is 409. The error code of isn't 409. Or, the doc doesn't say "If this field is specified and the generated name exists, the server will NOT return a 409". Create a large number of Pods with generateName, and cause name conflicts. Just question: if the API is correct in returning 409 for a GenerateName-conflict, how do we determine that error? We want to retry when GenerateName conflict returns in most cases because the retry may solve this conflict. latest master (maybe after ) <details</details<details</details<details</details<details</details<details</details
Sorry if not an appropriate SIG. /sig api-machinery
/assign
It seems related to . If it's expected behavior, the document isn't correct..?
is a mistake. /unassign
Hi do you think this is a good first issue to tackle? If so, I would like to try contributing here.
/assign /triage accepted
Forgive me to ping other people who might know. /cc /cc /cc
I suspect we just failed to update the documentation.
Thanks Okay. I'll create the PR for that. /kind documentation /assign


Question: if shares < MinShares { return MinShares } <ins> if shares > MaxShares { return MaxShares } </ins> return uint64(shares) }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: kubelet fails to create kubepods cgroup on larger machine with 512 cpus, fails with the following error: What you expected to happen: Run kubelet properly without any error How to reproduce it (as minimally and precisely as possible): Start the kubelet on a machine which has a more processors Anything else we need to know?: Environment: Kubernetes version (use ): 18.5 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig node
When I changed CgroupDriver to cgroupfs, kubelet is running is fine:
And here is the sample code which is creating this issue:
so this is essentially a runc/libcontainer problem or a problem with the host configuration - e.g. the host having less than the requested ? this line: hints that your snippet is for an older version of runc. rc8 renamed it to "Manager": is that the version of the kubelet too? k/k 1.18 is using rc10: so ideally the match test for kubelet 1.18.5 should be against rc10 of runc. and current master of k/k is at rc90: are you able to reproduce the same with rc90?
I tried running with master kublet which does use rc90 build and kubelet fails with the same error: complete log: I think this is happening because we are trying to create a cgroup more than a supported range for the Linux kernel: 2^18 = is the max value we can use for the CPUShare, so wondering whether can add logic into the kubelet to address this issue when we run on bigger machines(more than 256 processors)
/cc
interesting snippet in moby: ! where
i think we have a misconception here that CPU shares == # of CPU. "cpu shares" is a weighting proportion to give a container more resources in comparison to other running containers: the reason for 2^18 in the kernel is to have a larger integer scale to work with.

unlike docker that has (nanocpu / fractional cpus) (): does not have such an option. but it does expose (CpusetCpus) which accepts a such as . i have a wild guess that this same problem is not present if you use dockershim. to see more on what is happening you could: run the kubelet with --v=4 or more. patch the kubelet to add debug code.
sorry for the late reply. I think the issue is not in libcontainer, but in the Kubelet asking for a out of range value for cpu shares. I am not sure how it makes sense to set the number of shares in relation to the number of available CPUs, since it is a weight and not a limit. The error you are seeing is coming straight from systemd: Since there is already a check for the minimum value, I think it makes sense to add also a check for the maximum allowed (while we figure out why cpu shares are used this way). Could you give a try to the current patch? I've nothing close to 512 cores to validate it really solves the issue you've seen :smile: I'll open a PR once we are sure it solves the problem
You are right, this is coming from the systemd library which is getting used in the libcontainer With the above patch, I see kubelet is running fine but I don't think this is just good enough because while requesting the cpus in the pod is in terms of milicores which is 1000th of 1 cpu which is getting converted into and assigned to the pod's cgroup. e.g: if the user requests a cpu of which 1024/ = ~2 cpu share out 512 which brakes the current logic in the code. I think we need to add an additional logic for allocation based on the number of cpus present in the machine. Simple math should work fine if cpus are more than 256 in the system: e.g: if the request is 1024m 1024/(()/) = 512 shares
the value doesn't really limit the cpu time a process gets. It is a weight used for the cgroups at the same level. It looks to me that the Kubelet is using this mechanism simply to maintain some sort of priorities among the different pods (i.e.

if pod A has double millicpus than pod B, then A gets double weight than B). Before adding more complexity, I'd make sure why it is done this way, I think it is just wrong to put in relation to the number of cores. The cpu shares assigned to each pod though are not dependent on the number of cores, but they are converted directly from the spec file. You've reported the issue on systemd, and it happens since systemd adds another check that is not present with cgroupfs or the kernel. The kernel does the same clamping I've proposed above: so the patch is just to make sure the systemd cgroup driver behaves in the same way as cgroupfs; and I think it doesn't have any impact on the pods
I've opened a PR here:
Thanks for the PR, my only worry about - how k8s will prioritize when we have two pods requesting more than the MaxShares how this logic will prioritize it!
for now it works in the same way as in the cgroupfs case. Pod requests are not in relationship with the # of cores, that AFAICS happens only when setting up the node cgroups. I think a static allocation would work better for such cgroups, as anyway is a weight.


Question: - ./hack/build-go.sh script: <del> - KUBE_RACE="-race" KUBE_COVER="-cover -covermode=atomic" KUBE_TIMEOUT='-timeout 60s' ./hack/test-go.sh "" -p=4 </del> <ins> - KUBE_RACE="-race" KUBE_COVER="y" KUBE_GOVERALLS_BIN="$HOME/gopath/bin/goveralls" KUBE_TIMEOUT='-timeout 60s' ./hack/test-go.sh "" -p=4 </ins> - PATH=$HOME/gopath/bin:./third_party/etcd:$PATH ./hack/test-cmd.sh - PATH=$HOME/gopath/bin:./third_party/etcd:$PATH ./hack/verify-gendocs.sh - PATH=$HOME/gopath/bin:./third_party/etcd:$PATH ./hack/test-integration.sh

Right now, we execute unit tests in Travis with , and you can see the results on each line: but we don't have an easy place to go browse our code and figure out what's covered and what isn't. Is there some awesome way to do this? At the very least, can we get the actual line-by-line coverage results integrated into our CI build process so we can poke at these somehow?
cc
I'm willing to take a look into this to at least see if something is possible.
Looking into using Coveralls; there are a few other teams at Google already using this. Integration with Travis and Go are supported with the goveralls package:
Unit test coverage can now be viewed at - both master push builds and PRs will be shown there; additionally, Coveralls will update the status and post comments on PRs with coverage stats. (If we find these too spammy, we can disable either or both.) The top-level also has a badge/link to the coverage dashbaord.


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: It appears that validating admission requests for server-side apply are populating the for requests to create Namespace objects. Non server-side apply requests don't do this, and a missing namespace field is the only way to distinguish cluster-scoped resources from namespace-scoped resources. This appears to be a breaking behavioral change. See What you expected to happen: Requests to create namespaces would have a consistent signature between server-side apply and non-server-side apply contexts. How to reproduce it (as minimally and precisely as possible): See Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: kind OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig sig-api-machinery
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig sig-api-machinery Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig api-machinery
/wg api-expression
/assign
/assign
Requests made to namespaced URLs (URLs which contain a segment) populate the attribute in AdmissionReview requests. The documentation of the AdmissionReview field is as follows: From Update/Patch/Delete requests to specific namespaces have consistently populate this attribute for modifications to Namespace objects. Server-side apply is a type of patch request, so even when it is creating an object, that attribute will be populated. The difference is caused by a difference in the URL used by POST and apply-based creates. Creation via POST requests are made against the endpoint, so no attribute was populated. Namespace computed from request path: Namespace admission attributes for POST-based creates: Creation via server-side apply is done via a PATCH request to , so the attribute is populated. Namespace computed from request path: Namespace admission for apply-based creates: This is currently working as intended. If we want to make the behavior consistent between the two, I'd consider populating the namespace attribute for POST-based creation of Namespace objects.

Thanks for the thorough analysis Jordan, the problem OP is trying to solve is that they want to know if the object is namespaced or not, and used the presence of the field to detect that. According to your analysis, that seems wrong. The typical way to know if an object is namespaced is to use the discovery information, would that be the right way to solve that?
They could use discovery, or can continue to use the presence of the attribute, special-casing requests (namespaces are the only cluster-scoped resource which can have the attribute populated in AdmissionReview requests).
the documentation should be clearer, and I'd consider updating the POST-based create behavior to include this information to be consistent with update/patch/delete requests
do you want to update the doc and/or the post-based create behavior?
Sure! I am happy to take care of both.
where do you think it makes sense to add the namespace to the post-based behavior? We could add it to , but that might interfere with authorization, right? Would it be better to add the namespace directly to the ?
The info from the request body is not available when parsing requestinfo, only data in the URL. That's sort of what I expected, around the point we do this to pull the name from the incoming object:
Makes sense, I will give it a go, thank you!
that will also require sweeping admission plugins that check the namespace attribute. examples:
I think that we don't need to change anymore now. Do you know if there are other places in the docs where I should check?
yeah, I think we're good now. thanks for helping identify and clean this up


Question: "k8s.io/kubernetes/pkg/volume" ) <del> var errUnsupportedVolumeType = fmt.Errorf("unsupported volume type") </del> // This just exports required functions from kubelet proper, for use by volume // plugins. type volumeHost struct {

in there is a generic error variable defined that is used when a failure occurs and no other error is surfaced, it is a catch-all when something goes wrong. Think this should be reworded to something that is more reflective of the true issue and not so confusing for users. This error mostly comes when a builder can not be created and it is typically not true that the volume type is unsupported, but rather some other underlying issue, such as missing secrets or endpoints or etc... propose something like: or
are you working on this?
- yes, I would like to submit the PR for this

fyi When using PV/PVC for Persistent Volume volume mounts, if something goes wrong in the builder, a generic error is surfaced in the 'kubectl describe' with the following event error description *unsupported volume type": This only occurs when using the PV/PVC abstraction and not directly using a volume plugin. These type of errors are common when using volume plugins for mounts and would be useful for users to have an easy way to identify the errors rather than searching the logs for root cause. I think there are two issues: Why is the original error surfaced from the volume plugins (, , etc...) getting eaten when they return back to the I can see in the logs that 2 volume types are processed for any pod - the first is the actual underlying type (i.e. ) and this is where the actual real error is surfaced, then immediately after this the 2nd type is processed which is the general persistent claim type (i.e. ) and this when the first type error is eaten A simple work around would be to simply add a recorder event in the when there is a builder failure. This ensures the error is recorded and displayed as an event for the describer and will handle all like errors from other volume plugin types. full modifed function for reference from This will produce a better user experience when trying to see what went wrong - in this example, I am doing glusterfs PV/PVC but am missing end points which you can clearly see now in the events of the describe: rbd example: (Missing or Couldn't get secret)
I haven't had a chance to look at your proposal in detail yet, but I agree this is one of the most confusing things new users hit
another one is when gluster endpoints are missing, we get 'unsupported volume type' with your work extrapolating the permissions for NFS to automatically set supplemental groups, is there also a way to indicate more properly it's a permissions issue for it not mounting?
- the above discussion highlights the missing endpoints as the first example, it also discusses a similar rbd issue (missing secret file) but the point is, any of those type of volume plugin type of errors would be surfaced by adding the event in the
I like it! Is there a PR?


Question: configs[Mounttest] = Config{e2eRegistry, "mounttest", "1.0"} configs[MounttestUser] = Config{e2eRegistry, "mounttest-user", "1.0"} configs[Nautilus] = Config{e2eRegistry, "nautilus", "1.0"} <del> configs[NFSProvisioner] = Config{quayIncubator, "nfs-provisioner", "v2.2.0-k8s1.12"} </del> <ins> configs[NFSProvisioner] = Config{quayIncubator, "nfs-provisioner", "v2.2.2"} </ins> configs[Nginx] = Config{dockerLibraryRegistry, "nginx", "1.14-alpine"} configs[NginxNew] = Config{dockerLibraryRegistry, "nginx", "1.15-alpine"} configs[Nonewprivs] = Config{e2eRegistry, "nonewprivs", "1.0"}

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: https://k8s- Which test(s) are failing: all nfs provisioner tests Since when has it been failing: 10/1/2019 Testgrid link: https://k8s- Reason for failure: tbd Anything else we need to know:
cc
fix on the way, got merged but when it rebased the version wasn't updated (needs to be v2.2.2 for that selflink thing)


Question: verbs:     "get,watch,list", resources: "pods,deployments.extensions", expectedClusterRole: &rbac.ClusterRole{ <ins> TypeMeta: v1.TypeMeta{APIVersion: "rbac.authorization.k8s.io/v1", Kind: "ClusterRole"}, </ins> ObjectMeta: v1.ObjectMeta{ Name: clusterRoleName, },


Question: lister:                     informerFactory.Apiregistration().V1().APIServices().Lister(), APIRegistrationInformers:   informerFactory, serviceResolver:            c.ExtraConfig.ServiceResolver, <del> openAPIConfig:              openAPIConfig, </del> <ins> openAPIConfig:              c.GenericConfig.OpenAPIConfig, </ins> egressSelector:             c.GenericConfig.EgressSelector, proxyCurrentCertKeyContent: func() (bytes []byte, bytes2 []byte) { return nil, nil }, }

It looks like the openapi is not routed properly to set-up the apiservice so apply requests will fail because the fieldmanager is not set-up properly. /cc /sig api-machinery /priority important-soon /kind bug
/wg apply /cc
OpenAPIModels is set to nil for the kube-aggregator server config (which serves the APIServices API): This is done because the kube-aggregator has a special openapi handler that aggregates all the other server's openapi docs. The field controls (at least) two things: /openapi endpoint serving (which we want kube-aggregator to continue to customize) creation of the field manager (which we want kube-aggregator to behave consistently on) We need to decouple presence of OpenAPIConfig and construction of StaticOpenAPISpec (which is needed by the field manager) from construction/installation of OpenAPIVersionedService. That might be a little tricky to untangle, since construction of StaticOpenAPISpec includes paths/operations. The field manager only needs the gvk/schemas.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/lifecycle frozen This is not something we want closed by accident.
/assign
For reference:
Additional lines for reference: This is roughly how SSA consumes the openapi spec. openapi spec -openapi proto model -typeconverter -SSA field manager For built in types, openapi spec -openapi model openapi model -typeconverter A similar thing happens for CRDs As liggitt mentioned, SSA only requires a subset of the information in the openapi generation, and the openapi generation in the aggregator slightly different than the process for builtins and CRDs.
/unassign
/assign


Question: expectedEvents: []mockEvent{ {v1.EventTypeNormal, "Successfully assigned recycler-for-podRecyclerDeleted to 127.0.0.1"}, }, <del> expectedError: "recycler pod was deleted", </del> <ins> expectedError: "failed to recycle volume: recycler pod was deleted", </ins> }, { // Another recycler pod is already running <del> name:        "RecyclerRunning", existingPod: newPod("podOldRecycler", v1.PodRunning, ""), createPod:   newPod("podNewRecycler", v1.PodFailed, "mock message"), eventSequence: []watch.Event{ // Old pod succeeds newPodEvent(watch.Modified, "podOldRecycler", v1.PodSucceeded, ""), }, // No error = old pod succeeded. If the new pod was used, there // would be error with "mock message". expectedError: "", }, { // Another recycler pod is already running and fails name:        "FailedRecyclerRunning", existingPod: newPod("podOldRecycler", v1.PodRunning, ""), createPod:   newPod("podNewRecycler", v1.PodFailed, "mock message"), eventSequence: []watch.Event{ // Old pod failure newPodEvent(watch.Modified, "podOldRecycler", v1.PodFailed, "Pod was active on the node longer than specified deadline"), }, // If the new pod was used, there would be error with "mock message". expectedError: "Pod was active on the node longer than specified deadline", </del> <ins> name:          "RecyclerRunning", existingPod:   newPod("podOldRecycler", v1.PodRunning, ""), createPod:     newPod("podNewRecycler", v1.PodFailed, "mock message"), eventSequence: []watch.Event{}, expectedError: "old recycler pod found, will retry later", </ins> }, }

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: If a pod is already running in the default namespace with the expected name (i.e. ), the PV recycle method just assumes it's a recycler pod. However, if it is not a recycler, it could result in the volume being reused without the data being cleared. What you expected to happen: The recycled volume should not be reused until the data has been verifiably deleted. The easiest approach would be to make it an error to not run the recycler pod successfully, i.e. delete these lines: I think that should be safe, since the recycler will retry. Other errors in that method should also be considered fatal, e.g.:
/cc
How about generating a unique name for each recycler pod so that it doesn't matter what existing pods are named?
Could someone from your team pick this up?
Ack
OpenShift runs the recycler pod in a dedicated namespace, not visible to users. Maybe it could be a good idea to do so also in Kubernetes. Anyway, killing the old recycler is probably the best option - it could be forged by user to block recycling and cheat quota. /assign
Agree with running in a dedicated namespace, but it sounds like this feature is just being deprecated, so it might not be worth the effort. Thanks for picking this up.


Question: var map_PodExecOptions = map[string]string{ "":          "PodExecOptions is the query options to a Pod's remote exec call.", "stdin":     "Redirect the standard input stream of the pod for this call. Defaults to false.", <del> "stdout":    "Redirect the standard output stream of the pod for this call. Defaults to true.", "stderr":    "Redirect the standard error stream of the pod for this call. Defaults to true.", </del> <ins> "stdout":    "Redirect the standard output stream of the pod for this call.", "stderr":    "Redirect the standard error stream of the pod for this call.", </ins> "tty":       "TTY if true indicates that a tty will be allocated for the exec call. Defaults to false.", "container": "Container in which to execute the command. Defaults to only container if there is only one container in the pod.", "command":   "Command is the remote command to execute. argv array. Not executed within a shell.",

As per the PRs kubernetes/kubernetes kubernetes/kubernetes, the defaulting behaviour of the pod exec parameters had been changed however in the code there is a comment where it is mentioned it defaults to True The comment about the default value must be removed and the openapi spec must be fixed Present in the file as comment and also in the OpenApiSpec swagger If this issue seems appropriate can I take it up please <details</details<details</details<details</details<details</details<details</details<details</details
/sig docs
/sig node
/assign
/assign
/help /triage accepted
This request has been marked as needing help from a contributor. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help /triage accepted </details
/kind documentation


Question: return fmt.Errorf("rbd: failed to modprobe rbd error:%v", err) } // rbd map <del> l := len(rbd.mon) </del> <ins> l := len(rbd.Mon) </ins> // avoid mount storm, pick a host randomly start := rand.Int() % l // iterate all hosts until mount succeeds. for i := start; i < start+l; i++ { <del> mon := rbd.mon[i%l] </del> <ins> mon := rbd.Mon[i%l] </ins> glog.V(1).Infof("rbd: map mon %s", mon) <del> if rbd.secret != "" { </del> <ins> if rbd.Secret != "" { </ins> _, err = rbd.plugin.execCommand("rbd", <del> []string{"map", rbd.image, "--pool", rbd.pool, "--id", rbd.id, "-m", mon, "--key=" + rbd.secret}) </del> <ins> []string{"map", rbd.Image, "--pool", rbd.Pool, "--id", rbd.Id, "-m", mon, "--key=" + rbd.Secret}) </ins> } else { _, err = rbd.plugin.execCommand("rbd", <del> []string{"map", rbd.image, "--pool", rbd.pool, "--id", rbd.id, "-m", mon, "-k", rbd.keyring}) </del> <ins> []string{"map", rbd.Image, "--pool", rbd.Pool, "--id", rbd.Id, "-m", mon, "-k", rbd.Keyring}) </ins> } if err == nil { break

etcd cluster backing apiserver died spontaneously, after restart apiserver pulled up the pods on different machines. Node storage was backed by ext4 over rbd, which became mounted on several nodes at once. That could have possibly led to data corruption. I guess that the proper way to mount rbd volume is to take a lock on it first, which will at least fail if it's locked somewhere else, requiring manual intervention, but not destroying data. The lock should be optional, in case a proper cluster fs is mounted over the rbd. In this case there was clearly no need for accessing the data from several pods so ext4 is deemed a proper fs.
, Vladimir Pouzanov < :
want to get some detail. How is your pod-to-rbd mapping like? Since rbd image is a block device, it can be used by one pod at a time. I am not sure if locking is a proper way to go - if the pod holds a lock but dies in the middle of mounting, the rbd image (or any other volume type) is then excluded from access. In such case, a high level coordination at apiserver is a better place to resolve this problem.
Here's my rc: I guess it's better to have a lock to be cleaned up manually rather than to have non-cluster fs to be mounted in several locations at once?
thanks, this pod looks good to me. what would cause multiple nodes run the same pod?
I don't know the exact reason of why kubelet didn't unmount the old fs, but after apiserver and scheduler were restarted pod got allocated on different node.
It is a scary thing to see etcd down, it is meant to be reliable...
Unfortunately I don't have long-term logging in test cluster so I have no idea why it actually died. but using kubedns as primary node resolver was clearly a bad idea, once it died, nodes stuck with no resolver at all.
your thought on HA?
Wrong Tim St. Clair?
- you HAVE to be ready to handle network partitions - the old node might still be alive and even using the volume, but the apiserver can't see the node, so it thinks it deletes the pod and starts it elsewhere. If RBD corrupts itslef in this case, it is not useful and we should strip it from the codebase.

, Huamin Chen wrote:
that makes sense. rbd doesn't corrupt itself. This is inconsistency happens when you have two pods mounting the same rbd image. How does GCE PD handle/prevent this case?
It doesn't allow more than one read-write attachment. Multiple read-only attachments is OK. Better to fail than to allow corruption. , Huamin Chen wrote:
That's neat. I have a lock fencing for rbd, testing now.


Question: cmd.Flags().String("docker-password", "", i18n.T("Password for Docker registry authentication")) cmd.MarkFlagRequired("docker-password") cmd.Flags().String("docker-email", "", i18n.T("Email for Docker registry")) <del> cmd.MarkFlagRequired("docker-email") </del> cmd.Flags().String("docker-server", "https://index.docker.io/v1/", i18n.T("Server location for Docker registry")) cmdutil.AddInclude3rdPartyFlags(cmd) return cmd

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): Is this a BUG REPORT or FEATURE REQUEST? (choose one): <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Environment: Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: What happened: When I create a docker registry secret, the is also a MUST, I think we should make this parameter as optional as there is no need always put when creating a docker registry secret. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know:
Agree it should not be required
Did this change get reverted? I do not see as optional on stable.
I feel that this code should be changed because "--docker-email flag" is no longer required.
error: flag docker-email is required. On kubenetes 1.10.3, the above error message still exists
The field is still required. This is very frustrating as does not require an email but Kubernetes does? Is there a good reason for this?
Why is this closed? As far as I can see it is still required?


Question: // Accumulate NAT chains to keep. activeNATChains := map[utiliptables.Chain]bool{} // use a map as a set <del> // Accumulate new local ports that we have opened. newLocalPorts := map[localPort]closeable{} </del> <ins> // Accumulate the set of local ports that we will be holding open once this update is complete replacementPortsMap := map[localPort]closeable{} </ins> // Build rules for each service. for svcName, svcInfo := range proxier.serviceMap {

The symptom is the nodeport service is accessible thru hostip:hostport. But the port on host is not occupied. This is exposed by e2e on kubenet. I noticed in the kube- Kube-proxy did open the port. But later, kube-proxy closed the ports upon iptables-restore failure. It never reopen the ports. Hence causing the e2e failure. The code section in question: Not sure why failed. Error messge is not helpful since line 139 is ref:
Also of interest: did it exit with 1? This should be clear from the iptables commands it executed before the commit, which are hopefully in the logs (can you post them). If it's something like a non-exist chain, we should fix that. we retry on exit 1? Handwaving without looking at code: I'd expect something to get requeued, and to reinvoke syncProxyRules once again, in a time frame << test timeout.
Yeah, what was line 139 in the restore output? , Prashanth B wrote:
is No retry though. It is going to catch up in the next sync. So I think no retry for is needed.
FYI Jenkins job:
That iptables restore is 139 lines long, so it's unhappy about the last line? can you hunt ths? , Minhan Xia wrote:
The next sync is whenver the next service/endpoint changes right? Errors during the sync should cause an immediate retry of sync, preferably with backoff. If a service or endpoint never changes we should still recover. Yes we should hunt down the bad line, but it's not the last one right? COMMIT will fail if eg: Even though it's obviously not commit but the tcp/upd thing.
iptables resyncs every 30s, not ideal but in this case I don't expect that to fix the error I meant that we need to root cause it :) , Prashanth B wrote:
But in this case, iptables-restore is going to complain . Not line 2
Hmm
I manually restored the 139 lines above it worked fine - no errors. Even more curious. , Prashanth B wrote:
I also tried to execute the same sequence of iptables-restore. It worked. No errors.

Was this a one-time thing? Were there any other changes in your git repo? , Minhan Xia wrote:
This happened in kubenet suite. Only difference is . I looked at a few runs where e2e failed. This happened 3-4 times over the whole test, but not consecutively. In all-passed runs I checked, this never happened (Maybe I am lucky).
Of , Minhan Xia wrote:


Question: } for _, addr := range addrs { <del> if addr.IPNet.String() != keep { glog.V(5).Infof("Removing old address %s from %s", addr.IPNet.String(), BridgeName) </del> <ins> if !utilnet.IPNetEqual(addr.IPNet, keep) { glog.V(2).Infof("Removing old address %s from %s", addr.IPNet.String(), BridgeName) </ins> netlink.AddrDel(bridge, &addr) } }

This is at least partially responsible for - some tests can't run when dns is down. Logs from just before the crash: It got killed because Kubelet decided to kill it (readiness probe failed): It started happening quite recently - I'm trying to locate a build which caused this. Repro: run "Kubelet should not restart containers across restart" test - it caused this behavior for me multiple times today. cc - I applied the P0/v1.3 label/milestone, as I think it should be a blocker. Feel free to decrease the priority if you disagree
gce-serial 1511 seems to be the first one with this problem.
Or 1510 - hard to tell, as it was a broken run. There's no PR around that looks too suspicious though:/
I locally reverted and it seems that DNS is no longer failing. Sadly I have to run to catch a train, but maybe someone can pick it up here ?
Taking a look now. The PR should be unrelated...I am checking to see if the Federation DNS changes went in yet. Add nethealth container to prepull manifest
I search the restart counts for jobs from 1510-latest and gce-serial Job 1540 is the first one to show a non-zero kubedns restart count. If there is a commit responsible, it would be between 1539 and 1540 (June 21 7 AM -11 AM) 1539 end of run 1540 end of run Latest
The kubedns:1.3 container hasn't been updated since (Sat, 04 Jun 2016 03:29:37.847 GMT). GET
cc The kubedns container code is from June 04th. Dawn suggested that its networking issues that are causing kubedns container to fail its readiness check. I will need to add logic to print kubedns container logs during e2e tests, I am trying to search for any logs that might help, otherwise I will have to debug the next run live.
As reported, a simple kubelet restart triggers this issue - it looks like 10.0.0.1 is unreachable from the kubedns container.

The kubedns container blocks indefinitely waiting for a connection to the kubernetes service at 10.0.0.1 before setting up the healthz handlers - the readiness probe will fail until kubedns successfully connects to the API server.
- you can try reverting and running it again - it worked for me for some reason (it might be by accident though)
I think the root cause is that cbr0 doesn't have an IP address assigned after kubelet restart - all the pods running behind cbr0 are isolated from the network. Not only kube-dns, but after my kubelet restart, the other addon pods are also stuck in a crash loop.
This sounds like a blocker even more now.
Looks like a regression. I am investigating.
Found the root cause. There is a subnet ip mismatch. vs . Hence kubenet thinks they are different and clears the address. Will submit a fix soon.
i found coredns status is CrashLoopBackOff and i check the logs : i got tips from so i restart my docker , i check the logs , there is no error in logs again


Question: }}, }}, RestartPolicy: v1.RestartPolicyNever, <del> }}, metav1.CreateOptions{}) </del> <ins> }} _, err := f.ClientSet.CoreV1().Pods(ns).Create(context.TODO(), pod, metav1.CreateOptions{}) </ins> framework.ExpectNoError(err, "failed to create pod") <del> err = wait.PollImmediate(podRetryPeriod, podRetryTimeout, checkPodStatus(f, "e2e-test=proxy-endpoints")) framework.ExpectNoError(err, "Pod didn't start within time out period") </del> <ins> framework.ExpectNoError(e2epod.WaitForPodRunningInNamespace(f.ClientSet, pod), "Pod didn't start within time out period") </ins> framework.Logf("Creating service...") _, err = f.ClientSet.CoreV1().Services(ns).Create(context.TODO(), &v1.Service{

pull-kubernetes-e2e-kind-ipv6 - Not sure. Time out on waiting Pod to be started. No response /sig network
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: "//vendor:k8s.io/apimachinery/pkg/runtime/schema", "//vendor:k8s.io/apimachinery/pkg/types", "//vendor:k8s.io/apimachinery/pkg/util/intstr", <ins> "//vendor:k8s.io/apimachinery/pkg/util/sets", </ins> "//vendor:k8s.io/apimachinery/pkg/util/uuid", "//vendor:k8s.io/apimachinery/pkg/watch", "//vendor:k8s.io/client-go/pkg/api",

The test is failing in . https://k8s- https://k8s- https://k8s- The reason is that the test expects that all pause containers () should have . However, in many tests, we are starting test containers with . Those pause containers have . Ideally, containers started in previous test should be cleaned up, so in serial suite it should not affect later test. However, it is not always true when CRI is enabled. For example, in the oom message, we could see: There are 2 problems here: Why are containers not properly cleaned up when CRI enabled? I can't even find which test leaked pause container from the log, because there are many test using pause image, and some of them start a lot of containers, such as density test, restart test. The oom score adj test should not rely on process name . However, I can't find an alternative for now, because we don't report pause container status in our api, we don't even know the container id to inspect with docker or CRI. /cc
This test is still flaking in cri-serial. Some examples: https://k8s- https://k8s- Need to check the logs to see if the symptoms are the same.
It could be that with CRI, it takes kubelet longer to clean up. Or, even worse, that kubelet is not cleaning up properly. We need more information on this on failure. Dumping pods on the node Dumping containers () on the node. Accessing docker directly is not ideal, but it's okay because the test itself only applies to docker. Alternatively, we can get the current running containers from kubelet's /runningPods endpoint. The downside is that this may lag behind the actual state a little bit (seconds). We should also make the test suite run more frequently. It runs once a day, which makes the debugging round-trip overhead too high.
I can reproduce this process leak by running the "Network should recover from ip leaks" on my own node, without CRI enabled. The image used in the test was , and the docker version 1.11.2. The test creates 100 pods, then restarts the docker service by 6 times, with 20s interval in between.

After the test ends, there could be multiple processes running left on the node. Those processes were reparented to pid 1, and showed no containers running. /cc
On the side note, the test should not be blindly checking all processes. It should, instead, take a snapshot of the system before the test and only check the new processes created by the tests.
I filed a new issue for the docker bug since it's not unique to CRI and this test is catching it by accident: Leaving this issue to fix the test by not blindly checking all processes:
The docker bug is not a regression, and is tracked in . The problem of the test is not necessarily to be fixed in v1.6, at least it's not release blocker. :)


Question: "k8s.io/kubernetes/pkg/kubelet/util" ) <ins> func init() { utilruntime.Must(metricsfeatures.AddFeatureGates(utilfeature.DefaultMutableFeatureGate)) } </ins> const ( metricsPath         = "/metrics" cadvisorMetricsPath = "/metrics/cadvisor"

As per, , wire up metrics/slis for kubelet
/assign
/sig instrumentation
/sig node /triage accepted


Question: } func TestWriteEventError(t *testing.T) { <del> ref := &api.ObjectReference{ Kind:       "Pod", Name:       "foo", Namespace:  "baz", UID:        "bar", APIVersion: "version", } </del> type entry struct { timesToSendError int <del> attemptsMade     int </del> attemptsWanted   int err              error }

Have not reproduced locally even after a few tens of runs.
which version of Go are you using locally? This probably only repros on 1.5.1.
This was from Jenkins runs.
This is caused because an attempts counter is being read/written to without synchronization. I think this is caused by the event creation order not being guaranteed. I am looking a little deeper to track this down.
Got a fix with 130k+ runs and no failures. Will send out PR shortly.


Question: DeleteManagedDisk(diskURI string) error // Attaches the disk to the host machine. <del> AttachDisk(isManagedDisk bool, diskName, diskUri string, nodeName types.NodeName, cachingMode compute.CachingTypes) error </del> <ins> AttachDisk(isManagedDisk bool, diskName, diskUri string, nodeName types.NodeName, cachingMode compute.CachingTypes) (int32, error) </ins> // Detaches the disk, identified by disk name or uri, from the host machine. DetachDisk(diskName, diskUri string, nodeName types.NodeName) error

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /sig azure /assign


Question: diskURI := fmt.Sprintf("/subscriptions/%s/resourceGroups/%s/providers/Microsoft.Compute/disks/disk-name", c.SubscriptionID, c.ResourceGroup) <del> err := common.AttachDisk(true, "", diskURI, "node1", compute.CachingTypesReadOnly) </del> <ins> _, err := common.AttachDisk(true, "", diskURI, "node1", compute.CachingTypesReadOnly) </ins> if err != nil { fmt.Printf("TestAttachDisk return expected error: %v", err) } else {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /sig azure /assign


Question: # # $1 - server architecture kube::build::get_docker_wrapped_binaries() { <del> debian_iptables_version=v10.1 </del> <ins> debian_iptables_version=v10.2 </ins> ### If you change any of these lists, please also update DOCKERIZED_BINARIES ### in build/BUILD. And kube::golang::server_image_targets case $1 in

What happened: Kube-proxy ipvs mode CI failed http://k8s- But actually all test are passed in my local-up environment with the newest code. According to ci log, those pr might be related: seems like the most probable, Anyway, looking for help.
Friendly ping , Please take a look, looking forward for your advise.
thanks for the report! /sig network /sig gcp /kind failing-test
/area ipvs
Update: Seems like staging- image do have some problem, command will return err. Not likely to be ipset issue.
Suggestions?
The version is exactly the same in both versions of the image (v10 and v10.1). Where did you find that error message?
List of package differences (given by ) between v10 and v10.1:
Thanks, the error was found in ipvs ci environment kube-proxy log. And I also get this error in my local environment with
Maybe we got some kind of incompatible issue with ipset specific version.
It seems like the removal of is what's causing this failure, which appears to be due to depending on instead of . I'm not sure what caused that to switch.
ok, I think I have an idea of what happened here: was originally built against . At some point was built against what was then , i.e. pulling in the packages from , which included (and ). Later (probably due to security issues), we rebuilt , which by this point was now based on . In this release, they apparently switched to installing by default, which did not include a dependency on . We apparently did not rebuild , though I'm not sure why not. As a result, it was still based on the older debian release. By the time we built , had been released, and so was similarly based on and was thus missing . tl;dr: need to add a dependency on in apparently, and should build/push a (or maybe ). should probably be more careful with these "simple rebases", since they can unexpectedly change what packages are installed. Some automation to do would help here. (cc do you think this is a kubernetes 1.12-blocking issue?
/assign
Thanks a lot. Though I'm not really understand, but glad we found the root cause.

And I think this gonna block 1.12 release, ipvs feature already GA in 1.11, we need make sure this feature can work normally.
for thoughts.
Do you think this is a kubernetes 1.12-blocking issue?
If this causing core IPVS functionality to be broken, which it looks like it is, then I think this should block the 1.12 release Sent from my iPhone
/milestone v1.12 Will treat as release blocking now until we hear otherwise from SIG-Network and SIG-GCP
fix PR for the s is ; we'll need one more PR after that's merged to update the Makefiles to use the new images.
First PR has merged.
Thanks. Working on promoting those images to , though that probably won't happen till Tuesday PT.
... or maybe sooner than that, because is an overachiever.
Second PR:
You're the one up working on promoting images at 9pm :-) I'm just trying to set my dev environment up again and saw this going on.
I can't keep up with the 3 of you!!! - -- An old man
The CI working well again hours ago, thanks all of you, really appreciate it. : )
Awesome. We might want to get better monitoring on this job if IPVS is GA, since it took several weeks from the original PR merged before this issue was opened.
Anyone willing to monitor can PR email alerts per config here: Please consider doing so! It's ~hard~ impossible to track all of testgrid, we really need SIGs to step up on this!
Definitely make sense, I will raise a pr to do this work.
Thank you! Lion-Wei <:


Question: // more notes for this issue: // same issue in moby: https://github.com/moby/moby/issues/38935#issuecomment-744638345 // solution in hcsshim: https://github.com/microsoft/hcsshim/blob/master/internal/processorinfo/processor_count.go <del> func processorCount() int { </del> <ins> func ProcessorCount() int { </ins> if amount := getActiveProcessorCount(allProcessorGroups); amount != 0 { return int(amount) }

When submitting a windows pod using resource limits on a node with more than one processor group, the specified count is incorrect due to the usage of which does not take processor groups into account. For example: node with 96 cores cpu limit 94 returns This results in a CPU maximum of since the range is from [1, ]: Our initial goal for this was to limit a process to not use all available CPU and therefore render the node uncontrollable. A CPU maximum of instead of for that specific case. For the calculation, it should use the CPU count for all available processor groups instead of just the current one. Launch a node with a lot of CPUs, for example and schedule the following pod on it: Then SSH into the node and inspect the running container using . The value will be set to . (which just uses ) is able to correctly find out the number of CPUs available to a system. Using for calculation or incorporating that code might solve this issue. <details</details<detailsAzure in-tree </details<details</details<details</details<detailscontainerd://1.6.10 </details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig windows
There is already some code which uses GetActiveProcessorCount:
/assign
/triage accepted


Question: udpTimeoutMilliseconds: 123ms ` <del> expected := &componentconfig.KubeProxyConfiguration{ BindAddress: "9.8.7.6", ClientConnection: componentconfig.ClientConnectionConfiguration{ AcceptContentTypes: "abc", Burst:              100, ContentType:        "content-type", KubeConfigFile:     "/path/to/kubeconfig", QPS:                7, }, ClusterCIDR:      "1.2.3.0/24", ConfigSyncPeriod: metav1.Duration{Duration: 15 * time.Second}, Conntrack: componentconfig.KubeProxyConntrackConfiguration{ Max:                   4, MaxPerCore:            2, Min:                   1, TCPCloseWaitTimeout:   metav1.Duration{Duration: 10 * time.Second}, TCPEstablishedTimeout: metav1.Duration{Duration: 20 * time.Second}, </del> <ins> testCases := []struct { name               string bindAddress        string clusterCIDR        string healthzBindAddress string metricsBindAddress string }{ { name:               "IPv4 config", bindAddress:        "9.8.7.6", clusterCIDR:        "1.2.3.0/24", healthzBindAddress: "1.2.3.4:12345", metricsBindAddress: "2.3.4.5:23456", </ins> }, <del> FeatureGates:       "all", HealthzBindAddress: "1.2.3.4:12345", HostnameOverride:   "foo", IPTables: componentconfig.KubeProxyIPTablesConfiguration{ MasqueradeAll: true, MasqueradeBit: util.Int32Ptr(17), MinSyncPeriod: metav1.Duration{Duration: 10 * time.Second}, SyncPeriod:    metav1.Duration{Duration: 60 * time.Second}, </del> <ins> { name:               "IPv6 config", bindAddress:        "2001:db8::1", clusterCIDR:        "fd00:1::0/64", healthzBindAddress: "[fd00:1::5]:12345", metricsBindAddress: "[fd00:2::5]:23456", </ins> }, <del> MetricsBindAddress: "2.3.4.5:23456", Mode:               "iptables", OOMScoreAdj:        util.Int32Ptr(17), PortRange:          "2-7", ResourceContainer:  "/foo", UDPIdleTimeout:     metav1.Duration{Duration: 123 * time.Millisecond}, </del> } <del> options, err := NewOptions() assert.NoError(t, err) </del> <ins> for _, tc := range testCases { expected := &componentconfig.KubeProxyConfiguration{ BindAddress: tc.bindAddress, ClientConnection: componentconfig.ClientConnectionConfiguration{ AcceptContentTypes: "abc", Burst:              100, ContentType:        "content-type", KubeConfigFile:     "/path/to/kubeconfig", QPS:                7, }, ClusterCIDR:      tc.clusterCIDR, ConfigSyncPeriod: metav1.Duration{Duration: 15 * time.Second}, Conntrack: componentconfig.KubeProxyConntrackConfiguration{ Max:                   4, MaxPerCore:            2, Min:                   1, TCPCloseWaitTimeout:   metav1.Duration{Duration: 10 * time.Second}, TCPEstablishedTimeout: metav1.Duration{Duration: 20 * time.Second}, }, FeatureGates:       "all", HealthzBindAddress: tc.healthzBindAddress, HostnameOverride:   "foo", IPTables: componentconfig.KubeProxyIPTablesConfiguration{ MasqueradeAll: true, MasqueradeBit: util.Int32Ptr(17), MinSyncPeriod: metav1.Duration{Duration: 10 * time.Second}, SyncPeriod:    metav1.Duration{Duration: 60 * time.Second}, }, MetricsBindAddress: tc.metricsBindAddress, Mode:               "iptables", OOMScoreAdj:        util.Int32Ptr(17), PortRange:          "2-7", ResourceContainer:  "/foo", UDPIdleTimeout:     metav1.Duration{Duration: 123 * time.Millisecond}, } options, err := NewOptions() assert.NoError(t, err, "unexpected error for %s: %v", tc.name, err) yaml := fmt.Sprintf( yamlTemplate, tc.bindAddress, tc.clusterCIDR, tc.healthzBindAddress, tc.metricsBindAddress) config, err := options.loadConfig([]byte(yaml)) assert.NoError(t, err, "unexpected error for %s: %v", tc.name, err) if !reflect.DeepEqual(expected, config) { t.Fatalf("unexpected config for %s test, diff = %s", tc.name, diff.ObjectDiff(config, expected)) } } } </ins> <del> config, err := options.loadConfig([]byte(yaml)) assert.NoError(t, err) if !reflect.DeepEqual(expected, config) { t.Fatalf("unexpected config, diff = %s", diff.ObjectDiff(config, expected)) </del> <ins> // TestLoadConfigFailures tests failure modes for loadConfig() func TestLoadConfigFailures(t *testing.T) { testCases := []struct { name   string config string expErr string }{ { name:   "Decode error test", config: "Twas bryllyg, and ye slythy toves", expErr: "could not find expected ':'", }, { name:   "Bad config type test", config: "kind: KubeSchedulerConfiguration", expErr: "unexpected config type", }, } version := "apiVersion: componentconfig/v1alpha1" for _, tc := range testCases { options, _ := NewOptions() config := fmt.Sprintf("%sn%s", version, tc.config) _, err := options.loadConfig([]byte(config)) if assert.Error(t, err, tc.name) { assert.Contains(t, err.Error(), tc.expErr, tc.name) } </ins> } }

Is this a request for help? No. What keywords did you search in Kubernetes issues before filing this one? is:issue is:open ipv6 Is this a BUG REPORT or FEATURE REQUEST? (choose one): FEATURE REQUEST Kubernetes version (use ): N/A, these are unit test changes in latest master Environment: Cloud provider or hardware configuration: Kubernetes container dev env on Ubuntu 16.04.1 OS (e.g. from /etc/os-release): Ubuntu 16.04.1 Kernel (e.g. ): 4.4.0-78-generic -Ubuntu x8664 GNU/Linux Install tools: Others: What happened: Upon inspection, no IPv6 test cases found in kubernetes/cmd/kube- What you expected to happen: Expect to see some IPv6 test cases exercised for kube-proxy server code (kubernetes/cmd/kube-). Also need some code coverage for negative conditions in How to reproduce it (as minimally and precisely as possible): Run unit tests in kubernetes/cmd/kube- Anything else we need to know:
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
/sig networking
/area ipv6
/sig network


Question: LEAVE_UP=${2:-0} TEAR_DOWN=${3:-0} <del> HAVE_JQ=$(which jq) if [[ -z ${HAVE_JQ} ]]; then echo "Please install jq, e.g.: 'sudo apt-get install jq' or, " echo "'sudo yum install jq' or, " echo "if you're on a mac with homebrew, 'brew install jq'." exit 1 fi </del> # Exit on error set -e

We have an e2e test that runs the guestbook example. We should also have one that runs through the other example.


Question: // decisions made by Kubernetes. This is typically needed for resources not directly // managed by Kubernetes. type SchedulerExtender interface { <ins> // Name returns a unique name that identifies the extender. Name() string </ins> // Filter based on extender-implemented predicate functions. The filtered list is // expected to be a subset of the supplied list. failedNodesMap optionally contains // the list of failed nodes and failure reasons.

<!-- Please only use this template for submitting enhancement requests --What would you like to be : In pkg/scheduler/core/generic_scheduler# , add some debug info: each scheduler extender's score and its name for each pod, e.g. And we need add a function(e.g. ) in interface to get extender's name or similar identity. For , it might be enough to return its url . Why is this needed: If there are two or more scheduler extenders configured for scheduler(and we have now), it will be very useful to debug why kubelet nodes get so high/low score. <!-- DO NOT EDIT BELOW THIS LINE --/kind feature
/assign
Reiterating the mentions to trigger a notification: <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Why don't you send a PR instead?
Thanks. PR is sent. PTAL.


Question: COMMIT # Completed on Thu Jan 19 11:38:09 2017`, iptablesSaveCmd+version) <ins> stderrOutput := "#STDERR OUTPUT" // SaveInto() should should NOT capture stderr into the buffer </ins> fcmd := fakeexec.FakeCmd{ CombinedOutputScript: []fakeexec.FakeCombinedOutputAction{ // iptables version check

On some systems, iptables-save outputs the following warning to stderr: Note that the error message includes a newline. Because kube-proxy captures both stdout and stderr when calling , the newline might cause rules to be garbled. For example, the output might contain the following line: The parser treats this as a chain line, and panics, as predicted in
/sig network
FYI
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/assign
Just FYI here's another example of garbled output:
Anyway, fix is in Please review.
/assign
/remove-triage unresolved


Question: <ins> /* Copyright 2022 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package e2enode import ( "fmt" "time" "github.com/onsi/ginkgo/v2" "github.com/onsi/gomega" "github.com/onsi/gomega/gstruct" "github.com/onsi/gomega/types" v1 "k8s.io/api/core/v1" "k8s.io/apimachinery/pkg/api/resource" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" kubeletconfig "k8s.io/kubernetes/pkg/kubelet/apis/config" "k8s.io/kubernetes/pkg/kubelet/cm/cpumanager" "k8s.io/kubernetes/pkg/kubelet/cm/cpuset" "k8s.io/kubernetes/test/e2e/framework" e2emetrics "k8s.io/kubernetes/test/e2e/framework/metrics" e2epod "k8s.io/kubernetes/test/e2e/framework/pod" e2eskipper "k8s.io/kubernetes/test/e2e/framework/skipper" admissionapi "k8s.io/pod-security-admission/api" ) var _ = SIGDescribe("CPU Manager Metrics [Serial][Feature:CPUManager]", func() { f := framework.NewDefaultFramework("cpumanager-metrics") f.NamespacePodSecurityEnforceLevel = admissionapi.LevelPrivileged ginkgo.Context("when querying /metrics", func() { var oldCfg *kubeletconfig.KubeletConfiguration var testPod *v1.Pod var smtLevel int ginkgo.BeforeEach(func() { var err error if oldCfg == nil { oldCfg, err = getCurrentKubeletConfig() framework.ExpectNoError(err) } fullCPUsOnlyOpt := fmt.Sprintf("option=%s", cpumanager.FullPCPUsOnlyOption) _, cpuAlloc, _ := getLocalNodeCPUDetails(f) smtLevel = getSMTLevel() // strict SMT alignment is trivially verified and granted on non-SMT systems if smtLevel < 2 { e2eskipper.Skipf("Skipping CPU Manager %s tests since SMT disabled", fullCPUsOnlyOpt) } // our tests want to allocate up to a full core, so we need at last 2*2=4 virtual cpus if cpuAlloc < int64(smtLevel*2) { e2eskipper.Skipf("Skipping CPU Manager %s tests since the CPU capacity < 4", fullCPUsOnlyOpt) } framework.Logf("SMT level %d", smtLevel) // TODO: we assume the first available CPUID is 0, which is pretty fair, but we should probably // check what we do have in the node. cpuPolicyOptions := map[string]string{ cpumanager.FullPCPUsOnlyOption: "true", } newCfg := configureCPUManagerInKubelet(oldCfg, &cpuManagerKubeletArguments{ policyName:              string(cpumanager.PolicyStatic), enableCPUManager:        true, reservedSystemCPUs:      cpuset.NewCPUSet(0), enableCPUManagerOptions: true, options:                 cpuPolicyOptions, }, ) updateKubeletConfig(f, newCfg, true) }) ginkgo.AfterEach(func() { if testPod != nil { deletePodSyncByName(f, testPod.Name) } updateKubeletConfig(f, oldCfg, true) }) ginkgo.It("should report zero pinning counters after a fresh restart", func() { // we updated the kubelet config in BeforeEach, so we can assume we start fresh. // being [Serial], we can also assume noone else but us is running pods. ginkgo.By("Checking the cpumanager metrics right after the kubelet restart, with no pods running") matchResourceMetrics := gstruct.MatchKeys(gstruct.IgnoreExtras, gstruct.Keys{ "kubelet_cpu_manager_pinning_requests_total": gstruct.MatchAllElements(nodeID, gstruct.Elements{ "": timelessSample(0), }), "kubelet_cpu_manager_pinning_errors_total": gstruct.MatchAllElements(nodeID, gstruct.Elements{ "": timelessSample(0), }), }) ginkgo.By("Giving the Kubelet time to start up and produce metrics") gomega.Eventually(getCPUManagerMetrics, 1*time.Minute, 15*time.Second).Should(matchResourceMetrics) ginkgo.By("Ensuring the metrics match the expectations a few more times") gomega.Consistently(getCPUManagerMetrics, 1*time.Minute, 15*time.Second).Should(matchResourceMetrics) }) ginkgo.It("should report pinning failures when the cpumanager allocation is known to fail", func() { ginkgo.By("Creating the test pod which will be rejected for SMTAlignmentError") testPod = e2epod.NewPodClient(f).Create(makeGuaranteedCPUExclusiveSleeperPod("smt-align-err", 1)) // we updated the kubelet config in BeforeEach, so we can assume we start fresh. // being [Serial], we can also assume noone else but us is running pods. ginkgo.By("Checking the cpumanager metrics right after the kubelet restart, with pod failed to admit") matchResourceMetrics := gstruct.MatchKeys(gstruct.IgnoreExtras, gstruct.Keys{ "kubelet_cpu_manager_pinning_requests_total": gstruct.MatchAllElements(nodeID, gstruct.Elements{ "": timelessSample(1), }), "kubelet_cpu_manager_pinning_errors_total": gstruct.MatchAllElements(nodeID, gstruct.Elements{ "": timelessSample(1), }), }) ginkgo.By("Giving the Kubelet time to start up and produce metrics") gomega.Eventually(getCPUManagerMetrics, 1*time.Minute, 15*time.Second).Should(matchResourceMetrics) ginkgo.By("Ensuring the metrics match the expectations a few more times") gomega.Consistently(getCPUManagerMetrics, 1*time.Minute, 15*time.Second).Should(matchResourceMetrics) }) ginkgo.It("should not report any pinning failures when the cpumanager allocation is expected to succeed", func() { ginkgo.By("Creating the test pod") testPod = e2epod.NewPodClient(f).Create(makeGuaranteedCPUExclusiveSleeperPod("smt-align-ok", smtLevel)) // we updated the kubelet config in BeforeEach, so we can assume we start fresh. // being [Serial], we can also assume noone else but us is running pods. ginkgo.By("Checking the cpumanager metrics right after the kubelet restart, with pod should be admitted") matchResourceMetrics := gstruct.MatchKeys(gstruct.IgnoreExtras, gstruct.Keys{ "kubelet_cpu_manager_pinning_requests_total": gstruct.MatchAllElements(nodeID, gstruct.Elements{ "": timelessSample(1), }), "kubelet_cpu_manager_pinning_errors_total": gstruct.MatchAllElements(nodeID, gstruct.Elements{ "": timelessSample(0), }), }) ginkgo.By("Giving the Kubelet time to start up and produce metrics") gomega.Eventually(getCPUManagerMetrics, 1*time.Minute, 15*time.Second).Should(matchResourceMetrics) ginkgo.By("Ensuring the metrics match the expectations a few more times") gomega.Consistently(getCPUManagerMetrics, 1*time.Minute, 15*time.Second).Should(matchResourceMetrics) }) }) }) func getCPUManagerMetrics() (e2emetrics.KubeletMetrics, error) { // we are running out of good names, so we need to be unnecessarily specific to avoid clashes ginkgo.By("getting CPU Manager metrics from the metrics API") return e2emetrics.GrabKubeletMetricsWithoutProxy(framework.TestContext.NodeName+":10255", "/metrics") } func makeGuaranteedCPUExclusiveSleeperPod(name string, cpus int) *v1.Pod { return &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name: name + "-pod", }, Spec: v1.PodSpec{ RestartPolicy: v1.RestartPolicyNever, Containers: []v1.Container{ { Name:  name + "-cnt", Image: busyboxImage, Resources: v1.ResourceRequirements{ Requests: v1.ResourceList{ v1.ResourceCPU:    resource.MustParse(fmt.Sprintf("%d", cpus)), v1.ResourceMemory: resource.MustParse("64Mi"), }, Limits: v1.ResourceList{ v1.ResourceCPU:    resource.MustParse(fmt.Sprintf("%d", cpus)), v1.ResourceMemory: resource.MustParse("64Mi"), }, }, Command: []string{"sh", "-c", "sleep", "1d"}, }, }, }, } } func timelessSample(value interface{}) types.GomegaMatcher { return gstruct.PointTo(gstruct.MatchAllFields(gstruct.Fields{ // We already check Metric when matching the Id "Metric":    gstruct.Ignore(), "Value":     gomega.BeNumerically("==", value), "Timestamp": gstruct.Ignore(), })) } </ins>

Add to enable observability of the cpumanager. This is required to but it's also useful in general. To the best of my knowledge, it's very hard to observe the behavior of cpumanager without peeking into the pod and/or use the podresources interface (kubelet = 1.24). But even for the latter, no well known components expose metrics consuming this information. I believe anyway kubelet some expose some basic metrics about cpumanager behaviour, while deferring more fine grained metrics to external components (NFD, node_exporter, is fine)
/sig node /assign
/triage accepted /priority important-soon (as this is important for graduation for CPU Manager to GA)


Question: } }() <ins> retry := NewRetryWithDeadline(r.MaxInternalErrorRetryDuration, time.Minute, apierrors.IsInternalError, r.clock) </ins> for { // give the stopCh a chance to stop the loop, even in case of continue statements further down on errors select {

When etcd is unable to elect leader for 3 election cycles, etcd returns "no leader" error through watches. This can be seen in api-server logs as . Also, when apiserver receives this error, it terminates all watchers: . In large clusters, this can destabilize master, as all these watchers need to be recreated almost at the same time, overwhelming api-server quite often. I propose adding new api-server flag that allows to specify timeout, during which we are retrying to reestablish apiserver <-etcd watches, without terminating all api-server watchers. If we are unable to reestablish watches within this specified timeout, we start terminating all watchers. Losing etcd leader for ~2-3s does not destabilize api-server. Trigger prolonged reelection of etcd leader. No response Observed in all kubernetes versions 1.19-1.24 N/A <details</details<details</details<details</details<details</details
/sig scalability /sig api-machinery /priority important-soon
/wg reliability
/assign
/triage accepted
indeed, this is a source of instability. Is this coming from ?
I believe it's coming from this change. As I mentioned, the idea to mitigate it is to add some parameter that will allow us to ignore this error coming from etcd for specific duration without terminating all watchers, for example 10s be default.
With this comment thread in mind: It would probably be cleaner to allow for simply setting the "timeout" as part of WithRequireLeader option (instead of using the constant of 3). But that would require etcd changes, including the API and we want to workaround that in much smaller timeframe. But a comment about how to make that cleaner with some additional context from here would help.