Question: glog.Warningf("Warning: Unmount skipped because path does not exist: %v", mountPath) return nil } <del> corruptedMnt := isCorruptedMnt(pathErr) </del> <ins> corruptedMnt := IsCorruptedMnt(pathErr) </ins> if pathErr != nil && !corruptedMnt { return fmt.Errorf("Error checking path: %v", pathErr) }

/kind bug What happened: Kubelet failed to unmount volumes if the backend ISCSI volume is not available. This also applies to other network attached volumes. What you expected to happen: Kubelet should be able to unmount the volumes. A manual unmount works even when the backend network volume is not available. How to reproduce it (as minimally and precisely as possible): Mount an ISCSI/network volume and disconnect/de-provision ISCSI target. Anything else we need to know?: XFS shuts down the filesystem due to I/O errors. Kubelet uses "stat" calls to figure out if it is a valid mount point or it is already unmounted. Stat calls fail with EIO and Kubelet never proceeds to unmount the volume. Environment: Kubernetes version (use ): Kubernetes 1.10.5 Cloud provider or hardware configuration: Bare metal OS (e.g. from /etc/os-release): Centos 7.4 Kernel (e.g. ): Linux appserv37 3.10.0-693.el7.x8664 SMP Tue Aug 22 21:09:27 UTC 2017 x8664 x8664 x8664 GNU/Linux Install tools: Manual bring up Others:
/sig storage
IsLikelyNotMountPoint is using stat call to figure the filesystem is mounted or not. Stat calls are failing with EIO when there are I/O errors on the backend and Kubelet never proceeds to unmount the filesystem. Also, isLikelyNotMountPoint is not looking for "isCorruptedMnt" errors. All other calls have it.
looks like the nfs plugin is calling . Can iscsi use that?
didn't include error "Input/output error" () We should add it. I make a fix, PTAL, Thank you!
Thanks. Trying out the change. Will keep you posted if we need any other changes.
Stock ISCSI driver is working fine with EIO change. But more changes are required for making it work with Flexvolume drivers. I have the change list. I can club your change in or let me know if you want to go ahead and push this fix. I will push another PR to fix flexvolume issues.
Please club my change, I will close my PR. Thanks!


Question: apiVersion: v1 kind: ReplicationController metadata: <del> name: kube-dns-v18 </del> <ins> name: kube-dns-v19 </ins> namespace: kube-system labels: k8s-app: kube-dns <del> version: v18 </del> <ins> version: v19 </ins> kubernetes.io/cluster-service: "true" spec: replicas: __PILLAR__DNS__REPLICAS__ selector: k8s-app: kube-dns <del> version: v18 </del> <ins> version: v19 </ins> template: metadata: labels: k8s-app: kube-dns <del> version: v18 </del> <ins> version: v19 </ins> kubernetes.io/cluster-service: "true" spec: containers:

Observed weird memory spikes for both the dns healthz and kube-dns pods in production. Kube-dns didn't oom because there's enough headroom, but healthz did. I can't explain the spikes yet. I tried a bunch of different repros: no , bad nslookup command, sharing the node with cpu hogs etc, but was unable to trigger it. The memory grows, but stays under 10Mi. As a short term mitigation, we stole some ram from kube-dns, because it was always comfortably within limits. Related b/
can be closed -- healthz has been removed
-- close issue?


Question: "github.com/onsi/gomega" appsv1 "k8s.io/api/apps/v1" <ins> autoscalingv1 "k8s.io/api/autoscaling/v1" </ins> v1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/fields"

[ ] two weeks soak start date : testgrid-link [ ] two weeks soak end date : [ ] test promotion pr : kubernetes/kubernetes#? According to this APIsnoop query, there are still some remaining StatefulsetScale endpoints that are untested. with this query, you can filter untested endpoints by their category and eligibility for conformance. e.g below shows a query to find all conformance eligible untested, stable, apps endpoints sql-mode select distinct endpoint, right(useragent,73) AS useragent from testing.auditevent -- where useragent ilike '%subresource%' where endpoint ilike '%AppsV1NamespacedStatefulSetScale%' and releasedate::BIGINT round(((EXTRACT(EPOCH FROM NOW()))::numeric)1000,0) - and useragent like 'e2e%' order by endpoint limit 30; example endpoint | useragent -----------------------------------------|--------------------------------------------------------------------------- patchAppsV1NamespacedStatefulSetScale | [StatefulSetBasic] should have a working scale subresource [Conformance] readAppsV1NamespacedStatefulSetScale | [StatefulSetBasic] should have a working scale subresource [Conformance] replaceAppsV1NamespacedStatefulSetScale | [StatefulSetBasic] should have a working scale subresource [Conformance] (3 rows) If a test with these calls gets merged, test coverage will go up by 1 point* This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: asw.addVolumeToReportAsAttached(volumeName, nodeName) } <ins> func (asw *actualStateOfWorld) RemoveNodeFromAttachUpdates(nodeName types.NodeName) error { asw.Lock() defer asw.Unlock() _, nodeToUpdateExists := asw.nodesToUpdateStatusFor[nodeName] if nodeToUpdateExists { delete(asw.nodesToUpdateStatusFor, nodeName) return nil } return fmt.Errorf("node %q does not exist in volumesToReportAsAttached list", nodeName) } </ins> func (asw *actualStateOfWorld) AddVolumeNode( uniqueName v1.UniqueVolumeName, volumeSpec *volume.Spec, nodeName types.NodeName, devicePath string) (v1.UniqueVolumeName, error) { asw.Lock()

kubernetes version: 1.5.3 platform: aws deploy tool: kops my master's /var/log/kube-controller- is receiving this type of log: lots of times per second, this node "ip-172-20-114-85.ec2.internal" does not exist in the cluster anymore and is not shown if I shouldn't it be removed from status_updater?
Any work around? Any way to purge those old nodes from the NodeInformer cache?
AFAIK, sig-node doesn't maintain the node controller. Based on the files, the only approver is
It has nothing to do with the NodeController: cc
This was happening in our clusters as well, only the leader was doing it and and killing the container so it restarted made it stop happening.
did you kill the container, which then gets restarted by the scheduler, and that fixed it?
correct
Works for me
Same controller-manager log here using k8s 1.5. I deleted one of the nodes which ran out of resources and became stuck, something I'm still trying to pinpoint the cause of but it appears to be related to volumes.
This happens on k8s 1.6.1 as well, that replaced old nodes won't get cleaned up, generating large error logs. On CoreOS installation, I did systemctl restart kube-controller-manager. That fixed it.
Ping
This is happening on k8s 1.4.12 as well, restarting kube-controller-manager does not seem to fix it for me. It is rotating through two nodes that no longer exist, so I am hoping the cache will purge itself at some point and they will stop?
Just following up that the old nodes still not cleaned up
will work on a fix for this.
Let's make sure the fix also gets ported back to all affected branches (1.6, 1.5, and 1.4). CC
I was able to reproduce the problem by force deleting a node containing a pod with some volume attached to it. Did anyone run into a different failure mode?
I think that's how the bug surfaced for me. When I force killed my node we had just started running a StatefulSet with an attached persistent volume, and I think there was one running on the very node I killed.
The NodeInformer cache is actually reporting the correct state.

The problem is that inside the node status updater, there is another data structure keeping track of nodes that require a status update (see ActualStateOfWorld.GetVolumesToReportAttached(), called inside UpdateNodeStatuses() in ). The entry for the deleted node is never removed from the data structure; rather, the updater is set to try to update the dead node again next time, which leads to the message being logged every 100ms. The solution is to remove the corresponding node entry once the node is deleted. A fix is on its way.
A fix for 1.5 is on the way (PR )
Will 1.4 be included as well?
Here you go, my apologies.
Thank you very much! :)
To summarize, the fix for this is in 1.7 branch Releases 1.7.0 and greater 1.6 branch Releases 1.6.5 and greater 1.5 branch Releases 1.5.8 and greater (1.5.8 has not yet been cut, CC for ETA of that release). 1.4 branch Releases 1.4.13 and greater (1.4.13 has not yet been cut, CC for ETA of that release).
There is an edge case that the above fix did not handle: see PR is fixing that


Question: if err := options.Validate(args); err != nil { t.Fatalf("unexpected error: %v", err) } <del> if err := options.RunAnnotate(); err != nil { </del> <ins> if err := options.RunAnnotate(f); err != nil { </ins> t.Fatalf("unexpected error: %v", err) } }

and work in this pattern: first issuing a "GET" to the API server to retrieve the latest object, then adding the label/annotation to the object, and finally issuing an "PUT" with the modified object. The request will fail if the object stored in the server has been updated between the "GET" and "PUT", even if the update has no conflict with the label/annotate request. In such cases, instead of returning an error, should retry without disturbing the end user. There are two ways to implement this: in kubectl: simply retry the label/annotate process several times if the server rejected the request because the object has been updated. The only exception is when the is set, then kubectl shouldn't retry. the strategicMergePatch instead of calling Update at client-side. PATCH is more robust because PATCH does the "get, modify, update" in the server-side, so there is less chance to have a conflict, and PATCH will retry at the server-side (see ). We need to make some adaptions though: kubectl annotate/label has the , when it's false, kubectl should retrieve the latest object to confirm there is no existing annotation/label of the same key, and then proceed with issuing the patch. if the is set, [UPDATE:] we just need to specify the resourVersion in the patch, then it will be treated as a precondition by the UpdateResource() handler in the API server. I plan to take the first approach as an immediate fix.
Immediate need is to fix the flaky e2e test AIUI.
e2e? do we have an issue out there? is an immediate fix to the flake in test-
If this is for 1.1, it needs to be P0 at this point. Should it be?
Fwiw I use annotations both from cmd line as an admin and from scripts, to experiment with loadbalancer settings. In the case of the former, I've never really had a collision, and in the case of the latter my scripts know how to retry like we retry all other updates.
Thank you When used in scripts, the annotate and label commands have collisions frequently, e.g. in the test-, see . IMO, we shouldn't require users to retry in their scripts if we know there is a simple way to avoid the collisions. Anyway, and are sent to fix this.


Question: "io/ioutil" "math/big" "net" <del> "path" </del> <ins> "path/filepath" </ins> "strings" "time"

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --Cert path was generated with wrong slashes What happened: Paths generated with path and not filepath do not work always work on windows. will in general work, however volume guid paths will not This issue is related to rancher/k3s
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: # $4: value for variable 'cpulimit' # $5: pod name, which should be either etcd or etcd-events function prepare-etcd-manifest { <del> local host_name=$(hostname) local etcd_cluster="" local cluster_state="new" for host in $(echo "${INITIAL_ETCD_CLUSTER}" | tr "," "n"); do etcd_host="etcd-${host}=http://${host}:$3" if [[ -n "${etcd_cluster}" ]]; then etcd_cluster+="," cluster_state="existing" fi etcd_cluster+="${etcd_host}" done </del> local -r temp_file="/tmp/$5" cp "${KUBE_HOME}/kube-manifests/kubernetes/gci-trusty/etcd.manifest" "${temp_file}" <del> remove-salt-config-comments "${temp_file}" </del> sed -i -e "s@{{ *suffix *}}@$1@g" "${temp_file}" sed -i -e "s@{{ *port *}}@$2@g" "${temp_file}" sed -i -e "s@{{ *server_port *}}@$3@g" "${temp_file}" sed -i -e "s@{{ *cpulimit *}}@"$4"@g" "${temp_file}" <del> sed -i -e "s@{{ *host_name *}}@$host_name@g" "${temp_file}" sed -i -e "s@{{ *etcd_cluster *}}@$etcd_cluster@g" "${temp_file}" sed -i -e "s@{{ *cluster_state *}}@$cluster_state@g" "${temp_file}" </del> # Replace the volume host path. sed -i -e "s@/mnt/master-pd/var/etcd@/mnt/disks/master-pd/var/etcd@g" "${temp_file}" mv "${temp_file}" /etc/kubernetes/manifests

test grid: Data as of June 3rd, 9:00 PM Eastern. Latest data can be found here: Additional Context:
/sig release
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
cc Do we want to keep this job and fix it, or do something different considering the debs/rpms efforts?
I think we should keep this job as long as we build packages for Google infra (those are using different specs then packages for OBS).
fixed it for debs: The corresponding RPM job is also faster now:


Question: "foobar": "bar", "baz":    "blah", }) <ins> } func TestMakePodStatus(t *testing.T) { status := makePodStatus(map[string]interface{}{}) if status != "Pending" { t.Errorf("Expected 'Pending', got '%s'", status) } status = makePodStatus(map[string]interface{}{ "State": map[string]interface{}{ "Running": false, }, }) if status != "Stopped" { t.Errorf("Expected 'Stopped', got '%s'", status) } status = makePodStatus(map[string]interface{}{ "State": map[string]interface{}{ "Running": true, }, }) if status != "Running" { t.Errorf("Expected 'Running', got '%s'", status) } </ins> }

ci-cgroupv1-containerd-node-arm64-al2023-e2e-ec2-eks E2eNode Suite: [It] [sig-node] Summary API [NodeConformance] when querying /stats/summary should report resource usage through the stats api { failed [FAILED] Timed out after 180.001s. Expected <string: Summary to match fields: { .Node.SystemContainers: missing expected element runtime } In [It] at: @ 07/26/23 05:48:05.248 } arm test since begining cadvisor on arm? /sig node
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc
/assign it may be ec2 related, not arm related. We need to fix arm tests on gce to validate (failing currently).
Should be fixed, but I'm watching and will re-open if not.
Test is passing now, good to leave closed.


Question: t.Errorf("unexpected output: %s", buf.String()) } } <ins> func TestResourceErrors(t *testing.T) { testCases := map[string]struct { args  []string flags map[string]string errFn func(error) bool }{ "no args": { args:  []string{}, errFn: func(err error) bool { return strings.Contains(err.Error(), "you must provide one or more resources") }, }, "resources but no selectors": { args: []string{"pods"}, errFn: func(err error) bool { return strings.Contains(err.Error(), "resource(s) were provided, but no name, label selector, or --all flag specified") }, }, "multiple resources but no selectors": { args: []string{"pods,deployments"}, errFn: func(err error) bool { return strings.Contains(err.Error(), "resource(s) were provided, but no name, label selector, or --all flag specified") }, }, } for k, testCase := range testCases { f, tf, _ := NewAPIFactory() tf.Printer = &testPrinter{} tf.Namespace = "test" tf.ClientConfig = &restclient.Config{ContentConfig: restclient.ContentConfig{GroupVersion: testapi.Default.GroupVersion()}} buf := bytes.NewBuffer([]byte{}) cmd := NewCmdDelete(f, buf) cmd.SetOutput(buf) for k, v := range testCase.flags { cmd.Flags().Set(k, v) } err := RunDelete(f, buf, cmd, testCase.args, &DeleteOptions{}) if !testCase.errFn(err) { t.Errorf("%s: unexpected error: %v", k, err) continue } if tf.Printer.(*testPrinter).Objects != nil { t.Errorf("unexpected print to default printer") } if buf.Len() > 0 { t.Errorf("buffer should be empty: %s", string(buf.Bytes())) } } } </ins>

When users try to label/annotate multiple resources, we should tell them to add . Current output: Expected output:
and too
Instead of "--all is false", say "multiple resources were provided, but no label selector or --all flag specified"
Description updated based on suggestion
I think "resource(s) were provided, but no name, label selector, or --all flag specified" is better. I will send a pull request for this as my first PR.


Question: name = objectMeta.GenerateName } } <ins> return name, kind, nil } // NewForbidden is a utility function to return a well-formatted admission control error response func NewForbidden(a Attributes, internalError error) error { // do not double wrap an error of same type if apierrors.IsForbidden(internalError) { return internalError } name, kind, err := extractKindName(a) if err != nil { return apierrors.NewInternalError(errs.NewAggregate([]error{internalError, err})) } </ins> return apierrors.NewForbidden(kind, name, internalError) } <ins> // NewNotFound is a utility function to return a well-formatted admission control error response func NewNotFound(a Attributes) error { name, kind, err := extractKindName(a) if err != nil { return apierrors.NewInternalError(err) } return apierrors.NewNotFound(kind, name) } </ins>

Currently the admission controller returns a forbidden (403) error if the namespace doesn't exist. We should return a 404 error instead.


Question: if fileProjection.FsUser == nil { continue } <del> if err := os.Chown(fullPath, int(*fileProjection.FsUser), -1); err != nil { </del> <ins> if err := w.chown(fullPath, int(*fileProjection.FsUser), -1); err != nil { </ins> klog.Errorf("%s: unable to change file %s with owner %v: %v", w.logContext, fullPath, int(*fileProjection.FsUser), err) return err }

Install a cluster with v1.29.4 with linux node as master and worker node as windows kubeadm upgrade apply v1.30.0 fails with --ignore-preflight-errors` Describe Pod events: It seems that previously the pod was Pending as well, but this was ignored, because the jobs was successfully deleted in defer and return value was overridden with nil. Upgrade health check job expected to be scheduled only on Linux nodes using node selectors, rather than on Windows nodes. See What happened? No response_ <detailskubeadm version (use kubeadm version): v1.30.0 Environment: Kubernetes version (use kubectl version): v1.30.0 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Ubuntu 22.04.2 LTS, Windows Server 2022 Datacenter Kernel (e.g. uname -a): 5.19.0-1025-aws Container networking plugin (CNI): calico </details<details</details<details</details<details</details<details</details<details</details
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
fixed in 1.30.1 /close
Closing this issue. <detailsIn response to : kubeadm version (use kubeadm version): v1.30.0 Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/transfer kubeadm
we are seeing this issue for 1.30.1 version too with upgrade health check pods getting scheduled on windows worker nodes.
looks like I was mistaken and this issue is different. but the health check uses the pause image, which is multi os, supposedly. the error is strange. can you show the full log output. we might have to apply a node selector on the job spec and backport to 1.31 another bug fix.

I'm not sure if we should add a node selector on job spec. the job uses the pause image to create a container. It doesn't have specific requirements for the node. After I look at the projected volume plugin, I'm confused why the above event occurs. On windows nodes, the function is empty. It doesn't do anything. On linux nodes, the function will set the owner of the volume. FYI: If a log file from the kubelet is provided, it will be helpful to understand the issue.
thanks for checking I also think there should not be an error. maybe we are hitting a core kubelet bug. is the kubelet on the windows node also 1.30?
please ignore the above comment, it's incorrect. I may find the root cause. It's a bug in kubelet when kubelet is running on Windows and the pod configures the SecurityContext with a RunAsUser.
/sig storage /sig windows
/assign I'm trying to fix the kubelet bug.
so this is not a kubeadm bug and we can move it back to the k/k repo?
/transfer kubernetes
/retitle kubelet on Windows fails if a pod has SecurityContext with RunAsUser.
Because of historical reasons, the and in is only designed for the Linux, not Windows. See for more details. adds a new field to the PodSpec, which can be used to identify the OS of the pod. It's an optional field and no default value. If unset, the above fields are not restricted by the api-server. In other words, the following job is allowed to be created. But it will fails to start on Windows nodes. And the root cause is that the is called by the . To fix this issue, the AtomicWriter can check the current OS of the node and skip the if the OS is Windows. I want to confirm with sig-storage and sig-windows if this is the correct way to fix this issue. I'm not sure if the failing job is expected or not. If it's expected, kubeadm has to change the job manifest to make it work on only Linux nodes. Why I ask this question is that I saw , which is used to set the ownership of the volume, if the os is Windows, it will become a no-op.

/cc
/cc
/triage accepted
slack discussion is happening here: but we have agreed that this is a kubelet problem. on the kubeadm side you can use the --ignore-preflight-errors=CreateJob option to skip the check. unclear when or if a kubelet fix will be and to what versions it will be available.
To fix this issue, the AtomicWriter can check the current OS of the node and skip the os.Chown if the OS is Windows. I want to confirm with sig-storage and sig-windows if this is the correct way to fix this issue. please see: do you want to upstream your carry?
thank you. I raise a PR and add as a co-author. I add an e2e test. do you know which job can test windows?
is the e2e test job for Windows


Question: return gv.WithKind(policy.Spec.ParamKind.Kind) } <del> func (c *TypeChecker) checkExpression(expression string, hasParams, hasAuthorizer bool, types typeOverwrite) (*cel.Issues, error) { env, err := buildEnv(hasParams, hasAuthorizer, types) if err != nil { return nil, err } // We cannot reuse an AST that is parsed by another env, so reparse it here. // Compile = Parse + Check, we especially want the results of Check. // // Paradoxically, we discard the type-checked result and let the admission // controller use the dynamic typed program. // This is a compromise that is defined in the KEP. We can revisit this // decision and expect a change with limited size. _, issues := env.Compile(expression) return issues, nil } </del> // typesToCheck extracts a list of GVKs that needs type checking from the policy // the result is sorted in the order of Group, Version, and Kind func (c *TypeChecker) typesToCheck(p *v1beta1.ValidatingAdmissionPolicy) []schema.GroupVersionKind {

When I load a policy definition like this The object is updated with the following status While this error is reported, the policy does seem to be fully functioning This is similar to the documented examples No error reported on a valid spec Create a a policy that uses variables Observe the status of the newly created policy No response <details</details<detailsn/a </details<details</details<detailskind </details<details</details<details</details
/sig api-machinery
/assign Oops. Good find. Thank you. /triage accepted


Question: Help: "Response size distribution in bytes for each group, version, verb, resource, subresource, scope and component.", // Use buckets ranging from 1000 bytes (1KB) to 10^9 bytes (1GB). Buckets:        compbasemetrics.ExponentialBuckets(1000, 10.0, 7), <del> StabilityLevel: compbasemetrics.ALPHA, </del> <ins> StabilityLevel: compbasemetrics.STABLE, </ins> }, []string{"verb", "group", "version", "resource", "subresource", "scope", "component"}, )

I would like to propose the graduation of the following apiserver metrics from alpha to stable: With the exception of the metric these metrics have been stable for 2 years. The metric has been stable for 12 months.
/sig api-machinery /sig instrumentation
/assign /triage accepted
I'm okay with promoting all of these except I am hesitant about since it is relatively new. Can we omit that one?
I agree with the metric was patched a year ago as part of , so it would be best to wait a bit more until the majority of Kubernetes users catch up to the patched version and start using the new version of the metric in production.
A couple of renaming suggestions, since it is important to get it right before graduation: apiserverresponsesizes =apiserverresponsesizebytes aggregatorunavailableapiservice =aggregatorunavailableapiservices One concern regarding . It was very recently changed but the change was later reverted by In the end, the metric wasn't really changed, but it might raise the concern that some users may still need additional information from this metric so I am on the fence with this one. Side implementation note that we will need to keep in mind if we graduate is that the graduation should only affect the webhook admission metric and not the other, so we will need to pass the stability level to
Thank you and for the feedback. To clarify then, are the decisions to not graduate and final?
I'm actually okay with , the namespace addition was recent but it was a bad idea and was reverted. It has otherwise been in use for quite some time, so I'd be comfortable with promoting it.
I'm willing to make a PR for this one
+1 it wasn't a hard no from my side, just wanted to raise some concerns in case anyone had an objection.
/assign /unassign
Thank you for the feedback, and To summarize, we are in agreement to graduate all but the metric. Please correct me if I'm wrong.
/cc
since all the three metrics that are created with this function are graduating to stable, do you think I should still pass the to the function or should I just set the metrics created by this functions as stable by default? update: Seems like is designed to parse whatever's in front of . So putting a variable there (that is passed to the function) will cause the script to fail. Please correct me if I'm wrong.

update: Even setting the metrics to be stable by default will cause the script to fail and you'll get an error saying for the following line: It appears that the name of the stable metrics should be given to that object as a string and using the output of a function ( in this case) will cause the script to fail.
The PR is ready to be reviewed:


Question: Containers: []v1.Container{ { Name:    "pv-recycler", <del> Image:   "busybox:1.27", </del> <ins> Image:   "k8s.gcr.io/debian-base:v2.0.0", </ins> Command: []string{"/bin/sh"}, Args:    []string{"-c", "test -e /scrub && rm -rf /scrub/..?* /scrub/.[!.]* /scrub/*  && test -z "$(ls -A /scrub)" || exit 1"}, VolumeMounts: []v1.VolumeMount{

Noticed this issue when looking at logs to figure out which images we use in a e2e test and from where. for example: Looks like it is from here:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign /sig storage cc
see background in and
slack discussion -
Looks like we can switch to see below:
looks like this image supports all the arch(es) we care about as well: !


Question: var _ volume.Volume = &csiMountMgr{} func (c *csiMountMgr) GetPath() string { <del> dir := filepath.Join(getTargetPath(c.podUID, c.specVolumeID, c.plugin.host), "/mount") </del> <ins> dir := GetCSIMounterPath(filepath.Join(getTargetPath(c.podUID, c.specVolumeID, c.plugin.host))) </ins> klog.V(4).Info(log("mounter.GetPath generated [%s]", dir)) return dir }

In , we check mount on first level directory in pod volume directories. It is true for many volume plugins, but not for CSI plugin which . cc
/assign
Pardon my ignorance of kubelet source code, but this looks very much similar to an issue we have been running to many times recently, when developing a CSI plugin. The problem is that when CSI volume publish is called and the mount point is not propagated from the CSI plugin pod (either missing bidirectional flag in yaml or not supported by underlaying k8s cluster implementation), the kubelet nevertheless starts a pod using the volume. The volume is not there, so the app writes to root filesystem in fact, leading to all other kinds of problems - i.e. losing data, filling up root fs. IMHO what kubelet should do is to check that the mounted filesystem is indeed visible after publish call and if not it should fail the volume attach request. If this ticket was fixed, would it solve the issue we are seeing or should I open a new ticket? thanks
I think we can potentially add that check but we need to make sure first it can work with ephemeral CSI drivers. Ephemeral drivers would have to "stage" their directories somewhere, like /var/lib/kubelet/plugins/<csi driver/, and then bind mount it to the publish path. cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Hmm... That is a good question. Does csi require a mount to be made or only that the specified directory be created? Will have to go read through the spec...
/remove-lifecycle stale
The only place in the spec I could find was this comment: // For volumes with an access type of mount, the SP SHALL place the // mounted directory at target_path. So I think having Kubelet enforce that check would be ok with the spec, and a good thing to do.
cc CSI spec says that the target path has to be a mount. Shall we go ahead and enforce it then? Are we going to break drivers that may not be doing this?

Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
The issue mentioned here involves two functions . Now with PR , the issue in CheckVolumeExistenceOperation is solved by checking mount path instead of volume path. The only issue here is for getMountedVolumePathListFromDisk which is used during pod deletion. This function was introduced by PR . Now I am questioning whether we need to check the mount point or just need to check pod dirs exist before deleting the pod. Firstly, for some plugins, there might not be mount point. Secondly, without checking mount point, it means that pod cannot be deleted as long as there are pod volume dirs. In normal case, volume dirs should be cleanedup during volume teardown. For cases that kubelet restarts, we have reconciler try to clean up volume mounts if possible. So we can propose a couple of solutions here mount point check. It solves the problem, the downside is that there might be some rare case that reconciler could not clean up dirs after kubelet restarts. check mount point, and also check the next level "/mount" dir. This might still cause problem for some plugins that do not have mounts. cc
Discussed offline with For proposal 1., pod will not be deleted from api server if there are still volume dirs exist. So to make sure it works well, we need to have reliable way to clean up dirs. Normally when kubelet restarts, reconciler will either add it into actual state (if reconstruction works) or try to clean it up directly once unless volume is still in desired state. So if reconstruction does not work, and it fails to clean up during reconciler sync, there is no other chance to clean up the dirs.

Or if volume was in desired state and removed quickly afterwords, the dir might not get the chance to be cleaned up at all. We are thinking to improve this part, we could use mark volume "uncertain" in actual state feature. During reconstruction, we can remove the check for mounted points (since the mount might be gone during node reboots or driver does not have moutns at all). Reconciler could mark those volumes as uncertain (even if it was in desired state) to make sure those volume got cleaned up or remount whenever needed with retries. With this change, I think it is safe to remove the mount point check for pod deletion too.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: t.Logf("allocated: %v", found) } <ins> func TestRangeSize(t *testing.T) { testCases := []struct { name  string cidr  string addrs int64 }{ { name:  "supported IPv4 cidr", cidr:  "192.168.1.0/24", addrs: 256, }, { name:  "supported large IPv4 cidr", cidr:  "10.96.0.0/12", addrs: 1048576, }, { name:  "unsupported IPv4 cidr", cidr:  "192.168.1.0/1", addrs: 0, }, { name:  "supported IPv6 cidr", cidr:  "2001:db8::/48", addrs: 65536, }, { name:  "unsupported IPv6 mask", cidr:  "2001:db8::/1", addrs: 0, }, } for _, tc := range testCases { _, cidr, err := net.ParseCIDR(tc.cidr) if err != nil { t.Errorf("failed to parse cidr for test %s, unexpected error: '%s'", tc.name, err) } if size := RangeSize(cidr); size != tc.addrs { t.Errorf("test %s failed. %s should have a range size of %d, got %d", tc.name, tc.cidr, tc.addrs, size) } } } </ins> func TestForEach(t *testing.T) { _, cidr, err := net.ParseCIDR("192.168.1.0/24") if err != nil {

What happened: All the services are asking to be recreated because the service CIDR changed after upgrading from 1.16.3 ro 1.17.0 What you expected to happen: Smooth migration from to , which would not break service CIDR. How to reproduce it (as minimally and precisely as possible): Do an upgrade from version 1.16.3 to 1.17.0 Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: OS (e.g: ):Kernel (e.g. ):Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig node
do you mean that the service cidr changed without any intervention after the upgrade? EDIT 96.0.0/12 is the default service cidr when using kubeadm --service-cidr string Default: "10.96.0.0/12" EDIT if not using kubeadm the default is May it be possible you were using another service cidr before the upgrade and you didn't set it during the upgrade, so it fall back to the default?
Can you provide your kube-apiserver invocation?
xref which warnings if no explicit CIDR flag was passed but should not have changed behavior otherwise
Where do you see that? The default is specified here as 10.0.0.0/24 and did not change between 1.16.0 and 1.17.0:
I was assuming it was using kubeadm based on the log message 96.0.0/12 is the default service CIDR for kubeadm
ah, ok. If they are using kubeadm, that default also did not change between 1.16.0 and 1.17.0:
a few questions for you: how did you install your 1.16.x cluster? did you customize the service cidr in that deployment? how did you upgrade your 1.16.x cluster to 1.17.x? did you apply the same service cidr customization in that deployment?
16.x was a straight and normal install no change on service CIRD, and we have never touched this CIDR configuration. And kubeadm is supposed to be used during the upgrade from 1.16.x to 1.17.x.

We just followed the describe process: The logs system started showing the shared message.
can you reproduce?
can you paste ? That will show you kubeadm config
created a single node cluster with v1.16.4 and a custom service cidr: applied upgrade to v1.17.0: i do not see a change in the custom service cidr.
if you never touched the service CIDR on the initial install it should have been which is the kubeadm default. your config map shows . how did you adapt the IP to work with the kubeadm default service cidr () if you initially deployed without modifying said cidr? to me this suggests that a change was required at some point.
The only change is the update from 1.16.3 to 1.17.0, done yesterday. Please find below a snapshot from our logs: <img width="1635" alt="Screen Shot 2019-12-21 at 22 22 29" src="https://user-"
my point is that if you did not apply a custom service cidr on "kubeadm init", then this cidr was the default out of kubeadm and after upgrade the service cidr was preserved as your config map shows. what was the elasticsearch/logging service IP before upgrade? EDIT: also do you have the YAML that was used to create this service?
101.249.43 belongs to the service rubnet range at minimum the error message is wrong /kind bug
are those errors happening constantly or only happened during the upgrade? are you able to create new services within that range? let's say, with a service IP of 10.101.249.45 Also, can you paste your current service-cluster-cidr configured in the controller-manager and apiserver? i.e.
this is a bug in core: this call in the repair controller returns for an IP that is actually in the CIDR range: also indicates a missing unit test case. latest commits: this might have regressed with the dual-stack changes? cc /remove-triage support needs-information /sig network /remove-sig cluster-lifecycle
/priority important-longterm
I was able to reproduce the same behavior again, with a fresh install k8s release 1.16.4.

Just After installing metrics server, we could notice a shift in the service CIDR, but nothing in the logs.
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/cc
so the problem here is that an IPv4 CIDR such as translates to addresses. but this line here clamps the maximum to 2^16 () so for an address like the logic in the ip allocator fails in the function: as the offset ends up bigger than the maximum. it feels like the maximums for IPv4 and IPv6 should be different and larger. otherwise for CIDRs in the notation : can give incorrect results for IPv4 can give incorrect results for IPv6 cc who touched related logic.
I think that was touched in … was this bug preexisting or did that change behavior?
before PR () i do not see a clamp for IPv4: only one for IPv6 at 16bits.
We should: revert add unit tests for the missed cases get that into 1.17.1 reintroduce the reliance on the function carefully
What would be the safest path to recreate all the services whilst waiting a final solution ? Especially for production K8S.
to mitigate the error in 1.17.0 you need to modify the serivce CIDR to be , instead of . modify the ConfigMap CIDR to have modify the kubeadm static pod manifest for the api server () to have edit problematic Service objects and give them an IP in the range if needed.
Thank you for those instructions, I'm affected by this, does this have anything to do with a possibility of ClusterIP's not properly responding or does it only print out alerts?
hello, can you test if the ClusterIP is not responding? i don't think we got a confirmation if this is only an alert. my guess is that it is just that, but there might be side effects.
i see you logged i will comment there.
How do you "recreate" a service? Literally delete and create again? Or is there a command which would cause it to re-create itself without having to be deleted?


Question: return &runtimeapi.ListPodSandboxResponse{Items: result}, nil } <ins> // applySandboxLinuxOptions applies LinuxPodSandboxConfig to dockercontainer.HostConfig and dockercontainer.ContainerCreateConfig. func (ds *dockerService) applySandboxLinuxOptions(hc *dockercontainer.HostConfig, lc *runtimeapi.LinuxPodSandboxConfig, createConfig *dockertypes.ContainerCreateConfig, image string, separator rune) error { if lc == nil { return nil } // Apply security context. if err := applySandboxSecurityContext(lc, createConfig.Config, hc, ds.network, separator); err != nil { return err } // Set sysctls. hc.Sysctls = lc.Sysctls return nil } </ins> func (ds *dockerService) applySandboxResources(hc *dockercontainer.HostConfig, lc *runtimeapi.LinuxPodSandboxConfig) error { hc.Resources = dockercontainer.Resources{ MemorySwap: DefaultMemorySwap(),

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: is trying to setup DNS for Windows containers, so that Pods on Windows could confirm to dns policies. However, as posted Docker is not managing the networking here for windows. So it doens't really care about any network settings. So passing it to docker shim's hostconfig also doens;t make sense here. DNS for Windows containers will be set by CNI plugins. So PR reverts . This issue tracks the process of pushing DNS configure into CNI and passed that from Kubelet to container runtime. What you expected to happen: Pods on Windows confirm to dns policies. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: /sig windows
Need a linked issue & PR in CNI repo too so that capability arguments can be validated from kubelet
Fixed in by dns capabilities, Azure CNI has already supported this. /close
Closing this issue. <detailsIn response to : Fixed in by dns capabilities, Azure CNI has already supported this. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: continue } proxier.ipsetList[kubeLoadBalancerSourceIPSet].activeEntries.Insert(entry.String()) <ins> } else { // since nodeIP is not covered in any of SourceRange we need to explicitly block the lbIP access from k8s nodes. proxier.lbNoNodeAccessIPPortProtocolEntries = append(proxier.lbNoNodeAccessIPPortProtocolEntries, entry) </ins> } } // ipvs call

When using loadBalancerSourceRanges to allow access only from CIDRs outside the cluster segment, access from the K8s nodes (or pods with hostNetwork:true) should be rejected, but for proxy-mode=ipvs it works. /sig network /area kube-proxy /area ipvs Access from K8s nodes should be rejected in the same way as for proxy-mode=iptables Define a service with loadBalancerSourceRanges that allows access from a cidr outside the cluster, e.g. The K8s nodes have addresses on 192.168.1.0/24 and fd00::c0a8:100/120. Make sure a loadBalancerIP is assigned (in this example 10.0.0.0/32) and try to access it from a K8s node: Same for both ipv4 and ipv6. This is related to since a fix requires that the passed to the is correct. Please see v1.27.4 and master None Linux 6.4.4 None N/A N/A
wanted to work on this
Great! But please coordinate with I hope that one get accepted, and then ~the s (both families) passed to the proxiers should be usable.~ No, they are not, please see
Discussed this one in the SIG Network meeting today: we agree this is something we'd like to implement, however we weren't sure exactly how to do it with ipvs during the meeting. Whoever takes this on needs to come up with and communicate a clear plan of how it will be accomplished. /triage accepted
My comment is not feasible. For proxy-mode=ipvs the src and dst is both loadBalancerIP. So the nodeIP will not help :disappointed:
I did a small experimented with kind + metallb. I created a loadbalancer service backed by . loadBalancerIP: 172.18.255.200 podIP: 10.244.1.3 conntrack picks the src and dst both as loadBalancerIP. Tried to create an iptables rules matching source and destination with loadbalancerIP to drop packets but it didn't work. After some exploration I found and with the following rules I was able to prevent access to lbIP from k8s nodes / host network.

The solution looks hacky though, we only need this rule if source ranges are defined for the service and node ip is not covered by any of them. Will try to program this in the ipvs proxier.
Good find! :+1: I had missed the ipvs extension. I started to check nodeIP and the (flawed) option to experiment with setting a nodeIP as "src" for some routes to dodge the lbip-both-as-src-and-dst problem, and an old "vision" I have to not assign clusterIPs to at all, but have one rule to replace them all, like: It can be done, and in a cluster with 1000's of clusterIP's it would simplify


Question: // obj must be a pointer to an API type. An error is returned if the minimum // required fields are missing. Fields that are not required return the default // value and are a no-op if set. <del> // TODO: return bool instead of error </del> func Accessor(obj interface{}) (metav1.Object, error) { switch t := obj.(type) { case metav1.Object:

Anything else we need to know?: I notice there are TODOs in apimachinerypkgapimeta,return bool instead of error func CommonAccessor, ListAccessor, Accessor. If is it necessary to do,I push a PR for it. Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
cc
/sig api-machinery
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close


Question: // sets, the same network interface couldn't be added to more than one load balancer of // the same type. Omit those nodes (e.g. masters) so Azure ARM won't complain // about this. <ins> newBackendPoolsIDs := make([]string, 0, len(newBackendPools)) </ins> for _, pool := range newBackendPools { <del> backendPool := *pool.ID matches := backendPoolIDRE.FindStringSubmatch(backendPool) if len(matches) == 2 { lbName := matches[1] if strings.HasSuffix(lbName, InternalLoadBalancerNameSuffix) == isInternal { klog.V(4).Infof("Node %q has already been added to LB %q, omit adding it to a new one", nodeName, lbName) return nil } </del> <ins> if pool.ID != nil { newBackendPoolsIDs = append(newBackendPoolsIDs, *pool.ID) </ins> } } <ins> isSameLB, oldLBName, err := isBackendPoolOnSameLB(backendPoolID, newBackendPoolsIDs) if err != nil { return err } if !isSameLB { klog.V(4).Infof("Node %q has already been added to LB %q, omit adding it to a new one", nodeName, oldLBName) return nil } </ins> } newBackendPools = append(newBackendPools,

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: SLB backend pools are empty if outbound rules are defined in a separate backend pool. This is because, although we the check for not adding the backend pools from different LBs, the lbName is not checked: The new backend pool's lbName should be also checked in above codes. What you expected to happen: SLB backend pools should be updated correctly when outbound rules are defined in a separate backend pool. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Others: /sig azure /kind bug /assign


Question: # besides the default ones. extra_prune_whitelist= if [ -n "${KUBECTL_EXTRA_PRUNE_WHITELIST:-}" ]; then <del> extra_prune_whitelist=( "${KUBECTL_EXTRA_PRUNE_WHITELIST:-}" ) </del> <ins> read -ra extra_prune_whitelist <<< "${KUBECTL_EXTRA_PRUNE_WHITELIST}" </ins> fi prune_whitelist=( "${KUBECTL_PRUNE_WHITELIST[@]}"  "${extra_prune_whitelist[@]}" ) prune_whitelist_flags=$(generate_prune_whitelist_flags "${prune_whitelist[@]}")

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Regression in 1.17 introduced by When there are several items passed using KUBECTLEXTRAPRUNE_WHITELIST flag, cluster/addons/addon-manager/kube- generates wrong flags for kubectl: Which causes: What you expected to happen: It should generate: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: /sig release Environment: Kubernetes version (use ): 1.17.*
/assign
- FYI This seems like a regression in 1.17, so we should merge fix in 1.17.
cc for shellcheck-induced regressions
This happens because we don't have an "API" documented for these environment variables. Ideally we should not be depending on space splitting strings.
Not yet fixed for 1.17 /reopen
Reopened this issue. <detailsIn response to : Not yet fixed for 1.17 Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: GCE_HOSTS= GCE_IMAGE_CONFIG_PATH=test/e2e_node/jenkins/image-config.yaml GCE_ZONE=us-central1-f <del> GCE_PROJECT=kubernetes-jenkins </del> <ins> GCE_PROJECT=k8s-jkns-ci-node-e2e </ins> CLEANUP=true GINKGO_FLAGS='--focus="[Serial]" --skip="[Flaky]"' SETUP_NODE=false

I want to isolate these in their own project:
Right now just the CI jobs, but after we also need to do this for the PR jobs.


Question: var FakeMetadata = Metadata{ Uuid:             "83679162-1378-4288-a2d4-70e13ec132aa", <del> Name:             "test", </del> <ins> Hostname:         "test", </ins> AvailabilityZone: "nova", }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: The OpenStack cloud provider currently uses the name of the kubelet node to register with the API server. However the name is not guaranteed to conform to the DNS-1123 subdomain spec. Given a node name of , the following message appears in the kubelet logs: What you expected to happen: The OpenStack Cloud provider uses the instance id instead of the instance name. This will guarantee that the id conforms the DNS-1123 spec without making any assumptions on what the instance name may be. How to reproduce it (as minimally and precisely as possible): In the OpenStack console, rename a kubelet node to and attempt to register the kubelet node to the api server. Anything else we need to know?: Environment: OpenStack Newton Kubernetes version (use ):Cloud provider or hardware configuration: OpenStack OS (e.g. from /etc/os-release): Ubuntu 14.04.5 Kernel (e.g. ): Linux -cab9-44ff-9c99- 4.4.0-103-generic ~14.04.1-Ubuntu SMP Mon Dec 4 19:33:04 UTC 2017 x8664 x8664 x86_64 GNU/Linux Install tools: Others:
/sig openstack
/assign I prefer to conform hostname to DNS-1123 subdomain spec. Already submitted a PR on this.
I think the issue may be more than just the hostname. The hostname on my vm is set to Since the OpenStack cloud provider uses the instance name to register with the api server, I also get error messages like the following: It seems like for OpenStack specifically, it sets it's node name to the name returned from the OpenStack metadata server () which is not restricted to being DNS-1123 compliant.
The node name must match ([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9], not contain '/'.
I can confirm that Openstack can boot up instances with the name like . But this will break kubelet currently as stated above. For such cloud provider based nodes, the nodeName registered is retrieved from the provider, which should not be allowed when registering nodes for kubelet.

Agree. +1. I've an extra check in PR . PTAL. Thanks.
I think the question here is: Why did you choose to use as the nodename? This is the display name of the server, a property made for humans by humans. Essentially it is a field that everybody should be able to fill in whatever they think their server should be called. This doesn't need to be unique and has no guarantees about format or length or character format. We've had a number of people putting in unicode characters in the display name, for example. Furthermore, it can be changed during runtime of the server, without rebooting. Other possibilities for the nodename are: instance UUID, which is a string in a defined format, guaranteed to be unique and is only separated by dashes. Found in If you would like to have some way to associate your nodename with whatever is shown in the Horizon dashboard as server display name, there is . OpenStack already takes care to sanitize any weird display name. E.g. some weird display name like gets a reasonable hostname: <img width="745" alt="screen shot 2018-01-10 at 08 52 49" src="https://user-"What do you think about changing the source for selecting the nodename?
cc
Once upon a time, the cloud provider's (only!) role was to discover a dynamic list of nodes by regex. In that world, using the nodename made perfect sense because the admin could then configure it to be something like , with the obvious use of regex patterns to group the relevant nodes. Fast forward a few years, and this style of node registration is long gone, and there is no need to make the node name human readable, and in fact numerous reasons to choose other unique identifiers. There is still to removing "human-friendly" node names, however. In the end, I didn't see a good alternative that would please everyone (and meet the required technical character set/length constraints). I stuck with because at least it was under the control of the cluster admin, and so it could be obviously contrived to match our imposed k8s node name == openstack instance name constraint (once that constraint was communicated).


Question: export KUBE_CONFIG_FILE=${KUBE_CONFIG_FILE:-${KUBE_ROOT}/cluster/ubuntu/config-default.sh} source "${KUBE_CONFIG_FILE}" <del> source "${KUBE_ROOT}/cluster/common.sh" </del> tear_down_alive_resources check-pods-torn-down

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): Is this a BUG REPORT or FEATURE REQUEST? (choose one): <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Environment: Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: What happened: I was following to install a cluster but failed due to the following. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else do we need to know:
The current workaround solution is remove in ./kube-
please have a look
thanks for the quick response
Just FYI, I think that you should fix both kube-up and kube-down
i have already move: source "${KUBE_ROOT}" to the head of ,is there something can't work?
Thanks, I did not have a try but just a reminder, just check code, yes, I think it works. Thanks
Seems the kube-up still does not work, filed another issue here
can you help take a look? Thanks!
/sig cluster-lifecycle
hi, I still see the issue but wondering since you have closed considering planned deprecation of the script. Does this issue still requires a fix or any decision was made not to work on it? Thanks!
Thanks , I think that we can close this now as it is not supported any more.
sure and thank you for the quick response and closing it !!


Question: "affinity":                      "If specified, the pod's scheduling constraints", "schedulerName":                 "If specified, the pod will be dispatched by specified scheduler. If not specified, the pod will be dispatched by default scheduler.", "tolerations":                   "If specified, the pod's tolerations.", <del> "hostAliases":                   "HostAliases is an optional list of hosts and IPs that will be injected into the pod's hosts file if specified. This is only valid for non-hostNetwork pods.", </del> <ins> "hostAliases":                   "HostAliases is an optional list of hosts and IPs that will be injected into the pod's hosts file if specified.", </ins> "priorityClassName":             "If specified, indicates the pod's priority. "system-node-critical" and "system-cluster-critical" are two special keywords which indicate the highest priorities with the former being the highest priority. Any other name must be defined by creating a PriorityClass object with that name. If not specified, the pod priority will be default or zero if there is no default.", "priority":                      "The priority value. Various system components use this field to find the priority of the pod. When Priority Admission Controller is enabled, it prevents users from setting this field. The admission controller populates this field from PriorityClassName. The higher the value, the higher the priority.", "dnsConfig":                     "Specifies the DNS parameters of a pod. Parameters specified here will be merged to the generated DNS configuration based on DNSPolicy.",

there is a note above hostAliases that this feature does not work with hostNework=true, yet it seems it does: when testing with a hostNetwork pod + hostAliases, there are no errors in kubelet logs at --v=10 related to this, in fact i don't see any hostAlias entries at all in there. i could not find anything blocking this in the kubelet, in fact there is normal handling of hostNetwork: i could not find any validation performed on the hostAliases field of PodSpec if hostNetwork is true on the API level. i could not find any unit tests or e2e tests related to this. the godoc should contain valid information if the feature is not supposed to work, it must be patched to not have the entries in the hosts file (perhaps?) we should have tests that ensure this type of requirement is guaranteed these steps confirm that a static pod with hostNework (deployed by kubeadm) does make use of hostAliases. create a kubeadm single node cluster directly with or one with modify the to include:wait some time for the new static pod spec to resync find the PID of the kube-apiserverget the hosts file n/a latest master 1.30.0-alpha.0 not applicable <details</detailskubeadm containerd v1.6.14 kindnet add test: update godoc:
/sig node network cluster-lifecycle /kind documentation
as reported by the hostAliases sometimes are not mounted for hostNetwork static pods, which is concerning: related slack thread:
At minimum the comment is wrong I need to be updated and we need e2e tests :/
found this, support was a while back: i will PR the API docs.
We identified an edge case here with Kubernetes v1.29. If a cloud provider did not yet initialise the node's , the code for creating the file including the hostAliases does not get run. To be concrete, in: is empty in this case which leads to to be . I was able to trace it back to this change which leads to this behaviour for static pods: For confirmation: I reverted the above commit, rebuilt kubelet and it got back to the old behaviour.

there is no perfect solution, but reverting is wrong because opens another bug, we need to study this more carefully but in the meantime installers that know the nodes addresses beforehand should set the --node-ip flag or not use cloud-provider external
yeah, as i mentioned downstream, revering is out of the question. it fixes some long standing problems. we might have to document somewhere this limitation in a cloud provider env.
i think it might be possible for us to special case this so that the hostAlias management does not wait for the node IP in a CP env, under certain conditions. just a speculation.
PRs add test: update godoc:
/triage accepted looks like it is being worked on already


Question: operations:     []string{"add", "add", "add", "add", "add", "next", "next", "next", "next", "add", "next", "next", "next"}, expectedOutput: []string{"node-4", "node-5", "node-6", "node-7", "node-3", "node-8", "node-4"}, }, <ins> { name:           "remove zone and add new to ensure exhausted is reset correctly", nodesToAdd:     append(allNodes[3:5], allNodes[6:8]...), nodesToRemove:  allNodes[3:5], operations:     []string{"add", "add", "next", "next", "remove", "add", "add", "next", "next", "remove", "next", "next"}, expectedOutput: []string{"node-3", "node-4", "node-6", "node-7", "node-6", "node-7"}, }, </ins> } for _, test := range tests {

Testgrid: https://k8s- Sample failure log: https://k8s- This test job has always been flaky, but is now failing more often than it's passing. Diagnosing it is difficult, though, because even the "successful" runs have multiple failures in the log, and the reported test failures seem not to be independant failures in the logs, but rather related to multiple failures to deploy pods and storage in the test setup. Does anyone have more insight into what's going on here? This feels more like a google cloud or test config failure than anything else. /kind failing-test /kind flake /sig test-infrastructure /sig gcp /sig apps
/sig testing
also looks like https://k8s- has similar issues, so this is actually not a gke specific problem.
one of the kubelet log has not sure if that's helpful at all... /assign (redirect to get some help from dev-oncall to see if can find any leads here...)
/assign /cc
GitHub didn't allow me to assign the following users: artemvmin. Note that only and repo collaborators can be assigned. For more information please see <detailsIn response to : /assign /cc Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The suites are failing on both gce and gke, so I don't think it's a gke specific issue. Looking at one occurrence: https://k8s- The first test that started failing was: Because it couldn't schedule the pod. Looking at the scheduler logs, I only see these two lines referring to that pod and nothing else: This is odd because there is usually at least one more log message if scheduling failed or succeeded. This implies that the pod is stuck somewhere in the scheduler and there could potentially be a deadlock. There are no other scheduling logs about other pods after this. An odd pattern that I noticed was that for all the serial runs where a ton of tests fail, the test case that always runs before the subsequent failing tests is the equivalence cache tests. And there's always a line in the like: I wonder if there is some race condition regarding the pod deletion case.

I have seen this pattern in the most recent runs where a bunch of tests all fail: https://k8s- https://k8s- https://k8s- https://k8s- The first failing test case is different across all these runs, but the commonality between them all is that the equivalence cache tests run right before it.
can you plz help take a look at this and help triage within scheduling?
fyi, this is pretty easy to repro on a dev cluster on head. I just run the EquivalenceCache tests, and then create any Pod afterwards. The Pod fails to schedule about 50% of the time.
I am investigating it.
I identified the issue and sent a PR to address it. We need to cherrypick the fix to 1.12 as well. Thanks to for the initial investigation and finding reproduction steps!
Thanks a lot and for looking into this and giving a fox PR. /cc needs to be CP'ed into 1.12 as well.
It already got approved. Please create a cherry pick PR for it.


Question: } ginkgo.By("deleting the pod") <del> err = podClient.Delete(ctx, pod.Name, *metav1.NewDeleteOptions(30)) </del> <ins> err = podClient.Delete(ctx, pod.Name, *metav1.NewDeleteOptions(1)) </ins> framework.ExpectNoError(err, "failed to delete pod") }) ginkgo.It("should provide hostIPs as an env var", func(ctx context.Context) { <ins> if !utilfeature.DefaultFeatureGate.Enabled(kubefeatures.PodHostIPs) { e2eskipper.Skipf("PodHostIPs feature is not enabled") } </ins> podName := "downward-api-" + string(uuid.NewUUID()) env := []v1.EnvVar{ {

4 other Test Failures in ci-crio-cgroupv1-node-e2e-features. Test has failed 61 times (no passing result found). Earliest failure found at commit/ (Thu Oct 19 21:41 PDT). /cc { failed [FAILED] Error creating Pod: pods "pod-dualstack-host-ips9ddb9810-134c-4d35-a0c6-" is forbidden: violates PodSecurity "restricted:latest": host namespaces (hostNetwork=true) In [It] at: @ 10/23/23 01:38:06.024 } No response_ /sig network node
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
is also failing on these tests
the test has a bug, it checks the contrary
, let's start over, 1 commit for the e2e test, other for the promotion, and let's validate everything is alright before merging


Question: }, }, }, <del> expectedDNS: "newdns", </del> <ins> }, { desc:                    "ensurePublicIPExists shall delete DNS from PIP if DNS label is set empty", foundDNSLabelAnnotation: true, existingPIPs: []network.PublicIPAddress{{ Name: to.StringPtr("pip1"), PublicIPAddressPropertiesFormat: &network.PublicIPAddressPropertiesFormat{ DNSSettings: &network.PublicIPAddressDNSSettings{ DomainNameLabel: to.StringPtr("previousdns"), }, }, }}, expectedPIP: &network.PublicIPAddress{ Name: to.StringPtr("pip1"), ID: to.StringPtr("/subscriptions/subscription/resourceGroups/rg" + "/providers/Microsoft.Network/publicIPAddresses/pip1"), PublicIPAddressPropertiesFormat: &network.PublicIPAddressPropertiesFormat{ DNSSettings: nil, }, }, }, { desc:                    "ensurePublicIPExists shall not delete DNS from PIP if DNS label annotation is not set", foundDNSLabelAnnotation: false, existingPIPs: []network.PublicIPAddress{{ Name: to.StringPtr("pip1"), PublicIPAddressPropertiesFormat: &network.PublicIPAddressPropertiesFormat{ DNSSettings: &network.PublicIPAddressDNSSettings{ DomainNameLabel: to.StringPtr("previousdns"), }, }, }}, expectedPIP: &network.PublicIPAddress{ Name: to.StringPtr("pip1"), ID: to.StringPtr("/subscriptions/subscription/resourceGroups/rg" + "/providers/Microsoft.Network/publicIPAddresses/pip1"), PublicIPAddressPropertiesFormat: &network.PublicIPAddressPropertiesFormat{ DNSSettings: &network.PublicIPAddressDNSSettings{ DomainNameLabel: to.StringPtr("previousdns"), }, }, }, </ins> }, }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: After issue is fixed. Cloud provider will automatically reconcile the Azure Public Ip DNS record same as the service annotation. As a result, user finds their DNS record will be deleted if the annotation is not set. Such a change is a block to users who have set the DNS record on their public IP without using this annotation when they upgrade the cluster. Besides, if the public ip is a user's own public ip, it is also not reasonable to delete the DNS record. What you expected to happen: We may need to change the reconcile logic. Maybe only delete the DNS record, when the annotation exists but empty value How to reproduce it (as minimally and precisely as possible): Manually add the DNS label to Azure Public IP resource via Portal Environment: Kubernetes version (use ): 1.15.7 Cloud provider or hardware configuration: Azure
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig Azure Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cloud-provider
/area provider/azure
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig Azure Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
The fix helm install helm-nginx-ingress stable/nginx-ingress  --namespace ingress-nginx-ns  --set controller.replicaCount=2  --set controller.nodeSelector."beta.kubernetes.io/os"=linux  --set controller.service.annotations."service.beta.kubernetes.io/azure-dns-label-name"=MY-AZURE-DNS  --set defaultBackend.nodeSelector."beta.kubernetes.io/os"=linux


Question: }, }, { <del> description: "pod has tolerations, but none is for taint `notReady:NoExecute` or `unreachable:NoExecute`, expect add tolerations for `notReady:NoExecute` and `unreachable:NoExecute`", </del> <ins> description: "pod has alpha tolerations, expect add tolerations for `not-ready:NoExecute` and `unreachable:NoExecute`" + ", the alpha tolerations will not be touched", requestedPod: api.Pod{ Spec: api.PodSpec{ Tolerations: []api.Toleration{ { Key:               algorithm.DeprecatedTaintNodeNotReady, Operator:          api.TolerationOpExists, Effect:            api.TaintEffectNoExecute, TolerationSeconds: &defaultTolerationSeconds, }, { Key:               algorithm.DeprecatedTaintNodeUnreachable, Operator:          api.TolerationOpExists, Effect:            api.TaintEffectNoExecute, TolerationSeconds: &defaultTolerationSeconds, }, }, }, }, expectedPod: api.Pod{ Spec: api.PodSpec{ Tolerations: []api.Toleration{ { Key:               algorithm.DeprecatedTaintNodeNotReady, Operator:          api.TolerationOpExists, Effect:            api.TaintEffectNoExecute, TolerationSeconds: &defaultTolerationSeconds, }, { Key:               algorithm.DeprecatedTaintNodeUnreachable, Operator:          api.TolerationOpExists, Effect:            api.TaintEffectNoExecute, TolerationSeconds: &defaultTolerationSeconds, }, { Key:               algorithm.TaintNodeNotReady, Operator:          api.TolerationOpExists, Effect:            api.TaintEffectNoExecute, TolerationSeconds: &defaultTolerationSeconds, }, { Key:               algorithm.TaintNodeUnreachable, Operator:          api.TolerationOpExists, Effect:            api.TaintEffectNoExecute, TolerationSeconds: &defaultTolerationSeconds, }, }, }, }, }, { description: "pod has alpha not-ready toleration, expect add tolerations for `not-ready:NoExecute` and `unreachable:NoExecute`" + ", the alpha tolerations will not be touched", requestedPod: api.Pod{ Spec: api.PodSpec{ Tolerations: []api.Toleration{ { Key:               algorithm.DeprecatedTaintNodeNotReady, Operator:          api.TolerationOpExists, Effect:            api.TaintEffectNoExecute, TolerationSeconds: &defaultTolerationSeconds, }, }, }, }, expectedPod: api.Pod{ Spec: api.PodSpec{ Tolerations: []api.Toleration{ { Key:               algorithm.DeprecatedTaintNodeNotReady, Operator:          api.TolerationOpExists, Effect:            api.TaintEffectNoExecute, TolerationSeconds: &defaultTolerationSeconds, }, { Key:               algorithm.TaintNodeNotReady, Operator:          api.TolerationOpExists, Effect:            api.TaintEffectNoExecute, TolerationSeconds: &defaultTolerationSeconds, }, { Key:               algorithm.TaintNodeUnreachable, Operator:          api.TolerationOpExists, Effect:            api.TaintEffectNoExecute, TolerationSeconds: &defaultTolerationSeconds, }, }, }, }, }, { description: "pod has alpha unreachable toleration, expect add tolerations for `not-ready:NoExecute` and `unreachable:NoExecute`" + ", the alpha tolerations will not be touched", requestedPod: api.Pod{ Spec: api.PodSpec{ Tolerations: []api.Toleration{ { Key:               algorithm.DeprecatedTaintNodeUnreachable, Operator:          api.TolerationOpExists, Effect:            api.TaintEffectNoExecute, TolerationSeconds: &defaultTolerationSeconds, }, }, }, }, expectedPod: api.Pod{ Spec: api.PodSpec{ Tolerations: []api.Toleration{ { Key:               algorithm.DeprecatedTaintNodeUnreachable, Operator:          api.TolerationOpExists, Effect:            api.TaintEffectNoExecute, TolerationSeconds: &defaultTolerationSeconds, }, { Key:               algorithm.TaintNodeNotReady, Operator:          api.TolerationOpExists, Effect:            api.TaintEffectNoExecute, TolerationSeconds: &defaultTolerationSeconds, }, { Key:               algorithm.TaintNodeUnreachable, Operator:          api.TolerationOpExists, Effect:            api.TaintEffectNoExecute, TolerationSeconds: &defaultTolerationSeconds, }, }, }, }, }, { description: "pod has tolerations, but none is for taint `not-ready:NoExecute` or `unreachable:NoExecute`, expect add tolerations for `not-ready:NoExecute` and `unreachable:NoExecute`", </ins> requestedPod: api.Pod{ Spec: api.PodSpec{ Tolerations: []api.Toleration{

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: When discussing issue in , I noticed: although is using key , it has actually been changed to in for quite a while. I am thinking we may need to fix this inconsistency in 1.9, by moving out of alpha. Anything else we need to know?: Task tracker: [x] update [x] update kubernetes doc
/sig scheduling


Question: ALLOCATE_NODE_CIDRS=true <del> KUBE_PROMPT_FOR_UPDATE=y KUBE_SKIP_UPDATE=${KUBE_SKIP_UPDATE-"n"} </del> <ins> KUBE_PROMPT_FOR_UPDATE=${KUBE_PROMPT_FOR_UPDATE:-"n"} </ins> # How long (in seconds) to wait for cluster initialization. KUBE_CLUSTER_INITIALIZATION_TIMEOUT=${KUBE_CLUSTER_INITIALIZATION_TIMEOUT:-300}

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): Is this a BUG REPORT or FEATURE REQUEST? (choose one): FEATURE REQUEST <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): all :smile: Environment: Cloud provider or hardware configuration: GCE OS (e.g. from /etc/os-release): all Kernel (e.g. ): all Install tools: Others: What happened: uses the cli a lot and does a (in the commonly used function ) before doing things like . updates can introduce breaking changes (for example in the recent 132.0.0 release it removed a bunch of flags). This has caused cross-the-board test failures like . But this is not limited to just tests. Any kubernetes GCE users who rely on the and commands could have been broken. What you expected to happen: Stop updating gcloud automatically. We could either warn user to have the required dependencies installed, or pin to specific versions of dependencies like gcloud. How to reproduce it (as minimally and precisely as possible): download 1.4.4 release tarball run and Anything else do we need to know:
cc
/cc
cc I think it's reasonable to remove gcloud update calls from scripts, as I believe gcloud is updated out of band by k8s continuous testing. The scripts will still need to work with latest, so maybe they should check for a minimum version.
Good idea.
Yeah, we definitely want to continue using latest in CI so that we know if things break, but I think it's reasonable to remove the update itself from the kubernetes cluster util scripts.


Question: if gcloud compute addresses create "$1"  --project "${PROJECT}"  --region "${REGION}" -q > /dev/null; then <del> # successful operation </del> <ins> # successful operation - wait until it's visible start="$(date +%s)" while true; do now="$(date +%s)" # Timeout set to 15 minutes if [ $((now - start)) -gt 900 ]; then echo "Timeout while waiting for master IP visibility" exit 2 fi if gcloud compute addresses describe "$1" --project "${PROJECT}" --region "${REGION}" >/dev/null 2>&1; then break fi echo "Master IP not visible yet. Waiting..." sleep 5 done </ins> break fi

I'm seeing the following failures of Kubemark in presubmit: It calls stop-kubemark without clear reason. Such failure are e.g. here:
Ref
Actually, both these failures are a different issue. The logs that are problematic are: Why we don't have kubectl there?
I don't know why - ? I haven't seen those error recently, so I guess this is fixed. OTOH I have seen a number of runs where cluster startup failed, but there's a fun thing about those failures. All of them occurred on runs on worker-22:
I found something even more interesting. Extending the log: It seems that the cluster was created around 00:30 on 2016-09-20, MTV time (this is a default Jenkins time). But doing: gives the following result: Which seems to indicate that the address was never created around that time... cc
[FLAKE-PING] This flaky-test issue would love to have more attention.


Question: klog.Errorf("Failed to cast serviceInfo %q", svcName.String()) continue } <ins> isIPv6 := utilnet.IsIPv6(svcInfo.ClusterIP()) </ins> protocol := strings.ToLower(string(svcInfo.Protocol())) // Precompute svcNameString; with many services the many calls // to ServicePortName.String() show up in CPU profiles.

/kind bug What happened: When creating a node port application (in this case the example guestbook) on a host server that is dual stacked and has AAAA and A DNS entries, the node port on the v6 address is open to kube-proxy and hangs. What you expected to happen: The v6 address would refuse the connection and the client would fall back onto the v4 address which would connect correctly. How to reproduce it (as minimally and precisely as possible): kubeadm init with the above, and join in some workers. Then install the guestbook application Obtain the node port with and telnet to one of the servers on that port from a dual stacked client. The client will connect and refuse to respond to 'GET /' telnet directly to the IPv4 address of the server on the port and all works as expected. Anything else we need to know?: kube-proxy is noted as listening on the port. Environment: Kubernetes version (use ): ubuntu kubectl version Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-07-17T18:53:20Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-07-17T18:43:26Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: Brightbox OS (e.g. from /etc/os-release): Ubuntu 18.04.1 LTS Kernel (e.g. ): Linux srv-jidtj 4.15.0-29-generic -Ubuntu SMP Tue Jul 17 15:39:52 UTC 2018 x8664 x8664 x86_64 GNU/Linux Install tools: kubeadm 1.11 Others:
/sig network
Looks like kube-proxy in ipvs mode listens on the nodeport ports to ensure they're not in use by something else. It just binds and listens, it doesn't serve requests.

It binds on both ipv4 and ipv6 but never expects to receive connections because the iptables/ipvs rules intercept them, but on ipv4 clusters there is nothing stopping ipv6 connections coming in. The comment there does admit that only a bind is necessary to do this, but doing a full listen just made e2e tests easier. /cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen This now causes conformance failure on e2e tests due to the promotion of the Nodeport test in
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Still an issue on 1.16 when running in standard IPv4 mode
As I understand this thread, I have the same problem. I thought that it is a node configuration or some kubernetes flag that I didn't set but it seems it ain't.

I ask already on SO: Environment Kubernetes version (use kubectl version): Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-09-18T14:36:53Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-10-02T16:51:36Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: On premise, on virtual machines using hyper-v OS (e.g. from /etc/os-release): NAME="Ubuntu" VERSION="18.04.3 LTS (Bionic Beaver)" Kernel (e.g. uname -a): Linux kubernetes 5.0.0-29-generic # 31~18.04.1-Ubuntu SMP Thu Sep 12 18:29:21 UTC 2019 x8664 x8664 x86_64 GNU/Linux Install tools: kubeadm version: &version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-09-18T14:34:01Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}
/lifecycle frozen
/assign bowei
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
Looking through the comments, this looks like a bug that could have a targeted fix. /remove-triage unresolved /good-first-issue /help-wanted
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : Looking through the comments, this looks like a bug that could have a targeted fix.

/good-first-issue /help-wanted </details
I will attempt to solve this issue.
any success or progress? If you left this issue I would like to give it a go as a first time contributor. Thanks, Yuval
I'm still working on this as my first issue, and planning to produce a solution by 20th.
I wasn't able to tackle the issue and will not continue trying, feel free to take it.
/assign yuvalsteuer
Still, open -- anyone actively working on this? I would like to attempt if not. :) /assign rkc007
OK, again still open, doesn't look like anyone is working on this. I'd like to try it. The fix that occurs to me is simply to only listen to IPv6 addresses if the cluster is IPv6. If the e2e tests fail in this case (from looking at I don't think they will) then maybe they need to be updated. Does that make sense? What I'm not sure about is if we should do nothing with IPv6 ports or bind (to claim the port) but not listen.
/assign michaelbeaumont


Question: /* Release : v1.15 Testname: Pods, delete grace period <del> Description: Create a pod, make sure it is running. Using the http client send a 'delete' with gracePeriodSeconds=30. Pod SHOULD get deleted within 30 seconds. </del> <ins> Description: Create a pod, make sure it is running. Using the http client send a 'delete' with gracePeriodSeconds=30. Pod SHOULD get terminated within gracePeriodSeconds and removed from API server within a window. </ins> */ <del> ginkgo.It("should be submitted and removed [Flaky]", func() { </del> <ins> ginkgo.It("should be submitted and removed", func() { </ins> ginkgo.By("creating the pod") name := "pod-submit-remove-" + string(uuid.NewUUID()) value := strconv.Itoa(time.Now().Nanosecond())

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: Which test(s) are failing: [] [sig-node] Pods Extended [] Delete Grace Period should be submitted and removed [Conformance] Since when has it been failing: The ipv6 jobs have flakiness, but since I had 3 flakies in a row with this test Testgrid link: (IPv6),%20master%20(dev)%20%5Bnon-serial%5D&width=5&sort-by-flakiness= Reason for failure: It fails because is not able to delete de pod in 30 seconds, seems that this is the cause of that Nov 30 07:41:07 kind-worker kubelet[677]: W1130 07:41:07. 677 ] Failed to process watch event {EventType:0 Name:/docker/f35390ea97c6bae92f61d7d297fe7eacb4cfb3f2effafb23f3aca00e1a9d2341/kubepods/besteffort/podd8d966f8-2a03-4529-9d70-/243828b3fed0edf1137a9fa5ade6ffbbe468b79bbf212206480bbd37ab20fc7d WatchSource:0}: task 243828b3fed0edf1137a9fa5ade6ffbbe468b79bbf212206480bbd37ab20fc7d not found: not found Anything else we need to know: The job view seems to report the same test failures twice but the log output shows different timestamps
/sig node
/cc
interestingly, the serial ipv6 job doesn't look flaky: (IPv6),%20master%20(dev)&width=5&include-filter-by-regex=Delete%20Grace%20Period
this one is failing quite a lot in the 1.17 windows job - could someone confirm if it's a bug or a test issue? Thanks
or is it actually a completely different failure on windows? Since it's being tracked here as well
will close this issue because the patch addressing the issue was merged and to avoid having duplicates We can reopen if necessary /close
Closing this issue.

<detailsIn response to : will close this issue because the patch addressing the issue was merged and to avoid having duplicates We can reopen if necessary /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
by the way, this one is unrelated to the windows one :-)
ACK, thank you :)
Two failures seen on after the related PR merged, that was only a cleanup, not a root cause. /reopen
Reopened this issue. <detailsIn response to : two failures seen on after the related PR merged Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-kind failing-testing /kind flake
Those labels are not set on the issue: <detailsIn response to : /remove-kind failing-testing /kind flake Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-kind failing-test
I think that I'm making progress trying to deflake the ipv6 tests I have submitted PR to fix the portmap plugin issue that should be able to improve the situation
Retitling because I don't think this specific test failure is IPv6 specific anymore
triage: flaking in pull-kubernetes-e2e-gce: /priority important-soon This is the fourth flakiest test for pull-kubernetes-e2e-gce
debugging in
bumped priority, a conformance test must not be flaky. this is now the top flake in the e2e suite.
cc
debugging log messages merged 2019-12-19T08:38:10Z, will check for output in next failure
debug output from failure:
debug output from success:
failure in
based on my reading of this, the timeline was: 00:02:23. - we got a pod running 00:02:23. - deleted it with a 30 second grace period 00:02:54. - the kubelet finished handling the delete request (30.18 seconds) 00:02:54.

- test failed with a timeout (30.25 seconds) That seems like extremely tight tolerances, especially since the grace period refers to termination of the pod process, not removal from the API. I would be inclined to allow up to 2x the grace period to observe complete removal from the API.
opened to allow time for the kubelet to remove the terminated pod from the API
top flake according to
Opened to tag as [Flaky] and kick out of [Conformance]
Test is no longer run in blocking presubmit. - who owns resolving this and reintroducing the test before 1.18 is released? Investigation with combined timeline at shows that the kubelet stopped the container and destroyed it within ~15 seconds (observed deletion at 06:20:11., destroyed container at 06:20:46.), but then took another 30 before actually deleting the pod from the API (06:21:12.).
To evaluate the effectiveness of a potential fix, I would recommend opening a PR that removes the tag along with the proposed fix, and repeatedly on that PR to accumulate 10-20 test runs.
/assign assigning sig-node leads for visibility/routing
/milestone v1.19
/milestone v1.18 we need an ack from sig-leads whether this is release blocking or not
This issue is opened for 2 releases in a row. I guess the flake is not a release-blocking. In addition, there are many PRs linked to this issue, and many attempts happened to make the situation better. However, we should keep this issue open and make sure that the flake will disappear. WDYT? cc
The test has been flaky for multiple releases, but it was marked flaky and completely removed mid-1.18, which is new. I'm wanting to make sure that absence of any test signal around this is known/acceptable to sig-node.
Links to the investigations I was able to do: combined e2e/api/kubelet log from failure with extended timeout:
Added to agenda for 3/24 sig-node call


Question: # Persist %PSCORE% ENV variable for user convenience PSCORE="C:Program FilesPowerShellpwsh.exe" <del> LABEL image_version="1.29" </del> <ins> LABEL image_version="1.29-1" </ins> ENTRYPOINT ["cmd.exe", "/s", "/c"]

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI -- [sig-windows] [Feature:Windows] GMSA Kubelet [Slow] kubelet GMSA support when creating a pod with correct GMSA credential specs passes the credential specs down to the Pod's containers [sig-windows] [Feature:Windows] GMSA Full [Serial] [Slow] GMSA support works end to end ** pause image using in image was transitioned to the official pause image which no longer has in the image related to /sig windows /triage accepted


Question: testSvcLabels := map[string]string{"e2e-test": "proxy-endpoints"} framework.Logf("Creating pod...") <del> _, err := f.ClientSet.CoreV1().Pods(ns).Create(context.TODO(), &v1.Pod{ </del> <ins> pod := &v1.Pod{ </ins> ObjectMeta: metav1.ObjectMeta{ <del> Name: "agnhost", </del> <ins> Name:      "agnhost", Namespace: ns, </ins> Labels: map[string]string{ "e2e-test": "proxy-endpoints"}, },

pull-kubernetes-e2e-kind-ipv6 - Not sure. Time out on waiting Pod to be started. No response /sig network
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: "k8s.io/api/core/v1" "k8s.io/apimachinery/pkg/api/resource" "k8s.io/apimachinery/pkg/util/sets" <del> cloudprovider "k8s.io/cloud-provider" </del> <ins> "k8s.io/cloud-provider" </ins> volumehelpers "k8s.io/cloud-provider/volume/helpers" "k8s.io/kubernetes/pkg/util/mount" "k8s.io/kubernetes/pkg/volume"

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: We have pretty large clusters in aws and in production, we found an inefficiency in provisioning dynamic EBS volumes, where aws-ebs volume provisioner tries to look for available zone from aws directly. With scale, this can trigger massive from ec2, and affect cluster stability. This is also related to What you expected to happen: To simply get zones with available instances, we can get information from API server for efficiency. There might be a delay between node gets shutting down and such change is reflected in api server, but in case node is not ready or gets terminated, there will be higher level reconciliation triggered so we are safe. How to reproduce it (as minimally and precisely as possible): We only expect this to be hit when the cluster is huge (or aws account is huge) so not easy to reproduce. /assign /sig aws /sig cloud-provider
/assign


Question: } } <del> func TestPodAddressChanged(t *testing.T) { </del> <ins> func TestPodChanged(t *testing.T) { </ins> podStore := cache.NewStore(cache.DeletionHandlingMetaNamespaceKeyFunc) ns := "test" addPods(podStore, ns, 1, 1, 0)

This test has started failing on our e2e test suite, erroring out with "Failed to get endpoints..." https://k8s- https://k8s- Example failures https://k8s- https://k8s- cc cc
This may be related to our recently-merged endpoints change in . I'll look into it.
I verified that the flakiness introduced was a result of . A fix in should be merging soon. /assign
GitHub didn't allow me to assign the following users: joelsmith. Note that only can be assigned. <detailsIn response to : I verified that the flakiness introduced was a result of . A fix in should be merging soon. /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details