Question: } go func() { defer util.HandleCrash() <del> glog.Fatal(readOnlyServer.ListenAndServe()) </del> <ins> for { if err := readOnlyServer.ListenAndServe(); err != nil { glog.Errorf("Unable to listen for read only traffic (%v); will try again.", err) } time.Sleep(15 * time.Second) } </ins> }() }

I'm not using publicaddressoverride. At boot I'm trying to launch using: /usr/bin/kube-apiserver --logtostderr=true --v=0 --etcdservers=http://10.13.137.165:4001 --address=0.0.0.0 --port=8080 --kubeletport= --allowprivileged=true --portalnet=10.254.0.0/16 But this gets launched before eth0 has an ip address. This used to work. Now apiserver just refused to start (complaining about 127.0.0.1 being invalid). I think it needs to just keep trying until it gets an address and can get to work....
Adding --publicaddressoverride= as suggested doesn't help.... Nov 05 12:43:55 localhost kube-apiserver[713]: F1105 12:43:55. ] listen tcp 10.13.137.165:7080: bind: cannot assign requested address Which is a fatal error. I can't bind to it. It isn't up yet.
Bisect says either commit or
If publicaddressoverride is specified, I can put the read-only server bind in a goroutine and retry periodically. If it's not, though, since we can't change the public address once started, we'd have to delay construction until a suitable address shows up. That's pretty ugly; is it an important case to support?
I personally think delaying construction is fine. Being able to launch the apiserver on any node from the init system and have it just work 99% of the time seems like a good thing....
OK. Fix coming in 3, 2, 1...
See if does the trick for you.
commented there. it fixed it.


Question: // If non-nil, use this IP address for the node nodeIP net.IP <ins> // use this function to validate the kubelet nodeIP nodeIPValidator func(net.IP) error </ins> // If non-nil, this is a unique identifier for the node in an external database, eg. cloudprovider providerID string

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: When using the flag on the kubelet, the node's list no longer reports , , addresses. What you expected to happen: That I can explicitly set to an IP, but the other addresses will still show up. How to reproduce it (as minimally and precisely as possible): Create a new kops cluster, modify the kubelet to use the flag with the eth0 on the node. Run Anything else we need to know?: This change will probably take place in  This came up as a result of Environment: Kubernetes version (use ): 1.9.x Cloud provider or hardware configuration: AWS OS (e.g. from /etc/os-release): any Kernel (e.g. ): any Install tools: kops /sig network /sig node I'll make a PR for this change shortly


Question: if err := gce.(*gce_cloud.GCECloud).DetachDisk(pd.pdName); err != nil { return err } <ins> // Verify disk detached, retry if needed. go verifyDetached(pd, gce) return nil } // Verifys the disk device to be created has been succesffully attached, and retries if it fails. func verifyAttached(pd *gcePersistentDisk, sdBeforeSet util.StringSet, gce cloudprovider.Interface) (string, error) { devicePaths := getDiskByIdPaths(pd) for numRetries := 0; numRetries < maxRetries; numRetries++ { for numChecks := 0; numChecks < maxChecks; numChecks++ { if err := udevadmChangeToNewDrives(sdBeforeSet); err != nil { // udevadm errors should not block disk attachment, log and continue glog.Errorf("%v", err) } for _, path := range devicePaths { if pathExists, err := pathExists(path); err != nil { return "", err } else if pathExists { // A device path has succesfully been created for the PD glog.V(5).Infof("Succesfully attached GCE PD %q.", pd.pdName) return path, nil } } // Sleep then check again glog.V(5).Infof("Waiting for GCE PD %q to attach.", pd.pdName) time.Sleep(checkSleepDuration) } // Try attaching the disk again glog.Warningf("Timed out waiting for GCE PD %q to attach. Retrying attach.", pd.pdName) if err := gce.(*gce_cloud.GCECloud).AttachDisk(pd.pdName, pd.readOnly); err != nil { return "", err } } return "", fmt.Errorf("Could not attach GCE PD %q. Timeout waiting for mount paths to be created.", pd.pdName) } // Veify the specified persistent disk device has been succesfully detached, and retries if it fails. // This function is intended to be called asynchronously as a go routine. func verifyDetached(pd *gcePersistentDisk, gce cloudprovider.Interface) { defer util.HandleCrash() // Setting bufferSize to 0 so that when senders send, they are blocked until we recieve. This avoids the need to have a separate exit check. ch, err := detachCleanupManager.Start(pd.pdName, 0 /* bufferSize */) if err != nil { glog.Errorf("Error adding %q to detachCleanupManager: %v", pd.pdName, err) return } defer detachCleanupManager.Close(pd.pdName) devicePaths := getDiskByIdPaths(pd) for numRetries := 0; numRetries < maxRetries; numRetries++ { for numChecks := 0; numChecks < maxChecks; numChecks++ { select { case <-ch: glog.Warningf("Terminating GCE PD %q detach verification. Another attach/detach call was made for this PD.", pd.pdName) return default: allPathsRemoved := true for _, path := range devicePaths { if err := udevadmChangeToDrive(path); err != nil { // udevadm errors should not block disk detachment, log and continue glog.Errorf("%v", err) } if exists, err := pathExists(path); err != nil { glog.Errorf("Error check path: %v", err) return } else { allPathsRemoved = allPathsRemoved && !exists } } if allPathsRemoved { // All paths to the PD have been succefully removed glog.V(5).Infof("Succesfully detached GCE PD %q.", pd.pdName) return } // Sleep then check again glog.V(5).Infof("Waiting for GCE PD %q to detach.", pd.pdName) time.Sleep(checkSleepDuration) } } // Try detaching disk again glog.Warningf("Timed out waiting for GCE PD %q to detach. Retrying detach.", pd.pdName) if err := gce.(*gce_cloud.GCECloud).DetachDisk(pd.pdName); err != nil { glog.Errorf("Error on retry detach PD %q: %v", pd.pdName, err) return } } glog.Errorf("Could not detach GCE PD %q. One or more mount paths was not removed.", pd.pdName) } // Returns list of all /dev/disk/by-id/* paths for given PD. func getDiskByIdPaths(pd *gcePersistentDisk) []string { devicePaths := []string{ path.Join(diskByIdPath, diskGooglePrefix+pd.pdName), path.Join(diskByIdPath, diskScsiGooglePrefix+pd.pdName), } if pd.partition != "" { for i, path := range devicePaths { devicePaths[i] = path + diskPartitionSuffix + pd.partition } } return devicePaths } // Checks if the specified path exists func pathExists(path string) (bool, error) { _, err := os.Stat(path) if err == nil { return true, nil } else if os.IsNotExist(err) { return false, nil } else { return false, err } } // Calls "udevadm trigger --action=change" for newly created "/dev/sd*" drives (exist only in after set). // This is workaround for Issue #7972. Once the underlying issue has been resolved, this may be removed. func udevadmChangeToNewDrives(sdBeforeSet util.StringSet) error { sdAfter, err := filepath.Glob(diskSDPattern) if err != nil { return fmt.Errorf("Error filepath.Glob("%s"): %vrn", diskSDPattern, err) } for _, sd := range sdAfter { if !sdBeforeSet.Has(sd) { return udevadmChangeToDrive(sd) } } return nil } // Calls "udevadm trigger --action=change" on the specified drive. // drivePath must be the the block device path to trigger on, in the format "/dev/sd*", or a symlink to it. // This is workaround for Issue #7972. Once the underlying issue has been resolved, this may be removed. func udevadmChangeToDrive(drivePath string) error { glog.V(5).Infof("udevadmChangeToDrive: drive=%q", drivePath) // Evaluate symlink, if any drive, err := filepath.EvalSymlinks(drivePath) if err != nil { return fmt.Errorf("udevadmChangeToDrive: filepath.EvalSymlinks(%q) failed with %v.", drivePath, err) } glog.V(5).Infof("udevadmChangeToDrive: symlink path is %q", drive) // Check to make sure input is "/dev/sd*" if !strings.Contains(drive, diskSDPath) { return fmt.Errorf("udevadmChangeToDrive: expected input in the form "%s" but drive is %q.", diskSDPattern, drive) } // Call "udevadm trigger --action=change --property-match=DEVNAME=/dev/sd..." _, err = exec.New().Command( "udevadm", "trigger", "--action=change", fmt.Sprintf("--property-match=DEVNAME=%s", drive)).CombinedOutput() if err != nil { return fmt.Errorf("udevadmChangeToDrive: udevadm trigger failed for drive %q with %v.", drive, err) } </ins> return nil }

The "" E2E test will sometimes fail because it fails to delete the PD used in the test after the test is finished. The error: Detailed test log: The disk that fails to delete still shows up in : My attempts to manually try to delete the PD fail with: (the work around for ) doesn't change anything here:
is the disk still mounted ? It looks like the detach call did not succeed for some reason
I'll have to get another repro and verify.
okay cool. If you get a repro check to see if the disk is still mounted. If so then the cleanup code is not being called. If not then the clean up started, detach was called but failed for some reason; look in the logs for something like "googleapi: some error about detaching". It can also just be a race condition. That is, detach was called but is taking too long; if the disk does not detach then you cannot delete it.


Question: }, // -- <ins> // k8s.io/kubernetes/pkg/apis/flowcontrol/v1alpha1 gvr("flowcontrol.apiserver.k8s.io", "v1alpha1", "flowschemas"): { Stub:             `{"metadata": {"name": "va1"}, "spec": {"priorityLevelConfiguration": {"name": "name1"}}}`, ExpectedEtcdPath: "/registry/flowschemas/va1", }, // -- // k8s.io/kubernetes/pkg/apis/flowcontrol/v1alpha1 gvr("flowcontrol.apiserver.k8s.io", "v1alpha1", "prioritylevelconfigurations"): { Stub:             `{"metadata": {"name": "conf1"}, "spec": {"type": "Limited", "limited": {"assuredConcurrencyShares":3, "limitResponse": {"type": "Reject"}}}}`, ExpectedEtcdPath: "/registry/prioritylevelconfigurations/conf1", }, // -- </ins> // k8s.io/kubernetes/pkg/apis/storage/v1beta1 gvr("storage.k8s.io", "v1beta1", "volumeattachments"): { Stub:             `{"metadata": {"name": "va2"}, "spec": {"attacher": "gce", "nodeName": "localhost", "source": {"persistentVolumeName": "pv2"}}}`,

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: I am testing API Priority and Fairness. I launched it like this: After the logging said the cluster is running and told me how to set KUBECONFIG, I did so in another shell and went looking for FlowSchema objects. But kubectl erroneously told me there is no such thing. However, looking in the kube-apiserver log shows flowschemas and prioritylevelconfigurations being read and written; for example: Also, I am able to read them with against the insecure port. Interestingly, the kube-apiserver log also includes this complaint: I will attach the kube-apiserver log. What you expected to happen: I expected to list them. How to reproduce it (as minimally and precisely as possible): I think the report above is pretty minimal. Anything else we need to know?: Environment: Kubernetes version (use ): this is the branch, git commit 916edd922e528f2f27542d3ca5967ca11fe153f8Cloud provider or hardware configuration: bare metal, intel, 16 "CPUs", 64 GB of memory OS (e.g: ): 18.04.2 LTS (Bionic Beaver) Kernel (e.g. ): Linux mjs-dev-1 4.15.0-74-generic -Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019 x8664 x8664 x86_64 GNU/Linux Install tools: hack/local-up- Network plugin and version (if this is a network-related bug): Others: Docker version 18.09.7 go version go1.13.4 linux/amd64
/sig api-machinery
/cc /cc /cc /cc
/assign
the "" API group is not included in the list of known groups to register at startup: That means resources inside that group will not appear in discovery, and clients like kubectl will not know how to find them.
You can see all resources visible via discovery by running
just as said, for now we have to manually create an apiservice into the cluster to get kubectl work.

i dont think we're adding this to the list until the beta stage, can we make an exception for the case?
It is fine to add alpha things to the list I linked so they get created automatically if needed. They only get registered if the relevant runtime-config is enabled.
Interrupts control plane upgrade in lazy scenario where it's assumed dns name isn't changed properly to active node or or single node control plane
oh cool actually it succeeds on the second main node but the kube apiserver pod doesnt start
the hacky way being to adjust that api creation based on this?
And I removed the flags too
Wait what is this? This was to my api manifest during kubeadm upgrade?


Question: done } <ins> function kube::release::md5() { if which md5 >/dev/null 2>&1; then md5 -q "$1" else md5sum "$1" | awk '{ print $1 }' fi } </ins> # This will take binaries that run on master and creates Docker images # that wrap the binary in them. (One docker image per binary) function kube::release::create_docker_images_for_server() { # Create a sub-shell so that we don't pollute the outer environment ( <del> local binary_name; </del> <ins> local binary_name </ins> for binary_name in "${KUBE_DOCKER_WRAPPED_BINARIES[@]}"; do <del> echo "+++ Building docker image: ${binary_name}"; local docker_file_path="$1/Dockerfile"; local binary_file_path="$1/${binary_name}"; if [ -f ${docker_file_path} ]; then rm ${docker_file_path}; fi; printf " FROM scratch n ADD ${binary_name} /${binary_name} n ENTRYPOINT [ "/${binary_name}" ]n" >> ${docker_file_path}; local md5_sum=$(md5sum ${binary_file_path} | awk '{print $1}') local docker_image_tag=gcr.io/google_containers/$binary_name:$md5_sum docker build -t "${docker_image_tag}" ${1}; docker save ${docker_image_tag} > ${1}/${binary_name}.tar; echo $md5_sum > ${1}/${binary_name}.docker_tag; rm ${docker_file_path}; </del> <ins> kube::log::status "Starting Docker build for image: ${binary_name}" ( local docker_file_path="$1/${binary_name}.Dockerfile" local binary_file_path="$1/${binary_name}" if [ -f ${docker_file_path} ]; then rm ${docker_file_path} fi printf " FROM scratch n ADD ${binary_name} /${binary_name} n ENTRYPOINT [ "/${binary_name}" ]n" >> ${docker_file_path} local md5_sum=$(kube::release::md5 ${binary_file_path}) local docker_image_tag=gcr.io/google_containers/$binary_name:$md5_sum docker build -q -f "${docker_file_path}" -t "${docker_image_tag}" ${1} >/dev/null docker save ${docker_image_tag} > ${1}/${binary_name}.tar echo $md5_sum > ${1}/${binary_name}.docker_tag rm ${docker_file_path} ) & </ins> done <ins> wait || { kube::log::error "previous Docker build failed"; return 1; } kube::log::status "Docker builds done" </ins> ) }

Filing for This code is wrong The local needs to be on a separate line from the command execution, and we need to be resilient to not existing on OS X, like:
For once I was actually watching the output of a build and noticed that I was seeing an error about md5sum not being found. I'm building on a mac using boot2docker. The short_hash function () works fine on my mac as md5 does exist.
Actually, I'm just going to fix this as a follow-on to .
Thank you!


Question: Name: "should-be-removable-mutating-webhook.k8s.io", Rules: []v1beta1.RuleWithOperations{{ Operations: []v1beta1.OperationType{v1beta1.Create}, <ins> // This will not match any real resources so this webhook should never be called. </ins> Rule: v1beta1.Rule{ <del> APIGroups:   []string{"*"}, APIVersions: []string{"*"}, Resources:   []string{"*"}, </del> <ins> APIGroups:   []string{""}, APIVersions: []string{"v1"}, Resources:   []string{"invalid"}, </ins> }, }}, ClientConfig: v1beta1.WebhookClientConfig{

This flaked 3/4 times in the last week. /priority failing-test /priority important-soon /sig api-machinery /kind flake
/assign
Pinging here for latest status on this. This test is now one of our top flakes failing pull-kubernetes-e2e-gce job 40 times.
Here's the latest test run results for this test to get more logs. https://k8s-
The error message is really similar to the one in I'm looking into if the fix for that will also be sufficient to fix this.
We may want to reopen this until the fix is shown to actually work /reopen
Thanks for taking care of this. I will monitor this test and ping you back here to close when appropriate.
No failure 2 days since the fix. the fix seemed to have worked. So please go ahead and close the issue. Thanks much.
/close


Question: out.CloudProvider = in.CloudProvider out.NodeName = in.NodeName out.AuthorizationModes = *(*[]string)(unsafe.Pointer(&in.AuthorizationModes)) <ins> out.PrivilegedPods = in.PrivilegedPods </ins> out.Token = in.Token out.TokenTTL = (*v1.Duration)(unsafe.Pointer(in.TokenTTL)) out.APIServerExtraArgs = *(*map[string]string)(unsafe.Pointer(&in.APIServerExtraArgs))

Dear All: I'm not sure if I'm right. When i use whith .I found controller-manager logs Someone can help me? Thanks a lot.
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Please give more information about your set up cc
Sorry for report later. My openstack didn't support metadata service,but support config-drive service. More info:
Thanks Based on this, I think it's an openstack cloudprovider problem more than a kubeadm problem...
Thank you for help me. I will change the label.
/area cloudprovider
/remove-kind support
Should be fixed with you can specify a search order for metadata and config drive /close
Specifying a search order doesn't really solve the problem, especially if the metadata api isn't available which is what ZhenyangZhao said . I'm getting the same issue and since the config drive is my only option at the moment.
Like for this to be /open
can you please provide logs/versions for your problem in your environment? Thanks, Dims
Let me know if you need anything else Kubernetes version: 1.8.4 Docker version: 17.09.0-ce Linux: CentOS 7.4 Looking at the docker container log for the api-server (I don't have problems with the controller-manager):
and share the following:
Hi! Thank you for filing this issue. We appreciate your interest in Kubernetes and kubeadm. In order to make the kubernetes/kubernetes issue tracker more usable, we're routing issues to different repositories with different focuses. I think this issue belongs in the kubeadm repository. Please move this issue to the in order for the kubeadm team to be able to track this efficiently. If you disagree with my assessment of moving this issue, please speak up and/or reopen the issue. Finally, please try to reproduce the bug with the latest Kubernetes version, it might as well already have been fixed :) Thank you and have a good day!


Question: // TODO: document the feature (owning SIG, when to use this feature for a test) PodResources = framework.WithNodeFeature(framework.ValidNodeFeatures.Add("PodResources")) <ins> // ResourceHealthStatus (SIG-node, resource health Status for device plugins and DRA <https://kep.k8s.io/4680>) ResourceHealthStatus = framework.WithNodeFeature(framework.ValidNodeFeatures.Add("ResourceHealthStatus")) </ins> // TODO: document the feature (owning SIG, when to use this feature for a test) ProcMountType = framework.WithNodeFeature(framework.ValidNodeFeatures.Add("ProcMountType"))

July 24th. I think there is some suble way the test is named. Test is covered by and that works. I think this is mainly an issue where this test should not be run with these jobs. /sig node
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
what is the correct way to label a job now? This is apparently wrong and causing the job to be picked up in the unlabeled name.
You still need to have a corresponding feature in because, as you noticed, there are jobs which don't support skipping based on the feature gate.
I created an issue in test-infra. once all tasks are completed, it's okay only to use framework.WithFeatureGate. There may be a lot of jobs that need to be updated. for now, you have to add a feature label or modify all the failing jobs.
/assign
xref


Question: handler := proxy.NewUpgradeAwareHandler(location, proxyRoundTripper, true, upgrade, &responder{w: w}) handler.InterceptRedirects = utilfeature.DefaultFeatureGate.Enabled(genericfeatures.StreamingProxyRedirects) handler.RequireSameHostRedirects = utilfeature.DefaultFeatureGate.Enabled(genericfeatures.ValidateProxyRedirects) <ins> utilflowcontrol.RequestDelegated(req.Context()) </ins> handler.ServeHTTP(w, newReq) }

On a cluster with many APIs provided by aggregated apiservers, with controllers managing those APIs, the number of occupied seats effectively has a floor of 1 per watch. If the combined max-inflight limit is reduced low enough, priority levels tend to become saturated by these long-running watch requests and other requests are starved. Watch requests consume a seat only during initialization. a PriorityLevelConfiguration and a corresponding FlowSchema to match watches against a resource served by an aggregated apiserver. some watch requests. the metric climb on the priority level per watch until all seats are exhausted. Kubernetes version (use ): (OpenShift 4.9) /sig api-machinery
cc
D'oh! I can see two obvious possible ways to cope: one apiserver not manage watch requests the are delegated to another apiserver; or a marker into the notification stream so that the non-origin apiservers can detect when initialization is done.
I would even say that for regular (non-long-running) requests, it's not obvious that they should be managed by P&F even. [I agree that for watch initialization it's particularly bad though.] - do we have a simple way to detect if the request will be served from our kube-apiserver or from some other aggregated apiserver?
There is not currently a way to know that ahead of time in the filters today. It require a list/watch, but it ought not be excessively expensive to make a 99% correct prediction, but the actual decision is made by the aggregated apiserver later in the request.
/triage accepted
: is there a way that we can plumb that knowledge forward to the filters?


Question: local -r file="${1:-}" local -r prefix="${2:-}" local -r suffix="${3:-}" <ins> local -r dirname="$(dirname ${file})" local -r tmpfile="$(mktemp -t filtered.XXXX --tmpdir=${dirname})" </ins> touch "${file}" <del> awk "substr($0,0,length("${prefix}")) != "${prefix}" { print }" "${file}" > "${file}.filtered"  && mv "${file}.filtered" "${file}" echo "${prefix}${suffix}" >> "${file}" </del> <ins> awk "substr($0,0,length("${prefix}")) != "${prefix}" { print }" "${file}" > "${tmpfile}" echo "${prefix}${suffix}" >> "${tmpfile}" mv "${tmpfile}" "${file}" </ins> } function create-node-pki {

/sig cluster-lifecycle
Very rarely we are seeing known tokens files filled with the NULL character, with ASCII value 000. Restarting the master doesn't fix the issue because the is saved between restarts.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale


Question: local -r file="${1:-}" local -r prefix="${2:-}" local -r suffix="${3:-}" <ins> local -r dirname="$(dirname ${file})" local -r tmpfile="$(mktemp -t filtered.XXXX --tmpdir=${dirname})" </ins> touch "${file}" <del> awk "substr($0,0,length("${prefix}")) != "${prefix}" { print }" "${file}" > "${file}.filtered"  && mv "${file}.filtered" "${file}" echo "${prefix}${suffix}" >> "${file}" </del> <ins> awk "substr($0,0,length("${prefix}")) != "${prefix}" { print }" "${file}" > "${tmpfile}" echo "${prefix}${suffix}" >> "${tmpfile}" mv "${tmpfile}" "${file}" </ins> } function create-node-pki {

/sig cluster-lifecycle
Very rarely we are seeing known tokens files filled with the NULL character, with ASCII value 000. Restarting the master doesn't fix the issue because the is saved between restarts.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale


Question: local -r file="${1:-}" local -r prefix="${2:-}" local -r suffix="${3:-}" <ins> local -r dirname="$(dirname ${file})" local -r tmpfile="$(mktemp -t filtered.XXXX --tmpdir=${dirname})" </ins> touch "${file}" <del> awk "substr($0,0,length("${prefix}")) != "${prefix}" { print }" "${file}" > "${file}.filtered"  && mv "${file}.filtered" "${file}" echo "${prefix}${suffix}" >> "${file}" </del> <ins> awk "substr($0,0,length("${prefix}")) != "${prefix}" { print }" "${file}" > "${tmpfile}" echo "${prefix}${suffix}" >> "${tmpfile}" mv "${tmpfile}" "${file}" </ins> } function create-node-pki {

/sig cluster-lifecycle
Very rarely we are seeing known tokens files filled with the NULL character, with ASCII value 000. Restarting the master doesn't fix the issue because the is saved between restarts.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale


Question: func observeScheduleAttemptAndLatency(result, profile string, duration float64) { e2eSchedulingLatency.WithLabelValues(result, profile).Observe(duration) <ins> schedulingLatency.WithLabelValues(result, profile).Observe(duration) </ins> scheduleAttempts.WithLabelValues(result, profile).Inc() }

I would like to propose the graduation of the following scheduler metrics from alpha to stable: These metrics have been stable for at least 15 months.
/sig scheduling /sig instrumentation
/cc
IIRC, the last addition to these metrics was the profile name label. I don't foresee need for new labels. I don't see many precedents for graduation in core components. are there any defined criteria for graduation?
As requested, the following is how long it has been since a change has been made to each of the requested metrics: - 2 years; last change was to migrate to the metrics stability framework - 17 months; last change was a renaming of the metric - 15 months; last change was a renaming of the metric - 17 months; last change was to add a label - 17 months; last change was to add a label
It's up to the component owners to commit to graduating these metrics, since the component owners own the metrics and therefore need to agree to support them in accordance to stability requirements. If you guys are okay with promoting these metrics, I don't see a reason, from an instrumentation side, why we would object to promotion.
+1 to graduation then. These metrics are very mature. Although, I'm not sure if all metrics already adhere to naming conventions. and don't have a unit suffix, but they are unit-less. Is that fine
is a gauge so it's fine, is a histogram so it's also fine.
I'm recently working with metrics, I'm glad to provide some help if no one has interest about it.
There is already an open PR for this.
Oh sorry, didn't notice about that. Forget it.
These graduations sound good to me, but I would be inclined to rename . I feel that the part of the metric is unnecessary. Having instead and keeping for the algorithm latency is already descriptive enough to know the difference between both. If that's not the case, then the text of the metric can be used to guide the users.
I agree that is rather confusing, so +1 to removing the prefix. The metric doesn't include retries, for example. We have a separate metric that includes retries: We need to rename that one eventually to e2e, but it can only be done after the existing one has been deprecated and deleted for a few releases. /triage accepted
/assign
/cc
The PR for this is ready to be reviewed:


Question: <del> #!/usr/bin/python # Copyright 2015 Google Inc. All rights reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # #     http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. import os import sys import yaml def mutate_env(path, var, value): # Load the existing arguments if os.path.exists(path): args = yaml.load(open(path)) else: args = {} args[var] = value yaml.dump(args, stream=open(path, 'w'), default_flow_style=False) if __name__ == "__main__": mutate_env(sys.argv[1], sys.argv[2], sys.argv[3]) </del>

I tried to spin up GCE kube environment using kube-up and got the following error: The script would then hang on a subsequent command. The verify-prereqs for kube-up should require yaml support for python and fail early. To rectify, I had to do the following:
Thanks for the report Derek!
Yeah, this broke our Jenkins a while ago, and I thought I fixed it, but didn't. The real fix is in


Question: } } <del> func (h *hostpathDriver) GetDriverInfo() *DriverInfo { </del> <ins> func (h *hostPathDriver) GetDriverInfo() *DriverInfo { </ins> return &h.driverInfo } <del> func (h *hostpathDriver) SkipUnsupportedTest(pattern testpatterns.TestPattern) { </del> <ins> func (h *hostPathDriver) SkipUnsupportedTest(pattern testpatterns.TestPattern) { </ins> } <del> func (h *hostpathDriver) GetVolumeSource(readOnly bool, fsType string) *v1.VolumeSource { // hostpath doesn't support readOnly volume </del> <ins> func (h *hostPathDriver) GetVolumeSource(readOnly bool, fsType string) *v1.VolumeSource { // hostPath doesn't support readOnly volume </ins> if readOnly { return nil }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: https://k8s- What happened: The skip check for hostpath doesn't seem to be working for these test cases
can you take a look?
Oh, disruptive tests are the tests that can't be tested locally in my environment due to I will take a look.
I found that above issue is not related, I should have been checked more carefully. Sorry for that.
I found out that this was just a driver name issue, so below will fix it: hostpath -hostPath hostpathSymlink -hostPathSymlink We might be better changing the skip logic later, to make it skip not by name in order to avoid this kind of bug, though. I will make a PR for above change, soon.
I made a PR above. Also, it might be a bit out of scope of this fix, but are drivers tested with containerized volume server, like nfs, ceph, etc..., safe to be tested with subpath disruptive tests? I'm worrying about the case that test pod and containerized volume server are deployed on the same node. In such a case, the same behaviour expected or do we need to avoid this kind of situation?


Question: /* Testname: watch-consistency <ins> Release : v1.15 </ins> Description: Ensure that concurrent watches are consistent with each other by initiating an additional watch for events received from the first watch, initiated at the resource version of the event, and checking that all resource versions of all events match. Events are produced from writes on a background goroutine. */ <del> It("should receive events on concurrent watches in same order", func() { </del> <ins> framework.ConformanceIt("should receive events on concurrent watches in same order", func() { </ins> c := f.ClientSet ns := f.Namespace.Name

We should add a watch test that: a watch a background thread, makes a bunch of random changes the RV from the first item received by the watch. each subsequent item X, open a new watch at the previous RV and verify that the first thing received matches X. Record X's RV for the next iteration. /sig api-machinery
cc
/assign cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
/area conformance
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
A test for this was via , we just need to promote it to be a conformance test


Question: close(rc.stopCPU) close(rc.stopMem) close(rc.stopCustomMetric) <ins> rc.stopWaitGroup.Wait() </ins> // Wait some time to ensure all child goroutines are finished. time.Sleep(10 * time.Second) kind, err := kindOf(rc.kind)

https://k8s-
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
might be a culprit, but maybe not
This could be related to how the test initializes the framework or cleans it up, maybe.
Another failure in the same run, not just this one test:
I'm not positive but the stack trace might be wrong due to inlining optimizations. rc.framework == nil was my first guess, but that would fail earlier in that method.
Framework AfterEach runs namespace deletion, which may mean that .Namespace is nil (if inlining is indeed present).
I am looking into this right now.
It's unrelated. I reverted from the current master HEAD locally (via ), then did and finally and it did not help. The test still fails with the same message "Expected <*v1.Initializers | 0x0: nil not to be nil" (location: )
Note that I have executed one of the initializers e2e test (not autoscaling), because it is one of the tests reported by https://k8s- as panicked.
Looking a bit further I found out that commit the test above as already failing.
I have just realized that I have discovered yesterday a race condition in ResourceConsumer implementation, and might actually make it worse. Let me quickly make a PR out of a fix.
Let me try to recap what I have found out so far: The test that is being referred to at the top of this issue () fails "out of the box" and probably deserves a separate issue to track it Other failing tests that look like they have a similar failure pattern (NPE) fail because of trying to send a "consume CPU" request to a non-existent pod (unrelated to above, as far as I can tell) PR might cause the effect above (and probably does) because it actually deletes the pods that should have been deleted However, the root cause is that there is a race condition inside ResourceConsumer that makes "consume CPU" requests being sometimes sent already after "CleanUp" got called (fix prepared in PR )
Initializer was fixed in a different PR, depending on what cluster you test on.

, Andrzej Wasylkowski < :
Yeah, confirmed initializers test is now passing on GCE and disabled on GKE (which doesn't enable initializers configuration group yet) , Clayton Coleman <: The test that is being referred to at the top of this issue () fails "out of the box" and probably deserves a separate issue to track it Other failing tests that look like they have a similar failure pattern (NPE) fail because of trying to send a "consume CPU" request to a non-existent pod (unrelated to above, as far as I can tell) PR cause the effect above (and probably does) because it actually deletes the pods that should have been deleted However, the root cause is that there is a race condition inside ResourceConsumer that makes "consume CPU" requests being sometimes sent already after "CleanUp" got called (fix prepared in PR ) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub , or mute the thread
Clayton, can you assign this to me? I cannot do this myself.


Question: // name to the state for that level.  Every name referenced from a // member of `flowSchemas` has an entry here. priorityLevelStates map[string]*priorityLevelState <del> // the most recent update attempts, ordered by increasing age. // Consumer trims to keep only the last minute's worth of entries. // The controller uses this to limit itself to at most six updates // to a given FlowSchema in any minute. // This may only be accessed from the one and only worker goroutine. mostRecentUpdates []updateAttempt // watchTracker implements the necessary WatchTracker interface. WatchTracker </del> } type updateAttempt struct {

Two flakes recently: They fail with error like: or /assign
/sig scalability
In logs I have found the following slow calls: It's a bug that we use exempt priority level for clusterloader: I created It's quite worrisome that APF handler takes 1s for exempt request.
I checked a APF code and I think the flow for exempt requests is: genericfilters.WithPriorityAndFairness() workEstimate := workEstimator.EstimateWork(r) fcIfc.Handle() cfgCtlr.startRequest() () looks for matching flowschemas in cfgCtrl.flowSchemas returns for exempt for other requests it calculates sha256 for flowDistinguisher and queues request req.Finish updates metrics calls execFn updates headers handler.ServeHTTP which stops the timer for priorityandfairness For me, the most likely reason for slowness is lock contention on We know that the test is super cpu-intensive in that part and other requests are e.g. calculating sha256 keeping that lock. A potential improvement would be to change this lock to RWLock or return the lock before starting calculating sha256 (if we believe that this is the most expensive operation there). /cc
/assign
I think the lock-contention is even worse than what you wrote, as e.g.: is happening under this lock (which is also dispatching requests). I will take a deeper look later this week, but I think that switching to RWLock might do the job without any invasive changes.
may trigger the execution of some requests that were waiting, but they are not executed in the same goroutine. Yes, I think changing that Mutex to a RWMutex would be an easy way to greatly reduce contention.
I agree switching to RWMutex is likely to help a lot, if that's not enough I think with a little work we could e.g. switch to s.
(perhaps because that's what my code originally did? Reviewers thought the retirement dance was too tricky.)
/triage accepted /priority important-soon I will work on the fix.
/sig api-machinery
Wow! Thanks a lot for rapid help here! Let's wait for the next test run to see if it helped. /reopen
Reopened this issue. <detailsIn response to : Wow! Thanks a lot for rapid help here! Let's wait for the next test run to see if it helped.

/reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Yes, can you please report the filter latency regardless of whether it is bad or good? I would like to know how much of a difference that PR made.
Sure, will report the latency change in both cases.
I compared (the one where we saw issue) (first with the lock improvements) I tried to find requests that: use exempt PL (so shouldn't be throttled) has fl_priorityandfairness, key which means that priority and fairness filter took more than 100ms Sample matching request: In the old run, I see ~6k such requests, with the locking improvements there are no such requests. The latency of PATCH jobs seems to improve significantly: ! So overall it looks great. The only worrisome metric I see is that actual P&F throttling for other requests has grown: With locking improvements: ! Old: ! E.g. 'node-high' has now throttling latency of ~1.5s in spikes where in the previous test run it was more like ~800ms. I don't think the lock improvements touched this directly. I expect that this is either: per-test variance (if we rerun the old run again we would see different throttling latencies as well) we throttle exempt requests less (as expected) giving less resources to other requests. Anyway, I don't think this change is meaningful as per-test latency aggregates for PATCH nodes/status and PUT leases (the most common requests in node-high) look ~same as previously: ! ! I think we can close this for now as the issue seems to be resolved. Thanks a lot for your help here! /close
Closing this issue. <detailsIn response to : I compared In the old run, I see ~6k such requests, with the locking improvements there are no such requests. The latency of PATCH jobs seems to improve significantly: ! ! ! I expect that this is either: ! ! /close </details
: thanks for reporting!


Question: return []string{"crd", "crds"} } <ins> // Implement CategoriesProvider var _ rest.CategoriesProvider = &REST{} // Categories implements the CategoriesProvider interface. Returns a list of categories a resource is part of. func (r *REST) Categories() []string { return []string{"api-extensions"} } </ins> // Delete adds the CRD finalizer to the list func (r *REST) Delete(ctx context.Context, name string, deleteValidation rest.ValidateObjectFunc, options *metav1.DeleteOptions) (runtime.Object, bool, error) { obj, err := r.Get(ctx, name, &metav1.GetOptions{})

What would you like to be : Adding a "webhooks" group to kubectl to show mutating/validating webhooks would be super useful Why is this needed: There are a few reasons this is useful (to me): often cannot remember the right names for these resources, are very long and I often have a typo often want to enumerate all of them on the system. cc
/sig cli
I think this is a great idea, definitely something I could've used in the past. I'd be interested in taking a pass at implementing this if the proposal gets accepted.
Instead of building a category based on the types we have, I would build the category based on why people look at them. I suspect a category like which returned admission config admission config for various CSI types to list in CRD would be very helpful.
Webhook is far too generic. By putting admission webhooks in there we basically steal that word from the ecosystem to be used for other purposes. "admissionwebhooks" I could agree with.
Yeah, there are other webhooks, e.g. CRD conversion webhooks or authz webhooks...
I'd expect the short names "mwc" and "vwc" to be set up, are they not?
Unfortunately, they are not.
/triage accepted /assign
does category tolerate namespacing? i.e. would be tolerated? Still a bit long, but shorter than
It's a string, so you could theoretically namespace that.


Question: // test data newEtcdTestStorage := func(t *testing.T, prefix string) (*etcd3testing.EtcdTestServer, storage.Interface) { server, _ := etcd3testing.NewUnsecuredEtcd3TestClientServer(t) <del> storage := etcd3.New(server.V3Client, apitesting.TestCodec(codecs, example.SchemeGroupVersion), func() runtime.Object { return &example.Pod{} }, prefix, schema.GroupResource{Resource: "pods"}, identity.NewEncryptCheckTransformer(), true, etcd3.NewDefaultLeaseManagerConfig()) </del> <ins> storage := etcd3.New(server.V3Client, apitesting.TestCodec(codecs, examplev1.SchemeGroupVersion, example2v1.SchemeGroupVersion), func() runtime.Object { return &example.Pod{} }, prefix, schema.GroupResource{Resource: "pods"}, identity.NewEncryptCheckTransformer(), true, etcd3.NewDefaultLeaseManagerConfig()) </ins> return server, storage } server, etcdStorage := newEtcdTestStorage(t, "")

The test is flaking: /assign /sig api-machinery /priority important-soon
Ok - it's fairly easy to reproduce by just adding into the test I will validate the fix.
/triage accepted


Question: <ins> apiVersion: test.com/v1 kind: Example metadata: name: test namespace: default spec: test: test </ins>

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --Changing the scope field of a CRD ist not possible since it's immutable. But even after deleting and re-creating the CRD with changed scope it doesn't let me create resources of the CRD's kind. Re-creating CRD with changed scope should just work. Here is a full example that in the end leads to the error when trying to create a resource after re-creating the CRD. Kubernetes version (use ): v1.21.4 Cloud provider or hardware configuration: vSphere 7.0 OS (e.g: ): Centos 7 Kernel (e.g. ): 3.10.0-1160.42.2.el7.x86_64
/sig api-machinery
/assign /triage accepted
This appears to be working as intended. The example custom resource you use: It Cluster scoped because it has no namespace. But once the CRD is namespace scoped, you should not be able to create cluster scoped custom resources of that type. If instead you add a namespace to Example, it should be created, e.g.:
if the above seems to have answered your question - please feel free to close the issue :) Thank you!
kubectl will automatically use the namespace so this is definitely not the issue here. I can reproduce this issue with my above script in a fresh (v1.22.2) cluster. You can add the namespace in the last manifest, it makes no difference. From my understanding this really seems to be a bug.
Yes, that's true. Good catch. Okay, we need to root cause this.
I tried using curl to create the custom resource directly against the API and got the same result: So the problem seems to be in the API server (not in kubectl).
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale


Question: }{ {name: "one service", services: []string{service1}}, {name: "two services", services: []string{service1, service2}}, <ins> {name: "dot service", services: []string{service3}}, {name: "underscore service", services: []string{service4}}, {name: "at service", services: []string{service5}}, {name: "colon service", services: []string{service6}}, </ins> {name: "invalid service new line", services: []string{invalid1}, wantErr: true}, {name: "invalid service with dot", services: []string{invalid2}, wantErr: true}, {name: "long service", services: []string{strings.Repeat(service1, 100)}, wantErr: true}, <del> {name: "max number of services", services: []string{service1, service2, service1, service2, service1}, wantErr: true}, </del> <ins> {name: "max number of services", services: []string{service1, service2, service3, service4, service5}, wantErr: true}, </ins> } for _, tt := range tests { errs := validateServices(tt.services)

I enabled the on EKS 1.27.4 using Ubuntu 20.04.6. When I try to query the kubelet logs or log files from the system I get the error "input contains unsupported characters" The systemd kubelet service runs as a snap called and I think that might be the source of this problem (running the kubelet in a snap), but I'm not sure if it's intended to work. The fact that I couldn't see /var/log file query was surprising and the error message is misleading which is why I'm opening an issue here. I followed the same configuration steps on Amazon Linux 2 and it works as expected. Log query should work on Ubuntu or provide a clearer error message. This should be reproducible on any Kubernetes cluster running Ubuntu with Kubelet in a snap, but this is what I did. Install to make cluster creation and karpenter setup easier) Install Karpenter as a node autoscaler. This will automatically create a provisioner and AWSNodeTemplate objects. Modify the AWSNodeTemplate with userdata that can customize Ubuntu with log query enabled Deploy a workload that uses hostname spread to get more nodes. When the new Ubuntu node is created you can log into the node with session manager. I like to use You can verify that the kubelet is running and logging is enabled Now you can try to query logs This works with AL2. You can test with a different AWSNodeTemplate that uses AL2 instead <details</details<detailsEKS (but should happen on any provider) </details<details</details<details</details<detailscontainerd </details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
it's always best to talk to the product support before opening the upstream ticket (this repository in this case). the k8s docs do not have a guide for running the kubelet as a snap. it could be an entirely snap related bug, indeed. please communicate with EKS and the owners of the snap package.

if they confirm a kubernetes/kubernetes bug, you can reopen this ticket. thanks
/sig node /triage needs-information /close to discuss with other users:
Closing this issue. <detailsIn response to : /sig node /triage needs-information /close </details
After digging into the error a bit more I found the problem was actually the regex match for the service name. I opened to fix it.


Question: }, ) <del> docker_build( name = "busybox-net", base = ":busybox", debs = [ "@libc_deb//file", "@iptables_deb//file", "@iproute2_deb//file", "@libnetlink_deb//file", "@libxtables_deb//file", ], ) </del> # This list should roughly match kube::build::get_docker_wrapped_binaries() # in build/common.sh. DOCKERIZED_BINARIES = {

BUG REPORT Kubernetes version 2 Environment: Install tools: What happened: The code in takes care to supply the flag so it will wait on a lock to avoid interference between multiple processes modifying the table. If you read the source you discover it does this by creating a file . However, when is running in a container, is in a local filesystem, so it is not able to lock against processes running on the host or in other containers. And before you decide that the answer is to mount from the host, look at all the issues around leaking netns, e.g. Credit for pointing this out to me. Credit for pointing out locking functionality was to a few months back
Ironically, this is caused by a bugfix that moved to file-based locking. Previously, iptables would lock by reserving an abstract unix socket, which is tied to the network namespace.
Some discussion also in and following comments.
I experimented with this: but it didn't work on CoreOS, as the host wasn't using a version of iptables that used the file lock yet. Just ended up creating a directory call
I guess I should file this as an issue against kubeadm and any of the other sources of kube-proxy container config.
/assign
Should we leave this open (waiting for ... ?)
Waiting for I guess... (which I'm reviewing now)
i think this should not be marked as complete until the kubeadm fixes are in.
/reopen
/close fix in


Question: const ( // When kubelet is started with the "external" cloud provider, then // it sets this annotation on the node to denote an ip address set from the <del> // cmd line flag. This ip is verified with the cloudprovider as valid by </del> <ins> // cmd line flag (--node-ip). This ip is verified with the cloudprovider as valid by </ins> // the cloud-controller-manager AnnotationProvidedIPAddr = "alpha.kubernetes.io/provided-node-ip" )

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: When using the flag on the kubelet, the node's list no longer reports , , addresses. What you expected to happen: That I can explicitly set to an IP, but the other addresses will still show up. How to reproduce it (as minimally and precisely as possible): Create a new kops cluster, modify the kubelet to use the flag with the eth0 on the node. Run Anything else we need to know?: This change will probably take place in  This came up as a result of Environment: Kubernetes version (use ): 1.9.x Cloud provider or hardware configuration: AWS OS (e.g. from /etc/os-release): any Kernel (e.g. ): any Install tools: kops /sig network /sig node I'll make a PR for this change shortly


Question: func verifyPrefixOfSecretResource(t *testing.T, wantPrefix string, test *transformTest) { // implementing this brute force approach instead of fancy channel notification to avoid test specific code in prod. // wait for config to be observed <del> verifyIfKMSTransformersSwapped(t, wantPrefix, "", test) </del> <ins> verifyIfKMSTransformersSwapped(t, wantPrefix, test) </ins> // run storage migration secretsList, err := test.restClient.CoreV1().Secrets("").List(

transformation expandless | 14m1s Failed; === RUN TestEncryptionConfigHotReload TestEncryptionConfigHotReload When I see this flake, 10-09 is the first one.(But the job history in the testgrid only started from 10-03) some PR that may be related: /sig auth /cc
/cc
/triage accepted
/cc failed two times in recent 3 runs:
/reopen as of this is still .
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: t.Fatal("receive result timeout") } w.Stop() <ins> wg.Wait() </ins> } }

cc ref
There are no sig labels on this issue. Please add a sig label by either: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. See the . The in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
- can you please take a look, you introduced this test and it may mean real problem with your code
I took a quick look and this is a test issue. The problem is that the implementation explicitly assumes there is at most one call to dispatchEvent at a given time. And this particular test breaks this assumption: The problem is that, a given iteration is finished when is closed, which may actually happen as a result of even before is called (since it's a different goroutine). And then may be called concurrently with the next iteration, which triggers a race. The fix is simple, we should ensure that all goroutines are finished before starting the next iteration.
/cc


Question: return nil } <del> // waitForHTTPServers waits for all webservers to be up, on all protocols, and then validates them using the same probe logic as the rest of the suite. </del> <ins> // waitForHTTPServers waits for all webservers to be up, on all protocols sent in the input,  and then validates them using the same probe logic as the rest of the suite. </ins> func (k *kubeManager) waitForHTTPServers(model *Model) error { const maxTries = 10 framework.Logf("waiting for HTTP servers (ports 80 and 81) to become ready") testCases := map[string]*TestCase{} for _, port := range model.Ports { <ins> // Protocols is provided as input so that we can skip udp polling for windows </ins> for _, protocol := range model.Protocols { fromPort := 81 desc := fmt.Sprintf("%d->%d,%s", fromPort, port, protocol)

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Ive done preliminary testing with on calico powered clusters on some windows variants and found that the probing seems to work. Can we add wrapper to netpol suite so that we can run it on windows clusters ? Why is this needed: We need this because otherwise, CNIs such as EKS will give us This error is caused when, for example, pods that are landing on a windows node havent defined their target platform. Somehow i guess pods which are landing on linux nodes dont have this issue. Its also needed for the obvious reason that we probably want separate linux and windows test suites for policies, in general, even though we expect the logic to be identical (modulo hostports , which we dont currently test)
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig windows
/assign
grepping around, I can't figure out where those are supposed to go. Can you give me pointers?
(resolved in slack, matts digging into this)
ok, these are running now on eks, lots of failures. will dig
/assign
the netpol are not being applied for the ns: Instead of failing because should not be able to access , the connection succeeds.


Question: if add { if clientTraffic { klog.V(2).Infof("Adding rule for client MTU discovery from the network load balancer (%s) to instances (%s)", clientCidrs, instanceSecurityGroupID) <del> klog.V(2).Infof("Adding rule for client traffic from the network load balancer (%s) to instances (%s)", clientCidrs, instanceSecurityGroupID) </del> <ins> klog.V(2).Infof("Adding rule for client traffic from the network load balancer (%s) to instances (%s), port (%v)", clientCidrs, instanceSecurityGroupID, port) </ins> } else { <del> klog.V(2).Infof("Adding rule for health check traffic from the network load balancer (%s) to instances (%s)", clientCidrs, instanceSecurityGroupID) </del> <ins> klog.V(2).Infof("Adding rule for health check traffic from the network load balancer (%s) to instances (%s), port (%v)", clientCidrs, instanceSecurityGroupID, port) </ins> } } else { if clientTraffic { klog.V(2).Infof("Removing rule for client MTU discovery from the network load balancer (%s) to instances (%s)", clientCidrs, instanceSecurityGroupID) <del> klog.V(2).Infof("Removing rule for client traffic from the network load balancer (%s) to instance (%s)", clientCidrs, instanceSecurityGroupID) </del> <ins> klog.V(2).Infof("Removing rule for client traffic from the network load balancer (%s) to instance (%s), port (%v)", clientCidrs, instanceSecurityGroupID, port) </ins> } <del> klog.V(2).Infof("Removing rule for health check traffic from the network load balancer (%s) to instance (%s)", clientCidrs, instanceSecurityGroupID) </del> <ins> klog.V(2).Infof("Removing rule for health check traffic from the network load balancer (%s) to instance (%s), port (%v)", clientCidrs, instanceSecurityGroupID, port) </ins> } if clientTraffic {

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: After a rolling update of the cluster, NLB kept failing health checks because security group did not have rules for new EC2 instance. What you expected to happen: Security group should have been updated with both TCP rules for NLBs to work correctly. How to reproduce it (as minimally and precisely as possible): Create a service object with type LoadBalancer and adding the annotation for NLBs. FWIW, I had 6 services and each had their own NLBs. It’s possible that the bug happens when there are multiple NLBs causing multiple AWS updates. Perform a rolling update of the instance group. This causes the instances to be terminated and new ones to be brought up. AFAICT, this should result in two changes to the security group per EC2 instance. When the instance is terminated, the rules for that nodePort should be deleted. When the new node is created, the rule should be again. However, after the rolling update, the rule doesn’t get . As a result, the NLB’s health checks fail. Anything else we need to know?: As mentioned earlier, this happened on a cluster where there were multiple NLBs. This doesn’t seem to be a deterministic failure. The log has the following error message:I manually edited the Service object to see if it would simply retrigger an update on the cloud resources. It actually did. The rules got to the security group. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: AWS OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: kops 1.8.1 Others:
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
did you find any workarounds? we've run into this issue, where the security group rules got deleted. Specifically, we found that the security group missed the following: 1) TCP rule to allow traffic from NLB to Node Port 2) TCP rule to allow traffic from NLB to Healthcheck Port We these manually back in to fix, but were surprised these were deleted in the first place.

This stackoverflow also is relevant:
One workaround was to simply refresh the Kubernetes service object. We /updated a label in the Kubernetes service object and found that it would make the corresponding changes in AWS.
thanks have you been able to root cause why the rule doesn't get ? We're trying to re-produce the failure ourselves, too.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Any update on this bug fix release please?
Sorry to bother you with commenting on this issue. But I cannot find out in what version of Kubernetes this is expected to be released / was released. Any clarification highly appreciated.


Question: driverNodeID, maxVolumePerNode, accessibleTopology, err := csi.NodeGetInfo(ctx) if err != nil { <del> unregisterDriver(pluginName) return fmt.Errorf("error during CSI NodeGetInfo() call: %v", err) </del> <ins> glog.Error(log("registrationHandler.RegisterPlugin failed at CSI.NodeGetInfo: %v", err)) if unregErr := unregisterDriver(pluginName); unregErr != nil { glog.Error(log("registrationHandler.RegisterPlugin failed to unregister plugin due to previous: %v", unregErr)) return unregErr } return err </ins> } err = nim.AddNodeInfo(pluginName, driverNodeID, maxVolumePerNode, accessibleTopology) if err != nil { <del> unregisterDriver(pluginName) return fmt.Errorf("error updating CSI node info in the cluster: %v", err) </del> <ins> glog.Error(log("registrationHandler.RegisterPlugin failed at AddNodeInfo: %v", err)) if unregErr := unregisterDriver(pluginName); unregErr != nil { glog.Error(log("registrationHandler.RegisterPlugin failed to unregister plugin due to previous error: %v", unregErr)) return unregErr } return err </ins> } return nil } <del> // DeRegisterPlugin is called when a plugin removed it's socket, signaling </del> <ins> // DeRegisterPlugin is called when a plugin removed its socket, signaling </ins> // it is no longer available <del> // TODO: Handle DeRegistration </del> func (h *RegistrationHandler) DeRegisterPlugin(pluginName string) { <ins> glog.V(4).Info(log("registrationHandler.DeRegisterPlugin request for plugin %s", pluginName)) if err := unregisterDriver(pluginName); err != nil { glog.Error(log("registrationHandler.DeRegisterPlugin failed: %v", err)) } </ins> } func (p *csiPlugin) Init(host volume.VolumeHost) error {

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: When the Kubelet plugin watcher invokes method, right now the CSI code does nothing (method is not implemented). What you expected to happen: When the Kubelet plugin watcher no longer detects the socket file for a given driver, it invokes the method on the CSI registration handler, the CSI code should then clean up plugin information and remove the corresponding CSIDriver object. How to reproduce it (as minimally and precisely as possible): Delete the socket file of a CSI plugin exposed for registration, nothing happens. /sig storage
/assign


Question: return request, ok } <del> // GetVirtualFinish returns the expected virtual finish time of the request at // index J in the queue with estimated finish time G func (q *queue) GetVirtualFinish(J int, G float64) float64 { // The virtual finish time of request number J in the queue // (counting from J=1 for the head) is J * G + (virtual start time). // counting from J=1 for the head (eg: queue.requests[0] -> J=1) - J+1 jg := float64(J+1) * float64(G) return jg + q.virtualStart </del> <ins> // GetNextFinish returns the expected virtual finish time of the // oldest request in the queue with estimated duration G func (q *queue) GetNextFinish(G float64) float64 { // TODO: if we decide to generalize this function to return virtual finish time //  for the Nth oldest request waiting in the queue, we need to carefully //  evaluate and potentially improve the performance here. var oldestReq *request q.requests.Walk(func(r *request) bool { oldestReq = r return false }) if oldestReq == nil { // we should never be here, since the caller should ensure // that this queue has request(s) waiting to be served before // calling this function. return q.virtualStart } // the estimated service time of the oldest request is (G * request width) estimatedServiceTime := float64(G) * float64(oldestReq.Seats()) return q.virtualStart + estimatedServiceTime </ins> } func (q *queue) dump(includeDetails bool) debug.QueueDump {

I reviewed the current code to see what had been done to generalize request width. I found a few inconsistencies in . A really simple one is at line , where is subtracted from the product but should have been subtracted from just (and their difference multiplied by ). With that fixed, a queue's tracks the expected "virtual time" ( in the original paper) when the oldest waiting request will start. However, computes the expected finish "virtual time" of the J'th request by multiplying by but not . There should be a consistent notion of "virtual time" (AKA ), either always including seats or excluding them. Kubernetes version (use ): latest k8s/apiserver sources Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
BTW, I suspect it would be better if "virtualTime" is not multiplied by request width. I am working on writing.
/triage accepted
I am running into a dilemma. The whole point of "virtual time" --- R(t) in the original paper --- is that it is shared among all the queues. They all get work done at the same rate, dR/dt. But when we generalize so that requests have different widths (and thus work is the sum over requests of ), even restricting the virtual world so that a queue runs only one request at a time does not save us from the fact that different queues get work done at different rates. Thus, even discarding fairness, and executing one request at a time per queue in the virtual world, it looks like what can be shared among queues is not a matter of work done.
But the current implementation approach makes the assumption that all the non-empty queues get work done at the same rate. As long as we are willing to tolerate the lossages due to that fiction, the queues can share one R(t) meter.
I see both sides of this; I still think it is double counting to use to advance virtual time AND to occupy W seats. OTOH, I also think it is right that we should be fair over the work () -- not just duration. One crazy idea is to add the request to multiple queues, and not execute it until it gets to the head of all the queues it is in.

The number of queues it needs to be to would be (# queues) * W / seats, rounded up I guess.
I think "virtual time" is a fuzzy concept that does not come equipped with guidelines about how to think about it correctly. That is why I followed the style of reasoning in the original paper, it defines a specific function and shows why and how that particular function is useful. In the KEP revision I am preparing, I think the problem statement ()the equations defining intended behavior ()the implementation discussion () make sense. It is in the last of those that is introduced, and I think it is clear that this should be a measure that includes request width.
I don't think it's double counting, because those are different things. And yes - we should be fair over WD - this is exactly why I wanted to push virtual time by WD. - I've seen your KEP PR, just I'm under water. My Fridays are real working time, so I will take a look tomorrow.


Question: if forgetErr := sched.Cache().ForgetPod(assumedPod); forgetErr != nil { klog.Errorf("scheduler cache ForgetPod failed: %v", forgetErr) } <del> sched.recordSchedulingFailure(assumedPod, prebindStatus.AsError(), reason, prebindStatus.Message()) </del> // trigger un-reserve plugins to clean up state associated with the reserved Pod fwk.RunUnreservePlugins(pluginContext, assumedPod, scheduleResult.SuggestedHost) <ins> sched.recordSchedulingFailure(assumedPod, prebindStatus.AsError(), reason, prebindStatus.Message()) </ins> return }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: TestUnreservePlugin is flaky: What you expected to happen: Test to always pass. How to reproduce it (as minimally and precisely as possible): This is a flaky test, but the issue is that the unreserve plugin didn't get the change to execute before the test checked the unreserve call counter in the test. Assuming thread1 runs the scheduler, thread2 runs the tests, here is the sequence that caused this flake: thread2 creates a pod and blocks on a . thread1 tries to schedule the pod, executing the pre-bind plugin which denies the pod and . thread2 gets unblocked before thread1 gets to . Thread2 checks the counters, throwing an error since unreserve plugin call counter didn't get to be incremented. What we need is for the test to block until unreserve is called, not until we get a scheduling error. /sig scheduling
/help
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help </details
cc/


Question: return tabbedString(func(out io.Writer) error { w := NewPrefixWriter(out) w.Write(LEVEL_0, "Name:t%vn", ing.Name) <ins> printLabelsMultiline(w, "Labels", ing.Labels) </ins> w.Write(LEVEL_0, "Namespace:t%vn", ing.Namespace) w.Write(LEVEL_0, "Address:t%vn", loadBalancerStatusStringer(ing.Status.LoadBalancer, true)) def := ing.Spec.Backend

When command 'kubectl describe ing' Name: test-ing Namespace: practice Address: 10.10.01.10 Default backend: default-http-backend:80 (<error: endpoints "default-http-backend" not found) TLS: test terminates Rules: Host Path Backends ---- ---- -------- / test-svc:80 (<none) Annotations: nginx Events: <noneResult has no information about labels. But Pod, Deployment, Service resources has label information like below. Name: test-room-test-mjsbz Namespace: test Priority: 0 Node: aks-test-test-test/22.22.222. Start Time: Fri, 09 Jul 2021 05:45:26 +0000 Labels: app=test pod-template-hash=test role=mediatest Why ingress has no information about labels? Actually when delete ingresss with labels is working correctly. Kubernetes version (use ): 1.18.14 Cloud provider or hardware configuration: AKS (Azure) OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /sig
/sig
/sig cli
/assign
/assign
/triage accepted /remove-kind bug /kind feature


Question: Release : v1.27 Testname: OpenAPI V3 Aggregated APIServer Description: Create an Aggregated APIServer. The OpenAPI V3 for the aggregated apiserver MUST be aggregated by the aggregator and published. The specification MUST be round trippable. <ins> This test case is marked as serial due to potential conflicts with other test cases that set up a sample-apiserver. For more information, see: https://github.com/kubernetes/kubernetes/issues/119582#issuecomment-2215054411. </ins> */ <del> ginkgo.It("should contain OpenAPI V3 for Aggregated APIServer", func(ctx context.Context) { </del> <ins> f.It("should contain OpenAPI V3 for Aggregated APIServer", f.WithSerial(), func(ctx context.Context) { </ins> config, err := framework.LoadConfig() framework.ExpectNoError(err) aggrclient, err := aggregatorclient.NewForConfig(config)

e2e conformance jobs, seen in aws, capz-windows, and kind seen back to 1.28 shows flakes going back as far as history exists No response [FAILED] could not find group version resource for dynamic client and wardle/flunders (discovery error: <nil, discovery results: map[schema.GroupVersionResource]struct {}{schema.GroupVersionResource{Group:"", Version:"v1", Resource:"bindings"}:struct {}{}, schema.GroupVersionResource{Group:"", Version:"v1", Resource:"configmaps"}:struct {}{}, schema.GroupVersionResource{Group:"", Version:"v1", Resource:"endpoints"}:struct {}{}, schema.GroupVersionResource{Group:"", Version:"v1", Resource:"events"}:struct {}{}, schema.GroupVersionResource{Group:"", Version:"v1", Resource:"limitranges"}:struct {}{}, schema.GroupVersionResource{Group:"", Version:"v1", Resource:"persistentvolumeclaims"}:struct {}{}, schema.GroupVersionResource{Group:"", Version:"v1", Resource:"pods"}:struct {}{}, schema.GroupVersionResource{Group:"", Version:"v1", Resource:"podtemplates"}:struct {}{}, schema.GroupVersionResource{Group:"", Version:"v1", Resource:"replicationcontrollers"}:struct {}{}, schema.GroupVersionResource{Group:"", Version:"v1", Resource:"resourcequotas"}:struct {}{}, schema.GroupVersionResource{Group:"", Version:"v1", Resource:"secrets"}:struct {}{}, schema.GroupVersionResource{Group:"", Version:"v1", Resource:"serviceaccounts"}:struct {}{}, schema.GroupVersionResource{Group:"", Version:"v1", Resource:"services"}:struct {}{}, schema.GroupVersionResource{Group:"apps", Version:"v1", Resource:"controllerrevisions"}:struct {}{}, schema.GroupVersionResource{Group:"apps", Version:"v1", Resource:"daemonsets"}:struct {}{}, schema.GroupVersionResource{Group:"apps", Version:"v1", Resource:"deployments"}:struct {}{}, schema.GroupVersionResource{Group:"apps", Version:"v1", Resource:"replicasets"}:struct {}{}, schema.GroupVersionResource{Group:"apps", Version:"v1", Resource:"statefulsets"}:struct {}{}, schema.GroupVersionResource{Group:"", Version:"v1",

Resource:"localsubjectaccessreviews"}:struct {}{}, schema.GroupVersionResource{Group:"autoscaling", Version:"v2", Resource:"horizontalpodautoscalers"}:struct {}{}, schema.GroupVersionResource{Group:"batch", Version:"v1", Resource:"cronjobs"}:struct {}{}, schema.GroupVersionResource{Group:"batch", Version:"v1", Resource:"jobs"}:struct {}{}, schema.GroupVersionResource{Group:"", Version:"v1", Resource:"leases"}:struct {}{}, schema.GroupVersionResource{Group:"", Version:"v1", Resource:"endpointslices"}:struct {}{}, schema.GroupVersionResource{Group:"", Version:"v1", Resource:"events"}:struct {}{}, schema.GroupVersionResource{Group:"", Version:"v1", Resource:"ingresses"}:struct {}{}, schema.GroupVersionResource{Group:"", Version:"v1", Resource:"networkpolicies"}:struct {}{}, schema.GroupVersionResource{Group:"policy", Version:"v1", Resource:"poddisruptionbudgets"}:struct {}{}, schema.GroupVersionResource{Group:"", Version:"v1", Resource:"rolebindings"}:struct {}{}, schema.GroupVersionResource{Group:"", Version:"v1", Resource:"roles"}:struct {}{}, schema.GroupVersionResource{Group:"", Version:"v1", Resource:"csistoragecapacities"}:struct {}{}, schema.GroupVersionResource{Group:"", Version:"v1beta1", Resource:"flunders"}:struct {}{}}) or - [FAILED] could not find group version resource for dynamic client and wardle/flunders (discovery error: unable to retrieve the complete list of server APIs: stale GroupVersion discovery: , discovery results: map[schema.GroupVersionResource]struct {}{schema.GroupVersionResource{Group:"", Version:"v1", Resource:"bindings"}:struct {}{}, schema.GroupVersionResource{Group:"", Version:"v1", Resource:"configmaps"}:struct {}{}, schema.GroupVersionResource{Group:"", Version:"v1", Resource:"endpoints"}:struct {}{}, schema.GroupVersionResource{Group:"", Version:"v1", Resource:"events"}:struct {}{}, schema.GroupVersionResource{Group:"", Version:"v1", Resource:"limitranges"}:struct {}{}, schema.GroupVersionResource{Group:"", Version:"v1",

Resource:"persistentvolumeclaims"}:struct {}{}, schema.GroupVersionResource{Group:"", Version:"v1", Resource:"pods"}:struct {}{}, schema.GroupVersionResource{Group:"", Version:"v1", Resource:"podtemplates"}:struct {}{}, schema.GroupVersionResource{Group:"", Version:"v1", Resource:"replicationcontrollers"}:struct {}{}, schema.GroupVersionResource{Group:"", Version:"v1", Resource:"resourcequotas"}:struct {}{}, schema.GroupVersionResource{Group:"", Version:"v1", Resource:"secrets"}:struct {}{}, schema.GroupVersionResource{Group:"", Version:"v1", Resource:"serviceaccounts"}:struct {}{}, schema.GroupVersionResource{Group:"", Version:"v1", Resource:"services"}:struct {}{}, schema.GroupVersionResource{Group:"apps", Version:"v1", Resource:"controllerrevisions"}:struct {}{}, schema.GroupVersionResource{Group:"apps", Version:"v1", Resource:"daemonsets"}:struct {}{}, schema.GroupVersionResource{Group:"apps", Version:"v1", Resource:"deployments"}:struct {}{}, schema.GroupVersionResource{Group:"apps", Version:"v1", Resource:"replicasets"}:struct {}{}, schema.GroupVersionResource{Group:"apps", Version:"v1", Resource:"statefulsets"}:struct {}{}, schema.GroupVersionResource{Group:"", Version:"v1", Resource:"localsubjectaccessreviews"}:struct {}{}, schema.GroupVersionResource{Group:"autoscaling", Version:"v2", Resource:"horizontalpodautoscalers"}:struct {}{}, schema.GroupVersionResource{Group:"batch", Version:"v1", Resource:"cronjobs"}:struct {}{}, schema.GroupVersionResource{Group:"batch", Version:"v1", Resource:"jobs"}:struct {}{}, schema.GroupVersionResource{Group:"", Version:"v1", Resource:"leases"}:struct {}{}, schema.GroupVersionResource{Group:"", Version:"v1", Resource:"endpointslices"}:struct {}{}, schema.GroupVersionResource{Group:"", Version:"v1", Resource:"events"}:struct {}{}, schema.GroupVersionResource{Group:"", Version:"v1", Resource:"ingresses"}:struct {}{}, schema.GroupVersionResource{Group:"", Version:"v1", Resource:"networkpolicies"}:struct {}{}, schema.

GroupVersionResource{Group:"policy", Version:"v1", Resource:"poddisruptionbudgets"}:struct {}{}, schema.GroupVersionResource{Group:"", Version:"v1", Resource:"rolebindings"}:struct {}{}, schema.GroupVersionResource{Group:"", Version:"v1", Resource:"roles"}:struct {}{}, schema.GroupVersionResource{Group:"", Version:"v1", Resource:"csistoragecapacities"}:struct {}{}}) Unclear if the issue is discovery going stale, or the aggregated server not responding, or something else /sig api-machinery
last flake on sig-release-master-blocking testgrid is 03-05 As it does not flake on the board anymore and still flake in other CIs, remove it from release signal board.
/assign Based on , I don't see it flaking as far as history exists. It seemed to have started flaking around March/April 2023. This also corresponds to when aggregated discovery was enabled by default so it may be related. Will do some more digging.
/triage accepted
Hey Is this issue fixed? If yes, from which version of kubernetes will this fix be available from?
This still flakes in some CIs. The flake rate is very low, but it is a conformance test.
Thanks for the quick response. I see it is very flaky. Is there any active work going on to fix this? If yes, is there any estimated time by which it could be fixed?
do you have some update on this?
cc.

I see the flake in recent run of later find it flake in as well. (not flake now) or More flakes can be found in /kind flake <!-- If this is a flake, please add: /kind flake --/sig api-machinery a slack thread a similar issue: for windows CI
/cc
Likely a test job misconfiguration. /assign /triage accepted
It seems there was a previous related issue
/cc
From Cloud provider testgrid is testing against master k8s branch, while the test requires a change in kube-aggregator, which is not yet cherry-picked to cloud provider testgrid release-1.28
It not only flakes on windows CI.
hit in
Flake in as well.
/remove-sig windows
Flake in .
See
/unassign
this test is still flaking but rarely? from a kind job today
This is still flaking, see a recent duplicate:
Hello! This Flake has not been updated for a long time, so I'd like to check what's the status. The code freeze is starting 02:00 UTC Wednesday 10th July 2024 (about 4 weeks from now), and while there is still plenty of time, we want to ensure that each PR has a chance to be merged on time. As this is tagged for 1.31, is it still planned for this release?
/milestone v1.31
It seems it is failing in
cc. Hit the same in the presubmit which ran against one of my PRs. Did some digging into this. It turned out to be a conflict between the following test cases running in parallel: [It] [sig-api-machinery] Aggregator Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] [It] [sig-api-machinery] OpenAPIV3 should contain OpenAPI V3 for Aggregated APIServer Both test cases call . As a result, two cluster-scoped s are created, one for and one for . The existence of the causes to fail, as it results in returning instead of the expected . I also managed to reproduce this issue with hydrophone: See for test logs. Finally, the solution is to mark both test cases as [Serial]. I'll submit a PR shortly for this.

/assign
can we avoid to make them serial? do we have a way to make two tests to not able to run at the same time?
the only way, currently, is to mark (at least) one as . I have plans (more of a long-term vision, really) for more but for now your best recourse is to use . I suppose you could also leverage Kubernetes and have both tests get/remove some sort of agreed upon mutex in the cluster and hang out and wait until the mutex is cleared before running. This would allow them to run in parallel with other specs but would force them to serialize wrt each other if they ever try to run concurrently. While doable, this will introduce a fair bit of complexity and a very different new internal contract into the test suite. It's Almost Certainly a Bad Idea.
I agree that marking the tests as serial is currently our best option. We talked about running all conformance tests with a single invocation. That can be done with without skipping the serial tests: Ginkgo will run non-serial tests in parallel and then the serial ones.
let's mark the non-conformance as serial then, Serial are skipped by default in a lot of places and missing coverage in a conformance test is something I would prefer to avoid


Question: } func (saw *sampleAndWaterMarkHistograms) innerSet(updateXOrX1 func()) { <del> saw.Lock() when := saw.clock.Now() whenInt := saw.quantize(when) acc := saw.sampleAndWaterMarkAccumulator wellOrdered := !when.Before(acc.lastSet) if wellOrdered { </del> <ins> var when time.Time var whenInt int64 var acc sampleAndWaterMarkAccumulator var wellOrdered bool func() { saw.Lock() defer saw.Unlock() when = saw.clock.Now() whenInt = saw.quantize(when) acc = saw.sampleAndWaterMarkAccumulator wellOrdered = !when.Before(acc.lastSet) </ins> updateXOrX1() saw.relX = saw.x / saw.x1 <del> if acc.lastSetInt < whenInt { saw.loRelX, saw.hiRelX = acc.relX, acc.relX saw.lastSetInt = whenInt </del> <ins> if wellOrdered { if acc.lastSetInt < whenInt { saw.loRelX, saw.hiRelX = acc.relX, acc.relX saw.lastSetInt = whenInt } saw.lastSet = when </ins> } <ins> // `wellOrdered` should always be true because we are using // monotonic clock readings and they never go backwards.  Yet // very small backwards steps (under 1 microsecond) have been // observed // (https://github.com/kubernetes/kubernetes/issues/96459). // In the backwards case, treat the current reading as if it // had occurred at time `saw.lastSet` and log an error.  It // would be wrong to update `saw.lastSet` in this case because // that plants a time bomb for future updates to // `saw.lastSetInt`. </ins> if saw.relX < saw.loRelX { saw.loRelX = saw.relX } else if saw.relX > saw.hiRelX { saw.hiRelX = saw.relX } <del> saw.lastSet = when } saw.Unlock() </del> <ins> }() </ins> if !wellOrdered { <del> lastSetS := acc.lastSet.Format(time.RFC3339Nano) whenS := when.Format(time.RFC3339Nano) klog.Fatalf("Time went backwards from %s to %s for labelValues=%#+v", lastSetS, whenS, saw.labelValues) panic(append([]string{lastSetS, whenS}, saw.labelValues...)) </del> <ins> lastSetS := acc.lastSet.String() whenS := when.String() klog.Errorf("Time went backwards from %s to %s for labelValues=%#+v", lastSetS, whenS, saw.labelValues) </ins> } for acc.lastSetInt < whenInt { saw.samples.WithLabelValues(saw.labelValues...).Observe(acc.relX)

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: The apiserver is always restarting and I can't access it with any commands looks like . When I check the kubectl's status, the tells me that: And then I checked the container's status and log of the apiserver: status: log: I see that there is a line: And then I get the source code: What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): v1.19.3. But I can't use the now because of the problem, so I can't give you a formal result about the version. Cloud provider or hardware configuration: Ryzen 5 4500U 16GB memory 30GB SSD In the VMware Workstation 15.5.6 OS (e.g: ):Kernel (e.g. ):Install tools: Network plugin and version (if this is a network-related bug): Others:
The complete version for the log of apiserver:
/cc
: did you tweak that machine's clock backwards, or run any software that does this?
Time calculations should be written to use go's , not count on wall clock time always moving foward.
: The proximal problem is that uses and gets the answer that time went backwards.
sure, but should be immune to wall clock resets per
Exactly. It looks to me like this test for ordering is using the monotonic clock and somehow that is saying time went backwards.
does the quantization () factor into well-ordered detection? I'm not sure how the monotonic clock interacts with Unix / UnixNano
No, I see the quantization is buggy, it uses wall clock. But the ordering test does not depend on the quantization.
/triage accepted /priority urgent-critical
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /triage accepted /priority urgent-critical </details
very odd that this is repeatable. /priority critical-urgent
: it is very odd that this is repeatable. Do you have anything unusual in your environment ?

I see it is vmware workstation. Does that mess with the clock in any unusual way? I also wonder about the golang runtime; did you do your own build (if so, what version of golang) or fetch binaries from somewhere (where?); full output would be helpful.
I don't have anything unusual in my environment. I run it in the ubuntu driven by vmware workstation, because my real machine's operation system is Windows10 , and I need a virtual machine to run k8s. My computer is a laptop computer. I got the golang runtime by apt-get. I also got the distribution of k8s by apt-get too. I did not do my own build. ! Okay, I got my golang and k8s version now: !
I have exactly same problem in my local lab. Virt infra on Virtual Box 6.1.16 Guest OS
you try and, if so, does it fix your problem and does it log complaints about the monotic clock going backwards? you tell us about your host and the settings on your VM(s)?
exactly matches same error as posted by author of issue, I have NOT tried VM settings: 4 vCPUs 4GB RAM swap is OFF on OS level. Host HW: Core i9 9900K 32GB Ram BTW: I have managed to install 1.18.12 on same VM eventually - cause for me it is just lab infra to study, so 1.19.X is NOT that important for me to have on this infra.
: host OS? Is your VB "Version 6.1.16 r140961 (Qt5.6.3)"? VM settings match the following? <img width="616" alt="Screen Shot 2020-11-16 at 4 56 10 PM" src="https://user-"<img width="742" alt="Screen Shot 2020-11-16 at 4 55 58 PM" src="https://user-"<img width="720" alt="Screen Shot 2020-11-16 at 4 55 45 PM" src="https://user-"
OS Name: Microsoft Windows 10 Enterprise OS Version: 10.0.

N/A Build OS Manufacturer: Microsoft Corporation OS Configuration: Standalone Workstation OS Build Type: Multiprocessor Free ! ! ! !
I tried to reproduce the problem using Ubuntu 18.0.4.5 and Kubernetes 1.19.4 (installed by kubeadm, using Calico as the network plugin), but the problem did not strike. The guest was created from the Ubuntu live Server ISO with a very plain install, picking only OpenSSH from the things to add during the install. No explicit tweaks regarding timekeeping. In the guest: I am also running the exact same version of VBox as reported above. I have the ExtensionPack of the same version installed. My host is Windows 10 Pro version 2004. Intel Core i9-9900 (8 hyperthreaded cores, thus 16 logical processors), 32 GB RAM. Configured for automatic time sync. Last sync 34 minutes ago, with server
Same issue in my env: VM (CentOS 7.9.2009 4 CPU+12G Memory) on virtualbox(6.1) Host is Windows 10 (16G Memory, CPU Intel(R) Core(TM) i3- CPU @ 3.60GHz 3.60 GHz) docker version: docker-ce-19.03.9-3.el7.x8664 K8S version: 1.19.4 Part error log output from docker logs -f <kube-apiserver-containerid:
Hi, same here vitualbox 6.1.18 latest Linux master-1 5.4.0-72-generic latest /etc/os-release NAME="Ubuntu" VERSION="20.04.2 LTS (Focal Fossa)" ID=ubuntu IDLIKE=debian PRETTYNAME="Ubuntu 20.04.2 LTS" VERSIONID="20.04" HOMEURL="" SUPPORTURL="" BUGREPORTURL="" PRIVACYPOLICYURL="" VERSIONCODENAME=focal UBUNTU_CODENAME=focal tried with K8s 1st with 1.21.0-00 then 1.19.10-00 same Time went backwards message in the logs. Api server uses large amount of CPU at doing nothing. etcd cluster seems also to be totally lost. 3 masters setup.

thanks
Hm ... this must have something to do with the fact we're running in a VM. Funny enough ... you guys are reporting apiserver crashing. What led me to this issue is that my metrics-server crashes with this same log message instead :). I originally had ntpd installed on the VM, but removing it did not help. So ... if time goes backward, it must be something that VirtualBox does. I'll try to see if I can repro with HyperV. EDIT: I was able to get rid of this error with VirtualBox by using to disable some kind of automated clock sync that VirtualBox does by default. Don't forget to use NTPd, as you will need some time synchronization :). EDIT 2: Spoke too soon. That did not really help. It seems to have slightly delayed the issue, though, as it only presented itself after a couple of minutes... EDIT 3: So, the reason why it's still syncing time, is VBoxGuestAdditions. When they are running, the guest synces time with the host, even with the above time-sync setting disabled. One either has to stop the guest additions or pass them (but ... it's not really a configuration change: ). EDIT 4: The above doesn't seem to have an effect at all. Still, according to the metrics server, time goes backwards. Well ... let's try to completely disable the VirtualBox guest additions: (for my use case, this is totally fine, as I don't need any of the guest additions features such as shared folders or time sync). EDIT 5: Nothing helps. I reckon, either the check for "time went backwards" is somewhat buggy, or some odd time sync still takes place. For now, because I don't really need it, I am just disabling the metrics server. EDIT 6: Huh. Got it, finally. Never surrender, I guess :). Turns out, you gotta attack it from all sides. (see above) Add the argument to (see above) (this is new; turns out, even if one tells guest additions not to sync time, and disables the time syncing on the VM, something can still sync the time -- this setting makes the guest OS completely blind to the fact it is running in a VM).

I suggest to keep ntp and guest additions (with the disabled time sync) on at all times. EDIT 7: Also ... do no let your host sleep. When it sleeps and then wakes up, a lot of k8s cannot cope with it and begins rolling. It should recover, but if you can avoid sleeping, do. For me, this actually meant installing and using "Don't sleep" (alternative to "Caffeine" or similar) to keep it awake. For some reason, when you lock your computer, Windows will try to sleep. It should be possible to disable this mechanic (), but even though I did, it would still sleep. That's why I ended up with "Don't sleep". With all of this in place, it finally works as expected. None of the k8s' pods roll anymore.
I had exactly the same issue. Followed the steps listed above and it solved my issue. Thanks , much appreciated!


Question: for ix := range s.Columns { headers[ix] = s.Columns[ix].Header } <del> fmt.Fprintln(w, strings.Join(headers, "t")) </del> <ins> fmt.Fprintln(out, strings.Join(headers, "t")) </ins> s.lastType = t } parsers := make([]*jsonpath.JSONPath, len(s.Columns))

Compare the following two outputs (note the bad alignment in the last line): The custom-columns string is exactly the same in both cases, but something messes up the column widths when using openapi print columns. Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Columns printed when using are misaligned. What you expected to happen: Columns in all rows should be aligned. How to reproduce it (as minimally and precisely as possible): Deploy Service Catalog, patched with Run the above two commands Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"9+", GitVersion:"v1.9.0-alpha.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-11-15T23:06:17Z", GoVersion:"go1.9.1", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.0", GitCommit:"", GitTreeState:"dirty", BuildDate:"2017-10-17T15:09:55Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"} (same behavior with kubectl built from current master branch)
/sig api-machinery
/remove-sig api-machinery /sig cli PR is ready:
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: - name: test-volume # This AWS EBS volume must already exist. awsElasticBlockStore: <del> volumeID: aws://<availability-zone>/<volume-id> </del> <ins> volumeID: <volume-id> </ins> fsType: ext4 ``` <del> (Note: the syntax of volumeID is currently awkward; #10181 fixes it) </del> ### nfs An `nfs` volume allows an existing NFS (Network File System) share to be

changed the way volumes are adressed for AWS but the docs at aren't up to date yet.