Question: o.HPAClient = kubeClient.AutoscalingV1() // get the generator <del> o.generatorFunc = func(name string, mapping *meta.RESTMapping) (generate.StructuredGenerator, error) { </del> <ins> o.generatorFunc = func(name, refName string, mapping *meta.RESTMapping) (generate.StructuredGenerator, error) { if len(name) == 0 { name = refName } </ins> switch o.Generator { case generateversioned.HorizontalPodAutoscalerV1GeneratorName: return &generateversioned.HorizontalPodAutoscalerGeneratorV1{

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: the hpa name is always info.Namenot --name What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/assign
/sig cli


Question: // NodeMonitorGracePeriod is the amount of time which we allow a running node to be // unresponsive before marking it unhealthy. Must be N times more than kubelet's // nodeStatusUpdateFrequency, where N means number of retries allowed for kubelet <del> // to post node status. </del> <ins> // to post node status. This value should also be greater than the sum of // HTTP2_PING_TIMEOUT_SECONDS and HTTP2_READ_IDLE_TIMEOUT_SECONDS. </ins> NodeMonitorGracePeriod metav1.Duration // secondaryNodeEvictionRate is implicitly overridden to 0 for clusters smaller than or equal to largeClusterSizeThreshold LargeClusterSizeThreshold int32

The controller-manager node-lifecycle-controller watches node lease and update probeTimestamp , and it check node probeTimestamp is before . The node-monitor-grace-period default value is 40s. When the client connection lost, the lease watcher cannot receive new event. With the http2 health check, HTTP2PINGTIMEOUTSECONDS default is 30s, HTTP2READIDLETIMEOUTSECONDS default is 15s, it will delay 45s to kill http2 connection. During this period, node-lifecycle-controller will mark all nodes NotReady. Here are some logs When the http2 connection lost, the watch should stop as soon as possible. The reflector should relist resource, and node-lifecycle-controller should not mark all node NotReady. HTTP2PINGTIMEOUTSECONDS+HTTP2READIDLETIMEOUTSECONDS should be less than 40s. the controller-manager node-lifecycle-controller watch client srcIP:srcPort. Drop all tcp package send into srcIP:srcPort. for more than 40 seconds, all node will be marked NotReady node will be Ready after some seconds. No response <details</details<details</details<details</details<details</details<details</details<details</details
How about double-check node lease before marking node NotReady? The node-lifecycle-controller can get reference lease from apiserver with
/sig node
/sig api-machinery
/cc We have discussed this timeouts values before and myself, I also think got some metrics with lower timeouts for the HTTP2 connections. In this situation is clear that the defaults of the node watcher and the timeouts are related and having good defaults will be ideal, but the question is which values do we change ?
I think in internal network, the ping response almost return in 1 second, the HTTP2PINGTIMEOUT_SECONDS default value should be 5 seconds. And the total timeout will 35 seconds. WDYT?
/triage accepted any pointers in other places where those timeouts were changed for http2? Any notes from those discussions?
/priority important-longterm since it is not a regression I will keep it longterm priority and not higher. But we can reevaluate if we hear more reports about it.

/assign
I'm investigating problems similar to this, but I'm looking for a way to confirm this is what is happening - and if there are tips for how to recreate such a scenario, that would be very helpful. We get infrequent reports of "worker goes NotReady for a short time" incidents that appear to be triggered by events like failing over a LB sitting front of the kube-apiservers. We've done some tests with a simple client-go app and have recreated a scenario in which the app logs messages after 45 secs of "context exceeded" errors. The failures reported to us from customers don't have http2 errors in the kubelet logs, and our attempts to recreate with kubelet have never shown more more than a single error which works on subsequent retry.
Hi folks, we have a simulated recreate of the problem. This recreate involves injecting a 40 second delay in "connectivity" between the Kubernetes API server and etcd (we a 40 second delay to etcd defrag and defragged all etcd members at the same time). The connectivity between these two components is obviously critical to nodes successfully updating their leases in order to stay ready. We use 40 seconds because it matches . The test will usually cause one or more of the cluster nodes will flap their readiness state. Setting on the Kubernetes controller manager prevents the flap. Given this, I think that one or more of the Kubernetes default configurations needs to change in order to tolerate such "connectivity" problems in order to maintain healthy node state.
What's the proposal then? Do we update --node-monitor-grace-period to 50s? It seems the less risky change as the client-go defaults are more sensitive and impact the whole ecosystem, beyond kubernetes itself are you open to increase this default? It seems worth it to improve the reliability of cluster
I think we should update .
can someone send a PR to see how it will look like?
I have been following this discussion as we had encountered the bug as well. I have created a PR - Please let me know if it works
I'll add another data point from our testing. Namely, the simulated recreate noted depends on the Kubernetes controller manager (KCM) leader election configuration. Using the following leader election configuration for the KCM yields a KCM that has no leader after the delayed etcd defrag.

As a result, nodes won't be marked not ready unless the KCM leadership is restored before the nodes renew their lease. Extending the KCM leader election configuration to the following, makes the problem easier to recreate.
/assign Reassigning the issue to me


Question: // This way assumes you've set the POD_NAMESPACE environment variable using the downward API. // This check has to be done first for backwards compatibility with the way InClusterConfig was originally set up if ns := os.Getenv("POD_NAMESPACE"); ns != "" { <del> return ns, true, nil </del> <ins> return ns, false, nil </ins> } // Fall back to the namespace associated with the service account token, if available if data, err := ioutil.ReadFile("/var/run/secrets/kubernetes.io/serviceaccount/namespace"); err == nil { if ns := strings.TrimSpace(string(data)); len(ns) > 0 { <del> return ns, true, nil </del> <ins> return ns, false, nil </ins> } }

This is similar to , but that report (and fix) focussed on the in-cluster namespace vs configured .kube/config context namespace. I have a similar issue with the in-cluster namespace vs an explicit in the JSON/YAML file I'm passing to When run in-cluster, 1.5+ (including 1.6.0) refuses to allow me to create a resource in another namespace via an explicit property on a resource. 1.4.x honours when run in-cluster as expected. All versions honour when run out-of-cluster. I don't want to / can't provide an explicit arg or context declaration as suggested by the error text, because this is an automated deployment pipeline that creates resources in multiple namespaces. I just want the client to do-what-I-say without imposing additional client-side restrictions :stuckouttongue:
Fixed in


Question: } return libstrings.ToUpper(string(str[0])) + str[1:] } <ins> // getDiskLUN : deviceInfo could be a LUN number or a device path, e.g. /dev/disk/azure/scsi1/lun2 func getDiskLUN(deviceInfo string) (int32, error) { var diskLUN string if len(deviceInfo) <= 2 { diskLUN = deviceInfo } else { // extract the LUN num from a device path matches := lunPathRE.FindStringSubmatch(deviceInfo) if len(matches) == 2 { diskLUN = matches[1] } else { return -1, fmt.Errorf("cannot parse deviceInfo: %s", deviceInfo) } } lun, err := strconv.Atoi(diskLUN) if err != nil { return -1, err } return int32(lun), nil } </ins>

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: There is a drawback after PR: following code will fetch vmCache may lead to below error and the vmCache refresh time period would be 1 min, and may cost 1 min which is too long time How to fix this At least in Linux, we may still use parameter to get LUN, which is almost zero time. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): v1.12 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: /sig azure /assign
k8s 1.10.4, Azure (deployed via ACS-engine) do you know about any directly applicable workaround? This is hitting us quite hard. logs: logs: The last Attach operation reports success, yet we enter the same error loop over and over again.
thanks for reporting, I am fixing this issue, code should be ready in this week.
We have seen what looks like this issue - possible regression? It took 11m for the volume to attach.
it's more likely that disk could not be detected under dir inside vmss instance, is it a transient issue? Can this issue repro easily?
Seems to happen consistently on the 2nd pod of a new deploy. We have seen 3-4 times in a row deleting and re-deploying this project. Doesn't seem to be an issue for the kafka-0 and kafka-2 pods, just kafka-1. And only on a fresh deploy of the helm chart, deleting the pod does not reproduce.
what's your region and k8s version? it only happened on vmss, right?
The version we’re running is k8s 1.13.12 in useast2. And right this is on vmss. EDIT: I should also note that we're also using multi fault zones for each of the vmss pools we're deploying.
I'm having a similar issue, but I'm on 1.14.8.

I don't know if is related, but these disk attachments seem to not be working properly on aks with vmms.
confirm volume attach with vmss is failing bad..
pls open a separate issue, provide detailed error info, e.g. k8s version, pod error description, etc. thanks.
This took too long AKS - kubernetes version -1.14.8 location -west europe
thanks for the reporting, I could confirm that the is due to PR: , which is going to mitigate vmss API throttling issue, in that PR, it enlarge the getVMSS TTL from 1min to 10min. Fortunately, from k8s 1.15.0, won't have this issue, since we have refactored the disk detect logic, it won't depend on getVMSS operation to get the disk LUN. In brief, pls upgrade to latest version 1.15.5, 1.15.x won't have this issue, thanks.
cherry-pick PR: to fix this issue on 1.14
You mention a 1.14 cherry pick but I'm also seeing Is this correct to assume 1.13 will have this fix as well ?
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : <!-- This form is for bug reports and feature requests ONLY! --/kind bug Uncomment only one, leave it on its own line: /kind bug /kind feature What happened: There is a drawback after PR: following code will fetch vmCache may lead to below error and the vmCache refresh time period would be 1 min, and may cost 1 min which is too long time At least in Linux, we may still use parameter to get LUN, which is almost zero time. Anything else we need to know?: Kubernetes version (use ): v1.12 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.

</details
1.13 don't accept any cherry-pick, so only 1.14 could be cherry-picked now, target fixed version is 1.14.10
the delay still can be very big this is because azure api call for disk attach returns after at 35s-2min and kubelet goes to its exp. backoff retries so it can reach 2min timeout adding attached recording: called 3 disk simultaneous attach with interval ~2s and saw and So actual attachment takes ~10-30s (device appears in /dev/disk/azure/scsi1) , but due to azure api we can loop on it I saw also that disk attach timing is unstable and somtemes it can take 2.5min , what do you think about logic like below? is not waiting for attachdisk to return, instead it calls GET until it returns lun id (we saw it takes few seconds after attachdisk). after GET returns lun id , it updates node's disks with lunId and kubelet can continue with mount after waiting for /dev/disk/azure/scsi1/lun*
same here
It takes about 10 minutes to recover from the FailedMount messages.
what's your k8s version?
there is a new vhd disk feature based on azure file which could attach/detach disk < 1s, consider that as an option if you have concern about disk attach/detach time cost:
AKS v1.14.8
for this specific issue, aks 1.15.x won't have this issue.
hi ,I am not dare to upgrade.because my stateful rabbitmq with pvc can not started because of failmount of PVC!
you could scale your statefulset to 0 replica, and pvc disk would be detached. After upgrade, scale back to original count.
okay,let me have a try.
I use helm to can I restore to two replicas after upgrading k8s version?


Question: function kube::util::ensure-gnu-sed { # NOTE: the echo below is a workaround to ensure sed is executed before the grep. # see: https://github.com/kubernetes/kubernetes/issues/87251 <del> if LANG=C sed --help 2>&1 | grep -q "GNU|BusyBox"; then </del> <ins> sed_help="$(LANG=C sed --help 2>&1 || true)" if echo "${sed_help}" | grep -q "GNU|BusyBox"; then </ins> SED="sed" elif command -v gsed &>/dev/null; then SED="gsed"

/sig release cc filing this to track before I forget in this wall of meetings .the push-all target appears to have broken shell currently which means it's probably not only broken in CI but locally ...
Hi. I've commented on slack about this, so reposting here: Looking at [1], it looks like that function was not defined. folder, I'm not getting that error. [1]
Got it. Basically, that function is called when image used by the Image Promoter, it's busybox: TLDR: So, because the image is based on busybox, function. Thoughts?
maybe we should use a different image? we check for gnu sed because of compatibility issues?
Looking through the commit history, there are only 4 cases in which the GNU sed thing was mentioned / ensured, and the reason was pretty much so the scripts can run on Mac. From what I've read online, Macs have the FreeBSD version of sed, which behaves a bit differently. only uses sed to replace a few strings in some Dockerfile files, which is pretty standard, so I don't think the busybox version would have issues with that. Will check. I am currently trying to build the images inside the aforementioned container image above, and hit another issue after the sed thing. container: docker run --rm -it -v "{TARGET}:Z" -v "${KUBEROOT}" golang:"${GOLANGVERSION}"  /bin/bash -c " cd ${SRCDIR} &&  CGOENABLED=0 {ARCH} go build -a -installsuffix cgo --ldflags '-w' -o {SRC} ./{SRC}")" so the Image Promoter should also be able to build containers, but considering that docker CLI is already available inside the image, I think that'll be fine. Other than that, it also tries to mount the folder inside the new container (let's call it N) (which is probably already mounted inside the builder container (let's call it A)). But I'm having issues with that, the new container N doesn't see any file in the mounted volume. I'm looking into it, but if you have any ideas, that would be helpful.
cc WDYT re: not requiring GNU sed specifically?

Update: I can currently build and push images from inside the container, which is the same one used by the promoter. I've used the following command: docker run -t --privileged -v  -v /workspace/kubernetes/:/workspace/kubernetes  -v  --entrypoint bash  -c "cd kubernetes/test/images &&  make all-push WHAT=all-conformance REGISTRY=claudiubelu" build-logs This was done using this PR: However, there are a few catches here, of which I'm not sure they're being done in the Image Promoter: of all, the container must be able to build and run other contaienrs, which is why the was set. . Now, docker will always mount the folder / volume from the original host, and not from the current container's files and folders. has to run as, this is needed by the multiarch builds. This might already be set in the Image Promoter, but I'm not sure. I'd like to hear some answers about 2 and 3. PS: Also, there's another thing that's puzzling me, and it;s a being weird. There's a flake when running `. That function's code is: function kube::util::ensure-gnu-sed { if LANG=C sed --help 2&1 BusyBox"; then SED="sed" elif command -v gsed &/dev/null; then SED="gsed" else kube::log::error "Failed to find GNU sed as sed or gsed. If you are on Mac: brew install gnu-sed." &2 return 1 fi kube::util::sourcedvariable "${SED}" } But, sometimes, when running it, it runs the commands out of order: ++ kube::util::ensure-gnu-sed ++ grep -q 'GNU|BusyBox' ++ LANG=C ++ sed --help ++ command -v gsed ++ kube::log::error 'Failed to find GNU sed as sed or gsed. If you are on Mac: brew install gnu-sed.' And sometimes not: ++ kube::util::ensure-gnu-sed ++ LANG=C ++ sed --help ++ grep -q 'GNU|BusyBox' ++ SED=sed ++ kube::util::sourcedvariable sed ++ true
Addressed the last issue as well.

Seems to be fine now, didn't encounter that flake anymore, and I've rebuilt all the conformance images 5-6 times already. Please review the PR, and let's try to find some answers to the points I've raised above.


Question: } return resourceToRankFunc } <ins> func PodIsEvicted(podStatus api.PodStatus) bool { return podStatus.Phase == api.PodFailed && podStatus.Reason == reason } </ins>

Kubelet evicts pods whenever there is memory or disk pressure. It marks the pods as to evict them. To help with reducing disk pressure, kubelet should delete all containers belonging to pods right away. This will result in loss of logs. In the future, we can copy out logs from dead containers before deleting them. Logs can be rotated separately from dead containers. IMO, this feature is required for disk usage based evictions to work reliably in kubelet.


Question: cmd := fmt.Sprintf("mkdir %v -m 777 && ln -s %v %v", h.sourcePath, h.sourcePath, h.targetPath) privileged := true <del> // Launch pod to initialize hostpath directory and symlink </del> <ins> // Launch pod to initialize hostPath directory and symlink </ins> h.prepPod = &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name: fmt.Sprintf("hostpath-symlink-prep-%s", f.Namespace.Name),

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: https://k8s- What happened: The skip check for hostpath doesn't seem to be working for these test cases
can you take a look?
Oh, disruptive tests are the tests that can't be tested locally in my environment due to I will take a look.
I found that above issue is not related, I should have been checked more carefully. Sorry for that.
I found out that this was just a driver name issue, so below will fix it: hostpath -hostPath hostpathSymlink -hostPathSymlink We might be better changing the skip logic later, to make it skip not by name in order to avoid this kind of bug, though. I will make a PR for above change, soon.
I made a PR above. Also, it might be a bit out of scope of this fix, but are drivers tested with containerized volume server, like nfs, ceph, etc..., safe to be tested with subpath disruptive tests? I'm worrying about the case that test pod and containerized volume server are deployed on the same node. In such a case, the same behaviour expected or do we need to avoid this kind of situation?


Question: <del> 1.5 </del> <ins> 1.6 </ins>

Warning: The v0.1 and v1beta1 metadata server endpoints are deprecated and scheduled for shutdown on September 30, 2020. This test (which runs in most e2e jobs that run on GCE): Schedules a container that runs this: Resulting in a number of legacy metadata endpoints getting hit. To see if this test is going to start failing across all release branches on Sep 30th, we could try disabling the endpoints ahead of time via: In fact and support for expecting different results based on whether is set. My concern is the test expects 200 for some root endpoints, and 403 for disabled endpoints. However the e-mail I received as a GCP customer tells me any legacy endpoints may return 404. I'm inclined to suggest we remove the legacy endpoints from the check, bump the image version, and cherry-pick use of the new image version back to all release-branches. WDYT since you were involved with the earlier PRs /sig auth /sig cloud-provider /provider gcp /sig testing /priority important-soon (I'm tempted to critical-urgent this but it's not drop-everything yet) /milestone v1.20
/kind cleanup
Heyho I wanted to check how it is going? The code freeze is starting 12.11.2020 (about 3-4 weeks from now) and while there is still plenty of time, we want to ensure that each relevant issue has a chance to be considered. As the issue is tagged for 1.20, is it still planned for this release?
Bug Triage here, we're pretty late into code freeze and it doesn't look like there has been much activity recently. I am going to clear the milestone on this issue. /milestone clear
any update here?
Try to cleanup the legacy endpoints in
/milestone v1.21
/reopen This needs to stay open until the new image has been verified to work and has been promoted (ref: )
Reopened this issue. <detailsIn response to : /reopen This needs to stay open until the new image has been verified to work and has been promoted (ref: ) Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage accepted /assign


Question: fi if [ -z "$PROJECT" ]; then <del> echo "Could not detect Google Cloud Platform project.  Set the default project using 'gcloud config set project <PROJECT>'" </del> <ins> echo "Could not detect Google Cloud Platform project.  Set the default project using 'gcloud config set project <PROJECT>'" 1>&2 </ins> exit 1 fi echo "Project: $PROJECT (autodetected from gcloud config)"

I'm following the documentation on this doc: Everything works until this command: I get this error consistently from different VMs (a Debian and a CoreOS). Maybe I have to edit /etc/hosts to add the "external-ip" with the public IP of some server? Docs don't seem clear. Thanks, Riccardo
Its failing to look up your master correctly. You can try setting KUBERNETES_MASTER=http://<your IP hearBrendan On Sep 21, 2014 5:45 AM, "Riccardo Carlesso" wrote:


Question: fs.Var(utilflag.IPVar{Val: &o.config.BindAddress}, "bind-address", "The IP address for the proxy server to serve on (set to `0.0.0.0` for all IPv4 interfaces and `::` for all IPv6 interfaces)") fs.StringVar(&o.master, "master", o.master, "The address of the Kubernetes API server (overrides any value in kubeconfig)") fs.Int32Var(&o.healthzPort, "healthz-port", o.healthzPort, "The port to bind the health check server. Use 0 to disable.") <del> fs.Var(utilflag.IPVar{Val: &o.config.HealthzBindAddress}, "healthz-bind-address", "The IP address and port for the health check server to serve on (set to `0.0.0.0` for all IPv4 interfaces and `::` for all IPv6 interfaces)") fs.Var(utilflag.IPVar{Val: &o.config.MetricsBindAddress}, "metrics-bind-address", "The IP address and port for the metrics server to serve on (set to `0.0.0.0` for all IPv4 interfaces and `::` for all IPv6 interfaces)") </del> <ins> fs.Var(utilflag.IPVar{Val: &o.config.HealthzBindAddress}, "healthz-bind-address", "The IP address for the health check server to serve on (set to `0.0.0.0` for all IPv4 interfaces and `::` for all IPv6 interfaces)") fs.Int32Var(&o.metricsPort, "metrics-port", o.metricsPort, "The port to bind the metrics server. Use 0 to disable.") fs.Var(utilflag.IPVar{Val: &o.config.MetricsBindAddress}, "metrics-bind-address", "The IP address for the metrics server to serve on (set to `0.0.0.0` for all IPv4 interfaces and `::` for all IPv6 interfaces)") </ins> fs.Int32Var(o.config.OOMScoreAdj, "oom-score-adj", utilpointer.Int32PtrDerefOr(o.config.OOMScoreAdj, int32(qos.KubeProxyOOMScoreAdj)), "The oom-score-adj value for kube-proxy process. Values must be within the range [-1000, 1000]") fs.StringVar(&o.config.ResourceContainer, "resource-container", o.config.ResourceContainer, "Absolute name of the resource-only container to create and run the Kube-proxy in (Default: /kube-proxy).") fs.MarkDeprecated("resource-container", "This feature will be removed in a later release.")

What happened: From We used to allow using or in and . Now the validation on kubeproxyconfig disallow that (it only allows ): This seems to break backward compatibility for those two flags. Please comment if I'm missing something. cc How to reproduce it (as minimally and precisely as possible): Try with kube-proxy. Environment: Kubernetes version (use ):
/sig network /kind bug
Tried , the error is: and , the error is: So maybe we need to loosen and revert ?
Loosen the validation sounds good. was meant for loosen the validation as well (allowing not just ip, but also ip:port), but for some reason it doesn't work as intended. Would be great to fix that instead of revert :)
find the root cause of not working as expected. introduced the regression: I send a PR to fix the two issues we mentioned in this thread.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with .

/close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I believe this hasn't been fixed yet. /remove-lifecycle rotten /reopen
Reopened this issue. <detailsIn response to : I believe this hasn't been fixed yet. /remove-lifecycle rotten /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I agree There's also the issue of IPv6-only: Results in:
Thanks for your report. Do you mind opening a new issue regarding IPv6 since the original issue was fixed? Thanks.
Sure. Thanks
Please see:


Question: REGISTRY ?= staging-k8s.gcr.io IMAGE = $(REGISTRY)/pause <del> IMAGE_WITH_OS_ARCH = $(IMAGE)-$(OS)-$(ARCH) TAG = 3.4 </del> <ins> TAG = 3.4.1 </ins> REV = $(shell git describe --contains --always --match='v*') # Architectures supported: amd64, arm, arm64, ppc64le and s390x

We could either: where appropriate - least amount of code change but cumbersome OS version in the tag instead of the repository /cc
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig windows
/kind bug


Question: <ins> /* Copyright 2020 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package testsuites import ( "fmt" "strconv" "github.com/onsi/ginkgo" v1 "k8s.io/api/core/v1" errors "k8s.io/apimachinery/pkg/util/errors" "k8s.io/kubernetes/test/e2e/framework" e2epod "k8s.io/kubernetes/test/e2e/framework/pod" e2eskipper "k8s.io/kubernetes/test/e2e/framework/skipper" e2evolume "k8s.io/kubernetes/test/e2e/framework/volume" "k8s.io/kubernetes/test/e2e/storage/testpatterns" storageutils "k8s.io/kubernetes/test/e2e/storage/utils" utilpointer "k8s.io/utils/pointer" ) const ( rootDir         = "/mnt/volume1" rootDirFile     = "file1" rootDirFilePath = rootDir + "/" + rootDirFile subdir          = "/mnt/volume1/subdir" subDirFile      = "file2" subDirFilePath  = subdir + "/" + subDirFile ) type fsGroupChangePolicyTestSuite struct { tsInfo TestSuiteInfo } var _ TestSuite = &fsGroupChangePolicyTestSuite{} // InitFsGroupChangePolicyTestSuite returns fsGroupChangePolicyTestSuite that implements TestSuite interface func InitFsGroupChangePolicyTestSuite() TestSuite { return &fsGroupChangePolicyTestSuite{ tsInfo: TestSuiteInfo{ Name: "fsgroupchangepolicy", TestPatterns: []testpatterns.TestPattern{ testpatterns.DefaultFsDynamicPV, }, SupportedSizeRange: e2evolume.SizeRange{ Min: "1Mi", }, }, } } func (s *fsGroupChangePolicyTestSuite) GetTestSuiteInfo() TestSuiteInfo { return s.tsInfo } func (s *fsGroupChangePolicyTestSuite) SkipRedundantSuite(driver TestDriver, pattern testpatterns.TestPattern) { skipVolTypePatterns(pattern, driver, testpatterns.NewVolTypeMap(testpatterns.CSIInlineVolume, testpatterns.GenericEphemeralVolume)) } func (s *fsGroupChangePolicyTestSuite) DefineTests(driver TestDriver, pattern testpatterns.TestPattern) { type local struct { config        *PerTestConfig driverCleanup func() driver        TestDriver resource      *VolumeResource } var l local ginkgo.BeforeEach(func() { dInfo := driver.GetDriverInfo() if !dInfo.Capabilities[CapFsGroup] { e2eskipper.Skipf("Driver %q does not support FsGroup - skipping", dInfo.Name) } if pattern.VolMode == v1.PersistentVolumeBlock { e2eskipper.Skipf("Test does not support non-filesystem volume mode - skipping") } if pattern.VolType != testpatterns.DynamicPV { e2eskipper.Skipf("Suite %q does not support %v", s.tsInfo.Name, pattern.VolType) } _, ok := driver.(DynamicPVTestDriver) if !ok { e2eskipper.Skipf("Driver %s doesn't support %v -- skipping", dInfo.Name, pattern.VolType) } }) // This intentionally comes after checking the preconditions because it // registers its own BeforeEach which creates the namespace. Beware that it // also registers an AfterEach which renders f unusable. Any code using // f must run inside an It or Context callback. f := framework.NewDefaultFramework("fsgroupchangepolicy") init := func() { e2eskipper.SkipIfNodeOSDistroIs("windows") l = local{} l.driver = driver l.config, l.driverCleanup = driver.PrepareTest(f) testVolumeSizeRange := s.GetTestSuiteInfo().SupportedSizeRange l.resource = CreateVolumeResource(l.driver, l.config, pattern, testVolumeSizeRange) } cleanup := func() { var errs []error if l.resource != nil { if err := l.resource.CleanupResource(); err != nil { errs = append(errs, err) } l.resource = nil } if l.driverCleanup != nil { errs = append(errs, tryFunc(l.driverCleanup)) l.driverCleanup = nil } framework.ExpectNoError(errors.NewAggregate(errs), "while cleanup resource") } tests := []struct { name                              string // Test case name podfsGroupChangePolicy            string // 'Always' or 'OnRootMismatch' initialPodFsGroup                 int    // FsGroup of the initial pod changedRootDirFileOwnership       int    // Change the ownership of the file in the root directory (/mnt/volume1/file1), as part of the initial pod changedSubDirFileOwnership        int    // Change the ownership of the file in the sub directory (/mnt/volume1/subdir/file2), as part of the initial pod secondPodFsGroup                  int    // FsGroup of the second pod finalExpectedRootDirFileOwnership int    // Final expcted ownership of the file in the root directory (/mnt/volume1/file1), as part of the second pod finalExpectedSubDirFileOwnership  int    // Final expcted ownership of the file in the sub directory (/mnt/volume1/subdir/file2), as part of the second pod }{ // Test cases for 'Always' policy { name:                              "pod created with an initial fsgroup, new pod fsgroup applied to volume contents", podfsGroupChangePolicy:            "Always", initialPodFsGroup:                 1000, secondPodFsGroup:                  2000, finalExpectedRootDirFileOwnership: 2000, finalExpectedSubDirFileOwnership:  2000, }, { name:                              "pod created with an initial fsgroup, volume contents ownership changed in first pod, new pod with same fsgroup applied to the volume contents", podfsGroupChangePolicy:            "Always", initialPodFsGroup:                 1000, changedRootDirFileOwnership:       2000, changedSubDirFileOwnership:        3000, secondPodFsGroup:                  1000, finalExpectedRootDirFileOwnership: 1000, finalExpectedSubDirFileOwnership:  1000, }, { name:                              "pod created with an initial fsgroup, volume contents ownership changed in first pod, new pod with different fsgroup applied to the volume contents", podfsGroupChangePolicy:            "Always", initialPodFsGroup:                 1000, changedRootDirFileOwnership:       2000, changedSubDirFileOwnership:        3000, secondPodFsGroup:                  4000, finalExpectedRootDirFileOwnership: 4000, finalExpectedSubDirFileOwnership:  4000, }, // Test cases for 'OnRootMismatch' policy { name:                              "pod created with an initial fsgroup, new pod fsgroup applied to volume contents", podfsGroupChangePolicy:            "OnRootMismatch", initialPodFsGroup:                 1000, secondPodFsGroup:                  2000, finalExpectedRootDirFileOwnership: 2000, finalExpectedSubDirFileOwnership:  2000, }, { name:                              "pod created with an initial fsgroup, volume contents ownership changed in first pod, new pod with same fsgroup skips ownership changes to the volume contents", podfsGroupChangePolicy:            "OnRootMismatch", initialPodFsGroup:                 1000, changedRootDirFileOwnership:       2000, changedSubDirFileOwnership:        3000, secondPodFsGroup:                  1000, finalExpectedRootDirFileOwnership: 2000, finalExpectedSubDirFileOwnership:  3000, }, { name:                              "pod created with an initial fsgroup, volume contents ownership changed in first pod, new pod with different fsgroup applied to the volume contents", podfsGroupChangePolicy:            "OnRootMismatch", initialPodFsGroup:                 1000, changedRootDirFileOwnership:       2000, changedSubDirFileOwnership:        3000, secondPodFsGroup:                  4000, finalExpectedRootDirFileOwnership: 4000, finalExpectedSubDirFileOwnership:  4000, }, } for _, t := range tests { test := t testCaseName := fmt.Sprintf("(%s)[LinuxOnly], %s", test.podfsGroupChangePolicy, test.name) ginkgo.It(testCaseName, func() { init() defer cleanup() policy := v1.PodFSGroupChangePolicy(test.podfsGroupChangePolicy) podConfig := e2epod.Config{ NS:                     f.Namespace.Name, NodeSelection:          l.config.ClientNodeSelection, PVCs:                   []*v1.PersistentVolumeClaim{l.resource.Pvc}, FsGroup:                utilpointer.Int64Ptr(int64(test.initialPodFsGroup)), PodFSGroupChangePolicy: &policy, } // Create initial pod and create files in root and sub-directory and verify ownership. pod := createPodAndVerifyContentGid(l.config.Framework, &podConfig, true /* createInitialFiles */, "" /* expectedRootDirFileOwnership */, "" /* expectedSubDirFileOwnership */) // Change the ownership of files in the initial pod. if test.changedRootDirFileOwnership != 0 { ginkgo.By(fmt.Sprintf("Changing the root directory file ownership to %s", strconv.Itoa(test.changedRootDirFileOwnership))) storageutils.ChangeFilePathGidInPod(f, rootDirFilePath, strconv.Itoa(test.changedRootDirFileOwnership), pod) } if test.changedSubDirFileOwnership != 0 { ginkgo.By(fmt.Sprintf("Changing the sub-directory file ownership to %s", strconv.Itoa(test.changedSubDirFileOwnership))) storageutils.ChangeFilePathGidInPod(f, subDirFilePath, strconv.Itoa(test.changedSubDirFileOwnership), pod) } ginkgo.By(fmt.Sprintf("Deleting Pod %s/%s", pod.Namespace, pod.Name)) framework.ExpectNoError(e2epod.DeletePodWithWait(f.ClientSet, pod)) // Create a second pod with existing volume and verify the contents ownership. podConfig.FsGroup = utilpointer.Int64Ptr(int64(test.secondPodFsGroup)) pod = createPodAndVerifyContentGid(l.config.Framework, &podConfig, false /* createInitialFiles */, strconv.Itoa(test.finalExpectedRootDirFileOwnership), strconv.Itoa(test.finalExpectedSubDirFileOwnership)) ginkgo.By(fmt.Sprintf("Deleting Pod %s/%s", pod.Namespace, pod.Name)) framework.ExpectNoError(e2epod.DeletePodWithWait(f.ClientSet, pod)) }) } } func createPodAndVerifyContentGid(f *framework.Framework, podConfig *e2epod.Config, createInitialFiles bool, expectedRootDirFileOwnership, expectedSubDirFileOwnership string) *v1.Pod { podFsGroup := strconv.FormatInt(*podConfig.FsGroup, 10) ginkgo.By(fmt.Sprintf("Creating Pod in namespace %s with fsgroup %s", podConfig.NS, podFsGroup)) pod, err := e2epod.CreateSecPodWithNodeSelection(f.ClientSet, podConfig, framework.PodStartTimeout) framework.ExpectNoError(err) framework.Logf("Pod %s/%s started successfully", pod.Namespace, pod.Name) if createInitialFiles { ginkgo.By(fmt.Sprintf("Creating a sub-directory and file, and verifying their ownership is %s", podFsGroup)) cmd := fmt.Sprintf("touch %s", rootDirFilePath) var err error _, _, err = storageutils.PodExec(f, pod, cmd) framework.ExpectNoError(err) storageutils.VerifyFilePathGidInPod(f, rootDirFilePath, podFsGroup, pod) cmd = fmt.Sprintf("mkdir %s", subdir) _, _, err = storageutils.PodExec(f, pod, cmd) framework.ExpectNoError(err) cmd = fmt.Sprintf("touch %s", subDirFilePath) _, _, err = storageutils.PodExec(f, pod, cmd) framework.ExpectNoError(err) storageutils.VerifyFilePathGidInPod(f, subDirFilePath, podFsGroup, pod) return pod } // Verify existing contents of the volume ginkgo.By(fmt.Sprintf("Verifying the ownership of root directory file is %s", expectedRootDirFileOwnership)) storageutils.VerifyFilePathGidInPod(f, rootDirFilePath, expectedRootDirFileOwnership, pod) ginkgo.By(fmt.Sprintf("Verifying the ownership of sub directory file is %s", expectedSubDirFileOwnership)) storageutils.VerifyFilePathGidInPod(f, subDirFilePath, expectedSubDirFileOwnership, pod) return pod } </ins>

We need to add e2e that verifies volume permissions in pod with different values of fsGroupChangePolicy. /sig storage
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign


Question: NamespaceParam(cmdNamespace).DefaultNamespace(). FilenameParam(enforceNamespace, &o.FilenameOptions). ResourceTypeOrNameArgs(enforceNamespace, args...). <ins> SelectAllParam(o.All). </ins> SelectorParam(o.Selector). Latest(). Flatten().

It actually doesn't support --all.
/assign
ohh, thanks, will make a pr to address that right now..


Question: import ( "context" <ins> "encoding/json" </ins> "fmt" "math/rand" "time"

[x] two weeks soak end date : 26 Feb 2021 According to this APIsnoop query, there are still some remaining Deployment endpoints which are untested. with this query you can filter untested endpoints by their category and eligibility for conformance. e.g below shows a query to find all conformance eligible untested,stable,core endpoints Create a Deployment with a static label Read the Deployment Update the DeploymentScale Get the DeploymentScale to ensure it is updated Patch the DeploymentScale Read the DeploymentScale to ensure it is Patch Delete the Namespace and Deployment Due to the complexity of setting up the resources for APIService we have used the current e2e test as template. It has been extended in a for review. example endpoint | useragent ----------------------------------------|
patchAppsV1NamespacedDeploymentScale | (linux/amd64) kubernetes/$Format -- [sig-apps] Deployment Deployment should have a working scale subresource readAppsV1NamespacedDeploymentScale | (linux/amd64) kubernetes/$Format -- [sig-apps] Deployment Deployment should have a working scale subresource replaceAppsV1NamespacedDeploymentScale | (linux/amd64) kubernetes/$Format -- [sig-apps] Deployment Deployment should have a working scale subresource (3 rows) If a test with these calls gets merged, test coverage will go up by N points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted
Issue was presented in the Conformance meeting of 9 Feb 2021 and it was agreed that that method see to achieve the purpose and a PR can be created. A similar approach would be taken for the DepoymentScale endpoints, and this issue was created for that purpose.
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: klog.Errorf("%s: error determining whether payload should be written to disk: %v", w.logContext, err) return err } else if !should && len(pathsToRemove) == 0 { <del> klog.V(4).Infof("%s: no update required for target directory %v", w.logContext, w.targetDir) return nil </del> <ins> klog.V(4).Infof("%s: write not required for data directory %v", w.logContext, oldTsDir) // data directory is already up to date, but we need to make sure that // the user-visible symlinks are created. // See https://github.com/kubernetes/kubernetes/issues/121472 for more details. // Reset oldTsDir to empty string to avoid removing the data directory. shouldWrite = false oldTsDir = "" </ins> } else { klog.V(4).Infof("%s: write required for target directory %v", w.logContext, w.targetDir) } } <del> // (5) tsDir, err := w.newTimestampDir() if err != nil { klog.V(4).Infof("%s: error creating new ts data directory: %v", w.logContext, err) return err } tsDirName := filepath.Base(tsDir) // (6) if err = w.writePayloadToDir(cleanPayload, tsDir); err != nil { klog.Errorf("%s: error writing payload to ts data directory %s: %v", w.logContext, tsDir, err) return err } klog.V(4).Infof("%s: performed write of new data to ts data directory: %s", w.logContext, tsDir) </del> <ins> if shouldWrite { // (5) tsDir, err := w.newTimestampDir() if err != nil { klog.V(4).Infof("%s: error creating new ts data directory: %v", w.logContext, err) return err } tsDirName := filepath.Base(tsDir) </ins> <del> // (7) if setPerms != nil { if err := setPerms(tsDirName); err != nil { klog.Errorf("%s: error applying ownership settings: %v", w.logContext, err) </del> <ins> // (6) if err = w.writePayloadToDir(cleanPayload, tsDir); err != nil { klog.Errorf("%s: error writing payload to ts data directory %s: %v", w.logContext, tsDir, err) </ins> return err } <del> } </del> <ins> klog.V(4).Infof("%s: performed write of new data to ts data directory: %s", w.logContext, tsDir) </ins> <del> // (8) newDataDirPath := filepath.Join(w.targetDir, newDataDirName) if err = os.Symlink(tsDirName, newDataDirPath); err != nil { os.RemoveAll(tsDir) klog.Errorf("%s: error creating symbolic link for atomic update: %v", w.logContext, err) return err } </del> <ins> // (7) if setPerms != nil { if err := setPerms(tsDirName); err != nil { klog.Errorf("%s: error applying ownership settings: %v", w.logContext, err) return err } } </ins> <del> // (9) if runtime.GOOS == "windows" { os.Remove(dataDirPath) err = os.Symlink(tsDirName, dataDirPath) os.Remove(newDataDirPath) } else { err = os.Rename(newDataDirPath, dataDirPath) } if err != nil { os.Remove(newDataDirPath) os.RemoveAll(tsDir) klog.Errorf("%s: error renaming symbolic link for data directory %s: %v", w.logContext, newDataDirPath, err) return err </del> <ins> // (8) newDataDirPath := filepath.Join(w.targetDir, newDataDirName) if err = os.Symlink(tsDirName, newDataDirPath); err != nil { if err := os.RemoveAll(tsDir); err != nil { klog.Errorf("%s: error removing new ts directory %s: %v", w.logContext, tsDir, err) } klog.Errorf("%s: error creating symbolic link for atomic update: %v", w.logContext, err) return err } // (9) if runtime.GOOS == "windows" { if err := os.Remove(dataDirPath); err != nil { klog.Errorf("%s: error removing data dir directory %s: %v", w.logContext, dataDirPath, err) } err = os.Symlink(tsDirName, dataDirPath) if err := os.Remove(newDataDirPath); err != nil { klog.Errorf("%s: error removing new data dir directory %s: %v", w.logContext, newDataDirPath, err) } } else { err = os.Rename(newDataDirPath, dataDirPath) } if err != nil { if err := os.Remove(newDataDirPath); err != nil && err != os.ErrNotExist { klog.Errorf("%s: error removing new data dir directory %s: %v", w.logContext, newDataDirPath, err) } if err := os.RemoveAll(tsDir); err != nil { klog.Errorf("%s: error removing new ts directory %s: %v", w.logContext, tsDir, err) } klog.Errorf("%s: error renaming symbolic link for data directory %s: %v", w.logContext, newDataDirPath, err) return err } </ins> } // (10)

When kubelet is creating configmap volume for a pod, it finished writing dir and made symlink to . At the same time, the kubelet is restarted, unfortunately the configmap item symlink is not created. The restarted kubelet will try to configmap, . Kubelet should create absent symlink for configmap volume. can hack kubelet to help reproduce it: sleep some time before ` kubelet and start official kubelet will skip to create symlink for UserVisibleFiles No response_ <details</details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
/sig storage This seems like an issue for sig storage. Can someone take a look and confirm?
Can you please add the version details as well?
/remove-sig node
/assign


Question: } } if err := plugin.delContainerFromNetwork(plugin.netConfig, network.DefaultInterfaceName, namespace, name, id); err != nil { <ins> // This is to prevent returning error when TearDownPod is called twice on the same pod. This helps to reduce event pollution. if !hasCIDR { glog.Warningf("Failed to delete container from kubenet: %v", err) return nil } </ins> return err } delete(plugin.podCIDRs, id)

Kubelet log is full of errors from tearing down network at HEAD. From the log, a lot of them were the result of trying to access . Note that if you an exited container, the returned PID will be 0. Therefore, my guess is that kubelet tried to tear down the network after the container has already exited. I didn't dig further. could you take a look?
/cc
Is this a dup of ? We can add a safe guard to prevent TearDown to proceed, but it will not solve the root cause. IIUC,the root cause is the race between syncPod and podKiller.
See also
I don't think failed because of this. this is probably caused by , like you said. The fact that we are seeing more errors than before is likely due to , which enforces graceless namespace deletions. Before that, most objects in the e2e were deleted gracefully and the race condition was rarely hit. Now most objects will hit this path. It'd still be a good idea to add a safeguard to check whether the pid is 0.


Question: func (c *Fake$.type|publicPlural$) Create(ctx context.Context, $.type|private$Name string, $.inputType|private$ *$.inputType|raw$, opts $.CreateOptions|raw$) (result *$.resultType|raw$, err error) { obj, err := c.Fake. $if .namespaced$Invokes($.NewCreateSubresourceAction|raw$($.type|allLowercasePlural$Resource, $.type|private$Name, "$.subresourcePath$", c.ns, $.inputType|private$), &$.resultType|raw${}) <del> $else$Invokes($.NewRootCreateSubresourceAction|raw$($.type|allLowercasePlural$Resource, "$.subresourcePath$", $.inputType|private$), &$.resultType|raw${})$end$ </del> <ins> $else$Invokes($.NewRootCreateSubresourceAction|raw$($.type|allLowercasePlural$Resource, $.type|private$Name, "$.subresourcePath$", $.inputType|private$), &$.resultType|raw${})$end$ </ins> if obj == nil { return nil, err }

What happened: Test create method for non-namespaced subresources generate invalid code. I have the following code generation annotations: which generated test method with generated fake client: What you expected to happen: It should be: How to reproduce it (as minimally and precisely as possible): Generate clients for non-namespaced resources with subresources (like scale) like this: /sig api-machinery /area code-generator /assign
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : What happened: What you expected to happen: It should be: How to reproduce it (as minimally and precisely as possible): Generate clients for non-namespaced resources with subresources (like scale) like this: /sig api-machinery /area code-generator /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/triage accepted


Question: } } <del> // TODO(bsalamat): This way of updating equiv. cache has a race condition against // cache invalidations invoked in event handlers. This race has existed despite locks // in eCache implementation. If cache is invalidated after a predicate is executed // and before we update the cache, the updates should not be written to the cache. if eCacheAvailable { nodeName := info.Node().GetName() for predKey, result := range predicateResults { // update equivalence cache with newly computed fit & reasons // TODO(resouer) should we do this in another thread? any race? ecache.UpdateCachedPredicateItem(pod.GetName(), nodeName, predKey, result.Fit, result.FailReasons, equivCacheInfo.hash) } } </del> return len(failedPredicates) == 0, failedPredicates, nil }

Fix this TODO (raised by : Part of: One thoughts to fix this is, we can redesign the equivalence cache data structure to: bool instead of deleting the entry. certain version tag in the entry, update the version per WRITE op (Update and Invalidate). the version after eCache predicate, and check it before update eCache.
Another possible approach is: eCache before running a predicate. the predicate. the eCache for the predicate. eCache. Basically, in the existing code all predicates are run first and their results are collected and then at the end of the function the eCache is updated. Instead of that, we can move the eCache update for each predicate inside the for loop and next to each predicate execution and synchronize before start of the execution of each predicate. The downside to my suggested approach is that performance may suffer due to frequent locking and unlocking.
Prefer to bsalamat@ 's option which is simpler :). Comparing to performance improvement by eCache, the lock is acceptable to me.
yup same here, going with the locking is easier and is acceptable. - also using a flag for invalidation might just move the problem from eCache to the flag ?
Oh I missed this notification somehow, busy on cleaning scheduler integration test. Yep so if locking is acceptable, it should be cleanest way to fix this, optimistic control would be a more complicated story.
- are you handling this one ?
Already been taken by volunteer days ago :)


Question: // owner: @tkashem // beta: v1.29 <ins> // GA: v1.30 </ins> // // Allow Priority & Fairness in the API server to use a zero value for // the 'nominalConcurrencyShares' field of the 'limited' section of a

In 1.30, Priority and Fairness should allow a zero value, either via v1 or v1beta3 (with the roundtrip annotation) in create and update requests See:
/sig api-machinery /cc /assign
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /triage accpted Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage accepted


Question: ErrCloudInstance  = errors.New("cloud provider doesn't support instances.") ) <ins> var ( // aliased to allow mocking in tests lookupIP = net.LookupIP ) </ins> type NodeController struct { cloud              cloudprovider.Interface matchRE            string

It seems like this depends on the network condition as I first noticed this while on a plane, and it happens to me every so often but never on travis. Offenders are TestSyncNodeStatusDeletePods and TestSyncNodeStatusTransitionTime. This may be reproducible by disconnecting network connection but also happens sometimes while connected. is a call to net.LookupIP(), I'll investigate more.
This also occurs on OSX over a VPN (at least, for my VPN it does). Each call to net.LookupIP(name) takes 30s and fails correctly. As a result, the entire unit test times out. That same function call fails fast on Linux.
cc
For other runtime, we can just mock out net, time. I'd really like to know how to do mocking in go. Using interface seems overkill.


Question: // TestPreemptWithPermitPlugin tests preempt with permit plugins. func TestPreemptWithPermitPlugin(t *testing.T) { // Create a plugin registry for testing. Register only a permit plugin. <del> registry := framework.Registry{permitPluginName: NewPermitPlugin} </del> <ins> perPlugin := &PermitPlugin{} registry := framework.Registry{permitPluginName: NewPermitPlugin(perPlugin)} </ins> // Setup initial permit plugin for testing. plugins := &schedulerconfig.Plugins{

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: pull-kubernetes-integration Which test(s) are failing: TestPreemptWithPermitPlugin Since when has it been failing: 13:30 PST, Sep 9, 2019 Testgrid link: Reason for failure: Need investigations. Anything else we need to know: /sig scheduling
Another flaky It happens at 2019-08-06 05:43:47 +0000 UTC
The above repeated for 30 seconds.
I checked the code and log again. I think the reason is as following: When scheduler preempted , it failed to delete the pod(same as the case for , there might be something wrong with etcd), so it could not schedule because had not been deleted successfully.
/assign Find the root cause now, will send a PR soon.
Thanks for your work! Not sure whether you are still investigating it, the flaky test is impacting many PRs now, so I send a PR for it now, sorry for it.


Question: } o.ServerSideApply = cmdutil.GetServerSideApplyFlag(cmd) <ins> o.FieldManager = cmdutil.GetFieldManagerFlag(cmd) </ins> o.ForceConflicts = cmdutil.GetForceConflictsFlag(cmd) if o.ForceConflicts && !o.ServerSideApply { return fmt.Errorf("--force-conflicts only works with --server-side")

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: fails. What you expected to happen: I would expect the PatchOptions to not include FieldManager. The following fix will eliminate the problem. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): 1.16 and 1.17 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig cli
cc
/wg apply
cc
I agree, we clearly missed that. Would you mind sending the fix in the form of a PR? Also please include a test, because that shouldn't have happened if I had tested my code properly :-(
Clearly we should also cherry-pick that into 1.17. I'll send the PR on Monday if you prefer! Thanks.
Ok, I will send a PR. However, I can't include a test immediately because I don't understand how the test works.
Thanks, I’ll try to show you how to get started!
Any update we clearly need this promptly since code-freeze is coming! Thanks.
I'll send a PR on Monday if you haven't! Thanks.
Sorry, I haven't started working on this test. Thanks.


Question: "fmt" "time" <ins> apierrors "k8s.io/kubernetes/pkg/api/errors" </ins> "k8s.io/kubernetes/pkg/api/v1" legacyv1 "k8s.io/kubernetes/pkg/api/v1" metav1 "k8s.io/kubernetes/pkg/apis/meta/v1"

Three tests now create RoleBindings to give permissions to test service accounts: NodeProblemDetector Ingress PreStop Creating RBAC Rolebindings currently requires the user issuing the create call to have the required permissions. If RBAC is not enabled, and the cluster hasn't been configured with any RBAC bootstrapping, that won't be possible. We should make this test code conditional on whether RBAC is in use.
c.f.
Quick fix is to tolerate forbidden errors there and assume e2e setups using methods other than RBAC have already granted permissions to the service accounts
I'll put together a PR to tolerate the 403 in the test helper.
I think this can be closed now that the e2e RBAC setup is best effort and failures there don't affect the test if another authorizer is in use
I agree. My original thought was that we'd make the RBAC test stuff only happen on RBAC-enabled setups. But, since we're going to be pushing setups towards using RBAC, we might as well have those pieces stick around in the common code (as best-effort).


Question: } } } <ins> // TestFIFO_PopShouldUnblockWhenClosed checks that any blocking Pop on an empty queue // should unblock and return after Close is called. func TestFIFO_PopShouldUnblockWhenClosed(t *testing.T) { f := NewFIFO(testFifoObjectKeyFunc) c := make(chan struct{}) const jobs = 10 for i := 0; i < jobs; i++ { go func() { f.Pop(func(obj interface{}) error { return nil }) c <- struct{}{} }() } runtime.Gosched() f.Close() for i := 0; i < jobs; i++ { select { case <-c: case <-time.After(500 * time.Millisecond): t.Fatalf("timed out waiting for Pop to return after Close") } } } </ins>

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Race conditions found on 2 usages of sync.Cond in client-go/tools/cache.DeltaFIFO and client-go/tools/watch/eventProcessor. I don't know how harmful they are since the buggy interleaving is hard to trigger. They may lead to goroutine leaks to my understanding. They have similar causes and I put them in one issue here for ease of triage. When and are invoked concurrently on an empty queue, the in may miss the signal from . The is guarded by the for-loop below. It appears that should not be called if the FIFO queue is already closed, hence the check at line 475. However, the check above may race with : Specifically, when the queue is empty and not closed, will reach the . Between the check and , another goroutine can call and the broadcast on will be missed. Consequently, may block indefinintely and miss the broadcast on from . Note that holds while and hold . So can interleave with . Here is a stack trace found by our tool while running . The tool is based on the Go race detector. I did some research and found introduced the bug while trying to fix exactly the issue: "stop controller when the stop channel is closed and the queue is empty". The fix doesn't appear to solve the issue completely. One suggested fix: use in and instead of . This would enforce atomicity between the check and in . This is similar to DeltaFIFO. In between and : Another goroutine can call : This would cause the signal on to be missed by the above , leading to the goroutine being blocked unexpectedly. Stack trace from : One suggested fix: use mutex to protect in . What you expected to happen: No race condition. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): MacOS Kernel (e.g. ): Darwin Install tools: Network plugin and version (if this is a network-related bug): Others: This is my first issue filed.

Please let me know if I can provide further information.
/sig api-machinery
For issue 1, in probably the date race is not uncovered due to Run() being long running. If we use f.lock in f.Close() and f.IsClosed(), closedLock is not needed any more. Since f.lock is taken at the beginning of Pop(), we would need lockless version of IsClosed()
Oh..Right. I didn't see this point. Please ignore my suggested fix for issue 1 then.
/cc
Hi, I reproduced the goroutine leak with this test case. The control loop inside may not exit when is called on an empty queue due to the race condition. Also, I submitted a PR trying to fix it. :smile:


Question: svc.Spec.Type = v1.ServiceTypeLoadBalancer enableILB(svc) }) <ins> defer func() { ginkgo.By("Clean up loadbalancer service") e2eservice.WaitForServiceDeletedWithFinalizer(cs, svc.Namespace, svc.Name) }() </ins> svc = jig.WaitForLoadBalancerOrFail(namespace, serviceName, createTimeout) jig.SanityCheckService(svc, v1.ServiceTypeLoadBalancer) lbIngress := &svc.Status.LoadBalancer.Ingress[0]

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: ci-kubernetes-e2e-gce-scale-correctness Which test(s) are failing: [sig-network] Services should only allow access from service loadbalancer source ranges [Slow] Since when has it been failing: It has been failing since 8/28. Here is the k/k diff for additional context: Testgrid link: Reason for failure: Seems like they were two instances of the same test (at least in the latest run). One of them failed with the following error message: The other one with: /milestone v1.16 /priority critical-urgent /kind failing-test /cc /cc /cc
/sig network
- are you aware of any recent changes that might affect this tests (e.g. judging just by title)? It fails deterministically.
/remove-kind bug Actually
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
Those labels are not set on the issue: <detailsIn response to : /remove-kind node /assign - any chance this is related to ? This is a little wild guess, but from logs it seems that the namespace wasn't deleted, because objects other that pods were not deleted () Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Those labels are not set on the issue: <detailsIn response to : /remove-sig node Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Those labels are not set on the issue: <detailsIn response to : /remove-kind node /assign - any chance this is related to ? This is a little wild guess, but from logs it seems that the namespace wasn't deleted, because objects other that pods were not deleted () Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign Any chance this is related to ?

This is a little wild guess, but from logs it seems that the namespace wasn't deleted, because objects other that pods were not deleted ()
can you please let the 1.16 release team if you are going to be able to investigate and provide if you can help an approximate time for a fix?
Acked, I'm looking into the failure and will provide an update soon.
Fantastic! Thanks for the update
Yep, this is related to With service finalizer enabled by default, deleting the namespace (including services) now takes longer --- it now blocks on all the underlying load balancer resources to be cleaned up. It is expected that the latency for resource clean up might be longer on large cluster (such as in this test). I will soon send a PR to overwrite the  with something more reasonable in this case.
Sent to fix this.
Reopening this one to track the job's health /reopen
Reopened this issue. <detailsIn response to : Reopening this one to track the job's health /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Yesterday's run has failed due to problems with creating a test cluster. Hopefully we get an answer whether the fix has helped today.
Thanks for tracking this, seems like the latest run is green:
It looks good indeed, thanks /close
Closing this issue. <detailsIn response to : It looks good indeed, thanks </details


Question: waitGroup.Wait() verifyCalls(t, fakeDocker, []string{ <del> "list", "list", "stop", "create", "start", "inspect_container", "create", "start"}) </del> <ins> "list", "list", "list", "list", "inspect_container", "inspect_container", "stop", "create", "start", "inspect_container", "create", "start"}) </ins> // A map iteration is used to delete containers, so must not depend on // order here.

Observed in testing somehting unrelated - prio We need a test that captures this - we can't let this slip through, Maybe same as ?
I just reproduced this with newly built cluster from HEAD. This is definitely a regression introduced lately. Will look into it in detail.


Question: func initImageConfigs() (map[int]Config, map[int]Config) { configs := map[int]Config{} <del> configs[Agnhost] = Config{promoterE2eRegistry, "agnhost", "2.29"} </del> <ins> configs[Agnhost] = Config{promoterE2eRegistry, "agnhost", "2.30"} </ins> configs[AgnhostPrivate] = Config{PrivateRegistry, "agnhost", "2.6"} configs[AuthenticatedAlpine] = Config{gcAuthenticatedRegistry, "alpine", "3.7"} configs[AuthenticatedWindowsNanoServer] = Config{gcAuthenticatedRegistry, "windows-nanoserver", "v1"}

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --aks-engine-azure-windows-master-staging ServiceAccounts ServiceAccountIssuerDiscovery should support OIDC discovery of service account issuer [Conformance] /sig windows /cc
cc
/assign
/assign Let me know if you want to pair on this
We took a look at this for docker case and we believe it is happening due to Windows DNS networking limitations. DNS can be available sometime after the container starts due to the way networking is set up with docker. In containerd the setup steps are different so the network will be available when the container starts. In this case the test image does a lookup before the DNS is finished coming online and fails.
it fails in linux as well
From the link posted, it looks like the test cluster issuer might be misconfigured (--service-account-issuer is missing https, I'm not sure why this even passed kube-apiserver validation, double-checking that also): Try setting on kube-apiserver.
reminded me we couldn't actually add that validation, because we weren't previously validating schemes and it would've broken existing clusters with bad issuers set... So as far as fixing the tests make sure issuers have schemes, at least.
I guess the config in this case is managed behind AKS service, and AKS has been setting without https scheme?
Thanks! The failures on the Linux tests are a separate issue from the Windows failures on docker. I opened kubernetes-sigs/cloud-provider-azure to resolve the Linux failures (we have already done this for Windows: )
/reopen This is still an issue for windows docker. fixes it
Reopened this issue. <detailsIn response to : /reopen This is still an issue for windows docker. fixes it Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I'm glad this was fixed for open-source test configs, but I assume since the test said "aks" in the name this could be an issue for managed Azure clusters?

Clients should reject issuers without https instead of attempting to validate them, so we'd want all clusters to be using https scheme in the issuer URL, but at the same time, changing the issuer URL on an existing cluster invalidates previously-issued tokens, because kube-apiserver would no longer recognize the old issuer. I just want to confirm that what was happening with tests isn't foreshadowing a bigger issue for existing Azure clusters. can you/someone at Microsoft confirm? FYI
We use aks-engine (self-managed cluster instead of managed AKS cluster) to run the tests. For aks-engine, I opened a PR () to fix the service account issuer. For AKS, I can confirm that the service account issuer does not any scheme in the URL at all. Tagging who works on aks-engine. Tagging who works on AKS.
We confirmed that is being fixed in AKS.
/traige accepted /milestone v1.21 /priority important-soon
/triage accepted
Our ipv6 test is still failing: The service issuer has been set to .
that seems to be a different issue than Windows. could you open a new issue and tag and me?


Question: Protocol: v1.ProtocolTCP, }}, }}, <del> }, metav1.CreateOptions{}) framework.ExpectNoError(err, "failed to create Endpoint") </del> <ins> } w := &cache.ListWatch{ WatchFunc: func(options metav1.ListOptions) (watch.Interface, error) { options.LabelSelector = "test-endpoint-static=true" return f.ClientSet.CoreV1().Endpoints(testNamespaceName).Watch(context.TODO(), options) }, } endpointsList, err := f.ClientSet.CoreV1().Endpoints("").List(context.TODO(), metav1.ListOptions{LabelSelector: "test-endpoint-static=true"}) framework.ExpectNoError(err, "failed to list Endpoints") </ins> <del> // set up a watch for the Endpoint // this timeout was chosen as there was timeout failure from the CI endpointWatchTimeoutSeconds := int64(180) endpointWatch, err := f.ClientSet.CoreV1().Endpoints(ns).Watch(context.TODO(), metav1.ListOptions{LabelSelector: "testendpoint-static=true", TimeoutSeconds: &endpointWatchTimeoutSeconds}) framework.ExpectNoError(err, "failed to setup watch on newly created Endpoint") endpointWatchChan := endpointWatch.ResultChan() </del> <ins> ginkgo.By("creating an Endpoint") _, err = f.ClientSet.CoreV1().Endpoints(testNamespaceName).Create(context.TODO(), &testEndpoints, metav1.CreateOptions{}) framework.ExpectNoError(err, "failed to create Endpoint") </ins> ginkgo.By("waiting for available Endpoint") <del> for watchEvent := range endpointWatchChan { if watchEvent.Type == "ADDED" { break </del> <ins> ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second) defer cancel() _, err = watchtools.Until(ctx, endpointsList.ResourceVersion, w, func(event watch.Event) (bool, error) { switch event.Type { case watch.Added: if endpoints, ok := event.Object.(*v1.Endpoints); ok { found := endpoints.ObjectMeta.Name == endpoints.Name && endpoints.Labels["test-endpoint-static"] == "true" return found, nil } default: framework.Logf("observed event type %v", event.Type) </ins> } <del> } </del> <ins> return false, nil }) framework.ExpectNoError(err, "failed to see %v event", watch.Added) </ins> ginkgo.By("listing all Endpoints") <del> endpointsList, err := f.ClientSet.CoreV1().Endpoints("").List(context.TODO(), metav1.ListOptions{LabelSelector: "testendpoint-static=true"}) </del> <ins> endpointsList, err = f.ClientSet.CoreV1().Endpoints("").List(context.TODO(), metav1.ListOptions{LabelSelector: "test-endpoint-static=true"}) </ins> framework.ExpectNoError(err, "failed to list Endpoints") <del> foundEndpointService := false </del> <ins> eventFound := false </ins> var foundEndpoint v1.Endpoints for _, endpoint := range endpointsList.Items { <del> if endpoint.ObjectMeta.Name == testEndpointName && endpoint.ObjectMeta.Namespace == ns { foundEndpointService = true </del> <ins> if endpoint.ObjectMeta.Name == testEndpointName && endpoint.ObjectMeta.Namespace == testNamespaceName { eventFound = true </ins> foundEndpoint = endpoint break } } <del> framework.ExpectEqual(foundEndpointService, true, "unable to find Endpoint Service in list of Endpoints") </del> <ins> framework.ExpectEqual(eventFound, true, "unable to find Endpoint Service in list of Endpoints") </ins> ginkgo.By("updating the Endpoint") <del> foundEndpoint.ObjectMeta.Labels["testservice"] = "first-modification" _, err = f.ClientSet.CoreV1().Endpoints(ns).Update(context.TODO(), &foundEndpoint, metav1.UpdateOptions{}) </del> <ins> foundEndpoint.ObjectMeta.Labels["test-service"] = "updated" _, err = f.ClientSet.CoreV1().Endpoints(testNamespaceName).Update(context.TODO(), &foundEndpoint, metav1.UpdateOptions{}) </ins> framework.ExpectNoError(err, "failed to update Endpoint with new label") <ins> ctx, cancel = context.WithTimeout(context.Background(), 30*time.Second) defer cancel() _, err = watchtools.Until(ctx, endpointsList.ResourceVersion, w, func(event watch.Event) (bool, error) { switch event.Type { case watch.Modified: if endpoints, ok := event.Object.(*v1.Endpoints); ok { found := endpoints.ObjectMeta.Name == endpoints.Name && endpoints.Labels["test-endpoint-static"] == "true" return found, nil } default: framework.Logf("observed event type %v", event.Type) } return false, nil }) framework.ExpectNoError(err, "failed to see %v event", watch.Modified) </ins> ginkgo.By("fetching the Endpoint") <del> _, err = f.ClientSet.CoreV1().Endpoints(ns).Get(context.TODO(), testEndpointName, metav1.GetOptions{}) </del> <ins> endpoints, err := f.ClientSet.CoreV1().Endpoints(testNamespaceName).Get(context.TODO(), testEndpointName, metav1.GetOptions{}) </ins> framework.ExpectNoError(err, "failed to fetch Endpoint") <del> framework.ExpectEqual(foundEndpoint.ObjectMeta.Labels["testservice"], "first-modification", "label not patched") </del> <ins> framework.ExpectEqual(foundEndpoint.ObjectMeta.Labels["test-service"], "updated", "failed to update Endpoint %v in namespace %v label not updated", testEndpointName, testNamespaceName) </ins> endpointPatch, err := json.Marshal(map[string]interface{}{ "metadata": map[string]interface{}{ "labels": map[string]string{ <del> "testservice": "second-modification", </del> <ins> "test-service": "patched", </ins> }, }, "subsets": []map[string]interface{}{

[x] Original approval issue: [x] Two weeks soak end date: This issue is created to allow edit by /wip /hold This issue proposes the writing of a new test for the API endponts , , , with the intention of promoting it to Conformance. This issue contains a very basic mock test in order to start the conversation about what a good e2e test would look like for this endpoint. The code below is verified to be hitting the intended endpoint, show as per the queries in APIsnoop's live view of the cluster's audit logs. According to this APIsnoop query, there are still some remaining Endpoints endpoints which are untested. a Service in a namespace all Endpoints in all Namespaces find the created Service(1) /update the Namespaced Endpoint for the Service(1) the Namespaced Endpoint for the Service(1) the Namespaced Endpoint for the Service(1) Namespaced Endpoint(1) via a Collection with a LabelSelector Discover useragents: If a test with these calls gets merged, Conformance coverage will go up by 4 points* /sig testing /sig architecture /area conformance
/reopen Wait for Promotion PR to Merge.
Reopened this issue. <detailsIn response to : /reopen Wait for Promotion PR to Merge. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/fields" "k8s.io/apimachinery/pkg/util/uuid" <ins> "k8s.io/apimachinery/pkg/util/wait" </ins> clientset "k8s.io/client-go/kubernetes" <ins> "k8s.io/client-go/kubernetes/scheme" </ins> "k8s.io/kubernetes/pkg/kubelet/events" "k8s.io/kubernetes/test/e2e/feature" "k8s.io/kubernetes/test/e2e/framework" e2epod "k8s.io/kubernetes/test/e2e/framework/pod" e2eoutput "k8s.io/kubernetes/test/e2e/framework/pod/output" testutils "k8s.io/kubernetes/test/utils" <ins> "k8s.io/kubernetes/test/utils/format" </ins> imageutils "k8s.io/kubernetes/test/utils/image" admissionapi "k8s.io/pod-security-admission/api" <ins> "k8s.io/utils/ptr" </ins> ) const runAsUserNameContainerName = "run-as-username-container"

Install a cluster with v1.29.4 with linux node as master and worker node as windows kubeadm upgrade apply v1.30.0 fails with --ignore-preflight-errors` Describe Pod events: It seems that previously the pod was Pending as well, but this was ignored, because the jobs was successfully deleted in defer and return value was overridden with nil. Upgrade health check job expected to be scheduled only on Linux nodes using node selectors, rather than on Windows nodes. See What happened? No response_ <detailskubeadm version (use kubeadm version): v1.30.0 Environment: Kubernetes version (use kubectl version): v1.30.0 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Ubuntu 22.04.2 LTS, Windows Server 2022 Datacenter Kernel (e.g. uname -a): 5.19.0-1025-aws Container networking plugin (CNI): calico </details<details</details<details</details<details</details<details</details<details</details
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
fixed in 1.30.1 /close
Closing this issue. <detailsIn response to : kubeadm version (use kubeadm version): v1.30.0 Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/transfer kubeadm
we are seeing this issue for 1.30.1 version too with upgrade health check pods getting scheduled on windows worker nodes.
looks like I was mistaken and this issue is different. but the health check uses the pause image, which is multi os, supposedly. the error is strange. can you show the full log output. we might have to apply a node selector on the job spec and backport to 1.31 another bug fix.

I'm not sure if we should add a node selector on job spec. the job uses the pause image to create a container. It doesn't have specific requirements for the node. After I look at the projected volume plugin, I'm confused why the above event occurs. On windows nodes, the function is empty. It doesn't do anything. On linux nodes, the function will set the owner of the volume. FYI: If a log file from the kubelet is provided, it will be helpful to understand the issue.
thanks for checking I also think there should not be an error. maybe we are hitting a core kubelet bug. is the kubelet on the windows node also 1.30?
please ignore the above comment, it's incorrect. I may find the root cause. It's a bug in kubelet when kubelet is running on Windows and the pod configures the SecurityContext with a RunAsUser.
/sig storage /sig windows
/assign I'm trying to fix the kubelet bug.
so this is not a kubeadm bug and we can move it back to the k/k repo?
/transfer kubernetes
/retitle kubelet on Windows fails if a pod has SecurityContext with RunAsUser.
Because of historical reasons, the and in is only designed for the Linux, not Windows. See for more details. adds a new field to the PodSpec, which can be used to identify the OS of the pod. It's an optional field and no default value. If unset, the above fields are not restricted by the api-server. In other words, the following job is allowed to be created. But it will fails to start on Windows nodes. And the root cause is that the is called by the . To fix this issue, the AtomicWriter can check the current OS of the node and skip the if the OS is Windows. I want to confirm with sig-storage and sig-windows if this is the correct way to fix this issue. I'm not sure if the failing job is expected or not. If it's expected, kubeadm has to change the job manifest to make it work on only Linux nodes. Why I ask this question is that I saw , which is used to set the ownership of the volume, if the os is Windows, it will become a no-op.

/cc
/cc
/triage accepted
slack discussion is happening here: but we have agreed that this is a kubelet problem. on the kubeadm side you can use the --ignore-preflight-errors=CreateJob option to skip the check. unclear when or if a kubelet fix will be and to what versions it will be available.
To fix this issue, the AtomicWriter can check the current OS of the node and skip the os.Chown if the OS is Windows. I want to confirm with sig-storage and sig-windows if this is the correct way to fix this issue. please see: do you want to upstream your carry?
thank you. I raise a PR and add as a co-author. I add an e2e test. do you know which job can test windows?
is the e2e test job for Windows


Question: var ( // capabilities defaultLocalVolumeCapabilities = map[testsuites.Capability]bool{ <del> testsuites.CapPersistence: true, testsuites.CapFsGroup:     true, testsuites.CapBlock:       false, testsuites.CapExec:        true, testsuites.CapMultiPODs:   true, </del> <ins> testsuites.CapPersistence:      true, testsuites.CapFsGroup:          true, testsuites.CapBlock:            false, testsuites.CapExec:             true, testsuites.CapMultiPODs:        true, testsuites.CapSingleNodeVolume: true, </ins> } localVolumeCapabitilies = map[utils.LocalVolumeType]map[testsuites.Capability]bool{ utils.LocalVolumeBlock: { <del> testsuites.CapPersistence: true, testsuites.CapFsGroup:     true, testsuites.CapBlock:       true, testsuites.CapExec:        true, testsuites.CapMultiPODs:   true, </del> <ins> testsuites.CapPersistence:      true, testsuites.CapFsGroup:          true, testsuites.CapBlock:            true, testsuites.CapExec:             true, testsuites.CapMultiPODs:        true, testsuites.CapSingleNodeVolume: true, </ins> }, } // fstype

What happened: When we run csi hostpath driver e2e tests on a multi-node cluster some tests failed because they assume that volumes are node independent but this is not the case for some drivers like hostpath. Related PR; What you expected to happen: Skip the tests that assume volumes are node independent for the drivers that don't support node independent volumes. /sig storage /assign


Question: cmd.Execute() }() <del> if test.expectFatal { </del> <ins> switch { case recovered != nil && !sawFatal: t.Fatalf("got panic: %v", recovered) case test.expectFatal: </ins> if !sawFatal { t.Fatalf("%s: unexpected non-error", test.description) } if updated { t.Fatalf("%s: unexpected update", test.description) } <del> } if !test.expectFatal && sawFatal { </del> <ins> case !test.expectFatal && sawFatal: </ins> t.Fatalf("%s: unexpected error", test.description) <del> } if !reflect.DeepEqual(test.expected.Spec, test.node.Spec) && !updated { </del> <ins> case !reflect.DeepEqual(test.expected.Spec, test.node.Spec) && !updated: </ins> t.Fatalf("%s: node never updated", test.description) } })

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: kubectl drain crashed with panic when apiserver connection disappeared while there were pods that couldn't be evicted/deleted (sorry, the problem is a bit hard to reproduce and the log was lost) What you expected to happen: kubectl drain should exit with a proper error message How to reproduce it (as minimally and precisely as possible): Try kubectl drain multiple times, interrupting the connection during eviction/deletion Anything else we need to know?: The here can be in case if returns an error, and will panic upon accessing : This causes the c Environment: Kubernetes version (use ): The problem is present as of but is also present in earlier versions (I have a fix for this bug)
/priority important-longterm /sig cli


Question: assert.Equal(t, expected, status) // Advance the clock and stop the container. <del> fClock.SetTime(time.Now()) </del> <ins> fClock.SetTime(time.Now().Add(1 * time.Hour)) </ins> *expected.FinishedAt = fClock.Now().Unix() *expected.State = runtimeApi.ContainerState_EXITED *expected.Reason = "Completed"

Another flake i found earlier yesterday, just linking to it for posterity, not blocking me right now. TestContainerStatus
Could you post some error log here? I can't reproduce it with stress test
Hm, got one reproduce:


Question: return newReq, cancelFn } <del> // maybeWrapForConnectionUpgrades wraps the roundtripper for upgrades.  The bool indicates if it was wrapped func maybeWrapForConnectionUpgrades(restConfig *restclient.Config, rt http.RoundTripper, req *http.Request) (http.RoundTripper, bool, error) { if !httpstream.IsUpgradeRequest(req) { return rt, false, nil } tlsConfig, err := restclient.TLSConfigFor(restConfig) if err != nil { return nil, true, err } followRedirects := utilfeature.DefaultFeatureGate.Enabled(genericfeatures.StreamingProxyRedirects) requireSameHostRedirects := utilfeature.DefaultFeatureGate.Enabled(genericfeatures.ValidateProxyRedirects) upgradeRoundTripper := spdy.NewRoundTripper(tlsConfig, followRedirects, requireSameHostRedirects) wrappedRT, err := restclient.HTTPWrappersForConfig(restConfig, upgradeRoundTripper) if err != nil { return nil, true, err } return wrappedRT, true, nil } </del> // responder implements rest.Responder for assisting a connector in writing objects or errors. type responder struct { w http.ResponseWriter

What happened: upgrade requests to aggregated APIs (like websocket-based watch requests) on a kube-apiserver using --ssh-user hang What you expected to happen: Websocket-based watch requests work properly How to reproduce it (as minimally and precisely as possible): a kube-apiserver configured to use a node dialer to reach services/pods (e.g. with ) an aggregated API server (like the sample-apiserver) on a node other than the kube-apiserver a websocket-based watch against the aggregated API Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
/sig api-machinery /cc lavalamp
The code in question is here: handlingInfo.proxyRoundTripper is built here: and inherits the proxyDialerFn built in from the following construction chain: However, drops the custom dialer contained inside and from the returned round tripper. That means that upgrade requests fall back to use the default dialer, which cannot reach the aggregated API server
cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/assign would be good to resolve with the apiserver proxy work


Question: Context("when scheduling a busybox Pod with hostAliases", func() { podName := "busybox-host-aliases" + string(uuid.NewUUID()) <del> It("should write entries to /etc/hosts [NodeConformance]", func() { </del> <ins> /* Release : v1.13 Testname: Kubelet, hostAliases Description: Create a Pod with hostAliases and a container with command to output /etc/hosts entries. Pod's logs MUST have matching entries of specified hostAliases to the output of /etc/hosts entries. */ framework.ConformanceIt("should write entries to /etc/hosts [NodeConformance]", func() { </ins> podClient.CreateSync(&v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name: podName,

What happened: These tests are errantly listed as (ref: ) but aren't actually run as part of Conformance. They need to be moved to test/e2e/common to be included. To my eyes they may meet the criteria for conformance. There's definitely nothing about them that is node specific. They should be running as part of our regular CI, and they should probably be running as presubmits. Porting them to test/e2e/common would cause this to happen. /kind cleanup /sig node /area test /assign
GitHub didn't allow me to assign the following users: mkurylec. Note that only and repo collaborators can be assigned. For more information please see <detailsIn response to : What happened: These tests are errantly listed as (ref: ) but aren't actually run as part of Conformance. They need to be moved to test/e2e/common to be included. /sig node /area test /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: // // This is a separate function and not a method because not all types supported // by Generic are ordered and only those can be sorted. <del> func List[T ordered](s Set[T]) []T { </del> <ins> func List[T cmp.Ordered](s Set[T]) []T { </ins> res := make(sortableSliceOfGeneric[T], 0, len(s)) for key := range s { res = append(res, key)

Package use a custom-defined type . Go 1.21 support an official package cmp which has .
/sig api-machinery
cc
/assign xgp01
/triage accepted
/kind cleanup


Question: func (s *Scheduler) scheduleOne() { pod := s.config.NextPod() <del> glog.V(3).Infof("Attempting to schedule: %+v", pod) </del> <ins> glog.V(3).Infof("Attempting to schedule pod: %v/%v", pod.Namespace, pod.Name) </ins> start := time.Now() dest, err := s.config.Algorithm.Schedule(pod, s.config.NodeLister) if err != nil { <del> glog.V(1).Infof("Failed to schedule: %+v", pod) </del> <ins> glog.V(1).Infof("Failed to schedule pod: %v/%v", pod.Namespace, pod.Name) </ins> s.config.Error(pod, err) s.config.Recorder.Eventf(pod, api.EventTypeWarning, "FailedScheduling", "%v", err) s.config.PodConditionUpdater.Update(pod, &api.PodCondition{

In , there are two places where uses to output the content of pod. One place is , And the other one is , They are different from other places such as , which use to output the type . Below are my expirements. Output1 is: Output2 is: Output3 is: Output4 is: The result of my expirments shows that both and can't output useful information of a pod, it can only . can output the whole content of pod, but it seems too much! So, my opinion is output pod info use the format of to print instead of pod in both and . Like - Although this issue is tiny, I would like to ask you for comment. Thanks!
Fixed. I am going to close this issue. Thanks all!


Question: "//pkg/credentialprovider/aws:go_default_library", "//pkg/types:go_default_library", "//pkg/util/sets:go_default_library", <ins> "//pkg/util/wait:go_default_library", </ins> "//pkg/volume:go_default_library", "//vendor:github.com/aws/aws-sdk-go/aws", "//vendor:github.com/aws/aws-sdk-go/aws/awserr",

In a cloud provider environment it's possible to use up the API quota grinding the cluster to a halt: In a multi tenant environment, one user could consume the quota for an entire cluster. Some cloud providers allow throttling of API calls by type but Kube does not differentiate on the call types, so DOS still possible.
Adding some colors. Cloud providers like AWS/GCE/Azure have rate limits on API calls. GCE and Azure have fixed API calls per second, while AWS doesn’t offer fixed quota. Once API call quota are exhausted, no more API calls are allowed. Consequences include: If the error code path are not well tested, unexpected behavior will happen. A malicious user may launch DoS by triggering frequent API calls. The entire cluster would be impacted AWS cloudprovider has implemented API throttling at global level as well as on per call type level most recently. However, at least on AWS, quota for call types are different: query type API calls have higher quota while modification type API calls have lower quota. There should be strategies to differentiate these calls. Quota for Cloudprovider API calls should be supported on per account/project basis, so DoS can be averted. (Per This needs significant enhancements in Attach/Detach logic, volume plugin gets only pv.Spec, which does not include information about namespace of the PVC bound to it nor the user that initiated the operation. And only volume plugin (on top of cloud provider) can throttle API calls.) Throttling at per call type level should be supported.
I understand there is a problem, but I am not convinced on any particular solution. Can we get a more detailed enumeration on the code paths that invoke cloud provider backed interfaces, and where we would want some bucket to control rate for those calls? I don't think the existing object should bleed concepts from the underlying cloud provider, and I don't think its setup to support rate based quota models at this time.
/cc
I went through storage parts of the cloud provider and on AWS it looks like this: Deleter (in persistent volume controller) calls ec2.DeleteVolume() once per delete attempt Provisioner (in persistent volume controller) calls ec2.DescribeInstances() with filter "instance-state-name" == "running" to get all zones that contain a node. Once per provisioning attempt.

calls ec2.CreateVolume() once per provisioning attempt calls ec2.CreateTags() - up to 60 times per provisioning call (until it succeeds). Typically one attempt needed. calls ec2.DescribeVolumes() once per provisioning attempt When provisioning or deletion of a volume fails, the controller re-tries every 15 seconds until it succeeds. Typically it succeeds on the first attempt. PersistentVolume admission plugin: ec2.DescribeVolumes() once per new PV instance created (to fill its labels) Attach (in attach/detach controller) ec2.DescribeInstances() with filterprivate-dns-name" = nodeName once per attach attempt (to get list of already attached volumes) ec2.AttachVolume() once per attempt ec2.DescribeVolumes() with filter volumeId = volume every 10 seconds until it's attached Detach (in attach/detach controller) ec2.DescribeInstances() with filterprivate-dns-name" = nodeName once per attempt ec2.DetachVolume once per attempt ec2.DescribeVolumes() with filter volumeId = volume every 10 seconds until it's detached please confirm how often the controller re-tries on failure. Is it ? With many volumes being attached/detached at the same time, each volume polls its own DescribeVolumes() every 10 seconds while waiting for the operation to finish. We could save a lot of calls if we had one DescribeVolumes() call for all "interesting" volumes and distribute the results to proper places. In addition, some exponential backoff would be helpful, AWS is pretty slow when attaching/detaching volumes. Only storage was investigated!!!
The same issue also applies to ELB on AWS. Service controller invokes which calls:
It should be noted exponential backoff is disabled in - , I am not sure if it was intentional. Also, we should consider case where k8s cluster is in, inconsistent state. For example - if volume being deleted is still mounted to a pod, there would be a flood of API requests for deleting volume in pvcontroller (without any exponential backoff)
You're right, PV controller does not use retry logic of GoRoutineMap and retries provisioning and deletion on its own and currently without exponential backoff. The reason is that PV controller can react to deletion of object, e.g. when user removes a PVC then the controller does not need to retry provisioning a volume for it.

PV controller retries failed operation every 15 seconds by default. Some exponential backoff could be if 15 seconds are too short.
If Attach or Detach operation fails (e.g. due to throttling), there already is exponential backoff, starting with 0.5 seconds and doubling with each failed attempt, capped at 2 minutes. Summary: there already is exponential backoff for Attach/DetachVolume with we will have the same exponential backoff for Create/DeleteVolume the only fixed is DescribeVolumes() when we wait for a volume to get really attached/detached. I am going to fix it.