Question: return nil, fmt.Errorf("CSIPersistentVolumeSource not defined in spec") } <ins> func getReadOnlyFromSpec(spec *volume.Spec) (bool, error) { if spec.PersistentVolume != nil && spec.PersistentVolume.Spec.CSI != nil { return spec.ReadOnly, nil } return false, fmt.Errorf("CSIPersistentVolumeSource not defined in spec") } </ins> // log prepends log string with `kubernetes.io/csi` func log(msg string, parts ...interface{}) string { return fmt.Sprintf(fmt.Sprintf("%s: %s", csiPluginName, msg), parts...)

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: The field passed to is never set. In where is a object that is initialized in the method in Notice the field is never set. What you expected to happen: The field should be set to How to reproduce it (as minimally and precisely as possible): Create a CSI volume and set to true. Check the value of in the CSI call. It should be true, but is false. CC Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
I think is only used for mounting pod volume path to container, here(for ), we can set to , just like other volume plugins. I can help to fix this.
What about the case when a user wants to mount a volume for one of containers in a POD as read only, but the actual volume has ReadOnly False? I would like to suggest an alternative solution which passes the value of ReadOnly attribute, taken from POD's spec.
Thank you and for jumping on this! I feel like the other volume plugins have the wrong behavior here. The purpose of the call is to mount the volume. Therefore, the should receive the pod volume mount readonly field. That said, as mentioned , I think the fact that " is used for mounting pod volume path to container" might be hiding a bug: that is never set correctly. I haven't had a chance to validate this but my reading of the code suggests that the is never set. But because the container mount uses , even though the plugin did not get the correct readonly value, the mount ends up getting mounted with the correctly readonly value. Thoughts?
[MILESTONENOTIFIER] Milestone Issue: Up-to-date for process Note: This issue is marked as , and must be updated every 1 day during code freeze. Example update: <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release.

</details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
We discussed the proposed fixes in the Kubernetes CSI implementation meeting today (see ). We decided it would be better for CSI to be consistent with the other plugins (and still broken), and have CSI fixed, along with all the other plugins, whenever we fix the issue (likely post-1.10). Therefore, I will accept fix and we will close Then we can remove the 1.10 tag from this issue and track it independent of the 1.10 release.
Spoke with offline. Details here: Based on that conversation, I don't think there is anything else we need to do here.


Question: Build Kubernetes-Mesos. ```bash <del> $ git clone https://github.com/mesosphere/kubernetes-mesos k8sm $ mkdir -p bin && sudo docker run --rm -v $(pwd)/bin:/target  -v $(pwd)/k8sm:/snapshot -e GIT_BRANCH=release-0.4  mesosphere/kubernetes-mesos:build </del> <ins> $ git clone https://github.com/GoogleCloudPlatform/kubernetes $ cd kubernetes $ export KUBERNETES_CONTRIB=mesos $ make </ins> ``` Set some environment variables. The internal IP address of the master may be obtained via `hostname -i`. ```bash <del> $ export servicehost=$(hostname -i) $ export mesos_master=${servicehost}:5050 $ export KUBERNETES_MASTER=http://${servicehost}:8888 </del> <ins> $ export KUBERNETES_MASTER_IP=$(hostname -i) $ export KUBERNETES_MASTER=http://${KUBERNETES_MASTER_IP}:8888 </ins> ``` ### Deploy etcd Start etcd and verify that it is running: ```bash <del> $ sudo docker run -d --hostname $(uname -n) --name etcd -p 4001:4001 -p 7001:7001 coreos/etcd </del> <ins> $ sudo docker run -d --hostname $(uname -n) --name etcd -p 4001:4001 -p 7001:7001 quay.io/coreos/etcd:v2.0.12 </ins> ``` ```bash $ sudo docker ps <del> CONTAINER ID   IMAGE                COMMAND   CREATED   STATUS   PORTS                NAMES fd7bac9e2301   coreos/etcd:latest   "/etcd"   5s ago    Up 3s    2379/tcp, 2380/...   etcd </del> <ins> CONTAINER ID   IMAGE                        COMMAND   CREATED   STATUS   PORTS                NAMES fd7bac9e2301   quay.io/coreos/etcd:v2.0.12  "/etcd"   5s ago    Up 3s    2379/tcp, 2380/...   etcd </ins> ``` It's also a good idea to ensure your etcd instance is reachable by testing it ```bash <del> curl -L http://$servicehost:4001/v2/keys/ </del> <ins> curl -L http://${KUBERNETES_MASTER_IP}:4001/v2/keys/ </ins> ``` If connectivity is OK, you will see an output of the available keys in etcd (if any). ### Start Kubernetes-Mesos Services <del> Start the kubernetes-mesos API server, controller manager, and scheduler on a Mesos master node: </del> <ins> Update your PATH to more easily run the Kubernetes-Mesos binaries: ```bash $ export PATH="$(pwd)/_output/local/go/bin:$PATH" ``` Identify your Mesos master: depending on your Mesos installation this is either a `host:port` like `mesos_master:5050` or a ZooKeeper URL like `zk://zookeeper:2181/mesos`. In order to let Kubernetes survive Mesos master changes, the ZooKeeper URL is recommended for production environments. ```bash $ export MESOS_MASTER=<host:port or zk:// url> ``` Create a cloud config file `mesos-cloud.conf` in the current directory with the following contents: ```bash $ cat <<EOF >mesos-cloud.conf [mesos-cloud] mesos-master        = ${MESOS_MASTER} EOF ``` Now start the kubernetes-mesos API server, controller manager, and scheduler on the master node: </ins> ```bash <del> $ ./bin/km apiserver  --address=${servicehost}  --mesos_master=${mesos_master}  --etcd_servers=http://${servicehost}:4001  </del> <ins> $ km apiserver  --address=${KUBERNETES_MASTER_IP}  --etcd-servers=http://${KUBERNETES_MASTER_IP}:4001  </ins> --service-cluster-ip-range=10.10.10.0/24  --port=8888  <del> --cloud_provider=mesos  </del> <ins> --cloud-provider=mesos  --cloud-config=mesos-cloud.conf  </ins> --v=1 >apiserver.log 2>&1 & <del> $ ./bin/km controller-manager  --master=$servicehost:8888  --mesos_master=${mesos_master}  </del> <ins> $ km controller-manager  --master=${KUBERNETES_MASTER_IP}:8888  --cloud-provider=mesos  --cloud-config=./mesos-cloud.conf   </ins> --v=1 >controller.log 2>&1 & <del> $ ./bin/km scheduler  --address=${servicehost}  --mesos_master=${mesos_master}  --etcd_servers=http://${servicehost}:4001  --mesos_user=root  --api_servers=$servicehost:8888  </del> <ins> $ km scheduler  --address=${KUBERNETES_MASTER_IP}  --mesos-master=${MESOS_MASTER}  --etcd-servers=http://${KUBERNETES_MASTER_IP}:4001  --mesos-user=root  --api-servers=${KUBERNETES_MASTER_IP}:8888  </ins> --v=2 >scheduler.log 2>&1 & ``` <del> Also on the master node, we'll start up a proxy instance to act as a public-facing service router, for testing the web interface a little later on. ```bash $ sudo ./bin/km proxy  --bind_address=${servicehost}  --etcd_servers=http://${servicehost}:4001  --logtostderr=true >proxy.log 2>&1 & ``` </del> Disown your background jobs so that they'll stay running if you log out. ```bash

Hi We are updating docs to v1beta3. v1beta1 and v1beta2 will be abandoned soon. For the examples in getting-started-guides, we tend to use hyperlinks to point to examples instead of letting them have their own individual examples (see ). However, in the mesos guide, the examples contain mesos specific setup and knowledge, so I can't simply remove the texts and point to kubernetes/examples/guestbook, which is updated to v1beta3. My suggestion is that we update the guestbook example in kubernetes-mesos, and use a hyperlink in mesos getting-started-guide to replace the example texts. Could you help to make the changes? Thanks.
cc
We're going to delete v1beta1 and v1beta2 in O(days). See PR .
cc
we've initiated an effort to upstream some components into k8s. as part of that effort, we're upgrading the guestbook examples for k8s-mesos: the getting started guide for mesos will likely have to change in order to align with the move upstream. tracking this issue here: , Brian Grant wrote:
xref , James DeFelice wrote:
Ref
cc
xref
This needs to be moved to the v1 API.
Agreed. We have changes in review right now, should be merged soon. , Brian Grant wrote:


Question: sampleRegistry        = registry.SampleRegistry AdmissionWebhook         = ImageConfig{e2eRegistry, "webhook", "1.13v1"} <del> APIServer                = ImageConfig{e2eRegistry, "sample-apiserver", "1.0"} </del> <ins> APIServer                = ImageConfig{e2eRegistry, "sample-apiserver", "1.10"} </ins> AppArmorLoader           = ImageConfig{e2eRegistry, "apparmor-loader", "1.0"} BusyBox                  = ImageConfig{dockerLibraryRegistry, "busybox", "1.29"} CheckMetadataConcealment = ImageConfig{e2eRegistry, "metadata-concealment", "1.0"}

[sig-api-machinery] Aggregator Should be able to support the 1.7 Sample API Server using the current Aggregator More nuanced logs and details can be found at gave up waiting for apiservice wardle to come up successfully Expected error: <errors.errorString | 0xc420131150: { s: "timed out waiting for the condition", } timed out waiting for the condition not to have occurred /priority failing-test /priority important-soon /kind flake /sig-api-machinery /assign can you please help triage this issue?
has made changes that could have affected this recently. made one that'd be even more suspicious but I don't think it has merged yet :) :)
flakes show steady 2-3/day over the past week: which changes are you referring to? the line that's failing isn't doing any discovery, restmapping, etc, at all... just a straight get to the API:
audit log from the test run shows the requests are returning 404s, not 503s
ah, sorry, I misread the log and thought it was a discovery error. Hm.
hmm, there's a second smaller set of failures that are discovery related: looks like those started flaking after which went in on 5/4 looks like some discovery paths had internal retries that could have masked those 404 errors before. since the error that test is encountering is for an unrelated CRD API that is being created/deleted as part of another test, it makes more sense to switch that check to just ensure the extension API group under test is discovered, rather than assert there are no errors looking for other API groups that we know are being dynamically /removed opened to deflake the aggregator e2e check
/assign resolved the discovery issue. now the failure is the one noted in where calls to the extension API are responding with 404 errors
Kubelet log is reporting that the sample-apiserver exited with a return code of 2 and it also shows with a high restart count for the sample-apiserver.

When I extract the sample-apiserver log I'm seeing multiple "panic: Get https://10.0.0.1:443/api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentication: dial tcp 10.0.0.1:443: getsockopt: connection refused" Both failures today are in ipvs clusters which seems to have extensive networking problems. /assign
We are looking into finding the root cause into the problems with the ipvs clusters. I am pretty confident they were triggered by a bad PR, just need to identify which one.
Initial hypothesis is that PR caused it.
We think should fix the IPVS issues.
I see merged and this test is now passing in both pull-kubernetes-e2e-gce and ci-kubernetes-e2e-gci-gce. Hence closing the issue. Thanks all
/reopen The test is failing for the same error again in the following jobs https://k8s- It is one of the top flake from last week failing 47 jobs.
can you please take a look and triage? Thanks
this flake is now blocking this test from being promoted to Conformance suite in 1.11. Is it something we can investigate and resolve soon (today or tomorrow) to increase changes of getting this into 1.11? If not this test has to wait until 1.12 to be promoted.
all I see in the test run logs is 404 errors coming back from that check. can you see anything useful in the container logs for the aggregated server?
/cc
Update: I have been trying to reproduce this failure on my own test cluster (go run ...) and it has passed 128 times in a row so far. I think this means that the issue must be with either an interaction between e2e tests or an issue with the job running the tests.
Looking through recent results this crops up as a flake in presubmit and CI across many configurations... I don't think how we run the test jobs should be affecting this in the slightest. Not sure what's wrong here, but it looks like a pretty typical flakey test (some race / buggy setup/teardown?) nb that may not trivially map to the same way the tests are run though, and itself is a wrapper over
does the API server logs throw any light on whats happening on CI test cluster.

The teardown for the aggregator test is wrong, it tries to delete the deployment "sample-apiserver" when the deployment is actually named "sample-apiserver-deployment", but I don't think that's the issue. It's been like that since the test was . I'll look for conflicts with other tests.
We should probably go ahead and fix that anyway, it might be stomping on another test... Jenny Buckley <:
The namespace controller polls each of the api groups periodically, and it gets this internal error back from the endpoint a couple of times. Maybe related?
unclear... I opened to capture the state of the APIService, extension server pods, and those pod logs in failures cases
It flaked again since your PR merged https://k8s-
All I'm seeing is
I don't think that's related... I see 10-30 of those same rejections in apiserver logs of successful test runs (for example, search for ). I don't see anything in the test setup that grants the sample apiserver permission to run those checks, so it doesn't seem like that would be intermittent or flaky.
I am re-opening this issue since I still see intermittent failure of this test are your pending PRs ( and ) targeted towards these remaining issues? /reopen
this looks like 1.7 API Server image is causing more flakes other than just this test - The test has become lot less flaky after increasing the timeout. Since the official policy for GKE is 2 previous versions, and 1.11 is almost baked, this is the plan discussed with offline: leave the 1.7 test as it is right now create a similar test to the 1.7, but with 1.9 (implies creating a new sample-sever, uploading image, and creating the test). get that new test PR merged and running let it soak for several days, see if in 1.7 - 1.9 the flakiness (latency or whatever is causing it) went away.

if it did, evaluate removing the 1.7 and using the 1.9 let us know what you think of this approach
I think you misunderstood my comment... that only related to kubectl being used against a 1.7-level kube-apiserver serving discovery docs for the v1 API, not a generic 1.7-level extension API server
unrelatedly, moving the test to 1.9-level code once 1.11 is released seems reasonable to me
Thank you Jordan, that's the current plan! We will actually create a parallel 1.9-level test, to see if between 1.7 and 1.9 the internals of API Server fixed what's generating the flakiness and see next steps from there.
Sorry I didn't read the previous discussion close enough. The idea of creating a 1.9 test makes sense.
Pinging to see what are the next steps. Who is taking point to create a parallel test?
last I chated with someone on his team will build 1.9 API server image and push it to GCR. Once that image is available then can help writing the parallel test. I am not sure if the image step is done.
It's on the works. Buiding the sample-image showed to be more complicated than what it seemed to be. If it becomes a blocker let me know. Aish Sundar <:
I'd like to know where we are on this, because the 1.7 thing has come up in the context of multi-arch image builds
and
Hi Aaron and Aish, sorry for the delay. The process to compile, create and upload the sample API Server was not properly documented / maintained, and that plus more burning things is what made it take so long. We are trying to finally have it back (and documented by end of this week). Will keep you posted.
Image created with all the refactor happened on that area, trying upgrading the test at present from 1.7 to 1.9
Update: Upgraded the test, and now the test is not passing... working on this.

• Failure [38.067 seconds] [sig-api-machinery] Aggregator Should be able to support the 1.9 Sample API Server using the current Aggregator [It] creating a new flunders resource Expected error: <errors.StatusError | 0xc420c90cf0: { ErrStatus: { TypeMeta: {Kind: "", APIVersion: ""}, ListMeta: {SelfLink: "", ResourceVersion: "", Continue: ""}, Status: "Failure", Message: " "rest-flunder-" is forbidden: not yet ready to handle request", Reason: "Forbidden", Details: { Name: "rest-flunder-", Group: "", Kind: "flunders", UID: "", Causes: nil, RetryAfterSeconds: 0, }, Code: 403, }, } "rest-flunder-" is forbidden: not yet ready to handle request not to have occurred
not sure if it's related, but I don't think the test currently gives sufficient permissions to the aggregated server to perform delegated authn/authz checks. we might want to look at merging
tried your fix, same result. The image of the sample-apiserver 1.9 based that I've created is uploaded to If someone want's to give it a try, simply by upgrading 1.0 to 1.1 here:
/assign Can you try to figure out why the test is failing w/ the new image?
/cc
Now waiting on
Is there an ETA on ? This test is flaking quite a bit offlate - https://k8s-
upgrades to 1.10. however it requires more rbac permissions than before which is not fully understood. We will either resolve the permission issue or leave it for later investigation. The upgrade to 1.10 should be merged in a couple of days, if not today.
what issue are you seeing? is relevant?
Yes, it is relevant. It does not fix the issue by itself though.
As for should be ready to be merged.
/reopen I would like some manual confirmation this has been addressed
Reopening this issue. <detailsIn response to : /reopen I would like some manual confirmation this has been addressed Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.

</details
after some in-depth investigations, it turns out the 1.10 sample-apiserver would need the following rbac permissions (in addition to the system:auth-delegator role). I'm updating the PR to add these. apiVersion: kind: ClusterRole metadata: name: sample-apiserver rules: apiGroups: resources: '*' verbs: list apiGroups: '' resources: namespaces verbs: list
This test is failing again, FYI: https://k8s-
Hi Josh, TL;DR: we are upgrading to 1.10 and that's what people is talking about in the previous comments.
/reopen I would like to see the test pass with the fix before closing this issue out. thanks
Reopening this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hi please notice the test was renamed to: [sig-api-machinery] Aggregator Should be able to support the 1.10 Sample API Server using the current Aggregator, and has been passing since merged? https://k8s-
Ah cool thanks that explains it :) I was looking at in release-master-blocking and looks like it hasn't picked by the latest test since its 10/13 run. Closing this issue now based on the api-machinery dashboard now. Will reopen if the release job doesnt turn green the next run. Thanks much.
Is it worth removing the copy of this test that lives in ? Or is it still a release-blocking test?
I think the test will automatically pick the new version of the test in its next run (which should be sometime today, 10/20?). As to whether the test should be in the scale-correctness job , it is a ques for and


Question: expectErr: "--shutdown-delay-duration can not be negative value", }, { <ins> name: "Test when HSTSHeaders is valid", testOptions: &ServerRunOptions{ AdvertiseAddress:            net.ParseIP("192.168.10.10"), CorsAllowedOriginList:       []string{"10.10.10.100", "10.10.10.200"}, HSTSDirectives:              []string{"fakevalue", "includeSubDomains", "preload"}, MaxRequestsInFlight:         400, MaxMutatingRequestsInFlight: 200, RequestTimeout:              time.Duration(2) * time.Minute, MinRequestTimeout:           1800, JSONPatchMaxCopyBytes:       10 * 1024 * 1024, MaxRequestBodyBytes:         10 * 1024 * 1024, }, expectErr: "--strict-transport-security-directives invalid, allowed values: max-age=expireTime, includeSubDomains, preload. see https://tools.ietf.org/html/rfc6797#section-6.1 for more information", }, { </ins> name: "Test when ServerRunOptions is valid", testOptions: &ServerRunOptions{ AdvertiseAddress:            net.ParseIP("192.168.10.10"), CorsAllowedOriginList:       []string{"10.10.10.100", "10.10.10.200"}, <ins> HSTSDirectives:              []string{"max-age=31536000", "includeSubDomains", "preload"}, </ins> MaxRequestsInFlight:         400, MaxMutatingRequestsInFlight: 200, RequestTimeout:              time.Duration(2) * time.Minute,

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Add the "Strict-Transport-Security" header to the API server responses. Why is this needed: Nessus scanner treats this as a Medium Risk in kubernetes 1.18.10.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig api-machinery
/remove-sig api-machinery /sig auth


Question: FailedPlacementReason = "FailedPlacement" // FailedDaemonPodReason is added to an event when the status of a Pod of a DaemonSet is 'Failed'. FailedDaemonPodReason = "FailedDaemonPod" <ins> // SucceededDaemonPodReason is added to an event when the status of a Pod of a DaemonSet is 'Succeeded'. SucceededDaemonPodReason = "SucceededDaemonPod" </ins> ) // controllerKind contains the schema.GroupVersionKind for this controller type.

I can reproduce this reliably with Calico v3.25.0 CNI, installed straight from the upstream . It seems to be a regression in Kubernetes 1.27.0-rc.0 compared to 1.27.0-beta.0, at least I can't reproduce it with beta.0. Cluster is deployed, and everything is ready & healthy. After a series of node reboots some pods are stuck in state and never recover: This of course disrupts the CNI and affects the scheduled workloads. The daemonset is "happy" with 2 pods being out: If I delete the pod manually, the pod recovers without any issues. The pods are restarted. Bring up a cluster with graceful node shutdown, reboot nodes a few times (including multiple node at anoce). Additional detailed output: <details</details<details</details<detailsnone </details<detailsTalos Linux 1.4.0-alpha.3 </details<details</details<details</details<detailsCalico v3.25.0, also reproducible with 3.24.1 </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
I wonder if that is related to the change
cc this might a regression in 1.27.0-rc.0 with this PR
Even if it is, it sounds like a bug in daemonset instead. Why wouldn't it replace pods that are terminated? It can probably be reproduced in integration.
I think there might be something with the fact that Calico terminates with exit code '0', so it feels like a "successful" run. Still such pod should be restarted, so I'm not sure why it doesn't get restarted after a reboot.
let me ask to get more details about reproducing this locally. Are you running this on Kind? How do you trigger the the restarts? Can you share mode details on the node configuration, such as the graceful shutdown period?

I initially hit this problem under AWS, but then I narrowed it down to Calico CNI. I'm using Talos Linux, this is the full kubelet config, using 30s/10s graceful shutdown period: <details</detailsLet me try to build a reproduce instruction in the next comment.
I also have complete output of under the details in the issue itself.
The daemonset controller should still recreate the pod in that scenario. /sig apps
The reproducer is to follow the , but it should be same under any Talos platforms except Docker (which doesn't have a graceful node shutdown, and any kind of shutdown). I verified it with the assets from the latest . The smallest I can reproduce it with is 1+1 cluster, but it looks same way e.g. in 3+2 cluster: Once the cluster is up, all the pods are running & healthy (). If I issue a reboot of the worker node with , with shows that pod on the worker node quickly enters state, and never recovers.
At the same time I don't quite understand what is exactly wrong, as if I just do a daemonset with command, the pod also enters state, but it gets quickly restarted by the via transition.
I can also send kubelet logs, and anything else which might be helpful to debug the issue, I'm also open to do a debug session with screen sharing.
This is because of . The pod doesn't reach the phase in this case. The kubelet just recreates the containers. However, when a pod is actually marked as "Succeeded" then kubelet would stop looking at it. However, it should be the responsibility of the daemonset controller to recreate a pod.
Following the sig-apps advise opened the PR () with an integration test for DaemonSet. It passes for Failed pods, but fails for Succeeded. In other words, the succeeded pods are not restarted. I'm yet to investigate (probably tomorrow) why this is so, but it seems like a bug. In the meanwhile some expertise about DaemonSets would be helpful.
Thanks for reporting this. I suppose this might be related to the fact that pods (created by daemonset) can now be marked as in v1.27.

When graceful node shutdown happens, we terminate the pod with : which will trigger kubelet to terminate the pod in pod worker () and eventually I assume set a terminal phase on the pod. We should verify if that is what is happening here. In either case, +1 on comment that I would expect that daemonset controller should recreate a pod if it is marked completed.
I will note that the intent for graceful shutdown was always that pods could be marked Succeeded (vs just Failed), since the pod is indeed terminal. It's likely that this was previously racy (no guarantee succeeded made it to the apiserver) and fixed that. However, outside of graceful shutdown this should never occur - do we have any evidence a RestartAlways pod went succeeded in that case (wasn't clear from the thread)? That would be a serious bug.
I think Succeeded would also be possible in other scenarios, like deletion (deletionTimestamp!=nil). This I've tested for Job pods, but I think DS pods would behave the same. Except for deletion I think the behavior is limited to graceful node shutdown, which invokes the kill function, but respects the Succeeded phase: Other uses of the kill function unconditionally set Failed, such as the eviction manager: EDIT: oh, actually for Jobs we don't use restartPolicy=Always, but still deleted pods with restartPolicy=Always may still terminate with Succeeded phase as shown in this test:
Deletion with RestartAlways is ok, because the pod goes away reasonably fast (but we are adding latency in 1.27 to that path) as you note.


Question: "//staging/src/k8s.io/api/core/v1:go_default_library", "//staging/src/k8s.io/api/rbac/v1beta1:go_default_library", "//staging/src/k8s.io/api/storage/v1:go_default_library", <ins> "//staging/src/k8s.io/apimachinery/pkg/api/errors:go_default_library", </ins> "//staging/src/k8s.io/apimachinery/pkg/apis/meta/v1:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/runtime/schema:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/util/sets:go_default_library",

Failing on gke only: https://k8s- https://k8s-
/assign
Unfortunately the exact error message that its failing with is lost due to a typo: I have fixed that typo locally and am running the tests repeatedly to try to reproduce the flake. have been unable to reproduce the flake on HEAD of Kubernetes in GKE in 15 runs so far. Running 150 tests overnight.
Unable to repro on my own GKE cluster. Merged the error message fix, will monitor for the next flake to debug more


Question: } func (az *Cloud) isCurrentInstance(name types.NodeName, metadataVMName string) (bool, error) { <del> var err error </del> nodeName := mapNodeNameToVMName(name) if az.VMType == vmTypeVMSS { <del> // VMSS vmName is not same with hostname, use hostname instead. metadataVMName, err = os.Hostname() if err != nil { return false, err </del> <ins> // VMSS vmName is not same with hostname, construct the node name "{computer-name-prefix}{base-36-instance-id}". // Refer https://docs.microsoft.com/en-us/azure/virtual-machine-scale-sets/virtual-machine-scale-sets-instance-ids#scale-set-vm-computer-name. if ssName, instanceID, err := extractVmssVMName(metadataVMName); err == nil { instance, err := strconv.ParseInt(instanceID, 10, 64) if err != nil { return false, fmt.Errorf("failed to parse VMSS instanceID %q: %v", instanceID, err) } metadataVMName = fmt.Sprintf("%s%06s", ssName, strconv.FormatInt(instance, 36)) </ins> } } metadataVMName = strings.ToLower(metadataVMName) <del> return (metadataVMName == nodeName), err </del> <ins> return (metadataVMName == nodeName), nil </ins> } // InstanceID returns the cloud provider ID of the specified instance.

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Azure cloud provider Instances interfaces don't work expected on Windows containers. This is because Windows containers don't support HostNetwork mode, while Azure cloud provider depends on "os.Hostname()" to check whether the VMName from IMDS is same with NodeName: To fix it, the dependency on Hostname should be removed. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /sig cloud-provider /priority critical-urgent /area provider/azure /milestone v1.18


Question: } } <del> if diff := cmp.Diff(tt.want, got); diff != "" { </del> <ins> if got.LastTransitionTime.IsZero() && got.Type != "" { t.Errorf("unexpected empty LastTransitionTime in condition") } if diff := cmp.Diff(tt.want, got, cmpopts.IgnoreFields(api.PodCondition{}, "LastTransitionTime")); diff != "" { </ins> t.Errorf("unexpected field errors (-want, +got):n%s", diff) } })

When a scheduling gated pod is ungated, the condition's is consistently null until the pod is scheduled: Pods without a scheduling gate have the same condition but is populated: Submit a pod with a scheduling gate. Observe that after removal of the gate, the pod condition. I uncovered this while using Kueue and the ProvisioningRequest feature. Since this is reproducible with just the scheduling gate, I think k/k is the right project for the issue. <details</details<detailsGCP </details<details</details<details</details<details</details<details</details
/cc
/sig scheduling
When the pod is gated, what's the condition of the Pod? Is it empty?
The gated pod has just this condition. The transition time is also null, but is that expected since it is the first condition?
/triage accepted
The bug is here: It looks like at this point, the ObjectMeta.CreationTimestamp is already populated. We should copy that field.
/assign
Are you still on this, we meet the same problem.
It's been over a month, I assume you might no longer work on this? We encounter the same bug and hope this bug can be fixed as soon as possible(hopefully next release). Since there is no further update on this issue, I am proposing a protential fix on now, can you assign this issue to me and review the code once you are available? Thanks.
/unassign ahaysx /assign royliang
GitHub didn't allow me to assign the following users: royliang. Note that only with read permissions, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /unassign ahaysx /assign royliang Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Sorry I have been on vacation until today. Thanks for the fix
are you still working on this?
I’ve submitted the PR, but it seems lack of reviewer moving this forward.


Question: }), preemptedPodIndexes: map[int]struct{}{0: {}}, }, <ins> // This is identical with previous subtest except for setting enablePreFilter to true. // With this fake plugin returning Unschedulable in PreFilter, it's able to exercise the path // that in-tree plugins return Skip in PreFilter and their AddPod/RemovePod functions are also // skipped properly upon preemption. { name:            "basic pod preemption with preFilter", initTokens:      1, enablePreFilter: true, existingPods: []*v1.Pod{ initPausePod(&testutils.PausePodConfig{ Name:      "victim-pod", Namespace: testCtx.NS.Name, Priority:  &lowPriority, Resources: &v1.ResourceRequirements{Requests: v1.ResourceList{ v1.ResourceCPU:    *resource.NewMilliQuantity(200, resource.DecimalSI), v1.ResourceMemory: *resource.NewQuantity(200, resource.DecimalSI)}, }, }), }, pod: initPausePod(&testutils.PausePodConfig{ Name:      "preemptor-pod", Namespace: testCtx.NS.Name, Priority:  &highPriority, Resources: &v1.ResourceRequirements{Requests: v1.ResourceList{ v1.ResourceCPU:    *resource.NewMilliQuantity(200, resource.DecimalSI), v1.ResourceMemory: *resource.NewQuantity(200, resource.DecimalSI)}, }, }), preemptedPodIndexes: map[int]struct{}{0: {}}, }, </ins> { // same as the previous test, but the filter is unresolvable. name:         "basic pod preemption with unresolvable filter",

In k8s 1.27, scheduler introduced a new "short circuit" semantics in PreFilter to bypass a Pod quickly. A change introduced here is it relies on that plugin to be stored in , which makes sense as what it was explained . However, the info, although is populated upon each plugin's PreFilter invocation, is not injected into when some PreFilter returns unschedulable: So think about this workflow: pluginA returns Skip in its PreFilter pluginB return Unschedulable in its PreFilter note: pluginA is not included in due to aforementioned bug a Pod is claimed as unschedulable (failed by pluginB) during PostFilter, it's supposed to bypass pluginA's PreEnqueueExtension's AddPod/RemovePod, but it doesn't calling a "skipped" Plugin's AddPod/RemovePod during preemption, sometimes will abort the whole preemption as the logic tries to read info from cycleState, and some plugin's "skip" version doesn't fill cycleState properly, e.g., InterPodAffinity (skip immediately w/o write preFilterStateKey). Short circuiting using Skip is fine, but should injected properly at all cases. Described in the workflow above. It's firstly captured in an out-of-tree PostFilter plugin: , but can also happen in in-tree plugins. No response v1.27 <details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/kind regression /sig scheduling
cc
Among in-tree plugins, 3 plugins employ non-nil and may return Skip in their PreFilter() implementation w/o writing to cycleState, while return error immediately if is missing in their Add/RemovePod() functions.

So technically, as long as a scheduling flow hit the following criteria, it would hit the bug (preemption doesn't function): plugin 1/2/3 return Skip in PreFilter arbitrary PreFilter plugin after the "skipped" plugin returns Unschedulable in its PreFilter
/cc
/cc
Cherry-pick PRs: 1.28: 1.27:


Question: target discoveryendpoint.ResourceManager, ) DiscoveryAggregationController { discoveryScheme := runtime.NewScheme() <del> // Register conversion for apidiscovery apidiscoveryv2.SchemeBuilder.Register(apidiscoveryv2conversion.RegisterConversions) </del> utilruntime.Must(apidiscoveryv2.AddToScheme(discoveryScheme)) utilruntime.Must(apidiscoveryv2beta1.AddToScheme(discoveryScheme)) <ins> // Register conversion for apidiscovery utilruntime.Must(apidiscoveryv2conversion.RegisterConversions(discoveryScheme)) </ins> codecs := serializer.NewCodecFactory(discoveryScheme) return &discoveryManager{

Seeing data race flakes in unit tests: Appears to have just been in /sig api-machinery /priority critical-urgent /assign /cc
This code is mutating the package-level schemebuilder: I think it should be adding the conversions to the local scheme instead:
Fixed in (found several instances)
/triage accepted


Question: ginkgo.It("should verify ResourceQuota's priority class scope (cpu, memory quota set) against a pod with same priority class.", func() { _, err := f.ClientSet.SchedulingV1().PriorityClasses().Create(context.TODO(), &schedulingv1.PriorityClass{ObjectMeta: metav1.ObjectMeta{Name: "pclass9"}, Value: int32(1000)}, metav1.CreateOptions{}) <del> framework.ExpectEqual(err == nil || apierrors.IsAlreadyExists(err), true) </del> <ins> if err != nil && !apierrors.IsAlreadyExists(err) { framework.Failf("unexpected error while creating priority class: %v", err) } </ins> hard := v1.ResourceList{} hard[v1.ResourcePods] = resource.MustParse("1")

The tests are full of assertions which compare a boolean against true or false, often without any additional explanation. When those assertions fail, the error message is useless for understanding what went wrong, often just saying: Including an explanation is better, but then we still have the distracting noise about the comparison: All of those assertions should be replace with . Such locations can be found by grepping: UPDATE (2024-08-26): in cases where such an if check is not possible (typically in combination with or ), should be used instead of . They emit only the explanation given to them, without including the noise above. The Ginkgo linter (enabled in golangci-lint) points that out. Some progress has been made in the meantime: v1.23.0: 250 v1.24.0-beta.0: 177 However, we were down to 175 at some point, so the number is going back up. Example: Some other places compare values. Those should be replaced with a comparison assertion. Example: It makes sense to split this work into individual PRs, one per OWNERS file under test/e2e. Here's a list of directories: [ ] test/e2e/apimachinery [x] test/e2e/apps: [x] test/e2e/auth [x] test/e2e/autoscaling: [x] test/e2e/cloud/gcp [x] test/e2e/cloud [x] test/e2e/common/network [x] test/e2e/common/node: [x] test/e2e/common/storage: [x] test/e2e/framework: [x] test/e2e/framework/providers/vsphere [x] test/e2e/framework/volume [x] test/e2e/instrumentation/logging [x] test/e2e/instrumentation: [x] test/e2e/kubectl [x] test/e2e/lifecycle [x] test/e2e/node: [x] test/e2e/scheduling: [x] test/e2e/storage: (prior PR: ) [x] test/e2e/testing-manifests/storage-csi [x] test/e2e/upgrades/apps: [x] test/e2e/windows: , When creating a PR, use as title. Use one commit per PR, squash when addressing feedback.

Unless noted otherwise, will work on PRs. Help may be welcome. /sig testing /help Better output for developers when tests fail.
This request has been marked as needing help from a contributor. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : ### What would you like to be ? Example: Some other places compare values. Those should be replaced with a comparison assertion. Example: /sig testing /help </details
/good-first-issue
This request has been marked as suitable for new contributors. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /good-first-issue </details
/triage accepted
I would like to take this up as my first issue
/assign
I would like to take up this issue as my first issue
Hello I am already working on this issue , can you please take some other issue ?
Okay sure ,how do I un assign myself from this issue?
I don't think you are assigned to this issue.
Oh okay
Hey as discussed in pr I am assuming that you are only working for tests under area.

also finished the work on , see
and only one PR is to be made for the correction under test/e2e/storage/, so how can we distribute the task and also make only single PR. Or can we make multiple PR's
I have a TODO list to the issue description. I'll try to keep that up-to-date with what has been done and who is working on it.
you can continue with in
what is to be changed in this case !
can you inform on which directory you are working.
Currently working on apimachinery
Nothing, just leave it as-is.
But that isn't under - before starting somewhere else, please coordinate with
test/e2e/instrumentation/logging don't have any cases like "Expect.Equal([^,], *(true|false)", so I think it should be removed from tasks to do.
I am thinking that the output should be if !ok { framework.Failf(uexec.ExitError) } in this case, am I thinking reasonable here.
isn't valid Go. The first parameter must be a format string. The code checks whether err is a : Therefore the change should be:
Why don't we just add extra explanation into calls like for easy investigation? I might miss some point.
You have to consider the complete output when that case fails. Gomega will write print the actual value and the expected value, then in addition you get the same information in the custom message. Printing just the custom failure message is shorter and more readable, in particular for a comparison of booleans. For a comparison of integers I see two solutions: ExpectEqual with a fixed message that just explains what is being compared. An if check with Failf where the message includes the actual value. The later seems more useful when the expected value is zero because that value doesn't need to be printed when its obvious.
I am getting difficulty in figuring out what should be output here, what should the appropriate output here. Also what steps you took in knowing so, it would help to improve my understanding.
The "ok" value is the result of checking for a specific key in a map. What a developer therefore wants to know when that check fails is: what was the key what's in the map Both can be included in:
can I work for the section :- test/e2e/scheduling ?

And there was only one changes needed for test/e2e/node which I did but my PR was closed as it only contains trivial edits. That was my first PR, so can you please guide me further?
/assign I'll try to coordinate the work, but won't create PRs myself. I ticked of line items which had no remaining work. feel free to start working on .
I replied there and also suggested an improvement. For those following this issue: try to include more information than before in the error message when the assertion fails. Then it is more obvious to the reviewer that the change is useful without having to read this issue. Examples:
Another tip: don't just reference this issue as explanation for the PR. Reviewers are busy, they won't necessarily invest the time and follow the link to understand what the PR is about. Instead, take the first sentence from this issue and adapt it as needed to the specific changes in the code.
can I start working on test/e2e/common/storage ?
Okay.
for test/e2e/network/netpol , there are already message associated with all of the cases, for example :- So should I just change all of them to ? And same for all the cases in test/e2e/network except these two cases : framework.ExpectEqual(netutils.I sIPv4String(internalIPs[0]) != netutils.IsIPv4String(internalIPs[1]), true) framework.ExpectEqual(netutils.IsIPv4String(p.Status.PodIPs[0].IP) != netutils.IsIPv4String(p.Status.PodIPs[1].IP), true )
Yes.
Okay then I will be working on and
It might make sense to combine them in a single PR, depending on how many changes there are - just use common sense.
IMO : this will never fail because whether err will be nil or it will exist. So, we don't need to change this code. What I had done in this case was :
Note that that is redundant (= not wrong, just not needed). I later clarified that in
/reopen
Reopened this issue. <detailsIn response to : /reopen </details
Hi! I've created a PR to address the tests in e2e/apps It is my first PR so hopefully, everything is ok. Thanks a lot!
Hi can I take test/e2e/common/node?
Yes, please do.

Have we considered / helpers that can skip the comparison output? - is noisy and makes it harder to skim tests, especially given that it's a pretty commonly required pattern.
No, we haven't. Do you find it noisy because it's two lines instead of one? I don't have a problem with that. It seems to be a common approach in unit tests and I find it easy to read, partly because it matches how production code would be written.
It becomes three, plus extra indention - but more importantly stands out in a series of expectations. This draws extra scrutiny to those lines. many unit tests using testify would use to express this because of that.
Isn't that a good thing? is bad without any or no useful extra explanation. The if check makes it obvious that a good failure message is needed. Does print less useless boilerplate code than Gomega? We could use that with Ginkgo (), but then we still need to ensure that the optional explanation is useful.
xref
Hi can I take test/e2e/auth?
for test/e2e/auth, for example : Should I just change it to ?
feel free to work on test/e2e/auth. This one particular example is a bit different: here we are comparing string values, and for those it makes sense to print the actual value because the failure message then adds information. I would leave that one as it is.
OK. test/e2e/auth doesn't need replacing. I : <img width="731" alt="Screen Shot 2022-05-30 at 9 12 59 PM" src="https://user-"There are a lot of positions that need to be modified in . I'll take test/e2e/apps.
That looks like a good candidate. Please go for it. When you need to make up new failure messages, try to think about what information may be useful to have and add that. Then the resulting PR is clearly better than the status quo.
Hi can you please assign me a directory. I want to give it a try. Thanks,
Hi I already posted a PR for e2e/apps some days ago in this issue: If it is fine I'd love to proceed with that. Just to avoid us repeating work.
sorry for not noticing the overlap sooner. Let's go with the older PR.

you can take over test/e2e/storage from There is an incomplete PR for it in You can check out that branch, fix it and submit a new PR, or start from scratch.
it looks like is not active anymore. I've remove him from the todo list and you instead for test/e2e/common/storage.
OK. Thank you. Is there an incomplete PR for .
I in and it returns 0. What should I do?
needs modifying. <img width="1440" alt="Screen Shot 2022-05-31 at 2 40 29 PM" src="https://user-"Can I work on it?
Indeed, did test/e2e/common/storage in You can work on test/e2e/windows.
OK. I'm working on it currently.
Sure, I will look into it. Thanks,
: do you want to continue with test/e2e/apimachinery?
Yes. There's an older PR:
Hi is there anything I could take over?
Hey I am not sure how updated the list is, but I am available to help. Is there any part I could start or continue?
sorry for the late reply. I had (and still have) some big changes for the E2E tests going and had put this on the backseat for a while. If you are interested, how about doing test/e2e/apimachinery and test/e2e/autoscaling? Some work was done on the latter, but there are still several calls (again?).
Sure, I can take that!
Hey I'd also like to contribute on this issue. Is there anything I could take over?
feel free to take my part:
Perfect, then I will take part
Hi :smile: , I would like to contribute to this issue, there is anything that I could do?
I was reluctant to start new work on this because I had one big PR pending that touched a lot of test code (PR ), but that has been merged. you are taking test/e2e/autoscaling, right? test/e2e/apimachinery? you can do test/e2e/apps. That leave test/e2e_node in case someone wants to continue.
yes e2e/autoscaling. I had some trouble to get the tests going initially, but seems fine now!
Hi, Can I do something to contribute to this issue?

you could work on test/e2e_node.
Ok, I will start work on test/e2e_node thanks!
Hi , I would like to contribute. Is there anything I could take over?
thanks for your interest, but right now everything seems to be covered.
Hi Thanks. I am searching for "good-first-issue", feel free to assign one to me. I am exploring api-machinery, but happy to contribute in other sig as well.
maybe you can take a look at if you have some time :)
Hi , There is anything that I could do to contribute with this issue?
Hi , I was looking and found the test/e2e/apimachinery without doing it and only then did I see that it was already assigned if this is a problem just refuse my pr and if you have something else to do let me know.
The remaining work will probably finished in , so there isn't anything else to do. If someone wants to investigate writing a linter for this, then this would be a cool science project - not easy, though. I would start with looking at for this. It has the advantage that it is supported indirectly in golangci-lint ().
Hello I would like to try to do a liter for it. It will be a cool project whit a lot of opportunities to learn. There will be a separate issue for the linter or we continue the discussion here?
Let's continue here.
I ended up creating a separate issue for linter ideas: let's continue there.
Ok!
New tests have been again with this anti-pattern since the last time that I checked. It's clear that without a linter, we'll not get rid of it. So let's focus on that first.
is this issue still open
Yes, this is still open. Current status as of v1.28.0-rc.0: Two calls are in , the rest are in . We have such a linter now, with ginkgolinter in golangci-lint and (soon) the ability to prevent usage of , so it makes sense to continue with removing those calls. do you want to prepare one PR for all remaining calls? I think one PR is enough, no need to split it up.
also has it, which was intentional. I'm undecided whether that should be replaced.

Several of those calls are being touched in , let's wait for that PR to be merged before doing any further work.
no longer finds anything. We might be done... :tada: But not quite. What we have left now is checking a boolean via or , which has the same problem. In some cases, there is an extra message (which is still noisy, see ), in other cases there isn't even that (which makes the failure message fairly useless). We can and should replace some of those calls with the pattern.
is almost resolved. My work-in-progress branch forbids the usage of in favor of the new or plain Go. In some cases, moving the Go code that returns true/false into a matcher might be the best solution because that allows dumping the full actual value that was being checked. Here are the remaining issues: <details</detailsDoes anybody want to work on those?
I would love to!
: please go ahead. Assign PRs to me when you create them. Let's start with a few of those calls initially, do a round of reviews, then continue. /assign
You will have to start with my branch, current master doesn't have gomega with thew new API yet.
I'm starting new to kubernetes. Is there anything easy that I could do ?
Hey Is there anything left I can help you with in this issue ?
Thanks for your interest. There's not much to be done at the moment, though.
Should this be closed?
Oh wow! Looks like we are indeed done. Thanks everyone. /close
Closing this issue. <detailsIn response to : Oh wow! Looks like we are indeed done. Thanks everyone. </details


Question: dswp.deleteProcessedPod(volumeToMount.PodName) } <ins> // Cleanup orphanded entries from processedPods dswp.pods.Lock() orphanedPods := make([]volumetypes.UniquePodName, 0, len(dswp.pods.processedPods)) for k := range dswp.pods.processedPods { if _, ok := podsFromCache[k]; !ok { orphanedPods = append(orphanedPods, k) } } dswp.pods.Unlock() for _, orphanedPod := range orphanedPods { uid := types.UID(orphanedPod) _, podExists := dswp.podManager.GetPodByUID(uid) if !podExists && dswp.podStateProvider.ShouldPodRuntimeBeRemoved(uid) { dswp.deleteProcessedPod(orphanedPod) } } </ins> podsWithError := dswp.desiredStateOfWorld.GetPodsWithErrors() for _, podName := range podsWithError { if _, podExists := dswp.podManager.GetPodByUID(types.UID(podName)); !podExists {

If a pod is created and then deleted an entry in processedPods map remains forever (sometimes, it's racy). The problem comes from the fact that calls volume_manager asynchronously. If the last call happens after populator cleaned up resources, an entry is to processedPods and will never be seen. An entry in processedPods map must be deleted if pod doesn't exist and pod runtime should removed according to . To verify that the issue really exists I applaied a simple printing resources in map. Here is a reproducer: brand-new fresh k8s cluster with entries in : a pod, check its uid: that this uid is accurately reported: the pod, wait for a few mins, check for again: The entry will be kept forever because kubelet doesn't know anything about this pod anymore: so no more events will ever come to populater for this uid, and the loop in won't find it because it was already deleted from . n/a <details</details<detailsany </details<details</details<details</details<details</details<details</details
/sig storage
/triage accepted
/priority impoerant-soon
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /priority impoerant-soon Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/priority important-soon
/assign
PR () fixed this issue. It is ready to merge since Mar 21.
/assign


Question: wait.Wait() close(statuses) <ins> pred := componentStatusPredicate(options) </ins> reply := []api.ComponentStatus{} for status := range statuses { <del> reply = append(reply, status) </del> <ins> // ComponentStatus resources currently (v1.14) do not support labeling, however the filtering is executed // nonetheless in case the request contains Label or Field selectors (which will effectively filter out // all of the results and return an empty response). if matched := matchesPredicate(status, &pred); matched { reply = append(reply, status) } </ins> } return &api.ComponentStatusList{Items: reply}, nil } <ins> func componentStatusPredicate(options *metainternalversion.ListOptions) storage.SelectionPredicate { pred := storage.SelectionPredicate{ Label:       labels.Everything(), Field:       fields.Everything(), GetAttrs:    nil, IndexFields: []string{}, } if options != nil { if options.LabelSelector != nil { pred.Label = options.LabelSelector } if options.FieldSelector != nil { pred.Field = options.FieldSelector } } return pred } func matchesPredicate(status api.ComponentStatus, pred *storage.SelectionPredicate) bool { // currently no fields except the generic meta fields are supported for predicate matching fieldsSet := generic.AddObjectMetaFieldsSet(make(fields.Set, 2), &status.ObjectMeta, true) return pred.MatchesObjectAttributes( status.ObjectMeta.Labels, fieldsSet, ) } </ins> func (rs *REST) Get(ctx context.Context, name string, options *metav1.GetOptions) (runtime.Object, error) { servers := rs.GetServersToValidate()

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: tried to get listing of all componentstatuses using a labelSelector query selector. I am hitting the apiServer directly. GET /api/v1/componentstatuses What you expected to happen: GET /api/v1/componentstatuses with labelSelector should limit responses to only componentstatuses with the specified label How to reproduce it (as minimally and precisely as possible): run GET /api/v1/componentstatuses with and without a labelSelector against the apiServer. Anything else we need to know?: specifically my label selector is in the form . and i can confirm everything works as expected when i hit GET /api/v1/pods Environment: Kubernetes version (use ): Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.6+IKS", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-05-09T13:26:51Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: IBM OS (e.g: ): Mac Kernel (e.g. ): Darwin Kernel Version 18.5.0 Install tools: Using javascript npm request to talk to api server Network plugin and version (if this is a network-related bug): Others:
/sig api-machinery
i assume its because you cant add a label to componentstatus, but they still shouldnt return if i specify a labelSelector
that's indeed a bug i presume. but what use-case makes you label-querying componentstatus?
we have a bot to periodically run a query that finds all known endpoints on the ApiServer. Then for each of the found endpoints, find all resources with our label defined. sort of an inventory tracking system.
in a similar vein, the endpoint doesnt seem to respect
/priority backlog /help
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /priority backlog /help Instructions for interacting with me using PR comments are available .

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I will take a look at this unless no objections
thank you, yours!
a PR for the ComponentStatus API, could you pls kindly take a look? I will look into the metrics API probably over the weekend. Thanks
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: "/var/lib/kubelet/pods/some-pod/volumes/kubernetes.io~fc/fc-in-pod2", } for _, path := range mountPaths { <del> refs, _ := mount.GetMountRefs(fm, path) </del> <ins> refs, err := mount.GetMountRefs(fm, path) if err != nil { t.Errorf("couldn't get mountrefs. err: %v", err) } </ins> var globalPDPath string for _, ref := range refs { if strings.Contains(ref, "kubernetes.io/fc") {

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Calls to on unsupported platforms do not report any error and potentially misbehave. What you expected to happen: I would expect an error to be reported, that can be consumed by the caller. How to reproduce it (as minimally and precisely as possible): for and were failing on darwin, and no error was being reported. See: & Environment: OS (e.g. from /etc/os-release): I have been testing on osx/darwin, which is not a supported platform.
/sig storage
/assign


Question: Allocatable: nodeRes, Conditions: []v1.NodeCondition{ { <del> Type:   v1.NodeReady, Status: v1.ConditionTrue, </del> <ins> Type:              v1.NodeReady, Status:            v1.ConditionTrue, LastHeartbeatTime: metav1.Now(), </ins> }, }, },

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: Which test(s) are failing: TaintBasedEvictions Since when has it been failing: probably since it was in this PR? Testgrid link: Reason for failure: Anything else we need to know: /sig scheduling /kind flake /priority important-soon /milestone v1.17
You must be a member of the GitHub team to set the milestone. If you believe you should be able to issue the /milestone command, please contact your and have them propose you as an additional delegate for this responsibility. <detailsIn response to : <!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs -- TaintBasedEvictions probably since it was in this PR? Anything else we need to know: /sig scheduling /kind flake /priority important-soon /milestone v1.17 </details
cc.
Looking into this. We moved it to integration because it was flaking in e2e too
/assign damemi
Not sure yet, but I do see this in the logs of the failing tests: which, if I understand this, would remove all the taints and mark nodes as reachable (which would explain why the test is failing waiting for node taints: ). But I need to figure out why all the nodes are becoming unavailable
is that a full fix for this flake or just partial? I still saw it failed yesterday so just checking... :)
I think I also run into this on 13th Oct: Full logs:
Now gets merged, which might fix the flake. Let's keep an eye on this.
reported it still flaked: Can you continue looking into this?
I'll keep trying, from my previous debugging it seems that in some cases the test cluster is entering full disruption mode and removing all of the taints from the nodes. I know we specifically made changes to the test (using multiple nodes) to ensure it wouldn't become fully disrupted, do you have any idea what could still be causing that?
There is some flag controlling how the dispution thing is managed internally, maybe you can check how a node lifecycle manager is constructed.

The test still flakes after we merge , we still need to investigate the cause ,
/assign
Another flake:
These flakes are consistently (always) timeouts waiting on node taints:
I'm working on this. Will send out a PR soon.
/reopen We see another flake since got merged: However, don't panic :) The original "Entering master disruption mode" cause has been fixed - you won't see that message in the above log. This time it's a smaller issue. I will explain it in another PR.
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen The job is still flaky
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/close But for another test :)
Closing this issue. <detailsIn response to : /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: Filtered []string } <ins> func (templater *templater) FlagErrorFunc(exposedFlags ...string) func(*cobra.Command, error) error { return func(c *cobra.Command, err error) error { c.SilenceUsage = true switch c.CalledAs() { case "options": return fmt.Errorf("%snRun '%s' without flags.", err, c.CommandPath()) default: return fmt.Errorf("%snSee '%s --help' for usage.", err, c.CommandPath()) } } } </ins> func (templater *templater) ExposeFlags(cmd *cobra.Command, flags ...string) FlagExposer { cmd.SetUsageFunc(templater.UsageFunc(flags...)) return templater

The majority of kube binaries print usage when certain invalid arguments are presented. Unfortunately, most of our binaries have 2 pages of usage output, and the result of printing that is a very confusing and context disrupting action. In general, CLI commands try to minimize their output - for example: both print their minimal sets. Obviously, as commands grow the number of flags some of the short options become a bit silly (yes, congratulations, you have the entire alphabet of flags) and convey less useful information to a user. When errors are printed we also end up printing the error both above and below the command (in at least kubectl, kubelet, kube-apiserver, and controller-manager). I'm not even sure this was intentional, but coincidentally we print the errors twice and so it's both before and after the usage. Error feedback needs to minimize the amount of extraneous information presented to users for maximum clarity. Three are broadly four classes of help and error output: user explicitly requests help via or user provides an input that is not recognized (flag, command, or argument) and we can unambiguously know provided invalid input. User needs to change the input. user provides inputs that don't work well together or have conflicting meaning. User needs to change the input. user provided inputs that result in a runtime error, but that error can't be resolved by changing the input of the command. Recommended changes: our command and flag sets are so long, I don't think there is value in printing the full usage info in 2 and 3. Instead, I think we should print a minimal "usage: Run "command foo --help" to see usage info" message after printing the error. should stop printing error output twice (once as and once as ) in 2-4 and print as or (the former is more consistent with Posix guidelines). still need to clean up output.
this is really painful as an end user (and I've been tracking people who have these issues). I hit this while looking at and trying to clean it up, and realized we need to fix the broader problem.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with .

Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen /assign This is still a thing.
Reopened this issue. <detailsIn response to : /reopen /assign This is still a thing. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove lifecycle/rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This is still a thing. /assign /reopen
Reopened this issue. <detailsIn response to : This is still a thing. /assign /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
GitHub didn't allow me to assign the following users: sallyom.

Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : This is still a thing. /assign /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This is still very much a problem. Having to scroll through pages of options just to find the error output is terrible. I'm running kube-apiserver with systemd so it automatically restarts. This means the error itself ends up in the middle of hundreds of lines of unhelpful information. It's like finding a needle in a haystack to figure out what actually went wrong. If nothing else, a flag to turn it off would make it more manageable. /reopen /remove lifecycle/rotten
You can't reopen an issue/PR unless you authored it or you are a collaborator. <detailsIn response to : This is still very much a problem. Having to scroll through pages of options just to find the error output is terrible. I'm running kube-apiserver with systemd so it automatically restarts. This means the error itself ends up in the middle of hundreds of lines of unhelpful information. It's like finding a needle in a haystack to figure out what actually went wrong. If nothing else, a flag to turn it off would make it more manageable. /remove lifecycle/rotten Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen /remove-lifecycle rotten
Reopened this issue.

<detailsIn response to : /reopen /remove lifecycle/rotten Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: limitations under the License. */ <del> // Package options contains flags and options for initializing an apiserver </del> <ins> // Package options contains flags and options for initializing kube-apiserver </ins> package options import (

Currently as authentication and authorization are both standard config for all clusters, the kubeapiserver insecure http handler is unsecure, which means the cluster is vulnerable to the attack. So it should be deprecated, even removed totally.
/sig api-machinery
I think removing the insecure option is a good idea. Can you send an email the sig-apimachinery mailing list with a proposed timeline? I think this should be like other GA features and have a one year clock.
Sure, I will send it ASAP.
:+1:


Question: FilenameParam(flags.Filenames...). Flatten(). Do() <ins> checkErr(r.Err()) </ins> patch := cmdutil.GetFlagString(cmd, "patch") if len(flags.Filenames) == 0 && len(patch) == 0 {

I've reported this before.
From a Makefile I get exit status 255.
Just to be clear (sorry for the basic question), are you saying that when the service file is malformed the crash is unhelpful for debugging? Or that creating any/all services using is currently broken?
The YAML file is not malformed. I think there was something perhaps not quite right about firewalls/rules etc. and this caused to crash unhelpfully.


Question: pod     *v1.Pod mounter mount.Interface plugin  *nfsPlugin <del> volume.MetricsNil </del> <ins> volume.MetricsProvider </ins> } func (nfsVolume *nfs) GetPath() string {

/kind feature What happened: on a cluster with an nfs persistent volume present: What you expected to happen: I expected metrics concerning the nfs persistent volume to be returned e.g: (from a separate aws cluster) Anything else we need to know?: I'm assuming that this is just not implemented currently? I couldn't find a referencing issue Environment: Kubernetes version (use ):Cloud provider or hardware configuration: on-prem OS (e.g. from /etc/os-release):Kernel (e.g. ):
/sig instrumentation
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/open Metrics is missing for NFS driver.
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-lifecycle rotten
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/sig storage


Question: return r } <del> /* // ParseSelectorParam parses the given string as a resource selector. // This is a convenience function so you don't have to first check that it's a // validly formatted selector. func (r *Request) ParseSelectorParam(paramName, item string) *Request { if r.err != nil { return r </del> <ins> const ( // A constant that clients can use to refer in a field selector to the object name field. // Will be automatically emitted as the correct name for the API version. ObjectNameField = "metadata.name" PodHost         = "spec.host" ) type clientFieldNameToAPIVersionFieldName map[string]string func (c clientFieldNameToAPIVersionFieldName) filterField(field, value string) (newField, newValue string, err error) { newFieldName, ok := c[field] if !ok { return "", "", fmt.Errorf("%v - %v - no field mapping defined", field, value) </ins> } <del> var selector string var err error switch paramName { case "labels": var lsel labels.Selector if lsel, err = labels.Parse(item); err == nil { selector = lsel.String() } case "fields": var fsel fields.Selector if fsel, err = fields.ParseSelector(item); err == nil { selector = fsel.String() } default: err = fmt.Errorf("unknown parameter name '%s'", paramName) </del> <ins> return newFieldName, value, nil } type resourceTypeToFieldMapping map[string]clientFieldNameToAPIVersionFieldName func (r resourceTypeToFieldMapping) filterField(resourceType, field, value string) (newField, newValue string, err error) { fMapping, ok := r[resourceType] if !ok { return "", "", fmt.Errorf("%v - %v - %v - no field mapping defined", resourceType, field, value) </ins> } <del> if err != nil { r.err = err return r </del> <ins> return fMapping.filterField(field, value) } type versionToResourceToFieldMapping map[string]resourceTypeToFieldMapping func (v versionToResourceToFieldMapping) filterField(apiVersion, resourceType, field, value string) (newField, newValue string, err error) { rMapping, ok := v[apiVersion] if !ok { glog.Warningf("field selector: %v - %v - %v - %v: need to check if this is versioned correctly.", apiVersion, resourceType, field, value) return field, value, nil </ins> } <del> return r.setParam(paramName, selector) }*/ </del> <ins> newField, newValue, err = rMapping.filterField(resourceType, field, value) if err != nil { // This is only a warning until we find and fix all of the client's usages. glog.Warningf("field selector: %v - %v - %v - %v: need to check if this is versioned correctly.", apiVersion, resourceType, field, value) return field, value, nil } return newField, newValue, nil } var fieldMappings = versionToResourceToFieldMapping{ "v1beta1": resourceTypeToFieldMapping{ "nodes": clientFieldNameToAPIVersionFieldName{ ObjectNameField: "name", }, "minions": clientFieldNameToAPIVersionFieldName{ ObjectNameField: "name", }, "pods": clientFieldNameToAPIVersionFieldName{ PodHost: "DesiredState.Host", }, }, "v1beta2": resourceTypeToFieldMapping{ "nodes": clientFieldNameToAPIVersionFieldName{ ObjectNameField: "name", }, "minions": clientFieldNameToAPIVersionFieldName{ ObjectNameField: "name", }, "pods": clientFieldNameToAPIVersionFieldName{ PodHost: "DesiredState.Host", }, }, "v1beta3": resourceTypeToFieldMapping{ "nodes": clientFieldNameToAPIVersionFieldName{ ObjectNameField: "metadata.name", }, "minions": clientFieldNameToAPIVersionFieldName{ ObjectNameField: "metadata.name", }, "pods": clientFieldNameToAPIVersionFieldName{ PodHost: "spec.host", }, }, } </ins> // FieldsSelectorParam adds the given selector as a query parameter with the name paramName. func (r *Request) FieldsSelectorParam(s fields.Selector) *Request {

My broke it for v1beta3, then fixed it for v1beta3 and broke it for v1beta1. I'm working on a fix.
Honestly I'm very surprised that more people aren't complaining-- integration test logs are full of noise because of this.
Exact text of error log is "Failed to watch *api.Node: field label not supported: "
cc


Question: // WaitForPodsRunningReady waits up to timeout to ensure that all pods in // namespace ns are either running and ready, or failed but controlled by a <del> // replication controller. Also, it ensures that at least minPods are running // and ready. It has separate behavior from other 'wait for' pods functions in // that it requires the list of pods on every iteration. This is useful, for </del> <ins> // controller. Also, it ensures that at least minPods are running and // ready. It has separate behavior from other 'wait for' pods functions in // that it requests the list of pods on every iteration. This is useful, for </ins> // example, in cluster startup, because the number of pods increases while // waiting. // If ignoreLabels is not empty, pods matching this selector are ignored and

The logic to account for all pods being ready is incorrectly counting some pods as being controlled by a replication controller. This is causing the 1.3-1.5 upgrade tests to fail. See https://k8s- as an example. Since , the pre-flight checks fail and the rest of the tests never run. I think this was exposed by which changed some things from ReplicationControllers to Deployments. There's a pod being created by a deployment that also matches the selector of a replication controller, and is therefore being incorrectly counted by cc


Question: o.mapper, _ = f.Object() o.out = os.Stdout _, pods := createPods(false) <del> pendingPods, err := o.waitForDelete(pods, test.interval, test.timeout, test.getPodFn) </del> <ins> pendingPods, err := o.waitForDelete(pods, test.interval, test.timeout, false, test.getPodFn) </ins> if test.expectError { if err == nil {

In writing some documentation for I noticed some issues with the implementation. I think the simplest user experience is that the command doesn't return until all the user pods are drained off the machine. In particular this means wait until all graceful termination is finished infinitely retry evictions that are rejected due to insufficient PodDisruptionBudget The current implementation does not do this, in particular wait for graceful termination appears to be at 10 sec number of pods, which may be much shorter than the max graceful termination we only retry evictions number of times My suggestion is to change the behavior so that we compute how long to wait for graceful termination, by taking max(grace periods of all pods that we want to delete) (or the value of --grace-period command line argument instead, if that is specified), and wait exactly that amount of time plus a small buffer we retry evictions infinitely WDYT? I think we should do this in 1.5 because the current behavior is very non-obvious.
The problem here is how long should we wait, if the user doesn't specify a timeout. The default timeout used in drain is following how long the reapers have in , , and . Maybe this is not long enough. I can make the changes. Do we still need exponential back off?
To summarize discussion with there are two possible approaches (pseudocode below) 1) Goroutine approach: For each pod P that meets the , do the following: 2) Non-goroutine approach: where "not timed out yet" means you haven't finished waiting max(--grace period argument to kubectl drain, max graceful termination spec across all pods on the node)
Could you please triage this for 1.5 and apply blocker or non-blocker labels as appropriate along with priority.
The PR for this issue is waiting to be merged.
marking this as a release blocker due to being marked as such. Please update if needed cc:


Question: Opts:   []string{"rw", "relatime", "pagesize=2M"}, }, { <ins> Device: "/dev/hugepages", Type:   "hugetlbfs", Path:   "/mnt/hugepages-2Mi", Opts:   []string{"rw", "relatime", "pagesize=2Mi"}, }, { </ins> Device: "sysfs", Type:   "sysfs", Path:   "/sys",

What happened: Pods get stuck in terminating state, as kubelet failed to unmount volume: nodev on /var/lib/kubelet/pods/-120c-45c4-aeac- type hugetlbfs (rw,relatime,pagesize=2Mi) What you expected to happen: Pod to get deleted and volumes to be cleaned up. How to reproduce it (as minimally and precisely as possible): kvm based virtual machines on ubuntu 16.04. create a pod with hugepage volumes () Delete the pod. Anything else we need to know?: Changes are related to mentioned commit id: In this file : , line no 72 : pageSize, err := resource.ParseQuantity(strings.TrimPrefix(opt, prefix) + "i") The mount already has a pagesize 2Mi in my case as shown here : (rw,relatime,pagesize=2Mi) But the code here suffix/appends "i" assuming pagesize can be only of type 2M or 1024M resulting in failure to clean up this volume. Documentation from code: / NOTE: Adding suffix 'i' as result should be comparable with a medium size. // pagesize mount option is specified without a suffix, // e.g. pagesize=2M or pagesize=1024M for x86 CPUs pageSize, err := resource.ParseQuantity(strings.TrimPrefix(opt, prefix) + "i") if err != nil { return nil, fmt.Errorf("error getting page size from '%s' mount option: %v", opt, err) } when the pod is deployed, kubelet creates the mount and we can see from logs that it already has pagesize=2Mi for my environment.

Environment: Kubernetes version: kubectl version Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2020-03-25T14:58:59Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2020-04-16T11:48:36Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: Kvm hypervisor based virual machine setupOS (e.g: ): NAME="Ubuntu" VERSION="16.04.6 LTS (Xenial Xerus)" ID=ubuntu IDLIKE=debian PRETTYNAME="Ubuntu 16.04.6 LTS" VERSIONID="16.04" HOMEURL="" SUPPORTURL="" BUGREPORTURL="" VERSIONCODENAME=xenial UBUNTUCODENAME=xenial Kernel (e.g. ): Linux master 4.4.0-169-generic -Ubuntu SMP Tue Nov 12 10:38:00 UTC 2019 x8664 x8664 x8664 GNU/LinuxInstall tools: Network plugin and version (if this is a network-related bug): Others:
/assign bart0sh
/sig storage
What about this change ?
The change looks good to me. Would you be willing to submit a PR?
& : Thanks for the quick turnaround.
We are seeing this issue in 1.18.6 version, can you please push the fix to this branch as well.
Could we know when the problem was introduced? As I suppose we don't have the same issue with K8S 1.17.x release?
Hi same issue found in 1.18.2
It is not yet ported into v1.18.6 as well.
Can you submit your PR for release-1.18 branch?


Question: # This configuration file for Fluentd / td-agent is used # to watch changes to Docker log files that live in the # directory /var/lib/docker/containers/ and are symbolically <del> # linked to from the /varlog directory using names that capture the </del> <ins> # linked to from the /var/log directory using names that capture the </ins> # pod name and container name. These logs are then submitted to # Google Cloud Logging which assumes the installation of the cloud-logging plug-in. #

When I run Kubernetes cluster on GKE, it generates a lot of system logs coming from docker and kubelet. Even if my cluster runs just a few pods, I can see dozens or hundreds of log entries per second available in Google Cloud Logging in docker and kubelet logs. This makes the actual logs from my application get lost among them, if I do not set appropriate filter. Also, I am afraid that collecting massive amounts of logs imposes unnecessary workload on fluentd pod in my cluster. It would be good if I could just disable collection of kubelet/docker logs (and potentially other Kubernetes system logs) assuming that I do not want to have these logs exported.
/cc
since you tend to have a vested interest in node debug-ability, what's the minimum you'd want to have available here? From GKE's perspective, our users typically have no interest in these logs, so turning them off by default, even without an easy option to re-enable them, makes sense. Any user sophisticated enough to want them shouldn't have problems SSH'ing to the node for them. Is the story all that different in open-source kubernetes? Does anyone even care to have the kubelet and docker logs collected from the nodes? I've never used cloud logging to debug issues with them, it's always been easier to just SSH to the node. I'd lean towards not collecting them on the nodes in general. Also cc in case he has wisdom he'd like to impart
All I can say is that personally I have never accessed kubelet etc. logs "for real" when debugging -- as Alex says -- it is usually more direct just to SSH in. I would support not collecting these logs to improve cluster stability (but perhaps have a switch / technique to make it easy to collect them if the admin does care about this?).
One last call for to complain, because I'm starting on this now
Also
We want to avoid storing logs onto local disk in the future. In-order to achieve that, we need the existing cluster level logging infrastructure to work. Can the GKE problem be solved by moving system logs to a separate namespace? , Alex Robinson wrote:
How far in the future? Unless that's intended to happen soon, collecting the kubelet and docker logs in the meantime offers little value with a nontrivial resource (cpu+memory+egress) cost.

But yes, we can handle the system logs separately if we'd like. I think the most natural way would be to report them as compute engine logs, which could be done by adding an extra config option into the gcp output plugin.
, Alex Robinson wrote:
-- just to confirm, do you think this fix will be sufficient for you? In GKE, logs from containers will be listed under Container Engine, while logs from Docker and the Kubelet will be listed under Compute Engine, with the logs for each effectively being entirely seperated in the log viewer.
Yes, this will be sufficient in terms of not polluting Container Engine logs with system logs. I am still concerned about unnecessary load on fluentd caused by exporting Docker and Kubelet logs so ideally it would be great not to export these system logs if a user is not interested in them. But this is of slightly lower priority.
Hi, What's the status from this request? Is this available in 1.7.x or higher? Thanks.


Question: NegotiatedSerializer: resource.UnstructuredPlusDefaultContentConfig().NegotiatedSerializer, Client: fake.CreateHTTPClient(func(req *http.Request) (*http.Response, error) { switch p, m := req.URL.Path, req.Method; { <del> case strings.HasPrefix(p, pathCM) && m != "GET": pod := ioutil.NopCloser(bytes.NewReader(cmBytes)) return &http.Response{StatusCode: 200, Header: cmdtesting.DefaultHeader(), Body: pod}, nil case strings.HasPrefix(p, pathCM) && m != "PATCH": pod := ioutil.NopCloser(bytes.NewReader(cmBytes)) return &http.Response{StatusCode: 200, Header: cmdtesting.DefaultHeader(), Body: pod}, nil </del> <ins> case strings.HasPrefix(p, pathCM) && m == "GET": fallthrough case strings.HasPrefix(p, pathCM) && m == "PATCH": var body io.ReadCloser switch p { case pathCM + "/test0": body = ioutil.NopCloser(bytes.NewReader(configMapList[0])) case pathCM + "/test1": body = ioutil.NopCloser(bytes.NewReader(configMapList[1])) default: t.Errorf("unexpected request to %s", p) } return &http.Response{StatusCode: 200, Header: cmdtesting.DefaultHeader(), Body: body}, nil </ins> default: t.Fatalf("unexpected request: %#vn%#v", req.URL, req) return nil, nil

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: The dry run output listed all the resources that would be configured: deployments, HPAs, services, and an ingress: This was expected. However, it then listed all resources again (except for the HPAs) as being pruned: The config was then applied with: As expected, no resources were actually pruned. What you expected to happen: should actually display which resources are going to be pruned. How to reproduce it (as minimally and precisely as possible): a YAML file to with it: run + prune: should be listed as both configured and pruned it again with pruning: that the deployment is configured but not pruned, even though the dry run said it would be. Anything else we need to know?: Was trying to write a devops script that gave a warning before any resources would be pruned. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: GKE OS (e.g. from /etc/os-release): (Lubuntu 18.04) Kernel (e.g. ):Install tools: Others:
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I can reproduce it, and i will try to fix it.


Question: ginkgo.By("Creating statefulset") statefulset := e2estatefulset.CreateStatefulSet(client, manifestPath, namespace) <ins> defer e2estatefulset.DeleteAllStatefulSets(client, namespace) </ins> replicas := *(statefulset.Spec.Replicas) // Waiting for pods status to be Ready e2estatefulset.WaitForStatusReadyReplicas(client, statefulset, replicas)

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: The following issues are observed while running this e2e test case for vSphere CSI Migration feature validation. Issue-1 - test failed with Unexpected error: This is happening because YAML is using version. Issue-2 - After upgrading from . Test executed successfully but reported failure in the cleanup stage. when is called in DeleteAllStatefulSets, it does not find statefulset. moving call to in the defer block helps avoid printing this message. There may be a race between namespace deletion and after each execution. I am not familiar with recent e2e framework changes. What you expected to happen: Test should pass. How to reproduce it (as minimally and precisely as possible): execute using Kubernetes master branch builds. Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: vSphere Cloud Provider
/sig testing /sig storage
/remove-sig testing The framework and CI are SIG testing. SIG storage owns their tests.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten


Question: inkscape:cx="306.01712" inkscape:cy="-127.6824" inkscape:document-units="px" <del> inkscape:current-layer="layer1" </del> <ins> inkscape:current-layer="text4373" </ins> showgrid="false" inkscape:window-width="1519" inkscape:window-height="822"

The logo (, ) font does not match the others. I believe the png/pdf versions are correct.
What other ones? It should be the same font - they look the same to me...