Question: pod := downwardAPIVolumeForContainerResources(podName, "/etc/podinfo/memory_limit") e2epodoutput.TestContainerOutput(ctx, f, "downward API volume plugin", pod, 0, []string{ <del> fmt.Sprintf("67108864n"), </del> <ins> "134217728n", </ins> }) })

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: node-kubelet-master Which test(s) are flaking: [sig-storage] Downward API volume should provide container's memory limit [NodeConformance] [Conformance] [cos-stable1] [sig-storage] Downward API volume should provide container's cpu limit [NodeConformance] [Conformance] [ubuntu] Testgrid link: https://k8s- Reason for failure: Both of these tests are failing with same reason. Anything else we need to know: links to appreciated links to specific failures in spyglass appreciated <!-- Please see the deflaking doc () for more guidance! --/sig storage
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage
Hi this issue needs your attention.
Still flaking
Hi can you call attention of the relevant folks here as well ?
For the first test failure posted here on 09/30. The reason is not enough CPU available. The type of error only happened once, it never happen again for this test. For the test failure mentioned, more than half of the tests (150+) failed including this Downward API tests. It happened quite often for ci-kubernetes-e2e-windows-containerd. It seems something wrong with container runtime. The pods are all stuck on pulling image phase, no other events after that and then timed out. Seems containerd for windows have some issues.. So I think this specific test is not considered as flaky test.
Latest failure: This failure includes, lot of other failing test as well. So could, for another major reason. Time Test -- -- 01/12/2020, 06:31:17 [sig-storage] Downward API volume should provide container's memory limit [NodeConformance] [Conformance] Though most flakes are on windows related jobs. But, there are other instances too.

Do you think, above failure is for the same reason.
I am seeing downward api flakes in the Windows tests as well. If I broaden the search a bit it seems to happen for more than just memory limit but also: Test [sig-storage] Downward API volume should provide container's cpu request [NodeConformance] [Conformance] [sig-storage] Downward API volume should provide container's memory request [NodeConformance] [Conformance] [sig-storage] Downward API volume should provide container's cpu limit [NodeConformance] [Conformance] [sig-storage] Downward API volume should provide container's cpu request [NodeConformance] [Conformance] [sig-storage] Downward API volume should provide container's cpu request [NodeConformance] [Conformance] For Windows they seem to be failing with the following (both docker and containerd): I will can look into this from a Windows perspective. /assign
For Windows, I do not believe this is an issue with the test it's self. I ran this subset of tests continuously over the weekend and got the following results: The is "paging file is too small": I was able to force similar (but not exactly the same) errors by over consuming memory via a test pod. I believe this might be happening due to other tests running on the cluster at the same time. I am looking into this further.
A few tests failed similar to this over the weekend:
FYI , during this saw some of the issues w/ containerd leaks
/sig windows
For the aks-engine Containerd test cases we lowered the number of running to the tests concurrency value we had prior to 1.19 and this has gone away. I am continuing to looking into if this was a regression or if is releated.
We haven't seen this in a long time but it came up today:
Another failure in gce windows test:
another failure here:
/milestone v1.22
another failure:
cc
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen /remove-lifecycle rotten We saw this again in master
Reopened this issue.

<detailsIn response to : /reopen /remove-lifecycle rotten Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/milestone clear
We have a lead this might be related to working set bytes vs commit bytes and the way that windows handles memory.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/sig node
/remove-lifecycle rotten
/remove-sig It looks like sig storage issue. Please reapply if requires sig-node to triage.
This still flakes. An example: are you still working on this?
I haven't looked it from a Linux perspective. From Windows side, it seems this is caused by a mis-match between the commit memory and working set:
Posting some details here, since the issue above has been mentioned. We've been flaking on these downward API tests for quite a while, with errors like these: Logs taken from: Those error logs are actually from the Pod itself, which is supposed to run the agnhost golang binary. It doesn't always happen, but it does frequent enough to be visible in our Windows dashboards (8 failures in the past week, according to: (%20volume)%3F%20should%20provide%20container%27s).

Looking through the downward API E2E tests, we noticed that some memory limits are being set in 8 test cases (). The memory limit set is 64MB, which is a bit low for a Windows Pod, even if it's nanoserver-based (also, the agnhost golang binary is not that small either, it has 44.2 MB on Linux). IMO, we could increase the memory limit to 128MB and we should see fewer flakes.
+1


Question: }, metricsName: "scheduler_pending_pods", wants: ` <del> # HELP scheduler_pending_pods [ALPHA] Number of pending pods, by the queue type. 'active' means number of pods in activeQ; 'backoff' means number of pods in backoffQ; 'unschedulable' means number of pods in unschedulableQ. </del> <ins> # HELP scheduler_pending_pods [STABLE] Number of pending pods, by the queue type. 'active' means number of pods in activeQ; 'backoff' means number of pods in backoffQ; 'unschedulable' means number of pods in unschedulableQ. </ins> # TYPE scheduler_pending_pods gauge scheduler_pending_pods{queue="active"} 15 scheduler_pending_pods{queue="backoff"} 25

I would like to propose the graduation of the following scheduler metrics from alpha to stable: These metrics have been stable for at least 15 months.
/sig scheduling /sig instrumentation
/cc
IIRC, the last addition to these metrics was the profile name label. I don't foresee need for new labels. I don't see many precedents for graduation in core components. are there any defined criteria for graduation?
As requested, the following is how long it has been since a change has been made to each of the requested metrics: - 2 years; last change was to migrate to the metrics stability framework - 17 months; last change was a renaming of the metric - 15 months; last change was a renaming of the metric - 17 months; last change was to add a label - 17 months; last change was to add a label
It's up to the component owners to commit to graduating these metrics, since the component owners own the metrics and therefore need to agree to support them in accordance to stability requirements. If you guys are okay with promoting these metrics, I don't see a reason, from an instrumentation side, why we would object to promotion.
+1 to graduation then. These metrics are very mature. Although, I'm not sure if all metrics already adhere to naming conventions. and don't have a unit suffix, but they are unit-less. Is that fine
is a gauge so it's fine, is a histogram so it's also fine.
I'm recently working with metrics, I'm glad to provide some help if no one has interest about it.
There is already an open PR for this.
Oh sorry, didn't notice about that. Forget it.
These graduations sound good to me, but I would be inclined to rename . I feel that the part of the metric is unnecessary. Having instead and keeping for the algorithm latency is already descriptive enough to know the difference between both. If that's not the case, then the text of the metric can be used to guide the users.
I agree that is rather confusing, so +1 to removing the prefix. The metric doesn't include retries, for example. We have a separate metric that includes retries: We need to rename that one eventually to e2e, but it can only be done after the existing one has been deprecated and deleted for a few releases. /triage accepted
/assign
/cc
The PR for this is ready to be reviewed:


Question: Cgroups: []string{"cpu", "cpuacct", "cpuset", "devices", "freezer", "memory"}, RuntimeSpec: RuntimeSpec{ DockerSpec: &DockerSpec{ <del> Version:     []string{`1.1[1-3]..*`, `17.0[3,6,9]..*`, `18.0[6,9]..*`}, </del> <ins> Version:     []string{`1.1[1-3]..*`, `17.0[3,6,9]..*`, `18.0[6,9]..*`, `19.03..*`}, </ins> GraphDriver: []string{"aufs", "overlay", "overlay2", "devicemapper", "zfs"}, }, },

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Support for newer docker versions: 19.x Why is this needed: New users trying to use kubernetes install the latest docker version. It's inconvenient see the following warning and have to uninstall docker and then install an older version. Also, it's not clear what could happen if someone uses a newer version of docker. Lastly, I believe that kubernetes should be able to keep up with docker. Is there any roadmap for this, that I have missed?
/sig usability
you could work with sig-testing and sig-node to set up CI jobs when new versions of Docker shows up. Believe me, it's more inconvenient to debug failures in a new docker version as a end use than the downgrade. We have to have some CI coverage of a version before we ok its use by the community.
Hi I'm new to this, so I'm not sure how to apply your suggestion: Is there any documentation that I could use?
SIGs are teams working on specific areas of kubernetes All our test infrastructure for CI jobs ("Continuous Integration") is in: Out test-grid is here (please type docker in the search box on top to see the current CI jobs):
Since docker-ce 18.09 has reached EOL and there are no new versions planned (), it would be great to have official support for a newer version of docker-ce.
please see:
Hi folks, David from Docker here, we have run and validated Docker Engine 19.03 for Kubernetes 1.14 back in June, so Docker 19.03 is indeed supported for Kubernetes 1.14 and above: Let me know how else we can help.
thanks for confirming PR is here
Thank you Appreciate the PR, we typically test engines for major versions of Engine that are released like 19.03 with the latest version of Kubernetes that is available at the time. On our next engine, we will re-run the test and submit the results to testgrid. Let me know how I can better coordinate this.
Also let me know if there is anything else that is needed to close this issue out. Thanks.
hi, the PR should close this issue. i guess, one option is just creating a similar issue once the test job is available in test-grid and running green for a while? or me can then proceed to update the validators.

Ok thank you, that sounds like a good process we can follow moving forward.
Hi Any chance this can make its way into 1.16 at least? This way it may get picker up by Kops and not have to rely on some old and unpatched Docker version. At the moment Docker CE 18.09 is no longer actively supported
hi, we cannot backport this as it's not a critical fix. at least in the case of kubeadm the 19. versions of docker would only print a warning.
but doesn't the existence of a vulnerability make this critical enough?
the usage of Docker 19. is not blocked in any way in 1.16. you can ignore the and just use it. if something was blocking the usage of 19., then this is backport worthy.
I saw this as the first step before adding 19.03 as a validated external release. Without being documented as such Kops will not pick it up as default and will leave people with old and vulnerable Docker versions. If this I understand correctly the above comments, 19.03 has been validated for some time, just not documented.
i don't think kops uses the kubeadm validators that reside in k/k/cmd/kubeadm/app/... this was resolved in this PR the k8s docs for CRI are still out of date: so if someone wants to send a PR for that page please go ahead: (but AFAIK not all distros have moved to 19. yet)
I couldn't find any repos missing docker 19.03.4. Any chance to do the change in the changelog?


Question: InitSnapshottableTestSuite, InitSnapshottableStressTestSuite, InitVolumePerformanceTestSuite, <ins> InitReadWriteOncePodTestSuite, </ins> ) func getVolumeOpsFromMetricsForPlugin(ms testutil.Metrics, pluginName string) opCounts {

For the ReadWriteOncePod feature to graduate to beta, we need to have E2E test coverage for its current alpha behavior. See this section of KEP-2485 for more context on what cases we need to cover for alpha: This is a requirement for beta graduation, see:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage


Question: # See the License for the specific language governing permissions and # limitations under the License. <del> # Generates `types_swagger_doc_generated.go` files for API group </del> <ins> # This script generates `types_swagger_doc_generated.go` files for API group </ins> # versions. That file contains functions on API structs that return # the comments that should be surfaced for the corresponding API type # in our API docs.

What would you like to be : There are many useful scripts in "". For example, I often use and for creating simple k8s cluster quickly. However, there are only few explanation or nothing how to use it. I think it becomes more useful and helpful if some instruction or guidance is explained in each script. Why is this needed: It aims and expects to increase new k8s developers by describing how to use each script. My proposal Add the following description under the copyright for each script file. Overview of the script Simple usage Supplementary explanation(This is option) Other concern I would like to know the purpose of use about hack directory. (e.g. Where to put tools for developers(therefore there is no guidance?).)
I don't know who is in charge of each scripts of hack, but if he agrees the idea of this feature request , I am going to do this task myself!!
/sig testing
I believe there is a script generating swagger/openapi specfilters under hack, but I am still struggling with it. It could be of great help if there is some how-to doc for each scripts.
this basis, why not create a file under the hack folder, because a lot of kk CIs are derived from multiple script commands under the hack. There are already 268+ files under the hack file. For contributors unfamiliar with these, these comments and instructions are helpful to them, and it is also helpful for new contributors to handle pr ci error reports.
/cc What do you think
I think it's good idea to create (It may be easy to understand if the script's explanation is described by category or use in ) Apart from that, it is helpful that the script file itself has a brief description.
When we create , it may tough work if creating it including all scripts explanation at one time. (There are many scripts and I understand only some scripts now.) Regarding , current my idea is to add a description to each script little by little and update the at the same time. This means that will be completed little by little in conjunction with adding the description of each script. In the future, if someone create new script in hack, he also may have to add description of his script into
Self-documenting commands sound useful. A high level overview also sounds useful.

/cc /cc Regarding adding help information, I suggest that we can submit pr in one or several files as a unit, so that we can track and statistics. For the documentation, I think it should be concise and not too long. How about this advice? We can claim tasks through , the pr title yes , i agree with you. [x] kubernetes/hack/benchmark- [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] kubernetes/hack/boilerplate/BUILD [ ] kubernetes/hack/boilerplate/test/BUILD [ ] [ ] [ ] [ ] [ ] kubernetes/hack/BUILD [x] kubernetes/hack/build- [x] kubernetes/hack/build- [x] [ ] kubernetes/hack/conformance/BUILD [ ] [x] kubernetes/hack/dev-build-and- [x] kubernetes/hack/dev-build-and- [x] kubernetes/hack/dev-push- [ ] kubernetes/hack/e2e-internal/BUILD [ ] kubernetes/hack/e2e-internal/e2e-cluster- [ ] kubernetes/hack/e2e-internal/e2e- [ ] kubernetes/hack/e2e-internal/e2e-grow- [ ] kubernetes/hack/e2e-internal/e2e-shrink- [ ] kubernetes/hack/e2e-internal/e2e- [ ] kubernetes/hack/e2e-internal/e2e- [x] kubernetes/hack/e2e-node- [ ] kubernetes/hack/generate- [x] kubernetes/hack/generate- [ ] kubernetes/hack/gen-swagger-doc/gen-swagger- [x] kubernetes/hack/get- [x] kubernetes/hack/ginkgo- [x] kubernetes/hack/grab- [x] kubernetes/hack/install- [ ] kubernetes/hack/jenkins/benchmark- [ ] kubernetes/hack/jenkins/OWNERS [ ] [ ] kubernetes/hack/jenkins/test- [ ] kubernetes/hack/jenkins/verify- [ ] kubernetes/hack/lib/BUILD [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] [x] kubernetes/hack/lint- [x] kubernetes/hack/list-feature- [x] kubernetes/hack/local-up- [ ]

kubernetes/hack/make-rules/BUILD [ ] kubernetes/hack/make- [ ] kubernetes/hack/make- [ ] kubernetes/hack/make- [ ] kubernetes/hack/make- [ ] kubernetes/hack/make-rules/helpers/go2make/BUILD [ ] kubernetes/hack/make- [ ] kubernetes/hack/make- [ ] kubernetes/hack/make- [ ] kubernetes/hack/make-rules/helpers/go2make/testdata/dir-with- [ ] kubernetes/hack/make-rules/helpers/go2make/testdata/dir-with- [ ] kubernetes/hack/make-rules/helpers/go2make/testdata/dir-with-gofiles/README [ ] kubernetes/hack/make-rules/helpers/go2make/testdata/dir-with-gofiles/skipme/README [ ] kubernetes/hack/make-rules/helpers/go2make/testdata/dir-with-gofiles/skipme/subsubdir/README [ ] kubernetes/hack/make-rules/helpers/go2make/testdata/dir-with-gofiles/skipme/README [ ] kubernetes/hack/make-rules/helpers/go2make/testdata/dir-with-gofiles/README [ ] kubernetes/hack/make-rules/helpers/go2make/testdata/dir-with-gofiles/subdir/README [ ] kubernetes/hack/make-rules/helpers/go2make/testdata/dir-with-gofiles/README [ ] kubernetes/hack/make-rules/helpers/go2make/testdata/dir-with-gofiles/testdata/README [ ] kubernetes/hack/make-rules/helpers/go2make/testdata/dir-with-gofiles/testdata/subsubdir/README [ ] kubernetes/hack/make-rules/helpers/go2make/testdata/dir-with-gofiles/testdata/README [ ] kubernetes/hack/make-rules/helpers/go2make/testdata/dir-with-gofiles/README [ ] kubernetes/hack/make-rules/helpers/go2make/testdata/dir-with-gofiles/underscore/README [ ] kubernetes/hack/make-rules/helpers/go2make/testdata/dir-with-gofiles/underscore/subsubdir/README [ ] kubernetes/hack/make-rules/helpers/go2make/testdata/dir-with-gofiles/underscore/README [ ]

kubernetes/hack/make-rules/helpers/go2make/testdata/dir-with-gofiles/README [ ] kubernetes/hack/make-rules/helpers/go2make/testdata/dir-without-gofiles/README [ ] kubernetes/hack/make-rules/Makefile.manifest [ ] kubernetes/hack/make-rules/make- [ ] kubernetes/hack/make-rules/test- [ ] kubernetes/hack/make-rules/test-e2e- [ ] kubernetes/hack/make-rules/test-e2e- [x] kubernetes/hack/make-rules/test- [ ] kubernetes/hack/make- [ ] kubernetes/hack/make- [ ] kubernetes/hack/make- [ ] kubernetes/hack/make- [ ] kubernetes/hack/OWNERS [x] kubernetes/hack/pin- [x] kubernetes/hack/print-workspace- [x] kubernetes/hack/run-in- [ ] kubernetes/hack/testdata/auth- [x] [x] [x] kubernetes/hack/testdata/CRD/foo-- [x] kubernetes/hack/testdata/CRD/foo-deleted- [x] kubernetes/hack/testdata/CRD/foo-updated- [x] [x] kubernetes/hack/testdata/CRD/multi-crd-list-- [x] kubernetes/hack/testdata/CRD/multi-crd-list-deleted- [x] kubernetes/hack/testdata/CRD/multi-crd-list-updated- [x] kubernetes/hack/testdata/CRD/multi-crd- [x] [x] [x] kubernetes/hack/testdata/deployment-label- [x] kubernetes/hack/testdata/deployment-label- [x] kubernetes/hack/testdata/deployment-label- [x] kubernetes/hack/testdata/deployment-multicontainer- [x] kubernetes/hack/testdata/deployment- [x] kubernetes/hack/testdata/deployment- [x] kubernetes/hack/testdata/deployment- [x] kubernetes/hack/testdata/deployment-with- [x] [x] [x] [x] [x] [x] kubernetes/hack/testdata/filter/pod-apply- [x] kubernetes/hack/testdata/filter/pod-dont- [x]

kubernetes/hack/testdata/frontend- [x] kubernetes/hack/testdata/frontend- [x] kubernetes/hack/testdata/invalid-rc-with-empty- [x] kubernetes/hack/testdata/kubernetes- [x] [x] [x] [x] [x] [ ] kubernetes/hack/testdata/multi-resource- [ ] kubernetes/hack/testdata/multi-resource-json- [ ] kubernetes/hack/testdata/multi-resource- [ ] kubernetes/hack/testdata/multi-resource-list- [ ] kubernetes/hack/testdata/multi-resource- [ ] kubernetes/hack/testdata/multi-resource-rclist- [ ] kubernetes/hack/testdata/multi-resource- [ ] kubernetes/hack/testdata/multi-resource-svclist- [x] kubernetes/hack/testdata/multi-resource-yaml- [x] kubernetes/hack/testdata/multi-resource- [x] kubernetes/hack/testdata/null-propagation/deployment- [x] kubernetes/hack/testdata/null-propagation/deployment- [x] kubernetes/hack/testdata/OWNERS [x] kubernetes/hack/testdata/pod- [x] kubernetes/hack/testdata/pod- [x] kubernetes/hack/testdata/pod-with-api- [x] kubernetes/hack/testdata/pod-with-large- [x] kubernetes/hack/testdata/pod-with- [x] [x] [x] [x] [x] kubernetes/hack/testdata/prune- [x] kubernetes/hack/testdata/prune- [x] kubernetes/hack/testdata/recursive/deployment/deployment/nginx- [x] [x] [x] [x] [x] kubernetes/hack/testdata/recursive/pod/pod/busybox- [x] [x] [x] kubernetes/hack/testdata/recursive/pod- [x] kubernetes/hack/testdata/recursive/pod-modify/pod/busybox- [x] kubernetes/hack/testdata/recursive/pod- [x] kubernetes/hack/testdata/recursive/pod- [x] [x] kubernetes/hack/testdata/recursive/rc/rc/busybox- [x] [x] [x] kubernetes/hack/testdata/redis-slave- [x]

kubernetes/hack/testdata/retainKeys/deployment/deployment- [x] kubernetes/hack/testdata/retainKeys/deployment/deployment- [x] kubernetes/hack/testdata/rollingupdate-daemonset- [x] kubernetes/hack/testdata/rollingupdate- [x] kubernetes/hack/testdata/rollingupdate-statefulset- [x] kubernetes/hack/testdata/rollingupdate- [x] kubernetes/hack/testdata/scale-deploy-1.yaml [x] kubernetes/hack/testdata/scale-deploy-2.yaml [x] kubernetes/hack/testdata/scale-deploy-3.yaml [x] [x] kubernetes/hack/testdata/service- [x] kubernetes/hack/testdata/service- [x] kubernetes/hack/testdata/sorted-pods/sorted- [x] kubernetes/hack/testdata/sorted-pods/sorted- [x] kubernetes/hack/testdata/sorted-pods/sorted- [ ] [ ] [x] kubernetes/hack/test- [ ] kubernetes/hack/test- [x] kubernetes/hack/update- [ ] kubernetes/hack/update- [x] kubernetes/hack/update- [ ] kubernetes/hack/update-generated-api-compatibility- [x] kubernetes/hack/update-generated-device-plugin- [ ] kubernetes/hack/update-generated-device- [ ] kubernetes/hack/update-generated- [ ] kubernetes/hack/update-generated-kms- [ ] kubernetes/hack/update-generated- [x] kubernetes/hack/update-generated-kubelet-plugin-registration- [ ] kubernetes/hack/update-generated-kubelet-plugin- [x] kubernetes/hack/update-generated-pod-resources- [ ] kubernetes/hack/update-generated-pod- [ ] kubernetes/hack/update-generated-protobuf- [ ] kubernetes/hack/update-generated- [ ] kubernetes/hack/update-generated-runtime- [ ] kubernetes/hack/update-generated- [ ] kubernetes/hack/update-generated-swagger- [x] kubernetes/hack/update- [x] kubernetes/hack/update-import- [ ] kubernetes/hack/update-openapi- [x] kubernetes/hack/update- [ ] kubernetes/hack/update-vendor- [ ] kubernetes/hack/update- [ ] kubernetes/hack/update-workspace- [x] kubernetes/hack/verify-

[x] kubernetes/hack/verify-api- [x] kubernetes/hack/verify- [x] kubernetes/hack/verify- [x] kubernetes/hack/verify-cli- [x] kubernetes/hack/verify- [x] kubernetes/hack/verify-conformance- [x] kubernetes/hack/verify- [x] kubernetes/hack/verify-external-dependencies- [ ] kubernetes/hack/verify-flags/excluded- [x] kubernetes/hack/verify-flags/OWNERS [x] kubernetes/hack/OWNERS [ ] kubernetes/hack/verify-flags- [x] kubernetes/hack/verify-generated-device- [x] kubernetes/hack/verify-generated- [x] kubernetes/hack/verify-generated-files- [x] kubernetes/hack/verify-generated- [x] kubernetes/hack/verify-generated- [x] kubernetes/hack/verify-generated-kubelet-plugin- [x] kubernetes/hack/verify-generated-pod- [x] kubernetes/hack/verify-generated- [x] kubernetes/hack/verify-generated- [x] kubernetes/hack/verify-generated-swagger- [x] kubernetes/hack/verify- [x] kubernetes/hack/verify- [x] kubernetes/hack/verify- [x] kubernetes/hack/verify-import- [x] kubernetes/hack/verify-import- [x] kubernetes/hack/verify- [x] kubernetes/hack/verify- [x] kubernetes/hack/verify-no-vendor- [x] kubernetes/hack/verify-openapi- [x] kubernetes/hack/verify-pkg- [x] kubernetes/hack/verify-publishing- [x] kubernetes/hack/verify-readonly- [x] kubernetes/hack/verify- [x] kubernetes/hack/verify- [x] kubernetes/hack/verify-staging-meta- [x] kubernetes/hack/verify- [x] kubernetes/hack/verify-test- [x] kubernetes/hack/verify-test- [x] kubernetes/hack/verify-test- [x] kubernetes/hack/verify-typecheck- [x] kubernetes/hack/verify- [x] kubernetes/hack/verify-vendor- [x]

kubernetes/hack/verify- [x]
i will pick up , i will do it step by step.
I agree with your idea, and thanks the checkbox to check progress. there are many scripts, so let's work together!
/assign /assign
I will pick up hack/verify-
I will pick up hack/verify-
I will pick up hack/verify- (all of verify scripts). But there are large number of scripts(over 40), I want to split it into two or three PRs.
I will pick up hack/update-
I will pick up these scripts. kubernetes/hack/benchmark- kubernetes/hack/build- kubernetes/hack/build- kubernetes/hack/dev-build-and- kubernetes/hack/dev-build-and- kubernetes/hack/dev-push- kubernetes/hack/e2e-node- kubernetes/hack/generate-
I will pick up these scripts get- ginkgo- grab- install- lint- list-feature- local-up- pin- print-workspace- run-in- test- test-
How about this now? /help Marking this tag so that others can easily access it.
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : How about this now? /help Marking this tag so that others can easily access it. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I think we've finished adding simple explanations to the almost scripts. It seems that several scripts of update- does not complete yet. I think we may able to close after that work.
I will pick up these scripts:
/assign


Question: svcName := fmt.Sprintf("%s.%s.svc.%s", svc.Name, svc.Namespace, framework.TestContext.ClusterDNSDomain) // Service must resolve to IP cmd := fmt.Sprintf("nslookup %s", svcName) <del> _, stderr, err := framework.RunHostCmdWithFullOutput(pod.Namespace, pod.Name, cmd) // NOTE(claudiub): nslookup may return 0 on Windows, even though the DNS name was not found. In this case, // we can check stderr for the error. if err != nil || (framework.NodeOSDistroIs("windows") && strings.Contains(stderr, fmt.Sprintf("can't find %s", svcName))) { return fmt.Errorf("ExternalName service %q must resolve to IP", pod.Namespace+"/"+pod.Name) } return nil </del> <ins> return wait.PollImmediate(framework.Poll, ServiceReachabilityShortPollTimeout, func() (done bool, err error) { _, stderr, err := framework.RunHostCmdWithFullOutput(pod.Namespace, pod.Name, cmd) // NOTE(claudiub): nslookup may return 0 on Windows, even though the DNS name was not found. In this case, // we can check stderr for the error. if err != nil || (framework.NodeOSDistroIs("windows") && strings.Contains(stderr, fmt.Sprintf("can't find %s", svcName))) { framework.Logf("ExternalName service %q failed to resolve to IP", pod.Namespace+"/"+pod.Name) return false, nil } return true, nil }) </ins> } // CheckServiceReachability ensures that request are served by the services. Only supports Services with type ClusterIP, NodePort and ExternalName.

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: - Which test(s) are flaking: Testgrid link: https://k8s- https://k8s- Reason for failure: Anything else we need to know: Having some trouble with finding this in Triage, but here are two example failures: /sig network <!-- Please see the deflaking doc () for more guidance! --
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign I'll take time to see it.
Any updates on this? Should we triage-accept it?


Question: // If we need to (re-)create the pod sandbox, everything will need to be // killed and recreated, and init containers should be purged. if createPodSandbox { <del> if !shouldRestartOnFailure(pod) && attempt != 0 { </del> <ins> if !shouldRestartOnFailure(pod) && attempt != 0 && len(podStatus.ContainerStatuses) != 0 { </ins> // Should not restart the pod, just return. // we should not create a sandbox for a pod if it is already done. // if all containers are done and should not be started, there is no need to create a new sandbox. // this stops confusing logs on pods whose containers all have exit codes, but we recreate a sandbox before terminating it. <ins> // // If ContainerStatuses is empty, we assume that we've never // successfully created any containers. In this case, we should // retry creating the sandbox. </ins> changes.CreateSandbox = false return changes }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: If a pod's restart policy is "Never", and kubelet failed to create the PodSandbox (for whatever reason) once, the pod will be stuck at "ContainerCreating" forever. What you expected to happen: Since the user/application containers were never created, kubelet should retry the PodSandbox creation. The fact that the pod remains in a ContainerCreating state while kubelet explicitly does nothing to resolve the situation also confuses the user. How to reproduce it (as minimally and precisely as possible): I don't have an easy way to trigger a podsandbox creation failure off the top of my head, but this should be doable by messing with the CNI binaries/configurations. Anything else we need to know?: Relevant code snippet: Environment: Kubernetes version (use ): Only checked the master branch (1.16), but all supported release branches should have the issue Cloud provider or hardware configuration: not relevant OS (e.g: ): not relevant Kernel (e.g. ): not relevant Install tools: Network plugin and version (if this is a network-related bug): not relevant; though the initial PodSandbox creation failure could be caused by networking issues. Others:
thanks for posting! I'm looking at the code, and it seems like one fix could be checking if this pod has an associated sandbox - if it does, we want to respect the current behavior. If it doesn't, we want to attempt to create the sandbox. I was thinking something similar to: Thoughts? Is there a simplex fix?
Hi! Thank you for filing this - it matches the behaviour we've been seeing. Am I correct that this is a regression from (), and so only affects Kubernetes =1.13 ?
Your solution may not work for the one specific failure I checked, where the PodSandbox container was created, but call to the CNI plugin failed (for some reason; could be caused by a race condition). This left us a not-ready PodSandbox with no IP. Probably have to actually determine whether the user containers were created before. Need to check the code closer though.
Thanks. That's probably correct. I saw this in happening in 1.13
Gotcha - thanks! Taking a look at your pr now :)


Question: FakeMemoryCapacity     = 4026531840 FakeKernelVersion      = "3.16.0-0.bpo.4-amd64" FakeContainerOsVersion = "Debian GNU/Linux 7 (wheezy)" <del> FakeDockerVersion      = "1.5.0" </del> <ins> FakeDockerVersion      = "1.13.1" </ins> ) var _ cadvisor.Interface = new(Fake)

The current node e2e tests include images using Docker 1.11.x to , with the pending PR to test 18.06 () We should update the images to drop older docker versions. I think we can start by getting rid of 1.11 and 1.12, so that we still have three minor versions tested: , , and 1.13.x /cc (for cos and ubuntu)
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/assign Yang, could you take a look? Thanks


Question: vSphereInstance = nodeNameToVMName(nodeName) } <del> nodeExist, err := vs.NodeExists(vs.client, nodeName) if err != nil { glog.Errorf("Failed to check whether node exist. err: %s.", err) return err } if !nodeExist { glog.Warningf( "Node %q does not exist. DetachDisk will assume vmdk %q is not attached to it.", nodeName, volPath) return nil } </del> vm, vmDevices, _, dc, err := getVirtualMachineDevices(ctx, vs.cfg, vs.client, vSphereInstance) if err != nil { return err

When a vm is reported as no longer present in cloud provider and is deleted by node controller, there are no attempts to detach respective volumes. For example, if a VM is powered off or paused, and pods are migrated to other nodes. In the case of vSphere, the VM cannot be started again because the VM still holds mount points to volumes that are now mounted to other VMs. In order to re-join this node again, you will have to manually detach these volumes from the powered off vm before starting it.
Once the node is powered off, the controller-manager doesn't send a detachVolume request to vSphere cloud provider. The controller-manager couldn't send the detachVolume request because it checks until the volume is unmounted on the node for a particular timeout. You can see the code over here - Since the volume is never unmounted, the controller manager skips the detachVolume request and issues the attachVolume() on another successfully. I think the controller-manager needs send a detachVolume() request even if the volume is mounted when the node is powered OFF. This is serious problem in our case (vSphere) as volume is now attached on another node and the node which is powered OFF cannot be powered ON again, as the node tries to acquire a lock on the volume to mount it on this node during the power ON. The acquire lock fails as it is already attached/mounted on other node. Also there are a few workarounds for this. When the Node is powered OFF, the cloud provider can detach the volume in NodeExists() - This method is called in DisksAreAttached() method. However i don't think this is a viable solution. : Can you provide your inputs on this. CCing
CC If a node goes offline (for any reason including being powered off) the attach/detach controller will eventually detach the volume even if the node never reports it as safely unmounted. Once a node becomes inaccessible, the node controller waits 5 min 40 seconds, before deleting all the pods that were scheduled to that node. When a pod referencing an attachable volume on that node is deleted, the attach/detach controller tries to detach the volume, but if the node has not reported the volume as safely unmounted (which would be the case if the node was powered off), the attach/detach controller waits 6 minutes before triggering detach anyway.

Once the volume is detached, any attempts to attach it to some other node will finally succeed. This means that it can take up to 15 minutes for this situation to rectify itself. Can you verify what amount of time you waited?
So do you mean that the volume can be attached to another node only after the volume is detached on the node which is powered OFF. In my case, i have waited for significant amount of time. I am sure it is more than 15 mins. And I see the volume succesfully attached to the new node but still not detached from the powered OFF node. In this case I cannot restart the node as explained above. Only if I delete the attached nodes manually I can power ON the node again.
If the storage system supports multi node attach for the access mode, then the attach to the new node should succeed immediately. If the storage systems supports only single node attach for the access mode then the attach calls to the new node fail fail until it is detached from the first node (this is the case with GCE PDs). If you are never seeing the volume detach from the original (powered off) node (even after 15 minutes), that sounds like a bug. AFAIK, this case is working fine in the GCE PD case ( tested this recently). If you have a consistent repro, to help debug, could you grab and share your controller-manager logs showing the original attach, subsequent attach, and lack of detach from the first node.
Sure. I will attach the logs. Between in our case we only support single node attach for a volume.
Between I am testing it on v1.4.7. Is this a change which is recently made to kubernetes code base to address this issue and not present in v1.4.7?
Please verify the logs from controller-manager. I see there is detach volume request being issued. I almost waited for 45 mins, but I didn't send any request attempting to detach volume as in - As in , since it failed to update node status, it couldn't make any further request for detachVolume(). Also, from the logs I see it is the kubernetes control plane which decides whether the storage plane is multi node attach or single node attach by attempting to send an attachVolume request on the new node.

And its not the cloud provider which says it to kubernetes control plane what type of storage plane it is. Correct me if I am wrong on this. : Can you please verify this.
That log is level 5. If your cluster is set up with log level 4, you will not see that log. When a node is down, the following events will happen with code release 4 or before. Once a node becomes inaccessible, the node controller waits for about 5 mins, and then deletes all the pods that were scheduled to that node. Once pods are deleted, Volume manager's reconciler will try to detach the volumes that belong to the pods. However, it will fail to issue to request first because it thinks the volumes are still mounted to the nodes according to the node status. (node status can be updated by the kubelet but since the node is down, the status is not updated). the reconciler fails to issue the detach request for a periods time, it times out (currently it is set to 6 minutes) and will issue detach request anyway and it should find that the volume is already detached since node is already terminated. So it will probably take more than 10 mins for reconciler marks the volume is detached to the node. We recently have a PR that change the above behavior during node termination because this PR adds a periodic check that whether the volumes are still attached. When node is terminated, the reconciler will very quickly (in a few seconds) find out the node is terminated and volume is detached and then mark this change in the actual state. {"log":"W1223 00:27:26. 1 ] Node "node2" does not exist. DisksAreAttached will assume vmdk [[vsanDatastore1] kubevols/redis-master] are not attached to it.n","stream":"stderr","time":"2016-12-23T00:27:26.429562004Z"} {"log":"I1223 00:27:26.

1 ] VolumesAreAttached: check volume "[vsanDatastore1] kubevols/redis-master" (specName: "redis-master-data") is no longer attachedn","stream":"stderr","time":"2016-12-23T00:27:26.44262004Z"} Please let me know if you have any questions about this part. Can you please verify this.
: I am using v1.4.7. Can you let me know if this holds true for v1.4.7? I don't see any behavior as mentioned by in the logs. Also you can see there is no detach call made on the node. As per this - if fails to update the node status after various attempts and hence detach call doesn't pass through. Also, from the logs I see it is the kubernetes control plane which decides whether the storage plane is multi node attach or single node attach by attempting to send an attachVolume request on the new node. And its not the cloud provider which says it to kubernetes control plane what type of storage plane it is. Also can you comment on the above question?
+1 on this. Same goes for k8s v1.5.2. Detach never happens when powering off a node with PVs attached. Disks has to be manually detached before being able to turn on the node again.
The reason detach never happens is as follows, is terminated controller periodically check whether volumes are still attached by sending requests to cloud provider. The function DisksAreAttached will check whether the node still exist. If not, it will assume the volume is no longer attached. If node does exist, check the information returned by the cloud provider to see whether volume is still attached. In this step, I believe it returns as volume is detached volume is already detached. So explicit detach will not be issued since step 2 and 3 already marks it as detached. I will check the function to see if there is potential problem in checking volume detached or not.
Cheers then I understand. I will also have a look.
is a workaround but ideally k8s should call detach for every attach that was issued, if the disk is already detached it should be a no-op. The ideal behavior should be that for each attach there will be a detach (which could be redundant).

The assumption in the current model seems to be that as long as the volume provider does not consider the disk attached the detach does not need to be issued which ignores the configuration for the VM. If the VM still considers the disk attached it should be reconfigured as well. The pod in this case is fine after being rescheduled to a new node but the VM is left with an invalid configuration.
Ok! The workaround seems legit for the time being. Thanks!


Question: return false, err } <del> noxuWatch, err := resourceClient.Watch(metav1.ListOptions{ResourceVersion: createdInstance.GetResourceVersion()}) if err != nil { return false, err } defer noxuWatch.Stop() select { case watchEvent := <-noxuWatch.ResultChan(): if watch.Error == watchEvent.Type { return false, nil </del> <ins> // Wait for all versions of watch cache to be primed and also make sure we consumed the DELETE event for all // versions so that any new watch with ResourceVersion=0 does not get those events. This is source of some flaky tests. // When a client creates a watch with resourceVersion=0, it will get an ADD event for any existing objects // but because they specified resourceVersion=0, there is no starting point in the cache buffer to return existing events // from, thus the server will return anything from current head of the cache to the end. By accessing the delete // events for all versions here, we make sure that the head of the cache is passed those events and they will not being // delivered to any future watch with resourceVersion=0. for _, v := range versions { noxuWatch, err := resourceClientForVersion(crd, dynamicClientSet, ns, v).Watch( metav1.ListOptions{ResourceVersion: createdInstance.GetResourceVersion()}) if err != nil { return false, err </ins> } <del> if watch.Deleted != watchEvent.Type { return false, fmt.Errorf("expected DELETE, but got %#v", watchEvent) </del> <ins> defer noxuWatch.Stop() select { case watchEvent := <-noxuWatch.ResultChan(): if watch.Error == watchEvent.Type { return false, nil } if watch.Deleted != watchEvent.Type { return false, fmt.Errorf("expected DELETE, but got %#v", watchEvent) } case <-time.After(5 * time.Second): return false, fmt.Errorf("gave up waiting for watch event") </ins> } <del> return true, nil case <-time.After(5 * time.Second): return false, fmt.Errorf("gave up waiting for watch event") </del> } <ins> return true, nil </ins> } func DeleteCustomResourceDefinition(crd *apiextensionsv1beta1.CustomResourceDefinition, apiExtensionsClient clientset.Interface) error {

both in PR and CI runs see /assign
/assign
Interesting fact: is not flaky, only . The test codepath for these two are almost identical. I wonder if this is a real bug not a flaky test. still investigating.
still seeing flakes like https://k8s- the loop that ensures the watch cache is primed goes version-by-version, and creates and observes the delete event for that version. it does not ensure the delete event is drained from the watch caches for the other versions.
a better approach would be probably be to:
why the poll? Because the watch is not reliable? Otherwise, we have that structure now.
Moves the version loop inside the poll Today we loop over versions and poll per version, which can leave unconsumed delete events in prior versions
Cannot follow. Now we have the following without a poll:
At minimum, I would move the version loop inside the poll so that when we exit we know the deletion events from isWatchCachePrimed are consumed by the watch caches for all versions. If we wanted, we could move the create/delete up out of the isWatchCachePrimed function or pass versions into it to avoid creating/deleting an object per version
Ok, so we were talking about different code. good. To understand: we do a call per version, each creating and deleting an object, and waiting for the delete event with a watch call. a following watch will not see the events of the calls of itself (because they use the same versioned watch cache) and of any version in front of it in the version slice (is this true?), but it might see events of calls of versions later in the version slice. So with your outer poll and inner version loop we would make sure that we drain every version's cache for following rv=0 watch requests. Sounds plausible.
Where do we have the official watch axioms?
I think I know what is the problem here. We make sure the watch is primed for . then we move on to and make sure the watch is primed by creating another object in . The watch cache is drained for but the watch cache for is not and that is the problem here. will send a fix.
yep, same thing I described above. The solution looks good.
I didn't see your comment when I commented. Your description is accurate.


Question: ) var ( <ins> NodeName = prometheus.NewGaugeVec( prometheus.GaugeOpts{ Subsystem: KubeletSubsystem, Name:      NodeNameKey, Help:      "The node's name. The count is always 1.", }, []string{NodeLabelKey}, ) </ins> ContainersPerPodCount = prometheus.NewSummary( prometheus.SummaryOpts{ Subsystem: KubeletSubsystem,

What would you like to be : It would be useful if the kubelet exported a metric that contained its own name as a label, for instance: . This name should be consistent with the output of . Why is this needed: This would allow us to join the kubelet metrics against metrics exported by kube-state-metrics. An example of this can be seen in We've arranged in our cluster for the labels to be set to the node's name, but this is controlled by Prometheus and depends on your scrape & relabelling config. As such, in general we can't rely on this to be the case - for instance, with the prometheus-operator the label is a IP:Port combo. /cc
/sig instrumentation
This metric as it is right now doesn’t conform to the instrumentation guidelines, so yes we should changes this as part of the metrics overhaul targeted for 1.14.
you mean to add a label in kubelet metrics ?
precisely.
/assign


Question: package clientbuilder import ( <ins> "k8s.io/client-go/discovery" </ins> clientset "k8s.io/client-go/kubernetes" restclient "k8s.io/client-go/rest" "k8s.io/klog/v2"

Ran kube-controller-manager with and observed this in the log every 30 seconds: This is also happening in 1.20, 1.19, and 1.18 The GC and quota clients look up discovery information every 30 seconds. They have a QPS of 20 requests per second and a burst of 30 requests. Default configuration of kube-controller-manager runs without constant throttling warnings. /sig api-machinery
/assign /triage accepted


Question: "../examples/glusterfs": { "glusterfs-pod":       &api.Pod{}, "glusterfs-endpoints": &api.Endpoints{}, <ins> "glusterfs-service":   &api.Service{}, </ins> }, "../docs/user-guide/liveness": { "exec-liveness": &api.Pod{},

I create a glusterfs endpoint, but after I restart kube-apiserver, the endpoint is missing. Does Kubernetes persistence endpoints?
Do you mean and Endpoints object in the API or something else? , Bin Wang wrote:
Here is what I do: Then use I can see this line: But after I restart kube-apiserver, this endpoint is lost.
Do you have a Service named ? Otherwise it should persist, just the same as any other API object , Bin Wang wrote:
No. This endpoints is just used for persistent volumes, which is used by other pods. I followed . Is it necessary to define a service?
Can one of you guys try to repro?
What does that mean exactly? What kind of cluster are you running? Does anything persist and endure between restarts of the API server? When you restart the API server (why?), are you also restarting etcd or other services?
The cluster just have one master. The pods, secrets, controllers and services are recovered after the restart, but the endpoints are not. About why I restart the API server, well, it may fail (or something else). You cannot expect the api-server runs and never restart. After I restart it, I expect all the things are recovered. In my case, I just restart api-server through /etc/init.d/kube-apiserver (kill the process manually should do the same thing). Etcd is always running and the data in it is not lost.
Any update on this? I cannot use kubernetes in production if this problem is not solved. Or do you have a work around, like define a service with the glusterfs endpoints?
I'm running into this too on GKE so I'm not sure if it's the api-server restarting, but just now after a loadtest and intermittently a couple of other times, after my app tries to come back up it failes to mount from gluster due to the gluster endpoints having mysteriously disappeared.
Do you mean the whole object disappeared or that it exists but is empty?
In the original YAML "IP" should be "ip"
does not show the resource On Mar 27, 2017 8:16 PM, "Tim Hockin" <:
I am trying to repro, but no luck so far.

Just checked.. IP is does come up, and it works well enough as the pods are able to mount the gluster volumes, but every once in a while I find that they're gone. On Mar 27, 2017 8:18 PM, "Tim Hockin" <:
I haven't been able to either, it happens every once in a while. I think maybe they're deleted before I notice and I only notice because a pod is fails and on restart it fails to mount it's volume On Mar 27, 2017 8:21 PM, "Tim Hockin" <:
Just checked a cluster I created an hour ago, and yup all the pods that had been created are running smoothly but the gluster endpoints resource is gone , Jonathan <: — You are receiving this because you commented. Reply to this email directly, view it on GitHub , or mute the thread
Interesting. It appears this is intentional. If you create a service with clusterIP: None and an empty selector, this will not be activated. It is trying to prevent submarine problems, such as creating an Endpoints then later creating a Service and conflicting. , Jonathan Kosgei <:
So the fix is to create a corresponding service for the endpoints?
Yes. Endpoints are usually a derived value, and in hindsight they should probably have been a sub-resource of Service. On Mar 28, 2017 7:20 AM, "Jonathan Kosgei" <:
In case you're wondering (like I was) how to attach a service to the EP the was updated to include an example service for it.
I'm having this issue again in a very different setup, basically I'm using the Nginx Ingress controller and it uses endpoints and these endpoints keep disappearing intermittently breaking all ingress to the cluster. Opened an issue at the nginx-ingress repo Saving the endpoints with a and recreating them when resolution failure occurs fixes it immediately.
this will make kubelet notready if i create a Service named glusterfs-cluster, after reboot endpoint is not lost. but the kubelet will be notready , something wrong? hope response tks.
The correct explanation should be When the cluster starts Endpoint-Controller Will check all Endpoint list Then put Endpoint name in workqueue, worker get value from workqueue, Found Endpoint no corresponding Service,Just delete Endpoint.

yes, the manually created EP resource will not be reconciled by the controller. This is the cause. Usually we do not create EP, this kind of secondary resource directly. It is created for the corresponding service resource.


Question: }, } <del> for _, test := range tests { </del> <ins> for ix, test := range tests { </ins> glog.V(4).Infof("starting test %q", test.name) // Initialize the controller

Not many details in the logs: FAIL 2.721s
I ran tests a few times against recent master but couldn't see this, could it be an edgy case?
i'll look at it
should help a lot, still there will be a short window where the test could get wrong. Patch will follow when is merged.
Today I saw a very similar error to the one reported in , but seeing as that was closed as a dupe of this, adding details here:
this looks like something new. Let's keep this issue open for this unit test flake: I filled for integration test flake you reported.


Question: podutil.VisitPodSecretNames(pod, func(name string) bool { hasSecrets = true return false <del> }) </del> <ins> }, podutil.AllContainers) </ins> if hasSecrets { return admission.NewForbidden(a, fmt.Errorf("a mirror pod may not reference secrets")) }

Fully described in , and will visit these fields for all containers, regardless of whether they're hidden behind a feature gate (e.g. ). This is usually correct, these methods are often used to drop fields and in this case should visit all containers that are persisted regardless of whether they're disabled, but the callers of these methods should signal that explicitly rather than relying on the implicit behavior. /cc /priority important-longterm /kind bug /sig node
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : Fully described in , and will visit these fields for all containers, regardless of whether they're hidden behind a feature gate (e.g. ). /priority important/long-term /kind bug /sig node Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
are you interested in taking this one, too?
VisitPodSecretNames and VisitPodConfigmapNames already specify AllContainers for VisitContainers Can you let me know what else needs to be done?
Callers should specify whether they want to visit all containers or not. There are valid reasons for each, so we shouldn't assume the desired behavior, even if 100% of the current users want to visit all containers. Jordan's original comment in has some examples of current uses. I don't think this needs to be as configurable as , so my naive first attempt to implement this would probably just be a new to and


Question: - [Cluster Requirements](#cluster-requirements) - [Decide where you will store your data](#decide-where-you-will-store-your-data) - [Host Path](#host-path) <ins> - [SELinux](#selinux) </ins> - [GCE Persistent Disk](#gce-persistent-disk) - [Create the MySQL Password Secret](#create-the-mysql-password-secret) - [Deploy MySQL](#deploy-mysql)

Version: Running this example on RHEL 7.2 (assume same for centos), and using hostPath, I get a permissions error when the mysql container does a . If I set selinux to permissive on the host it works. If I manually run the container with it works. I was not able to get it to work when adding to the deployment. My fix was to locate the PVs in by changing the local- file, and doing . I this comment to local- If this issue is valid I will create a quick PR to doc this selinux solution.
See pr


Question: rbac.NewClusterBinding("system:kube-controller-manager").Users(user.KubeControllerManager).BindingOrDie(), rbac.NewClusterBinding("system:kube-dns").SAs("kube-system", "kube-dns").BindingOrDie(), rbac.NewClusterBinding("system:kube-scheduler").Users(user.KubeScheduler).BindingOrDie(), <ins> rbac.NewClusterBinding("system:aws-cloud-provider").SAs("kube-system", "aws-cloud-provider").BindingOrDie(), </ins> // This default binding of the system:node role to the system:nodes group is deprecated in 1.7 with the availability of the Node authorizer. // This leaves the binding, but with an empty set of subjects, so that tightening reconciliation can remove the subject.

Recently introduced fix of tainting the node if EBS volume on a node is stuck for too long in "attaching" state does not work on RBAC enabled clusters because of missing service account permissions. /sig storage /sig aws


Question: "//vendor/github.com/prometheus/client_golang/prometheus:go_default_library", "//vendor/github.com/rubiojr/go-vhd/vhd:go_default_library", "//vendor/k8s.io/api/core/v1:go_default_library", <ins> "//vendor/k8s.io/apimachinery/pkg/api/resource:go_default_library", </ins> "//vendor/k8s.io/apimachinery/pkg/types:go_default_library", "//vendor/k8s.io/apimachinery/pkg/util/errors:go_default_library", "//vendor/k8s.io/apimachinery/pkg/util/sets:go_default_library",

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: According to , some volumes(e.g. aws, nfs, ceph) has already supported PV resizing, need to add PV resize support for azure disk What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: PV resize doc is here: Update: For azure disk, there is blocking issue: it does not support disk resize on a Running VM. I will try to work on resizing an azure disk which is not in attached state. May dig into more details later. Environment: Kubernetes version (use ): v1.10.0 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: /sig azure /assign
Hi , is PV size grow feature still in alpha in master now? How do I open this feature, set in kubelet, kube-apiserer and kube-controller-manager config?
I have figured out that should be set in kubelet, kube-apiserer, kube-controller-manager, kube-scheduler config, and also need to enable in apiserver admission control. It works well.
should we go ahead and close this issue?
also file needs to be updated to allow resizing. This plugin provides additional checks for resizing.
For azure disk, there is blocking issue: it does not support disk resize on a Running VM. I would suggest work on first, could you review , it works well in my testing env.
Adding priority important-soon to this issue so that it can stay in 1.10. If that is incorrect, please update. priority important-soon
/priority important-soon
PR has been merged, can this issue be closed?
yup, looks like it's in. Thanks for flagging! /close
you can't close an active issue unless you authored it or you are assigned to it, Can only assign issues to org members and/or repo collaborators.. <detailsIn response to : yup, looks like it's in. Thanks for flagging! /close Instructions for interacting with me using PR comments are available .

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Awww shucks, looks like I don't have magic powers. please close this when you have a chance?
has been merged, while for azure disk work is just in investigation state mainly because there is blocking issue: it does not support disk resize on a Running VM.
could someone move this issue out of milestone v1.10, thanks.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Why was this closed? This is not fixed by that crappy PR. This is the kubernetes repo, and this refers to k8s apis being implemented correctly iyam. The PR that you closed this with does NOT support disk resize on a Running VM. Your words Please reopen this.
pls track this issue for online resize feature:


Question: } else { for i := 0; i < aLen; i++ { // Validate PV <del> if !reflect.DeepEqual(expectedBindings[i].pv, bindings[i].pv) { t.Errorf("binding.pv doesn't match [A-expected, B-got]: %s", diff.ObjectDiff(expectedBindings[i].pv, bindings[i].pv)) </del> <ins> if diff := cmp.Diff(expectedBindings[i].pv, bindings[i].pv); diff != "" { t.Errorf("binding.pv doesn't match (-want, +got):n%s", diff) </ins> } // Validate PVC <del> if !reflect.DeepEqual(expectedBindings[i].pvc, bindings[i].pvc) { t.Errorf("binding.pvc doesn't match [A-expected, B-got]: %s", diff.ObjectDiff(expectedBindings[i].pvc, bindings[i].pvc)) </del> <ins> if diff := cmp.Diff(expectedBindings[i].pvc, bindings[i].pvc); diff != "" { t.Errorf("binding.pvc doesn't match (-want, +got):n%s", diff) </ins> } } }

When scheduling pod, we encountered the following event , then the scheduler will backoff to reschedule again, but this add more waiting time. From the event message, we can see it is the PVC update conflict leads to the , so need to add retry logic on PVC updating.### Need to add retry logic on PVC updating, to avoid the backoff, to improve the scheduling efficiency. a PVC and then create Pod; scheduing; the PVC annotation or label before the Pod scheduled success; will encounter this issue. No response <details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage note that this is not a supported client/server skew: Server Version: version.Info{Major:"1", Minor:"18+", GitVersion:"v1.18.4-tke.23", GitCommit:"", GitTreeState:"clean", BuildDate:"2022-03-15T09:25:31Z", GoVersion:"go1.15.10", Compiler:"gc", Platform:"linux/amd64"}
/sig scheduling
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue with Mark this issue as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close not-planned [1]:
Closing this issue, marking it as "Not Planned". <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue with Mark this issue as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: // using local traffic health check or nodes health check. Attempt to delete // both to prevent leaking. hcNames = append(hcNames, loadBalancerName) <del> hcNames = append(hcNames, makeNodesHealthCheckName(clusterID)) </del> <ins> hcNames = append(hcNames, MakeNodesHealthCheckName(clusterID)) </ins> } errs := utilerrors.AggregateGoroutines( func() error { <del> fwName := makeFirewallName(loadBalancerName) </del> <ins> fwName := MakeFirewallName(loadBalancerName) </ins> err := ignoreNotFound(gce.DeleteFirewall(fwName)) if isForbidden(err) && gce.OnXPN() { glog.V(4).Infof("ensureExternalLoadBalancerDeleted(%v): do not have permission to delete firewall rule (on XPN). Raising event.", loadBalancerName)

A couple of gce-scale-performance runs failed due to failing to delete the network, which in turn was due to a couple of leaked firewall-rules (I manually deleted the network later to fix it): Ref: https://k8s- cc (assigning to you, feel free to reassign as apt)
I digged up a bit and seems like the firewall is created by the service-controller as part of , which should've been deleted in but wasn't. Feel free to close the bug if you think it's just a one-off failure.
Took a quick look, both and runs have these firewall rules initially but doesn't, which indicates something happen in the run. But oddly there is no build log, logs in artifacts are empty as well. Likely that run is badly broken so cluster got terminated forcibly in the middle, and resource are leaked. Will observe this suite a bit longer. /assign /unassign
I think it wasn't caused by this job. We're running this one () and on alternating days in the project. And they both create clusters (network, etc) with the same name. And I think of gce-scale-correctness leaked the firewall (and hence the network) as it runs some correctness tests around load-balancer. That likely caused this job to fail.
Ref The resource leak was likely caused by test timeout --- GCE operations took too long to finish and the cluster was terminated ahead. Because heathcheck and firewall cleanup in gce controller are coupled with targetpool deletion, deleting cluster while targetpool deletion is pending will leak the resources.
Fyi - this affected gce-scale-performance again (ref: run 35 of https://k8s-): Is there a way to make sure they get deleted even in case of a timeout? I have a dirty hack in my mind - of deleting those as part of kube-down. Maybe you have some better suggestions?
Consider the frequency this happens, I think it makes sense to implement some cleanup logic within the e2e test. Will give it a try.
Turned out we already have such cleanup logic, it is just we use an incorrect firewall name and forget to cleanup node health check resources as well. Sent to fix.
Thanks for fixing!
Sure thing, thanks for reporting!
Reopening as it seems like the issue is not fixed yet.

For e.g run-16 of https://k8s- failed due to this: This is also affecting our job, both of them being release-blocking jobs. Could you PTAL?
By any chance we could make and use different names so that it will be easier to target the leaking point? Also, in that way one leaking job will not affect the other. Or maybe we could also make different runs use different network names? The scalability job is expensive so it might be great if newer runs wouldn't be affected by the leaked resources from previous run. I haven't figured out how exactly and when the resource are leaked, but given that the operation cycle of service controller and e2e test process proceed in an asynchronous manner (plus always fail ESIPP tests ), there might be chances we will leak resource even after we put cleanup logic in e2e test. Will take a deeper look later. Also what you suggested before --- deleting those as part of kube-down --- may also be a viable solution.
I thought about this option, but it's better for them to share the same name for network and other resources (instance-prefix, etc) because that way we ensure that if one of them leaks some resources, the other will delete them during it's startup. Else we'll run out of quota in the scale-testing project (for e.g wrt routes or VMs) by trying to start another 5k-node cluster while the previous one still exists. I went to pantheon and deleted the network and that went through successfully (though fails). Wonder if the delete command in kube-down can be modified to delete all sub-resources of the network (like firewalls, etc) before deleting it?
Sounds reasonable to me. We may need yet another env to trigger it? As deleting everything correlated to that network sounds a bit dangerous.
Well, took another look, seems like the actual thing that prevented us from deleting the network was mostly firewall rules, and deleting those sounds safe enough if we are going to delete the network anyway. Will try to make a PR for this.
Thanks for fixing the issue with firewalls. However seems like the latest run of gce-scale-correctness still leaked the network (but doesn't seem due to firewall this time): ref: https://k8s-
I thought we're deleting the subnets in this case too when we're using custom subnets mode.

Yep, we are supposed to. I think the scale CI hit the same issue as described. I've sent a PR for fixing it
For the time-being I manually deleted the subnets and then the network to unblock the next scale job run. But it would be good to have this fixed soon. cc
Closing this issue as we're now deleting all remaining firewall rules in the network before deleting the network itself in kube-down. For other networking resources in general, we have issue .


Question: }) } } <del> // TestDropDisabledFields tests if the drop functionality is working fine or not with feature gate switch func TestDropDisabledFields(t *testing.T) { testCases := []struct { name               string enableXValidations bool crd                *apiextensions.CustomResourceDefinition oldCRD             *apiextensions.CustomResourceDefinition expectedCRD        *apiextensions.CustomResourceDefinition }{ { name:               "For creation, FG disabled, no XValidations, no field drop", enableXValidations: false, crd:                &apiextensions.CustomResourceDefinition{}, oldCRD:             nil, expectedCRD:        &apiextensions.CustomResourceDefinition{}, }, { name:               "For creation, FG disabled, empty XValidations, no field drop", enableXValidations: false, crd: &apiextensions.CustomResourceDefinition{ ObjectMeta: metav1.ObjectMeta{Name: "foos.sigs.k8s.io", Annotations: map[string]string{v1beta1.KubeAPIApprovedAnnotation: "valid"}, ResourceVersion: "1"}, Spec: apiextensions.CustomResourceDefinitionSpec{ Validation: &apiextensions.CustomResourceValidation{}, }, }, oldCRD: nil, expectedCRD: &apiextensions.CustomResourceDefinition{ ObjectMeta: metav1.ObjectMeta{Name: "foos.sigs.k8s.io", Annotations: map[string]string{v1beta1.KubeAPIApprovedAnnotation: "valid"}, ResourceVersion: "1"}, Spec: apiextensions.CustomResourceDefinitionSpec{ Validation: &apiextensions.CustomResourceValidation{}, }, }, }, { name:               "For creation, FG disabled, set XValidations, drop XValidations", enableXValidations: false, crd: &apiextensions.CustomResourceDefinition{ ObjectMeta: metav1.ObjectMeta{Name: "foos.sigs.k8s.io", Annotations: map[string]string{v1beta1.KubeAPIApprovedAnnotation: "valid"}, ResourceVersion: "1"}, Spec: apiextensions.CustomResourceDefinitionSpec{ Validation: &apiextensions.CustomResourceValidation{ OpenAPIV3Schema: &apiextensions.JSONSchemaProps{ Type: "object", XValidations: apiextensions.ValidationRules{ { Rule:    "size(self) > 0", Message: "openAPIV3Schema should contain more than 0 element.", }, }, Dependencies: apiextensions.JSONSchemaDependencies{ "test": apiextensions.JSONSchemaPropsOrStringArray{ Schema: &apiextensions.JSONSchemaProps{ Type: "object", XValidations: apiextensions.ValidationRules{ { Rule:    "size(self) > 0", Message: "size of scoped field should be greater than 0.", }, }, }, }, }, Properties: map[string]apiextensions.JSONSchemaProps{ "subRule": { Type: "object", XValidations: apiextensions.ValidationRules{ { Rule:    "isTest == true", Message: "isTest should be true.", }, }, Properties: map[string]apiextensions.JSONSchemaProps{ "isTest": { Type: "boolean", }, }, }, }, }, }, }, }, oldCRD: nil, expectedCRD: &apiextensions.CustomResourceDefinition{ ObjectMeta: metav1.ObjectMeta{Name: "foos.sigs.k8s.io", Annotations: map[string]string{v1beta1.KubeAPIApprovedAnnotation: "valid"}, ResourceVersion: "1"}, Spec: apiextensions.CustomResourceDefinitionSpec{ Validation: &apiextensions.CustomResourceValidation{ OpenAPIV3Schema: &apiextensions.JSONSchemaProps{ Type: "object", Dependencies: apiextensions.JSONSchemaDependencies{ "test": apiextensions.JSONSchemaPropsOrStringArray{ Schema: &apiextensions.JSONSchemaProps{ Type: "object", }, }, }, Properties: map[string]apiextensions.JSONSchemaProps{ "subRule": { Type: "object", Properties: map[string]apiextensions.JSONSchemaProps{ "isTest": { Type: "boolean", }, }, }, }, }, }, }, }, }, { name:               "For creation, FG enabled, set XValidations, update with XValidations", enableXValidations: true, crd: &apiextensions.CustomResourceDefinition{ ObjectMeta: metav1.ObjectMeta{Name: "foos.sigs.k8s.io", Annotations: map[string]string{v1beta1.KubeAPIApprovedAnnotation: "valid"}, ResourceVersion: "1"}, Spec: apiextensions.CustomResourceDefinitionSpec{ Validation: &apiextensions.CustomResourceValidation{ OpenAPIV3Schema: &apiextensions.JSONSchemaProps{ Type: "object", XValidations: apiextensions.ValidationRules{ { Rule:    "size(self) > 0", Message: "openAPIV3Schema should contain more than 0 element.", }, }, Dependencies: apiextensions.JSONSchemaDependencies{ "test": apiextensions.JSONSchemaPropsOrStringArray{ Schema: &apiextensions.JSONSchemaProps{ Type: "object", XValidations: apiextensions.ValidationRules{ { Rule:    "size(self) > 0", Message: "size of scoped field should be greater than 0.", }, }, }, }, }, Properties: map[string]apiextensions.JSONSchemaProps{ "subRule": { Type: "object", XValidations: apiextensions.ValidationRules{ { Rule:    "isTest == true", Message: "isTest should be true.", }, }, Properties: map[string]apiextensions.JSONSchemaProps{ "isTest": { Type: "boolean", }, }, }, }, }, }, }, }, oldCRD: nil, expectedCRD: &apiextensions.CustomResourceDefinition{ ObjectMeta: metav1.ObjectMeta{Name: "foos.sigs.k8s.io", Annotations: map[string]string{v1beta1.KubeAPIApprovedAnnotation: "valid"}, ResourceVersion: "1"}, Spec: apiextensions.CustomResourceDefinitionSpec{ Validation: &apiextensions.CustomResourceValidation{ OpenAPIV3Schema: &apiextensions.JSONSchemaProps{ Type: "object", XValidations: apiextensions.ValidationRules{ { Rule:    "size(self) > 0", Message: "openAPIV3Schema should contain more than 0 element.", }, }, Dependencies: apiextensions.JSONSchemaDependencies{ "test": apiextensions.JSONSchemaPropsOrStringArray{ Schema: &apiextensions.JSONSchemaProps{ Type: "object", XValidations: apiextensions.ValidationRules{ { Rule:    "size(self) > 0", Message: "size of scoped field should be greater than 0.", }, }, }, }, }, Properties: map[string]apiextensions.JSONSchemaProps{ "subRule": { Type: "object", XValidations: apiextensions.ValidationRules{ { Rule:    "isTest == true", Message: "isTest should be true.", }, }, Properties: map[string]apiextensions.JSONSchemaProps{ "isTest": { Type: "boolean", }, }, }, }, }, }, }, }, }, { name:               "For update, FG disabled, oldCRD XValidation in use, don't drop XValidations", enableXValidations: false, crd: &apiextensions.CustomResourceDefinition{ ObjectMeta: metav1.ObjectMeta{Name: "foos.sigs.k8s.io", Annotations: map[string]string{v1beta1.KubeAPIApprovedAnnotation: "valid"}, ResourceVersion: "1"}, Spec: apiextensions.CustomResourceDefinitionSpec{ Validation: &apiextensions.CustomResourceValidation{ OpenAPIV3Schema: &apiextensions.JSONSchemaProps{ Type: "object", XValidations: apiextensions.ValidationRules{ { Rule:    "size(self) > 0", Message: "openAPIV3Schema should contain more than 0 element.", }, }, Dependencies: apiextensions.JSONSchemaDependencies{ "test": apiextensions.JSONSchemaPropsOrStringArray{ Schema: &apiextensions.JSONSchemaProps{ Type: "object", XValidations: apiextensions.ValidationRules{ { Rule:    "size(self) > 0", Message: "size of scoped field should be greater than 0.", }, }, }, }, }, Properties: map[string]apiextensions.JSONSchemaProps{ "subRule": { Type: "object", XValidations: apiextensions.ValidationRules{ { Rule:    "isTest == true", Message: "isTest should be true.", }, }, Properties: map[string]apiextensions.JSONSchemaProps{ "isTest": { Type: "boolean", }, }, }, }, }, }, }, }, oldCRD: &apiextensions.CustomResourceDefinition{ ObjectMeta: metav1.ObjectMeta{Name: "foos.sigs.k8s.io", Annotations: map[string]string{v1beta1.KubeAPIApprovedAnnotation: "valid"}, ResourceVersion: "1"}, Spec: apiextensions.CustomResourceDefinitionSpec{ Validation: &apiextensions.CustomResourceValidation{ OpenAPIV3Schema: &apiextensions.JSONSchemaProps{ Type: "object", Dependencies: apiextensions.JSONSchemaDependencies{ "test": apiextensions.JSONSchemaPropsOrStringArray{ Schema: &apiextensions.JSONSchemaProps{ Type: "object", XValidations: apiextensions.ValidationRules{ { Rule:    "size(self) > 0", Message: "size of scoped field should be greater than 0.", }, }, }, }, }, }, }, }, }, expectedCRD: &apiextensions.CustomResourceDefinition{ ObjectMeta: metav1.ObjectMeta{Name: "foos.sigs.k8s.io", Annotations: map[string]string{v1beta1.KubeAPIApprovedAnnotation: "valid"}, ResourceVersion: "1"}, Spec: apiextensions.CustomResourceDefinitionSpec{ Validation: &apiextensions.CustomResourceValidation{ OpenAPIV3Schema: &apiextensions.JSONSchemaProps{ Type: "object", XValidations: apiextensions.ValidationRules{ { Rule:    "size(self) > 0", Message: "openAPIV3Schema should contain more than 0 element.", }, }, Dependencies: apiextensions.JSONSchemaDependencies{ "test": apiextensions.JSONSchemaPropsOrStringArray{ Schema: &apiextensions.JSONSchemaProps{ Type: "object", XValidations: apiextensions.ValidationRules{ { Rule:    "size(self) > 0", Message: "size of scoped field should be greater than 0.", }, }, }, }, }, Properties: map[string]apiextensions.JSONSchemaProps{ "subRule": { Type: "object", XValidations: apiextensions.ValidationRules{ { Rule:    "isTest == true", Message: "isTest should be true.", }, }, Properties: map[string]apiextensions.JSONSchemaProps{ "isTest": { Type: "boolean", }, }, }, }, }, }, }, }, }, { name:               "For update, FG disabled, oldCRD has no XValidations, drop XValidations", enableXValidations: false, crd: &apiextensions.CustomResourceDefinition{ ObjectMeta: metav1.ObjectMeta{Name: "foos.sigs.k8s.io", Annotations: map[string]string{v1beta1.KubeAPIApprovedAnnotation: "valid"}, ResourceVersion: "1"}, Spec: apiextensions.CustomResourceDefinitionSpec{ Validation: &apiextensions.CustomResourceValidation{ OpenAPIV3Schema: &apiextensions.JSONSchemaProps{ Type: "object", XValidations: apiextensions.ValidationRules{ { Rule:    "size(self) > 0", Message: "openAPIV3Schema should contain more than 0 element.", }, }, }, }, }, }, oldCRD: &apiextensions.CustomResourceDefinition{ ObjectMeta: metav1.ObjectMeta{Name: "foos.sigs.k8s.io", Annotations: map[string]string{v1beta1.KubeAPIApprovedAnnotation: "valid"}, ResourceVersion: "1"}, Spec: apiextensions.CustomResourceDefinitionSpec{ Validation: &apiextensions.CustomResourceValidation{ OpenAPIV3Schema: &apiextensions.JSONSchemaProps{ Type: "object", }, }, }, }, expectedCRD: &apiextensions.CustomResourceDefinition{ ObjectMeta: metav1.ObjectMeta{Name: "foos.sigs.k8s.io", Annotations: map[string]string{v1beta1.KubeAPIApprovedAnnotation: "valid"}, ResourceVersion: "1"}, Spec: apiextensions.CustomResourceDefinitionSpec{ Validation: &apiextensions.CustomResourceValidation{ OpenAPIV3Schema: &apiextensions.JSONSchemaProps{ Type: "object", }, }, }, }, }, { name:               "For update, FG enabled, oldCRD has XValidations, updated to newCRD", enableXValidations: true, crd: &apiextensions.CustomResourceDefinition{ ObjectMeta: metav1.ObjectMeta{Name: "foos.sigs.k8s.io", Annotations: map[string]string{v1beta1.KubeAPIApprovedAnnotation: "valid"}, ResourceVersion: "1"}, Spec: apiextensions.CustomResourceDefinitionSpec{ Validation: &apiextensions.CustomResourceValidation{ OpenAPIV3Schema: &apiextensions.JSONSchemaProps{ Type: "object", XValidations: apiextensions.ValidationRules{ { Rule:    "size(self) > 0", Message: "openAPIV3Schema should contain more than 0 element.", }, }, }, }, }, }, oldCRD: &apiextensions.CustomResourceDefinition{ ObjectMeta: metav1.ObjectMeta{Name: "foos.sigs.k8s.io", Annotations: map[string]string{v1beta1.KubeAPIApprovedAnnotation: "valid"}, ResourceVersion: "1"}, Spec: apiextensions.CustomResourceDefinitionSpec{ Validation: &apiextensions.CustomResourceValidation{ OpenAPIV3Schema: &apiextensions.JSONSchemaProps{ Type: "object", XValidations: apiextensions.ValidationRules{ { Rule:    "old data", Message: "old data", }, }, }, }, }, }, expectedCRD: &apiextensions.CustomResourceDefinition{ ObjectMeta: metav1.ObjectMeta{Name: "foos.sigs.k8s.io", Annotations: map[string]string{v1beta1.KubeAPIApprovedAnnotation: "valid"}, ResourceVersion: "1"}, Spec: apiextensions.CustomResourceDefinitionSpec{ Validation: &apiextensions.CustomResourceValidation{ OpenAPIV3Schema: &apiextensions.JSONSchemaProps{ Type: "object", XValidations: apiextensions.ValidationRules{ { Rule:    "size(self) > 0", Message: "openAPIV3Schema should contain more than 0 element.", }, }, }, }, }, }, }, { name:               "For update, FG enabled, oldCRD has no XValidations, updated to newCRD", enableXValidations: true, crd: &apiextensions.CustomResourceDefinition{ ObjectMeta: metav1.ObjectMeta{Name: "foos.sigs.k8s.io", Annotations: map[string]string{v1beta1.KubeAPIApprovedAnnotation: "valid"}, ResourceVersion: "1"}, Spec: apiextensions.CustomResourceDefinitionSpec{ Validation: &apiextensions.CustomResourceValidation{ OpenAPIV3Schema: &apiextensions.JSONSchemaProps{ Type: "object", XValidations: apiextensions.ValidationRules{ { Rule:    "size(self) > 0", Message: "openAPIV3Schema should contain more than 0 element.", }, }, }, }, }, }, oldCRD: &apiextensions.CustomResourceDefinition{ ObjectMeta: metav1.ObjectMeta{Name: "foos.sigs.k8s.io", Annotations: map[string]string{v1beta1.KubeAPIApprovedAnnotation: "valid"}, ResourceVersion: "1"}, Spec: apiextensions.CustomResourceDefinitionSpec{ Validation: &apiextensions.CustomResourceValidation{ OpenAPIV3Schema: &apiextensions.JSONSchemaProps{ Type: "object", }, }, }, }, expectedCRD: &apiextensions.CustomResourceDefinition{ ObjectMeta: metav1.ObjectMeta{Name: "foos.sigs.k8s.io", Annotations: map[string]string{v1beta1.KubeAPIApprovedAnnotation: "valid"}, ResourceVersion: "1"}, Spec: apiextensions.CustomResourceDefinitionSpec{ Validation: &apiextensions.CustomResourceValidation{ OpenAPIV3Schema: &apiextensions.JSONSchemaProps{ Type: "object", XValidations: apiextensions.ValidationRules{ { Rule:    "size(self) > 0", Message: "openAPIV3Schema should contain more than 0 element.", }, }, }, }, }, }, }, } for _, tc := range testCases { t.Run(tc.name, func(t *testing.T) { defer featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, features.CustomResourceValidationExpressions, tc.enableXValidations)() old := tc.oldCRD.DeepCopy() dropDisabledFields(tc.crd, tc.oldCRD) // old crd should never be changed if diff := cmp.Diff(tc.oldCRD, old); diff != "" { t.Fatalf("old crd changed from %v to %v", tc.oldCRD, old) } if diff := cmp.Diff(tc.expectedCRD, tc.crd); diff != "" { t.Fatalf("unexpected crd: %v", tc.crd) } }) } } </del>

This is the tracking issue of promoting CRD validation rules to GA: Listed the blocking issues/features below: [x] [x] [x] [x] Issues be aware of but should not be a GA blocker [ ] [ ] [ ] [ ] [ ] /sig api-machinery /triage accepted
/assign
/cc


Question: Time_Key    timestamp Time_Format %Y-%m-%dT%H:%M:%S.%L%z <del> # ---------- [PARSER] Name   json Format json Time_Key time Time_Format %d/%b/%Y:%H:%M:%S %z [PARSER] Name         docker Format       json Time_Key     time Time_Format  %Y-%m-%dT%H:%M:%S.%L Time_Keep    On </del> [PARSER] Name        syslog-rfc5424 Format      regex

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --The default fluent-bit configuration for GCE Windows nodes produces unparsed JSON logs and does not forward container logs. Container log output is produced. fluent-bit on a given windows image. a GCE Windows e2e cluster and view the stackdriver output for pods. Kubernetes version (use ): 1.20+ Cloud provider or hardware configuration: GKE/GCE OS (e.g: ): Windows Kernel (e.g. ): Any version Install tools: fluent-bit Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig windows


Question: func TestEnsurePublicIPExists(t *testing.T) { testCases := []struct { <del> desc          string existingPIPs  []network.PublicIPAddress expectedPIP   *network.PublicIPAddress expectedID    string expectedDNS   string expectedError bool </del> <ins> desc                    string existingPIPs            []network.PublicIPAddress inputDNSLabel           string foundDNSLabelAnnotation bool expectedPIP             *network.PublicIPAddress expectedID              string expectedError           bool </ins> }{ { desc:         "ensurePublicIPExists shall return existed PIP if there is any",

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: After issue is fixed. Cloud provider will automatically reconcile the Azure Public Ip DNS record same as the service annotation. As a result, user finds their DNS record will be deleted if the annotation is not set. Such a change is a block to users who have set the DNS record on their public IP without using this annotation when they upgrade the cluster. Besides, if the public ip is a user's own public ip, it is also not reasonable to delete the DNS record. What you expected to happen: We may need to change the reconcile logic. Maybe only delete the DNS record, when the annotation exists but empty value How to reproduce it (as minimally and precisely as possible): Manually add the DNS label to Azure Public IP resource via Portal Environment: Kubernetes version (use ): 1.15.7 Cloud provider or hardware configuration: Azure
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig Azure Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cloud-provider
/area provider/azure
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig Azure Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
The fix helm install helm-nginx-ingress stable/nginx-ingress  --namespace ingress-nginx-ns  --set controller.replicaCount=2  --set controller.nodeSelector."beta.kubernetes.io/os"=linux  --set controller.service.annotations."service.beta.kubernetes.io/azure-dns-label-name"=MY-AZURE-DNS  --set defaultBackend.nodeSelector."beta.kubernetes.io/os"=linux


Question: } pv.Spec.Capacity = v1.ResourceList{ <del> v1.ResourceName(v1.ResourceStorage): resource.MustParse(fmt.Sprintf("%dGi", sizeGB)), </del> <ins> v1.ResourceName(v1.ResourceStorage): resource.MustParse(fmt.Sprintf("%dG", sizeGB)), </ins> } return pv, nil }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: Create 50Gi podVolumeClaim with glusterfs provisioner. What you expected to happen: I expected 50Gi volume would be created, instead of 54Gi. How to reproduce it (as minimally and precisely as possible): storage class with glusterfs as dynamic provisioner. podVolumeClaim. Anything else we need to know?: Commit, that cause it: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig storage
Is it better to check binary/decimal format?
make sense
The bug doesn't exist in the commit you linked. Heketi can't provision volume in chunks of GiB that you are requesting, so it is never going to provision 50GiB if you asked for 50GiB. It will round the value upto GB and provision it. For example, if you are asking for 50GiB it will provision 51GiB after rounding upto to GBs. The problematic line in code is - That line incorrectly assumes returned integer from function to be in GiB whereas it is in GB, it should be relatively easy to change that line. So If you request 50GiB it will give you 54GB and 51GiB.
Yeah, inside the container i got:
Yep, the problem is in how the size is represented in PV. can you make a PR for fixing it?
Yes, it would be great.
Completely agree to what wrote. thanks for the PR.
/assign


Question: docker run --rm --privileged multiarch/qemu-user-static:register --reset curl -sSL --retry 5 https://github.com/multiarch/qemu-user-static/releases/download/v2.5.0/x86_64_qemu-${QEMUARCH}-static.tar.xz | tar -xJ -C ${TEMP_DIR} endif <del> # This cross-compiles cni for all architectures # TODO(freehan): Push the latest cni for all arches to storage.googleapis.com so we may just download the binaries docker run -it -v ${TEMP_DIR}/cni-bin:/cnibin golang:1.6 /bin/bash -c " git clone https://github.com/containernetworking/cni  && cd cni  && git checkout $(CNI_RELEASE)  && GOARCH=$(ARCH) ./build  && cp bin/* /cnibin" </del> <ins> # Download CNI curl -sSL --retry 5 https://storage.googleapis.com/kubernetes-release/network-plugins/cni-${ARCH}-${CNI_RELEASE}.tar.gz | tar -xz -C ${TEMP_DIR}/cni-bin </ins> docker build -t ${REGISTRY}/hyperkube-${ARCH}:${VERSION} ${TEMP_DIR} rm -rf "${TEMP_DIR}"

We should update cni binaries to containernetworking/cni and release binaries for all arches. I do not have push access to the repo, so I guess some googler have to do it.
Thanks. I have this on my plate.


Question: "time" "k8s.io/api/core/v1" <ins> apiequality "k8s.io/apimachinery/pkg/api/equality" </ins> "k8s.io/apimachinery/pkg/api/errors" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/labels"

Currently, endpoints objects containing no subsets are decoded by the go client as subsets:[] (when requested individually) or as subsets:null (when requested in a list of endpoints). Because the endpoints controller is fed via a lister/watcher, it gets the subsets:null version fed to it. The subsets computation then returns an empty slice, which fails reflect.DeepEqual, which triggers a write attempt
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals


Question: 1. Docker, using one of the following configurations: * **macOS** You can either use Docker for Mac or docker-machine. See installation instructions [here](https://docs.docker.com/docker-for-mac/). <del> **Note**: You will want to set the Docker VM to have at least 3GB of initial memory or building will likely fail. (See: [#11852]( http://issue.k8s.io/11852)). </del> <ins> **Note**: You will want to set the Docker VM to have at least 4.5GB of initial memory or building will likely fail. (See: [#11852]( http://issue.k8s.io/11852)). </ins> * **Linux with local Docker**  Install Docker according to the [instructions](https://docs.docker.com/installation/#installation) for your OS. * **Remote Docker engine** Use a big machine in the cloud to build faster. This is a little trickier so look at the section later on. 2. **Optional** [Google Cloud SDK](https://developers.google.com/cloud/sdk/)

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Current instruction state that Kubernetes requires 3GB to build in Docker on osx. Attempting to build () with 3GB, 3.5GB, and 4.0GB fail. What you expected to happen: I expect osx users to be able to build Kubernetes, and the instructions to indicate that at least 4.5GB of memory are required. How to reproduce it (as minimally and precisely as possible): On osx setup docker with 3GB memory, and then attempt to build Kubernetes () and it will fail. Anything else we need to know?: Environment: Kubernetes version (use ): upstream/master OS (e.g. from /etc/os-release): osx - 10.13.4
/sig contributor-experience
/sig docs