Question: t.Errorf("Expected original,added groups, got %#v", user.GetGroups()) } } <ins> func TestAuthenticatedGroupAdder(t *testing.T) { tests := []struct { name         string inputUser    user.Info expectedUser user.Info }{ { name: "add", inputUser: &user.DefaultInfo{ Name:   "user", Groups: []string{"some-group"}, }, expectedUser: &user.DefaultInfo{ Name:   "user", Groups: []string{"some-group", user.AllAuthenticated}, }, }, { name: "don't double add", inputUser: &user.DefaultInfo{ Name:   "user", Groups: []string{user.AllAuthenticated, "some-group"}, }, expectedUser: &user.DefaultInfo{ Name:   "user", Groups: []string{user.AllAuthenticated, "some-group"}, }, }, { name: "don't add for anon", inputUser: &user.DefaultInfo{ Name:   user.Anonymous, Groups: []string{"some-group"}, }, expectedUser: &user.DefaultInfo{ Name:   user.Anonymous, Groups: []string{"some-group"}, }, }, { name: "don't add for unauthenticated group", inputUser: &user.DefaultInfo{ Name:   "user", Groups: []string{user.AllUnauthenticated, "some-group"}, }, expectedUser: &user.DefaultInfo{ Name:   "user", Groups: []string{user.AllUnauthenticated, "some-group"}, }, }, } for _, test := range tests { adder := authenticator.Request( NewAuthenticatedGroupAdder( authenticator.RequestFunc(func(req *http.Request) (user.Info, bool, error) { return test.inputUser, true, nil }), ), ) user, _, _ := adder.AuthenticateRequest(nil) if !reflect.DeepEqual(user, test.expectedUser) { t.Errorf("got %#v", user) } } } </ins>

When an auth proxy tries to proxy an anonymous request, the user.Info ends up with the group system:authenticated, which is wrong.


Question: } } <ins> func TestApfWatchHandlePanic(t *testing.T) { preExecutePanicingFilter := newFakeWatchApfFilter(1) preExecutePanicingFilter.preExecutePanic = true postExecutePanicingFilter := newFakeWatchApfFilter(1) postExecutePanicingFilter.postExecutePanic = true testCases := []struct { name   string filter *fakeWatchApfFilter }{ { name:   "pre-execute panic", filter: preExecutePanicingFilter, }, { name:   "post-execute panic", filter: postExecutePanicingFilter, }, } onExecuteFunc := func() { time.Sleep(5 * time.Second) } postExecuteFunc := func() {} for _, test := range testCases { t.Run(test.name, func(t *testing.T) { apfHandler := newApfHandlerWithFilter(t, test.filter, onExecuteFunc, postExecuteFunc) handler := func(w http.ResponseWriter, r *http.Request) { defer func() { if err := recover(); err == nil { t.Errorf("expected panic, got %v", err) } }() apfHandler.ServeHTTP(w, r) } server := httptest.NewServer(http.HandlerFunc(handler)) defer server.Close() if err := expectHTTPGet(fmt.Sprintf("%s/api/v1/namespaces/default/pods?watch=true", server.URL), http.StatusOK); err != nil { t.Errorf("unexpected error: %v", err) } }) } } </ins> // TestContextClosesOnRequestProcessed ensures that the request context is cancelled // automatically even if the server doesn't cancel is explicitly. // This is required to ensure we won't be leaking goroutines that wait for context

As visible in this graph: ! pod-startup time in 5k-node tests increased around beginning of June (only 99% though). It seems to be connected to "run-to-watch" phase. We should investigate and see what we can do with this. /sig scalability /kind bug
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : As visible in this graph: ! /kind bugs Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage accepted Once PR that merged around that time is P&F support for watch-initialitation: As a first step, we may want to confirm/reject if it's this PR and see if we aren't somehow misconfigured here.
I've briefly checked test-infra/perf-tests to see if we had any relevant changes around that time, but I don't see any. Most likely it's related to k/k changes.
/assign
Good news is that we can also see it regression in kubemark (although not as bad as in regular 5k tests): http://perf- But this can help us with finding root cause faster/cheaper. ! I've picked two runs, one before this change, one after: 26th May: Timeline: 12th June: Timeline: First run vs second run I can see significantly higher spike on waiting time in P&F, but according to timeline it seems like it was during huge service and not pod startup latency: 26th May: ! 12th June: ! I can see that number of apiserver goroutines significantly increased (from ~500k to ~900k). I would expect to have 1 more goroutine per watch, we have around 200k watches). ! vs ! Worst 100 pods with slowest latency runtowatch: I've picked the slowest pod "test-2xppkn-33/latency-deployment-60--p7nwz" to analyse.

This pod was bound to node gce-scale-cluster-minion-group-4-txww: Kubelet log for this pod: CL2 logs for this pod: In apiserver logs we can see: So it looks like updating pod status was delayed by P&F by ~500 ms for this pod. Timeline for this pod: 18:37:46. scheduled 18:37:48. container started 18:37:49. kubelet gets pod (1.3s latency with 610 ms delay from P&F) 18:37:50. patch status of pod (545ms latency with 500ms delay from P&F) 18:37:51. observed by CL2 as running To compare with previous run we have this timeline for pod "test-rcdjgd-26/latency-deployment-69--6s95x 2.724031741s": 18:57:08. scheduled on node gce-scale-cluster-minion-group-3-l21s 18:57:10. container started 18:57:10. kubelet gets pod (364 ms latency with 345 ms delay from P&F) 18:57:11. patched status od pod (360 ms latency with 254 ms delay from P&F) So to compare, for these two pods major difference was "get pod" + "patch status" in kubelet: 1.8s vs 700ms. 1s of 1.8s of latency was due to P&F and 0.6s of 0.7s was due to P&F.
In both runs I don't see significant waiting time (99th percentile) on any of Priority levels during pod startup latency phase: 26th May: ! 12th June: !
1) Do we have data about 100% waiting time in P&F? With 99% it my not be enough to to see that data. 2) If really waiting time is small, then the question is what exactly. (a) it might be related to the number of goroutines (b) it might be locking (c) it might be something else... I have just opened the WIP PR: that should help with first two (at least to some extent). Will you be able to test if that helps?

Max waiting time in P&F for pod startup latency phase: 26th May: ! 12th June: ! Buckets are ..., 0.2, 0.5, ... so I don't think it's significant. will test it.
This is definitely non-negligible regression that we should address in 1.22 timeframe. I'm actually have a promissing results for it. /milestone v1.22 /priority critical-urgent
OK - so we proved that addresses the problem: it reduces back to previous value the number of goroutines ! it brings back pod startup time back to less than 3.4s on 99th percentile it brings back API call latencies I'm going to work with reviewers on merging this to fix the regression. - FYI


Question: configs[CheckMetadataConcealment] = Config{e2eRegistry, "metadata-concealment", "1.2"} configs[CudaVectorAdd] = Config{e2eRegistry, "cuda-vector-add", "1.0"} configs[CudaVectorAdd2] = Config{e2eRegistry, "cuda-vector-add", "2.0"} <del> configs[DebianIptables] = Config{buildImageRegistry, "debian-iptables", "buster-v1.3.0"} </del> <ins> configs[DebianIptables] = Config{buildImageRegistry, "debian-iptables", "buster-v1.4.0"} </ins> configs[EchoServer] = Config{e2eRegistry, "echoserver", "2.2"} configs[Etcd] = Config{gcRegistry, "etcd", "3.4.13-0"} configs[GlusterDynamicProvisioner] = Config{dockerGluster, "glusterdynamic-provisioner", "v1.0"}

What happened: The latest release of the debian-base image, v2.1.3, contains a vulnerable version of the apt package. CVE-2020- impacts apt version 1.8.2.1. What you expected to happen: Image should be updated to use at least apt 1.8.2.2 to clear the vulnerability. How to reproduce it (as minimally and precisely as possible):
/sig release
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
kube-proxy is using debain as base image. /assign Let me check other components
checked the base image in and I seems that no new debian or ubuntu official image is built for this CVE ?
That is correct, the purpose of this issue is to get a new image built and released, one which contains a resolution for the CVE.
per comments of @ tianon in Will update it once debain releases it.


Question: } if metricFamily == nil { <del> return Histogram{}, fmt.Errorf("metric %q not found", metricName) </del> <ins> return nil, fmt.Errorf("metric %q not found", metricName) </ins> } if metricFamily.GetMetric() == nil { <del> return Histogram{}, fmt.Errorf("metric %q is empty", metricName) </del> <ins> return nil, fmt.Errorf("metric %q is empty", metricName) </ins> } if len(metricFamily.GetMetric()) == 0 { <del> return Histogram{}, fmt.Errorf("metric %q is empty", metricName) </del> <ins> return nil, fmt.Errorf("metric %q is empty", metricName) </ins> } <del> return Histogram{ // Histograms are stored under the first index (based on observation). // Given there's only one histogram registered per each metric name, accessing // the first index is sufficient. metricFamily.GetMetric()[0].GetHistogram(), }, nil </del> <ins> vec := make(HistogramVec, 0) for _, metric := range metricFamily.GetMetric() { if LabelsMatch(metric, lvMap) { if hist := metric.GetHistogram(); hist != nil { vec = append(vec, &Histogram{hist}) } } } return vec, nil </ins> } func uint64Ptr(u uint64) *uint64 {

The new scheduler performance test was introduced back in 1.18, back then, the metric it supported are all . However, along with , some metrics are updated to be , and hence it's no longer appropriate to return the first Histogram: However, it's very common (for instance Preemption) that the metrics may hold multiple histograms: Additionally, metrics and have been removed, we should update them to : /sig scheduling
/triage accepted /assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale


Question: # These are the default versions (image tags) for their respective base images. readonly __default_distroless_iptables_version=v0.5.1 <del> readonly __default_go_runner_version=v2.3.1-go1.22-bookworm.0 </del> <ins> readonly __default_go_runner_version=v2.3.1-go1.22.0-bookworm.0 </ins> readonly __default_setcap_version=bookworm-v1.0.1 # These are the base images for the Docker-wrapped binaries.

https://prow.ppc64le- Tests are not getting executed due to improper go version. It has been failing since past 4 days. No response No response No response /sig node
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This seems to be failing due to improper version addition in dependency files where is instead of at some places. I will raise a PR for it.
/assign
fixing that


Question: if path, subscript, ok := fieldpath.SplitMaybeSubscriptedPath(internalFieldPath); ok { switch path { case "metadata.annotations": <del> for _, msg := range validation.IsQualifiedName(strings.ToLower(subscript)) { allErrs = append(allErrs, field.Invalid(fldPath, subscript, msg)) } </del> <ins> allErrs = append(allErrs, ValidateQualifiedName(strings.ToLower(subscript), fldPath)...) </ins> case "metadata.labels": <del> for _, msg := range validation.IsQualifiedName(subscript) { allErrs = append(allErrs, field.Invalid(fldPath, subscript, msg)) } </del> <ins> allErrs = append(allErrs, ValidateQualifiedName(subscript, fldPath)...) </ins> default: allErrs = append(allErrs, field.Invalid(fldPath, path, "does not support subscript")) }

After got merged, we have new helper function here: It will be good for us to use this simpler helper function to replace previous function. /good-first-issue
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : After got merged, we have new helper function here: </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig usability
I would like to take this one /assign
I worked in on this as well, but kept your changed files untouched (as written in my PR description). same for you. I hope this is OK for you.
This should be reopened until and got merged (my PR excluded the changes from the other PRs).
/reopen per last comment
Reopened this issue. <detailsIn response to : This should be reopened until and got merged (my PR excluded the changes from the other PRs). per last comment Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
can you assign me
/assign
/assign
I'd love to work on this issue.

/assign
Hi, , I am a first commitor for kubernetes and I have committed a PR in , could you help me to review it, or give me some suggestions. Thanks a lot.
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: "//pkg/controller:go_default_library", "//pkg/controller/util/endpoint:go_default_library", "//pkg/features:go_default_library", <ins> "//pkg/util/labels:go_default_library", </ins> "//staging/src/k8s.io/api/core/v1:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/api/equality:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/api/errors:go_default_library",

bug (): Originally posted by in cc /sig network /priority critical-urgent
fix in
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.


Question: } } <del> if !probeGracePeriodInUse(oldPodSpec) { // Set pod-level terminationGracePeriodSeconds to nil if the feature is disabled and it is not used VisitContainers(podSpec, AllContainers, func(c *api.Container, containerType ContainerType) bool { if c.LivenessProbe != nil { c.LivenessProbe.TerminationGracePeriodSeconds = nil } if c.StartupProbe != nil { c.StartupProbe.TerminationGracePeriodSeconds = nil } // cannot be set for readiness probes return true }) } </del> // If the feature is disabled and not in use, drop the hostUsers field. if !utilfeature.DefaultFeatureGate.Enabled(features.UserNamespacesSupport) && !hostUsersInUse(oldPodSpec) { // Drop the field in podSpec only if SecurityContext is not nil.

Kubernetes does not respect the of a probe. I found this while fixing the e2e tests. Test result: /kind regression /kind bug /sig node /priority important-soon The of a container's probe should override the pod's . a pod with long pod-level terminationGracePeriodSeconds and short probe-level terminationGracePeriodSeconds. for the container not respecting the probe-level terminationGracePeriodSeconds No response <details</details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I guess the probe's terminationGracePeriodSeconds field is dropped for some reason.
/assign


Question: # When a 'git archive' is exported, the '$Format:%D$' below will look # something like 'HEAD -> release-1.8, tag: v1.8.3' where then 'tag: ' # can be extracted from it. <del> if [[ '$Format:%D$' =~ tag: (v[^ ]+) ]]; then </del> <ins> if [[ '$Format:%D$' =~ tag: (v[^ ,]+) ]]; then </ins> KUBE_GIT_VERSION="${BASH_REMATCH[1]}" fi fi

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Download the kube release source artifact from and do a build. All the kube binaries have a extraneous comma in the version: $ /usr/bin/kube-proxy --version Kubernetes v1.9.3, What you expected to happen: kube binaries built from git archive source tgz to show the correct version without extra comma. this was introduced in git attribute export-subst on creates a string format like ''HEAD -release-1.9, tag: v1.9.3, origin/release-1.9' . so, the regex capture should ignore ','
/sig release
The issue is that our regex didn't handle the case where the tag isn't the last part of the string filled in by : One the 1.9.3 source tarball linked, this looks like This should be a pretty easy fix - I don't think commas are allowed in semantic version strings. /assign
/kind bug /priority important-soon
This PR likely affects v1.10 as well. I have a fix out:
Confirmed: this affects v1.10.0-beta.0 (trying ): Though applying my fix from reveals a different bug: The issue then is that there are multiple tags matching that commit when it was built: who could guess that hacks would break?
disregard the second part of my comment above (at least for now); v1.10.0-beta.1 has a more reasonable string: (but it still demonstrates the original bug this issue mentions)
we need a sig lead to status/approved-for-milestone here
/status approved-for-milestone sure, good to be fixed :-)
aaaaaand priority increased and re- to milestone....
/priority critical-urgent
Master branch is fixed. Creating cherry-pick PRs now: 1.9: 1.8: 1.7:
[MILESTONENOTIFIER] Milestone Issue Labels Incomplete Action required: This issue requires label changes. priority: Must specify exactly one of , or .

<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
Cherry pick PRs are in. Next patch release of each branch should be fixed. /close


Question: which won't work properly on Windows. See https://github.com/moby/buildkit/issues/1560 - The base image for all the Windows images is nanoserver, which is ~10 times smaller than Windows Servercore. Most binaries added to the image will work out of the box, but some will not due to missing dependencies <del> (**atention**: the image will still build successfully, even if the added binaries will not work). </del> <ins> (**attention**: the image will still build successfully, even if the added binaries will not work). </ins> For example, `coredns.exe` requires `netapi32.dll`, which cannot be found on a nanoserver image, but we can copy it from a servercore image (see the agnhost image's `Dockerfile_windows` file as an example). A good rule of thumb is to use 64-bit applications instead of 32-bit as they have fewer dependencies.

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI -- [sig-windows] [Feature:Windows] GMSA Kubelet [Slow] kubelet GMSA support when creating a pod with correct GMSA credential specs passes the credential specs down to the Pod's containers [sig-windows] [Feature:Windows] GMSA Full [Serial] [Slow] GMSA support works end to end ** pause image using in image was transitioned to the official pause image which no longer has in the image related to /sig windows /triage accepted


Question: } func (s *sourceEtcd) run() { <del> watching := s.helper.Watch(s.key, 0) </del> <ins> boundPods := api.BoundPods{} err := s.helper.ExtractToList(s.key, &boundPods) if err != nil { glog.Errorf("etcd failed to retrieve the value for the key %q. Error: %v", s.key, err) return } // Push update. Maybe an empty PodList to allow EtcdSource to be marked as seen s.updates <- kubelet.PodUpdate{boundPods.Items, kubelet.SET, kubelet.EtcdSource} index, _ := s.helper.ResourceVersioner.ResourceVersion(&boundPods) watching := s.helper.Watch(s.key, index) </ins> for { select { case event, ok := <-watching.ResultChan():

To reproduce: Start a cluster via hack/local-up- Start some pods Kill the cluster "docker ps -a" should show the pod containers running still Start the cluster again "etcdctl ls /registry" should show that there is no boundpods directory -- only minions and services "docker ps -a" should show the old pod containers still running Start a new pod After, say, 10s, "docker ps -a" should show the old containers dead /cc
I suspect the cause of this might be : which guarantees that the pod sync logic is never called if is nil. Need to get a repro to confirm.


Question: h.driverInfo.Config.ClientNodeName = node.Name } <del> func (h *hostpathDriver) DeleteVolume(volType testpatterns.TestVolType) { </del> <ins> func (h *hostPathDriver) DeleteVolume(volType testpatterns.TestVolType) { </ins> } <del> // HostpathSymlink type hostpathSymlinkDriver struct { </del> <ins> // HostPathSymlink type hostPathSymlinkDriver struct { </ins> node       v1.Node sourcePath string targetPath string

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: https://k8s- What happened: The skip check for hostpath doesn't seem to be working for these test cases
can you take a look?
Oh, disruptive tests are the tests that can't be tested locally in my environment due to I will take a look.
I found that above issue is not related, I should have been checked more carefully. Sorry for that.
I found out that this was just a driver name issue, so below will fix it: hostpath -hostPath hostpathSymlink -hostPathSymlink We might be better changing the skip logic later, to make it skip not by name in order to avoid this kind of bug, though. I will make a PR for above change, soon.
I made a PR above. Also, it might be a bit out of scope of this fix, but are drivers tested with containerized volume server, like nfs, ceph, etc..., safe to be tested with subpath disruptive tests? I'm worrying about the case that test pod and containerized volume server are deployed on the same node. In such a case, the same behaviour expected or do we need to avoid this kind of situation?


Question: // This file tests the Taint feature. import ( <ins> "errors" </ins> "fmt" "testing" "time" v1 "k8s.io/api/core/v1" <ins> apierrors "k8s.io/apimachinery/pkg/api/errors" </ins> "k8s.io/apimachinery/pkg/api/resource" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime/schema"

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: Which test(s) are failing: TaintBasedEvictions Since when has it been failing: probably since it was in this PR? Testgrid link: Reason for failure: Anything else we need to know: /sig scheduling /kind flake /priority important-soon /milestone v1.17
You must be a member of the GitHub team to set the milestone. If you believe you should be able to issue the /milestone command, please contact your and have them propose you as an additional delegate for this responsibility. <detailsIn response to : <!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs -- TaintBasedEvictions probably since it was in this PR? Anything else we need to know: /sig scheduling /kind flake /priority important-soon /milestone v1.17 </details
cc.
Looking into this. We moved it to integration because it was flaking in e2e too
/assign damemi
Not sure yet, but I do see this in the logs of the failing tests: which, if I understand this, would remove all the taints and mark nodes as reachable (which would explain why the test is failing waiting for node taints: ). But I need to figure out why all the nodes are becoming unavailable
is that a full fix for this flake or just partial? I still saw it failed yesterday so just checking... :)
I think I also run into this on 13th Oct: Full logs:
Now gets merged, which might fix the flake. Let's keep an eye on this.
reported it still flaked: Can you continue looking into this?
I'll keep trying, from my previous debugging it seems that in some cases the test cluster is entering full disruption mode and removing all of the taints from the nodes. I know we specifically made changes to the test (using multiple nodes) to ensure it wouldn't become fully disrupted, do you have any idea what could still be causing that?
There is some flag controlling how the dispution thing is managed internally, maybe you can check how a node lifecycle manager is constructed.

The test still flakes after we merge , we still need to investigate the cause ,
/assign
Another flake:
These flakes are consistently (always) timeouts waiting on node taints:
I'm working on this. Will send out a PR soon.
/reopen We see another flake since got merged: However, don't panic :) The original "Entering master disruption mode" cause has been fixed - you won't see that message in the above log. This time it's a smaller issue. I will explain it in another PR.
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen The job is still flaky
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/close But for another test :)
Closing this issue. <detailsIn response to : /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: // ErrorNotVmssInstance indicates an instance is not belongint to any vmss. ErrorNotVmssInstance = errors.New("not a vmss instance") <del> scaleSetNameRE = regexp.MustCompile(`.*/subscriptions/(?:.*)/Microsoft.Compute/virtualMachineScaleSets/(.+)/virtualMachines(?:.*)`) </del> <ins> scaleSetNameRE        = regexp.MustCompile(`.*/subscriptions/(?:.*)/Microsoft.Compute/virtualMachineScaleSets/(.+)/virtualMachines(?:.*)`) vmssMachineIDTemplate = "/subscriptions/%s/resourceGroups/%s/providers/Microsoft.Compute/virtualMachineScaleSets/%s/virtualMachines/%s" </ins> ) // scaleSet implements VMSet interface for Azure scale set.

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: When useInstanceMetadata is set to true for vmss nodes, their providerID is wrong. What you expected to happen: Their providerID should be in format azure:///subscriptions/<subscription-id/resourceGroups/<rg-name/providers/Microsoft.Compute/virtualMachineScaleSets/<scaleset-name/virtualMachines/<instance-id How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig azure
adding priority important-soon so that this issue can stay in 1.10. Please update if incorrect. /priority important-soon


Question: func SetObjectDefaults_Service(in *v1.Service) { SetDefaults_Service(in) <ins> for i := range in.Spec.Ports { a := &in.Spec.Ports[i] if a.Protocol == "" { a.Protocol = "TCP" } } </ins> } func SetObjectDefaults_ServiceList(in *v1.ServiceList) {

It looks like the openapi is not routed properly to set-up the apiservice so apply requests will fail because the fieldmanager is not set-up properly. /cc /sig api-machinery /priority important-soon /kind bug
/wg apply /cc
OpenAPIModels is set to nil for the kube-aggregator server config (which serves the APIServices API): This is done because the kube-aggregator has a special openapi handler that aggregates all the other server's openapi docs. The field controls (at least) two things: /openapi endpoint serving (which we want kube-aggregator to continue to customize) creation of the field manager (which we want kube-aggregator to behave consistently on) We need to decouple presence of OpenAPIConfig and construction of StaticOpenAPISpec (which is needed by the field manager) from construction/installation of OpenAPIVersionedService. That might be a little tricky to untangle, since construction of StaticOpenAPISpec includes paths/operations. The field manager only needs the gvk/schemas.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/lifecycle frozen This is not something we want closed by accident.
/assign
For reference:
Additional lines for reference: This is roughly how SSA consumes the openapi spec. openapi spec -openapi proto model -typeconverter -SSA field manager For built in types, openapi spec -openapi model openapi model -typeconverter A similar thing happens for CRDs As liggitt mentioned, SSA only requires a subset of the information in the openapi generation, and the openapi generation in the aggregator slightly different than the process for builtins and CRDs.
/unassign
/assign


Question: klog.Errorf("Failed to create govmomi client. err: %+v", err) return err } <ins> setVCenterInfoMetric(connection) </ins> return nil } m := session.NewManager(connection.Client)

<!-- Please only use this template for submitting enhancement requests --What would you like to be : We need reporting of version to be shown in the metrics. Why is this needed: The goal behind this is to add some info about the underlying infrastructure for the kubernetes cluster. Some customers were experiencing issues during the cluster upgrades, but nothing in the upgrades themselves or kubernetes code for provider seemingly could cause those. The issue is storage related, and happening on a variety of releases with a significant delay from any other underlying infrastructure change, to which the kubernetes cluster has no relation and a way to know about. So knowing used hostnames, and versions associated with it over time would be extremely beneficial for debugging in similar circumstances. Some of the use cases we face: 1) A is enabled in a customer environment, and so there occurs a periodic automated vMotion for VMs backing kuberentes . We'd like to know on which host the actual kubernetes was located at any moment, and if any vCenter version transitions happened over time, new hosts /removed/renamed.
/sig cloud-provider /wg provider-vsphere
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig cloud-provider /wg provider-vsphere Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/area provider/vmware


Question: endpointsConfig.Channel("api"), ) <del> config.NodeRef = &api.ObjectReference{ </del> <ins> config.NodeRef = &v1.ObjectReference{ </ins> Kind:      "Node", Name:      hostname, UID:       types.UID(hostname),

Is this a BUG REPORT or FEATURE REQUEST? (choose one): bug Kubernetes version (use ): master Environment: Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: What happened: When testing , I found kube-proxy can not record event with following errors. What you expected to happen: No errors How to reproduce it (as minimally and precisely as possible): cc
I am still getting this with the latest build
, thanks for your feedback, I'll check it.


Question: lun, err := common.AttachDisk(true, "", diskURI, test.nodeName, compute.CachingTypesReadOnly) assert.Equal(t, test.expectedLun, lun, "TestCase[%d]: %s", i, test.desc) <del> assert.Equal(t, test.expectedErr, err != nil, "TestCase[%d]: %s", i, test.desc) </del> <ins> assert.Equal(t, test.expectedErr, err != nil, "TestCase[%d]: %s, return error: %v", i, test.desc, err) </ins> } }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Pods with PVs could not be moved from one node to another. According to the controller-manager logs, K8s tried to detach to Azure disks from the same node. One of the detach operation waited for this lock (). While it waited for the lock, the controller-manager pod was restarted (for whatever reason, not related to this bug, lets assume an operator restarted the pod). After the restart, K8s only tried to attach the disk to the new node but is not trying to detach it first. What you expected to happen: After a restart of the controller-manager, it should continue unfinished detach operations How to reproduce it (as minimally and precisely as possible): 2 Pods with PVs running on one node Move both pods to another node restart the controller-manager shortly after the move Anything else we need to know?: Environment: Kubernetes version (use ): 1.15.1 Cloud provider or hardware configuration: Azure OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig azure
/sig cloud-provider
/area provider/azure that's expected behavior, k8s controller won't continue detach volume if controller manager restarted, user needs to detach that disk volume manually.
Could you please elaborate why this behaviour is expected. I would assume that the controller manager tries to reach the desired state after a restart? Then it would need to detect that the volume is still attached to the wrong node and should try to detach it once more, before it continuing to attach the (still attached) volume? This leads to many ... Why is it required to resolve this situation by manual interaction?
I can't imagine that "manual intervention" is "desired behaviour" (yes, you wrote "expected", but that doesn't matter, if you make no proposal, how to improve this). If this isn't fixed, we can't use Azure, period. We can't have clusters that need to be babysitted / you cannot expect people running workloads on Azure clusters and babysit them and manually fixing what should/can fix itself.

Level-triggered behaviour and resilience is what made Kubernetes so successful... please bring that also to Azure's KCM/CCM.
do you know why controller manager is restarted, it's not a common scenario, we cannot guarantee disk attach/detach functionality works if controller manager restarted at that time.
related to
I think there can be many reasons why the controller-manager is restarted during running detach or attach operations e. g. the hosting machine turn into error state, if you run the controller-manager in k8s -vertical pod autoscaling,... etc. I understand you can't guarantee that attach/detach works when the controller-manager is restarted, but I don't understand why the controller-manager can't recover the situation autonomously after a restart. After restart it tries to attach the volume of the rescheduled pod and this does not work because the volume is still attached to another machine. This is indicated by a error from the arm and the controller-manager could react to it properly, right?
This is an upstream issue, and there is a discussion in , let's check whether we can fix it in cloud provider driver in that thread. Thanks.
I will try to fix this issue in azure disk driver, thanks all for the discussion. details: /assign
Thank you very much It means a lot to us, that there are no manual operations. I wonder how you guys deal with it in AKS, since Azure uses the same controllers, right? Anyway, it helps tremendously, that the clusters are self-healing.
AKS use same kube-controller-manager, while it won't be restarted easily.
OK, but how? I understand that AKS only supports 100 nodes. So, I guess you have some upper limit and just hope, nobody stresses it, because if this happens, you it either restarts or you need to scale it up and then it gets restarted as well. Also, regular maintenance, low-level as well as Kubernetes updates, all must impact it. Considering the probably thousands over thousands of clusters (I don't know, but I expect you run many), any such problem, even if unlikely by itself, must be in your interest to fix, simply because the number of clusters increases the likelihood that it happens, I would think. Anyway, thank you for making this change
Yes, that's reasonable, I will try to fix it as my top priority.


Question: # NOTE: All functions that return lists should use newlines. # bash functions can't return arrays, and spaces are tricky, so newline # separators are the preferred pattern. <del> # To transform a string of newline-separated items to an array, use mapfile -t: # mapfile -t FOO <<< "$(kube::golang::dups a b c a)" </del> <ins> # To transform a string of newline-separated items to an array, use kube::util::read-array: # kube::util::read-array FOO < <(kube::golang::dups a b c a) </ins> # # ALWAYS remember to quote your subshells. Not doing so will break in # bash 4.3, and potentially cause other issues.

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --This issue was moved from What happened: building kubernetes with make fails on mac after What you expected to happen: building should not fail How to reproduce it (as minimally and precisely as possible): build from a mac Anything else we need to know?: mapfile is not available on macOS bash. we should have a version check at minimum if we're going to use unsupported features (some of the code already does this) cc
/sig release
Ugh. Does the containerized build at least still work on Mac? Ideally we'd stick to that as the recommended approach, since we can at least guarantee a correct environment. I guess this is really Apple's fault, for refusing to upgrade the bash version on Macs (so they can avoid GPL) - the default on most Macs is 3.2.57, which dates to 2007 according to this blog post: I don't personally think we should support something that old. None of our CI uses a bash that old. It's ripe for breakage. You can install a newer version of bash via homebrew.
the containerized build does not because this code runs on the host. we should put a version check then, so we get a more understandable error.
Up until this point we have supported it by using instead, macOS's old system bash is annoying, but errors from deep within our bash waste more time supporting confused developers
IIRC even in CI you ran into a bash version problem in fact just yesterday (?) we could add a snippet like this to the main build if we don't want to switch to :
I worry about requiring brew and can imagine some folks systems might be managed in a way that disallow this. Still it's reaaaaally painful having to debug and trying to keep things running with the crufty OSX dev userspace. Given there are multiple snippets of brew install suggestions, it would be better imho to put them in one common top level place (ie: )
fair, On systems with restricted at least should work, if that doesn't work i'd expect the setup is just too restrictive to build at all (?)

The check doesn't have to require homebrew per se either, even just "Sorry building on macOS is unsupported" is more helpful than:
Fix inbound:
/assign


Question: health kubernetes {{ pillar['dns_domain'] }} in-addr.arpa ip6.arpa { pods insecure <ins> upstream </ins> fallthrough in-addr.arpa ip6.arpa ttl 30 }

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: ci-kubernetes-e2e-gci-gce-serial Which test(s) are failing: [sig-network] DNS configMap nameserver [IPv4] Forward external name lookup should forward externalname lookup to upstream nameserver [Slow][Serial] Since when has it been failing: Failing since 6/10 at 16:58. . Decent chance that the failure is due to Testgrid link: Reason for failure: /cc /priority critical-urgent /kind failing-test /milestone v1.15 /cc
This is blocking the release /cc
I noticed in the it thinks it's testing 1.16 alpha:
/assign
should fix it. Sorry for the mess-up!


Question: return fmt.Errorf("--since must be greater than 0") } <del> if logsOptions.TailLines != nil && *logsOptions.TailLines < 0 { return fmt.Errorf("TailLines must be greater than or equal to 0") </del> <ins> if logsOptions.TailLines != nil && *logsOptions.TailLines < -1 { return fmt.Errorf("--tail must be greater than or equal to -1") </ins> } return nil

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When attempting to get all log lines using only 10 lines of logs are returned What you expected to happen: All log lines should be returned How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: This was introduced in in an attempt to fix another issue where selectors would ignore tail values !=-1. I think the problem here is the logic cannot tell if the was set explicitly by the command line or is simply the initial default value. Environment: Kubernetes version (use ): 1.13.3 Cloud provider or hardware configuration: Kubeadm OS (e.g: ): 18.04.2 LTS (Bionic Beaver) Kernel (e.g. ): Linux ubuntu 4.15.0-45-generic -Ubuntu SMP Tue Jan 29 16:28:13 UTC 2019 x8664 x8664 x86_64 GNU/Linux Install tools: Others:
/sig cli
Maybe is not a bug. See the kubectl help. --tail=-1: Lines of recent log file to display. Defaults to -1 with no selector, showing all log lines otherwise 10, if a selector is provided. I think i mean a selector is provided the --tail should not use -1 ?
Thanks for you reply. There are some background in below tickets, --tail=-1 is one of the option. Please check.
The help indicates that the default is 10 if the selector is used. But if someone explicitly asks for -1 it should still work, at least in my opinion.
Encountering similar behaviour to this myself. Is there any chance whatever fixes get applied can also be applied to the branch? Since that also has strange behaviour around logs with label selectors and setting .

See
This ticket has two issues: First, It's said "TailLines must be greater than or equal to 0", however we can specify -1. kubectl logs -l app=test --tail=-3 error: TailLines must be greater than or equal to 0 kubectl logs -l app=test --tail=-1 : Wed Apr 10 06:51:34 UTC 2019 : Wed Apr 10 06:51:35 UTC 2019 : Wed Apr 10 06:51:36 UTC 2019 : Wed Apr 10 06:51:37 UTC 2019 : Wed Apr 10 06:51:38 UTC 2019 : Wed Apr 10 06:51:39 UTC 2019 : Wed Apr 10 06:51:40 UTC 2019 : Wed Apr 10 06:51:41 UTC 2019 : Wed Apr 10 06:51:42 UTC 2019 : Wed Apr 10 06:51:43 UTC 2019 Second, --tail=-1 is the legacy usage which show all lines, here just return 10 lines. kubectl logs -l app=test --tail=-1 The behavior after fix: Test cases ./kubectl logs -l app=test --tail=-3 ./kubectl logs -l app=test --tail=-1 ./kubectl logs -l app=test --tail=2 ./kubectl logs -l app=test --tail=0
Interesting. I think this also causes to fail: ^-- expected record ^-- some later record (there are only 2 pods running, so is not relevant) ^-- adding makes it work
A bit strange that it finds records a few seconds older as well.. but that's not a big concern, and not relevant for this issue. vs. Although that only happens when the workaround is used: :thinking: P.S.


Question: }, } <del> err := verifyRunAsNonRoot(pod, &pod.Spec.Containers[0], int64(0)) assert.NoError(t, err) runAsUser := types.UnixUserID(0) RunAsNonRoot := false podWithContainerSecurityContext := &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ UID:       "12345678", Name:      "bar", Namespace: "new", </del> <ins> rootUser := types.UnixUserID(0) runAsNonRootTrue := true runAsNonRootFalse := false imageRootUser := int64(0) imageNonRootUser := int64(123) for _, test := range []struct { desc      string sc        *v1.SecurityContext imageUser int64 fail      bool }{ { desc:      "Pass if SecurityContext is not set", sc:        nil, imageUser: imageRootUser, fail:      false, </ins> }, <del> Spec: v1.PodSpec{ Containers: []v1.Container{ { Name:            "foo", Image:           "busybox", ImagePullPolicy: v1.PullIfNotPresent, Command:         []string{"testCommand"}, WorkingDir:      "testWorkingDir", SecurityContext: &v1.SecurityContext{ RunAsNonRoot: &RunAsNonRoot, RunAsUser:    &runAsUser, }, }, </del> <ins> { desc: "Pass if RunAsNonRoot is not set", sc: &v1.SecurityContext{ RunAsUser: &rootUser, </ins> }, <ins> imageUser: imageRootUser, fail:      false, </ins> }, <del> } err2 := verifyRunAsNonRoot(podWithContainerSecurityContext, &podWithContainerSecurityContext.Spec.Containers[0], int64(0)) assert.EqualError(t, err2, "container's runAsUser breaks non-root policy") RunAsNonRoot = false podWithContainerSecurityContext = &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ UID:       "12345678", Name:      "bar", Namespace: "new", </del> <ins> { desc: "Pass if RunAsNonRoot is false (image user is root)", sc: &v1.SecurityContext{ RunAsNonRoot: &runAsNonRootFalse, }, imageUser: imageRootUser, fail:      false, </ins> }, <del> Spec: v1.PodSpec{ Containers: []v1.Container{ { Name:            "foo", Image:           "busybox", ImagePullPolicy: v1.PullIfNotPresent, Command:         []string{"testCommand"}, WorkingDir:      "testWorkingDir", SecurityContext: &v1.SecurityContext{ RunAsNonRoot: &RunAsNonRoot, }, }, </del> <ins> { desc: "Pass if RunAsNonRoot is false (RunAsUser is root)", sc: &v1.SecurityContext{ RunAsNonRoot: &runAsNonRootFalse, RunAsUser:    &rootUser, </ins> }, <ins> imageUser: imageNonRootUser, fail:      false, </ins> }, <ins> { desc: "Fail if container's RunAsUser is root and RunAsNonRoot is true", sc: &v1.SecurityContext{ RunAsNonRoot: &runAsNonRootTrue, RunAsUser:    &rootUser, }, imageUser: imageNonRootUser, fail:      true, }, { desc: "Fail if image's user is root and RunAsNonRoot is true", sc: &v1.SecurityContext{ RunAsNonRoot: &runAsNonRootTrue, }, imageUser: imageRootUser, fail:      true, }, } { pod.Spec.Containers[0].SecurityContext = test.sc err := verifyRunAsNonRoot(pod, &pod.Spec.Containers[0], int64(0)) if test.fail { assert.Error(t, err, test.desc) } else { assert.NoError(t, err, test.desc) } </ins> } <del> err3 := verifyRunAsNonRoot(podWithContainerSecurityContext, &podWithContainerSecurityContext.Spec.Containers[0], int64(0)) assert.EqualError(t, err3, "container has runAsNonRoot and image will run as root") </del> }

What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): kubelet securitycontext Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): 1.7.0-beta/1.8.0 (HEAD) Environment: Cloud provider or hardware configuration: gce OS (e.g. from /etc/os-release): gci Kernel (e.g. ): Install tools: Others: docker runtime (rkt seems to have a different code path, but I haven't checked it) What happened: containers could not run as root when "RunAsNonRoot=false" What you expected to happen: containers could run as root How to reproduce it (as minimally and precisely as possible): add a securitycontext to a container, don't set privileged, do set RunAsNonRoot, use an image that runs as root. The PodSecurityPolicy admission controller will always set a securitycontext if it's not specified, and cause this to be hit more often. Anything else we need to know: The bug is in the checks here: I have a fix in but that going in depends on an extension being granted. We may want to fix the bug regardless.
/sig node
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
I'll send a fix shortly.


Question: return nil } <ins> overwrite := cmdutil.GetFlagBool(cmd, "overwrite") </ins> helper := resource.NewHelper(info.Client, info.Mapping) <del> patcher := NewPatcher(encoder, decoder, info.Mapping, helper) </del> <ins> patcher := NewPatcher(encoder, decoder, info.Mapping, helper, overwrite) </ins> patchBytes, err := patcher.patch(info.Object, modified, info.Source, info.Namespace, info.Name) if err != nil {

This issue tracks the implementation of the --overwrite flag in kubectl apply, per in .
Changed the title to address only the --overwrite flag, since deals with the --force flag.
progress on this issue is blocked waiting for to merge.


Question: // Total number of available pods (ready for at least minReadySeconds) targeted by this statefulset. // This is a beta field and enabled/disabled by StatefulSetMinReadySeconds feature gate. <ins> // +optional </ins> AvailableReplicas int32 `json:"availableReplicas" protobuf:"varint,11,opt,name=availableReplicas"` }

Looking at the for , it defines as a required attribute. But looking at the of , it states . Should this in fact be a required attribute? I come about this from using , a python package that generates models from the swagger definitions and thus builds a that requires an attribute that is not enabled. That only truly required attributes be required in the swagger definition Reproduction is not applicable as this is an issue with a spec No response Appeared in 1.22 (I believe) and affects current versions. Does not affect 1.21 n/a No response No response No response No response
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
alpha: has that is planned for removal in beta. ( results in fields as non-required in the OpenAPI) beta: removes the this might be a question whether feature gate controlled fields must remain until the feature gate is removed (and how would validation work in that case). ~old feature gates graduations to beta enabled a feature by default.~ a topic related to ? but it does not apply to feature gates. /kind documentation /sig apps
cc (since you are the author)
new fields to existing types must always be marked as +optional, since clients are not required to specify them
please open a PR adding back the +optional tag, regenerate openapi, and open picks back to 1.23 and 1.22


Question: channel <- podUpdate expectPodUpdate(t, ch, CreatePodUpdate(kubetypes.ADD, TestSource, CreateValidPod("foo", "new"))) <del> // add an invalid update podUpdate = CreatePodUpdate(kubetypes.UPDATE, TestSource, &v1.Pod{ObjectMeta: metav1.ObjectMeta{Name: "foo"}}) </del> <ins> // add an invalid update, pod with the same name podUpdate = CreatePodUpdate(kubetypes.ADD, TestSource, CreateValidPod("foo", "new")) </ins> channel <- podUpdate expectNoPodUpdate(t, ch) }

Which test(s) are flaking: TestInvalidPodFiltered Reason for failure: This is flaking rarely enough that it is not caught by our CI jobs, which currently tolerate up to 2 unit test failures per run (!). With that toleration removed in , this flake has been seen (). Reproducible with the following steps: a test binary for the affected package: the affected test using the tool: failures are seen, look at the logs to see the details about the error: /sig node cc
/assign
It is a bit funny that if I change to the following(wait for 3 seconds until receive event from channel): The test would consistently fail. I am not sure what the invalid update means at , so I cannot think of an ideal fix. I would like to defer to the members from sig-node. /unassign
Isn't this asserting a negative? I don't think it can work, especially not without a wait or some way of forcing the config mux storage merger thingy to run. The multiple levels is kind of mindbending, really. I come to the same conclusion as Maybe the focus needs to be on the invalid update? I do not understand why it is invalid from the context. Tracing it backwards in time, v1.8 was the last time this passed. v1.9 changed what validation means, I think I have a solution. /assign
Thanks for going deep down the commit history and found out the problem !
Made if either of you wish to review, and confirm my understanding.


Question: persister: persister, } <del> return cache.setClient(issuer, clientID, provider), nil </del> <ins> return cache.setClient(clusterAddress, issuer, clientID, provider), nil </ins> } type oidcAuthProvider struct {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: We using go-code kubectl cli package. We are a long running app serving to many different Kubernetes clusters. Each cluster has a different user credentials and bearer token to access it. But they all use the same issuer and clientid. The oidc cache is keyed off of these two only. So what happens is when we switch to another configuration it pulls in the oidc provider for the other cluster and that bearer token does not have authority to this other cluster. The cluster config has the correct info in it but the oidc provider ignores that and used the stale data. What you expected to happen: oidc provider would always use the appropriate current config context to access the cluster How to reproduce it (as minimally and precisely as possible): In same app instance create two cluster configs to two different clusters with same oidc issuer and clientid by DIFFERENT credentials. Where the credentials on one are not authorized for the other cluster. Try to get namespaces on each, one will work and the other won't. Anything else we need to know?: Environment: Kubernetes version (use ): latest go-code from as a package called from go application.Cloud provider or hardware configuration: IBM Kubernetes OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten


Question: } else { if _, ok := err.(*core.FitError); ok { glog.V(4).Infof("Unable to schedule %v %v: no fit: %v; waiting", pod.Namespace, pod.Name, err) <ins> } else if errors.IsNotFound(err) { if errStatus, ok := err.(errors.APIStatus); ok && errStatus.Status().Details.Kind == "node" { nodeName := errStatus.Status().Details.Name // when node is not found, We do not remove the node right away. Trying again to get // the node and if the node is still not found, then remove it from the scheduler cache. _, err := factory.client.CoreV1().Nodes().Get(nodeName, metav1.GetOptions{}) if err != nil && errors.IsNotFound(err) { node := v1.Node{ObjectMeta: metav1.ObjectMeta{Name: nodeName}} factory.schedulerCache.RemoveNode(&node) // invalidate cached predicate for the node if factory.enableEquivalenceClassCache { factory.equivalencePodCache.InvalidateAllCachedPredicateItemOfNode(nodeName) } } } </ins> } else { glog.Errorf("Error scheduling %v %v: %v; retrying", pod.Namespace, pod.Name, err) }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: In case scheduler misses a node delete event, or cleaning the node from scheduler cache errors out, node will stay in scheduler cache forever and scheduler will try to schedule pods on it and keeps seeing an error similar to: E1123 00:00:00 7 ] Error scheduling namespace pod-1: node 'node-xyz' not found; retrying When scheduler sees these "not found" errors for a node, it should delete the node from its cache. In order to tolerate transient errors, it is better if it deletes the node when it keeps seeing this error multiple times and over some period of time. /sig scheduling
I can help fix it.
In all my Kubernetes controllers I back up watch handlers with a full scan every now and than to reconciliate as a safety net. I guess that would help with thess kind of errors as well. Thanks for looking into this.
Thanks for the suggestion. That sounds like a better approach. I think we should have periodic full scan to deal with similar scenarios. We should probably do it for all cached objects, not just nodes.
Sorry, I do not know that how we can do periodic full scan because seems has not function. and it consumes a lot of resources for doing periodic full scan also.
Okay, just removing the node from the cache - like you suggested - is then probably the most straightforward way to do it.
I also suspected that periodic resync would be prohibitively expensive, but I don't have any concrete numbers. If it is not acceptable, as said, we can stick to the first solution of removing the node when scheduler faces a "node not found" error. We should not remove the node right away. I'd suggest trying again to get the node and if the node is still not found, then remove it from the scheduler cache. Doing the same thing for Pods would be useful too, but that would be a separate PR.
Closing from merge of


Question: to pull the images from. Sample file: ```yaml <del> dockerLibraryRegistry: your-awesome-registry </del> e2eRegistry: your-awesome-registry promoterE2eRegistry: your-awesome-registry gcRegistry: your-awesome-registry

Part of umbrella to migrate kubernetes e2e test images/registries to community-owned infrastructure: The following images are used by 1.20 e2e tests We should migrate them to community owned infrastructure (ref: for deciding how to implement a mirror registry) We should consider doing this for release branches as well, which may use different images/tags. So these wouldn't necessarily be cherry-picks Two possible approaches: change the registry in source (I would prefer eliminating the extra "docker library" registry, and consolidate to a single e2e test image registry) start running tests with a config that swaps out for some other registry /sig testing /sig release /area release-eng /wg k8s-infra /milestone v1.21
/kind cleanup
/triage accepted /help cc: Give a shout if any of you are interested in working on this!
I'm in! /assign
/remove-help
cc
/cc
I've had 2 PRs opened for this, before it was decited that the test jobs would use the registry mirror configuration. Has there been any consensus on the approach? If so, I can un-abandon these PRs if necessary:
ooh thanks for sharing I don't think there has been any consensus, though I personally think your work on kubernetes/kubernetes is relevant, especially that you've covered everything except gluster and redis.
We already have the redis image here [1], we'll just have to trigger a job for it afterwards. Indeed, I was missing the gluster image, I only considered the image for it.
so I've been thinking about this for a bit and I'm a bit concerned with the visibility of this setup to test authors -- I'm not sure if this is straightforward / intuitive enough for them to follow. alternatively, are there other lighter (in terms of maintenance) solutions that we can pursue?
Hm, what do you mean by the visibility of this setup of test authors? If I'm not mistaken, the current goal is to not rely so heavily on dockerhub for testing anymore. For this purpose we are mirroring these images into our own registries, and then we are going to update to use our own registry instead of the dockerhub one. Test authors shouldn't be impacted by this, IMO. From their point of view, it's the same image, from a different source.

And in terms of maintenance... hm, techically, if someone would want to use an newer image, say , so it would be consistent.
yes, I think we're on the same page on this yeah what I'm wondering is: are there ways we make it even less significant work on the first step of modifying and files? if that step could disappear altogether, especially in scenario of adding new images, that would be great.
I don't really see how. Other images are doing something similar too. For example, the pause image also has its BASEIMAGEs and VERSION in its Makefile. Not all images are built exactly the same, or have the same base image, or are built for the same arch types. Some of them are from scratch, others are from an alpine image, others are from a different alpine image, others are from a debian image... and then we also have windows base images. So, having those BASEIMAGE files addresses this issue. Plus, it also allows us to control what platforms we build for and what we test for. For example, the s390x support wasn't there from the start, it was afterwards. Another example, we want to add the base images at some point. Anyways, modifying the BASEIMAGE files is not very common, but when it is, there's a good reason behind it, so it can't really be optional. Plus, adding / modifying the BASEIMAGE and VERSION files is not significant work. For example, mirroring the 7 images in this PR [1] probably took me ~10 minutes or so. From what I can tell, you want some fast way to just say: "mirror me this image entirely onto our registry". I don't know if we have something like that, and if we do, it'll probably also mirror images with arch types we don't need, and most likely won't have images for platforms we actually need (for example, the ` images on dockerhub does not have any Windows images, but we need them). The test images building process for the test images was not designed for mirroring images, but it can easily do it and we can use it to do so. In addition to that, we can even extend those manifest lists with Windows support [2]. [1] [2]
oh right, apologies for the confusion.

I was referring to having a Dockerfile as a whole, but your explanation is very helpful for me to gain more context about image usage in our infrastructure -- thank you for elaborating I agree the what isn't significant, but the where worries me a bit. I guess in some ways I kind of wish that there is way for this work to simply exist in without touching k/k proper, though now I'm not too sure if that would be overall less complex than this approach. yeah, I was trying to find more solutions towards this direction though I haven't found anything significant which aligns with this topic.
/priority important-soon
In order to complete this, we are going to need these 2 trivial PRs: Retrigger image builder jobs for some images: Small fix for Windows httpd images:
seems like much of the work has been done by Claudiu or I do not have access to finish -- I'll unassign myself to keep this aligned with the state of things /unassign
/assign who has done a lot of the work for this


Question: return nil, fmt.Errorf("Error creating LB floatingip %+v: %v", floatIPOpts, err) } } <ins> status := &v1.LoadBalancerStatus{} </ins> if floatIP != nil { <del> status.Ingress = append(status.Ingress, v1.LoadBalancerIngress{IP: floatIP.FloatingIP}) </del> <ins> status.Ingress = []v1.LoadBalancerIngress{{IP: floatIP.FloatingIP}} } else { status.Ingress = []v1.LoadBalancerIngress{{IP: loadbalancer.VipAddress}} </ins> } if lbaas.opts.ManageSecurityGroups {

When creating a service passing the annotation , two IPs are created in the service, a private and a public: Having a private IP is not a desired scenario for service discovery in federated global environments. If we have, for example, a cluster in GKE and join a local OpenStack cluster, this private IP is written in the DNS zone but the GKE cluster won't be able to access it. We could add an annotation to say we don't want the private IP to be registered in service's LoadBalancer Ingress field, like: or And when it's false, only the public IP would be written in the service, therefore in the DNS zone /kind feature
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig openstack /sig federation
'' sounds better.
I don't like adding annotations just to make things work. I counter-propose just not publishing the private IP whenever there is also a floating-ip (without a configuration option to control this behaviour).


Question: }, }, }, <ins> VendorExtensible: spec.VendorExtensible{ Extensions: spec.Extensions{ "x-kubernetes-map-type": "atomic", }, }, </ins> }, } }

I created an Ingress resource with a port number manually, and then tried to use an Operator with server-side apply to update the resource from a port number to a port name. This failed with the error message "cannot set both port name & port number". I expected the port number to be cleared as the operator would take over the ownership of the ServiceBackendPort. To simulate the behavior in the operator in a most simple example I use : ingress- ingress- This will print: It seems to me that the merge type for the struct ServiceBackendPort should to be set to "atomic" , so that the Operator updating the port should take over the whole struct, and two applications shouldn't update single fields in it. Red Hat OpenShift on AWS OS version of the client: NAME="Fedora Linux" VERSION="40 (Workstation Edition)" Linux 6.9.6-200.fc40.x8664 SMP PREEMPTDYNAMIC Fri Jun 21 15:48:21 UTC 2024 x86_64 GNU/Linux
/sig network
So, on one hand you made a problem when you used two different owners to manage one aspect of the object. On the other hand, I think you're right that it SHOULD work. Probably. is not a super common tag, and ingress way predates it, so no surprise it isn't used. I ACK the issue and reproduced it. I ran update-codegen and built a KinD cluster from HEAD.
this may be something you want to look at for DRA API - I didn't even realize this was a thing until today :)
The API uses for , which is very similar. So using it for LGTM. SSA will auto-migrate existing resources using once the change is released.
/triage accepted
"this" being "add "? I can see where this might be useful, probably on all one-of structs and perhaps on those containing a discriminator because otherwise SSA leaves fields set which should be cleared when some other owner takes over. Is this important enough to put into I don't see a big need for SSA in general and even less for SSA with different owners at the moment - this will change once we do more with status.
No, not urgent. This is the first time it has come up in my awareness ever. But now that I know about it, I could see how it applies to dra.

Thank to everyone involved here to resolve this! This will make things simpler in the Keycloak operator once it is released.
Thanks for filing a good (and easy to fix :) bug. Easy repro means we can jump on it.


Question: package customresource_test import ( <ins> "context" "fmt" </ins> "reflect" "strings" "testing"

What happened: Using a command such as: fails with the following error: This bug happens on each run and started occurring as of v1.15.0. With v1.14 or older, I don't have such problem, i.e. it works flawlessly. What you expected to happen: The command succeeds: How to reproduce it (as minimally and precisely as possible): Create a CRD that has the scale subresource and try to scale a CR using . Example of such CRD can be found . Anything else we need to know?: Regarding the bug, it seems that it started appearing as of . Before the PR, we always fetched the resource and it worked because it had the resource version, while now we are only fetching it in case preconditions are provided (preconditions are not provided by default/when using ). Here is the output of a command: If I manually craft the request that has the resource version (or use the flag), it works. Is it as expected that the Scale resource requires the resource version in order for update to succeed? Shouldn't it always pick the latest resource version in case it isn't provided? On that note, I also noticed that if you want to update a CR by crafting the request manually and using , you also need to specify the resource version or you will get the same error. I'm not sure if this is expected and related, but I'm writing it down as it might be related. As far as I know, this is not required for some native resources at all, but I'm not sure is this the case for CRDs. Environment: Kubernetes version (use ):
/sig api-machinery cc
/sub
could you try to see if it fix the problem?
I've created a test cluster based on and I can confirm that I'm able to scale CustomResources using as usual. I used an to test this, such as: Thanks a lot for your time spent fixing this!
Bug triage for 1.17 here with a gentle reminder that code freeze for this release is on November 14. Is this issue still intended for 1.17? I see you move it to 1.17 so I just wanted to be sure.
I guess could be merged now?

It is in the review queue and should still make 1.17
a kind reminder that code freeze for this release is on November 14.
I wanted to circle back around on this. We are 4 days out from code freeze. If this is still good based on just note here. Judging from the looks of it we are still good
Yes this should merge early in the week
Code freeze is tomorrow, this still going to make it?
yes, is still planned


Question: waitForDetachCallCount(t, 0 /* expectedDetachCallCount */, fakePlugin) } <ins> // Populates desiredStateOfWorld cache with one node/volume/pod tuple. // The node starts as healthy. // // Calls Run() // Verifies there is one attach call and no detach calls. // Deletes the pod from desiredStateOfWorld cache without first marking the node/volume as unmounted. // Verifies that the volume is NOT detached after maxWaitForUnmountDuration. // Marks the node as unhealthy. // Verifies that the volume is detached after maxWaitForUnmountDuration. func Test_Run_OneVolumeDetachOnUnhealthyNode(t *testing.T) { // Arrange volumePluginMgr, fakePlugin := volumetesting.GetTestVolumePluginMgr(t) dsw := cache.NewDesiredStateOfWorld(volumePluginMgr) asw := cache.NewActualStateOfWorld(volumePluginMgr) fakeKubeClient := controllervolumetesting.CreateTestClient() fakeRecorder := &record.FakeRecorder{} fakeHandler := volumetesting.NewBlockVolumePathHandler() ad := operationexecutor.NewOperationExecutor(operationexecutor.NewOperationGenerator( fakeKubeClient, volumePluginMgr, fakeRecorder, fakeHandler)) informerFactory := informers.NewSharedInformerFactory(fakeKubeClient, controller.NoResyncPeriodFunc()) nsu := statusupdater.NewFakeNodeStatusUpdater(false /* returnError */) nodeLister := informerFactory.Core().V1().Nodes().Lister() reconciler := NewReconciler( reconcilerLoopPeriod, maxWaitForUnmountDuration, syncLoopPeriod, false, dsw, asw, ad, nsu, nodeLister, fakeRecorder) podName1 := "pod-uid1" volumeName1 := v1.UniqueVolumeName("volume-name1") volumeSpec1 := controllervolumetesting.GetTestVolumeSpec(string(volumeName1), volumeName1) nodeName1 := k8stypes.NodeName("worker-0") node1 := &v1.Node{ ObjectMeta: metav1.ObjectMeta{Name: string(nodeName1)}, Status: v1.NodeStatus{ Conditions: []v1.NodeCondition{ { Type:   v1.NodeReady, Status: v1.ConditionTrue, }, }, }, } informerFactory.Core().V1().Nodes().Informer().GetStore().Add(node1) dsw.AddNode(nodeName1, false /*keepTerminatedPodVolumes*/) volumeExists := dsw.VolumeExists(volumeName1, nodeName1) if volumeExists { t.Fatalf( "Volume %q/node %q should not exist, but it does.", volumeName1, nodeName1) } generatedVolumeName, podErr := dsw.AddPod(types.UniquePodName(podName1), controllervolumetesting.NewPod(podName1, podName1), volumeSpec1, nodeName1) if podErr != nil { t.Fatalf("AddPod failed. Expected: <no error> Actual: <%v>", podErr) } // Act ch := make(chan struct{}) go reconciler.Run(ch) defer close(ch) // Assert waitForNewAttacherCallCount(t, 1 /* expectedCallCount */, fakePlugin) verifyNewAttacherCallCount(t, false /* expectZeroNewAttacherCallCount */, fakePlugin) waitForAttachCallCount(t, 1 /* expectedAttachCallCount */, fakePlugin) verifyNewDetacherCallCount(t, true /* expectZeroNewDetacherCallCount */, fakePlugin) waitForDetachCallCount(t, 0 /* expectedDetachCallCount */, fakePlugin) // Act // Delete the pod and the volume will be detached even after the maxWaitForUnmountDuration expires as volume is // not unmounted and the node is healthy. dsw.DeletePod(types.UniquePodName(podName1), generatedVolumeName, nodeName1) time.Sleep(maxWaitForUnmountDuration * 5) // Assert waitForNewDetacherCallCount(t, 0 /* expectedCallCount */, fakePlugin) verifyNewAttacherCallCount(t, false /* expectZeroNewAttacherCallCount */, fakePlugin) waitForAttachCallCount(t, 1 /* expectedAttachCallCount */, fakePlugin) verifyNewDetacherCallCount(t, true /* expectZeroNewDetacherCallCount */, fakePlugin) waitForDetachCallCount(t, 0 /* expectedDetachCallCount */, fakePlugin) // Act // Mark the node unhealthy node2 := node1.DeepCopy() node2.Status.Conditions[0].Status = v1.ConditionFalse informerFactory.Core().V1().Nodes().Informer().GetStore().Update(node2) // Assert -- Detach was triggered after maxWaitForUnmountDuration waitForNewDetacherCallCount(t, 1 /* expectedCallCount */, fakePlugin) verifyNewAttacherCallCount(t, false /* expectZeroNewAttacherCallCount */, fakePlugin) waitForAttachCallCount(t, 1 /* expectedAttachCallCount */, fakePlugin) verifyNewDetacherCallCount(t, false /* expectZeroNewDetacherCallCount */, fakePlugin) waitForDetachCallCount(t, 1 /* expectedDetachCallCount */, fakePlugin) } </ins> func Test_ReportMultiAttachError(t *testing.T) { type nodeWithPods struct { name     k8stypes.NodeName

A/D controller tries to detach volumes that are still mounted after 6 minutes. The conditions to trigger the detach are: I.e. all containers are finished. But that does not guarantee that volumes are unmounted. When a CSI driver can't unmount a volume for 6 minutes, the volume is force detached by A/D controller, corrupting the volume and leaving volume mounts on the node in a bad state. Note that the node is perfectly healthy, it just can't unmount a volume from whatever reason. A/D controller detaches volumes after a Pod is deleted from the API server, i.e. the volumes are unmounted or someone force-deleted the pods. Very artificial reproducer: a Pod with an attachable CSI volume. the node, make the volume mount busy, e.g.: Every unmount of the volume fails and kubelet/the CSI driver periodically retry. the pod 6 minutes, see the volume is detached without unmounting first. KCM logs: Note that the Pod is still in the API server: And the node is healthy: Note that some CSI drivers (e.g. the cloud ones) do not allow detaching a volume while it's still mounted. Not all CSI driver have such possibility and they rely on Kubernetes not to call ControllerUnpublish before NodeUnpublish / NodeUnstage succeeds. <detailsToday's master </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage /priority important-longterm
Right now, a volume is detached from a node when: (all pods that use the volume are deleted from the API server OR all their container are stopped) AND (the volume is unmounted OR 6 minutes passed). Potential solution 1: Issue detach of a mounted volume only when all pods are force-deleted from the API server.

More formally: Detach when [(All pods that use the volume are deleted from the API server OR all their container are stopped) AND the volume is unmounted] OR [all pods that use the volume are deleted from the API server AND 6 minutes passed]. Is this change subtle enough that it does not break existing behavior (e.g. tooling on top of Kubernetes)?
Potential solution 2: Add a new flag to CSIDriver, say . We would need to extend internal Detach() call to include a flag if the volume to detach is mounted or not (A/D controller knows that) and the CSI volume plugin would check CSIDriver and potentially reject detach of a mounted volume. This way, it would not be possible to ever force detach a mounted volume, even when a pod is force-deleted! Potential solution 3: As a variant, we could add a cmdline option to the external-attacher, like . When that cmdline option is specified, the external attacher would check , if the volume to detach is still mounted, and refuse such detach (with a reasonable error message). CSI driver vendors could opt-in into this behavior. This way, it would not be possible to ever force detach a mounted volume, even when a pod is force-deleted! Potential solution 4: As a variant, we could add to CSIDriver, where each driver could put their own timeout before force-detaching a volume, with an option to use infinity.
Note to self: check , it might help on the CSI side.
Solution 3 will not work well with ungraceful node shutdown - attach/detach controller does not watch pods and force-deleting pods is the main point of ungraceful node shutdown KEP. /me will try if solution 1 would work when NodeStage times out (is Pod deleted from the API server before or after NodeUnstage is called?)
was right, a pod is deleted (without force) from the API server after NodeUnpublish succeeds, it does not wait for NodeUnstage to finish. Therefore we cannot distinguish if a pod was force-deleted (and we should start 6 minute timer) or pod was deleted without any force and only NodeUnstage is taking more time than expected. Solution 1 is then not possible.
Now we are down to solution 2 and 4.

For solution 2, if driver does not have a way to check whether volume is mounted or not, the behavior will not change and we still force detach after 6 mins. And it might happen in the middle of unstage ..
Summing up yesterday's call: A/D controller should check if a node is unhealthy before force detaching volumes from it. It should never force detach a volume from a healthy node, not even after 6 minutes. Question what healthy means is open and we should talk to sig-node. is a working assumption.
I agree with both of these. I'd just emphasize that that particular definition of unhealthy definitely includes perfectly healthy nodes in a network partition. This change is no worse than the status quo ante for those nodes, and a definite improvement on the healthy nodes, so I wholeheartedly support it.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/remove-lifecycle rotten
/assign


Question: // We expect an error here since it's a cannot connect test. // Dump debug information if the error was nil. if err == nil { <del> // Collect pod logs when we see a failure. logs, logErr := e2epod.GetPodLogs(f.ClientSet, f.Namespace.Name, podClient.Name, "client") if logErr != nil { framework.Failf("Error getting container logs: %s", logErr) } </del> <ins> pods, policies, logs := collectPodsAndNetworkPolicies(f, podClient) framework.Failf("Pod %s should not be able to connect to service %s, but was able to connect.nPod logs:n%snn Current NetworkPolicies:nt%vnn Pods:nt %vnn", podClient.Name, service.Name, logs, policies.Items, pods) </ins> <del> // Collect current NetworkPolicies applied in the test namespace. policies, err := f.ClientSet.NetworkingV1().NetworkPolicies(f.Namespace.Name).List(context.TODO(), metav1.ListOptions{}) if err != nil { framework.Logf("error getting current NetworkPolicies for %s namespace: %s", f.Namespace.Name, err) } </del> <ins> // Dump debug information for the test namespace. framework.DumpDebugInfo(f.ClientSet, f.Namespace.Name) } } </ins> <del> // Collect the list of pods running in the test namespace. podsInNS, err := e2epod.GetPodsInNamespace(f.ClientSet, f.Namespace.Name, map[string]string{}) if err != nil { framework.Logf("error getting pods for %s namespace: %s", f.Namespace.Name, err) </del> <ins> func checkNoConnectivityByExitCode(f *framework.Framework, ns *v1.Namespace, podClient *v1.Pod, service *v1.Service) { err := e2epod.WaitForPodCondition(f.ClientSet, ns.Name, podClient.Name, "terminated", framework.PodStartTimeout, func(pod *v1.Pod) (bool, error) { statuses := pod.Status.ContainerStatuses if len(statuses) == 0 || statuses[0].State.Terminated == nil { return false, nil </ins> } <del> pods := []string{} for _, p := range podsInNS { pods = append(pods, fmt.Sprintf("Pod: %s, Status: %sn", p.Name, p.Status.String())) </del> <ins> if statuses[0].State.Terminated.ExitCode != 0 { return true, fmt.Errorf("pod %q container exited with code: %d", podClient.Name, statuses[0].State.Terminated.ExitCode) </ins> } <del> framework.Failf("Pod %s should not be able to connect to service %s, but was able to connect.nPod logs:n%snn Current NetworkPolicies:nt%vnn Pods:nt %vnn", podClient.Name, service.Name, logs, policies.Items, pods) </del> <ins> return true, nil }) // We expect an error here since it's a cannot connect test. // Dump debug information if the error was nil. if err == nil { pods, policies, logs := collectPodsAndNetworkPolicies(f, podClient) framework.Failf("Pod %s should not be able to connect to service %s, but was able to connect.nPod logs:n%snn Current NetworkPolicies:nt%vnn Pods:nt%vnn", podClient.Name, service.Name, logs, policies.Items, pods) </ins> // Dump debug information for the test namespace. framework.DumpDebugInfo(f.ClientSet, f.Namespace.Name) } } <ins> func collectPodsAndNetworkPolicies(f *framework.Framework, podClient *v1.Pod) ([]string, *networkingv1.NetworkPolicyList, string) { // Collect pod logs when we see a failure. logs, logErr := e2epod.GetPodLogs(f.ClientSet, f.Namespace.Name, podClient.Name, "client") if logErr != nil { framework.Failf("Error getting container logs: %s", logErr) } // Collect current NetworkPolicies applied in the test namespace. policies, err := f.ClientSet.NetworkingV1().NetworkPolicies(f.Namespace.Name).List(context.TODO(), metav1.ListOptions{}) if err != nil { framework.Logf("error getting current NetworkPolicies for %s namespace: %s", f.Namespace.Name, err) } // Collect the list of pods running in the test namespace. podsInNS, err := e2epod.GetPodsInNamespace(f.ClientSet, f.Namespace.Name, map[string]string{}) if err != nil { framework.Logf("error getting pods for %s namespace: %s", f.Namespace.Name, err) } pods := []string{} for _, p := range podsInNS { pods = append(pods, fmt.Sprintf("Pod: %s, Status: %sn", p.Name, p.Status.String())) } return pods, policies, logs } </ins> // Create a server pod with a listening container for each port in ports[]. // Will also assign a pod label with key: "pod-name" and label set to the given podName for later use by the network // policy.

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: The NetworkPolicy e2e test fails regardless of which CNI plugin is enabled. The test assumes the client Pod can recheck connectivity after updating its label, but the client won't restart after the first failure, so the second check will always fail. What you expected to happen: The test should succeed if a CNI plugin implements NetworkPolicy correctly. How to reproduce it (as minimally and precisely as possible): a CNI that supports NetworkPolicy. the e2e test by focusing "should allow ingress access from updated pod". test will always fail. Anything else we need to know?: Environment: Kubernetes version (use ): 16.3 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig testing /assign
/remove-sig testing /sig network
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/remove-triage unresolved
I see this in Calico 3.10 as well confirming that its definitely an issue.


Question: } { t.Run(tc.desc, func(t *testing.T) { set := burst(newStatefulSet(0)) <ins> set.Spec.VolumeClaimTemplates[0].ObjectMeta.Labels = map[string]string{"test": "test"} </ins> parallelScale(t, set, tc.replicas, tc.desiredReplicas, assertBurstInvariants) }) }

The kube-controller-manage may crash when processing StatefulSet with spec.podManagementPolicy is Parallel. With logs: we expect the kube-controller-manage will work well. The following example can reproduce this problem: After digging into the source code, we found that when using the pod management policy, the statefulset controller will use to start parallel processing. But the which is called during parallel processing may directly update the claims' labels without a lock. This finally causes a concurrent map-accessing problem. <details</details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig apps
Thanks for the report. Do we know when this issue was introduced? cc
Maybe it was imported by this pr
We can reproduce this issue in unit testing.
The reported versions is and it seems that PR got backported further /priority important-soon
/assign
The original report looks like could have introduced it, yes. I wonder if we intentionally modify template without creating a copy (and as it seems, assign the exact same labels every time we use this template for any pod): Regardless, that was probably safe before the pod creations were parallelized. I don't think the proposed unit test catches the bug (more on that below), but we can probably reproduce it by timing pod state transitions so that many pods are simultaneously pending and missing a PVC. I ran this test on (before the cherry-pick of parallel statefulset scaling), and it also failed. This is because the test itself introduces a concurrent map write. It probably doesn't replicate the issue, and I don't think it can be merged, as it'll fail even if the original PR is reverted. For reproduction we'd need to catch a concurrent write without adding one in test code, just as it happened based on the original report.
Turns out we never tested this path at all, since the test PVC didn't have any labels of its own.

After adding labels, existing tests are sufficient to detect race condition. PR showing that improved test detects the issue: PR with fix and improved test:
Sweet! thanks !


Question: assert.Equal(t, http.StatusOK, resp.StatusCode) } <ins> func TestTrimURLPath(t *testing.T) { tests := []struct { path, expected string }{ {"", ""}, {"//", ""}, {"/pods", "pods"}, {"pods", "pods"}, {"pods/", "pods"}, {"good/", "good"}, {"pods/probes", "pods"}, {"metrics", "metrics"}, {"metrics/resource", "metrics/resource"}, {"metrics/hello", "metrics/hello"}, } for _, test := range tests { assert.Equal(t, test.expected, trimURLPath(test.path), fmt.Sprintf("path is: %s", test.path)) } } </ins>

<!-- Please only use this template for submitting enhancement requests --What would you like to be : 1) add metrics into kubelet to monitor ; 2) add throttling to . Why is this needed: Kubelet provides lots of its functionalities through its http server, and currently there is no server-side monitoring and throttling mechanisms. Adding metrics into kubelet to monitor its http server will improve its observability and debuggability; adding throttling will prevent kubelet being overloaded from serving too many http requests.
/sig node


Question: return WaitConditionToBe(ctx, c, name, v1.NodeReady, true, timeout) } <ins> func WaitForNodeSchedulable(c clientset.Interface, name string, timeout time.Duration, wantSchedulable bool) bool { framework.Logf("Waiting up to %v for node %s to be schedulable: %t", timeout, name, wantSchedulable) for start := time.Now(); time.Since(start) < timeout; time.Sleep(poll) { node, err := c.CoreV1().Nodes().Get(context.TODO(), name, metav1.GetOptions{}) if err != nil { framework.Logf("Couldn't get node %s", name) continue } if IsNodeSchedulable(node) == wantSchedulable { return true } } framework.Logf("Node %s didn't reach desired schedulable status (%t) within %v", name, wantSchedulable, timeout) return false } </ins> // CheckReady waits up to timeout for cluster to has desired size and // there is no not-ready nodes in it. By cluster size we mean number of schedulable Nodes. func CheckReady(ctx context.Context, c clientset.Interface, size int, timeout time.Duration) ([]v1.Node, error) {

User cordons a node with a pvc bound to a local persistent volume User deletes a stateful set pod from that node The pod is in pending because the pvc is still bound to a local pv on the cordoned node (expected) The node dies and the pvc is deleted The pod is still stuck in pending because the pvc is missing (bug) The pod now needs to be manually deleted so that the pod+pvc can be recreated After the pvc was deleted, the statefulset controller should have recreated it and bound it to a new node's local pv so that the pending pod could be scheduled. Currently the only way to get the pod out of pending is to manually delete it so that it recreates itself and its pvc We used for the local PVs, but the problem is not limited to this particular provisioner create a stateful set with a pvc that is bound to a local PV use to cordon a node with a local pv delete a statefulset pod from that node delete the pvc from that node the pod is now stuck in pending due to a missing pvc There was a related issue but this only fixed the issue when the order of operations was delete pvc -delete pod. but in this case, it is delete pod -delete pvc <details</details<detailsTested on aws and gcp </details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig apps storage
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The Kubernetes project currently lacks enough contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
This issue was fixed by The new issues look different. /close
Closing this issue. <detailsIn response to : This issue was fixed by The new issues look different. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details