Question: }, Spec: batchv1.CronJobSpec{ Schedule:          "* * * * ?", <del> ConcurrencyPolicy: batchv1.AllowConcurrent, </del> <ins> ConcurrencyPolicy: "Allow", </ins> JobTemplate: batchv1.JobTemplateSpec{ ObjectMeta: metav1.ObjectMeta{ Labels:      map[string]string{"a": "b"},

in TestControllerV2SyncCronJob, there are many listing the properties that don't matter (like nil, false). and many constants F and T, which is unnecessary. As per 's request, We shouldn't be listing the properties that don't matter (like nil, false). And get rid of constants F and T check No response <details</details<detailsNone </details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign I would like to work on the optimization
It's not an optimization. It's just about readability.
/remove-kind bug /kind cleanup
/sig apps


Question: return "", err } <del> attached = true </del> <ins> attachEnded = true </ins> return hostDevice, nil }

Environment: CoreOS 815.0.0, k8s 1.1.0 () It happens to me quite often that a volume marked as "available" on AWS remains in a "detached" state according to the kubelet: Deleting the pod and recreating it eventually succeeds.
I have some fixes in progress that should fix this
/cc
I can reliably reproduce similar issue, I am not sure it's the same: create a AWS EBS, let's say it has ID Create a pod that uses this EBS: Create the pod. It succeeds eventually: Delete the pod Create the pod again. It will be forever IMO, it's caused by not removing the detached volume from internal map of attached volumes . Subsequent will find it there and it won't even try to attach it again. Patch is available in referenced PR.
can you please confirm that your bugs is the same as I can reproduce? Or do you see different bug, only with similar symptoms?
It is
Was this fixed by ?
I think you referenced the wrong issue / PR, didn't you? As far as I know this is not fixed yet.
Yes, I meant to link . mentioned there that it might fix this issue.
Oh right! It made it to v1.2.0-alpha.4
did not fix this issue (but fixed the DeviceMapping cache invalidation)
I think, what is happening is that you run into an error or timeout while waiting for the disk operation. That leaves the devicecache yet again in an inconsistent state. I have a fix here that just discards the cache on any error. Additionally, it is totally possible that there is concurrent operations which lead to an error and bork the cache. The fix: Unfortunately, this is in our tests still not 100% rock solid. Especially in timeout cases. I'm missing the capability to properly test this to engage in a bigger refactoring of this whole issue. A workaround to actually fixing it properly would be to add a TTL to the cache, so that it eventually starts working again.
Just for information for people who might use the GitHub search, as of Kubernetes 1.1.7 the errors logged by the look like: Deleting the pod or restarting doesn't help, the instance keeps failing attaching the EBS with the same errors if the pod get rescheduled on the same host.

thanks for the fix!
Oh... the same host. I bet I messed up that case... I'm going to move into 1.2 so I take another look at this!
What's the status on this?
I just don't think this is fixable with the volume cache we have. I proposed a PR to remove the cache, and replace it with a map of active attachment operations in progress. A little less efficient, but a lot more correct (presuming passes testing).
How to check what K8 version has this fix?
Is there any progress on this issue? I'm running K8s version v1.5.2, and I'm having similar issues. awsElasticBlockStore volumes are virtually unusable due to one of three errors I'm getting: My volume claim is ReadWriteOnce, so this next error confuses me: And finally: [SchedulerPredicates failed due to persistentvolumeclaims "mysql" not found, which is unexpected., SchedulerPredicates failed due to persistentvolumeclaims "mysql" not found, which is unexpected., SchedulerPredicates failed due to persistentvolumeclaims "mysql" not found, which is unexpected.] Unable to mount volumes for pod "staging-connect-mysql-<id(<id)": timeout expired waiting for volumes to attach/mount for pod "default"/"<id". list of unattached/unmounted volumes=[mysql] Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod "default"/"<id". list of unattached/unmounted volumes=[mysql]
Also having the same issue as ^ Should we be using PV+VPC instead of claiming the EBS volume directly from the deployment spec?


Question: "github.com/GoogleCloudPlatform/kubernetes/pkg/api" "github.com/GoogleCloudPlatform/kubernetes/pkg/client" <ins> "github.com/GoogleCloudPlatform/kubernetes/pkg/client/cache" "github.com/GoogleCloudPlatform/kubernetes/pkg/controller/framework" </ins> "github.com/GoogleCloudPlatform/kubernetes/pkg/fields" "github.com/GoogleCloudPlatform/kubernetes/pkg/labels" <ins> "github.com/GoogleCloudPlatform/kubernetes/pkg/runtime" </ins> "github.com/GoogleCloudPlatform/kubernetes/pkg/util" "github.com/GoogleCloudPlatform/kubernetes/pkg/util/wait" <ins> "github.com/GoogleCloudPlatform/kubernetes/pkg/watch" </ins> "github.com/golang/glog" . "github.com/onsi/ginkgo"

Currently the density tests are ok if all the pods come up eventually. We should verify there are no failure/error events while the pods are coming up. This could indicate issues dealing with the number of pods and should be investigated.
has been merged, and there's already an issue filed for followup actions, so I think we can close this.


Question: // before Remove event, in which case the state would change from Assumed to Added. case ok && !cache.assumedPods[key]: if currState.pod.Spec.NodeName != pod.Spec.NodeName { <del> glog.Errorf("Pod %v removed from a different node than previously added to.", key) </del> <ins> glog.Errorf("Pod %v was assumed to be on %v but got added to %v", key, pod.Spec.NodeName, currState.pod.Spec.NodeName) </ins> glog.Fatalf("Schedulercache is corrupted and can badly affect scheduling decisions") } err := cache.removePod(currState.pod)

/kind bug There are several error messages in the scheduler cache that are confusing to read, at least one of these surfaced up to end users via Events. I noticed these two in particular but there may be others: Whether or not these should actually be surfacing as events, the messages should be cleaned up.
cc
Seems that these are internal or implement details in scheduler, do our end users can fully understand what the events really mean? I think it is depend on our own implement. idea?
I believe, the problem is with in the message but I am not able to reproduce it, can you please share steps to reproduce this.
No unfortunately, one of our users reported seeing it when there system was in a bad state, not in a reproducible way. The point of the issue is just that the grammar in those messages is bad, which should be fixable even if you can't get them to fire.


Question: source := &v1.HostPathVolumeSource{ Path: "/tmp", } <del> pod := testPodWithHostVol(volumePath, source, true) </del> <ins> // we can't spawn privileged containers on Windows, nor do we need to. privileged := !framework.NodeOSDistroIs("windows") pod := testPodWithHostVol(volumePath, source, privileged) </ins> pod.Spec.Containers[0].Args = []string{ fmt.Sprintf("--new_file_0644=%v", filePath),

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --/sig testing /sig windows /area conformance Which jobs are failing: N/A Which test(s) are failing: [sig-storage] HostPath should support r/w [NodeConformance] [sig-storage] HostPath should support subPath [NodeConformance] Since when has it been failing: Before 2019-03-07. Since then, the Windows nodes' kubelet service has been passed the , which allows the test to pass. Testgrid link: (older than 2019-03-07) Reason for failure: The tests are trying to spawn pods with a HostPath volume ( folder has 777 file permissions by default, meaning it should pass without privileged access. The tests have turned green in the linked testgrid because the Windows Kubelet flag ` has been set to true, but it shouldn't have been necessary in the first place. Anything else we need to know:
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale


Question: "net/http" "github.com/GoogleCloudPlatform/kubernetes/pkg/health" <ins> "github.com/golang/glog" </ins> ) type HealthyMinionRegistry struct {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --Exclude node from LoadBalancer (via kubectl label node node1) leads to failing reconciliation of the existing load balancer services in the cluster with following error: The legacy provider azure is not using an AvailablilitySet as it it running with those cloudprovider config: Creating new load balancer services have the effect that they are stuck in due to same issue. Service reconciliation should succeed. Create an Azure k8s cluster with with an Azure cloud controller manager based on legacy-cloud-provider azure in the corresponding k8s version Configure the cloud controller manager to use the feature gate Create a service of type LoadBalancer in the cluster and wait until it was created Label a node with Check the cloud-controller-manager logs New load balancer services will now stuck in Pending state Maybe related to but should be fixed with Kubernetes version (use ): Cloud provider or hardware configuration: Azure OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /kind bug /sig cloud-provider /area provider/azure cc
/assign
/triage accepted
/assign
, thanks for the fix. Can you also create a cherry-pick to kubernetes/kubernetes
we hit a similar issue in in-tree provider as well, could you please help cherry-pick the fix to 1.20 and later release? Thanks.
Here are the cherry-picks to k/k:


Question: "k8s.io/kubernetes/pkg/api/legacyscheme" "k8s.io/kubernetes/pkg/controller" "k8s.io/kubernetes/pkg/master/ports" // add the kubernetes feature gates _ "k8s.io/kubernetes/pkg/features" )

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --BUG REPORT: /kind bug What happened: Kubernetes distro testing v1.12.0-rc.2 shows in setups with self-hosted control planes (i.e. apiserver, controller-manager, scheduler run as pods) that previously worked on v1.11.3 and prior, the kube-controller-manager now crash loops. Notably, the kube-controller-manager is run with: which has been fine through to v1.11.3. Full . What you expected to happen: Controller manager may not assume it has root. Or should clarify if an additional mount is now required. May have been introduced in cc How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): v1.12.0-rc.2 Cloud provider or hardware configuration: GCE, DigitalOcean, AWS OS (e.g. from /etc/os-release): Container Linux 1855.4.0 Kernel (e.g. ): 4.14.67-coreos Install tools: Typhoon
Seems like distros like Typhoon, Tectonic, and Tectonic/Openshift may work around this explicitly passing some other path to kube-controller-manager's or creating an emptyDir mount for . Need to understand the expectations around these certs (multiple pods, restarts, etc.) A release note seems needed to cover the new requirements. Appears this was also noticed by juju in
/sig release
cc
WDYT? Can we get folks to write a note to cover this problem in known issues?
I've an mount to address that crashloop, which seems to work preliminary (). was able to proceed with self-signed cert creation and hit a new bug I've filed separately EDIT: Perhaps a suitable release note for this change is:
the cert should not need to be persisted to disk.

I don't think adding a writeable directory requirement just for a self-signed cert is a very good idea /assign
Possible fixes enable by default if no cert is provided, rather than self-signing the self-sign cert in memory only, no disk persistence needed 't enable by default at all 2 would be ideal but is likely too big a change at this point. 3 would be unfortunate. 1 could be a good compromise if the code change is contained.
It's up to the SIG if this is a 1.12.0 blocker, or a Known Issue and 1.12.1 material. I'm inclined to think the latter.
thanks for the clarifications. I would prefer to avoid my current workaround of giving it a writeable location so those all sound fine to me. is there a full story around these controller-manager changes somewhere? I could mint actual certs, mount from a secret, update clients. If there's merit. Like, how big of a deal is it that this is now always enabled vs before? I'll probably read through the various diffs after vacation.
If this is going to get some text in release notes Known Issues (seems like it should), somebody needs to draft the text and make a suggestion of it in ASAP.
/assign
Do we want to fix this for 1.12.1 (on friday)?
Self signing in memory means you get a different cert each time it starts. It's probably better than nothing but it's not how things are supposed to work (it's too inconvenient to manually verify the cert out of band in places users should trust it). OTOH if you want to do that, maybe you'll provide a writable directory in the first place, or even a real cert. So I guess in-memory is probably OK.
Second thought: is a terrible default for the cert path. It should have the word "certs" in the path somewhere. If folks (inadvisably) run apiserver and controller manager on the same system, they should at least not both try to use certs from the same place, or worse, write them there. I suggest as the default.
What components are now verifying these certs or supposed to be? I'd expect something to be failing since random certs are generated each time the pod restarts, but don't see any impact (implying this isn't yet improving security).

Seems like controller-manager just an HTTPS endpoint, but the HTTP one is still what's used right?
I guess that aligns with the comment
The autogenerated certs only have SANs for localhost and 127.0.0.1.
The auto generated certs are self signed, right? If so, and callers won't be able to validate the signer, which SANs they have don't really matter.
Yeah, they're self-signed. So looks like the 1.12 controller-manager serves on HTTP/ (always on) and HTTPS/ (unless --secure-port=0). I guess the point of the original change was so some client could now choose to use the HTTPS endpoint with insecureSkipVerify. Main reason I was interested in the SAN is to get some clue about how a client would be expected to access these controller-manager pods (for contrast, apiserver SAN has DNS:kubernetes and an internal record that always resolves to a kube-apiserve pod).
is this updated for the openstack cloud provider too?
This issue wasn't related to any particular cloud provider. The cloud provider / hardware config field is just a typical part of filing an issue in case it had been related.
I have a writeable (I've tried other writeable directories too.) I'm encountering this: Should I post elsewhere in a new thread, or continue here?


Question: Adding new fields or API types *is* expected to modify these fixtures. To regenerate them, run: ```sh <del> UPDATE_COMPATIBILITY_FIXTURE_DATA=true go test ./vendor/k8s.io/api -run //HEAD </del> <ins> UPDATE_COMPATIBILITY_FIXTURE_DATA=true go test k8s.io/api -run //HEAD </ins> ``` ## Previous versions

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --I tried rebasing an old open PR, and got merge conflicts on api testdata. I read and tried the command listed there for regenerating the test data. It failed as follows. I expected successful regeneration of the api test data. Checkout and issue the command exhibited above. Kubernetes version (use ): is at commit right nowCloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/cc
ah, latest go versions changed the invocation to
fixed in
/triage accepted


Question: t.Errorf("unexpected error: %v", err) } if len(times) != 1 { <del> t.Errorf("expected 2 start times, got: , got: %v", times) </del> <ins> t.Errorf("expected 1 start times, got: %v", times) </ins> } else if !times[0].Equal(T2) { t.Errorf("expected: %v, got: %v", T1, times[0]) }

Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: Unit test error message match the conventional Go style: For example: if got, want := set.Status.Replicas, 3; got != want { t.Errorf("set.Status.Replicas = %v; want %v", got, want) }
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig testing
/assign
/reopen


Question: }), }, }, <ins> { // https://github.com/kubernetes/kubernetes/issues/98040 name: "cluster-scoped bad child, namespaced good child, missing parent", steps: []step{ // setup createObjectInClient("", "v1", "pods", "ns1", makeMetadataObj(pod2ns1, pod1ns1)),     // good child createObjectInClient("", "v1", "nodes", "", makeMetadataObj(node1, pod1nonamespace)), // bad child // 2,3: observe bad child processEvent(makeAddEvent(node1, pod1nonamespace)), assertState(state{ graphNodes: []*node{ makeNode(node1, withOwners(pod1nonamespace)), makeNode(pod1nonamespace, virtual)}, pendingAttemptToDelete: []*node{ makeNode(pod1nonamespace, virtual)}, // virtual parent queued for deletion }), // 4,5: observe good child processEvent(makeAddEvent(pod2ns1, pod1ns1)), assertState(state{ graphNodes: []*node{ makeNode(node1, withOwners(pod1nonamespace)), makeNode(pod2ns1, withOwners(pod1ns1)), makeNode(pod1nonamespace, virtual)}, pendingAttemptToDelete: []*node{ makeNode(pod1nonamespace, virtual),     // virtual parent queued for deletion makeNode(pod2ns1, withOwners(pod1ns1)), // mismatched child queued for deletion }, }), // 6,7: process attemptToDelete of bad virtual parent coordinates processAttemptToDelete(1), assertState(state{ graphNodes: []*node{ makeNode(node1, withOwners(pod1nonamespace)), makeNode(pod2ns1, withOwners(pod1ns1)), makeNode(pod1nonamespace, virtual)}, pendingAttemptToDelete: []*node{ makeNode(pod2ns1, withOwners(pod1ns1))}, // mismatched child queued for deletion }), // 8,9: process attemptToDelete of good child processAttemptToDelete(1), assertState(state{ clientActions: []string{ "get /v1, Resource=pods ns=ns1 name=podname2",    // get good child, returns 200 "get /v1, Resource=pods ns=ns1 name=podname1",    // get missing parent, returns 404 "delete /v1, Resource=pods ns=ns1 name=podname2", // delete good child }, graphNodes: []*node{ makeNode(node1, withOwners(pod1nonamespace)), makeNode(pod2ns1, withOwners(pod1ns1)), makeNode(pod1nonamespace, virtual)}, absentOwnerCache: []objectReference{pod1ns1}, // missing parent cached }), // 10,11: observe deletion of good child // steady-state is bad cluster child and bad virtual parent coordinates, with no retries processEvent(makeDeleteEvent(pod2ns1, pod1ns1)), assertState(state{ graphNodes: []*node{ makeNode(node1, withOwners(pod1nonamespace)), makeNode(pod1nonamespace, virtual)}, absentOwnerCache: []objectReference{pod1ns1}, }), }, }, { // https://github.com/kubernetes/kubernetes/issues/98040 name: "cluster-scoped bad child, namespaced good child, late observed parent", steps: []step{ // setup createObjectInClient("", "v1", "pods", "ns1", makeMetadataObj(pod1ns1)),              // good parent createObjectInClient("", "v1", "pods", "ns1", makeMetadataObj(pod2ns1, pod1ns1)),     // good child createObjectInClient("", "v1", "nodes", "", makeMetadataObj(node1, pod1nonamespace)), // bad child // 3,4: observe bad child processEvent(makeAddEvent(node1, pod1nonamespace)), assertState(state{ graphNodes: []*node{ makeNode(node1, withOwners(pod1nonamespace)), makeNode(pod1nonamespace, virtual)}, pendingAttemptToDelete: []*node{ makeNode(pod1nonamespace, virtual)}, // virtual parent queued for deletion }), // 5,6: observe good child processEvent(makeAddEvent(pod2ns1, pod1ns1)), assertState(state{ graphNodes: []*node{ makeNode(node1, withOwners(pod1nonamespace)), makeNode(pod2ns1, withOwners(pod1ns1)), makeNode(pod1nonamespace, virtual)}, pendingAttemptToDelete: []*node{ makeNode(pod1nonamespace, virtual),      // virtual parent queued for deletion makeNode(pod2ns1, withOwners(pod1ns1))}, // mismatched child queued for deletion }), // 7,8: process attemptToDelete of bad virtual parent coordinates processAttemptToDelete(1), assertState(state{ graphNodes: []*node{ makeNode(node1, withOwners(pod1nonamespace)), makeNode(pod2ns1, withOwners(pod1ns1)), makeNode(pod1nonamespace, virtual)}, pendingAttemptToDelete: []*node{ makeNode(pod2ns1, withOwners(pod1ns1))}, // mismatched child queued for deletion }), // 9,10: process attemptToDelete of good child processAttemptToDelete(1), assertState(state{ clientActions: []string{ "get /v1, Resource=pods ns=ns1 name=podname2", // get good child, returns 200 "get /v1, Resource=pods ns=ns1 name=podname1", // get late-observed parent, returns 200 }, graphNodes: []*node{ makeNode(node1, withOwners(pod1nonamespace)), makeNode(pod2ns1, withOwners(pod1ns1)), makeNode(pod1nonamespace, virtual)}, }), // 11,12: late observe good parent processEvent(makeAddEvent(pod1ns1)), assertState(state{ graphNodes: []*node{ makeNode(node1, withOwners(pod1nonamespace)), makeNode(pod2ns1, withOwners(pod1ns1)), makeNode(pod1ns1)}, // warn about bad node reference events: []string{`Warning OwnerRefInvalidNamespace ownerRef [v1/Pod, namespace: , name: podname1, uid: poduid1] does not exist in namespace "" involvedObject{kind=Node,apiVersion=v1}`}, pendingAttemptToDelete: []*node{ makeNode(node1, withOwners(pod1nonamespace))}, // queue bad cluster-scoped child for delete attempt }), // 13,14: process attemptToDelete of bad child // steady state is bad cluster-scoped child remaining with no retries, good parent and good child in graph processAttemptToDelete(1), assertState(state{ clientActions: []string{ "get /v1, Resource=nodes name=nodename", // get bad child, returns 200 }, graphNodes: []*node{ makeNode(node1, withOwners(pod1nonamespace)), makeNode(pod2ns1, withOwners(pod1ns1)), makeNode(pod1ns1)}, }), }, }, { // https://github.com/kubernetes/kubernetes/issues/98040 name: "namespaced good child, cluster-scoped bad child, missing parent", steps: []step{ // setup createObjectInClient("", "v1", "pods", "ns1", makeMetadataObj(pod2ns1, pod1ns1)),     // good child createObjectInClient("", "v1", "nodes", "", makeMetadataObj(node1, pod1nonamespace)), // bad child // 2,3: observe good child processEvent(makeAddEvent(pod2ns1, pod1ns1)), assertState(state{ graphNodes: []*node{ makeNode(pod2ns1, withOwners(pod1ns1)), makeNode(pod1ns1, virtual)}, pendingAttemptToDelete: []*node{ makeNode(pod1ns1, virtual)}, // virtual parent queued for deletion }), // 4,5: observe bad child processEvent(makeAddEvent(node1, pod1nonamespace)), assertState(state{ graphNodes: []*node{ makeNode(pod2ns1, withOwners(pod1ns1)), makeNode(node1, withOwners(pod1nonamespace)), makeNode(pod1ns1, virtual)}, pendingAttemptToDelete: []*node{ makeNode(pod1ns1, virtual),                   // virtual parent queued for deletion makeNode(node1, withOwners(pod1nonamespace)), // mismatched child queued for deletion }, }), // 6,7: process attemptToDelete of good virtual parent coordinates processAttemptToDelete(1), assertState(state{ clientActions: []string{ "get /v1, Resource=pods ns=ns1 name=podname1", // lookup of missing parent, returns 404 }, graphNodes: []*node{ makeNode(node1, withOwners(pod1nonamespace)), makeNode(pod2ns1, withOwners(pod1ns1)), makeNode(pod1ns1, virtual)}, pendingGraphChanges: []*event{makeVirtualDeleteEvent(pod1ns1)}, // virtual parent not found, queued virtual delete event pendingAttemptToDelete: []*node{ makeNode(node1, withOwners(pod1nonamespace)), // mismatched child still queued for deletion }, }), // 8,9: process attemptToDelete of bad cluster child processAttemptToDelete(1), assertState(state{ clientActions: []string{ "get /v1, Resource=nodes name=nodename", // lookup of existing node }, graphNodes: []*node{ makeNode(node1, withOwners(pod1nonamespace)), makeNode(pod2ns1, withOwners(pod1ns1)), makeNode(pod1ns1, virtual)}, pendingGraphChanges: []*event{makeVirtualDeleteEvent(pod1ns1)}, // virtual parent virtual delete event still enqueued }), // 10,11: process virtual delete event for good virtual parent coordinates processPendingGraphChanges(1), assertState(state{ graphNodes: []*node{ makeNode(node1, withOwners(pod1nonamespace)), makeNode(pod2ns1, withOwners(pod1ns1)), makeNode(pod1nonamespace, virtual)}, // missing virtual parent replaced with alternate coordinates, still virtual absentOwnerCache: []objectReference{pod1ns1}, // cached absence of missing parent pendingAttemptToDelete: []*node{ makeNode(pod2ns1, withOwners(pod1ns1)), // good child of missing parent enqueued for deletion makeNode(pod1nonamespace, virtual),     // new virtual parent coordinates enqueued for deletion }, }), // 12,13: process attemptToDelete of good child processAttemptToDelete(1), assertState(state{ clientActions: []string{ "get /v1, Resource=pods ns=ns1 name=podname2",    // lookup of good child "delete /v1, Resource=pods ns=ns1 name=podname2", // delete of good child }, graphNodes: []*node{ makeNode(node1, withOwners(pod1nonamespace)), makeNode(pod2ns1, withOwners(pod1ns1)), makeNode(pod1nonamespace, virtual)}, absentOwnerCache: []objectReference{pod1ns1}, pendingAttemptToDelete: []*node{ makeNode(pod1nonamespace, virtual), // new virtual parent coordinates enqueued for deletion }, }), // 14,15: observe deletion of good child processEvent(makeDeleteEvent(pod2ns1, pod1ns1)), assertState(state{ graphNodes: []*node{ makeNode(node1, withOwners(pod1nonamespace)), makeNode(pod1nonamespace, virtual)}, absentOwnerCache: []objectReference{pod1ns1}, pendingAttemptToDelete: []*node{ makeNode(pod1nonamespace, virtual), // new virtual parent coordinates enqueued for deletion }, }), // 16,17: process attemptToDelete of bad virtual parent coordinates // steady-state is bad cluster child and bad virtual parent coordinates, with no retries processAttemptToDelete(1), assertState(state{ graphNodes: []*node{ makeNode(node1, withOwners(pod1nonamespace)), makeNode(pod1nonamespace, virtual)}, absentOwnerCache: []objectReference{pod1ns1}, }), }, }, </ins> } alwaysStarted := make(chan struct{})

What happened: create a namespaced parent object (for example a ) delete the parent object create a cluster-scoped object (for example. an ) with the parent as an , which is unresolvable since the parent is namespaced. create a namespaced object (for example another ) in the same namespace, with the parent as an , which should be valid. The namespaced child and the cluster-scoped child are not garbage collected. What you expected to happen: The namespaced child should be garbage collected immediately after being created, the cluster-scoped child should not be. Reproduction: Setup: parent- cluster-scoped- namespaced- Scenario 1: one terminal, run another, a third, run See that the cluster scoped child and namespaced child are both created but not deleted. Scenario 2: one terminal, run another, run See that the namespaced child is created and immediately deleted. My expectation is that the garbage collection of should not depend on , however scenarios 1 and 2 do not seem consistent. Environment: Kubernetes version (use ): 1.20.0 Cloud provider or hardware configuration: Kind 1.20.0 OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/assign
Thanks for the detailed report, reproduced the issue with a unit test in
/priority important-soon /triage accepted /sig api-machinery


Question: } ?> ``` <del> To play with the service itself, find the name of a frontend, grab the external IP of that host from the Google Cloud Console, and visit http://<host-ip>:8080, note you may need to open the firewall for port 8080 using the console or the gcloud tool. </del> <ins> To play with the service itself, find the name of a frontend, grab the external IP of that host from the Google Cloud Console, and visit http://<host-ip>:8080. Note, you may need to open the firewall for port 8080 using the console or the gcloud tool. </ins>

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: The image () used to run the per-pod infra/pause container logs to stdout about once per second. What you expected to happen: Pause container not logging anything. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Kubelet does not expose, nor manage these logs. Writing the logs wastes resources and takes up disk space, which may not be properly accounted by kubelet. The pause image used for the e2e tests seems much better: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/assign Will also rebuild with


Question: export PROJECT="k8s-e2e-gke-prod-parallel" export CLOUDSDK_BUCKET="gs://cloud-sdk-testing/rc" export JENKINS_USE_SERVER_VERSION="y" <ins> export CLOUDSDK_API_ENDPOINT_OVERRIDES_CONTAINER="https://container.googleapis.com/" </ins> export GINKGO_TEST_ARGS="--ginkgo.skip=[Slow]|[Serial]|[Disruptive]|[Flaky]|[Feature:.+]" export GINKGO_PARALLEL="y" export ZONE="asia-east1-b"

With recent migration of Kubelet to structured logging introduced e2e tests that enable JSON in Kubelet. Organized migration of other components haven't yet started and is on best effort bases, yet some line were already migrated. As code-base will be slowly migrated, we need to expand the test to avoid errors like I propose to expand existing e2e test suite ( and ) to cover all control plane components (apiserver, controller-manager and scheduler). /cc
/sig instrumentation
/help
If all components have this flag do we just want to enable it on all components? We already increase the log level on all components in CI, it should be pretty straightforward to expand this to all: pick a new environment variable name for the script with a more appropriate name for all components instead of just kubelet set it on the job in addition to the old kubelet one, by sending a PR to test-infra send a PR to the kind script to set the value in all places where we set verbosity instead of just kubelet, switching to the new variable name
/triage accepted
/area logging
So currently the e2e tests do not actually read and test what the logs return the current job just makes sure that kubelet is running Would we want to write similar tests for the different control planes while we start migration or do you have something else in mind with respect to the automated testing ?
Yes, currently we just test if Kubelet is running to test that flag is working and if enabling json doesn't cause panics. This exact issue proposes to expand this to other control plane, but your idea would be a great follow up. Would be great to propose a test plan that will specify what we want to test and how. If you have ideas some ideas could you collect hem in some issue so we can discuss.
There's no portable way to get these logs. There are non portable ways but those cannot be used in a standard e2e test.
Oh so for the standard e2e tests it might be good enough to just test the startups of these control planes with the logging enabled.
I was thinking that we could get started with maybe writing similar to one for the remaining components and then maybe find a way of checking the logs generated as we have bigger discussions a) kube-apiserver --Completed b) etcd --- TODO c) kube-scheduler ---TODO d) kube-controller-manager --TODO
sound good.

FYI Etcd is not a kubernetes component as so it's out of scope of this issue.
/wg structured-logging As getting core components logs is not portable, we could test a container with same logging library and using Kubenetes logs API to access logs. Created to track it
it still might not be a terrible idea to write a simple test script to probe the exported logs from a cluster with json logging enabled for any non-json lines once we know that everything should be. I think this could probably be hacked up with jq or something, but I wouldn't position it like a normal e2e test.
Could you give a hint how to implement and run such script?
/priority important-soon This issue is based on feedback from Kubelet migration retrospection, it is required before organizing migration of next component. /cc
Would this follow the same flow as in where we check if json logging is enabled and then run jq separately or within the same flow. To be more specific: Seems to be getting the data and validating it so in that case would we be running after grabbing the logs ?
For now let's focus on the original tasks of having ( and ) test scenarios, also configure other components. We just need to update Kind script to also pass to other components. Example:
Would like to take this one! /cc
Great, let me know if you need any help? /remove-help
just created a PR:
Components to be picked: [x] kube-scheduler [ ] kube-apiserver [ ] controller-manager
Attempted to cover the remaining components in this
It would be great if we had e2e setup for all components before we start structured migration next week. will you be able to work on it this week? Please let me know if there is anything I can help with.
Thanks for help with control plane components, for 1.23 we are planning to migrate kube-proxy to structured logging. Could you also try to enable JSON logging on kube-proxy?
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Confirmed that in last CI run Logs from kubelet and control plane components are written in JSON: Great Job Could you take a look into changing log format of kube-proxy?


Question: percentageOfNodesToScore: percentageOfNodesToScore, } } <ins> // To expose more information about the scheduling process but not has a big drop on performance, // we use a dynamic strategy to dump depending on the logging level: // * for logging level ranged in [4, 6), dump the scores for topM nodes/plugins. And make it as performant as possible. // * for logging level ranged in [6, 10), dump the scores for topN nodes, and probably show all plugins // * for logging level >= 10, dump the scores for all nodes/plugins. func logNodeAndPluginScores(nodeScores framework.NodeScoreList, scoresMap framework.PluginToNodeScores, pod *v1.Pod) { if klog.V(10).Enabled() { for plugin, nodeScoreList := range scoresMap { for _, nodeScore := range nodeScoreList { klog.InfoS("Plugin scored node for pod", "pod", klog.KObj(pod), "plugin", plugin, "node", nodeScore.Name, "score", nodeScore.Score) } } return } // Log the top M node scores for the pod topMNodes := minNodeScoresToDump if klog.V(6).Enabled() { topMNodes = moreNodeScoresToDump } // Use the min-heap to maintain a top M node score list nsh := make(nodeScoreHeap, 0, topMNodes) for i, nodeScore := range nodeScores { if len(nsh) < int(topMNodes) || nodeScore.Score > nsh[0].Score { if len(nsh) == int(topMNodes) { heap.Pop(&nsh) } heap.Push(&nsh, &nodeScores[i]) } } // Build a map of Nodes->PluginScores on that node nodeScoresMap := make(nodeToPluginScores, topMNodes) for _, nodeScore := range nsh { nodeScoresMap[nodeScore.Name] = make(pluginScoreList, 0) } // Convert the scoresMap (which contains Plugins->NodeScores) to the Nodes->PluginScores map for plugin, nodeScoreList := range scoresMap { for _, nodeScore := range nodeScoreList { // Get the top M nodes' plugin scores if _, ok := nodeScoresMap[nodeScore.Name]; ok { nodeScoresMap[nodeScore.Name] = append(nodeScoresMap[nodeScore.Name], pluginScore{Name: plugin, Score: nodeScore.Score}) } } } // Log the top N plugin scores for the top M nodes topNPlugins := minPluginScoresToDump if klog.V(6).Enabled() { // Log all plugins' socre if -v6 topNPlugins = -1 } for name, pluginScores := range nodeScoresMap { sort.Slice(pluginScores, func(i int, j int) bool { return pluginScores[i].Score > pluginScores[j].Score }) // Get the top N plugin socres only if logging level ranged in [4, 6) if topNPlugins > 0 && len(pluginScores) > int(topNPlugins) { nodeScoresMap[name] = pluginScores[:topNPlugins] } } scoresMessage := fmt.Sprintf("Top %d plugins for pod on node", topMNodes) for len(nsh) > 0 { nodeScore := heap.Pop(&nsh).(*framework.NodeScore) pluginScores := nodeScoresMap[nodeScore.Name] klog.InfoS(scoresMessage, "pod", klog.KObj(pod), "node", nodeScore.Name, "score", nodeScore.Score, "plugins", pluginScores) } } // nodeScoreHeap is a min-heap ordered by the score of nodes. The // scheduler uses this to find the top-N scoring nodes when the // logging verbose level is ranged in [4, 10) type nodeScoreHeap []*framework.NodeScore // var _ heap.Interface = &nodeScoreHeap{} func (nsh nodeScoreHeap) Len() int { return len(nsh) } func (nsh nodeScoreHeap) Less(i, j int) bool { return nsh[i].Score < nsh[j].Score } func (nsh nodeScoreHeap) Swap(i, j int) { nsh[i], nsh[j] = nsh[j], nsh[i] } func (nsh *nodeScoreHeap) Push(v interface{}) { *nsh = append(*nsh, v.(*framework.NodeScore)) } func (nsh *nodeScoreHeap) Pop() interface{} { ns := (*nsh)[nsh.Len()-1] *nsh = (*nsh)[:nsh.Len()-1] return ns } </ins>

/sig scheduling A way to check the scoring of each node to figure out why a node has a high scheduling priority. proposal 1: add some debug level logs to Kube-scheduler, so we can know node score on each scoring step. proposal 2: a simple way to decrease the priority of one node(in my case a node with more than 200 pods is with the highest scheduling priority for an unknown reason) proposal 3:? All new pods are scheduled to one of five nodes in my cluster. I have checked both CPU/memory requested usage and all five nodes are bare metal and running about 150 pods or 200 pods. Pods on node request 80-93% of the memory. We have a similar issue as Currently, I think is the easiest way.
This is useful. how do you suggest we log this information? note that there are two dimensions for each pod: plugin and node score. Also, we need to limit the log length, we can't simply log the score for each node in clusters with large number of nodes.
I prefer to log only the top 5 or 10 node scores in the log. (I have not read the code yet., will dig into it later.)
We actually merged something like this that I wrote in , but had to revert it because of performance issue in large clusters. That could be a good starting point My general approach was to 1) invert the already available to map (rather than , which it currently does) and then 2) log the top 3 highest scoring plugins for each node However discovered performance issues after this in (we believe due to the many writes this process took) and it was reverted in I think we decided that some performance hit is reasonable, especially given log levels =4, but this was ~50% drop.
thanks for the information. So with current kube-scheduler, I can set log level to 10 to get every node score for each plugin. It helps in some cases.
yes, that's the current case. Though creates a lot of other noise in the logs, so I think we agreed that it is okay for us to add this kind of summary at a lower level like
I think we can use this chance to standardize the logging logic to dump scores dynamically depending on log level: for logging level ranged in , dump the scores for topM nodes/plugins.

And make it as performant as possible. for logging level ranged in , dump the scores for topN nodes, and probably show all plugins for logging level = 10, dump the scores for all nodes/plugins.
/assign
/triage accepted
/reopen Some problems that were mentioned by are not fixed yet. /cc
Reopened this issue. <detailsIn response to : /reopen After taking a glance, I can spot several issues: A consensus on the value and . This is not fully discussed yet. Further dig into the performance result. The test runs on v=1 and v=4, but actually they're the same if it's running in the context of integration framework. I'd expect to see the result comparing before/after on v=4 and v=6. Further review on the code, like adding UTs and some logic nits. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Since is merged to revert the change,
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/remove-lifecycle rotten
Just FYI: this may be related to in the future.

In , we cannot add all scores to the event message (because the event msg has the limit on the length), so I am thinking we need to sort node-scores by total score and add the score of top N nodes to the event msg. And, if we do so, we can easily achieve to log the top M Node's score by using this sorted node-scores.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/lifecycle rotten
/remove-lifecycle rotten
/remove-lifecycle rotten
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue with Mark this issue as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close not-planned [1]:
Closing this issue, marking it as "Not Planned". <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue with Mark this issue as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: } if event.Type == watch.Error { util.HandleError(fmt.Errorf("error from watch during sync: %v", errors.FromObject(event.Object))) <ins> // Clear the resource version, this may cause us to skip some elements on the watch, // but we'll catch them on the synchronize() call, so it works out. *resourceVersion = "" </ins> continue } glog.V(4).Infof("Got watch: %#v", event) rc, ok := event.Object.(*api.ReplicationController) if !ok { <ins> if status, ok := event.Object.(*api.Status); ok { if status.Status == api.StatusFailure { glog.Errorf("failed to watch: %v", status) // Clear resource version here, as above, this won't hurt consistency, but we // should consider introspecting more carefully here. (or make the apiserver smarter) // "why not both?" *resourceVersion = "" continue } } </ins> util.HandleError(fmt.Errorf("unexpected object: %#v", event.Object)) continue }

wrote: I just ran across this the other day. but I am seeing the error in the service. Here is a sample of the output. I am running v0.8.2. I can get the behavior to go away temporarily by rebooting the node or restarting the service but it always seems to come back eventually.


Question: for i := range podList.Items { p := podList.Items[i] <del> if podutil.IsPodReady(&p) { </del> <ins> if podutil.IsPodReady(&p) && p.ObjectMeta.DeletionTimestamp.IsZero() { </ins> pod = &p return true, nil }

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: all of the e2e jobs that run this test Which test(s) are flaking: ` Testgrid link: Reason for failure: mostly: Anything else we need to know: links to appreciated links to specific failures in spyglass appreciated /sig apps xref:
approx:
my suspicion: /
I can't find anything else that seems relevant in the time frame, and it is the only change to the test file in that time frame.
merged recently, not sure if it's relevant
Not sure what the procedure it, but we can revert or try some other image instead of Pause, if you think it helps. I don't have a good explanation why the change would make the test flake like this.
I don't have a good explanation, just that it affects the test and is in the time frame that seems to fit the flake spike It may take a bit to pin down what exactly happened here :/ I can't look for a little bit.
I tried to reproduce this behaviour on a test cluster in AWS, but couldn't. I prepared a PR to revert the change, just incase.
Good news and bad news kind of thing... The same test fails in the revert PR also, with the previous EchoServer image:
I did an experiment reverting the second suspect in I couldn't reproduce the issue anymore even though I tried. You can check the results in the PR history.
I believe I have found the issue here. More information and fix incoming
Thanks for the update
I just about have a fix ready here. I was able to narrow the commit range to by looking at when this flake first occurred in It looks like was correct that this was introduced in . I believe what is happening is that in the test we are getting getting a with a set (the one that we previously successfully issued an eviction for) before waiting for 3 pods to be ready again in the namespace (). Before the aforementioned PR caused us to skip checking PDBs when is set we would still report back that eviction was not possible because our with did not meet the other criteria (i.e.

was not , , or ), which caused us to still check PDBs, which would show us that we did not have the budget to delete (even though it was not really taking into account the that we were actually attempting to evict). I think there are a few things we can do to address this: we should check that not only is the , but also that it does not have set. above where we call so that we do not attempt to get a before all of the pods are ready. in that the does not have a . Just doing (3) by itself would address our current problem, but (1) and (2) also seem to make sense in the context of what we are trying to check here.
Thanks everyone and apologies for the false alarm :-)
/lifecycle active /priority critical-urgent (this is blocking merges of other fixes) /assign
/milestone v1.19


Question: var pod, pod1, pod2 *v1.Pod It("should assign CPUs as expected based on the Pod spec", func() { <del> oldCfg = enableCPUManagerInKubelet(f) </del> cpuCap, cpuAlloc, cpuRes = getLocalNodeCPUDetails(f) <del> // Skip CPU Manager tests if the number of allocatable CPUs < 1. if cpuAlloc < 1 { framework.Skipf("Skipping CPU Manager tests since the number of allocatable CPUs < 1") </del> <ins> // Skip CPU Manager tests altogether if the CPU capacity < 2. if cpuCap < 2 { framework.Skipf("Skipping CPU Manager tests since the CPU capacity < 2") </ins> } <ins> // Enable CPU Manager in the kubelet. oldCfg = enableCPUManagerInKubelet(f) </ins> By("running a non-Gu pod") ctnAttrs = []ctnAttribute{ {

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Test failed when running CPU Manager node e2e test on Ubuntu 16.04 locally, and show this panic message: What you expected to happen: Tests passed. How to reproduce it (as minimally and precisely as possible): Run this node e2e test, use the following command: Anything else we need to know?: I think this is because the following codes: is . I will send a PR to fix this after it is confirmed. /sig node /assign PTAL, Thanks!
Thanks for the bug report. Reproduced successfully. It looks like the response body may have changed from empty map to null, which caused this to break. In any case, suggested adding the fix to his pending PR here:
Hi, have fixed this, lgtm.


Question: return T1 } <ins> func weekAfterTheHour() time.Time { T1, err := time.Parse(time.RFC3339, "2016-05-26T10:00:00Z") if err != nil { panic("test setup error") } return T1 } </ins> func justBeforeThePriorHour() time.Time { T1, err := time.Parse(time.RFC3339, "2016-05-19T08:59:00Z") if err != nil {

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): No What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): , , Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Environment: Cloud provider or hardware configuration: bare metal OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: , deployed with Others: What happened: I am using a scheduledjob with . An instance of my job has taken a long time to finish, thus skipping many start times. Ever since, the scheduledjob will not schedule anything new. Controller manager log has: What you expected to happen: I expect the scheduledjob not to be stuck not scheduling anything anymore. With , I expect that it should just re-schedule something as soon as the previously running job has completed. The "Too many missed start times to list" safeguard appears to apply mostly to cases where job concurrency is allowed to avoid creating a lot of jobs in case of timestamp issues. ` How to reproduce it (as minimally and precisely as possible): The issue shows up after about 100 minutes of this running. Anything else do we need to know:
cc
sorry for the long delay in response, there were a few issues solved in later versions of ScheduledJob, can you verify this against latest 1.4.x? I've tried cherry-picking all of the crucial ones, like: and I'll try to verify this on my own as well.

I am still getting the same issue with
thanks for the info, will dig deeper into this one.
Thanks a lot any chance a potential fix could make it in 1.5?
if possible - yes, I'll try to backport it to 1.5
did you look into this? If not, let me know if you want me to pick it up and work on a PR
yeah, go ahead and ping me in the PR.
I just thought of something which might make sense. We've modified the error message to make it more "actionable", and the user can set a deadline to avoid running into this. However, the error message is still only controller level AFAIK and thus not exposed to the user. From the user's perspective, there is not way to tell why the CronJob is not scheduling the next instance. For these reasons, maybe we should also emit an event in the corresponding CronJob? I can send a small PR for this modification.


Question: } for _, test := range tests { <del> authenticator := bearertoken.New(bootstrap.NewTokenAuthenticator(bootstrapSecrets{test.secret})) </del> <ins> authenticator := group.NewAuthenticatedGroupAdder(bearertoken.New(bootstrap.NewTokenAuthenticator(bootstrapSecrets{test.secret}))) </ins> // Set up an API server controlPlaneConfig := framework.NewIntegrationTestControlPlaneConfig() controlPlaneConfig.GenericConfig.Authentication.Authenticator = authenticator

While moving APF to beta (see comments on ), we discovered that the integration tests commonly present users which are not members of any groups. We had thought that the system enforced membership in either the authenticated or unauthenticated group. If this is supposed to be a system invariant (as we thought), we'd like to ask SIG Auth: Please fix the integration tests Please add e.g. logging so that if real, operational clusters hit this there's at least a log message and people will file bugs. /sig auth
Can I suggest also, if this is a real invariant, that it be documented in some place(s) likely to be read?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale /priority important-longterm /triage accepted
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/lifecycle frozen
fixes the authenticator configured in one of the integration test API server startup helpers which resolved this issue for . Remaining tests that still need fixing that are encountering unexpected users look to be ones set up to specifically test auth paths:
open to fix remaining integration test users


Question: metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/runtime/schema" <ins> "k8s.io/apimachinery/pkg/types" "k8s.io/apimachinery/pkg/util/uuid" </ins> admissionapi "k8s.io/pod-security-admission/admission/api" "k8s.io/pod-security-admission/api" "k8s.io/pod-security-admission/metrics"

When updating policy labels on a namespace, existing pods are evaluated to see if they are compatible with the new policy. If there are too many pods, or if evaluation takes too long, we short circuit and warn that not all pods were checked. For that reason, we want to prioritize checking a variety of pods before checking essentially identical pods, so that unique warnings can be surfaced before reaching the pod count cutoff or evaluation timeout. Pods created by the same controller are most likely to be identical (e.g. replicas from a replicaset or job, copies of pods from a statefulset or daemonset, etc) cc
This is an interesting fix I'd like to take this up Can I take this up? I am fairly new to Kubernetes source code so might need some help
I am guessing this is where the logic you mentioned should be implemented right?
thanks for the interest; this is targeted at the beta version... right now, effort is going toward the alpha-blocking work for 1.22
Should I wait for the code freeze before working on this? Sorry for lame questions
If you want to start looking at it, that's fine, but it's possible the code in the ValidateNamespace function will change before we're ready to start merging 1.23 work, and there isn't really reviewer bandwidth to take a look at work-in-progress stuff for next release yet.
Sure, I'll just for code freeze before I start with implementation By the time I'll just assign this to myself /assign
Does it make sense to take it up now? I think we already have a couple of RC builds after the code freeze
sure, it's fine to start work in a branch in your fork and have a WIP PR open, even before master reopens for 1.23 work
Awesome, I'll start working on it Thank you
Hello! Are you still working on this? If you don't have time, I can take this issue. If not, maybe I can give some help
Since this was not an urgent issue I was working on some other issues last week I have started on this, Will be raising a PR maybe today or tomorrow I Will ask you if I get stuck Thank you


Question: expectedResult:    true, runtimeCostBudget: 10, // enough for one lazy evaluation but not two, should pass }, <ins> { name: "single boolean variable in expression", variables: []NamedExpressionAccessor{ &testVariable{ name:       "fortuneTelling", expression: "true", }, }, attributes:     endpointCreateAttributes(), expression:     "variables.fortuneTelling", expectedResult: true, }, { name: "variable of a list", variables: []NamedExpressionAccessor{ &testVariable{ name:       "list", expression: "[1, 2, 3, 4]", }, }, attributes:     endpointCreateAttributes(), expression:     "variables.list.sum() == 10", expectedResult: true, }, { name: "variable of a map", variables: []NamedExpressionAccessor{ &testVariable{ name:       "dict", expression: `{"foo": "bar"}`, }, }, attributes:     endpointCreateAttributes(), expression:     "variables.dict['foo'].contains('bar')", expectedResult: true, }, { name: "variable of a list but confused as a map", variables: []NamedExpressionAccessor{ &testVariable{ name:       "list", expression: "[1, 2, 3, 4]", }, }, attributes:           endpointCreateAttributes(), expression:           "variables.list['invalid'] == 'invalid'", expectErr:            true, expectedErrorMessage: "found no matching overload for '_[_]' applied to '(list(int), string)'", }, { name: "list of strings, but element is confused as an integer", variables: []NamedExpressionAccessor{ &testVariable{ name:       "list", expression: "['1', '2', '3', '4']", }, }, attributes:           endpointCreateAttributes(), expression:           "variables.list[0] == 1", expectErr:            true, expectedErrorMessage: "found no matching overload for '_==_' applied to '(string, int)'", }, </ins> } for _, tc := range cases { t.Run(tc.name, func(t *testing.T) {

Variables field is to ValidatingAdmissionPolicy in 1.28. Currently the usage of variables would expect a operator associated. e.g. (thanks to Mariam Fahmy for raising the issue) Above will fail with error: Meanwhile the will pass. The reason is when variables be evaluated to non dyn type(like bool or string), it will require special handling(like casting) since we use dyn type everywhere. It might confuse user.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage accepted /sig api-machinery /cc
Tests I'd expect to see verify this behavior: PASS: A variable is of type bool, an expression returns the variable PASS: A variable is of type string, a messageExpression returns the variable PASS: A variable is of type dyn, at runtime the dyn is a bool, an expression returns the variable PASS: A variable is of typedyn, at runtime the dyn is a string, an messageExpression returns the variable PASS: A expression is of type dyn, at runtime the dyn is of type bool PASS: A messageExpression is of type dyn, at runtime the dyn is of type string COMPILE ERROR: A variable is of type bool, an expression tries to perform a non-bool (e.g. string replace) operation on the variable, this should resultsin a compile error COMPILE ERROR: A variable is of type string, an expression tries to perform a non-string (e.g. less than) operation on the variable, this should result in a compile error RUNTIME ERROR: A variable is of type dyn, at runtime the dyn is a bool, an expression tries to perform a non-bool (e.g. string replace) operation on the variable, this should result in a runtime error I'd also expect to see some cases covering variables of other types (maps, lists, duration, quantity, ...) to make sure we're over-fitting for bool and string.
/retitle ValidatingAdmissionPolicy: variable expression type not recognized as return type in expression that evaluates the variable
/assign


Question: //TODO: Handle errors by taking corrective measures w.wg.Add(1) <del> go func() { </del> <ins> func() { </ins> defer w.wg.Done() if event.Op&fsnotify.Create == fsnotify.Create {

What happened: When I restart pod with my driver and driver-registrar, node looses NodeID annotation. How to reproduce it (as minimally and precisely as possible): Kubernetes without any alpha features (local-up- in this case) HostPath driver for everything to be Running the node has annotation the node pod (deleting all pods here, we don't care about attacher/provisioner). will start a new node pod with new registrar. Wait until the new pod gets Running. the node does not have the annotation It's 80% reproducible in my local cluster. New driver-registrar deletes socket of the old registrar and creates new one. I can see in logs: So the old socket was deregistered very shortly before new one got registered. IMO these two operations collide with each other and the deregistration "wins" somehow. Environment: Kubernetes version (use ): today's master /kind bug /sig storage
Yeah, InstallCSIDriver runs in parallel with UninstallCSIDriver. UninstallCSIDriver updates the node while InstallCSIDriver is getting the current value. In most cases, InstallCSIDriver gets the old value (still with annotations) and does not update the node. Some locks are missing there. But even with the locks it will be racy. We must make sure that UninstallCSIDriver grabs the lock before InstallCSIDriver, so the annotation is always in the end.
We should see if we can handle this at the layer so that it works for everyone. We could use where key is the CSI driver name to ensure that two operations are not started in parallel for the same driver. But that's not the hard part, like you mentioned, the tricky part is ensuring the correct order of operations between create/delete events.
/milestone v1.13 /priority critical-urgent
I am investigating this today. Will update with fix/finding on a PR.
A best practice for the driver should be to remove the registration sockets when the driver is uninstalled: this would generally remedy this problem for that particular driver in normal uninstall-install cases. We do it for GCE PD here: I still think the underlying problem should be resolved though :+1:
Is there an incoming PR for this?
has been working on this today. He believes that he has the problem understood and is able to repro it.

He is working on a fix/verification (which takes time since it requires multiple runs to repro/verify). He will add more information soon as he can.
After spending a good amount of time chasing the cause for this issue, it came down to the pluginwatcher code that can handle fsnotify CREATE and fsnotify DELETE events concurrently here: The at that line number causes both CREATE and DELETE events to be handled concurrently. This can create race conditions in plugins by calling and concurrently. This is fixed in this PR (see for detail of fix).
that is a great suggestion, will make not of it for documentation.
hi,I test in v1.13 and the problem existed! my csi plugin running,but the node annotation can not labeled! [root kubectl get po -n kube-system -o wide -w NAME READY STATUS RESTARTS AGE IP NODE csi-attacher-cinderplugin-0 2/2 Running 0 4d 111.111.111.116 111.111.111.116 csi-nodeplugin-cinderplugin-cpq7m 2/2 Running 0 14s 111.111.111.118 111.111.111.118 csi-nodeplugin-cinderplugin-h48dh 2/2 Running 0 14s 111.111.111.116 111.111.111.116 Annotations: then cause my pod can not attached I0409 01:50:59. 1 ] Saving attach error to "csi-46866cc91b27a53461216c2dab85709a01f1704004457f6e3b9f0f4beb26bf8e" I0409 01:50:59. 1 ] Ignoring VolumeAttachment "csi-46866cc91b27a53461216c2dab85709a01f1704004457f6e3b9f0f4beb26bf8e" change I0409 01:50:59. 1 ] Saved attach error to "csi-46866cc91b27a53461216c2dab85709a01f1704004457f6e3b9f0f4beb26bf8e" I0409 01:50:59.

1 ] Error processing "csi-46866cc91b27a53461216c2dab85709a01f1704004457f6e3b9f0f4beb26bf8e": failed to attach: node "111.111.111.116" has no NodeID annotation
AttachVolume.Attach failed for volume "pvc--758e-11e9-bfc0-" : node "rancher196" has no NodeID annotation
Docker 18.9.6 Kubelet v1.12.0 CentOS Linux 7 3.10.0-957.10.1.el7.x86_64 ttachVolume.Attach failed for volume "pvc--75ed-11e9-bfc0-" : node "crawler203" has no NodeID annotation Dashboard/volume/pvc--75ed-11e9-bfc0- Volume Details State:Detached Health: Frontend:Block Device Attached Node & Endpoint : Size:2 Gi Actual Size:0 Bi Base Image: Engine Image:rancher/longhorn-engine:v0.4.1 Created:10 minutes ago Latest Backup:
I don't find code in kubelet 1.12.0 about this problem. strange!


Question: } // DeleteMirrorPod provides a mock function with given fields: podFullName <del> func (_m *MockManager) DeleteMirrorPod(podFullName string) error { </del> <ins> func (_m *MockManager) DeleteMirrorPod(podFullName string, _ *types.UID) (bool, error) { </ins> ret := _m.Called(podFullName) var r0 error if rf, ok := ret.Get(0).(func(string) error); ok { <del> r0 = rf(podFullName) </del> <ins> return true, rf(podFullName) </ins> } else { r0 = ret.Error(0) } <del> return r0 </del> <ins> return false, r0 </ins> } // DeleteOrphanedMirrorPods provides a mock function with given fields:

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: of a static pod (really the mirror pod) results in multiple attempts to re-create the mirror pod before it finally succeeds (note that in some runs it goes into Running or Terminating state -- it just depends upon the exact timing of the . Note that in the below trace the pod name is elided from the output. Note that the actual static pod continues to run as normal. What you expected to happen: Mirror pod should start without additional state transitions How to reproduce it (as minimally and precisely as possible): Create a static pod, it, and Anything else we need to know?: Environment: Kubernetes version (use ): 1.14/OpenShift 4.2 (master has the same code) Cloud provider or hardware configuration: AWS (not relevant) OS (e.g: ): Not relevant Kernel (e.g. ): Not relevant Install tools: Not relevant Network plugin and version (if this is a network-related bug): N/A Others:
/sig node
More details: The bug appears to be a race condition between deleting the old mirror pod, the API server deleting its record of it, and creating a replacement that's not related to skew. I suspect the reason I didn't see it bouncing back and forth in my earlier tests was simply that I wasn't running the kubectl get node frequently enough; if I run it continuously, I see it bounce. The sequence is: 1) Somebody kubectl deletes the mirror pod. 2) The static pod code in (around line 1630) notices that the pod has been marked deleted (its deletion timestamp is not nil and it is a mirror pod), deletes it (in DeleteMirrorPod in , via the apiserverClient), and creates a new one (again, via the apiserverClient). 3) The next time around, the static pod code may see either the old pod (that has been deleted) or the new one as the mirror pod.

If the old pod is the one that's seen, it sees that there's a deletion timestamp, so it deletes it again -- which, since pod deletion is by name (UID isn't involved) results in the new pod being deleted and yet another new pod being created. If the new pod is the one that's seen, then we break out with the spin cycle complete. Still TBD is why the mirror pod isn't getting replaced right away. But here are traces of the relevant sequences: Bad sequence: Good sequence: The messages at lines 7 and 18 were printed immediately after the call to apiServerClient...Create(). But the key is at lines 9 and 19. At line 9, note that the UID (...) is the UID of the old mirror pod, which has a deletion timestamp and therefore the mirror code deletes the pod -- but the pod that's actually in existence at the time is the new pod. At line 19, however, note that the mirror pod is now the new one, which is not marked for deletion, and is therefore left alone. The solution (I'll open a PR shortly) is to check both the pod name and the UID to make sure that we're in fact trying to delete the pod we think we want to delete.
Thanks for the detailed issue :) Really awesome!
I'm seeing this issue on an upgrade from v1.14.4 to v1.15.1 following the on a CentOS 7 system. After installing the updated kubelet package and restarting kubelet per the instructions, the cluster went into a loop of continually restarting the etcd pod from one control plane node and the scheduler pod from another control plane node. This continued for approximately 10 minutes before the etcd pod finally stayed up. It was another 5 minutes before the scheduler pod finally ran. I noticed v1.15.2 was released although the change log doesn't show this issue included. Was there a plan to back-port this fix to 1.15 and 1.14 (not to the same extent, but I have seen this behavior on a 1.14.4 cluster)? Since this behavior can happen to the apiserver pod, it has the potential to be very disruptive to a cluster.
The actual etcd pod, or the mirror pod (which is more likely)?

I hadn't specifically planned on backporting it, but it shouldn't be a problem; this code has been stable.
Ah, good point--the actual pod does appear to remain running. That makes it less important than I was initially thinking. However, it is a bit disconcerting for someone who happens to be looking at the state of the cluster (or for monitoring) to see the pod appear to be stuck in a restart loop. If it were just a few iterations, it might not matter too much. However, I'm seeing this happen for minutes at a time. For example, to verify that the real pod remained available I did a on the apiserver pod for one master, and output showed it in the restart loop for 15 minutes before it settled down. IMO that's worth backporting if it's low risk.
I opened a cherrypick against 1.15 for this.
Appreciated. Thanks!


Question: // ResourceList is a set of (resource name, quantity) pairs. type ResourceList map[ResourceName]resource.Quantity <del> // Get is a convenience function, which returns a 0 quantity if the // resource list is nil, empty, or lacks a value for the requested resource. // Treat as read only! func (rl ResourceList) Get(name ResourceName) *resource.Quantity { if rl == nil { return &resource.Quantity{} } q := rl[name] return &q } </del> // Node is a worker node in Kubernetenes // The name of the node according to etcd is in ObjectMeta.Name. type Node struct {

No other types have helper methods, and this does weird things with pointers.
It's a clean build/test without that helper method. See .


Question: : ${PROJECT:="kubernetes-jenkins"} ;; <ins> # Runs only the elasticsearch logging tests on GCE. kubernetes-e2e-gce-es-logging) : ${E2E_CLUSTER_NAME:="jenkins-gce-e2e-es-logging"} : ${E2E_NETWORK:="e2e-es-logging"} : ${GINKGO_TEST_ARGS:="--ginkgo.focus=[Feature:Elasticsearch]"} : ${KUBE_GCE_INSTANCE_PREFIX:="e2e-es-logging"} : ${PROJECT:="kubernetes-es-logging"} # Enable elasticsearch logging addon KUBE_LOGGING_DESTINATION="elasticsearch" ;; </ins> # Runs only the autoscaling tests on GCE. kubernetes-e2e-gce-autoscaling) : ${E2E_CLUSTER_NAME:="jenkins-gce-e2e-autoscaling"}

It failed in gce suite. I honestly have no idea under which team it falls:
Another failure: bumping to p0. - can you find an owner, or at least an appropriate team for this?
owns logging (see team description).
Cannot reproduce failure so far. Leaving it to run overnight...
Still cannot reproduce the exact failure... Increase verbosity...
There were a lot of such failures over the weekend:
Moving this test to flaky (), as it's only 66% stable and is blocking the build quite often: please prioritize this.
This test was passing 100% of the time until Build (Dec 5, 2015 4:13:38 PM). After that, it failed 50% of the time (10 failures in 20 runs). Clearly a PR that merged shortly before the above time introduced the flakyness. That PR should be rolled back.
hmm - I looked into PRs that were merged around that time, and actually nothing looks suspicious to me
This test is 100% successful in flaky test suite so far. I suspect some other e2e test is impacting this one. Adding more sanity checks... (root)/Kubernetes%%20suite/ClusterlevelloggingusingElasticsearchshouldcheckthatlogsfrompodsonallnodesareingestedinto_Elasticsearch/
Let it run in flaky suite overnight. If everything goes well, will unmark it.
I can't follow any of the links but if I could get a bit more information about what is going wrong I could look into the Elasticsearch test if you think that is the issue rather than something else having a bad impact on it.
Thanks for offering to help. Here are the failure symptoms in Jenkins: cluster in red status. I bump into this once with my test cluster. It is cause by es split brain problem. Fix already merged. all log lines from one node. I cannot reproduce this. My guess is the fluentd pod on a node is not working properly. So I validation for fluentd pods. Can you think of anything else?
Can you please point me to the fix for the split brain problem? Is that ? So that works by making sure there are always two master nodes? Is 2 really a good idea for quorum? As suggested here a choice of 2 might not be desirable?

Missing log lines indeed mean a synthetic logging pod did not start for some reason -- without more log information it's hard to work out exactly why this occurred. If I get a moment I will try to do a repo. Thanks.
minimummasternodes set to 2 does not mean having 2 master nodes. It means the es cluster needs to have at least 2 nodes before electing a master. If network partition happened to a es non-master node, it will wait until it could reach out to at least one other node to start electing master. Hence it will prevent the split brain problem. Based on the log in jenkins, all synthetic logging pod ran successfully.
It gets flaky again... I am able to reproduce the problem by running DaemonRestart and Elasticsearch test together. It looks like the DaemonRestart test somehow disrupts the ES cluster. ES cluster gets into bad state, hence failing the es test. Confirmed with This particular test restarts all the containers.
It's a recent regression. Found the cause and will upload a fix soon.
- please monitor this test and close this bug if there won't be any failures in some time.
Test is green for the last 40 builds. Closing the issue.
This test is still marked as at . please send a PR to promote it out of ?
Since is the owner of the issue. I'll let him decide whether the test is ready.
The test has been successful lately. Want to add one more check to improve resilience. I had a pending PR to remove label. Will try to merge it on Monday if everything goes well during the weekend.
I think that is because elasticsearch query return unexpect results. should solve it.
Weirdly hit the same unrelated pr twice, but it looks like the fix just got merged
Still flaking:
what's next on this? Seems to not be resolved...
Moved back into flaky in , is trying for repro/solidifying the test IIUC
Trying to reproduce the problem. It is running successful in flaky suite and my local cluster so far... Will try to run it with other tests together.
Two types of failure: cluster in bad state. This only happens when running with other disruptive tests. Never failed in suite where it is not running with disruptive tests. ES resilience can be improved by log lines in Elasticsearch.

I am able to reproduce this once. It is due to fluentd is hanged. Filed a issue to upstream. Request to mark this test as [Feature] since it depends on two 3rd party components.
The risk here is that ES is a default cluster addon. IMO it's fine not testing it for critical builds, but the risk is we'll ship something that straight crashloops. So if we move it to feature I'd ask that you add a smaller test that verifies basic functionality to a blocking suite (ping es, check that it accepts some logs). Or make it non-default.
Good point. I will work on first. It has to be done anyway. It will also make future ES updates easier. BTW, ES addon is not default for GCE/GKE.
Really? my e2e cluster always has it, why are we testing with it then?
For GCE:
Yeah - another place where test and 'production' configs differ, but I guess we need to test elasticsearch for 'fully opensource' version, thus it's justified...
That's [Feature] then, just like we test flannel. Not something to bring up as an addon on every e2e clusters.
We probably need some suite with all 'important' options (e.g. Features) turned on, on top of dedicated suite for every feature. Ugh...


Question: images []container.Image } <del> // set updates image cache. </del> <ins> // set sorts the input list and updates image cache. // 'i' takes ownership of the list, you should not reference the list again // after calling this function. </ins> func (i *imageCache) set(images []container.Image) { i.Lock() defer i.Unlock() <ins> // The image list needs to be sorted when it gets read and used in // setNodeStatusImages. We sort the list on write instead of on read, // because the image cache is more often read than written sort.Sort(sliceutils.ByImageSize(images)) </ins> i.images = images } <del> // get gets a sorted (by image size) image list from image cache. // There is a potentical data race in this function. See PR #60448 // Because there is deepcopy function available currently, move sort // function inside this function </del> <ins> // get gets image list from image cache. // NOTE: The caller of get() should not do mutating operations on the // returned list that could cause data race against other readers (e.g. // in-place sorting the returned list) </ins> func (i *imageCache) get() []container.Image { i.Lock() defer i.Unlock() <del> sort.Sort(sliceutils.ByImageSize(i.images)) </del> return i.images }

See Today kubelet image manager returns the internal image list cache to caller directly. However, in we sort the image list, which introduces a data race. We should let image manager return a copy of image list instead of returning the list directly. /cc
There are no sig labels on this issue. Please add a sig label. <detailsA sig label can be by either: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. See the . The in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Code Freeze for 1.10 begins in 2 hours. If this issue represents a bug or change for 1.10, then it needs SIG approval in the form of a status/approved-for-milestone label. It also needs a priority/ Thanks!


Question: // A null selector selects no pods. // An empty selector ({}) also selects no pods, which differs from standard behavior of selecting all pods. // In policy/v1, an empty selector will select all pods in the namespace. <del> // +patchStrategy=replace </del> // +optional <del> Selector *metav1.LabelSelector `json:"selector,omitempty" patchStrategy:"replace" protobuf:"bytes,2,opt,name=selector"` </del> <ins> Selector *metav1.LabelSelector `json:"selector,omitempty" protobuf:"bytes,2,opt,name=selector"` </ins> // An eviction is allowed if at most "maxUnavailable" pods selected by // "selector" are unavailable after the eviction, i.e. even in absence of

PDB was upgraded to v1 in 1.21. Despite the changelog saying that the behavior of the v1beta1 implementation should have stayed the same, it appears that the for the field was set to . I believe this is the change in question: When running with a starting config that had a with both a key and a key, and a patch that had a with just a key, the behavior between 1.20 and 1.21 differs. What we get is the key gets deleted, rather than merged with the patch's key to make one unified selector. In 1.20, the behavior we'd get is thus, which is desirable for us because it lets us inject a base assumption about our PDBs into everything we do: This is my repro case: : Compare by using the command from earlier in this report: I also brought this up in Slack: I don't think we want to switch to using just for PDBs, because that wouldn't solve issues where our users want to use without having to re-specify the from our base. Putting aside the fact that changing the behavior of the v1beta1 implementation was probably not intentional, it also seems undesirable to change how selectors work in general. I don't know how widely scoped this change will be across various resource types, too, so we're starting down quite a bit of nasty discovery time. <detailsThis is the version I used to demonstrate my repro case, but I confirmed that this occurs even in kubectl v1.21.0. For my "happy" repro case, this is what the version output looks like: </details<detailsAWS (the hard way) </detailsNo response No response No response No response
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment.

<detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
as requested, I am tagging you and on this one
We should narrowly revert the strategic merge patch selector behavior for v1beta1 to match pre-1.21 behavior for backwards compatibility. More broadly, it is unexpected for multiple actors to be writing to a single selector. The pre-1.21 strategic merge patch behavior for v1beta1 PDB was already inconsistent w.r.t. merging portions of the selector and replacing others (it would merge and replace ) For that reason, server-side apply treats selector writes as owning the entire selector, and the v1 PodDisruptionBudget API will do the same for strategic merge patch. edit: reworded the inconsistencies with the < 1.21 selector handling
More broadly, it is unexpected for multiple actors to be writing to a single selector. The pre-1.21 strategic merge patch behavior for v1beta1 PDB has problems with data getting left in the selector incorrectly if the same user submits a patch with two keys, and then submits a patch with only a single key. Would that be considered a bug in the implementation, or was there some inherent, design-caused race condition in patch application that I don't understand? Cuz if I were to sequentially apply patches with the old strategic merge, I would think the selector would just accumulate keys, which seems fine to me–but I can see how a race condition would turn it into chaos
edit: I was wrong about the previous behavior... the issue was that it already inconsistently merged the selector... it would merge and replace , which is still pretty incoherent If you really want to tweak field-level things, jsonpatch can still do surgical insertion/appending/removal of specific fields.
Should we expect to use jsonpatch for anything that can have a selector, going forward? If it changed for PDBs then it should change for everything else, I would assume edit: we discussed more in slack. The answer is no, the fields for other resource types' label selectors are not changing, so we only need to treat PDBs as a special case.


Question: // serialTestsRegexp is the default skip value if running in parallel. Will not // override an explicit E2E_SKIP value. <del> serialTestsRegexp = "[Serial]" </del> <ins> serialTestsRegexp = "[Serial]" </ins> )

What happened: When using sonobuoy to run some community tests, I realized that ginkgo was not invoked properly. More precisely, when invoking sonobuoy like this: I saw the following in the sonobuoy logs: Because I am running the tests in parallel, and sonobuoy does not set , the code defaults to (see ) for , and is set to . This is not the intended regex (should be ) and all the tests are skipped (because all test names have an ). What you expected to happen: should be set to so that only the tests with the "tag" are skipped. How to reproduce it (as minimally and precisely as possible): Use the sonobuoy command provided above. Environment:
I opened a PR for this
/sig testing


Question: lb := NewLoadBalancerRR() lb.OnUpdate([]api.Endpoints{{JSONBase: api.JSONBase{ID: "echo"}, Endpoints: []string{net.JoinHostPort("127.0.0.1", port)}}}) <del> p := NewProxier(lb) </del> <ins> p := NewProxier(lb, "127.0.0.1") </ins> proxyPort, err := p.addServiceOnUnusedPort("echo") if err != nil {

kube-proxy listens on all ports, but should be configurable. create a flag to specify which interface to listen on. Thanks for finding this bug.
For context, some providers apparently do external IPs by having a whole interface that gets 1:1 NAT without any firewalling. We do not want service ports on those interfaces. , Brian Ketelsen wrote: