Question: }, wantRequeuedPods: sets.New("pod1"), }, <ins> { name:         "Pods with PodTopologySpread should be requeued when a Pod with matching label is scheduled", initialNodes: []*v1.Node{st.MakeNode().Name("fake-node").Label("node", "fake-node").Capacity(map[v1.ResourceName]string{v1.ResourceCPU: "2"}).Obj()}, initialPod:   st.MakePod().Name("pod1").Label("key", "val").Container("image").Node("fake-node").Obj(), pods: []*v1.Pod{ // - Pod2 will be rejected by the PodTopologySpread plugin. st.MakePod().Name("pod2").Label("key", "val").SpreadConstraint(1, "node", v1.DoNotSchedule, st.MakeLabelSelector().Exists("key").Obj(), ptr.To(int32(3)), nil, nil, nil).Container("image").Obj(), }, triggerFn: func(testCtx *testutils.TestContext) error { // Trigger an assigned Pod add event. pod := st.MakePod().Name("pod3").Label("key", "val").Node("fake-node").Container("image").Obj() if _, err := testCtx.ClientSet.CoreV1().Pods(testCtx.NS.Name).Create(testCtx.Ctx, pod, metav1.CreateOptions{}); err != nil { return fmt.Errorf("failed to create Pod %q: %w", pod.Name, err) } return nil }, wantRequeuedPods: sets.New("pod2"), }, </ins> } for _, tt := range tests {

/sig scheduling /kind cleanup Ref: A test helper function creates a goroutine to pop a Pod from the scheduling queue. This goroutine never stops until the scheduling queue is ed or is somehow stopped for waiting for a new Pod. Ideally, when reaching timeout, we should somehow stop from waiting for Pod to be enqueued to activeQ.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
We can replace this function with and it should be enough? I think we no longer need to maintain this function.
Not enough, the core of this problem is that we cannot cancel once we start . Even if we change function to whatever, we will hit this problem. So, we need:
Or, alternatively and much more easily, maybe we can just replace all with and remove . kills the entire test if reaching timeout and we could care less about leaked goroutine then.
I don't think we can do this. From the test case: The expected behavior of this case is that we receive a timeout to make sure no pods are to . So, reaching timeout is acutally a success. So, I think: option 1: using sth like this in : option2: do sth in the test code This part is where it confused me. The current code is fine as we closed schdulerQueue after called , and after I think there is no test case using this this function anymore(seeems that you refactored the test), so are we trying to solve a potential problem?
Oh, then we can just remove NextPod in the PR.
/assign
The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/close
Closing this issue. <detailsIn response to : /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

/sig scheduling /kind bug AssignedPodAdded(): AssignedPodUpdated(): When assigned Pods are created/updated, cluster events (Pod/Add or Pod/Update) should be delivered for all plugins that subscribe those events. But, actually pre-filters Pods to receive those events based on Pods' affinity and . - meaning only PodAffinity/NodeResourceFit receive them actually. The problem here is similar to - in in-tree plugins, PodAffinity, PodTopologySpread and NodeResourceFit subscribe Pod add and/or update events. And, Pods rejected by PodTopologySpread will never be requeued to activeQ by Pod-related events. The same happens for custom plugins registering pod add and/or update events. All Pods will get chances to get back to activeQ by Pod-related events. Run ↓ in No response <detailsmaster </details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
The Kubernetes project currently lacks enough contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale


Question: MASTER_NAME="${INSTANCE_PREFIX}-master" MASTER_TAG="${INSTANCE_PREFIX}-master" MINION_TAG="${INSTANCE_PREFIX}-minion" <del> CLUSTER_IP_RANGE="10.245.0.0/16" MINION_IP_RANGES=($(eval echo "10.245.{1..${NUM_MINIONS}}.0/24")) </del> <ins> CLUSTER_IP_RANGE="${KUBE_GCE_CLUSTER_CLASS_B:-10.245}.0.0/16" MINION_IP_RANGES=($(eval echo "${KUBE_GCE_CLUSTER_CLASS_B:-10.245}.{1..${NUM_MINIONS}}.0/24")) </ins> MASTER_IP_RANGE="${MASTER_IP_RANGE:-10.246.0.0/24}" MINION_SCOPES=("storage-ro" "compute-rw") # Increase the sleep interval value if concerned about API rate limits. 3, in seconds, is the default. POLL_SLEEP_INTERVAL=3

This is an offshoot of . cc Here's a log from a single showing that it's getting it's context screwed over by the other clusters coming up at the same time, even those its local environment never changed: Here's the relevant part of the new script: ( is in a separate commit and allows the override of on the GCE so that this script can work.)
I think this can be resolved by having kube- set the context by exporting KUBECONFIG, so a subsequent run doesn't overwrite it.
After some digging with it's a bit trickier than I thought. If we want to be able to run kube-up in parallel from the same branch/working directory, needs to create a separate kubeconfig for each run. But we also want kube-up to set config such that you can immediately run against the created cluster. Probably best to defer until is resolved.
cc And the script is sitting in


Question: } } <ins> if fieldPatchStrategy == replaceDirective { return false, nil } </ins> // Check the individual keys. return mapsHaveConflicts(leftType, rightType, fieldType) default:

Extracted from The RawExtension type allows an API object to inline arbitrary JSON. strategicpatch.CreateTwoWayMergePatch returns an error trying to map that inlined JSON to struct fields. It should be able to compute a patch for an object containing a field of that type.
cc
We need to decide what strategic merge patch should do in the following scenarios: unknown fields (can be encountered when the API or a user-defined object has a new field the compiled-in struct doesn't have yet) unmergeable fields (like RawExtension) cc
The current behavior is: unknown fields: trying to merge a unknown field will fail when lookup the metadata in the struct tags unmergeable fields: will be replaced as raw byte array Maybe we can use generic json merge for unmergeable fields?
Did that change since 1.5? In 1.5, we were getting an error during patch computation for objects with a RawExtension field
Sorry. This behavior is in 1.4.7. e.g . It will error in 1.5.
Yeah, the field gets replaced. Even in 1.4.7, an unknown field (like initcontainers in pods) or an object including a RawExtension (anything with a field of type or ) would error
My personal favor would be to instruct the patch to ignore such field, seems the cleanest approach to me. Similarly to how we allow annotating deletion and merging.
wouldn't it be confusing to submit a patch with stuff in that field, have the patch accepted, and have none of your submitted changes take effect? seems like the least surprising thing would be to treat struct fields like or as
How about both, iow. having as a default and as an opt-in. This would allow the resource author to decide how his resource should be treated. Eventually on we could return the ignored fields. Having said that we could start with right away. I think I could work on a patch implementing that if majority agrees (
I think is a good place to start when computing/applying patches for known struct fields like RawExtension. I'll open a separate issue for client-side handling of computing patches for unknown fields
Ignore the field sounds like a really good solution. WDYT of printing a warning to stdeer?
what does ignoring instead of replacing get us?
Sorry I was reading the comment on my phone between other things. Ignore my comment.


Question: origCtx, store, _ := testSetup(t) ctx, cancel := context.WithCancel(origCtx) w := store.watcher.createWatchChan(ctx, "/abc", 0, false, false, newTestTransformer(), storage.Everything) <del> // make resutlChan and errChan blocking to ensure ordering. </del> <ins> // make resultChan and errChan blocking to ensure ordering. </ins> w.resultChan = make(chan watch.Event) w.errChan = make(chan error) // The event flow goes like:

While working on generalizing tests to somehow better define spec for storage interface (and share them across all implementation of storage.interface - in particular etcd3 and watchcache), I found an unexpected inconsistency in Delete operation in etcd3. What is happening is that it effectively returns the state of the object from just before the deletion: However, this is inconsistent with how our watch is working, which is returning the state of the object from just before the deletion, but it sets the RV to the moment of deletion (which Delete call isn't doing). This sounds like a bug to me, but given this is a long-standing behavior we may need to take additional care when fixing this. /kind bug /sig api-machinery /priority important-soon /cc
I opened as a fix for it.
/triage accepted


Question: framework.AddOrUpdateLabelOnNode(cs, nodeName, k, v) framework.ExpectNodeHasLabel(cs, nodeName, k, v) defer framework.RemoveLabelOffNode(cs, nodeName, k) // make the nodes have balanced cpu,mem usage <del> createBalancedPodForNodes(f, cs, ns, nodeList.Items, podRequestedResource, 0.6) </del> <ins> err := createBalancedPodForNodes(f, cs, ns, nodeList.Items, podRequestedResource, 0.6) framework.ExpectNoError(err) </ins> By("Trying to launch the pod, now with podAffinity.") labelPodName := "pod-with-podaffinity" pod = createPausePod(f, pausePodConfig{

Chronically failing test. Perhaps covers deprecated behavior? /sig scalability /assign
shows up in
/sig scheduling is one of scheduling e2e test.
/assign I'll check this one.
According to the log, the balanced pods were not created by scheduler, the cpu/mem usage was unchanged.
This test failing every other run. Fix or disable? Blocking 1.7?
similar to , move to 1.8, but I'll still working on that.


Question: return nil } <del> // List names of registered log formats </del> <ins> // List names of registered log formats (sorted) </ins> func (lfr *LogFormatRegistry) List() []string { formats := make([]string, 0, len(lfr.registry)) for f := range lfr.registry { formats = append(formats, f) } <ins> sort.Strings(formats) </ins> return formats }

What happened: 50% of builds have a different man page content What you expected to happen: It should be possible to get bit-identical build results - might need a patch similar to - knows these things. How to reproduce it (as minimally and precisely as possible): build kubernetes1.19 openSUSE package a few times and compare results. e.g. with Anything else we need to know?: Environment: OS: openSUSE-Tumbleweed While working on for , I found that when building the kubernetes1.19 package, there was sometimes such a difference between builds:
/sig release
/sig scheduling


Question: return avoidTimestampEqualities } <ins> func equalIgnoringValueAtPath(a, b any, path []string) bool { if len(path) == 0 { // found the value to ignore return true } aMap, aOk := a.(map[string]any) bMap, bOk := b.(map[string]any) if !aOk || !bOk { // Can't traverse into non-maps, ignore return true } if len(aMap) != len(bMap) { return false } pathHead := path[0] for k, aVal := range aMap { bVal, ok := bMap[k] if !ok { return false } if k == pathHead { if !equalIgnoringValueAtPath(aVal, bVal, path[1:]) { return false } } else if !avoidTimestampEqualities.DeepEqual(aVal, bVal) { return false } } return true } </ins> // IgnoreManagedFieldsTimestampsTransformer reverts timestamp updates // if the non-managed parts of the object are equivalent func IgnoreManagedFieldsTimestampsTransformer(

When continuously re-applying the same object via a controller or kubectl with SSA the resourceVersion increases every time. I expected the resourceVersion to stay the same if the client always sends the same object via SSA, especially if the resulting object doesn't change (apart from resourceVersion & managedFields, but these are managed by the apiserver). Apply this CRD: <details</summary</detailsApply this CR: <details</summary</detailsGet state of the CR after first apply Apply the CR again Get state of the CR after second apply Diff =The resourceVersion and the "time" field of the managedField entry of the "test-manager" increases. The use case we have is a lot more complex than this simple scenario. The tl;dr is that we want to implement caching for our SSA calls, so that we don't have SSA calls on every reconcile of our controller if nothing actually changes. This is relatively hard to do if the resourceVersion increases after every apply. I took a closer look at the APIserver and ~ the following happens: "incoming" SSA patch is applied over the live object from etcd this leads to a change because has which means the field will be dropped Because of that change the "time" field in the managedField entry gets set to "time.Now()" later during admission the namespace field gets defaulted again (I also tried this with a mutating webhook, same result) On a high-level: client always applies the same object defaulting always results in the same object resourceVersion and managedFields[].time continuously go up I think there are many similar cases that would lead to this behavior. Basically whenever the admission chain mutates a field previously set via SSA. Notes: This only happens if there is 1s between each apply, otherwise the managedField[].time is identical and then the resourceVersion stays the same (because we then don't write to etcd) <detailsI also tried this with client-go v0.30 and kube-apiserver v1.30. Same result. </details<detailsRunning this natively on Apple Silicon M2. But can't imagine this makes a difference. Also got the same results in Docker Desktop on Mac. </details
(cc might be interesting for you)
/sig api-machinery
It looks less a bug but more of a intended behavior to me.

/cc /triage accepted
Bypassing the writes triggered by noop applies would break storage version migration (), so we're committed to supporting this behavior as a feature.
I am confused. Are you saying this is intended behavior for a NOP SSA apply to increment resourceVersion? This seems to not actually happen in all cases, only for certain objects/edge cases (see ). For example, try this: No resourceVersion is incremented above). Surprisingly I couldn't find any clear statement that SSA doesn't increment the resource version, but does say "a no-op apply is not very much more work for the API server than an extra GET" which seems impossible if resourceVersion is incremented
is probably only updating it because either the fieldManager changes or in the patch body the apiVersion is different (or both, perhaps)?
We need this in storage version migration to make sure we correctly handles the CRD schema changes. See Joe's comments here - If it helps to understand this better, we have tests to validate this.
I may have misspoke. When storage versions changes, a NOP SSA apply will trigger a resourceVersion bump. If the storage version is unchanged and an update would be a NOP for etcd, I believe we short-circuit the write (?). This might explain the configMap case.
For the reported issue: We've seen cases like this before where the time gets bumped causing a NOP to become a write. This could be improved. Note that this isn't technically a bug because the API server is allowed to bump the resourceVersion for any write (the discussion about storage version changes causing this is a good example of where this can happen in a way that the caller wouldn't expect). But I'd like to see the etcd write optimized away for cases like this.
Sounds similar to perhaps which had some investigation. FWIW as a controller author my expectation is that repeatedly applying does not repeatedly trigger resourceVersion increments. It's fine if it happens rarely (storage version change, etc) but if it happens consistently that means our own writes re-trigger reconciliation and we have an infinite loop of writes. AFAIK all the marketing around writing SSA controllers has been that you just apply what you want and don't try to do a get+diff+apply. Even if we did want to do that, it doesn't really work well currently ().

If we aren't able to rely on NOP writes (usually) being NOPs, I don't see how we are supposed to use SSA to write a controller at all?
This is a really good point. I agree.
looks like it was intended fix this sort of problem. I'm looking into why it doesn't work for this case.
doesn't work for CRs since the managed fields are inside of a unstructured object and are not typed as (for the slow path equality check in IgnoreManagedFieldsTimestampsTransformer).
is a draft proposal for a fix. Feedback welcome on how to make this fix cleaner. I'll test carefully before moving it out of draft.


Question: cancel:   nil, } w.emit = func(e watch.Event) { <ins> if curLen := int64(len(w.outgoing)); w.outgoingHWM.Update(curLen) { // Monitor if this gets backed up, and how much. glog.V(1).Infof("watch (%v): %v objects queued in outgoing channel.", reflect.TypeOf(e.Object).String(), curLen) } </ins> // Give up on user stop, without this we leak a lot of goroutines in tests. select { case w.outgoing <- e:

https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite} Previous issues for this test:
This is pretty much the same failure, that are happening in "gce-enormous" suite and we currently debugging them with The main reason of those failures is that we are terminating all watchers for a particular resource type from time to time, and some of them are actually causing a snowball effect. After more looking into it, my hypothesis is that our machinery for aggregating watchers in apiserver sometimes is just to slow. In particular, I some log in my local branch and what I'm observing is that in etcd_watcher, we are: accumulating objects in incoming channel but also accumulating objects in outgoing channel, and the accumulation there is usually higher [I will send a PR to put those logs into master - they seem to be pretty useful for debugging such issues.] So my current hypothesis is that dispatching watch events to watchers is what is too slow. We will continue debugging.
The logs that I seem to confirm my hypothesis - outgoing channel is generally more full then the incoming channel in etcd_watcher: vs I'm working on the PR that should at least improve the situation.
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.

https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
Please apply the label to this issue if it does not need to be fixed before we launch 1.4. Otherwise please advise on how to proceed with respect to 1.4.
This was a regression - I fixed it by reverting an offending PR: This seems to be fixed now.
I have 33 pods in my container (1 is substantive, the other 32 just run on log files). I get this all the time now. I'm running 1.5.4.
- to be honest, I don't undertsand what your wrote. what is "container" in "33 pods in my container"? Also, if you believe there is a bug somewhere, please open a new issue instead of adding comment to more than half-year-old one.


Question: } cfg := rest.CopyConfig(restConfig) <del> cfg.Host = u.Host </del> <ins> cfg.Host = u.Scheme + "://" + u.Host </ins> cfg.APIPath = u.Path return complete(cfg)

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: In the case of an admission webhook that specifies a url, we only use the Host and Path fields of the parsed url, (url is defined as ) When a service based webhook is constructed, we do define a scheme for the config If the scheme is not specified when the REST client is constructed, then it defaults to http, unless CA, Cert, or config.Insecure is set What you expected to happen: Admission Webhooks should honor the scheme part of the url provided in their client config
/sig api-machinery
/assign


Question: memoryCapacity := node.Status.Capacity["memory"] memoryLimit := memoryCapacity.Value() <del> matchResourceMetrics := gstruct.MatchAllKeys(gstruct.Keys{ </del> <ins> matchResourceMetrics := gomega.And(gstruct.MatchKeys(gstruct.IgnoreMissing, gstruct.Keys{ </ins> "scrape_error": gstruct.Ignore(), "node_cpu_usage_seconds_total": gstruct.MatchAllElements(nodeID, gstruct.Elements{ "": boundedSample(1, 1e6),

Node e2e features is failing after PR for graduating swap to beta is merged. After is merged. Test needs to be adjusted for the extra field for swap usage. No response /sig node
/cc
/assign
/triage accepted /priority critical-urgent


Question: return finalURL.String() } <ins> // Similar to finalURL(), but if the request contains name of an object // (e.g. GET for a specific Pod) it will be substited with "<name>". func (r Request) finalURLTemplate() string { if len(r.resourceName) != 0 { r.resourceName = "<name>" } return r.finalURL() } </ins> // Watch attempts to begin watching the requested location. // Returns a watch.Interface, or an error. func (r *Request) Watch() (watch.Interface, error) {

Currently we do not measure latency for REST operations. This is important for any component that performs non-watch operations (scheduler, controllers, kubelet). This is also prerequisite to actually make sure that all API calls take less than 1 sec ( ).
The apiserver is already , including for REST operations. You can scrape the metrics by hitting the /metrics HTTP endpoint.
Sorry, I was not clear in the description. I'm more interested on the client side and with more details so that I can measure things like: how long it takes to write bindings in scheduler how long it takes to update status in kubelet I'd think about it as a counterpart for apiserver stats.


Question: } // computeReplicasForMetrics computes the desired number of replicas for the metric specifications listed in the HPA, <del> // returning the maximum  of the computed replica counts, a description of the associated metric, and the statuses of </del> <ins> // returning the maximum of the computed replica counts, a description of the associated metric, and the statuses of </ins> // all metrics computed. func (a *HorizontalController) computeReplicasForMetrics(hpa *autoscalingv2.HorizontalPodAutoscaler, scale *autoscalingv1.Scale, metricSpecs []autoscalingv2.MetricSpec) (replicas int32, metric string, statuses []autoscalingv2.MetricStatus, timestamp time.Time, err error) {

What happened: I setup custom metrics based autoscaling with multiple metrics. One of those metrics was not available. HPA reports all metrics as "unknown" (even the ones that are still available) and refuses to operate. What you expected to happen: Pods would be automatically scaled based on metrics that were available, ignoring missing metrics (treating them as zero). A partial failure in the metrics system should not prevent auto-scaling from proceeding with the data that it has. For example, CPU % metrics come from the k8s metrics server whereas the other metrics come from the prometheus adapter. If the prometheus adapter goes away we can still use the CPU metric as a lower bound on the number of replicas. If you consider, e.g. , instead of aborting the whole metric gathering process there, it could have stored a placeholder or zero metric value when it fails to fetch the metric. How to reproduce it (as minimally and precisely as possible): Setup a custom metrics server and tell HPA to scale based on several metrics, one of which is not actually available. Environment: kops 1.8 on AWS Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-01-18T10:09:24Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.8", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-02-09T21:23:25Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
/sig autoscaling
cc
/kind bug /priority important-longterm
this return returns nil value of statuses variable. statuses variable contains the value of all metrics.
as long as we failed to get one metric of all metrics, the func computeReplicasForMetrics returns, the scale lose efficacy, it is not highly available. when failed to get metric, should we record this event and do not return:)
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale Still having this problem.
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
this issue was resolved or it works as designed? thanks,
I was testing this last week and it is not resolved. I don't think it is reasonable to say this is working as designed.
Seems there is a PR to fix it, but it hasn't been reviewed or merged.
yes, I looked into the master branch code, not fixed yet, just curious about it, it would not be fixed or as dobesv said it will be merged/fixed latter with that PR.

/reopen
You can't reopen an issue/PR unless you authored it or you are a collaborator. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen /remove-lifecycle rotten /lifecycle frozen /priority backlog
Reopened this issue. <detailsIn response to : /reopen /remove-lifecycle rotten /lifecycle frozen /priority backlog Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue was discussed at SIG Autoscaling yesterday: I'm going to raise a new PR later this week given the code's changed so much since the original PR was raised by that it doesn't make sense to just rebase it.
Thanks, that would be excellent!


Question: return nil, framework.NewStatus(framework.Unschedulable, msg) } } <del> state.SkipFilterPlugins = skipPlugins </del> return result, nil }

In k8s 1.27, scheduler introduced a new "short circuit" semantics in PreFilter to bypass a Pod quickly. A change introduced here is it relies on that plugin to be stored in , which makes sense as what it was explained . However, the info, although is populated upon each plugin's PreFilter invocation, is not injected into when some PreFilter returns unschedulable: So think about this workflow: pluginA returns Skip in its PreFilter pluginB return Unschedulable in its PreFilter note: pluginA is not included in due to aforementioned bug a Pod is claimed as unschedulable (failed by pluginB) during PostFilter, it's supposed to bypass pluginA's PreEnqueueExtension's AddPod/RemovePod, but it doesn't calling a "skipped" Plugin's AddPod/RemovePod during preemption, sometimes will abort the whole preemption as the logic tries to read info from cycleState, and some plugin's "skip" version doesn't fill cycleState properly, e.g., InterPodAffinity (skip immediately w/o write preFilterStateKey). Short circuiting using Skip is fine, but should injected properly at all cases. Described in the workflow above. It's firstly captured in an out-of-tree PostFilter plugin: , but can also happen in in-tree plugins. No response v1.27 <details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/kind regression /sig scheduling
cc
Among in-tree plugins, 3 plugins employ non-nil and may return Skip in their PreFilter() implementation w/o writing to cycleState, while return error immediately if is missing in their Add/RemovePod() functions.

So technically, as long as a scheduling flow hit the following criteria, it would hit the bug (preemption doesn't function): plugin 1/2/3 return Skip in PreFilter arbitrary PreFilter plugin after the "skipped" plugin returns Unschedulable in its PreFilter
/cc
/cc
Cherry-pick PRs: 1.28: 1.27:


Question: fsckErrorsCorrected = 1 // 'fsck' found errors but exited without correcting them fsckErrorsUncorrected = 4 <ins> // Error thrown by exec cmd.Run() when process spawned by cmd.Start() completes before cmd.Wait() is called (see - k/k issue #103753) errNoChildProcesses = "wait: no child processes" </ins> ) // Mounter provides the default implementation of mount.Interface

While investigating some slowness in mounting/unmounting operations of PVs (in this case EFS volumes) for pods, we observed errors like these: fortunately it does recover quickly on Kubelet’s next retry: Looking through the Kubelet code I found out that the issue is not related to the umount calls themselves, but rather a race condition inside where we call umount. Because of the way golang’s function (called inside ) is implemented, it for the child process to finish after it is . Since there isn't any locking on the child process across those function calls, it's possible that the child process exits event before the Wait() call starts. In such case the wait call returns with a error (). This error is more likely to happen especially if the gap between the Start() and Wait() calls is more (which could for e,g happen if the goroutine had CPU cycles snatched away from it for some time in between those calls). A similar issue also exists in the . While this is a self-recovering issue (because of Kubelet retries), it does seem to increase the pod startup/cleanup latency upto a 1s. Open to suggestions for a fix. The simplest one occurring to me is to treat the error as a special one on which we don't actually return an error for the mount/unmount calls. /sig storage /sig scalability /assign
Another thing is when analyzing a distribution of such errors across different nodes, we observed a skew. The top 10% of the nodes were seeing more than 90% of these errors. And these are nodes that are running hotter (w.r.t the pod churn) compared to others: ! Given this, I'm wondering if those errors are higher on those nodes just because of statistical significance or other causes like more GC pauses happening on those nodes.
/triage accepted
^ I've opened this PR to fix the Mount/Unmount functions in the file. In general we might want to make a similar fix to other invocations of exec.Run() and exec.CombinedOutput() within our codebase. But that'll be a bigger change. Happy to open another issue for it.
Sorry for opening the PR against wrong repo, here's the right one -


Question: # TODO(dawnchen): Check $CONTAINER_RUNTIME to decide which # cloud_config yaml file should be passed <ins> # TODO(mbforbes): Make $1 required. # TODO(mbforbes): Document required vars (for this and call chain). # $1 version </ins> function create-node-instance-template { <del> create-node-template "${NODE_INSTANCE_PREFIX}-template" "${scope_flags[*]}"  </del> <ins> local suffix="" if [[ -n ${1:-} ]]; then suffix="-${1}" fi create-node-template "${NODE_INSTANCE_PREFIX}-template${suffix}" "${scope_flags[*]}"  </ins> "kube-env=${KUBE_TEMP}/node-kube-env.yaml"  "user-data=${KUBE_ROOT}/cluster/gce/coreos/node.yaml" }

See issue as the roll-up for node upgrades. tl;dr: Update MIG template, then call a MIG rollingupdate. More details: When upgrading nodes, an in-place upgrade must worry about leftover state on the boot disk, VM configuration like IP tables, and the idempotency of salt. Getting a fresh VM instead would remove these concerns and provide a clean state for a new node. Furthermore, a versioned node spec () includes the tuple of (kubelet, kube-proxy, docker, kernel/OS). In the general case, changing docker or the kernel/OS means we have to stop the running pods. Once know we have to stop the pods, the simplest upgrade mechanism is to totally down the node and provision a new one. Provisioning a new node is also a great backup to have for node repairs. To be clear, though, this issue is entirely separate from node repairs; once we have the capability to grow the cluster by adding new nodes, this mechanism, and other repair mechanisms, will build on top of that capability. This was blocked by: nodes joining a running cluster (template-only metadata) References: The most relevant discussion regarding the worries of complications with state while doing in-place upgrades is .
FWIW, I think we need this as a fallback to repair completely borked nodes? That is, it may not be necessary for upgrades per se, but could end up being independently useful.
Thanks for the comment, I updated the description to be clear about what this issue is addressing. Having the ability to add new nodes to a cluster will both enable this upgrade path as well as give us a great "hammer" to use for repairs.
Working on this now.


Question: # This file creates a standard build environment for building cross # platform go binary for the architecture kubernetes cares about. <del> FROM golang:1.11.1 </del> <ins> FROM golang:1.11.2 </ins> ENV GOARM 7 ENV KUBE_DYNAMIC_CROSSPLATFORMS

go 1.11.2 is out 11/2 and this bug tracks upgrade to the latest version. Is it something we should absolutely needed in the 1.13 timeframe? are there any that should absolutely take? how long does the associated image updates and push take? If we chose to upgrade in 1.13, we should do it before code slush targeted for Friday 11/6. /assign can you please clarify the above ques and and help evaluate timing of the upgrade. /cc /sig release /sig testing /priority important-soon /kind cleanup
/cc I’ve already got a branch cooking for this, but waiting for rules_go support first. I think said in Slack that this will happen this week.
Ahh so is the earliest expected timeframe when we can upgrade falls into Code slush period for 1.13 (post 11/9?) Depending on when it lands we might want to see if its safe to wait for 1.13.1. are you aware of any critical fixes that make this compelling for 1.13 timeframe?
The risk for patch golang point releases is typically low (the minor releases, like 1.10 -1.11, are higher risk). I don't see anything that's critical or of known impact, but at the same time, I see compiler fixes and go/types package fixes. They are more subtle and could present issues later. I'd suggest that if we can tolerate getting it in, even if we're in slush, we should.
Ack that :) Plz ping this bug as soon as we are ready to build and push the upgraded images. we'll will make a final call depending on the timeframe and CI signal going in. thanks!


Question: "github.com/GoogleCloudPlatform/kubernetes/pkg/api" "github.com/GoogleCloudPlatform/kubernetes/pkg/client" <del> "github.com/GoogleCloudPlatform/kubernetes/pkg/kubelet/leaky" </del> "github.com/GoogleCloudPlatform/kubernetes/pkg/labels" "github.com/GoogleCloudPlatform/kubernetes/pkg/registry/pod"

Kubelet now only reports ContainerStatus. PodStatus / PodCondition is observed at master side based on a set of ContainerStatus. There is no other way to get PodIP except net container for master side. This is very confusion to the end user, , , ... Now we are finalizing on v1beta3 API, including PodStatus & ContainerStatus. We can make net container completely invisible to the end users by finishing the following steps: 1) Expose network related errors at API level 2) Generate PodStaus directly from Kubelet
3) once PodStatus generated by kubelet, PodIP can be removed from ContainerStatus.
I think reporting PodStatus rather than just container status is most important. How hard is that?
Used to be hard since there are many dependencies when I first report Status back. Now it is totally feasible. I already started to write PRs for this. :-)
One more PR required to close this one.


Question: e2enode "k8s.io/kubernetes/test/e2e/framework/node" e2epod "k8s.io/kubernetes/test/e2e/framework/pod" imageutils "k8s.io/kubernetes/test/utils/image" <ins> k8utilnet "k8s.io/utils/net" </ins> ) const (

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: https://k8s-(IPv6),%20master%20(dev) Which test(s) are failing: [sig-scheduling] SchedulerPredicates [Serial] validates that there is no conflict between pods with same hostPort but different hostIP and protocol [Conformance] Since when has it been failing: Testgrid link: https://k8s-(IPv6),%20master%20(dev) Reason for failure: The tests checks for 0.0.0.0 that is IPv4 only Anything else we need to know: Probably I should check with SIG-Testing how can we add an ipv6 job and update the KEP accordingly EDIT Probably I should check with SIG-Testing how can we add an ipv6 job as part of the k/k presubmits, and update the KEP accordingly
/assign /cc
/sig scheduling

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Introduce a mechanism that allows specifying the IP protocol version used in the e2e testing. Why is this needed: This is a followup on , my understanding is that in order to discriminate between IPv4 or IPv6 we have to use the test labels and this adds complexity to the testing automation and doesn't allow to send this information to the context of the tests. Having a mechanism to specify the IP protocol will simplify the testing for IPv4 and IPv6 and will allow reusing most of the code.
/sig testing /cc /assign Does this make sense? If affirmative, what should be the best way to add this functionality? Checking the code I think that or can provide this, don't know if checking some of the cluster config parameters, ...
the testing-commons group (sig-testing) maintains the framework, but ideally the decision on how to proceed in terms of adding ipv6 detection should come from sig-network. possibly having this as agenda for a sig-network meeting is a good start, then discussing it in a testing-commons meeting. /sig network the linked issue is a big one, so i don't have all the context here. but IMO existing tests should just work for ipv6 without the requirement for a new "IPV6" tag in test names. dual-stack or protocol specific tests should have their descriptions contain the details properly (i.e. what is this test doing) we might want to add an utility function under: (there is a PR to move it BTW ) then such an utility might be useful only to tests that need to detect the protocol. one of the goals is to refactor the code in testcontext / framework, so adding this logic there is less preferred.
/area e2e-test-framework
I think that we can add an option to test_context to specify the cluster IP family the same way that exists one for the cloud provider this is simpler because when setting up the CI jobs you will know beforehand what IP family you want to test, it also will allow reusing all the e2e code for IPv6 and Dual Stack (cc: and by default will be IPv4, so it´s backward compatible.
Being able to override and specify exactly what you want the test to beleive about your cluster is nice, but ideally it should generally work without specifying more flags.

Surely we can detect this?
I was thinking during some time about this, detecting the IP family is very simple having IPv4 or IPv6 only clusters, but with the introduction of dual stack I´m afraid about the supportability and reliability of the autodetecting mechanisms in the long term. If you see the dual stack PRs there are still a lot of unknowns about the cluster behavior. My point is, let´s have something first stable that works and is simple, and once dual stack is implemented, we can replace the flag with a function in the framework . I guess that migrating from a flag variable to a function is straight forward, is just replacing by , what do you think?
in is as part of the context. do you mean adding as a function which will auto-detecting the ip family. less flags for the framework binary is preferred.
once we have this I will open a new issue to start cleaning up and , we no longer need to discriminate based on tags


Question: sleep 1 done <del> # Wait for the node to not have SchedulingDisabled=True and also to have # Ready=True. </del> <ins> # Uncordon the node. echo "== Uncordon ${instance}. == " >&2 local uncordon_rc "${KUBE_ROOT}/cluster/kubectl.sh" uncordon "${instance}"  && uncordon_rc=$? || uncordon_rc=$? if [[ "${uncordon_rc}" != 0 ]]; then echo "== FAILED to uncordon ${instance} ==" return ${uncordon_rc} fi # Wait for the node to have Ready=True. </ins> echo "== Waiting for ${instance} to become ready. ==" >&2 while true; do <del> local cordoned=$("${KUBE_ROOT}/cluster/kubectl.sh" get node "${instance}" --output='jsonpath={.status.conditions[?(@.type == "SchedulingDisabled")].status}') </del> local ready=$("${KUBE_ROOT}/cluster/kubectl.sh" get node "${instance}" --output='jsonpath={.status.conditions[?(@.type == "Ready")].status}') <del> if [[ "${cordoned}" == 'True' ]]; then echo "Node ${instance} is still not ready: SchedulingDisabled=${ready}" elif [[ "${ready}" != 'True' ]]; then </del> <ins> if [[ "${ready}" != 'True' ]]; then </ins> echo "Node ${instance} is still not ready: Ready=${ready}" else echo "Node ${instance} Ready=${ready}"

https://k8s- /kind bug /priority failing-test /priority important-soon /sig testing /milestone v1.11 cc /assign for triage
GitHub didn't allow me to assign the following users: for, triage. Note that only and repo collaborators can be assigned. <detailsIn response to : ## Failing Job https://k8s- /priority failing-test /priority important-soon /sig testing /milestone v1.11 Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I don't think this is sig-testing. Looks like , , /unassign /remove-sig testing
/sig cluster-lifecycle /assign can you see if this is a cluster lifecycle issue please?
Chiming in for cluster-lifecycle, this is weird. Somehow the nodes are being put back into a cordoned state after the upgrade. The upgrade checks that after a node is upgraded, it becomes both ready and schedulable. Here's an example from the linked failure: But just a few lines below that, we can see the the cluster validation after the upgrade shows that 2 nodes are still unschedulable. Does someone from the node team know why this might be happening? Is it a race condition? It worked for one node, but not two in this case. In other runs, I see it working for two nodes and only leaving one unschedulable. cc
/cc can either of you please help investigate the node side of this issue to unblock This is currently on of the few tests failing our sig-release upgrade blocking suite. If this is not fixed we will have to delay beta for 1.11 which is slated for 5/15.
/cc
cc
/cc I am also running into this issue when trying to add a new upgrade test. First, I think the reason that the test log said the node was ready is because the code is looking at a wrong field for SchedulingDisabled.

The unschedulable field is a field in not a node status condition, so I think we should instead use: local cordoned={KUBE_ROOT}" get node "${instance}" --output='jsonpath=="{.spec.unschedulable}"' I am also puzzled on why recreating instance just rebooted the node. According to 'gcloud instance-groups managed recreate-instances' doc, that command should delete the instance first and create a new one. For the passed cluster upgrade test, that seems the case, but for the failed ones, I saw node was just rebooted and so still got the old unschedulable state from the API server. While trying to figure out why recreating instance just rebooted node sometimes, would it make sense we explicitly use 'gcloud instance-groups managed delete-instances' and 'gcloud instance-groups managed create-instances' ?
We use recreate-instances to maintain the node identity and name. How are you determining that it just rebooted and not recreated? Did the nodes stop recreating their node objects on reboot and no longer clear out the field? Is this a consequence of ? Since is no longer unique across , the node object doesn't get recreated? Would that mean the upgrade script has to uncordon the node now?
I think theory at is right. After was merged, .spec.externalID is mapping to spec.instanceID which wouldn't be unique after recreate. In this case, Kubelet won't delete old node object and recreate new one. I think this is what we want: we want to stop the node to delete and create itself, we want to stop the node to mess up its taints and labels. Instead, we want the controller(s) to explicitly set the taints and labels to a node. also updated upgrade script to use boot id to detect the new boot or recreation: In this very case, we should update upgrade script to explicitly clear the unschedulable field once new boot id was detected.
Thanks, I'll work on a PR and send it out today.

From my manual run, I ssh-ed into the unschedulable node after the test and saw the kubelet log contains multiple restarts and logged the event that the node was previously registered and then later changed to NodeNotSchedulable based on the spec got from the API server: 1754: May 08 18:03:18 e2e-test-jiayingz-minion-group-ssbv kubelet[1073]: I0508 18:03:18. 1073 ] Node e2e-test-jiayingz-minion-group-ssbv was previously registered 1755: May 08 18:03:18 e2e-test-jiayingz-minion-group-ssbv kubelet[1073]: I0508 18:03:18. 1073 ] Successfully registered node e2e-test-jiayingz-minion-group-ssbv ... 1789: May 08 18:03:18 e2e-test-jiayingz-minion-group-ssbv kubelet[1073]: I0508 18:03:18. 1073 ] Event(v1.ObjectReference{Kind:"Node", Namespace:"", Name:"e2e-test-jiayingz-minion-group-ssbv", UID:"e2e-test-jiayingz-minion-group-ssbv", APIVersion:"", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'NodeNotSchedulable' Node e2e-test-jiayingz-minion-group-ssbv status is now: NodeNotSchedulable
That fits with the current theory. The instance is being recreated, but the kubelet is no longer re-registering itself since the externalID is the same.
/cc
was sufficient to fix this issue or are we also waiting on to b merged? I am still waiting for the latest run to verify if the issue is fixed.
/reopen There's a bug with my fix. It fails if the uncordon command happens in the window where the node object doesn't exist. I will have a fix out shortly.
you can't re-open an issue/PR unless you authored it or you are assigned to it. <detailsIn response to : /reopen There's a bug with my fix. It fails if the uncordon command happens in the window where the node object doesn't exist.

I will have a fix out shortly. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/reopen
Note that this bug is flaky. It doesn't happen 100% of the time. Looks like we got 100% pass in the latest run: https://k8s- Here's an upgrade test that's hitting this flake: https://k8s-
cc Please see my comments at for the change Before the change, when instance is recreated, kubelet will delete the node object and a new one will be created. Now, kubelet will not delete the old one if the node name is the same, I think. This might give some trouble for volumes because the old node will contain the outdated volumesAreAttached information. But we do sync up with the cloud provider and update this information eventually. I am not sure whether this old object might cause other problems or not.
Can you move that to another issue? It is a legitimate bug that the upgrade logic should not rely on kubelet recreating the node object. I agree that no longer doing that may lead to other issues, but that should be a separate discussion.
I see the test failing in the last few runs. Is it a different issue plaguing the test now? https://k8s-
/reopen
[MILESTONENOTIFIER] Milestone Issue Needs Approval Action required: This issue must have the label applied by a SIG maintainer. If the label is not applied within 1 day, the issue will be moved out of the v1.11 milestone. <details<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
Here are the since the last failure.
Spoke to offline and SkewTest is a overall for the second half of the suite. So if the other tests in the job pass, this one should turn the corner. so closing this bug for now.


Question: createConfig.HostConfig.NetworkMode = dockercontainer.NetworkMode(networkMode) } else { // Todo: Refactor this call in future for calling methods directly in security_context.go <del> modifyHostNetworkOptionForContainer(false, podSandboxID, createConfig.HostConfig) </del> <ins> modifyHostOptionsForContainer(false, podSandboxID, createConfig.HostConfig) </ins> } return nil

https://k8s- https://k8s- cc: /kind bug /priority failing-test
/cc
Looking at the k8s-merge-robot in PR , it says: "pull-kubernetes-cross Skipped" (That's under "k8s-merge-robot merged commit into kubernetes:master 17 hours ago") Can someone tell why that test was skipped there? Wasn't the bot supposed to catch that breakage at that point instead? Just sent to fix this, someone please review. Thanks, Filipe
because the cross build is pretty expensive and takes a while, we only run it when we think a PR is likely to cause problems, based on the files it touches (regex at ). it's obviously not perfect - nothing in your PR immediately suggested there'd be a cross-build failure.
Ah ok I see thanks for the clarification!


Question: oldPod := oldObj.(*api.Pod) newPod := newObj.(*api.Pod) if core.QuotaPod(oldPod) && !core.QuotaPod(newPod) { <del> options.ReplenishmentFunc(options.GroupKind, newPod.Namespace, newPod) </del> <ins> options.ReplenishmentFunc(options.GroupKind, newPod.Namespace, oldPod) </ins> } } }

master-informing: capz-windows-containerd-master capz-e2e.[It] Conformance Tests conformance-tests ! 2023-03-10 04:56:51 +0000 UTC No response /sig testing
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : ### Which jobs are failing? ### Since when has it been failing? /sig testing Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig windows /triage accepted
no major change between passing test on 3/9 17:55 PST and failing test on 3/9 20:56 PST
containerd logs have error about failing to unpack azure-cloud-manager on both nodes:
kubelet logs confirm this:
looks to be related to bumping capz to 1.8 which uses the cloud-provider oot image: caused by upgrading to 1.8 capz in combination with using OOT
This looks to be affecting our release jobs as well:
looks like the image being built is linux only () and is being pulled by windows. When setting up the snapshot in containerd it is failing (as it is missing files required by windows)
For the passing test the image is multi-arch manifest, so the windows image pulls properly (azure-cloud-node-manager:)
There is a missing env in the see


Question: if _, exists, _ := p.podBackoffQ.Get(pInfo); exists { return fmt.Errorf("pod is already present in the backoffQ") } <del> if podSchedulingCycle > p.moveRequestCycle && isPodUnschedulable(pod) { p.backoffPod(pod) p.unschedulableQ.addOrUpdate(pInfo) p.nominatedPods.add(pod, "") return nil } // If a move request has been received and the pod is subject to backoff, move it to the BackoffQ. if p.isPodBackingOff(pod) && isPodUnschedulable(pod) { err := p.podBackoffQ.Add(pInfo) if err != nil { klog.Errorf("Error adding pod %v to the backoff queue: %v", pod.Name, err) } else { p.nominatedPods.add(pod, "") </del> <ins> // Every unschedulable pod is subject to backoff timers. p.backoffPod(pod) // If a move request has been received, move it to the BackoffQ, otherwise move // it to unschedulableQ. if p.moveRequestCycle >= podSchedulingCycle { if err := p.podBackoffQ.Add(pInfo); err != nil { // TODO: Delete this klog call and log returned errors at the call site. err = fmt.Errorf("error adding pod %v to the backoff queue: %v", pod.Name, err) klog.Error(err) return err </ins> } <del> return err </del> <ins> } else { p.unschedulableQ.addOrUpdate(pInfo) </ins> } <del> err := p.activeQ.Add(pInfo) if err == nil { p.nominatedPods.add(pod, "") p.cond.Broadcast() } return err </del> <ins> p.nominatedPods.add(pod, "") return nil </ins> } // flushBackoffQCompleted Moves all pods from backoffQ which have completed backoff in to activeQ

We've noticed a strange behavior of the scheduler in GCE 5K node scalability tests. There were periods of time, lasting about 6s, when scheduler didn't schedule any new pods. It's ruining our 5s pod-startup-latency SLO. See the example below, pod density-latency-pod-2325-dgqrv created on 09:08:59, scheduler didn't schedule it until 09:09:06. An interesting pattern can be found in the around that time. Before 09:09:00 scheduler schedules ~5 pods / s. Then it goes silent for about 6s and at 09:09:06 it outputs about 30 "bound successfully" statements and continues scheduling pods with ~5 pods / s pace. We checked api-server logs for that silent period of time and it turned out that scheduler was doing something, it was constantly sending GET and PUT request for a metrics-server and heapster pod it was trying to schedule. ! We dug deeper and found that scheduler was actually trying to schedule these two pods for over 12h, below is the plot of number of calls per minute that scheduler issued to api-server for these two pods: ! As we can see there are long periods of time when scheduler issues over 1500 requests per minute. That seems strangely a lot given that the requests are only for the two pods. During the whole run scheduler issued over 400K GETs and over 200K PUTs for these two pods. ! We figured out that scheduler wasn't able to schedule these two pods because there was not enough resources in the cluster. While it's understandable that scheduler wasn't able to schedule the pods, I'd expect scheduler to not spam the system with so many retries, which, as we can see in this example, can potentially block / slow-down scheduling of other pods. I've increased the scheduler log verbosity level in , we should have more logs from the tomorrow's run. Will post them here once available.
/sig scheduling
Reiterating the mentions to trigger a notification: <detailsIn response to : /sig scheduling </details
/assign for triage
What is the version of your cluster? If you are using the HEAD, we have a back-off mechanism to the scheduler to delay rescheduling unschedulable pods.

It starts with 1 second delay and doubles it every time the pod is not schedulable up to 10 seconds.
It's from head, so apparently something doesn't work...
Yes, we're using head. The test run on Jan 21 and the cluster was built from head on the same day (commit to be more precise). As you can see in the logs it doesn't look like scheduler is using any back-off mechanism. Definitely not the one you described, it issues requests to api server much more often than 1 per second per pod. !
An update on more logs - we won't have more logs. We tried increasing scheduler's log level to v4 and it resulted in the log growing by few orders of magnitude and it was clogged with mostly useless stuff, e.g. it logged "Node gce-scale-cluster-minion-group-XXX is a potential node for preemption." over 250 000 000 times. We reverted the log level change in If more scheduler logs are needed to debug this we probably should add some new, more useful, ones.
cc/
/assign I can look into this next Tuesday
Friendly ping, any updates?
Whoops. Thanks for the ping. Log level v4 should not be needed. Although the log line you called out should probably get pushed into V9... First off, that's some really great debugging. Thanks for the detailed report. If the scheduler is getting into tight busy loops, that could in theory starve the scheduling thread, causing the observed 6 second pause in scheduling. The bug is in  A pod only gets marked for backoff if . And can increment very quickly in a high churn cluster. This would result in a pod being retried in a tight loop. One (unrelated) observation that stood out to me was this: This is being printed by the package and shows a trace with no steps logged to it, which means that we are hitting one of these four error cases in . Most likely, the PVC for those heapster & metrics-server pods are in a bad/unexpected state.
I think fixes the root problem here. If we eliminate the API server spam, the main scheduling thread hopefully will not get blocked for 6 second periods. Are you able to test with this patch?
Thanks Jonathan!

I've reserved a scale-testing slot for Thursday (Feb 21) to test the fix. Will update this issue once I have the results.
I reproduced the issue (non-schedulable heapster pod due to not enough resources) on a 5K node cluster. Then I run the load test, first on the version without the fix and the on the version with the fix. Below are the graphs of the number of scheduler requests (for the unschedulable pod) per minute. Without fix: ! With fix: ! As we can see on the graphs, the fix definitely helps, without it there were minutes were scheduler was calling api server up to 2K times per minute (2K QPM). With the fix it didn't exceed 100 QPM. But even with the fix the average QPM is ~60 (per single pod) which is 1 query per second per pod on average. is average 60 QPM (1 QPS) reasonable here or should we expect something less from the rate-limiting mechanism?
By default the scheduler rate limit is 20 qps for all scheduler operations. Default backoff delay for unschedulable pods is 1 second and goes exponentially higher every time a pod is retried, but does not exceed 10 seconds. This 10 second max backoff time also means that if a pod is not retried for 10 seconds, for example because there are many other pods in front of it in the queue, its backoff time is reset to 1 second. So the next time it is retried and is found unschedulable it will be backed off for 1 second. I am basically trying to say that 1 qps is not unreasonable. We also an optimization () that should lower qps for unschedulable pods further down. This last PR got merged only a few hours ago. With that PR merged, I would expect to see even lower qps as the scheduler will not update pod status every time it retries scheduling a pod.
Thanks Bobby! That makes sense. In that case I believe we can close this issue once merges.
/milestone v1.14 We're entering and this looks relevant to the 1.14 release. Please if I am incorrect
Thanks for fixing!


Question: return ret, nil } <ins> // crdObjectConverter is a converter that supports field selectors for CRDs. type crdObjectConverter struct { unstructured.UnstructuredObjectConverter clusterScoped bool } func (c crdObjectConverter) ConvertFieldLabel(version, kind, label, value string) (string, string, error) { // We currently only support metadata.namespace and metadata.name. switch { case label == "metadata.name": return label, value, nil case !c.clusterScoped && label == "metadata.namespace": return label, value, nil default: return "", "", fmt.Errorf("field label not supported: %s", label) } } </ins> func (c *crdHandler) updateCustomResourceDefinition(oldObj, _ interface{}) { oldCRD := oldObj.(*apiextensions.CustomResourceDefinition) glog.V(4).Infof("Updating customresourcedefinition %s", oldCRD.Name)

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: got an error when trying to use a fieldSelector when watching a custom resource What you expected to happen: no error How to reproduce it (as minimally and precisely as possible): a CRD and then a CR for it. to watch with a fieldSelector, such as Anything else we need to know?: The customresource implementation has a defined, and it looks like it should support at least a field selector for and , but I'm guessing we need a bit more in place to make this work for . Environment: Kubernetes version (use ): HEAD (commit ) /sig api-machinery I can try to submit a PR to fix once I finish my current work item, if nobody picks this up sooner.
/area third-party-resource
cc
This is a more general form of . I don't have immediate plans to work on either of them, so you're welcome to take it.
sounds good. Do you want me to close this as a dupe?
I think there's a useful distinction between the two issues. If we support only , that would be enough to but this one tracks the idea of supporting arbitrary field selectors. By the way, if I recall correctly, when I applied the hack mentioned in the other thread to skip the step for CRDs, I think I saw that actually arbitrary field selectors worked too (not just ). That would mean enabling this might be easy on a technical level, although we may need to think through whether this could tie our hands if we ever wanted to add versioning support for CRD in the future. This issue is a better home for that discussion than the other one.
This would help a lot with debugging in prow. Today, the only thing we can do is bubble up information from the spec of a prowjob as labels and use a label selector.
What about just having 's just return the passed in label and value without any "conversion" for now?
I don't think we'd want to do it for all Unstructured, since there's no guarantee all Unstructured objects are CRD objects. However, it should be pretty easy to arrange for ConvertFieldLabel to be a no-op for CRD objects, which makes sense currently because CRD only allows one version for a given resource.

We just need to make sure this will not prevent us from doing more with CRD versioning in the future. My suspicion is that it should be fine, but I haven't investigated properly.
ok, I can put together a PR to make this change. any gut reactions?
PR:
I am sorry to comment on this closed thread. Currently, I am working on a CRD and needs to support arbitrary field selectors. I've tried to do it by learning from what you do for . But it always throws error: . So, I am wondering if I am doing something wrong or it only supports limited field selectors for CRD, which is stated . Here is my code snippet:
CRDs still only support namespace and name for field selectors
Any plan to support arbitrary field selector for CRDs?
Not currently

Follow a TPR an instance a single instance return 400. (watch a collection of TPR works) master and kubectl version: 1.7.1 CRD has the same issue. /assign
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
I was able to reproduce this with CRD on master. The problem is that kubectl implements a watch on a single object as a watch on the collection with a field selector that filters by . However, field selectors don't work on CRD objects because they are stored as , which does not support : This method is used to transform the field selector from versioned to internal field names: As a test, I bypassed that check for Unstructured objects, and watches on single CRD objects started working. Given that CRD doesn't support multiple versions anyway, it seems feasible to support this with a properly-targeted override of specific to CRD objects. cc
If we could support name and namespace generally on all objects containing object meta, I think I'm in favor of the concept.
I think it's feasible to fix this for CRD at some point. However, given that TPR is already gone in master, I don't think it's worth fixing for TPR unless it's somehow a regression in 1.7. Do you have any objections to that?
Having CRD support is good enough for me and service catalog. Will you backport the fix for CRD to 1.7?
At this stage I wouldn't expect that since it isn't a regression (I don't think) for either extension mechanism.
cc
I am also getting the same error when i try to watch CRDs and filter on a specific field. Im a currently using the to watch for CRDs. On a related note, i noticed that the takes a . Is there a way to pass a label selector so that i can watch CRDs using labels?
Also is this issue slated for 1.8?

(For CRDs that is)
The general issue of supporting field selectors for CRD is tracked in . I don't think it's slated for 1.8. The methods on should take a , which can contain an optional .


Question: framework.Failf("Error retrieving logs: %v", err) } <ins> if waitErr != nil { framework.Logf("Pod log:n%s", logOutput) framework.Failf("Pod did not stop running: %v", waitErr) } </ins> By("Verifying logs") verifyLogMessage(logOutput, "^Accepted client connection$") verifyLogMessage(logOutput, "^Received expected client data$")

https://k8s- Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends no data, and disconnects [Conformance] {Kubernetes e2e suite}
So far I haven't found anything in the logs. The pfpod container starts, the e2e test receives all the data from it. At that time, the container should exit and the pod should be stopped. For some reason, either the container doesn't stop, or the kubelet doesn't see it stop. It's hard to tell which because the logs don't have enough information to make that determination.
https://k8s- Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends no data, and disconnects [Conformance] {Kubernetes e2e suite}
Might be the same issue as
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
Builds: kubernetes-e2e-gce-etcd3 kubernetes-e2e-gci-gce Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends no data, and disconnects [Conformance] {Kubernetes e2e suite}
https://k8s- Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends no data, and disconnects [Conformance] {Kubernetes e2e suite} Happened on a presubmit run in .
https://k8s- Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends no data, and disconnects [Conformance] {Kubernetes e2e suite}
Adding to assignee list b/c of ssh tunnel flakes (i.e.

this issue has multiple, different root causes)
Builds: ci-kubernetes-e2e-gce-etcd3 ci-kubernetes-e2e-gci-gce-etcd3 ci-kubernetes-e2e-gci-gce-proto kubernetes-e2e-gce-etcd3 kubernetes-e2e-gci-gce-kubenet kubernetes-e2e-gci-gce-proto kubernetes-e2e-gci-gce-release-1.4 kubernetes-e2e-gci-gke-prod-smoke Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends no data, and disconnects [Conformance] {Kubernetes e2e suite}
https://k8s- Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends no data, and disconnects [Conformance] {Kubernetes e2e suite}
Closing this issue as the corresponding test has been removed from cc:


Question: _, err := client.CoreV1().Secrets(namespace).Create(secret) framework.ExpectNoError(err, "creating secret %q in namespace %q", secretName, namespace) <ins> // kubectl create -f clusterrole.yaml _, err = client.RbacV1beta1().ClusterRoles().Create(&rbacv1beta1.ClusterRole{ // role for listing ValidatingWebhookConfiguration/MutatingWebhookConfiguration/Namespaces ObjectMeta: metav1.ObjectMeta{Name: "sample-apiserver-reader"}, Rules: []rbacv1beta1.PolicyRule{ rbacv1beta1helpers.NewRule("list").Groups("").Resources("namespaces").RuleOrDie(), rbacv1beta1helpers.NewRule("list").Groups("admissionregistration.k8s.io").Resources("*").RuleOrDie(), }, }) framework.ExpectNoError(err, "creating cluster role %s", "sample-apiserver-reader") _, err = client.RbacV1beta1().ClusterRoleBindings().Create(&rbacv1beta1.ClusterRoleBinding{ ObjectMeta: metav1.ObjectMeta{ Name: "wardler:" + namespace + ":sample-apiserver-reader", }, RoleRef: rbacv1beta1.RoleRef{ APIGroup: "rbac.authorization.k8s.io", Kind:     "ClusterRole", Name:     "sample-apiserver-reader", }, Subjects: []rbacv1beta1.Subject{ { APIGroup:  "", Kind:      "ServiceAccount", Name:      "default", Namespace: namespace, }, }, }) framework.ExpectNoError(err, "creating cluster role binding %s", "wardler:"+namespace+":sample-apiserver-reader") </ins> // kubectl create -f authDelegator.yaml _, err = client.RbacV1beta1().ClusterRoleBindings().Create(&rbacv1beta1.ClusterRoleBinding{ ObjectMeta: metav1.ObjectMeta{

[sig-api-machinery] Aggregator Should be able to support the 1.7 Sample API Server using the current Aggregator More nuanced logs and details can be found at gave up waiting for apiservice wardle to come up successfully Expected error: <errors.errorString | 0xc420131150: { s: "timed out waiting for the condition", } timed out waiting for the condition not to have occurred /priority failing-test /priority important-soon /kind flake /sig-api-machinery /assign can you please help triage this issue?
has made changes that could have affected this recently. made one that'd be even more suspicious but I don't think it has merged yet :) :)
flakes show steady 2-3/day over the past week: which changes are you referring to? the line that's failing isn't doing any discovery, restmapping, etc, at all... just a straight get to the API:
audit log from the test run shows the requests are returning 404s, not 503s
ah, sorry, I misread the log and thought it was a discovery error. Hm.
hmm, there's a second smaller set of failures that are discovery related: looks like those started flaking after which went in on 5/4 looks like some discovery paths had internal retries that could have masked those 404 errors before. since the error that test is encountering is for an unrelated CRD API that is being created/deleted as part of another test, it makes more sense to switch that check to just ensure the extension API group under test is discovered, rather than assert there are no errors looking for other API groups that we know are being dynamically /removed opened to deflake the aggregator e2e check
/assign resolved the discovery issue. now the failure is the one noted in where calls to the extension API are responding with 404 errors
Kubelet log is reporting that the sample-apiserver exited with a return code of 2 and it also shows with a high restart count for the sample-apiserver.

When I extract the sample-apiserver log I'm seeing multiple "panic: Get https://10.0.0.1:443/api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentication: dial tcp 10.0.0.1:443: getsockopt: connection refused" Both failures today are in ipvs clusters which seems to have extensive networking problems. /assign
We are looking into finding the root cause into the problems with the ipvs clusters. I am pretty confident they were triggered by a bad PR, just need to identify which one.
Initial hypothesis is that PR caused it.
We think should fix the IPVS issues.
I see merged and this test is now passing in both pull-kubernetes-e2e-gce and ci-kubernetes-e2e-gci-gce. Hence closing the issue. Thanks all
/reopen The test is failing for the same error again in the following jobs https://k8s- It is one of the top flake from last week failing 47 jobs.
can you please take a look and triage? Thanks
this flake is now blocking this test from being promoted to Conformance suite in 1.11. Is it something we can investigate and resolve soon (today or tomorrow) to increase changes of getting this into 1.11? If not this test has to wait until 1.12 to be promoted.
all I see in the test run logs is 404 errors coming back from that check. can you see anything useful in the container logs for the aggregated server?
/cc
Update: I have been trying to reproduce this failure on my own test cluster (go run ...) and it has passed 128 times in a row so far. I think this means that the issue must be with either an interaction between e2e tests or an issue with the job running the tests.
Looking through recent results this crops up as a flake in presubmit and CI across many configurations... I don't think how we run the test jobs should be affecting this in the slightest. Not sure what's wrong here, but it looks like a pretty typical flakey test (some race / buggy setup/teardown?) nb that may not trivially map to the same way the tests are run though, and itself is a wrapper over
does the API server logs throw any light on whats happening on CI test cluster.

The teardown for the aggregator test is wrong, it tries to delete the deployment "sample-apiserver" when the deployment is actually named "sample-apiserver-deployment", but I don't think that's the issue. It's been like that since the test was . I'll look for conflicts with other tests.
We should probably go ahead and fix that anyway, it might be stomping on another test... Jenny Buckley <:
The namespace controller polls each of the api groups periodically, and it gets this internal error back from the endpoint a couple of times. Maybe related?
unclear... I opened to capture the state of the APIService, extension server pods, and those pod logs in failures cases
It flaked again since your PR merged https://k8s-
All I'm seeing is
I don't think that's related... I see 10-30 of those same rejections in apiserver logs of successful test runs (for example, search for ). I don't see anything in the test setup that grants the sample apiserver permission to run those checks, so it doesn't seem like that would be intermittent or flaky.
I am re-opening this issue since I still see intermittent failure of this test are your pending PRs ( and ) targeted towards these remaining issues? /reopen
this looks like 1.7 API Server image is causing more flakes other than just this test - The test has become lot less flaky after increasing the timeout. Since the official policy for GKE is 2 previous versions, and 1.11 is almost baked, this is the plan discussed with offline: leave the 1.7 test as it is right now create a similar test to the 1.7, but with 1.9 (implies creating a new sample-sever, uploading image, and creating the test). get that new test PR merged and running let it soak for several days, see if in 1.7 - 1.9 the flakiness (latency or whatever is causing it) went away.

if it did, evaluate removing the 1.7 and using the 1.9 let us know what you think of this approach
I think you misunderstood my comment... that only related to kubectl being used against a 1.7-level kube-apiserver serving discovery docs for the v1 API, not a generic 1.7-level extension API server
unrelatedly, moving the test to 1.9-level code once 1.11 is released seems reasonable to me
Thank you Jordan, that's the current plan! We will actually create a parallel 1.9-level test, to see if between 1.7 and 1.9 the internals of API Server fixed what's generating the flakiness and see next steps from there.
Sorry I didn't read the previous discussion close enough. The idea of creating a 1.9 test makes sense.
Pinging to see what are the next steps. Who is taking point to create a parallel test?
last I chated with someone on his team will build 1.9 API server image and push it to GCR. Once that image is available then can help writing the parallel test. I am not sure if the image step is done.
It's on the works. Buiding the sample-image showed to be more complicated than what it seemed to be. If it becomes a blocker let me know. Aish Sundar <:
I'd like to know where we are on this, because the 1.7 thing has come up in the context of multi-arch image builds
and
Hi Aaron and Aish, sorry for the delay. The process to compile, create and upload the sample API Server was not properly documented / maintained, and that plus more burning things is what made it take so long. We are trying to finally have it back (and documented by end of this week). Will keep you posted.
Image created with all the refactor happened on that area, trying upgrading the test at present from 1.7 to 1.9
Update: Upgraded the test, and now the test is not passing... working on this.

• Failure [38.067 seconds] [sig-api-machinery] Aggregator Should be able to support the 1.9 Sample API Server using the current Aggregator [It] creating a new flunders resource Expected error: <errors.StatusError | 0xc420c90cf0: { ErrStatus: { TypeMeta: {Kind: "", APIVersion: ""}, ListMeta: {SelfLink: "", ResourceVersion: "", Continue: ""}, Status: "Failure", Message: " "rest-flunder-" is forbidden: not yet ready to handle request", Reason: "Forbidden", Details: { Name: "rest-flunder-", Group: "", Kind: "flunders", UID: "", Causes: nil, RetryAfterSeconds: 0, }, Code: 403, }, } "rest-flunder-" is forbidden: not yet ready to handle request not to have occurred
not sure if it's related, but I don't think the test currently gives sufficient permissions to the aggregated server to perform delegated authn/authz checks. we might want to look at merging
tried your fix, same result. The image of the sample-apiserver 1.9 based that I've created is uploaded to If someone want's to give it a try, simply by upgrading 1.0 to 1.1 here:
/assign Can you try to figure out why the test is failing w/ the new image?
/cc
Now waiting on
Is there an ETA on ? This test is flaking quite a bit offlate - https://k8s-
upgrades to 1.10. however it requires more rbac permissions than before which is not fully understood. We will either resolve the permission issue or leave it for later investigation. The upgrade to 1.10 should be merged in a couple of days, if not today.
what issue are you seeing? is relevant?
Yes, it is relevant. It does not fix the issue by itself though.
As for should be ready to be merged.
/reopen I would like some manual confirmation this has been addressed
Reopening this issue. <detailsIn response to : /reopen I would like some manual confirmation this has been addressed Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.

</details
after some in-depth investigations, it turns out the 1.10 sample-apiserver would need the following rbac permissions (in addition to the system:auth-delegator role). I'm updating the PR to add these. apiVersion: kind: ClusterRole metadata: name: sample-apiserver rules: apiGroups: resources: '*' verbs: list apiGroups: '' resources: namespaces verbs: list
This test is failing again, FYI: https://k8s-
Hi Josh, TL;DR: we are upgrading to 1.10 and that's what people is talking about in the previous comments.
/reopen I would like to see the test pass with the fix before closing this issue out. thanks
Reopening this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hi please notice the test was renamed to: [sig-api-machinery] Aggregator Should be able to support the 1.10 Sample API Server using the current Aggregator, and has been passing since merged? https://k8s-
Ah cool thanks that explains it :) I was looking at in release-master-blocking and looks like it hasn't picked by the latest test since its 10/13 run. Closing this issue now based on the api-machinery dashboard now. Will reopen if the release job doesnt turn green the next run. Thanks much.
Is it worth removing the copy of this test that lives in ? Or is it still a release-blocking test?
I think the test will automatically pick the new version of the test in its next run (which should be sometime today, 10/20?). As to whether the test should be in the scale-correctness job , it is a ques for and


Question: import ( "fmt" <ins> "io" </ins> "io/ioutil" "os" "path/filepath"

The reboot test may cause corrupted checkpoint left on the node. https://k8s- This prevents the sandbox from being stopped properly, so kubelet cannot create a new one. One option is to log and remove the corrupted checkpoint, while cleaning up the pod sandbox resources with best effort (i.e., may leak resources). Another option is to not clean up, and force a human being to intervene and remove the corrupt checkpoint manually. The proper solution perhaps is to scan the checkpoints at startup (or periodically) and repair/remove corrupt checkpoints. /cc related:
/cc
should this block ?
If we can scan the checkpoints at startup time, then repair the corrupted checkpoint, which means, we can recover the state from the runtime, why we introduce the checkpoint at the first place? :-) I am ok with your idea upon the corrupted checkpoints, removing the bad ones as the workaround. Anyway, Kubelet will fire an event of failedSync with the proper message, some manual intervene (the remedy system) can be fired based on the event to resolve the leak resource issue.
I think this shouldn't be the blocker for v1.6, but has an acceptable workaround to address this, and I would like to include that if no one disagreed with that approach.
Adding to milestone to track
Moving this back to 1.6. The checkpoint implementation use , which does not sync at all. No wonder the checkpoints would be corrupted in the reboot tests. I reproduced this issue with a simple go program + reboots on a VM. The problem went away once I forced a I will upload a PR to fix this. /cc


Question: }, }, }, <ins> { storageClasses: []*storagev1.StorageClass{ makeDefaultStorageClass(classGold, &modeImmediate), makeStorageClass(classCopper, &modeImmediate), }, tests: []controllerTest{ { name:            "15-7 - pvc storage class is not changed if claim is not bound but already set annotation "volume.beta.kubernetes.io/storage-class"", initialVolumes:  novolumes, expectedVolumes: novolumes, initialClaims:   newClaimArray("claim15-7", "uid15-7", "1Gi", "", v1.ClaimPending, nil, v1.BetaStorageClassAnnotation), expectedClaims:  newClaimArray("claim15-7", "uid15-7", "1Gi", "", v1.ClaimPending, nil, v1.BetaStorageClassAnnotation), expectedEvents:  noevents, errors:          noerrors, test:            testSyncClaim, }, }, }, </ins> } for _, test := range tests { runSyncTests(t, test.tests, test.storageClasses, nil)

We observed that when deplying manifests to a cluster running 1.26.1-gke.200, the StorageClass and PVC get created successfully. However, the PVC is stuck in a Pending state and no PV is provisioned. The workaround is to add storageClassName to one of the pending PVCs on cluster, and it did go from Pending to properly provisioned a PV and reaching Bound state. We looked at the logs and found that according to the doc: , Is this annotation is fully deprecated in the new 1.26.1 release? If yes can we add this detail to documentation or release note? the PVC should not stuck in a Pending state and PV is provisioned in 1.26.1 a 1.26.1-gke.200 cluster in GKE via yaml The error comes form the code: and There are some hypotheses here: The annotation does not work on 1.26.1 any more, so a default storage class is being populated to the PVC. For some reason, the did not work, so the PVC was deployed with an empty StorageClass nil. Then the kube controller manager tried to update the PVC with the default StorageClass, which triggered the error: <details</details<detailsGKE </details<details</details<details</details<details</details<details</details
/sig storage
Annotation support seem to be still intact. Check the e2e test
From the point of view of the code, this annotation is still in use. I'll try to see if I can reproduce it.
It's a bug introduced in The feature is enabled in 1.26 by default.
/triage accepted


Question: return err } <del> if !reflect.DeepEqual(existing.Data, data) { </del> <ins> if !apiequality.Semantic.DeepEqual(existing.Data, data) { </ins> existing.Data = data _, err = client.ConfigMaps(metav1.NamespaceSystem).Update(existing) }

Please write a test that: Starts kube-apiserver Adds a webhook which blocks everything it is possible to block. Re-starts kube-apiserver Verifies that kube-apiserver comes back up Verifies that it can remove the broken webhook. This would have caught
cc
Hi I'm new to kubernetes but eager to contribute. I think this will be a good first issue for me but I'll need some guide on how to proceed I will need the first two points explained and also which file I'm supposed to be making my changes to.
Here is a good example of how to start a realistic kube-apiserver in integration tests:
"Verifies that it can remove the broken webhook." sounds dangerous. The apiserver should come up and it should be possible to remove a webhook via the API. But removing a potentially broken (or intentionally DoS'ed) webhook asks for security attacks.
I interpreted "Verifies that it can remove the broken webhook" as just ensuring that no webhook can be registered which could potentially deny its own (or another webhook in the same namespace's) deletion. Broken webhooks definitely shouldn't be treated differently in terms of determining who can delete them, but validation should probably prevent webhooks from applying to creates, updates and deletes of webhook objects, wildcard selectors should exclude webhook objects, etc. Also, this behavior would need to be documented. I think the safest way to ensure that a webhook does not get deleted is with RBAC rules, rather than using a validating webhook to prevent its deletion. I can't think of a legitimate use case for webhooks which affect the admission of webhook objects, it seems like any workflow which relies on that could be implemented in another, easier to reason about way. thoughts?
I have the same understanding. Excluding webhooks from webhooks seems reasonable. But it's only one piece in the (restart when webhook broke) puzzle. We reconcile RBAC rules, create namespaces (or keep them when they exist already). So I expect some more exceptions.
Is this still something you are working on?
If you can make a webhook which prevents the apiserver from starting at all, this is a problem, no matter what the webhook is verifying/doing, because you can't recover from that state.

I'm not suggesting that the system should automatically remove a broken webhook, which would be a security problem, rather that it needs to be possible for an appropriately authenticated sysadmin to do so. Especially since making a broken webhook non-broken may require executing API calls that the broken webhook is preventing.
/assign
as you assigned this to you, can you sketch what the plan is for this?
The plan was to use an integration test, (i.e. ) which I implemented in but it turns out that since the integration test apiserver does not follow exactly the same startup chain as the real one, that it is insufficient to ensure the apiserver can actually come up with a broken webhook installed. The integration test doesn't even fail against the code before merged. So currently the plan is to do it as an e2e test, probably with the [Slow] and [Destructive] tags.
it should be 99.9% the same. What are you missing? I think an integration test would be much better than an e2e test. We have much more control there.
I think the bug fixed was something a controller (bootstrap controller?) was doing, not the apiserver
those bootstrap controller are also executed by the testserver. So I would expect that the bug could be hit in testserver integration tests too. Compare with the real start code and for the test server start code. They are the same. The only difference is that we tweak the options beforehand a bit to run with a random port and random etcd prefix.
Hmm, then I think the reason that the test doesn't fail pre- is because the post start hook changed by that PR isn't called unless api-aggregation is being used. I guess the integration test for this will have to do that as well. Although the one in is still somewhat valuable to make sure new startup namespace creations in non aggregated apiservers are done correctly.
- you probably already thought about it, but one more thing to ensure here is that: apiserver can starte responding "OK" to "/healthz" even if all webhooks are broken [that means e.g. finishing all post-startup hooks] Without that, apiserver would be consistently being killed + restarted by Kubelet on which it is running (at least in some setups).

Yeah, the post-start hooks are the thing that inspired this issue :) Wojciech Tyczynski <:
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Reopened this issue. <detailsIn response to : /reopen : — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub , or mute the thread </details
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available .

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Reopened this issue. <detailsIn response to : /reopen /remove-lifecycle rotten Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: nopreserve  bool }{ "copy to pod no preserve user and permissions": { <del> expectedCmd: []string{"tar", "--no-same-permissions", "--no-same-owner", "-xf", "-", "-C", "."}, </del> <ins> expectedCmd: []string{"tar", "--no-same-permissions", "--no-same-owner", "-xmf", "-", "-C", "."}, </ins> nopreserve:  true, }, "copy to pod preserve user and permissions": { <del> expectedCmd: []string{"tar", "-xf", "-", "-C", "."}, </del> <ins> expectedCmd: []string{"tar", "-xmf", "-", "-C", "."}, </ins> nopreserve:  false, }, }

What happened: Copying a folder to a pod, e.g. postgresql-16-1-m7wjw results in the following error: time stamp 2019-02-12 12:17:06 is 0. s in the future What you expected to happen: Successful file upload to a given pod. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: perhaps this might come useful Environment: Cloud provider or hardware configuration: OS (e.g. from /etc/os-release):Kernel (e.g. ):Install tools: Others:
/sig cli
Ran into this as well. That and it's annoying that cannot be used to create a directory structure when copying files into a container.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten