Question: func HashContainer(container *api.Container) uint64 { hash := adler32.New() <del> fmt.Fprintf(hash, "%#v", *container) </del> <ins> util.DeepHashObject(hash, *container) </ins> return uint64(hash.Sum32()) }

Is this to follow pointers?
Context: The follows pointers when printing. Unlike JSON marshaling/unmarshaling it also handles private fields.

I am experiencing some weird issues when enabling HTTP liveness probes for some containers. I configure the following for all HTTP-serving containers (this is part of a template used by a replication controller with a single replica): When this is configured, the following issues start to occur: One pod is already running without problems. When I add another replication controller which creates a new pod (also with the liveness probe set), the first pod (the one that was already running) is restarted. This happens even when the two pods are in different namespaces. But when a pod's HTTP server fails completely (connection to the port is refused), the liveness probe will not trigger and the pod will not be restarted. This behavior occurs consistently when I start with no replication controllers and then add them one by one. When I disable the liveness probes on the affected pods/templates, they are no longer restarted in this manner.
You say you're starting two replication controllers-- can you verify that their label selectors do not intersect?
Yes, each replication controller has a selector over a single label in the form of so the labels do not overlap. Also, this problem occurs even when the apps are in different namespaces and goes away when the liveness probe is removed.
Thanks-- last question-- Do the pods have to be scheduled on the same node for this to occur? Your second point I believe is a problem here: we treat an error during HTTP GET as healthy for some inexplicable reason. I can't really explain your first observation, unless perhaps the pods and containers have the same names, but in different namespaces AND kubelet has a namespace handling bug?
cc
"Unknown" != "Healthy" I think it's an open question about whether we want to treat a GET failure as "Unhealthy" we certainly could... --brendan , Daniel Smith wrote:
Giving this to Saad. I think it's pretty clear that if kubelet can't do an HTTP GET, then the pod isn't healthy-- they're on the same host, HTTP get can't fail for network reasons. Our internal system makes this assumption. (I broke it once!)
tl;dr... When a second replication controller is the hash of the container of the first replication controller changes causing it to be killed and recreated. The hash changes because the LivenessProbe pointer changes. Need to figure out why that happens.

Spent time trying to understand why creating two replication controllers, both with http liveness probes, causes the container from the first replication controller to be killed and then recreated when the second replication controller is created. It turns out, as soon as the second replication controller is created, the hash of the container from the first replication controller changes and thus the container is killed: Let's see how the expected and actual hashes are computed and how they end up differing. The actual hash is pulled from the docker container name (dockerContainers.FindPodContainer): The expected hash is created from the object. Specifically, this code (): Expected hash before second replication controller is : Expected hash after second replication controller is : The difference between the container before and after is: So the hash changes because the LivenessProbe pointer in the object changes. Need to figure out why this happens.
Great detective work-- I think it's clear that we shouldn't be including pointers in our hash, so we need to revisit that and explicitly hash only the things that we care about.
Adding second replication controller changes the pointer of LivenessProbe in container config is another mystery. We need to make sure not randomly update other fields too.
Dug in to this some more: confirmed that the value of the original container's LivenessProbe doesn't actually change when the second replication controller is created--only the pointer to the LivenessProbe object changes. It turns out that the instance of the  object being used to compute the expected hash is not guaranteed to be the same instance for the lifespan of the actual docker container. In fact, every time there is an etcd update (e.g. a new pod is created) a new boundpods object is created and although the values of the existing pod/container container objects don't change, because new instances are created the pointers do change (by design). So the fix for this is to ensure that the expected hash is computed only on container values not pointers. One way to ensure this would be to make sure the container object specifies no pointer fields. However, LivenessProbe and Lifecycle, since they are optional, must be able to be set to . Thus, pointer fields can't simply be converted to value fields. Another approach would be to only hash some subset of the values (maybe the name and image) of the container.

However, containers should be destroyed/recreated (meaning the hash must change) if the actual values of any of the container fields changes (e.g. the cpu, memory, volume mounts, liveness probe path, etc.), not just the container name/image. Thus, pointer fields can't simply be disregarded for the hash computation. Instead the fix must modify the expected hash to be computed on the complete deep values of the entire container object (including the actual LivenessProbe values, instead of the pointer). The easiest way to do this is to marshal the container object to json and compute the hash from the resulting byte array.


Question: import ( "fmt" <ins> "time" </ins> . "github.com/onsi/ginkgo" . "github.com/onsi/gomega"

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: https://k8s- https://k8s-
/assign /priority critical-urgent /kind failing-test cc
/cc this is the symptom you said had an internal fix coming by end of week (ie: today)?
No this is purely oss
cc


Question: var map_RollingUpdateDeployment = map[string]string{ "":               "Spec to control the desired behavior of rolling update.", <del> "maxUnavailable": "The maximum number of pods that can be unavailable during the update. Value can be an absolute number (ex: 5) or a percentage of desired pods (ex: 10%). Absolute number is calculated from percentage by rounding up. This can not be 0 if MaxSurge is 0. By default, a fixed value of 1 is used. Example: when this is set to 30%, the old RC can be scaled down to 70% of desired pods immediately when the rolling update starts. Once new pods are ready, old RC can be scaled down further, followed by scaling up the new RC, ensuring that the total number of pods available at all times during the update is at least 70% of desired pods.", </del> <ins> "maxUnavailable": "The maximum number of pods that can be unavailable during the update. Value can be an absolute number (ex: 5) or a percentage of desired pods (ex: 10%). Absolute number is calculated from percentage by rounding down. This can not be 0 if MaxSurge is 0. By default, a fixed value of 1 is used. Example: when this is set to 30%, the old RC can be scaled down to 70% of desired pods immediately when the rolling update starts. Once new pods are ready, old RC can be scaled down further, followed by scaling up the new RC, ensuring that the total number of pods available at all times during the update is at least 70% of desired pods.", </ins> "maxSurge":       "The maximum number of pods that can be scheduled above the desired number of pods. Value can be an absolute number (ex: 5) or a percentage of desired pods (ex: 10%). This can not be 0 if MaxUnavailable is 0. Absolute number is calculated from percentage by rounding up. By default, a value of 1 is used. Example: when this is set to 30%, the new RC can be scaled up immediately when the rolling update starts, such that the total number of old and new pods do not exceed 130% of desired pods. Once old pods have been killed, new RC can be scaled up further, ensuring that total number of pods running at any time during the update is atmost 130% of desired pods.", }

API documentation says is rounded up, but it is actually rounded down: The code shows it is rounded down, not up: and
Yes, the documentation is obsolete. We changed maxUnavailable to round down so that we wouldn't accidentally infer 100% unavailability for non-100 percentages. For example, maxSurge=25% and maxUanavailable=25% with a Deployment of 1 replica meant that we could scale down 1 before thus leading in 100% maxUnavailable (implicit).


Question: } var pod *corev1.Pod <del> leaveStdinOpen := o.LeaveStdinOpen waitForExitCode := !leaveStdinOpen && restartPolicy == corev1.RestartPolicyNever </del> <ins> waitForExitCode := !o.LeaveStdinOpen && (restartPolicy == corev1.RestartPolicyNever || restartPolicy == corev1.RestartPolicyOnFailure) </ins> if waitForExitCode { <del> pod, err = waitForPod(clientset.CoreV1(), attachablePod.Namespace, attachablePod.Name, opts.GetPodTimeout, podCompleted) </del> <ins> // we need different exit condition depending on restart policy // for Never it can either fail or succeed, for OnFailure only // success matters exitCondition := podCompleted if restartPolicy == corev1.RestartPolicyOnFailure { exitCondition = podSucceeded } pod, err = waitForPod(clientset.CoreV1(), attachablePod.Namespace, attachablePod.Name, opts.GetPodTimeout, exitCondition) </ins> if err != nil { return err } <del> } // after removal is done, return successfully if we are not interested in the exit code if !waitForExitCode { </del> <ins> } else { // after removal is done, return successfully if we are not interested in the exit code </ins> return nil }

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: gce-master-scale-correctness Which test(s) are failing: [sig-cli] Kubectl client Simple pod should return command exit codes Since when has it been failing: 02/05 Testgrid link: https://k8s- Reason for failure: Anything else we need to know: /sig cli
/priority critical-urgent /assign
The test seems to be flaking quite a lot recently ! Friendly ping, sig-cli, could you take a look?
FTR, these are really blocking tests and it's getting more urgent given the forthcoming 1.18 release
Full log of the failure: It looks like the finishes with error. Strange, I thought is supposed to create a pod. Would really appreciate if someone from sig-cli could take a look
4 out of last 8 runs failed: https://k8s- /sig scalability
Another flaky, related test:
SIG CLI Leads -- any update on this? This is affecting scalability signal. /assign
-- Please keep tabs on this one.
yep, monitoring
It looks like there's a race between when the pod gets created and when we try to connect to it, thus we're seeing but further down in the logs the pod is , I'll be looking more into ti.
Thanks, Any update on this?
/cc
Sorry I got sidetracked yesterday and I'm back investigating this just now...
Probable fix in
Will continue to monitor here and reopen if needed :+1:
Unfortunately - it seems that the problem still exists - the test failed today (an hour ago): :( ^^
I'll check it, thx Wojtek for pointing that out.
- I'm not familiar with kubectl code, but I took a quick look and what I found is that this waitForPod() thing is happening only for pods with RestartPolicyNever: I must be missing something here, but for the case that failed it's not really the case, which means we don't wait for it?
This particular one is using , see the actual invocation that failed: That's why I started with this approach.
Sorry - the one for which the bug was originally opened.

But the today`s failure is actually slightly different (the same test, different case - sorry): The restart policy for this one is
Gotcha, so it looks like we progressed, I'll check the possible paths for this one, something tells me we're missing another timeout somewhere :wink:
OK - I think I know what is going on here. Here is what is happening (for the test case from two comments above): the test is setting "-i", so interactive mode is switched on that option triggers that Attach option defaults to true: over the Run() command, we get to the handleAttachPod: handleAttachPod is calling waitForPod And waitForPod is broken. The problem is the preconditionFunc: which is below passed to the UntilWithSync function UntilWithSync underneath is creating an informer and calling that function against its state: But the informer is initialized from apiserver watchcache, so it may be lagging. In large clusters we are observing lags in the order of small hundreds of milliseconds and that seem to be enough. What is happening is that: informer is initialized (but the pod creation is not yet reflected in it) the preconditionFunc is called the state doesn't contain that pod (yet) the whole thing fails with that error I actually believe, that just removing the preconditionFunc should actually solve the problem. I opened for that.
^^ - please take a look at the PR that I assigned to you (it's literally one-liner as of now - if you want to add tests or anything like that, feel free to pick it up).
Actually I went ahead, and fixed couple other commands too - so it's no longer one-liner. But still short one :)


Question: if config.Timeout == 0 { config.Timeout = defaultTimeout } <ins> if config.Burst == 0 && config.QPS < 100 { // discovery is expected to be bursty, increase the default burst // to accommodate looking up resource info for many API groups. // matches burst set by ConfigFlags#ToDiscoveryClient(). // see https://issue.k8s.io/86149 config.Burst = 100 } </ins> codec := runtime.NoopEncoder{Decoder: scheme.Codecs.UniversalDecoder()} config.NegotiatedSerializer = serializer.NegotiatedSerializerWrapper(runtime.SerializerInfo{Serializer: codec}) if len(config.UserAgent) == 0 {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: With client-go v0.17.0, I observed many "Throttling request" when DiscoveryClient is used to get API group resources from API server. This didn't happen with client-go kubernetes-1.16.0. It looks like originally this kind of discovery requests are not throttled because no rate limiter provided and QPS == 0. But with rate limiter is always set if nil rate limiter is provided. is particularly suspicious because it is always true because DefaultQPS is nonzero. What you expected to happen: Discovery requests should not be throttled as before. How to reproduce it (as minimally and precisely as possible): Create DiscoveryClient and then call with it. Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig api-machinery
The NewTokenBucketRateLimiter was via Always negotiate a decoder using ClientNegotiator /cc
To be clear, that was fixing a bug where the documented defaults for rate limiting were not taking effect. I don't think we should completely disable throttling until more of the server-side priority work is completed, but discovery is expected to be bursty, so I could see defaulting to a different amount in the discovery client constructor, similar to
Yeah, everyone must be rate limited. I think we could give discovery a big burst value.
opened


Question: Long:    create_long, Example: create_example, Run: func(cmd *cobra.Command, args []string) { <del> err := RunCreate(f, out, cmd, filenames) cmdutil.CheckErr(err) </del> <ins> cmdutil.CheckErr(ValidateArgs(cmd, args)) cmdutil.CheckErr(RunCreate(f, out, cmd, filenames)) </ins> }, } cmd.Flags().VarP(&filenames, "filename", "f", "Filename, directory, or URL to file to use to create the resource") return cmd } <ins> func ValidateArgs(cmd *cobra.Command, args []string) error { if len(args) != 0 { return cmdutil.UsageError(cmd, "Unexpected args: %v", args) } return nil } </ins> func RunCreate(f *Factory, out io.Writer, cmd *cobra.Command, filenames util.StringList) error { schema, err := f.Validator() if err != nil {

cc
When deleting/stopping using file, you don't specify resource (in this case 'pod'), as it's inferred from the file.
Thanks and sorry for confusion
Ok, but should not work as well.


Question: Expect(err).NotTo(HaveOccurred()) pst := statefulSetTester{c: c} <del> pst.saturate(ps) </del> nn := framework.TestContext.CloudConfig.NumNodes nodeNames, err := framework.CheckNodesReady(f.ClientSet, framework.NodeReadyInitialTimeout, nn)

What happened: What you expected to happen: The statefulset should have scaled down to 0. Instead, even after 10 minutes: How to reproduce it (as minimally and precisely as possible): Run the e2e test on a gce cluster running the latest and greatest kubernetes. cc
if the petset has replicas=0 it's not going to do anything
When I a running petset, and set its scale to 0, that seems to work and delete the pets, and we should have a functioning scaler. Am I misunderstanding something? I'm using the same teardown process for the e2e test I linked in the issue as the other tests (), but it times out sometimes after network partition.
yeah is scale in the petset controller noticing deletion timestamp? because the test code you showed doesn't care
As I mentioned yesterday, it isn't even issuing a delete request even after 10 minutes, because the pods continue to be "running". I'll check the controller manager logs today. I suspect it might be the backoff in the rate limited work-queue.
Hmm, the AGE fields on your pets don't make sense if it isn't even issuing a delete?
In the output I pasted above, at the 7 minute mark, it created the new replacement pet-0, after the network partition was healed. Then tried to scale down the StatefulSet and waited 10 minutes for the pods to disappear.
I see a bunch of different failures modes on the bug: 3 08:58:44.475: Failed waiting for pods to enter running: timed out waiting for the condition Failed to scale statefulset to 0 in 10m0s. Remaining pods: [] 3 08:11:30.728: Unexpected error: <nilBasically 3 translates to We should clarify. 1 and 2 both have events like guessing the node controlelr shouldn't be setting timestamp?
Looking into the case 2: The controller manager log goes: So I guess even deletions are being blocked by the health/readiness.
Both issues appear to be due to the testsuite and the way we use the "initialized" annotation in tests. I'll send a PR to fix these shortly. Thanks


Question: } return false, err } <del> job := *sharedJob </del> <ins> // make a copy so we don't mutate the shared cache job := *sharedJob.DeepCopy() </ins> // if job was finished previously, we don't want to redo the termination if IsJobFinished(&job) {

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --ci-kubernetes-integration-master I can't seem to find a match for beyond that? Most likely related to as it was merged during that set of commits Example Spyglass link: /cc /sig apps /priority critical-urgent /milestone v1.21
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
FYI: the error message is a red herring and isn't the real reason why this test is failing. When the integration tests are run with instead of , the mutation detector is also started, which catches a bug in the job controller where we modify the shared cache. This root cause isn't present in the logs, but you can reproduce this bug with the following command: Then apply the job controller patch in and re-run the test, you'll see that the test passes. You'll also notice that passes with/without the bug fix patch when run with . cc
Test is back on green. Thanks !


Question: cleanup() { echo "Cleaning up..." <del> kill "${APISERVER_PID}" kill "${CTLRMGR_PID}" kill "${KUBELET_PID}" kill "${PROXY_PID}" kill "${SCHEDULER_PID}" </del> <ins> sudo kill "${APISERVER_PID}" sudo kill "${CTLRMGR_PID}" sudo kill "${KUBELET_PID}" sudo kill "${PROXY_PID}" sudo kill "${SCHEDULER_PID}" </ins> kill "${ETCD_PID}" rm -rf "${ETCD_DIR}"

/kind failing-test <!-- If this is a flake, please add: /kind flake --<!-- Please assign a SIG using: /sig SIG-NAME --
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
looks like started a few days ago ! /sig cluster-lifecycle
cc
fix was sent earlier
/sig cluster-lifecycle
thanks :) i saw it yesterday and forgot to raise the issue. i should have checked the PR queue :)


Question: // validation with resource counting, but we did this before QoS was even defined. // let's not make that mistake again with other resources now that QoS is defined. requiredSet := quota.ToSet(required).Intersection(validationSet) <del> missingSet := sets.NewString() </del> <ins> missingSetResourceToContainerNames := make(map[string]sets.String) </ins> for i := range pod.Spec.Containers { <del> enforcePodContainerConstraints(&pod.Spec.Containers[i], requiredSet, missingSet) </del> <ins> enforcePodContainerConstraints(&pod.Spec.Containers[i], requiredSet, missingSetResourceToContainerNames) </ins> } for i := range pod.Spec.InitContainers { <del> enforcePodContainerConstraints(&pod.Spec.InitContainers[i], requiredSet, missingSet) </del> <ins> enforcePodContainerConstraints(&pod.Spec.InitContainers[i], requiredSet, missingSetResourceToContainerNames) </ins> } <del> if len(missingSet) == 0 { </del> <ins> if len(missingSetResourceToContainerNames) == 0 { </ins> return nil } <del> return fmt.Errorf("must specify %s", strings.Join(missingSet.List(), ",")) </del> <ins> var resources = sets.NewString() for resource := range missingSetResourceToContainerNames { resources.Insert(resource) } var errorMessages = make([]string, 0, len(missingSetResourceToContainerNames)) for _, resource := range resources.List() { errorMessages = append(errorMessages, fmt.Sprintf("%s for: %s", resource, strings.Join(missingSetResourceToContainerNames[resource].List(), ","))) } return fmt.Errorf("must specify %s", strings.Join(errorMessages, "; ")) </ins> } // GroupResource that this evaluator tracks

I deployed a pod with and on a namesapace with a set with and elements and I get the error: The pod to be deployed Create a new namespace Create (using a manifest file named resource-quota-) and verify a named compute-resources Content of the resource-quota- file Create (using a manifest file named ) a pod named high-priority with the and for both and . Content of the file The pod is deployed correctly if I delete the <details</details<detailsaws </details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig Scheduling
/wg API Expression
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /wg API Expression Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/wg <API Expression
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /wg <API ExpressionInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/wg "API Expression"
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /wg "API Expression" Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/wg APIExpression
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /wg APIExpression Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.

</details
/wg wg-api-expression
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /wg wg-api-expression Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
hey, v1.18.20 is out of support now, you should try with a newer version. Besides, I tried with an old version v1.18.19, but worked as well.
Are u using some mutating webhook which inject some sidecar into the pod without setting resources fields?
It's probably that, but IDK which WebHook is doing that. The feature you suggested would really help.
hi the fix pr has been merged, maybe u could use it to locate which container is causing the problem.
Thanks


Question: CommandGroups: groups, Filtered:      filters, } <ins> cmd.SetFlagErrorFunc(templater.FlagErrorFunc()) </ins> cmd.SetUsageFunc(templater.UsageFunc()) cmd.SetHelpFunc(templater.HelpFunc()) return templater

The majority of kube binaries print usage when certain invalid arguments are presented. Unfortunately, most of our binaries have 2 pages of usage output, and the result of printing that is a very confusing and context disrupting action. In general, CLI commands try to minimize their output - for example: both print their minimal sets. Obviously, as commands grow the number of flags some of the short options become a bit silly (yes, congratulations, you have the entire alphabet of flags) and convey less useful information to a user. When errors are printed we also end up printing the error both above and below the command (in at least kubectl, kubelet, kube-apiserver, and controller-manager). I'm not even sure this was intentional, but coincidentally we print the errors twice and so it's both before and after the usage. Error feedback needs to minimize the amount of extraneous information presented to users for maximum clarity. Three are broadly four classes of help and error output: user explicitly requests help via or user provides an input that is not recognized (flag, command, or argument) and we can unambiguously know provided invalid input. User needs to change the input. user provides inputs that don't work well together or have conflicting meaning. User needs to change the input. user provided inputs that result in a runtime error, but that error can't be resolved by changing the input of the command. Recommended changes: our command and flag sets are so long, I don't think there is value in printing the full usage info in 2 and 3. Instead, I think we should print a minimal "usage: Run "command foo --help" to see usage info" message after printing the error. should stop printing error output twice (once as and once as ) in 2-4 and print as or (the former is more consistent with Posix guidelines). still need to clean up output.
this is really painful as an end user (and I've been tracking people who have these issues). I hit this while looking at and trying to clean it up, and realized we need to fix the broader problem.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with .

Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen /assign This is still a thing.
Reopened this issue. <detailsIn response to : /reopen /assign This is still a thing. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove lifecycle/rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This is still a thing. /assign /reopen
Reopened this issue. <detailsIn response to : This is still a thing. /assign /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
GitHub didn't allow me to assign the following users: sallyom.

Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : This is still a thing. /assign /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This is still very much a problem. Having to scroll through pages of options just to find the error output is terrible. I'm running kube-apiserver with systemd so it automatically restarts. This means the error itself ends up in the middle of hundreds of lines of unhelpful information. It's like finding a needle in a haystack to figure out what actually went wrong. If nothing else, a flag to turn it off would make it more manageable. /reopen /remove lifecycle/rotten
You can't reopen an issue/PR unless you authored it or you are a collaborator. <detailsIn response to : This is still very much a problem. Having to scroll through pages of options just to find the error output is terrible. I'm running kube-apiserver with systemd so it automatically restarts. This means the error itself ends up in the middle of hundreds of lines of unhelpful information. It's like finding a needle in a haystack to figure out what actually went wrong. If nothing else, a flag to turn it off would make it more manageable. /remove lifecycle/rotten Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen /remove-lifecycle rotten
Reopened this issue.

<detailsIn response to : /reopen /remove lifecycle/rotten Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: v1 "k8s.io/api/core/v1" apierrors "k8s.io/apimachinery/pkg/api/errors" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" <ins> "k8s.io/apimachinery/pkg/runtime" </ins> "k8s.io/apimachinery/pkg/util/sets" "k8s.io/client-go/informers" "k8s.io/client-go/kubernetes"

I'm writing a scheduler plugin using the scheduler framework, and in my plugin I'd like to add a using the provided by the . Adding such a Indexer lead to a runtime crash like The reason for this is that I'm not sure the scheduler framework wants to explicitely disallow adding Indexers using the facilities. If this is the case, it should be more explicit somehow because the current behaviour is surprising. If the scheduler framework doesn't want to add this limitation, then adding indexers should not lead to runtime crashes add a new scheduler plugin trying to add indexers. Sample code (relevant minimal portions only) I'm not sure this is a real bug and not a deliberate decision; a research was inconcusive, and a (admittedly quick) search in the past issues was not fruitful either. The fix could be as trivial as <detailsconsole $ kubectl version Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2020-01-18T23:30:10Z", GoVersion:"go1.13.5", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.5+", GitCommit:"", GitTreeState:"clean", BuildDate:"2022-05-10T16:30:48Z", GoVersion:"go1.17.5", Compiler:"gc", Platform:"linux/amd64"} </detailsbut applies also on latest master (see above) <detailsany </details<detailsany </details<detailsnot relevant </details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scheduling


Question: isReady, err := podRunningReady(p) expectNoError(err) <del> Expect(isReady).To(BeTrue()) </del> <ins> Expect(isReady).To(BeTrue(), "pod should be ready") </ins> <del> Expect(getRestartCount(p) == 0).To(BeTrue()) </del> <ins> restartCount := getRestartCount(p) Expect(restartCount == 0).To(BeTrue(), "pod should have a restart count of 0 but got %v", restartCount) </ins> }) It("with readiness probe that fails should never be ready and never restart", func() {

If the container_probe test runs on a node without the nginx image, it sometimes takes 90s to get the pod in running. Most of this time goes in pulling the image, so I'm assuming it's registry related hiccup. Using might make it more reliable. Example failure: Note the following event in the logs:
a) we should not be pulling from docker in our tests - too hit-or-miss b) we should replace nginx (132 MB) with (4.5 MB) whenever possible. , Prashanth B wrote:


Question: config.DialFromEndpointContainer("udp", config.NodeIP, config.NodeUDPPort, config.MaxTries, 0, config.EndpointHostnames()) }) <ins> // This test ensures that in a situation where multiple services exist with the same selector, // deleting one of the services does not affect the connectivity of the remaining service ginkgo.It("should function for multiple endpoint-Services with same selector", func() { config := e2enetwork.NewNetworkingTestConfig(f, false, false) ginkgo.By("creating a second service with same selector") svc2, httpPort := createSecondNodePortService(f, config) // original service should work ginkgo.By(fmt.Sprintf("dialing(http) %v (endpoint) --> %v:%v (config.clusterIP)", config.EndpointPods[0].Name, config.ClusterIP, e2enetwork.ClusterHTTPPort)) config.DialFromEndpointContainer("http", config.ClusterIP, e2enetwork.ClusterHTTPPort, config.MaxTries, 0, config.EndpointHostnames()) ginkgo.By(fmt.Sprintf("dialing(http) %v (endpoint) --> %v:%v (nodeIP)", config.EndpointPods[0].Name, config.NodeIP, config.NodeHTTPPort)) config.DialFromEndpointContainer("http", config.NodeIP, config.NodeHTTPPort, config.MaxTries, 0, config.EndpointHostnames()) // Dial second service ginkgo.By(fmt.Sprintf("dialing(http) %v (endpoint) --> %v:%v (svc2.clusterIP)", config.EndpointPods[0].Name, svc2.Spec.ClusterIP, e2enetwork.ClusterHTTPPort)) config.DialFromEndpointContainer("http", svc2.Spec.ClusterIP, e2enetwork.ClusterHTTPPort, config.MaxTries, 0, config.EndpointHostnames()) ginkgo.By(fmt.Sprintf("dialing(http) %v (endpoint) --> %v:%v (nodeIP)", config.EndpointPods[0].Name, config.NodeIP, httpPort)) config.DialFromEndpointContainer("http", config.NodeIP, httpPort, config.MaxTries, 0, config.EndpointHostnames()) ginkgo.By("deleting the original node port service") config.DeleteNodePortService() // Second service should continue to function unaffected ginkgo.By(fmt.Sprintf("dialing(http) %v (endpoint) --> %v:%v (svc2.clusterIP)", config.EndpointPods[0].Name, svc2.Spec.ClusterIP, e2enetwork.ClusterHTTPPort)) config.DialFromEndpointContainer("http", svc2.Spec.ClusterIP, e2enetwork.ClusterHTTPPort, config.MaxTries, 0, config.EndpointHostnames()) ginkgo.By(fmt.Sprintf("dialing(http) %v (endpoint) --> %v:%v (nodeIP)", config.EndpointPods[0].Name, config.NodeIP, httpPort)) config.DialFromEndpointContainer("http", config.NodeIP, httpPort, config.MaxTries, 0, config.EndpointHostnames()) }) </ins> ginkgo.It("should update endpoints: http", func() { config := e2enetwork.NewNetworkingTestConfig(f, false, false) ginkgo.By(fmt.Sprintf("dialing(http) %v --> %v:%v (config.clusterIP)", config.TestContainerPod.Name, config.ClusterIP, e2enetwork.ClusterHTTPPort))

What happened: indicates that we're missing an e2e test for. This specific issue is just a tracker to create a new E2E test that demonstrated .
/sig network
/assign
btw, should this be a part of conformance?
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
/remove-triage unresolved
I can start working on this.
Sure/ are you working full time on it ? If so I'll drop the issue from my work ... Otherwise we can collaborate as I already have some code for this. Let me know. Was hoping to have this done in the next week or so.
I would love to collaborate, especially if you already have some code. But if you want to drop this from your work, I don't mind taking it on full time either. I can pick up from where ever you left off.
ok, if youve got full bandwidth go for it. if make progress on it this wk ill just drop it . I actually dont know where my old branch is for this :), so i guess youll have to start from scratch. One thing to note ~ Might be seeing if any tests can be reused for this or extended in a simple way by just adding a secondary service, rather then writing a full blown test from scratch. Let me know how it goes thanks !
I will let you know how it progresses. Thanks !
thanks for getting this taken care of :) looks like the PR is LGTM.
Thanks for letting me take a stab at it!


Question: // IDMappingRange specifies a mapping between container IDs and node IDs type IDMappingRange struct { <del> // ContainerID is the starting container ID </del> <ins> // ContainerID is the starting container UID or GID </ins> ContainerID int <del> // HostID is the starting host ID </del> <ins> // HostID is the starting host UID or GID </ins> HostID int <del> // Length is the length of the ID range </del> <ins> // Length is the length of the UID/GID range </ins> Length int }

Which jobs are failing: Which test(s) are failing: Since when has it been failing: Testgrid link: Reason for failure: Anything else we need to know: /cc /milestone v1.19 /priority critical-urgent /sig testing
/assign /area kubeadm
/sig cluster-lifecycle
i went trough all the flakes we have at the moment and adjusted a number timeouts. here is the PR:
was green for a bit since merge, but noticed a timeout on this morning. Do you think it is worth bumping the timeout there as well? It looks like the timeout was quite a large outlier compared to the successful runs (more than double the duration), so maybe we should address underlying problem.
lets wait for some more singlal before bumping the timeouts on init. locally for me takes a couple of minutes, so i would like to know if this is a temporary GCE/networking issue.
actually something is not right, because in this job: also this job showed that even 10m is not enough for it errored with: but looking at the log timestamps only ~2 minutes have passed. i will have another look later today or tomorrow.
/retitle [Flaky Test] kubeadm-kinder-upgrade-1-18-master (ci-kubernetes-e2e-kubeadm-kinder-upgrade-1-18-master) /priority important-soon /remove-kind failing-test /kind flake the test seems to be timeouting occasionally now. this appeared to be a temporary network speed issue. in the meantime i have created this PR which attempts to catch network flakes early and reduce overall network traffic:
the kubeadm e2e jobs started requiring more than 40 minutes: https://k8s- so a lot of test jobs are now failing. this could be due to: resource limits in Prow more conformance tests were during the weekend or next week i will try bumping all jobs to ~60 minutes.
/remove-priority important-soon (prio is already critical)
xref
hi, the jobs seem to be now at the state where they were before the ticket was logged - i.e. mostly green. the Prow issue that was discussed in is definitely a factor, so i'm not going to bump the jobs to 60minutes for now.

we do still see 2 occasional types of flakes: one of them is known and we don't have a fix for it as it's complicated. for the other one i'm going to send a PR and ping you for a quick lgtm. thanks.
thanks for following up! I will make sure to take a look at your incoming PR :+1:
i haven't seen the same failure in the last 10 runs: https://k8s- please re-open this ticket if needed. thanks.
/close
Closing this issue. <detailsIn response to : /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I am seeing some pretty regular failures here again. Could you take a look? /reopen
Reopened this issue. <detailsIn response to : I am seeing some pretty regular failures here again. Could you take a look? Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
the job is taking more than 40 minutes. so this is related to:
The runs in last few days have been well under 60m, except for
i'm suspecting slow CPU/network for some of the recent failures.
/remove-sig testing SIG testing does not own kinder or upgrades
looks like the most prominent recent flakes are caused by the inability to prepare the node image or upgrade all nodes in less than 10 minutes. i will send a PR to increase these timeouts.
Is that test now named ?
yes. i see some occasional flakes of not being able to complete the conformance suite in 25 minutes (parallel).
/retitle [Flaky Test] kubeadm-kinder-upgrade-1-18-1-19 (ci-kubernetes-e2e-kubeadm-kinder-upgrade-1-18-1-19)
i can conform that the suite requires mostly more than 25 minutes in later releases (either due to more tests or slower infra). this PR increases the e2e test suite in kubeadm to 35 minutes and will close this ticket: we should create new / separate tickets with flaky tasks in the job.


Question: return metav1.StatusReasonUnknown } <ins> func reasonAndCodeForError(err error) (metav1.StatusReason, int32) { if status := APIStatus(nil); errors.As(err, &status) { return status.Status().Reason, status.Status().Code } return metav1.StatusReasonUnknown, 0 } </ins> // ErrorReporter converts generic errors into runtime.Object errors without // requiring the caller to take a dependency on meta/v1 (where Status lives). // This prevents circular dependencies in core watch code.

Checked a method error using when the error was caused by a validating webhook that returns a 403 response. In this particular case, the error was generated from a validating webhook implemented with controller-runtime that returns controller-runtime's response. to return true. returned false Many (but not all) of the other family of functions in suffer the same kind of problem. I have changes ready to submit in a PR if we can get agreement that this is a problem and that it should be fixed in the functions that ought to check the of an erorr value that is or wraps an Kubernetes version (use ): 19.3 for the server
/sig api-machinery
are you going to send a PR for this? The issue description looks reasonable. /triage accepted
Thanks, PR is submitted.
/assign


Question: return "", "", fmt.Errorf("invalid ip config ID %s", ipConfigurationID) } <del> prefix := matches[1] suffix := matches[2] nodeName := fmt.Sprintf("%s-%s", prefix, suffix) </del> <ins> nicResourceGroup, nicName := matches[1], matches[2] if nicResourceGroup == "" || nicName == "" { return "", "", fmt.Errorf("invalid ip config ID %s", ipConfigurationID) } nic, rerr := as.InterfacesClient.Get(context.Background(), nicResourceGroup, nicName, "") if rerr != nil { return "", "", fmt.Errorf("GetNodeNameByIPConfigurationID(%s): failed to get interface of name %s: %s", ipConfigurationID, nicName, rerr.Error().Error()) } vmID := "" if nic.InterfacePropertiesFormat != nil && nic.VirtualMachine != nil { vmID = to.String(nic.VirtualMachine.ID) } if vmID == "" { klog.V(2).Infof("GetNodeNameByIPConfigurationID(%s): empty vmID", ipConfigurationID) return "", "", nil } </ins> <del> vm, err := as.getVirtualMachine(types.NodeName(nodeName), azcache.CacheReadTypeDefault) </del> <ins> matches = vmIDRE.FindStringSubmatch(vmID) if len(matches) != 2 { return "", "", fmt.Errorf("invalid virtual machine ID %s", vmID) } vmName := matches[1] vm, err := as.getVirtualMachine(types.NodeName(vmName), azcache.CacheReadTypeDefault) </ins> if err != nil { <del> return "", "", fmt.Errorf("cannot get the virtual machine by node name %s", nodeName) </del> <ins> return "", "", fmt.Errorf("cannot get the virtual machine by node name %s", vmName) </ins> } asID := "" if vm.VirtualMachineProperties != nil && vm.AvailabilitySet != nil { asID = to.String(vm.AvailabilitySet.ID) } if asID == "" { <del> return "", "", fmt.Errorf("cannot get the availability set ID from the virtual machine with node name %s", nodeName) </del> <ins> return vmName, "", nil </ins> } asName, err := getAvailabilitySetNameByID(asID) if err != nil { return "", "", fmt.Errorf("cannot get the availability set name by the availability set ID %s", asID) } <del> return nodeName, strings.ToLower(asName), nil </del> <ins> return vmName, strings.ToLower(asName), nil </ins> }

What happened: introduced new dependency on Availability Sets which is too restrictive for some existing installations: What you expected to happen: Azure cloud provider to successfully manage LBs, like it had been doing before. Instead, it died [with][1]: How to reproduce it (as minimally and precisely as possible): Fails every time in OpenShift CI. Can you point us to documentation around the infrastructure options that the Azure cloud provider expects for LB management? https://kubernetes- doesn't say anything about NIC names, and it doesn't have specifics around what things should look like without Availability Sets, or whether "without Availability Sets" is not supported (when previous iterations of the cloud provider had no problem with our lack of Availability Sets). Documenting cloud-provider assumptions about host hardware would have also helped avoid ; CC /sig cloud-provider /area provider/azure [1]:
/triage accepted
/assign
Notes on OpenShift's Azure infrastructure in ARM templates [here][1], in case that helps clarify the infra we use today. Those are OpenShift 4.6 docs, and 4.6 is based on Kubernetes 1.19, and things worked fine. The trouble we've been having when we took the same Azure-infra approach and tried to use it with Kubernetes 1.20, after openshift/kubernetes. [1]:
could you share an example of resourceID (please replace your subscription/resourceName with fake ones)?
Example CI run [here][1] from the NIC-renaming openshift/installer. Installer logs [here][2] include: I'm hoping that the subscription name is not sensitive, because we've been dumping them in public logs for a long time now . Should be IDs for any other resources you're interested in in those logs as well. [1]: [2]: https://gcsweb-
Looks like this is about the master node. Are you using standard LB or a basic one? For standard LB, the master node would be excluded from the LB by default.
[Standard][1] (and [here][2]). Is this documented somewhere? We've been using standard LBs [for years][3], with no problems until the bump to the Kube 1.20 Azure cloud-provider. [1]: [2]: [3]:
Are you referring to the built-in logic of Kube to never target master nodes with a service type LoadBalancer?

That used to be a problem (), but my understanding is that it has since been fixed.
Can we re-open this? kubernetes-sigs/cloud-provider-azure was [not sufficient][1] to get the cloud-provider happy on our infra, which again, contains no Availability Sets. There are some more notes about our infrastructure [here][2] (with discussion in [rhbz][3]), in case that helps. [1]: [2]: [3]:
who has a better grasp on this than me, also just opened . Sounds like fixing that broader issue might obsolete this ticket, because "the cloud-provider expects Availability Sets which OpenShift infra doesn't have in order to figure out which nodes to remove" doesn't matter if the cloud-provider isn't removing any OpenShift nodes in the first place.
can you please confirm the standalone VM scenario is supported? It's a critical use case for CAPZ (our control plane VMs in zone-enabled regions are not placed in availability sets)

What happened: introduced new regex which is too restrictive for some existing installations: This new restriction causes LBs to fail to sync if just one ipConf fails regex. What you expected to happen: This change was to enable optional behavior, any ipConf that fails regex should just be ignored. This will allow existing installs to work as they did prior to the change. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
Suggest putting a continue here: This will restore previous functionality and appears to have no undesirable impact.
/sig cloud-provider /area provider/azure
Unclear if we should also put a continue here as well:
/assign
could you share an example of resourceID (please replace your subscription/resourceName with fake ones)? /triage accepted
Here's the error we're seeing: Just the string


Question: // f must run inside an It or Context callback. f := framework.NewDefaultFramework("stress") <del> init := func() *stressTest { </del> <ins> init := func() { </ins> cs = f.ClientSet <del> l := &stressTest{} </del> <ins> l = &stressTest{} </ins> // Now do the more expensive test initialization. l.config, l.driverCleanup = driver.PrepareTest(f)

/sig storage What happened: When you interrupt the e2e storage stress test, namespaced objects are cleaned up, but non-namespaced objects like StorageClasses are not cleaned up. What you expected to happen: Non-namespaced objects like StorageClasses are cleaned up so long as the user doesn't interrupt a second time. How to reproduce it (as minimally and precisely as possible):
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: pvName := pvc.Spec.VolumeName pv, err := b.pvCache.GetPV(pvName) if err != nil { <del> return false, err </del> <ins> if _, ok := err.(*errNotFound); ok { err = nil } return true, false, err </ins> } pv, err = b.tryTranslatePVToCSI(pv, csiNode) if err != nil { <del> return false, err </del> <ins> return false, true, err </ins> } err = volumeutil.CheckNodeAffinity(pv, node.Labels) if err != nil { klog.V(4).Infof("PersistentVolume %q, Node %q mismatch for Pod %q: %v", pvName, node.Name, podName, err) <del> return false, nil </del> <ins> return false, true, nil </ins> } klog.V(5).Infof("PersistentVolume %q, Node %q matches for Pod %q", pvName, node.Name, podName) } klog.V(4).Infof("All bound volumes for Pod %q match with Node %q", podName, node.Name) <del> return true, nil </del> <ins> return true, true, nil </ins> } // findMatchingVolumes tries to find matching volumes for given claims,

What would you like to be : Scheduler VolumeBinding plugin reports error when a PVC doesn't exist for example: We need to take a second look at the VolumeBinding plugin and make sure that errors are returned only when there is a system error, not a user error. Why is this needed: To fix the scheduler's error reporting metrics. /sig scheduling /sig storage /cc
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign I'll take a look.
If the PVC object does not exist, the plugin will return status. I guess you mean the plugin will return an error when a referenced PV does not exist? I've created a PR for this:
Thanks! The errors I am seeing are on 1.18 clusters, can we look at 1.18 and try to patch? This is still throwing the scheduler's error metrics off.
sure
I've submitted a patch for the 1.18 release branch:
I guess on the master branch should be reviewed and merged too? cc
hello! the fix you made only apply for 1.18? how about 1.19 and 1.17? thanks!
after the fix on the master branch () is merged, I'll cherry-pick it into the 1.19 branch (the code are almost the same). I'm not sure if it's necessary to fix the 1.17 branch. It's 3 versions behind 1.20.
hi, This is because of this patch: Do you think this fix needs to be cherry-picked into 1.19 and 1.18 branch too?
can you send a patch for 1.19?
ah, here it is
to sum up: master: 1.19: 1.18: and I think this change should be backported to 1.19/1.18 too: I'll send PRs soon.

summary of the issue that an error will be reported instead of UnschedulableAndUnresolvable when PVC not found: master: already implemented, see 1.19: 1.18: can't backport as it does not implement the PreFilter extension point


Question: if readyIn < 0 { framework.Failf("Pod became ready before startupProbe succeeded") } <del> if readyIn > 5*time.Second { framework.Failf("Pod became ready in %v, more than 5s after startupProbe succeeded. It means that the delay readiness probes were not initiated immediately after startup finished.", readyIn) </del> <ins> if readyIn > 25*time.Second { framework.Failf("Pod became ready in %v, more than 25s after startupProbe succeeded. It means that the delay readiness probes were not initiated immediately after startup finished.", readyIn) </ins> } })

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Seems to be impacting a few KIND jobs on the master branch the most, but also assorted other jobs see the triage link above Reported by Sample failure: /sig node <!-- Please see the deflaking doc () for more guidance! --
xref probing tests
/assign ref /triage accepted
Thanks Elana, still trying today... Does anyone know if we can use the kubelet logs in e2e tests?
Not very portably in cluster e2e tests at least. Maybe in node_e2e instead.
Have you considered that the issue is not with the test but with the underlying code in kubelet like detailed in this issue Like detailed in the issue I can reliably reproduce (on k8s 1.18) the startup probe passing within a few seconds, but the container is only marked as ready after about a minute.
there is already a fix in the 1.21 release for that (part of a bigger PR): And actually this test was to verify that part of the code, but unfortunately sometimes it doesn't respond as quick as it should (but never on my PC, only in ci/cd tests)...
Some recent flakes:
one of the most common flakes in kind jobs currently failures like: Mar 25 17:03:42.508: Pod became ready in 42.007286524s, more than 5s after startupProbe succeeded. It means that the delay readiness probes were not initiated immediately after startup finished.
Maybe we could speed up the review on
Thank you I will take a look. I'm making a pass through flakes tonight trying to identify if anything is on kind's end
I don't think it's kind... since running on my PC with kind it works 100%, I think the issue is either the resources... or contention in the queue as mentioned to in the comment
/milestone v1.22
commented on PR with the suggested fix
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.

</details
Yeah...
Thanks for your great work in keeping things in check
FYI
I have (finally) understood what's happening, and I can even explain why the deflake made things worse... Please have a look at the sleep before the worker starts to process probes: So, you have to remember that we have one worker per pod and per probe, and since we have these probe settings for the test: We have (PeriodSeconds=10) starting sometimes immediately (if the Kubelet has started since more than 10s) or after a quick sleep of rand10s. On the contrary, will likely sleep most of the time for rand60s. This causes a delay in the processing of notifications: If we are lucky, this delay is less than 5s and the test passes: Now you can understand why the deflaked settings were actually worse:
The reason I didn't spot these was that I usually start my cluster once for all and iterate on the tests... so the kubelet had plenty of time to start and I never ended in that sleep.
For the solution, I suggest we either remove the sleep completely, or we find something that would make all 3 probe workers start at the same time to avoid sync issues... For the record, I was able to spot this issue thanks to the following log error (because the channel was overflowing): What do you think ?
I have found another issue in that code: If has already ran (and failed), we go to L245 and don't probe via and therefore wait for the next tick, making the test fail.
This seems consistent with what I found when I was adding e2es for the new probe-level termination grace period and why I did my timing math the way I did.
/reopen I don't think this is fixed, I just hit it again and gubernator is still showing plenty of flakes:
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
how to check the exact commit used for each test?

For instance, I am not sure (from the name) that should include my latest fix...
In your PR the fix wasn't there:
you may send cherry-pick to 1.21 for which is using kubelet 1.21, not 1.22.
is using kubelet. Cherry-pick to 1.20 may be needed as well.
I have found this one that failed even with the fix: corresponding to:
I don't know what could explain this atm... but it seems we still have delay propagating container statuses.
In the job artifacts there are multiple places with this info including the clone records/logs, and In testgrid it's a row.
yeah afaict since the tests ran on my PR after your fix was merged into master, it showed the flake was still present :( Prow will always use the latest master when running CI.
Sorry for doubting what you said... I was probably in deny and hoped this annoyance was fixed once for all :(
Plan: we'll see if Clayton's pod lifecycle refactor and/or the jitter refactor for probes affects this.
Clayton's PR jitter refactor
since isn't (likely) coming to 1.22 we could cut off 90% of flakes by increasing to 15s delay:
it seems it fails even after merging so let's continue our discussion on slack
this is not a regression, our suggestion is not block release on it. feel free to remove the milestone
/milestone v1.23
In Kubelet maybe, but in the flakiness of the e2e tests it was. Unfortunately we can't easily explore the current state :/
In the release blocking jobs the frequency is not horrible now, so unblocking is probably fine, it is not as bad as it was. Outside of the release cycle this is still the flakiest test in some jobs, see e.g. where it's the only flake source we have on record currently.
if we end up needing it, we have a timing/period adjustment PR we can take that should help with the flakiness with


Question: // the newer overlapping ones (only sync the oldest one). New/old is determined by when the // deployment's selector is last updated. func (dc *DeploymentController) handleOverlap(d *extensions.Deployment) error { <del> selector, err := unversioned.LabelSelectorAsSelector(d.Spec.Selector) if err != nil { return fmt.Errorf("deployment %s/%s has invalid label selector: %v", d.Namespace, d.Name, err) } </del> deployments, err := dc.dLister.Deployments(d.Namespace).List(labels.Everything()) if err != nil { return fmt.Errorf("error listing deployments in namespace %s: %v", d.Namespace, err) } overlapping := false for _, other := range deployments { <del> if !selector.Empty() && selector.Matches(labels.Set(other.Spec.Template.Labels)) && d.UID != other.UID { </del> <ins> foundOverlaps, err := util.OverlapsWith(d, other) if err != nil { return err } if foundOverlaps { </ins> deploymentCopy, err := util.DeploymentDeepCopy(other) if err != nil { return err

A developer accidentally made two deployments with labels that overlapped, such that one of the deployment's selectors matched the pods from both deployments. In the repro below the labels are exactly the same, but this also happened one one set of labels is a subset of another. Expected: An error? Actual: Cluster CPU went past 100% and everything else broke. Crazy thrashing with kubernetes shutting down pods & restarting new ones so quickly that we could not even get logs. They never made it past the Pending state. We are running kubernetes 1.2 on GKE. Here's a minimal repro, just apply it (on a cluster you don't need! D: )
See also , . My guess is that this is due to pod relabeling. cc
this shouldn't be an issue in 1.4 anymore
I just ran into this issue and I'm on v1.4.4. My cluster goes into a crazy Pod creation frenzy even though the number of Pod replicas in both the deployments is set to 1 because one of the overlapping Deployment seems to be "redeploying" itself thousands of times. On a single node k8s it quickly reached the limit of 110 pods and makes the cluster unusable until I delete one of the deployments. This is a very nasty bug as it will overrun the whole cluster. I my case the matching labels are only partial. Both deployments have the label "run=nginx" but one of them has a second selected label as well. What happens is that one of the deployments gets the annotation: "" And that Deployment goes into a loop of "redeploying" itself so it tries to create hundreds of pods and overruns the cluster. Looking at the live Deployment yaml it shows "generation : 6616". I've attached the two test Deployments files to reproduce this and I noticed that the ORDER you deploy them in MATTERS! $ kubectl create -f deployment- (and wait until it is fully deployed) $ kubectl create -f deployment- (and watch all hell break loose) I recommend using a disposable cluster even though, if you are quick enough, you can recover by deleting one of the Deployments (it will take a while!) and then its ReplicaSet.

Client/Server versions: Client Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.4", GitCommit:"", GitTreeState:"clean", BuildDate:"2016-10-21T02:48:38Z", GoVersion:"go1.7.1", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.4", GitCommit:"", GitTreeState:"clean", BuildDate:"2016-10-21T02:42:39Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}
Yes it's a bug. deployment-one has selector & label run=my-nginx deployment-two has selector & label run=my-nginx, other-label=somevalue deployment-two was updated later In the code we look for overlapping deployments by checking if one's selector matches the other's label: In this case, deployment-one selector matches deployment-two (but not the other way around), so only deployment-one knows there's an overlapping deployment. The we choose the deployment that's last updated to ignore (skip syncing): In this case, deployment-one assumes deployment-two will skip syncing itself, so deployment-one continues. But deployment-two doesn't know there's an overlapping deployment. Updating the statement for finding overlapping deployment should solve it.
So you mean that the original/older Deployment always "wins" and the newer deployed one doesn't get any Pods launched or the Deployment "relaunched" even though after deployment the older Deployment is the one in violation (like in my use case)?
Sent the
In this case they overlap with each other (I mean, not only the older one is in violation). The one whose selector is last updated should be the one which stops syncing. Otherwise it'll be surprising when someone adds a new deployment or update a deployment's selector causes my existing deployment to stop working. The bug is because the latter one doesn't know it overlap with the former one. With the fix it can detect other overlapping deployments correctly.
Hello Guys! I think that I have a relevant question related to this issue. I would appreciate any answer. That configuration, which generate this bad behavior. I mean to have a common label between two deploys wouldn't be required to implement a canary release?

If the canary release was just a different nginx's version? I am asking this because I have this configuration ,even though I don't have this mess hapening, my deployment never reachs my desired state. If I describe this deployment I got a warning: !!!WARNING!!! This deployment has overlapping label selector with deployment "my-nginx" and won't behave as expected. Please fix it before continue. I am using kubernetes 1.5.1, have I miss something related to concept or implementation of canary release?
can you post the deployment yaml files you are running? Seems like they are overlapping. Are both deployments broken or is it just one of them?
Hello Just one was broken, unfortunately I overwrite the files with another deployment configuration. Probably. I was my mistake, because now everything is working. I got my canary release through the files that follows: I am sharing because I got here ins this thread with this problem.These configurations could be useful to someone with the same problem. Thank you for your attention to my question.

Deployment stable apiVersion: extensions/v1beta1 kind: Deployment metadata: name: ise spec: replicas: 3 template: metadata: labels: app: "ise" track: "stable" spec: containers: name: "nginx" image: "nginx:1.8.1" ports: containerPort: 80 lifecycle: preStop: exec: command: ["/usr/sbin/nginx","-s","quit"] livenessProbe: httpGet: path: / port: 80 scheme: HTTP initialDelaySeconds: 10 periodSeconds: 30 timeoutSeconds: 5 readinessProbe: httpGet: path: / port: 80 scheme: HTTP initialDelaySeconds: 5 timeoutSeconds: 1 Deployment canary apiVersion: extensions/v1beta1 kind: Deployment metadata: name: ise-canary spec: replicas: 1 template: metadata: labels: app: "ise" track: "canary" spec: containers: name: "nginx" image: "nginx:1.9.12" ports: containerPort: 80 lifecycle: preStop: exec: command: ["/usr/sbin/nginx","-s","quit"] livenessProbe: httpGet: path: / port: 80 scheme: HTTP initialDelaySeconds: 10 periodSeconds: 30 timeoutSeconds: 5 readinessProbe: httpGet: path: / port: 80 scheme: HTTP initialDelaySeconds: 5 timeoutSeconds: 1 Service apiVersion: v1 kind: Service metadata: name: ise spec: ports: port: 80 selector: app: ise type: LoadBalancer
Between two overlapping deployments, one being broken is expected today - you should also see an event about it. This won't be an issue in releases after 1.6. I am assuming deployments in old clusters (1.6+) in the future will contain owner references plus maybe we will have selector generation. Related discussion in and , Adriano S. Fonseca < :


Question: # See the License for the specific language governing permissions and # limitations under the License. <del> FROM golang:1.12.0 AS builder </del> <ins> FROM golang:1.15.3 AS builder </ins> COPY elasticsearch_logging_discovery.go go.mod go.sum / RUN CGO_ENABLED=0 GOOS=linux GO111MODULE=on go build -a -ldflags "-w" -o /elasticsearch_logging_discovery /elasticsearch_logging_discovery.go

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: es-image as part of fluent-elasticsearch cluster addon does not build What you expected to happen: in the es-image folder should work How to reproduce it (as minimally and precisely as possible): cd cluster/addons/fluentd-elasticsearch/es-image make Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/assign /triage accepted
/sig instrumentation


Question: assert.True(t, reflect.DeepEqual(updatedIngress2.Spec, fedIngress.Spec), "Spec of updated ingress is not equal") assert.Equal(t, updatedIngress2.ObjectMeta.Annotations["A"], fedIngress.ObjectMeta.Annotations["A"], "Updated annotation not transferred from federated to cluster ingress.") <del> // Test add cluster t.Log("Adding a second cluster") </del> fedIngress.Annotations[staticIPNameKeyWritable] = "foo" // Make sure that the base object has a static IP name first. fedIngressWatch.Modify(&fedIngress) <del> clusterWatch.Add(cluster2) t.Log("Checking that the ingress got created in cluster 2") </del> <ins> t.Log("Checking that the ingress got created in cluster 2 after a global ip was assigned") </ins> createdIngress2 := GetIngressFromChan(t, cluster2IngressCreateChan) assert.NotNil(t, createdIngress2) assert.True(t, reflect.DeepEqual(fedIngress.Spec, createdIngress2.Spec), "Spec of created ingress is not equal")

The Stop() method in the Federated Informer races with deleteCluster() in how they both try to close (stopChan). Thanks for the heads up
Fixed by Thanks Maru.


Question: return azureDiskList } <ins> // getDiskLinkByDevName get disk link by device name from devLinkPath, e.g. /dev/disk/azure/, /dev/disk/by-id/ func getDiskLinkByDevName(io ioHandler, devLinkPath, devName string) (string, error) { dirs, err := io.ReadDir(devLinkPath) glog.V(12).Infof("azureDisk - begin to find %s from %s", devName, devLinkPath) if err == nil { for _, f := range dirs { diskPath := devLinkPath + f.Name() glog.V(12).Infof("azureDisk - begin to Readlink: %s", diskPath) link, linkErr := io.Readlink(diskPath) if linkErr != nil { glog.Warningf("azureDisk - read link (%s) error: %v", diskPath, linkErr) continue } if libstrings.HasSuffix(link, devName) { return diskPath, nil } } return "", fmt.Errorf("device name(%s) is not found under %s", devName, devLinkPath) } return "", fmt.Errorf("read %s error: %v", devLinkPath, err) } </ins> func scsiHostRescan(io ioHandler, exec mount.Exec) { scsi_path := "/sys/class/scsi_host/" if dirs, err := io.ReadDir(scsi_path); err == nil {

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: In a customer , customer is unable to use azure disk in StatefulSet since /dev/sd changed after detach/attach disk. Currently we are using to reference the data disk which could be changed in some condition, see code: What you expected to happen: How to reproduce it (as minimally and precisely as possible): 5 StatefulSet with 1 replica and 1 azure disk mount each 1 StatefulSet and recreate the same StatefulSet would be azure disks could not be usable since /dev/sd changed Update: It seems that on azure china, this issue could repro very easily. Anything else we need to know?: I have checked the official doc: Environment: Kubernetes version (use ): from v1.6.0 to v1.10.0 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: /sig azure /assign I am working on a fix for this issue, would use instead of
We changed how we find disks on nodes quite some time back. What server version is this?
yes, I have already checked the code, the point is we are still using (code is ) to "mount -bind" k8s path, while could be changed when VM is attach/detaching data disks, see And I have also checked related AWS, GCE code, they are using other than ,And I have also checked related AWS, GCE code, they are using other than , see This issue could be stably repro on Azure China, while I could not repro that on other region up to now. I have already got a fix for this, still in testing now.
also add for awareness, I am working on a fix to use other than .
I would not use "/dev/disk/by-id" instead utilize the special UDEV rules we have created for our VMs -- They have the advantage that they create symlinks for all of the disks attached to the VM and the LUN assigned to a disk remains the same. The sysmlinks you find under /dev/disk/azure. For a master-node you will see a layout like the following.

In scsi1 you find the additonal (data) disks A newly assigned disk is goign to appear as lun1,lun2,... Since we are not able to guaranty the order in which disks get assigned to a VM I would go and base the code on this already created layout
thanks for the check. So do you mean we use , instead? actually in Azure VM with UDEV rules created, they have no difference, they link to the same /dev/sd* (see below layout). While I also got some customer cases(e.g. coreos, see ), they don't have such UDEV rules created, which means does not exist, I would say would be more common, every linux distro have that path. From debugging perspective, the path is more intuitive.
BTW, I have a azure disk debugging doc: which uses , it's really intuitive for debugging perpurse.
it is right that at the end we point to /dev/sdi like in the sample above the advantage we have with the lun0 is that it acts as an abstraction layer where we can be sure that an application is always utilizing the disk with lun0, independent what actuall device name gets assigned to it. So at the moment it is ../../../sdi and after a reboot it could become ../../../sdk. But thanks to these UDEV rules /dev/disk/azure/scsi1/lun0 will always point to the right device behind it. I also would favour the usage of the lun over the disk-id if from a maintanance/support point of view as we can directly correlate what lun belongs to what disk. Otherwise I have at first fo to the VM directly to find out what disk might have an issue Regarding the centos issue, if we have the waagent installed on it then those new UDEV rules should be available as well. So it wonders me that we do not have them working well. At least I had not seen any customer SR who had reported an issue with those rules specifically on CentOS
It's not centos, it's coreos issue, see If we use , then we would heavily depends on UDEV rules.

I agree that one depends on these special UDEV rules in this case , but on each Azure VM the agent is to be installed anyway, so that is background why I suggested to use the LUNs
I think proposal is the cleanest and safest. However we will have to tell all Azure users that they have to have azure UDEV rules configured in nodes, even if WAAGENT is not installed.
I did hit the issue, using an ACS based Kubernetes cluster running 1.7.7. We are already in the process of moving over to AKS based clusters, so was simply hoping that there would be some escalated effort to bring 1.8.7 into AKS given that the azure-disk provisioner is the default StorageClass.Not sure if you are looking for any more debug info at this time, but I still have the affected pod active and should be able to get you some more information, so just let me know. that's great, I would like to take a look to make sure it's that issue, could you follow this to collect the data.
Sadly the node became NotReady, and the cluster redeployed the pod to a separate node, which has rectified the issue. We have been seeing the issue with far frequency, so I will make sure to gather the data next time it pops up.
That said, the reason I was confident it was this issue previously is that when I exec'd into the shell, I got the "Input / Output error" upon trying to "ls" the mount point. Furthermore, attempting to re-mount that point gave an error to the extent of "Special device /dev/sde does not exist"
Could you recall the VM had restarted before you hit that disk issue? Or just after attach/detach disk, that issue come up?


Question: rbacv1helpers.NewClusterBinding("system:kube-controller-manager").Users(user.KubeControllerManager).BindingOrDie(), rbacv1helpers.NewClusterBinding("system:kube-dns").SAs("kube-system", "kube-dns").BindingOrDie(), rbacv1helpers.NewClusterBinding("system:kube-scheduler").Users(user.KubeScheduler).BindingOrDie(), <del> rbacv1helpers.NewClusterBinding("system:aws-cloud-provider").SAs("kube-system", "aws-cloud-provider").BindingOrDie(), </del> rbacv1helpers.NewClusterBinding("system:volume-scheduler").Users(user.KubeScheduler).BindingOrDie(), // This default binding of the system:node role to the system:nodes group is deprecated in 1.7 with the availability of the Node authorizer.

As of , all Kubernetes clusters with RBAC bootstrap roles for AWS, regardless of whether they are running on AWS. The hardcoded RBAC bootstrap policy should only include roles that are used by all clusters, regardless of addons, configuration, or cloud provider. The ClusterRole(Binding)s should be removed from the bootstrap policy, and instead configured by the provisioning tools. /sig aws /sig cloud-provider /sig auth
/assign I can help do it. :)


Question: } // separate out the nodes that have the label from the ones that don't <del> otherNodes := []string{} labeledNodes := map[string]string{} for _, node := range nodes { if labels.Set(node.Labels).Has(s.label) { label := labels.Set(node.Labels).Get(s.label) labeledNodes[node.Name] = label } else { otherNodes = append(otherNodes, node.Name) } } </del> <ins> labeledNodes, nonLabeledNodes := s.getNodeClassificationByLabels(nodes) </ins> podCounts := map[string]int{} for _, pod := range nsServicePods { label, exists := labeledNodes[pod.Spec.NodeName]

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Kubernetes version (use ): master What happened: The functionality is hidden in a large function, and not explicit, testable. What you expected to happen: The logic of should read more clearly, with tests for individual filters.
has a PR for this .
Few more things include removing limits in entries like
yeah, im not sure why the thing is there, after all they are just appending to the slice and the limit wont have an effect , from what i can tell, will it? probably good idea to remove unless im missing something.
Its part of part of PR too.


Question: return true } if isTagged { <ins> hostname, _ := dockerref.SplitHostname(named) </ins> // Check the RepoTags for an exact match for _, tag := range inspected.RepoTags { <del> if tag == image { // We found a specific tag that we were looking for return true </del> <ins> // Deal with image with hostname specified if len(hostname) > 0 { if strings.HasSuffix(image, tag) { return true } } else { if tag == image { // We found a specific tag that we were looking for return true } </ins> } } }

I'm launching master components from static pods with kubelet. won't launch my pods. Gives this error: I suspect this very recent change: Can you advise anything
- Can you please check if works for you? fyi, i am off on vacation for a week so response may be a bit slow
Yes, it works. Thanks.


Question: SubPathExpr: "$(POD_NAME)", }, { <del> Name:      "workdir2", </del> <ins> Name:      "workdir1", </ins> MountPath: "/testcontainer", }, },

/kind cleanup /sig storage /sig node /sig testing Description: Change e2e tests to use EmptyDir instead of HostPath as pre-requisite to promoting VolumeSubpathEnvExpansion to Conformance. Related Issues: Conformance Tests:
/cc
/assign


Question: return nil } <ins> // Stop retrying if we exceed statusUpdateRetries - the DaemonSet will be requeued with a rate limit. if i >= StatusUpdateRetries { break } </ins> // Update the set with the latest resource version for the next poll if toUpdate, getErr = dsClient.Get(context.TODO(), ds.Name, metav1.GetOptions{}); getErr != nil { // If the GET fails we can't trust status.Replicas anymore. This error

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --Almost every update to a DaemonSet can lead to 2~3 error logs in kube-controller-manager even when the DaemonSet manages very few Pods, for example, there were 3 error logs when updating a daemonset in a small cluster of 2 Nodes. Other controllers like deployment don't have this issue when they manage much more Pods. After looking at DaemonSet controller code, it seems the retry logic in "storeDaemonSetStatus" was not implemented correctly. It never really retried and always made an useless GET call. DaemonSet controller should produce less error logs in normal case. Create or update a DaemonSet to trigger Pod recreation. Kubernetes version (use ): v1.20.4 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig apps /assign created to fix the retry logic.
Same problem with me. </brCould you explain why other controllers(like deployment controller) is not easy to get conflict when updating status? I think there are many in every controller.
the number of workers doesn't matter here as the workqueue guarantees that they will not process the same resource at the same time. The conflict comes from multiple successive processing of a resource because of the update delay of the resource's cache in . For example, when a DaemonSet manages 2 Pods and they are getting ready. first Pod is ready and triggers updating Daemonset, whose resourceVersion is in Lister. updates DaemonSet's ready count to 1 with resourceVersion set to and it succeeds. The resourceVersion in server becomes and the Informer/Lister hasn't received the update.

second Pod is ready and triggers updating Daemonset, whose resourceVersion is still in Lister. updates DaemonSet's ready count to 2 with resourceVersion set to and it fails with the "the object has been modified" error. To avoid this error, retry is the most straightforward way but it is implemented differently in controllers. In deployment controller, it actually has the same conflict but it only logs the error with so normally you don't see it, and it requeues the resource and re-processes it upon conflict. DaemonSet controller and Replicaset controller retry directly when calling fails. However, the former made a mistake when checking the loop's boundary condition and it never retried. is fixing it.
Very clear explanation. Thanks.


Question: switch status.State { case kubecontainer.ContainerStateCreated: <del> // nothing to do but wait for it to start </del> <ins> // The main sync loop should have created and started the container // in one step. If the init container is in the 'created' state, // it is likely that the container runtime failed to start it. To // prevent the container from getting stuck in the 'created' state, // restart it. changes.InitContainersToStart = append(changes.InitContainersToStart, i) </ins> case kubecontainer.ContainerStateRunning: if !types.IsRestartableInitContainer(container) {

It was consistently observed in a testing environment that a node-critical pod failed to start. event was present but was missing( is an init container). Also the following error was present but strangely only once: After some digging it seemed throwing an error matched the observations above. Eventually the init container and the pod to start The following tests in recreate the problem by and show how depending on the behavior is different. When the feature gate is enabled the last init container gets stuck in and never gets started. Therefore the pod get's stuck and never progresses The problem seems to stem from: There case get's skipped and the created container is expected to get started by the previous one by the following lines or But the provided tests show that this does not happen. Also if there is only one init container with the following logic should execute but won't because in this case this get's executed <details</details<details</details<details</details<details</details<details</details<details</details
/sig node can you help checking this issue report as according to it is a regression introduced with Thanks in advance!
I'll take a look. /assign
I guess this is a valid bug. /triage accepted /priority important-soon Can you provide a more specific scenario for this? Why do container runtime fail to start the container? (How can I reproduce it in e2e test?)
^ I guess this is the wrong assumption. If the container is in the state, the kubelet should resync it.
During node bootstrapping we had a bug where got restarted. So it is possible the following to happen: container gets created -restarts -container fails to start
We have a similar case. I have not yet looked at how containerd behaves at the moment, because it happens rarely and only at node startup. Pod stuck in the status. In the suppress status we have the following message: Alse we have the next events: Node characteristics: And it's like the error is reproduced on 1.29. The neighbouring cluster with k8s 1.28 is not affected by this problem
we have observed a similar behaviour but intermittently while upgrading our k8s cluster from k8s 1.29.0 to 1.29.6.

During upgrade process, when the node gets rebooted the "calico-node" pod of that node got stuck with below error forever until i restart the pod manually. Note: We are not observing this every time. Please let me know for any additional info required.
/cc


Question: return fmt.Sprintf("No rollout history found in %s %q", resource, name), nil } // Sort the revisionToChangeCause map by revision <del> var revisions []string for k := range historyInfo.RevisionToTemplate { revisions = append(revisions, strconv.FormatInt(k, 10)) </del> <ins> var revisions []int64 for r := range historyInfo.RevisionToTemplate { revisions = append(revisions, r) </ins> } <del> sort.Strings(revisions) </del> <ins> sliceutil.SortInts64(revisions) </ins> return tabbedString(func(out io.Writer) error { fmt.Fprintf(out, "%s %q:n", resource, name) fmt.Fprintf(out, "REVISIONtCHANGE-CAUSEn") <del> errs := []error{} </del> for _, r := range revisions { // Find the change-cause of revision r <del> r64, err := strconv.ParseInt(r, 10, 64) if err != nil { errs = append(errs, err) continue } changeCause := historyInfo.RevisionToTemplate[r64].Annotations[ChangeCauseAnnotation] </del> <ins> changeCause := historyInfo.RevisionToTemplate[r].Annotations[ChangeCauseAnnotation] </ins> if len(changeCause) == 0 { changeCause = "<none>" } <del> fmt.Fprintf(out, "%st%sn", r, changeCause) </del> <ins> fmt.Fprintf(out, "%dt%sn", r, changeCause) </ins> } <del> return errors.NewAggregate(errs) </del> <ins> return nil </ins> }) }

Ref and should sort revisions correctly (10 should be after 9, not 1).
I can help with this bug. Looks like trivial fix in PrintRolloutHistory
Thanks
are you still taking care of this one ? I am searching for some easy to do PR to just get familiar with the process and put my first PR in.
I created pr some time ago to close this issue
great thanks :-)
you can look for help-wanted issues or check and
thanks


Question: volumeSpec := volume.NewSpecFromPersistentVolume(pv, false) migratable, err := expc.csiMigratedPluginManager.IsMigratable(volumeSpec) if err != nil { <del> klog.V(4).Infof("failed to check CSI migration status for PVC: %s with error: %v", util.ClaimToClaimKey(pvc), err) </del> <ins> klog.V(4).Infof("failed to check CSI migration status for PVC: %s with error: %v", key, err) </ins> return nil } // handle CSI migration scenarios before invoking FindExpandablePluginBySpec for in-tree

I migrated an existing EBS volume from in-tree plugin to external aws-ebs-csi-driver (). Resizing PVC twice in 6 hour window causes kube-controller-manager to crash. Exception message: Kube-controller-manager shouldn't crash in such case. If external aws-ebs-csi-driver fails to provision the pvc it should just log an warning or error and continue operating as usual running cluster version 1.22.9 using in-tree aws ebs driver. pvc: Install aws-ebs-csi-driver following featureGate CSIMigrationAWS on all cluster components (with draining nodes before enabling featureGate on kubelets - according to warning ) we have ebs volume that has been migrated to aws-ebs-csi-driver. pvc by increasing .spec.resources.request.storage to 2Gi. PVC, PV and EBS volume are correctly resized. once again not later than 6 hours after first resize. Increase .spec.resources.request.storage to 3Gi. Now kube-controller-manager throws an exception and crashes: EBS CSI Driver logs: I tried also enabling alpha featureGate InTreePluginAWSUnregister but the issue still occurs. <details</details<detailsAWS </details<details</details<details</details<detailsdocker </details<detailsaws-ebs-csi-driver version 1.10.0 </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cloud-provider
/sig storage some situation where the sync loop passes a nil parameter to
/assign
SetClaimResizer may return nil pvc and non-nil error, ClaimToClaimKey naively assumes non-nil as described above, will create PR
But the posted bug is I think 1.22 specific and should never happen on master or 1.24 and hence this fix against master is kind of okay and should fix the issue but at the same time code in is still kinda broken in 1.22 because it returns if patching fails.