Question: if err := c.updateInstanceSecurityGroupForNLBTraffic(sgID, sgPerms, healthRuleAnnotation, "tcp", healthCheckPorts, vpcCIDRs); err != nil { return err } <del> if err := c.updateInstanceSecurityGroupForNLBTraffic(sgID, sgPerms, clientRuleAnnotation, "tcp", clientPorts, clientCIDRs); err != nil { </del> <ins> if err := c.updateInstanceSecurityGroupForNLBTraffic(sgID, sgPerms, clientRuleAnnotation, clientProtocol, clientPorts, clientCIDRs); err != nil { </ins> return err } } else { if err := c.updateInstanceSecurityGroupForNLBTraffic(sgID, sgPerms, healthRuleAnnotation, "tcp", nil, nil); err != nil { return err } <del> if err := c.updateInstanceSecurityGroupForNLBTraffic(sgID, sgPerms, clientRuleAnnotation, "tcp", nil, nil); err != nil { </del> <ins> if err := c.updateInstanceSecurityGroupForNLBTraffic(sgID, sgPerms, clientRuleAnnotation, clientProtocol, nil, nil); err != nil { </ins> return err } }

What happened: What you expected to happen: kubernetes creating an AWS NLB for UDP How to reproduce it (as minimally and precisely as possible): Create service with annotation and port with Anything else we need to know?: AWS recently announced that UDP is available within their NLB Environment: Kubernetes version (use ): 1.11.9 Cloud provider or hardware configuration: AWS
/sig aws
/kind feature
/remove-kind bug
/milestone v1.16
You must be a member of the GitHub team to set the milestone. If you believe you should be able to issue the /milestone command, please contact your and have them propose you as an additional delegate for this responsibility. <detailsIn response to : /milestone v1.16 Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/milestone v1.16
I am taking up this Feature Development. How can this be assigned to me?
/assign
+1
+1 I am experiencing the same issue. NLB also supports UDP_TCP type, any service using "nlb" should be able to mix TCP and UDP port configs.
Any progress? I can collaborate on this, if you need a hand.
+1
+1 On Thu, Jul 18, 2019 at 12:58 Adam Murphy <:
+1
I just ran into this. Any updates?
Attempting to fix it with
Thanks, The Changes looks good.
This should be closed I think.
This feature only for 1.16?
Could feature back ported into 1.15 1.14 1.13
Could someone review please?
/sig cloud-provider
Any update on this?
As we are in the code freeze and this issue is still not fixed, I'll move it to the 1.17 milestone. I'm also going to add the PR that fixes this issue to the milestone. If there are any concerns, please ping me or ping SIG-Release on Slack. /milestone v1.17
Do you have a timeline when this will be released?
This is a breaking issue in multiple versions but the is not prioritized. Does anyone have a workaround at the moment?
For AWS you can associate elastic IP with pod. Need make aws cli call.

If you need static private IP then you can create DaemonSet attach NIC with static IP as second NIC to EC2 instance and node some label (like ). Later you start pod with network, get node label, bound to required host IP. If you need static public and private IPs, then you can customize node-group init script and attach NIC with permanent private IP and Elastic IP at node startup. Later simple start pod with network
Bug triage for release 1.17 here. Since this issue has been tagged for the milestone, we would like to check it's status. Code freeze is on Nov. 14. Will this issue be resolved before then? If not, we should move it out of the milestone.
Hi bug triage team for 1.17, will this issue be resolved before code freeze on Nov. 14 (Next Thursday)?
/milestone v1.18 (v1.17 code freeze)
looking for a first issue to work, can I take this
gave some good feedback to the original PR. Does this need a reset with a fresh PR? I also have a need for this. I don't have a strong urge to write, or reuse, an operator for NLBs in AWS for UDP. This will eventually get done in k8s anyways, so I'd rather put the effort there.
BTW, you can always setup an NLB manually to forward UDP to a nodePort service on your k8s instances. The issue with that is that you have to pick a NodePort service for UDP before hand, and manually handle the NLB and the target groups. One cluster upgrade and your target group behind the NLB disappears since all of the instances get replaced. It's not ideal but in a pinch...
I don't mind updating a the PR, seems like a good first issue for me.
/assign
+1
+1
+1
+1
+1
Hello ! Bug Triage team here for the 1.18 release. This is a friendly reminder that code freeze is scheduled for 5 March, which is a few weeks from now. Is this issue still intended for milestone 1.18? Thanks in advance!

C/O Code freeze for release 1.18 is starting in 5 March (less than 1 week from now) and we want to ensure that each issue has a chance to be finished on time. Is this still on track for this deadline?
/remove-kind bug
Is there any work remaining to be done on this? Willing to contribute if so, would really love to see this make it into 1.18!
Thanks for the reply This has been a long running issue, and given that there has not been any recent activity, would it make more sense to once again move the milestone? We are 2 days away from code freeze (5 March).
I wonder what "activity" you are expecting.
We are now in code freeze for 1.18, so I am moving this to the next milestone. /milestone v1.19
Hello, as of now, I see is 65% completed. May we please have an update on estimated release date of UDP support for AWS NLB?
People have been trying to be constructive here. But the team doesn't even try to tell us what is pending. Sad.
Would also love to see this feature for EKS - and am willing to help out. Even if we could get it in 1.19 it would still take a long time to become available in EKS (which is currently at 1.15). So the sooner we can start work on this - the better :-)
+1 Also hope we could get in 1.19 and happy to help out!
any update on this pretty please ?
whats the update here?
From what I can see, the main issues that remain unresolved since the first PR last year: is still pending if you need a service that supports both TCP and UDP, e.g., DNS. DNS is also a more-complicated example because it supports TCP and UDP on the same port number, and that'll need more refactoring here, as AWS treats that as a single multi-protocol target. Edit: To be clear, not really a blocker for UDP support, but it's been raised in the discussion earlier. AWS NLB can only do TCP-based health checks (including HTTP and HTTPS), so your service needs to have a health-check TCP port listening. Current PRs silently assume it's on the same port number as the UDP service you're advertising.

If you're using , then it'll use the health check from  instead anyway. So this is probably only an issue for LoadBalancer Services with set. See "Things to know" on the .


Question: ds:            newDaemonSet("ds"), shouldEnqueue: true, }, <ins> { test: "Node conditions changed", oldNode: func() *v1.Node { node := newNode("node1", nil) node.Status.Conditions = []v1.NodeCondition{ {Type: v1.NodeOutOfDisk, Status: v1.ConditionTrue}, } return node }(), newNode:       newNode("node1", nil), ds:            newDaemonSet("ds"), shouldEnqueue: true, }, { test: "Node conditions not changed", oldNode: func() *v1.Node { node := newNode("node1", nil) node.Status.Conditions = []v1.NodeCondition{ {Type: v1.NodeOutOfDisk, Status: v1.ConditionTrue}, {Type: v1.NodeMemoryPressure, Status: v1.ConditionFalse}, {Type: v1.NodeDiskPressure, Status: v1.ConditionFalse}, {Type: v1.NodeNetworkUnavailable, Status: v1.ConditionFalse}, {Type: v1.NodeInodePressure, Status: v1.ConditionFalse}, } return node }(), newNode: func() *v1.Node { node := newNode("node1", nil) node.Status.Conditions = []v1.NodeCondition{ {Type: v1.NodeOutOfDisk, Status: v1.ConditionTrue}, {Type: v1.NodeInodePressure, Status: v1.ConditionFalse}, } return node }(), ds:            newDaemonSet("ds"), shouldEnqueue: false, }, </ins> } for _, c := range cases { manager, podControl, _ := newTestController()

deamonset would not schedule pod on nodes of OutofDisk state,on the other hand, in it's updateNode func, it return directly if there is no change in labels or taints of the node. That is, ds controller could not make sense when a node turn into OutofDisk state, or recover from it. Unless we restart the controller-manager component
/assign
if other conditions in func "nodeShouldRunDaemonPod" changed? say, a node does not have enough cpu or memory resource to create the pod of ds, later I delete pods on it, and it satisfy the request now, but the ds controller could not feel it
, thanks for your cases, they're reasonable to me; let me find solutions for them.


Question: return err } <ins> // the latest view of the namespace asserts that namespace is no longer deleting.. if namespace.DeletionTimestamp.IsZero() { return nil } </ins> // if the namespace is already finalized, delete it if finalized(namespace) { <del> err = kubeClient.Core().Namespaces().Delete(namespace.Name, nil) </del> <ins> var opts *api.DeleteOptions uid := namespace.UID if len(uid) > 0 { opts = &api.DeleteOptions{Preconditions: &api.Preconditions{UID: &uid}} } err = kubeClient.Core().Namespaces().Delete(namespace.Name, opts) </ins> if err != nil && !errors.IsNotFound(err) { return err }

See for full dumps. The line number of corresponds to:
From a quick look, perhaps the namespace object returned from retryOnConflictError() just above no longer has a DeletionTimestamp and thus "*namespace.DeletionTimestamp" triggers the panic?
Thanks for the report, I will investigate.


Question: APIVersion: "v1", }, ObjectMeta: metav1.ObjectMeta{ <del> Name: config.Prefix + "-injector", </del> <ins> Name: podName, </ins> Labels: map[string]string{ "role": config.Prefix + "-injector", },

Failing Job: These tests started flaking recently (they were consistently green for a long time): Mounted volume expand[Slow] Should verify mounted devices can be resized Volume expand [Slow] Verify if editing PVC allows resize Subpath [Volume type: gcePDPVC] should support restarting containers using file as subpath [Slow] Subpath [Volume type: gcePDPVC] should fail for new directories when readOnly specified in the volumeSource Subpath [Volume type: gcePDPVC] should fail if subpath with backstepping is outside the volume [Slow] Volumes PD should be mountable with ext4 Affected runs: https://k8s- https://k8s- https://k8s- /kind bug /sig storage /priority important-soon /milestone v1.12 /cc - Could you PTAL or delegate? Thanks.
Volume expansion tests have been fixed by
/assign can you investigate the remaining failures?
[MILESTONENOTIFIER] Milestone Issue Needs Approval Action required: This issue must have the label applied by a SIG maintainer. If the label is not applied within 4 days, the issue will be moved out of the v1.12 milestone. <details<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
Reopening this issue, as a bunch of storage-related e2es started flaking in our large-scale jobs recently: https://k8s- https://k8s- /reopen
I think the latest failures are caused by gce API rate limiting: /assign
It seems to only be failing on scalability test suites. I wonder if it's due to some different cidr configuration
/sig scalability
/kind failing-test
/status approved-for-milestone Friendly ping. This is affecting release-blocking scalability job.
hey what needs to be done here? by whom?
Could you comment on what needs to be done here and by whom?
can you guys take a look at the API quota issue?

This is happening consistently in the scale test environment
as Code Freeze is today, this needs to be of priority . Please triage! cc
It looks like the latest run on 9/1 has passed.
Marking closed as the fixes have been merged and the test runs are now passing /close
Closing this issue. <detailsIn response to : Marking closed as the fixes have been merged and the test runs are now passing /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: // phase for each node // it's initialized in the PreFilter phase podVolumesByNode map[string]*scheduling.PodVolumes <ins> sync.Mutex </ins> } func (d *stateData) Clone() framework.StateData {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: during phase from the plugin when the is updated: What you expected to happen: Not crash. How to reproduce it (as minimally and precisely as possible): I am suspecting that the issue is in the naive implementation of the 's method: Most likely the is cloned during filter phase, e.g. in the called during in the : ... and when is cloned during filtering, new mutex is created, but is not cloned properly, this mean that potentially there are two or more different mutexes protecting the same map from writes. Anything else we need to know?: I think potential solution is to have mutex in the or change the 's . Environment: Kubernetes version (use ): 1.19.0 Cloud provider or hardware configuration: not relevant OS (e.g: ): not relevant Kernel (e.g. ): not relevant Install tools: not relevant Network plugin and version (if this is a network-related bug): not relevant Others: not relevant /sig scheduling
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scheduling
It looks like relevant code is from and mutex per was considered intially. CC
IIUC, the may (edit: if the node has nominated pods) be duplicated in for every node before passing to extension point. So we cann't rely on 's lock to prevent the state from being modified concurrently. Am I right?
I may have given an incorrect analysis as VolumeBinding's plugin-specific state are always using the same pointer, sorry for the confusion. I removed the confusing comments. It's actually different from what was discussed in . Your and 's analysis make more sense to me.

For nodes that have nominated nodes (cloned CycleState) or not (original CycleState), the original mutex may not be able to protect the CycleState: I will take another look tomorrow.
can you please post the actual core dump, the description in the original comment doesn't seem complete. If the "concurrent map" fatal error is indeed happening at , then it means we are operating on two different CycleStates instances at the same time, which may happen when adding nominated pods () Using the CycleState lock to synchronize access to the plugin's state in the map is not proper, the plugin state itself should have its private lock, this is certainly a bug.
/cc


Question: . "github.com/onsi/gomega" ) <del> // Flaky issue #17873 var _ = Describe("Cluster level logging using Elasticsearch [Flaky]", func() { </del> <ins> var _ = Describe("Cluster level logging using Elasticsearch [Feature:Elasticsearch]", func() { </ins> f := NewFramework("es-logging") BeforeEach(func() {

It failed in gce suite. I honestly have no idea under which team it falls:
Another failure: bumping to p0. - can you find an owner, or at least an appropriate team for this?
owns logging (see team description).
Cannot reproduce failure so far. Leaving it to run overnight...
Still cannot reproduce the exact failure... Increase verbosity...
There were a lot of such failures over the weekend:
Moving this test to flaky (), as it's only 66% stable and is blocking the build quite often: please prioritize this.
This test was passing 100% of the time until Build (Dec 5, 2015 4:13:38 PM). After that, it failed 50% of the time (10 failures in 20 runs). Clearly a PR that merged shortly before the above time introduced the flakyness. That PR should be rolled back.
hmm - I looked into PRs that were merged around that time, and actually nothing looks suspicious to me
This test is 100% successful in flaky test suite so far. I suspect some other e2e test is impacting this one. Adding more sanity checks... (root)/Kubernetes%%20suite/ClusterlevelloggingusingElasticsearchshouldcheckthatlogsfrompodsonallnodesareingestedinto_Elasticsearch/
Let it run in flaky suite overnight. If everything goes well, will unmark it.
I can't follow any of the links but if I could get a bit more information about what is going wrong I could look into the Elasticsearch test if you think that is the issue rather than something else having a bad impact on it.
Thanks for offering to help. Here are the failure symptoms in Jenkins: cluster in red status. I bump into this once with my test cluster. It is cause by es split brain problem. Fix already merged. all log lines from one node. I cannot reproduce this. My guess is the fluentd pod on a node is not working properly. So I validation for fluentd pods. Can you think of anything else?
Can you please point me to the fix for the split brain problem? Is that ? So that works by making sure there are always two master nodes? Is 2 really a good idea for quorum? As suggested here a choice of 2 might not be desirable?

Missing log lines indeed mean a synthetic logging pod did not start for some reason -- without more log information it's hard to work out exactly why this occurred. If I get a moment I will try to do a repo. Thanks.
minimummasternodes set to 2 does not mean having 2 master nodes. It means the es cluster needs to have at least 2 nodes before electing a master. If network partition happened to a es non-master node, it will wait until it could reach out to at least one other node to start electing master. Hence it will prevent the split brain problem. Based on the log in jenkins, all synthetic logging pod ran successfully.
It gets flaky again... I am able to reproduce the problem by running DaemonRestart and Elasticsearch test together. It looks like the DaemonRestart test somehow disrupts the ES cluster. ES cluster gets into bad state, hence failing the es test. Confirmed with This particular test restarts all the containers.
It's a recent regression. Found the cause and will upload a fix soon.
- please monitor this test and close this bug if there won't be any failures in some time.
Test is green for the last 40 builds. Closing the issue.
This test is still marked as at . please send a PR to promote it out of ?
Since is the owner of the issue. I'll let him decide whether the test is ready.
The test has been successful lately. Want to add one more check to improve resilience. I had a pending PR to remove label. Will try to merge it on Monday if everything goes well during the weekend.
I think that is because elasticsearch query return unexpect results. should solve it.
Weirdly hit the same unrelated pr twice, but it looks like the fix just got merged
Still flaking:
what's next on this? Seems to not be resolved...
Moved back into flaky in , is trying for repro/solidifying the test IIUC
Trying to reproduce the problem. It is running successful in flaky suite and my local cluster so far... Will try to run it with other tests together.
Two types of failure: cluster in bad state. This only happens when running with other disruptive tests. Never failed in suite where it is not running with disruptive tests. ES resilience can be improved by log lines in Elasticsearch.

I am able to reproduce this once. It is due to fluentd is hanged. Filed a issue to upstream. Request to mark this test as [Feature] since it depends on two 3rd party components.
The risk here is that ES is a default cluster addon. IMO it's fine not testing it for critical builds, but the risk is we'll ship something that straight crashloops. So if we move it to feature I'd ask that you add a smaller test that verifies basic functionality to a blocking suite (ping es, check that it accepts some logs). Or make it non-default.
Good point. I will work on first. It has to be done anyway. It will also make future ES updates easier. BTW, ES addon is not default for GCE/GKE.
Really? my e2e cluster always has it, why are we testing with it then?
For GCE:
Yeah - another place where test and 'production' configs differ, but I guess we need to test elasticsearch for 'fully opensource' version, thus it's justified...
That's [Feature] then, just like we test flannel. Not something to bring up as an addon on every e2e clusters.
We probably need some suite with all 'important' options (e.g. Features) turned on, on top of dedicated suite for every feature. Ugh...


Question: expectedFail:         false, }, { <ins> // Canonicalize node names. machines: []string{"NODE0", "node1"}, fakeNodeHandler: &FakeNodeHandler{ CreateHook: func(fake *FakeNodeHandler, node *api.Node) bool { if node.Name == "NODE0" { return false } return true }, }, retryCount:           1, expectedRequestCount: 2, expectedCreateCount:  2, expectedFail:         false, }, { </ins> // No machine to register. machines: []string{}, fakeNodeHandler: &FakeNodeHandler{

NodeController is unable to register the value of , because it contains mixed case from the OS (in this case OSX). I'm not positive that all Linux systems are guaranteed to lowercase hostname -f, but it's likely that automated processes may end up collecting hostnames from sources that preserve case and passing that to the kubelet or master. I believe we should lowercase names detected from hostname -f and the values the node controller users - it's a minor thing, but internally we can guarantee that we treat node names as insensitive and in their canonical interior form.
cc
I'll take a look tmr.


Question: }) framework.ExpectNoError(err, "failed to locate ResourceQuota %q in namespace %q", patchedResourceQuota.Name, ns) <del> err = wait.PollUntilContextTimeout(ctx, 5*time.Second, 5*time.Minute, true, func(ctx context.Context) (bool, error) { </del> <ins> // the resource_quota_controller ignores changes to the status so we have to wait for a full resync of the controller // to reconcile the status again, this full resync is set every 5 minutes by default so we need to poll at least one // minute more just in case we we start to poll just after the full resync has happened and he have to wait until // next full resync. // Ref: https://issues.k8s.io/121911 err = wait.PollUntilContextTimeout(ctx, 5*time.Second, 6*time.Minute, true, func(ctx context.Context) (bool, error) { </ins> resourceQuotaResult, err := rqClient.Get(ctx, rqName, metav1.GetOptions{}) <del> framework.ExpectNoError(err) </del> <ins> if err != nil { return false, nil } </ins> <del> if apiequality.Semantic.DeepEqual(resourceQuotaResult.Spec.Hard.Cpu(), resourceQuotaResult.Status.Hard.Cpu()) { gomega.Expect(*resourceQuotaResult.Status.Hard.Cpu()).To(gomega.Equal(resource.MustParse("1")), "Hard cpu value for ResourceQuota %q is %s not 1.", repatchedResourceQuota.Name, repatchedResourceQuota.Status.Hard.Cpu().String()) gomega.Expect(*resourceQuotaResult.Status.Hard.Memory()).To(gomega.Equal(resource.MustParse("1Gi")), "Hard memory value for ResourceQuota %q is %s not 1Gi.", repatchedResourceQuota.Name, repatchedResourceQuota.Status.Hard.Memory().String()) </del> <ins> if *resourceQuotaResult.Spec.Hard.Cpu() == *resourceQuotaResult.Status.Hard.Cpu() { if *resourceQuotaResult.Status.Hard.Cpu() != resource.MustParse("1") { framework.Logf("Hard cpu status value for ResourceQuota %q is %s not 1.", repatchedResourceQuota.Name, resourceQuotaResult.Status.Hard.Cpu().String()) return false, nil } if *resourceQuotaResult.Status.Hard.Memory() != resource.MustParse("1Gi") { framework.Logf("Hard memory status value for ResourceQuota %q is %s not 1Gi.", repatchedResourceQuota.Name, resourceQuotaResult.Status.Hard.Memory().String()) return false, nil } </ins> framework.Logf("ResourceQuota %q Spec was unchanged and /status reset", resourceQuotaResult.Name) return true, nil } <ins> framework.Logf("ResourceQuota %q Spec and Status does not match: %#v", resourceQuotaResult.Name, resourceQuotaResult) </ins> return false, nil }) <del> framework.ExpectNoError(err) </del> <ins> if err != nil { framework.Failf("Error waiting for ResourceQuota %q to reset its Status: %v", patchedResourceQuota.Name, err) } </ins> }) })

TestGrid: /kind flake /sig api-machinery
checking there is a ginkgo panic, independently of this issue, is it normal that we panic on these cases? coming back to the issue, the error comes from I couldn't spend much time, but we need to understand why the ratelimiter errors
It started to flake more after 2023/10/10. Before , we use will not return context err. After , we use and it may fail for .
There is something odd here, why they will exceed the deadline , it is just the second try I noticed previous loops use a different timeout is it possible that we are using the wrong context here? if that is the case, why it cancel the request in flight?
Using assertions inside a callback as in is wrong. An error should be returned instead. Or better yet, don't use ... An assertion inside a is not ideal either, but I think would handle it better than .
The context is the one from That looks right to me. In , there is a "normal" , but given the lack of information in the log output it is impossible to tell how long it has been polling. seems normal to me, too, if is called shortly before the 5 min timeout. The then isn't the number of times that polled, it's the number of times that the rate limiter retried the .
mentioned that as well and This may finally be changed to something like below:
ok, so it really times out, there is a 5 mins gap, between the error and the previous log Can anybody get the test right per Patrick suggestions? asserting inside the loops and creating panics is definitely wrong
don't use these jobs to debug this problem, those create a multi master cluster and there is a loadbalancer in between that makes it hard to debug,
+ should work together to fix this issue. FYI:
Reporting the timeout will become better, but it doesn't help when the expected state really isn't reached.
Given that there is additional checking in it might be best to use a custom matcher that is specific to this particular test.
Another idea that I keep coming back to, but haven't turned into an issue or some actual code: if some failure occurs, then we need to know about the state of the cluster and events that may have been relevant for the test. was written to gather such information.

I think it would be nicer if this was part of a report after a failure. That report then can also be more detailed than log messages that get written regardless of whether the test passes or not. One way to do that would be to gather events in that watcher in memory instead of logging them and when generating the failure message in the custom matcher, include that information. That would be limited to this test. Ideally, the framework should make that easier or even do it automatically - not sure yet how... perhaps by manipulating the Gingko failure handler while a framework instance is active. There's one potential downside of not dumping information during a successful run: it becomes impossible to compare a "good" run with a "bad" run. But that's already true today for the CI logs because we discard the log output from "good" runs. For running tests locally, a flag could enable dumping this information also for passing tests. It could even be on by default and only get turned off in the CI? Or perhaps always be enabled?
that is already dumped today, isn't it?
yeah, that's broken. it will panic on the first GET call from the RQ client if it fails. first thing to fix here is to remove all the assertions from the PollUntilContextTimeout handler function. there could be log messages and/or state can be stored, then once the poll is done state can be asserted outside of the handler.
Something is dumped. But I am not sure how complete it is. I think it also only dumps events that still exist. If some were garbage-collected, then they won't show up. Not sure whether that is a problem. Anyway, definitely needs to be looked into.
I don't know the owner of the resource quota controller, can someone check why the resource quota controller takes more than 5 minutes and fails the test?
/assign Having a look
From kube-controller-manager logs:
the flake is caused by a metrics-server addon deployment failure? RS controller fails to deploy metrics server pods because of a quota problem -the RQ controller's discovery client fails to get a fresh state for the metrics APIservice. i could be reading this wrong... there is also this:
is an example where this test is PASSING while metrics server discovery is failing.

We should probably still figure out why the metrics server is not starting (cc but it MIGHT be a red herring here.
I filtered and then diffed the kube-controller-manager logs between a passing test and a failing test, looking for any difference in behavior in the RQ controller, I was not able to observe any difference in behavior from the logs..
/sig instrumentation /cc
/assign Found the race, tricky one, checking The test fail after modifying the status and waiting for the resourcequotacontroller to reconcile the spec to status again. However, the controller ignores updates to the status on purpose assuming is the only owner of the status and waits for the full reconcile loop to recover it If we observer this failure, the first check is at 03:30:44.387 and it times out at 03:35:44.388 if we check the kube-controller-manager logs we can see that the resourcequotacontroller loop runs just before and after we just need to take this into account on the test so the poll loop always get one of the full calculation usage The problem is that this period is random and it seems the defaul is 12 hours ???? on tests is 3 mins what is the value of this field for kubeadm? I answer myself, I just checked in a kind cluster and is the default
ok, my bad, I didn't follow the code right, the resync period for the resource quota controller is an option in and the default is 5 minutes that matches with the observation on the failed test , everything adds up now
/triage accepted


Question: } func TestCPUManagerAdd(t *testing.T) { <ins> testPolicy := NewStaticPolicy( &topology.CPUTopology{ NumCPUs:    4, NumSockets: 1, NumCores:   4, CPUDetails: map[int]topology.CPUInfo{ 0: {CoreID: 0, SocketID: 0}, 1: {CoreID: 1, SocketID: 0}, 2: {CoreID: 2, SocketID: 0}, 3: {CoreID: 3, SocketID: 0}, }, }, 0) </ins> testCases := []struct { description string <del> regErr      error </del> updateErr   error <ins> policy      Policy expCPUSet   cpuset.CPUSet </ins> expErr      error }{ { description: "cpu manager add - no error", <del> regErr:      nil, </del> updateErr:   nil, <ins> policy:      testPolicy, expCPUSet:   cpuset.NewCPUSet(3, 4), </ins> expErr:      nil, }, { description: "cpu manager add - policy add container error", <del> regErr:      fmt.Errorf("fake reg error"), </del> updateErr:   nil, <del> expErr:      fmt.Errorf("fake reg error"), </del> <ins> policy: &mockPolicy{ err: fmt.Errorf("fake reg error"), }, expCPUSet: cpuset.NewCPUSet(1, 2, 3, 4), expErr:    fmt.Errorf("fake reg error"), </ins> }, { description: "cpu manager add - container update error", <del> regErr:      nil, </del> updateErr:   fmt.Errorf("fake update error"), <del> expErr:      nil, </del> <ins> policy:      testPolicy, expCPUSet:   cpuset.NewCPUSet(1, 2, 3, 4), expErr:      fmt.Errorf("fake update error"), </ins> }, } for _, testCase := range testCases { mgr := &manager{ <del> policy: &mockPolicy{ err: testCase.regErr, }, </del> <ins> policy: testCase.policy, </ins> state: &mockState{ assignments:   state.ContainerCPUAssignments{}, <del> defaultCPUSet: cpuset.NewCPUSet(), </del> <ins> defaultCPUSet: cpuset.NewCPUSet(1, 2, 3, 4), </ins> }, containerRuntime: mockRuntimeService{ err: testCase.updateErr,

/kind bug What happened: Sometimes when an deployment creates new pods they get scheduled on a node successfully but the containers fail to start with: What you expected to happen: The scheduler shouldn't put pods on nodes without enough resources. How to reproduce it (as minimally and precisely as possible): I don't know how it got into this situation. Deleting the pods and have the RS recreate them works around this issue. Anything else we need to know?: We're using the CPUManager with static policy which might contribute to this problem. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: AWS OS (e.g. from /etc/os-release): coreos /sig scheduling
/sig node (for CPU manager)
do the Kubelet logs seem similar to what's reported in this PR description?
/assign
Hi! :) No, I don't see these 'still running' messages. This is what I've got in the kubelet log (slightly redacted):
Can you share more information on this? Deployment manifest, platform details, frequency of these errors?
Fortunately we have open source the platform deployment: - but in short it's: AWS (c5.4xlarge instances for the workers where this issue is happening) multi master w/ kubeadm Enabled CPUManager and set to static policy Running 5 ASG: Controller, GPU-enabled workers and 3x regular workers (one per AZ) Using cluster autoscaler and HPA We have this problem with some pods pretty much all the time. Since we have pretty varying workloads, the CA often moves pods around and scales up/down which works around the problem by deleting the affected pods. So sometimes all pods are healthy, sometime maybe 15% of the pods have this issue. The biggest issue for us currently is that it's blocking rolling upgrades: Since new pods take forever to start, it blocks our CI. There are several deployments affected. All use HPA and all have resource limits of 2 cpus. One has 5Gi, the other 2Gi memory limits but that shouldn't matter. Here is one deployment manifest (slightly edited for internals) Why does this "Internal PreStartContainer hook" even exists? Shouldn't the scheduler be what knows about available resources and decides about this? Is there some doc that could help me understand how this is suppose to work?

I'd imaging if there is some limits the scheduler doesn't know about (like exclusively available cores or something), it would explain the issue but it would be unclear how this can work at all?
I've just realize that only one specific node (which is suppose to be the same as all other nodes) seems to be affected on which all pods that require full CPUs (triggering the static allocation) fail. Going to taint and drain the node now and will decomision it ~tomorrow, so let me know if you want me to pull something from the node first. Unfortunately I don't have long enough log rentention to see what kubelet might have logged before it got into this state.
My kubelet args:
Maybe relevant: Now that I'm trying to drain the node, I realize that there are ~100 argo () workflows objects "mapped" to the node. I don't know the details and how they are related to the node, I only know that I can't drain the node because of the workflows (that's another issue with argo or CRDs in general, see ). Maybe this is somehow blocking the cpus?
Ah, I misunderstood that error. There are ~100 completed pods that were created by argo. Could it by simply that somehow the finished pods don't free their cpu resources somehow?
We have fixed the completed pods related issue before:
would you mind digging into the cpu manager state checkpointed to disk? This will confirm or refute for us that we failed to release CPUs from a terminal container. The state file is located by default in and the content is simply utf-8 JSON. We should be able to see some container-to-cpuset mappings in there. If we can verify that some of the mappings are for terminal container IDs then that's where we should focus the search. My next question would be, were the PreStopContainer and PostStopContainer callbacks invoked, and if so, was there a failed write to the checkpoint file. PreStopContainer is not guaranteed to be called, in case the container process exits on its own. PostStopContainer is called as part of container GC. If there's some other problem interfering with the GC logic then that may have prevented cleanup of our CPU manager state too. Thanks for helping us to debug this remotely!

Ah damn, looks like the cluster autoscaler terminated that node already so unfortunately can't provide the cpumanagerstate.. Would be nice to have the relevant bits exposed as Prometheus metrics. I'll update this issue if this happens again.
"Good" news, problem just re-appeared - this time on two nodes. Here is one: cpumanagerstate: One of the containers doesn't exist at all and another one is exited: Here is the state field from for the exited container: It has "": "koko-intelligence--np2zh", set and according to k8s this pod is still running on that host: I found this in the kubelet log for this container:
Oh and it's indeed running, now in container which is also properly accounted for in the state file. Now regarding the non-existent container, I also see it in the kubelet log: Looks like this belongs to koko-gateway-track--6mljm, which doesn't exist anymore. But most likely because it just got gracefully removed eventually. Maybe some race condition preventing it to remove it and leading to these inconsistencies?
This is awesome info — thanks for updating. I won’t have time to look at this immediately, but perhaps can help trace it. I suspect some kind of race in reconcileState.
Thanks for the info I will take a look at this. /assign
/ Do you have a rough estimate for this? Fully understand if you have other things on your plate but then I'd probably revert the static policy since this becomes a bigger and bigger problem for us (it looks like it got worse over time, now even nodes are affected that were just brought up).
/ Is there some way I can help you debug/fix this? Seems unrelated to anything special in our setup from what I can tell. We had to disable this for all our deployments because it was causing severe issues.
/ Do you have any update/ETA on this?
Note that this is fixed in 1.12, but backports to 1.11 and 1.10 are not merged yet:
This still happens occasionally in 1.12 (with ) . provides cleanup for stalled entries.
Out of curiousity, does anyone still seeing this issue utilize containers in their pod Specs?

If so, the curlprit is likely addressed by the following:
The issue is still existing in kubernetes v1.14.4, without init containers to pod spec. Some way below may reproduce it: container crash on a node frequently and then delete the pod. a new pod and schedule to this node. Is this already fixed on master branch?


Question: # Taint Windows nodes by default to prevent Linux workloads from being # scheduled onto them. WINDOWS_NODE_TAINTS="${WINDOWS_NODE_TAINTS:-node.kubernetes.io/os=win1809:NoSchedule}" <ins> # Whether to set up a private GCE cluster, i.e. a cluster where nodes have only private IPs. GCE_PRIVATE_CLUSTER="${KUBE_GCE_PRIVATE_CLUSTER:-false}" </ins>

Public IPs are expensive resources. Given that we're running scalability tests on 5k-node clusters, we are unnecessarily consuming 5k IPs which is a lot. We should switch our tests so that nodes have only private IP addresses. We are going to do that by configuring NAT so that nodes can talk to outside world (e.g. to download image from dockerhub). But that also requires changing some correctness tests which assume public IPs.
To summarize recent work: setting up private gce clusters with nat, feature is gated via the env test and dashboard with correctness tests on gce private cluster - https://k8s- 27 tests fail on fixing the tests - should fix majority of them
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale /lifecycle frozen
- is there anything missing to be done here?
No, we can close this one. It was done long time ago. Thanks for catching /close
Closing this issue. <detailsIn response to : No, we can close this one. It was done long time ago. Thanks for catching Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: "k8s.io/client-go/restmapper" "k8s.io/client-go/util/retry" featuregatetesting "k8s.io/component-base/featuregate/testing" <ins> basemetrics "k8s.io/component-base/metrics" "k8s.io/component-base/metrics/testutil" </ins> "k8s.io/controller-manager/pkg/informerfactory" "k8s.io/klog/v2" kubeapiservertesting "k8s.io/kubernetes/cmd/kube-apiserver/app/testing" podutil "k8s.io/kubernetes/pkg/api/v1/pod" "k8s.io/kubernetes/pkg/controller/garbagecollector" jobcontroller "k8s.io/kubernetes/pkg/controller/job" <ins> "k8s.io/kubernetes/pkg/controller/job/metrics" </ins> "k8s.io/kubernetes/pkg/features" "k8s.io/kubernetes/test/integration/framework" "k8s.io/utils/pointer"

Occasionally job completions are counted twice (or potentially multiple times?) in the metric. This happens for about 20% of jobs, but may depend on the Job specificity, which results in an overestimation of the number of finished jobs. The metric reflects the actual number of completed jobs. a kind cluster with prometheus monitoring stack (for example follow this: ). the number of completed jobs by inspecting the metric and delete a job multiple times in a loop Example job: Example repro script: . Check the number of completed jobs by inspecting the metric and the job Yes, in the kube-controller-logs we find the following log lines which are related and probably indicate repeated updated which results in double counting. <details</details<detailsReproducible with kind </details<details</details<details</details<details</details<details</details
/assign
FYI
/sig apps
/triage accepted


Question: // GetOriginalConfiguration retrieves the original configuration of the object // from the annotation, or nil if no annotation was found. <del> func GetOriginalConfiguration(info *resource.Info) ([]byte, error) { annots, err := info.Mapping.MetadataAccessor.Annotations(info.Object) </del> <ins> func GetOriginalConfiguration(mapping *meta.RESTMapping, obj runtime.Object) ([]byte, error) { annots, err := mapping.MetadataAccessor.Annotations(obj) </ins> if err != nil { return nil, err }

From an e2e test failure: The command should retry resource version mismatches like scale and rollingupdate. cc
/cc fyi.
From perspective , Isn't the failure intentional (i.e. optimistic mutate) ? or... maybe I'm (probably) missing something. I would think its the user's job to retry... ?
The desired behavior that we settled on previously is to detect version mismatches at the client and error. User can force replacement using --force. See , and .
What is the mismatch here? If it is just resourceVersion, kubectl should retry. It should only bail on a meaningful conflict. We're working on eliminating flaky tests (and flaky parts of the system). Also, that log message isn't very useful.
per discussion on , we can't determine the correct behavior when the resource version changes without more information, so best to catch version mismatch at the client and bail, and let the user replace the object with --force if they want. We can also make the error message less verbose.
I don't see any mention of resourceVersion in . It shouldn't be serialized in the annotation, because the user shouldn't specify it in their config. The resourceVersion-based precondition is intended to guard against concurrent read-modify-write operations. kubectl apply should be able retry the entire diff&patch in the case of this type of conflict. That said, it's not obvious to me from the log message what the actual problem was in this case.
I'm sorry, my mistake. You're right that resourceVersion is not in the annotation, and that we should retry on resourceVersion failure. I was confusing this with the api version change issue discussed in . This is actually a much simpler issue, in the sense that it's an optimistic lock conflict. If we encounter a lock conflict, then we should fetch a fresh copy of the object, and try again to apply the new configuration supplied by the user. If we don't get a merge conflict when we resubmit, then all's well. If we get a merge conflict because the object has changed out from under the user, then we report it in those terms, and the user can decide what to do (e.g., replace using --force). Does this fit your expectation of how it should be handled?
This is causing test flake:
yes, your understanding is correct.

Do you have time to fix this? We're trying to get all our tests fixed. Additionally this sounds like a legit system bug, not just a flaky test problem.
Maybe. Swamped right now, but can fit it in given enough notice. When is the fix needed?
asap. We'll find someone else.
This should just be a generic GET/Update retry loop, eg:
Go for it. Jack Greenfield | Google , Prashanth B wrote:
Chao, do you have cycles to take a look?
I thought Patch is retried in the server. I'll investigate.
Patch is retried at server-side five times before an error is returned: , so I expect this issue to occur very rarely. In fact, it has passed last ~400 kubernetes-gce-e2e tests. If we want to make this test less flaky, I'd recommend increase the number of the retries at the server-side. What do you guys think?
It'll fail if an actual conflicting change was made, though. Do we know what the nature of the conflict was? Maybe first we need to add some log messages to figure out what the conflict was next time it occurs. , Chao Xu wrote:
This happened last week:
you are right. I thought an actual conflicting change would result in a different error message but it's not. I'll add the logs.
In I a log entry if there is a meaningful conflict. That said, I don't think we have a meaningful conflict in this particular e2e test. After taking a closer look at the implementation of , I think the submitted is precise, so we shouldn't retry at the client side if server reports a meaningful conflict. [edit] And if the conflict is not meaningful, we should just rely on the server-side patch handler to retry.
does the milestone on this need to be changed to v1.2?
How often is the test flaking?
Last time I checked, I'll check it again.
Last time I checked the results was two days ago, in the past 48 hours, the test has consecutively passed another 124 builds ( to ).
I only see 3 reports of failures since October, so I'm going to say, no, it doesn't need to be fixed in 1.2.
any progress on this now?

I returned back to the kubectl apply issues, and hope I can fix all apply related issues before 1.3.0 release.
I'll try to check the flake rate again. IMO if the flake rate is non-negligible, we should increase the retry limits of Patch, or more aggressively, just remove the limit.
Infinite retry still scares me, mostly because we may have failure modes that could race for a very long time. , Chao Xu wrote:
I don't think anyone has worked on this. Though the original failing test/logs weren't recorded, it appears to be a failure of "should apply a new configuration to an existing RC": There is a retry loop in the Patch handler in apiserver, but if resourceVersion is specified, the retry loop will always fail in the case of a conflict. (We should detect that and bail out, but I don't think we do.) In the case of such a conflict, apply should diff again, create another patch, and try again. I agree with that it shouldn't retry forever. We should use the same retry limits we use for other commands.
we have an infinite loop in GuaranteedUpdate (), maybe we should apply a retry limit there as well?
Agree. Just note that the test flake doesn't specify resourceVersion.
Since this is causing flakey test failures, it is something we should focus on. Is this something I should pick up?
FYI: has a PR out for this under review
Making this a P3 (no release block) since I looks like we haven't seen many instances since Feb 2016


Question: "strings" "sync" <ins> "github.com/golang/glog" </ins> "k8s.io/apimachinery/pkg/util/sets" "k8s.io/kubernetes/plugin/pkg/scheduler/algorithm" "k8s.io/kubernetes/plugin/pkg/scheduler/algorithm/predicates" "k8s.io/kubernetes/plugin/pkg/scheduler/algorithm/priorities" schedulerapi "k8s.io/kubernetes/plugin/pkg/scheduler/api" <del> "github.com/golang/glog" </del> ) // PluginFactoryArgs are passed to all plugin factory functions.

Weight is an int (see ). Priority can take the range 1-10. Total priority for a node is sum of weight  priority for each priority function, and the sum is accumulated into an int (see ). So obviously the total priority can overflow if someone sets the weight(s) too high. The weights of all priority functions are compiled in (or specified in config file), so we can check to make sure maximum total priority can't overflow an int (take sum over all priority functions of 10  weight of that priority function, and make sure that's smaller than int(MaxUint 1) which according to StackOverflow is the way to find out what is max int (math package has max Int32, Int64, etc. but not max Int). Note that scheduler extender is OK because although the policy runs in a separate process, the weight is specified in scheduler config file (ExtenderConfig struct) and the priority is bounded 1-10 as usual.
We really should also make "score" be an abstract datatype rather than a raw int that is manipulated directly, so we can do things like make sure it is in the range 1-10 (or 0-10 if that's what we want); today the range is inconistent (we say 1-10, but sometimes use 0; probably we should change it to 0-9..).
I volunteer myself to do this.
I think we can check weight overflow in after we get all PriorityConfig. what do you think about this?
As for "score" I think we can use
Fixed by - closing.
Tim that PR was closed, not merged. I don't think this is fixed yet.
whoops, good catch, just removing milestone then. There was a shift that happened yesterday where a bunch of items suddenly came into the 1.6 query.
do we still want this fixed? I can re-pick this.
AFAIK it's still broken. cc/
Ok, I will open a PR tomorrow.


Question: pair := topologyPair{key: c.TopologyKey, value: tpVal} cnt = *s.TopologyPairToPodCounts[pair] } <ins> cnt = adjustForMaxSkew(cnt, int64(c.MaxSkew)) </ins> score += float64(cnt) * s.TopologyNormalizingWeight[i] } }

Spin-off from What would you like to be : Use maxSkew in TopologySpreadingContraint to affect relative weight of the constraint. Exact formula to be discussed. Some ideas: , intuitive but too coarse-grained at low numbers. Goes: 100%, 50%, 33%, 25%, 20% some linearity. For , goes: 100%, 66%, 50%, 40%, 33% . For , goes: 100%, 75%, 60%, 50%, 42% . Goes: 100%, 70%, 57%, 50%, 44% Why is this needed: When using for , it will be useful for operators to control how strong the constraints should be compared to explicit user requests. Pods would be able to control the weight of constraints per-topology.
/sig scheduling
1 maybe the most compatible option with the current zone:node ratio (2:1):
Not quite. The ratio is correct, but the weight is different. original: prop 1:
I realized that using as weight is quite counterintuitive. Instead, I propose that we use to water down the counts per topology. This way, the behavior still doesn't change for . But as increases, counts become more similar. For example, given the counts and , we get the new counts: . Then we proceed with the counts normalization as usual. Note that when the minimum count is zero in a topology, this procedure has the effect of giving the max score (100) for nodes (or zones) that would satisfy the condition. That won't be true when the minimum count is not zero, but enforcing it would be quite costly. This proposal still gets us closer to that and reduce score differentiation among nodes within the skew. When using constraints against multiple topologies, another side effect is that topologies with bigger would have a reduced effect in the overall score.
That sounds better than the initial proposal which seemed to deviate from the original semantics of the API.
WDYT ?
Sounds good to me. Per constraint on large-scope topology (e.g., zone), I assume it will still be credited with the matching number on its sub topologies (e.g., node), right? (It's just we're using the newCount formula - by taking the maxSkew specified on its constraint into consideration.)

Note that this doesn't replace increasing the weight as described in the parent issue. I think the issue title needs updating.
Understood. Default weight is tracked in another issue. This one talks about Scoring.
Right, just wanted to explicitly mention that to clarify the overall plan since this issue started in the context of increasing the default weight :)
Could you rename that one and clarify that it depends on this issue being resolved first?
can you clarify why that issue is dependent on this one? aren't we going to increase the weight regardless?
I was referring to . But you are right, it doesn't depend on this. kubernetes/enhancements does


Question: now time.Time // expectations <del> expectCreate               bool expectDelete               bool expectActive               int expectedWarnings           int expectErr                  bool expectRequeueAfter         bool jobStillNotFoundInLister   bool jobPresentInCJActiveStatus bool jobCreateError             error </del> <ins> expectCreate             bool expectDelete             bool expectActive             int expectedWarnings         int expectErr                bool expectRequeueAfter       bool jobStillNotFoundInLister bool jobCreateError           error </ins> }{ <del> "never ran, not valid schedule, A":      {A, F, errorSchedule, noDead, F, F, justAfterThePriorHour(), justBeforeTheHour(), F, F, 0, 1, F, F, F, T, nil}, "never ran, not valid schedule, F":      {f, F, errorSchedule, noDead, F, F, justAfterThePriorHour(), justBeforeTheHour(), F, F, 0, 1, F, F, F, T, nil}, "never ran, not valid schedule, R":      {f, F, errorSchedule, noDead, F, F, justAfterThePriorHour(), justBeforeTheHour(), F, F, 0, 1, F, F, F, T, nil}, "never ran, not time, A":                {A, F, onTheHour, noDead, F, F, justAfterThePriorHour(), justBeforeTheHour(), F, F, 0, 0, F, T, F, T, nil}, "never ran, not time, F":                {f, F, onTheHour, noDead, F, F, justAfterThePriorHour(), justBeforeTheHour(), F, F, 0, 0, F, T, F, T, nil}, "never ran, not time, R":                {R, F, onTheHour, noDead, F, F, justAfterThePriorHour(), justBeforeTheHour(), F, F, 0, 0, F, T, F, T, nil}, "never ran, is time, A":                 {A, F, onTheHour, noDead, F, F, justAfterThePriorHour(), *justAfterTheHour(), T, F, 1, 0, F, T, F, T, nil}, "never ran, is time, F":                 {f, F, onTheHour, noDead, F, F, justAfterThePriorHour(), *justAfterTheHour(), T, F, 1, 0, F, T, F, T, nil}, "never ran, is time, R":                 {R, F, onTheHour, noDead, F, F, justAfterThePriorHour(), *justAfterTheHour(), T, F, 1, 0, F, T, F, T, nil}, "never ran, is time, suspended":         {A, T, onTheHour, noDead, F, F, justAfterThePriorHour(), *justAfterTheHour(), F, F, 0, 0, F, F, F, T, nil}, "never ran, is time, past deadline":     {A, F, onTheHour, shortDead, F, F, justAfterThePriorHour(), justAfterTheHour().Add(time.Minute * time.Duration(shortDead+1)), F, F, 0, 0, F, T, F, T, nil}, "never ran, is time, not past deadline": {A, F, onTheHour, longDead, F, F, justAfterThePriorHour(), *justAfterTheHour(), T, F, 1, 0, F, T, F, T, nil}, "prev ran but done, not time, A":                {A, F, onTheHour, noDead, T, F, justAfterThePriorHour(), justBeforeTheHour(), F, F, 0, 0, F, T, F, T, nil}, "prev ran but done, not time, F":                {f, F, onTheHour, noDead, T, F, justAfterThePriorHour(), justBeforeTheHour(), F, F, 0, 0, F, T, F, T, nil}, "prev ran but done, not time, R":                {R, F, onTheHour, noDead, T, F, justAfterThePriorHour(), justBeforeTheHour(), F, F, 0, 0, F, T, F, T, nil}, "prev ran but done, is time, A":                 {A, F, onTheHour, noDead, T, F, justAfterThePriorHour(), *justAfterTheHour(), T, F, 1, 0, F, T, F, T, nil}, "prev ran but done, is time, F":                 {f, F, onTheHour, noDead, T, F, justAfterThePriorHour(), *justAfterTheHour(), T, F, 1, 0, F, T, F, T, nil}, "prev ran but done, is time, R":                 {R, F, onTheHour, noDead, T, F, justAfterThePriorHour(), *justAfterTheHour(), T, F, 1, 0, F, T, F, T, nil}, "prev ran but done, is time, suspended":         {A, T, onTheHour, noDead, T, F, justAfterThePriorHour(), *justAfterTheHour(), F, F, 0, 0, F, F, F, T, nil}, "prev ran but done, is time, past deadline":     {A, F, onTheHour, shortDead, T, F, justAfterThePriorHour(), *justAfterTheHour(), F, F, 0, 0, F, T, F, T, nil}, "prev ran but done, is time, not past deadline": {A, F, onTheHour, longDead, T, F, justAfterThePriorHour(), *justAfterTheHour(), T, F, 1, 0, F, T, F, T, nil}, "still active, not time, A":                {A, F, onTheHour, noDead, T, T, justAfterThePriorHour(), justBeforeTheHour(), F, F, 1, 0, F, T, F, T, nil}, "still active, not time, F":                {f, F, onTheHour, noDead, T, T, justAfterThePriorHour(), justBeforeTheHour(), F, F, 1, 0, F, T, F, T, nil}, "still active, not time, R":                {R, F, onTheHour, noDead, T, T, justAfterThePriorHour(), justBeforeTheHour(), F, F, 1, 0, F, T, F, T, nil}, "still active, is time, A":                 {A, F, onTheHour, noDead, T, T, justAfterThePriorHour(), *justAfterTheHour(), T, F, 2, 0, F, T, F, T, nil}, "still active, is time, F":                 {f, F, onTheHour, noDead, T, T, justAfterThePriorHour(), *justAfterTheHour(), F, F, 1, 0, F, T, F, T, nil}, "still active, is time, R":                 {R, F, onTheHour, noDead, T, T, justAfterThePriorHour(), *justAfterTheHour(), T, T, 1, 0, F, T, F, T, nil}, "still active, is time, suspended":         {A, T, onTheHour, noDead, T, T, justAfterThePriorHour(), *justAfterTheHour(), F, F, 1, 0, F, F, F, T, nil}, "still active, is time, past deadline":     {A, F, onTheHour, shortDead, T, T, justAfterThePriorHour(), *justAfterTheHour(), F, F, 1, 0, F, T, F, T, nil}, "still active, is time, not past deadline": {A, F, onTheHour, longDead, T, T, justAfterThePriorHour(), *justAfterTheHour(), T, F, 2, 0, F, T, F, T, nil}, </del> <ins> "never ran, not valid schedule, A": { concurrencyPolicy:          "Allow", schedule:                   errorSchedule, deadline:                   noDead, jobCreationTime:            justAfterThePriorHour(), now:                        justBeforeTheHour(), expectedWarnings:           1, jobPresentInCJActiveStatus: true, }, "never ran, not valid schedule, F": { concurrencyPolicy:          "Forbid", schedule:                   errorSchedule, deadline:                   noDead, jobCreationTime:            justAfterThePriorHour(), now:                        justBeforeTheHour(), expectedWarnings:           1, jobPresentInCJActiveStatus: true, }, "never ran, not valid schedule, R": { concurrencyPolicy:          "Forbid", schedule:                   errorSchedule, deadline:                   noDead, jobCreationTime:            justAfterThePriorHour(), now:                        justBeforeTheHour(), expectedWarnings:           1, jobPresentInCJActiveStatus: true, }, "never ran, not time, A": { concurrencyPolicy:          "Allow", schedule:                   onTheHour, deadline:                   noDead, jobCreationTime:            justAfterThePriorHour(), now:                        justBeforeTheHour(), expectRequeueAfter:         true, jobPresentInCJActiveStatus: true}, "never ran, not time, F": { concurrencyPolicy:          "Forbid", schedule:                   onTheHour, deadline:                   noDead, jobCreationTime:            justAfterThePriorHour(), now:                        justBeforeTheHour(), expectRequeueAfter:         true, jobPresentInCJActiveStatus: true, }, "never ran, not time, R": { concurrencyPolicy:          "Replace", schedule:                   onTheHour, deadline:                   noDead, jobCreationTime:            justAfterThePriorHour(), now:                        justBeforeTheHour(), expectRequeueAfter:         true, jobPresentInCJActiveStatus: true, }, "never ran, is time, A": { concurrencyPolicy:          "Allow", schedule:                   onTheHour, deadline:                   noDead, jobCreationTime:            justAfterThePriorHour(), now:                        *justAfterTheHour(), expectCreate:               true, expectActive:               1, expectRequeueAfter:         true, jobPresentInCJActiveStatus: true, }, "never ran, is time, F": { concurrencyPolicy:          "Forbid", schedule:                   onTheHour, deadline:                   noDead, jobCreationTime:            justAfterThePriorHour(), now:                        *justAfterTheHour(), expectCreate:               true, expectActive:               1, expectRequeueAfter:         true, jobPresentInCJActiveStatus: true, }, "never ran, is time, R": { concurrencyPolicy:          "Replace", schedule:                   onTheHour, deadline:                   noDead, jobCreationTime:            justAfterThePriorHour(), now:                        *justAfterTheHour(), expectCreate:               true, expectActive:               1, expectRequeueAfter:         true, jobPresentInCJActiveStatus: true, }, "never ran, is time, suspended": { concurrencyPolicy:          "Allow", suspend:                    true, schedule:                   onTheHour, deadline:                   noDead, jobCreationTime:            justAfterThePriorHour(), now:                        *justAfterTheHour(), jobPresentInCJActiveStatus: true, }, "never ran, is time, past deadline": { concurrencyPolicy:          "Allow", schedule:                   onTheHour, deadline:                   shortDead, jobCreationTime:            justAfterThePriorHour(), now:                        justAfterTheHour().Add(time.Minute * time.Duration(shortDead+1)), expectRequeueAfter:         true, jobPresentInCJActiveStatus: true, }, "never ran, is time, not past deadline": { concurrencyPolicy:          "Allow", schedule:                   onTheHour, deadline:                   longDead, jobCreationTime:            justAfterThePriorHour(), now:                        *justAfterTheHour(), expectCreate:               true, expectActive:               1, expectRequeueAfter:         true, jobPresentInCJActiveStatus: true, }, "prev ran but done, not time, A": { concurrencyPolicy:          "Allow", schedule:                   onTheHour, deadline:                   noDead, ranPreviously:              true, jobCreationTime:            justAfterThePriorHour(), now:                        justBeforeTheHour(), expectRequeueAfter:         true, jobPresentInCJActiveStatus: true, }, "prev ran but done, not time, F": { concurrencyPolicy:          "Forbid", schedule:                   onTheHour, deadline:                   noDead, ranPreviously:              true, jobCreationTime:            justAfterThePriorHour(), now:                        justBeforeTheHour(), expectRequeueAfter:         true, jobPresentInCJActiveStatus: true, }, "prev ran but done, not time, R": { concurrencyPolicy:          "Replace", schedule:                   onTheHour, deadline:                   noDead, ranPreviously:              true, jobCreationTime:            justAfterThePriorHour(), now:                        justBeforeTheHour(), expectRequeueAfter:         true, jobPresentInCJActiveStatus: true, }, "prev ran but done, is time, A": { concurrencyPolicy:          "Allow", schedule:                   onTheHour, deadline:                   noDead, ranPreviously:              true, jobCreationTime:            justAfterThePriorHour(), now:                        *justAfterTheHour(), expectCreate:               true, expectActive:               1, expectRequeueAfter:         true, jobPresentInCJActiveStatus: true, }, "prev ran but done, is time, F": { concurrencyPolicy:          "Forbid", schedule:                   onTheHour, deadline:                   noDead, ranPreviously:              true, jobCreationTime:            justAfterThePriorHour(), now:                        *justAfterTheHour(), expectCreate:               true, expectActive:               1, expectRequeueAfter:         true, jobPresentInCJActiveStatus: true, }, "prev ran but done, is time, R": { concurrencyPolicy:          "Replace", schedule:                   onTheHour, deadline:                   noDead, ranPreviously:              true, jobCreationTime:            justAfterThePriorHour(), now:                        *justAfterTheHour(), expectCreate:               true, expectActive:               1, expectRequeueAfter:         true, jobPresentInCJActiveStatus: true, }, "prev ran but done, is time, suspended": { concurrencyPolicy:          "Allow", suspend:                    true, schedule:                   onTheHour, deadline:                   noDead, ranPreviously:              true, jobCreationTime:            justAfterThePriorHour(), now:                        *justAfterTheHour(), jobPresentInCJActiveStatus: true, }, "prev ran but done, is time, past deadline": { concurrencyPolicy:          "Allow", schedule:                   onTheHour, deadline:                   shortDead, ranPreviously:              true, jobCreationTime:            justAfterThePriorHour(), now:                        *justAfterTheHour(), expectRequeueAfter:         true, jobPresentInCJActiveStatus: true, }, "prev ran but done, is time, not past deadline": { concurrencyPolicy:          "Allow", schedule:                   onTheHour, deadline:                   longDead, ranPreviously:              true, jobCreationTime:            justAfterThePriorHour(), now:                        *justAfterTheHour(), expectCreate:               true, expectActive:               1, expectRequeueAfter:         true, jobPresentInCJActiveStatus: true, }, "still active, not time, A": { concurrencyPolicy:          "Allow", schedule:                   onTheHour, deadline:                   noDead, ranPreviously:              true, stillActive:                true, jobCreationTime:            justAfterThePriorHour(), now:                        justBeforeTheHour(), expectActive:               1, expectRequeueAfter:         true, jobPresentInCJActiveStatus: true, }, "still active, not time, F": { concurrencyPolicy:          "Forbid", schedule:                   onTheHour, deadline:                   noDead, ranPreviously:              true, stillActive:                true, jobCreationTime:            justAfterThePriorHour(), now:                        justBeforeTheHour(), expectActive:               1, expectRequeueAfter:         true, jobPresentInCJActiveStatus: true, }, "still active, not time, R": { concurrencyPolicy:          "Replace", schedule:                   onTheHour, deadline:                   noDead, ranPreviously:              true, stillActive:                true, jobCreationTime:            justAfterThePriorHour(), now:                        justBeforeTheHour(), expectActive:               1, expectRequeueAfter:         true, jobPresentInCJActiveStatus: true, }, "still active, is time, A": { concurrencyPolicy:          "Allow", schedule:                   onTheHour, deadline:                   noDead, ranPreviously:              true, stillActive:                true, jobCreationTime:            justAfterThePriorHour(), now:                        *justAfterTheHour(), expectCreate:               true, expectActive:               2, expectRequeueAfter:         true, jobPresentInCJActiveStatus: true, }, "still active, is time, F": { concurrencyPolicy:          "Forbid", schedule:                   onTheHour, deadline:                   noDead, ranPreviously:              true, stillActive:                true, jobCreationTime:            justAfterThePriorHour(), now:                        *justAfterTheHour(), expectActive:               1, expectRequeueAfter:         true, jobPresentInCJActiveStatus: true, }, "still active, is time, R": { concurrencyPolicy:          "Replace", schedule:                   onTheHour, deadline:                   noDead, ranPreviously:              true, stillActive:                true, jobCreationTime:            justAfterThePriorHour(), now:                        *justAfterTheHour(), expectCreate:               true, expectDelete:               true, expectActive:               1, expectRequeueAfter:         true, jobPresentInCJActiveStatus: true, }, "still active, is time, suspended": { concurrencyPolicy:          "Allow", suspend:                    true, schedule:                   onTheHour, deadline:                   noDead, ranPreviously:              true, stillActive:                true, jobCreationTime:            justAfterThePriorHour(), now:                        *justAfterTheHour(), expectActive:               1, jobPresentInCJActiveStatus: true, }, "still active, is time, past deadline": { concurrencyPolicy:          "Allow", schedule:                   onTheHour, deadline:                   shortDead, ranPreviously:              true, stillActive:                true, jobCreationTime:            justAfterThePriorHour(), now:                        *justAfterTheHour(), expectActive:               1, expectRequeueAfter:         true, jobPresentInCJActiveStatus: true, }, "still active, is time, not past deadline": { concurrencyPolicy:          "Allow", schedule:                   onTheHour, deadline:                   longDead, ranPreviously:              true, stillActive:                true, jobCreationTime:            justAfterThePriorHour(), now:                        *justAfterTheHour(), expectCreate:               true, expectActive:               2, expectRequeueAfter:         true, jobPresentInCJActiveStatus: true, }, </ins> // Controller should fail to schedule these, as there are too many missed starting times // and either no deadline or a too long deadline. <del> "prev ran but done, long overdue, not past deadline, A": {A, F, onTheHour, longDead, T, F, justAfterThePriorHour(), weekAfterTheHour(), T, F, 1, 1, F, T, F, T, nil}, "prev ran but done, long overdue, not past deadline, R": {R, F, onTheHour, longDead, T, F, justAfterThePriorHour(), weekAfterTheHour(), T, F, 1, 1, F, T, F, T, nil}, "prev ran but done, long overdue, not past deadline, F": {f, F, onTheHour, longDead, T, F, justAfterThePriorHour(), weekAfterTheHour(), T, F, 1, 1, F, T, F, T, nil}, "prev ran but done, long overdue, no deadline, A":       {A, F, onTheHour, noDead, T, F, justAfterThePriorHour(), weekAfterTheHour(), T, F, 1, 1, F, T, F, T, nil}, "prev ran but done, long overdue, no deadline, R":       {R, F, onTheHour, noDead, T, F, justAfterThePriorHour(), weekAfterTheHour(), T, F, 1, 1, F, T, F, T, nil}, "prev ran but done, long overdue, no deadline, F":       {f, F, onTheHour, noDead, T, F, justAfterThePriorHour(), weekAfterTheHour(), T, F, 1, 1, F, T, F, T, nil}, </del> <ins> "prev ran but done, long overdue, not past deadline, A": { concurrencyPolicy:          "Allow", schedule:                   onTheHour, deadline:                   longDead, ranPreviously:              true, jobCreationTime:            justAfterThePriorHour(), now:                        weekAfterTheHour(), expectCreate:               true, expectActive:               1, expectedWarnings:           1, expectRequeueAfter:         true, jobPresentInCJActiveStatus: true, }, "prev ran but done, long overdue, not past deadline, R": { concurrencyPolicy:          "Replace", schedule:                   onTheHour, deadline:                   longDead, ranPreviously:              true, jobCreationTime:            justAfterThePriorHour(), now:                        weekAfterTheHour(), expectCreate:               true, expectActive:               1, expectedWarnings:           1, expectRequeueAfter:         true, jobPresentInCJActiveStatus: true, }, "prev ran but done, long overdue, not past deadline, F": { concurrencyPolicy:          "Forbid", schedule:                   onTheHour, deadline:                   longDead, ranPreviously:              true, jobCreationTime:            justAfterThePriorHour(), now:                        weekAfterTheHour(), expectCreate:               true, expectActive:               1, expectedWarnings:           1, expectRequeueAfter:         true, jobPresentInCJActiveStatus: true, }, "prev ran but done, long overdue, no deadline, A": { concurrencyPolicy:          "Allow", schedule:                   onTheHour, deadline:                   noDead, ranPreviously:              true, jobCreationTime:            justAfterThePriorHour(), now:                        weekAfterTheHour(), expectCreate:               true, expectActive:               1, expectedWarnings:           1, expectRequeueAfter:         true, jobPresentInCJActiveStatus: true, }, "prev ran but done, long overdue, no deadline, R": { concurrencyPolicy:          "Replace", schedule:                   onTheHour, deadline:                   noDead, ranPreviously:              true, jobCreationTime:            justAfterThePriorHour(), now:                        weekAfterTheHour(), expectCreate:               true, expectActive:               1, expectedWarnings:           1, expectRequeueAfter:         true, jobPresentInCJActiveStatus: true, }, "prev ran but done, long overdue, no deadline, F": { concurrencyPolicy:          "Forbid", schedule:                   onTheHour, deadline:                   noDead, ranPreviously:              true, jobCreationTime:            justAfterThePriorHour(), now:                        weekAfterTheHour(), expectCreate:               true, expectActive:               1, expectedWarnings:           1, expectRequeueAfter:         true, jobPresentInCJActiveStatus: true, }, </ins> <del> "prev ran but done, long overdue, past medium deadline, A": {A, F, onTheHour, mediumDead, T, F, justAfterThePriorHour(), weekAfterTheHour(), T, F, 1, 0, F, T, F, T, nil}, "prev ran but done, long overdue, past short deadline, A":  {A, F, onTheHour, shortDead, T, F, justAfterThePriorHour(), weekAfterTheHour(), T, F, 1, 0, F, T, F, T, nil}, </del> <ins> "prev ran but done, long overdue, past medium deadline, A": { concurrencyPolicy:          "Allow", schedule:                   onTheHour, deadline:                   mediumDead, ranPreviously:              true, jobCreationTime:            justAfterThePriorHour(), now:                        weekAfterTheHour(), expectCreate:               true, expectActive:               1, expectRequeueAfter:         true, jobPresentInCJActiveStatus: true, }, "prev ran but done, long overdue, past short deadline, A": { concurrencyPolicy:          "Allow", schedule:                   onTheHour, deadline:                   shortDead, ranPreviously:              true, jobCreationTime:            justAfterThePriorHour(), now:                        weekAfterTheHour(), expectCreate:               true, expectActive:               1, expectRequeueAfter:         true, jobPresentInCJActiveStatus: true, }, </ins> <del> "prev ran but done, long overdue, past medium deadline, R": {R, F, onTheHour, mediumDead, T, F, justAfterThePriorHour(), weekAfterTheHour(), T, F, 1, 0, F, T, F, T, nil}, "prev ran but done, long overdue, past short deadline, R":  {R, F, onTheHour, shortDead, T, F, justAfterThePriorHour(), weekAfterTheHour(), T, F, 1, 0, F, T, F, T, nil}, </del> <ins> "prev ran but done, long overdue, past medium deadline, R": { concurrencyPolicy:          "Replace", schedule:                   onTheHour, deadline:                   mediumDead, ranPreviously:              true, jobCreationTime:            justAfterThePriorHour(), now:                        weekAfterTheHour(), expectCreate:               true, expectActive:               1, expectRequeueAfter:         true, jobPresentInCJActiveStatus: true, }, "prev ran but done, long overdue, past short deadline, R": { concurrencyPolicy:          "Replace", schedule:                   onTheHour, deadline:                   shortDead, ranPreviously:              true, jobCreationTime:            justAfterThePriorHour(), now:                        weekAfterTheHour(), expectCreate:               true, expectActive:               1, expectRequeueAfter:         true, jobPresentInCJActiveStatus: true, }, </ins> <del> "prev ran but done, long overdue, past medium deadline, F": {f, F, onTheHour, mediumDead, T, F, justAfterThePriorHour(), weekAfterTheHour(), T, F, 1, 0, F, T, F, T, nil}, "prev ran but done, long overdue, past short deadline, F":  {f, F, onTheHour, shortDead, T, F, justAfterThePriorHour(), weekAfterTheHour(), T, F, 1, 0, F, T, F, T, nil}, </del> <ins> "prev ran but done, long overdue, past medium deadline, F": { concurrencyPolicy:          "Forbid", schedule:                   onTheHour, deadline:                   mediumDead, ranPreviously:              true, jobCreationTime:            justAfterThePriorHour(), now:                        weekAfterTheHour(), expectCreate:               true, expectActive:               1, expectRequeueAfter:         true, jobPresentInCJActiveStatus: true, }, "prev ran but done, long overdue, past short deadline, F": { concurrencyPolicy:          "Forbid", schedule:                   onTheHour, deadline:                   shortDead, ranPreviously:              true, jobCreationTime:            justAfterThePriorHour(), now:                        weekAfterTheHour(), expectCreate:               true, expectActive:               1, expectRequeueAfter:         true, jobPresentInCJActiveStatus: true, }, </ins> // Tests for time skews // the controller sees job is created, takes no actions <del> "this ran but done, time drifted back, F": {f, F, onTheHour, noDead, T, F, *justAfterTheHour(), justBeforeTheHour(), F, F, 0, 0, F, T, F, F, errors.NewAlreadyExists(schema.GroupResource{Resource: "jobs", Group: "batch"}, "")}, </del> <ins> "this ran but done, time drifted back, F": { concurrencyPolicy:  "Forbid", schedule:           onTheHour, deadline:           noDead, ranPreviously:      true, jobCreationTime:    *justAfterTheHour(), now:                justBeforeTheHour(), expectRequeueAfter: true, jobCreateError:     errors.NewAlreadyExists(schema.GroupResource{Resource: "jobs", Group: "batch"}, "")}, </ins> // Tests for slow job lister <del> "this started but went missing, not past deadline, A": {A, F, onTheHour, longDead, T, T, topOfTheHour().Add(time.Millisecond * 100), justAfterTheHour().Add(time.Millisecond * 100), F, F, 1, 0, F, T, T, T, nil}, "this started but went missing, not past deadline, f": {f, F, onTheHour, longDead, T, T, topOfTheHour().Add(time.Millisecond * 100), justAfterTheHour().Add(time.Millisecond * 100), F, F, 1, 0, F, T, T, T, nil}, "this started but went missing, not past deadline, R": {R, F, onTheHour, longDead, T, T, topOfTheHour().Add(time.Millisecond * 100), justAfterTheHour().Add(time.Millisecond * 100), F, F, 1, 0, F, T, T, T, nil}, </del> <ins> "this started but went missing, not past deadline, A": { concurrencyPolicy:          "Allow", schedule:                   onTheHour, deadline:                   longDead, ranPreviously:              true, stillActive:                true, jobCreationTime:            topOfTheHour().Add(time.Millisecond * 100), now:                        justAfterTheHour().Add(time.Millisecond * 100), expectActive:               1, expectRequeueAfter:         true, jobStillNotFoundInLister:   true, jobPresentInCJActiveStatus: true, }, "this started but went missing, not past deadline, f": { concurrencyPolicy:          "Forbid", schedule:                   onTheHour, deadline:                   longDead, ranPreviously:              true, stillActive:                true, jobCreationTime:            topOfTheHour().Add(time.Millisecond * 100), now:                        justAfterTheHour().Add(time.Millisecond * 100), expectActive:               1, expectRequeueAfter:         true, jobStillNotFoundInLister:   true, jobPresentInCJActiveStatus: true, }, "this started but went missing, not past deadline, R": { concurrencyPolicy:          "Replace", schedule:                   onTheHour, deadline:                   longDead, ranPreviously:              true, stillActive:                true, jobCreationTime:            topOfTheHour().Add(time.Millisecond * 100), now:                        justAfterTheHour().Add(time.Millisecond * 100), expectActive:               1, expectRequeueAfter:         true, jobStillNotFoundInLister:   true, jobPresentInCJActiveStatus: true, }, </ins> // Tests for slow cronjob list <del> "this started but is not present in cronjob active list, not past deadline, A": {A, F, onTheHour, longDead, T, T, topOfTheHour().Add(time.Millisecond * 100), justAfterTheHour().Add(time.Millisecond * 100), F, F, 1, 0, F, T, F, F, nil}, "this started but is not present in cronjob active list, not past deadline, f": {f, F, onTheHour, longDead, T, T, topOfTheHour().Add(time.Millisecond * 100), justAfterTheHour().Add(time.Millisecond * 100), F, F, 1, 0, F, T, F, F, nil}, "this started but is not present in cronjob active list, not past deadline, R": {R, F, onTheHour, longDead, T, T, topOfTheHour().Add(time.Millisecond * 100), justAfterTheHour().Add(time.Millisecond * 100), F, F, 1, 0, F, T, F, F, nil}, </del> <ins> "this started but is not present in cronjob active list, not past deadline, A": { concurrencyPolicy:  "Allow", schedule:           onTheHour, deadline:           longDead, ranPreviously:      true, stillActive:        true, jobCreationTime:    topOfTheHour().Add(time.Millisecond * 100), now:                justAfterTheHour().Add(time.Millisecond * 100), expectActive:       1, expectRequeueAfter: true, }, "this started but is not present in cronjob active list, not past deadline, f": { concurrencyPolicy:  "Forbid", schedule:           onTheHour, deadline:           longDead, ranPreviously:      true, stillActive:        true, jobCreationTime:    topOfTheHour().Add(time.Millisecond * 100), now:                justAfterTheHour().Add(time.Millisecond * 100), expectActive:       1, expectRequeueAfter: true, }, "this started but is not present in cronjob active list, not past deadline, R": { concurrencyPolicy:  "Replace", schedule:           onTheHour, deadline:           longDead, ranPreviously:      true, stillActive:        true, jobCreationTime:    topOfTheHour().Add(time.Millisecond * 100), now:                justAfterTheHour().Add(time.Millisecond * 100), expectActive:       1, expectRequeueAfter: true, }, </ins> } for name, tc := range testCases { name := name

in TestControllerV2SyncCronJob, there are many listing the properties that don't matter (like nil, false). and many constants F and T, which is unnecessary. As per 's request, We shouldn't be listing the properties that don't matter (like nil, false). And get rid of constants F and T check No response <details</details<detailsNone </details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign I would like to work on the optimization
It's not an optimization. It's just about readability.
/remove-kind bug /kind cleanup
/sig apps


Question: return } <ins> // Only node names that exist in previous version of the EndpointSlice // deprecatedTopology fields may be retained in new version of the // EndpointSlice. prevNodeNames := getDeprecatedTopologyNodeNames(oldEPS) </ins> for i := range newEPS.Endpoints { ep := &newEPS.Endpoints[i] <del> //Silently clear out DeprecatedTopology </del> <ins> newTopologyNodeName, ok := ep.DeprecatedTopology[corev1.LabelHostname] if ep.NodeName == nil && ok && prevNodeNames.Has(newTopologyNodeName) && len(apivalidation.ValidateNodeName(newTopologyNodeName, false)) == 0 { // Copy the label previously used to store the node name into the nodeName field, // in order to make partial updates preserve previous node info ep.NodeName = &newTopologyNodeName } // Drop writes to this field via the v1 API as documented </ins> ep.DeprecatedTopology = nil } } } <ins> // getDeprecatedTopologyNodeNames returns a set of node names present in // deprecatedTopology fields within the provided EndpointSlice. func getDeprecatedTopologyNodeNames(eps *discovery.EndpointSlice) sets.String { if eps == nil { return nil } var names sets.String for _, ep := range eps.Endpoints { if nodeName, ok := ep.DeprecatedTopology[corev1.LabelHostname]; ok && len(nodeName) > 0 { if names == nil { names = sets.NewString() } names.Insert(nodeName) } } return names } </ins>

On the first update of an EndpointSlice using the v1 API, we drop deprecatedTopology with EndpointSlice strategy. Although we only do this if at least 1 endpoint has been modified, there is no guarantee that node name has been populated for all endpoints within the EndpointSlice. This is most noticeable if the first update using the v1 API is only adding endpoint(s). In that case the EndpointSlice controller does not update EndpointSlices and instead just adds the new endpoint(s). That will cause node names to be dropped from the deprecated topology field without having the newer node name field populated. Node name to be retained in deprecated topology until the new field was populated. an EndpointSlice with the v1beta1 API. an endpoint in that EndpointSlice: No response This is most notable after upgrading a 1.20 cluster to 1.21, but can technically occur in any 1.21 or newer cluster. Any Any Any N/A N/A
I'm working on a fix for this, hoping to have it out later today. /triage accepted /sig network /cc
I think this is better characterized as a controller bug, but is the safest mitigation I can think of in the short term, since we don't want to rewrite 100% of endpointslices on startup. Can you open a separate issue to work through the controller reconciliation so that v1 clients can depend on the nodeName field more reliably?


Question: } // Combine the provided plugins with the ones from component config. <del> // TODO(#86789): address disabled plugins. var plugins schedulerapi.Plugins plugins.Append(provider.FrameworkPlugins) plugins.Append(c.plugins) c.plugins = &plugins </del> <ins> var defaultPlugins schedulerapi.Plugins defaultPlugins.Append(provider.FrameworkPlugins) defaultPlugins.Apply(c.plugins) c.plugins = &defaultPlugins </ins> var pluginConfig []schedulerapi.PluginConfig pluginConfig = append(pluginConfig, provider.FrameworkPluginConfig...) pluginConfig = append(pluginConfig, c.pluginConfig...)

What would you like to be : Actually disable scheduler plugins according to the disabled list in the scheduler's component config. The semantics are defined here: Part of /sig scheduling /priority important-soon /assign
The following semantics seems vague, and I think we should reach consensus on some cases when disabled plugins are involved. Suppose we have two default plugins, and , the runs before , here are the cases we could meet with the scheduling framework, plugins which run after all the default plugins plugins which should run ahead of the plugins plugin which should run ahead of all the plugins -enable default or custom plugin should return errors
Do you have any other cases not mentioned here?
Those cases sound right, I don't have others.
Is this actually what we want? I don't think so. The comment needs rewording.
I plan to revisit the comments after


Question: nodeLabelRole  = "kubernetes.io/role" nicFailedState = "Failed" <del> storageAccountNameMaxLength = 24 </del> <ins> storageAccountNameMaxLength   = 24 frontendIPConfigNameMaxLength = 80 loadBalancerRuleNameMaxLength = 80 </ins> ) var errNotInVMSet = errors.New("vm is not in the vmset")

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When the specified internal LB uses a subnet with longer name, the LB configuration will fail with following error: What you expected to happen: When constructing the frontend IP configuration name and the load balancing rules name, the subnet name should be truncated if too long. How to reproduce it (as minimally and precisely as possible): Use a subnet that has long name (70 charactors) and create ILB service. Anything else we need to know?: Environment: Kubernetes version (use ): 1.15 Cloud provider or hardware configuration: azure OS (e.g: ): Ubuntu 16.04 Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig cloud-provider /area provider/azure /assign


Question: t.Errorf("%s: on name collision wanted new name %s got %s", test.name, expectedName, created.Name) } <del> // Second name collision should have incremented collisionCount to 2 </del> <ins> // Second name collision will be caused by an identical revision, so no need to do anything </ins> _, err = history.CreateControllerRevision(test.parent, test.revision, &collisionCount) if err != nil { t.Errorf("%s: %s", test.name, err) } <del> if collisionCount != 2 { </del> <ins> if collisionCount != 1 { </ins> t.Errorf("%s: on second name collision wanted collisionCount 1 got %d", test.name, collisionCount) } }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT: What happened: When a statefulset object is created, a lot of controllerrevisions are created at the same time. the controllerrevisions have differenct hash value but the same Data.Raw. How to fix it: two controllerrevisions are seen as the same , when [ControllerRevisionHashLabel] , Controllerrevision.Data.Raw , Controllerrevision.Data.Object are strictly the same // this line shoud be to fix the bug like below: Anything else we need to know?: Environment: Kubernetes version (use ): 1.7.8 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
I would help fix it
/kind bug /sig apps
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
StatefulSet uses the following code to construct its history object (controllerrevision): However, collisionCount shouldn't be blindly increased when there's name collision. Because controller uses a cache store, the name collision could be due to a controllerrevision created but not yet observed by the controller. The right way to handle it is to first check if the controllerrevision with the same name has the same content as the one we're going to create, like how DaemonSet controller handles it:


Question: <ins> The MIT License (MIT) Copyright (c) 2015-2016 Manfred Touron Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. </ins>

we have one in k/k and one in k/utils, both need to be in LICENSES /assign
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/area code-organization
I think we should include anything matching the LICENSE/COPYRIGHT/COPYING patthers under or subdirs in k/k or dependencies... a quick sweep showed these:
/sig release ?


Question: noIncreaseMetrics = sets.NewString() // Always evaluate all SELinux metrics to check that the other metrics are not unexpectedly increased. for name := range allMetricNames { <del> if expectedIncreaseNames.Has(name) { </del> <ins> expectIncrease := false // allMetricNames can include "{volume_plugin="XXX"}", while expectedIncreaseNames does not. // Compare them properly. Value of volume_plugin="XXX" was already checked in grabMetrics(), // we can ignore it here. for expectedIncreaseMetricName := range expectedIncreaseNames { if strings.HasPrefix(name, expectedIncreaseMetricName) { expectIncrease = true break } } if expectIncrease { </ins> if metrics[name] <= initialValues[name] { noIncreaseMetrics.Insert(name) } } else { <ins> // Expect the metric to be stable </ins> if initialValues[name] != metrics[name] { return false, fmt.Errorf("metric %s unexpectedly increased to %v", name, metrics[name]) }

[It] [sig-storage] CSI Mock selinux on mount metrics SELinuxMount metrics [LinuxOnly][Feature:SELinux][Feature:SELinuxMountReadWriteOncePod][Serial] warning is not bumped on two Pods with the same context on RWO volume [sig-storage] [It] [sig-storage] CSI Mock selinux on mount metrics SELinuxMount metrics [LinuxOnly][Feature:SELinux][Feature:SELinuxMountReadWriteOncePod][Serial] warning is bumped on two Pods with a different context on RWO volume [sig-storage] Sample failure: October 25, 2023 It fails since The PR a label to some SELinux metric and the tests do not count with that. No response /sig storage
/assign /triage accepted


Question: claimSize := actualStateOfWorld.GetClaimSize(volumeToMount.VolumeName) // only fetch claimSize if it was not set previously <del> if volumeToMount.VolumeSpec.PersistentVolume != nil && claimSize == nil { </del> <ins> if volumeToMount.VolumeSpec.PersistentVolume != nil && claimSize == nil && !volumeToMount.VolumeSpec.InlineVolumeSpecForCSIMigration { </ins> pv := volumeToMount.VolumeSpec.PersistentVolume pvc, err := og.kubeClient.CoreV1().PersistentVolumeClaims(pv.Spec.ClaimRef.Namespace).Get(context.TODO(), pv.Spec.ClaimRef.Name, metav1.GetOptions{}) if err != nil {

The following test failures started happening on 30th March: Kubernetes e2e suite.[sig-storage] In-tree Volumes [Driver: aws] [Testpattern: Inline-volume (default fs)] volumes should allow exec of files on the volume Kubernetes e2e suite.[sig-storage] In-tree Volumes [Driver: aws] [Testpattern: Inline-volume (default fs)] volumes should store data Kubernetes e2e suite.[sig-storage] In-tree Volumes [Driver: aws] [Testpattern: Inline-volume (ext4)] volumes should allow exec of files on the volume Kubernetes e2e suite.[sig-storage] In-tree Volumes [Driver: aws] [Testpattern: Inline-volume (ext4)] volumes should store data See This seems related to the changes in kubelet fails with the following: Tests passing See the e2e logs above No response <detailsLatest CI build </details<detailsAWS </details<details</details<details</details<details</details<detailsEBS CSI driver 1.5.0 </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc
/sig storage
/assign
not having debugging kops builds for awhile, are kubelet logs attached in prow? I don't see them...
Spoke with offline. It looks like we don't have kubelet logs captured in these tests. I need complete kubelet logs to make sense of these errors because testgrid shows inline volume tests as failing and code in question should not be called or inline volumes (because they can not be expanded). If that code fails, it would have been failing before too. I will wait for logs from and in the meanwhile try and see if I can reproduce these failures in single node AWS cluster setup that I can build manually.
thanks
Looks like this is happening for inline volumes if CSI migration is enabled for them. We construct a "fake" object on the fly for migration enabled plugins even though volume is inline, which is causing this condition to fail. It should be simple fix.
opened to fix the problem. The code change is just one line but has tests.


Question: import ( "k8s.io/kubernetes/pkg/api/unversioned" <ins> "k8s.io/kubernetes/pkg/api/v1" </ins> "k8s.io/kubernetes/pkg/runtime" versionedwatch "k8s.io/kubernetes/pkg/watch/versioned" )

It seems to me like all of the garbage collection integration tests produce a ton of log messages of the form: