Question: TimeoutSeconds: &timeoutSeconds, } <del> r.metrics.numberOfWatches.Inc() </del> w, err := r.listerWatcher.Watch(options) if err != nil { switch err {

What happened: After upgrading to kubernetes 1.12.5 we observe failing nodes, that are caused by kubelet eating all over the memory after some time. ! I use image to run kubelet on 102 clusters and since a week we see some nodes leaking memory, caused by kubelet. I investigated some of these kubelets with strace and pprof. With 3s of running strace I saw = 50 openat() calls to the same file from the same threadid (pid) from kubelet: If I do pprof kubelet it shows clientgo metrics and compress is taking over most of the compute time. Memory profile png: ! The reflector metrics seem to be the problem: What you expected to happen: I expect that kubelet does not need so much memory How to reproduce it (as minimally and precisely as possible): I don't know Anything else we need to know?: One of the affected clusters has only 120 Pods and 3 Pods are in CrashLoopBackOff state, one pod since 6 days on an affected node which was investigated. Environment: Kubernetes version (use ): v1.12.5 Cloud provider or hardware configuration: AWS OS (e.g. from /etc/os-release):Kernel (e.g. ): Linux ip-172-31-10- 4.14.63-coreos SMP Wed Aug 15 22:26:16 UTC 2018 x86_64 Intel(R) Xeon(R) CPU E5-2686 v4 @ 2.30GHz GenuineIntel GNU/Linux Install tools: custom
/sig node
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I think it might be a side effect of , but I couldn't find any numbers in there. The comments mention an offline discussion, was this brought up?
What happens (I only investigated it very briefly) is that kubelet creates a bunch of reflectors for every pod — secret mounts, configmap mounts, token mounts etc. Each reflector registers a number of histograms, gauges, etc. and even when the reflector is stopped and removed the metrics are never removed.

After a few days on a busy cluster you get millions of metrics bloating up kubelet and everything else using the reflectors.
This was broken since v1.12.0, so probably unrelated to .
correct that PR is only in v1.14.0-alpha.2 v1.14.0-alpha.1 and master
For everyone that finds this issue and needs a patch to disable the reflector metrics:
Do you have same metrics after disabling ReflectorMetricsProvider?
no, it drops the reflector metrics
is this indirectly caused by switching to watch-based managers?
Yeah - switching to watch resulted in more extensive using of a reflector. It seems that those metrics are directly coming from reflector (I wasn't aware of those). So it's highly probably that this change cause it. I think that we don't really need those metrics, so if we could switch them off in Kubelets, that should solve the problem.
is sent out for review to fix that. We should probably cherrypick it to previous releases if we agree that's the right way to solve it.
Actually, I realized that I don't fully understand the problem. It should be a problem in default Kubernetes (what we build from this repo), because those metrics are not switched on for kubelet: metrics factory is defaulted to noop the prometheus metrics need to be explicitly linked and initialized (as they are in apiserver e.g.: )
Wouldn't it be better to rewrite the reflector metrics so they're aggregated in a better way rather than relying on people not accidentally enabling them? Or at least put a huge warning on top saying "this will leak memory like crazy" just so the same situation doesn't repeat in other code using client-go?
- that's a separate issue whether the metrics are reasonable or not (and node sig-node related). One thing that I would like to confirm: do you customize k8s yourself? I would like to ensure that this issue you is in your setup and not in upstream k8s.
Our production configuration is this one: Kubelet config: We run with an image called , but it's just a mirror of . So upstream Kubernetes.
I try to figure out an issue why these metrics are even there and I found issues from the past showing leaks already in older versions: Sounds for, that we need a postmortem to make this not happening again.

I don't find any issue in kubernetes nor in client-go, that could reasonable explain why these were introduced in the first place. I guess I just did not found it, and I hope someone can highlight where the decision came from.
I am running Kubernetes v1.12.3 and I don't see reflector metrics being used by kubelet ( shows nothing). FWIW, the cluster is created using kops. Any idea, how are reflector metrics enabled?
and myself will also look into this as well and see what we can find.
Thanks
The issue only exists if hyperkube is used when running kubelet, as hyperkube has imported the package for apiserver and controller-manager. If kubelet binary is used, then there're no reflector metrics (this is same with your findings). Since the metrics is not exposed by kubelet cmd, what do you think of disabling it for kubelet subcommand in hyperkube? I can't reproduce the issue (with hyperkube used) on my own cluster: ! Since both apiserver and controller-manager have also imported the package, have you also observed same issues for them?
+1
Yes, we observe the same thing for apiserver and controller-manager as well.
I would also suggest not using imports for side effects because this usually creates hard to debug and non-obvious problems like this.
So you mean disable the reflector metrics totally?
No, I suggest changing the code so that the end users (kubelet, apiserver, etc) would have to explicitly enable the metrics by calling a function instead of having this happen as a side effect of an import statement.
LGTM. WDYT? We should probably do this for both kubelet and control plane services (apiserver and controller-manager).
I'm fine with that.
+1
+1
Why wouldn't we want to just actually disable this metric wholesale? Enabling this flag would basically be the same thing as saying enable a memory leak, no?
/assign
Han points out that, since the metric has a random suffix each time, it's not very useful for monitoring anyway.
/sig api-machinery


Question: } r := &Reflector{ <del> name:                   "test-reflector", listerWatcher:          lw, store:                  NewFIFO(MetaNamespaceKeyFunc), initConnBackoffManager: bm, clock:                  clock, watchErrorHandler:      WatchErrorHandler(DefaultWatchErrorHandler), </del> <ins> name:              "test-reflector", listerWatcher:     lw, store:             NewFIFO(MetaNamespaceKeyFunc), backoffManager:    bm, clock:             clock, watchErrorHandler: WatchErrorHandler(DefaultWatchErrorHandler), </ins> } stopCh := make(chan struct{})

Currently, reflector uses two backoff managers: One is used for initial list call and other for watch call. Imagine following scenaraion when k8s apiserver is under very heavy load: the beginning, reflector spams k8s apiserver (with exponential backoff) with calls until one of them succeeds (due to APF throtling and receiving ) tries to run request and uses separate backoffManager - so again spams k8s apiserver with requests Watch does not succeed during Etcd compaction window (2.5 - 5 minutes usually) at some point it will receive and go back to point 1, spamming k8s apiserver again with List calls (because first backoffManager for List got reseted after 2 minutes of retries for Watch requests) I propose we combine these two backoff managers into one. If we combine these two backoffManagers into one, it will work like this: At the beginning, reflector spams k8s apiserver (still with exponential backoff) with calls until one of them succeeds Tries to make first watch call instantly (as there is no backoff on this path) If it fails, it will makre retries every (30, 60)s If watch request fails all the times during Etcd compaction window it will go back to List call, but already being throttled with (30, 60)s window. /sig scalability
/cc
/assign
I think the proposed solution makes sense to me. Probably before we start, we should to understand why we ended up with two back off managers by looking into PRs touching them. Maybe the code authors had some specific case in mind.
for thoughts too I agree it makes sense, but I don't know the historical context here.
/sig api-machinery
/triage accepted


Question: counts: map[string]int64{ "events.foo.bar": 799, }, <del> seatsExpected: 5, </del> <ins> seatsExpected: 8, }, { name:       "request verb is list, resource version match is Exact", requestURI: "http://server/apis/foo.bar/v1/events?resourceVersion=foo&resourceVersionMatch=Exact&limit=399", requestInfo: &apirequest.RequestInfo{ Verb:     "list", APIGroup: "foo.bar", Resource: "events", }, counts: map[string]int64{ "events.foo.bar": 699, }, seatsExpected: 8, </ins> }, { name:       "request verb is list, resource version match is NotOlderThan, limit not specified",

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --I noticed that does not make sense to me. Looking at how LIST requests are served (which is NOT easy, there is so much dependency injection), I concluded that the worst case number of objects touched in the apiserver equals the pagination limit only if all of the following are true. The LIST request is served from etcd rather than the watch cache. The pagination FeatureGate is enabled. The request has a specified limit and it is no greater than the number of objects in etcd (restricted by namespace if relevant). The request does not have a label selector or field selector. I expected the list work estimator to use the pagination limit only under the conditions listed above. Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
I commented on the PR.
Thank you both Mike and Wojciech! /triage accepted


Question: } } <del> var podsNeededForSaturation int milliCpuPerPod := nodeMaxAllocatable / maxNumberOfPods if milliCpuPerPod < minPodCPURequest { milliCpuPerPod = minPodCPURequest } framework.Logf("Using pod capacity: %vm", milliCpuPerPod) for name, leftAllocatable := range nodeToAllocatableMap { framework.Logf("Node: %v has cpu allocatable: %vm", name, leftAllocatable) podsNeededForSaturation += (int)(leftAllocatable / milliCpuPerPod) } By(fmt.Sprintf("Starting additional %v Pods to fully saturate the cluster CPU and trying to start another one", podsNeededForSaturation)) // As the pods are distributed randomly among nodes, // it can easily happen that all nodes are saturated // and there is no need to create additional pods. // StartPods requires at least one pod to replicate. if podsNeededForSaturation > 0 { framework.ExpectNoError(testutils.StartPods(cs, podsNeededForSaturation, ns, "overcommit", *initPausePod(f, pausePodConfig{ Name:   "", Labels: map[string]string{"name": ""}, Resources: &v1.ResourceRequirements{ Limits: v1.ResourceList{ v1.ResourceCPU: *resource.NewMilliQuantity(milliCpuPerPod, "DecimalSI"), }, Requests: v1.ResourceList{ v1.ResourceCPU: *resource.NewMilliQuantity(milliCpuPerPod, "DecimalSI"), </del> <ins> By("Starting Pods to consume most of the cluster CPU.") // Create one pod per node that requires 70% of the node remaining CPU. fillerPods := []*v1.Pod{} for nodeName, cpu := range nodeToAllocatableMap { requestedCPU := cpu * 7 / 10 fillerPods = append(fillerPods, createPausePod(f, pausePodConfig{ Name: "filler-pod-" + nodeName, Resources: &v1.ResourceRequirements{ Limits: v1.ResourceList{ v1.ResourceCPU: *resource.NewMilliQuantity(requestedCPU, "DecimalSI"), }, Requests: v1.ResourceList{ v1.ResourceCPU: *resource.NewMilliQuantity(requestedCPU, "DecimalSI"), }, }, Affinity: &v1.Affinity{ NodeAffinity: &v1.NodeAffinity{ RequiredDuringSchedulingIgnoredDuringExecution: &v1.NodeSelector{ NodeSelectorTerms: []v1.NodeSelectorTerm{ { MatchExpressions: []v1.NodeSelectorRequirement{ { Key:      "node", Operator: v1.NodeSelectorOpIn, Values:   []string{nodeName}, }, }, }, }, </ins> }, }, <del> }), true, framework.Logf)) </del> <ins> }, })) </ins> } <ins> // Wait for filler pods to schedule. for _, pod := range fillerPods { framework.ExpectNoError(framework.WaitForPodRunningInNamespace(cs, pod)) } By("Creating another pod that requires unavailable amount of CPU.") // Create another pod that requires 50% of the largest node CPU resources. // This pod should remain pending as at least 70% of CPU of other nodes in // the cluster are already consumed. </ins> podName := "additional-pod" conf := pausePodConfig{ Name:   podName, Labels: map[string]string{"name": "additional"}, Resources: &v1.ResourceRequirements{ Limits: v1.ResourceList{ <del> v1.ResourceCPU: *resource.NewMilliQuantity(milliCpuPerPod, "DecimalSI"), </del> <ins> v1.ResourceCPU: *resource.NewMilliQuantity(nodeMaxAllocatable*5/10, "DecimalSI"), </ins> }, }, } WaitForSchedulerAfterAction(f, createPausePodAction(f, conf), podName, false) <del> verifyResult(cs, podsNeededForSaturation, 1, ns) </del> <ins> verifyResult(cs, len(fillerPods), 1, ns) </ins> }) // Test Nodes does not have any label, hence it should be impossible to schedule Pod with

cc cc Seeing a failure on our downgrade test. Can someone please look at this? https://k8s- Example failure https://k8s- Seeing timeouts on the downgrade test
Since we're so late in the release process and this is the first time we're seeing this fail, going to mark as non-release-blocking for now.
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/scheduling: Issue will be escalated to these SIGs if needed. priority/critical-urgent: Never automatically move out of a release milestone; continually escalate to contributor and SIG through all available channels. kind/bug: Fixes a bug discovered during the current release. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details
It fails in more places than just the downgrade job, but when the downgrade job fails that test, that timeout appears to be the consistent error that it fails with
/assign I looked at the logs. The test creates a few pods to saturate the existing nodes and then creates an additional pod and ensures that this last pod stays pending. The problem we observe is that those saturating pods are often not scheduled, causing a test timeout. Those pods use all the resources on the nodes. So, any small changes to the amount of available node resources or even rounding errors could cause those saturating pods to not schedule. I will test this theory by cutting down the amount of resources that the saturating pods use, but I would like to check if the problem is fixed before submitting the PR. Where can I get the steps these cluster downgrade tests take?
The test setups are in the test-infra repo. E.g.
see my analysis in .


Question: // NewAllocatorCIDRRange creates a Range over a net.IPNet, calling allocatorFactory to construct the backing store. func NewAllocatorCIDRRange(cidr *net.IPNet, allocatorFactory allocator.AllocatorFactory) (*Range, error) { <del> max := integer.Int64Min(utilnet.RangeSize(cidr), 1<<16) base := utilnet.BigForIP(cidr.IP) </del> <ins> max := RangeSize(cidr) base := bigForIP(cidr.IP) </ins> rangeSpec := cidr.String() r := Range{

What happened: All the services are asking to be recreated because the service CIDR changed after upgrading from 1.16.3 ro 1.17.0 What you expected to happen: Smooth migration from to , which would not break service CIDR. How to reproduce it (as minimally and precisely as possible): Do an upgrade from version 1.16.3 to 1.17.0 Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: OS (e.g: ):Kernel (e.g. ):Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig node
do you mean that the service cidr changed without any intervention after the upgrade? EDIT 96.0.0/12 is the default service cidr when using kubeadm --service-cidr string Default: "10.96.0.0/12" EDIT if not using kubeadm the default is May it be possible you were using another service cidr before the upgrade and you didn't set it during the upgrade, so it fall back to the default?
Can you provide your kube-apiserver invocation?
xref which warnings if no explicit CIDR flag was passed but should not have changed behavior otherwise
Where do you see that? The default is specified here as 10.0.0.0/24 and did not change between 1.16.0 and 1.17.0:
I was assuming it was using kubeadm based on the log message 96.0.0/12 is the default service CIDR for kubeadm
ah, ok. If they are using kubeadm, that default also did not change between 1.16.0 and 1.17.0:
a few questions for you: how did you install your 1.16.x cluster? did you customize the service cidr in that deployment? how did you upgrade your 1.16.x cluster to 1.17.x? did you apply the same service cidr customization in that deployment?
16.x was a straight and normal install no change on service CIRD, and we have never touched this CIDR configuration. And kubeadm is supposed to be used during the upgrade from 1.16.x to 1.17.x.

We just followed the describe process: The logs system started showing the shared message.
can you reproduce?
can you paste ? That will show you kubeadm config
created a single node cluster with v1.16.4 and a custom service cidr: applied upgrade to v1.17.0: i do not see a change in the custom service cidr.
if you never touched the service CIDR on the initial install it should have been which is the kubeadm default. your config map shows . how did you adapt the IP to work with the kubeadm default service cidr () if you initially deployed without modifying said cidr? to me this suggests that a change was required at some point.
The only change is the update from 1.16.3 to 1.17.0, done yesterday. Please find below a snapshot from our logs: <img width="1635" alt="Screen Shot 2019-12-21 at 22 22 29" src="https://user-"
my point is that if you did not apply a custom service cidr on "kubeadm init", then this cidr was the default out of kubeadm and after upgrade the service cidr was preserved as your config map shows. what was the elasticsearch/logging service IP before upgrade? EDIT: also do you have the YAML that was used to create this service?
101.249.43 belongs to the service rubnet range at minimum the error message is wrong /kind bug
are those errors happening constantly or only happened during the upgrade? are you able to create new services within that range? let's say, with a service IP of 10.101.249.45 Also, can you paste your current service-cluster-cidr configured in the controller-manager and apiserver? i.e.
this is a bug in core: this call in the repair controller returns for an IP that is actually in the CIDR range: also indicates a missing unit test case. latest commits: this might have regressed with the dual-stack changes? cc /remove-triage support needs-information /sig network /remove-sig cluster-lifecycle
/priority important-longterm
I was able to reproduce the same behavior again, with a fresh install k8s release 1.16.4.

Just After installing metrics server, we could notice a shift in the service CIDR, but nothing in the logs.
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/cc
so the problem here is that an IPv4 CIDR such as translates to addresses. but this line here clamps the maximum to 2^16 () so for an address like the logic in the ip allocator fails in the function: as the offset ends up bigger than the maximum. it feels like the maximums for IPv4 and IPv6 should be different and larger. otherwise for CIDRs in the notation : can give incorrect results for IPv4 can give incorrect results for IPv6 cc who touched related logic.
I think that was touched in … was this bug preexisting or did that change behavior?
before PR () i do not see a clamp for IPv4: only one for IPv6 at 16bits.
We should: revert add unit tests for the missed cases get that into 1.17.1 reintroduce the reliance on the function carefully
What would be the safest path to recreate all the services whilst waiting a final solution ? Especially for production K8S.
to mitigate the error in 1.17.0 you need to modify the serivce CIDR to be , instead of . modify the ConfigMap CIDR to have modify the kubeadm static pod manifest for the api server () to have edit problematic Service objects and give them an IP in the range if needed.
Thank you for those instructions, I'm affected by this, does this have anything to do with a possibility of ClusterIP's not properly responding or does it only print out alerts?
hello, can you test if the ClusterIP is not responding? i don't think we got a confirmation if this is only an alert. my guess is that it is just that, but there might be side effects.
i see you logged i will comment there.
How do you "recreate" a service? Literally delete and create again? Or is there a command which would cause it to re-create itself without having to be deleted?


Question: } } <ins> // TestDeltaFIFO_replaceWithDeleteDeltaIn tests that a `Sync` delta for an // object `O` with ID `X` is added when .Replace is called and `O` is among the // replacement objects even if the DeltaFIFO already stores in terminal position // a delta of type `Delete` for ID `X`. Not adding the `Sync` delta causes // SharedIndexInformers to miss `O`'s create notification, see https://github.com/kubernetes/kubernetes/issues/83810 // for more details. func TestDeltaFIFO_replaceWithDeleteDeltaIn(t *testing.T) { oldObj := mkFifoObj("foo", 1) newObj := mkFifoObj("foo", 2) f := NewDeltaFIFO(testFifoObjectKeyFunc, keyLookupFunc(func() []testFifoObject { return []testFifoObject{oldObj} })) f.Delete(oldObj) f.Replace([]interface{}{newObj}, "") actualDeltas := Pop(f) expectedDeltas := Deltas{ Delta{Type: Deleted, Object: oldObj}, Delta{Type: Sync, Object: newObj}, } if !reflect.DeepEqual(expectedDeltas, actualDeltas) { t.Errorf("expected %#v, got %#v", expectedDeltas, actualDeltas) } } </ins> func TestDeltaFIFO_requeueOnPop(t *testing.T) { f := NewDeltaFIFO(testFifoObjectKeyFunc, nil)

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: I did a review of and found out a case where a SharedIndexInformer can miss a notification. Follows an example scenario where this could happen. 1) A SharedIndexInformer is running, and its reflector is WATCHing. 2) A delete notification for an API object X with namespaced name N1 and UID U1 which was in the informer's cache is received. 3) The reflector on X; the relevant store is a DeltaFIFO and its Delete method is . 4) The DeltaFIFO's Delete method invokes , which . Notice that the store of the reflector, i.e., the DeltaFIFO, is NOT the SharedIndexInformer's store/cache: . Assume that BEFORE X's deletion is popped from the DeltaFIFO and applied to the informer's cache, the following sequence of events take place: 5) The WATCH is interrupted because of a network partition. 6) While the reflector is not watching, a new API object Y with namespaced name N1 (same as X) and UID U2 is created. 7) The partition disappears and the reflector does a relist against the API server; there's a "hole" between the last watch notification and the list results; assume Y is in the LIST results. 8) The reflector will load the LIST results in the store/DeltaFIFO with which in turn calls the DeltaFIFO's method. The DeltaFIFO's replace method attempts to add to the DeltaFIFO a delta of type for every object in the LIST results: when processing Y it will which should . But won't enqueue Y's delta, because it enqueues deltas of type only if there's not already a delta of type for the given object ID as the last delta for that object ID (see and ), to avoid recreating a deleted object during a periodic resync triggered by the SharedIndexInformer resync period. Object IDs are namespaced names, and remember that the last delta for Y's namespaced name in the DeltaFIFO is X's deletion.

The consequence is that Y's creation delta is not to the DeltaFIFO: Y is not to the SharedIndexInformer's cache and its creation notification handler does not execute. The SharedIndexInformer missed Y's creation. What you expected to happen: I expected SharedIndexInformers to get all notifications from the API server. Anything else we need to know? Missed notifications should be rare: for them to occur a delta must stay in the DeltaFIFO without being processed for long enough for a new List against the API server to be completed. Also, touches the reflector code that handles list and watches, and it might introduce changes relevant to this bug. Proposed solution: There's a straightforward fix: get rid of the check that skips enqueuing of a delta for object ID X if the last delta to the DeltaFIFO (and still in it) for X is of type ; is the code to remove. Notice that such check applies to deltas of type only. deltas are triggered only by relists and periodic resyncs. In the case of relists the check is harmful as it's the cause of this bug. In the case of periodic resyncs, that check is useless because it will either evaluate to or not execute; in fact, during a period resync, and this condition is checked before the check that would be removed, OTOH, if this condition is true and processing goes on the check that would be removed can only evaluate to (hence the delta for X is enqueued). If this solution has consensus I'd be happy to open a PR.
/assign
/sig api-machinery
cc
cc
xref ,
cc
/cc
This looks similar to failures we were seeing in kube-proxy while doing a rolling master upgrade. It was repeatable enough that someone hacked a workaround that I later spotted - this is close enough to the description that I’m suspicious that it was real /priority critical-urgent Since any missed informer event can cause all sorts of safety violations.
Note that rolling master upgrades break watches en masse, so a backlogged informer (contention / other locking / busy machine) would be more likely so see it the more caches it has.
A great first step would be to craft a test that demonstrates the issue repeatably, to help verify potential fixes resolve it.

IYO what would be the most appropriate kind of test, a unit test for the DeltaFIFO or something higher-level like integration/e2e tests?
The lower-level the test will be the better. So ideally unit test.
the lowest level that demonstrates the problematic behavior. if the issue is with the interaction between reflector and shared informer, then ideally something at the shared informer level
When the reflector calls it is expected that for each object O with ID X in a delta is to the DeltaFIFO, but that is (erroneously) not the case if the DeltaFIFO already stores a delta for ID X . So it's the DeltaFIFO which causes troubles and should be tested. But that alone does not show the user-visible problem; for a test to fail because a SharedIndexInformer missed a notification, the test would have to be at the SharedIndexInformer level.
That seems reasonable. I'd expect a DeltaFIFO test that sets up the described scenario and exercises both the Relist/Replace and Resync cases described here:
Sounds good, I'm at work.
is the testcase literally just adding this to But for some reason this hangs in the Pop(f).
Yes, the test looks very similar to what you posted, but there's a mistake: the expected result must be a single with the two that in your example are the only elements of the two 1-element Deltas that make up , respectively.
(that would be caught and fail the test immediately by changing to )
Aha, you're totally right :) Had to run off to lunch before spending more time on it.
DeltaFIFO only de-dupes deletions, but your example would do effectively extend that to compressing Delete+Sync=Sync. You used to be able to build smarter compression off DeltaFIFO (eg ) but nobody actually used the DeltaCompressor stuff so we removed it. And nobody above DeltaFIFO expects compressed events now :( In your example I'd be a bit worried about the implications of not deleting the object, since the Sync-ed one probably has a different UUID and thus technically not be the same object as the deleted one. Might not be a big issue, but this is critical path code so I'd personally rather be a bit conservative.
I agree about not compressing.

It's a little hard to reason about what this sync event should be when we know that relists can currently time-travel backwards. Edit: it looks like shared informer code will if it follows a delete event.
I was this morning in the "sig-api-machinery" channel of the "Kubernetes" Slack team. My reading of the code had me expecting that I'd get a delete followed by an add.
This issue is specifically focused on how a queued delete event (typically via directly observing a delete watch event or via absence in a relist/replace) followed by an observed add (via presence in a relist) is handled by DeltaFIFO. your question is slightly broader/different, and has multiple facets: UID changes considered when handling reflector object add/delete/update events? They are not; if an object is deleted and recreated between the reflector dropping a watch and relisting, the reflector will enqueue an Sync event with the object with the new uid (which the shared informer will turn into an Update event) delete/add events is actively compressed into an update event? I don't think so, but it's been a while since I've traipsed through reflector/fifo code, so I'm not positive (and since the Update with a new uid can occur from scenario 1, informer clients have to be prepared for uid to change on update anyway)
Listeners need to handle Sync too. And the only two in-tree users of NewDeltaFIFO are and and they look like they correctly handle Sync.
Edit: yep you're right, I was looking at slightly older code.
if this happens, I think it should happen on top of the DeltaFIFO, not in the DF. Let's keep DF simpler since it's (a) complicated enough and (b) critical path code. probably the right place for compression, if any happens. IMHO it's correct that Deletes don't get compressed, eg that Delete + Add is not compressed into Update or something like that.
I agree, though that specific example can happen naturally if it occurs in a window while the reflector isn't getting watch events.

Just to make sure I understand this properly, does this scenario involve an object in the cache with name "x" and UID "u1," then the reflector loses its watch, reestablishes it and lists all the relevant objects in the cluster, finds one named "x" with UID "u2," and since it already has an "x" in its cache, translates this discovery into a call?
exactly (both namespace and name are part of the cache key, uid is not)
does this mean we couldn't use the argument to DeltaFIFO's Replace() to work around whatever is talking about (if that problem still exists)?
clients can only use resourceVersion to check equality, not older-than/newer-than
As noted in Slack, there is now careful commentary in about what is promised. There is no promise that a delete followed by an add is delivered as two notifications; it is allowed to be delivered as one update notification.


Question: # Force delete a pod on a dead node kubectl delete pod foo --grace-period=0 --force <del> # Delete a pod with UID 1234-56-7890-234234-456456. kubectl delete pod 1234-56-7890-234234-456456 </del> # Delete all pods kubectl delete pods --all`) )

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): delete pod delete pod uid Is this a BUG REPORT or FEATURE REQUEST? (choose one): maybe a bug report: the "kubectl delete pod [uid]" doesn't work. Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.1+", GitCommit:"", GitTreeState:"dirty", BuildDate:"2017-01-10T00:30:53Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.1+", GitCommit:"", GitTreeState:"dirty", BuildDate:"2017-01-10T00:26:06Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"} What happened: First, use "kubectl get pod [podname] -o yaml" to get the details of the pod, obtaining the uid under Then use "kubectl delete pod [uid]" to delete the pod, but failed finally. the info showed like this: Error from server (NotFound): pods "-dbf8-11e6-818c-" not found. Did I get the wrong uid? What you expected to happen: It should delete pod successfully just like using podname. How to reproduce it (as minimally and precisely as possible): just do like mentioned in "What happened"
I don't think we provide pod deletion by UID... /cc
If you type "kubectl delete pod -h", there is a piece of specifications describing like this: "# Delete a pod with UID 1234-56-7890--. kubectl delete pod 1234-56-7890--" So is this a mistake?
I don't know. It could be a feature that I am not aware of. Folks from the will know better.

kubectl delete pod 1234-56-7890-- It seems this help message exists even in v1.0. Do we support this before?
AFAIK, apiserver doesn't support operations using the UID as the key. Unless kubectl implement additional logic to map the UID to a (name, namespace) tuple, it'd not be able to do that.
Delete by UID is no longer supported, this is an issue with the help text that needs to be adjusted.


Question: }, ExpectedTaints: []*v1.Taint{notReadyTaint}, }, <ins> { Name: "Ready is unknown", Node: &v1.Node{ ObjectMeta: metav1.ObjectMeta{ Name:              "node0", CreationTimestamp: metav1.Date(2012, 1, 1, 0, 0, 0, 0, time.UTC), Labels: map[string]string{ kubeletapis.LabelZoneRegion:        "region1", kubeletapis.LabelZoneFailureDomain: "zone1", }, }, Status: v1.NodeStatus{ Conditions: []v1.NodeCondition{ { Type:               v1.NodeReady, Status:             v1.ConditionUnknown, LastHeartbeatTime:  metav1.Date(2015, 1, 1, 12, 0, 0, 0, time.UTC), LastTransitionTime: metav1.Date(2015, 1, 1, 12, 0, 0, 0, time.UTC), }, }, }, }, ExpectedTaints: []*v1.Taint{unreachableTaint}, }, </ins> } for _, test := range tests {

In autoscaling tests, we verify that when a node suddenly disappears from the cluster, Cluster Autoscaler will add another one to replace it. With TaintNodesByCondition (enabled by default in ), this is now failing: https://k8s- Every time a node disappears, a subset of pods become NodeLost and aren't rescheduled on a new node. /cc /sig node
I can work on this.
just got merged. Let's check result of autoscaling testings tomorrow: https://k8s-
latest autoscaling testing is green. I think we can close this issue.
Thanks for fixing this!


Question: Subsystem:         KubeletSubsystem, Name:              AssignedConfigKey, Help:              "The node's understanding of intended config. The count is always 1.", <del> DeprecatedVersion: "1.23.0", </del> <ins> DeprecatedVersion: "1.22.0", </ins> StabilityLevel:    metrics.ALPHA, }, []string{ConfigSourceLabelKey, ConfigUIDLabelKey, ConfigResourceVersionLabelKey, KubeletConfigKeyLabelKey},

This shipped in 1.22 as deprecated in 1.22. As a hacky fix to unbreak tests on 1.23 dev, I bumped the deprecation policy one release: We will need to revert that before 1.23 ships and figure out a better solution. /kind cleanup /milestone v1.23 /sig node
/triage accepted
I want to try it. Can I take this issue?
I have explained how to fix this on PR:
/reopen reverted in as fix did not work
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
Hi, Im the bug triage shadow. Since we're in 1.23 milestone code freeze, could we move this issue to milestone 1.24?
this is critical for 1.23 and I think it just got forgotten... we need to get this fixed before test freeze
/priority important-soon


Question: } // setCDIDevices adds a set of CDI devices to the claim info. <del> func (info *ClaimInfo) addDevice(driverName string, device state.Device) { </del> <ins> func (info *ClaimInfo) addDevice(driverName string, deviceState state.Device) { </ins> if info.DriverState == nil { info.DriverState = make(map[string]state.DriverState) } driverState := info.DriverState[driverName] <del> driverState.Devices = append(driverState.Devices, device) </del> <ins> driverState.Devices = append(driverState.Devices, deviceState) </ins> info.DriverState[driverName] = driverState }

When adding the field to , old checkpoint files became invalid because the checksum that gets calculated by hashing the output in no longer matches. This caused kubelet to fail to start with: Here's what the checksum in a unit test gets calculcated for: Note the new . Not sure what the right behavior is. Perhaps kubelet should have ignored the invalid checkpoint? Revert the update in The state.DRAManagerCheckpoint contains a version field, which is set to v1.I think that's where things fall apart: that version must exactly match the type that gets serialized, which is not the case here. The type changes over time, but the version is fixed. Let's reconsider what should get checkpointed and how we can make that stable. Also, is checkpointing really worth it? What bad things happen if we remove it? master n/a No response No response No response /sig node /cc
/cc
We originally didn't have checkpointing and it was by in to fix
The reasons for checkpoint were: (what Kevin mentioned) of the pod resource API DAR extensions concerns raised on what happen if kubelet restart and we get GetPodResourcesRequest. The request should return the DRA claimed devices but if we didn't checkpoint the DRA cache is empty and we would return empty resources force delete see [2] My original PR didn't checkpoint the ResourceHandle is was in PR [2]. Maybe we should exclude ResourceHandle from the checksum hash calculation [1] [2] -
So the cache makes sense. What doesn't make sense to me is how the consistency of that on-disk file is checked and how it is versioned. Shouldn't the checksum be calculated for the bytes on disk (version-independent) instead of the in-memory representation (version-dependent)? Or is the goal to protect against "I can't read this data anymore" instead of "bit has flipped on disk"? The assumption is that the content can always be decoded from what was written in the past into the type which is used now. This works with JSON if the newer type is a true superset of the old one, but that's not necessarily guaranteed. The proper way of doing this is to define a scheme which includes conversion between different versions.

This cannot be done when embedding API types from because conversion between current v1alpha2 and a future v1beta1 can only be done in the apiserver, not client-side. I don't see a quick solution here. In the short-term, how do we deal with version upgrades? Require that users drain the node (thus clearing the cache) before upgrading or downgrading kubelet?
I am still trying to understand the problem. the problem that we checksum the version? the problem that we don't have conversion mechanism to move from version X -version Y on upgrade? the issue related only for upgrade or there are other cases which checksum changes? In general we the version to checkpoint file so that in the future when the DRA checkpoint struct changed we can have mechanism to convert old checkpoint version to new checkpoint version (in the initial PR we didn't implement conversion mechanism as I thought it will be on DRA checkpoint struct change) . If you take a look on the CPU Manager checkpoint [1] you will see that it detecting if the scheme matches to V1 first and if not it assume V2. We wanted that it will part of the checkpoint file so we won't need such detection and in the file we will know that correct version of the DRA checkpoint struct. Now if someone change the DRA checkpoint scheme and didn't implement migration flow (similar to CPU manager [2]) or change the version that indeed an issue, [1] - [2] -
The problem is that after an upgrade of kubelet from <1.30 to 1.30, it won't be able to read an existing, non-empty checkpoint file. Worse, it will refuse to start because of that. The root cause is the lack of proper versioning and lack of conversion code. Any version change potentially has this problem (including downgrades). I'm not aware of other failure scenarios.
Since it is not a GA feature, we cannot fix this in 1.30. But marking it as important long-term. /priority important-longterm
This has to be solved before graduating from alpha
I intend to move all version-dependent code into drivers (). This will simplify the checkpoint content a bit, but the overall questions around its design remain.

/kind feature /remove-kind bug /triage accepted I think a good forum to discuss this could be the KEP for DRA, though we can keep this tracker here
Isn't it because checkpoint structure was changed without incrementing version and adding conversion code? Anyway, I'd propose to remove version from the checkpoint structure and comment it to prevent changing it without changing its version. the implementation should be similar to the cpu maanager checkpointing. If it's been working for CPU (and device?) managers for some time, it should hopefully work for DRA as well. Any other ideas?
Yes. I haven't looked into this again. Can you implement your proposal on top of and submit a PR for review, or do you want to discuss this first?
if you're ok with my proposal I'm ok to implement it. Otherwise it needs to be discussed further.
I don't fully understand the implications without looking into this further, so I can say neither yes nor no to it at this point. do you have an opinion?


Question: framework.ExpectEqual(endpointSubsetOnePorts.Port, int32(8080), "failed to patch Endpoint") ginkgo.By("deleting the Endpoint by Collection") <del> err = f.ClientSet.CoreV1().Endpoints(ns).DeleteCollection(context.TODO(), metav1.DeleteOptions{}, metav1.ListOptions{LabelSelector: "testendpoint-static=true"}) </del> <ins> err = f.ClientSet.CoreV1().Endpoints(testNamespaceName).DeleteCollection(context.TODO(), metav1.DeleteOptions{}, metav1.ListOptions{LabelSelector: "test-endpoint-static=true"}) </ins> framework.ExpectNoError(err, "failed to delete Endpoint by Collection") ginkgo.By("waiting for Endpoint deletion") <del> for watchEvent := range endpointWatchChan { if watchEvent.Type == "DELETED" { break </del> <ins> ctx, cancel = context.WithTimeout(context.Background(), 30*time.Second) defer cancel() _, err = watchtools.Until(ctx, endpoints.ResourceVersion, w, func(event watch.Event) (bool, error) { switch event.Type { case watch.Deleted: if endpoints, ok := event.Object.(*v1.Endpoints); ok { found := endpoints.ObjectMeta.Name == endpoints.Name && endpoints.Labels["test-endpoint-static"] == "true" return found, nil } default: framework.Logf("observed event type %v", event.Type) </ins> } <del> } </del> <ins> return false, nil }) framework.ExpectNoError(err, "failed to see %v event", watch.Deleted) ginkgo.By("fetching the Endpoint") _, err = f.ClientSet.CoreV1().Endpoints(testNamespaceName).Get(context.TODO(), testEndpointName, metav1.GetOptions{}) framework.ExpectError(err, "should not be able to fetch Endpoint") </ins> }) })

[x] Original approval issue: [x] Two weeks soak end date: This issue is created to allow edit by /wip /hold This issue proposes the writing of a new test for the API endponts , , , with the intention of promoting it to Conformance. This issue contains a very basic mock test in order to start the conversation about what a good e2e test would look like for this endpoint. The code below is verified to be hitting the intended endpoint, show as per the queries in APIsnoop's live view of the cluster's audit logs. According to this APIsnoop query, there are still some remaining Endpoints endpoints which are untested. a Service in a namespace all Endpoints in all Namespaces find the created Service(1) /update the Namespaced Endpoint for the Service(1) the Namespaced Endpoint for the Service(1) the Namespaced Endpoint for the Service(1) Namespaced Endpoint(1) via a Collection with a LabelSelector Discover useragents: If a test with these calls gets merged, Conformance coverage will go up by 4 points* /sig testing /sig architecture /area conformance
/reopen Wait for Promotion PR to Merge.
Reopened this issue. <detailsIn response to : /reopen Wait for Promotion PR to Merge. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: {kubeNodePortSetTCP, string(KubeNodePortChain), string(KubeMarkMasqChain), "dst", utilipset.ProtocolTCP}, {kubeNodePortLocalSetUDP, string(KubeNodePortChain), "RETURN", "dst", utilipset.ProtocolUDP}, {kubeNodePortSetUDP, string(KubeNodePortChain), string(KubeMarkMasqChain), "dst", utilipset.ProtocolUDP}, <del> {kubeNodePortSetSCTP, string(KubeNodePortChain), string(KubeMarkMasqChain), "dst,dst", utilipset.ProtocolSCTP}, </del> {kubeNodePortLocalSetSCTP, string(KubeNodePortChain), "RETURN", "dst,dst", utilipset.ProtocolSCTP}, <ins> {kubeNodePortSetSCTP, string(KubeNodePortChain), string(KubeMarkMasqChain), "dst,dst", utilipset.ProtocolSCTP}, </ins> } // In IPVS proxy mode, the following flags need to be set

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --ipvs mode didn't preserve client IP for NodePort Service with protocol SCTP and ExternalTrafficPolicy Local. iptables mode works fine. ipvs mode with TCP and UDP protocol works fine. Client IP for NodePort Service should be preseved when ExternalTrafficPolicy is Local regardless of the mode and the protocol. kube-proxy mode to ipvs a NodePort service with protocol SCTP and ExternalTrafficPolicy Local the NodePort service, the client IP was not preserved This is caused by wrong order of iptables rules that match kubeNodePortLocalSetSCTP and kubeNodePortSetSCTP: Kubernetes version (use ): v1.22.0 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig network /assign
/triage accepted


Question: Do() found := 0 <del> r.IgnoreErrors(errors.IsNotFound).Visit(func(r *resource.Info) error { </del> <ins> err := r.IgnoreErrors(errors.IsNotFound).Visit(func(r *resource.Info) error { </ins> found++ if err := resource.NewHelper(r.Client, r.Mapping).Delete(r.Namespace, r.Name); err != nil { return err

We should support multiple objects on the command line, but, failing that, some kind of error message would be useful. /cc
Fixing, just fixed the equiv in create


Question: v1 "k8s.io/api/core/v1" "k8s.io/apimachinery/pkg/types" <ins> schedulerapi "k8s.io/kubernetes/pkg/scheduler/api" </ins> internalcache "k8s.io/kubernetes/pkg/scheduler/internal/cache" )

What would you like to be : a default normalize score function in the scheduling framework. the plugin and log an error when the normalizing phase returns an invalid range of scores. Why is this needed: Per discussion in , we should enforce the range of score returned by the scoring plugin. /sig scheduling /priority important-soon /assign


Question: expectedToFail: false, config:         goodConflictPlugins2, }, <ins> "bad-plugins-config": { expectedToFail: true, config:         deprecatedPluginsConfig, errorString:    "profiles[0].pluginConfig[1]: Invalid value: "NodeResourcesLeastAllocated": was removed in version "kubescheduler.config.k8s.io/v1beta2" (KubeSchedulerConfiguration is version "kubescheduler.config.k8s.io/v1beta2")", }, </ins> } for name, scenario := range scenarios {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --Have a legacy scheduler config, upgrade to v1beta2 since we are at v1beta2 now, scheduler failed to start with some unreadable raw data, accurate and readable error message. config scheduler config like what has been said above. Kubernetes version (use ): latest. Cloud provider or hardware configuration: OS (e.g: ): all Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /sig scheduling /assign
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: ### fake-gitserver <del> Fakes a git server. When doing `git clone localhost:8000`, you will clone an empty git repo named `8000` on local. You can also use `git clone localhost:8000 my-repo-name` to rename that repo. </del> <ins> Fakes a git server. When doing `git clone http://localhost:8000`, you will clone an empty git repo named `localhost` on local. You can also use `git clone http://localhost:8000 my-repo-name` to rename that repo. Access to the service with the backing pod will show you below information. ```console curl -w "n" http://localhost:8000 I am a fake git server ``` </ins> Usage:

What happened: Follow the "README", I hit below issue, the command there need a prefix "http" or "https". What you expected to happen: clone a fake repo successfully. How to reproduce it (as minimally and precisely as possible): follow the "README" here. create a pod create a service git clone localhost:8000 Anything else we need to know?: Environment: Kubernetes version (use ): latest Cloud provider or hardware configuration: OS (e.g: ): all Kernel (e.g. ): all
/assign
/sig testing
/kind documentation


Question: err := p.client.PullImage(opts, docker.AuthConfiguration{}) if err == nil { <ins> // Sometimes PullImage failed with no error returned. exist, ierr := p.IsImagePresent(image) if ierr != nil { glog.Warningf("Failed to inspect image %s: %v", image, ierr) } if !exist { return fmt.Errorf("image pull failed for unknown error") } </ins> return nil }

I set a non-exist image in rc and watch event. kubelet is supposed to throw failed pull event and actually we get failed pull event. But I noticed there is an "successfully pulled image" event.


Question: package ipvs import ( <ins> "fmt" </ins> "sync" "time" <del> "fmt" </del> "k8s.io/apimachinery/pkg/util/wait" "k8s.io/klog" utilipvs "k8s.io/kubernetes/pkg/util/ipvs"

What happened: We're running pod behind and sending logs via UDP to it. It is k8s v1.13.3 with ipvs services. When pod gets restarted logs getting blackholed. Simplified example on a node: So, old pod had IP 192.168.129.116, new pod - 192.168.128.193. Weight correctly forwards all new traffic to new pod, but there are some which are still sending traffic to non-existing IP: Minimal value for UDP connection expiration is 1s: But logs are being sent at much faster rates, so these broken connections are never expire. Issue introduced by Looks like this supposed to keep old pod running, while there are still connections to it But is there a reason to still keep these realServer entries, while old pod already cleaned up and no such IP exist? What you expected to happen: would cleanup all non-existing realServers from ipvs entries. How to reproduce it (as minimally and precisely as possible): Repro steps are same as for host-port issue: Anything else we need to know?: sysctls are default: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: on-premise OS (e.g: ): Debian 9 Kernel (e.g. ): 4.9.0-8-amd64 Install tools: Others:
/sig network
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by
cc
We observe the same behavior in k8s v1.12.7 and it hits us hard. In our case Nginx in a pod uses coredns service IP as DNS resolver address and every time coredns pods get new addresses DNS queries are blackholed. Query rate is higher than ipvs UDP connection timeout (30s in our case), so the connection to a non-existent ip address stays forever. Kube-proxy keeps non-existent real server entry in ipvs service and conntrack entry is never getting removed either. Kube-proxy logs: ipvsadm -ln:
I've been putting some thoughts on the issue (we actually have the same one). Currently, we handle graceful termination the same way for udp as we do for tcp. However on heavily loaded UDP services, (DNS for instance) udp ports are reused quickly and old backends are not deleted fast enough.

There are two possible solutions: use more aggressive timeouts (30s instead of the default 300s). Not perfect but better immediately delete real servers and associated connections when a backend is not ready anymore (very good for DNS queries but this means no graceful termination for udp) / what do you think? I'm not sure what the exact behavior with iptables I can create a detailed issue with options to have a wider discussion. Or even directly send an email to the sig-network group
I like the idea. BTW, iptables also does not support graceful termination for UDP connections.
this is very easy to implement: change the test here: make sure is set (which we do : )
If we agree this is the way to go, I can implement it (if iptables works this way this is the best solution, I agree) We could still tweak timeouts for TCP (defaults are very long: 900s, which can lead to complete blackholing for 15mn on a crash of a target because the connection is never closed)
I agree.
/area ipvs
If you have a test environment I have a test image with the fix discussed just above: Commit here: This is not perfect, but I believe it is much better. What happens is the following: udp target pod becomes NotReady IP is removed from RealServers immediately Because of any new packet reusing the port will generate a write error and the connection will be removed from the ipvs connection table (if the application retries you are fine) I think in addition to that we should set much more aggressive UDP timeouts (5s? instead of the default: 300s) and make it configurable with a flag for specific use-case (I haven't seen a use case for long lived udp "connections" with very low amount of traffic)
Sorry, i was unable to quick test this on local dev cluster. Tried to replace image with but it seems it does not work with k8s 1.13. It then generates a lot of the same messages, and no any ipvs VIPs are created. I've also tried on fresh new cluster with the same result. Then tried but was unable to even start the cluster (with modified config from our 1.13) Can you please cherry-pick this to 1.13 or 1.14 or am I doing something wrong?
If this issue has been triaged, please comment .

If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
We tested this patch to kubernetes v1.12.7 and it works well in production. BTW, I don't know much about SCTP. But question is do we need to threat it as UDP or TCP in this case? But it seems more close to UDP.
Thanks for testing! The PR is limiting graceful termination to TCP which seems the safest. I don't know SCTP but making it behave like TCP would be an easy change.
This feels like a fundamental problem w/ endpoints... ideally the proxier would remove the realserver when the pod is nonexistent rather than relying on arbitrary timeouts. Otherwise, there is no way to support graceful termination with UDP based protocols. Not a big deal for most UDP things, but I imagine e.g. QUIC users would be affected. Immediately terminating udp connections is in line with the iptables behavior, which seems to be okay for most people, so the change seems reasonable until there is a better way to determine pod state.
The PR that was just merged will improve things a lot for UDP but I completely agree with you on the fundamental problem with endpoints. I think we should have 3 different states: Ready (accepting traffic), in which case we should route traffic to the endpoint NotReady (terminating for instance), in which case we should stop routing new traffic to the endpoint but should keep existing connections "Gone" (no in the endpoint object anymore), in which case we should remove the endpoint altogether Today the endpoint object is too simple to handle these differences. I hope the new EndpointCondition status in the EndpointSlice API will help address this problem. More than happy to integrate this in IPVS when it is released (cc / )


Question: package podlogs import ( <ins> "bytes" </ins> "crypto/tls" <ins> "crypto/x509" "encoding/base64" </ins> "io/ioutil" "net/url" <ins> "path" </ins> "strings" "testing" "time" "k8s.io/apiserver/pkg/server/dynamiccertificates" "k8s.io/kubernetes/cmd/kube-apiserver/app/options" "k8s.io/kubernetes/test/integration/framework" )

When we want to rotate certificates for aggregated API servers, we have to restart (kill) pods in order for API server to pick the new certificates. It would be very useful to implement some sort of self-awareness for the aggregated API servers, so they can pick-up new certificates when they're updated, without user interaction. If we have certificate files in pods, that are provided by some volumeSource, that can be persistent storage or a Secret, when file gets changed on the volume, the kubelet updates file in the pod usually in matters of few seconds. However, aggregated API servers are not self-aware of such change and users have to manually restart the pods. This brings additional complexity, especially if you want to automate the process. The first problem is that you need to know what pods you need to restart. But the potentially biggest problem is that you need additional permissions to do so. That would include RBAC permissions for listing pods, listing deployments and for deleting pods. That can make a security issue in some setups. By making aggregated API servers sort of self-aware, this could be avoided. Create a health check somewhere in , that utilizes a file watcher to watch for certificates changes. When certificate is changed, fail a health check. Failing health check will restart the pod without user interaction. This could be the most easiest and simplest solution to the problem. Such feature should/could be optional and guarded by a flag, let's name it . Some delay could be required between restarting pods. We probably don't want to kill all pods in the same time. This is up to be discussed. I would like to work on this issue, so feel free to assign it to me. /kind feature /sig api-machinery /area apiserver /cc
this could also be done by a purpose-built container that watched the same input files. as you point out, you likely would not want to restart all apiservers at once, which makes it unclear whether you'd want to build in the restart logic to the individual servers
That coordination would certainly work and be a valuable tool for someone to build to help manage rolling generic processes if we never get a pod restart. However, being able to have a "usually won't all go down together" option built-in is easy to understand and will resolve the problem until then.

It isn't a stretch to say that if your process isn't fulfilling the current intent of the pod, it isn't healthy. This gives us a thing between dynamic config reloading and nothing. A container that can coordinate based on a service would be really cool, but I think that'd be a separate thing.
/cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: storageframework.CapOfflineExpansion:    true, storageframework.CapOnlineExpansion:     true, storageframework.CapSingleNodeVolume:    true, <ins> storageframework.CapReadWriteOncePod:    true, </ins> // This is needed for the // testsuites/volumelimits.go `should support volume limits`

For the ReadWriteOncePod feature to graduate to beta, we need to have E2E test coverage for its current alpha behavior. See this section of KEP-2485 for more context on what cases we need to cover for alpha: This is a requirement for beta graduation, see:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage


Question: wantError: true, }, { <ins> name:      "Pod copy: specifying attach without existing or newly created container", args:      "mypod --set-image=*=busybox --copy-to=my-debugger --attach", wantError: true, }, { </ins> name: "Node: interactive session minimal args", args: "node/mynode -it --image=busybox", wantOpts: &DebugOptions{

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Using without specifying should modify existing container images without trying to create a new one. Instead it tries to create a new container and returns an error that isn't set: What you expected to happen: This command should change all container images to instead of returning an error. /sig cli /kind bug /priority important-soon
/assign /triage accepted
My kubernetes cluster version is 1.19.0,and i use kubectl-debug version 0.1.1 try to debug my pod,bug i got error: <img width="1470" alt="image" src="https://user-"Who one know about that


Question: } } } <del> func TestConstantMaps(t *testing.T) { pkg, err := importer.Default().Import("crypto/tls") if err != nil { fmt.Printf("error: %sn", err.Error()) return } discoveredVersions := map[string]bool{} discoveredCiphers := map[string]bool{} for _, declName := range pkg.Scope().Names() { if strings.HasPrefix(declName, "VersionTLS") { discoveredVersions[declName] = true } if strings.HasPrefix(declName, "TLS_") && !strings.HasPrefix(declName, "TLS_FALLBACK_") { discoveredCiphers[declName] = true } } acceptedCiphers := allCiphers() for k := range discoveredCiphers { if _, ok := acceptedCiphers[k]; !ok { t.Errorf("discovered cipher tls.%s not in ciphers map", k) } } for k := range acceptedCiphers { if _, ok := discoveredCiphers[k]; !ok { t.Errorf("ciphers map has %s not in tls package", k) } } for k := range discoveredVersions { if _, ok := versions[k]; !ok { t.Errorf("discovered version tls.%s not in version map", k) } } for k := range versions { if _, ok := discoveredVersions[k]; !ok { t.Errorf("versions map has %s not in tls package", k) } } } </del>

Once we upgrade to go1.14, we can use the new CipherSuites and InsecureCipherSuites functions and CipherSuiteName function to build the supported cipher suite flags. /sig api-machinery
/sig auth
/cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
as the project was upgraded to go-1.14+ is that still something that needs to be done?
yes, these manually maintained maps should be swapped to the stdlib impls:
/assign
/triage accepted Thanks,


Question: return false } <ins> type MockedFakeELBV2 struct { LoadBalancers []*elbv2.LoadBalancer TargetGroups  []*elbv2.TargetGroup Listeners     []*elbv2.Listener // keys on all of these maps are ARNs LoadBalancerAttributes map[string]map[string]string Tags                   map[string][]elbv2.Tag RegisteredInstances    map[string][]string // value is list of instance IDs } func (m *MockedFakeELBV2) AddTags(request *elbv2.AddTagsInput) (*elbv2.AddTagsOutput, error) { for _, arn := range request.ResourceArns { for _, tag := range request.Tags { m.Tags[aws.StringValue(arn)] = append(m.Tags[aws.StringValue(arn)], *tag) } } return &elbv2.AddTagsOutput{}, nil } func (m *MockedFakeELBV2) CreateLoadBalancer(request *elbv2.CreateLoadBalancerInput) (*elbv2.CreateLoadBalancerOutput, error) { accountID := 123456789 arn := fmt.Sprintf("arn:aws:elasticloadbalancing:us-east-1:%d:loadbalancer/net/%x/%x", accountID, rand.Uint64(), rand.Uint32()) newLB := &elbv2.LoadBalancer{ LoadBalancerArn:  aws.String(arn), LoadBalancerName: request.Name, Type:             aws.String(elbv2.LoadBalancerTypeEnumNetwork), VpcId:            aws.String("vpc-abc123def456abc78"), } m.LoadBalancers = append(m.LoadBalancers, newLB) return &elbv2.CreateLoadBalancerOutput{ LoadBalancers: []*elbv2.LoadBalancer{newLB}, }, nil } func (m *MockedFakeELBV2) DescribeLoadBalancers(request *elbv2.DescribeLoadBalancersInput) (*elbv2.DescribeLoadBalancersOutput, error) { findMeNames := make(map[string]bool) for _, name := range request.Names { findMeNames[aws.StringValue(name)] = true } findMeARNs := make(map[string]bool) for _, arn := range request.LoadBalancerArns { findMeARNs[aws.StringValue(arn)] = true } result := []*elbv2.LoadBalancer{} for _, lb := range m.LoadBalancers { if _, present := findMeNames[aws.StringValue(lb.LoadBalancerName)]; present { result = append(result, lb) delete(findMeNames, aws.StringValue(lb.LoadBalancerName)) } else if _, present := findMeARNs[aws.StringValue(lb.LoadBalancerArn)]; present { result = append(result, lb) delete(findMeARNs, aws.StringValue(lb.LoadBalancerArn)) } } if len(findMeNames) > 0 || len(findMeARNs) > 0 { return nil, awserr.New(elbv2.ErrCodeLoadBalancerNotFoundException, "not found", nil) } return &elbv2.DescribeLoadBalancersOutput{ LoadBalancers: result, }, nil } func (m *MockedFakeELBV2) DeleteLoadBalancer(*elbv2.DeleteLoadBalancerInput) (*elbv2.DeleteLoadBalancerOutput, error) { panic("Not implemented") } func (m *MockedFakeELBV2) ModifyLoadBalancerAttributes(request *elbv2.ModifyLoadBalancerAttributesInput) (*elbv2.ModifyLoadBalancerAttributesOutput, error) { attrMap, present := m.LoadBalancerAttributes[aws.StringValue(request.LoadBalancerArn)] if !present { attrMap = make(map[string]string) m.LoadBalancerAttributes[aws.StringValue(request.LoadBalancerArn)] = attrMap } for _, attr := range request.Attributes { attrMap[aws.StringValue(attr.Key)] = aws.StringValue(attr.Value) } return &elbv2.ModifyLoadBalancerAttributesOutput{ Attributes: request.Attributes, }, nil } func (m *MockedFakeELBV2) DescribeLoadBalancerAttributes(request *elbv2.DescribeLoadBalancerAttributesInput) (*elbv2.DescribeLoadBalancerAttributesOutput, error) { attrs := []*elbv2.LoadBalancerAttribute{} if lbAttrs, present := m.LoadBalancerAttributes[aws.StringValue(request.LoadBalancerArn)]; present { for key, value := range lbAttrs { attrs = append(attrs, &elbv2.LoadBalancerAttribute{ Key:   aws.String(key), Value: aws.String(value), }) } } return &elbv2.DescribeLoadBalancerAttributesOutput{ Attributes: attrs, }, nil } func (m *MockedFakeELBV2) CreateTargetGroup(request *elbv2.CreateTargetGroupInput) (*elbv2.CreateTargetGroupOutput, error) { accountID := 123456789 arn := fmt.Sprintf("arn:aws:elasticloadbalancing:us-east-1:%d:targetgroup/%x/%x", accountID, rand.Uint64(), rand.Uint32()) newTG := &elbv2.TargetGroup{ TargetGroupArn:  aws.String(arn), TargetGroupName: request.Name, Port:            request.Port, Protocol:        request.Protocol, } m.TargetGroups = append(m.TargetGroups, newTG) return &elbv2.CreateTargetGroupOutput{ TargetGroups: []*elbv2.TargetGroup{newTG}, }, nil } func (m *MockedFakeELBV2) DescribeTargetGroups(request *elbv2.DescribeTargetGroupsInput) (*elbv2.DescribeTargetGroupsOutput, error) { var targetGroups []*elbv2.TargetGroup if request.LoadBalancerArn != nil { targetGroups = []*elbv2.TargetGroup{} for _, tg := range m.TargetGroups { for _, lbArn := range tg.LoadBalancerArns { if aws.StringValue(lbArn) == aws.StringValue(request.LoadBalancerArn) { targetGroups = append(targetGroups, tg) break } } } } else if len(request.Names) != 0 { targetGroups = []*elbv2.TargetGroup{} for _, tg := range m.TargetGroups { for _, name := range request.Names { if aws.StringValue(tg.TargetGroupName) == aws.StringValue(name) { targetGroups = append(targetGroups, tg) break } } } } else if len(request.TargetGroupArns) != 0 { targetGroups = []*elbv2.TargetGroup{} for _, tg := range m.TargetGroups { for _, arn := range request.TargetGroupArns { if aws.StringValue(tg.TargetGroupArn) == aws.StringValue(arn) { targetGroups = append(targetGroups, tg) break } } } } else { targetGroups = m.TargetGroups } return &elbv2.DescribeTargetGroupsOutput{ TargetGroups: targetGroups, }, nil } func (m *MockedFakeELBV2) ModifyTargetGroup(request *elbv2.ModifyTargetGroupInput) (*elbv2.ModifyTargetGroupOutput, error) { var matchingTargetGroup *elbv2.TargetGroup dirtyGroups := []*elbv2.TargetGroup{} for _, tg := range m.TargetGroups { if aws.StringValue(tg.TargetGroupArn) == aws.StringValue(request.TargetGroupArn) { matchingTargetGroup = tg break } } if matchingTargetGroup != nil { dirtyGroups = append(dirtyGroups, matchingTargetGroup) if request.HealthCheckEnabled != nil { matchingTargetGroup.HealthCheckEnabled = request.HealthCheckEnabled } if request.HealthCheckIntervalSeconds != nil { matchingTargetGroup.HealthCheckIntervalSeconds = request.HealthCheckIntervalSeconds } if request.HealthCheckPath != nil { matchingTargetGroup.HealthCheckPath = request.HealthCheckPath } if request.HealthCheckPort != nil { matchingTargetGroup.HealthCheckPort = request.HealthCheckPort } if request.HealthCheckProtocol != nil { matchingTargetGroup.HealthCheckProtocol = request.HealthCheckProtocol } if request.HealthCheckTimeoutSeconds != nil { matchingTargetGroup.HealthCheckTimeoutSeconds = request.HealthCheckTimeoutSeconds } if request.HealthyThresholdCount != nil { matchingTargetGroup.HealthyThresholdCount = request.HealthyThresholdCount } if request.Matcher != nil { matchingTargetGroup.Matcher = request.Matcher } if request.UnhealthyThresholdCount != nil { matchingTargetGroup.UnhealthyThresholdCount = request.UnhealthyThresholdCount } } return &elbv2.ModifyTargetGroupOutput{ TargetGroups: dirtyGroups, }, nil } func (m *MockedFakeELBV2) DeleteTargetGroup(request *elbv2.DeleteTargetGroupInput) (*elbv2.DeleteTargetGroupOutput, error) { newTargetGroups := []*elbv2.TargetGroup{} for _, tg := range m.TargetGroups { if aws.StringValue(tg.TargetGroupArn) != aws.StringValue(request.TargetGroupArn) { newTargetGroups = append(newTargetGroups, tg) } } m.TargetGroups = newTargetGroups delete(m.RegisteredInstances, aws.StringValue(request.TargetGroupArn)) return &elbv2.DeleteTargetGroupOutput{}, nil } func (m *MockedFakeELBV2) DescribeTargetHealth(request *elbv2.DescribeTargetHealthInput) (*elbv2.DescribeTargetHealthOutput, error) { healthDescriptions := []*elbv2.TargetHealthDescription{} var matchingTargetGroup *elbv2.TargetGroup for _, tg := range m.TargetGroups { if aws.StringValue(tg.TargetGroupArn) == aws.StringValue(request.TargetGroupArn) { matchingTargetGroup = tg break } } if registeredTargets, present := m.RegisteredInstances[aws.StringValue(request.TargetGroupArn)]; present { for _, target := range registeredTargets { healthDescriptions = append(healthDescriptions, &elbv2.TargetHealthDescription{ HealthCheckPort: matchingTargetGroup.HealthCheckPort, Target: &elbv2.TargetDescription{ Id:   aws.String(target), Port: matchingTargetGroup.Port, }, TargetHealth: &elbv2.TargetHealth{ State: aws.String("healthy"), }, }) } } return &elbv2.DescribeTargetHealthOutput{ TargetHealthDescriptions: healthDescriptions, }, nil } func (m *MockedFakeELBV2) DescribeTargetGroupAttributes(*elbv2.DescribeTargetGroupAttributesInput) (*elbv2.DescribeTargetGroupAttributesOutput, error) { panic("Not implemented") } func (m *MockedFakeELBV2) ModifyTargetGroupAttributes(*elbv2.ModifyTargetGroupAttributesInput) (*elbv2.ModifyTargetGroupAttributesOutput, error) { panic("Not implemented") } func (m *MockedFakeELBV2) RegisterTargets(request *elbv2.RegisterTargetsInput) (*elbv2.RegisterTargetsOutput, error) { arn := aws.StringValue(request.TargetGroupArn) alreadyExists := make(map[string]bool) for _, targetID := range m.RegisteredInstances[arn] { alreadyExists[targetID] = true } for _, target := range request.Targets { if !alreadyExists[aws.StringValue(target.Id)] { m.RegisteredInstances[arn] = append(m.RegisteredInstances[arn], aws.StringValue(target.Id)) } } return &elbv2.RegisterTargetsOutput{}, nil } func (m *MockedFakeELBV2) DeregisterTargets(request *elbv2.DeregisterTargetsInput) (*elbv2.DeregisterTargetsOutput, error) { removeMe := make(map[string]bool) for _, target := range request.Targets { removeMe[aws.StringValue(target.Id)] = true } newRegisteredInstancesForArn := []string{} for _, targetID := range m.RegisteredInstances[aws.StringValue(request.TargetGroupArn)] { if !removeMe[targetID] { newRegisteredInstancesForArn = append(newRegisteredInstancesForArn, targetID) } } m.RegisteredInstances[aws.StringValue(request.TargetGroupArn)] = newRegisteredInstancesForArn return &elbv2.DeregisterTargetsOutput{}, nil } func (m *MockedFakeELBV2) CreateListener(request *elbv2.CreateListenerInput) (*elbv2.CreateListenerOutput, error) { accountID := 123456789 arn := fmt.Sprintf("arn:aws:elasticloadbalancing:us-east-1:%d:listener/net/%x/%x/%x", accountID, rand.Uint64(), rand.Uint32(), rand.Uint32()) newListener := &elbv2.Listener{ ListenerArn:     aws.String(arn), Port:            request.Port, Protocol:        request.Protocol, DefaultActions:  request.DefaultActions, LoadBalancerArn: request.LoadBalancerArn, } m.Listeners = append(m.Listeners, newListener) for _, tg := range m.TargetGroups { for _, action := range request.DefaultActions { if aws.StringValue(action.TargetGroupArn) == aws.StringValue(tg.TargetGroupArn) { tg.LoadBalancerArns = append(tg.LoadBalancerArns, request.LoadBalancerArn) break } } } return &elbv2.CreateListenerOutput{ Listeners: []*elbv2.Listener{newListener}, }, nil } func (m *MockedFakeELBV2) DescribeListeners(request *elbv2.DescribeListenersInput) (*elbv2.DescribeListenersOutput, error) { if len(request.ListenerArns) == 0 && request.LoadBalancerArn == nil { return &elbv2.DescribeListenersOutput{ Listeners: m.Listeners, }, nil } else if len(request.ListenerArns) == 0 { listeners := []*elbv2.Listener{} for _, lb := range m.Listeners { if aws.StringValue(lb.LoadBalancerArn) == aws.StringValue(request.LoadBalancerArn) { listeners = append(listeners, lb) } } return &elbv2.DescribeListenersOutput{ Listeners: listeners, }, nil } panic("Not implemented") } func (m *MockedFakeELBV2) DeleteListener(*elbv2.DeleteListenerInput) (*elbv2.DeleteListenerOutput, error) { panic("Not implemented") } func (m *MockedFakeELBV2) ModifyListener(request *elbv2.ModifyListenerInput) (*elbv2.ModifyListenerOutput, error) { modifiedListeners := []*elbv2.Listener{} for _, listener := range m.Listeners { if aws.StringValue(listener.ListenerArn) == aws.StringValue(request.ListenerArn) { if request.DefaultActions != nil { // for each old action, find the corresponding target group, and remove the listener's LB ARN from its list for _, action := range listener.DefaultActions { var targetGroupForAction *elbv2.TargetGroup for _, tg := range m.TargetGroups { if aws.StringValue(action.TargetGroupArn) == aws.StringValue(tg.TargetGroupArn) { targetGroupForAction = tg break } } if targetGroupForAction != nil { newLoadBalancerARNs := []*string{} for _, lbArn := range targetGroupForAction.LoadBalancerArns { if aws.StringValue(lbArn) != aws.StringValue(listener.LoadBalancerArn) { newLoadBalancerARNs = append(newLoadBalancerARNs, lbArn) } } targetGroupForAction.LoadBalancerArns = newLoadBalancerARNs } } listener.DefaultActions = request.DefaultActions // for each new action, add the listener's LB ARN to that action's target groups' lists for _, action := range request.DefaultActions { var targetGroupForAction *elbv2.TargetGroup for _, tg := range m.TargetGroups { if aws.StringValue(action.TargetGroupArn) == aws.StringValue(tg.TargetGroupArn) { targetGroupForAction = tg break } } if targetGroupForAction != nil { targetGroupForAction.LoadBalancerArns = append(targetGroupForAction.LoadBalancerArns, listener.LoadBalancerArn) } } } if request.Port != nil { listener.Port = request.Port } if request.Protocol != nil { listener.Protocol = request.Protocol } modifiedListeners = append(modifiedListeners, listener) } } return &elbv2.ModifyListenerOutput{ Listeners: modifiedListeners, }, nil } func (m *MockedFakeELBV2) WaitUntilLoadBalancersDeleted(*elbv2.DescribeLoadBalancersInput) error { panic("Not implemented") } func (m *MockedFakeEC2) maybeExpectDescribeSecurityGroups(clusterID, groupName string) { tags := []*ec2.Tag{ {Key: aws.String(TagNameKubernetesClusterLegacy), Value: aws.String(clusterID)}, {Key: aws.String(fmt.Sprintf("%s%s", TagNameKubernetesClusterPrefix, clusterID)), Value: aws.String(ResourceLifecycleOwned)}, } m.On("DescribeSecurityGroups", &ec2.DescribeSecurityGroupsInput{Filters: []*ec2.Filter{ newEc2Filter("group-name", groupName), newEc2Filter("vpc-id", ""), }}).Maybe().Return([]*ec2.SecurityGroup{{Tags: tags}}) m.On("DescribeSecurityGroups", &ec2.DescribeSecurityGroupsInput{}).Maybe().Return([]*ec2.SecurityGroup{{Tags: tags}}) } func TestNLBNodeRegistration(t *testing.T) { awsServices := newMockedFakeAWSServices(TestClusterID) awsServices.elbv2 = &MockedFakeELBV2{Tags: make(map[string][]elbv2.Tag), RegisteredInstances: make(map[string][]string), LoadBalancerAttributes: make(map[string]map[string]string)} c, _ := newAWSCloud(CloudConfig{}, awsServices) awsServices.ec2.(*MockedFakeEC2).Subnets = []*ec2.Subnet{ { AvailabilityZone: aws.String("us-west-2a"), SubnetId:         aws.String("subnet-abc123de"), Tags: []*ec2.Tag{ { Key:   aws.String(c.tagging.clusterTagKey()), Value: aws.String("owned"), }, }, }, } awsServices.ec2.(*MockedFakeEC2).RouteTables = []*ec2.RouteTable{ { Associations: []*ec2.RouteTableAssociation{ { Main:                    aws.Bool(true), RouteTableAssociationId: aws.String("rtbassoc-abc123def456abc78"), RouteTableId:            aws.String("rtb-abc123def456abc78"), SubnetId:                aws.String("subnet-abc123de"), }, }, RouteTableId: aws.String("rtb-abc123def456abc78"), Routes: []*ec2.Route{ { DestinationCidrBlock: aws.String("0.0.0.0/0"), GatewayId:            aws.String("igw-abc123def456abc78"), State:                aws.String("active"), }, }, }, } awsServices.ec2.(*MockedFakeEC2).maybeExpectDescribeSecurityGroups(TestClusterID, "k8s-elb-aid") nodes := []*v1.Node{makeNamedNode(awsServices, 0, "a"), makeNamedNode(awsServices, 1, "b"), makeNamedNode(awsServices, 2, "c")} fauxService := &v1.Service{ ObjectMeta: metav1.ObjectMeta{ Name: "myservice", UID:  "id", Annotations: map[string]string{ "service.beta.kubernetes.io/aws-load-balancer-type": "nlb", }, }, Spec: v1.ServiceSpec{ Ports: []v1.ServicePort{ { Name:       "http", Port:       8080, NodePort:   31173, TargetPort: intstr.FromInt(31173), Protocol:   v1.ProtocolTCP, }, }, SessionAffinity: v1.ServiceAffinityNone, }, } _, err := c.EnsureLoadBalancer(context.TODO(), TestClusterName, fauxService, nodes) if err != nil { t.Errorf("EnsureLoadBalancer returned an error: %v", err) } for _, instances := range awsServices.elbv2.(*MockedFakeELBV2).RegisteredInstances { if len(instances) != 3 { t.Errorf("Expected 3 nodes registered with target group, saw %d", len(instances)) } } _, err = c.EnsureLoadBalancer(context.TODO(), TestClusterName, fauxService, nodes[:2]) if err != nil { t.Errorf("EnsureLoadBalancer returned an error: %v", err) } for _, instances := range awsServices.elbv2.(*MockedFakeELBV2).RegisteredInstances { if len(instances) != 2 { t.Errorf("Expected 2 nodes registered with target group, saw %d", len(instances)) } } _, err = c.EnsureLoadBalancer(context.TODO(), TestClusterName, fauxService, nodes) if err != nil { t.Errorf("EnsureLoadBalancer returned an error: %v", err) } for _, instances := range awsServices.elbv2.(*MockedFakeELBV2).RegisteredInstances { if len(instances) != 3 { t.Errorf("Expected 3 nodes registered with target group, saw %d", len(instances)) } } } func makeNamedNode(s *FakeAWSServices, offset int, name string) *v1.Node { instanceID := fmt.Sprintf("i-%x", int64(0x02bce90670bb0c7cd)+int64(offset)) instance := &ec2.Instance{} instance.InstanceId = aws.String(instanceID) instance.Placement = &ec2.Placement{ AvailabilityZone: aws.String("us-east-1c"), } instance.PrivateDnsName = aws.String(fmt.Sprintf("ip-172-20-0-%d.ec2.internal", 101+offset)) instance.PrivateIpAddress = aws.String(fmt.Sprintf("192.168.0.%d", 1+offset)) var tag ec2.Tag tag.Key = aws.String(TagNameKubernetesClusterLegacy) tag.Value = aws.String(TestClusterID) instance.Tags = []*ec2.Tag{&tag} s.instances = append(s.instances, instance) testProviderID := "aws:///us-east-1c/" + instanceID return &v1.Node{ ObjectMeta: metav1.ObjectMeta{ Name: name, }, Spec: v1.NodeSpec{ ProviderID: testProviderID, }, } } </ins> func newMockedFakeAWSServices(id string) *FakeAWSServices { s := NewFakeAWSServices(id) s.ec2 = &MockedFakeEC2{FakeEC2Impl: s.ec2.(*FakeEC2Impl)}

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Using a pod like the nginx-ing-cntlr with an AWS Network Load Balancer (NLB), and then draining the node its running on to force a migration to another node pool, does not reliably deregister the drained nodes from the NLB Target group. This causes the nodeport used by the NLB to remain open, and the drained node ends up continuing to funnel traffic for other pods cause of being used. This leads to connection timeouts for clients, and downtime on migrations. Switching to a Classic ELB, instead of an NLB, with connection draining enabled does work, and the instances that are drained are practically removed immediately from the instances targeted by the ELB, as expected. There seems to be some sort of issue with NLB's as this is not happening, or happening rarely (only saw it once in many tests but couldn't recreate it). The node + LB set seems to be working as I see in the controller-mgr logs that the nodes are updated as I went from 11 nodes to 8, after attempting to drain + migrate a nodepool of 3 nodes with nginx-ing-cntlr all running on them: However, this update does not seem to be translating into deregistering instances from the NLB. And i've waited minutes & hours in case timeouts were playing a hand, but that didn't seem to help. What you expected to happen: On node draining, with being used, that the NLB would deregister the instances as expected. How to reproduce it (as minimally and precisely as possible): Launch a Deployment + NLB service on a node and drain it to force it over to another node pool. Watch the instances in the AWS NLB Target group never get out of "healthy" status or be drained + removed. Anything else we need to know?: Environment: AWS EKS Kubernetes version (use ): 1.12 & 1.13 Cloud provider or hardware configuration: AWS OS (e.g: ): AWS Linux 2 Kernel (e.g.

): 4.14.123-111.109.amzn2.x8664 Install tools: Network plugin and version (if this is a network-related bug): core-dns v1.2.6 Others:
/sig aws
Related:
/sig cloud-provider
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: "testing" v1 "k8s.io/api/core/v1" <del> apierrors "k8s.io/apimachinery/pkg/api/errors" </del> metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime" clientsetfake "k8s.io/client-go/kubernetes/fake" <del> core "k8s.io/client-go/testing" </del> kubeadmapi "k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm" kubeadmscheme "k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/scheme" kubeadmapiv1beta2 "k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta2"

What happened: Running fails consistently with timeout errors What you expected to happen: Unit tests should complete without any errors. If this was a configuration problem (e.g. unsupported go version), then I should see a meaningful error rather than just a timeout error. How to reproduce it (as minimally and precisely as possible): Run Anything else we need to know?: This issue has been present in my system for months. Environment: Kubernetes version (use ): master Cloud provider or hardware configuration: OS (e.g: ): Ubuntu 20.04 LTS (Focal Fossa) Kernel (e.g. ): Linux desktop 5.4.0-37-generic -Ubuntu SMP Wed Jun 3 18:57:02 UTC 2020 x8664 x8664 x86_64 GNU/Linux Install tools: Network plugin and version (if this is a network-related bug): Others: go version go1.13.12 linux/amd64 (snap)
/sig testing
/assign
Issues targeted at the specific problematic packages and responsible SIGs will be more effective in getting this addressed
/remove-sig testing Unless we expect some broad testing change (like reconfiguring go test) this isn't SIG testing. This is SIG storage, node, and cluster lifecycle
/sig node /sig cluster-lifecycle /sig storage tagged the right sigs, though per-package issues are far more likely to drive getting this fixed
fix for cmd/kubeadm/app/phases/uploadconfig:
I'm going to open two more issues for the other two packages and work on them
sounds good, in that case please close this one.
New issues for reconciler and csi packages: ,
Closing. This problem has come up in the past, the solution is to just increase the timeout. I missed this the first time I read the doc.


Question: var name string = p.Name if name == "" { <del> name = fmt.Sprintf("%s-%s:%d", family, p.Protocol, p.ContainerPort) </del> <ins> name = fmt.Sprintf("%s-%s-%s:%d:%d", family, p.Protocol, p.HostIP, p.ContainerPort, p.HostPort) </ins> } // Protect against a port name being used more than once in a container.

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: A container ports spec such as: Results in this error: From . Because of this, the port mappings are not setup correctly on the host when using a CNI-compliant portmap solution. What you expected to happen: Auto-generated port names should never result in duplicates How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: /sig node Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.7", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-01-16T21:59:57Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.11", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-04-05T17:16:46Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: VirtualBox OS (e.g. from /etc/os-release): Ubuntu Kernel (e.g. ): Linux node2 4.4.0-104-generic -Ubuntu SMP Mon Dec 11 12:16:42 UTC 2017 x8664 x8664 x86_64 GNU/Linux Install tools: kubeadm Others:
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with .

Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
/assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
I see that the problem is legit, but is this setup frequently used? why not using services instead of hostports? I mean, it will complicate more the solution but will provide a good abstraction and remove the dependency on the node.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
Why is this still an issue?
Issues go stale after 90d of inactivity.


Question: framework.ConformanceIt("should run a job to completion when tasks sometimes fail and are locally restarted", func() { ginkgo.By("Creating a job") // One failure, then a success, local restarts. <del> // We can't use the random failure approach used by the // non-local test below, because kubelet will throttle // frequently failing containers in a given pod, ramping // up to 5 minutes between restarts, making test timeouts // due to successive failures too likely with a reasonable // test timeout. </del> <ins> // We can't use the random failure approach, because kubelet will // throttle frequently failing containers in a given pod, ramping // up to 5 minutes between restarts, making test timeout due to // successive failures too likely with a reasonable test timeout. </ins> job := jobutil.NewTestJob("failOnce", "fail-once-local", v1.RestartPolicyOnFailure, parallelism, completions, nil, backoffLimit) job, err := jobutil.CreateJob(f.ClientSet, f.Namespace.Name, job) framework.ExpectNoError(err, "failed to create job in namespace: %s", f.Namespace.Name)

Which jobs are flaking: pull-kubernetes-e2e-kind Which test(s) are flaking: Testgrid link: Reason for failure: Anything else we need to know: /sig apps
Failure at
Hi I looked into the failures and think it was because the test required the job succeeds 3 times with pseudorandom 50% failure chance within 15 minutes, while there is an exponential back-off delay (10s, 20s, 40s …) capped at 6 minutes before recreating failed pods. As 7 consecutive failures (1/128 chance) could exceed the timeout, the test failed intermittently because of "timed out waiting for the condition". For example, in the failure of , the job pods succeeded and failed in the following orders: After the last failure at , the delay has increased to 640s so no pod was created any more before the test failed at . I think it can be fixed by using a deterministic approach to make the job pods succeed and fail alternately, and its executing time can be reduced to tens of seconds stably. Assigning myself. /assign