Question: finishedCondition = newCondition(batch.JobFailed, v1.ConditionTrue, "BackoffLimitExceeded", "Job has reached the specified backoff limit") } else if pastActiveDeadline(&job) { finishedCondition = newCondition(batch.JobFailed, v1.ConditionTrue, "DeadlineExceeded", "Job was active longer than specified deadline") <ins> } else if job.Spec.ActiveDeadlineSeconds != nil && !jobSuspended(&job) { syncDuration := time.Duration(*job.Spec.ActiveDeadlineSeconds)*time.Second - time.Since(job.Status.StartTime.Time) klog.V(2).InfoS("Job has activeDeadlineSeconds configuration. Will sync this job again", "job", key, "nextSyncIn", syncDuration) jm.queue.AddAfter(key, syncDuration) </ins> } var prevSucceededIndexes, succeededIndexes orderedIntervals

Prior to 1.24.0 setting in a Job object () would kill the Job and Pods if they ran longer than the integer specified. I tested today with 1.24.0 and it's not working. The Job defined below (for testing) should be killed 10 seconds, but isn't. It's actually being terminated after about 55 seconds. The Job to be killed after 10 seconds as per . It's worked like this in 1.23.x. On a cluster running 1.24.0 with the following YAML file called . Create the Job. On Kubernetes clusters earlier than 1.24.0 it would fail after 10 seconds and the following command should show in the events 10 seconds after the Pod is created. In K8s 1.24.0 it's terminated after about a minute instead of 10 seconds. It works as expected in 1.23.6 and other point versions of 1.23. It works as expected, and documented, in all versions of 1.23 I've tested on. Docs: <details</details<detailsAll and also local clusters. </details<detailsN/A </details<detailsN/A </details<detailsN/A </details<detailsN/A </details
/sig apps
I just searched for at locally and could not see any logic that compares running time with and marks a pod for termination. Do I need to look somewhere else?
It's here Here's the quote: "Another way to terminate a Job is by setting an active deadline. Do this by setting the field of the Job to a number of seconds. The applies to the duration of the job, no matter how many Pods are created. Once a Job reaches , all of its running Pods are terminated and the Job status will become type: Failed with reason: DeadlineExceeded." I may be misunderstanding it, but I don't think I am as it works the way described in older versions. The Job owns the Pod, you set on the Job to a value, let's say . This will cause the Job to be killed if it's still running after 10 seconds. As the Job owns the Pod(s) it cleans them up as well. This is exactly how it works in previous versions.

The job controller for sure has the code for cleanup on active deadline still in place +1 from me as well. The behavior for sure is not the same on 1.23.x vs 1.24.0 using the same job manifest. And the documentation v/s the behavior is not the same.
Seem to be a side effect of feature gate. Disabling that gives me the old behavior as in 1.23.x You can get the same behavior even in 1.23.x with the feature flag enabled.
The weird bit as you see is the pod actually completes but when it processes the completion event it notices the activeDeadline value and marks deadline Exceeded while the pod actually ran to completion
/assign
I think I have figured out why this happened. It is a side effect of feature gate (thanks ). : when we call with multiple times, the workqueue will only keep the duration that will be triggered nearest from now. (when there is no feature gate or do not enable it): job-controller mainly relies on the first time sync to add the job after its : (unless the field has been changed then it will be add after ). : the feature gate makes to be and it will be add after for the job when a pod of it has been updated. Then the duration which has been after in the first time sync will be overwrite to . And after the next syncJob, there will be no one to addAfter the job again. WDYT
I have came to the same conclusion as you are based on what is in the and . So, I did a dirty prototype of tracking the activeDeadline in an entirely different to avoid conflicts and it seem to be working well. If this is an acceptable change to and I would be more than happy to raise a Draft PR for review along with some additional UT to cover the active deadline cases
Emm... Usually it is not suggested to have multiple active queues in one controller, which will cause an object to be handled by multiple workers at the same time and also make the progress more complicated. My suggestion is do at the end of syncJob every time an unfinished job is reconciled, the duration should be . This is clear for everyone to understand, brings less change and no extra burden on the controller.
I was too trigger happy and did not think about the cases you mentioned above.

Thanks for the suggestion of the approach I will took a stab at your suggestion and here is a draft of the changes to get your suggestion working. I did a quick test and that seem to work fine. There were one too many termination point in call and the has no access to the object of the key. So, instead of reading it took a deferred approach as we already had named return parameters in place on the
Other approach is to use the and fetch the job via key if the is not set to This will do an additional fetch but much simpler in change
This is essentially the same as . The solution is actually very simple, but I didn't have time to implement it Can someone take it? /priority critical-urgent /triage accepted
/assign
Let me open a PR. Do you want to just add the item back to queue or retain the check for that I have in the above ?
just add it back to the queue every time there is a job sync. It's a noop if the item is already there.


Question: // Output: // clusters: //   alfa: <del> //     LocationOfOrigin: "" </del> //     certificate-authority: path/to/my/cert-ca-filename //     disable-compression: true //     insecure-skip-tls-verify: true //     server: https://alfa.org:8080 //   bravo: <del> //     LocationOfOrigin: "" </del> //     server: https://bravo.org:8080 // contexts: //   alfa-as-black-mage: <del> //     LocationOfOrigin: "" </del> //     cluster: alfa //     namespace: zulu //     user: black-mage-via-auth-provider //   alfa-as-white-mage: <del> //     LocationOfOrigin: "" </del> //     cluster: alfa //     user: white-mage-via-cert //   bravo-as-black-mage: <del> //     LocationOfOrigin: "" </del> //     cluster: bravo //     namespace: yankee //     user: black-mage-via-auth-provider

yaml.Marshal a kubeconfig, as done in the example tests here LocationOfOrigin is not seen in output, as comment says "never serialized." Click "run example" doesn't suffer from this as it does some conversion <details</details<details</details<details</details<details</details<details</details<details</details
/assign
/sig api-machinery
/assign /triage accepted
is what's known as internal type. It is not intended to be serialized by code. The serialization version is in and does not have this field. I could get behind removing all json tags from the internal version, but I wouldn't refine tags that shouldn't be used.


Question: c = f.ClientSet ns = f.Namespace.Name framework.ExpectNoError(framework.WaitForAllNodesSchedulable(c, framework.TestContext.NodeSchedulableTimeout)) <ins> }) setupFunc := func(allowExpansion bool, blockVolume bool) (*v1.PersistentVolumeClaim, *storage.StorageClass, error) { </ins> test := testsuites.StorageClassTest{ <del> Name:                 "default", ClaimSize:            "2Gi", AllowVolumeExpansion: true, </del> <ins> Name:      "default", ClaimSize: "2Gi", </ins> } <del> resizableSc, err = createStorageClass(test, ns, "resizing", c) framework.ExpectNoError(err, "Error creating resizable storage class") Expect(resizableSc.AllowVolumeExpansion).NotTo(BeNil()) Expect(*resizableSc.AllowVolumeExpansion).To(BeTrue()) pvc = newClaim(test, ns, "default") pvc.Spec.StorageClassName = &resizableSc.Name pvc, err = c.CoreV1().PersistentVolumeClaims(pvc.Namespace).Create(pvc) framework.ExpectNoError(err, "Error creating pvc") }) </del> <ins> if allowExpansion { test.AllowVolumeExpansion = true } if blockVolume { test.VolumeMode = v1.PersistentVolumeBlock } sc, err := createStorageClass(test, ns, "resizing", c) framework.ExpectNoError(err, "Error creating storage class for resizing") tPVC := newClaim(test, ns, "default") tPVC.Spec.StorageClassName = &sc.Name tPVC, err = c.CoreV1().PersistentVolumeClaims(tPVC.Namespace).Create(tPVC) if err != nil { return nil, sc, err } return tPVC, sc, nil } </ins> AfterEach(func() { framework.ExpectNoError(framework.DeletePersistentVolumeClaim(c, pvc.Name, pvc.Namespace)) <del> framework.ExpectNoError(c.StorageV1().StorageClasses().Delete(resizableSc.Name, nil)) </del> <ins> framework.ExpectNoError(c.StorageV1().StorageClasses().Delete(storageClassVar.Name, nil)) </ins> }) It("should not allow expansion of pvcs without AllowVolumeExpansion property", func() { <del> test := testsuites.StorageClassTest{ Name:      "no-expansion", ClaimSize: "2Gi", } regularSC, err := createStorageClass(test, ns, "noexpand", c) framework.ExpectNoError(err, "Error creating non-expandable storage class") </del> <ins> pvc, storageClassVar, err = setupFunc(false /* allowExpansion */, false /*BlockVolume*/) framework.ExpectNoError(err, "Error creating non-expandable PVC") </ins> <del> defer func() { framework.ExpectNoError(c.StorageV1().StorageClasses().Delete(regularSC.Name, nil)) }() Expect(regularSC.AllowVolumeExpansion).To(BeNil()) </del> <ins> Expect(storageClassVar.AllowVolumeExpansion).To(BeNil()) </ins> <del> noExpandPVC := newClaim(test, ns, "noexpand") noExpandPVC.Spec.StorageClassName = &regularSC.Name noExpandPVC, err = c.CoreV1().PersistentVolumeClaims(noExpandPVC.Namespace).Create(noExpandPVC) framework.ExpectNoError(err, "Error creating pvc") defer func() { framework.ExpectNoError(framework.DeletePersistentVolumeClaim(c, noExpandPVC.Name, noExpandPVC.Namespace)) }() pvcClaims := []*v1.PersistentVolumeClaim{noExpandPVC} </del> <ins> pvcClaims := []*v1.PersistentVolumeClaim{pvc} </ins> pvs, err := framework.WaitForPVClaimBoundPhase(c, pvcClaims, framework.ClaimProvisionTimeout) framework.ExpectNoError(err, "Failed waiting for PVC to be bound %v", err) Expect(len(pvs)).To(Equal(1)) By("Expanding non-expandable pvc") newSize := resource.MustParse("6Gi") <del> noExpandPVC, err = expandPVCSize(noExpandPVC, newSize, c) </del> <ins> pvc, err = expandPVCSize(pvc, newSize, c) </ins> Expect(err).To(HaveOccurred(), "While updating non-expandable PVC") }) It("Verify if editing PVC allows resize", func() { <ins> pvc, storageClassVar, err = setupFunc(true /* allowExpansion */, false /*BlockVolume*/) framework.ExpectNoError(err, "Error creating non-expandable PVC") </ins> By("Waiting for pvc to be in bound phase") pvcClaims := []*v1.PersistentVolumeClaim{pvc} pvs, err := framework.WaitForPVClaimBoundPhase(c, pvcClaims, framework.ClaimProvisionTimeout)

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: I created a block EBS volume using dynamic provisioning. Then I resized the volume from 10GB to 20 GB. Both PV and block volume reflects the change, but the PVC's capacity is still 10GB. This could be caused by in-tree resize put the PVC condition under state, but no one is acting on the condition since it is block volume: /cc /sig storage What you expected to happen: PVC's capacity should be updated to 20GB How to reproduce it (as minimally and precisely as possible): Easy to reproduce using following specs: StorageClass: Claim: Pod: Anything else we need to know?: This will affect whether CSI migration should handle migration scenario for block volume resizing. Environment: Kubernetes version (use ):1.13 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: kops Network plugin and version (if this is a network-related bug): Others:
/sig aws
/assign


Question: return finalURL } <ins> // finalURLTemplate is similar to URL(), but will make all specific parameter values equal // - instead of name or namespace, "{name}" and "{namespace}" will be used, and all query // parameters will be reset. This creates a copy of the url so as not to change the // underlying object. func (r Request) finalURLTemplate() url.URL { newParams := url.Values{} v := []string{"{value}"} for k := range r.params { newParams[k] = v } r.params = newParams u := r.URL() if u == nil { return url.URL{} } segments := strings.Split(u.Path, "/") groupIndex := 0 index := 0 trimmedBasePath := "" if r.c.base != nil && strings.Contains(u.Path, r.c.base.Path) { p := strings.TrimPrefix(u.Path, r.c.base.Path) if !strings.HasPrefix(p, "/") { p = "/" + p } // store the base path that we have trimmed so we can append it // before returning the URL trimmedBasePath = r.c.base.Path segments = strings.Split(p, "/") groupIndex = 1 } if len(segments) <= 2 { return *u } const CoreGroupPrefix = "api" const NamedGroupPrefix = "apis" isCoreGroup := segments[groupIndex] == CoreGroupPrefix isNamedGroup := segments[groupIndex] == NamedGroupPrefix if isCoreGroup { // checking the case of core group with /api/v1/... format index = groupIndex + 2 } else if isNamedGroup { // checking the case of named group with /apis/apps/v1/... format index = groupIndex + 3 } else { // this should not happen that the only two possibilities are /api... and /apis..., just want to put an // outlet here in case more API groups are added in future if ever possible: // https://kubernetes.io/docs/concepts/overview/kubernetes-api/#api-groups // if a wrong API groups name is encountered, return the {prefix} for url.Path u.Path = "/{prefix}" u.RawQuery = "" return *u } // switch segLength := len(segments) - index; segLength { switch { // case len(segments) - index == 1: // resource (with no name) do nothing case len(segments)-index == 2: // /$RESOURCE/$NAME: replace $NAME with {name} segments[index+1] = "{name}" case len(segments)-index == 3: if segments[index+2] == "finalize" || segments[index+2] == "status" { // /$RESOURCE/$NAME/$SUBRESOURCE: replace $NAME with {name} segments[index+1] = "{name}" } else { // /namespace/$NAMESPACE/$RESOURCE: replace $NAMESPACE with {namespace} segments[index+1] = "{namespace}" } case len(segments)-index >= 4: segments[index+1] = "{namespace}" // /namespace/$NAMESPACE/$RESOURCE/$NAME: replace $NAMESPACE with {namespace},  $NAME with {name} if segments[index+3] != "finalize" && segments[index+3] != "status" { // /$RESOURCE/$NAME/$SUBRESOURCE: replace $NAME with {name} segments[index+3] = "{name}" } } u.Path = path.Join(trimmedBasePath, path.Join(segments...)) return *u } </ins> func (r *Request) tryThrottleWithInfo(ctx context.Context, retryInfo string) error { if r.rateLimiter == nil { return nil

Hello, we have been using client-go to measure the latency for all client-go traffic from Cilium to kube-apiserver. We have that since v1.24.0 the URL path is no longer aggregated in a way that this cardinality would no longer happen as presented in the following image: <img width="330" alt="Screen Shot 2022-08-08 at 10 18 24 AM" src="https://user-"It seems this functionality was removed as a part of as it seems that PR only took into account that users of the would only be using it to measure latency to a particular host and not for a particular URL path as we are doing in . Would it be reasonable to revert commit in order to fix this regression? Thanks cc
/sig api-machinery /sig instrumentation
But reverting will bring back the cardinality problem, will not? The latency of each URL path is available in the apiserver metrics ...
We have never any cardinality problem with the previous code since we were not using the "host" field which is what I assume the original issue about cardinality was trying to fix. We only used the "path" of an URL, which was aggregated in a way that would not be facing any cardinality, like the figure attached in this GH description. Now, starting with = 1.24.0, each and is actually a different value which does cause the cardinality problem to appear.
ah, I see the disconnection, I wasn't understanding the issue sorry. The Interface uses the url as parameter to keep backwards compatibility, but the method that gathers the metrics only uses the host to avoid the cardinality problem. You have to update the method that gathers the metrics to only use the host part of the url
Filed
should fix the code in kubernetes ? however, that will break consumer of the Interface that still wants to process the path, but we know using the path will cause issues with cardinality :shrug:
I would be fine with adding a new label to mitigate that issue since originally the cardinality explosion that was reported was caused by the query string and not the path: When trimming down the whole URL to only the host we even had a discussion to keep the path, but we decided not to since we didn't have any use case for this data.

If you have a use case where you need the path in order to get more granularity on the latency of your network then I would be fine with adding the label to Kubernetes, but another path forward could be to also remove the path information from cilium if deemed unnecessary. In the most common scenarios, you will only need the host to get the network latency based on different proxy routes.
yeah we can't really do much about it, it is either we leak the unsanitized paths or we prevent the users from using the path completely. Since we want to not break the interface and keep the URL parameter, we might just have to keep sanitizing the paths for our consumers even though we are not using the path anymore.
IIUIC the RESTClient is used in the dynamic client-go too, for these users with 1900 CRDs, the path is really going to explode for them
Then it seems that the best option to not break users would be to revert , but at the same time I don't really like the idea of spending CPU time sanitizing something that will never be used in the RESTClient
A suggestion would be to create a new interface similar to the that would have the following method: Thus, those users with 1900 CRDs would use the instead of the .
after digging a little bit the underlying issue it looks that the problem was initially caused by a kubelet bug as pointed out . The dependents of client-go don't have access to all the fields of the struct which uses heavily to generate the aggregated URL.
well, it is clear now that current status is wrong, we pass the whole URL to the interface and rely on the to use only the Host part of the URL :smile: I'm fine with whatever you decide, to restore or just completely strip the Path Sorry for the unfortunate consequence, at that time I didn't consider that the Interface/Method drift and thought that function was orphan
/triage accepted /assign


Question: // Temp clear time stamps for comparison because actual values don't matter for comparison recvEvent.FirstTimestamp = expectedEvent.FirstTimestamp recvEvent.LastTimestamp = expectedEvent.LastTimestamp <ins> recvEvent.ReportingController = expectedEvent.ReportingController </ins> // Check that name has the right prefix. if n, en := recvEvent.Name, expectedEvent.Name; !strings.HasPrefix(n, en) { t.Errorf("%v - Name '%v' does not contain prefix '%v'", messagePrefix, n, en)

From the Deprecated API Migration Guide: , since v1.19, the has been using the new reportingController and reportingInstance to replace the decrecated source.component and But when I checked the Kube events in a cluster with version 1.26.0, those new fields are empty and the supposed to be deprecated fields have not been renamed. Seeing the values in the new fields like reportingController and reportingInstance an Kubernetes cluster with version 1.26.0 some Kube events the Kube events details for the field of reportingController, reportingInstance, source.component, No response <detailsServer Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2022-12-11T00:15:47Z", GoVersion:"go1.19.4", Compiler:"gc", Platform:"linux/amd64"} </details<detailsAzure </details<details</details<details</details<details</details<details</details
/sig instrumentation
I have also encountered this issue in my local 1.26.0 Kubernetes cluster, which does not align with the deprecation guide described in I believe I can investigate this further
/assign
Thank you for making the PR to fix this! Also a comment in the PR for the doc update.
/triage accepted


Question: <ins> apiVersion: v1 kind: Secret metadata: name: test </ins>

What happened: I get a diff, when I specify a file directly, but on recursive mode, no diff is shown anymore: What you expected to happen: Recursive mode should work How to reproduce it (as minimally and precisely as possible): Have a bunch of YAML manifest, but with name collisions, e.g. Service , Deployment , etc Anything else we need to know?: I think the problem is that all objects have the same name, which should not be a problem per se. Only heapster-binding has a different name, and looking at the generated temp files, only and are present: Environment: Kubernetes version (use ): $ kubectl version --short Client Version: v1.13.0 Server Version: v1.13.0 <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
/sig cli
I think the files are overwriting each other?
Yeah, good catch, I'll try to fix that asap
Thanks! By the way, very nice work with dry-run and diff. I was waiting for this.
Feel free to look at the PR and give your opinion!


Question: defer r.mu.Unlock() r.Service = new(api.Service) <del> *r.Service = *svc </del> <ins> clone, err := api.Scheme.DeepCopy(svc) if err != nil { return nil, err } r.Service = clone.(*api.Service) </ins> r.List.Items = append(r.List.Items, *svc) return svc, r.Err }

~L60 does a shallow copy, and ends up aliasing slices.
Is this bug still valid?
yes, but now it in L78: This is not a huge deal, but it is wrong. It should call
ok I'll fix it


Question: return p.Status.Phase == v1.PodSucceeded }) failedPods = getValidPodsWithFilter(jobCtx, jobCtx.uncounted.Failed(), func(p *v1.Pod) bool { <del> if feature.DefaultFeatureGate.Enabled(features.JobPodFailurePolicy) && jobCtx.job.Spec.PodFailurePolicy != nil { if !isPodFailed(p, jobCtx.job) { return false } _, countFailed, _ := matchPodFailurePolicy(jobCtx.job.Spec.PodFailurePolicy, p) return countFailed } else { return isPodFailed(p, jobCtx.job) } </del> <ins> return isPodFailed(p, jobCtx.job) </ins> }) return succeededPods, failedPods }

Pod backoff calculation excludes pods that match an rule in the failure policy: This could lead to fast retries, overwhelming the control plane, if most failures match an rule. The semantics of is not to count towards the limit, but the backoff should still be respected. /sig apps Recreations to be throttled. a Job with failure policy that matches every kind of failure. a container that always fails You would observe immediate recreation. No response_ 27+ <detailsN/A </details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
cc


Question: <ins> /* Copyright 2014 Google Inc. All rights reserved. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package cloudprovider import ( "testing" ) func TestGetRegion(t *testing.T) { gce := &GCECloud{ zone: "us-central1-b", } zones, ok := gce.Zones() if !ok { t.Fatalf("Unexpected missing zones impl") } zone, err := zones.GetZone() if err != nil { t.Fatalf("unexpected error %v", err) } if zone.Region != "us-central1" { t.Errorf("Unexpected region: %s", zone.Region) } } </ins>

Comments that the PR bot adds solely to trigger email notification of a label change, should be immediately deleted by the bot after it adds them. I have no idea what team to assign this to... cc/
, David Oppenheimer wrote:
(Motivation, of course, is to avoid having PR threads that are 90% bot comments...)
should really be an issue over in that's where (some of) the bots live. Lets make a list of which comments we want deleted immediately. [x] Label size. [x] Needs rebase. [x] test this stale CI [x] test this pending CI [x] PR changed after LGTM, removing LGTM. [x] Comment about blocking based on the paths touched (old docs) [x] unapproved cherrypick comment [x] cherrypicks must set a milestone [x] ok to test from bot I don't want to make it all comments. There are way too many times people have asked me "Why did the bot add 'do-not-merge'?" and I can just link them the comment 2 lines above which tells exactly why....
"Test this" with an issue is also useful, isn't it? On Apr 7, 2016 6:30 PM, "Eric Paris" wrote:
OK to test Add to whitelist On Apr 7, 2016 6:30 PM, "Eric Paris" wrote:
I was envisioning deleting the by the bot. (every 48 hours after LGTM, when retesting right before merge) not those by users (with an issue) After LGTM the bot will add , that one I can remove. The request to comes from the jenkins bot. And in that I am clueless...
any interest in a review (I'm trying to not just self merge all my own bot changes). This deleted hundreds and hundreds of comments.
Looks like reviewed it since your last message:

The contains instructions for downloading kubectl. The doesn't contain these instructions but it should if it's now the recommended way of running Kubernetes locally.
+1 Just ran into this.
cc
By the way, we encourage anyone to propose improvements/fixes to docs. You can do so pretty easily by clicking the pencil icon in the top right corner of every page of
kubectl download instructions to the guide in


Question: // case 0 { devAddresses: map[string][]string{"eth0": {"1.2.3.4"}, "lo": {"127.0.0.1"}}, <del> expectIPs:    []string{"1.2.3.4", "127.0.0.1"}, </del> <ins> expectIPs:    []string{"1.2.3.4"}, </ins> }, // case 1 { devAddresses: map[string][]string{"lo": {"127.0.0.1"}}, <del> expectIPs:    []string{"127.0.0.1"}, </del> <ins> expectIPs:    []string{}, </ins> }, // case 2 {

What happened: For services with type NodePort (and LoadBalancer) ipvs entries for localhost (ipv4 and ipv6) are created. Here is an example for a service with so the same nodeport is used for ipv4 and ipv6; The localhost entries can't be used and most likely they never will. What you expected to happen: Since "::1" most likely will never work and 127.0.0.1 will not work without an unlikely Linux kernel update these entries must not be created. Further, since the use of localhost addresses for nodeports is the only know reason to require (), this requirement shall be removed for proxy-mode=ipvs. How to reproduce it (as minimally and precisely as possible): Start a cluster with proxy-mode=ipvs and some services Do; Anything else we need to know?: That ipv6 access to [::1]:nodeport does not work (and most likely never will) is discussed in For 127.0.0.1 is is a bit more complicated as discussed in , with the excellent deep-dive into the kernel by in This may seem like a minor issue but clean-up of useless things is always good. I also have (vague) plans to propose improvements for the address handling for proxy-mode=ipvs and the localhost entries becomes an unwanted problem. If this issue is accepted as a bug I'll be happy to do a PR. Environment: Kubernetes version (use ): all supporting proxy-mode=ipvs Cloud provider or hardware configuration: any OS (e.g: ): any Kernel (e.g. ): any Install tools: none Network plugin and version (if this is a network-related bug): any Others: /kind cleanup /sig network /area ipvs
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
test in ipv6 + iptables , ::1 is not work for NodePort. /assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with .

Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/area kube-proxy
/remove lifecycle-stale
/remove-lifecycle stale <sigh...
Since it is highly unlikely that the kernel will be updated to allow this for ipv4 and it will never work for ipv6 I would really want to remove this from the ipvs proxier.
Do you think I can go ahead and disable the setup for localhost node-ports in the ipvs proxier? Since it have never worked it would not be a user facing change. Or should we discuss it on the sig/network meeting?
I personally think that this was implicitly discussed and confirmed here , it is an implementation detail of IPVS and a known issue that never worked in IPVS, so it will be safe to disable directly without further discussion. However, I defer to for a final decision /assign
I made a PR. ~It's just 3 lines~ Now 8 lines. I forgot the config setting.


Question: }, } for _, tc := range cases { <del> testNamespace := "test_projected_namespace" tc.configMap.ObjectMeta = metav1.ObjectMeta{ Namespace: testNamespace, Name:      tc.name, } </del> <ins> t.Run(tc.name, func(t *testing.T) { testNamespace := "test_projected_namespace" tc.configMap.ObjectMeta = metav1.ObjectMeta{ Namespace: testNamespace, Name:      tc.name, } </ins> <del> source := makeProjection(tc.name, tc.mode, "configMap") source.Sources[0].ConfigMap.Items = tc.mappings source.Sources[0].ConfigMap.Optional = &tc.optional testPodUID := types.UID("test_pod_uid") pod := &v1.Pod{ObjectMeta: metav1.ObjectMeta{Namespace: testNamespace, UID: testPodUID}} client := fake.NewSimpleClientset(tc.configMap) _, host := newTestHost(t, client) var myVolumeMounter = projectedVolumeMounter{ projectedVolume: &projectedVolume{ sources: source.Sources, podUID:  pod.UID, plugin: &projectedPlugin{ host:         host, getConfigMap: host.GetConfigMapFunc(), </del> <ins> source := makeProjection(tc.name, tc.mode, "configMap") source.Sources[0].ConfigMap.Items = tc.mappings source.Sources[0].ConfigMap.Optional = &tc.optional testPodUID := types.UID("test_pod_uid") pod := &v1.Pod{ObjectMeta: metav1.ObjectMeta{Namespace: testNamespace, UID: testPodUID}} client := fake.NewSimpleClientset(tc.configMap) tempDir, host := newTestHost(t, client) defer os.RemoveAll(tempDir) var myVolumeMounter = projectedVolumeMounter{ projectedVolume: &projectedVolume{ sources: source.Sources, podUID:  pod.UID, plugin: &projectedPlugin{ host:         host, getConfigMap: host.GetConfigMapFunc(), }, </ins> }, <del> }, source: *source, pod:    pod, } </del> <ins> source: *source, pod:    pod, } </ins> <del> actualPayload, err := myVolumeMounter.collectData() if err != nil && tc.success { t.Errorf("%v: unexpected failure making payload: %v", tc.name, err) continue } if err == nil && !tc.success { t.Errorf("%v: unexpected success making payload", tc.name) continue } if !tc.success { continue } if e, a := tc.payload, actualPayload; !reflect.DeepEqual(e, a) { t.Errorf("%v: expected and actual payload do not match", tc.name) } </del> <ins> actualPayload, err := myVolumeMounter.collectData() if err != nil && tc.success { t.Errorf("%v: unexpected failure making payload: %v", tc.name, err) return } if err == nil && !tc.success { t.Errorf("%v: unexpected success making payload", tc.name) return } if !tc.success { return } if e, a := tc.payload, actualPayload; !reflect.DeepEqual(e, a) { t.Errorf("%v: expected and actual payload do not match", tc.name) } }) </ins> } }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: I run the unit tests for , but I found it would not clean the tempdir it created. ! What you expected to happen: automatically clean up when finishing unit tests. How to reproduce it (as minimally and precisely as possible): go test $ Anything else we need to know?: Environment: k8s 1.15 Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig storage /sig testing
/assign Assign the issue to you since you have fired a pull request.


Question: subjectAltName = @alt_names [alt_names] IP.1 = 127.0.0.1 <ins> DNS.1 = webhook-test.default.svc </ins> EOF cat > client.conf << EOF

xref: , The package is failing tests and doesn't seem to work as expected when run under go1.9. It throws some x509 certificate errors. However, the same tests run successfully under go1.8. go1.8 successful output: go1.9 failing output: /priority important-soon
/assign


Question: } legacyregistry.MustRegister(RunPodSandboxDuration) legacyregistry.MustRegister(RunPodSandboxErrors) for _, collector := range collectors { legacyregistry.CustomMustRegister(collector) }

In commit we podresources-related metrics, to make sure the feature was monitoring-friendly. Because of a implementation bug, the metrics have been , they are updated correctly, but they are never registered, so they don't show up when doing like the podresources related metrics should be reported in the endpoint in the kubelet the kubelet, even the simplistic environment of the tests is sufficient sure to enable the and feature gates (on by default in kube = 1.25 at least) the podresources endpoint issuing and calls. The actual output is not relevant the metrics: The bug was introduced in the original commit <details</details<detailsNot relevant. </details<detailsNot relevant </details<detailsNot relevant </details<detailsNot relevant </details<detailsNot relevant </details
/sig node /assign
/cc /triage-accepted
/triage accepted /cc


Question: createRequest.Scheme = aws.String("internal") } <ins> var allocationIDs []string if eipList, present := annotations[ServiceAnnotationLoadBalancerEIPAllocations]; present { allocationIDs = strings.Split(eipList, ",") if len(allocationIDs) != len(subnetIDs) { return nil, fmt.Errorf("Error creating load balancer: Must have same number of EIP AllocationIDs (%d) and SubnetIDs (%d)", len(allocationIDs), len(subnetIDs)) } } </ins> // We are supposed to specify one subnet per AZ. // TODO: What happens if we have more than one subnet per AZ? <del> createRequest.SubnetMappings = createSubnetMappings(subnetIDs) </del> <ins> createRequest.SubnetMappings = createSubnetMappings(subnetIDs, allocationIDs) </ins> for k, v := range tags { createRequest.Tags = append(createRequest.Tags, &elbv2.Tag{

Is this a BUG REPORT or FEATURE REQUEST?: /kind feature /sig aws Anything else we need to know?: AWS NLBs support static IP addresses by associating an EIP per AZ. An already allocated EIP can be supplied with the . A user can then supply EIP allocation IDs in the service definition of kubernetes. By that the service will be available via those static IPs.
eip's for white listing are really important in the enterprise, we need this feature now, as a result we have to manually create nlb's, would really like to see this
This is also important for hosting providers, who might have huge numbers of apex domain DNS records pointed at EIPs. Associating an already-allocated EIP with a new Kubernetes-generated NLB would be huge!
This feature would save us weeks of work reimplementing the cloud controller just so we can preserve a special ELB that has static IP addresses... I really wish it was available
I'd like to contribute time to working on this. Is anyone else actively working on adding support?
That's awesome news. Thanks for working on this. Do you need assistance in testing this in an AWS environment? I might help out with that ...
Sure, the PR has been approved for testing so the binaries are available. I'm moving the code into our soak environment to get it some more exposure this week.
I've been trying with: But I only go a NLB without the EIPs deployed. Didn't find any errors in the logs. Do I have to enable a feature flag for it or kind of?
The reviewer found a typo where I may not be collecting the correct annotation. I'm waiting for the new build to complete and then rolling to our soak test. I'm passing through any AWS error and also error if the number of EIP and Subnets don't match. Let me get it online with the changes from code review and I'll dig in deeper. Also, are you using the binaries directly from the build system or just Kubernetes 1.12?
I checked with the fixes you provided yesterday. Works :) Created a service which spawns over 3AZs and supplied 3 EIP allocations. NLBs gets created correctly.
Can you also comment on the PR? I'm sure having multiple successful tests would be helpful to the reviewer.
Issues go stale after 90d of inactivity.

Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Hey, thanks for doing this, I was just looking for a way to associate one eip to a nlb, one question though why do you need one eip for each AZ? now the nlb creates a 1 public IP that balance to 3 AZ using traefik in my case when I deploy the service, thanks
Due to the fact that EIPs are represented by network hardware in AWS's datacenter you'd like to have redundant instances. If you run with only one EIP and the AZ of this EIP goes down, you won't be able to reach the (still up and running) traefik deployment in the other two AZs.
Hi, I've seen the PR () has been waiting for an update for three months. do you need a hand? do you want me to pick it up? I am really interested on this to be merged :smile:
Can you help write the tests, ? I'm completely lost there to be honest.
Hi I've found out that test-coverage on the aws provider is low and there is no tests on the so I can't find tests on NLB. I am going to open a new issue and work on improving coverage for that and hopefully I'll find out how to add tests for the new annotation. As you can see below current coverage for and is low and can't find tests for NLB:
Thanks for the help. I thought I was just missing being able to locate them.
+1 need this ASAP We are hosted on EKS so only official version supported. Any idea when this will be merged?
.... As a workaround, I'm just manually creating a second NLB (with EIPs) and changing the listeners to forward the traffic to the automatically created target groups (from the original svc-linked NLB).
Waiting for this as well on EKS. Thanks!
would be great to have that in place. thanks! I can confirm that the manual workaround by works fine, but it is hard to automate that flow.
+1 Waiting for this as well on EKS.

+1
I have created a node project that runs as a cronjob and monitors a SOURCE target group and clones the targets to a DESTINATION target group. So when you create a LoadBalancer of type 'nlb' in kubernetes, all you need to do is manually create a new 'nlb' in AWS Console, assign your EIP to your NLB for a specific 'subnet'(availability zone), then create a target group with no targets; use the ARN from this target group as the destination and use the ARN of the kubernetes target group as the SOURCE. More information here: Public docker image:
I have hard time getting this working with NLB using ingress controller : Even subnets are not taking effect here , its not passing my configurations in the API that creates the NLB kind: Service apiVersion: v1 metadata: name: ingress-nginx namespace: ingress-nginx labels: ingress-nginx ingress-nginx annotations: nlb "eipalloc-,eipalloc-,eipalloc-" "subnet-,subnet-" spec: type: LoadBalancer selector: ingress-nginx ingress-nginx ports: name: http port: 80 targetPort: http name: https port: 443 targetPort: https =======================================================``
I am also getting the same error can anyone help me with this ??
what version of k8s you are using ? this is not supported before 1.15 as far as I know so my issue was that I tried to implement on EKS
Hi i am using the latest version of k8 1.15 .5
I am not using EKS with my solution but the problem is the same as you mentioned
New to EKS. Could you elaborate how the connectivity from NLB that we have created manually (as you mentioned ) to the ingress-nginx pod?
Has it worked for you in k8s 1.15 EKS?
How to get Allocation IDs?
my elastic eip ids cannot be found at service creation, so it just sits in pending forever. if I resort to listing the subnets on their own annotation, (not using subnet discovery) same thing. no matter what I do it seems to not find me elastic ip addresses. Are you sure nlb type is supported to work with Elastic IPs?

From the AWS docs it seem they are pretty heavily trying to discourage use of the nlb type (Cloud Provider Load Balancer) in favor of pushing the new AWS Load balancer.
See available annotations: https://kubernetes- If you have done that: what do your aws-lb-controller logs tell you? I guess there’s an access problem with your iam role. Is your aws-lb-controller iam autorized to list and modify eip’s? Are the eip annotations exactly correct, can they be found by the controller? The controller’s log stream should have some kind of an error there. I have successfully assigned eips using service annotations to provision an aws nlb, but pay attention, you need to install the aws-lb-controller yourself. The kubernetes built-in aws library does not support eips.
Here is the annotation needed for assigning a static Elastic IP to your NLB using the AWS Load Balancer (Ingress) Controller https://kubernetes- New Controller link https://kubernetes-


Question: fi fi <ins> if [[ "${CLUSTER_DNS_CORE_DNS:-}" == "true" ]]; then update-coredns-config fi </ins> echo "== Validating cluster post-upgrade ==" "${KUBE_ROOT}/cluster/validate-cluster.sh"

Which jobs are failing: Multiple jobs on . Which test(s) are failing: on the following jobs: Since when has it been failing: Since 5/17. Testgrid links: See above. Reason for failure: All the failures mentioned above seem to have to have the same root cause, namely failure while waiting for CoreDNS to be ready. Example: /sig testing /priority critical-urgent /kind failing test /milestone v1.15 /cc
Some of this (possible all of it) is due to , which is dependent on being merged.
/sig cluster-lifecycle
Not the gce jobs thoght. Only the kinder job. So the gce jobs are failing with a problem outside of kubeadm/kinder but possibly still related to the corefile plugin change
/assign (At kubecon this week)
The gce failures appear to show the same behavior, but with kube-up instead of kubeadm. I.e. it appears that kube-up may also be retaining old configs when upgrading/downgrading. If thats true, then we may need to integrate the migration tools into kube-up as well.
I have opened which aims to fix all the failing tests except , which will be fixed via
Great job so far and Would it be possible to ask for the open PRs to be merged and/or a solution found soon? We have these tests failing for quite some time and code freeze coming up in the next days. Thank you in advance for your efforts!
thanks for the heads up. let me know if you don't have the time for the refactor that i requested and i can try to take over there. the alternative is to rollback the coredns version which i assume is not ideal, due to a variety of fixes in the new version.
is there any more work planned for this issue? I see that some of the kinder jobs are still failing after merged:
merging and will resolve and close this issue.
Thank you for the update !
updates: kinder tests were fixed via
/unassign i was here for: it's a bit flaky (which is due to the tooling), but green.
Since and we're moved to 1.16, and we're waiting for to merge, can we move this issue to 1.16?
? is that the right PR?

Good catch, I meant
The GCE test failures were fixed by and /close
Closing this issue. <detailsIn response to : The GCE test failures were fixed by and /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: } // Inject error to underlying layer and check if cacher is not bypassed. <del> backingStorage.err = errDummy </del> <ins> backingStorage.injectError(errDummy) </ins> err = cacher.GetList(context.TODO(), "pods/ns", storage.ListOptions{ ResourceVersion: "0", Predicate:       pred,

As Jordan pointed out in: watchcache treats RV=0 and RV="" the same way, even though it's not what we document: This should be fixed. /kind bug /sig api-machinery /priority important-soon
/assign
/triage accepted


Question: // SetCRDCondition sets the status condition. It either overwrites the existing one or creates a new one. func SetCRDCondition(crd *CustomResourceDefinition, newCondition CustomResourceDefinitionCondition) { <ins> newCondition.LastTransitionTime = metav1.NewTime(time.Now()) </ins> existingCondition := FindCRDCondition(crd, newCondition.Type) if existingCondition == nil { <del> newCondition.LastTransitionTime = metav1.NewTime(time.Now()) </del> crd.Status.Conditions = append(crd.Status.Conditions, newCondition) return } <del> if existingCondition.Status != newCondition.Status { existingCondition.Status = newCondition.Status </del> <ins> if existingCondition.Status != newCondition.Status || existingCondition.LastTransitionTime.IsZero() { </ins> existingCondition.LastTransitionTime = newCondition.LastTransitionTime } <ins> existingCondition.Status = newCondition.Status </ins> existingCondition.Reason = newCondition.Reason existingCondition.Message = newCondition.Message }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: the for a CRDs Established condition is always null What you expected to happen: it should have a timestamp How to reproduce it (as minimally and precisely as possible): create a CRD and then examine its yaml/json. For example: Anything else we need to know?: Environment: community master
/assign
/area custom-resource
/sig api-machinery /area custom-resources
/assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
edit: actually, that issue is about kubectl display and clock skew. this issue is about the condition timestamp updating, which still requires correction
Closing this issue. <detailsIn response to : duplicate of Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hello I'm a bug triage shadow for the 1.16 release cycle and considering this issue is tagged for 1.16, I'd like to check its status. Code freeze is starting on August 29th (about 3 days from now), which means that this PR must merged by then. Do we still target this issue to be fixed for 1.16?
yes, will take a look at it tomorrow /assign


Question: 1 /* expectedSetUpCallCount */, testKubelet.volumePlugin)) // Remove pod <del> // TODO: this may not be threadsafe (technically waitForVolumeUnmount) kubelet.podWorkers.(*fakePodWorkers).removeRuntime = map[types.UID]bool{pod.UID: true} </del> <ins> // TODO: technically waitForVolumeUnmount kubelet.podWorkers.(*fakePodWorkers).setPodRuntimeBeRemoved(pod.UID) </ins> kubelet.podManager.SetPods([]*v1.Pod{}) assert.NoError(t, kubelet.volumeManager.WaitForUnmount(pod))

pull-kubernetes-unit TestVolumeUnmountAndDetachControllerDisabled It failed on but after that the test is passing DATA RACE It seems that the data race is caused by below code It failing of test is Flaky, it was not reproducible on my local machine too, I have used below command to run the test output on my local machine I would like to work on this issue, is it possible to reproduce this issue locally?
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig testing
SIG Testing owns test frameworks, CI infrastructure, and other things to support testing. But this test is owned by SIG storage I believe since it tests their code, as we own tests that test our code.


Question: return 0, framework.NewStatus(framework.Error, fmt.Sprintf("injecting failure for pod %v", p.Name)) } <del> score := 10 </del> <ins> score := 1 </ins> if sp.numScoreCalled == 1 { // The first node is scored the highest, the rest is scored lower. sp.highScoreNode = nodeName <del> score = 100 </del> <ins> score = framework.MaxNodeScore </ins> } return score, nil }

What would you like to be : a default normalize score function in the scheduling framework. the plugin and log an error when the normalizing phase returns an invalid range of scores. Why is this needed: Per discussion in , we should enforce the range of score returned by the scoring plugin. /sig scheduling /priority important-soon /assign


Question: command = append(command, baseFlags[component]...) if component == apiServer { <ins> // Use first address we are given if len(s.API.AdvertiseAddresses) > 0 { command = append(command, fmt.Sprintf("--advertise-address=%s", s.API.AdvertiseAddresses[0])) } </ins> // Check if the user decided to use an external etcd cluster if len(s.Etcd.Endpoints) > 0 { command = append(command, fmt.Sprintf("--etcd-servers=%s", strings.Join(s.Etcd.Endpoints, ",")))

Is this a request for help? I think it is an issue either with software or documentation, but I am not quite sure. I have started with a question on stackoverflow: Is this a BUG REPORT or FEATURE REQUEST? (choose one): I think it is a bug or request to improve documentation Kubernetes version (use ): 0 Environment: Cloud provider or hardware configuration: Vagrant OS (e.g. from /etc/os-release): Ubuntu 16.04 Kernel (e.g. ): Install tools: kubeadm init/join Others: What happened: I have got 2 VMs nodes. Both see each other either by hostname (through /etc/hosts) or by ip address. One has been provisioned with kubeadm as a master. Another as a worker node. Following the instructions () I have weave-net. The list of pods looks like the following: CrashLoopBackOff appears for each worker node connected. I have spent several ours playing with network interfaces, but it seems the network is fine. I have found similar question on stackoverflow, where the answer advised to look into the logs and no follow up. So, here are the logs: What you expected to happen: I would expect the weave-net to be in Running state How to reproduce it (as minimally and precisely as possible): I have not done anything special, just followed the documentation on Getting Started. If it is essencial, I can share Vagrant project, which I used to provision everything. Please, let me know if you need one.
Please, let me know if it needs to be reported on weave-net project.
I reported a similar issue here:
I wonder if there is any working step by step instruction on how to get kubernetes cluster up and running on a set of virtual machines? I am happy to downgrade the version of kubernetes if it is an option. I am totally confused with all of these options: kube-deploy, kube-up, and other, I found from 3rd parties..
I have the same issue. I'm using VirtualBox to run 2 VM based on minimal Centos 7 image. All VMs are attached to 2 interfaces, a NAT and an host-only network. The two VMs are able to connect to each other using the host-only network interfaces.

I tried also with the instructions about Calico and Canal, and I cannot make them work either.
It seems Calico has got similar issue: What could I try to progress this issue further?
Okay I ran into this exact same issue and here is how I fixed it. This problem seems to be due to kube-proxy looking at the wrong network interface. If you look at the kube-proxy logs on a worker node you will most likely see something like: This is the wrong network interface. The kube-proxy should be looking at the master node's IP address not the NAT IP address. As far as I know the kube-proxy gets this value from the Kube API Server when starting up. If you look at the Kube API Server's documentation it states that if flag isn't set it will default to and if isn't set it will default to host's default interface. Which in my case and yours seems to be the NAT interface, which isn't what we want. So what I did was set the Kube API Server's flag and everything started working. So right after Step 2 and before Step 3 of You will need to update your and add the flag to point to your master node's IP address. So for example: My master node's IP address is , which means right after Step 2 I do: I am not to sure if this is a valid long term solution, because if the default kube- changes then those changes wouldn't get reflected by doing what I am doing. Ideally, I think the user would want some way to set these flags via kubeadm or maybe the user should be responsible for parsing the JSON themselves. Thoughts? However, it still may be a good idea to update Step 2 of to atleast mention to the users that they can update the kube component flags by modifying the their json found at: .
I am looking at this right now. We have a way of reproducing it with , where we have already applied a fix that makes kubelet pickup desride IP for pods in host network namespaces, however somehow service IP still points to 10.0.2.15.
I encountered this issue too. Adding --advertise-address when starting kube-apiserver solved this issue.
I've already started diving into the code, and you saved me time to realise how this part worked exactly, thank you so much!

This looks like an easy fix for kubeadm, PR is coming shortly.
Ok, so it turns out that this flag is not enough, we still have an issue reaching service IP. The simplest solution to this is to run with . To enable this, you can use .
I am starting kubeadm with --advertise-address option: kubeadm init --api-advertise-addresses=$masteraddress where my $masteraddress is not NAT interface. It still not enough to remove this issue.
"Ok, so it turns out that this flag is not enough, we still have an issue reaching kubernetes service IP. The simplest solution to this is to run kube-proxy with --proxy-mode=userspace. To enable this, you can use kubectl -n kube-system edit ds kube-proxy-amd64 && kubectl -n kube-system delete pods -l name=kube-proxy-amd64." Could you, please, explain what flag are you setting, to what command in the getting started tutorial? What does your last command do?
see Also, you can do this for now:
We are going to fix this shortly. If you can hop on Slack, I'd be happy to help.
As this thread is getting quite noisy, here is a recap. First, find out what IP address you want to use on the master, it's probably the one on the second network interface. For this example I'll use . Next, run . Now, you want to append to in the static pod manifest, you can do it like this: And finally, you need to update flags in daemonset and append , which can be done like this:
I'm working on docs for this. Also, has provided a Vagrant+Ansible implementation, and latest fixes are in .
can you help me with similar problem, I have multiple eth*. And I am using for private network, but . For now I fixed issue with weave via open port 443 for eth0 and eth1. But found next issue: I could not access to the instances from the cluster(example to redis that hosts on separate instance). I found that weave uses network. And I am looking for solution to change this. Do you know how can I change it? Example to ?
I didn't have to update the for my kube-proxy.

All I had to do was make sure the flag for the Kube API Server was set to a value listed in the flag. I'm just curious as to why you needed to set in your setup and I don't? Any ideas? I looked over the and your Vagrant setup looks very similar to mine, except you are using Ubuntu vs my Centos.
No, it uses . So for Weave we have , which expands to - . Kubernetes service range is , which expands to - . And you have , which expands to - . I don't see an overlap here, which is what you seem to be suggesting, but please correct me if I missunderstood. I think your problem is different, and please file another issue. Your previous issue didn't not describe the problem, please try to describe the problem. How many VMs are you running on? If it's a single-node setup, you won't notice anything. If it's not, I'm quite curious and would like to take a closer look, could you share your Vagrantfile?
after applying your changes I was able to get kube-dns and weave-net working! However, kube-scheduler-kubernetes won't run:
sorry, that was a copy/paste bug, I've updated the comments now, it's should had been not .
Yeah I figured that out shortly after my comment. I have everything working as expected now. Thank you to everyone who participated in fixing this! :)
you are welcome! We should eliminate the need for these work-arounds soon, at least as far as week can get without total hacks.
Thank you for the feedback. I'm getting closer to a working solution. With my basic Vagrant 2-node cluster (Centos) I now have weave up and running. The kube-dns service is however still not working. Steps taken: 1) kubeadm init on master with api-advertise-addresses flag set. 2) Update flags in pod manifest and daemonsets per guidelines above. 3) Join node. 4) Install weave on master. kube-dns is still stuck in ContainerCreating state: I experimented with both bridged network settings and private network settings but the problem still persists.
could you provide the output of ?
I thought it should be .

After update api server to use specific ip address the wave could not connect to 100.64.0.1:443. Digital Ocean has also Anchor Ip to eth0 with mask: 10.10.0.0/16: Updated: So I have theory that somehow pointed to 159.203.188.178 even when I specify api server address to 10.136.26.23
sure, see below. I assume the last cni-related errors propagate before weave is installed. The local NAT-interface address is still visible in . Is this an issue?
You a right, it's that in the current release, but it changed in master. Yes, there is an issue with that in some DO regions... you need to pass , but not with the build you are using. If you are online now, it'd be easy if you could find me on Slack. Thanks.
just as a sanity check, could you kill the DNS pod, i.e. , may there is some bug on the CNI end and somehow it's not getting picked up for this pod...
Hmm no luck. After deletion:
resolved my issue via adding a routing rule to use eth1 for node machines to kubernetes service range ip. Example:
Despite following the workaround steps ((1) specify flag when running , (2) specify in manifest, (3) append to daemonset), I'm seeing the exact same issue as That is, after installing the weave pod network, is still stuck in : did you ever resolve your problem? Does anyone else have any ideas?
but weave-net was up 15m ago and dns last event was 25m ago. try to restart dns pod.
It would be easier if folks could hop on #kubeadm channel in Slack and we can debug step-by-step. It became really hard to pick individual setup differences in this thread. On Tue, 18 Oct 2016, 14:50 Michael Nikitochkin, wrote:
I did restart the pod with no luck (by the way, when you say restart a pod, I assume you mean deleting the pod (and have the replica set replace it), right)? It is stuck in , although it appears to have reached slightly farther, beeing able to at least create containers. However But I guess I'll follow advice and take it on the slack channel.

So I actually did manage to get it working at last. The last missing piece of the puzzle was to follow the advice of point 5) in If anyone is interested, my ended up looking as follows:
I am very happy to hear you got it working, we planning to add a proper vagrant how-to, once we have tested latest packages. Have you referred to the Ansible code in k8s-playgound repo linked from the docs? On Wed, 19 Oct 2016, 07:54 Peter Gardfjäll, wrote:
yes, I did have a look at that code and just adapted their snippet to my particular setup.
: thanks for the hint. I can also confirm that I have a working system now.
Have you enabled userspace mode in kube-proxy also? On Wed, 19 Oct 2016, 20:49 bach1, wrote:
I run a similar setup based on CoreOS boxes(two nics, eth0 nat and eth1 private) on Vagrant. I had to make two changes to get the networking to work: Set the --node-ip on each kubelet process to the private network ip for each node Enable masquerading on the proxy to respond over the correct interface(--masquerade-all)
Pease checkout , there is code to handle firewalld rules and SELinux stuff (if you are on CentOS), as well as VirtualBox issues (applies to Ubuntu as well as CentOS). We are planing to provide plain text documentation soon too. Find me on Slack if you still have problems. It's always a good idea to create a new issue rather then comment on an existing one that seems like it describes your problem.
I am seeing a similar issue.

Here are the details: [root ~]# kubectl version Client Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2016-09-26T18:16:57Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2016-09-26T18:10:32Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"} [root ~]# kubeadm version kubeadm version: version.Info{Major:"1", Minor:"5+", GitVersion:"v1.5.0-alpha.0.1534+-dirty", GitCommit:"", GitTreeState:"dirty", BuildDate:"2016-09-27T18:10:39Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"} [root ~]# CentOS 7.2. Kernel Version: [root ~]# uname -a Linux 3.10.0-327.18.2.el7.x8664 SMP Thu May 12 11:03:55 UTC 2016 x8664 x8664 x8664 GNU/Linux These are based on an old version of OpenStack and KVM. if that is relevant, I can find the details
letting you know that workaround command is now being rejected: What could I do to fix it?
i am having the same issue as If you leave validation on, you get this error. i included the input from and both kind and apiVersion are clearly there. { "kind": "List", "apiVersion": "v1", "metadata": {}, "items": [ { "spec": { "template": { "spec": { "containers": [ { "command": [ "--cluster-cidr=10.32.0.0/12" ] } ] } } } } ] } it seems to not get the outermost portion of the input which is where the kind and apiVersion are both at.

Hi, I can confirm the problem still exists on a two nodes kubernetes cluster installed with Kubeadm. The nodes are Ubuntu Xenial 16.04 connected to each other using a tinc VPN. Everything works fine on the tainted master node but on the second (worker only) node every CNI network plugin container keep crashing. (I tried Flannel and Weake) It seems like the network-fabric container is unable to reach the API server on 16.96.0.1 I tried the workaround proposed but couldn't get it through becouse of the issue psoted by and apparentely the first two steps are insufficient.
you need add route to 16.96.0.1 via your you VPN interface.
In my case, my apiserver's advertise-address was correct, but I still had the crashloop issue with the weave pods. Adding "--proxy-mode=userspace" using 's neat jq trick solved the issue for me. Though the issue is solved, I don't have a good understanding of why it solved. I looked at the kube-proxy documentation() and it says: Should we expect any side effects as we are setting it to the "older" option? Thanks.
Hi everyone, I just finished a Vagrant box with centos 7.2 and kubeadm with weave-net and dashboard. I include all solved issues in this post. Thank you to and ! replace all occurences of kube-proxy-amd64 by kube-proxy maybe ... just check it with a get pods in the global namespace
Moved to kubernetes/kubeadm
I'm also experiencing the same issue: Any ideas on how to deal with the issue ?
what do the container's logs say?
Please open a new issue in the kubeadm repo to track that
would you mind sending me the cmd for getting this logs out into a gist for example ? that would help. I will raise a new ticket asap.
either: or should do
Same problem, solved it with


Question: create 0644 root root } EOF <del> done </del> } # Finds the master PD device; returns it in MASTER_PD_DEVICE

GCI log rotation configuration missed "" file. We should just configure log rotation to rotate all "*.log" files in /var/log.
I cherrypicked your change last night. Please do what you need to do to confirm this is resolved.
Thank you, will do.
Verified. Closing issue.


Question: "fmt" "io" "io/ioutil" <ins> "net" </ins> "net/http" "os" "regexp"

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: The Kubernetes master created routes in the neutron router that assign an IPv6 nexthop for a IPv4 prefix. This is not possible and causes a reachability problem between the pods. Example route: What you expected to happen: When describing a node the master has the following information It should detect the address type and choose the correct InternalIP How to reproduce it (as minimally and precisely as possible): Deploy on a Openstack cloud with IPv6 enabled using the following playbook Anything else we need to know?: The wrong next hop is coming from calling the function and nowhere is a check if the address is IPv4 or IPv6 Environment: Kubernetes version (use ):Cloud provider or hardware configuration: Openstack OS (e.g. from /etc/os-release): Ubuntu Xenial Kernel (e.g. ):Install tools: Others: /sig openstack
- Thank you for this detailed analysis. Seems to me that the best way to address this would be to add an IP family argument to getAddressByName() to let it know what V4/V6 family to look for for a next-hop address. In the calls to getAddressByName(), this argument could be set based on the family of route.DestinationCIDR.
Issue is probably related.


Question: const ( // CloudProviderName is the value used for the --cloud-provider flag <del> CloudProviderName      = "azure" </del> <ins> CloudProviderName = "azure" // AzureStackCloudName is the cloud name of Azure Stack AzureStackCloudName    = "AZURESTACKCLOUD" </ins> rateLimitQPSDefault    = 1.0 rateLimitBucketDefault = 5 backoffRetriesDefault  = 6

<!-- Please only use this template for submitting enhancement requests --What would you like to be : PR() already proves that old api-version on Azure stack works on global Azure too, we could support azure stack in azure cloud provider lib, and then make sure following components works in Azure stack: CSI drivers: azure disk, azure file CCM Since it's mainly api-version change, we could also add e2e tests against global Azure too. Why is this needed: /assign /priority important-soon /sig cloud-provider /area provider/azure
there is blocking issue here(also need azure stack supports latest data model struct): just making the change in interfaces and not go-sdk but facing this error, since we don't have a switch inside go-sdk, we only do switch in azure cloud provider lib,
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale


Question: kube::release::create_tarball "${package_name}" "${release_stage}/.." } <del> # This is all the stuff you need to run/install kubernetes.  This includes: #   - precompiled binaries for client </del> <ins> # This is all the platform-independent stuff you need to run/install kubernetes. # Arch-specific binaries will need to be downloaded separately (possibly by # using the bundled cluster/get-kube-binaries.sh script). # Included in this tarball: </ins> #   - Cluster spin up/down scripts and configs for various cloud providers <del> #   - tarballs for server binary and salt configs that are ready to be uploaded </del> <ins> #   - Tarballs for salt configs that are ready to be uploaded </ins> #     to master by whatever means appropriate. <del> function kube::release::package_full_tarball() { kube::log::status "Building tarball: full" </del> <ins> #   - Examples (which may or may not still work) #   - The remnants of the docs/ directory function kube::release::package_final_tarball() { kube::log::status "Building tarball: final" # This isn't a "full" tarball anymore, but the release lib still expects # artifacts under "full/kubernetes/" </ins> local release_stage="${RELEASE_STAGE}/full/kubernetes" rm -rf "${release_stage}" mkdir -p "${release_stage}" <del> # Copy all of the client binaries in here, but not test or server binaries. # The server binaries are included with the server binary tarball. local platform for platform in "${KUBE_CLIENT_PLATFORMS[@]}"; do local client_bins=("${KUBE_CLIENT_BINARIES[@]}") if [[ "${platform%/*}" == "windows" ]]; then client_bins=("${KUBE_CLIENT_BINARIES_WIN[@]}") fi mkdir -p "${release_stage}/platforms/${platform}" cp "${client_bins[@]/#/${LOCAL_OUTPUT_BINPATH}/${platform}/}"  "${release_stage}/platforms/${platform}" done </del> <ins> mkdir -p "${release_stage}/client" cat <<EOF > "${release_stage}/client/README" Client binaries are no longer included in the Kubernetes final tarball. Run cluster/get-kube-binaries.sh to download client and server binaries. EOF </ins> # We want everything in /cluster except saltbase.  That is only needed on the # server.

Either include non-arch specific stuff in every tar or break that out too. ref cc
interested in helping with this one?
Yes, I may help
OK, I did not actually understand what was in , apparently. Alternate proposal: [ ] update all references that need just server binaries to download the server tarball directly [ ] same for places that just need (related to ) [ ] remove all arch-specific binary artifacts from and replace with a script which downloads the client/server tarballs as necessary. The alternative option, including all of the non-arch-specific code and docs in an arch-specific starts to get very tricky, since we run into a large cross product of client x server platforms/archs.


Question: # # We specify -X to avoid a race condition that can cause minion failure to # install.  See https://github.com/saltstack/salt-bootstrap/issues/270 <del> wget -q -O - https://bootstrap.saltstack.com | sh -s -- -X </del> <ins> curl -L --connect-timeout 20 --retry 6 --retry-delay 10 https://bootstrap.saltstack.com | sh -s -- -X </ins>

When wgetting to this address during the installation process, this just fails silently. Turns out the issue is that if you take a raw and curl that address you get a TLS error. I've fixed it by changing to , but obviously that's more risky.
Related (filed after an email exchange around wget) I don't have easy access to the vSphere so I can't verify how it relates to my debian install.
Yes, that is exactly the error I was experiencing.
Does curl work? Or is it busted too?
Curl is not installed on the image by default. I'm building a new image based on Debian 7.7 where I pull in curl by default, so we can switch to using that everywhere (instead of an awkward mix of wget and curl).
Thanks for reporting this I hadn't seen this yet. I uploaded a new VMDK that includes curl and updated the deployment scripts to use curl. I verified that deployment works again for me.
I can confirm that this works now with latest build from master on ESX. Probably still won't work with v0.4.3 (even w new image) since the scripts still used wget back then.
We are probably going to fork for v0.5 soon. But I can backport if this trips folks up. sent from tiny keyboard On Nov 12, 2014 9:09 AM, "Ben Smith" wrote:


Question: return &runtimeapi.ListContainersResponse{Containers: result}, nil } <ins> func (ds *dockerService) getContainerCleanupInfo(containerID string) (*containerCleanupInfo, bool) { ds.cleanupInfosLock.RLock() defer ds.cleanupInfosLock.RUnlock() info, ok := ds.containerCleanupInfos[containerID] return info, ok } func (ds *dockerService) setContainerCleanupInfo(containerID string, info *containerCleanupInfo) { ds.cleanupInfosLock.Lock() defer ds.cleanupInfosLock.Unlock() ds.containerCleanupInfos[containerID] = info } func (ds *dockerService) clearContainerCleanupInfo(containerID string) { ds.cleanupInfosLock.Lock() defer ds.cleanupInfosLock.Unlock() delete(ds.containerCleanupInfos, containerID) } </ins> // CreateContainer creates a new container in the given PodSandbox // Docker cannot store the log to an arbitrary location (yet), so we create an // symlink at LogPath, linking to the actual path of the log.

What happened: Windows kubelet panic occasionally on nodes which have high volume of pods run and delete. Full logs: Preliminary investigate result: How to reproduce it (as minimally and precisely as possible): Run high volume of pods and delete them when they completed in Windows nodes. Environment: Kubernetes version (use ): 1.17.7 Cloud provider or hardware configuration: Azure AKS OS (e.g: ): Windows Server 2019 Kernel (e.g. ): 10.0..1282
/assign I would like to give this a shot.
/remove-sig windows I think this is not a windows specific problem. /sig node /area kubelet
Hi! Thanks for your report, I have filed to address this issue.
This issue is a bit urgent for our situation, it occurs every few days in production clusters. Seems the PR is pending a few days, is it possible to increase the priority to merge it trunk and 1.17/1.18 branch? Thanks
I understand how urgent this issue is, but I am not privileged to do anything further :(
Do you know anyone who can help?
looks like a simple fix, add a lock for , I could help.
There is already a fix by , the problem is the PR is pending, not sure if you or someone can increase the priority to merge it trunk and 1.17/1.18 branch asap. Our AKS clusters are affecting by this frequently.
Thanks guys.


Question: } func (fp *tokenFilter) PreFilter(ctx context.Context, state *framework.CycleState, pod *v1.Pod) (*framework.PreFilterResult, *framework.Status) { <del> return nil, nil </del> <ins> if !fp.EnablePreFilter || fp.Tokens > 0 { return nil, nil } return nil, framework.NewStatus(framework.Unschedulable) </ins> } func (fp *tokenFilter) AddPod(ctx context.Context, state *framework.CycleState, podToSchedule *v1.Pod,

In k8s 1.27, scheduler introduced a new "short circuit" semantics in PreFilter to bypass a Pod quickly. A change introduced here is it relies on that plugin to be stored in , which makes sense as what it was explained . However, the info, although is populated upon each plugin's PreFilter invocation, is not injected into when some PreFilter returns unschedulable: So think about this workflow: pluginA returns Skip in its PreFilter pluginB return Unschedulable in its PreFilter note: pluginA is not included in due to aforementioned bug a Pod is claimed as unschedulable (failed by pluginB) during PostFilter, it's supposed to bypass pluginA's PreEnqueueExtension's AddPod/RemovePod, but it doesn't calling a "skipped" Plugin's AddPod/RemovePod during preemption, sometimes will abort the whole preemption as the logic tries to read info from cycleState, and some plugin's "skip" version doesn't fill cycleState properly, e.g., InterPodAffinity (skip immediately w/o write preFilterStateKey). Short circuiting using Skip is fine, but should injected properly at all cases. Described in the workflow above. It's firstly captured in an out-of-tree PostFilter plugin: , but can also happen in in-tree plugins. No response v1.27 <details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/kind regression /sig scheduling
cc
Among in-tree plugins, 3 plugins employ non-nil and may return Skip in their PreFilter() implementation w/o writing to cycleState, while return error immediately if is missing in their Add/RemovePod() functions.

So technically, as long as a scheduling flow hit the following criteria, it would hit the bug (preemption doesn't function): plugin 1/2/3 return Skip in PreFilter arbitrary PreFilter plugin after the "skipped" plugin returns Unschedulable in its PreFilter
/cc
/cc
Cherry-pick PRs: 1.28: 1.27:


Question: // loadbalancer's backend hosts. In this case, request will not hit the loadbalancer but loop back directly. // Need to add the following rule to allow request on host. if allowFromNode { <del> writeLine(proxier.natRules, append(args, "-s", fmt.Sprintf("%s/32", ingress.IP), "-j", "ACCEPT")...) </del> <ins> writeLine(proxier.natRules, append(args, "-s", utilproxy.ToCIDR(net.ParseIP(ingress.IP)), "-j", "ACCEPT")...) </ins> } // If the packet was able to reach the end of firewall chain, then it did not get DNATed.

We are working migrating ipv4 to ipv6, we should support ipv6 for IPVS. We should do the following things: remove ipv4 constraints of Node IPs() remove ipv4 in pkg/util/ipvs() use ipv6 conntrack tools use ipv6 iptables tools <!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind feature /sig network /assign /cc
/area ipv6
FYI: I've just posted a PR (PR ) to fix the way that kube-proxy was calling conntrack without the required "-f ipv6" CLI flag whenever conntrack was being executed for a service that's using IPv6 addresses. There were a couple of lines in that needed to be changed. I don't think I can add UT coverage for those lines until more support for IPv6 addresses is to Cc
ACK! I will take care of , please feel free to ping me when you want my hand :)
Sounds good, is all yours. :) Likewise, let me know if I can help.
Sure, mate.


Question: mountEtcHostsFile = mountEtcHostsFile && (mount.MountPath != etcHostsPath) vol, ok := podVolumes[mount.Name] if !ok || vol.Mounter == nil { <del> glog.Warningf("Mount cannot be satisfied for container %q, because the volume is missing or the volume mounter is nil: %q", container.Name, mount) continue </del> <ins> glog.Errorf("Mount cannot be satisfied for container %q, because the volume is missing or the volume mounter is nil: %+v", container.Name, mount) return nil, fmt.Errorf("cannot find volume %q to mount into container %q", mount.Name, container.Name) </ins> } relabelVolume := false

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: The makeMounts function adding volumes to the volumes list if the volume doesn't exist or mounter doesn't exist. This is potentially dangerous behavior because an application can write to the mount path (if it already exists in the container image) expecting it to be backed by a persistent volume, but in fact, is only backed by the container rootfs, which is ephemeral. We probably have not seen this issue manifest in the wild because the volume not existing or mounter being nil should never happen, and also the application would probably see a path/file does not exist error first. What you expected to happen: If we can't mount all of the user's requested volumes into the container, we should fail to start the container. /assign /sig storage node /cc


Question: if err != nil { return err } <del> if visitCount == 0 { </del> <ins> if visitCount == 0 && !isForDelete { </ins> return errNoMatchingResources } return err

What happened: given an empty namespace (no pods), if running: , kubectl errors with the message . What you expected to happen: kubectl should exit with status 0, since the condition to wait for is for all pods to be deleted, and that condition is already satisfied. How to reproduce it (as minimally and precisely as possible): run the example kubectl command Anything else we need to know?: Environment: Kubernetes version (use ): Taking the liberty of skipping other version information, since this is (I assume) purely a client-side bug
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cli
Seems like didn't fix it correctly: Test in this PR always passed because the always return nil error and visitCount == 1, and the type conversion always fail here:
thanks for the quick fix!
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale - still an issue, I believe the PR is waiting for re-review
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle rotten
/remove-lifecycle rotten

One use-case of is to wait for a lot of resources that you have just deleted. Unfortunately, this workflow is racy because will fail when encountering a resource that doesn't exist (yes, it's probably already gone). I suspect we should just consider 404 resources to be already gone and integrate that condition as part of the normal flow. /sig cli /kind bug
/assign
Duplicate of ?
yes, I tried to add in in But that PR seems not complete.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
The PR merged, I think we're good now!

Hi. A similar issue was reported here: but the bot had closed it due to inactivity. So, creating a new one with details again. I am facing the same problem on my end. If a resource does not exist or is deleted, but you are using wait --for condition delete, it fails. There is a flag in delete to avoid such errors. A similar condition of flag must be in as well so that it does not fail with errors like below. Thanks.
/sig cli
/assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Sorry I forget to do this, I'll try to submit a PR this week.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Yes, this would be nice to have.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at .

/lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-contributor-experience at . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Any chance this can be revisited? Thanks!


Question: return isNodeReady(&nodeList.Items[0]) } <del> func restartDbus() error { cmd := "systemctl restart dbus" _, err := runCommand("sh", "-c", cmd) return err } </del> func systemctlDaemonReload() error { cmd := "systemctl daemon-reload" _, err := runCommand("sh", "-c", cmd)

Any test that runs Gracefulshutdown seems to be having issues. Reference: In this case, they are failing consistently. The tests were flaky but as of 9/29, they have been failing ever since. and I were looking into these failures. We are able to reproduce on a GCP instance but we are seeing failures to sending dbus shutdown signals. We have looked into this code but it does not look like any code is responsible for this failure. One thing we did see is that there was a new release of COS around then. We are seeing failures in the logs. Running this test locally we can sometimes see issues with dbus. Running this locally (after copying the image config): /sig node
/cc
/cc
/triage accepted /priority important-soon
Looking in the logs of the last successful job: And viewing the kernel logs, I don't see any errors related to dbus. Last successful job linux version: The next job fails and the main difference seems to be a linux kernel update.
/cc
/cc
I was able to repro this on COS M109 as follows, while the below works fine on COS M105. Will dig a bit deeper if this is an issue in COS or systemd upstream regression.
thank you for looking into this! (adding who authored the Dbus restart code) What we are seeing here would be something other than a regression. This problem affects Google's COS, Fedora CoreOS (and its RHEL-based equivalent), Fedora Workstations, and most likely any other Linux distribution, whether container optimised or not, that is not Debian or based on Debian, such as Ubuntu. Both Debian and Ubuntu are unaffected as they handle Dbus restarts slightly differently. TL;DR Restarting the Dbus broker service at runtime is generally not recommended or supported. Many systemd components (but also other software and services) that rely on Dbus wouldn't reconnect to a new Unix socket, which would have the same path (location) as before, following the Dbus broker restart. This can lead to anything from authentication timeouts (since the systemd-logind and the polkitd services would be impacted) to the system generally broken following Dbus broker restart - this is especially the case for desktop (such as Fedora Workstation) systems where there are a lot of variety of services that rely on working Dbus connectivity.

Different software might handle this differently and choose to reconnect on a socket write or read timeout. However, systemd at large is not one of these. Lennart has confirmed this on a number of occasions per (selected from a sample of issues): Debian and its derivatives, such as Ubuntu or Linux Mint, etc., are not affected as they opt to trigger the restart (or activation, if you wish) of key system services (such as the systemd-logind) following a Dbus broker restart. I recommend removing the Dbus broker restart from the test code base to fix the failing tests. If there were no objections, then I would be happy to do this work. Additionally, we could see about porting Debian's behaviour to Google's COS and Fedora CoreOS (and, as such, the Red Hat RHEL-based equivalent) around restarting systemd-logind on Dbus restart. But that would be a bonus work and outside of this issue's scope. Typically on a Fedora CoreOS distribution, the default list of Dbus connections just after system startup might look as follows: <details<summaryDbus connections list on Fedora CoreOS after startup</summary</detailsSeveral key systemd services are connected, such as the systemd-logind and the systemd-homed, plus the polkitd service. Restarting Dbus would reduce this list to the following: <details<summaryDbus connections list on Fedora CoreOS after Dbus broker restart</summary</detailsThere is almost nothing connected following the restart. Any attempt to introspect the /org/freedesktop/login1 object would now fail per: <details<summaryObject introspection run</summary</detailsRestarting the systemd-logind service causes a fresh process to reconnect to the Dbus bus per: <details<summaryDbus connections list on Fedora CoreOS after systemd-logind restart</summary</detailsThis resolves the problem of accessing the /org/freedesktop/login1 object per: <details<summaryObject introspection run</summary</detailsHowever, other services might still be affected and would need to be restarted. I mentioned that Debian and its derivatives are not affected as they trigger the restart of key systemd services, especially the systemd-logind service.

This can be seen in the following output - pay close attention to the PID of the systemd-logind service on the list (using Debian 12 "bookworm" as an example): <details<summaryDbus connections list on Debian 12 following Dbus broker restart</summary</detailsNote that for this particular Debian 12 system, the initial state of the connections was different. The following is the state immediately after the system starts up, this is a desktop flavour with a lot of other software that relies on Dbus, and that will also fail to reconnect correctly per: <details<summaryDbus connections list on Debian 12 after startup</summary</detailsWhen the Dbus broker is restarted, and a service that exposes an object hasn't reconnected to the new Unix socket, the following breakdown in connection between the client, the Dbus bus, and the service of interest in question can be seen. Any attempt to access the /org/freedesktop/login1 object will not work: <details<summaryObject introspection output</summary</detailsThis failure can also be seen on a bit lower level too, per: <details<summaryThe busctl strace output</summary</detailsA test has been in the past to catch issues following the Dbus broker service restart as part of a genuine issue that consisted of adding support for reconnecting to the Dbus socket in case of errors or connection failure. This test would work fine on Debian and Ubuntu, as these Linux distributions aren't affected as much as we already learned. Specific commit: This was in response to a reported issue: Given that tests are nowadays commonly run on either Googe Container-optimised OS (COS) or Fedora CoreOS, the addition of the Dbus broker restart would have side effects: Any use of would either appear as hang or take a long time to time out Attempt to sign into a given compute instance using SSH would have taken a long time or appear as a hang connection Dispatching requests to Dbus to send a signal to some other participating services can also take a long time before eventually timing out Some combination of the above might help explain why a lot of our tests were timing out either on connection (SSH) or execution (Ginko, etc.).

Related code: The code responsible for sending a signal (SIGHUP) to systemd-logind to ask it to reload its configuration: The above is an interesting take on sending the SIGHUP signal to a process since we don't need to find out what PID the process in question has and just ask Dbus to do the work for us - a bit unorthodox and relies on the service that when breaks would render other interactions with the system and other services unreliable. Related upstream issues and Pull Requests for reference: Linux distributions used for testing: <details<summaryA list of Linux distributions showing the /etc/os-release file content</summary</details
I took the time to diagnose it this week, and I tried to find a way around it. The previous purpose was to fix that kubelet could not work after dbus restart. Currently, this issue has been fixed in kubelet for a long time, and I have no opinion on deleting this test.
Hi This is working without issues. The issue is not between kubelet and Dbus, but between Dbus and other services that do not handle the Dbus broker being restarted. We are good on the kubelet front. :smile:


Question: } expected := []metav1.PartialObjectMetadata{ <del> *newPartialObjectMetadata("group/version", "TheKind", "ns-foo", "name-foo"), </del> *newPartialObjectMetadata("group/version", "TheKind", "ns-foo", "name-bar"), *newPartialObjectMetadata("group/version", "TheKind", "ns-foo", "name-baz"), <ins> *newPartialObjectMetadata("group/version", "TheKind", "ns-foo", "name-foo"), </ins> } if !equality.Semantic.DeepEqual(listFirst.Items, expected) { t.Fatal(diff.ObjectGoPrintDiff(expected, listFirst.Items))

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: I used to fake a clientset to test my code. I found that it became very slow to init the clientset with a big number of objects when I was trying to simulate a large scale cluster, and the time seemed exponential increasing. For example: Initializing a clientset with: 5000 objects took 300ms objects took 2.58s objects took 11.82s objects took 54.64s. It's because the fake clientset used a slice to store each kind of objects, so it's quite slow to init the clientset with massive objects because it checked existence of an object by traversing all objects and comparing with them before adding one, which leads to O(n^2) time complexity. What you expected to happen: The fake clientset is very useful for development, it should have a reasonable time complexity to allow developers to use it for large scale scenario. How to reproduce it (as minimally and precisely as possible): Run this code: You would see the time spent on each operation, like: Anything else we need to know?: Environment: Kubernetes version (use ): 1.15.0-1.18.0 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I had PR that uses map to store objects, which can reduce the init time to a reasonable value (tens of ms for objects). /assign