Question: http_archive( name = "io_bazel_rules_go", <del> sha256 = "91b79f4758fd16f2c6426279ce00c1d2d8577d61c519db39675ed84657e1a95e", urls = mirror("https://github.com/bazelbuild/rules_go/releases/download/0.17.4/rules_go-0.17.4.tar.gz"), </del> <ins> sha256 = "87a089eabf919de29eb4711daa52ffbc4b22b2c045949c20503357a3cadf1037", urls = mirror("https://github.com/bazelbuild/rules_go/releases/download/0.17.5/rules_go-0.17.5.tar.gz"), </ins> ) load("@io_bazel_rules_go//go:deps.bzl", "go_register_toolchains", "go_rules_dependencies")

Something has changed between 320 and 321 runs and the request latency has increased significantly. The graphs below shows request latency (non watch) over time for a single test run: : ! : ! In my opinion they looks significantly different: in 320 most of the spikes are lower than 0.8s while 321 most of the spikes exceeds 1.6s. I checked few runs below 320 and have latency pattern similar to 320 and few runs after 321 and they have similar pattern to the 321. We are seeing similar latency growth in kubemark tests. The most significant change between 321 and 320 was go 1.11.5 -1.12 migration. /cc
/sig scalability
With we are trying to prepare small scale repro that we can use for further debugging.
was also in that range (cc for verification that nothing in that PR increased server load in non-alpha clusters)
did tests at head with reverting to go1.11 and metrics seemed better He is working on confirming that and should paste results then, but it seems to be pretty clear that it's go regression. We're also working on providing smaller-scale repro case to work closer with golang team (they used to be really helpful when debugging regressions in previous versions (IIRC 1.9 is where there were previous one).
I double checked and didn't find anything suspicious. The alpha function wasn't invoked in that window, and the PR is a no-op to non-alpha clusters
Regarding what wrote: ! I was mostly investigating the regression on kubemark-5000. Some of its latencies increased x3. I run two tests: one with checkout, and the other one with some commit checkout but without (Update to use golang 1.12). The 99th latency percentile of node listing was 3400.152ms in first case and f 952.576ms in second one.
Look into other metrics (I can't remember which exactly, delete pods? post pods? or sth like that) - which increased by almost 10x.
This is the graph that I had on my mind, but yeah, it's more like 4x not 10x. But still quite huge drop: !

Reproducing the issue: kubernetes repo - First broken version (go1.12) Last working version (go1.11) perf-tests repo - This repo is home for clusterloader2 - tool used for running the scalability tests. kubernetes directory run “make quick-release”. This will build kubernetes based on checkout version. GCE cluster. kubemark test. This step creates kubemark cluster (on top of the cluster created in step 4), run cluster loader test and tear down the cluster. The test run metrics will be exported to directory provided by the --report-dir flag. Inside this directory there will be APIResponsivenessload file containing 50, 90, and 90 percentiles of all apiserver call latencies. The latencies affected the most are: Listing nodes { "Resource": "nodes", "Scope": "cluster", "Subresource": "", "Verb": "LIST" } Pod deletion { "Resource": "pods", "Scope": "namespace", "Subresource": "", "Verb": "DELETE" } Listing services { "Resource": "services", "Scope": "cluster", "Subresource": "", "Verb": "LIST" } I would suggest their 99th percentiles as the issue metric. As for comparison: -Go1.11 run: Listing nodes - 162.409ms Pod deletion - 41.302ms Listing services - 28.277ms -Go1.12 run: Listing nodes - 293.724ms Pod deletion - 56.2ms Listing services - 46.183ms down GCE cluster
We're trying to dig deeper into it to both root cause it to a smaller part of the system, but it is quite tricky. Unfortunately, we don't have much more than guesses right now, though. Some observations: I looked into kubemark-5000 runs into DELETE pods calls, and it seems that all of them for which we have traces (500ms+) - there are ~5000 of those in each of recent runs [previously 99th percentile was <200ms), there is a significant big of "Object deleted from database". That may suggest some etcd issues, but we don't see logs related to long-running-calls on etcd side.

That may suggest some issues on the etcd client side, but we don't have anything concrete yet.It's also possible that golang upgrade changes something in serialization/deserialization - that's a second hypothesis (we've seen a bunch of traces of "Decoded delete options" (deserialization is also done in etcd client)), but I'm not convinced this is the case as of now
We've spent significant time with golang folks and already have some findings. To summarize: the microbenchmark from enabled Go team to find some regression on their side - the faulty commit from golang is: to cite Go folks: "To summarize the change: I removed the "fast" path for small page-level allocations (which currently always need to acquire the heap lock) in order to simplify code for subsequent changes. In effect, span allocations now all go through a balanced tree rather than just a linked list for small spans (I think it was all spans under 128 pages?)." We aren't fully convinced that this is for sure the same thing that is causing regression in our scalability tests, but analyzing traces, profiles, golang tracesetc. suggests that problems are indeed around memory allocations or GC. So with high probabilty this is the same thing. We are discussing with the way to prove it's the first thing and working with the to see what we can do with it.
It appeared that the commit isn't the most faulty one for us - running scalability tests against that commit doesn't show significant drop in metrics. In the lack of better ideas, we decided to bisect changes in golang by running full scalability suite on 5k-node cluster against them. This will take couple more days though...
Just to update this thread - we're bisecting Go 1.12 branch and there aren't that many remaining commits. We already know that: is a good one is a bad one I hope that we will know the exact faulty commit on Monday.
Looking through the following stand out for me /me places bets on one of those 2 :)
That is our bet too - we're working with Go folks to confirm and fix/mitigate this.
Any luck? (curious!)

I'm sorry for delay with update here - the current update is as following: after the next round of bisections, we know that: is a good one is a bad one During my discussions with golang folks, the outcome was that we believe that is the faulty one, as it adds a bunch of syscall to second-slowest allocation path (built on top of the previous one) should only help So the next step was to verify and indeed it is a bad one (and actually as expected, the metrics were worse than in the so far confirmed oldest bad commit: ) This is what we already know. What we're doing now is that golang folks provided us two patches that they think should mitigate the issue and I'm trying to confirm if that is really true. We should have results of this one later today. The two potential outcomes are: it solves our issue; then I will be working with golang folks to patch 1.12 with those it doesn't solve our issue: then I will be working on final confirmation that it's exactly this commit (by checking the previous one, and the next one to confirm it really helps) and working further with golang folks on what else we can do to mitigate it (and also I will try to understand deeper what exactly is happening there, I just have some rough understanding now).
Very very cool! thanks
cc
Just to give an update on that.

we fully confirmed that the regression is due to (using go1.12.1 with manually reverted that commit reverts the behavior) however, according to golang fokls that commit was fixing some things, and there are couple commits built on top of them after 1.12, so it's not-trivial to revert it (we are still looking for other options)the patches that golang team provided helps, but don't fix the problem (they roughly reduce the problem to 50%) yesterday I got another two patches that may help and we're validating them nowin the meantime we're also working with golang folks to try gathering more useful data for them for debugging - they already have some patches to apply in golang and I'm working on piping them through k8s and our build scripts - I hope I will have them ready today for an over-night run
To give an update on that - the current state of it is as following: that particular commit can't be just reverted, as it was fixing some issue however, golang folks have a PR that is mitigating that issue in a different way we have verified with custom build of that fully mitigates regression on our side The current plan is as following: has been merged to head of Go [that is the mitigation commit] it will be cherrypicked to 1.12 branch additionally in 1.12 we will revert the offending commit: We expect a new patch release of Go with those two commits to be released on May 1st. is currently my main point-of-contact coordinating that on golang side.
thanks for the update ! we will still be in 1.15 alpha* phases so that's great news.
- the trick is that go 1.12 is also in 1.14, so we should also cherrypick 1.14 branch - currently there is visible regression in 1.14
agree. i'd support that
The commits that we were waiting for are already there: Now we are waiting for next 1.12 patch release of Go to update Kubernetes to use, the ETA is May 1st. - given that I will be on vacation for the rest of this week, would you be able to help with that?
yes i can, please let me know what needs to be done (so i can figure out if i have access before you head out)
- thanks a lot.

I didn't do upgrade of go for 2 years or so, so I'm not entirely sure how (if?) the process changed in the meantime. - was doing the last one: , so I'm sure he will be able to help with that.
Ah just update to latest golang when it gets out. yes, i can do that. i was wondering if you wanted me to kick off some CI runs or something. Have a good vacation
- thanks a lot! Yes - it wasn't clear, but I was testing that and it helps us, so no need to run another tests (we will get final confirmation from postsubmits).
I can also help out to get this upgrade in when it’s released. Thanks for all your work on this
Looks like 1.12.5 went out this afternoon. PR open (), but expect tests to fail: Waiting for the docker hub image to publish. The PR is merged, so this should happen shortly. Need a new version of the bazel rules_go to be published.
Just to clarify the above (from : "oh, so I saw the auto update ran here: but it looks like the update for the tags hasn't merged yet: "
Yea, it takes some time for the images to get published.
Looks like new images are live.
Reopening this so we can verify that everything is back to normal in CI
/assign
For me it's pretty clear that this was resolved: http://perf- http://perf- Now there is one more thing we should do, which is cherrypick it to 1.14 branch. ^^
Opened pick:
has SIG Scalability done any one-off custom scale/perf testing with the new golang patch release and the k8s 1.14 branch content? Do you see this cherry pick as a 1.14.2 release blocker? Or would a week or two of focused testing with updated golang in the 1.14 k8s branch and a 1.14.3 release by end of May be a sufficiently safe yet expedient response?
We were doing a bunch of tests to discover the real regression in golang 1.12 - the regression happened within 1.14 release timeframe and the bisection was done on that branch. So yes - I'm 100% confident that it helps.

Tough question - I would love to see it in 1.14.2, otoh given how long it took to debug and fix it, I can see release team pushing it to 1.14.3. But if we could have that in 1.14.2, it would be extremely helpful.
just merged and we let 1.14 soak for one or two days before we cut . Let's keep this issue open until we have some data on the latencies after that merge. SIG Scalability, can you please provide some data / graphs about the latency in a (couple of) days for the branch?
is there any evidence we'll be able to observe in for validation? Or is it possible for SIG Scalability to trigger any custom runs now that the cherry-pick has merged to the 1.14 branch?
We aren't running large tests on release branches due to cost of them... We don't have any free quota to run it now, due to huge number of regressions that we have at head that we're currently debugging. But I'm 100% confident with this change. What I was personally doing in the past: I was running tests against commit when we bumped to 1.12.0 with 1.12.5 (which was from beginning of March, so still in 1.14 timeline) and the effect was exactly the same as the effect against head we got after merging it. I also understand changes in Go between 1.12.4 and 1.12.5 and their effect on performance And we also understand this regression, and why the fixes help, which has nothing to do with any changes that are happening to kubernetes. So I'm entirely sure that if you didn't observe any correctness issues, from performance perspective it is good-to-go.
Thanks for your answer!
With this being fixed in head and 1.14.2 released with the fix, I'm closing this one as fixed.


Question: // https://github.com/kubernetes/kubernetes/issues/72236 proxier.filterRules.Write( "-A", string(kubeIPVSFilterChain), <del> "-m", "set", "--match-set", proxier.ipsetList[kubeLoadBalancerSet].Name, "dst,dst", "-j", "ACCEPT") </del> <ins> "-m", "set", "--match-set", proxier.ipsetList[kubeLoadBalancerSet].Name, "dst,dst", "-j", "RETURN") </ins> proxier.filterRules.Write( "-A", string(kubeIPVSFilterChain), <del> "-m", "set", "--match-set", proxier.ipsetList[kubeClusterIPSet].Name, "dst,dst", "-j", "ACCEPT") </del> <ins> "-m", "set", "--match-set", proxier.ipsetList[kubeClusterIPSet].Name, "dst,dst", "-j", "RETURN") </ins> proxier.filterRules.Write( "-A", string(kubeIPVSFilterChain), <del> "-m", "set", "--match-set", proxier.ipsetList[kubeExternalIPSet].Name, "dst,dst", "-j", "ACCEPT") </del> <ins> "-m", "set", "--match-set", proxier.ipsetList[kubeExternalIPSet].Name, "dst,dst", "-j", "RETURN") </ins> proxier.filterRules.Write( "-A", string(kubeIPVSFilterChain), "-m", "conntrack", "--ctstate", "NEW",

Kubernetes e2e suite.[It] [sig-network] LoadBalancers should only allow access from service loadbalancer source ranges [Slow] Sept 1st This PR merged around that time ! /sig network /area ipvs
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I agree that is a prime suspect. I can't run "ESIPP" (is there a way?) tests locally, but I will try to run the second test and test manually.
Reproduced and found. The was ; The "ACCEPT" should be replaced with "RETURN".


Question: Annotations: annotations, Labels:      cronJob.Spec.JobTemplate.Labels, OwnerReferences: []metav1.OwnerReference{ <del> *metav1.NewControllerRef(cronJob, appsv1.SchemeGroupVersion.WithKind("CronJob")), </del> <ins> *metav1.NewControllerRef(cronJob, cronJob.GroupVersionKind()), </ins> }, }, Spec: cronJob.Spec.JobTemplate.Spec,

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: of Job created by using command will be . This apiVersion is wrong. What you expected to happen: Set the to How to reproduce it (as minimally and precisely as possible): CronJob manifest: <details</detailsApply CronJob and execute command Anything else we need to know?: Environment: Kubernetes version (use ):
/sig cli
This was introduced by . /assign
GitHub didn't allow me to assign the following users: odino. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : This was introduced by . Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I created fix pull request:


Question: framework.ExpectNoError(err, "failed to list ReplicaSets") framework.ExpectEqual(len(rsList.Items), 0, "filtered list should have no replicas") } <ins> func testRSStatus(f *framework.Framework) { ns := f.Namespace.Name c := f.ClientSet rsClient := c.AppsV1().ReplicaSets(ns) // Define ReplicaSet Labels rsPodLabels := map[string]string{ "name": "sample-pod", "pod":  WebserverImageName, } labelSelector := labels.SelectorFromSet(rsPodLabels).String() rsName := "test-rs" replicas := int32(1) w := &cache.ListWatch{ WatchFunc: func(options metav1.ListOptions) (watch.Interface, error) { options.LabelSelector = labelSelector return rsClient.Watch(context.TODO(), options) }, } rsList, err := c.AppsV1().ReplicaSets("").List(context.TODO(), metav1.ListOptions{LabelSelector: labelSelector}) framework.ExpectNoError(err, "failed to list Replicasets") ginkgo.By("Create a Replicaset") rs := newRS(rsName, replicas, rsPodLabels, WebserverImageName, WebserverImage, nil) testReplicaSet, err := c.AppsV1().ReplicaSets(ns).Create(context.TODO(), rs, metav1.CreateOptions{}) framework.ExpectNoError(err) ginkgo.By("Verify that the required pods have come up.") err = e2epod.VerifyPodsRunning(c, ns, "sample-pod", false, replicas) framework.ExpectNoError(err, "Failed to create pods: %s", err) ginkgo.By("Getting /status") rsResource := schema.GroupVersionResource{Group: "apps", Version: "v1", Resource: "replicasets"} rsStatusUnstructured, err := f.DynamicClient.Resource(rsResource).Namespace(ns).Get(context.TODO(), rsName, metav1.GetOptions{}, "status") framework.ExpectNoError(err, "Failed to fetch the status of replicaset %s in namespace %s", rsName, ns) rsStatusBytes, err := json.Marshal(rsStatusUnstructured) framework.ExpectNoError(err, "Failed to marshal unstructured response. %v", err) var rsStatus appsv1.ReplicaSet err = json.Unmarshal(rsStatusBytes, &rsStatus) framework.ExpectNoError(err, "Failed to unmarshal JSON bytes to a replicaset object type") framework.Logf("Replicaset %s has Conditions: %v", rsName, rsStatus.Status.Conditions) ginkgo.By("updating the Replicaset Status") var statusToUpdate, updatedStatus *appsv1.ReplicaSet err = retry.RetryOnConflict(retry.DefaultRetry, func() error { statusToUpdate, err = rsClient.Get(context.TODO(), rsName, metav1.GetOptions{}) framework.ExpectNoError(err, "Unable to retrieve replicaset %s", rsName) statusToUpdate.Status.Conditions = append(statusToUpdate.Status.Conditions, appsv1.ReplicaSetCondition{ Type:    "StatusUpdate", Status:  "True", Reason:  "E2E", Message: "Set from e2e test", }) updatedStatus, err = rsClient.UpdateStatus(context.TODO(), statusToUpdate, metav1.UpdateOptions{}) return err }) framework.ExpectNoError(err, "Failed to update status. %v", err) framework.Logf("updatedStatus.Conditions: %#v", updatedStatus.Status.Conditions) ginkgo.By("watching for the ReplicaSet status to be updated") ctx, cancel := context.WithTimeout(context.Background(), rsRetryTimeout) defer cancel() _, err = watchtools.Until(ctx, rsList.ResourceVersion, w, func(event watch.Event) (bool, error) { if rs, ok := event.Object.(*appsv1.ReplicaSet); ok { found := rs.ObjectMeta.Name == testReplicaSet.ObjectMeta.Name && rs.ObjectMeta.Namespace == testReplicaSet.ObjectMeta.Namespace && rs.ObjectMeta.Labels["name"] == testReplicaSet.ObjectMeta.Labels["name"] && rs.ObjectMeta.Labels["pod"] == testReplicaSet.ObjectMeta.Labels["pod"] if !found { framework.Logf("Observed replicaset %v in namespace %v with annotations: %v & Conditions: %v", rs.ObjectMeta.Name, rs.ObjectMeta.Namespace, rs.Annotations, rs.Status.Conditions) return false, nil } for _, cond := range rs.Status.Conditions { if cond.Type == "StatusUpdate" && cond.Reason == "E2E" && cond.Message == "Set from e2e test" { framework.Logf("Found replicaset %v in namespace %v with labels: %v annotations: %v & Conditions: %v", rs.ObjectMeta.Name, rs.ObjectMeta.Namespace, rs.ObjectMeta.Labels, rs.Annotations, rs.Status.Conditions) return found, nil } framework.Logf("Observed replicaset %v in namespace %v with annotations: %v & Conditions: %v", rs.ObjectMeta.Name, rs.ObjectMeta.Namespace, rs.Annotations, rs.Status.Conditions) } } object := strings.Split(fmt.Sprintf("%v", event.Object), "{")[0] framework.Logf("Observed %v event: %+v", object, event.Type) return false, nil }) framework.ExpectNoError(err, "failed to locate replicaset %v in namespace %v", testReplicaSet.ObjectMeta.Name, ns) framework.Logf("Replicaset %s has an updated status", rsName) ginkgo.By("patching the Replicaset Status") payload := []byte(`{"status":{"conditions":[{"type":"StatusPatched","status":"True"}]}}`) framework.Logf("Patch payload: %v", string(payload)) patchedReplicaSet, err := rsClient.Patch(context.TODO(), rsName, types.MergePatchType, payload, metav1.PatchOptions{}, "status") framework.ExpectNoError(err, "Failed to patch status. %v", err) framework.Logf("Patched status conditions: %#v", patchedReplicaSet.Status.Conditions) ginkgo.By("watching for the Replicaset status to be patched") ctx, cancel = context.WithTimeout(context.Background(), rsRetryTimeout) defer cancel() _, err = watchtools.Until(ctx, rsList.ResourceVersion, w, func(event watch.Event) (bool, error) { if rs, ok := event.Object.(*appsv1.ReplicaSet); ok { found := rs.ObjectMeta.Name == testReplicaSet.ObjectMeta.Name && rs.ObjectMeta.Namespace == testReplicaSet.ObjectMeta.Namespace && rs.ObjectMeta.Labels["name"] == testReplicaSet.ObjectMeta.Labels["name"] && rs.ObjectMeta.Labels["pod"] == testReplicaSet.ObjectMeta.Labels["pod"] if !found { framework.Logf("Observed replicaset %v in namespace %v with annotations: %v & Conditions: %v", rs.ObjectMeta.Name, rs.ObjectMeta.Namespace, rs.Annotations, rs.Status.Conditions) return false, nil } for _, cond := range rs.Status.Conditions { if cond.Type == "StatusPatched" { framework.Logf("Found replicaset %v in namespace %v with labels: %v annotations: %v & Conditions: %v", rs.ObjectMeta.Name, rs.ObjectMeta.Namespace, rs.ObjectMeta.Labels, rs.Annotations, cond) return found, nil } framework.Logf("Observed replicaset %v in namespace %v with annotations: %v & Conditions: %v", rs.ObjectMeta.Name, rs.ObjectMeta.Namespace, rs.Annotations, cond) } } object := strings.Split(fmt.Sprintf("%v", event.Object), "{")[0] framework.Logf("Observed %v event: %+v", object, event.Type) return false, nil }) framework.ExpectNoError(err, "failed to locate replicaset %v in namespace %v", testReplicaSet.ObjectMeta.Name, ns) framework.Logf("Replicaset %s has a patched status", rsName) } </ins>

[X] test pr : kuberenetes/kubernetes 25 May [x] two weeks soak end date : 8 June [x] test promotion pr : kubernetes/kubernetes According to this APIsnoop query, there are still some remaining ReplicaSet endpoints that are untested. Create a watch to track replica set events Create a replica set with a static label. Confirm that the pods are running. Get the replica set status. Parse the response and confirm that the replica set status conditions can be listed. Update the replica set status. Confirm via the watch that the status has been updated. Patch the replica set status. Confirm via the watch that the status has been patched. Using an existing as a template for a new for daemon set lifecycle test. Due to a test flake with patching the replica set status conditions the final “watch/validation” check is still be written. [It] should validate Replicaset Status endpoints Apr 6 15:16:30.093: INFO: labels: name=sample-pod,pod=httpd STEP: Create a ReplicaSet STEP: Verify that the required pods have come up. Apr 6 15:16:30.099: INFO: Pod name sample-pod: Found 0 pods out of 1 Apr 6 15:16:35.108: INFO: Pod name sample-pod: Found 1 pods out of 1 STEP: ensuring each pod is running STEP: Getting /status Apr 6 15:16:35.136: INFO: ReplicaSet test-rs has Conditions: [] Apr 6 15:16:35.136: INFO: ReplicaSet test-rs Status.Replicas: 1 Apr 6 15:16:35.136: INFO: ReplicaSet test-rs Status.ReadyReplicas: 1 Apr 6 15:16:35.136: INFO: ReplicaSet test-rs Status.AvailableReplicas: 1 STEP: updating the ReplicaSet Status Apr 6 15:16:35.147: INFO: updatedStatus.Conditions: []v1.ReplicaSetCondition{v1.ReplicaSetCondition{Type:"StatusUpdate", Status:"True", LastTransitionTime:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(time.Location)(nil)}},

Reason:"E2E", Message:"Set from e2e test"}} STEP: watching for the daemon set status to be updated Apr 6 15:16:35.149: INFO: Observed event: ADDED Apr 6 15:16:35.149: INFO: Observed event: MODIFIED Apr 6 15:16:35.149: INFO: Observed event: MODIFIED Apr 6 15:16:35.149: INFO: Observed event: MODIFIED Apr 6 15:16:35.149: INFO: Found replica set test-rs in namespace replicaset-2037 with labels: map[name:sample-pod pod:httpd] annotations: map[] & Conditions: [{StatusUpdate True 0001-01-01 00:00:00 +0000 UTC E2E Set from e2e test}] Apr 6 15:16:35.149: INFO: Replica set test-rs has an updated status STEP: get ReplicaSet state Apr 6 15:16:35.152: INFO: rs1.Conditions: []v1.ReplicaSetCondition{v1.ReplicaSetCondition{Type:"StatusUpdate", Status:"True", LastTransitionTime:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(time.Location)(nil)}}, Reason:"E2E", Message:"Set from e2e test"}} Apr 6 15:16:35.152: INFO: ReplicaSet test-rs Status.FullyLabeledReplicas: 1 Apr 6 15:16:35.152: INFO: ReplicaSet test-rs Status.Replicas: 1 Apr 6 15:16:35.152: INFO: ReplicaSet test-rs Status.ReadyReplicas: 1 Apr 6 15:16:35.152: INFO: ReplicaSet test-rs Status.AvailableReplicas: 1 STEP: patching the ReplicaSet Status Apr 6 15:16:35.158: FAIL: Failed to patch replica set status.

"test-rs" is invalid: [status.fullyLabeledReplicas: Invalid value: 1: cannot be greater than status.replicas, status.readyReplicas: Invalid value: 1: cannot be greater than status.replicas, status.availableReplicas: Invalid value: 1: cannot be greater than status.replicas] Unexpected error: <*errors.StatusError | 0xc00024d4a0: { ErrStatus: { TypeMeta: {Kind: "", APIVersion: ""}, ListMeta: { SelfLink: "", ResourceVersion: "", Continue: "", RemainingItemCount: nil, }, Status: "Failure", Message: " "test-rs" is invalid: [status.fullyLabeledReplicas: Invalid value: 1: cannot be greater than status.replicas, status.readyReplicas: Invalid value: 1: cannot be greater than status.replicas, status.availableReplicas: Invalid value: 1: cannot be greater than status.replicas]", Reason: "Invalid", Details: { Name: "test-rs", Group: "apps", Kind: "ReplicaSet", UID: "", Causes: [ { Type: "FieldValueInvalid", Message: "Invalid value: 1: cannot be greater than status.replicas", Field: "status.fullyLabeledReplicas", }, { Type: "FieldValueInvalid", Message: "Invalid value: 1: cannot be greater than status.replicas", Field: "status.readyReplicas", }, { Type: "FieldValueInvalid", Message: "Invalid value: 1: cannot be greater than status.replicas", Field: "status.availableReplicas", }, ], RetryAfterSeconds: 0, }, Code: 422, }, } "test-rs" is invalid: [status.fullyLabeledReplicas: Invalid value: 1: cannot be greater than status.replicas, status.readyReplicas: Invalid value: 1: cannot be greater than status.replicas, status.availableReplicas: Invalid value: 1: cannot be greater than status.replicas] occurred The error looks to be part of the . Looking for suggestions on whats likely to be happening and how to best resolve the flake would be appreciated.

This query shows the endpoints hit within a short period of running the e2e test If a test with these calls gets merged, test coverage will go up by 2 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: local message=$1 local match=$2 <del> if echo "$message" | grep -q "$match"; then </del> <ins> if grep -q "${match}" <<< "${message}"; then </ins> echo "FAIL!" echo "message:$message" echo "has:$match"

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug for lack of better idea hack/make-rules/test-cmd-, line 1443 is failing, even when the debug output shows the expected text: from the test log: What happened: See for example https://k8s-
This should be owned by whoever owns the "test-cmd runkubectlget_tests". This doesn't look like a failure in the testing framework or the infrastructure. I believe this is
the linked test is actually failing on the next test, not the .missing test... from the test log: this is very odd:
the issue appears to be with
potential fix in
seems like the same issue for runpluginstests:
same fix will hopefully resolve the issue


Question: "k8s.io/klog" "k8s.io/utils/mount" utilpath "k8s.io/utils/path" <ins> utilstrings "k8s.io/utils/strings" </ins> v1 "k8s.io/api/core/v1" "k8s.io/apimachinery/pkg/types"

In , we check mount on first level directory in pod volume directories. It is true for many volume plugins, but not for CSI plugin which . cc
/assign
Pardon my ignorance of kubelet source code, but this looks very much similar to an issue we have been running to many times recently, when developing a CSI plugin. The problem is that when CSI volume publish is called and the mount point is not propagated from the CSI plugin pod (either missing bidirectional flag in yaml or not supported by underlaying k8s cluster implementation), the kubelet nevertheless starts a pod using the volume. The volume is not there, so the app writes to root filesystem in fact, leading to all other kinds of problems - i.e. losing data, filling up root fs. IMHO what kubelet should do is to check that the mounted filesystem is indeed visible after publish call and if not it should fail the volume attach request. If this ticket was fixed, would it solve the issue we are seeing or should I open a new ticket? thanks
I think we can potentially add that check but we need to make sure first it can work with ephemeral CSI drivers. Ephemeral drivers would have to "stage" their directories somewhere, like /var/lib/kubelet/plugins/<csi driver/, and then bind mount it to the publish path. cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Hmm... That is a good question. Does csi require a mount to be made or only that the specified directory be created? Will have to go read through the spec...
/remove-lifecycle stale
The only place in the spec I could find was this comment: // For volumes with an access type of mount, the SP SHALL place the // mounted directory at target_path. So I think having Kubelet enforce that check would be ok with the spec, and a good thing to do.
cc CSI spec says that the target path has to be a mount. Shall we go ahead and enforce it then? Are we going to break drivers that may not be doing this?

Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
The issue mentioned here involves two functions . Now with PR , the issue in CheckVolumeExistenceOperation is solved by checking mount path instead of volume path. The only issue here is for getMountedVolumePathListFromDisk which is used during pod deletion. This function was introduced by PR . Now I am questioning whether we need to check the mount point or just need to check pod dirs exist before deleting the pod. Firstly, for some plugins, there might not be mount point. Secondly, without checking mount point, it means that pod cannot be deleted as long as there are pod volume dirs. In normal case, volume dirs should be cleanedup during volume teardown. For cases that kubelet restarts, we have reconciler try to clean up volume mounts if possible. So we can propose a couple of solutions here mount point check. It solves the problem, the downside is that there might be some rare case that reconciler could not clean up dirs after kubelet restarts. check mount point, and also check the next level "/mount" dir. This might still cause problem for some plugins that do not have mounts. cc
Discussed offline with For proposal 1., pod will not be deleted from api server if there are still volume dirs exist. So to make sure it works well, we need to have reliable way to clean up dirs. Normally when kubelet restarts, reconciler will either add it into actual state (if reconstruction works) or try to clean it up directly once unless volume is still in desired state. So if reconstruction does not work, and it fails to clean up during reconciler sync, there is no other chance to clean up the dirs.

Or if volume was in desired state and removed quickly afterwords, the dir might not get the chance to be cleaned up at all. We are thinking to improve this part, we could use mark volume "uncertain" in actual state feature. During reconstruction, we can remove the check for mounted points (since the mount might be gone during node reboots or driver does not have moutns at all). Reconciler could mark those volumes as uncertain (even if it was in desired state) to make sure those volume got cleaned up or remount whenever needed with retries. With this change, I think it is safe to remove the mount point check for pod deletion too.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: } } <del> var info syscall.Sysinfo_t syscall.Sysinfo(&info) procs := int64(info.Procs) rlimit.NumOfRunningProcesses = &procs </del> <ins> // Prefer to read "/proc/loadavg" when possible because sysinfo(2) // returns truncated number when greater than 65538. See // https://github.com/kubernetes/kubernetes/issues/107107 if procs, err := runningTaskCount(); err == nil { rlimit.NumOfRunningProcesses = &procs } else { var info syscall.Sysinfo_t syscall.Sysinfo(&info) procs := int64(info.Procs) rlimit.NumOfRunningProcesses = &procs } </ins> rlimit.Time = v1.NewTime(time.Now()) return rlimit, nil } <ins> func runningTaskCount() (int64, error) { // Example: 1.36 3.49 4.53 2/3518 3715089 bytes, err := ioutil.ReadFile("/proc/loadavg") if err != nil { return 0, err } fields := strings.Fields(string(bytes)) if len(fields) < 5 { return 0, fmt.Errorf("not enough fields in /proc/loadavg") } subfields := strings.Split(fields[3], "/") if len(subfields) != 2 { return 0, fmt.Errorf("error parsing fourth field of /proc/loadavg") } return strconv.ParseInt(subfields[1], 10, 64) } </ins>

Condition PIDPressure is not set to true when the system thread number exceeds even it's reaching the limit. Condition PIDPressure is set to true correctly See cause below The bug is due to , which truncates any process number greater than because of type conversion. See kernel code . Reading it from should be working. v1.23.1 <detailsNA </details<details</details<details</details<details</details<details</details
/sig node
/triage accepted /priority important-longterm


Question: }) } } <ins> func TestCalculatePodRequests(t *testing.T) { containerRestartPolicyAlways := v1.ContainerRestartPolicyAlways testPod := "test-pod" tests := []struct { name             string pods             []*v1.Pod container        string resource         v1.ResourceName expectedRequests map[string]int64 expectedError    error }{ { name:             "void", pods:             []*v1.Pod{}, container:        "", resource:         v1.ResourceCPU, expectedRequests: map[string]int64{}, expectedError:    nil, }, { name: "pod with regular containers", pods: []*v1.Pod{{ ObjectMeta: metav1.ObjectMeta{ Name:      testPod, Namespace: testNamespace, }, Spec: v1.PodSpec{ Containers: []v1.Container{ {Name: "container1", Resources: v1.ResourceRequirements{Requests: v1.ResourceList{v1.ResourceCPU: *resource.NewMilliQuantity(100, resource.DecimalSI)}}}, {Name: "container2", Resources: v1.ResourceRequirements{Requests: v1.ResourceList{v1.ResourceCPU: *resource.NewMilliQuantity(50, resource.DecimalSI)}}}, }, }, }}, container:        "", resource:         v1.ResourceCPU, expectedRequests: map[string]int64{testPod: 150}, expectedError:    nil, }, { name: "calculate requests with special container", pods: []*v1.Pod{{ ObjectMeta: metav1.ObjectMeta{ Name:      testPod, Namespace: testNamespace, }, Spec: v1.PodSpec{ Containers: []v1.Container{ {Name: "container1", Resources: v1.ResourceRequirements{Requests: v1.ResourceList{v1.ResourceCPU: *resource.NewMilliQuantity(100, resource.DecimalSI)}}}, {Name: "container2", Resources: v1.ResourceRequirements{Requests: v1.ResourceList{v1.ResourceCPU: *resource.NewMilliQuantity(50, resource.DecimalSI)}}}, }, }, }}, container:        "container1", resource:         v1.ResourceCPU, expectedRequests: map[string]int64{testPod: 100}, expectedError:    nil, }, { name: "container missing requests", pods: []*v1.Pod{{ ObjectMeta: metav1.ObjectMeta{ Name:      testPod, Namespace: testNamespace, }, Spec: v1.PodSpec{ Containers: []v1.Container{ {Name: "container1"}, }, }, }}, container:        "", resource:         v1.ResourceCPU, expectedRequests: nil, expectedError:    fmt.Errorf("missing request for %s in container %s of Pod %s", v1.ResourceCPU, "container1", testPod), }, { name: "pod with restartable init containers", pods: []*v1.Pod{{ ObjectMeta: metav1.ObjectMeta{ Name:      testPod, Namespace: testNamespace, }, Spec: v1.PodSpec{ Containers: []v1.Container{ {Name: "container1", Resources: v1.ResourceRequirements{Requests: v1.ResourceList{v1.ResourceCPU: *resource.NewMilliQuantity(100, resource.DecimalSI)}}}, }, InitContainers: []v1.Container{ {Name: "init-container1", Resources: v1.ResourceRequirements{Requests: v1.ResourceList{v1.ResourceCPU: *resource.NewMilliQuantity(20, resource.DecimalSI)}}}, {Name: "restartable-container1", RestartPolicy: &containerRestartPolicyAlways, Resources: v1.ResourceRequirements{Requests: v1.ResourceList{v1.ResourceCPU: *resource.NewMilliQuantity(50, resource.DecimalSI)}}}, }, }, }}, container:        "", resource:         v1.ResourceCPU, expectedRequests: map[string]int64{testPod: 150}, expectedError:    nil, }, } for _, tc := range tests { t.Run(tc.name, func(t *testing.T) { requests, err := calculatePodRequests(tc.pods, tc.container, tc.resource) assert.Equal(t, tc.expectedRequests, requests, "requests should be as expected") assert.Equal(t, tc.expectedError, err, "error should be as expected") }) } } </ins>

Calculate sidecar containers resource in pod autoscaler. From 1.28 user could run Sidecar containers in , those sidecar containers remain running during the entire life of the Pod, it should be calculated in . If missing the sidecar resources, we will get a wrong pod resource utilization. We should also correct the related Cadvisor metrics.
/assign /sig autoscaling
/triage accepted /priority important-soon


Question: "github.com/golang/glog" "github.com/skynetservices/skydns/metrics" "github.com/skynetservices/skydns/server" <ins> "github.com/spf13/pflag" </ins> "k8s.io/kubernetes/cmd/kube-dns/app/options" "k8s.io/kubernetes/pkg/api/unversioned" clientset "k8s.io/kubernetes/pkg/client/clientset_generated/internalclientset" "k8s.io/kubernetes/pkg/client/restclient" kclientcmd "k8s.io/kubernetes/pkg/client/unversioned/clientcmd" kdns "k8s.io/kubernetes/pkg/dns" <ins> "k8s.io/kubernetes/pkg/version" </ins> ) type KubeDNSServer struct {

This makes it so much easier to tell what you are debugging. is a not great version--the stringification elides some useful info--but it's a start.
And also flag values.
is the same thing but adds flags.
will handle flags for everything that uses .
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
/remove-lifecycle rotten
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
/cc I think you might be interested in this. Its absence makes debugging harder.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten Adding a good first issue label on this because the idea is clear. If anyone wants to work on this, please see and as examples on how to add the version info. go through all binaries and check if these exist.

If they don't, add these in the glog lines. /good-first-issue
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /remove-lifecycle rotten go through all binaries and check if these exist. If they don't, add these in the glog lines. /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hey may I take this one?
We would love to get this fixed but would prefer if you choose one good-first-issue out of this and first. Feel free to send a PR for one issue and if the other issue hasn't been claimed yet, feel free to send a PR for that one as well. :)
/kind cleanup
please, sync up together, since your PRs are overlapping from kubeadm's perspective :)
, I remove kubeadm's perspective , thanks.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.

</details
/remove-lifecycle rotten
+1
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
/assign
I believe this issue can be closed now, from what I could see all the intended binaries now include version logging.
thanks! Were there some other CLs for binaries besides kubemark?
I count/see the following: Which I think covers everything that makes sense; notable exclusions being hyperkube, kubeadm, kubectl, and a few of the other misc things under /cmd
Seems the list is complete. Should this issue be closed rather still open?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available .


Question: appslisters "k8s.io/client-go/listers/apps/v1beta1" corelisters "k8s.io/client-go/listers/core/v1" "k8s.io/client-go/tools/cache" <ins> "k8s.io/client-go/tools/record" </ins> podutil "k8s.io/kubernetes/pkg/api/v1/pod" "k8s.io/kubernetes/pkg/controller" "k8s.io/kubernetes/pkg/controller/history"

An issue to make bot happy for When StatefulSet finds any of its pods in phase Failed it will delete it and create it again. While doing so it will emit delete+create event which is really confusing for the user as he has no idea what's happening. /sig apps


Question: REGISTRY?=gcr.io/google-containers IMAGE?=debian-hyperkube-base <del> TAG=0.7 </del> <ins> TAG=0.8 </ins> ARCH?=amd64 CACHEBUST?=1

the new hyperkube image has changed without notifying the bash environment now links to sh, there is only one thing worst than an error, and it is a SILENT ERROR this means that anyone that execute kubectl inside the hyperkube and leverage from a bash environment now get the weirdest exceptions , in my case troubles where caused by flexvolume scripted in bash which stopped working without reason and with weird errors logged ( but also there are some other cluster-images based on your hyperkube image, and if you link bash to sh it will spread headache around :) ) while for version 1.7.7 the result was different so , or remove bash, or repristinate bash, link bash-sh is a NO-NO
/assign /sig node


Question: // case 3 { devAddresses: map[string][]string{"encap0": {"10.20.30.40"}, "lo": {"127.0.0.1"}, "docker0": {"172.17.0.1"}}, <del> expectIPs:    []string{"10.20.30.40", "127.0.0.1", "172.17.0.1"}, </del> <ins> expectIPs:    []string{"10.20.30.40", "172.17.0.1"}, </ins> }, // case 4 { devAddresses: map[string][]string{"encaps9": {"10.20.30.40"}, "lo": {"127.0.0.1"}, "encap7": {"10.20.30.31"}}, <del> expectIPs:    []string{"10.20.30.40", "127.0.0.1", "10.20.30.31"}, </del> <ins> expectIPs:    []string{"10.20.30.40", "10.20.30.31"}, </ins> }, // case 5 { devAddresses: map[string][]string{"kube-ipvs0": {"1.2.3.4"}, "lo": {"127.0.0.1"}, "encap7": {"10.20.30.31"}}, <del> expectIPs:    []string{"127.0.0.1", "10.20.30.31"}, </del> <ins> expectIPs:    []string{"10.20.30.31"}, </ins> }, // case 6 { devAddresses: map[string][]string{"kube-ipvs0": {"1.2.3.4", "2.3.4.5"}, "lo": {"127.0.0.1"}}, <del> expectIPs:    []string{"127.0.0.1"}, </del> <ins> expectIPs:    []string{}, </ins> }, // case 7 {

What happened: For services with type NodePort (and LoadBalancer) ipvs entries for localhost (ipv4 and ipv6) are created. Here is an example for a service with so the same nodeport is used for ipv4 and ipv6; The localhost entries can't be used and most likely they never will. What you expected to happen: Since "::1" most likely will never work and 127.0.0.1 will not work without an unlikely Linux kernel update these entries must not be created. Further, since the use of localhost addresses for nodeports is the only know reason to require (), this requirement shall be removed for proxy-mode=ipvs. How to reproduce it (as minimally and precisely as possible): Start a cluster with proxy-mode=ipvs and some services Do; Anything else we need to know?: That ipv6 access to [::1]:nodeport does not work (and most likely never will) is discussed in For 127.0.0.1 is is a bit more complicated as discussed in , with the excellent deep-dive into the kernel by in This may seem like a minor issue but clean-up of useless things is always good. I also have (vague) plans to propose improvements for the address handling for proxy-mode=ipvs and the localhost entries becomes an unwanted problem. If this issue is accepted as a bug I'll be happy to do a PR. Environment: Kubernetes version (use ): all supporting proxy-mode=ipvs Cloud provider or hardware configuration: any OS (e.g: ): any Kernel (e.g. ): any Install tools: none Network plugin and version (if this is a network-related bug): any Others: /kind cleanup /sig network /area ipvs
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
test in ipv6 + iptables , ::1 is not work for NodePort. /assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with .

Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/area kube-proxy
/remove lifecycle-stale
/remove-lifecycle stale <sigh...
Since it is highly unlikely that the kernel will be updated to allow this for ipv4 and it will never work for ipv6 I would really want to remove this from the ipvs proxier.
Do you think I can go ahead and disable the setup for localhost node-ports in the ipvs proxier? Since it have never worked it would not be a user facing change. Or should we discuss it on the sig/network meeting?
I personally think that this was implicitly discussed and confirmed here , it is an implementation detail of IPVS and a known issue that never worked in IPVS, so it will be safe to disable directly without further discussion. However, I defer to for a final decision /assign
I made a PR. ~It's just 3 lines~ Now 8 lines. I forgot the config setting.


Question: logger.Info("Stopping to monitor node resources of the plugin", "reason", context.Cause(ctx), "err", ctx.Err(), "recover", r) }() <ins> backOff := flowcontrol.NewBackOffWithJitter(retryPeriod, maxRetryPeriod, backoffFactor) backOffID := "retry" </ins> // Keep trying until canceled. for ctx.Err() == nil { logger.V(5).Info("Calling NodeListAndWatchResources")

has to retry after failures. This currently uses a fixed 5 second delay to avoid busy-looping. Exponential backoff might be better. /sig node /triage-accepted /priority important-longterm /lifecycle frozen /assign
/triage accepted
/assign


Question: fakeRuntime.Stop() }() fakeRecorder := &record.FakeRecorder{} <del> rtSvc := createRemoteRuntimeService(endpoint, t) </del> <ins> rtSvc := createRemoteRuntimeService(endpoint, t, oteltrace.NewNoopTracerProvider()) </ins> kubeDep := &Dependencies{ Auth:                 nil, CAdvisorInterface:    cadvisor,

spans are not being parented to other spans - propagation of context seems broken. More context: spans should be parented to other spans enable kubelet tracing at v1.28 and hook up a tracing backend. No response <detailsv1.28 at least </details<detailsNot sure this matters. </details<details</details<details</details<details</details<details</details
/sig instrumentation /assign
/triage accepted
!
repro steps:
/assign Thanks for helping to look into this! Feel free to post the findings here you shared with me on slack
GitHub didn't allow me to assign the following users: fatsheep9146. Note that only with read permissions, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Thanks! I will post this ASAP
The broken issue is due to the pr , which try to propagate context brought by pod worker to syncPod function in order to let the syncPod can be interrupted in time by other asynchronous routines, for example eviction. But in the line we did not pass the context which contain the parent span and trace into function, so the trace is broken. So I want to fix this by passing the into , which is also desired in comment But passing the ctx into syncPod may have something other to do as described in comments, so I tried to fix as comments advised.
cc


Question: } # Ensure that we have a password created for validating to the master.  Will <del> # read from the kubernetes auth-file for the current context if available. </del> <ins> # read from kubeconfig for the current context if available. </ins> # # Assumed vars #   KUBE_ROOT

Retrying after gave no joy.
Trying did not help.
will fix it. The problem is that your kubeconfig's current-context doesn't have a corresponding context entry. I'll see if I can make it gce's kube-up little more robust to invalid kubeconfig entries.
Thank you kindly for your help again. My brain is simply not large enough to remember all this.


Question: init() defer func() { <del> volume.TestCleanup(f, convertTestConfig(l.config)) </del> <ins> volume.TestServerCleanup(f, convertTestConfig(l.config)) </ins> cleanup() }()

<!-- Please only use this template for submitting enhancement requests --What would you like to be : In block volume tests, wait for the Pod to be deleted instead of letting it be handled by the framework's namespace deletion Why is this needed: I think the regression caused by could have been caught if the tests check for Pod deletion because the Pod object shouldn't be deleted if TearDown fails.
/assign
This issue is not specific to block tests. We should in ideally wait for pod deletion at the end of all types of tests to confirm cleanup logic is functioning
I checked the tests under test/e2e/storage/testsuites/ where DeletePodWithWait is used to wait for the deletion.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten is this still an issue? One way to check is to make a breaking change in cleanup and see if the pull job fails.
I went through all block volume usage in e2e tests: All of them but one wait for their pods to be deleted. I'm fixing the remaining test in


Question: defer cancel() // Invalidate the cache right after updating <del> if err = ss.deleteCacheForNode(vmName); err != nil { return nil, err } </del> <ins> defer ss.deleteCacheForNode(vmName) </ins> klog.V(2).Infof("azureDisk - update(%s): vm(%s) - detach disk(%s, %s)", nodeResourceGroup, nodeName, diskName, diskURI) return ss.VirtualMachineScaleSetVMsClient.Update(ctx, nodeResourceGroup, ssName, instanceID, newVM, "detach_disk")

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: vmss dirty cache issue, caused by What you expected to happen: clean vmss cache should happen after disk attach/detach operation, now it's before those operations, which would lead to dirty cache. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /kind bug /assign /priority important-soon /sig cloud-provider /area provider/azure


Question: shortDesc: "Aging shows pods by age" longDesc: > Aging shows pods from the current namespace by age. <del> Once we have plugin support for global flags through env vars (planned for V1) we'll be able to switch between namespaces using the --namespace flag. </del> command: ./aging.rb

/kind bug /sig cli What happened: Tried to list the age of pods in the namespace in an otherwise empty cluster $ kubectl --namespace kube-system plugin aging The Magnificent Aging Plugin. No pods What you expected to happen: That the pods in the namespace would have been listed How to reproduce it (as minimally and precisely as possible): the and from into . the command: Anything else we need to know?: No Environment: Kubernetes version (use ): v1.8.2 Cloud provider or hardware configuration: minikube & GKE OS (e.g. from /etc/os-release): Mac OSX El Capitan Kernel (e.g. ): Darwin Kernel Version 15.6.0 Install tools: Google Cloud SDK
hi for cli issue, could you please file it in this repo next time ? thanks !!
Will the k8s github bots be able to resolve references in commit messages made in kubernetes/kubernetes PRs be able to find the corresponding issue ticket in that other project? Semi-OT but related: I know there's an overall intention to refactoring k8s proper into many distinct projects, so it would be helpful if either the contributor guide contained a mapping of each and every module making up a k8s release with its corresponding github project and protocol governing the component. Or if each directory (module) in kubernetes/kubernetes contains its own explaining the contribution protocol. Right now documentation and information about ownership, contribution and protocols of various project modules are a sprawl and a bit of a mess, creating a significant barrier to entry (contribution). What's the proper place and form to discuss these sort of things?
thanks, I think those are some good suggestions, as far as I know you could send an email to our mailling-list(google-groups) or create an new issue then cc


Question: // by REST storage - these typically indicate programmer // error by not using pkg/api/errors, or unexpected failure // cases. <del> runtime.HandleError(fmt.Errorf("apiserver received an error that is not an metav1.Status: %v", err)) </del> <ins> runtime.HandleError(fmt.Errorf("apiserver received an error that is not an metav1.Status: %#+v", err)) </ins> return &metav1.Status{ TypeMeta: metav1.TypeMeta{ Kind:       "Status",

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: I am suffering from mysterious timeouts. See for the evidence from my latest investigation. It includes this line from the function in . The relevant line is this: I still do not know the type of the error in question or where it is coming from. It would be helpful if the log message included the type of the error object --- such as with or, if that does not include the type's package name, perhaps something that would really nail the type. What you expected to happen: I would expect more information about an unexpected type of error. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): 1.10.3 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
I think this timeout is from resourcequota admission. Did you enable admission plugin? will create a pr to make ResourceQuota return err.
thanks for the suggestion, I will respond in the original issue.
I appreciate removing a source of errors of unexpected type, but this issue is about what to do whenever an unexpected type of error does arrive. I suggest formatting it with instead of .
/reopen That PR did not really fix this issue.


Question: rename := o.OldDriverName != "" && o.NewDriverName != "" && o.OldDriverName != o.NewDriverName <ins> substKubeletRootDir := func(s string) string { return strings.ReplaceAll(s, "/var/lib/kubelet/", e2eframework.TestContext.KubeletRootDir+"/") } </ins> patchVolumes := func(volumes []v1.Volume) { if !rename { return

What happened: The CSI hostpath driver used in the e2e tests expects the root directory of nodes to be set to , , however nodes can be configured to use different paths. The tests should find out the root directory from kubelet to allow for this. Several test failures related to CSI Volumes: What you expected to happen: Test should be expected to pass. How to reproduce it (as minimally and precisely as possible): Configure kubelet on cluster nodes to use the non-default path. Run the storage-csi e2e tests. Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig testing
/sig storage /remove-sig testing This is a SIG storage specific issue.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
How? I've asked about that before and the conclusion was that there is no portable way to do that (i.e. no official API). It seems to fall onto cluster admins to configure CSI driver deployments so that this particular aspect matches the cluster. For the tests that means that there has to be a parameter for the kubelet data directory that users of then have to set.
We could add and respect a flag of course then all nodes have to be the same at least. Reminder though
That's indeed another complication.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale Tracked for csi-driver-host-path in There's a bigger issue about both mock and hostpath driver in which includes "support non-default kubelet path".
The current hostpath driver in k/k testing supports non-default paths, but the suite doesn't use that yet.

/assign
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale


Question: return admission.NewForbidden(a, fmt.Errorf("node %q can not create pods that reference a service account", nodeName)) } hasSecrets := false <del> podutil.VisitPodSecretNames(pod, func(name string) (shouldContinue bool) { hasSecrets = true; return false }) </del> <ins> podutil.VisitPodSecretNames(pod, func(name string) (shouldContinue bool) { hasSecrets = true; return false }, podutil.AllContainers) </ins> if hasSecrets { return admission.NewForbidden(a, fmt.Errorf("node %q can not create pods that reference secrets", nodeName)) } hasConfigMaps := false <del> podutil.VisitPodConfigmapNames(pod, func(name string) (shouldContinue bool) { hasConfigMaps = true; return false }) </del> <ins> podutil.VisitPodConfigmapNames(pod, func(name string) (shouldContinue bool) { hasConfigMaps = true; return false }, podutil.AllContainers) </ins> if hasConfigMaps { return admission.NewForbidden(a, fmt.Errorf("node %q can not create pods that reference configmaps", nodeName)) }

Fully described in , and will visit these fields for all containers, regardless of whether they're hidden behind a feature gate (e.g. ). This is usually correct, these methods are often used to drop fields and in this case should visit all containers that are persisted regardless of whether they're disabled, but the callers of these methods should signal that explicitly rather than relying on the implicit behavior. /cc /priority important-longterm /kind bug /sig node
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : Fully described in , and will visit these fields for all containers, regardless of whether they're hidden behind a feature gate (e.g. ). /priority important/long-term /kind bug /sig node Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
are you interested in taking this one, too?
VisitPodSecretNames and VisitPodConfigmapNames already specify AllContainers for VisitContainers Can you let me know what else needs to be done?
Callers should specify whether they want to visit all containers or not. There are valid reasons for each, so we shouldn't assume the desired behavior, even if 100% of the current users want to visit all containers. Jordan's original comment in has some examples of current uses. I don't think this needs to be as configurable as , so my naive first attempt to implement this would probably just be a new to and


Question: scheme := runtime.NewScheme() err := v2beta1scheme.AddToScheme(scheme) require.NoError(t, err) <del> v2scheme.SchemeBuilder.Register(v2scheme.RegisterConversions) </del> err = v2scheme.AddToScheme(scheme) require.NoError(t, err) <ins> err = v2scheme.RegisterConversions(scheme) require.NoError(t, err) </ins> fuzzer := fuzz.NewWithSeed(2374375)

Seeing data race flakes in unit tests: Appears to have just been in /sig api-machinery /priority critical-urgent /assign /cc
This code is mutating the package-level schemebuilder: I think it should be adding the conversions to the local scheme instead:
Fixed in (found several instances)
/triage accepted


Question: // that the request should be executed then `execute()` will be // invoked once to execute the request; otherwise `execute()` will // not be invoked. <ins> // Handle() should never return while execute() is running, even if // ctx is cancelled or times out. </ins> Handle(ctx context.Context, requestDigest RequestDigest, noteFn func(fs *flowcontrol.FlowSchema, pl *flowcontrol.PriorityLevelConfiguration),

As visible in this graph: ! pod-startup time in 5k-node tests increased around beginning of June (only 99% though). It seems to be connected to "run-to-watch" phase. We should investigate and see what we can do with this. /sig scalability /kind bug
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : As visible in this graph: ! /kind bugs Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage accepted Once PR that merged around that time is P&F support for watch-initialitation: As a first step, we may want to confirm/reject if it's this PR and see if we aren't somehow misconfigured here.
I've briefly checked test-infra/perf-tests to see if we had any relevant changes around that time, but I don't see any. Most likely it's related to k/k changes.
/assign
Good news is that we can also see it regression in kubemark (although not as bad as in regular 5k tests): http://perf- But this can help us with finding root cause faster/cheaper. ! I've picked two runs, one before this change, one after: 26th May: Timeline: 12th June: Timeline: First run vs second run I can see significantly higher spike on waiting time in P&F, but according to timeline it seems like it was during huge service and not pod startup latency: 26th May: ! 12th June: ! I can see that number of apiserver goroutines significantly increased (from ~500k to ~900k). I would expect to have 1 more goroutine per watch, we have around 200k watches). ! vs ! Worst 100 pods with slowest latency runtowatch: I've picked the slowest pod "test-2xppkn-33/latency-deployment-60--p7nwz" to analyse.

This pod was bound to node gce-scale-cluster-minion-group-4-txww: Kubelet log for this pod: CL2 logs for this pod: In apiserver logs we can see: So it looks like updating pod status was delayed by P&F by ~500 ms for this pod. Timeline for this pod: 18:37:46. scheduled 18:37:48. container started 18:37:49. kubelet gets pod (1.3s latency with 610 ms delay from P&F) 18:37:50. patch status of pod (545ms latency with 500ms delay from P&F) 18:37:51. observed by CL2 as running To compare with previous run we have this timeline for pod "test-rcdjgd-26/latency-deployment-69--6s95x 2.724031741s": 18:57:08. scheduled on node gce-scale-cluster-minion-group-3-l21s 18:57:10. container started 18:57:10. kubelet gets pod (364 ms latency with 345 ms delay from P&F) 18:57:11. patched status od pod (360 ms latency with 254 ms delay from P&F) So to compare, for these two pods major difference was "get pod" + "patch status" in kubelet: 1.8s vs 700ms. 1s of 1.8s of latency was due to P&F and 0.6s of 0.7s was due to P&F.
In both runs I don't see significant waiting time (99th percentile) on any of Priority levels during pod startup latency phase: 26th May: ! 12th June: !
1) Do we have data about 100% waiting time in P&F? With 99% it my not be enough to to see that data. 2) If really waiting time is small, then the question is what exactly. (a) it might be related to the number of goroutines (b) it might be locking (c) it might be something else... I have just opened the WIP PR: that should help with first two (at least to some extent). Will you be able to test if that helps?

Max waiting time in P&F for pod startup latency phase: 26th May: ! 12th June: ! Buckets are ..., 0.2, 0.5, ... so I don't think it's significant. will test it.
This is definitely non-negligible regression that we should address in 1.22 timeframe. I'm actually have a promissing results for it. /milestone v1.22 /priority critical-urgent
OK - so we proved that addresses the problem: it reduces back to previous value the number of goroutines ! it brings back pod startup time back to less than 3.4s on 99th percentile it brings back API call latencies I'm going to work with reviewers on merging this to fix the regression. - FYI


Question: }{ "No Discovery Error": { discoveryError:    nil, <del> possibleResources: nil, </del> <ins> possibleResources: possibleDiscoveryResource(), </ins> expectedErr:       false, }, "Discovery Calls Partially Failed": { <del> discoveryError: new(discovery.ErrGroupDiscoveryFailed), possibleResources: []*metav1.APIResourceList{ { GroupVersion: "create/v1", APIResources: []metav1.APIResource{ { Name:       "jobs", Verbs:      []string{"create", "list", "watch", "delete"}, ShortNames: []string{"jz"}, Categories: []string{"all"}, }, }, }, }, expectedErr: false, </del> <ins> discoveryError:    new(discovery.ErrGroupDiscoveryFailed), possibleResources: possibleDiscoveryResource(), expectedErr:       false, </ins> }, } for name, test := range tcs {

In we observed a crashlooping controller cause a cluster cold boot failure, as controller-manager wasn't able to stay up long enough to run the deployment that would have fixed the problem. We should audit everything that runs in kube-controller-manager and ensure that they all fail gracefully enough to prevent crashloops, or make them individually restart in that case, instead of the process. Longer term, we should consider classifying the controllers by run level or bootstrap order (), and splitting them into separate binaries or changing the start-up order in kcm to make sure this sort of thing doesn't happen in the future.
Same issue with metrics-server, using minikube. Also reproduced on our test cluster. Steps to reproduce: Set up localkube Install metrics-server Stop minikube: sudo minikube stop Start minikube: sudo -E minikube start --vm-driver=none Expected: Kubernetes is running Actual: Kubernetes is dead Logs: Workaround: To fix the system, we must delete the metrics-server: Now kubernetes will start normally. After that we can reinstall the metrics server, and it will be ok.
Related:
This will be a hard problem to fix permanently. In all likelihood, we should stop allowing Fatalf() in kubernetes/kubernetes, but as you can see we do this to ourselves right here...
/kind bug
I also think this bug would benefit from an e2e test where we misconfigure things in a variety of ways and verify that KCM still comes up and stays up: [ ] misconfigured apiservice [ ] broken service attached to that apiservice [ ] RBAC rules that disallow things. maybe everything! [ ] webhooks that disallow everything [ ] yr example here
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: proxyTransport   http.RoundTripper } <ins> // ServiceNodePort includes protocol and port number of a service NodePort. type ServiceNodePort struct { // The IP protocol for this port. Supports "TCP" and "UDP". Protocol api.Protocol // The port on each node on which this service is exposed. // Default is to auto-allocate a port if the ServiceType of this Service requires one. NodePort int32 } </ins> // NewStorage returns a new REST. func NewStorage(registry Registry, endpoints endpoint.Registry, serviceIPs ipallocator.Interface, serviceNodePorts portallocator.Interface, proxyTransport http.RoundTripper) *ServiceRest {

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --BUG REPORT /kind bug What happened: I tried to submit a Service Replace () to the kube-apiserver and this results the kube-apiserver container crashing and restarting. PUT localhost:8001/api/v1/namespaces/<namespace/services/<servicenameWhat you expected to happen: I expected the server not to crash and to get validation errors if the request was invalid. How to reproduce it (as minimally and precisely as possible): Create a service, get the resourceVersion and clusterIP from the created resource GET localhost:8001/api/v1/namespaces/<namespace/services/<servicenameSubmit a replace command to PUT localhost:8001/api/v1/namespaces/<namespace/services/<servicenamewith body like If the same request is submitted with the nodePort fields in the ports array then the request succeeds. I get that I may be sending an invalid request without a node port (though shouldn't the merge deal with that?) but it shouldn't take down the API server. Anything else we need to know?: Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-05-10T23:29:08Z", GoVersion:"go1.8.1", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-10-11T23:16:41Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: Bare metal cluster OS (e.g. from /etc/os-release): Ubuntu 16.04.3 LTS (Xenial Xerus) Kernel (e.g. ): 4.4.0-98-generic Install tools: Others:
thanks for reporting this! Seems like this is similar to and should be fixed by . can you PTAL that PR to see if it's good to merge?
/sig api-machinery
Duplicate of

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Re-applying a NodePort service that has two ports fails and crashes the API server: What you expected to happen: Do not crash, service is configured correctly. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: on VBox provider in macOS OS (e.g. from /etc/os-release): macOS/Minikube Kernel (e.g. ): Install tools: Others:
Seems that in func updateNodePorts, it ignored protocol when Detecting duplicate nodeport, I would have a fix on it :)
/sig network
FYI: I think this would be fixed in
/cc Maybe this is the time to have another review on and merge it :)
/assign
Fine, That makes sense

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug /sig network /cc /assign What happened: a ClusterIP type service the service. Change it's type to NodePort and add a Port. Note: the new port has same port number but protocol with the existing Port. the updated service. The TCP Port was eaten. What you expected to happen: The updated service has both TCP and UDP ports. How to reproduce it (as minimally and precisely as possible): See the step above. Anything else we need to know?: If I create a NodePort service with the above two ports, it works. I write a test case to verify this case. I got a panic. The root cause is as below. We should take another look at this logic. BTW: changed some related code but it doesn't change the logic here. So I believe this bug is not introduced by . I also tested this on v1.5.1, it has the same issue. Environment: Kubernetes version (use ): HEAD Cloud provider or hardware configuration**: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
I will be OOO the next week. Since this bug is not so urgent, I will fix it ASAP when I'm back. Folks who are interested in this can take it :) BTW, I can join in the discussion from my phone.
I'm not sure this is a bug. In the first configuration yaml, you're defining the ports array twice. I'm the second, you're only defining it once. The ports array should only be defined once.
Did you mean the below yaml by ? Please note that the two ports have same port number but different protocols. This is allowed.
Yes, that's the one I meant. But note you have "ports:" twice.
Ah, thanks for pointing it out! This might be the cause. But I remembered I also tested the case without the second line, it had the same issue. I can't verify this now. Will test it when I'm back to my keyboard.
I try the case without the second line, the issue also exists.

So I believe there is a bug here. Seems like there might be two issues here: a port is lost as is described in this issue. we shouldn't allow to configure two lines in the manifest. i.e. we should add validation for this case.
/remove-priority awaiting-more-evidence
can you please provide (or edit) example YAMLs then to allow for attempts at duplication?
Update: reproducing without the extra line. a ClusterIP type service the service (). Change it's type to NodePort and add a Port which has the same port number but different protocol with the existing Port. the updated service. We can see that the UDP Port was not created. I have dug into the code and will send a patch to fix it.
PR sent. Reviews welcome! Thanks!
There are/were bugs in the patch implementation wrt ports, because they need compound merge keys.
Yeah I had some discussios with in . But we also do have a bug in our service updating codes. See:


Question: if isHTTPErrorCode(err, http.StatusNotFound) { return false, true, nil } <del> return false, false, fmt.Errorf("error getting load balancer's target pool: %v", err) } if translateAffinityType(affinityType) != tp.SessionAffinity { </del> <ins> // Err on the side of caution in case of errors. Caller should notice the error and retry. // We never want to end up recreating resources because gce api flaked. return true, false, fmt.Errorf("error getting load balancer's target pool: %v", err) } // TODO: If the user modifies their Service's session affinity, it *should* // reflect in the associated target pool. However, currently not setting the // session affinity on a target pool defaults it to the empty string while // not setting in on a Service defaults it to None. There is a lack of // documentation around the default setting for the target pool, so if we // find it's the undocumented empty string, don't blindly recreate the // target pool (which results in downtime). Fix this when we have formally // defined the defaults on either side. if tp.SessionAffinity != "" && translateAffinityType(affinityType) != tp.SessionAffinity { glog.Infof("LoadBalancer target pool %v changed affinity from %v to %v", name, tp.SessionAffinity, affinityType) </ins> return true, true, nil } return true, false, nil

Restarting master causes a recreation of fwrule and target pool for type=loadbalancer services. We are saved in the non restart case because we compare a cached verson of the service to decided if we should even call into the cloudprovider and update resources. This is happening because our reconciliation routines are confused about session affintiy and lbip, leading them to recreate, since inplace update isn't allowed. The impact is at best a few seconds of radio silence, at worst who knows? Repro: Create a loadbalanced service, eg: Wait for it to acquire a vip. Try it out Ssh into master and restart kube-controller-manager: Watch the wgets, you'll notice radio silence after a bit. This is because the targetpool and forwarding rule get recreted, for 2 reasons: affinity mismatch So this check fails: Lbip mismatch: So this check fails: (lbip is retrieved here: )
the session affinity fix is trivial. The easiest change for the lb ip fix is to not actually check the ip, but simply backoff as long as the forwarding rule exists. This is "bad" because of the way we allocate a static-ip, assign it to the fwd rule, then de-allocate it. The alternative is to use to reconcile, but that's racy.
cherry pick


Question: // Partially update a node using strategic merge patch kubectl patch node k8s-node-1 -p '{"spec":{"unschedulable":true}}' <ins> // Update a container's image; spec.containers[*].name is required because it's a merge key kubectl patch pod valid-pod -p '{"spec":{"containers":[{"name":"kubernetes-serve-hostname","image":"new image"}]}}' </ins> .fi .RE

The current documentation is very vague and doesn't explain what can be updated in this way. Can I rename or stop the pod, change number of replicas in RC etc. For example it seems to be impossible to change pod name:
cc
After an in person discussion with we think the docs should point people to the swagger-ui to find if a field is mutable. I'll add this point into the docs.


Question: obj.LeaderElection.LockObjectName = kubescedulerconfigv1alpha1.SchedulerDefaultLockObjectName } <del> if obj.PercentageOfNodesToScore == 0 { // by default, stop finding feasible nodes once the number of feasible nodes is 50% of the cluster. obj.PercentageOfNodesToScore = 50 } </del> if len(obj.FailureDomains) == 0 { obj.FailureDomains = kubeletapis.DefaultFailureDomains }

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Kubernetes scheduler has that specifies what percentage of the cluster nodes should be scored in each scheduling cycle. The scheduler stops scanning more nodes once that many nodes are found feasible for running the pod under consideration. The default value of this config option is 50 (for 50%). We should let the scheduler determine this value based on the size of the cluster when this config option is not provided. I'd propose this formula for finding the percentage: . This is a linear formula that gives us 50 when the cluster size is 100 nodes and 10 for a 5000 node cluster. The scheduler already has a lower bound of 50 nodes per scheduling cycle. As a part of this change, we should also add a 5% lower bound for the formula (If the result of the formula is below 5, we should return 5). Why is this needed: To improve scheduler performance in large clusters. <!-- DO NOT EDIT BELOW THIS LINE --/kind feature /sig scheduling
/assign /remove-help
/assign wgliang
Thanks


Question: Delete(name string) error Create(pod *api.Pod) (*api.Pod, error) Update(pod *api.Pod) (*api.Pod, error) <ins> Watch(label, field labels.Selector, resourceVersion string) (watch.Interface, error) </ins> Bind(binding *api.Binding) error }

PodInterface should have Watch method behaving in the same way as ServiceInterface.Watch does. Use case: a monitoring system (namely Prometheus) expects to get the list of monitored servers performing a DNS-SD query. Special service (similar to kube2sky shipped with Kubernetes) mirrors status of running Pods into etcd, which is in turn monitored by SkyDNS server and exported as DNS-SD. Schematically the data flow looks like this: Kubernetes -kube2sky -etcd -SkyDNS -Prometheus. The biggest challenge is to deliver list of all running pods from Kubernetes to kube2sky. Currently kube2sky can only poll Kubernetes with PodInterface.List() each N seconds, which is not efficient. Having a Watch method would be much better.
cc Do you know why we do not support watch?
Every object should support Watch.
Watching status doesn't currently work -- that's .
I agree with OP. This is just an accident of history; some boilerplate in the client isn't completely filled out. As Brian mentions, not every field is currently watchable but that's being addressed elsewhere.
This is relevant to your interests potentially w/ your work on
This is a pretty simple change if someone wants to get their feet wet. :)
I'll work on it.


Question: # `go test` does not install the things it builds. `go test -i` installs # the build artifacts but doesn't run the tests.  The two together provide # a large speedup for tests that do not need to be rebuilt. <del> printf "%sn" "${@}" | grep -Ev $cover_ignore_dirs | xargs -I{} -n1 -P${KUBE_COVERPROCS}  bash -c "set -o pipefail; _pkg="{}"; _pkg_out=${_pkg////_};  go test -i ${goflags[@]:+${goflags[@]}}  ${KUBE_RACE}  ${KUBE_TIMEOUT}  -cover -covermode="${KUBE_COVERMODE}"  -coverprofile="${cover_report_dir}/${_pkg}/${cover_profile}"  "${KUBE_GO_PACKAGE}/${_pkg}"  ${testargs[@]:+${testargs[@]}} go test ${goflags[@]:+${goflags[@]}}  ${KUBE_RACE}  ${KUBE_TIMEOUT}  -cover -covermode="${KUBE_COVERMODE}"  -coverprofile="${cover_report_dir}/${_pkg}/${cover_profile}"  "${KUBE_GO_PACKAGE}/${_pkg}"  ${testargs[@]:+${testargs[@]}}  | tee ${junit_filename_prefix:+"${junit_filename_prefix}-$_pkg_out.stdout"}  | grep "${go_test_grep_pattern}""  && test_result=$? || test_result=$? </del> <ins> printf "%sn" "${@}"  | grep -Ev $cover_ignore_dirs  | xargs -I{} -n 1 -P ${KUBE_COVERPROCS}  bash -c "set -o pipefail; _pkg="$0"; _pkg_out=${_pkg////_};  go test -i ${goflags[@]:+${goflags[@]}}  ${KUBE_RACE}  ${KUBE_TIMEOUT}  -cover -covermode="${KUBE_COVERMODE}"  -coverprofile="${cover_report_dir}/${_pkg}/${cover_profile}"  "${KUBE_GO_PACKAGE}/${_pkg}"  ${testargs[@]:+${testargs[@]}} go test ${goflags[@]:+${goflags[@]}}  ${KUBE_RACE}  ${KUBE_TIMEOUT}  -cover -covermode="${KUBE_COVERMODE}"  -coverprofile="${cover_report_dir}/${_pkg}/${cover_profile}"  "${KUBE_GO_PACKAGE}/${_pkg}"  ${testargs[@]:+${testargs[@]}}  | tee ${junit_filename_prefix:+"${junit_filename_prefix}-$_pkg_out.stdout"}  | grep "${go_test_grep_pattern}""  {}  && test_result=$? || test_result=$? </ins> produceJUnitXMLReport "${junit_filename_prefix}"

Haven't had time to debug, but just recording here.
Got the same issue here, the root cause is that in On BSD, the -I option has constraint of 255 bytes limit, so the arguments got truncated. This did not happen on Linux as GNU has no such limitation. -I replstr Execute utility for each input line, replacing one or more occur- rences of replstr in up to replacements (or 5 if no -R flag is specified) arguments to utility with the entire line of input. The resulting arguments, after replacement is done, will not be allowed to grow beyond 255 bytes; this is implemented by concate- nating as much of the argument containing replstr as possible, to the constructed arguments to utility, up to 255 bytes. One solution is to extract the bash call command to a function, likely to be , and change the code to something like all parameters need to be passed in to the function, still considering on how to optimize the call.
we have to put the newfunc somewhere rather than test-, is a candidate, as the script need to be sourced before the function call is available. and test- can not be source. Another option is to drop the xargs call, a function named runTestIterations is in test-, which is called when coverage is not enabled, evaluating the possibility of adapting this function.
The other problem with this solutions is the hugely long path with tons of subdirs created by . Eg. which is very painful.


Question: import ( "bytes" "context" <ins> "encoding/json" </ins> "errors" "fmt" "math/rand"

According to this APIsnoop query, there are still some remaining Endpoints endpoints which are untested. Create a Service in a namespace List all Endpoints in all Namespaces find the created Service(1) Replace/update the Namespaced Endpoint for the Service(1) Patch the Namespaced Endpoint for the Service(1) Get the Namespaced Endpoint for the Service(1) Delete Namespaced Endpoint(1) via a Collection with a LabelSelector Discover useragents: If a test with these calls gets merged, Conformance coverage will go up by 4 points /sig testing /sig architecture /area conformance
When you create a selector, race conditions can occur between controllers and our test code. We have services that test endpoint code, we could do these as explicit tests that are NOT handled by the endpoints controller. Let's try this with are managed by the endpoints controller w/ a selector. Endpoints normally go along with services: we want to create a service sans selector. This keeps it out of the endpoints controller, so we can test directly. Agreed for backlog. Double check that we don't already have test for that, whether it's being done by the endpoints controller that doesn't hit the API. Look at ALL the user agents that hit the endpoint.
/reopen /open Needs promotion!
Reopened this issue. <detailsIn response to : /reopen /open Needs promotion! Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/retitle Promote CoreV1Endpoints test - +4 coverage
This doesn't have an APISnoop PR because (the exact process wasn't in place properly when the original mock test ticket was made)
/retitle Write CoreV1Endpoints test+promote - +4 coverage
please close this Issue in favour of and place in In Progress Issues on org dashboard
/close Close this Issue in favour of and place in In Progress Issues on org dashboard
Closing this issue. <detailsIn response to : /close Close this Issue in favour of and place in In Progress Issues on org dashboard Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/close Close this Issue in favour of and place in In Progress Issues on org dashboard


Question: }) } } <ins> func TestInvalidOpenAPIV3Document(t *testing.T) { tests := map[string]struct { spec *spec3.OpenAPI }{ "nil document correctly returns Unsupported error": { spec: nil, }, "empty document correctly returns Unsupported error": { spec: &spec3.OpenAPI{}, }, "minimal document correctly returns Unsupported error": { spec: &spec3.OpenAPI{ Version: "openapi 3.0.0", Paths:   nil, }, }, "document with empty Paths correctly returns Unsupported error": { spec: &spec3.OpenAPI{ Version: "openapi 3.0.0", Paths:   &spec3.Paths{}, }, }, } gvk := schema.GroupVersionKind{ Group:   "batch", Version: "v1", Kind:    "Job", } for tn, tc := range tests { t.Run(tn, func(t *testing.T) { verifier := &queryParamVerifierV3{ finder: NewCRDFinder(func() ([]schema.GroupKind, error) { return []schema.GroupKind{}, nil }), root:       &fakeRoot{tc.spec}, queryParam: QueryParamFieldValidation, } err := verifier.HasSupport(gvk) if err == nil { t.Errorf("Expected not supports error, but none received.") } }) } } // fakeRoot implements Root interface; manually specifies the returned OpenAPI V3 document. type fakeRoot struct { spec *spec3.OpenAPI } func (f *fakeRoot) GroupVersions() ([]schema.GroupVersion, error) { // Unused return nil, nil } // GVSpec returns hard-coded OpenAPI V3 document. func (f *fakeRoot) GVSpec(gv schema.GroupVersion) (*spec3.OpenAPI, error) { return f.spec, nil } func (f *fakeRoot) GVSpecAsMap(gv schema.GroupVersion) (map[string]interface{}, error) { // Unused return nil, nil } </ins>

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Following the example here: Everything here works when you use , but when using 1.27.1, I see the following panic: What you expected to happen: When downgrading, this works as expected: How to reproduce it (as minimally and precisely as possible): <!-- Please make sure you are able to reproduce the bug using a supported version and version skew of Kubernetes. See for which versions and are currently supported. --You will need a Kubernetes cluster with Agones installed (I'm testing on 1.25.x, as you can see above), then follow this quickstart: Anything else we need to know?: The is implemented as an API Extension, not a CRD -- which is probably where the issue lies. Looking at the line: There seems to be an assuming that there will be a under and , in - there should be some nil checks in there. Environment: Kubernetes client and server versions (use ): See above Cloud provider or hardware configuration: GKE OS (e.g: ): Linux
seems like an another edge case; Due to the server version is old and openapiv3 is returning empty?
Thanks for opening this issue /triage accepted
I'm digging into the Agones code, but I'm 99.4% sure we don't even implement openapiv3 at all for our api endpoint
Yep, confirmed -- I see a bunch of 404's in my API Extension logs for requests for
/transfer kubernetes /kind regression
/sig cli /assign
I can also work on tomorrow if Sean can not get time to look at it
If nothing else, this does also force me to publish an openapi spec for our API extension
not sure if needed to handle nils inside the schema better, or if didn't fall back to v2 correctly in the absence of v3 data
It appears to be the former, where an OpenAPI V3 document is returned without , while the code assumes this field is non-nil.
seems like that should still fall back to v2
Can we get a kubectl -v8 to see what actually happens with downloading the OpenAPI?
As requested!

Thanks for the quick response. It looks like the OpenAPI V3 document is almost completely empty The code is expecting an Open API document with . I will have a fix out shortly.
I was digging through that - I found that really interesting. We only expose an basically empty spec for the api extension. It's always "just worked", so never needed to update it. I am curious where K8s is even getting the something that it is outputting for , but that's a side note.