Question: ${ETCD_CMD}  --name="etcd-$(hostname)"  --debug  <ins> --force-new-cluster  </ins> --data-dir=${DATA_DIRECTORY}  --listen-client-urls http://127.0.0.1:${ETCD_PORT}  --advertise-client-urls http://127.0.0.1:${ETCD_PORT}

It appears that the API server is no longer able to talk to etcd after an upgrade. See https://k8s- for an example. Interesting bits from api server log:
I have debugged a lot of 2.3.7 -3.0.14 upgrades and adding to every invocation helps a ton. Any chance we can twiddle this knob when we run these tests? Perhaps via instance metadata?
shows etcd starting, and that 3.0.14 stays running for ~10 hours, and apparently appending to its log. It looks like someone is talking to it. The failure modes I saw in testing upgrades were generally that etcd would crash, which is not what we see here.
One thing just occurred to me. We should probably have a test suite like these upgrades that simply tries to upgrade etcd without changing the version of k8s. This would let us isolate etcd failures from failures of other types.
- I think there is still one flag fix in the "migration" script that you found but didn't create a PR for. Maybe this is also reason of this failure? Can you please finish the loop and finally create that PR so that we can have the script in head updated?
I'm reopening for verifying if the problem is fixed.
I have re-built . I'll see if I can abort the current run of https://k8s-
We are currently using "3.0.14-alpha.1" image. So if you want to try a different image, I think you should update manifest files to the new image. Or rebuild 3.0.14-alpha.1 image :) [Unless I'm missing something how the test works.]
D'oh. Thanks, I just assumed the default tag in the Makefile was correct.
Should I be looking into this job: https://k8s- From the logs that pasted at the beginning (and which appear in all logs) and seems that the problem is with certificated files, not with etcd itself. - did anything change around how certs are handled? Any ideas why those tests may be facing this issue?
Actually, looking at the log that is causing it to fail: I very strongly suspect that the victim of those failures is this PR: - can you please take a look if the hypothesis sounds reasonable to you?

I have seen something related, probably the same issue. I was testing 1.4.7, persisted the etcd data dir, restarted the cluster on master and wasn't able to access any resources created when the cluster was running in 1.4.7. I was able to see those resources in etcd by using curl directly on etcd. Using etcd 3.0.15
Ok, ya this was caused by . I tried to revert and got some nasty conflicts. I'm hoping that will be ready by the EOD and will fix this issue.
thanks!
can you confirm that your PR would have caused the same problem if was running etcd v2? I'm trying to rule out any interaction with etcd v3 or our scripts related to it.
yes, I don't think this problem is etcd related, as far as i can tell right now.
- is this issue still relevant? Or can we close as obsolete?


Question: zone := testContext.CloudConfig.Zone region, err := gcecloud.GetGCERegion(zone) if err != nil { <del> glog.Fatalf("error parsing GCE region from zone %q: %v", zone, err) </del> <ins> return fmt.Errorf("error parsing GCE/GKE region from zone %q: %v", zone, err) </ins> } managedZones := []string{zone} // Only single-zone for now cloudConfig.Provider, err = gcecloud.CreateGCECloud(testContext.CloudConfig.ProjectID, region, zone, managedZones, "" /* networkUrl */, tokenSource, false /* useMetadataServer */) if err != nil { <del> glog.Fatal("Error building GCE provider: ", err) </del> <ins> return fmt.Errorf("Error building GCE/GKE provider: ", err) </ins> } <del> } if testContext.Provider == "aws" { </del> <ins> case "aws": </ins> awsConfig := "[Global]n" if cloudConfig.Zone == "" { <del> glog.Fatal("gce-zone must be specified for AWS") </del> <ins> return fmt.Errorf("gce-zone must be specified for AWS") </ins> } awsConfig += fmt.Sprintf("Zone=%sn", cloudConfig.Zone) if cloudConfig.ClusterTag == "" { <del> glog.Fatal("--cluster-tag must be specified for AWS") </del> <ins> return fmt.Errorf("--cluster-tag must be specified for AWS") </ins> } awsConfig += fmt.Sprintf("KubernetesClusterTag=%sn", cloudConfig.ClusterTag) var err error cloudConfig.Provider, err = cloudprovider.GetCloudProvider(testContext.Provider, strings.NewReader(awsConfig)) if err != nil { <del> glog.Fatal("Error building AWS provider: ", err) </del> <ins> return fmt.Errorf("Error building AWS provider: ", err) </ins> } <del> } // Disable skipped tests unless they are explicitly requested. if config.GinkgoConfig.FocusString == "" && config.GinkgoConfig.SkipString == "" { // TODO(ihmccreery) Remove [Skipped] once all [Skipped] labels have been reclassified. config.GinkgoConfig.SkipString = `[Flaky]|[Skipped]|[Feature:.+]` </del> } <del> gomega.RegisterFailHandler(ginkgo.Fail) c, err := loadClient() if err != nil { glog.Fatal("Error loading client: ", err) } </del> <ins> return nil } // There are certain operations we only want to run once per overall test invocation // (such as deleting old namespaces, or verifying that all system pods are running. // Because of the way Ginkgo runs tests in parallel, we must use SynchronizedBeforeSuite // to ensure that these operations only run on the first parallel Ginkgo node. // // This function takes two parameters: one function which runs on only the first Ginkgo node, // returning an opaque byte array, and then a second function which runs on all Ginkgo nodes, // accepting the byte array. var _ = ginkgo.SynchronizedBeforeSuite(func() []byte { // Run only on Ginkgo node 1 </ins> // Delete any namespaces except default and kube-system. This ensures no // lingering resources are left over from a previous test run. if testContext.CleanStart { <ins> c, err := loadClient() if err != nil { glog.Fatal("Error loading client: ", err) } </ins> deleted, err := deleteNamespaces(c, nil /* deleteFilter */, []string{api.NamespaceSystem, api.NamespaceDefault}) if err != nil { <del> t.Errorf("Error deleting orphaned namespaces: %v", err) </del> <ins> Failf("Error deleting orphaned namespaces: %v", err) </ins> } glog.Infof("Waiting for deletion of the following namespaces: %v", deleted) if err := waitForNamespacesDeleted(c, deleted, namespaceCleanupTimeout); err != nil { <del> glog.Fatalf("Failed to delete orphaned namespaces %v: %v", deleted, err) </del> <ins> Failf("Failed to delete orphaned namespaces %v: %v", deleted, err) </ins> } }

Both the GKE and Parallel tests recently had a run where they timed out. (after 90 minutes!) This is like 10x longer than the parallel run should be taking.
And the build project, too:
any ideas?
p0
yup, investigating this is next on my queue, right after I figure out why the PR jenkins is failing so many runs.
This happened again: In fact I've seen at least few cases like that: "at some point tests just stop and completely nothing is happening to the end" The same happened here - completely nothing happened during last hour of the run,
While the failure rates of the individual test jobs is not very high (e.g. kubernetes-e2e-gce-parallel : MTTF (Last 7 Days) 7 hr 27 min ... the fact that we gate auto-merging on about 7 of these jobs means that a merge fails almost every hour due to this.
I looked at 4 of the builds in kubernetes-e2e-gce-parallel, comparing the test cases marked "N/A" (i.e. no test result given) instead of their usual "PASSED" or "SKIPPED" report. So far, there's been only one test case that's been consistently N/A in all four of the builds I looked at: I'm not sure what might be happening, but I'm tempted to disable this test in parallel runs now while we investigate what's actually wrong.
OK, that test has been disabled. I'm hoping that the PR builds recover (along with some git fixes as referenced in ). Now on to figuring out why this is causing us problems...
Had a successful failure in build 8902. Grepping out the Ginkgo node number shows this test being the problem: So it looks like it got stuck in ? Is there anything in here that should hang without deadline?
I had another run get stuck in exactly the same place: What's worrying is that another parallel build (http://go/k8s-test/job/kubernetes-e2e-gce-parallel/7960/) timed out again early this morning, but this networking test was not enabled. This suggests that there's still something else triggering the problem.
I may have a partial fix for this in .
Have we seen this since merged? Can we close this bug?
- I think we had a lot of other problems during that time.

I would suggest leaving it open during the weekend and close it on Monday if we don't see any problems.
I don't think we've seen this in a while. Please reopen if we're still having mysterious timeouts in Jenkins.
We're seeing this again in . It looks like someone put "Networking should function for intra-pod communication" test back in the parallel run recently. I think that we should revert whatever that PR was.
That test had been fine. More worrying is that my fix from seems to have vanished.
OK, looks like that was removed in , referencing issue . I'm not sure if the same issue is happening again (where operations mysteriously hang forever) or if it's something else. any thoughts?
I think your answer is here: The timeout field, and all references to it in clients were removed, based on the fact that clients should use request.Timeout() instead, but the clients were not updated to do that. I will send a PR shortly.
It seems like very few of the e2e tests are using this method, but we aren't seeing issues on most of the tests. I wonder if there's something in particular we're hitting with the networking test?
Could this be this magic number in ? I guess this means the checks only happen for about a 75 seconds. If one or two of the nodes are spinning up very slowly, this will trip over . Just a thought. Either way I think this needs to be configurable, I'll work on that angle.
I agree this "15" should probably be bigger. However, I'm pretty sure, that this shouldn't cause the test hang forever - it may cause test failures, but hanging forever test shouldn't happen...
Anyway - I've sent out out for review - we will see if that helps...
OK - it won't help, because it doesn't work :( - I asked if there is any reasonably easy way to do it...
Another instance: Again - the "Networking should function for intra-pod communication" is among tests that didn't run...
assigning this issue to you since you're working on a PR
- sure, although I'm not sure if this will solve the issue...
you can assign back to me if it doesn't :)
Unfortunately my PR didn't help. This just happened again: - reassigning back to you
Another instance:
And again:
Are folks still seeing this?

I'm having a really hard time getting it to occur on-demand.
OK, I did find another timeout - The list of suspicious tests does not include the networking test this time: Kubectl client Kubectl patch should add annotations for pods in rc [Conformance] Kubectl client Simple pod should support exec through an HTTP proxy Pods should be submitted and removed [Conformance] Secrets should be consumable from pods [Conformance] Will have to keep searching for builds timing out to see if any new pattern emerges.
Looking more carefully, is another timeout. (I really wish it'd fail on my debug PR...) Suspicious tests: Job should scale a job down Kubectl client Simple pod should support exec through an HTTP proxy I'm gonna start suspecting that Kubectl HTTP proxy test now... are we doing anything in this test that might be blocking?
has a very curious failure: only 3 tests ran at all. It seems we are running the cluster-sanity checks in for every Ginkgo node, so there are probably cases like this run where it'll race and fail in mysterious ways (or possibly even deadlock)? We should probably use and (as documented ) to run the sanity checks, cleanup, and log dumping only once, rather than once per Ginkgo node. This may solve the mysterious timeouts.
Bumping down to P1 because this seems pretty rare nowadays (but I'll still address the issues I noted).
It seems like this issue is way too general: jobs timing out without a clear reason should have an action of making that reason clear. Is there any action we can take there?
I think the fixes I mentioned in will address the mysterious timeouts.
Happened here:
Hoping this one is gone for good, but I'll track and see.
Reopening; saw this again with
Final lines of that before being killed: Another run which timed out is kubernetes-pull-build-test-e2e-gce/. Final lines from that: Plus my earlier suspicions in make me think the kubectl proxy test is lacking a necessary timeout somewhere.
Moving discussion of that test case to .
With open, can we close this in favor of ?
Closing for now. If fixing doesn't fix, I'll keep looking...


Question: // volume *should* be attached to this node until it is safely unmounted. desiredVolumes := vm.desiredStateOfWorld.GetVolumesToMount() mountedVolumes := vm.actualStateOfWorld.GetGloballyMountedVolumes() <del> volumesToReportInUse := make( []v1.UniqueVolumeName, 0, /* len */ len(desiredVolumes)+len(mountedVolumes) /* cap */) desiredVolumesMap := make( map[v1.UniqueVolumeName]bool, len(desiredVolumes)+len(mountedVolumes) /* cap */) </del> <ins> volumesToReportInUse := make([]v1.UniqueVolumeName, 0, len(desiredVolumes)+len(mountedVolumes)) desiredVolumesMap := make(map[v1.UniqueVolumeName]bool, len(desiredVolumes)+len(mountedVolumes)) </ins> for _, volume := range desiredVolumes { if volume.PluginIsAttachable {

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: is not sent in a stable sort order. this makes node status patch requests larger than needed, and makes debugging nodes harder than necessary. What you expected to happen: stable sort of volumesInUse How to reproduce it (as minimally and precisely as possible): always Anything else we need to know?: nope Environment: n/a
/sig node
- this is a good getting started item.
I've already submitted a PR in . PTAL. Thanks.


Question: expectDelete: false, }, { <del> description:   "orphaned DS-managed pod with --force", node:          node, expected:      cordonedNode, pods:          []corev1.Pod{orphanedDsPod}, rcs:           []corev1.ReplicationController{}, args:          []string{"node", "--force"}, expectFatal:   false, expectDelete:  true, expectWarning: "WARNING: deleting Pods not managed by ReplicationController, ReplicaSet, Job, DaemonSet or StatefulSet: default/bar", </del> <ins> description:           "orphaned DS-managed pod with --force", node:                  node, expected:              cordonedNode, pods:                  []corev1.Pod{orphanedDsPod}, rcs:                   []corev1.ReplicationController{}, args:                  []string{"node", "--force"}, expectFatal:           false, expectDelete:          true, expectWarning:         "WARNING: deleting Pods not managed by ReplicationController, ReplicaSet, Job, DaemonSet or StatefulSet: default/bar", expectOutputToContain: "node/node drained", </ins> }, { <del> description:  "DS-managed pod with --ignore-daemonsets", node:         node, expected:     cordonedNode, pods:         []corev1.Pod{dsPod}, rcs:          []corev1.ReplicationController{rc}, args:         []string{"node", "--ignore-daemonsets"}, expectFatal:  false, expectDelete: false, </del> <ins> description:           "DS-managed pod with --ignore-daemonsets", node:                  node, expected:              cordonedNode, pods:                  []corev1.Pod{dsPod}, rcs:                   []corev1.ReplicationController{rc}, args:                  []string{"node", "--ignore-daemonsets"}, expectFatal:           false, expectDelete:          false, expectOutputToContain: "node/node drained", </ins> }, { <del> description:   "DS-managed pod with emptyDir with --ignore-daemonsets", node:          node, expected:      cordonedNode, pods:          []corev1.Pod{dsPodWithEmptyDir}, rcs:           []corev1.ReplicationController{rc}, args:          []string{"node", "--ignore-daemonsets"}, expectWarning: "WARNING: ignoring DaemonSet-managed Pods: default/bar", expectFatal:   false, expectDelete:  false, </del> <ins> description:           "DS-managed pod with emptyDir with --ignore-daemonsets", node:                  node, expected:              cordonedNode, pods:                  []corev1.Pod{dsPodWithEmptyDir}, rcs:                   []corev1.ReplicationController{rc}, args:                  []string{"node", "--ignore-daemonsets"}, expectWarning:         "WARNING: ignoring DaemonSet-managed Pods: default/bar", expectFatal:           false, expectDelete:          false, expectOutputToContain: "node/node drained", </ins> }, { <del> description:  "Job-managed pod with local storage", node:         node, expected:     cordonedNode, pods:         []corev1.Pod{jobPod}, rcs:          []corev1.ReplicationController{rc}, args:         []string{"node", "--force", "--delete-emptydir-data=true"}, expectFatal:  false, expectDelete: true, </del> <ins> description:           "Job-managed pod with local storage", node:                  node, expected:              cordonedNode, pods:                  []corev1.Pod{jobPod}, rcs:                   []corev1.ReplicationController{rc}, args:                  []string{"node", "--force", "--delete-emptydir-data=true"}, expectFatal:           false, expectDelete:          true, expectOutputToContain: "node/node drained", </ins> }, { <del> description:  "Ensure compatibility for --delete-local-data until fully deprecated", node:         node, expected:     cordonedNode, pods:         []corev1.Pod{jobPod}, rcs:          []corev1.ReplicationController{rc}, args:         []string{"node", "--force", "--delete-local-data=true"}, expectFatal:  false, expectDelete: true, </del> <ins> description:           "Ensure compatibility for --delete-local-data until fully deprecated", node:                  node, expected:              cordonedNode, pods:                  []corev1.Pod{jobPod}, rcs:                   []corev1.ReplicationController{rc}, args:                  []string{"node", "--force", "--delete-local-data=true"}, expectFatal:           false, expectDelete:          true, expectOutputToContain: "node/node drained", </ins> }, { <del> description:  "Job-managed terminated pod", node:         node, expected:     cordonedNode, pods:         []corev1.Pod{terminatedJobPodWithLocalStorage}, rcs:          []corev1.ReplicationController{rc}, args:         []string{"node"}, expectFatal:  false, expectDelete: true, </del> <ins> description:           "Job-managed terminated pod", node:                  node, expected:              cordonedNode, pods:                  []corev1.Pod{terminatedJobPodWithLocalStorage}, rcs:                   []corev1.ReplicationController{rc}, args:                  []string{"node"}, expectFatal:           false, expectDelete:          true, expectOutputToContain: "node/node drained", </ins> }, { <del> description:  "RS-managed pod", node:         node, expected:     cordonedNode, pods:         []corev1.Pod{rsPod}, replicaSets:  []appsv1.ReplicaSet{rs}, args:         []string{"node"}, expectFatal:  false, expectDelete: true, </del> <ins> description:           "RS-managed pod", node:                  node, expected:              cordonedNode, pods:                  []corev1.Pod{rsPod}, replicaSets:           []appsv1.ReplicaSet{rs}, args:                  []string{"node"}, expectFatal:           false, expectDelete:          true, expectOutputToContain: "node/node drained", </ins> }, { description:  "naked pod",

When draining a node, using kubectl drain [NODE], the output resembles this: Expected output should indicate that the node was drained, for example: kubectl drain [NODE] No response <details</details<detailsCan reproduce locally using and printing the output </details<details</details<details</details<details</details<details</details
/sig cli
/triage accepted


Question: # Specialized tests which should be skipped by default for projects. GCE_DEFAULT_SKIP_TESTS=( "${REBOOT_SKIP_TESTS[@]}" <del> "AutoscalingsSuite" </del> "Reboot" "ServiceLoadBalancer" )

Per our email discussion while getting 1.1 out the door. It looks like this never happened, though, which means we're not running autoscaling tests against GCE on the 1.1 branch, (we were running it on GKE with the 1.1-features job, but I'm removing that job in ). I'd once again like to (strongly) suggest that we mirror test suites running against release branch to jobs running against master, so that we avoid this problem of accidentally not running tests when we think we are. I'd much rather have a perhaps-unnecessary slow/HPA/whatever suite than continually try to figure out where slow/HPA/whatever tests are running, depending on if I'm looking at release jobs versus jobs running against master. cc
+1 We should be running the autoscaling tests in slow or continuous.


Question: <ins> #!/bin/bash # Copyright 2016 The Kubernetes Authors. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # #     http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. echo "Removing empty directories from etcd..." cleanup_empty_dirs () { if [[ $(${ETCDCTL} ls $1) ]]; then for SUBDIR in $(${ETCDCTL} ls -p $1 | grep "/$") do cleanup_empty_dirs ${SUBDIR} done else echo "Removing empty key $1 ..." ${ETCDCTL} rmdir $1 fi } while true do echo "Starting cleanup..." cleanup_empty_dirs "/registry" echo "Done with cleanup." sleep ${SLEEP_SECOND} done </ins>  No newline at end of file

I came upon a cluster that had had the namespace code thoroughly exercised. No namespaces currently existed, but many had been created and deleted in the past. Its etcd had lots of empty directories under each resource that had held items when those namespaces were alive (secrets, serviceaccounts, pods...). Do we need to be doing something to clean these empty directories up? It seems like this etcd was consuming a lot of memory.
or might know the answer to this question.
Right. We should delete them. It might get tricky for concurrent cases like A checks if dir is empty, B creates, then A deletes it. What does it look like currently in creating and deleting namespace?
How much memory does a directory entry take? Does it really sound possible to run out of memory while snapshotting on account of too many empty directories?
-- have you guys encountered this? I think you run with more namespaces than we do. But maybe yours are mostly populated?
Yes, we've seen it. c.f.
Approximate memory use we saw from empty dirs:
cc
The patch we debated carrying is here: Our hope was to just wait to etcd v3 where this would not be a problem.
can you link to your clean-up script here?
I needed another PR to run the pod automatically on master.


Question: for _, tc := range testCases { t.Run(tc.testName, func(t *testing.T) { <del> recycleSlices(tc.startingSlices) </del> <ins> startingSlices := tc.startingSlices recycleSlices(startingSlices) unorderedSlices := [][]*discovery.EndpointSlice{startingSlices.toCreate, startingSlices.toUpdate, startingSlices.toDelete} for _, actual := range unorderedSlices { sort.Slice(actual, func(i, j int) bool { return actual[i].Name < actual[j].Name }) } </ins> <del> expectEqualSlices(t, tc.startingSlices.toCreate, tc.expectedSlices.toCreate) expectEqualSlices(t, tc.startingSlices.toUpdate, tc.expectedSlices.toUpdate) expectEqualSlices(t, tc.startingSlices.toDelete, tc.expectedSlices.toDelete) </del> <ins> expectEqualSlices(t, startingSlices.toCreate, tc.expectedSlices.toCreate) expectEqualSlices(t, startingSlices.toUpdate, tc.expectedSlices.toUpdate) expectEqualSlices(t, startingSlices.toDelete, tc.expectedSlices.toDelete) </ins> }) } }

seen in the PR updating to go1.15, but this unit test should not be flaky /assign
was the flaky run
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
can reproduce locally with:
looks like toCreateByAddrType is a map which doesn't have a defined iteration order, so the address family order of toUpdate is not well-defined
/assign
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.


Question: t.Errorf("error creating strategy for %s: %v", k, err) } errs := s.Validate(nil, v.groups) <del> if v.pass && len(errs) > 0 { </del> <ins> if v.expectedError == "" && len(errs) > 0 { </ins> t.Errorf("unexpected errors for %s: %v", k, errs) } <del> if !v.pass && len(errs) == 0 { t.Errorf("expected no errors for %s but got: %v", k, errs) </del> <ins> if v.expectedError != "" && len(errs) == 0 { t.Errorf("expected errors for %s but got: %v", k, errs) } if v.expectedError != "" && len(errs) > 0 && !strings.Contains(errs[0].Error(), v.expectedError) { t.Errorf("expected error for %s: %v, but got: %v", k, v.expectedError, errs[0]) </ins> } } }

Is this a BUG REPORT or FEATURE REQUEST?: This is request for improvements. What happened: When PodSecurityPolicy provider validates or fields of a pod, it shows an error like "999 is not an allowed group" if the group doesn't belong to the range(s). What you expected to happen: It should include to the error message also the allowed values -- the ranges against what we were validating this group. This will simplify user experience and debugging. How to reproduce it (as minimally and precisely as possible): run kubernetes cluster with PSP support create a PSP that allows fsGroup/SupplementalGroups in some ranges grant to a user access to this PSP try to create a pod that has fsGroup/SupplementalGroups outside of the allowed ranges Anything else we need to know?: In order to improve the current behavior, we need to modify function here: Reported-by: /kind help
This looks like a good first issue! My suggestion would be something like . I would also suggest to merge overlaying ranges in the log message, like as , to keep the log as concise as possible. I will look into it.
Maybe something like ? Thank you!
We can use the same approach that strategy uses:
Sure, that is probably the most consistent solution.
/remove-help


Question: EnableServiceLinks: tc.enableServiceLinks, }, } <del> podIP := "1.2.3.4" podIPs := []string{"1.2.3.4,fd00::6"} </del> <ins> podIP := "" if len(tc.podIPs) > 0 { podIP = tc.podIPs[0] } </ins> if tc.staticPod { testPod.Annotations[kubetypes.ConfigSourceAnnotationKey] = "file" } <del> result, err := kl.makeEnvironmentVariables(testPod, tc.container, podIP, podIPs) </del> <ins> result, err := kl.makeEnvironmentVariables(testPod, tc.container, podIP, tc.podIPs) </ins> select { case e := <-fakeRecorder.Events: assert.Equal(t, tc.expectedEvent, e)

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --In a dual-stack cluster with IPv4 as default family (first kubelet IP), a container is created with a env var set to . When the container is running, properly contains the IPv4 But at runtime, this env var (used in container args) is expanded to the pod IPv6 instead of the IPv4. When using , it should always return the value actually exposed as and not something else. Kubernetes version:Cloud provider or hardware configuration: Bare metal cluster OS (e.g: ): Ubuntu 20.04 Kernel (e.g. ): Linux 5.8.0-59-generic 66~20.04.1-Ubuntu SMP Thu Jun 17 11:14:10 UTC 2021 x8664 x8664 x86_64 GNU/Linux Network plugin and version (if this is a network-related bug): cilium 1.10 Others: CRI-O 1.21 + crun.
Note: I have a configuration that matches the ones that had IP ordering issue before was fixed. So it may be related.
/sig network
Just a guess, but I wonder if it may be caused by
That's it. The fix from properly sort IP when computing the Pod status, but the sorted IPs are then replaced by computed IPs as this log shows.
/triage accepted Thanks for the detailed report
/assign


Question: namespaceName := attr.GetNamespace() if len(namespaceName) == 0 && attr.GetResource().Resource != "namespaces" { // If the request is about a cluster scoped resource, and it is not a <del> // namespace, it is exempted from all webhooks for now. </del> <ins> // namespace, it is never exempted. </ins> // TODO: figure out a way selective exempt cluster scoped resources. // Also update the comment in types.go <del> return false, nil </del> <ins> return true, nil </ins> } namespaceLabels, err := m.GetNamespaceLabels(attr) // this means the namespace is not found, for backwards compatibility,

If I am not mistaken, means that cluster-scoped resources other than namespaces are broken with admission webhooks. There is even a unit test checking that:
To clear any ambiguity around the word "broken", the current code behavior is that most cluster-scoped resources are not subjected to any webhooks. I wrote them like this because I was concerned with the potential bootstrapping deadlocks: a webhook that enacts policies for cluster-scoped resources might cause bootstrapping deadlocks for the cluster or for its own deployment. Thinking out loud, perhaps for cluster-scoped resources, the apiserver should only call the webhooks if the resources' labels match the , so that the critical bootstrapping resources can be excluded from the webhooks, thus breaking the bootstrapping deadlocks. This aligns with what we do for (see ). The downside is that the field name "namespaceSelector" becomes inappropriate.
I think we still need namespaceSelectors. We don't want to label pods. But I agree we actually want another objectLabelSelector. For namespaces these two coincide, i.e. we only have to define how they are applied if both are set or we forbid both at the same time. I tend to the later.
raised up a security issue regarding . A user can change labels to skip webhooks. doesn't have this issue because a user usually doesn't have permission to change the namespace's labels.
Users normally cannot create cluster-scoped objects either. But techincally it's a valid point. On the other hand, my main use-case are CRDs. They are not critical to the cluster health (maybe self-hosted clusters are different with etcd operator), i.e. you normally cannot shoot yourself into the foot with a webhook on them. For CRDs we do not need a selector at all. In other words: IMO we should enable webhooks for cluster-scoped objects, even if we have no in place.
Sounds reasonable. I'll create a PR ()
We would like to have the ability to apply the admission controller on specific namespaces only, and not on cluster level resources. This is useful if the list of possible resource Kinds that we want to apply the controller for is large, and we want to limit when it runs as much as possible to avoid affecting the whole cluster in case there's an issue.

I think this can be done by adding a SkipClusterResources flag (or something similar) in addition to the NamespaceSelector, to the Webhook struct. What do you think?
I think adding a flag is reasonable. It doesn't have the security issue as the idea does (). The expressiveness of is limited compared to , e..g., admins cannot use to run webhooks on some cluster-level resources while skipping others. However it's not a big drawback, because admins can specify more detailed GVK to limit the scope of webhook in that case.
Zeroth thought: s/SkipClusterResources/scope=[clusterboth]/g First thought: can't you just exclude cluster scoped resources, or include only namespace scoped resources in the webhook definition? Second thought: can a namespace selector be crafted that excludes the empty string?
0: Are you sure that's cleaner? The namespace selector picks which namespaces to use, and skipCluster decides whether to include cluster resources or not. Not setting it maintains current behavior (skipCluster=false) and whether cluster level resources are included is very clear. scope=cluster/namespace/both will require us to default to "both" which I find slightly more confusing. Also what to do if I specify cluster AND include namespace selector - is that an invalid spec or do we ignore the selector? 1: Yes. This is the current best thing we can do. If the list of resources is large though (for example, if I am implementing an alternative resource quota controller) then this solution requires us to maintain and update a long list of resources. This may be ok, since as long as our controller can't support certain types of resources it doesn't make sense to watch for them... 2: Doesn't look like it - the current code specifically checks if the attribute is a cluster level resource and skips the entire label matching process.
Re: 0: fair point, but let's at least name it in the positive: matchClusterScopedResources=true|false or something like that.
I proposed it in the negative to make it backwards compatible, so that "false" keeps current behavior. I agree that negatively named booleans are not not bad. I'll treat it as true if unspecified and use matchClusterScopedResources.
In today's semantic, such a namespace selector will exclude namespaced objects whose namespace has no label.


Question: } wait.Wait() } <del> if controller.Status.Replicas != activePods { controller.Status.Replicas = activePods </del> <ins> if controller.Status.Replicas != numActivePods { controller.Status.Replicas = numActivePods </ins> _, err = rm.kubeClient.ReplicationControllers(controller.Namespace).Update(&controller) if err != nil { return err

I've noticed a issue where if you have multiple replicated slave pods that don't have a HOST assigned and you reduce the number of replicas in the replicationController definition for those pods, sometimes the currently assigned (and running) pods get killed and the unassigned pod will get assigned to run in the killed pods place. The issue can be reproduced using the guestbook example provided in the repo. Steps to reproduce: (Note this is not always deterministic and may need multiple attempts) the master pod the master service the replicated slaves the Slave Controller definition to only have 1 replicas the Slave Controller with new definition The expected behavior should be that the scheduler is able to determine that unassigned pods are more viable candidates when reducing number of replicas. It seems to be more likely that since this issue is not deterministic, the ordering in which the command returns result is not always deterministically in the same order
: please can you look at this or suggested someone else? Thank you.
Unassign myself from this bug since I cannot get working on this soon. Also remove kubelet from the label since it is not kubelet issue. This is obviously a replicaController issue. cc/ Instead of destroying pod in pending state, replicaController unbound running pod from a node. Kubelet running on such node has this new desire state and kill unwanted pod to match it.
Yes, the replication controller should be smarter, taking into account Phase, Readiness, recency, ... This will be a significant problem for auto-scalers, but I expect manual scaling to be infrequent. I doubt we'll get to this soon. A PR would be welcome. Place to change is here:
Hi, I am final year Master's student of International Institute of Information Technology, Hyderabad. I am familiar with docker and containers. I have basic understanding of Go language with some hands on Docker API. This project seems interesting and a solution to this will also be a good addition to Kubernetes. So I would like to pick this as my GSoC project. I have started reading about Replication Controller and Pods and going though the code. I would appreciate if someone has any inputs for me on this. Thanks, Swapnil Patil


Question: "fmt" "time" <ins> "k8s.io/apimachinery/pkg/api/errors" </ins> "k8s.io/apimachinery/pkg/api/meta" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" <ins> "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" </ins> "k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/runtime/schema" "k8s.io/apiserver/pkg/endpoints/handlers/fieldmanager/internal" openapiproto "k8s.io/kube-openapi/pkg/util/proto" "sigs.k8s.io/structured-merge-diff/fieldpath" "sigs.k8s.io/structured-merge-diff/merge" <ins> "sigs.k8s.io/yaml" </ins> ) // FieldManager updates the managed fields and merge applied

What happened: When sending a request to the server side apply endpoint (), if the apiVersion provided in the object is different from the apiVersion in the url, it returns a 500 error, like: What you expected to happen: The request should fail if the apiVersion provided in the object is different from the apiVersion in the url, but it should fail before trying to merge, and be a 4xx error. Probably a bad request error. Anything else we need to know?: The check should probably be done somewhere near this . The issue, though, is that the patch is given as a byte array, and needs to be deserialized into an object to access the apiVersion field. How to reproduce it (as minimally and precisely as possible): With a kube-apiserver built from head, running with :
/kind bug /sig api-machinery /wg apply /priority important-longterm /assign


Question: func NewFramework(fns []RegisterPluginFunc, opts ...runtime.Option) (framework.Framework, error) { registry := runtime.Registry{} plugins := &schedulerapi.Plugins{} <del> var pluginConfigs []schedulerapi.PluginConfig </del> for _, f := range fns { <del> f(&registry, plugins, pluginConfigs) </del> <ins> f(&registry, plugins) </ins> } <del> return runtime.NewFramework(registry, plugins, pluginConfigs, opts...) </del> <ins> return runtime.NewFramework(registry, plugins, nil, opts...) </ins> } // RegisterPluginFunc is a function signature used in method RegisterFilterPlugin() // to register a Filter Plugin to a given registry. <del> type RegisterPluginFunc func(reg *runtime.Registry, plugins *schedulerapi.Plugins, pluginConfigs []schedulerapi.PluginConfig) </del> <ins> type RegisterPluginFunc func(reg *runtime.Registry, plugins *schedulerapi.Plugins) </ins> // RegisterQueueSortPlugin returns a function to register a QueueSort Plugin to a given registry. func RegisterQueueSortPlugin(pluginName string, pluginNewFunc runtime.PluginFactory) RegisterPluginFunc {

/sig scheduling /triage accepted The method in only modifies the slice of PluginConfig internally. Instead, the method should return the appended slice so that callers can override the original. For example, here:
/assign
ref:
Hey are you still working on this, if not I'd like to try it out!
Hi I'm still working on this. Thanks for your interest about this issue.


Question: return nil } <ins> // ChainExists is part of iptables.Interface func (*FakeIPTables) ChainExists(table iptables.Table, chain iptables.Chain) (bool, error) { return true, nil } </ins> // EnsureRule is part of iptables.Interface func (*FakeIPTables) EnsureRule(position iptables.RulePosition, table iptables.Table, chain iptables.Chain, args ...string) (bool, error) { return true, nil

(update: likely this is benign, but it seems really weird when your reading kube proxy logs) in healthy, IPTABLES jobs, the various ip6tables-restore commands are happy....MEANWHILE in ipvs jobs (healthy, or non healthy, we see that some of the iptables dont exist which makes sense... why is ipvs kube proxy tyrying to do iptables-restore operations on the table ? why is it trying to do iptables stuff at all ? It seems like we get error messages when running ipv6. not sure why though? in particular i found these errors when looking at the test failures for You wouldnt see error messages for ip6tables-restore when running ipvs tests. This may be my own ignorance however, i.e. maybe this is normal. I guess every job has this (wether or not the job passes or fails) ? Here is an example from a passing test.... Similar logs from a failing tests that i originally saw this message ... has this failure, it is reproduced via the corresponding ci job ... the logs in the kube-proxy for one of the nodes look like this: ill add more links as i collect more data. Kubernetes version (use ): prow (main branch as of feb 12 2021)
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig network
So, this happens in HEALTHY jobs to, so i guess its not a driver for an actual bug.... but its still weird to me :) ... updating original issue...
any idea on why those ipv6 errors are printed in ipvs code ?
The proxier is configured in dual-stack mode in the CI Since we enabled the dual-stack gate recently, this is the expected behaviour
I don't get similar printouts myself so I can't say anything for sure. By first check the old re-occuring (and very annoying) "KUBE-MARK-DROP" problem.

This chain is usually created by "kubeadm" but so far kubeadm is unaware of dual-stack and has only created this chain for ipv4 (in iptables). Then it needs to be created for IPv6 before kube-proxy is started; In my test cluster I always do this in a prep-stage before starting K8s.
The ci nodes doesn't have the kernel ip6tables nat modules loaded, hence they fail all the operation related
hmmm ok so is the issue here that we need to load that module in the CI jobs?
signing here because I want to at least find why ip6tables restore is called on kube-proxy ipvs things :D
ipvs does uses iptables too I think that kube-proxy should implement something similar to what implemented in kubelet The nat table for IPv6 was in 3.7 and it seems that it may be not present in some systems. /cc
/assign
I think it's mostly just about whether IPv6 is enabled, not about the kernel version.
IIRC this specific error in the CI is about the nat module only Anyway, you fixed it :clap: /unassign /assign


Question: klog.ErrorS(err, "getNodeAnyway function failed") return false, nil, "" } <ins> podCopy := pod.DeepCopy() </ins> cpuAvailable := node.Status.Allocatable.Cpu().MilliValue() memAvailable := node.Status.Allocatable.Memory().Value() <del> cpuRequests := resource.GetResourceRequest(pod, v1.ResourceCPU) memRequests := resource.GetResourceRequest(pod, v1.ResourceMemory) </del> <ins> cpuRequests := resource.GetResourceRequest(podCopy, v1.ResourceCPU) memRequests := resource.GetResourceRequest(podCopy, v1.ResourceMemory) </ins> if cpuRequests > cpuAvailable || memRequests > memAvailable { <del> klog.V(3).InfoS("Resize is not feasible as request exceeds allocatable node resources", "pod", pod.Name) return false, nil, v1.PodResizeStatusInfeasible </del> <ins> klog.V(3).InfoS("Resize is not feasible as request exceeds allocatable node resources", "pod", podCopy.Name) return false, podCopy, v1.PodResizeStatusInfeasible </ins> } // Treat the existing pod needing resize as a new pod with desired resources seeking admit.

mutates the pod object in place (vs creating a copy first). That's illegal and can cause data races or potentially panics (the panic is unlikely here but I can't rule it out). This line should be: This should be fixed because we might serialize the object in the middle of this from somewhere else and if the object changes midway the protobuf object we create may not be valid, leading to panics in kubelets with this gate turned on. /sig node /priority critical-urgent /kind bug /assign No mutation of pod objects from within methods (in fact no mutation of should occur in most loops). N/A No response <details</details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
ACK. I'll fix it shortly.
is similar. is my try for both data races that we found.
One small , thanks.
Please review
This Issue/PR has been marked as release blocker. Please have the fix merged by eod today otherwise the feature has to be reverted. For the Issue/PR owner: please have a revert/feature disable PR open today in case the fix failed to get in. Thank you! See the -- Kubernetes 1.27 release manager


Question: } func containerIsNetworked(containerName string) bool { <del> return false </del> <ins> return containerName == PodInfraContainerName </ins> } // Returns nil as both Seccomp and AppArmor security options are not valid on Windows

Kubernetes version: master branch. Environment: OSX. What happened: cc/
I just took as an example. When running on OSX, always returns . However, on Linux, the actual calling path is , which will return when is . When get , will not run, which will call methods TearDownPod. And this is why on OSX calls to testing.MockNetworkPlugin.TearDownPod are missing.
Thanks. Could you file a PR to fix this?


Question: } return strings.Contains(output, "This is a test file.") }, 1*time.Minute, 1*time.Second).Should(gomega.BeTrue()) }) }) })

Cluster api provides a framework to run the e2e tests which uses the conformance image: The gmsa tests use bash to install the webhook: The test will fail when run in the conformance image because of the bash: we should install the component another way run this e2e conformance image with /sig windows fyi <details</details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Conformance image:
Hi does the "work e2e" test case pass? Deploying GMSA webhook is the same for both "work e2e" and this test case.
no, they both fail due to bash not being present in the conformance image. The gmsa tests themselves pass if I run locally
/sig testing


Question: } } <ins> func TestHasMountRefs(t *testing.T) { testCases := map[string]struct { mountPath string mountRefs []string expected  bool }{ "plugin mounts only": { mountPath: "/var/lib/kubelet/plugins/kubernetes.io/some-plugin/mounts/volume-XXXX", mountRefs: []string{ "/home/somewhere/var/lib/kubelet/plugins/kubernetes.io/some-plugin/mounts/volume-XXXX", "/var/lib/kubelet/plugins/kubernetes.io/some-plugin/mounts/volume-XXXX", "/mnt/kubelet/plugins/kubernetes.io/some-plugin/mounts/volume-XXXX", "/mnt/plugins/kubernetes.io/some-plugin/mounts/volume-XXXX", }, expected: false, }, "extra local mount": { mountPath: "/var/lib/kubelet/plugins/kubernetes.io/some-plugin/mounts/volume-XXXX", mountRefs: []string{ "/home/somewhere/var/lib/kubelet/plugins/kubernetes.io/some-plugin/mounts/volume-XXXX", "/local/data/kubernetes.io/some-plugin/mounts/volume-XXXX", "/mnt/kubelet/plugins/kubernetes.io/some-plugin/mounts/volume-XXXX", "/mnt/plugins/kubernetes.io/some-plugin/mounts/volume-XXXX", }, expected: true, }, } for name, test := range testCases { actual := HasMountRefs(test.mountPath, test.mountRefs) if actual != test.expected { t.Errorf("for %s expected %v but got %v", name, test.expected, actual) } } } </ins> func TestMountOptionFromSpec(t *testing.T) { scenarios := map[string]struct { volume            *volume.Spec

See This happens on ubuntu images on GKE when the node ephemeral storage is on local SSD. /sig storage /assign
/triage accepted


Question: ForceUserspaceProxy bool SyncPeriod          time.Duration nodeRef             *api.ObjectReference // Reference to this node. <ins> MasqueradeAll       bool </ins> } // NewProxyServer creates a new ProxyServer object with default parameters

Related to . As a safety mesure, it could be very useful to actually include a flag to proxy to MASQUERADE everything (acting very close to a userspace proxy). I think it's not very hard to do and a very good way to diagnose or even fallback in case of problem (I'm thinking about vxlan cases). Something like "--snat-clients" or "--snat-pods" could be appropriate.
/cc
Looking at the current code: It looks like we SNAT almost everything already now? Would adding a set-mark rule (behind the flag of course) at line 548 be sufficient? Deprecated Public IPs, Loadbalancer IPs, and NodePorts all appear to be marked for SNAT already (also endpoint loopback traffic).
I think that is the flag, yeah. , Benjamin Elder wrote:
Ok, I'll push out a PR for that in a moment. I should be able to take care of that and the cleanup flag (PR out but needs some more work) today. On Aug 20, 2015 2:12 PM, "Tim Hockin" wrote:


Question: "k8s.io/apimachinery/pkg/labels" "k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/runtime/schema" <ins> "k8s.io/apimachinery/pkg/util/sets" </ins> "k8s.io/apimachinery/pkg/util/validation/field" celconfig "k8s.io/apiserver/pkg/apis/cel" "k8s.io/apiserver/pkg/features"

A client is allowed to add custom finalizers to Custom Resources with a finalizer name that's not fully qualified (e.g. or ). However they're prevented from doing the same on builtin resource types, returning the following error: Invalid value: "": name is neither a standard finalizer name nor is it fully qualified Per , custom finalizer names are expected to be in format "consist of a domain name, a forward slash and the name of the finalizer" (e.g. ). Therefore, I expected the enforcement to be the same for CRs and builtin types. a custom resource on the cluster it to add the edit does not fail the same on a builtin resource (e.g. Namespace or Pod), observe it fails No response Reproducible at least starting from 1.21 and still on latest versions. <details</details<detailsN/A kind cluster </details<detailsN/A - kind cluster accessed via macOS </details<details</details<details</details<details</details
/area api /cc /sig api-machinery
/assign /triage accepted
There is no validation of finalizer in and . Please unassign me if you guys have started to work on it. /assign
it is called from here on custom resources:
lacks validation for finalizer name. I have raised a pr for this. PTAL.


Question: // PodKiller handles requests for killing pods type PodKiller interface { // KillPod receives pod speficier representing the pod to kill <del> KillPod(pair *kubecontainer.PodPair) </del> <ins> KillPod(podPair *kubecontainer.PodPair) </ins> // PerformPodKillingWork performs the actual pod killing work via calling CRI // It returns after its Close() func is called and all outstanding pod killing requests are served PerformPodKillingWork()

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Right after calling , returns false even the pod is not killed yet. What you expected to happen: In kubelet, after is complete, the pod should be pending termination or be killed. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Related to Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node /assign


Question: KUBE_PROMPT_FOR_UPDATE=y KUBE_SKIP_UPDATE=${KUBE_SKIP_UPDATE-"n"} <del> # Suffix to append to the staging path used for the server tars. Useful if # multiple versions of the server are being used in the same project # simultaneously (e.g. on Jenkins). KUBE_GCS_STAGING_PATH_SUFFIX=${KUBE_GCS_STAGING_PATH_SUFFIX-""} </del> # How long (in seconds) to wait for cluster initialization. KUBE_CLUSTER_INITIALIZATION_TIMEOUT=${KUBE_CLUSTER_INITIALIZATION_TIMEOUT:-300}

We seem to have an occasional consistency issue between the client and server versions on our Jenkins GCE e2e project. Build 6295 failed this morning because the client version was newer than the server version: The server version matches the build was was immediately running in 6294, which was aborted early. I wonder if (somehow) the cleanup didn't work properly? (Which is a bit worrying if so.) Here's the version from 6294: A short while later, build 6297 failed, this time with the client version older than the server version: This one makes a little bit more sense: there was a new build published between downloading the client/test tarballs and setting up the cluster. I'm guessing that somewhere in the cluster setup scripts the latest CI release file was read again, rather than using the version read for the client/test tarballs. (I haven't yet worked out where exactly it's read, but it should be an easier fix.) cc
A side issue: we should probably figure out why the build git client is being marked dirty. Since we're checking out the head version from GitHub, it should ideally be a clean client.
The -dirty bit is now fixed by .
This is caused by different Jenkins jobs uploading to the same GCS bucket and clobbering each other. In we see: Over in
This also explains the issues seen in ; the job is also running in the same project.
Ah, good sleuthing. We avoided this issue in the PR builds by setting (used ), but maybe we should always be setting to include the Jenkins job name (and optionally executor/node number).
The original idea was to run every e2e job in it's own project, to avoid these sorts of name clashes. But we seem to have regressed and now have many jobs sharing the same projects. Is project-per-job not a better approach? It also helps with resource leaks and the likes. Just a thought.
I'd argue this is actually uncovering a (dev) deployment bug; I should be able to deploy different versions of Kubernetes to the same project. I know you and I disagree (and probably always will) on the strategy here.

:smile:
fwiw I think I agree with that if project-per-job avoids this issue and helps identify resource leaks, it's a pragmatic near-term improvement in the state of the world that would result in more benefit than the cost of changing all the everythings to support multi-job-in-project. Re: devs deploying different versions of k8s to the same project, P3 feature request?
I was just about to suggest the same thing. I will file a separate issue, and fix this in the short term an easier way. is out to make it easier to isolate a run to a project.
Also it seems like a reasonable thing (higher than P2) to want to deploy multiple clusters to the same project safely in prod, not just dev. Disagree?
Multiple clusters in prod using well-defined build versions such as v1.1.4 = P0. Multiple clusters in prod using our cooked-up test-building system = non-goal?
SGTM; this is prod code, though, not test code; it's used every time we call , regardless of whether it's a CI build or released version.
FYI: kubernetes-e2e-gce-reboot/ has one reboot test failed due to kube-proxy image change. But that build doesn't include pr yet.
looks like it. That run used version , but the first build with was .


Question: github.com/containerd/ttrpc v1.0.2/go.mod h1:UAxOpgT9ziI0gJrmKvgcZivgxOp8iFPSk8httJEt98Y= github.com/containerd/typeurl v1.0.1 h1:PvuK4E3D5S5q6IqsPDCy928FhP0LUIGcmZ/Yhgp5Djw= github.com/containerd/typeurl v1.0.1/go.mod h1:TB1hUtrpaiO88KEK56ijojHS1+NeF0izUACaJW2mdXg= <del> github.com/containernetworking/cni v0.8.0 h1:BT9lpgGoH4jw3lFC7Odz2prU5ruiYKcgAjMCbgybcKI= github.com/containernetworking/cni v0.8.0/go.mod h1:LGwApLUm2FpoOfxTDEeq8T9ipbpZ61X79hmU3w8FmsY= </del> <ins> github.com/containernetworking/cni v0.8.1 h1:7zpDnQ3T3s4ucOuJ/ZCLrYBxzkg0AELFfII3Epo9TmI= github.com/containernetworking/cni v0.8.1/go.mod h1:LGwApLUm2FpoOfxTDEeq8T9ipbpZ61X79hmU3w8FmsY= </ins> github.com/coredns/corefile-migration v1.0.11 h1:ptBYGW2ADXIB7ZEBPrhhTvNwJLQfxE3Q9IUMBhJCEeI= github.com/coredns/corefile-migration v1.0.11/go.mod h1:RMy/mXdeDlYwzt0vdMEJvT2hGJ2I86/eO0UdXmH9XNI= github.com/coreos/bbolt v1.3.2/go.mod h1:iRUV2dpdMOn7Bo10OQBFzIJO9kkE559Wcmn+qkEiiKk=

is identified with with severity rating as high. This is found using snyk scan while running against k/k master We're not using directly however, it gets tagged for vulnerability scans as we use . The fix is already present in and we can bump the dep. Bump dep to which has the mentioned CVE fix. and check Kubernetes version (use ): n/a Cloud provider or hardware configuration: any OS (e.g: ): n/a Kernel (e.g. ): n/a Install tools: n/a Network plugin and version (if this is a network-related bug): n/a Others: n/a /sig architecture /area code-organization for the dependency
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: "k8s.io/apimachinery/pkg/fields" "k8s.io/apimachinery/pkg/labels" "k8s.io/apimachinery/pkg/runtime/schema" <ins> "k8s.io/apimachinery/pkg/util/wait" </ins> "k8s.io/apimachinery/pkg/watch" restclient "k8s.io/client-go/rest" "k8s.io/client-go/tools/record"

the timing is suspicious. Your pull merged, right?
it did, i will take a look.


Question: if opts.AllowNonStructural || len(structuralschema.ValidateStructural(nil, ss)) == 0 { schema = ss <ins> // This adds ValueValidation fields (anyOf, allOf) which may be stripped below if opts.StripValueValidation is true schema = schema.Unfold() </ins> if opts.StripValueValidation { schema = schema.StripValueValidations() } if opts.StripNullable { schema = schema.StripNullable() } <del> schema = schema.Unfold() </del> } } }

With in a CRD we get: This is because gnostic's OpenAPI proto types have no nullable support. Luckily CRDs keep working because in the upper error case we don't stop serving, just set the SSA schema to nil.
Thanks for reporting!
I think it should be "This is because gnostic's OpenAPI v2 proto types have no nullable support.". The problem is that we're not supposed to pass-in v3.
Source:
OK, We'll write a test and fix this, thanks Stefan!
/reopen This still happens in slightly different shape: This comes from :
That's a fairly different bug but yes, thanks!
Is this related to the stating that use of "anyOf" wouldn't be allowed after 1.16? We're seeing issue on OpenFaaS after going to 1.17.5 and 1.18. Usually upgrading K8s versions hasn't caused breaking changes and we're trying to figure out where to start with this one.
No. That post was about structural schemas. v1 CRDs are limited to structural schemas. v1beta1 CRDs can continue to create non-structural schemas until v1beta1 is no longer served (in 1.22) Also, is not completely disallowed, specific restrictions inside an clause (like ) are what are not allowed ("logical constraints using oneOf, allOf, anyOf, not are allowed").
extracted the newly found issue to to track separately with isolated reproducer /close
Closing this issue. <detailsIn response to : extracted the newly found issue to to track separately Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: Containers: []api.Container{ { Name:    "tester", <del> Image:   "busybox", </del> <ins> Image:   "gcr.io/google_containers/busybox", </ins> Command: []string{"sleep", "600"}, Lifecycle: &api.Lifecycle{ PreStop: &api.Handler{

Testgrid; https://k8s- Sample Failure: The problem seems to be that Kube-anywhere isn't working since commit /sig cluster-lifecycle /kind failing-test /priority important-soon cc:
/milestone v1.13
sent possible fix PR:
sent new PR might get merged once github is back on track.
Reopening, because the test isn't passing yet. /reopen
Reopening this issue. <detailsIn response to : Reopening, because the test isn't passing yet. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
true, i was almost sure that the solution in that last PR won't be sufficient... will investigate more tomorrow.
/assign
I see https://k8s- job failing as well. Will the 2 open PRs address this failure as well?
all the kubeadm-gce-x tests are broken at the moment due to the e2e refactor. we will hopefully decide on how to approach the issue today.
latest efforts:
the tests are working on the cluster up/down side, except that they fail when dumping the node logs. looking into it.
should be green by tomorrow.
Looks good, thanks! One flake in the last run - let's keep an eye on it but it should be fine. /cc /close
Closing this issue. <detailsIn response to : Looks good, thanks! One flake in the last run - let's keep an eye on it but it should be fine. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: --pv-recycler-pod-template-filepath-hostpath="": The file path to a pod definition used as a template for HostPath persistent volume recycling. This is for development and testing only and will not work in a multi-node cluster. --pv-recycler-pod-template-filepath-nfs="": The file path to a pod definition used as a template for NFS persistent volume recycling --pv-recycler-timeout-increment-hostpath=30: the increment of time added per Gi to ActiveDeadlineSeconds for a HostPath scrubber pod.  This is for development and testing only and will not work in a multi-node cluster. <del> --pvclaimbinder-sync-period=10m0s: The period for syncing persistent volumes and persistent volume claims </del> <ins> --pvclaimbinder-sync-period=15s: The period for syncing persistent volumes and persistent volume claims </ins> --replicaset-lookup-cache-size=4096: The the size of lookup cache for replicatsets. Larger number = more responsive replica management, but more MEM load. --replication-controller-lookup-cache-size=4096: The the size of lookup cache for replication controllers. Larger number = more responsive replica management, but more MEM load. --resource-quota-sync-period=5m0s: The period for syncing quota usage status in the system

Immediately after this test was reenabled in we got huge numbers of parallel flakes in . This might be a coincidence. I'll make a PR to move it to flaky. If things haven't improved after it's moved then I'll revert, else I'll assign this issue to !
Do you have any logs? references , but that has nothing to do with NFS nor PersistentVolumes.
. It's pretty clear this test caused the flakes we were seeing.
do you have any logs?
It hasn't flaked since your fix went in, but we should keep the issue open until it's moved out of the flaky suite.
The fix Joe Finney developed was merged on 5/30. The signature for the problem in the log file referenced here is: 6 18:56:07.895: Expected PersistentVolume to be bound, but got nil ClaimRef: kind:"" apiVersion:""` It looks like from this dashboard the last instance of this error log signature for the test PersistentVolumes NFS test to the flaky suite. was on 5/30 right around the time the code was merged. I found the off the dashboard at https://k8s- Since then the error has not occurred with that signature since I am tempted to close this bug, but keep the the test suite as marked "Flaky". The merge bot should start flagging the new bugs for this test going forward.
If it's no longer flaky we should move it out of the flaky suite. While it's in the flaky suite it won't block merges if it breaks and mergebot won't open bugs for it. I'm worried that if we just close this issue it'll sit in the flaky suite and break and noone will notice for a long time.
This bug has not been reproduced for past 2 months, can we close it?
It looks like the test is still in the flaky suite, although with a different name. Once it's moved out of there then this can be closed.
[FLAKE-PING] This flaky-test issue would love to have more attention...
[FLAKE-PING] This flaky-test issue would love to have more attention...
Not reproduced for three months, closing.

Consider reducing the PVClaimBinderSyncPeriod from 10 minutes down to something more reasonable, so that volumes don't sit around unbound for half an hour if it fails twice. This would reduce the impact of issues like until the binder/recycler/provosioner consolidation/refactor is complete. CC
+100
do you remember what was the reason for 10 minutes?
I checked provisioner and recycler, lowering the sync period should not harm them. Still, it will impact deleter a bit. Now we try to delete a volume three times every sync-period (10 minutes), mostly because if user deletes a pod that uses a volume and deletes also the PVC bound to it, the volume is still attached to the node for some time and all deletion attempts fail. I've seen AWS detach taking around 5 minutes in some cases. So, if we lower the timer, we should also increase the number of retries. Or have a different retry timer for deletion.


Question: case "${platform}" in "linux/arm") export CGO_ENABLED=1 <del> export CC=arm-linux-gnueabihf-gcc </del> <ins> export CC=${KUBE_LINUX_ARM_CC:-arm-linux-gnueabihf-gcc} </ins> ;; "linux/arm64") export CGO_ENABLED=1 <del> export CC=aarch64-linux-gnu-gcc </del> <ins> export CC=${KUBE_LINUX_ARM64_CC:-aarch64-linux-gnu-gcc} </ins> ;; "linux/ppc64le") export CGO_ENABLED=1 <del> export CC=powerpc64le-linux-gnu-gcc </del> <ins> export CC=${KUBE_LINUX_PPC64LE_CC:-powerpc64le-linux-gnu-gcc} </ins> ;; "linux/s390x") export CGO_ENABLED=1 <del> export CC=s390x-linux-gnu-gcc </del> <ins> export CC=${KUBE_LINUX_S390X_CC:-s390x-linux-gnu-gcc} </ins> ;; esac fi

<!-- Please only use this template for submitting enhancement requests --What would you like to be : There should be a way to override the default gcc compliers mentioned in the file with custom gcc compilers for different platforms. Also, cross compilation should be supported on platforms other than amd64. Why is this needed: Right now, there seems to be no way to override the platform specific cc flags that kubernetes sets. We should be able to override those values. For example, bottlerocket sets customized CC flags for different platforms using this . We should be able to override the default values in kubernetes using environment variables or build time flags and build kubernetes with customized gcc variants that bottlerocket uses. Also, we need an option to set gcc compilers for amd64 platform, so that amd64 targets can be built on different platforms.
/sig release
Sure, that will require cross building the build toolset itself, which nobody has taken up yet.


Question: #   foo: true #   bar: false for gate in $(echo ${FEATURE_GATES} | tr ',' ' '); do <del> echo $gate | sed -e 's/(.*)=(.*)/  1: 2/' </del> <ins> echo $gate | ${SED} -e 's/(.*)=(.*)/  1: 2/' </ins> done fi >>/tmp/kube-proxy.yaml

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: After running (on master branch), a new file named left as a surprise. and the file is not matched by git ignore so it's tracked by git which is really disturbing. I tried to find out where it's created but failed. Any one has a context w/ it? Environment: OS (e.g. from /etc/os-release): Darwin Kernel Version 17.6.0: Tue May 8 15:22:16 PDT 2018; root:xnu-4570.61.1~1/RELEASEX8664 x86_64
/sig instrumentation
the only piece of code in local-up-cluster that touches the file is
can't reproduce this. Two questions: Can you see this everytime? What is the SHA of the commit on master when you tried this? Can check with that too.
finally i realize that the cause is command is built receiving different flags between Linux and OSX. on Linux, you'll find that for linux sed the is explained as: '--in-place[=SUFFIX]' This option specifies that files are to be edited in-place. GNU 'sed' does this by creating a temporary file and sending output to this file rather than to the standard output.(1). so the is optional. but for OSX, it is: Edit files in-place, saving backups with the specified extension. If a zero-length extension is given, no backup will be saved. It is not recommended to give a zero- length extension when in-place editing files, as you risk corruption or partial content in situations where disk space is exhausted, etc. the difference is that on OSX, flag is requiring a variable , so when we're using on OSX in the script: the which is supposed to be received as a flag will be a suffix for the backup file. then the magic happened.will raise a fix for this. /assign
linking the difference between and


Question: package azure import ( <ins> "net/http" </ins> "strings" "github.com/Azure/azure-sdk-for-go/services/compute/mgmt/2019-07-01/compute"

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: check disk status before disk azure disk There is little possibility that attach disk would cost 10min, and disk could be attached, deleted in same time, that would cause race condition. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /kind bug /assign /priority important-soon /sig cloud-provider /area provider/azure


Question: return &effectiveVersion{} } binaryVersion := version.MustParse(binaryVer) <del> return newEffectiveVersion(binaryVersion) </del> <ins> return newEffectiveVersion(binaryVersion, false) } func defaultBuildBinaryVersion() *version.Version { verInfo := baseversion.Get() return version.MustParse(verInfo.String()).WithInfo(verInfo) </ins> } // DefaultBuildEffectiveVersion returns the MutableEffectiveVersion based on the // current build information. func DefaultBuildEffectiveVersion() MutableEffectiveVersion { <del> verInfo := baseversion.Get() binaryVersion := version.MustParse(verInfo.String()).WithInfo(verInfo) </del> <ins> binaryVersion := defaultBuildBinaryVersion() </ins> if binaryVersion.Major() == 0 && binaryVersion.Minor() == 0 { return DefaultKubeEffectiveVersion() } <del> return newEffectiveVersion(binaryVersion) </del> <ins> return newEffectiveVersion(binaryVersion, true) </ins> } // DefaultKubeEffectiveVersion returns the MutableEffectiveVersion based on the // latest K8s release. func DefaultKubeEffectiveVersion() MutableEffectiveVersion { binaryVersion := version.MustParse(baseversion.DefaultKubeBinaryVersion).WithInfo(baseversion.Get()) <del> return newEffectiveVersion(binaryVersion) </del> <ins> return newEffectiveVersion(binaryVersion, false) </ins> } // ValidateKubeEffectiveVersion validates the EmulationVersion is equal to the binary version at 1.31 for kube components.

Starting in 1.31, --version no longer works to modify the build ID of the running components. /kind regression /priority important-soon Running kube-apiserver v1.31.0 with ignores the modified build ID and reports In 1.28, 1.29, and 1.30 (since ), this allowed modifying the build ID of the running server. /sig api-machinery /assign Run v1.31.0 with No response 31.0 n/a <details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I think this broke in DefaultBuildEffectiveVersion() is grabbed in NewAPIServerCommand(), before the flag is parsed, so the GitVersion propagated into the snapshot effectiveVersion doesn't honor
cc


Question: "k8s.io/apiserver/pkg/server/healthz" "k8s.io/apiserver/pkg/server/mux" "k8s.io/apiserver/pkg/server/routes" <ins> "k8s.io/client-go/kubernetes/scheme" </ins> "k8s.io/client-go/tools/events" "k8s.io/client-go/tools/leaderelection" cliflag "k8s.io/component-base/cli/flag"

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Kube-Scheduler exposes metrics in a secure manner (TLS and authentication required) on port by default. If one does not authenticate itself, the error message displayed is: What you expected to happen: Kube-Scheduler should emit a clear and understandable error message if authentication fails. Kube-Controller-Manager does that the following way: How to reproduce it (as minimally and precisely as possible): up a Kubernetes Cluster to access the exposed metrics endpoint (https://metrics-server:/metrics) without authentication Anything else we need to know?: xref: Environment: Kubernetes version (use ): 1.18.6 Cloud provider or hardware configuration: Bare metal OS (e.g: ): Debian GNU/Linux 10 (buster) Kernel (e.g. ): 4.19.0-9-amd64 Install tools: Kubeadm (Kube-Scheduler )
/sig scheduling instrumentation
If ok, I can try this fix /assign
/assign


Question: // CreatePodsOnNode creates a new pod accorting to the spec on the specified node. CreatePodsOnNode(nodeName, namespace string, template *api.PodTemplateSpec, object runtime.Object) error // DeletePod deletes the pod identified by podID. <del> DeletePod(namespace string, podID string) error </del> <ins> DeletePod(namespace string, podID string, object runtime.Object) error </ins> } // RealPodControl is the default implementation of PodControlInterface.

Currently RC emits an event when it's creating a pod: But nothing is emitted when a pod is deleted. We should create event for scaling both ways. cc


Question: go_library( name = "go_default_library", <del> srcs = ["controller-manager.go"], </del> <ins> srcs = [ "controller-manager.go", "providers.go", ], </ins> importpath = "k8s.io/kubernetes/cmd/cloud-controller-manager", deps = [ "//cmd/cloud-controller-manager/app:go_default_library",

make verify WHAT=typecheck-providerless outputs /cc /milestone v1.19
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
see example successful run:
For reference -
fix in
screaming


Question: // IPVS service was changed, update the existing one // During updates, service VIP will not go down glog.V(3).Infof("IPVS service %s was changed", svcName) <del> if err := proxier.ipvs.UpdateVirtualServer(appliedVirtualServer); err != nil { </del> <ins> if err := proxier.ipvs.UpdateVirtualServer(vs); err != nil { </ins> glog.Errorf("Failed to update IPVS service, err:%v", err) return err }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: IPVS proxier update service too slow, it waits until a sync period. What you expected to happen: Any k8s service update should trigger IPVS virtual service update. How to reproduce it (as minimally and precisely as possible): a k8s service its session affinity type see if ipvs virtual service has changed - NO! Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: /sig network /area ipvs


Question: # check right pod exists kube::test::get_object_assert 'pods a' "{{${id_field}}}" 'a' # apply svc and don't prune pod a by overwriting whitelist <del> kubectl apply --prune -l prune-group=true -f hack/testdata/prune/svc.yaml -w core/v1/Service 2>&1 "${kube_flags[@]}" </del> <ins> kubectl apply --prune -l prune-group=true -f hack/testdata/prune/svc.yaml --prune-whitelist core/v1/Service 2>&1 "${kube_flags[@]}" </ins> kube::test::get_object_assert 'service prune-svc' "{{${id_field}}}" 'prune-svc' kube::test::get_object_assert 'pods a' "{{${id_field}}}" 'a' # apply svc and prune pod a with default whitelist

When writing the docs for , I found there is a flag with short hand flag has been add to in 1.5. The flag is mostly used by add-on manger. I think we only give the flag that is very commonly used a shorthand flag. I think we should keep and save it for other flags in the future. We can make the change before it goes out in 1.5. cc:


Question: return nil, err } <del> if fieldPatchStrategy == mergeDirective { </del> <ins> // Delete lists are handled the same way regardless of what the field's patch strategy is if fieldPatchStrategy == mergeDirective || isDeleteList { </ins> return mergeSlice(typedOriginal, typedPatch, schema, fieldPatchMergeKey, mergeOptions, isDeleteList) } else { return typedPatch, nil

This page explains that the deleteFromPrimitiveList should delete the specified items, but the specified items are kept, and the other items are deleted. The specified items in the deleteFromPrimitiveList should be removed. The result should be instead: No response <details</details<details</details<details</details<details</details<details</details<details</details
/sig api-machinery
Related:
/assign Could you take a look Antoine please? Thanks! /triage accepted
Simplest way to reproduce: It doesn't work with any kubectl version from 1.18 to 1.23 that I've tested.
I tried the exact same object, changing the finalizers works like a charm.
cc Want to give this one a try? :-)
Sure, I'll take a look at it... /assign
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale


Question: MUST cause pods created by that RC to be orphaned. release: v1.9 file: test/e2e/apimachinery/garbage_collector.go <ins> - testname: Namespace, apply changes to a namespace status codename: '[sig-api-machinery] Namespaces [Serial] should apply changes to a namespace status [Conformance]' description: Getting the current namespace status MUST succeed. The reported status phase MUST be active. Given the patching of the namespace status, the fields MUST equal the new values. Given the updating of the namespace status, the fields MUST equal the new values. release: v1.25 file: test/e2e/apimachinery/namespace.go </ins> - testname: namespace-deletion-removes-pods codename: '[sig-api-machinery] Namespaces [Serial] should ensure that all pods are removed when a namespace is deleted [Conformance]'

[x] test approval issue : [x] test pr : [x] two weeks soak end date : 1 Aug 2022 [x] test promotion pr : According to this APIsnoop query, there is still some remaining NamespaceStatus endpoints which are untested. readCoreV1NamespaceStatuspatchCoreV1NamespaceStatus Using a number of existing e2e test practices a new has been created for 2 namespace status endpoints. The e2e logs for this test are listed below. This query shows the endpoints hit within a short period of running the e2e test If a test with these calls gets merged, test coverage will go up by 3 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: if len(missingPods) > 0 { if usageRatio < 1.0 { <del> // on a scale-down, treat missing pods as using 100% of the resource request </del> <ins> // on a scale-down, treat missing pods as using exactly the target amount </ins> for podName := range missingPods { metrics[podName] = metricsclient.PodMetric{Value: targetUtilization} }

Change the fallback consumption from to when pod metrics are missing on a scale down. Background from : The fallback value of 100% is supposed to be a safe overestimate of the desired value to dampen the magnitude of a potential scale down. However, if a metric target is set to more than 100% then this fallback typically becomes an underestimate that amplifies a potential scale down. Note that a metric target of over 100% is a valid HPA configuration. We could choose a fallback of instead. However, any safety margin would be arbitrary and I think it's better to opt for simplicity vs. precision in this corner case.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig autoscaling
/assign
/assign


Question: ./cmd/libs/go2idl/conversion-gen                   | grep "^$(PRJ_SRC_PATH)"                          | xargs go list -e -f '{{.Dir}} {{.Dir}}/*.go');   <del> echo $$DIRECT $$INDIRECT | sed 's/ / nt/g';        </del> <ins> echo $$DIRECT $$INDIRECT | sed 's/ / =,/g' | tr '=,' 'nt';  </ins> ) | sed "s|$$(pwd -P)/||" > $@.tmp;                     cmp -s $@.tmp $@ || cat $@.tmp > $@ && rm -f $@.tmp

I don't know where the at the beginning of the target comes from, but probably from assumptions about sed version that are wrong. I had to update to gnu-sed 4.2.2 to build.
Also ootb on a Mac doesn't work with the rules specified (on el capitan):
does work
I take it back - updating sed did not appear to work. EDIT: I take that back, I had to clean after fixing paste.
most recent OSX version is UNIX certified, a POSIX requires as file argument when reading input from stdin. For , is only guaranteed to work in GNU sed, you need to insert literal newline and tab character there for portability.
Can someone with a mac send a patch? I don't have a mac...
It's update and every should end in instead of having no arguments. , Tim Hockin wrote:
(user should update sed, make needs changes to paste) , Clayton Coleman wrote:
I think we should stick to portable way for , it's too risky for relying on GNUism. If you don't mind, I will make a PR.
Sure, thanks , Cuong Manh Le wrote:
Are you sure the worked? I pull the latest from master, then run the same command above, I got: File is empty.
Make sure you are in your GOPATH in the correct spot for Kube On Jul 14, 2016, at 2:47 AM, Cuong Manh Le wrote: Are you sure the Makefile worked? I pull the latest from master, then run the same command above, I got: F0714 13:36:50. ] Error: Failed making a parser: unable to add directory "": unable to import "": expected 'package', found 'EOF' make: * [Makefile.generatedfiles:201: gendeepcopy] Error 255 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub , or mute the thread .
If you get the "expected 'package', found 'EOF'" message it is usually because something somewhere failed, and left an empty .go file. make cleangenerated , Cuong Manh Le wrote:
From BSD sed man page: Maybe try that?
It's not about ERE ( enables it), it's about expanding escape sequences in RHS of search and replace .


Question: CSINodeInfo:                    {Default: true, PreRelease: featuregate.GA, LockToDefault: true}, // remove in 1.19 BlockVolume:                    {Default: true, PreRelease: featuregate.GA, LockToDefault: true}, // remove in 1.20 StorageObjectInUseProtection:   {Default: true, PreRelease: featuregate.GA}, <del> SupportIPVSProxyMode:           {Default: true, PreRelease: featuregate.GA}, </del> SupportPodPidsLimit:            {Default: true, PreRelease: featuregate.GA, LockToDefault: true}, // remove in 1.21 SupportNodePidsLimit:           {Default: true, PreRelease: featuregate.GA, LockToDefault: true}, // remove in 1.21 HyperVContainer:                {Default: false, PreRelease: featuregate.Alpha},

When should it be removed? /sig network /kind cleanup
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/assign
Feature is GA since 1.11 - Any reference to this Feature Gate needs to be removed from the code, README and Docs: If you're a new contributor, please step in, take this issue, make the PRs, assign it to me with /assign and I'll be glad to help you through the journey of contributing :) /good-first-issue /help
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : Feature is GA since 1.11 - /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
are you willing to take this one?
Yes. I'm a new contributor.I will to try to complete it.
/assign Welcome! Let me know if you need any kind of help. Please take a look also into the Kubernetes Contributors documentation - When you're ready to create your first PR, please assign it to me and I'll make the first pass!
/sig docs We don't remove GA feature gates from the website; we move them to


Question: []string{"requestKind"}, ) kubectlExeRegexp = regexp.MustCompile(`^.*((?i:kubectl.exe))`) <ins> metrics = []resettableCollector{ requestCounter, longRunningRequestGauge, requestLatencies, requestLatenciesSummary, responseSizes, DroppedRequests, currentInflightRequests, } </ins> ) const (

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: endpoint of different k8s components expose Prometheus metrics that are not for them. For example, exposes metrics: or : What you expected to happen: Prometheus metrics only for to appear on kubelet's endpoint. How to reproduce it (as minimally and precisely as possible): Access the endpoint of all Kubernetes components Anything else we need to know?: This problem comes from usage of functions in different packages that automatically register Prometheus metrics. Just importing that package or some package that depends on it can cause those metrics to appear in unexpected places. In the list bellow you can find the different packages that have init functions and affected binaries: [ ] () cloud-controller-manager hyperkube kube-apiserver kube-controller-manager kube-proxy kube-scheduler [ ] () cloud-controller-manager hyperkube kube-apiserver kube-controller-manager kube-proxy kube-scheduler [ ] cloud-controller-manager hyperkube kube-apiserver kube-controller-manager [ ] cloud-controller-manager hyperkube kube-apiserver kube-controller-manager kubelet [ ] cloud-controller-manager hyperkube kube-apiserver kube-controller-manager kube-proxy kube-scheduler [ ] cloud-controller-manager hyperkube kube-apiserver kube-controller-manager kube-proxy kube-scheduler kubectl kubelet [ ] cloud-controller-manager hyperkube kube-apiserver kube-controller-manager kube-proxy kube-scheduler kubelet [ ] hyperkube kube-apiserver kube-controller-manager [ ] hyperkube kube-apiserver [ ] cloud-controller-manager hyperkube kube-apiserver kube-controller-manager kubelet [ ] cloud-controller-manager hyperkube kube-apiserver kube-controller-manager kube-proxy kube-scheduler kubelet Environment: Kubernetes version (use ):Cloud provider or hardware configuration: N/A OS (e.g. from /etc/os-release): N/A Kernel (e.g. ): N/A Install tools: N/A Others: N/A /sig instrumentation
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: allErrs = append(allErrs, ValidateImmutableField(service.Spec.ClusterIP, oldService.Spec.ClusterIP, field.NewPath("spec", "clusterIP"))...) } <ins> // TODO(freehan): allow user to update loadbalancerSourceRanges allErrs = append(allErrs, ValidateImmutableField(service.Spec.LoadBalancerSourceRanges, oldService.Spec.LoadBalancerSourceRanges, field.NewPath("spec", "loadBalancerSourceRanges"))...) </ins> allErrs = append(allErrs, ValidateService(service)...) return allErrs }

a service, type LoadBalancer. that and add a field with a value that the iptables rules do not appear It does seem that the GCE firewall rule is created. patch?
Is this a regression?
No. This has never been "right" as far as I can tell. , Phillip Wittrock wrote:
can this be closed now?
Nope. The root cause has not been fixed yet.


Question: selectPolicyFn = min // Use the default policy otherwise to produce a highest possible change } for _, policy := range scalingRules.Policies { <del> replicasDeletedInCurrentPeriod := getReplicasChangePerPeriod(policy.PeriodSeconds, scaleEvents) periodStartReplicas := currentReplicas + replicasDeletedInCurrentPeriod </del> <ins> replicasAddedInCurrentPeriod := getReplicasChangePerPeriod(policy.PeriodSeconds, scaleUpEvents) replicasDeletedInCurrentPeriod := getReplicasChangePerPeriod(policy.PeriodSeconds, scaleDownEvents) periodStartReplicas := currentReplicas - replicasAddedInCurrentPeriod + replicasDeletedInCurrentPeriod </ins> if policy.Type == autoscalingv2.PodsScalingPolicy { proposed = periodStartReplicas - policy.Value } else if policy.Type == autoscalingv2.PercentScalingPolicy {

We created an hpa as following: Then started a deployment with 4 replicas and made the CPU utilization about 10%, which proposed to double the replicas, and it did work: Then we changed the CPU utilization to about 1%, which would trigger scaling down and would kill 2 pods only for scaleDown behavior: Then changed the CPU utilization to about 20%, which would trigger scaling up: It did scale up, and for our expectation, it should be scaled to 620/5=24 but would be limited by scaleUp behavior as 4(1+300%)=16, but in fact, it was just scaled up to 8 It would be scaled up to 16 in last scaling actions Start an hpa with a scaleUp behavior which has little long period and a little great Percent value, then start a deployment with a container whose CPU utilization could be controlled easily. Change the CPU utilization to make it scale up within limit, scale down a little, and scale up again without limit. I think it is caused by , this function seems that it just considered about scaleUp event history when calculate scaleUp limit (so does calculate scaleDown limit). I think it is a bit unreasonable. Does this action match the design? Or it does be a bug? <details</details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig autoscaling
From what I understand, we observed the following behaviour: , there are pods. The HPA scales up new pods, so we have in total. , pods are scaled down, so we have in total. HPA calculates the desired Replicas with . Since all scaling events happened within the policies , the policy value of kicks in and should allow scaling up to . But it only scaled to .

I think that there is a bug in the following code: See the comments in the following snippets for an explanation: With a of 2, the is as follows: I will think of a possible solution to the problem and would like to contribute once I've made some progress on that :)
In fact, I knew this was caused by the calculating method of "getReplicasChangePerPeriod" and mentioned it in the "Anything else we need to know" of this issue. What I wonder now is that if it is an expected action or it is just a bug. If it quite is a bug, I hope I could contribute as well.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale


Question: ID:   "bar", }, }, <del> NewConflictErr("foo", "bar", errors.New("failure")): api.Status{ </del> <ins> NewConflictErr("foo", "bar", errors.New("failure")): { </ins> Status:  api.StatusFailure, Code:    http.StatusConflict, Reason:  "conflict",

https://travis- Seems surprising-- I will look into it later if no one else does first.
Got another flake. Looking into it.


Question: } } <del> func waitForDetachAndGrabMetrics(oldMetrics *storageControllerMetrics, metricsGrabber *metrics.Grabber) *storageControllerMetrics { </del> <ins> func waitForDetachAndGrabMetrics(oldMetrics *storageControllerMetrics, metricsGrabber *metrics.Grabber, pluginName string) *storageControllerMetrics { </ins> backoff := wait.Backoff{ Duration: 10 * time.Second, Factor:   1.2,

Which jobs are flaking: () () Which test(s) are flaking: Reason for failure: Timeout: Anything else we need to know: This test flaking about 10% of the time in this job. /sig storage /kind flake /priority important-soon cc
/remove-kind failing-test
/assign
I noticed one issue with the test is that it's not filtering metrics by plugin name. So it will just count the last plugin's metric. I'm not sure if the metrics collected are in any sorted order or not. If not, then when you have a suite that runs multiple plugins, then you might be pulling from the wrong metric.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-lifecycle rotten
still very prevalent in sig-release dashboards
I have pushed which should hopefully fix this.
Also PR against 1.17 branch -