Question: run(ctx, initializers) }, OnStoppedLeading: func() { <del> klog.Fatalf("leaderelection lost") </del> <ins> klog.ErrorS(nil, "leaderelection lost") klog.FlushAndExit(klog.ExitFlushTimeout, 1) </ins> }, })

kube-controller-manager yields whole stack trace when it loses leaderelection. kube-controller-manager to only log and to exit (without logging 17K lines of stack trace that is not useful for anyone). kube-controller-manager yields around 17K lines of stack trace (omitted in the above example). No response <detailsv1.20.12 </details<details</details<details</details<details</details<details</details<details</details
/sig api-machinery
/assign /triage accepted
has Fatalf always triggered a stack dump of all goroutines?
/cc
Seems like does a dump of all goroutines Could we and then maybe? /cc in case you have thoughts about this
If I read the right, it only printed the current goroutine to stderr. Dumping everything seems to have been later. make sense to me.
Dumping of all goroutines was in
I have similar doubts. I want to fix it. /assign
kube-scheduler seems to have a similar issue (but a shorter stack trace)
When replacing (useful because it doesn't do structured logging, not just because of the stack trace problem), remember to use + the new . Updated migration instructions are pending in There's also - reverting the klog change might be a faster way to address this.


Question: <ins> #!/usr/bin/env bash # Copyright 2021 The Kubernetes Authors. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # #     http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # This script validates that only a restricted set of packages are importing # github.com/prometheus/* # NOTE: this is not the same as verify-imports which can only verify # that within a particular package the imports made are allowed. # # This is also not the same thing as verify-import-boss, which is pretty # powerful for specifying restricted imports but does not scale to checking # the entire source tree well and is only enabled for specific packages. # # See: https://github.com/kubernetes/kubernetes/issues/99876 set -o errexit set -o nounset set -o pipefail KUBE_ROOT=$(dirname "${BASH_SOURCE[0]}")/.. source "${KUBE_ROOT}/hack/lib/init.sh" source "${KUBE_ROOT}/hack/lib/util.sh" # See: https://github.com/kubernetes/kubernetes/issues/89267 allowed_prometheus_importers=( ./cluster/images/etcd-version-monitor/etcd-version-monitor.go ./pkg/volume/util/operationexecutor/operation_generator_test.go ./staging/src/k8s.io/component-base/metrics/collector.go ./staging/src/k8s.io/component-base/metrics/collector_test.go ./staging/src/k8s.io/component-base/metrics/counter.go ./staging/src/k8s.io/component-base/metrics/counter_test.go ./staging/src/k8s.io/component-base/metrics/desc.go ./staging/src/k8s.io/component-base/metrics/gauge.go ./staging/src/k8s.io/component-base/metrics/gauge_test.go ./staging/src/k8s.io/component-base/metrics/histogram.go ./staging/src/k8s.io/component-base/metrics/histogram_test.go ./staging/src/k8s.io/component-base/metrics/http.go ./staging/src/k8s.io/component-base/metrics/labels.go ./staging/src/k8s.io/component-base/metrics/legacyregistry/registry.go ./staging/src/k8s.io/component-base/metrics/metric.go ./staging/src/k8s.io/component-base/metrics/opts.go ./staging/src/k8s.io/component-base/metrics/processstarttime_others.go ./staging/src/k8s.io/component-base/metrics/registry.go ./staging/src/k8s.io/component-base/metrics/registry_test.go ./staging/src/k8s.io/component-base/metrics/summary.go ./staging/src/k8s.io/component-base/metrics/testutil/metrics.go ./staging/src/k8s.io/component-base/metrics/testutil/metrics_test.go ./staging/src/k8s.io/component-base/metrics/testutil/promlint.go ./staging/src/k8s.io/component-base/metrics/testutil/testutil.go ./staging/src/k8s.io/component-base/metrics/value.go ./staging/src/k8s.io/component-base/metrics/wrappers.go ./test/e2e/apimachinery/flowcontrol.go ./test/e2e_node/dynamic_kubelet_config_test.go ./test/e2e_node/resource_metrics_test.go ./test/instrumentation/main_test.go ./test/integration/apiserver/flowcontrol/concurrency_test.go ./test/integration/metrics/metrics_test.go ) # Go imports always involve a double quoted string of the package path # https://golang.org/ref/spec#Import_declarations # # If you *really* need a string literal that looks like "github.com/prometheus/.*" # somewhere else that actually isn't an import, you can use backticks / a raw # string literal instead (which cannot be used in imports, only double quotes). # # NOTE: we previously had an implementation that checked for an actual import # as a post-processing step on the matching files, which is cheap enough and # accurate, except that it's difficult to guarantee we check for all supported # GOOS, GOARCH, and other build tags, and we want to prevent all imports. # So we dropped this, in favor of only the grep call. # See: https://github.com/kubernetes/kubernetes/pull/100552 really_failing_files=() all_failing_files=() while IFS='' read -r filepath; do # convert from file to package, and only insert unique results # we want to minimize the amount of `go list` calls we need to make if ! kube::util::array_contains "$filepath" "${allowed_prometheus_importers[@]}"; then # record a failure if not really_failing_files+=("$filepath") fi all_failing_files+=("$filepath") done < <(cd "${KUBE_ROOT}" && grep  --exclude-dir={_output,vendor}  --include='*.go'  -R .  -l  -Ee '"github.com/prometheus/.*"'  | LC_ALL=C sort -u) # check for any files we're allowing to fail that are no longer failing, so we # can enforce that the list shrinks allowed_but_not_failing=() for allowed_file in "${allowed_prometheus_importers[@]}"; do if ! kube::util::array_contains "$allowed_file" "${all_failing_files[@]}"; then allowed_but_not_failing+=("$allowed_file") fi done # we will exit with this at the end of the script depending on the checks below exit_code=0 # check for files we've allow-listed that no longer need to be if [ -n "${allowed_but_not_failing[*]}" ]; then { echo "ERROR: Some files allow-listed to import prometheus are no longer failing and should be removed." echo "Please remove these files from allowed_prometheus_importers in hack/verify-prometheus-imports.sh" echo "" echo "Non-failing but allow-listed files:" for non_failing_file in "${allowed_but_not_failing[@]}"; do echo "  ${non_failing_file}" done } >&2 exit_code=1 fi # check for files that fail but are not allow-listed if [ -n "${really_failing_files[*]}" ]; then { echo "ERROR: Some files are importing packages under github.com/prometheus/* but are not allow-listed to do so." echo "" echo "See: https://github.com/kubernetes/kubernetes/issues/89267" echo "" echo "Failing files:" for failing_file in "${really_failing_files[@]}"; do echo "  ${failing_file}" done } >&2 exit_code=2 fi exit "$exit_code" </ins>

See: xref: /sig testing /sig architecture /area code-organization /assign
/triage accepted /priority important-soon (maybe critical? we don't want to lapse on these imports)
Not a hard blocker for the release I think, we could backport this, but this should really happen in the milestone, and likely shouldn't take long
Tentatively is way too slow for this use case and is only capable of restricting imports within a particular package. We'll need a different approach.
I have a prototype of doing this much faster with ... bash, not sure what to call it yet and whether it's worth making general It seems the main thing we need here is check that imports are only in this list: Which we can do with a non-generic script that is pretty simple and fast.
AFAIK is planning to land this today
filed:
NOTE: this was implemented for the actively pursued SIG instrumentation restrictions. It was not implemented for all rules, it's not clear that the utility was high enough for all of these. We can add more restrictions using any of these alternative tools in the future as necessary.
What rules weren't implemented? Do you want to leave this open and move to v1.22 for those, or close this in v1.21 with a new issue (or not) for those rules?
There are a number of package groups in: It will take a bit to identify if any of them were actually used by any visibility rules, and if any of those are necessary (we have visibility rules on nearly every target, but most of that is pointless or implicit)
Er: and the prometheus bit is the only part that was actively worked on in the past few years. Other changes were due to targets referenced by the package groups being moved or removed in the sources and therefore needing updating there as well. For the most part active import management was done with import boss or importverifier, but I cannot say with certainty yet that there was no other useful visibility usage. I'm looking currently to verify. I looked previously but can't recall the details.
these were used in the past during the development of moving kubectl to staging, but now that it is staged that shouldn't be necessary I think. Everything else seems to either be the prometheus effort, or related to the kubectl move. cc


Question: "k8s.io/kubernetes/pkg/kubelet/cm/cpuset" "k8s.io/kubernetes/pkg/kubelet/cm/topologymanager" "k8s.io/kubernetes/pkg/kubelet/cm/topologymanager/bitmask" <ins> "k8s.io/kubernetes/pkg/kubelet/metrics" </ins> ) const (

Add to enable observability of the cpumanager. This is required to but it's also useful in general. To the best of my knowledge, it's very hard to observe the behavior of cpumanager without peeking into the pod and/or use the podresources interface (kubelet = 1.24). But even for the latter, no well known components expose metrics consuming this information. I believe anyway kubelet some expose some basic metrics about cpumanager behaviour, while deferring more fine grained metrics to external components (NFD, node_exporter, is fine)
/sig node /assign
/triage accepted /priority important-soon (as this is important for graduation for CPU Manager to GA)


Question: package scheduling <del> import "github.com/onsi/ginkgo" </del> <ins> import ( "time" "github.com/onsi/ginkgo" v1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/util/sets" clientset "k8s.io/client-go/kubernetes" "k8s.io/kubernetes/test/e2e/framework" e2epod "k8s.io/kubernetes/test/e2e/framework/pod" ) </ins> // SIGDescribe annotates the test with the SIG label. func SIGDescribe(text string, body func()) bool { return ginkgo.Describe("[sig-scheduling] "+text, body) } <ins> // WaitForStableCluster waits until all existing pods are scheduled and returns their amount. func WaitForStableCluster(c clientset.Interface, masterNodes sets.String) int { timeout := 10 * time.Minute startTime := time.Now() allPods := getAllPods(c) scheduledSystemPods, currentlyNotScheduledSystemPods := getSystemPods(c) systemPods := scheduledSystemPods + currentlyNotScheduledSystemPods // Wait for system pods to be scheduled, and for pods in all other namespaces to be deleted for currentlyNotScheduledSystemPods != 0 || systemPods != allPods { time.Sleep(2 * time.Second) scheduledSystemPods, currentlyNotScheduledSystemPods := getSystemPods(c) systemPods = scheduledSystemPods + currentlyNotScheduledSystemPods allPods = getAllPods(c) if startTime.Add(timeout).Before(time.Now()) { framework.Failf("Timed out after %v waiting for stable cluster.", timeout) break } } return scheduledSystemPods } // getAllPods lists all pods in the cluster, with succeeded and failed pods filtered out and returns the count func getAllPods(c clientset.Interface) int { allPods, err := c.CoreV1().Pods(metav1.NamespaceAll).List(metav1.ListOptions{}) framework.ExpectNoError(err, "listing all pods in kube-system namespace while waiting for stable cluster") // API server returns also Pods that succeeded. We need to filter them out. currentPods := make([]v1.Pod, 0, len(allPods.Items)) for _, pod := range allPods.Items { if pod.Status.Phase != v1.PodSucceeded && pod.Status.Phase != v1.PodFailed { currentPods = append(currentPods, pod) } } allPods.Items = currentPods return len(allPods.Items) } // getSystemPods lists the pods in the kube-system namespace and returns the number of scheduled and unscheduled pods func getSystemPods(c clientset.Interface) (int, int) { systemPods, err := c.CoreV1().Pods(metav1.NamespaceSystem).List(metav1.ListOptions{}) framework.ExpectNoError(err, "listing all pods in kube-system namespace while waiting for stable cluster") scheduledSystemPods, currentlyNotScheduledSystemPods := e2epod.GetPodsScheduled(masterNodes, systemPods) return len(scheduledSystemPods), len(currentlyNotScheduledSystemPods) } </ins>

What would you like to be : is used at the beginning of some e2e scheduler tests to wait until the cluster is in a stable state. It currently waits until all existing pods are scheduled, but it should also wait until pods from previous tests are actually deleted. We can do this by assuming that pods should exist only in the kube-system namespace, and so WaitForStableCluster should wait until there are no pods in any other namespace. Why is this needed: /help /sig testing /sig scheduling
/milestone v1.17 /priority critical-urgent
/cc
I'll take this one, can't figure out how to get ci bot to assign to me...
/assign
/remove-help


Question: t.Fatalf("unexpected error while writing: %v", err) } } <ins> func TestWriteAgainAfterUnexpectedExit(t *testing.T) { testCases := []struct { name       string payload    map[string]FileProjection simulateFn func(targetDir string, payload map[string]FileProjection) error }{ { name: "process killed before creating user visible files", payload: map[string]FileProjection{ "foo": {Mode: 0644, Data: []byte("foo")}, "bar": {Mode: 0644, Data: []byte("bar")}, }, simulateFn: func(targetDir string, payload map[string]FileProjection) error { for filename := range payload { path := filepath.Join(targetDir, filename) if err := os.RemoveAll(path); err != nil { return err } } return nil }, }, } for _, tc := range testCases { tc := tc t.Run(tc.name, func(t *testing.T) { targetDir, err := utiltesting.MkTmpdir("atomic-write") if err != nil { t.Fatalf("unexpected error creating tmp dir: %v", err) } defer func() { err := os.RemoveAll(targetDir) if err != nil { t.Errorf("%v: unexpected error removing tmp dir: %v", tc.name, err) } }() writer := &AtomicWriter{targetDir: targetDir, logContext: "-test-"} err = writer.Write(tc.payload, nil) if err != nil { t.Fatalf("unexpected error writing payload: %v", err) } err = tc.simulateFn(targetDir, tc.payload) if err != nil { t.Fatalf("failed to simulate the unexpected exit: %v", err) } err = writer.Write(tc.payload, nil) if err != nil { t.Fatalf("unexpected error writing payload again: %v", err) } checkVolumeContents(targetDir, tc.name, tc.payload, t) }) } } </ins>

When kubelet is creating configmap volume for a pod, it finished writing dir and made symlink to . At the same time, the kubelet is restarted, unfortunately the configmap item symlink is not created. The restarted kubelet will try to configmap, . Kubelet should create absent symlink for configmap volume. can hack kubelet to help reproduce it: sleep some time before ` kubelet and start official kubelet will skip to create symlink for UserVisibleFiles No response_ <details</details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
/sig storage This seems like an issue for sig storage. Can someone take a look and confirm?
Can you please add the version details as well?
/remove-sig node
/assign


Question: return deployments, nil } <ins> // ReplicaSetsByRevision sorts a list of ReplicaSet by revision, using their creation timestamp or name as a tie breaker. // By using the creation timestamp, this sorts from old to new replica sets. type ReplicaSetsByRevision []*apps.ReplicaSet func (o ReplicaSetsByRevision) Len() int      { return len(o) } func (o ReplicaSetsByRevision) Swap(i, j int) { o[i], o[j] = o[j], o[i] } func (o ReplicaSetsByRevision) Less(i, j int) bool { revision1, err1 := Revision(o[i]) revision2, err2 := Revision(o[j]) if err1 != nil || err2 != nil || revision1 == revision2 { return controller.ReplicaSetsByCreationTimestamp(o).Less(i, j) } return revision1 < revision2 } </ins>

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: when deployment controller cleans up old ReplicaSet, it sorts all old ReplicaSets by createTimeStamp and delete the oldest one, which may cause Deployment roll back to a wrong version. How to reproduce it (as minimally and precisely as possible): Suppose we set the revisionHistoryLimit of a Deployment to 3, and currently we have 3 Replicas: r1, r2, r3 (the createTimeStamp and Revision both are r3 r2 r1); if we upgrade and roll back 3 times continuously, there will be an expected case: upgrade:   ReplicaSet will be: r4, r3, r2   createTimeStamp: r4 r3 r2   reversion: r4 r3 r2 rollback: should rollback to v3   ReplicaSet will be: r4, r3, r2   createTimeStamp: r4 r3 r2   reversion: r3 r4 r2 upgrade:   ReplicaSet will be: r5, r4, r3   createTimeStamp: r5 r4 r3   reversion: r5 r3 r4 rollback: should rollback to v3   ReplicaSet will be: r4, r3, r2   createTimeStamp: r5 r4 r3   reversion: r3 r5 r4 upgrade:   ReplicaSet will be: r6, r5, r4   createTimeStamp: r6 r5 r4   reversion: r6 r5 r4 rollback: should rollback to r3, but instead, we rollbacked to r5, and this may cause error. What you expected to happen: when deployment controller cleans up old ReplicaSet, it should sort old ReplicaSets by Revision
/sig api-machinery /sig apps
does this make sense to you? /assign /cc
?
/remove-sig api-machinery
This requirement makes sense. Unassign API machinery - SIG Apps should own this.
Issues go stale after 90d of inactivity. Mark the issue as fresh with .

Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: "value": "{{ pillar.get('storage_backend', 'etcd3') }}" }, { "name": "TARGET_VERSION", <del> "value": "{{ pillar.get('etcd_version', '3.5.3') }}" </del> <ins> "value": "{{ pillar.get('etcd_version', '3.5.4') }}" </ins> }, { "name": "DO_NOT_MOVE_BINARIES",

Please publish 3.5.4-0 image and update to to benefit from fix for , while still keeping v3.5.3 critical correctness fixes (, ). For recent PRs for update to etcd v3.5.3 see and
/sig release /sig api-machinery /kind bug
/cc /triage accepted
Hey, Can I work on this ? I'm a beginner, but all the good first issues are mostly assigned. Can you guide me on doing this ?
I am working on this. /assign
Hi I would like to work on Make sense to backport in v1.22, v1.23 & v1.24? Thanks!!
hi, it makes sense to me, but requires approval from dependency maintainers. i can only approve the kubeadm changes.
was this backported ?


Question: <del> This file is autogenerated, but we've stopped checking such files into the repository to reduce the need for rebases. Please run hack/generate-docs.sh to populate this file. </del>

kubectl edit is not working on HEAD. It's working on 1.5.


Question: <del> This file is autogenerated, but we've stopped checking such files into the repository to reduce the need for rebases. Please run hack/generate-docs.sh to populate this file. </del>

Rolling-update tasks which worked since v1.0 are now getting stuck mid-way since v1.2 I'm rolling a multi-container pod from to . The target number of replicas is 4. The first 2 pods are rolled correctly, but the job hangs and times out (5min) after the first 2 iterations: Controller manager logs: Note the time it took to scale down on the second iteration, although all the pods marked new transitioned to a state in less than 10sec. My RC selector matches 3 labels. After the timeout the 2 RCs look like:
Can you please share the exact command you are running that leads to timeout? cc
I first get the current RC definition in YAML format, replace the image tag and label, and finally pass it to:
Were you able to reproduce this consistently (i.e. this always fail)?
Yes, 100%. It eventually goes through if I increase the timeout but I still have to wait a couple of minutes between each pod. I switched this deployment task to the API with the Rolling-update strategy and it's extremely fast.
Yes, we'd recommend using  since it's faster, declarative and has more features, such as you can rollback to a previous reversion even when the rollout/rolling-update is done. When you're using , is it possible that there are overlapping resources fighting with , such as replicasets, deployments, or replication controllers with the same label selector?
labels are not conflicting. I have 2 other rcs in that namespace and each of them has a different label (see my first comment). Nothing has changed in that deployment procedure since Kubernetes v1.0, and v1.2 is the first version that seems to struggle. I can post my rc definition here, just in case you can't reproduce with a generic 3-container pod.
Yes, posting the RC definition would help, since I cannot reproduce it on my end. Thanks!
The following definition times out as described above. It resembles our real definition, minus a log forwarding container which I replaced by something generic like postgres.

Secrets: Create the rc and secrets: Now delete the rc without killing the pods: In the definition, rename the rc to anything else, and recreate it: In the definition, rename the rc back to the original name "issue23276" and bump the "deployment" number in the label, then perform a rolling-update: It remains stuck in this state:
cc
Quick observations: In 1.1 and 1.2, rolling update takes pod readiness into account before proceeding. Are the pods ready? Use "kubectl get -o yaml pod ...". Cpu limits are now enforced using cpu quota and 10m is a very small limit. Specify "requests" instead.
In and specifically refererring to , it seems that this pod is never scaled down (based on the following output from . Can you inspect the container logs from the old pods for anything weird?
This limit is low on purpose, this container only starts . In our production environment we use it for log rotation with the same CPU limits. When deployed using the API everything works and the rolling update takes less than a minute. All pods are running as expected and don't show any error.
Same problem here. We don't use any limits, but simple scale down more then 3 pods is hangs. One moment about rolling-update issue: it appears only in case of renaming original rc; if use it by default all ok.
should help in identifying if the error comes from the scaler.
Must add to message, that with bigger time-out (--timeout=30m), rolling-update proceeds after 3 pods but much slower. On master I have following log for 5 pods RC: Message like appear every time before update makes progress after a ~5 minutes freeze.
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list .
/sig cli
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or .

/lifecycle stale
We should consider deprecating the rolling-update command.
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close


Question: ProbeRemove ) <ins> var ErrNoPluiginMatched = errors.New("no volume plugin matched") </ins> // VolumeOptions contains option information about a volume. type VolumeOptions struct { // The attributes below are required by volume.Provisioner

In the events , the PVC has been successfully resized, but a warning-level log is still printed. And by checking the logs of kube-controller-manager, can see the following output. Do not print events with warning level. To expand a a CSI PVC, and then use "kubectl describe pvc" to view its events. (I am using xsky CSI to expand pvc, but based on my understanding of Kubernetes code, this issue can occur with any CSI.) ! ! Based on Kubernetes v1.27.3, I some logs in the func located in . These logs printed all the plugins before expansion, and I noticed that was not among them. In my understanding, it is normal for CSI to be absent from the plugins here because CSI expansion is scheduled through external-resizer instead of kube-controller-manager. Therefore, there should not be any warning-level logs printed here either. v1.27.3 (In version 1.18, the issue can also be reproduced) <details</detailsLinux debian-hk 5.10.0-23-amd64 SMP Debian 5.10.179-1 (2023-05-12) x86_64 GNU/Linux kind v0.20.0 <details</details<details</details
/sig storage
We can change the ExternalExpanding event from Warning to Info. are you interested in making this change?
/triage accepted
Yes, I will submit the pull request in the next few days.
Hi Have you submitted the PR? I have a bugfix branch tested on v1.29.1. I would like to cherry-pick it to master if this is not fixed yet or not started.
You can go do it. Thank you!


Question: "flag" "fmt" "io/ioutil" <ins> "net/http" </ins> "os" "reflect" "sort"

The kubecfg -c option is used to pass a file pointer to a representation of a model object on disk (pod, replicationController, service) for use by the operation. It would be nice if I could do the following: Where kubecfg sees the http:// and does a network operation to fetch resource for me. This would let users who want to build tutorials on how to use kubernetes to avoid steps to manually download the files locally via curl and unzip, etc. and would also let scripters that use the kubecfg client to avoid needing write access on the terminal where they run kubecfg to perform basic actions.


Question: return } <del> ch <- metrics.NewLazyConstMetric(containerStartTimeDesc, metrics.GaugeValue, float64(s.StartTime.UnixNano())/float64(time.Second), s.Name, pod.PodRef.Name, pod.PodRef.Namespace) </del> <ins> ch <- metrics.NewLazyMetricWithTimestamp(s.StartTime.Time, metrics.NewLazyConstMetric(containerStartTimeDesc, metrics.GaugeValue, float64(s.StartTime.UnixNano())/float64(time.Second), s.Name, pod.PodRef.Name, pod.PodRef.Namespace)) </ins> } func (rc *resourceMetricsCollector) collectContainerCPUMetrics(ch chan<- metrics.Metric, pod summary.PodStats, s summary.ContainerStats) {

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI -- (containerd) (crio) [sig-node] ResourceMetricsAPI [NodeFeature:ResourceMetrics] when querying /resource/metrics should report resource usage through the resource metrics api Ever since this commit was merged (containerd) (crio) We need to see what the commit is breaking this particular NodeFeatures test on both crio and containerd.
/sig node
/cc
/cc
/triage accepted
ah, let's use the RT's issue /close
Closing this issue. <detailsIn response to : ah, let's use the RT's issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: if err != nil { return results, err } <ins> allErrs := []error{} </ins> for _, apiGroup := range serverGroupList.Groups { preferredVersion := apiGroup.PreferredVersion apiResourceList, err := discoveryClient.ServerResourcesForGroupVersion(preferredVersion.GroupVersion) if err != nil { <del> return results, err </del> <ins> allErrs = append(allErrs, err) continue </ins> } groupVersion := unversioned.GroupVersion{Group: apiGroup.Name, Version: preferredVersion.Version} for _, apiResource := range apiResourceList.APIResources {

I have been having the kube-controller-manager (running in a container) crash quite frequently with this error: I cant fathom what seems to cause it. It seems to happen consistently for a while (CrashLoopBackoff occurs) and then fixes itself after an undetermined amount of time. Kubectl version: Docker version: Networking: Calico (with the calico k8s policy agent) Please do let me know if there is any other information that I can supply. I appreciate I haven't given much to go off of. I am not sure what the steps to reproduce are...
cc
It looks like a bug. I'll take a look. Thanks for reporting it.
Weirdly, the description showed up in the email github sent me but it appears absent on the github issue?
The description was there when I read it but disappeared now.
Well thats very odd! If you have it on your email, could you post it on here please (and I will update the description)? I unfortunately haven't got the error message to hand now...
I took the liberty of just editing the OP.
thanks a lot! Not sure what happened there!
could you paste the returned output at "server-ip:port/api" and "server-ip:port/apis"?
Sure.
Thanks "" has an empty preferred version and that's causing the error you reported. Is "" created via the thirdparty resource api? ()
aha! Good spot! Yes, it was created via that API. This is the Yaml file I used to create it: Here is the output of
Thanks for the confirmation Your configuration is correct. The preferred version is left empty by the thirdparty resource controller, while our discovery client has an assumption that the preferred version won't be empty. I'll patch the discovery client to tolerate an empty preferred version, and will discuss with on if we should make a default preferred version for a thirdparty resource.
Thanks a lot Great job on working it out :)
Sent to fix.
What would be a workaround for this before the release, how can one specify the preferredVersion?
AFAIK, there is no workaround. There is no way to specify the preferredVersion for the thirdPartyResource.

I have a quick workaround to fix this : 1) Change the kubernetes to reflect the following : from : thirdpartyresources=true to : thirdpartyresources=false 2) Then restart 'kubelet' service OR reboot the K8S Master server systemctl restart kubelet 3) Revert back the changes done to kube- from : thirdpartyresources=false to : thirdpartyresources=true 4) Verify whether the above change on the above manifest file has been automatically picked up by the kubernetes API ps aux | grep -i thirdpartyresources 5) Finally you may restart kubelet service systemctl restart kubelet 6) Now verify whether the 'kube-controller-manager' is in a 'Running' state kubectl get pods --all-namespaces | grep -i controller-manager Hope this fix works for you guys as well !!


Question: t.Errorf("Failed to create node, err: %v", err) } } <ins> // Regularly send heartbeat event to APIServer so that the cluster doesn't enter fullyDisruption mode. // TODO(Huang-Wei): use "NodeDisruptionExclusion" feature to simply the below logic when it's beta. var heartbeatChans []chan struct{} for i := 0; i < nodeCount; i++ { heartbeatChans = append(heartbeatChans, make(chan struct{})) } for i := 0; i < nodeCount; i++ { // Spin up <nodeCount> goroutines to send heartbeat event to APIServer periodically. go func(i int) { for { select { case <-heartbeatChans[i]: return case <-time.Tick(2 * time.Second): nodes[i].Status.Conditions = []v1.NodeCondition{ { Type:              v1.NodeReady, Status:            v1.ConditionTrue, LastHeartbeatTime: metav1.Now(), }, } updateNodeStatus(cs, nodes[i]) } } }(i) } </ins> neededNode := nodes[1] if test.pod != nil { test.pod.Name = fmt.Sprintf("testpod-%d", i)

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: Which test(s) are failing: TaintBasedEvictions Since when has it been failing: probably since it was in this PR? Testgrid link: Reason for failure: Anything else we need to know: /sig scheduling /kind flake /priority important-soon /milestone v1.17
You must be a member of the GitHub team to set the milestone. If you believe you should be able to issue the /milestone command, please contact your and have them propose you as an additional delegate for this responsibility. <detailsIn response to : <!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs -- TaintBasedEvictions probably since it was in this PR? Anything else we need to know: /sig scheduling /kind flake /priority important-soon /milestone v1.17 </details
cc.
Looking into this. We moved it to integration because it was flaking in e2e too
/assign damemi
Not sure yet, but I do see this in the logs of the failing tests: which, if I understand this, would remove all the taints and mark nodes as reachable (which would explain why the test is failing waiting for node taints: ). But I need to figure out why all the nodes are becoming unavailable
is that a full fix for this flake or just partial? I still saw it failed yesterday so just checking... :)
I think I also run into this on 13th Oct: Full logs:
Now gets merged, which might fix the flake. Let's keep an eye on this.
reported it still flaked: Can you continue looking into this?
I'll keep trying, from my previous debugging it seems that in some cases the test cluster is entering full disruption mode and removing all of the taints from the nodes. I know we specifically made changes to the test (using multiple nodes) to ensure it wouldn't become fully disrupted, do you have any idea what could still be causing that?
There is some flag controlling how the dispution thing is managed internally, maybe you can check how a node lifecycle manager is constructed.

The test still flakes after we merge , we still need to investigate the cause ,
/assign
Another flake:
These flakes are consistently (always) timeouts waiting on node taints:
I'm working on this. Will send out a PR soon.
/reopen We see another flake since got merged: However, don't panic :) The original "Entering master disruption mode" cause has been fixed - you won't see that message in the above log. This time it's a smaller issue. I will explain it in another PR.
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen The job is still flaky
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/close But for another test :)
Closing this issue. <detailsIn response to : /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: func TestWatchSucceedsWithoutArgs(t *testing.T) { _, s := framework.RunAMaster(t) <del> defer s.Close() </del> <ins> // TODO: Uncomment when fix #19254 // defer s.Close() </ins> resp, err := http.Get(s.URL + "/api/v1/namespaces?watch=1") if err != nil {

http://pr- is another failure.
Also seen in http://pr-
Again. Not assigned to anyone, so I am sending to for routing
Again: Is there anyone working on this? I will spend some time looking into this.
I think that the name of this issue is misleading - the issue seems to be unrelated to etcd_watcher Basically, if you look into this part of log: So it seems that, the WaitGroup that is problematic is in httptest.Server. So my feeling is that we are simply talking to a server that is already being close and this is ~roughly duplicate of We should just find where exactly the problem is and try to work-around it.
I think this is exactly what we had before. I will send a PR with the fix today.


Question: } } <del> const networkContainerName = "k8snet" </del> <ins> const networkContainerName = "net" </ins> func (kl *Kubelet) networkContainerExists(manifest *api.ContainerManifest) (string, bool, error) { pods, err := kl.ListContainers()

release-master-blocking Conformance-GCE-master-kubetest2Kubernetes e2e suite.[It] [sig-network] Services should be able to change the type from ExternalName to ClusterIP [Conformance] ci-kubernetes-gce-conformance-latest-kubetest2.Overall kubetest2.Test 07-11 15:51 EEST No response /sig cloud-provider cc
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I remember some timeout has changed with ginkgo2 and something moved from 24hours to 1 hours, or something like that, is it possible that this is the problem here, in this job?
The Grinko timeout change could also be the root problem with this job:
hmm, we have to revisit the flags
hmm, this seems correct
that was a good observations, seems that job is failing because gathering the logs takes too much time !
there is no timeout set in the kubetest2 command here (not sure if this would be the place to set a timeout)
it seems kubetest has to be migrated to ginkgov2 it should use this as basis I'm on vacation this week, can somebody take over and open the issue and patch kubetest2?
looks like this file needs to get updated: add line 72:
oh, you already commented this :D
I will have a look later
Related: umbrella issue:
last testgrid run passed :)


Question: store.updateCache(add) } } <ins> func TestHistogramCacheReadWait(t *testing.T) { registry := k8smetrics.NewKubeRegistry() if err := registry.Register(metrics.WatchCacheReadWait); err != nil { t.Errorf("unexpected error: %v", err) } ctx := context.Background() testedMetrics := "apiserver_watch_cache_read_wait_seconds" store := newTestWatchCache(2, &cache.Indexers{}) defer store.Stop() // In background, update the store. go func() { if err := store.Add(makeTestPod("foo", 2)); err != nil { t.Errorf("unexpected error: %v", err) } if err := store.Add(makeTestPod("bar", 5)); err != nil { t.Errorf("unexpected error: %v", err) } }() testCases := []struct { desc            string resourceVersion uint64 want            string }{ { desc:            "resourceVersion is non-zero", resourceVersion: 5, want: ` # HELP apiserver_watch_cache_read_wait_seconds [ALPHA] Histogram of time spent waiting for a watch cache to become fresh. # TYPE apiserver_watch_cache_read_wait_seconds histogram apiserver_watch_cache_read_wait_seconds_bucket{resource="pods",le="0.005"} 1 apiserver_watch_cache_read_wait_seconds_bucket{resource="pods",le="0.025"} 1 apiserver_watch_cache_read_wait_seconds_bucket{resource="pods",le="0.05"} 1 apiserver_watch_cache_read_wait_seconds_bucket{resource="pods",le="0.1"} 1 apiserver_watch_cache_read_wait_seconds_bucket{resource="pods",le="0.2"} 1 apiserver_watch_cache_read_wait_seconds_bucket{resource="pods",le="0.4"} 1 apiserver_watch_cache_read_wait_seconds_bucket{resource="pods",le="0.6"} 1 apiserver_watch_cache_read_wait_seconds_bucket{resource="pods",le="0.8"} 1 apiserver_watch_cache_read_wait_seconds_bucket{resource="pods",le="1"} 1 apiserver_watch_cache_read_wait_seconds_bucket{resource="pods",le="1.25"} 1 apiserver_watch_cache_read_wait_seconds_bucket{resource="pods",le="1.5"} 1 apiserver_watch_cache_read_wait_seconds_bucket{resource="pods",le="2"} 1 apiserver_watch_cache_read_wait_seconds_bucket{resource="pods",le="3"} 1 apiserver_watch_cache_read_wait_seconds_bucket{resource="pods",le="+Inf"} 1 apiserver_watch_cache_read_wait_seconds_sum{resource="pods"} 0 apiserver_watch_cache_read_wait_seconds_count{resource="pods"} 1 `, }, { desc:            "resourceVersion is 0", resourceVersion: 0, want:            ``, }, } for _, test := range testCases { t.Run(test.desc, func(t *testing.T) { defer registry.Reset() if _, _, _, err := store.WaitUntilFreshAndGet(ctx, test.resourceVersion, "prefix/ns/bar"); err != nil { t.Errorf("unexpected error: %v", err) } if err := testutil.GatherAndCompare(registry, strings.NewReader(test.want), testedMetrics); err != nil { t.Errorf("unexpected error: %v", err) } }) } } </ins>

Part of From For now posted as a "help wanted" issue, however if we cannot hit the timeline for 1.30 code freeze () I would take it over so we don't miss. Still issue is simple enough that it can be quickly implemented. Implementation tips: Add a metric around waitUnitlFreshAndBlock method Metric should be a histogram with latency buckets similar to api latency Metric should have label for resource version. A way to monitor watch cache impact on request latency.
/sig api-machinery /triage accepted /help
This request has been marked as needing help from a contributor. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /sig api-machinery /triage accepted /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: R: &etcd.Response{ EtcdIndex: 10, Node: &etcd.Node{ <ins> Dir: true, </ins> Nodes: []*etcd.Node{ { <ins> Key:   "/directory1", </ins> Value: `{"name": "directory1"}`, Dir:   true, Nodes: []*etcd.Node{ { <ins> Key:           "/foo", </ins> Value:         `{"id":"foo","kind":"Pod","apiVersion":"v1beta1"}`, <ins> Dir:           false, ModifiedIndex: 1, }, { Key:           "/baz", Value:         `{"id":"baz","kind":"Pod","apiVersion":"v1beta1"}`, Dir:           false, </ins> ModifiedIndex: 1, }, }, }, { <ins> Key:   "/directory2", </ins> Value: `{"name": "directory2"}`, Dir:   true, Nodes: []*etcd.Node{ { <ins> Key:           "/bar", </ins> Value:         `{"id":"bar","kind":"Pod","apiVersion":"v1beta1"}`, ModifiedIndex: 2, },

kubectl returns resources in a random order, which makes it hard to look for things when users have a lot resources. Returning resources in deterministic order should be a helpful and trivial change (hopefully). We have an issue about pagination , but that's a larger and separate issue.
We should be cause to not do this on the * namespace - I think thats a special case. No disagreement for other list calls
What is the default etcd response order? Last created or last modified? Sent from my iPhone
I think the order is by key if sort param is given. If not given, the order is by last created (maybe, but we shouldn't trust it). It seems we are not trying to get sorted result from etcd.
Today, I think it's random order in the child key space in etcd (because the node structure uses a map), so you get golang map iteration order without sort. I don't know if sort on server is any cheaper than sort on client today, so we may want to do this in rest / registry rather than by passing etcd.
Apart from performance, sorting in clients means we can also take control of how responses are sorted, so for * namespace, we can sort by namespace, then by resource ID. , Clayton Coleman wrote: Master of Electrical and Computer Engineering Carnegie Mellon University
We can always hint sort to the registry storage layer in the future, so sorting on the client now won't kill us.
Just took a closer look at our rest/registry code to implement the sort (in client), but implementation-wise, it seems much easier to pass sort to etcd (one line change). Also, namespace is a prefix on resource, so sort across all namespace with etcd key will essentially sort resources by namespace then by Name. If we are to implement sort on client side, we can either: sort package: have each list object, ie.e PodList, ControllerList, implement sort interface. (We won't do this). have ListMeta implement sort interface, if there is a way to find enclosing structs (so we can use items field) have api.Object implement sort interface, verify object is list before sorting. sort algorithm ourself, and do it in rest. Is there other golang or kubernetes idom that I haven't been aware of that can be applied to this use case?

----- Original Message ----- I think sorting it on etcd is fine for now - that's the simplest possible thing that works.
Is there any material perf impact in etcd to enable it? Sent from my iPhone
Probably not material. Can always switch away if necessary
Famous last words :-) Sent from my iPhone
Got my hat ready


Question: apiextensions.CRDRemoveFinalizer(crd, apiextensions.CustomResourceCleanupFinalizer) crd, err = c.crdClient.CustomResourceDefinitions().UpdateStatus(crd) <ins> if apierrors.IsNotFound(err) || apierrors.IsConflict(err) { // deleted or changed in the meantime, we'll get called again return nil } </ins> if err != nil { return err }

k8s version: v1.9.3 When I create customresourcedefinition, There is error like Operation cannot be fulfilled on "": the object has been modified; please apply your changes to the latest version and try again . apiVersion: kind: CustomResourceDefinition metadata: name: annotations: "": crd-install labels: app: istio-pilot spec: group: names: kind: VirtualService listKind: VirtualServiceList plural: virtualservices singular: virtualservice categories: istio-io networking-istio-io scope: Namespaced version: v1alpha3 So what is the reason of it?
a few questions: where are you seeing the error, in the apiserver log or in the output of the kubectl command? what kubectl command are you using to create the object? are you running HA apiservers?
The log is shown by apiserver. I run k8s in HA apiservers. The command I used is "kubectl apply -f "+file.
API servers attempt to update CRD status to indicate whether the names were available. In an HA apiserver setup, the API servers race to do so, and all but the first lose the race with a resource conflict error. That is not unexpected. Opened to avoid logging normal API responses in that scenario.
Thanks ! But I want to know whether the CRD has been set successfully in HA apiservers ? I use the command " kubectl get crd -o yaml" and get the following results. ! I see its status is true. But when i create a virtualservices, it returns error. Error from server (Timeout): error when creating "cms-api-": Timeout: request did not complete within allowed duration ! So i want to know how to solve this error.
/assign
Excuse me. Do you have some greate ideas to solve this issue??
There is a PR for this sent by
I see your request is timed-out not failed. In HA, I expect the request to fail in the first seconds of the CRD creation, but after that it should go through. I need more logs during this timeout or a consistent way to reproduce it.


Question: } configFileName := filepath.Join(dir, "config.yaml") <del> if err := os.WriteFile(configFileName, []byte(` </del> <ins> if err := atomicWriteFile(configFileName, []byte(` </ins> apiVersion: apiserver.config.k8s.io/v1alpha1 kind: AuthorizationConfiguration authorizers:

ci-kubernetes-integration-master --- FAIL: TestMultiWebhookAuthzConfig (9.23s) (03-20) I cannot search for previous flake. updated recently /sig auth /cc
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Updated the description, the actual cause of the flake is an unexpected change in the file reload timestamp metric
interestingly, doesn't show any failures of this test for this reason (there are a few failures on a PR that broke a bunch of integration tests for a different reason)
Fix open in , switched the file write to an atomic write
No flake after 03-29. Thanks.


Question: func waitForDeploymentUpdatedReplicasGTE(c clientset.Interface, ns, deploymentName string, minUpdatedReplicas int32, desiredGeneration int64) error { return testutil.WaitForDeploymentUpdatedReplicasGTE(c, ns, deploymentName, minUpdatedReplicas, desiredGeneration, poll, pollLongTimeout) } <ins> // Deployment should have a working scale subresource func testDeploymentSubresources(f *framework.Framework) { ns := f.Namespace.Name c := f.ClientSet deploymentName := "test-new-deployment" framework.Logf("Creating simple deployment %s", deploymentName) d := e2edeployment.NewDeployment("test-new-deployment", int32(1), map[string]string{"name": WebserverImageName}, WebserverImageName, WebserverImage, appsv1.RollingUpdateDeploymentStrategyType) deploy, err := c.AppsV1().Deployments(ns).Create(context.TODO(), d, metav1.CreateOptions{}) framework.ExpectNoError(err) // Wait for it to be updated to revision 1 err = e2edeployment.WaitForDeploymentRevisionAndImage(c, ns, deploymentName, "1", WebserverImage) framework.ExpectNoError(err) err = e2edeployment.WaitForDeploymentComplete(c, deploy) framework.ExpectNoError(err) _, err = c.AppsV1().Deployments(ns).Get(context.TODO(), deploymentName, metav1.GetOptions{}) framework.ExpectNoError(err) ginkgo.By("getting scale subresource") scale, err := c.AppsV1().Deployments(ns).GetScale(context.TODO(), deploymentName, metav1.GetOptions{}) if err != nil { framework.Failf("Failed to get scale subresource: %v", err) } framework.ExpectEqual(scale.Spec.Replicas, int32(1)) framework.ExpectEqual(scale.Status.Replicas, int32(1)) ginkgo.By("updating a scale subresource") scale.ResourceVersion = "" // indicate the scale update should be unconditional scale.Spec.Replicas = 2 scaleResult, err := c.AppsV1().Deployments(ns).UpdateScale(context.TODO(), deploymentName, scale, metav1.UpdateOptions{}) if err != nil { framework.Failf("Failed to put scale subresource: %v", err) } framework.ExpectEqual(scaleResult.Spec.Replicas, int32(2)) ginkgo.By("verifying the deployment Spec.Replicas was modified") deployment, err := c.AppsV1().Deployments(ns).Get(context.TODO(), deploymentName, metav1.GetOptions{}) if err != nil { framework.Failf("Failed to get deployment resource: %v", err) } framework.ExpectEqual(*(deployment.Spec.Replicas), int32(2)) ginkgo.By("Patch a scale subresource") scale.ResourceVersion = "" // indicate the scale update should be unconditional scale.Spec.Replicas = 4    // should be 2 after "UpdateScale" operation, now Patch to 4 deploymentScalePatchPayload, err := json.Marshal(autoscalingv1.Scale{ Spec: autoscalingv1.ScaleSpec{ Replicas: scale.Spec.Replicas, }, }) framework.ExpectNoError(err, "Could not Marshal JSON for patch payload") _, err = c.AppsV1().Deployments(ns).Patch(context.TODO(), deploymentName, types.StrategicMergePatchType, []byte(deploymentScalePatchPayload), metav1.PatchOptions{}, "scale") framework.ExpectNoError(err, "Failed to patch deployment: %v", err) deployment, err = c.AppsV1().Deployments(ns).Get(context.TODO(), deploymentName, metav1.GetOptions{}) framework.ExpectNoError(err, "Failed to get deployment resource: %v", err) framework.ExpectEqual(*(deployment.Spec.Replicas), int32(4), "deployment should have 4 replicas") } </ins>

[x] two weeks soak end date : 26 Feb 2021 According to this APIsnoop query, there are still some remaining Deployment endpoints which are untested. with this query you can filter untested endpoints by their category and eligibility for conformance. e.g below shows a query to find all conformance eligible untested,stable,core endpoints Create a Deployment with a static label Read the Deployment Update the DeploymentScale Get the DeploymentScale to ensure it is updated Patch the DeploymentScale Read the DeploymentScale to ensure it is Patch Delete the Namespace and Deployment Due to the complexity of setting up the resources for APIService we have used the current e2e test as template. It has been extended in a for review. example endpoint | useragent ----------------------------------------|
patchAppsV1NamespacedDeploymentScale | (linux/amd64) kubernetes/$Format -- [sig-apps] Deployment Deployment should have a working scale subresource readAppsV1NamespacedDeploymentScale | (linux/amd64) kubernetes/$Format -- [sig-apps] Deployment Deployment should have a working scale subresource replaceAppsV1NamespacedDeploymentScale | (linux/amd64) kubernetes/$Format -- [sig-apps] Deployment Deployment should have a working scale subresource (3 rows) If a test with these calls gets merged, test coverage will go up by N points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted
Issue was presented in the Conformance meeting of 9 Feb 2021 and it was agreed that that method see to achieve the purpose and a PR can be created. A similar approach would be taken for the DepoymentScale endpoints, and this issue was created for that purpose.
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: } } <del> func (fopg *fakeOperationGenerator) GenerateMountVolumeFunc(waitForAttachTimeout time.Duration, volumeToMount VolumeToMount, actualStateOfWorldMounterUpdater ActualStateOfWorldMounterUpdater, isRemount bool) (volumetypes.GeneratedOperations, error) { </del> <ins> func (fopg *fakeOperationGenerator) GenerateMountVolumeFunc(waitForAttachTimeout time.Duration, volumeToMount VolumeToMount, actualStateOfWorldMounterUpdater ActualStateOfWorldMounterUpdater, isRemount bool) volumetypes.GeneratedOperations { </ins> opFunc := func() (error, error) { startOperationAndBlock(fopg.ch, fopg.quit) return nil, nil } return volumetypes.GeneratedOperations{ OperationFunc: opFunc, <del> }, nil </del> <ins> } </ins> } func (fopg *fakeOperationGenerator) GenerateUnmountVolumeFunc(volumeToUnmount MountedVolume, actualStateOfWorld ActualStateOfWorldMounterUpdater, podsDir string) (volumetypes.GeneratedOperations, error) { opFunc := func() (error, error) {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: In this test, volume mount failed because the driver-registrar took a minute to come up. However, it seems like volume mount operation is retrying without any sort of backoff. This message appears in 400+ times in less than a minute. What you expected to happen: Operation executor should not be retrying operations so frequently. <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
I'll take a look at this. /assign
Problem is that reconciler depends on operation executor to do backoff. In the normal case operation executor returns a method that does the actual work. Any errors by that method result in exponential backoff. But in this case, the error is coming from the code creating the mount method which is not handled by the back off logic, and is called again and again by the reconciler resulting in log and event spam. This didn't use to be a problem because initialization errors were uncommon (plugins were built in), but with CSI the driver may not exist and these initialization errors are expected. I'll look in to cleaning this up.
Is this something that will affect users in 1.13 and needs CPing?
This will result in kubelet log spam and possibly kubernetes event spam. It happens when someone creates a pod referencing a CSI volume before the CSI driver for that volume is installed. Which happens frequently in E2E tests because they install/remove driver with every test. But should be rare in real world usage since most users are likely to install a driver first before using it.
If release team is worried about this, we could create a quick PR to suppress logging/event generation of this these errors. Otherwise we plan to address this more holistically for 1.14 and 1.13.1.


Question: tc.runTest(t) } <ins> func TestReplicaCalcDuringRollingUpdateWithMaxSurge(t *testing.T) { tc := replicaCalcTestCase{ currentReplicas:  2, expectedReplicas: 2, podPhase:         []v1.PodPhase{v1.PodRunning, v1.PodRunning, v1.PodRunning}, resource: &resourceInfo{ name:     v1.ResourceCPU, requests: []resource.Quantity{resource.MustParse("1.0"), resource.MustParse("1.0"), resource.MustParse("1.0")}, levels:   []int64{100, 100}, targetUtilization:   50, expectedUtilization: 10, expectedValue:       numContainersPerPod * 100, }, } tc.runTest(t) } </ins> // TestComputedToleranceAlgImplementation is a regression test which // back-calculates a minimal percentage for downscaling based on a small percentage // increase in pod utilization which is calibrated against the tolerance value.

What happened: The bug described in is still present in 1.14.8 When a change is made to the running deployment that requires a rolling update to be executed, both the current replicaset and the new replicaset is scaled out to max number of pods regardless of actual load. What you expected to happen: The bug to be fixed How to reproduce it (as minimally and precisely as possible): Follow this guide The cluster does NOT have to be an EKS cluster, the bug is not related to the underlying platform or infrastructure. Anything else we need to know?: This is bug is really critical, and should be addressed and hotfixed as soon as possible. Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-08-19T11:13:54Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"windows/amd64"} Server Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.8", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-10-15T12:02:12Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: AWS EC2 OS (e.g: ): Container Linux by CoreOS 2247.5.0 (Rhyolite) Kernel (e.g. ): 4.19.78-coreos Install tools: kube-aws v.0.14.2 ()
This bug does only apply to the and API version, not the API.
same issue i think it was really critical iuuse so hope hot fixed asap
/sig autoscaling
I've been looking into this and have made some interesting observations about the behaviour within our systems at least: Firstly, during the update, we see even though is greater in size than the current replica count, and this continues to increase over time. This suggests the bug may be somewhere in here: is set to and so I believe is being increased (to greater than ) by the method. Secondly, we see the scale doubling (approximately) during the rolling update.

The observation I've made it that it seems to be getting updated to the total count of the old and new replicasets pods combined, as if it is using the number of pods that match the label matcher for the deployment as the desired count when it finally updates the scale. I have yet to find a piece of code that verifies this though, merely an observation. Hopefully this helps narrow down the bug, it may also be a wild goose chase I can do more testing if anyone has any ideas, just let me know
Just for information, I am seeing similar behaviour using a custom metric, not CPU - I am using . In our dev environment the config is of 1, of 3. Even when there are no requests coming in, hpa is scaling to 3 or 4 pods then cooling back down to 3 pods after 5 minutes.
CPU Utilization of Pod-1 is . CPU Utilization of Pod-2 is . So CPU Utilization of Deployment is . So calculated number of replicas is . But is , so as a result, new number of replicas is . New Pod-3 is created due to RollingUpdate with and . CPU Utilization of Pod-1 is . CPU Utilization of Pod-2 is . CPU Utilization of Pod-3 is because metrics-server can't provide metrics of freshly maded Pod-3 and hpa-controller treats pod's resource utilization as when deployment tend to scale in. So CPU Utilization of Deployment is . So new number of replicas is in spite of .
I'd like to fix this issue in .
as workaround, can we just set to solve it?
, just to make sure this is not forgotten Specifying also results in an uncontrolled roll out. Please verify that the fix in apply to the and scenario as well. Or is your about just a typo? Your PR actually specifies , which is what I reported in this bug.
, if I'm not mistaken this bug is related to , not . So specifying won't help. Edit: There is actually no such thing as a , which wouldn't make any logical sense either.
Sorry, I meant but copy pasted the incorrect text. I think is simply a typo. So, as workaround, can we just set to solve it?
Unfortunately is really not a good workaround.

Say you have a deployment with 200 pods, would slow down the rolling update dramatically, as only one new pod could be created at a time, instead of all 200 at once. If the startup time for your pod is 10 seconds, replacing all 200 pods would take 10 seconds with , but 2000 seconds (33.3 minutes) with And according to the documentation, setting to anything except for 0 will reduce the number of pods (basically reducing capacity) during a rolling update. We cannot tolerate any reduced capacity during a rolling update, as the service powered by the pods might get saturated with incoming customer requests. Example: when this is set to 30%, the old ReplicaSet can be scaled down to 70% of desired pods immediately when the rolling update starts. Once new pods are ready, old ReplicaSet can be scaled down further, followed by scaling up the new ReplicaSet, ensuring that the total number of pods available at all times during the update is at least 70% of desired pods. My understanding is that the setting is to ensure that your cluster (worker nodes) are not saturated during a rolling update. In our case we do have enough free capacity in the cluster to always be able to immediately replace all pods for any running service.
I'm sorry. is typo. is correct.
Unfortunately no. When I try scenario described in with , HPA scale deployment regardless of current load. become depends on timing.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
What is the status of this issue? I still believe it's a critical bug that needs to be addressed.
+1
We may be running into the same or a problem that may be connected to this. We run a deployment with The HPA, which targets the deployment, is set to scale on a custom metric which is scraped every 30 seconds (as the value is a calculated as a it takes 60 seconds for the first value to appear).

When the rolling update runs, even with completely idle deployment at 1 replicas, the HPA starts to add replicas to the deployment until it either hits the max replicas (which is 8 for this deployment) or until the first Pod generates a metric. The in method works as intended. It uses 100% of the target value for pods missing metrics, so, with 2 replicas running, one at 100% and the other one at 1% it still returns 2 replicas as the desired replicas count. I believe the problem is related to how the current number of replicas is taken from the deployment spec during HPA reconciliation (). During a rolling update, we saw the deployment spec and the deployment status being out of sync for long periods of time (over 3 minutes). This does not always happen, but it happens most of the times we do a rolling update. The HPA status is also confusing, as it indicates that the metric is above target when in fact is well below target: Should the HPA reconciliation use the current number of replicas from the status rather that from the spec?
thanks for the report. I am able to reproduce this on 1.14.10 with your test yaml. I will find the root cause and come back to you shortly.
/assign josephburnett
This issue is reproduced in my environment (v1.15.9) too, and it's resolved by replacing the metric type from to . I suspect the formula calculating the average is wrong.
I've cherry picked on top of 1.14.8 and it indeed fixes the bug. I've approved the PR and requested to fix it in as well.
, thx a lot! So in the next release of 1.14, 1.15, 1.16 and 1.17 this bug will be fixed?
Unless I'm mistaken this won't be cherry-picked back to 1.14 as it's fallen out of support as of the GA release of 1.17 as per . At best it'll be getting back to 1.15.
I'll work on backporting this to 1.15 - 1.18.
Cherry pick process (for reference): 1:15: 16: 17: 18:
This bug is affecting us badly , Is this fix merged in 1.16.x release ?

if not is there a temporary solution to fix this issue
This fix is available from 1.16.9.


Question: } // probeConnectivity execs into a pod and checks its connectivity to another pod.. <del> func (k *kubeManager) probeConnectivity(nsFrom string, podFrom string, containerFrom string, addrTo string, protocol v1.Protocol, toPort int) (bool, string, error) { </del> <ins> func (k *kubeManager) probeConnectivity(nsFrom string, podFrom string, containerFrom string, addrTo string, protocol v1.Protocol, toPort int, timeoutSeconds int) (bool, string, error) { </ins> port := strconv.Itoa(toPort) var cmd []string <ins> timeout := fmt.Sprintf("--timeout=%vs", timeoutSeconds) </ins> switch protocol { case v1.ProtocolSCTP: <del> cmd = []string{"/agnhost", "connect", net.JoinHostPort(addrTo, port), "--timeout=1s", "--protocol=sctp"} </del> <ins> cmd = []string{"/agnhost", "connect", net.JoinHostPort(addrTo, port), timeout, "--protocol=sctp"} </ins> case v1.ProtocolTCP: <del> cmd = []string{"/agnhost", "connect", net.JoinHostPort(addrTo, port), "--timeout=1s", "--protocol=tcp"} </del> <ins> cmd = []string{"/agnhost", "connect", net.JoinHostPort(addrTo, port), timeout, "--protocol=tcp"} </ins> case v1.ProtocolUDP: <del> cmd = []string{"/agnhost", "connect", net.JoinHostPort(addrTo, port), "--timeout=1s", "--protocol=udp"} </del> <ins> cmd = []string{"/agnhost", "connect", net.JoinHostPort(addrTo, port), timeout, "--protocol=udp"} if framework.NodeOSDistroIs("windows") { framework.Logf("probing UDP for windows may result in cluster instability for certain windows nodes with low CPU/Memory, depending on CRI version") } </ins> default: framework.Failf("protocol %s not supported", protocol) }

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Ive done preliminary testing with on calico powered clusters on some windows variants and found that the probing seems to work. Can we add wrapper to netpol suite so that we can run it on windows clusters ? Why is this needed: We need this because otherwise, CNIs such as EKS will give us This error is caused when, for example, pods that are landing on a windows node havent defined their target platform. Somehow i guess pods which are landing on linux nodes dont have this issue. Its also needed for the obvious reason that we probably want separate linux and windows test suites for policies, in general, even though we expect the logic to be identical (modulo hostports , which we dont currently test)
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig windows
/assign
grepping around, I can't figure out where those are supposed to go. Can you give me pointers?
(resolved in slack, matts digging into this)
ok, these are running now on eks, lots of failures. will dig
/assign
the netpol are not being applied for the ns: Instead of failing because should not be able to access , the connection succeeds.


Question: options.Limit = 0 options.Continue = "" options.ResourceVersion = requestedResourceVersion <ins> options.ResourceVersionMatch = requestedResourceVersionMatch </ins> result, err := p.PageFn(ctx, options) return result, paginatedResult, err }

We recently started using the list pager from client-go as part of Calico, and have seen reports of the following error showing up: No error, just beautiful paged list calls. Create 1,000 resources of a particular type, such as a NetworkPolicy. (Need 500 to trigger paging) Do a of all those resources with set. Originally filed here (but maybe that repo is not monitored): Reference issue in Calico: projectcalico/calico (comment) Reference Calico list pager code: We think this is the required fix: <detailsclient-go version (but bug seems to be in master too). </details<details</details<details</details<details</details<details</details<details</details
/sig api-machinery
/triage accepted


Question: package glusterfs import ( <del> "math/rand" </del> "os" <ins> "path" </ins> "github.com/golang/glog" "k8s.io/kubernetes/pkg/api"

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: pull-kubernetes-e2e-aks-engine-azure Which test(s) are failing: Since when has it been failing: Testgrid link: No Testgrid, but job history: Reason for failure: Anything else we need to know: ref:
/assign cc: /milestone v1.16 /priority important-soon
/sig azure
cc
Related with
Update: has been merged, but a new release with this change is not available yet. Pending on aks-engine's next release.
/sig cloud-provider
v1.16.0-alpha.1 is available in
Will submit a PR after testing this in our own prow first.


Question: useMetricsAPI:   true, recommendations: []timestampedRecommendation{}, } <ins> tc1.runTest(t) </ins> <del> tc.runTest(t) // Reuse the data structure above, now testing "unscaling". // Now, we test that no scaling happens if we are in a very close margin to the tolerance </del> target = math.Abs(1/(requestedToUsed*(1-defaultTestingTolerance))) + .004 finalCPUPercentTarget = int32(target * 100) <del> tc.CPUTarget = finalCPUPercentTarget tc.initialReplicas = startPods tc.expectedDesiredReplicas = startPods tc.expectedConditions = statusOkWithOverrides(autoscalingv2.HorizontalPodAutoscalerCondition{ Type:   autoscalingv2.AbleToScale, Status: v1.ConditionTrue, Reason: "ReadyForNewScale", }) tc.runTest(t) </del> <ins> tc2 := testCase{ minReplicas:             0, maxReplicas:             1000, initialReplicas:         startPods, expectedDesiredReplicas: startPods, CPUTarget:               finalCPUPercentTarget, reportedLevels: []uint64{ totalUsedCPUOfAllPods / 10, totalUsedCPUOfAllPods / 10, totalUsedCPUOfAllPods / 10, totalUsedCPUOfAllPods / 10, totalUsedCPUOfAllPods / 10, totalUsedCPUOfAllPods / 10, totalUsedCPUOfAllPods / 10, totalUsedCPUOfAllPods / 10, totalUsedCPUOfAllPods / 10, totalUsedCPUOfAllPods / 10, }, reportedCPURequests: []resource.Quantity{ resource.MustParse(fmt.Sprint(perPodRequested+100) + "m"), resource.MustParse(fmt.Sprint(perPodRequested-100) + "m"), resource.MustParse(fmt.Sprint(perPodRequested+10) + "m"), resource.MustParse(fmt.Sprint(perPodRequested-10) + "m"), resource.MustParse(fmt.Sprint(perPodRequested+2) + "m"), resource.MustParse(fmt.Sprint(perPodRequested-2) + "m"), resource.MustParse(fmt.Sprint(perPodRequested+1) + "m"), resource.MustParse(fmt.Sprint(perPodRequested-1) + "m"), resource.MustParse(fmt.Sprint(perPodRequested) + "m"), resource.MustParse(fmt.Sprint(perPodRequested) + "m"), }, useMetricsAPI:   true, recommendations: []timestampedRecommendation{}, expectedConditions: statusOkWithOverrides(autoscalingv2.HorizontalPodAutoscalerCondition{ Type:   autoscalingv2.AbleToScale, Status: v1.ConditionTrue, Reason: "ReadyForNewScale", }), } tc2.runTest(t) </ins> } func TestScaleUpRCImmediately(t *testing.T) {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: Resync period for HorizontalPodAutoscaler controller is set to 15 seconds in order to update HPAs every 15 seconds. Sometimes updates are skipped and period between two updates is 30 seconds. What you expected to happen: HPA should be updated every 15 seconds. How to reproduce it (as minimally and precisely as possible): Add debugs to HPA controller showing how often HPAs are processed. Create HPA. Anything else we need to know?: HPA controller uses rate RateLimitingQueue: New requests are always with 15 seconds delay: If request for the given HPA is in the queue then newly request is skipped. Requests are and removed from the queue at the same moment so there is a race condition. See following scenario: 00:00:00 - Resync adds request to queue to be executed at 00:00:15 00:00:15 - Resync adds next request to queue to be executed at 00:00:30 but it is skipped because previous request is still in the queue 00:00:15 - Request at 00:00:00 is executed 00:00:30 - Resync adds next request to queue to be executed at 00:00:45 00:00:45 - Request at 00:00:30 is executed In above scenario HPA wasn't updated between 00:00:15 and 00:00:45. Environment: Kubernetes version (use ): HEAD Cloud provider or hardware configuration: GKE <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
/sig autoscaling
/assign krzysztof-jastrzebski


Question: restartCommand, []string{kubeletHealthCheckURL}, "kubelet.log", <ins> e.monitorParent, </ins> true /* restartOnExit */) return server, server.start() } <del> // A server manages a separate server process started and killed with // commands. type server struct { // name is the name of the server, it is only used for logging. name string // startCommand is the command used to start the server startCommand *exec.Cmd // killCommand is the command used to stop the server. It is not required. If it // is not specified, `sudo kill` will be used to stop the server. killCommand *exec.Cmd // restartCommand is the command used to restart the server. If provided, it will be used // instead of startCommand when restarting the server. restartCommand *exec.Cmd // healthCheckUrls is the urls used to check whether the server is ready. healthCheckUrls []string // outFilename is the name of the log file. The stdout and stderr of the server // will be redirected to this file. outFilename string // restartOnExit determines whether a restart loop is launched with the server restartOnExit bool // Writing to this channel, if it is not nil, stops the restart loop. // When tearing down a server, you should check for this channel and write to it if it exists. stopRestartingCh chan<- bool // Read from this to confirm that the restart loop has stopped. ackStopRestartingCh <-chan bool } // newServer returns a new server with the given name, commands, health check // URLs, etc. func newServer(name string, start, kill, restart *exec.Cmd, urls []string, outputFileName string, restartOnExit bool) *server { return &server{ name:            name, startCommand:    start, killCommand:     kill, restartCommand:  restart, healthCheckUrls: urls, outFilename:     outputFileName, restartOnExit:   restartOnExit, } } // commandToString format command to string. func commandToString(c *exec.Cmd) string { if c == nil { return "" } return strings.Join(append([]string{c.Path}, c.Args[1:]...), " ") } func (s *server) String() string { return fmt.Sprintf("server %q start-command: `%s`, kill-command: `%s`, restart-command: `%s`, health-check: %v, output-file: %q", s.name, commandToString(s.startCommand), commandToString(s.killCommand), commandToString(s.restartCommand), s.healthCheckUrls, s.outFilename) </del> <ins> func adjustConfigForSystemd(config string) string { return strings.Replace(config, "%", "%%", -1) </ins> } <del> // start starts the server by running its commands, monitors it with a health // check, and ensures that it is restarted if applicable. // // Note: restartOnExit == true requires len(s.healthCheckUrls) > 0 to work properly. func (s *server) start() error { glog.Infof("Starting server %q with command %q", s.name, commandToString(s.startCommand)) errCh := make(chan error) // Set up restart channels if the server is configured for restart on exit. var stopRestartingCh, ackStopRestartingCh chan bool if s.restartOnExit { if len(s.healthCheckUrls) == 0 { return fmt.Errorf("Tried to start %s which has s.restartOnExit == true, but no health check urls provided.", s) } stopRestartingCh = make(chan bool) ackStopRestartingCh = make(chan bool) s.stopRestartingCh = stopRestartingCh s.ackStopRestartingCh = ackStopRestartingCh </del> <ins> // getLogFiles gets logs of interest either via journalctl or by creating sym // links. Since we scp files from the remote directory, symlinks will be // treated as normal files and file contents will be copied over. func (e *E2EServices) getLogFiles() { // Nothing to do if report dir is not specified. if framework.TestContext.ReportDir == "" { return </ins> } <del> // This goroutine actually runs the start command for the server. go func() { defer close(errCh) // Create the output filename outPath := path.Join(framework.TestContext.ReportDir, s.outFilename) outfile, err := os.Create(outPath) if err != nil { errCh <- fmt.Errorf("failed to create file %q for `%s` %v.", outPath, s, err) return } else { glog.Infof("Output file for server %q: %v", s.name, outfile.Name()) } defer outfile.Close() defer outfile.Sync() // Set the command to write the output file s.startCommand.Stdout = outfile s.startCommand.Stderr = outfile // Death of this test process should kill the server as well. attrs := &syscall.SysProcAttr{} // Hack to set linux-only field without build tags. deathSigField := reflect.ValueOf(attrs).Elem().FieldByName("Pdeathsig") if deathSigField.IsValid() { deathSigField.Set(reflect.ValueOf(syscall.SIGTERM)) } else { errCh <- fmt.Errorf("failed to set Pdeathsig field (non-linux build)") return } s.startCommand.SysProcAttr = attrs // Start the command err = s.startCommand.Start() if err != nil { errCh <- fmt.Errorf("failed to run %s: %v", s, err) return } if !s.restartOnExit { glog.Infof("Waiting for server %q start command to complete", s.name) // If we aren't planning on restarting, ok to Wait() here to release resources. // Otherwise, we Wait() in the restart loop. err = s.startCommand.Wait() if err != nil { errCh <- fmt.Errorf("failed to run start command for server %q: %v", s.name, err) return </del> <ins> glog.Info("Fetching log files...") journaldFound := isJournaldAvailable() for targetFileName, logFileData := range e.logFiles { targetLink := path.Join(framework.TestContext.ReportDir, targetFileName) if journaldFound { // Skip log files that do not have an equivalent in journald-based machines. if len(logFileData.journalctlCommand) == 0 { continue </ins> } <del> } else { usedStartCmd := true for { glog.Infof("Running health check for service %q", s.name) // Wait for an initial health check to pass, so that we are sure the server started. err := readinessCheck(s.name, s.healthCheckUrls, nil) if err != nil { if usedStartCmd { glog.Infof("Waiting for server %q start command to complete after initial health check failed", s.name) s.startCommand.Wait() // Release resources if necessary. } // This should not happen, immediately stop the e2eService process. glog.Fatalf("restart loop readinessCheck failed for %s", s) } else { glog.Infof("Initial health check passed for service %q", s.name) } // Initial health check passed, wait until a health check fails again. stillAlive: for { select { case <-stopRestartingCh: ackStopRestartingCh <- true return case <-time.After(time.Second): for _, url := range s.healthCheckUrls { resp, err := http.Head(url) if err != nil || resp.StatusCode != http.StatusOK { break stillAlive } } } } if usedStartCmd { s.startCommand.Wait() // Release resources from last cmd usedStartCmd = false } if s.restartCommand != nil { // Always make a fresh copy of restartCommand before // running, we may have to restart multiple times s.restartCommand = &exec.Cmd{ Path:        s.restartCommand.Path, Args:        s.restartCommand.Args, Env:         s.restartCommand.Env, Dir:         s.restartCommand.Dir, Stdin:       s.restartCommand.Stdin, Stdout:      s.restartCommand.Stdout, Stderr:      s.restartCommand.Stderr, ExtraFiles:  s.restartCommand.ExtraFiles, SysProcAttr: s.restartCommand.SysProcAttr, } // Run and wait for exit. This command is assumed to have // short duration, e.g. systemctl restart glog.Infof("Restarting server %q with restart command", s.name) err = s.restartCommand.Run() if err != nil { // This should not happen, immediately stop the e2eService process. glog.Fatalf("restarting server %s with restartCommand failed. Error: %v.", s, err) } } else { s.startCommand = &exec.Cmd{ Path:        s.startCommand.Path, Args:        s.startCommand.Args, Env:         s.startCommand.Env, Dir:         s.startCommand.Dir, Stdin:       s.startCommand.Stdin, Stdout:      s.startCommand.Stdout, Stderr:      s.startCommand.Stderr, ExtraFiles:  s.startCommand.ExtraFiles, SysProcAttr: s.startCommand.SysProcAttr, } glog.Infof("Restarting server %q with start command", s.name) err = s.startCommand.Start() usedStartCmd = true if err != nil { // This should not happen, immediately stop the e2eService process. glog.Fatalf("Restarting %s with startCommand failed. Error: %v.", s, err) } </del> <ins> glog.Infof("Get log file %q with journalctl command %v.", targetFileName, logFileData.journalctlCommand) out, err := exec.Command("sudo", append([]string{"journalctl"}, logFileData.journalctlCommand...)...).CombinedOutput() if err != nil { glog.Errorf("failed to get %q from journald: %v, %v", targetFileName, string(out), err) } else { if err = ioutil.WriteFile(targetLink, out, 0644); err != nil { glog.Errorf("failed to write logs to %q: %v", targetLink, err) </ins> } } <del> } }() return readinessCheck(s.name, s.healthCheckUrls, errCh) } // kill runs the server's kill command. func (s *server) kill() error { glog.Infof("Kill server %q", s.name) name := s.name cmd := s.startCommand // If s has a restart loop, turn it off. if s.restartOnExit { s.stopRestartingCh <- true <-s.ackStopRestartingCh } if s.killCommand != nil { return s.killCommand.Run() } if cmd == nil { return fmt.Errorf("could not kill %q because both `killCommand` and `startCommand` are nil", name) } if cmd.Process == nil { glog.V(2).Infof("%q not running", name) return nil } pid := cmd.Process.Pid if pid <= 1 { return fmt.Errorf("invalid PID %d for %q", pid, name) } // Attempt to shut down the process in a friendly manner before forcing it. waitChan := make(chan error) go func() { _, err := cmd.Process.Wait() waitChan <- err close(waitChan) }() const timeout = 10 * time.Second for _, signal := range []string{"-TERM", "-KILL"} { glog.V(2).Infof("Killing process %d (%s) with %s", pid, name, signal) cmd := exec.Command("sudo", "kill", signal, strconv.Itoa(pid)) // Run the 'kill' command in a separate process group so sudo doesn't ignore it attrs := &syscall.SysProcAttr{} // Hack to set unix-only field without build tags. setpgidField := reflect.ValueOf(attrs).Elem().FieldByName("Setpgid") if setpgidField.IsValid() { setpgidField.Set(reflect.ValueOf(true)) } else { return fmt.Errorf("Failed to set Setpgid field (non-unix build)") } cmd.SysProcAttr = attrs _, err := cmd.Output() if err != nil { glog.Errorf("Error signaling process %d (%s) with %s: %v", pid, name, signal, err) </del> continue } <del> select { case err := <-waitChan: if err != nil { return fmt.Errorf("error stopping %q: %v", name, err) </del> <ins> for _, file := range logFileData.files { if _, err := os.Stat(file); err != nil { // Expected file not found on this distro. continue } if err := copyLogFile(file, targetLink); err != nil { glog.Error(err) } else { break </ins> } <del> // Success! return nil case <-time.After(timeout): // Continue. </del> } } <del> return fmt.Errorf("unable to stop %q", name) </del> } <del> func adjustConfigForSystemd(config string) string { return strings.Replace(config, "%", "%%", -1) </del> <ins> // isJournaldAvailable returns whether the system executing the tests uses // journald. func isJournaldAvailable() bool { _, err := exec.LookPath("journalctl") return err == nil </ins> } func copyLogFile(src, target string) error {

In node e2e, we have the option to keep the services running after test for further debug. However, we set to for the services, which means that once the parent exits, they will receive and exit with the parent if they don't trap the signal. With this, option will never work. If we still need this feature, we should not set if is false. /cc
We still need this feature. It helps with debugging test failures by keeping all the daemons running upon failure. , Lantao Liu wrote:


Question: if err == nil { t.Errorf("%s: Expected failure", protoStr) } <ins> if string(buffer.Bytes()) != stderrOutput { t.Errorf("%s: Expected output '%s', got '%v'", protoStr, stderrOutput, string(buffer.Bytes())) } </ins> } func TestSaveIntoIPv4(t *testing.T) {

On some systems, iptables-save outputs the following warning to stderr: Note that the error message includes a newline. Because kube-proxy captures both stdout and stderr when calling , the newline might cause rules to be garbled. For example, the output might contain the following line: The parser treats this as a chain line, and panics, as predicted in
/sig network
FYI
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/assign
Just FYI here's another example of garbled output:
Anyway, fix is in Please review.
/assign
/remove-triage unresolved


Question: // The configuration of NodeKiller. NodeKiller NodeKillerConfig <ins> // The Default IP Family of the cluster ("ipv4" or "ipv6") IPFamily string </ins> } // NodeKillerConfig describes configuration of NodeKiller -- a utility to

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: https://k8s-(IPv6),%20master%20(dev) Which test(s) are failing: [sig-scheduling] SchedulerPredicates [Serial] validates that there is no conflict between pods with same hostPort but different hostIP and protocol [Conformance] Since when has it been failing: Testgrid link: https://k8s-(IPv6),%20master%20(dev) Reason for failure: The tests checks for 0.0.0.0 that is IPv4 only Anything else we need to know: Probably I should check with SIG-Testing how can we add an ipv6 job and update the KEP accordingly EDIT Probably I should check with SIG-Testing how can we add an ipv6 job as part of the k/k presubmits, and update the KEP accordingly
/assign /cc
/sig scheduling

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Introduce a mechanism that allows specifying the IP protocol version used in the e2e testing. Why is this needed: This is a followup on , my understanding is that in order to discriminate between IPv4 or IPv6 we have to use the test labels and this adds complexity to the testing automation and doesn't allow to send this information to the context of the tests. Having a mechanism to specify the IP protocol will simplify the testing for IPv4 and IPv6 and will allow reusing most of the code.
/sig testing /cc /assign Does this make sense? If affirmative, what should be the best way to add this functionality? Checking the code I think that or can provide this, don't know if checking some of the cluster config parameters, ...
the testing-commons group (sig-testing) maintains the framework, but ideally the decision on how to proceed in terms of adding ipv6 detection should come from sig-network. possibly having this as agenda for a sig-network meeting is a good start, then discussing it in a testing-commons meeting. /sig network the linked issue is a big one, so i don't have all the context here. but IMO existing tests should just work for ipv6 without the requirement for a new "IPV6" tag in test names. dual-stack or protocol specific tests should have their descriptions contain the details properly (i.e. what is this test doing) we might want to add an utility function under: (there is a PR to move it BTW ) then such an utility might be useful only to tests that need to detect the protocol. one of the goals is to refactor the code in testcontext / framework, so adding this logic there is less preferred.
/area e2e-test-framework
I think that we can add an option to test_context to specify the cluster IP family the same way that exists one for the cloud provider this is simpler because when setting up the CI jobs you will know beforehand what IP family you want to test, it also will allow reusing all the e2e code for IPv6 and Dual Stack (cc: and by default will be IPv4, so it´s backward compatible.
Being able to override and specify exactly what you want the test to beleive about your cluster is nice, but ideally it should generally work without specifying more flags.

Surely we can detect this?
I was thinking during some time about this, detecting the IP family is very simple having IPv4 or IPv6 only clusters, but with the introduction of dual stack I´m afraid about the supportability and reliability of the autodetecting mechanisms in the long term. If you see the dual stack PRs there are still a lot of unknowns about the cluster behavior. My point is, let´s have something first stable that works and is simple, and once dual stack is implemented, we can replace the flag with a function in the framework . I guess that migrating from a flag variable to a function is straight forward, is just replacing by , what do you think?
in is as part of the context. do you mean adding as a function which will auto-detecting the ip family. less flags for the framework binary is preferred.
once we have this I will open a new issue to start cleaning up and , we no longer need to discriminate based on tags


Question: - list - watch - apiGroups: <ins> - discovery.k8s.io resources: - endpointslices verbs: - get - list - watch - apiGroups: </ins> - apps resources: - controllerrevisions

Users are not able to view by default as part of the or roles like they are with Users with or roles should be able to view like they are with Have a user with permissions on a namespace. Try to run the command I asked about this in slack and was told that it was an oversight that it wasn't to the default roles. Kubernetes version (use ): 1.21 Cloud provider or hardware configuration: n/a OS (e.g: ): n/a Kernel (e.g. ): n/a Install tools: n/a Network plugin and version (if this is a network-related bug): n/a Others: n/a
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig security
/sig auth


Question: // calculatePodResourceRequest returns the total non-zero requests. If Overhead is defined for the pod and the // PodOverhead feature is enabled, the Overhead is added to the result. <ins> // podResourceRequest = max(sum(podSpec.Containers), podSpec.InitContainers) + overHead </ins> func calculatePodResourceRequest(pod *v1.Pod, resource v1.ResourceName) int64 { var podRequest int64 for i := range pod.Spec.Containers {

The scheduler does not properly calculate pod requests for a resource as - it ignores init containers completely in . This would allow a workload to get "free" scheduling above the limit of containers (mitigated by quota, which calculates this properly) and so the scheduler would make non-optimal fits. Both kubelet and scheduler enforce the , so you can't actually schedule, but you sometimes get steered to the wrong node if you have more init container resource usage than containers. in should use and the similar helpers instead of writing its own method to calculate the request for resources. /sig scheduling
/priority important-soon
Let me follow this issue. /assign
fyi
After I checked the implementation of function, I think we just need to use it to instead of the function in the scheduler.
/priority critical-urgent
Is the pod phase not considered? When the pod phase is running, the requested resource is actually considered as . This change seems to reduce the resource utilization of the node.
Actually, I don't think we need to consider the pod.Phase. The resource request & limit are mainly used in the scheduling process. The scenario you said is not belong to the responsibility of kube-scheduler
Correct. Setting appropriate requests/limits that maintain good node usage is users responsibility. Considering the max(initContainers, sum(containers)) is the documented behavior.
I understand. Thank you for your comments!


Question: KUBE_DELETE_NETWORK=${KUBE_DELETE_NETWORK:-true} MASTER_OS_DISTRIBUTION=${KUBE_MASTER_OS_DISTRIBUTION:-${KUBE_OS_DISTRIBUTION:-gci}} <del> NODE_OS_DISTRIBUTION=${KUBE_NODE_OS_DISTRIBUTION:-${KUBE_OS_DISTRIBUTION:-debian}} </del> <ins> NODE_OS_DISTRIBUTION=${KUBE_NODE_OS_DISTRIBUTION:-${KUBE_OS_DISTRIBUTION:-gci}} </ins> if [[ "${MASTER_OS_DISTRIBUTION}" == "coreos" ]]; then MASTER_OS_DISTRIBUTION="container-linux" fi

The debian-based CVM image is going to be deprecated in September. We should: Change the default OS image for GCE cluster setup script. This means that all the GCE CI builds will use the new default OS image. Add deprecation warning to the cluster setup script if CVM is specified. Maintain the node e2e test coverage for CVM for 1.8. /cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
The CVM tests were turned off a while ago.