Question: el     int }{ {[]kubecontainer.Image{}, 0}, <del> {buildByImageSize(), 3}, </del> <ins> {buildByImageSize(), 4}, </ins> {nil, 0}, }

What happened: When two or more images on a given node have the same size, their order inside the field set by the process is not deterministic. This problem has the side effect of continually triggering controllers which assume this list to be always equal on node updates unless images are or removed. What you expected to happen: The order of images should be stable even when two images have the same size, using the image ID as a second sort field would help this. How to reproduce it (as minimally and precisely as possible): Choose a node and run: Run a few times: It should be possible to see and images alternating on the output. Anything else we need to know?: Our particular use case make it really common for multiple images to have the same size on a given node. I came across this problem while debugging slow updates to DaemonSets and finding out that the kube-controller-manager was leaking memory until it eventually dies due to OOM: <img width="800" alt="Screen Shot 2019-06-30 at 20 27 22" src="https://user-"I was able to track this leak to a large number of watched elements in memory and by taking a look at the logs with increased verbosity I figured it was due to too many Node notifications being processed by the DaemonSet controller: <img width="800" alt="Screen Shot 2019-06-30 at 20 36 23" src="https://user-"It's possible that other controllers in the wild that watch nodes and only ignore the conditions timestamps on updates to face the same problem. I'll send a simple PR to fix this shortly. Environment: Kubernetes version (use ): But by taking a look at the code this problem also happens on master.
/sig node
Nice debugging here :)


Question: ETCDCTL=$(which etcdctl) KUBECTL="${KUBE_OUTPUT_HOSTBIN}/kubectl" UPDATE_ETCD_OBJECTS_SCRIPT="${KUBE_ROOT}/cluster/update-storage-objects.sh" <del> DISABLE_ADMISSION_PLUGINS="ServiceAccount,NamespaceLifecycle,LimitRanger,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,PersistentVolumeLabel,DefaultStorageClass" </del> <ins> DISABLE_ADMISSION_PLUGINS="ServiceAccount,NamespaceLifecycle,LimitRanger,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,PersistentVolumeLabel,DefaultStorageClass,StorageObjectInUseProtection" </ins> function startApiServer() { local storage_versions=${1:-""}

What happened: The preparation step of the e2e tests were failed sometimes on my local environment like What you expected to happen: These tests can be passed without any errors. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): v1.13.1 Cloud provider or hardware configuration: OpenStack OS (e.g. from /etc/os-release): 16.04.4 LTS (Xenial Xerus) Kernel (e.g. ): 4.4.0-138-generic Install tools: kubeadm Others:
/sig storage
After discussing on the PR I have known I need to enable StorageObjectInUseProtection plugin of admission controller which is disabled by default. The corresponding e2e tests require the plugin is enabled, so I think it is better to enable the plugin by default for most users.
Investigation of StorageObjectInUseProtection plugin of admission controller under kubernetes/kubernetes: cluster/centos/config- Enabled cluster/gce/config- Enabled cluster/gce/config- Enabled cluster/kubemark/gce/config- Enabled hack/local-up- Enabled cluster/juju/layers/kubernetes- Disabled This is the reason why we don't face this issue on the upstream CI system, and I it on my local environment which is deployed with kubeadm.


Question: }() }() <del> postTimeoutFn() </del> <ins> defer postTimeoutFn() </ins> tw.timeout(err) } }

Unit test in package flakes there is a race condition between the go routine that executes the request handler and the timeout go routine that waits for the handler to finish /sig api-machinery
/assign
I have a PR in the works
should fix it, I can easily reproduce it with the PR, the flake is not reproducible any more cc
/triage accepted


Question: appsv1 "k8s.io/api/apps/v1" batchv1 "k8s.io/api/batch/v1" corev1 "k8s.io/api/core/v1" <ins> apierrors "k8s.io/apimachinery/pkg/api/errors" </ins> "k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/runtime/schema" "k8s.io/apiserver/pkg/admission"

Address the TODO here:
I'm interested in knocking this out, but am not entirely sure where this would be propagated. Seems related to / ?


Question: credentialprovider.RegisterCredentialProvider("google-dockercfg-url", &credentialprovider.CachingDockerConfigProvider{ Provider: &dockerConfigUrlKeyProvider{ <del> metadataProvider{Client: http.DefaultClient}, </del> <ins> metadataProvider{Client: &http.Client{Timeout: 10 * time.Second}}, </ins> }, Lifetime: 60 * time.Second, })

http://pr-
Something holding the port already?
The previous masterless case just barely passed with a 12.5 second timeout (25 tries at 0.5 second interval) I think we just need to increase the timeout.
I randomly sampled 5 green builds and they all have this test (starting kubelet in masterful/less mode) finished in 3s or less. Judging for the successful masterless mode startup before the failure, kubelet spent the most time trying to register credential providers (a 10s gap in the log). The same roughly applies to the failed "masterful mode", based on where the log message stopped. From , the registering process basically tries to access the metadata url to see if the credential provider is enabled. I think this may just be the slow response from the metadata server.
I am wondering even if this issue might be related to a slow response from the metadata server, why it does not show "Registering credential provider: .dockercfg" always in failed logs?
If there is a possibility that ReadUrl in may take longer than expected, would it be preferred to use a Timeout (0) with the http client to not get stuck longer than expected?
I am unsure why it didn't show that, but even in the passing round we see a 10 second gap, which almost maxed out the timeout. I think adding a timeout is reasonable, but I am not sure what value is reasonable given that the previous request took 10s. do you have any idea?
Even I am not sure exactly what should be a reasonable value. My guess would be that if in normal cases, average response time is x seconds (perhaps a guestimate could be computed based on "normal" successful tests), then x could be a starting point, or otherwise another option is choosing something randomly (like 5s). And I think that if at some point, the server is responding in 10s, and even though the test passes, its not normal and something is going wrong.
uses a 10 second timeout, but the request will be retried. Given that we don't retry this request, a short timeout is worrisome. Maybe we should add both retry and timeout in this case.
SGTM.
I reopen this one since pr was reverted.

Noticed a similar hang in: The logs are a bit different from those observed in
if kubelet couldn't start up properly, the test itself should've timed out and logged "Timed out waiting for kubelet to answer at http://127.0.0.1:/healthz; tried 25 waiting 0.5 between each". I am not sure if this failure is related.
failure is not relevant to this issue. The most recent report of such flake was over a week ago. I think the cause is the slow metadata url response in GCE, and that should've been fixed by the recent rollout. I am going to close this issue. Feel free to reopen if there is another occurrence.


Question: verifyActions(t, expected, cl.Actions()) } <ins> func TestCleanerExpiredAt(t *testing.T) { cleaner, cl, secrets, err := newTokenCleaner() if err != nil { t.Fatalf("error creating TokenCleaner: %v", err) } secret := newTokenSecret("tokenID", "tokenSecret") addSecretExpiration(secret, timeString(2*time.Second)) expected := []core.Action{} verifyFunc := func() { secrets.Informer().GetIndexer().Add(secret) cleaner.evalSecret(secret) verifyActions(t, expected, cl.Actions()) } // token has not expired currently verifyFunc() if cleaner.queue.Len() != 0 { t.Errorf("not using the queue, the length should be 0, now: %v", cleaner.queue.Len()) } var conditionFunc = func() (bool, error) { if cleaner.queue.Len() == 1 { return true, nil } return false, nil } err = wait.Poll(100*time.Millisecond, wait.ForeverTestTimeout, conditionFunc) if err != nil { t.Fatalf("secret is put back into the queue, the queue length should be 1, error: %vn", err) } // secret was eventually deleted expected = []core.Action{ core.NewDeleteAction( schema.GroupVersionResource{Version: "v1", Resource: "secrets"}, api.NamespaceSystem, secret.ObjectMeta.Name), } verifyFunc() } </ins>

What happened: kube-controller-manager with enabled option --controllers=,tokencleaner is not removing newly created tokens. It works only after kube-controller-manager restart or after editing secret with token. What you expected to happen: It should remove all expired tokens from kube-system namespace, without restarting or touching secrets. How to reproduce it (as minimally and precisely as possible): enable tokencleaner in kube-controller-manager start controller manager create bootstrap token, you can use script wait 5 minutes token is still there restart kube-controller-manager OR edit newly created token token should be gone after a while Anything else we need to know?: It could be related with Environment: Kubernetes version (use ): Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-12-03T20:56:12Z", GoVersion:"go1.11.2", Compiler:"gc", Platform:"linux/amd64"}
Do you see the following log ? thanks
no errors at all
/sig auth


Question: driverInfo DriverInfo } <del> var _ TestDriver = &hostpathSymlinkDriver{} var _ PreprovisionedVolumeTestDriver = &hostpathSymlinkDriver{} var _ InlineVolumeTestDriver = &hostpathSymlinkDriver{} </del> <ins> var _ TestDriver = &hostPathSymlinkDriver{} var _ PreprovisionedVolumeTestDriver = &hostPathSymlinkDriver{} var _ InlineVolumeTestDriver = &hostPathSymlinkDriver{} </ins> <del> // InitHostpathSymlinkDriver returns hostpathSymlinkDriver that implements TestDriver interface func InitHostpathSymlinkDriver() TestDriver { return &hostpathSymlinkDriver{ </del> <ins> // InitHostPathSymlinkDriver returns hostPathSymlinkDriver that implements TestDriver interface func InitHostPathSymlinkDriver() TestDriver { return &hostPathSymlinkDriver{ </ins> driverInfo: DriverInfo{ <del> Name:        "hostpathSymlink", </del> <ins> Name:        "hostPathSymlink", </ins> MaxFileSize: testpatterns.FileSizeMedium, SupportedFsType: sets.NewString( "", // Default fsType

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: https://k8s- What happened: The skip check for hostpath doesn't seem to be working for these test cases
can you take a look?
Oh, disruptive tests are the tests that can't be tested locally in my environment due to I will take a look.
I found that above issue is not related, I should have been checked more carefully. Sorry for that.
I found out that this was just a driver name issue, so below will fix it: hostpath -hostPath hostpathSymlink -hostPathSymlink We might be better changing the skip logic later, to make it skip not by name in order to avoid this kind of bug, though. I will make a PR for above change, soon.
I made a PR above. Also, it might be a bit out of scope of this fix, but are drivers tested with containerized volume server, like nfs, ceph, etc..., safe to be tested with subpath disruptive tests? I'm worrying about the case that test pod and containerized volume server are deployed on the same node. In such a case, the same behaviour expected or do we need to avoid this kind of situation?


Question: if mounter.(*nfsMounter).readOnly { t.Errorf("The volume source should not be read-only and it is.") } <ins> mntDevs, err := fake.List() if err != nil { t.Errorf("fakeMounter.List() failed: %v", err) } if len(mntDevs) != 1 { t.Errorf("unexpected number of mounted devices. expected: %v, got %v", 1, len(mntDevs)) } else { if mntDevs[0].Type != "nfs" { t.Errorf("unexpected type of mounted devices. expected: %v, got %v", "nfs", mntDevs[0].Type) } if mntDevs[0].Device != expectedDevice { t.Errorf("unexpected nfs device, expected %q, got: %q", expectedDevice, mntDevs[0].Device) } } </ins> log := fake.GetLog() if len(log) != 1 { t.Errorf("Mount was not called exactly one time. It was called %d times.", len(log))

In bare-metal IPv6 environment, any pod consuming storage resource is not able to run. The storage should be consumable, and the pod should become in status. Running the in a bare-metal IPv6 configurations. I have replaced the cloud-based storage resource with a mounted to a folder on my localhost. Steps alongside logs are below :- $kubectl get svc $kubectl describe svc nfs-server $kubectl describe pv nfs $kubectl get pod -w $kubectl get pod -w $kubectl describe pod nfs-busybox-6r96h I have built the master branch to make sure it was not fixed up to this moment.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage
/sig storage /tirage accepted /assign


Question: <ins> #!/bin/sh # Copyright 2015 The Kubernetes Authors. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # #     http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # Install prerequisites. apt-get update apt-get install -y -q --no-install-recommends  curl ca-certificates make g++ sudo bash # Install Fluentd. /usr/bin/curl -sSL https://toolbelt.treasuredata.com/sh/install-ubuntu-xenial-td-agent2.sh | sh # Change the default user and group to root. # Needed to allow access to /var/log/docker/... files. sed -i -e "s/USER=td-agent/USER=root/" -e "s/GROUP=td-agent/GROUP=root/" /etc/init.d/td-agent # Install the Elasticsearch Fluentd plug-in. # http://docs.fluentd.org/articles/plugin-management td-agent-gem install --no-document fluent-plugin-kubernetes_metadata_filter -v 0.24.0 td-agent-gem install --no-document fluent-plugin-elasticsearch -v 1.5.0 # Remove docs and postgres references rm -rf /opt/td-agent/embedded/share/doc  /opt/td-agent/embedded/share/gtk-doc  /opt/td-agent/embedded/lib/postgresql  /opt/td-agent/embedded/bin/postgres  /opt/td-agent/embedded/share/postgresql apt-get remove -y make g++ apt-get autoremove -y apt-get clean -y rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* </ins>

It's 400+mb and we need to pull it on every node. This leads to a lot of image pulling flake (), and also gets very expensive in some countries where bandwidth is sold at a premium. We took a stab at halving the size with: , but then realized that we were using an unsupported base image We tried to optimize to ~300Mb by building a smaller ubuntu:15 base, but that's still huge. Every other mandatory system daemon is built on busybox and so takes ~10Mb.
I can try again now that the agent supports
Newer Docker images using the spec from the official repository are based on AlpineLinux and are around ~38MB:


Question: fi } <del> # Verify and find the various tar files that we are going to use on the server. # # Vars set: #   SERVER_BINARY_TAR #   SALT_TAR function find-release-tars { SERVER_BINARY_TAR="${KUBE_ROOT}/server/kubernetes-server-linux-amd64.tar.gz" if [[ ! -f "$SERVER_BINARY_TAR" ]]; then SERVER_BINARY_TAR="${KUBE_ROOT}/_output/release-tars/kubernetes-server-linux-amd64.tar.gz" fi if [[ ! -f "$SERVER_BINARY_TAR" ]]; then echo "!!! Cannot find kubernetes-server-linux-amd64.tar.gz" exit 1 fi SALT_TAR="${KUBE_ROOT}/server/kubernetes-salt.tar.gz" if [[ ! -f "$SALT_TAR" ]]; then SALT_TAR="${KUBE_ROOT}/_output/release-tars/kubernetes-salt.tar.gz" fi if [[ ! -f "$SALT_TAR" ]]; then echo "!!! Cannot find kubernetes-salt.tar.gz" exit 1 fi } </del> # Take the local tar files and upload them to S3.  They will then be # downloaded by the master as part of the start up script for the master. #

I suspect we're going through an if/else clause where KUBETARURL isn't defined, and not defaulting it, though the function header seems to indicate it should be set in the callee.
Will take a look.
Is this in Jenkins, or on a local run? The Jenkins push job is borked as well, but it looks like it's for a different reason: Which I'm taking care of in .
Jobs are generating incredibly large log files (300 Megs?) with lots of repeated lines like this: Failure trying to curl release .sha1 /jenkins-master-data/jobs/kubernetes-push-gce-step1-deploy/workspace/kubernetes/hack/e2e-internal/../../cluster/../cluster/../c line 283: KUBETARURL: unbound variable Failure trying to curl release .sha1 /jenkins-master-data/jobs/kubernetes-push-gce-step1-deploy/workspace/kubernetes/hack/e2e-internal/../../cluster/../cluster/../c line 283: KUBETARURL: unbound variable Failure trying to curl release .sha1 I think this needs to be resolved, since it already caused an out-of-disk in Jenkins.
Okiedokie. Yes, this was my fault. silently expects and to be set in . I the extra expectation that be set, but didn't think about how would then also have to set that. Working on a fix.
(For anyone who cares, culprit is .)


Question: <ins> /* Copyright 2023 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package metrics import ( "strings" "testing" "k8s.io/component-base/metrics/legacyregistry" "k8s.io/component-base/metrics/testutil" ) func TestRecordEncryptionConfigAutomaticReloadFailure(t *testing.T) { expectedValue := ` # HELP apiserver_encryption_config_controller_automatic_reload_failures_total [ALPHA] Total number of failed automatic reloads of encryption configuration. # TYPE apiserver_encryption_config_controller_automatic_reload_failures_total counter apiserver_encryption_config_controller_automatic_reload_failures_total 1 ` metrics := []string{ namespace + "_" + subsystem + "_automatic_reload_failures_total", } encryptionConfigAutomaticReloadFailureTotal.Reset() RegisterMetrics() RecordEncryptionConfigAutomaticReloadFailure() if err := testutil.GatherAndCompare(legacyregistry.DefaultGatherer, strings.NewReader(expectedValue), metrics...); err != nil { t.Fatal(err) } } func TestRecordEncryptionConfigAutomaticReloadSuccess(t *testing.T) { expectedValue := ` # HELP apiserver_encryption_config_controller_automatic_reload_success_total [ALPHA] Total number of successful automatic reloads of encryption configuration. # TYPE apiserver_encryption_config_controller_automatic_reload_success_total counter apiserver_encryption_config_controller_automatic_reload_success_total 1 ` metrics := []string{ namespace + "_" + subsystem + "_automatic_reload_success_total", } encryptionConfigAutomaticReloadSuccessTotal.Reset() RegisterMetrics() RecordEncryptionConfigAutomaticReloadSuccess() if err := testutil.GatherAndCompare(legacyregistry.DefaultGatherer, strings.NewReader(expectedValue), metrics...); err != nil { t.Fatal(err) } } func TestEncryptionConfigAutomaticReloadLastTimestampSeconds(t *testing.T) { testCases := []struct { expectedValue string resultLabel   string timestamp     int64 }{ { expectedValue: ` # HELP apiserver_encryption_config_controller_automatic_reload_last_timestamp_seconds [ALPHA] Timestamp of the last successful or failed automatic reload of encryption configuration. # TYPE apiserver_encryption_config_controller_automatic_reload_last_timestamp_seconds gauge apiserver_encryption_config_controller_automatic_reload_last_timestamp_seconds{status="failure"} 1.689101941e+09 `, resultLabel: "failure", timestamp:   1689101941, }, { expectedValue: ` # HELP apiserver_encryption_config_controller_automatic_reload_last_timestamp_seconds [ALPHA] Timestamp of the last successful or failed automatic reload of encryption configuration. # TYPE apiserver_encryption_config_controller_automatic_reload_last_timestamp_seconds gauge apiserver_encryption_config_controller_automatic_reload_last_timestamp_seconds{status="success"} 1.689101941e+09 `, resultLabel: "success", timestamp:   1689101941, }, } metrics := []string{ namespace + "_" + subsystem + "_automatic_reload_last_timestamp_seconds", } RegisterMetrics() for _, tc := range testCases { encryptionConfigAutomaticReloadLastTimestampSeconds.Reset() encryptionConfigAutomaticReloadLastTimestampSeconds.WithLabelValues(tc.resultLabel).Set(float64(tc.timestamp)) if err := testutil.GatherAndCompare(legacyregistry.DefaultGatherer, strings.NewReader(tc.expectedValue), metrics...); err != nil { t.Fatal(err) } } } </ins>

ref - /sig auth
/triage accepted
/retitle encryption-at-rest: add success or failure metric when encryption configuration attempts to dynamically update
/assign


Question: <del> #!/bin/bash # Copyright 2014 Google Inc. All rights reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # #     http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # Assumes a running Kubernetes test cluster; verifies that the monitoring setup # works. Assumes that we're being called by hack/e2e-test.sh (we use some env # vars it sets up). set -o errexit set -o nounset set -o pipefail KUBE_ROOT=$(dirname "${BASH_SOURCE}")/../.. : ${KUBE_VERSION_ROOT:=${KUBE_ROOT}} : ${KUBECTL:="${KUBE_VERSION_ROOT}/cluster/kubectl.sh"} : ${KUBE_CONFIG_FILE:="config-test.sh"} export KUBECTL KUBE_CONFIG_FILE source "${KUBE_ROOT}/cluster/kube-env.sh" source "${KUBE_VERSION_ROOT}/cluster/${KUBERNETES_PROVIDER}/util.sh" prepare-e2e MONITORING="${KUBE_ROOT}/cluster/addons/cluster-monitoring" KUBECTL="${KUBE_ROOT}/cluster/kubectl.sh" BIGRAND=$(printf "%xn" $(( $RANDOM << 16 | $RANDOM ))) # random 2^32 in hex MONITORING_FIREWALL_RULE="monitoring-test-${BIGRAND}" function setup { # This only has work to do on gce and gke if [[ "${KUBERNETES_PROVIDER}" == "gce" ]] || [[ "${KUBERNETES_PROVIDER}" == "gke" ]]; then detect-project if ! "${GCLOUD}" compute firewall-rules create "${MONITORING_FIREWALL_RULE}"  --project "${PROJECT}"  --network "${NETWORK}"  --quiet  --allow tcp:80 tcp:8083 tcp:8086 tcp:9200; then echo "Failed to set up firewall for monitoring" && false fi fi "${KUBECTL}" create -f "${MONITORING}/" } function cleanup { "${KUBECTL}" stop rc monitoring-influx-grafana-controller &> /dev/null || true "${KUBECTL}" stop rc monitoring-heapster-controller &> /dev/null || true "${KUBECTL}" delete -f "${MONITORING}/" &> /dev/null || true # This only has work to do on gce and gke if [[ "${KUBERNETES_PROVIDER}" == "gce" ]] || [[ "${KUBERNETES_PROVIDER}" == "gke" ]]; then detect-project if "${GCLOUD}" compute firewall-rules describe "${MONITORING_FIREWALL_RULE}" &> /dev/null; then "${GCLOUD}" compute firewall-rules delete  --project "${PROJECT}"  --quiet  "${MONITORING_FIREWALL_RULE}" || true fi fi } function influx-data-exists { local max_retries=10 local retry_delay=30 #seconds local influx_ip=$("${KUBECTL}" get pods -l name=influxGrafana -o template -t {{range.items}}{{.currentState.hostIP}}:{{end}} | sed s/://g) local influx_url="http://$influx_ip:8086/db/k8s/series?u=root&p=root" local ok="false" for i in `seq 1 10`; do if curl --retry $max_retries --retry-delay $retry_delay -G $influx_url --data-urlencode "q=select * from stats limit 1"  && curl --retry $max_retries --retry-delay $retry_delay -G $influx_url --data-urlencode "q=select * from machine limit 1"; then echo "retrieved data from InfluxDB." ok="true" break fi sleep 5 done if [[ "${ok}" != "true" ]]; then echo "failed to retrieve stats from InfluxDB. monitoring test failed" exit 1 fi } function wait-for-pods { local running=false for i in `seq 1 20`; do sleep 20 if "${KUBECTL}" get pods -l name=influxGrafana -o template -t {{range.items}}{{.currentState.status}}:{{end}} | grep Running &> /dev/null  && "${KUBECTL}" get pods -l name=heapster -o template -t {{range.items}}{{.currentState.status}}:{{end}} | grep Running &> /dev/null; then running=true break fi done if [ running == false ]; then echo "giving up waiting on monitoring pods to be active. monitoring test failed" exit 1 fi } trap cleanup EXIT # Remove any pre-existing monitoring services. cleanup # Start monitoring pods and services. setup # Wait for a maximum of 5 minutes for the influx grafana pod to be running. echo "waiting for monitoring pods to be running" wait-for-pods # Wait for some time to let heapster push some stats to InfluxDB. echo "monitoring pods are running. waiting for stats to be pushed to InfluxDB" sleep 60 # Check if stats data exists in InfluxDB echo "checking if stats exist in InfluxDB" influx-data-exists echo "monitoring setup works" exit 0 </del>

Due to changes in field selectors (see ) heapster was almost certainly busted, but continues to pass. This seems incorrect.


Question: type testResult bool func init() { <del> // Turn off colors by default to make it easier to collect console output in Jenkins // Override colors off with --ginkgo.noColor=false in the command-line config.DefaultReporterConfig.NoColor = true </del> // Turn on verbose by default to get spec names config.DefaultReporterConfig.Verbose = true <ins> // Turn on EmitSpecProgress to get spec progress (especially on interrupt) config.GinkgoConfig.EmitSpecProgress = true </ins> // Randomize specs as well as suites config.GinkgoConfig.RandomizeAllSpecs = true }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --/kind bug Got two different failure scenarios: ://k8s- ://k8s- /cc Please reassign if the above is wrong. Thanks!
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
Both these failures seems to be test infra issues IIUC. There are no failures in the GPU related logic.
/cc
Seems like it's fixed in
Yes, the PR GPU job had a flake with reusing a build which is a new feature that all e2e jobs will use soon. Hopefully it should be fixed now.
Once gets picked up things should be fixed. I'm still monitoring this. /assign
I have a PR out to revert using shared builds since rolling out an updated CI image will take a bit longer and the patch to kubetest needs to be rolled out.
/sig testing
This is fixed. /close


Question: corev1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/labels" <ins> "k8s.io/apiserver/pkg/authentication/group" </ins> "k8s.io/apiserver/pkg/authentication/request/bearertoken" bootstrapapi "k8s.io/cluster-bootstrap/token/api" "k8s.io/kubernetes/plugin/pkg/auth/authenticator/token/bootstrap"

While moving APF to beta (see comments on ), we discovered that the integration tests commonly present users which are not members of any groups. We had thought that the system enforced membership in either the authenticated or unauthenticated group. If this is supposed to be a system invariant (as we thought), we'd like to ask SIG Auth: Please fix the integration tests Please add e.g. logging so that if real, operational clusters hit this there's at least a log message and people will file bugs. /sig auth
Can I suggest also, if this is a real invariant, that it be documented in some place(s) likely to be read?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale /priority important-longterm /triage accepted
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/lifecycle frozen
fixes the authenticator configured in one of the integration test API server startup helpers which resolved this issue for . Remaining tests that still need fixing that are encountering unexpected users look to be ones set up to specifically test auth paths:
open to fix remaining integration test users


Question: import ( "context" "fmt" <ins> "log" </ins> "net" "net/url" <ins> "os" </ins> "path" "strings" "sync" "time" grpcprom "github.com/grpc-ecosystem/go-grpc-prometheus" <ins> "go.etcd.io/etcd/client/pkg/v3/logutil" </ins> "go.etcd.io/etcd/client/pkg/v3/transport" clientv3 "go.etcd.io/etcd/client/v3" "go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc" <ins> "go.uber.org/zap" "go.uber.org/zap/zapcore" </ins> "google.golang.org/grpc" "k8s.io/apimachinery/pkg/runtime"

The project uses a lot of CRDs - in some cases almost 2,000. I do mean here, not custom resources. We've noticed that API server memory consumption balloons when many CRDs are installed - for example with ~1,900 CRDs we see the API server using up to ~8Gi RSS. A little profiling shows that around 1.5Gi of that memory is being used by the etcd3 client's logger. With Kubernetes built from with the CRDs at loaded I'm seeing the API server using up to ~7.5GiB RSS. I see it dip to ~5.5GiB, presumably when garbage collection runs. You can see taking up a ton of memory below. Zap is the logger used by the etcd client. Note that this is a sample, so it's actually using more than the indicated ~800MiB. ! I'm seeing the API server using up to ~6GiB RSS. I see it dip to ~4.5GiB, presumably due to GC runs. You can see no longer appears below. ! I would not expect a logger to be responsible for such a significant amount of the API server's memory consumption. You can get the above profile by: out . to build a kind node. to start a control plane for the above build. to allow anonymous pprof access the CRDs from I used to see the RSS but I imagine would show it too. See for details on how I disabled the logger to get the "without logger" numbers. No response <details</details<detailsN/A </details<details</details<details</details<details</details<details</details
/sig api-machinery
Since you looked at the code, and I'd rather not spend time doing that. Any chance you can tell if it's because it's being mis-used?
This is a scale dimension where the apiserver has historically . With the OpenAPI discovery performance being improved it looks like this might be a hint at what the next bottleneck might be? Any idea how many etcd clients were instantiated? I suspect it's quite a lot.
Sadly I didn't yet look deeply past turning it off. I was slightly optimistic that we might just be able to disable it without any meaningful impact.

What I do know is that the offending code seems to be on the etcd3 client end - i.e. it defaults to creating its own logger when one isn't supplied. You can see that code at
I don't - will try to find out.
The zap logs might tell you if you have the data. If my memory of the code is right, you should get 1 etcd client per CRD right now (plus some for builtin types).
/triage accepted
Based on my read of the code, you're right. I see roughly: All of /apis is handled by a crdHandler When a request comes in it gets an instance of the crd (from an informer) Gets serving info from a map of CRDs - if there is none it... Calls customresource.NewStorage which creates a genericregistry.Store Calls its CompleteWithOptions method That calls its “Decorator”, which is a genericregistry.StorageWithCacher That calls generic.NewRawStorage which just wraps factory.Create factory.Create calls newETCD3Storage which calls newETCD3Client I'm experiencing technical difficulties getting lsof running in my kind container but /proc/pid/net confirms we have a connection to etcd per CRD: Note that 1,937 - 60 = 1,877 - the number of CRDs we have loaded (the 1,878 above includes the header line).


Question: "k8s.io/kubernetes/pkg/kubelet/cm" "k8s.io/kubernetes/pkg/kubelet/config" kubecontainer "k8s.io/kubernetes/pkg/kubelet/container" <ins> kubelettypes "k8s.io/kubernetes/pkg/kubelet/types" </ins> utilnode "k8s.io/kubernetes/pkg/util/node" )

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Pods remaining in running but not ready state after node become unready for few seconds. What you expected to happen: Pods should become ready when node is ready. How to reproduce it (as minimally and precisely as possible): Created large deployment. Make node unready for few seconds. Pods will remain not ready. However is requires node lifecycle controler to be in eviction pass, so my guess it will not be reproducible in 100% of cases. Anything else we need to know?: I achieve this state lock with kubemark run. There is chance that vanilla k8s is unaffected. Environment: Kubernetes version (use ): 18 Cloud provider or hardware configuration: gce - kubemark OS (e.g: ): linux Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
My guess of what happened is: nodes become not ready. lifecycle controller store node statuses become ready again pass (based on stored status) marked pods not ready even though nodes were ready at that time
/cc
kube-controller-manager log: The error was created here: Assuming that node still existed at that time, it didn't changed execution of the logic. OTOH support for kubemark provider would be nice.
There is a possibility that node lifecycle controller didn't spot node updates. I'm bumping up log level for this case:
Apiserver logs (ragrding node status): Updating pod (from node hollow-node-pz68p):
Can someone give a hint what sets the pod ready status to true, when node is ready again?
I also find this bug and It is introduced by when I rollback this PR, This bug is fixed
If you have a chance, can you see if the following fixes the issue you (without reverting ) ?
I haven't find a way to reproduce this issue every time, so it is hard for me to say if it fixes the issue. Can you say more about what issue this patch addresses? Is there a problem with statusManager having wrong/old pod status?
I've re-run the test.

There was also a problem with hollow-nodes. This time test didn't have problem with pods. k8s build included patch posted above. It looks like your patch helps. I still would like to hear some backstory of what happens there.
My patch refines the code to achieve what intended to do. Can you say more about problem with hollow-node ? Please post log snippet if you have it. Thanks
hi, . I have already test your fix in and it can solve this bug. Still, I have a suggestion. Could you use UpdatePod() function in podManager instead of directly changing pod status in podManager by p.Status=status? If some other codes also use this direct way to modify pod infos in podManager, it will be very hard to trace pod infos are modified by which code , still there is no any log show it has been changed.
- FYI
I have log for status update over w.r.t. using UpdatePod() function in podManager, the review / test may take more time. I will think about it meanwhile.
Is this issue the same problem? Sure looks like it: I've been following that issue in hopes of a fix as we see this happen a few times a day (several large clusters). Very happy to see progress.
Can you attach kubelet log (without the patch) somewhere and share the URL ? Thanks
We are also hit by this on a regular basis with kublets v1.15.2 The following issues are reporting the same bug I think: As pointed out in this inconsistency seems to be trigged when a kubelet reconnects to the apiserver. At least this is what we observe.
this as well /cc
Maybe add a bug label? I believe there are at least 4 independent bug reports about this rather serious inconsistency that requires a kubelet restart to fix.
Reiterating the mentions to trigger a notification: <detailsIn response to : Maybe add a bug label? I believe there are at least 4 independent bug reports about this rather serious inconsistency that requires a kubelet restart to fix. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Ah, nvm. It is already attached. Sorry for the noise.

I got confused with the other related issues. All good.
Is a different fix for the same underlying problem? I wouldn't normally ask, but we're trying to understand if there are two different problem and fixes we need, or competing solutions for the same problem.


Question: panic(fmt.Sprintf("b: %v", err)) } return util.StringDiff(string(ab), string(bb)) <ins> } </ins> <del> // An alternate diff attempt, in case json isn't showing you // the difference. (reflect.DeepEqual makes a distinction between // nil and empty slices, for example.) </del> <ins> // ObjectGoPrintDiff is like ObjectDiff, but uses go's %#v formatter to print the // objects, in case json isn't showing you the difference. (reflect.DeepEqual makes // a distinction between nil and empty slices, for example, even though nothing else // really does.) func ObjectGoPrintDiff(a, b Object) string { </ins> return util.StringDiff( fmt.Sprintf("%#v", a), fmt.Sprintf("%#v", b),

For example: struct field tag not compatible with reflect.StructTag.Get missing argument for Errorf("%v"): format reads arg 2, have only 1 args range variable pod enclosed by function struct field tag not compatible with reflect.StructTag.Get arg minion for printf verb %s of wrong type: unreachable code unreachable code
Those all look really important
I'm going to fix this unless I hear that someone else is working on this in the next couple minutes.


Question: }, } } <ins> func newPodWithInitContainers(name string, containers []v1.Container, initContainers []v1.Container) *v1.Pod { return &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name: name, }, Spec: v1.PodSpec{ Containers:     containers, InitContainers: initContainers, }, } } </ins>

What happened: A pod that has init container with request for cpu and memory, and containers that make no request for cpu or memory reports a BestEffort QoS class. This is incorrect. What you expected to happen: Containers that make requests independent of lifecycle phase should impact quality of service calculation. Found in 1.12, looks to still be present in 1.14 dev. /cc
/priority important-soon
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/lifecycle frozen
I think this doesn't work very well with the current cpu manager logic. The problem is that a pod can't now have QoS class Guaranteed if it has init containers which don't have matching requests/limits. This means that normal containers in such a pod don't then get dedicated cpus even though the init containers are long gone when the app containers (with matching requests/limits) are finally started. This is quite non-intuitive. I would propose separating at least Guaranteed class calculation so that the init container resource requests wouldn't affect the calculation. What do you think?


Question: }, Type: corev1.SecretTypeBootstrapToken, Data: map[string][]byte{ <del> bootstrapapi.BootstrapTokenIDKey:               []byte(tokenID), </del> <ins> bootstrapapi.BootstrapTokenIDKey:               []byte(validTokenID), </ins> bootstrapapi.BootstrapTokenSecretKey:           []byte("invalid"), bootstrapapi.BootstrapTokenUsageAuthentication: []byte("true"), <del> bootstrapapi.BootstrapTokenExpirationKey:       []byte(bootstraputil.TimeStringFromNow(-time.Hour)), </del> <ins> bootstrapapi.BootstrapTokenExpirationKey:       []byte(tokenExpiredTime), </ins> }, } type request struct {

Originally tracked at . After fixing "not include test files in import-boss" in , we have a regression here: is using . We need remove dependencies from integration tests to e2e tests again. /assign /sig testing
/sig auth as per the required change
/kind cleanup /area test
It would be helpful to outline an argument as to why this is the case. I can imagine reasons, but none are mentioned here.
Description updated.
/triage accepted /priority backlog /milestone v1.21 This seems a simple enough fix to land (and thank you for doing this) but marking low enough priority to make it clear we may back out and wait for v1.22 if we don't have bandwidth for this
So while I think we should probably do this anyhow I want to note: This issue lists the reasons: we have no intention to stage e2e (there's been much discussion about this since then) we generally don't want anything non-informational (required jobs) running only sometimes anyhow, it's just not reliable Decoupling these seems like a good idea from a purely code-organization perspective, but the reasons in the previous issue are no longer valid.
/milestone v1.22 The PR didn't land in time. At this point I view this as nice to have. Even if it's low impact, let's wait until next release.


Question: } if len(matchedPluginNames) == 0 { <del> return nil, fmt.Errorf("no volume plugin matched") </del> <ins> return nil, ErrNoPluiginMatched </ins> } if len(matchedPluginNames) > 1 { return nil, fmt.Errorf("multiple volume plugins matched: %s", strings.Join(matchedPluginNames, ","))

In the events , the PVC has been successfully resized, but a warning-level log is still printed. And by checking the logs of kube-controller-manager, can see the following output. Do not print events with warning level. To expand a a CSI PVC, and then use "kubectl describe pvc" to view its events. (I am using xsky CSI to expand pvc, but based on my understanding of Kubernetes code, this issue can occur with any CSI.) ! ! Based on Kubernetes v1.27.3, I some logs in the func located in . These logs printed all the plugins before expansion, and I noticed that was not among them. In my understanding, it is normal for CSI to be absent from the plugins here because CSI expansion is scheduled through external-resizer instead of kube-controller-manager. Therefore, there should not be any warning-level logs printed here either. v1.27.3 (In version 1.18, the issue can also be reproduced) <details</detailsLinux debian-hk 5.10.0-23-amd64 SMP Debian 5.10.179-1 (2023-05-12) x86_64 GNU/Linux kind v0.20.0 <details</details<details</details
/sig storage
We can change the ExternalExpanding event from Warning to Info. are you interested in making this change?
/triage accepted
Yes, I will submit the pull request in the next few days.
Hi Have you submitted the PR? I have a bugfix branch tested on v1.29.1. I would like to cherry-pick it to master if this is not fixed yet or not started.
You can go do it. Thank you!


Question: // Check iptables chain and rules epIpt := netlinktest.ExpectedIptablesChain{ string(kubeServicesChain): { <ins> {JumpChain: "RETURN", SourceAddress: "127.0.0.0/8"}, </ins> {JumpChain: string(kubeLoadBalancerChain), MatchSet: kubeLoadBalancerSet}, {JumpChain: string(kubeMarkMasqChain), MatchSet: kubeClusterIPSet}, {JumpChain: string(kubeMarkMasqChain), MatchSet: kubeExternalIPSet},

Loopback addresses (127.0.0.0/8, ::1/128) can't be used by services in proxy-mode=ipvs it makes sense to dismiss those early, add RETURN rule to KUBE-NODE-PORT,KUBE-SERVICES No response <details</details25 <details</detailsnone <details</details<details</details<details</details<details</details
/sig network
As mentioned in , this is not a bug since the nodeport and ephemeral ranges overlap.
/kind cleanup /kind remove-bug
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /kind cleanup /kind remove-bug </details
/remove-kind bug
The title of this issue should be changed to something like; Which would indicate that this is not a bug fix but an improvement. Also remove the invalid use-case in the description and add references to comments in like I think to dismiss nodePort connects to localhost early for proxy-mode=ipvs is an improvement, and in the majority of systems traffic to localhost is more common than traffic to nodePorts so this change will also improve overall performance. This is not a fix for overlapping ephemeral and nodePort ranges! So if the title and description is not updated the correct triage is to close this issue as it violates the non overlapping rule for ephemeral/nodeport ranges, please see
/area ipvs
Ok, i will change issue title and descripiton
/triage accepted


Question: type conditionalTransport struct { oauthTransport *oauth2.Transport <ins> persister      restclient.AuthProviderConfigPersister </ins> } func (t *conditionalTransport) RoundTrip(req *http.Request) (*http.Response, error) { if len(req.Header.Get("Authorization")) != 0 { return t.oauthTransport.Base.RoundTrip(req) } <del> return t.oauthTransport.RoundTrip(req) </del> <ins> res, err := t.oauthTransport.RoundTrip(req) if err != nil { return nil, err } if res.StatusCode == 401 { glog.V(4).Infof("The credentials that were supplied are invalid for the target cluster") emptyCache := make(map[string]string) t.persister.Persist(emptyCache) } return res, nil </ins> }

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): No What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): kubectl "does not allow access" Is this a BUG REPORT or FEATURE REQUEST? (choose one): Bug report <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Environment: Cloud provider or hardware configuration: GCP GKE 1.4.5 OS (e.g. from /etc/os-release): Not sure, using gci Kernel (e.g. ): Not sure, using gci Install tools: GKE web UI Others: N/A What happened: In juggling multiple gcloud accounts with GKE clusters I accidentally used with the wrong application default credentials and it cached bad creds in After pointing gcloud at the right application default credentials and re-running , it continued trying and failing to use the bad creds cached in ; at this point I was stuck and confused for a few hours To repair, I had to manually edit to remove , and then started working again (or else wait ~1h for the access token to expire and refresh itself) What you expected to happen: After pointing gcloud at the right application default credentials should just work, or at least provide an error msg to the user that can help them figure out that they need to revoke the bad access token and how to do it How to reproduce it (as minimally and precisely as possible): Good scenario: just works Bad scenario: user makes one mistake and gets stuck
+1 Removing helped, thanks.
I can look into fixing this if no-one else has already made a start on it?
feel free to take it, thanks!

Ok, I think I have an understanding of what is happening call to is storing credentials in a json file in a known location. a subsequent credential-requiring kubectl command, the GCP plugin is being loaded, and the application default credentials are being exchanged for a token. That token is then cached in the .kube/config file. So where do things go wrong? As far as I can tell, if I switch to a cluster-2 without refreshing the application default credentials from it's separate account, those existing credentials will still be exchanged successfully for an oauth2 token and cached against cluster-2. Those tokens will then fail against a call to cluster-2 - because they were intended for cluster-1. The only sure-fire fix I can think of would be to ensure the application default credentials client ID is stored in users[].user.authprovider. Doing this would at least enable the ability to verify that the default credentials were valid for that particular cluster, and if not, prompt the user with a message to correctly aquire the application default credentials for it. Lets call this option 1. Of course, there is a problem here with determining what the Client ID is in the first place. I think this would require that a call to would store the client ID in the generated config. This would require exposing the client id from oauth2/google. Otherwise, perhaps storing the client ID the first time around would be enough - assuming the user at least manages to authenticate successfully against the cluster in the first instance. Then only oauth2/google would require a change, and not the gcloud command. The other option (option 2) would be, upon failure, check if is true and if-so, prompt the user that their application default credentials may be invalid for the target cluster, and to either prompt or clear the currently cached credentials in .kube/config. This would probably mean extending the auth provider interface such that a function may be provided that can be called upon failure that enables the auth provider plugin to do some checking of the state and to provide a reasonable message as to what might be wrong with the credentials that have been provided. I feel like option 2 is probably the better option in the long term. Different auth plugins are likely to have different failure modes. Providing a way to communicate back to the user about possible edge cases seems a little more flexible. Any thoughts?

edit: On closer inspection this seems possible by wrapping the oauth2.Transport call in WrapTransport. I would then be able to check for an authorization failure and provide a reasonable error message based on the existing configuration.
Just come across this, its particularly strange to have to edit the config file, when there is a folder in the folder.
Is it really fixed? I still can reproduce it on k8s 1.14 with description provided in initial comment.
I also reproduced this, unfortunately. Tried to use kubectl to access a cluster from an unauthorised account. Then, switched to the authorised account but the invalid authentication was cached. To resolve it, I set the to an expired date.
Believe I'm hitting this issue as well
I just ran into this as well, gcloud 311.0.0, kubectl 1.18.6.
I've got the same issue. I need to change the expiry time to 1 year back as pointed out.
Solution that usually works for me So it will use different credentials when switching contexts
You'll need  installed ().


Question: if err != nil { return apierrors.NewInternalError(err) } <ins> if len(patchObj) == 0 { return nil } // if a non-empty patch was provided, and we have no object we can apply it to (e.g. a DELETE admission operation), error if attr.VersionedObject == nil { return apierrors.NewInternalError(fmt.Errorf("admission webhook %q attempted to modify the object, which is not supported for this operation", h.Name)) } </ins> objJS, err := runtime.Encode(a.plugin.jsonSerializer, attr.VersionedObject) if err != nil { return apierrors.NewInternalError(err)

We had mutating webhooks on delete operation. After updating to Kubernetes 1.11, it is failing. Stacktrace: I tried to debug and this is what I found: is nil. Calling causes this error. I see that the code path has not changed since 1.10. But we don't see the same error in 1.10 minikube cluster. I can't explain that. We have stopped using mutating webhook on delete operation anyway, since it does not make much sense. But I thought I should file a bug just in case. /kind bug /sig api-machinery
cc:
thanks for the report... is this on a custom resource or a native resource? edit: nevermind, found in the stack
It is a CRD.
fix is in , pick to 1.11.x in conversion of unstructured objects (used internally for CRDs) changed in 1.11 as part of CRD versioning, which made them actually go through the conversion stack
/assign


Question: hostname = hostnameCandidate } } <del> if len(hostname) > hostnameMaxLen { hostname = hostname[:hostnameMaxLen] glog.Errorf("hostname for pod:%q was longer than %d. Truncated hostname to :%q", pod.Name, hostnameMaxLen, hostname) </del> <ins> hostname, err := truncatePodHostnameIfNeeded(pod.Name, hostname) if err != nil { return "", "", err </ins> } hostDomain := ""

Docker daemon complains "invalid hostname format" if the truncated host name ends with "." The truncation occurs .
Docker checks hostname with the regexp (See ): It means that the hostname must end with an alnum, not or . This issue has been here for a long time. This usually happens to static pod, because we compose a looooong name for static pod. We didn't hit this just because we are lucky. :P Will send a PR to fix this. /cc
: Needed for v1.5? please add milestone if so
Thanks for reminding. :) This is a bug fix in kubelet. Mark 1.5. /cc
Are you working on a fix?
Yeah, the fix is here


Question: to.LogWriter = ginkgo.GinkgoWriter } else { test := ginkgo.CurrentGinkgoTestDescription() <ins> // Clean up each individual component text such that // it contains only characters that are valid as file // name. </ins> reg := regexp.MustCompile("[^a-zA-Z0-9_-]+") <ins> var components []string for _, component := range test.ComponentTexts { components = append(components, reg.ReplaceAllString(component, "_")) } </ins> // We end the prefix with a slash to ensure that all logs // end up in a directory named after the current test. // <del> // TODO: use a deeper directory hierarchy once gubernator // supports that (https://github.com/kubernetes/test-infra/issues/10289). </del> <ins> // Each component name maps to a directory. This // avoids cluttering the root artifact directory and // keeps each directory name smaller (the full test // name at one point exceeded 256 characters, which was // too much for some filesystems). </ins> to.LogPathPrefix = framework.TestContext.ReportDir + "/" + <del> reg.ReplaceAllString(test.FullTestText, "_") + "/" </del> <ins> strings.Join(components, "/") + "/" </ins> } podlogs.CopyAllLogs(ctx, cs, ns, to)

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Currently test names are formed by concatenating the ginkgo context's together. The testname is used to create a directory. This creates a limit on the length of the the testname, as most operating systems can only support a filename of up to 255 bytes. This concatenating is done by the logging library (I think) A way to break up the long test names by ginkgo context. Why is this needed: This test () failed on PR Current workarounds are simple by shortening the test names. However this is not desirable if we wish to support more nesting of ginkgo contexts in order to reduce boilerplate testing code by sharing common setup. This is also not desirable because it limits the ability to communicate the intent of the .
/sig testing
why are we using the testname to create a directory? even if we do need to create a directory per test we should probably use something more unique. sometimes multiple storage test entries wind up with the same test name
/sig storage
I assume this is about the artifacts, for example in The test name is guaranteed to be unique per test run. It was chosen because then developers can find the artifacts for a certain test. The alternative might be to split at component texts, see
In this example, the directory might then become (I'm guessing a bit what the component texts are - I'm not sure).
Oh, there's an old todo about doing exactly that. It wasn't possible at the time because gubernator couldn't handle deeply nested directories:
Thank you all!


Question: } func (c *LRUExpireCache) Get(key lru.Key) (interface{}, bool) { <del> c.lock.RLock() defer c.lock.RUnlock() </del> <ins> c.lock.Lock() defer c.lock.Unlock() </ins> e, ok := c.cache.Get(key) if !ok { return nil, false

Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): No What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): "LRUExpireCache" Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT What happened: requires write lock since needs to manipulate its list to track recently used item. Currently it uses read lock so it may introduce race condition. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else do we need to know:
This was fixed in


Question: &PetSet{}, &PetSetList{}, &v1.ListOptions{}, <ins> &v1.DeleteOptions{}, </ins> ) versionedwatch.AddToGroupVersion(scheme, SchemeGroupVersion) }

It seems to me like all of the garbage collection integration tests produce a ton of log messages of the form:


Question: package factory <del> import "testing" </del> <ins> import ( "k8s.io/kubernetes/plugin/pkg/scheduler/algorithm" "k8s.io/kubernetes/plugin/pkg/scheduler/api" "testing" ) </ins> func TestAlgorithmNameValidation(t *testing.T) { algorithmNamesShouldValidate := []string{

Weight is an int (see ). Priority can take the range 1-10. Total priority for a node is sum of weight  priority for each priority function, and the sum is accumulated into an int (see ). So obviously the total priority can overflow if someone sets the weight(s) too high. The weights of all priority functions are compiled in (or specified in config file), so we can check to make sure maximum total priority can't overflow an int (take sum over all priority functions of 10  weight of that priority function, and make sure that's smaller than int(MaxUint 1) which according to StackOverflow is the way to find out what is max int (math package has max Int32, Int64, etc. but not max Int). Note that scheduler extender is OK because although the policy runs in a separate process, the weight is specified in scheduler config file (ExtenderConfig struct) and the priority is bounded 1-10 as usual.
We really should also make "score" be an abstract datatype rather than a raw int that is manipulated directly, so we can do things like make sure it is in the range 1-10 (or 0-10 if that's what we want); today the range is inconistent (we say 1-10, but sometimes use 0; probably we should change it to 0-9..).
I volunteer myself to do this.
I think we can check weight overflow in after we get all PriorityConfig. what do you think about this?
As for "score" I think we can use
Fixed by - closing.
Tim that PR was closed, not merged. I don't think this is fixed yet.
whoops, good catch, just removing milestone then. There was a shift that happened yesterday where a bunch of items suddenly came into the 1.6 query.
do we still want this fixed? I can re-pick this.
AFAIK it's still broken. cc/
Ok, I will open a PR tomorrow.


Question: "--use-service-account-credentials=true", "--controllers=*,bootstrapsigner,tokencleaner", "--allocate-node-cidrs=true", <del> "--cluster-cidr=bar", </del> <ins> "--cluster-cidr=10.0.1.15/16", "--node-cidr-mask-size=24", }, }, { cfg: &kubeadmapi.MasterConfiguration{ Networking:        kubeadmapi.Networking{PodSubnet: "2001:101:115::/48"}, CertificatesDir:   testCertsDir, KubernetesVersion: "v1.7.0", }, expected: []string{ "kube-controller-manager", "--address=127.0.0.1", "--leader-elect=true", "--kubeconfig=" + kubeadmconstants.KubernetesDir + "/controller-manager.conf", "--root-ca-file=" + testCertsDir + "/ca.crt", "--service-account-private-key-file=" + testCertsDir + "/sa.key", "--cluster-signing-cert-file=" + testCertsDir + "/ca.crt", "--cluster-signing-key-file=" + testCertsDir + "/ca.key", "--use-service-account-credentials=true", "--controllers=*,bootstrapsigner,tokencleaner", "--allocate-node-cidrs=true", "--cluster-cidr=2001:101:115::/48", "--node-cidr-mask-size=64", </ins> }, }, }

/kind bug What happened: Currently, if the user sets the cidr with --pod-network-cidr with kubeadm, the kube-controller-manager is passed the cidr with the --cluster-cidr arg. --node-cidr-mask-size is not passed to the kube-controller-manager and the controller-manager used the default value of 24 The default value of 24 if good for IPv4, but for IPv6 the controller-manager throws an error because the node-cidr-mask-size is incorrect. What you expected to happen: The user should be able to specify the node-cidr-mask-size for IPv6 How to reproduce it (as minimally and precisely as possible): Use Kubeadm specify an IPv6 address with --pod-network-cidr. with a subnet greater than 24. For example kubeadm init --pod-network-cidr=2001:101::0/64 Anything else we need to know?: Environment: IPv6 Kubernetes version (use ): Cloud provider or hardware configuration**: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/area ipv6 /area kubeadm
/assign rpothier


Question: // Check validates http connectivity type, direct or via proxy. func (hst HTTPProxyCheck) Check() (warnings, errors []error) { <del> url := fmt.Sprintf("%s://%s", hst.Proto, hst.Host) </del> <ins> u := (&url.URL{Scheme: hst.Proto, Host: hst.Host}).String() </ins> <del> req, err := http.NewRequest("GET", url, nil) </del> <ins> req, err := http.NewRequest("GET", u, nil) </ins> if err != nil { return nil, []error{err} }

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: When setting an HTTPPROXY with an IPv6 address, the is no HTTPProxy warning. What you expected to happen: There should be a warning that a proxy is being used. How to reproduce it (as minimally and precisely as possible): Use Kubeadm with IPv6 addresses, and IPv6 proxy in the environment. Such as: NOPROXY=2001:420:2c50:2021::/64 Anything else we need to know?: Environment: Kubernetes version (use ): 9 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/assign rpothier
/sig cluster-lifecycle
/area ipv6


Question: const ( ProviderName     = "openstack" AvailabilityZone = "availability_zone" <ins> defaultTimeOut   = 60 * time.Second </ins> ) var ErrNotFound = errors.New("Failed to find object")

What happened: We use OpenStack as our cloud provider. We meet an issue that some controllers stuck because all workers routine stuck at http call to OpenStack services because of rabbitmq issues. Stuck routine call stack: Checked the openstack cloud provider that there is no HttpClient.Timeout set which means no timeout during http call. I think we should set at least the default http timeout value. Environment: Kubernetes version (use ): v1.6.3 Cloud provider or hardware configuration**: OpenStack
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig openstack /kind feature PTAL PR .


Question: expectedResult: results.Failure, }, { // Probe successful. <del> podStatus:      &runningStatus, expectContinue: true, </del> <ins> podStatus: &runningStatus, expectContinue: map[string]bool{ liveness.String():  true, readiness.String(): true, startup.String():   true, }, </ins> expectSet:      true, expectedResult: results.Success, },

What happened: liveness/readiness probe fails while pod is terminated. Also it happened only once during the pod termination. The issue started happening after upgrading version to v1.7 from v1.6.X How to reproduce it (as minimally and precisely as possible): execute to delete pod, so status of the nginx-podA1 is changed to , right after that it seems Liveness and Readiness Probe is executed and failed, but only once. Nginx reverse proxy is running in the pod. so I just use method for liveness and readiness Here is my Deployment config. Here is Events log by Environment: Kubernetes version: 1.7.2
/kind bug
/sig node
PING
Since the upgrade to 1.7, it seems our deployment rollouts have a higher failure rate. Occasionally, the pod would come up, but no readiness probe ever gets started. It stays in that state, blocking the entire deployment. I usually have to delete the pod so it is rescheduled, and a new readiness probe is fired to check. I wonder if these are related issues here.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
/reopen /remove-lifecycle rotten We see this consistently with all pods who define a liveness or readiness probe. Whenever we roll out a new deployment, the pods who are terminated will emit a failed liveness/readiness probe AFTER they have been terminated.

We have considered adding a hook that just sleeps for 2-3 seconds, but it seems like a band-aid solution to something that should not happen in the first place. Is this an impossible-to-solve race condition between kubernetes moving parts?
You can't reopen an issue/PR unless you authored it or you are a collaborator. <detailsIn response to : /reopen /remove-lifecycle rotten Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Do you still experience this? This happens to us running kubernetes 1.11.4.
On k8s 1.12.7 and this is happening to me as well.
Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.6", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-07-26T10:04:08Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"} still happen. I think just add a phase "Terminateing" in pod, when an pod is in Terminateing just stop check.
I think it still happening in 1.13, why this issue was closed? Logs, !
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
still experience this on kubernetes 1.12.4. delete event and probe failed event so it seems that after the pod is deleted, the probe keeps running.I am wondering if the probe is started before the pod deletion.
/area kubelet
/priority important-soon
Any progress on this issue? It seems like some kind bug.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with .

Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-lifecycle rotten /assign
GitHub didn't allow me to assign the following users: ashleyschuett. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /remove-lifecycle rotten /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
16.9 here. Same issue.
/assign
I am also facing this issue. We have pods which have a lot of cleanup to do during shutdown, it can take up to 5 mins to terminate gracefully. During this time the livelinessProbe is detecting failure and restarting the pod. not really what we want. I am unable to prevent the service that handles the liveliness check from stopping while the cleanup is happening. It would be better if the pod was immediately removed from the service and the probes stopped while the shutdown is performed. Basically this ends up that k8s never actually is able to terminate the pod.
I will take that point and propose a shutdown probe to API sig.

Same here at 1.18.8
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
kubernetes_version 1.19.3. Same issue.
Same here, kubernetes 1.19.4.
/remove-lifecycle rotten
/assign
Should I consider a cherry-pick for 1.20 and 1.19? (maybe 1.18 too?)
we're running 1.16 and being hit by this continuously when some of our elixir apps shutdown so cherry-picking it in 1.18 would at least put it closer in our update path :angel:


Question: github.com/vishvananda/netlink/nl # github.com/vishvananda/netns v0.0.0-20171111001504-be1fbeda1936 => github.com/vishvananda/netns v0.0.0-20171111001504-be1fbeda1936 github.com/vishvananda/netns <del> # github.com/vmware/govmomi v0.20.1 => github.com/vmware/govmomi v0.20.1 </del> <ins> # github.com/vmware/govmomi v0.20.3 => github.com/vmware/govmomi v0.20.3 </ins> github.com/vmware/govmomi github.com/vmware/govmomi/find github.com/vmware/govmomi/list

Expectation- We should be able to create PVCs via a storageclass with storagePolicyName defined while using a singed ca cert for secure communications with vCenter. Bug- The PVC created with a storageclass with storagePolicyName defined fails with a error when PVCs created with a storageclass without the storagPolicyName work against the same vCenter and datastore. Scenario: Utilizing the vSphere in-tree plugin on kubernetes 1.12.7. Utilizing secure communication with vSphere (signed ca cert imported into the cluster) We are able to create PVCs with the following storage class: However, if we include a storagePolicyName we receive a failure: Here is the yaml for our sc1 storage class: /sig vmware
I think we are having issue with creating PBM client using certificate set on the vCenter Client. We need to create a PBM client to get list of compatible datastore for given storage policy. I think call is failing at - Can you please provide controller-manager logs for further debugging. Make sure log level is set to 4.
Sorry for the delayed response. I'm not confident I got the log level correct, however this is what I find:
I believe I have log level 4 now set:
For comparison, here is a PVC created w/o storagePolicyName on the same datastore which succeeds:
thanks for posting logs. Now this is confirmed, issue is happening while creating PBM client when we use certificate based authentication. One more thing I want to confirm regarding the certificate, are you using solution user as mentioned here -
Can you also provide value of in the file? Is it set to true or false / 1 or 0 ? Also confirm version of vCenter you are using.
is unset, as for the solution user I'd have to leave that to to expand on.
I don't believe we used any solution user because we simply used "" as the default user during installation. please correct me if I'm wrong.
How is the certificate and private key retrieved for the user . I am successfully able to connect to the PBM using vpxd certificte and key. I ssh into the vCenter and executed following commands to retrieve certificate and private key for vpxd. with vpxd cert and private key, I do not see this issue. In addition I also set while making connection.

Can you try setting Insecure to false? That's the flag setting that triggers the cert error.
with , I see the exact same error - , while creating VC client.
Isn't that because it's self-signed? In the case of the issue in this ticket the cert is signed by a trusted CA /cc to confirm.
The cert is self-signed and its CA isn't trusted.
I used vpxd certificate and issuer is I require to set .
Is there any way to make it work if insecure=false?
- then I’d say that’s expected? If the root CA isn’t trusted, then it is insecure?
well, while that's expected, I don't know if there's a method to add the root CA to a trusted list.
Correct me if I’m wrong but isn’t the trusted list the root CA store of the K8S node’s OS in this case? I’d imagine whatever CA Root signed the cert would need to be to the OS of choice’s CA store.
you are correct, we need to download trusted root certificate and add them in the store on node vm. For now on my mac, I have downloaded trusted root certificate from the vCenter -and them in the keychain and flagged as "Always Trust". ! After this, I am able to connect to VC client with or without specifying this flag at all using VPXD certificate and private key. So certificate needs to be to the CA trust store on the master node VM. after adding trusted root certificate from vCenter to the trust store, you should be able to make a connection using .
After importing the root cert by placing it in /etc/ssl/certs on the master node, I'm able to provision a PVC while specifying a storage policy. Thanks for all the help.
Can you close this issue and work with KK to get this automated in the product?
It turns out that importing the root cert into the host was just a workaround, and the actual bug was fixed in That fix is included in govmomi v0.21.0.


Question: err := ds.checkpointManager.GetCheckpoint(podSandboxID, checkpoint) // Return empty portMappings if checkpoint is not found if err != nil { <del> if err == utilstore.ErrKeyNotFound { </del> <ins> if err == errors.ErrCheckpointNotFound { </ins> return nil, nil } <ins> errRem := ds.checkpointManager.RemoveCheckpoint(podSandboxID) if errRem != nil { glog.Errorf("Failed to delete corrupt checkpoint for sandbox %q: %v", podSandboxID, errRem) } </ins> return nil, err } _, _, _, checkpointedPortMappings, _ := checkpoint.GetData()

/kind bug are failing. seems like the likely cause. cc /assign
ping
seems one test still failing.


Question: "type": "array" } }, <del> "required": [ "sources" ], </del> "type": "object" }, "io.k8s.api.core.v1.QuobyteVolumeSource": {

What happened: I created a Pod with a ProjectedVolume with an empty array of , but Kubernetes transformed that value to . Some clients are happy with either value (e.g. Go) while others are not (e.g. Python). For example, the following is accepted and stored by Kubernetes, but cannot be read by the Python client: The field has been transformed to : Even a JSON patch has no effect: What you expected to happen: I expected the required (OpenAPI) field of core/v1.ProjectedVolumeSource to be returned as the empty array I submitted. How to reproduce it (as minimally and precisely as possible): Create the above Pod: See that has been transformed to : See that the officially-supported Python client is unable to deserialize the Pod: Anything else we need to know?: I recognize there's a lot of layers involved in the steps above. The ineffectiveness of the JSON patch is what makes me think the culprit is somewhere in the field spec or API somehow. /sig api-machinery Environment: Kubernetes version (use ): [official-clients]:
/remove-sig api-machinery /sig storage the issue is in the API definition, which does not indicate the sources field is optional (even though it is)
/cc


Question: "//pkg/client/metrics/prometheus:go_default_library", "//pkg/version/prometheus:go_default_library", "//plugin/cmd/kube-scheduler/app:go_default_library", <ins> "//vendor/k8s.io/apiserver/pkg/util/flag:go_default_library", </ins> "//vendor/k8s.io/apiserver/pkg/util/logs:go_default_library", ], )

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: I pull the latest code, and run with below cmd. But the cluster is not up. And I notice the kube-, What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): GitCommit:"" Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig cluster-lifecycle
/assign
Can you please try the fix above?
/assign
/unassign
your pr does not fix it yet? If not, I can help.
The fix from works for me! o/
ok, thanks
reopen , make less modifications.
sorry, i miss hyper...


Question: "type": "string" }, "namespaceSelector": { <del> "description": "NamespaceSelector decides whether to run the webhook on an object based on whether the namespace for that object matches the selector. If the object itself is a namespace, the matching is performed on object.metadata.labels. If the object is other cluster scoped resource, it is not subjected to the webhook.nnFor example, to run the webhook on any objects whose namespace is not associated with "runlevel" of "0" or "1";  you will set the selector as follows: "namespaceSelector": {n  "matchExpressions": [n    {n      "key": "runlevel",n      "operator": "NotIn",n      "values": [n        "0",n        "1"n      ]n    }n  ]n}nnIf instead you want to only run the webhook on any objects whose namespace is associated with the "environment" of "prod" or "staging"; you will set the selector as follows: "namespaceSelector": {n  "matchExpressions": [n    {n      "key": "environment",n      "operator": "In",n      "values": [n        "prod",n        "staging"n      ]n    }n  ]n}nnSee https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/ for more examples of label selectors.nnDefault to the empty LabelSelector, which matches everything.", </del> <ins> "description": "NamespaceSelector decides whether to run the webhook on an object based on whether the namespace for that object matches the selector. If the object itself is a namespace, the matching is performed on object.metadata.labels. If the object is another cluster scoped resource, it never skips the webhook.nnFor example, to run the webhook on any objects whose namespace is not associated with "runlevel" of "0" or "1";  you will set the selector as follows: "namespaceSelector": {n  "matchExpressions": [n    {n      "key": "runlevel",n      "operator": "NotIn",n      "values": [n        "0",n        "1"n      ]n    }n  ]n}nnIf instead you want to only run the webhook on any objects whose namespace is associated with the "environment" of "prod" or "staging"; you will set the selector as follows: "namespaceSelector": {n  "matchExpressions": [n    {n      "key": "environment",n      "operator": "In",n      "values": [n        "prod",n        "staging"n      ]n    }n  ]n}nnSee https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/ for more examples of label selectors.nnDefault to the empty LabelSelector, which matches everything.", </ins> "$ref": "#/definitions/io.k8s.apimachinery.pkg.apis.meta.v1.LabelSelector" }, "rules": {

If I am not mistaken, means that cluster-scoped resources other than namespaces are broken with admission webhooks. There is even a unit test checking that:
To clear any ambiguity around the word "broken", the current code behavior is that most cluster-scoped resources are not subjected to any webhooks. I wrote them like this because I was concerned with the potential bootstrapping deadlocks: a webhook that enacts policies for cluster-scoped resources might cause bootstrapping deadlocks for the cluster or for its own deployment. Thinking out loud, perhaps for cluster-scoped resources, the apiserver should only call the webhooks if the resources' labels match the , so that the critical bootstrapping resources can be excluded from the webhooks, thus breaking the bootstrapping deadlocks. This aligns with what we do for (see ). The downside is that the field name "namespaceSelector" becomes inappropriate.
I think we still need namespaceSelectors. We don't want to label pods. But I agree we actually want another objectLabelSelector. For namespaces these two coincide, i.e. we only have to define how they are applied if both are set or we forbid both at the same time. I tend to the later.
raised up a security issue regarding . A user can change labels to skip webhooks. doesn't have this issue because a user usually doesn't have permission to change the namespace's labels.
Users normally cannot create cluster-scoped objects either. But techincally it's a valid point. On the other hand, my main use-case are CRDs. They are not critical to the cluster health (maybe self-hosted clusters are different with etcd operator), i.e. you normally cannot shoot yourself into the foot with a webhook on them. For CRDs we do not need a selector at all. In other words: IMO we should enable webhooks for cluster-scoped objects, even if we have no in place.
Sounds reasonable. I'll create a PR ()
We would like to have the ability to apply the admission controller on specific namespaces only, and not on cluster level resources. This is useful if the list of possible resource Kinds that we want to apply the controller for is large, and we want to limit when it runs as much as possible to avoid affecting the whole cluster in case there's an issue.

I think this can be done by adding a SkipClusterResources flag (or something similar) in addition to the NamespaceSelector, to the Webhook struct. What do you think?
I think adding a flag is reasonable. It doesn't have the security issue as the idea does (). The expressiveness of is limited compared to , e..g., admins cannot use to run webhooks on some cluster-level resources while skipping others. However it's not a big drawback, because admins can specify more detailed GVK to limit the scope of webhook in that case.
Zeroth thought: s/SkipClusterResources/scope=[clusterboth]/g First thought: can't you just exclude cluster scoped resources, or include only namespace scoped resources in the webhook definition? Second thought: can a namespace selector be crafted that excludes the empty string?
0: Are you sure that's cleaner? The namespace selector picks which namespaces to use, and skipCluster decides whether to include cluster resources or not. Not setting it maintains current behavior (skipCluster=false) and whether cluster level resources are included is very clear. scope=cluster/namespace/both will require us to default to "both" which I find slightly more confusing. Also what to do if I specify cluster AND include namespace selector - is that an invalid spec or do we ignore the selector? 1: Yes. This is the current best thing we can do. If the list of resources is large though (for example, if I am implementing an alternative resource quota controller) then this solution requires us to maintain and update a long list of resources. This may be ok, since as long as our controller can't support certain types of resources it doesn't make sense to watch for them... 2: Doesn't look like it - the current code specifically checks if the attribute is a cluster level resource and skips the entire label matching process.
Re: 0: fair point, but let's at least name it in the positive: matchClusterScopedResources=true|false or something like that.
I proposed it in the negative to make it backwards compatible, so that "false" keeps current behavior. I agree that negatively named booleans are not not bad. I'll treat it as true if unspecified and use matchClusterScopedResources.
In today's semantic, such a namespace selector will exclude namespaced objects whose namespace has no label.


Question: }, } <del> o.DeleteFlags.AddFlags(cmd) </del> o.PrintFlags.AddFlags(cmd) o.RecordFlags.AddFlags(cmd)

The command provides a flag but this doesn't do anything. most likely the flag shouldn't exist. I have no idea how kustomize could be useful here to start a single pod. This creates a pod, and shows that is ignored: No response <details</details<detailsAzure, K3OS</details<detailsOSX 11.6.1 (Big Sur) </detailsNo response No response No response
/sig cli
This comes from the inclusion of DeleteFlags: brings in FilenameFlags: has Kustomize, resulting in the flag: I suspect there are many flags from DeleteFlags that don't really apply to . : Not sure what this would do : seems to be ignored and : doesn't seem to be used : doesn't seem to be used : also probably doesn't apply There might be others as well...
It seems to delete object which created by command.
I agree, As far as I can see, , and have this deleteFlags exposure.
/assign
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage accepted


Question: t.Errorf("error adding pod to meta: %v", err) } <ins> newState, err := getPreFilterState(cycleState) if err != nil { t.Errorf("failed to get preFilterState from cycleState: %v", err) } if !reflect.DeepEqual(newState.topologyToMatchedAntiAffinityTerms, test.expectedAntiAffinity) { t.Errorf("State is not equal, got: %v, want: %v", newState.topologyToMatchedAntiAffinityTerms, test.expectedAntiAffinity) } if !reflect.DeepEqual(newState.topologyToMatchedAffinityTerms, test.expectedAffinity) { t.Errorf("State is not equal, got: %v, want: %v", newState.topologyToMatchedAffinityTerms, test.expectedAffinity) } </ins> if !reflect.DeepEqual(allPodsState, state) { t.Errorf("State is not equal, got: %v, want: %v", state, allPodsState) }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Maybe it should be What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig Scheduling
I feel it is a valid error. But need confirmation from sig-scheduling maintainers. ... is it okay if I work on this?
ok
Yeah, this is a bug in preemption
/assign


Question: <ins> /* Copyright 2017 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package printers import ( "bytes" "fmt" "reflect" "testing" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" metav1alpha1 "k8s.io/apimachinery/pkg/apis/meta/v1alpha1" "k8s.io/apimachinery/pkg/runtime" "k8s.io/kubernetes/pkg/api" ) var testNamespaceColumnDefinitions = []metav1alpha1.TableColumnDefinition{ {Name: "Name", Type: "string", Format: "name", Description: metav1.ObjectMeta{}.SwaggerDoc()["name"]}, {Name: "Status", Type: "string", Description: "The status of the namespace"}, {Name: "Age", Type: "string", Description: metav1.ObjectMeta{}.SwaggerDoc()["creationTimestamp"]}, } func testPrintNamespace(obj *api.Namespace, options PrintOptions) ([]metav1alpha1.TableRow, error) { if options.WithNamespace { return nil, fmt.Errorf("namespace is not namespaced") } row := metav1alpha1.TableRow{ Object: runtime.RawExtension{Object: obj}, } row.Cells = append(row.Cells, obj.Name, obj.Status.Phase, "<unknow>") return []metav1alpha1.TableRow{row}, nil } func testPrintNamespaceList(list *api.NamespaceList, options PrintOptions) ([]metav1alpha1.TableRow, error) { rows := make([]metav1alpha1.TableRow, 0, len(list.Items)) for i := range list.Items { r, err := testPrintNamespace(&list.Items[i], options) if err != nil { return nil, err } rows = append(rows, r...) } return rows, nil } func TestPrintRowsForHandlerEntry(t *testing.T) { printFunc := reflect.ValueOf(testPrintNamespace) testCase := map[string]struct { h             *handlerEntry opt           PrintOptions obj           runtime.Object includeHeader bool expectOut     string expectErr     string }{ "no tablecolumndefinition and includeheader flase": { h: &handlerEntry{ columnDefinitions: []metav1alpha1.TableColumnDefinition{}, printRows:         true, printFunc:         printFunc, }, opt: PrintOptions{}, obj: &api.Namespace{ ObjectMeta: metav1.ObjectMeta{Name: "test"}, }, includeHeader: false, expectOut:     "testtt<unknow>n", }, "no tablecolumndefinition and includeheader true": { h: &handlerEntry{ columnDefinitions: []metav1alpha1.TableColumnDefinition{}, printRows:         true, printFunc:         printFunc, }, opt: PrintOptions{}, obj: &api.Namespace{ ObjectMeta: metav1.ObjectMeta{Name: "test"}, }, includeHeader: true, expectOut:     "ntesttt<unknow>n", }, "have tablecolumndefinition and includeheader true": { h: &handlerEntry{ columnDefinitions: testNamespaceColumnDefinitions, printRows:         true, printFunc:         printFunc, }, opt: PrintOptions{}, obj: &api.Namespace{ ObjectMeta: metav1.ObjectMeta{Name: "test"}, }, includeHeader: true, expectOut:     "NAMEtSTATUStAGEntesttt<unknow>n", }, "print namespace and withnamespace true, should not print header": { h: &handlerEntry{ columnDefinitions: testNamespaceColumnDefinitions, printRows:         true, printFunc:         printFunc, }, opt: PrintOptions{ WithNamespace: true, }, obj: &api.Namespace{ ObjectMeta: metav1.ObjectMeta{Name: "test"}, }, includeHeader: true, expectOut:     "", expectErr:     "namespace is not namespaced", }, } for name, test := range testCase { buffer := &bytes.Buffer{} err := printRowsForHandlerEntry(buffer, test.h, test.obj, test.opt, test.includeHeader) if err != nil { if err.Error() != test.expectErr { t.Errorf("[%s]expect:n %vn but got:n %vn", name, test.expectErr, err) } } if test.expectOut != buffer.String() { t.Errorf("[%s]expect:n %vn but got:n %vn", name, test.expectOut, buffer.String()) } } } </ins>

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: 1when error has occurwe should not print header 2 when i get ns use --all-namespaces, i get NAMESPACE column, we should return error: storageclass is not namespaced What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration**: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/assign
/sig cli


Question: status api.PodPhase test   string }{ <del> {&api.Pod{Spec: desiredState, Status: currentState}, api.PodPending, "waiting"}, </del> { &api.Pod{ Spec: desiredState,

I'm actually not sure if it's by design. But certainly replication controller doesn't behave like it expects this.
If restart policy is to restart always, they should be restarted, but in that case they shouldn't be reported as failed.
This is a known issue, and by design based on upper-level will observe the status reported from kubelet, but not check with RestartPolicy and generate PodCondition / PodStatus. When RestartPolicy is Always, even Kubelet reports all containers are at failed state, apiserver shouldn't mark Pod as failed state. This is why initially I want kubelet report Pod state / status / condiction.
As a quick fix, we could let apiserver populate the right state / status based on containerStatus and RestartPolicy for now. In a long term, I still think we should make kubelet smarter and atomic and self-contained component, not just a wrapper around docker.
Short term fix should be P1 then we can de-prio for a "realer" fix.
Yes, I am working on the short term fix, and should have it included in 0.6 cut.
are you still working on this?
Yes, I am working on this. Should finish it today.


Question: if cpuUsage <= 0 { framework.Failf("Pod %s/%s reported usage is %v, but it should be greater than 0", p.Namespace, p.Name, cpuUsage) } <del> if cpuUsage >= .5*1.05 { framework.Failf("Pod %s/%s reported usage is %v, but it should not exceed limit by > 5%%", p.Namespace, p.Name, cpuUsage) </del> <ins> // Jobs can potentially exceed the limit for a variety of reasons on Windows. // Softening the limit margin to 8% and allow a occasional overload. // https://github.com/kubernetes/kubernetes/issues/124643 if cpuUsage >= .8*1.05 { framework.Failf("Pod %s/%s reported usage is %v, but it should not exceed limit by > 8%%", p.Namespace, p.Name, cpuUsage) </ins> } } })

Failed error: Since migration to containerd 1.7.x No response The expected cpu usage is set to 0.5, but the actual cpu usage could be around 0.52-0.6, hence exceeding the limit more than the threshold of 5%. /sig windows
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /sig sig-windows Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig windows
cc:
/triage accepted I've reached out to some folks to see if the variance in the job limit is expected.
Jobs can potentially exceed the limit for a variety of reasons. If they do they get put into a deficit, so over time they would appear to be at the correct limit. Since we run for a short period of time we this is likely what we bumping into a scenario where the job happen to go over the limit. I would be in favor of softening the in this case since it appears we are just slightly over it occasionally. We can add a comment in the code that mention that it will even out over time.
Hi thank you for agreeing to soften the 5% check. Just check in to see the timeline for getting the limit updated? If needed, I can help with making the changes in a PR.
if you want to take a go at make this update, it would be appreciated!
/assign