Question: // finalURLTemplate is similar to URL(), but will make all specific parameter values equal // - instead of name or namespace, "{name}" and "{namespace}" will be used, and all query <del> // parameters will be reset. This creates a copy of the request so as not to change the // underlying object.  This means some useful request info (like the types of field // selectors in use) will be lost. // TODO: preserve field selector keys </del> <ins> // parameters will be reset. This creates a copy of the url so as not to change the // underlying object. </ins> func (r Request) finalURLTemplate() url.URL { <del> if len(r.resourceName) != 0 { r.resourceName = "{name}" } if r.namespaceSet && len(r.namespace) != 0 { r.namespace = "{namespace}" } </del> newParams := url.Values{} v := []string{"{value}"} for k := range r.params {

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: In Kubernetes 1.11.x, the controller-manager metrics are providing histogram records for every api endpoint ever called during namespace cleanup. For every namespace that is deleted since a controller has been the leader, 1000+ new lines of metrics are created / served indefinitely. When creating/deleting namespaces at a rate of 100/hour, that's 100,000+ new metrics lines being served for every request of controller-manager/metrics. spun up a 1.10 environment, and says it looks like the namespace names were replaced with {namespace}: In 1.11 this has changed (short clip of 1 histogram): What you expected to happen: Don't generate so many metrics. How to reproduce it (as minimally and precisely as possible): To replicate in 1.11.x, simply create and then delete a namespace and query the metrics endpoint of your controller-manager. Anything else we need to know?: Repro steps and bug reported by /sig api-machinery Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
did some more debugging and believes that the root cause is He says that PR switched to the new dynamic client so when a request is made, this function is called: He explains that function in the old client checks if is set, the following is an example call: In the new client, the following is an example call: In this new dynamic client, the delete operation calls: whereas in the old client, it ran: The , he says, is the critical component, which sets the request client namespace field without namespace being set, when is called then the metrics are populated without renaming the Namespace to which causes tons of unique metrics to be generated and cause metrics data grow very quickly. This seems to be a pretty critical bug. Will mark it for v1.12. Please triage. /milestone v1.12 CC
- friendly reminder that Code Freeze is next week; if this is critical enough for 1.12 it needs to be triaged accordingly.

This isn't a regression (1.11 had it), but it's a big enough bug that I'd take a fix for it in 1.12. I think that a fix should take the form of fixing the to handle cases where the abspath is set. That would fix it for all future callers instead of just a one off for this client. That method is only called for metrics and our api paths follow a very fixed format. I expect that it should be fairly easy to recognize and you only have to do the analysis if you have no hits for the namespace being set.
please apply a priority/critical-urgent label to keep track for 1.12
/assign
GitHub didn't allow me to assign the following users: wenjiaswe. Note that only and repo collaborators can be assigned. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I applied the label. We will have to un-apply it if a targeted fix doesn't show up very soon :)
If no one has started working on it, I am on this now. I will try fix it ASAP.
please do!
/assign
GitHub didn't allow me to assign the following users: wenjiaswe. Note that only and repo collaborators can be assigned. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
please update on a daily basis, as we're in Code Freeze. :)
I have reproduced the problem and also wrote a unit test case for confirmation. Basically with the current dynamic client implementation, the actually string of namespace is in url here: , so when finalURLTemplate function checks (), it returns false and would not replace namespace with "{namespace}". I am working on fixing it by replacing the url segment after /namespaces/ by . I should be able to submit the PR tomorrow if not today.


Question: IMAGE=staging-k8s.gcr.io/kube-addon-manager ARCH?=amd64 TEMP_DIR:=$(shell mktemp -d) <del> VERSION=v8.7 </del> <ins> VERSION=v8.8 </ins> KUBECTL_VERSION?=v1.10.7 <del> ifeq ($(ARCH),amd64) BASEIMAGE?=bashell/alpine-bash endif ifeq ($(ARCH),arm) BASEIMAGE?=arm32v7/debian endif ifeq ($(ARCH),arm64) BASEIMAGE?=arm64v8/debian endif ifeq ($(ARCH),ppc64le) BASEIMAGE?=ppc64le/debian endif ifeq ($(ARCH),s390x) BASEIMAGE?=s390x/debian endif </del> <ins> BASEIMAGE=k8s.gcr.io/debian-base-$(ARCH):0.3.2 </ins> .PHONY: build push

Is this a BUG REPORT or FEATURE REQUEST?: /kind feature Alpine fixed a RCE vulnerability in apk in 3.8.1 () Update the base image used for addon-manager () to pick it up. Note: I'd send a PR if it depended directly on alpine image, not bashell/alpine.
There are no sig labels on this issue. Please add a sig label by either: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. See the . The in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assing
Ping


Question: name: new-kms-provider-for-configmaps cachesize: 1000 endpoint: unix:///@new-kms-provider.sock <ins> - resources: - '*.*' providers: - kms: name: kms-provider-to-encrypt-all cachesize: 1000 endpoint: unix:///@new-encrypt-all-kms-provider.sock - identity: {} </ins> ` <ins> // start new KMS Plugin _ = mock.NewBase64Plugin(t, "@new-encrypt-all-kms-provider.sock") </ins> // update encryption config and wait for hot reload if err := os.WriteFile(filepath.Join(test.configDir, encryptionConfigFileName), []byte(encryptionConfigWithoutOldProvider), 0644); err != nil { t.Fatalf("failed to update encryption config, err: %v", err) } <ins> wantPrefixForEncryptAll := "k8s:enc:kms:v1:kms-provider-to-encrypt-all:" </ins> // wait for config to be observed <del> verifyIfKMSTransformersSwapped(t, wantPrefixForSecrets, test) </del> <ins> verifyIfKMSTransformersSwapped(t, wantPrefixForSecrets, wantPrefixForEncryptAll, test) </ins> // confirm that reading secrets still works _, err = test.restClient.CoreV1().Secrets(testNamespace).Get(

transformation expandless | 14m1s Failed; === RUN TestEncryptionConfigHotReload TestEncryptionConfigHotReload When I see this flake, 10-09 is the first one.(But the job history in the testgrid only started from 10-03) some PR that may be related: /sig auth /cc
/cc
/triage accepted
/cc failed two times in recent 3 runs:
/reopen as of this is still .
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: } } <del> output, err := exec.Command("cmd", "/c", "mklink", "/D", target, bindSource).CombinedOutput() </del> <ins> // There is an issue in golang where EvalSymlinks fails on Windows when passed a // UNC share root path without a trailing backslash. // Ex: SERVERshare will fail to resolve but SERVERshare will resolve // containerD on Windows calls EvalSymlinks so we'll add the backslash when making the symlink if it is missing. // https://github.com/golang/go/pull/42096 fixes this issue in golang but a fix will not be available until // golang v1.16 mklinkSource := bindSource if !strings.HasSuffix(mklinkSource, "") { mklinkSource = mklinkSource + "" } output, err := exec.Command("cmd", "/c", "mklink", "/D", target, mklinkSource).CombinedOutput() </ins> if err != nil { <del> klog.Errorf("mklink failed: %v, source(%q) target(%q) output: %q", err, bindSource, target, string(output)) </del> <ins> klog.Errorf("mklink failed: %v, source(%q) target(%q) output: %q", err, mklinkSource, target, string(output)) </ins> return err } <del> klog.V(2).Infof("mklink source(%q) on target(%q) successfully, output: %q", bindSource, target, string(output)) </del> <ins> klog.V(2).Infof("mklink source(%q) on target(%q) successfully, output: %q", mklinkSource, target, string(output)) </ins> return nil }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Deploy a Windows cluster with containerd configured, SMB mount does not work by trying following project: It reports following error: What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): v1.18.6 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /sig windows /sig storage
/cc
I was able to repro your issue using the azurefile-csi-driver for Windows. I also tried to set up a repo where I would mount an azure file share to a Windows node manually, create symlinks to the share and then schedule a pod to the node using a hostPath volume and the container was able to start up fine using contianerd. Here is the pod-spec i used I tried to mount the Azure file share with both New-PsDrive and New-SmbGlobalMapping and saw date being written the the in the azure file share in both instances. I used mklink /D to create the symlinks - and also tried creating a symlink chain because it looks like that is how they get set up when when using azurefile-csi-driver and csi-proxy Right now I think this issue may be specific to either how the csi-proxy is setting up the SMB share for Windows or how paths are getting passed around between the various components.
this is not related to csi-proxy, in-tree azure file does not use csi-proxy, and it does not work
hostPath on containerd Windows does not work too:
I captured containerd logs for this and see I spoke with and he thinks this may be related to a known issue where go-lang is getting confused on Windows if a symlink points directly to the root of a SBM share (which is does in this case). He suggested adding a trailing  to see if this fixes the issue. I will try that out now and report back.

I tested some updates to CSI proxy to ensure the symlink to the sharepath had a trailing backslash and it looks like that resolved the issue. On my node running contianerd i have the following symlinks Previously the  on pvc--f195-4d9e-841b- was missing.
The core issue here is that containerd calls on the mount source path, and that function has a bug which prevents it from working when the path is a UNC share root. If the path has a trailing appended, then works properly. I have filed a bug on Go to fix this, and am working on a fix: golang/go
/close
Closing this issue. <detailsIn response to : /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is not resolved, right?
I see this issue is resolved in csi-proxy v0.2.2, while it's not resolved for the in-tree azure file driver.
/triage accepted
Ah, good point. yes, this is not fully resolved.


Question: }, }, { <ins> name: "Pod copy: explicit attach", args: "mypod --image=busybox --copy-to=my-debugger --attach -- sleep 1d", wantOpts: &DebugOptions{ Args:           []string{"sleep", "1d"}, Attach:         true, CopyTo:         "my-debugger", Image:          "busybox", Namespace:      "default", ShareProcesses: true, TargetNames:    []string{"mypod"}, }, }, { </ins> name: "Pod copy: replace single image of existing container", args: "mypod --image=busybox --container=my-container --copy-to=my-debugger", wantOpts: &DebugOptions{

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Using without specifying should modify existing container images without trying to create a new one. Instead it tries to create a new container and returns an error that isn't set: What you expected to happen: This command should change all container images to instead of returning an error. /sig cli /kind bug /priority important-soon
/assign /triage accepted
My kubernetes cluster version is 1.19.0,and i use kubectl-debug version 0.1.1 try to debug my pod,bug i got error: <img width="1470" alt="image" src="https://user-"Who one know about that


Question: } // generateReadBlockCmd generates the corresponding command lines to read from a block device with the given file path. <del> func generateReadBlockCmd(fullPath string, numberOfCharacters int) []string { </del> <ins> func GenerateReadBlockCmd(fullPath string, numberOfCharacters int) []string { </ins> var commands []string commands = []string{"head", "-c", strconv.Itoa(numberOfCharacters), fullPath} return commands

<!-- Feature requests are unlikely to make progress as an issue. Instead, please suggest enhancements by engaging with SIGs on slack and mailing lists. A proposal that works through the design along with the implications of the change can be opened as a KEP: --In block volume tests were skipped because is not using , there are more details about how these functions are used in This is a test that uses the , if tested using the it should have , in addition the test cluster should have the Things that are needed for a first time contributor a development kubernetes cluster, preferably on a public cloud instance although I'm not sure if kind can be used a CSI driver like PD CSI Driver, the storage class should be setup to use the xfs fstype the volume snapshot CRDs and controllers installed in the cluster (kubernetes-csi/external-snapshotter#usage) /sig storage /area testing /good-first-issue
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : <!-- Feature requests are unlikely to make progress as an issue. --/area testing /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : #### What would you like to be : /area testing /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hey team! I'd like to further investigate on this issue if possible?

/assign
/area test hey sure, I've more info about how to test it, it requires these components: a development kubernetes cluster, preferably on a public cloud instance although I'm not sure if can be used a CSI driver like , the storage class should be setup to use the fstype the volume snapshot CRDs and controllers installed in the cluster ()
thanks a lot !! I might be a bit slow though, I'm pretty new to the community. I appreciate the documentation you shared, the assignation and how to replicate it, I'll test it out and if I have some doubts, I'll get back with you :D
Hey are you still working on this issue? I am interested in this
Yes, I'm still working on this, but would totally be open to doing pair-programming if you're open to it? I would benefit from you for solving the problem since I'm still pretty new to the codebase and I'm still understanding the environment.
Hey Are you still working on this, I would like to do pair-programming if possible.
Hi there definitely!! Ping me out in slack in the Kubernetes workspace with "Diego Goytia"


Question: if !ok { cacheSize = f.Options.DefaultWatchCacheSize } <ins> // depending on cache size this might return an undecorated storage </ins> ret.Decorator = genericregistry.StorageWithCacher(cacheSize) }

Currently we signal undecorated storage by simply affecting the undecoratedStorage to the decorator. This is a blocker for kubernetes/enhancements as we need to enable watch cache for events (which currently hardcoded and not configurable) cc /assign /sig api-machinery


Question: <ins> #!/bin/bash # Copyright 2014 Google Inc. All rights reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # #     http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # Build the fluentd-elasticsearch image and push # to google/fluentd-elasticsearch. sudo docker build -t kubernetes/fluentd-elasticsearch . sudo docker push kubernetes/fluentd-elasticsearch </ins>

After PR , I did a bit experiments on kube-dns graceful termination and got some unintended result as below: I guess it is because the other container was terminated before . DNS query from clients failed because is the actual connection point. After a bit discussion with and two solutions for this are: another program wrap up the dnsmasq and capture the SIGTERM signal. out a way to modify the sig mask of process without touching the original codes.
Something I have been meaning to try: Put in Dockerfile Build and push image We should deliver SIGCONT rather than SIGTERM. I did a quick test and it seems to work. Otherwise trapping SIGTERM in the shell and ignoring it is well documented... , Zihong Zheng wrote:
Thanks, I will take a look and test these.


Question: <ins> /* Copyright 2019 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package e2e_node import ( "fmt" "os" "time" "k8s.io/api/core/v1" "k8s.io/apimachinery/pkg/api/resource" "k8s.io/apimachinery/pkg/util/uuid" kubeapi "k8s.io/kubernetes/pkg/apis/core" kubeletconfig "k8s.io/kubernetes/pkg/kubelet/apis/config" evictionapi "k8s.io/kubernetes/pkg/kubelet/eviction/api" "k8s.io/kubernetes/test/e2e/framework" . "github.com/onsi/ginkgo" . "github.com/onsi/gomega" ) var _ = framework.KubeDescribe("SystemNodeCriticalPod [Slow] [Serial] [Disruptive] [NodeFeature:SystemNodeCriticalPod]", func() { f := framework.NewDefaultFramework("system-node-critical-pod-test") // this test only manipulates pods in kube-system f.SkipNamespaceCreation = true Context("when create a system-node-critical pod", func() { tempSetCurrentKubeletConfig(f, func(initialConfig *kubeletconfig.KubeletConfiguration) { diskConsumed := resource.MustParse("200Mi") summary := eventuallyGetSummary() availableBytes := *(summary.Node.Fs.AvailableBytes) initialConfig.EvictionHard = map[string]string{string(evictionapi.SignalNodeFsAvailable): fmt.Sprintf("%d", availableBytes-uint64(diskConsumed.Value()))} initialConfig.EvictionMinimumReclaim = map[string]string{} }) // Place the remainder of the test within a context so that the kubelet config is set before and after the test. Context("", func() { var staticPodName, mirrorPodName, podPath string ns := kubeapi.NamespaceSystem BeforeEach(func() { By("create a static system-node-critical pod") staticPodName = "static-disk-hog-" + string(uuid.NewUUID()) mirrorPodName = staticPodName + "-" + framework.TestContext.NodeName podPath = framework.TestContext.KubeletConfig.StaticPodPath // define a static pod consuming disk gradually // the upper limit is 1024 (iterations) * 10485760 bytes (10MB) = 10GB err := createStaticSystemNodeCriticalPod( podPath, staticPodName, ns, busyboxImage, v1.RestartPolicyNever, 1024, "dd if=/dev/urandom of=file${i} bs=10485760 count=1 2>/dev/null; sleep .1;", ) Expect(err).ShouldNot(HaveOccurred()) By("wait for the mirror pod to be running") Eventually(func() error { return checkMirrorPodRunning(f.ClientSet, mirrorPodName, ns) }, time.Minute, time.Second*2).Should(BeNil()) }) It("should not be evicted upon DiskPressure", func() { By("wait for the node to have DiskPressure condition") Eventually(func() error { if hasNodeCondition(f, v1.NodeDiskPressure) { return nil } msg := fmt.Sprintf("NodeCondition: %s not encountered yet", v1.NodeDiskPressure) framework.Logf(msg) return fmt.Errorf(msg) }, time.Minute*2, time.Second*4).Should(BeNil()) By("check if it's running all the time") Consistently(func() error { err := checkMirrorPodRunning(f.ClientSet, mirrorPodName, ns) if err == nil { framework.Logf("mirror pod %q is running", mirrorPodName) } else { framework.Logf(err.Error()) } return err }, time.Minute*8, time.Second*4).ShouldNot(HaveOccurred()) }) AfterEach(func() { By("delete the static pod") err := deleteStaticPod(podPath, staticPodName, ns) Expect(err).ShouldNot(HaveOccurred()) By("wait for the mirror pod to disappear") Eventually(func() error { return checkMirrorPodDisappear(f.ClientSet, mirrorPodName, ns) }, time.Minute, time.Second*2).Should(BeNil()) }) }) }) }) func createStaticSystemNodeCriticalPod(dir, name, namespace, image string, restart v1.RestartPolicy, iterations int, command string) error { template := ` apiVersion: v1 kind: Pod metadata: name: %s namespace: %s spec: priorityClassName: system-node-critical containers: - name: %s image: %s restartPolicy: %s command: ["sh", "-c", "i=0; while [ $i -lt %d ]; do %s i=$(($i+1)); done; while true; do sleep 5; done"] ` file := staticPodPath(dir, name, namespace) podYaml := fmt.Sprintf(template, name, namespace, name, image, string(restart), iterations, command) f, err := os.OpenFile(file, os.O_RDWR|os.O_TRUNC|os.O_CREATE, 0666) if err != nil { return err } defer f.Close() _, err = f.WriteString(podYaml) return err } </ins>

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: Eviction manager evicts the below static and critical pod when the node observed MemoryPressure. The k8s server version is v1.13.2, and the pod priority is enabled. As far as I can see, the eviction manager does not evict such pods. What you expected to happen: Static pods with are not deleted as well as those with a critical pod annotation. How to reproduce it (as minimally and precisely as possible): pod with system-node-critical priority using by kubelet option memory load to the target node until eviction manager observes memory pressure. critical and static pod is evicted. Anything else we need to know?: When I set critical-pod annotation (and enables ) insted of pod priority, the eviction manager DOES NOT evict the static and critical pod. Environment: Kubernetes version (use ): v1.13.2 Cloud provider or hardware configuration: baremetal
/sig node
from enable priority is , more than I can not understand why it has happened
Right. It seems the expected behavior and the actual behavior are different. will remove next k8s v1.14 (it's written in the document), but I'd like to wait for it until the critical priority can prevent eviction.
I see this as well. Start from the bottom and read up. It ranks ibm-master-proxy-static which is a static pod that has priority system-node-critical. It actually is always ranked first as well.
It can be replicated by causing disk pressure (using ) And then viewing the eviction manager logs. `journalctl -u kubelet | grep eviction There's also the piece that when the static pod is evicted kubelet never ends up starting it back up again.
Enabling ExperimentalCriticalPodAnnotation=true makes it not fail and detect properly
Setting on the static pod also works. It appears that a static pod doesn't get set based on . If this is expected then it should be documented. Although it seems like the priority class name should be honored for static pods.
I've confirmed that the problem occurs when a static pod has set to but does not set accordingly.

doesn't see the priority as being set and thus thinks the pod isn't critical which ultimately causes the eviction. I would have expected to translate to .
The code fix is available in master branch right now, which means it will be auto included in the upcoming 1.14 release. For lower releases, you need to wait for a while before cherrypick PRs get approved and merged.
Cherrypick PRs have been created and waiting for approval. 1.13 - 1.12 - 1.11 -
Your patch does not work for me. The case in that you completely replaces static pods with the mirror pods. But static pods contains important annotations such as . For static pods is , for mirror - . As a result, eviction manager supposes that pod is not static.
I doubt it. A mirror pod should be annotated with : Otherwise the PR won't pass the e2e test.
When I get static pod through I see . But when I try to debug eviction manager, I see that pod has annotation
Thanks for the info. Then we are sort of in a dilemma - on one hand annotation is needed to check if it's a static pod, on the other hand we also need the corresponding mirror pod to get the priority value. I will provide a supplemental fix later.
can we please hold on those cherry-picks until the supplemental fix is also available?
sure.
Here is a detailed explanation on why previous PR passed the e2e test: As , the critical static pod failed on IsStaticPod() check (because of mirror pod is tagged as " api") And it's reasonable that it's being marked as "to be deleted": However, as , there is some weird log showing: It's because the pod we passed onto () is a mirror pod, and it's deleted (or probably a dummy delete) by container runtime without any error. To take an analogy, it's sort of like a "controllee" (mirror pod) gets deleted, but "controller" (static pod) is still alive, so it causes data inconsistency, and the above error msg popped up. And in user's perspective, the static pod is "not" evicted, then it "looks" like previous PR works.
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available .

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Status of cherry pick PRs: 1.13 - - merged 1.12 - - merged 1.11 - - merged
/close As all PRs have been cherrypicked.
Closing this issue. <detailsIn response to : /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
When will this be released for 1.11.x? We are getting this same issue with and believe it's related: The api log:
Maybe in v1.11.10 (but I didn't see it in v1.11.10-beta.0 yet)
Thanks, I assume this is pretty high priority if it's causing to be evicted?
Yes. Prior to get it in 1.11.x, you can work it around by manually setting the priority value (not only setting priority name) to the control plane pods (apiserver, controller-manager and scheduler).
I think the priority value is correct:
Did you get that from ? If so, that's not sufficient. You should work around by hardcoding it into your static pod yaml file.
I did get it from , will try hardcoding. Thanks!
Were you successful ? I'm running into this same problem (unfortunately on a 1.10 cluster, which doesn't have this patch) and received the error: Looks like it comes from
I see this as well but the actual disk is not even full.


Question: kubectl replace "${kube_flags[@]}" --force -f /tmp/tmp-valid-pod.json # Post-condition: spec.container.name = "replaced-k8s-serve-hostname" kube::test::get_object_assert 'pod valid-pod' "{{(index .spec.containers 0).name}}" 'replaced-k8s-serve-hostname' <ins> ## check replace --grace-period requires --force output_message=$(! kubectl replace "${kube_flags[@]}" --grace-period=1 -f /tmp/tmp-valid-pod.json 2>&1) kube::test::if_has_string "${output_message}" '--grace-period must have --force specified' ## check replace --timeout requires --force output_message=$(! kubectl replace "${kube_flags[@]}" --timeout=1s -f /tmp/tmp-valid-pod.json 2>&1) kube::test::if_has_string "${output_message}" '--timeout must have --force specified' </ins> #cleaning rm /tmp/tmp-valid-pod.json

When replacing an existing pod, the command fails as it tries to create a pod with a name that is still used by terminating pod. How does the option works? deletes a pod (e.g. mypod) creates a new pod with the same name (mypod) What happens? As the command tries to recreate the pod, it fails as the is in state. Thus it can not be created. The problem is the does not wait for a pod to be deleted. Reproducer: run mypod --image=yapei/hello-openshift --generator=run-pod/v1 get pod mypod -o yaml replace -f --force The last command ends with:
option introduce by
With set to positive value, what should be the default behaviour of the command? When set to , should the command hang for and then create the pod? Or is there a way to create a callback (inside a scheduler?) that creates the pod once the current one is deleted?
You could watch for the resource to be deleted and create after you see the deletion happen
have you ever tried to set , I think this will do a non graceful deletion. should make sure the old resource is deleted before it creates a new resource, if not, that's a bug.
Though the help says the is to be used only with , it returns the error above.
Then I think there are two issues we need to track: 1. option should be ignored or just returns error if users are not using option. 2. should always make sure the old resource has been deleted before create the new resource.


Question: eventsRule(), }, }, <ins> { // a role making the csrapprover controller approve a node client CSR ObjectMeta: metav1.ObjectMeta{Name: "system:certificates.k8s.io:certificatesigningrequests:nodeclient"}, Rules: []rbac.PolicyRule{ rbac.NewRule("create").Groups(certificatesGroup).Resources("certificatesigningrequests/nodeclient").RuleOrDie(), }, }, { // a role making the csrapprover controller approve a node client CSR requested by the node itself ObjectMeta: metav1.ObjectMeta{Name: "system:certificates.k8s.io:certificatesigningrequests:selfnodeclient"}, Rules: []rbac.PolicyRule{ rbac.NewRule("create").Groups(certificatesGroup).Resources("certificatesigningrequests/selfnodeclient").RuleOrDie(), }, }, } if utilfeature.DefaultFeatureGate.Enabled(features.RotateKubeletServerCertificate) { roles = append(roles, rbac.ClusterRole{ // a role making the csrapprover controller approve a node server CSR requested by the node itself ObjectMeta: metav1.ObjectMeta{Name: "system:certificates.k8s.io:certificatesigningrequests:selfnodeserver"}, Rules: []rbac.PolicyRule{ rbac.NewRule("create").Groups(certificatesGroup).Resources("certificatesigningrequests/selfnodeserver").RuleOrDie(), }, }) </ins> } addClusterRoleLabel(roles) return roles }

/kind feature From: Would it make sense to add these rules to the bootstrappolicy? To me they seem generic enough, and they wouldn't bind to something. Consumers can do the binding to their subjects as they like. I'm asking because it's already duplicated twice for two early adopters: I don't think those ClusterRoles are specific to kubeadm and GKE at all; but the ClusterRoleBindings will surely be, so therefore we'd create the roles but not the bindings.
Wouldn't we expect and encourage people using the CSR capability for their nodes to use node identities that match the node authorizer, in which case we won't need the client or server renewal CSR roles?
The node authorizer doesn't grant these, since otherwise you could never prevent a compromised node from renewing its credentials
Realistically, with these roles you still wouldn't. People would grant them to the node group, not each individual node, but I guess that would be a tradeoff for the deployment mechanism.
You could individually revoke this role from the group and stop renewals without disrupting regular operation
I can definitely see myself revoking a binding from to multiple specific bindings in case of a compromise. Anyway, this issue is more "should we create these by default or not". I've already mentioned my opinion.
Creating the roles seems reasonable. I don't think I want bindings. I think the binding should be up the particular deployer.
Perfect, that's exactly my proposal. I'll go ahead and make this change...
/assign
are you working on this or should I send a PR?


Question: framework.ExpectNoError(err, "failed to list DaemonSets") framework.ExpectEqual(len(dsList.Items), 0, "filtered list should have no daemonset") }) <ins> ginkgo.It("should verify changes to a daemon set status", func() { label := map[string]string{daemonsetNameLabel: dsName} labelSelector := labels.SelectorFromSet(label).String() dsClient := f.ClientSet.AppsV1().DaemonSets(ns) cs := f.ClientSet w := &cache.ListWatch{ WatchFunc: func(options metav1.ListOptions) (watch.Interface, error) { options.LabelSelector = labelSelector return dsClient.Watch(context.TODO(), options) }, } dsList, err := cs.AppsV1().DaemonSets("").List(context.TODO(), metav1.ListOptions{LabelSelector: labelSelector}) framework.ExpectNoError(err, "failed to list Daemon Sets") ginkgo.By(fmt.Sprintf("Creating simple DaemonSet %q", dsName)) testDaemonset, err := c.AppsV1().DaemonSets(ns).Create(context.TODO(), newDaemonSetWithLabel(dsName, image, label), metav1.CreateOptions{}) framework.ExpectNoError(err) ginkgo.By("Check that daemon pods launch on every node of the cluster.") err = wait.PollImmediate(dsRetryPeriod, dsRetryTimeout, checkRunningOnAllNodes(f, testDaemonset)) framework.ExpectNoError(err, "error waiting for daemon pod to start") err = checkDaemonStatus(f, dsName) framework.ExpectNoError(err) ginkgo.By("Getting /status") dsResource := schema.GroupVersionResource{Group: "apps", Version: "v1", Resource: "daemonsets"} dsStatusUnstructured, err := f.DynamicClient.Resource(dsResource).Namespace(ns).Get(context.TODO(), dsName, metav1.GetOptions{}, "status") framework.ExpectNoError(err, "Failed to fetch the status of daemon set %s in namespace %s", dsName, ns) dsStatusBytes, err := json.Marshal(dsStatusUnstructured) framework.ExpectNoError(err, "Failed to marshal unstructured response. %v", err) var dsStatus appsv1.DaemonSet err = json.Unmarshal(dsStatusBytes, &dsStatus) framework.ExpectNoError(err, "Failed to unmarshal JSON bytes to a daemon set object type") framework.Logf("Daemon Set %s has Conditions: %v", dsName, dsStatus.Status.Conditions) ginkgo.By("updating the DaemonSet Status") var statusToUpdate, updatedStatus *appsv1.DaemonSet err = retry.RetryOnConflict(retry.DefaultRetry, func() error { statusToUpdate, err = dsClient.Get(context.TODO(), dsName, metav1.GetOptions{}) framework.ExpectNoError(err, "Unable to retrieve daemon set %s", dsName) statusToUpdate.Status.Conditions = append(statusToUpdate.Status.Conditions, appsv1.DaemonSetCondition{ Type:    "StatusUpdate", Status:  "True", Reason:  "E2E", Message: "Set from e2e test", }) updatedStatus, err = dsClient.UpdateStatus(context.TODO(), statusToUpdate, metav1.UpdateOptions{}) return err }) framework.ExpectNoError(err, "Failed to update status. %v", err) framework.Logf("updatedStatus.Conditions: %#v", updatedStatus.Status.Conditions) ginkgo.By("watching for the daemon set status to be updated") ctx, cancel := context.WithTimeout(context.Background(), dsRetryTimeout) defer cancel() _, err = watchtools.Until(ctx, dsList.ResourceVersion, w, func(event watch.Event) (bool, error) { if ds, ok := event.Object.(*appsv1.DaemonSet); ok { found := ds.ObjectMeta.Name == testDaemonset.ObjectMeta.Name && ds.ObjectMeta.Namespace == testDaemonset.ObjectMeta.Namespace && ds.Labels[daemonsetNameLabel] == dsName if !found { framework.Logf("Observed daemon set %v in namespace %v with annotations: %v & Conditions: %v", ds.ObjectMeta.Name, ds.ObjectMeta.Namespace, ds.Annotations, ds.Status.Conditions) return false, nil } for _, cond := range ds.Status.Conditions { if cond.Type == "StatusUpdate" && cond.Reason == "E2E" && cond.Message == "Set from e2e test" { framework.Logf("Found daemon set %v in namespace %v with labels: %v annotations: %v & Conditions: %v", ds.ObjectMeta.Name, ds.ObjectMeta.Namespace, ds.ObjectMeta.Labels, ds.Annotations, ds.Status.Conditions) return found, nil } framework.Logf("Observed daemon set %v in namespace %v with annotations: %v & Conditions: %v", ds.ObjectMeta.Name, ds.ObjectMeta.Namespace, ds.Annotations, ds.Status.Conditions) } } object := strings.Split(fmt.Sprintf("%v", event.Object), "{")[0] framework.Logf("Observed %v event: %+v", object, event.Type) return false, nil }) framework.ExpectNoError(err, "failed to locate daemon set %v in namespace %v", testDaemonset.ObjectMeta.Name, ns) framework.Logf("Daemon set %s has an updated status", dsName) ginkgo.By("patching the DaemonSet Status") daemonSetStatusPatch := appsv1.DaemonSet{ Status: appsv1.DaemonSetStatus{ Conditions: []appsv1.DaemonSetCondition{ { Type:   "StatusPatched", Status: "True", }, }, }, } payload, err := json.Marshal(daemonSetStatusPatch) framework.ExpectNoError(err, "Failed to marshal JSON. %v", err) _, err = dsClient.Patch(context.TODO(), dsName, types.MergePatchType, payload, metav1.PatchOptions{}, "status") framework.ExpectNoError(err, "Failed to patch daemon set status", err) ginkgo.By("watching for the daemon set status to be patched") ctx, cancel = context.WithTimeout(context.Background(), dsRetryTimeout) defer cancel() _, err = watchtools.Until(ctx, dsList.ResourceVersion, w, func(event watch.Event) (bool, error) { if ds, ok := event.Object.(*appsv1.DaemonSet); ok { found := ds.ObjectMeta.Name == testDaemonset.ObjectMeta.Name && ds.ObjectMeta.Namespace == testDaemonset.ObjectMeta.Namespace && ds.Labels[daemonsetNameLabel] == dsName if !found { framework.Logf("Observed daemon set %v in namespace %v with annotations: %v & Conditions: %v", ds.ObjectMeta.Name, ds.ObjectMeta.Namespace, ds.Annotations, ds.Status.Conditions) return false, nil } for _, cond := range ds.Status.Conditions { if cond.Type == "StatusPatched" { framework.Logf("Found daemon set %v in namespace %v with labels: %v annotations: %v & Conditions: %v", ds.ObjectMeta.Name, ds.ObjectMeta.Namespace, ds.ObjectMeta.Labels, ds.Annotations, ds.Status.Conditions) return found, nil } framework.Logf("Observed daemon set %v in namespace %v with annotations: %v & Conditions: %v", ds.ObjectMeta.Name, ds.ObjectMeta.Namespace, ds.Annotations, ds.Status.Conditions) } } object := strings.Split(fmt.Sprintf("%v", event.Object), "{")[0] framework.Logf("Observed %v event: %v", object, event.Type) return false, nil }) framework.ExpectNoError(err, "failed to locate daemon set %v in namespace %v", testDaemonset.ObjectMeta.Name, ns) framework.Logf("Daemon set %s has a patched status", dsName) }) </ins> }) // randomPod selects a random pod within pods that causes fn to return true, or nil

[x] test approval issue : [X] test pr : 26 May [x] two weeks soak end date : 9 June [x] test promotion pr : kubernetes/kubernetes According to this APIsnoop query, there are still some remaining DaemonSet endpoints which are untested. Create a watch to track daemon set events that DaemonSet with a static label Create a daemon set with a static label. Confirm that the pods are running. Get the daemon set status. Parse the response and confirm that the daemon set status conditions can be listed. Update the daemon set status. Confirm via the watch that the status has been updated. Patch the daemon set status. Confirm via the watch that the status has been patched. Using an existing as a template for a new which validates that three new endpoints are hit. This query shows the endpoints hit within a short period of running the e2e test If a test with these calls gets merged, test coverage will go up by 3 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: framework.Logf("Sleeping 10 seconds and crossing our fingers that scheduler will run in that time.") time.Sleep(10 * time.Second) <del> verifyResult(c, labelPodName, 2, 0, ns) </del> <ins> verifyResult(c, labelPodName, 3, 1, ns) </ins> }) // test the pod affinity successful matching scenario with multiple Label Operators.

https://k8s- Failed: [] SchedulerPredicates [Serial] validates that InterPodAntiAffinity is respected if matching 2 {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that InterPodAntiAffinity is respected if matching 2 {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that InterPodAntiAffinity is respected if matching 2 {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that InterPodAntiAffinity is respected if matching 2 {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that InterPodAntiAffinity is respected if matching 2 {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that InterPodAntiAffinity is respected if matching 2 {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that InterPodAntiAffinity is respected if matching 2 {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that InterPodAntiAffinity is respected if matching 2 {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that InterPodAntiAffinity is respected if matching 2 {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that InterPodAntiAffinity is respected if matching 2 {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that InterPodAntiAffinity is respected if matching 2 {Kubernetes e2e suite}
Looking into this...
trying to track this flake down. It looks like the pod is launched despite AntiAffinity in . In my local setup I saw the same, but noticed that the new admission controller was not activate.

I have activate it and now get this:
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that InterPodAntiAffinity is respected if matching 2 {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that InterPodAntiAffinity is respected if matching 2 {Kubernetes e2e suite}
Should be fixed by


Question: docker_pull( name = "debian-hyperkube-base-amd64", <del> digest = "sha256:1a05a58432254268c31ef5c8d9c21f3d01a40611b14707de6ac2772c0793bd13", </del> <ins> digest = "sha256:fc1b461367730660ac5a40c1eb2d1b23221829acf8a892981c12361383b3742b", </ins> registry = "gcr.io", repository = "google-containers/debian-hyperkube-base-amd64", <del> tag = "0.7",  # ignored, but kept here for documentation </del> <ins> tag = "0.8",  # ignored, but kept here for documentation </ins> ) docker_pull(

the new hyperkube image has changed without notifying the bash environment now links to sh, there is only one thing worst than an error, and it is a SILENT ERROR this means that anyone that execute kubectl inside the hyperkube and leverage from a bash environment now get the weirdest exceptions , in my case troubles where caused by flexvolume scripted in bash which stopped working without reason and with weird errors logged ( but also there are some other cluster-images based on your hyperkube image, and if you link bash to sh it will spread headache around :) ) while for version 1.7.7 the result was different so , or remove bash, or repristinate bash, link bash-sh is a NO-NO
/assign /sig node


Question: <ins> # This configuration file for Fluentd / td-agent is used # to watch changes to Docker log files that live in the # directory /var/lib/docker/containers/ which are then submitted to # Elasticsearch (running on the machine ES_HOST:9200) which # assumes the installation of the fluentd-elasticsearch plug-in. # See https://github.com/uken/fluent-plugin-elasticsearch for # more information about the plug-in. This file needs to be # patched to replace ES_HOST with the name of the actual # machine running Elasticsearch. # Maintainer: Satnam Singh (satnam@google.com) # # Exampe # ====== # A line in the Docker log file might like like this JSON: # # {"log":"2014/09/25 21:15:03 Got request with path wombatn", #  "stream":"stderr", #   "time":"2014-09-25T21:15:03.499185026Z"} # # The time_format specification below makes sure we properly # parse the time format produced by Docker. This will be # submitted to Elasticsearch and should appear like: # $ curl 'http://elasticsearch:9200/_search?pretty' # ... # { #      "_index" : "logstash-2014.09.25", #      "_type" : "fluentd", #      "_id" : "VBrbor2QTuGpsQyTCdfzqA", #      "_score" : 1.0, #      "_source":{"log":"2014/09/25 22:45:50 Got request with path wombatn", #                 "stream":"stderr","tag":"docker.container.all", #                 "@timestamp":"2014-09-25T22:45:50+00:00"} #    }, # ... <source> type tail format json time_key time path /var/lib/docker/containers/*/*-json.log time_format %Y-%m-%dT%H:%M:%S tag docker.container.all </source> <match docker.container.*> type elasticsearch log_level info include_tag_key true host ES_HOST port 9200 logstash_format true flush_interval 5s </match> </ins>

After PR , I did a bit experiments on kube-dns graceful termination and got some unintended result as below: I guess it is because the other container was terminated before . DNS query from clients failed because is the actual connection point. After a bit discussion with and two solutions for this are: another program wrap up the dnsmasq and capture the SIGTERM signal. out a way to modify the sig mask of process without touching the original codes.
Something I have been meaning to try: Put in Dockerfile Build and push image We should deliver SIGCONT rather than SIGTERM. I did a quick test and it seems to work. Otherwise trapping SIGTERM in the shell and ignoring it is well documented... , Zihong Zheng wrote:
Thanks, I will take a look and test these.


Question: export KUBE_ROOT=$(dirname "${BASH_SOURCE}")/.. source "${KUBE_ROOT}/hack/lib/init.sh" <ins> if [[ ! -f "${KUBE_ROOT}/vendor/BUILD" ]]; then echo "${KUBE_ROOT}/vendor/BUILD does not exist." echo echo "Run ./hack/update-bazel.sh" exit 1 fi </ins> # Remove generated files prior to running kazel. # TODO(spxtr): Remove this line once Bazel is the only way to build. rm -f "${KUBE_ROOT}/pkg/generated/openapi/zz_generated.openapi.go"

At this point I don't know if it is a valid file or not?
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
I reproduce it on my env. After removing , will not create and update ; Manually creating an empty file , will successfully rewrite and update ; /cc Seems file will not be created automatically if missing.
This is working as intended, or at least as designed. We only create new files if there are go sources in that directory, and doesn't have any go sources, so won't be auto-created. We have a solely for the and rules, but those are only managed in existing files. Without , we'd have a lot of lines in , which would be painful. I learned last week that the say to remove all of , which mucks with this process a little bit. I a step in the instructions there to recreate the file, but maybe should just be doing that? I'm not super excited about adding special cases there, but is kinda weird, so maybe it's unavoidable.
the more automatic the better
Specifically, this would cause many merge conflicts.
I'm a little confused, since this has been the way things have worked since merged in April, but 4+ people asking me about this in the last week have convinced me that this needs to be more automatic. Sent .


Question: return canonicalVolumePath, nil } <ins> // getPathFromFileNotFound returns the path from a fileNotFound error func getPathFromFileNotFound(err error) (string, error) { if soap.IsSoapFault(err) { fault := soap.ToSoapFault(err) f, ok := fault.VimFault().(types.FileNotFound) if !ok { return "", fmt.Errorf("%v is not a FileNotFound error", err) } return f.File, nil } return "", fmt.Errorf("%v is not a soap fault", err) } </ins> func setdatastoreFolderIDMap( datastoreFolderIDMap map[string]map[string]string, datastore string,

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: Provsioning a PVC using a storage class with vSphere as the cloud provider failed with error "runtime error: index out of range". What you expected to happen: The PVC should have been created. How to reproduce it (as minimally and precisely as possible): Create a PVC using a storage class with vSphere as the cloud provider and a VMware vCenter Server where the default language is not English. Anything else we need to know?: K8s in conjunction with vSphere PVC relies on English error messages from the VMware API in order to determine the canonical path for a newly created volume (VMDK file). The faulting code (v1.11.3, still the same on master) can be found here (starting at line 312): The line "canonicalVolumePath = match[1]" (340) in the above code is the problem because "File (.*?) was not found" (English) never matches for a different language (e.g. German: "Datei [DATASTORE] kubevols/kube- wurde nicht gefunden") and a “runtime error: index out of range” oocurs in the "kube-controller-manager". This causes the already created volume (VMDK file) not to be reported back to the K8s system which in turn will try to recreate it repeatedly in a loop (fails with "Cannot complete the operation because the file or folder [DATASTORE] /kubevols/kubernetes-dynamic-pvc-[UUID] already exists"). Find below the corresponding log excerpt (kube-controller-manager): Environment: Kubernetes version (use ):Cloud provider or hardware configuration: vSphere OS (e.g. from /etc/os-release): n/a Kernel (e.g. ): n/a Install tools: n/a Others: n/a <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
/assign


Question: return nil } if err != nil { <del> klog.V(5).Infof("Error getting PVC %q (uid: %q) from informer : %v", util.GetPersistentVolumeClaimQualifiedName(pvc), pvc.UID, err) </del> <ins> klog.V(5).Infof("Error getting PVC %q from informer : %v", key, err) </ins> return err } pv, err := expc.getPersistentVolume(ctx, pvc) if err != nil { <del> klog.V(5).Infof("Error getting Persistent Volume for PVC %q (uid: %q) from informer : %v", util.GetPersistentVolumeClaimQualifiedName(pvc), pvc.UID, err) </del> <ins> klog.V(5).Infof("Error getting Persistent Volume for PVC %q (uid: %q) from informer : %v", key, pvc.UID, err) </ins> return err } if pv.Spec.ClaimRef == nil || pvc.Namespace != pv.Spec.ClaimRef.Namespace || pvc.UID != pv.Spec.ClaimRef.UID { <del> err := fmt.Errorf("persistent Volume is not bound to PVC being updated : %s", util.ClaimToClaimKey(pvc)) </del> <ins> err := fmt.Errorf("persistent Volume is not bound to PVC being updated : %s", key) </ins> klog.V(4).Infof("%v", err) return err }

I migrated an existing EBS volume from in-tree plugin to external aws-ebs-csi-driver (). Resizing PVC twice in 6 hour window causes kube-controller-manager to crash. Exception message: Kube-controller-manager shouldn't crash in such case. If external aws-ebs-csi-driver fails to provision the pvc it should just log an warning or error and continue operating as usual running cluster version 1.22.9 using in-tree aws ebs driver. pvc: Install aws-ebs-csi-driver following featureGate CSIMigrationAWS on all cluster components (with draining nodes before enabling featureGate on kubelets - according to warning ) we have ebs volume that has been migrated to aws-ebs-csi-driver. pvc by increasing .spec.resources.request.storage to 2Gi. PVC, PV and EBS volume are correctly resized. once again not later than 6 hours after first resize. Increase .spec.resources.request.storage to 3Gi. Now kube-controller-manager throws an exception and crashes: EBS CSI Driver logs: I tried also enabling alpha featureGate InTreePluginAWSUnregister but the issue still occurs. <details</details<detailsAWS </details<details</details<details</details<detailsdocker </details<detailsaws-ebs-csi-driver version 1.10.0 </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cloud-provider
/sig storage some situation where the sync loop passes a nil parameter to
/assign
SetClaimResizer may return nil pvc and non-nil error, ClaimToClaimKey naively assumes non-nil as described above, will create PR
But the posted bug is I think 1.22 specific and should never happen on master or 1.24 and hence this fix against master is kind of okay and should fix the issue but at the same time code in is still kinda broken in 1.22 because it returns if patching fails.


Question: }, scheme.ParameterCodec) var stdout, stderr bytes.Buffer <ins> Logf("ExecWithOptions: execute(POST %s %s)", req.URL()) </ins> err = execute("POST", req.URL(), config, options.Stdin, &stdout, &stderr, tty) if options.PreserveWhitespace { return stdout.String(), stderr.String(), err }

We should fine grained debug statements to ExecWithOptions in the framework class , so that in tests where theres alot of execs where hangs can happen i.e. on windows, we can see where tests are getting clogged up. Mostly i want this for , but i suppose its a generic improvement
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig testing /good-first-issue
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /sig testing /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale


Question: golang.org/x/lint v0.0.0-20200302205851-738671d3881b/go.mod h1:3xt1FjdF8hUf6vQPIChWIBhFzV8gjjsPE/fR3IyQdNY= golang.org/x/mobile v0.0.0-20190312151609-d3739f865fa6/go.mod h1:z+o9i4GpDbdi3rU15maQ/Ox0txvL9dWGYEHz965HBQE= golang.org/x/mobile v0.0.0-20190719004257-d2bd2a29d028/go.mod h1:E/iHnbuqvinMTCcRqshq8CkpyQDoeVncDDYHnLhea+o= <ins> golang.org/x/mobile v0.0.0-20201217150744-e6ae53a27f4f/go.mod h1:skQtrUTUwhdJvXM/2KKJzY8pDgNr9I/FOMqDVRPBUS4= </ins> golang.org/x/mod v0.0.0-20190513183733-4bf6d317e70e/go.mod h1:mXi4GBBbnImb6dmsKGUJ2LatrhH/nqhxcFungHvyanc= golang.org/x/mod v0.1.0/go.mod h1:0QHyrYULN0/3qlju5TqG8bIK38QM8yzMo5ekMj3DlcY= golang.org/x/mod v0.1.1-0.20191105210325-c90efee705ee/go.mod h1:QqPTAvyqsEbceGzBzNggFXnrqF1CaUcvgkdR5Ot7KZg= golang.org/x/mod v0.1.1-0.20191107180719-034126e5016b/go.mod h1:QqPTAvyqsEbceGzBzNggFXnrqF1CaUcvgkdR5Ot7KZg= <ins> golang.org/x/mod v0.1.1-0.20191209134235-331c550502dd/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= </ins> golang.org/x/mod v0.2.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= <ins> golang.org/x/mod v0.3.1-0.20200828183125-ce943fd02449/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= </ins> golang.org/x/mod v0.4.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= golang.org/x/mod v0.4.1/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= golang.org/x/mod v0.4.2 h1:Gz96sIWK3OalVv/I/qNygP42zyoKp3xptRVCWRFEBvo=

For structured logging migration we have developed a tool to klog invocations For a given directory logcheck validates two things: directory is fully migrated - check if there are non-structured logging calls like usage of and is correct - check if key values passed are paired. We run logcheck as part of Kubernetes verify scripts on directories listed in Directories listed in are marked as migrated to avoid regressions from organized migration, however there are also other code locations that already use new functions that are not yet ready to be marked as migrated (components that are not part of organized migrations are done as best effort). We should split checking if directory was migrated from correctness of new functions and scan whole codebase for correctness. Plan: Add a flag to logcheck for requiring that directory was migrated. Add to verify-strucured- running logcheck of whole k/k codebase to check correctness This will ensure higher quality of migration PRs by validating correctness of directories that were not yet in .structuredlogging /wg structured-logging /cc
I'd like to pick this up. Do I need to wait till it is triage/accepted?
/triage accepted
Feel free to pick it up :P Usually I don't want to self triage, but this issue is purely Structured Logging WG related.
/assign
[Case 2: Correct usage] If the directory is not marked as migrated, we check that it's usage of is correct and don't care about usage of . By default, CI will test for correctness of whole k/k codebase instead of testing for complete migration (which is the current behavior). And to test for migration, a new flag must be passed explicitly. is this the expectation?
To clarify, let me rephrase. Current state is What we want: [Case 2: Correct usage] tested on whole repository
If this is the case, I can start implementing it.
SG, thanks for the help.
/milestone v1.22 Part of work go graduate Structured Logging KEP to beta.
I don't think work here has finished. We still need to implement those changes in k/k repo: Cut new klog Bump klog in k/k Update the script ./hack/verify-structured- /reopen
Reopened this issue. <detailsIn response to : I don't think work here has finished.

We still need to implement those changes in k/k repo: Update the script /reopen </details
Why do we need to cut a new klog and bump in k/k? Can't we make latest logcheck available to k/k without it? I don't know the process.
Hmm, We could try without updating to update. Please send PR that will update ./hack/verify-structured- If updating klog is nesesery feel free to contact for help
If we do both, we'd have to run logcheck twice (with and without flag) which means test time will increase.
This depends on implementation, increasing running time is ok as long it's reasonable, for example x1.5, but not x10.
Due to v1.22 code freeze, I'm moving this issue to the next release. /milestone v1.23
We would like to have static analysis setup before we start migrating logs in 1.23. will you be able to work on it this week? Do you need any help?
Yes, I can. PR kubernetes/kubernetes/pull/ is up for enabling static analysis. It has some unresolved conversations. It'd be helpful if I could get another round of review on that PR. Before we merge that we'd need to fix existing logging issues which are listed here: If I could get help fixing those it'd be great. Else I can fix it too.
I see two issues that need to be addressed: Fix handling of in klog code, this should be done by creating an issue to klog () Fix remaining errors Upgrade klog version to pick up fix for Exit


Question: err := store.Delete(ctx, tt.key, out, nil, storage.ValidateAllObjectFunc, nil) if tt.expectNotFoundErr { if err == nil || !storage.IsNotFound(err) { <del> t.Errorf("%s: expecting not found error, but get: %s", tt.name, err) </del> <ins> t.Errorf("expecting not found error, but get: %s", err) </ins> } return } if err != nil { <del> t.Fatalf("%s: Delete failed: %v", tt.name, err) </del> <ins> t.Fatalf("Delete failed: %v", err) </ins> } <del> ExpectNoDiff(t, fmt.Sprintf("%s: incorrect pod:", tt.name), tt.expectedObj, out) </del> <ins> // We expect the resource version of the returned object to be // updated compared to the last existing object. if storedObj.ResourceVersion == out.ResourceVersion { t.Errorf("expecting resource version to be updated, but get: %s", out.ResourceVersion) } out.ResourceVersion = storedObj.ResourceVersion ExpectNoDiff(t, "incorrect pod:", tt.expectedObj, out) </ins> }) } }

While working on generalizing tests to somehow better define spec for storage interface (and share them across all implementation of storage.interface - in particular etcd3 and watchcache), I found an unexpected inconsistency in Delete operation in etcd3. What is happening is that it effectively returns the state of the object from just before the deletion: However, this is inconsistent with how our watch is working, which is returning the state of the object from just before the deletion, but it sets the RV to the moment of deletion (which Delete call isn't doing). This sounds like a bug to me, but given this is a long-standing behavior we may need to take additional care when fixing this. /kind bug /sig api-machinery /priority important-soon /cc
I opened as a fix for it.
/triage accepted


Question: e2epod "k8s.io/kubernetes/test/e2e/framework/pod" e2eresource "k8s.io/kubernetes/test/e2e/framework/resource" e2eservice "k8s.io/kubernetes/test/e2e/framework/service" <ins> e2etestfiles "k8s.io/kubernetes/test/e2e/framework/testfiles" </ins> "k8s.io/kubernetes/test/e2e/network/common" )

What happened: We've been doing some release readiness and auditing all tagged ginkgo tests... Looks like The test seems to expect a GOPATH when running. Im seeing some failures in sonobuoy and suspect a possible relation there but havent dove into it. Either way, its clear this test might need some updating/fixing/cleanuping and parts can be rewritten/improved. What you expected to happen: No reliance on GOPATH for an e2e test ... I dont remember the whole history on this, but no tests that rely on static files currently look in gopath, from what i can tell ? instead we plumbed in a repo-root/ option a few years ago. There may be a state of the art in terms of how we do this.... but i thought it was at one point which was an option passed in. How to reproduce it (as minimally and precisely as possible): Try to run this in sonobuoy or any other environment where there's no GOPATH or non-standard source trees, pretty sure it might not work. Anything else we need to know?: I think the fix here is to just embed the YAML in golang like we do in other tests. along the way there's other cleanup we can do to improve this test. Environment: Kubernetes version (use ): master
/sig network
/assign
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
the YAML files are pretty small, so +1 to embed them in i don't think testing the YAML from the separate repository () directly is required here. ref /help /kind cleanup /priority backlog
/remove-help
/assign Also I should say, I talked to Jay on slack and he is ok with me taking a look at it, if thats ok.
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at .

/lifecycle stale
/remove-lifecycle stale
xref: go 1.16 has built in support for embedding go 1.17 looks to be removing GOPATH
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/help wanted
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/remove-lifecycle rotten still a problem. FYI re: GOPATH removal
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at .

/lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/assign


Question: "sync" "time" <ins> "github.com/aws/aws-sdk-go/aws" </ins> "github.com/aws/aws-sdk-go/aws/awserr" "github.com/aws/aws-sdk-go/aws/request" "github.com/golang/glog"


Question: // TODO(mtaufen): below is ported from the old kubelet_node_status_test.go code, potentially add more test coverage for NodeAddress setter in future func TestNodeAddress(t *testing.T) { <ins> type cloudProviderType int const ( cloudProviderLegacy cloudProviderType = iota cloudProviderExternal cloudProviderNone ) </ins> cases := []struct { <del> name                  string hostnameOverride      bool nodeIP                net.IP externalCloudProvider bool nodeAddresses         []v1.NodeAddress expectedAddresses     []v1.NodeAddress shouldError           bool </del> <ins> name                string hostnameOverride    bool nodeIP              net.IP cloudProviderType   cloudProviderType nodeAddresses       []v1.NodeAddress expectedAddresses   []v1.NodeAddress expectedAnnotations map[string]string shouldError         bool </ins> }{ { name:   "A single InternalIP",

I am reproducing this in OpenShift using the OpenStack cloud provider, but nothing here is specific to either OpenShift or OpenStack. When migrating a cluster which uses in-tree cloud provider to use external cloud-provider, we use the following process: cloud controllers in KCM CCM kubelets with --cloud-provider=external Between steps 1&2 the node-lifecycle, routes, and service controllers are not running anywhere, but this situation isn't unsafe for the cluster and we can keep it short. Between steps 2&3 is more interesting. The node controller is running in CCM, and the kubelets are running with --cloud-provider=openstack (or whatever, just not external). The main loop of node-controller will ignore a newly created node because . However, the loop does not have a similar exclusion. . Note that we shortcut if the kubelet is using an external cloud provider, but not if we don't. So between steps 2&3 both kubelet and CCM's node controller are managing Node.Addresses. If the legacy cloud provider and external CCM don't return the same list of NodeAddresses this causes flapping. Running OpenShift on OpenStack we have 2 problems. Firstly, our legacy cloud provider and external CCM produce different lists. We don't think this would be a problem in practise, except for the second problem. Kubelet adds the annotation to the node containing the node IP. This allows the CCM node controller to apply the same filtering based on node IP as kubelet does for the legacy cloud provider. This means that, although the lists are different, the node IP is still listed first which is the only thing that matters in practise. The problem is that kubelet only adds the annotation when --cloud-provider=external which, between steps 2&3, it is not. This means that Node.Addresses flaps between a correct and a broken configuration, as the NodeAddresses returned by the external CCM cannot be filtered by node IP without the annotation. The ideal solution here would be to ensure that only 1 controller is managing Node.Addresses during this period in the same way we do for Node creation with the external CCM taint. This would fix the issue entirely.

A fix with a much smaller scope would be to have kubelet add the node IP annotation unconditionally so, although Node.Address can potentially flap, it will at least be between 2 valid configurations. Given that this should only ever be a problem during the upgrade window, the smaller fix may be more expedient. Node.Addresses should remain valid at all times when kubelet is configured with --cloud-provider=openstack but the CCM is also running. Good luck No response 24 OpenStack (but as described, not OpenStack-specific) No response No response No response No response
/sig cloud-provider
we've an agenda item for next week's (11 May) SIG cloud provider meeting about this topic
/triage accepted
Would it work to swap steps 2 and 3?
Good question. I don't think so. That would leave us without the Node, Node lifecycle, and Service controllers until all kubelets were updated. This would mean: No cluster scale-up or scale-down No creation, deletion, or modification to LB Services I think we could handle this in OpenShift, although it's a bit fragile and I suspect updating all the kubelets can take a really long time on a big cluster. However, I think this would break cluster-api or any deployment tooling which rolls out updates by creating new machines. I think this would never complete: the new nodes would never come up because the node controller isn't running.
I am still trying to understand the issues here while reviewing your pr: , but I agreed with your comments above: That would leave us without the Node, Node lifecycle, and Service controllers until all kubelets were updated. This would mean:


Question: Addresses:  []string{"1.2.3.6", "1.2.3.7"}, Conditions: discoveryv1.EndpointConditions{Ready: utilpointer.BoolPtr(true)}, TargetRef:  &corev1.ObjectReference{Kind: "Pod", Name: "test-124"}, <ins> NodeName:   utilpointer.StringPtr("node-2"), </ins> }, }, Ports: []discoveryv1.EndpointPort{

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --panic as below when an Endpoint doesn't have zone. should print zone as if an Endpoint's zone is unset. not set "" for Nodes. Pods and export them as a service. kubectl to describe the corresponding EndpointSlice. Kubernetes version (use ): v1.21.0 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig network /sig cli /assign


Question: } func (cm *containerManagerImpl) GetCPUs(podUID, containerName string) []int64 { <del> return cm.cpuManager.GetCPUs(podUID, containerName).ToSliceNoSortInt64() </del> <ins> if cm.cpuManager != nil { return cm.cpuManager.GetCPUs(podUID, containerName).ToSliceNoSortInt64() } return []int64{} </ins> } func (cm *containerManagerImpl) GetAllocatableCPUs() []int64 { <del> return cm.cpuManager.GetAllocatableCPUs().ToSliceNoSortInt64() </del> <ins> if cm.cpuManager != nil { return cm.cpuManager.GetAllocatableCPUs().ToSliceNoSortInt64() } return []int64{} </ins> } func (cm *containerManagerImpl) ShouldResetExtendedResourceCapacity() bool {

Iff the cpumanager feature gate (default: enabled) is explicitely disabled, calling podresources API causes a NPE in the kubelet. The stacktrace looks like This applies to any podresources API (either or ) Only triggers if the cpumanager feature gate is explicitely disabled (it is enabled by default in 1.20+) The podresources API should return either error or a valid value, not NPE. The simplest possible reproducer is probably to apply this patch And then run the podresources e2e tests We should probably add another e2e test to explicitly check this dlow Kubernetes version (use ): any since 1.20 (possibly introduced in commit ) Cloud provider or hardware configuration: any OS (e.g: ): any Kernel (e.g. ): any Install tools: N/A Network plugin and version (if this is a network-related bug): N/A Others: N/A
/sig node /triage accepted
possible workaround (not tested but everything suggests it should work) enable the cpumanager feature gate set the cpumanager policy to "none"


Question: storage "k8s.io/api/storage/v1" apierrors "k8s.io/apimachinery/pkg/api/errors" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" <del> "k8s.io/apimachinery/pkg/labels" </del> "k8s.io/apimachinery/pkg/util/sets" utilfeature "k8s.io/apiserver/pkg/util/feature" clientset "k8s.io/client-go/kubernetes"

What happened: With a StorageClass with , it's not clear why a PVC is not bound when there is no available PV. On Pod, there is a clear event what's wrong: On the PVC side we get only this event: Note that the message says "waiting for first consumer'. But the pod already exists. The reason why it's not bound yet is that the StorageClass has no dynamic provisioning configured () and there are no free PVs to satisfy the claim. What you expected to happen: An event "1 node(s) didn't find available persistent volumes to bind" on the PVC.
/sig scheduling /sig storage
I tried to fix this in VolumeBinding predicate, The predicate is called for each pod - node combination, so it would generate event for each node. Which is too much and it's even wrong when there is a node that has an available PV. We need to evaluate all nodes and then, if the pod is still unschedulable, distribute events to all unbound PVCs of that pod. Scheduler does not offer any way how to track unbound PVCs at that stage - whole volume logic is hidden in VolumeBinding predicate. One option would be to extend  struct to contain also list of objects where to send events (except the pod itself): Scheduler would then send all failed scheduling events it sends to a pod to all the , without any interpretation. is used to manage duplicates easily - each object (its UID) can be in the map only once. Simple array would work too, but then it needs extra work in removing duplicates. Not sure how easy it is to detect duplicate instances. is extensible to allow any object type. predicate would put all unbound PVCs there. does not need any special handling of each PVC. Message "2 node(s) didn't find available persistent volumes to bind" sounds just right in case a pod uses multiple PVCs, each of them might be bind-able on different nodes (node A may satisfy PVC1, but not PVC2, while node B may satisfy PVC2 only). what do you think about such design?
We used to have extension point, which I believe is the right moment for recording the event for unbounds PVCs. But recently it's been refactored so doesn't fit. Do you think this case can be a good example of using as an informational extension point?

Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale wdyt?
I have some other concerns. What if the pod is deleted? Should the scheduler send events that PVC is not in volume binding? The PV controller must be updated to stop emit event in this scenario. It might be simpler to solve on the PV controller side. In this function, we can check if the PVC is being used by a pod. If it is, we can send event like , then people can go to check the progress of pod "foo/bar". If the pod "foo/bar" is deleted, the PV controller can emit "waiting for first consumer ..." event again. What do you think? cc
Sounds better than it's now. My only issue may be performance - PV controller has to check if a pod exists, i.e. list all pods in the same namespace as PVC being reconciled. I am probably too worried.
current attach detach controller has a custom indexer to index pods by PVC keys to avoid iterating all pods every time to find pods which reference given PVC. And there are other places we need to check if a PVC is used by a pod or not from pod informer, e.g. pvcprotectioncontroller pvcontroller we can try to share this custom indexer among these volume controllers in the future. This will improve the overall performance of the IMO. What do you think
Yes, using the indexer looks like a good idea.
/remove-sig scheduling I also agree that solving in PVC side seems like a better idea. Thanks
/assign unbound claims are reconciled in the PV controller every 15 seconds (default), we may have bad worst-case performance (e.g. too many pods in the same namespace). we can add an indexer in the PV controller locally to solve this issue first (also optimize ), then find a way to in manager to improve the overall performance.


Question: mountArgs = append(mountArgs, mountpoint) mountArgs = append(mountArgs, "-r") mountArgs = append(mountArgs, cephfsVolume.path) <ins> mountArgs = append(mountArgs, "--id") mountArgs = append(mountArgs, cephfsVolume.id) </ins> glog.V(4).Infof("Mounting cmd ceph-fuse with arguments (%s)", mountArgs) command := exec.Command("ceph-fuse", mountArgs...)

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: I tried to use pvs provisioned by the cephfs storageclass build from PV and pvc are OK but it failed when mounting. here is the output of the ceph-fuse mount command: I try the ceph-fuse mounting outside kubernetes environment, and found that I have to pass the --id argument to make ceph-fuse work correctly: so I add the --id option in file on line 401: and cephfs volumes can mount corretly with ceph-fuse then. Anything else we need to know?: My kubernetes cluster version is v1.6.7, running on arm64 servers, with ceph version 10.1.2, the kernel version is 4.4.58 . I backported the latest version to mount cephfs volumes with ceph-fuse.
/sig storage
Does this problem still exist in the latest k8s version?
yes, I've tested on kubernetes v1.10.0-alpha.3 with ceph version 10.2.10 and the problem still exists
OK, let me see it ,and feedback soon. =)
Hi, one question is that do you use 'admin' as user or other user to mount cephfs by fuse ?
when no "--id" specified, the default user is admin, which will result in bug when user is not admin.
/assign


Question: }, wantErr: false, }, <ins> { name: "DualSocketHT - non unique Core'ID's", args: &cadvisorapi.MachineInfo{ NumCores: 12, Topology: []cadvisorapi.Node{ {Id: 0, Cores: []cadvisorapi.Core{ {Id: 0, Threads: []int{0, 6}}, {Id: 1, Threads: []int{1, 7}}, {Id: 2, Threads: []int{2, 8}}, }, }, {Id: 1, Cores: []cadvisorapi.Core{ {Id: 0, Threads: []int{3, 9}}, {Id: 1, Threads: []int{4, 10}}, {Id: 2, Threads: []int{5, 11}}, }, }, }, }, want: &CPUTopology{ NumCPUs:    12, NumSockets: 2, NumCores:   6, CPUDetails: map[int]CPUInfo{ 0:  {CoreID: 0, SocketID: 0}, 1:  {CoreID: 1, SocketID: 0}, 2:  {CoreID: 2, SocketID: 0}, 3:  {CoreID: 3, SocketID: 1}, 4:  {CoreID: 4, SocketID: 1}, 5:  {CoreID: 5, SocketID: 1}, 6:  {CoreID: 0, SocketID: 0}, 7:  {CoreID: 1, SocketID: 0}, 8:  {CoreID: 2, SocketID: 0}, 9:  {CoreID: 3, SocketID: 1}, 10: {CoreID: 4, SocketID: 1}, 11: {CoreID: 5, SocketID: 1}, }, }, wantErr: false, }, { name: "OneSocketHT fail", args: &cadvisorapi.MachineInfo{ NumCores: 8, Topology: []cadvisorapi.Node{ {Id: 0, Cores: []cadvisorapi.Core{ {Id: 0, Threads: []int{0, 4}}, {Id: 1, Threads: []int{1, 5}}, {Id: 2, Threads: []int{2, 2}}, // Wrong case - should fail here {Id: 3, Threads: []int{3, 7}}, }, }, }, }, want:    &CPUTopology{}, wantErr: true, }, { name: "OneSocketHT fail", args: &cadvisorapi.MachineInfo{ NumCores: 8, Topology: []cadvisorapi.Node{ {Id: 0, Cores: []cadvisorapi.Core{ {Id: 0, Threads: []int{0, 4}}, {Id: 1, Threads: []int{1, 5}}, {Id: 2, Threads: []int{2, 6}}, {Id: 3, Threads: []int{}}, // Wrong case - should fail here }, }, }, }, want:    &CPUTopology{}, wantErr: true, }, </ins> } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) {

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: CPU Manager uses topology from cAdvisor which uses which is different from in terms of coreID's. assigns coreID's in unique manner (platform-wise) whereas has coreID's from CPU registers which are unique only on socket level. This causes some unexpected cpu assignment on platforms with more than one socket. All tests written all "false positive" Example output: From What you expected to happen: Fix either by changing the way CPU manger sees topology (use ) or redesign CPU Manager to handle coreID's as socket only unique How to reproduce it (as minimally and precisely as possible): kubelet on dual socket node G pod that requests 2 cores pod gets 2 cores on different sockets
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig node


Question: Kind string // Name is empty for some verbs, but if the request directly indicates a name (not in body content) then this field is filled in. Name string <del> // Parts are the path parts for the request relative to /{resource}/{name} </del> <ins> // Parts are the path parts for the request, always starting with /{resource}/{name} </ins> Parts []string <ins> // Raw is the unparsed form of everything other than parts. // Raw + Parts = complete URL path Raw []string } // URLPath returns the URL path for this request, including /{resource}/{name} if present but nothing // following that. func (info APIRequestInfo) URLPath() string { p := info.Parts if n := len(p); n > 2 { // Only take resource and name p = p[:2] } return path.Join("/", path.Join(info.Raw...), path.Join(p...)) </ins> } type APIRequestInfoResolver struct {

The apiserver proxy rewrites links to the form "/proxy/ns/default/...": But this form stopped working recently, seemingly around The "/proxy/ns/default/..." form always returns "404: Page Not Found", similar to the problem seen in If you remove "/ns/default" from the URL, it works.
/cc
I can investigate this as well.
I see the issue, sending a PR.
The proxy path needs to be built with knowledge of the apiversion of the incoming proxy request, and it needs to be updated to use the knowledge from the restmapper. This is a little trickier than I have time for right now, but it should definitely not be hardcode appending the /ns form. Not clear why this is not also caught by our test case. I can try and look at this more in the AM unless someone else wants to take a look in interim. /cc
Let me steal this one from you, I think I can fix it.
Have at it. /cc me on your PR
Sorry, I see you've already got a PR out!
No problem. I would be happy if you finished my PR relative to the commit I started. It will be a few days before I can go back to looking at this issue. I think v1beta1/v1beta2 proxy urls in go-restful need to always but the namespace scope in the path as the ultimate fix. I do want the proxy to have access to the restmapper per my PR. ----- Original Message ----- From: "Daniel Smith" To: "GoogleCloudPlatform/kubernetes" Cc: "Derek Carr" Sent: Friday, February 13, 2015 4:56:19 PM Subject: Re: [kubernetes] apiserver proxy stopped accepting namespaces () Sorry, I see you've already got a PR out! Reply to this email directly or view it on GitHub:
Any updates ?
This is breaking monitoring since it requires the proxy. Bumping up the priority to P0.
Sorry for dropping this. Working on a fix.


Question: var _ volume.VolumePlugin = &downwardAPIPlugin{} <del> var wrappedVolumeSpec = volume.Spec{ Volume: &api.Volume{VolumeSource: api.VolumeSource{EmptyDir: &api.EmptyDirVolumeSource{Medium: api.StorageMediumMemory}}}, </del> <ins> func wrappedVolumeSpec() volume.Spec { return volume.Spec{ Volume: &api.Volume{VolumeSource: api.VolumeSource{EmptyDir: &api.EmptyDirVolumeSource{Medium: api.StorageMediumMemory}}}, } </ins> } func (plugin *downwardAPIPlugin) Init(host volume.VolumeHost) error {

Data as of June 3rd, 9:00 PM Eastern. Latest data can be found here: Prow link to the 2 jobs: * Additional Context:
/sig storage cc
I'm working on , it's hard to get a job yaml right at the first time. I'll check iscsi. /assign
/triage accepted /kind failing-test
ci-kubernetes-e2e-storage-kind-disruptive fixed: has 1 green run pull-kubernetes-e2e-storage-kind-disruptive fixed: , it fails when it should fail, it's green otherwise, I'll continue with iscsi.
/reopen it's still red, but at least it installs now:
Reopened this issue. <detailsIn response to : /reopen it's still red, but at least it installs now: Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
e2e-gce-iscsi-serial job passes now: e2e-gce-iscsi is getting greener: I'll let it run for few more days to see if today's failures are flakes. /reopen
Reopened this issue. <detailsIn response to : It's getting greener: /reopen </details
iscsi-serial is green:


Question: "time" "k8s.io/api/core/v1" <ins> schedulerapi "k8s.io/api/scheduling/v1beta1" "k8s.io/apimachinery/pkg/api/errors" </ins> "k8s.io/apimachinery/pkg/api/resource" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/fields"

https://k8s- The following tests are broken: NVIDIA GPU Device Plugin PriorityLocalStorageEvictionOrdering PriorityMemoryEvictionOrdering Looks like some new validation was to prevent two things: PodPriority directly system-critical priority classes outside of the kube-system namespace. To fix this, we need to create new priority classes for these tests. cc /kind failing-test /sig node /sig scheduling /priority important-soon
/kind bug
/assign
This doesn't appear to have fixed the tests. Tests now panic during the Aftereach in which the PriorityClass is deleted. /reopen
Panic:
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: }) /* <del> Testname: var-expansion-subpath-lifecycle Description: Verify should not change the subpath mount on a container restart if the environment variable changes 1.	valid subpathexpr starts a container running 2.	test for valid subpath writes 3.	container restarts 4.	delete cleanly </del> <ins> Release : v1.19 Testname: VolumeSubpathEnvExpansion, subpath lifecycle Description: Verify should not change the subpath mount on a container restart if the environment variable changes 1.	valid subpathexpr starts a container running 2.	test for valid subpath writes 3.	container restarts 4.	delete cleanly </ins> */ <del> ginkgo.It("should not change the subpath mount on a container restart if the environment variable changes [sig-storage][Slow]", func() { </del> <ins> framework.ConformanceIt("should not change the subpath mount on a container restart if the environment variable changes [sig-storage][Slow]", func() { </ins> suffix := string(uuid.NewUUID()) podName := fmt.Sprintf("var-expansion-%s", suffix)

Is this a BUG REPORT or FEATURE REQUEST?: /kind api-change /sig storage /sig node /sig testing Description: Promote e2e tests for VolumeSubpathEnvExpansion to Conformance as it is GA approved. Related Issues: GA Upgrade: Feature Request: KEP:
/assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: golang.org/x/tools v0.0.0-20191216052735-49a3e744a425/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28= golang.org/x/tools v0.0.0-20191216173652-a0e659d51361/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28= golang.org/x/tools v0.0.0-20191227053925-7b8e75db28f4/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28= <ins> golang.org/x/tools v0.0.0-20200117012304-6edc0a871e69/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28= </ins> golang.org/x/tools v0.0.0-20200117161641-43d50277825c/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28= golang.org/x/tools v0.0.0-20200117220505-0cba7a3a9ee9/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28= golang.org/x/tools v0.0.0-20200122220014-bf1340f18c4a/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28=

For structured logging migration we have developed a tool to klog invocations For a given directory logcheck validates two things: directory is fully migrated - check if there are non-structured logging calls like usage of and is correct - check if key values passed are paired. We run logcheck as part of Kubernetes verify scripts on directories listed in Directories listed in are marked as migrated to avoid regressions from organized migration, however there are also other code locations that already use new functions that are not yet ready to be marked as migrated (components that are not part of organized migrations are done as best effort). We should split checking if directory was migrated from correctness of new functions and scan whole codebase for correctness. Plan: Add a flag to logcheck for requiring that directory was migrated. Add to verify-strucured- running logcheck of whole k/k codebase to check correctness This will ensure higher quality of migration PRs by validating correctness of directories that were not yet in .structuredlogging /wg structured-logging /cc
I'd like to pick this up. Do I need to wait till it is triage/accepted?
/triage accepted
Feel free to pick it up :P Usually I don't want to self triage, but this issue is purely Structured Logging WG related.
/assign
[Case 2: Correct usage] If the directory is not marked as migrated, we check that it's usage of is correct and don't care about usage of . By default, CI will test for correctness of whole k/k codebase instead of testing for complete migration (which is the current behavior). And to test for migration, a new flag must be passed explicitly. is this the expectation?
To clarify, let me rephrase. Current state is What we want: [Case 2: Correct usage] tested on whole repository
If this is the case, I can start implementing it.
SG, thanks for the help.
/milestone v1.22 Part of work go graduate Structured Logging KEP to beta.
I don't think work here has finished. We still need to implement those changes in k/k repo: Cut new klog Bump klog in k/k Update the script ./hack/verify-structured- /reopen
Reopened this issue. <detailsIn response to : I don't think work here has finished.

We still need to implement those changes in k/k repo: Update the script /reopen </details
Why do we need to cut a new klog and bump in k/k? Can't we make latest logcheck available to k/k without it? I don't know the process.
Hmm, We could try without updating to update. Please send PR that will update ./hack/verify-structured- If updating klog is nesesery feel free to contact for help
If we do both, we'd have to run logcheck twice (with and without flag) which means test time will increase.
This depends on implementation, increasing running time is ok as long it's reasonable, for example x1.5, but not x10.
Due to v1.22 code freeze, I'm moving this issue to the next release. /milestone v1.23
We would like to have static analysis setup before we start migrating logs in 1.23. will you be able to work on it this week? Do you need any help?
Yes, I can. PR kubernetes/kubernetes/pull/ is up for enabling static analysis. It has some unresolved conversations. It'd be helpful if I could get another round of review on that PR. Before we merge that we'd need to fix existing logging issues which are listed here: If I could get help fixing those it'd be great. Else I can fix it too.
I see two issues that need to be addressed: Fix handling of in klog code, this should be done by creating an issue to klog () Fix remaining errors Upgrade klog version to pick up fix for Exit