Question: pair := topologyPair{key: c.TopologyKey, value: node.Labels[c.TopologyKey]} s.TpPairToMatchNum[pair] = new(int32) } <ins> filteredNodes = append(filteredNodes, n) </ins> } processNode := func(i int) { <del> nodeInfo := allNodes[i] </del> <ins> nodeInfo := filteredNodes[i] </ins> node := nodeInfo.Node() for _, constraint := range constraints {

We know that nodeAffinity/nodeSelector is honored during the calculating of PodTopolgySpread, however, when some existing pods match the incoming pod's topologySpreadConstratints, while also excluded by the nodeAfffinity, things get a bit tricky. Raising this issue to discuss if we should read it as a bug. Here are the repro steps: you have a 3-node cluster, worker1 and worker2 belong to zoneA, and worker3 belongs to zoneB: make a deployment with 3 replicas. Suppose 2 land on worker2, and 1 land on worker. a new deployment with topologySpreadConstraints as well as nodeAffinity (not schedule to worker2): test pod will land on worker3. This behavior concerns me: it looks like the 2 pods on worker2 are taken into consideration, even though worker2 is excluded by nodeAffinity, so the pods on zoneA/zoneB distribute as 2:2. However, shouldn't we disregard the matching pods that are excluded by nodeAffinity? In that case, the test pod should land on worker1 to reach a pods distribution as 1:1. Please share your thoughts. /sig scheduling /kind bug
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
cc/
That is not the issue, the 2 pods on work2 aren't actually being taken into consideration; the problem is similar to what describes. What happened is that zoneA is now not "visible" at all, and so it is as if we have a single zone. The solution should apply to both cases.
I think we should consider it as a bug. Imagine in this situation, all of these pods are labeled with , then we want to apply a like this: is excluded in scheduling. But now when we try to calculate the skew, we will still calculate the pods in . Since : is , then we will schedule the new pod to or , but since they are resource exhausted, then it will be in pending state.

But if we filtered out the , the pod will be scheduled to successfully. I think it's the right algorithm.
it's not the same as , even if worker2 becomes invisible, there are still 2 zones:
ah, I didn't notice existed. What part of the code is causing this behavior? I would have assumed that the spreading would happen between and . The following code shows that we should be skipping counting the pods on worker2
it's the that is passed to processNode(): We should pass on a filtered list of nodes to processNode(). included the fix in , and that bits should be extracted out to be a standalone PR, if we get to a consensus on this issue is a bug.
I still don't get where the bug is. I think iterating over all node is the right way and the line I pointed to above should filter out the nodes that don't match the affinity. Passing a filteredList (as in exclude nodes that were filtered by previous filters) is certainly not the right solution because we need to count pods on all nodes. Let alone the filteredlist is different based on how we order the filters, which should never be a factor in the semantics of the PodTopologySpread.
The bug lies in the preFilterState's calculation. For the above case, it's expected to be , but it ended up with . The code I posted above is in PodTopologySpread's PreFitler, not Filter.
ok, this is indeed a bug, we need to check the node affinity constraints again inside the second loop.
actually, doesn't seem efficient that we have two loops in the first place
It seems we are doing it right in the score function, excluding those nodes:
Feel the same. Let's fix the correctness issue first, and then improve the efficiency.
Do we know of a use case where this would be useful? It's kind of odd that there are matching pods in a node where we don't intend to schedule. Maybe during some sort of migration?
It's about clarifying/correcting the semantics of constraints (nodeAffinity, and the to-be- ones like Unschedulable taint) embedded in PodTopologySpread. So IMO we should fix the correctness issue, no matter there is a use case or not.
One thing to be aware of is whether a potential fix could render some pods unschedulable, thus be a breaking change.


Question: } ], "livenessProbe": { <del> "httpGet": { "host": "127.0.0.1", "port": {{ etcd_livenessprobe_port }}, "path": "/health" </del> <ins> "exec": { "command": [ "/bin/sh", "-c", "set -x; exec /usr/local/bin/etcdctl --endpoints=127.0.0.1:{{ port }} {{ etcdctl_certs }} --command-timeout=15s endpoint health" ] </ins> }, "initialDelaySeconds": {{ liveness_probe_initial_delay }}, <del> "timeoutSeconds": 15 </del> <ins> "timeoutSeconds": 15, "periodSeconds": 5, "failureThreshold": 5 </ins> }, "ports": [ { "name": "serverport",

Currently, etcd livenessProbe is defined here: This means that it proves /health endpoint 3 times (default failureThreshold value) and if all of them fails, it restarts etcd server. /health endpoint returns "unhealthy" if one of the conditions is met (): there is an active Alarm (there are two kinds of alarm: NOSPACE and CORRUPT) there is no raft leader the latency of a QGET request doesn't exceed 1s The problem is that in most of those cases, etcd's restart isn't a right behavior: if there is NOSPACE alarm, the restart will not free that space if there is no raft leader, it means that it cannot talk to other etcd members, so restart will not fix them if the request latency 1s, the etcd cluster is overloaded which is bad, but after restart we will get similar amount of requests (if not more), so restart isn't a right action e.g. in we see that event-etcd gets restarted by kubelet when it's overloaded. If we do not restart etcd at that point, we will likely simply finish work to be done and heal ourselves. I think we should change livenessProbe to detect cases where restart will actually help. /cc
I think that's not necessary true, that may potentially help sometimes. But I agree that's probably pretty rare. /sig api-machinery
Do we have an analysis of why the 3 healthchecks in row failed in Was it timeout, timeout, timeout or something else? Do you say that if etcd is overloaded then restarting it will help? Wouldn't it be even more overloaded and run into restart-loop because it will never become /healthy in time again?
Sorry, typo. I tried to say that in none of those cases restart will help (including 1s latency), but in fact I wrote something else. I changed initial post.
should fix etcd such that it does not declares as unhealthy as a result of overload (while its still making a progress). Ideally the not-healthchecking traffic should be throttled / load-shed, such that healthcheck stays responsive. this is implemented, I would consider giving higher timeout (e.g. 30sec), such that HC request have higher chance to reach beginning of the processing queue and get responded.

is a usual disjoint between 2 purpuses of healthchecking: a) autohealing -by restart -where we need to be very conservative about timeouts, as cost is high. b) measuring SLO & alerting -where we want to measure customer's happiness, and we cannot consider request waiting for 15s+ seconds as healthy/happy state. The lower tolerance more represents customers needs. In practical application we should optimize etcd healthcheck for a), while b) should be based on prometheous metrics.
This is basically the difference between "livez" and "readyz". "livez" is whether the app is running (for autohealing) "readyz" is whether it's useful from customer point of view (e.g. for SLO) In the ideal world, those should be two separate handlers.
In addition to doubling the timeout we can also consider halving the probing period (from default 10s to 5s) and doubling failure thresholds (from 3 failed probes in row to 6). It would decrease the probability of etcd being restarted in case of temporarily overloaded master.
There is no option to keep using "/health" endpoint and changing timeout to something higher, as this is hardcoded in etcd: Also I think we should stop restarting etcd on NOSPACE or CORRUPT alarms. This way we will stop restarting on alarms and have ability to control timeout (and setting it to some higher value). Once etcd starts exposing some better endpoint that can be used for liveness probes, we will switch to that. WDYT?
/cc /triage accepted


Question: if err != nil && !apierrors.IsConflict(err) && !apierrors.IsNotFound(err) { klog.V(3).Infof("Error deleting Secret: %v", err) } <ins> } else if ttl > 0 { tc.queue.AddAfter(o, ttl) </ins> } }

What happened: kube-controller-manager with enabled option --controllers=,tokencleaner is not removing newly created tokens. It works only after kube-controller-manager restart or after editing secret with token. What you expected to happen: It should remove all expired tokens from kube-system namespace, without restarting or touching secrets. How to reproduce it (as minimally and precisely as possible): enable tokencleaner in kube-controller-manager start controller manager create bootstrap token, you can use script wait 5 minutes token is still there restart kube-controller-manager OR edit newly created token token should be gone after a while Anything else we need to know?: It could be related with Environment: Kubernetes version (use ): Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-12-03T20:56:12Z", GoVersion:"go1.11.2", Compiler:"gc", Platform:"linux/amd64"}
Do you see the following log ? thanks
no errors at all
/sig auth


Question: t.Fatalf("Error creating test service: %v", err) } } <ins> func getPatchBytes(oldSvc, newSvc *v1.Service) ([]byte, error) { oldData, err := json.Marshal(oldSvc) if err != nil { return nil, fmt.Errorf("failed to Marshal oldData for svc %s/%s: %v", oldSvc.Namespace, oldSvc.Name, err) } newData, err := json.Marshal(newSvc) if err != nil { return nil, fmt.Errorf("failed to Marshal newData for svc %s/%s: %v", newSvc.Namespace, newSvc.Name, err) } patchBytes, err := strategicpatch.CreateTwoWayMergePatch(oldData, newData, v1.Service{}) if err != nil { return nil, fmt.Errorf("failed to CreateTwoWayMergePatch for svc %s/%s: %v", oldSvc.Namespace, oldSvc.Name, err) } return patchBytes, nil } </ins>

Why PatchService method is used for deleting finalizers in cloud-controller-manager, This method updates with service/status, It seems to go unnoticed by the API server, resulting in the API server being unaware when the finalizer field on the service is deleted, leading to a delay in the release of service-related NodePorts. service nodeport be released promptly when service is deleted. create a loadblancer service with nodeport seted,then delete it and recreate. cloud-controller-manager add a finalizer on service. delete this service and then cloud-controller remove finalizer. but nodeport is not released when service deleted. ref i change like this, it seems like releasing nodeport when service is deleted : No response <details</details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cloud-provider
Are you sure? that method updates the status and remove the finalizer ... otherwise , who removes the finalizer?
I apologize for any confusion. What I meant was that the update of the status and the removal of the finalizer are performed through updating service/status. Therefore, removing the finalizer does not release resources.It seems like removal of the finalizer by service/status, store.AfterDelete is nil. i think remove the finalizer should by patch servcie and updates the status should by patch service/status.Now we patch both by servcie/status.
no need to apologize, let me see if I can reproduce this locally first, Services api allocation is complex , it overrides the standard AfterDelete and release all allocated resources there /assign
/sig network ack the problem
will look for a possibly-related past cloud provider issue.
this will be fixed by , it is a problem in the apiserver logic
Cool! I forgot that you can patch finalizers thru /status. I repro'ed it on both ClusterIP and NodePort.

Saw this in the logs: So the repair loop, yet again, saves our butt but also hides a bug.
we need metrics here about to have visibility of the repairs
I also found this problem in versions 1.23 and 1.27
yep, I started the backports , but we can only backport to the supported versions, 1.23 and 1.24 are EOL


Question: } By("creating the pod") <del> pod, err := f.ClientSet.CoreV1().Pods(f.Namespace.Name).Create(pod) </del> <ins> var podClient *framework.PodClient podClient = f.PodClient() pod = podClient.Create(pod) </ins> By("waiting for pod running") <del> err = framework.WaitTimeoutForPodRunningInNamespace(f.ClientSet, pod.Name, pod.Namespace, framework.PodStartShortTimeout) </del> <ins> err := framework.WaitTimeoutForPodRunningInNamespace(f.ClientSet, pod.Name, pod.Namespace, framework.PodStartShortTimeout) </ins> Expect(err).NotTo(HaveOccurred(), "while waiting for pod to be running") By("creating a file in subpath")

Which jobs are failing: All new tests introduced for api change Which test(s) are failing: New alpha subpathExpr tests [Feature:VolumeSubpathEnvExpansion][NodeAlphaFeature:VolumeSubpathEnvExpansion] Existing tests are working fine and the whole suite works correctly in the gce suite https://k8s- Since when has it been failing: 21st February 2019 (since merge of PR) Testgrid link: https://k8s- Reason for failure: Investigation shows that node e2e tests use a different framework pod creation pattern Doing some comparisons of tests that work and those that don't reveals differences in the standard client set as and the existing ones which do work go through the intermediate which uses The version has an additional function specifically for node tests
/sig storage
/sig node /sig test
/assign
/cc
/sig test


Question: return []string{devicePath}, nil } <ins> func TestextractTransportname(t *testing.T) { fakeIscsiadmOutput := []string{ "# BEGIN RECORD 2.0-873n" + "iface.iscsi_ifacename = defaultn" + "iface.transport_name = tcpn" + "iface.initiatorname = <empty>n" + "# END RECORD", "# BEGIN RECORD 2.0-873n" + "iface.iscsi_ifacename = defaultn" + "iface.transport_name = cxgb4in" + "iface.initiatorname = <empty>n" + "# END RECORD", "# BEGIN RECORD 2.0-873n" + "iface.iscsi_ifacename = defaultn" + "iface.transport_name = <empty>n" + "iface.initiatorname = <empty>n" + "# END RECORD", "# BEGIN RECORD 2.0-873n" + "iface.iscsi_ifacename = defaultn" + "iface.initiatorname = <empty>n" + "# END RECORD"} transportName := extractTransportname(fakeIscsiadmOutput[0]) if transportName != "tcp" { t.Errorf("extractTransportname: Could not extract correct iface.transport_name 'tcp', got %s", transportName) } transportName = extractTransportname(fakeIscsiadmOutput[1]) if transportName != "cxgb4i" { t.Errorf("extractTransportname: Could not extract correct iface.transport_name 'cxgb4i', got %s", transportName) } transportName = extractTransportname(fakeIscsiadmOutput[2]) if transportName != "tcp" { t.Errorf("extractTransportname: Could not extract correct iface.transport_name 'tcp', got %s", transportName) } transportName = extractTransportname(fakeIscsiadmOutput[3]) if transportName != "" { t.Errorf("extractTransportname: Could not extract correct iface.transport_name '', got %s", transportName) } } </ins> func TestWaitForPathToExist(t *testing.T) { devicePath := []string{"/dev/disk/by-path/ip-127.0.0.1:3260-iqn.2014-12.com.example:test.tgt00-lun-0", "/dev/disk/by-path/pci-0000:00:00.0-ip-127.0.0.1:3260-iqn.2014-12.com.example:test.tgt00-lun-0"} <del> exist := waitForPathToExistInternal(devicePath[0], 1, "default", fakeOsStat, filepath.Glob) </del> <ins> exist := waitForPathToExistInternal(devicePath[0], 1, "tcp", fakeOsStat, filepath.Glob) </ins> if exist == false { t.Errorf("waitForPathToExist: could not find path %s", devicePath[0]) }

open-iscsi software initiators are currently broken for non- interfaces due to the fact that the devicePath (in ) is assumed to use the prefixing scheme for all interfaces not named . This causes waitForPathToExist to timeout after 10 seconds since the device path that it is waiting for is not the - prefixed path that is created, resulting in host logging such as: or K8S event logs such as: open-iscsi support for hardware transports was in the following commit: In this commit, software iSCSI support was assumed to be done through only the use of the default interface. According to the documentation at (Section 5.1.1) both software and offload/hardware interfaces are intended to be configured using named interface specifications and is only a backwards-compatibility vehicle. Configuration of cannot be done via the use of as this approach is specifically forbidden in the documentation: Leaving this here to help others associate various failure modes with this issue and possible resolution: In my case I am just learning and this is my first iSCSI deployment. My initial error was that I was receiving an message at discovery-time even though I had used to define a name. This approach failed because I didn't realize that iscsid needed to be bounced before it would pick up changes to that file. I had defined a custom specification and was using that to accomplish the task of setting my initiator name. I was able to unwind this mess and return to the use of once I read the code and understood what it was attempting to do.
i'm marking this as a feature request against the iscsi volume driver. RFE: support iscsi interfaces named something besides 'default'.
In theory the fix should be easy, just move the path check from iface name to transport_name. I will take a look asap, unfortunately I am travelling till monday
Sorry about the previous comment, I was logged in to the wrong account. In theory the fix should be easy, just move the path check from iface name to transport_name. I will take a look asap, unfortunately I am travelling till monday. can you share your iface file ? It will help me validate
I no longer have the file precisely as it was. However, I did have some version of what I had used committed into my VCS. While not 100% sure this is what I was using while encountering the failure it does looks to be about what I'd expect:


Question: $(warning ***** finding all +k8s:conversion-gen tags) endif CONVERSION_DIRS := $(shell                                 <del> grep '^// *+k8s:conversion-gen=' $(ALL_K8S_TAG_FILES)  </del> <ins> grep --color=never '^// *+k8s:conversion-gen=' $(ALL_K8S_TAG_FILES)  </ins> | cut -f1 -d:                                      | xargs -n1 dirname                                | sort -u

Using the grep from OSX, fails when GREPOPTIONS/GREPCOLOR is set as shown below (). Workaround : unset GREPOPTIONS/GREPCOLOR Note that using GNU grep using brew install will show that GREPOPTIONS shows immediately that GREP_OPTIONS may be interfering with make ()
oh, here's badly manged conversion-
Will look, thanks. For make-related bugs, please @ me directly.
Will do. thanks
This breaks building kubernetes inside Alpine Linux:
I'm pressing more and more on the idea that the only supported build is the dockerized build. Trying to accomodate every random environment is just too taxing. , Marc Villacorta wrote:
I agree with We can support the stuff in to run on mac to get you dockerized but everything else is only supported in there. I've worked around a few BSD/OSX things in but it isn't fun. We should try to keep that stuff isolated.
I agree, with the usual caveat that we need to be doing all of the build in one container with as little as possible happening outside. OS X tends to have old and broken versions of important programs, which is terrible. We discussed setting up CI for it but it's not going to happen in the near term. Alpine/Busybox purposefully omit features from their versions for good reasons, but it's still more work to support than it's worth.
Ok, I will re-think my workflow.
How did you find yourself building in Alpine, in I may ask? , Marc Villacorta wrote:
nothing is forcing me to use Alpine I just like the distro. I run stuff on CoreOS and my aim is to solve the build-deliver-run pipeline with a single small container. By the way, I have the problem installing the GNU package :-)


Question: for _, minion := range matches.Items { if _, ok := minionMap[minion.Name]; !ok { glog.Infof("Create minion in registry: %s", minion.Name) <del> s.kubeClient.Minions().Create(&minion) </del> <ins> _, err = s.kubeClient.Minions().Create(&minion) if err != nil { glog.Errorf("Create minion error: %s", minion.Name) } </ins> } delete(minionMap, minion.Name) } for minionID := range minionMap { glog.Infof("Delete minion from registry: %s", minionID) <del> s.kubeClient.Minions().Delete(minionID) </del> <ins> err = s.kubeClient.Minions().Delete(minionID) if err != nil { glog.Errorf("Delete minion error: %s", minionID) } </ins> } return nil }

gives but gives If minions were already in etcd from some previous launch of the apiserver/controller things work just fine no matter what the order. So to reproduce you have to clear everything from etcd before you try...
can you take a look at this? Probably just need to retry in background until etcd comes up?
Yep, there is no control loop for static machine config. Adding one should be trivial.
Blocked by Basically, apiserver returns inconsistent response.
I have observed a similar sort of thing, but what it appeared to be is more like the is not listening on it's socket at the time tries to connect to it and it never retries. So my work-around is to run before it like so: What I then observed is that the connection is indeed being refused at first attempt, but works on the second.
what do you see in the logs? I was seeing that it basically fails to connect to once and doesn't retry (although it said that it will try to retry).
should fix it, sorry for the trouble wondering where did you see that "it will try to retry"?
that wasn't the exact wording there... let me see if I can find out what exactly it said.
Ok, so what I'm seeing now is: I hope that PIDs would be a good enough proof to what order services have started up: I could provide some systemd graph dumps, but that would probably be too verbose, PIDs seem clearer. So really don't see how got //apisevercontroller-managerapiserver` didn't open the socket yet...


Question: // TestListSandboxes creates several sandboxes and then list them to check // whether the correct metadatas, states, and labels are returned. func TestListSandboxes(t *testing.T) { <del> ds, _, _ := newTestDockerService() </del> <ins> ds, _, fakeClock := newTestDockerService() </ins> name, namespace := "foo", "bar" configs := []*runtimeapi.PodSandboxConfig{} for i := 0; i < 3; i++ {

https://k8s- Pod has a restart count 1 FYI, all relevant messages about this pod in the From the sync loop timeline, some containers died. Check the log again: The sandbox was GC'd right after it was created. This led to kubelet killing all the containers in the pod since the sandbox needed to be recreated. Need to check whether this was caused by the container state emulated by fakedockerclient, or by a real bug. /cc
/cc FYI.
kuberuntime GC routine has a minAge protection such that it can't GC any pod sandbox that was created no more than 30 seconds ago). This doesn't seem to have be respected from the log. I suspect that the fakedockerclient was not returning a proper timestamp.
GC logic might be faulty. I see a similar failure in https://k8s- EDIT: The node name is The sandbox was created and removed within 2 seconds
- by looking at the logs, can you tell if there's an easy repro?
So given that this happens also in non-kubemark suites, it seems like a release blocker to me. Anyway - - do you think it's releated to number of pod on a node, or it's just that it happens from time to time, and in scalability suites (no matter if it's kubemark or not) we have more of those tries (due to scale)?
I think the timestamp comparison logic might be wrong in the sandbox garbage collector. This would only happen (in my theory) when the the snapshot of all containers/sandboxes on the node are taken when: sandbox is created but not running (started) yet. other containers in the pod haven't not been created. In this case, the fail-safe (minAge) kicks in, and apparently failed in the tests discussed above. This would more likely to happen when during batch creation of more pods. kubelet would usually heal itself by restarting all container in the pod (which has the undesirable side-effect of killing the user containers of course). I think other tests might not catch this because 1) they don't perform large batch creation, and 2) they don't check the restart count.
Oh the above comment was completely based on my theory.

Reproducing this deterministically on a working node may need some hacking in the kubelet code (e.g., inject a long enough sleep after sandbox creation). I will try writing a unit test first to verify the behavior.
Can't reproduce this in my cluster with artificially injected latency... Adding more debug information in For people who are interested in checking the logs and identify the victim pod, here are the instructions: Download the artifacts: E.g., Find the (density) pod with restart count 0:
Didn't reproduce the problem either. I think so. GC can't distinguish just-created sandboxes from failed sandboxes. Especially when the time from sandbox creation to ready is longer than GC's minAge. I'm wondering should runtime (e.g. dockershim) filter out those created-but-not-ready sandboxes from sandbox list? or introduce another sandbox state in CRI?
has been merged 11 hours ago and scalability suite hasn't flaked since. Will monitor and see if there is any need to reopen.
OK. But because , there are still potential problems under big pressure.
First, this issue is no longer a release blocker after the fix. Let's open a separate issue to discuss potential improvement, instead of commenting on a closed issue :) Also, FWIW, we used the same logic in the old docker manager, so this is also not a regression.
Agreed. Opened for this.


Question: deletionTimestampNotPastGracePeriod := metav1.NewTime(fakeClock.Now()) testCases := map[string]struct { <del> pod   *api.Pod usage corev1.ResourceList </del> <ins> pod                *api.Pod usage              corev1.ResourceList podOverheadEnabled bool </ins> }{ "init container CPU": { pod: &api.Pod{

assume that: cluster with only one node with 4G allocatable memory Define test runtimeclass with PodOverhead Define resourceQuota test under test namespace with 4Gi memory limit Create Pod with test runtimeClass under test namespace podOverhead resource not counted into resourceQuota used info podOverhead resource counted into node requests info try to create one more pod with 2G memory requests, resourceQuota check passed but failed to schedule because it requests 2G+200M memory podOverhead resource should be counted into resourceQuota used info to keep resourceQuota check and schedule check be consistent. Kubernetes version 1.16
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/sig node
ref:
-- can you verify that you have the feature gate enabled for pod overhead for api-server, as well as kubelet and scheduler? Example:
I made a cluster with the following command, using crio as a container runtime( is not available with Docker) And the issue was reproduced.
overhead feature enabled, the overhead resource has been counted into requests during scheduling but not counted into resourcequota
Currently, PodOverhead is defined as a fixed resource. this value looks like the average usage value and being into requests resource during scheduling. type Overhead struct { PodFixed *ResourceList } My use case is kata_container, the VMM related processes consume much overhead resource when pod running with high workload. A small overhead(small cgroup limits) will trigger oom but a big value result that big overhead requests take into account during scheduling. So how about define PodOverhead including limits and requests like that in Pod resources definition. It could be helpful resources: limits: memory: 1Gi requests: memory: 200M
CC: FYI


Question: } if err = watcher.AddWatch(path, inotify.InOpen|inotify.InDeleteSelf); err != nil { klog.ErrorS(err, "Unable to watch lockfile") <ins> watcher.Close() </ins> return err } go func() {

kubelet increased more than 100MB in memory in 4 days. kubelet to not used that amount of RAM I have reproduced it using 3 cronjobs which create hello world pods each minute and another cronjob that writes a line to a file each minute. Kubernetes version (use ): v1.18.1 Cloud provider or hardware configuration happens on both virtualbox and baremetal OS (e.g: ): centos 7 Kernel (e.g. ): 18.0-147.3.1 I have seen some existing issues like but I do not have any dangling resources associated with the cgroup.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
/assign This is a duplicate issue, there has been a fixed PR but it has not been merged.

What happened: i wrote a shell then the kubelet's RSS increases continuously and does not release i make some trace and found some watched objects have never been released even these objects have not existed in real word i notice the fucntion readEvent is producer for these abnormal objects and print the length of every eventsome of which is beyond the length of buf even when i increase the array by 10 timesthe problem is still there and the limits are broken through again and again. so i realized the process of reading cannot be blocked. now i a queue for the buf of every inotify event and another goroutine to consume the queue. everythings seems good for now. What you expected to happen: fixed the bug How to reproduce it (as minimally and precisely as possible): running the shell : Anything else we need to know?: Environment: Kubernetes version (use ):1.17.9 Cloud provider or hardware configuration: OS (e.g: ):Kernel (e.g. ):Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig node /area kubelet
/triage unresolved
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
I didn't repeat the question. Is there a way of stable recurrence? Thx!
you could use the shell i'v paste above, or make "add" more faster than "delete".....
/assign
/reopen
You can't reopen an issue/PR unless you authored it or you are a collaborator. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available .

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The version mentioned is quite old. Can you please try to repro this on supported version of Kubernetes? /unassign /triage needs-information /remove-triage unresolved
The Kubernetes project currently lacks enough contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue with Mark this issue as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close not-planned [1]:
Closing this issue, marking it as "Not Planned". <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue with Mark this issue as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: visibility = ["//visibility:public"], deps = [ "//pkg/kubelet/apis/stats/v1alpha1:go_default_library", <ins> "//pkg/kubelet/events:go_default_library", </ins> "//pkg/kubelet/runtimeclass/testing:go_default_library", "//pkg/master/ports:go_default_library", "//pkg/util/slice:go_default_library",

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: For a Pod with , kubelet would attempt to create a new sandbox for it even after all container succeeded. Logs as below: This issue is similar to which fixed the case of pods whose RestartPolicy are Never. It caused unnecessary CRI and CNI calls, confusing logs and conflicts between the routine that creates the new sandbox and the routine that kills the Pod. It's likely also the root cause of because sometimes the routine that creates the second sandbox and the routine that kills unwanted pod run concurrently, then the netns would be gone when calling CNI. What you expected to happen: kubelet shouldn't create a new sandbox for pod with RestartPolicyOnFailure if all containers succeeded How to reproduce it (as minimally and precisely as possible): Create a pod like this: Anything else we need to know?: Environment: Kubernetes version (use ): v1.18.2 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig node
Created which may fix this.
/assign


Question: framework.ExpectNodeHasLabel(cs, nodeName, k, v) defer framework.RemoveLabelOffNode(cs, nodeName, k) <del> By("Trying to create a pod(pod4) with hostport 80 and hostIP 0.0.0.0(empty string here) and expect scheduled") creatHostPortPodOnNode(f, "pod4", ns, "", v1.ProtocolTCP, nodeSelector, true) </del> <ins> port := int32(54322) By(fmt.Sprintf("Trying to create a pod(pod4) with hostport %v and hostIP 0.0.0.0(empty string here) and expect scheduled", port)) creatHostPortPodOnNode(f, "pod4", ns, "", port, v1.ProtocolTCP, nodeSelector, true) </ins> <del> By("Trying to create another pod(pod5) with hostport 80 but hostIP 127.0.0.1 on the node which pod4 resides and expect not scheduled") creatHostPortPodOnNode(f, "pod5", ns, "127.0.0.1", v1.ProtocolTCP, nodeSelector, false) </del> <ins> By(fmt.Sprintf("Trying to create another pod(pod5) with hostport %v but hostIP 127.0.0.1 on the node which pod4 resides and expect not scheduled", port)) creatHostPortPodOnNode(f, "pod5", ns, "127.0.0.1", port, v1.ProtocolTCP, nodeSelector, false) </ins> }) })

The test uses port 80, which is often in use on clusters (in our case, masters have an ingress controller listening on 80). The test should use something in the high range that is unlikely to conflict or use a smarter algorithm for finding a port to validate. fails in openshift
I am looking into it.
I looked into the test and it seems that another option would be for the test to not use 127.0.0.1 to avoid such conflict with infra (or any other important) pods on 127.0.0.1. As the test is using 127.0.0.1 and 127.0.0.2, I think we could just use 127.0.0.2 and 127.0.0.3 instead.
so it seems there are following options we can use: 127.0.0.2 and 127.0.0.3 instead of 127.0.0.1 and 127.0.0.2. pick some high random port number. something like following to pick unused port: and then extract random port from it.
with option 1 as it seems safer than 2 to me. Let me know if there is any concern, or if we would want to go with 2 or 3 option or something else.


Question: }, "io.k8s.apimachinery.pkg.runtime.RawExtension": { "description": "RawExtension is used to hold extensions in external versions.nnTo use this, make a field which has RawExtension as its type in your external, versioned struct, and Object in your internal struct. You also need to register your various plugin types.nn// Internal package: type MyAPIObject struct {ntruntime.TypeMeta `json:",inline"`ntMyPlugin runtime.Object `json:"myPlugin"`n} type PluginA struct {ntAOption string `json:"aOption"`n}nn// External package: type MyAPIObject struct {ntruntime.TypeMeta `json:",inline"`ntMyPlugin runtime.RawExtension `json:"myPlugin"`n} type PluginA struct {ntAOption string `json:"aOption"`n}nn// On the wire, the JSON will look something like this: {nt"kind":"MyAPIObject",nt"apiVersion":"v1",nt"myPlugin": {ntt"kind":"PluginA",ntt"aOption":"foo",nt},n}nnSo what happens? Decode first uses json or yaml to unmarshal the serialized data into your external MyAPIObject. That causes the raw JSON to be stored, but not unpacked. The next step is to copy (using pkg/conversion) into the internal struct. The runtime package's DefaultScheme has conversion functions installed which will unpack the JSON stored in RawExtension, turning it into the correct object type, and storing it in the Object. (TODO: In the case where the object is of an unknown type, a runtime.Unknown object will be created and stored.)", <del> "properties": { "Raw": { "description": "Raw is the underlying serialization of this object.", "format": "byte", "type": "string" } }, "required": [ "Raw" ], </del> "type": "object" }, "io.k8s.apimachinery.pkg.util.intstr.IntOrString": {

The Service Catalog includes the type in . The openapi schema generated for it is wrong. It has two problems: it includes the field, which is marked as required it doesn't allow additional properties to be included under it Is this a BUG REPORT or FEATURE REQUEST?: BUG REPORT /kind bug What happened: If the Service Catalog openapi schema is exposed to kubectl (this currently requires a few tweaks and a patch), it reports validation errors when creating a ServiceInstance resource. What you expected to happen: The schema should allow to include arbitrary properties and not require the property to be included. How to reproduce it (as minimally and precisely as possible): not easy at the moment (I'll add instructions later if I'm unable to fix this myself) Anything else we need to know?: I'd like to tackle this problem myself, but I need guidance. Should openapi-gen handle this automatically for all occurences of RawExtension, or should each field of type be json-tagged individually? Environment: Kubernetes version (use ):
I've heard you're the goto guy for the openapi-gen. Please help? :)
/sig api-machinery
OK, I've found I can specify a custom schema for RawExtension the way it's done in Time and MicroTime. The remaining problem is how to specify that the RawExtension object can be of any type. Using is not supported currently - if I use it in the schema, the API server dies with:
/assign /cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
We've just hit this. /remove-lifecycle rotten /lifecycle frozen
hitting this
According to this: the raw extension can define its openapi type as: or alternatively what do you think?

do you still work on this or you want me to send a PR?
Would changing the definition of RawExtension to this help the situation? (adding ) I noticed that this removes Raw from the openapi defintion of it, without changing how it is marshalled or unmarshalled (since we have custom functions for that) I'll try making a PR for this


Question: service, err = t.CreateService(service) Expect(err).NotTo(HaveOccurred()) }) <ins> It("should create endpoints for unready pods", func() { serviceName := "never-ready" ns := f.Namespace.Name t := NewServerTest(c, ns, serviceName) defer func() { defer GinkgoRecover() errs := t.Cleanup() if len(errs) != 0 { framework.Failf("errors in cleanup: %v", errs) } }() service := t.BuildServiceSpec() service.Annotations = map[string]string{endpoint.TolerateUnreadyEndpointsAnnotation: "true"} rcSpec := rcByNameContainer(t.name, 1, t.image, t.Labels, api.Container{ Name:  t.name, Image: t.image, Ports: []api.ContainerPort{{ContainerPort: int32(80), Protocol: api.ProtocolTCP}}, ReadinessProbe: &api.Probe{ Handler: api.Handler{ Exec: &api.ExecAction{ Command: []string{"/bin/false"}, }, }, }, }) By(fmt.Sprintf("createing RC %v with selectors %v", rcSpec.Name, rcSpec.Spec.Selector)) _, err := t.createRC(rcSpec) ExpectNoError(err) By(fmt.Sprintf("creating Service %v with selectors %v", service.Name, service.Spec.Selector)) _, err = t.CreateService(service) ExpectNoError(err) By("Verifying pods for RC " + t.name) ExpectNoError(framework.VerifyPods(t.Client, t.Namespace, t.name, false, 1)) svcName := fmt.Sprintf("%v.%v", serviceName, f.Namespace.Name) By("waiting for endpoints of Service with DNS name " + svcName) createExecPodOrFail(f.Client, f.Namespace.Name, "exec") cmd := fmt.Sprintf("wget -qO- %v", svcName) var stdout string if pollErr := wait.PollImmediate(framework.Poll, kubeProxyLagTimeout, func() (bool, error) { var err error stdout, err = framework.RunHostCmd(f.Namespace.Name, "exec", cmd) if err != nil { framework.Logf("expected un-ready endpoint for Service %v, stdout: %v, err %v", t.name, stdout, err) return false, nil } return true, nil }); pollErr != nil { framework.Failf("expected un-ready endpoint for Service %v within %v, stdout: %v", t.name, kubeProxyLagTimeout, stdout) } }) </ins> }) // updateService fetches a service, calls the update function on it,

This is mostly motivated by petset. Currently we create a governing service that is responsible for the network identity of each pet. The requirements of this governing service differ from a normal service in the following ways: have internal protocols to notice peer health, and a list of different checks (eg: ). In kube, peers use DNS to communicate. So we don't want a readiness check failure to remove the pet from DNS and abruptly cut it off from the set. "governing service" is headless so kube-proxy doesn't need to NAT the service ip. will not be using this Service, so it doesn't matter that unready pets show up in DNS under the svcName sub domain. In addition to these, there's one important reason some pets need to have DNS before readiness. To provide the guarantees required to safely bootstrap a pet, we need: 0 starts before pet1.. 't start pet1 till pet0 is running/ready pet1 starts, it sees pet0 in DNS The easiest way to guarantee 3 today is to do one of: poll in the entrypoint of pet0, till it sees itself poll in the init container of pet0, till it sees itself poll in the init containers of pet {1..N-1} till it sees all previous pets Third option is not a great idea because we don't know who else is supposed to exist, and for the first 2 to work with a readiness probe, we need a DNS record before the Service is ready.
I have a wip that does this through an annotation:
+1. this is useful even in traditional n rc's and n+1 services setup for clustered systems like n nodes zookeeper/etcd. Currently, there isn't a easy way to specify a readiness check in such setups, because the check in these cases would look for whether the pod member has joined the cluster successfully as follower, which wouldn't work as they discover each other through them appearing in service endpoints. this prevents one from being able to do a rolling upgrade of cluster members safely, because there is no readiness feedback after a member pod is upgraded.
Suggestion to hold this for alpha in 1.5, and in 1.6 spend a bit of time talking about how services are best intermixed with stateful sets (there's a lot of correlation between them).

That avoids us breaking people in 1.5 and gets a bit more think time.
SGTM
See also
Can we claim that if a pod is requesting pet DNS (i.e DNS name via the hostname/subdomain annotation) it never wants kube-style readiness, because it has its own internal protocol for peer health? If that's true we can get rid of the annotation and teach kubedns to always serve records for pet DNS ignoring readiness. My intuition tells me the answer to that question is no (i.e there exist pets that use DNS for discovery, and don't have an internal protocol for readiness, so they must use kube readiness to remove themselves from the peer group), and so the ability to toggle it on/off is useful, but I also want others to consider it.
I'd love to nix an annotation but I have not the expertise to answer this one.
I don't have a strong feel either way - I'd say readiness controls on a service feel orthogonal to independent names though. Since we don't have per port readiness, we need to consider that at some point we might need more sophistication on services or pods to control membership (unless we make label flipping easier). I would like to see us tease apart the "leader / follower" service member concept a bit more because that might inform how we handle readiness (seems like a separate bit, but we may want atomic changes).
They are somewhat orthogonal, but you presumably only want to discover healthy peers, unless you have a way to probe every peer you discover for health. Applying this policy based on endpoint info also feels a little weird, because it's harder to create overlapping Servics with different behavior. I think doing the common thing always and by default is fine, as long as we can annotate our way back to what we have today, but what form would the opposite of today's Service level annotation take? i.e something that says "even though my endpoints need pet names, apply the regular rediness behavior to their DNS records"
How bad would it be to start without an annotation, assume that pet-names == tolerate-unready, and see if anyone screams? Who should we consult?
SGTM, that's also what I was prposing in the previous comment
I guess the question is why do we even want to do that? Why make a change that couples logic vs leaving as is with decoupled logic?

We have people who want unready endpoints for deployments to form gossip pools - so I guess that counts as a use case where pet names is not coupled to unready. Is leaving the annotation on for another release really a problem? , Prashanth B <:
That's exactly the kind of use case I was looking for, something that doesn't require pet names. I get the feeling there will be other apps but I couldn't produce an example.
JBoss EAP uses it to form initial quorum for jgroups, which isn't a strong consensus system. So they use deployments and a minimum set to form an initial quorum and deal with split brain like jgroups normally does (infinispan has multiple consistent modes, some of which might prefer loose pools). Arguably smaller sets and some infinispan quorums might prefer statefulsets, but those use cases are also more tolerant of partitions in general. , Prashanth B <:
we are using tolerate-unready-endpoints annotation in our zookeeper deployments (not using pet/stateful set). Without unready endpoints, it is impossible to define a meaningful pod readiness check for zk, and without a readiness check, it is hard to do rolling updates (one ensemble member at a time).
The question is whether there's a need to have StatefulSets without tolerate-unready-endpoints. It sounds like maybe yes, which makes me grumpy, but I'll acquiesce if Clayton and Prashanth say so. , ravilr <:
isn't it enough to establish a need to tolerate-unready without statefulsets?
I have two services pointing to my etcd stateful set - one that is endpoints (peers) and tolerates unready. I have one that is for naive clients that uses readiness. On Dec 1, 2016, at 2:59 AM, Prashanth B <: isn't it enough to establish a need to tolerate-unready without statefulsets? — You are receiving this because you commented. Reply to this email directly, view it on GitHub , or mute the thread
fine.
/sig network
Right. This is exactly why etcd operator uses the alpha feature. Can we make it beta in 1.8 release?
/cc
We had discussed promoting since we need it for StatefulSet and no one has come up with anything better.

We should make a decision and include it in our GA roadmap for workload controllers.
Be sure to include sig-network On Jul 6, 2017, at 2:40 PM, Anthony Yeh <: had discussed promoting tolerate-unready-endpoints since we need it for StatefulSet and no one has come up with anything better. We should make a decision and include it in our GA roadmap for workload controllers. — You are receiving this because you commented. Reply to this email directly, view it on GitHub , or mute the thread
This is also useful for ingresses that need to retain the source IP using OnlyLocal node ports, which is important for firewalling with things like Calico. The problem that we see now is when a deployment occurs, the pod is removed from the node port before the PreStop completes, which doesn't give the upstream AWS load balancer time for the pod to fail out using normal health checks.
/close
Headless Services are a thing now


Question: metrics.LabelName:  name, metrics.LabelImage: image, "pod_name":         podName, <ins> "pod":              podName, </ins> "namespace":        namespace, "container_name":   containerName, <ins> "container":        containerName, </ins> } return set }

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What's happening: Currently, the metrics use the label "podname" to refer to the name of the pod in a timeseries and "containername" to refer to container names within that pod. kube-state-metrics, however, uses "pod" to refer to the names of pods and "container" to refer to names of containers. This difference makes it difficult to do joins on KSM metrics and cadvisor metrics, leading me to write queries like this one (which calculates the percentage of real utilization of quota per container -- namespace and podname templated in Grafana): With normalized labels I could just write Hence, I propose renaming the "podname" labels to "pod" and "containername" labels to "container" to normalize between the two data sources. I can attempt to write a patch for this. Why propose this here rather than telling KSM to match the cadvisor metrics? I reported this against KSM first: The indicate the usage of "container" (vs. containername), "pod" (vs. podname), etc.: Fixing this would ensure the cadvisor metrics conform with the community guidelines. Environment: Kubernetes version (use ): 1.8.7-21 with internal patches Kube-state-metrics image version: 1.2.0
/sig instrumentation
I just this to be discussed in the sig-instrumentation meeting, personally I'm all for it and we've discussed this a number of times. we should figure out the timeline for node-metrics being reworked, if that is going to take a while then I think we should make these changes.
considering that now basically has a dependent change that has to go through first, I think that's ok
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with .


Question: func TestNamespaceController(t *testing.T) { cluster1 := NewCluster("cluster1", api_v1.ConditionTrue) cluster2 := NewCluster("cluster2", api_v1.ConditionTrue) <ins> ns1 := api_v1.Namespace{ ObjectMeta: api_v1.ObjectMeta{ Name:     "test-namespace", SelfLink: "/api/v1/namespaces/test-namespace", }, } </ins> fakeClient := &fake_federation_release_1_4.Clientset{} RegisterFakeList("clusters", &fakeClient.Fake, &federation_api.ClusterList{Items: []federation_api.Cluster{*cluster1}})

In PR the specific resource types are handled explicitly. Need to delete them in more generic way, in a similar way that namespace controller does.
- switching this to 1.6 as it's too late for 1.5. ok? (please switch it right back if you disagree)
did fix this?
Please close or move to next-candidate with work remaining to be done.
Yes fixed by


Question: legacyMode bool } <ins> func detachDiskLogError(pd *gcePersistentDisk) { err := pd.manager.DetachDisk(pd, "/dev/disk/by-id/google-"+pd.pdName) if err != nil { glog.Warningf("Failed to detach disk: %v (%v)", pd, err) } } </ins> // SetUp attaches the disk and bind mounts to the volume path. func (pd *gcePersistentDisk) SetUp() error { if pd.legacyMode {

Repro: Create a pod that attempts to mount a GCE PD that is mis-formatted Delete the pod Expected: GCE PD should be detached Observed: GCE PD is not detached.
Chatted with offline and understand the problem with more detail: The basic problem is when filesystem on PD is unformatted, the mount is going to fail. Kubernetes doesn't do cleanup by detaching the PD. Two potential solutions were proposed: 1) use built-in utility: safeformatand_mount in GCE image to format and mount PD to avoid extra step to format the PD disk. 2) Handle mount failure properly. I think we might need handle both anyway. Brendan mentioned that you are looking at a potential new controller for PD, whose responsible is attaching and detaching PD disks. Do you have any latest status on this topic?
No latest status. I haven't done any actual work on it yet. The idea would be to scan the pods assigned to each node, and attaches (or detaches) any PDs that the node needs (or no longer needs). I'm still coming up to speed on a lot of the architectural concepts, so I'd love some guidance.
Now that Tim's volume re-factor has gone in, I'm going to re-visit this issue. Assigning to me.


Question: --- <del> - createCoreV1NamespacedPersistentVolumeClaim </del> - createCoreV1NamespacedServiceAccountToken - createCoreV1Node <del> - createCoreV1PersistentVolume </del> - createStorageV1CSINode - createStorageV1StorageClass - createStorageV1VolumeAttachment <del> - deleteCoreV1CollectionNamespacedPersistentVolumeClaim - deleteCoreV1CollectionPersistentVolume - deleteCoreV1NamespacedPersistentVolumeClaim </del> - deleteCoreV1Node <del> - deleteCoreV1PersistentVolume </del> - deleteStorageV1CollectionCSINode - deleteStorageV1CollectionStorageClass - deleteStorageV1CollectionVolumeAttachment

[X] test approval issue : [X] test pr : [x] two weeks soak end date : 11 Sept 2023 [x] test promotion pr : [x] remove endpoint from According to following APIsnoop query, there are two PersistentVolume and three PersistentVolumeClaim endpoints that are untested. Using a number of existing e2e test practices a new has been created to provide future Conformance coverage for the 13 endpoints. The e2e logs for this test are listed below. This query shows the following PV and PVC endpoints are hit within a short period of running this e2e test. If a test with these calls gets merged, test coverage will go up by 13 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted
/cc
/reopen For tracking the rest of the steps to Promotion
Reopened this issue. <detailsIn response to : /reopen For tracking the rest of the steps to Promotion Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: tag kube-scheduler </source> <del> <filter kubernetes.**> type kubernetes_metadata </filter> </del> <ins> #<filter kubernetes.**> #  type kubernetes_metadata #</filter> </ins> # Example: # I0603 15:31:05.793605       6 cluster_manager.go:230] Reading config from path /etc/gce.conf

It's 400+mb and we need to pull it on every node. This leads to a lot of image pulling flake (), and also gets very expensive in some countries where bandwidth is sold at a premium. We took a stab at halving the size with: , but then realized that we were using an unsupported base image We tried to optimize to ~300Mb by building a smaller ubuntu:15 base, but that's still huge. Every other mandatory system daemon is built on busybox and so takes ~10Mb.
I can try again now that the agent supports
Newer Docker images using the spec from the official repository are based on AlpineLinux and are around ~38MB:


Question: if err != nil { return nil, false, err } <del> cr = obj.(*unstructured.Unstructured) </del> <ins> cr := obj.(*unstructured.Unstructured) </ins> newScale, _, err := scaleFromCustomResource(cr, r.specReplicasPath, r.statusReplicasPath, r.labelSelectorPath) if err != nil { return nil, false, apierrors.NewBadRequest(err.Error()) } return newScale, false, err }

What happened: Using a command such as: fails with the following error: This bug happens on each run and started occurring as of v1.15.0. With v1.14 or older, I don't have such problem, i.e. it works flawlessly. What you expected to happen: The command succeeds: How to reproduce it (as minimally and precisely as possible): Create a CRD that has the scale subresource and try to scale a CR using . Example of such CRD can be found . Anything else we need to know?: Regarding the bug, it seems that it started appearing as of . Before the PR, we always fetched the resource and it worked because it had the resource version, while now we are only fetching it in case preconditions are provided (preconditions are not provided by default/when using ). Here is the output of a command: If I manually craft the request that has the resource version (or use the flag), it works. Is it as expected that the Scale resource requires the resource version in order for update to succeed? Shouldn't it always pick the latest resource version in case it isn't provided? On that note, I also noticed that if you want to update a CR by crafting the request manually and using , you also need to specify the resource version or you will get the same error. I'm not sure if this is expected and related, but I'm writing it down as it might be related. As far as I know, this is not required for some native resources at all, but I'm not sure is this the case for CRDs. Environment: Kubernetes version (use ):
/sig api-machinery cc
/sub
could you try to see if it fix the problem?
I've created a test cluster based on and I can confirm that I'm able to scale CustomResources using as usual. I used an to test this, such as: Thanks a lot for your time spent fixing this!
Bug triage for 1.17 here with a gentle reminder that code freeze for this release is on November 14. Is this issue still intended for 1.17? I see you move it to 1.17 so I just wanted to be sure.
I guess could be merged now?

It is in the review queue and should still make 1.17
a kind reminder that code freeze for this release is on November 14.
I wanted to circle back around on this. We are 4 days out from code freeze. If this is still good based on just note here. Judging from the looks of it we are still good
Yes this should merge early in the week
Code freeze is tomorrow, this still going to make it?
yes, is still planned


Question: metrics:                    metrics, unfinishedWorkUpdatePeriod: updatePeriod, } <del> go t.updateUnfinishedWorkLoop() </del> <ins> // Don't start the goroutine for a type of noMetrics so we don't consume // resources unnecessarily if _, ok := metrics.(noMetrics); !ok { go t.updateUnfinishedWorkLoop() } </ins> return t }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When we scale Rancher to 1000's of clusters we end up spending a large percentage of our time waking up and doing nothing across 10s of thousands of goroutines. This was seen as constantly consuming 2-3 cores while the system was in an 'idle' state with approx. 78k queues being active. What you expected to happen: I would expect these goroutines to not be running if there is no metrics for them to write to. How to reproduce it (as minimally and precisely as possible): Calling in a loop Anything else we need to know?: Environment: Kubernetes version (use ): v1.19.1 Cloud provider or hardware configuration: DO droplets OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig api-machinery
/cc
/assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: "pkg/version", ], tags = ["automanaged"], <ins> vendor_prefix = openapi_vendor_prefix, </ins> vendor_targets = [ "k8s.io/api/admission/v1alpha1", "k8s.io/api/admissionregistration/v1alpha1",

As part of moving federation out of tree (), the federation build will need to generate ` for itself and vendored kubernetes. The following steps are required: [x] Discover targets for generation () [x] Update the file's bazel rules to configurably enable a build for vendored kube () [x] Support generation of the file to a vendored path () [x] Update kubernetes to use latest kazel and kube-openapi () /sig multicluster testing /area federation /assign cc:
/cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale


Question: func (proxier *Proxier) OnEndpointsSynced() { proxier.mu.Lock() proxier.endpointsSynced = true <ins> proxier.setInitialized(proxier.servicesSynced && proxier.endpointsSynced) </ins> proxier.mu.Unlock() <ins> // Sync unconditionally - this is called once per lifetime. </ins> proxier.syncProxyRules() }

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Many service related test cases in test case are flaky. Especially related cases. CI Link: http://k8s- What you expected to happen: IPVS should sync rules within . How to reproduce it (as minimally and precisely as possible): a cluster with ipvs proxy mode, and specify . a service with backends. check the delay from create service to sync ipvs rules. (This issue is probabilistic, about 50% possibility to happened.)
/sig network /area ipvs
/assign


Question: func NewDefaultStatefulSetControl( podControl StatefulPodControlInterface, statusUpdater StatefulSetStatusUpdaterInterface, <del> controllerHistory history.Interface) StatefulSetControlInterface { return &defaultStatefulSetControl{podControl, statusUpdater, controllerHistory} </del> <ins> controllerHistory history.Interface, recorder record.EventRecorder) StatefulSetControlInterface { return &defaultStatefulSetControl{podControl, statusUpdater, controllerHistory, recorder} </ins> } type defaultStatefulSetControl struct { podControl        StatefulPodControlInterface statusUpdater     StatefulSetStatusUpdaterInterface controllerHistory history.Interface <ins> recorder          record.EventRecorder </ins> } // UpdateStatefulSet executes the core logic loop for a stateful set, applying the predictable and

An issue to make bot happy for When StatefulSet finds any of its pods in phase Failed it will delete it and create it again. While doing so it will emit delete+create event which is really confusing for the user as he has no idea what's happening. /sig apps


Question: Properties: map[string]spec.Schema{ "maxUnavailable": { SchemaProps: spec.SchemaProps{ <del> Description: "The maximum number of pods that can be unavailable during the update. Value can be an absolute number (ex: 5) or a percentage of total pods at the start of update (ex: 10%). Absolute number is calculated from percentage by rounding up. This can not be 0 if MaxSurge is 0. By default, a fixed value of 1 is used. Example: when this is set to 30%, the old RC can be scaled down by 30% immediately when the rolling update starts. Once new pods are ready, old RC can be scaled down further, followed by scaling up the new RC, ensuring that at least 70% of original number of pods are available at all times during the update.", </del> <ins> Description: "The maximum number of pods that can be unavailable during the update. Value can be an absolute number (ex: 5) or a percentage of total pods at the start of update (ex: 10%). Absolute number is calculated from percentage by rounding down. This can not be 0 if MaxSurge is 0. By default, a fixed value of 1 is used. Example: when this is set to 30%, the old RC can be scaled down by 30% immediately when the rolling update starts. Once new pods are ready, old RC can be scaled down further, followed by scaling up the new RC, ensuring that at least 70% of original number of pods are available at all times during the update.", </ins> Ref:         spec.MustCreateRef("#/definitions/intstr.IntOrString"), }, },

API documentation says is rounded up, but it is actually rounded down: The code shows it is rounded down, not up: and
Yes, the documentation is obsolete. We changed maxUnavailable to round down so that we wouldn't accidentally infer 100% unavailability for non-100 percentages. For example, maxSurge=25% and maxUanavailable=25% with a Deployment of 1 replica meant that we could scale down 1 before thus leading in 100% maxUnavailable (implicit).


Question: return false } <del> // Needs to allocate additional devices. if m.allocatedDevices[resource] == nil { m.allocatedDevices[resource] = sets.New[string]() } </del> // Allocates from reusableDevices list first. if allocateRemainingFrom(reusableDevices) { return allocated, nil

It is suspected to be a concurrent bug of kubelet. When we delete the last pod that requires GPU on the node and create two or more pods that require GPU at the same time, kubelet may panic on as following. I think it's a concurrency issue. a pod with GPU request, then kubelet will allocate resource for it. Kubelet try call device plugin to choose preferred device for it. It will unlock the mutex!. we delete the last pod that requires GPU. the second pod with GPU request, kubelet will allocate resource for it also. Kubelet will call to recalculate device (delete allocated device if pod deleted). It will update the allocatedDevices map of ManagerImpl, because of mutex is unlock by step 1. Kubelet continues with the first pod. If allocatedDevices has changed and GPU device is nil, kubelet will panic in the closure allocateRemainingFrom. Don't panic. Delete the last pod that requires GPU on the node and create two or more pods that require GPU at the same time. No response <details</details<detailsexternal </details<details</detailsNOOP <detailsContainerd </detailsNOOP
/assign
/sig node
Is this reproduced in the master branch?
It is possible from the code. I will try to reproduce it later. Hope someone can give me some ideas also.
/cc Could you take a look? The issue scenario looks real to me.
hi, this looks legit, but OTOH a key assumption in the resource management allocation is that all the pods are admitted serially (e.g. NOT concurrently) one after the another. In other words, a key assumption is we never try allocating resources while another allocation is ongoing, e.g. not completed. So is not obvious to me which flows are conflicting. [EDIT] could still be if there is a client accessing the podresources API:
/triage accepted /priority important-soon I think the issue described here is real but I'm not sure how often it appears. Panics are always bad, but how often do we know this is happening? Let's start with as priority [EDIT] subejct to further adjustments as the investigation progresses.
Sure. Kubelet will allocate resources to pods when admitting. Admit is sequential.

As long as it is not device re-allocate, the problem will not occur. Podresources API is suspicious.
I deployed the dcgm exporter and it does call the interface of podresources.
This is a genuine and reproducible issue. We have observed the kubelet crashing due to such a panic when inserting a device into a map (): In our observation, right after deleting the pod, the podresources API was called. This removed the state of for the GPU resource. This happened when the was called as soon as the mutex got unlocked in the . Moreover, such a race seems being triggered when Device Plugin periodically calls podresources API.
Thanks, this information matches with other findings. The podresources endpoint intentionally triggers to report the current up to date allocation; otherwise the device manager would clean up lazily the allocation state when a new admission is attempted. We will work on a fix.
Just to add we've also seen this a few times on 1.23 and 1.24 clusters in EKS with the nvidia-device-plugin + DCGM-Exporter
This looks like a real problem. I unit tests to verify that the PR works.
not taking for 1.29, but may consider for a cherry-pick later. Depending on the fix complexity
This issue is labeled with but has not been updated in over 90 days, and should be re-triaged. Important-soon issues must be staffed and worked on either currently, or very soon, ideally in time for the next release. You can: Confirm that this issue is still relevant with (org members only) Deprioritize it with or Close this issue with For more details on the triage process, see /remove-triage accepted
/triage accepted /priority important-longterm because low capacity
/remove-priority important-soon


Question: } // createBatchPodSequential creates pods back-to-back in sequence. <del> func createBatchPodSequential(f *framework.Framework, pods []*v1.Pod) (time.Duration, []e2emetrics.PodLatencyData) { </del> <ins> func createBatchPodSequential(f *framework.Framework, pods []*v1.Pod, podType string) (time.Duration, []e2emetrics.PodLatencyData) { var ( mutex       = &sync.Mutex{} watchTimes  = make(map[string]metav1.Time, 0) stopCh      = make(chan struct{}) firstCreate metav1.Time lastRunning metav1.Time init        = true ) // the controller watches the change of pod status controller := newInformerWatchPod(f, mutex, watchTimes, podType) go controller.Run(stopCh) defer close(stopCh) </ins> batchStartTime := metav1.Now() e2eLags := make([]e2emetrics.PodLatencyData, 0) <ins> createTimes := make(map[string]metav1.Time) </ins> for _, pod := range pods { create := metav1.Now() <del> f.PodClient().CreateSync(pod) </del> <ins> createTimes[pod.Name] = create p := f.PodClient().Create(pod) framework.ExpectNoError(wait.PollImmediate(2*time.Second, framework.PodStartTimeout, podWatchedRunning(watchTimes, p.Name))) </ins> e2eLags = append(e2eLags, <del> e2emetrics.PodLatencyData{Name: pod.Name, Latency: metav1.Now().Time.Sub(create.Time)}) </del> <ins> e2emetrics.PodLatencyData{Name: pod.Name, Latency: watchTimes[pod.Name].Time.Sub(create.Time)}) </ins> } <del> batchLag := metav1.Now().Time.Sub(batchStartTime.Time) </del> <ins> for name, create := range createTimes { watch, ok := watchTimes[name] framework.ExpectEqual(ok, true) if !init { if firstCreate.Time.After(create.Time) { firstCreate = create } if lastRunning.Time.Before(watch.Time) { lastRunning = watch } } else { init = false firstCreate, lastRunning = create, watch } } batchLag := lastRunning.Time.Sub(batchStartTime.Time) </ins> sort.Sort(e2emetrics.LatencySlice(e2eLags)) return batchLag, e2eLags } <ins> // podWatchedRunning verifies whether the pod becomes Running, as the watchTime was set by informer func podWatchedRunning(watchTimes map[string]metav1.Time, podName string) wait.ConditionFunc { return func() (done bool, err error) { if _, found := watchTimes[podName]; found { return true, nil } return false, nil } } </ins> // verifyLatencyWithinThreshold verifies whether 50, 90 and 99th percentiles of a latency metric are // within the expected threshold. func verifyLatencyWithinThreshold(threshold, actual e2emetrics.LatencyMetric, metricName string) error {

Test: [sig-node] Density [Serial] [Slow] create a sequence of pods latency/resource should be within limit when create 10 pods with 50 background pods /kind flake <!-- If this is a flake, please add: /kind flake --/sig node
/triage accepted /priority important-soon
Although the pod polling exhausted 4 times (interval 2s, total 6s+) to get pod Running, but from the kubelet log, the time from the pod found to Running status reported is almost less than 5s, so wo may reduce the poll interval to 1s, or make this time more relaxed
http://perf- ! This is caused by as . The commit will be reverted, and it make fail yesterday. ! EDITED I mean if this flake points out there are some other performance downgrades in v1.24. Only recently, no version 99% -- -- v1.20 ~2200ms v1.21 ~2200ms v1.22 ~2300ms v1.23 ~2650ms v1.24(before) ~2200ms v1.24(after ) ~6600ms
Editing title as this was also failing on containerd: /milestone v1.24 I think this confirms that the revert above fixed the issue, as the failures stopped after the revert merged. /close
Closing this issue. <detailsIn response to : Editing title as this was also failing on containerd: Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: w: &webhook.GenericWebhook{ RestClient:     rc, InitialBackoff: initialBackoff, <ins> ShouldRetry:    retryOnError, </ins> }, name: fmt.Sprintf("dynamic_%s", PluginName), }

What would you like to be : The api server should retry sending audit events to configured webhook sinks regardless of the type of HTTP error code it might receive from the webhooks. Why is this needed: While some components may live with failed data transmissions, this is a particularly sensitive topic for the audit area. The API server audit mechanisms must ensure that no other reasonable option exists before dropping an audit event that may be the only trace of malicious behavior in the cluster. It seems that the current implementation may be improved further in this direction. Currently, when the API server is configured for non-blocking sending of audit events to a webhook audit sink backend, it will retry up to 5 times with exponential backoff, should there be an error in communicating with the webhook, as implemented . However, not all errors will trigger a retry. Some are considered transient and yield retry. Others simply stop the retry cycle. For example, it retries on receiving HTTP code , but not on receiving . The list of error codes that will yield a retry is actually quite small - three HTTP errors and one network error. Any other error will not trigger a retry and the audit events will be lost. The code comments explain that the retry will consider only recoverable failures, which may make sense in other cases, but as far as sending audit events to an audit backend webhook is concerned, it is not sufficient policy. A more robust approach would be to consider any error and give the backend service a chance to recover instead of relying on the error codes semantics and the webhook implementations interpretation of it. That is more inline with having the transmission of audit events as top priority. /sig auth /sig api-machinery


Question: } isListFromCache := !shouldListFromStorage(query, &listOptions) <del> count, err := e.countGetterFn(key(requestInfo)) </del> <ins> numStored, err := e.countGetterFn(key(requestInfo)) </ins> switch { case err == ObjectCountStaleErr: // object count going stale is indicative of degradation, so we should

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --I noticed that does not make sense to me. Looking at how LIST requests are served (which is NOT easy, there is so much dependency injection), I concluded that the worst case number of objects touched in the apiserver equals the pagination limit only if all of the following are true. The LIST request is served from etcd rather than the watch cache. The pagination FeatureGate is enabled. The request has a specified limit and it is no greater than the number of objects in etcd (restricted by namespace if relevant). The request does not have a label selector or field selector. I expected the list work estimator to use the pagination limit only under the conditions listed above. Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
I commented on the PR.
Thank you both Mike and Wojciech! /triage accepted


Question: for _, pod := range ns.Pods { framework.Logf("creating/updating pod %s/%s", ns.Name, pod.Name) <ins> // note that we defer the logic of pod (i.e. node selector) specifics to the model // which is aware of linux vs windows pods </ins> kubePod, err := k.createPod(pod.KubePod()) if err != nil { return err } <del> createdPods = append(createdPods, kubePod) </del> <ins> createdPods = append(createdPods, kubePod) </ins> _, err = k.createService(pod.Service()) if err != nil { return err

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Ive done preliminary testing with on calico powered clusters on some windows variants and found that the probing seems to work. Can we add wrapper to netpol suite so that we can run it on windows clusters ? Why is this needed: We need this because otherwise, CNIs such as EKS will give us This error is caused when, for example, pods that are landing on a windows node havent defined their target platform. Somehow i guess pods which are landing on linux nodes dont have this issue. Its also needed for the obvious reason that we probably want separate linux and windows test suites for policies, in general, even though we expect the logic to be identical (modulo hostports , which we dont currently test)
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig windows
/assign
grepping around, I can't figure out where those are supposed to go. Can you give me pointers?
(resolved in slack, matts digging into this)
ok, these are running now on eks, lots of failures. will dig
/assign
the netpol are not being applied for the ns: Instead of failing because should not be able to access , the connection succeeds.


Question: "github.com/pkg/errors" authorizationv1beta1 "k8s.io/api/authorization/v1beta1" rbacv1beta1 "k8s.io/api/rbac/v1beta1" <del> apierrors "k8s.io/apimachinery/pkg/api/errors" </del> metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime/schema" "k8s.io/apimachinery/pkg/util/wait"

What would you like to be : The function in should check for GKE and allow a missing SubjectAccessReview endpoint. If we get a 404 on another provider an error should still be returned. GKE doesn't enable the SAR endpoint. Without this endpoint, we cannot determine if the policy engine has adjusted as expected. In this case, we currently wait one second and hope it's up to date without returning an error and that it has adjusted as expected. Why is this needed: A 404 on that endpoint is not expected behavior for any provider other than GKE so it doesn't make any sense to implicitly allow 404s and hope that everything is ok. If it is a special, allowable, condition for GKE we should make that explicit in the code. The problem with doing it now is that checking the provider requires access to the TestContext which forces us to import the entire package. This goes against the goal of At the same time, we don't want to cause every test/caller of this to have to (1) know that GKE acts differently in this situation and code that one special path. This may be the lesser of the evils though, but regardless of how it is accomplished, at some point, the code should be validating this flow. Open to other solutions to this; maybe a specific test to test this code path (in which case this helper can be assumed to not be responsible for it?) Not sure if that is considered sufficient as it creates an awkward coupling between a helper function's assumptions and the existence of particular test.
/sig testing /area test /area test-framework
I think that comment no longer applies, and we can just remove that GKE special case. IIRC, GKE didn't initially enable serving the group, but it does now.
OK; I will: confirm GKE handling paste link here for history Put up PR removing the special case entirely


Question: acceptPodName := framework.CreateExecPodOrFail(cs, namespace, "execpod-accept", nil) dropPodName := framework.CreateExecPodOrFail(cs, namespace, "execpod-drop", nil) <del> accpetPod, err := cs.Core().Pods(namespace).Get(acceptPodName, metav1.GetOptions{}) </del> <ins> acceptPod, err := cs.Core().Pods(namespace).Get(acceptPodName, metav1.GetOptions{}) </ins> Expect(err).NotTo(HaveOccurred()) dropPod, err := cs.Core().Pods(namespace).Get(dropPodName, metav1.GetOptions{}) Expect(err).NotTo(HaveOccurred())

Sub-issue for For AWS ELB, when an ELB is first created, is made it is very normal that it takes 2 minutes to be created, but the DNS name is reported fairly quickly.


Question: // OBSERVATION: pvc is "Pending", will retry switch { case delayBinding && !pvutil.IsDelayBindingProvisioning(claim): <del> ctrl.eventRecorder.Event(claim, v1.EventTypeNormal, events.WaitForFirstConsumer, "waiting for first consumer to be created before binding") </del> <ins> if err = ctrl.emitEventForUnboundDelayBindingClaim(claim); err != nil { return err } </ins> case v1helper.GetPersistentVolumeClaimClass(claim) != "": if err = ctrl.provisionClaim(claim); err != nil { return err

What happened: With a StorageClass with , it's not clear why a PVC is not bound when there is no available PV. On Pod, there is a clear event what's wrong: On the PVC side we get only this event: Note that the message says "waiting for first consumer'. But the pod already exists. The reason why it's not bound yet is that the StorageClass has no dynamic provisioning configured () and there are no free PVs to satisfy the claim. What you expected to happen: An event "1 node(s) didn't find available persistent volumes to bind" on the PVC.
/sig scheduling /sig storage
I tried to fix this in VolumeBinding predicate, The predicate is called for each pod - node combination, so it would generate event for each node. Which is too much and it's even wrong when there is a node that has an available PV. We need to evaluate all nodes and then, if the pod is still unschedulable, distribute events to all unbound PVCs of that pod. Scheduler does not offer any way how to track unbound PVCs at that stage - whole volume logic is hidden in VolumeBinding predicate. One option would be to extend  struct to contain also list of objects where to send events (except the pod itself): Scheduler would then send all failed scheduling events it sends to a pod to all the , without any interpretation. is used to manage duplicates easily - each object (its UID) can be in the map only once. Simple array would work too, but then it needs extra work in removing duplicates. Not sure how easy it is to detect duplicate instances. is extensible to allow any object type. predicate would put all unbound PVCs there. does not need any special handling of each PVC. Message "2 node(s) didn't find available persistent volumes to bind" sounds just right in case a pod uses multiple PVCs, each of them might be bind-able on different nodes (node A may satisfy PVC1, but not PVC2, while node B may satisfy PVC2 only). what do you think about such design?
We used to have extension point, which I believe is the right moment for recording the event for unbounds PVCs. But recently it's been refactored so doesn't fit. Do you think this case can be a good example of using as an informational extension point?

Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale wdyt?
I have some other concerns. What if the pod is deleted? Should the scheduler send events that PVC is not in volume binding? The PV controller must be updated to stop emit event in this scenario. It might be simpler to solve on the PV controller side. In this function, we can check if the PVC is being used by a pod. If it is, we can send event like , then people can go to check the progress of pod "foo/bar". If the pod "foo/bar" is deleted, the PV controller can emit "waiting for first consumer ..." event again. What do you think? cc
Sounds better than it's now. My only issue may be performance - PV controller has to check if a pod exists, i.e. list all pods in the same namespace as PVC being reconciled. I am probably too worried.
current attach detach controller has a custom indexer to index pods by PVC keys to avoid iterating all pods every time to find pods which reference given PVC. And there are other places we need to check if a PVC is used by a pod or not from pod informer, e.g. pvcprotectioncontroller pvcontroller we can try to share this custom indexer among these volume controllers in the future. This will improve the overall performance of the IMO. What do you think
Yes, using the indexer looks like a good idea.
/remove-sig scheduling I also agree that solving in PVC side seems like a better idea. Thanks
/assign unbound claims are reconciled in the PV controller every 15 seconds (default), we may have bad worst-case performance (e.g. too many pods in the same namespace). we can add an indexer in the PV controller locally to solve this issue first (also optimize ), then find a way to in manager to improve the overall performance.


Question: func addDebugFlags(cmd *cobra.Command, opt *DebugOptions) { cmd.Flags().BoolVar(&opt.ArgsOnly, "arguments-only", opt.ArgsOnly, i18n.T("If specified, everything after -- will be passed to the new container as Args instead of Command.")) <del> cmd.Flags().BoolVar(&opt.Attach, "attach", opt.Attach, i18n.T("If true, wait for the Pod to start running, and then attach to the Pod as if 'kubectl attach ...' were called.  Default false, unless '-i/--stdin' is set, in which case the default is true.")) </del> <ins> cmd.Flags().BoolVar(&opt.Attach, "attach", opt.Attach, i18n.T("If true, wait for the container to start running, and then attach as if 'kubectl attach ...' were called.  Default false, unless '-i/--stdin' is set, in which case the default is true.")) </ins> cmd.Flags().StringVarP(&opt.Container, "container", "c", opt.Container, i18n.T("Container name to use for debug container.")) cmd.Flags().StringVar(&opt.CopyTo, "copy-to", opt.CopyTo, i18n.T("Create a copy of the target Pod with this name.")) cmd.Flags().BoolVar(&opt.Replace, "replace", opt.Replace, i18n.T("When used with '--copy-to', delete the original Pod"))

This issue tracks implementing node debugging with privileged containers for , as described in the This feature allows administrators to start a privileged container on a node in host namespaces rather than having to SSH to the node. For example: /kind feature /sig cli
/assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: // TODO: This value should be defined in an other, generic authz package instead of here // Starting from v1.8, CSRAutoApprovalClusterRoleName is automatically created by the API server on startup CSRAutoApprovalClusterRoleName = "system:certificates.k8s.io:certificatesigningrequests:nodeclient" <ins> // NodeSelfCSRAutoApprovalClusterRoleName is a role defined in default 1.8 RBAC policies for automatic CSR approvals for automatically rotated node certificates NodeSelfCSRAutoApprovalClusterRoleName = "system:certificates.k8s.io:certificatesigningrequests:selfnodeclient" </ins> // NodeAutoApproveBootstrapClusterRoleBinding defines the name of the ClusterRoleBinding that makes the csrapprover approve node CSRs NodeAutoApproveBootstrapClusterRoleBinding = "kubeadm:node-autoapprove-bootstrap" <ins> // NodeAutoApproveCertificateRotationClusterRoleBinding defines name of the ClusterRoleBinding that makes the csrapprover approve node auto rotated CSRs NodeAutoApproveCertificateRotationClusterRoleBinding = "kubeadm:node-autoapprove-certificate-rotation" </ins> ) // AllowBootstrapTokensToPostCSRs creates RBAC rules in a way the makes Node Bootstrap Tokens able to post CSRs

Reported in Scenario (CentOS 7): install 1.7.x packages for kubelet on nodes. install cluster with kubeadm init stable-1.7 use kubeadm to upgrade control plane to 1.8.0-rc.1 on nodes which still run 1.7.x kubelet, install new rpms for where certificate-rotation is enabled. systemctl daemon-reload, systemctl restart kubelet. result: kubelet is in wait loop, waiting for csr to be approved. node is in NotReady state. The kubelet blocks on startup, despite having a valid kubeconfig file. It should start up, attempt to rotate in the background, but not block.
I think this is because of the bootstrap kubeconfig flag. If you only provide the cert rotation flag it should only bootstrap in the background. That's what we where doing for bootkube where we need to start static pods while we're doing our initial cert rotation
a bit of debug shown that returns on the last line . So, kubelet has valid cert, but still wants to rotate it. That in theory should be a valid scenario. Question is: why it's not automatically approved then ? Even if cert will be in proper place, does that mean that automatic rotation wouldn't work for expired certs ? Or I'm missing something here.
no, it's because when a file is provided containing only cert/key data, the cert manager blocks on Start() until a CSR is approved
proposed fix for unblocking Start() in , which runs rotation in the background if you have a starting certificate
/cc
what does set up for auto-approval of node self certificate requests during an upgrade to 1.8?
for cert manager hang for kubeadm auto-approval setup issues upgrading to 1.8
looks like kubeadm upgrade instructions indicate approval of CSRs for upgrades nodes is expected: http://kubernetes-io-vnext- I'd expect that only to be required once certs expired, not automatically required as part of upgrade
Yeah, I that step to the kubeadm upgrade instructions last-minute today when the problem was discovered and we didn't think we would have a solution in time for 1.8.0. Manually approving the CSRs seems to work fine, but is a less-than-stellar upgrade experience.

lowering priority a notch, since the upgrade procedure is at least documented, and does not default to true for all installations (specific deployment methods like kubeadm enable it)
In case it helps, kubeadm 1.8 code for starting the controller manager is , and the RBAC rules for auto-approval are .
Also, I'm not sure if we have any other docs that need to be updated regarding upgrading kubelet, since this isn't a kubeadm-specific problem.
the combination of enabling rotation and not setting up auto-approval is what bit kubeadm here is a deployment-specific choice, and is not true by default. is the only deployment I see that enables it on upgrade to 1.8 the auto-approval rules at do not enable nodes to rotate their client certificates, that only sets up auto-approval for the initial TLS bootstrapping token I agree the kubelet should behave better in this scenario if it already has a valid credential (like ), but the configured setup by kubeadm doesn't seem correct
Ah, understood. It wasn't until I was looking at the RBAC rules myself just now that I realized we must have missed that. Definitely needs to get fixed in kubeadm.
I've created a PR for the documentation fix for kubeadm upgrades: We still need a kubeadm code fix to add the RBAC rules, hopefully to get into 1.8.1.
Yeah, not creating the ClusterRoleBinding between and was an oversight, but isn't that critical in any way. We'll make that change in v1.8.1, and document things meanwhile. Upgrading your kubelets is a manual process anyway right now, so it's reasonable to add one extra step to create this binding. Is it expected that a kubelet that already has credentials should renew them on restart if is set?
Not in a blocking way, only in the background if needed because of expiration, see
Reopening. Blocking on start is still an issue if auto approval is not enabled.
Fix in
Should this have the v1.8 milestone attached to it?
is everything here now fixed or are we waiting for
ping on the status of
- needs rebase and the mentioned in there is merged. What else needs to be done?

[MILESTONENOTIFIER] Milestone Removed From Issue Important: Code freeze is in effect and only issues with may remain in the v1.9 milestone. <details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
I think this issue is now solved right?


Question: tests := []struct { name                                string volumePaths                         []string <del> expectedVolumesNeedReportedInUse    []string </del> expectedVolumesNeedDevicePath       []string expectedVolumesFailedReconstruction []string verifyFunc                          func(rcInstance *reconciler, fakePlugin *volumetesting.FakeVolumePlugin) error

remove and was introduced in to fix This is to overcome the situation that volume is to DSW at first and then deleted before it is to ASW, which causes mounted volume neither in DSW nor ASW and won't execute unmount/detach: The reason the volume cannot be to ASW in time is , which is . However, there is another PR also solved the issue. What it did is to check whether ASW already has the volume before deleting it from DSW, this makes sure that volume can be detached/unmounted after it is deleted from DSW. I suggest that we can keep the second solution and remove related code to make the reconstruction process and reconcile loop cleaner?
/sig storage
cc
Should I proceed with this issue? Your guidance is appreciated.
I think we'd better wait for the feedback from
cc
Sorry for a late reply. This is how behind we are with reviews :-(. Reading Now that is on by default, it reconstructs volumes first and always puts them to ASW, it should never happen that a reconstructed volume is in DSW and not in ASW, right? So we can remove easily. I would not be so sure without , because the code is 5 years old and who knows all the implications. You could be right though, it may not be necessary there too.
/triage accepted
/assign