Question: } func TestValidateCustomResourceDefinitionValidationRuleCompatibility(t *testing.T) { <del> allValidationsErrors := []validationMatch{ </del> <ins> allRuleValidationsErrors := []validationMatch{ </ins> invalid("spec", "validation", "openAPIV3Schema", "properties[x]", "x-kubernetes-validations[0]", "rule"), invalid("spec", "validation", "openAPIV3Schema", "properties[obj]", "x-kubernetes-validations[0]", "rule"), invalid("spec", "validation", "openAPIV3Schema", "properties[obj]", "properties[a]", "x-kubernetes-validations[0]", "rule"),

Noticed a copy & paste issue when reading through the CEL code. Basically here: This should be instead of: The impact of this issue should be that messageExpressions are always evaluated with the CEL environment of the compatibility version. But stored (i.e. pre-existing) messageExpressions should be evaluated with a CEL environment with the "max" version instead. For users of Kubernetes this means that messageExpressions deployed with one minor version of Kubernetes might not work anymore if they rollback e.g. by one minor version. messageExpressions should be treated the same way as rules in that regard. Deploy Kubernetes 1.30 Deploy CRD that uses messageExpression that is using Sets (which became available with Kubernetes 1.30 that is using compatibilty version 1.29) Rollback to Kubernetes 1.29 Update something in the CRD =CRD update will fail because messageExpression is evaluated with the wrong CEL environment (it should still work) No response <detailsAt least on Kubernetes main, but I would assume this isuse is there since CEL was implemented </details<details</details
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
cc Will open a PR in a bit


Question: * `skuName`: Azure storage account Sku tier. Default is empty. * `location`: Azure storage account location. Default is empty. <del> * `storageAccount`: Azure storage account name. If storage account is not provided, all storage accounts associated with the resource group are searched to find one that matches `skuName` and `location`. If storage account is provided, `skuName` and `location` are ignored. </del> <ins> * `storageAccount`: Azure storage account name. If storage account is not provided, all storage accounts associated with the resource group are searched to find one that matches `skuName` and `location`. If storage account is provided, it must reside in the same resource group as the cluster, and `skuName` and `location` are ignored. </ins> ### User provisioning requests

Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2016-09-26T18:16:57Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2016-12-14T00:52:01Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"} Environment: Cloud provider or hardware configuration: Azure OS (e.g. from /etc/os-release): Ubuntu 16.04 LTS Kernel (e.g. ): 4.4.0-28-generic Install tools: Azure ACS Engine What happened: PVC keeps pending and returns "no key found for storage account" What you expected to happen: Storage class can retrieve the storage account key and successfully create the blob and pv. How to reproduce it (as minimally and precisely as possible): 1 - Create a new storage account and create a blob container named vhds 2 - Create a storage class as below 3 - Create the pvc 4 - Check kube-controller-manager logs: I0122 19:02:49. 1 ] no key found for storage account artyoudbs I0122 19:02:49. 1 ] failed to provision volume for claim "default/kaui-db" with StorageClass "slow": failed to find a matching storage account Anything else do we need to know: I can retrieve storage account keys using azure-cli: I've already used storageclass and pvc in azure with success before.
I was trying with a storage account in a different resource group. With the storage account in the same RG it works. Is it expected or a bug?
you have to use the storage account in the same resource group. Otherwise, the resource group is not authorized to retrieve the storage account's key
So you're saying the permission to attach a disk to a given VM is implicit by nature of them being in the same RG? I'm not sure that's accurate.

I've had non-k8s VMs previously that use an attached disk from a different RG.
I am not sure about attach, but listing storage account and keys appears to be limited to the same RG.
Sure, but you shouldn't need to do those things if the user is walking up with a disk URI for a disk that just happens to live in a different storage account. You don't need storage account keys for attach.
Is there a specific "is in this resource group" check that you implemented? I would've assumed you just take the URI and try to attach it...
you are right, attach or walking up the disk URI doesn't check RG or storage account. The particular issue here happened during provisioning using a provided storage account that lived out of the RG. In such case, when the provsioner tried to find the account key, it failed.
Ah, yeah, I think it seems reasonable to say that dynamic provisioning only works when the targetted storage account is in the same resource group.
+1 So the doc needs some words about this case to stress storage account, if provided, must be in the same RG.
PR


Question: }} } <del> func podAddressChanged(oldPod, newPod *v1.Pod) bool { </del> <ins> func podChanged(oldPod, newPod *v1.Pod) bool { // If the pod's readiness has changed, the associated endpoint address // will move from the unready endpoints set to the ready endpoints. // So for the purposes of an endpoint, a readiness change on a pod // means we have a changed pod. if podutil.IsPodReady(oldPod) != podutil.IsPodReady(newPod) { return true } </ins> // Convert the pod to an EndpointAddress, clear inert fields, // and see if they are the same. newEndpointAddress := podToEndpointAddress(newPod)

This test has started failing on our e2e test suite, erroring out with "Failed to get endpoints..." https://k8s- https://k8s- Example failures https://k8s- https://k8s- cc cc
This may be related to our recently-merged endpoints change in . I'll look into it.
I verified that the flakiness introduced was a result of . A fix in should be merging soon. /assign
GitHub didn't allow me to assign the following users: joelsmith. Note that only can be assigned. <detailsIn response to : I verified that the flakiness introduced was a result of . A fix in should be merging soon. /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: "github.com/golang/glog" <ins> "k8s.io/apimachinery/pkg/api/errors" </ins> "k8s.io/apimachinery/pkg/util/strategicpatch" "k8s.io/kubernetes/pkg/api" "k8s.io/kubernetes/pkg/api/v1"

kubernetes version: 1.5.3 platform: aws deploy tool: kops my master's /var/log/kube-controller- is receiving this type of log: lots of times per second, this node "ip-172-20-114-85.ec2.internal" does not exist in the cluster anymore and is not shown if I shouldn't it be removed from status_updater?
Any work around? Any way to purge those old nodes from the NodeInformer cache?
AFAIK, sig-node doesn't maintain the node controller. Based on the files, the only approver is
It has nothing to do with the NodeController: cc
This was happening in our clusters as well, only the leader was doing it and and killing the container so it restarted made it stop happening.
did you kill the container, which then gets restarted by the scheduler, and that fixed it?
correct
Works for me
Same controller-manager log here using k8s 1.5. I deleted one of the nodes which ran out of resources and became stuck, something I'm still trying to pinpoint the cause of but it appears to be related to volumes.
This happens on k8s 1.6.1 as well, that replaced old nodes won't get cleaned up, generating large error logs. On CoreOS installation, I did systemctl restart kube-controller-manager. That fixed it.
Ping
This is happening on k8s 1.4.12 as well, restarting kube-controller-manager does not seem to fix it for me. It is rotating through two nodes that no longer exist, so I am hoping the cache will purge itself at some point and they will stop?
Just following up that the old nodes still not cleaned up
will work on a fix for this.
Let's make sure the fix also gets ported back to all affected branches (1.6, 1.5, and 1.4). CC
I was able to reproduce the problem by force deleting a node containing a pod with some volume attached to it. Did anyone run into a different failure mode?
I think that's how the bug surfaced for me. When I force killed my node we had just started running a StatefulSet with an attached persistent volume, and I think there was one running on the very node I killed.
The NodeInformer cache is actually reporting the correct state.

The problem is that inside the node status updater, there is another data structure keeping track of nodes that require a status update (see ActualStateOfWorld.GetVolumesToReportAttached(), called inside UpdateNodeStatuses() in ). The entry for the deleted node is never removed from the data structure; rather, the updater is set to try to update the dead node again next time, which leads to the message being logged every 100ms. The solution is to remove the corresponding node entry once the node is deleted. A fix is on its way.
A fix for 1.5 is on the way (PR )
Will 1.4 be included as well?
Here you go, my apologies.
Thank you very much! :)
To summarize, the fix for this is in 1.7 branch Releases 1.7.0 and greater 1.6 branch Releases 1.6.5 and greater 1.5 branch Releases 1.5.8 and greater (1.5.8 has not yet been cut, CC for ETA of that release). 1.4 branch Releases 1.4.13 and greater (1.4.13 has not yet been cut, CC for ETA of that release).
There is an edge case that the above fix did not handle: see PR is fixing that


Question: runcmd: - modprobe configs <del> # Setup the installation target at make it executable - mkdir -p /home/kubernetes/bin/nvidia - mount --bind /home/kubernetes/bin/nvidia /home/kubernetes/bin/nvidia - mount -o remount,exec /home/kubernetes/bin/nvidia # Compile and install the nvidia driver (precompiled driver installation currently fails) - docker run --net=host --pid=host -v /dev:/dev -v /:/root -v /home/kubernetes/bin/nvidia:/usr/local/nvidia -e NVIDIA_INSTALL_DIR_HOST=/home/kubernetes/bin/nvidia -e NVIDIA_INSTALL_DIR_CONTAINER=/usr/local/nvidia -e NVIDIA_DRIVER_VERSION=460.91.03 --privileged gcr.io/cos-cloud/cos-gpu-installer:latest # Run the installer again, as on the first try it doesn't detect the libnvidia-ml.so # on the second attempt we detect it and update the ld cache. - docker run --net=host --pid=host -v /dev:/dev -v /:/root -v /home/kubernetes/bin/nvidia:/usr/local/nvidia -e NVIDIA_INSTALL_DIR_HOST=/home/kubernetes/bin/nvidia -e NVIDIA_INSTALL_DIR_CONTAINER=/usr/local/nvidia -e NVIDIA_DRIVER_VERSION=460.91.03 --privileged gcr.io/cos-cloud/cos-gpu-installer:latest </del> <ins> # Install GPU drivers - https://cloud.google.com/container-optimized-os/docs/how-to/run-gpus - cos-extensions install gpu - mount --bind /var/lib/nvidia /var/lib/nvidia - mount -o remount,exec /var/lib/nvidia /var/lib/nvidia # Run nvidia-smi to verify installation - /var/lib/nvidia/bin/nvidia-smi </ins> # Remove build containers. They're very large. - docker rm -f $(docker ps -aq) # Standard installation proceeds

I can't find a single instance of this test passing in testgrid. Run via and potentially other suites. /sig node /kind failing-test
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
The test fails because there is no GPUs reported by kubelet [1]. Looking at the node info right before the failure, the node does not have any gpu capacity [2]. The artifacts only have kubelet log, but not device plugin or driver installer log. So I am not sure if there's any error on them. Is there any way to see the device plugin or driver installer log? [1] [2]
/assign I have a partial fix here - The actual feature seems broken too tho - hidden by the fact that it's been failing to provision properly for a really long time :(
Since merged this test has been relatively stable. Going to close this issue now (kept it around while I was on PTO to take a look when I got back). Seems like there are some infrequent flakes that we should follow up on after we remove DKK. /close
Closing this issue. <detailsIn response to : Since merged this test has been relatively stable. Going to close this issue now (kept it around while I was on PTO to take a look when I got back). Seems like there are some infrequent flakes that we should follow up on after we remove DKK. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen This is failing again, I'm not sure what regressed but it was 2wks ago.
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I keep seeing errors like But our serial suite doesn't have GPUs attached.

I'd expect this to be skipped: do you have any idea why sometimes these tests pass, sometimes they get skipped, and sometimes they time out? It makes no sense to me:
Wrote to try to dump what's matching and not causing these tests to get skipped
The tests get skipped when they run on nodes without GPUs attached, which is 2 of 3 runs every time, but cos-stable2 has 2 GPUs (). The failures happen because something in the GPU setup process broke again (I assume a bad image bump bc we don't pin), and so the test fails waiting for the GPUs to show up on the node after installing the plugin, but they are attached to the node.
I think I have a fix here -


Question: } else { // TODO: Make all nodePorts jump to the firewall chain. // Currently we only create it for loadbalancers (#33586). <ins> // Fix localhost martian source error loopback := "127.0.0.0/8" if isIPv6 { loopback = "::1/128" } writeLine(proxier.natRules, append(args, "-s", loopback, "-j", string(KubeMarkMasqChain))...) </ins> writeLine(proxier.natRules, append(args, "-j", string(svcXlbChain))...) }

/kind bug /sig network If I set up a service with a NodePort and set , and try to access it as it works. If I try to access it as , it triggers a martian source error - the packet gets forwarded as 127.0.0.1. We need some masquerade in there. Applies to iptables mode. Maybe ipvs mode, not sure?
If we set for NodePort, we don't do any masq for packets currently. I have tested that masq will resolve the issue in iptables mode, but my question is "do we still need to preserve source ip of packets for nodeport when externalTrafficPolicy=Local"? If we do masq, we will lose the source IP, sadly.
BTW, with the fix of , is reachable The iptables created by is Any comments?
this is not entirely true. key "externalTrafficPolicy=Local" needs to be a solution to the problem of obtaining the real IP of the client, but this is not happening. If we do masq, we will not lose the source IP, because now he doesn't work (version 1.8.1) the problem with the inaccessibility of the actual source IP () in version 1.8.1 is still relevant and is a very big problem for deployment in kubernetes most web services on a production. besides, in my cluster key "externalTrafficPolicy : false" leads to a hollow stop traffic to the pods.
Without masq, nodeport for localhost when set "externalTrafficPolicy=Local" won't work. If service is not reachable, say "how to preserve source IP" won't make sense. Listen, masq is used for someone who want to visit nodeport via localhost when set "externalTrafficPolicy=Local" and don't require preserving source IP. If source IP is really important to you, you can just ignore the .
I do not understand what do you mean "you can just ignore the 127.0.0.1". could you describe in detail how to configure the service to get the real source IP?

type service, externalTrafficPolicy=[Local | Cluster] type service, externalTrafficPolicy=Local, but don't try to access type service, externalTrafficPolicy=Local
i wrote above, NodePort type service with externalTrafficPolicy=Local blocking the transmission of traffic to the pod. I'm trying to connect to node_ip:port. maybe this is a bug? I could demonstrate this.
oh, sorry )) i'm connecting to wrong node )) yes, now traffic is gone. but ip is not source! with "externalTrafficPolicy=Cluster" i getting ip 10.1.29.21 (node ip) with "externalTrafficPolicy=Local" i getting ip 10.4.62.1 (pods network ip - not real source ip)
You probably should check the iptables applied in your host, is there any masq rules created by other processes other than kube-proxy? It preserves the source IP of client - I double checked it. BTW, you can create another thread for discussing your problems. I feel like this issue is orthogonal with your question.
Will setting ClusterIP type service and externalTrafficPolicy=Local for a service to a DaemonSet, route traffic only to the daemonSet pod in that node when i access the service using DaemonSetServiceIP?
no. local service does not work for cluster IP


Question: func gvkP(g, v, k string) *schema.GroupVersionKind { return &schema.GroupVersionKind{Group: g, Version: v, Kind: k} } <ins> func gvk(g, v, k string) schema.GroupVersionKind { return schema.GroupVersionKind{Group: g, Version: v, Kind: k} } </ins>

Improve the to ensure that the expected storage version for a resource is one understood by previous releases. We have API testdata from prior versions available in When checking each group/version/kind in the test: accumulate all the versions of the group/kind available in previous (non-HEAD) releases if there is a beta or stable version of the group/kind available in previous releases, then the storage version must be in the set of previously available versions It is too easy to accidentally change the storage version prematurely and edit the test/integration/etcd test to match the bug cc
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: if protocol == "udp" { // TODO: It would be enough to pass 1s+epsilon to timeout, but unfortunately // busybox timeout doesn't support non-integer values. <del> cmd = fmt.Sprintf("echo 'hostName' | timeout -t 2 nc -w 1 -u %s %d", targetIP, targetPort) </del> <ins> cmd = fmt.Sprintf("echo 'hostName' | nc -w 1 -u %s %d", targetIP, targetPort) </ins> } else { ipPort := net.JoinHostPort(targetIP, strconv.Itoa(targetPort)) // The current versions of curl included in CentOS and RHEL distros // misinterpret square brackets around IPv6 as globbing, so use the -g // argument to disable globbing to handle the IPv6 case. <del> cmd = fmt.Sprintf("timeout -t 15 curl -g -q -s --connect-timeout 1 http://%s/hostName", ipPort) </del> <ins> cmd = fmt.Sprintf("curl -g -q -s --max-time 15 --connect-timeout 1 http://%s/hostName", ipPort) </ins> } // TODO: This simply tells us that we can reach the endpoints. Check that

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug /sig testing What happened: Some commands used in tests are Linux specific and do not exist or do not behave the same on Windows nodes. This can cause those tests to fail on Windows nodes. Some of the tests are: [sig-network] Networking Granular Checks: Pods should function for node-pod communication: http [NodeConformance] [sig-network] Networking Granular Checks: Pods should function for node-pod communication: udp [NodeConformance] [sig-network] Networking Granular Checks: Services [Slow] should function for node-Service: http [sig-network] Networking Granular Checks: Services [Slow] should function for node-Service: udp [sig-network] Networking Granular Checks: Services [Slow] should update nodePort: http [Slow] [sig-network] Networking Granular Checks: Services [Slow] should update nodePort: udp [Slow] [sig-network] NetworkPolicy NetworkPolicy between server and client should support a 'default-deny' policy [Feature:NetworkPolicy] [sig-network] NetworkPolicy NetworkPolicy between server and client should enforce policy based on PodSelector [Feature:NetworkPolicy] [sig-network] NetworkPolicy NetworkPolicy between server and client should enforce policy based on NamespaceSelector [Feature:NetworkPolicy] [sig-network] NetworkPolicy NetworkPolicy between server and client should enforce policy based on Ports [Feature:NetworkPolicy] [sig-network] NetworkPolicy NetworkPolicy between server and client should enforce multiple, stacked policies with overlapping podSelectors [Feature:NetworkPolicy] [sig-network] NetworkPolicy NetworkPolicy between server and client should support allow-all policy [Feature:NetworkPolicy] [sig-network] NetworkPolicy NetworkPolicy between server and client should allow ingress access on one named port [Feature:NetworkPolicy] [sig-network] NetworkPolicy NetworkPolicy between server and client should allow egress access on one named port [Feature:NetworkPolicy] What you expected to happen: The commands used by those tests should be platform agnostic, so they can pass using Windows Nodes as well. How to reproduce it (as minimally and precisely as possible): Run the mentioned tests using Windows Nodes. Anything else we need to know?

: Environment: Kubernetes version (use ): v1.12.0-rc.2 Cloud provider or hardware configuration: N/A OS (e.g. from /etc/os-release): Windows Server 1803 compute nodes Kernel (e.g. ): N/A Install tools: N/A Others:


Question: func (w ConditionalWait) IsConditionMet(info *resource.Info, o *WaitOptions) (runtime.Object, bool, error) { endTime := time.Now().Add(o.Timeout) for { <ins> if len(info.Name) == 0 { return info.Object, false, fmt.Errorf("resource name must be provided") } </ins> resourceVersion := "" gottenObj, err := o.DynamicClient.Resource(info.Mapping.Resource).Namespace(info.Namespace).Get(info.Name, metav1.GetOptions{}) switch {

Is this a BUG REPORT or FEATURE REQUEST?: What happened: I created a job with metadata.generateName twice and waited for completion, but got: panic: interface conversion: runtime.Object is unstructured.UnstructuredList, not unstructured.Unstructured What you expected to happen: kubectl to wait for the last job to complete. How to reproduce it (as minimally and precisely as possible): kubectl create -f kubectl create -f kubectl wait --for=condition=complete -f (shortened): apiVersion: batch/v1 kind: Job metadata: generateName: my-job- Anything else we need to know?: Environment: Kubernetes version (use ): 1.11.3 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
I have to choose a sig, but I am not sure which one is right...
Reiterating the mentions to trigger a notification: <detailsIn response to : I have to choose a sig, but I am not sure which one is right... Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cli /sig apps
/assign
kubectl get -f also doesn't work as expected.
/remove-sig apps


Question: continue } err = config.client.CoreV1().PersistentVolumes().Delete(backgroundCtx, pv.Name, metav1.DeleteOptions{}) <ins> if errors.Is(err, context.Canceled) { continue } </ins> framework.ExpectNoError(err) pvConfig := makeLocalPVConfig(config, localVolume) localVolume.pv, err = e2epv.CreatePV(backgroundCtx, config.client, f.Timeouts, e2epv.MakePersistentVolume(pvConfig)) <ins> if errors.Is(err, context.Canceled) { continue } </ins> framework.ExpectNoError(err) } }

master-informing: gce-cos-master-serial ! 2023-06-08 20:01:55 +0000 UTC { failed [FAILED] PV Create API error: Post "https://35.247.13.135/api/v1/persistentvolumes": context canceled In [DeferCleanup (Each)] at: test/e2e/storage/persistentvolumes- @ 06/09/23 18:05:53.165 } No response_ /sig storage cc
I think we did something wrong in /triage accepted
/assign
do you have a link to a full log? It's not clear where comes from (will look closer on Monday), but as the test is shutting down, not allowing new PVs to be created may be normal.
Never mind, the description has links to failing jobs.
ok - this looks like an actual bug with the code as written. what's happening is the "Stop and wait.." at line 492 is calling of the passed to the "goroutine [that] recycle[s] unbound PVs". Ordinarily that signal triggers the and the goroutine exits. In this case (and it will be flaky) it triggers while the case is running and it causes the on line 533 to error with . I don't know what the desired behavior is - should a error be ignored since we're signaling that it's time to leave? or should we actually wait for the to complete and use a different (timeout??) context for that instead.
one potential (if complex) fix is to pass two contexts down to the goroutine. one, to signal “you should stop when you are next at the select statement” and another , that is passed down to CreatePV that signals “stop immediately”. The would then take a Ginkgo context with a timeout. when the first launches it cancels and then if the timeout expires it cancels there might well be a simpler way to track all this but the very nature of what this goroutine is doing seems to need some sort of two-signal approach. alternatively: we can say that within this goroutine is always safe to ignore. it’s simpler as long as there are no test pollution concerns about forcibly canceling the CreatePV.
Patrick is the most knowledgable person in this area, we should wait for his response
Sounds good.

This might be a one off - or they may be a class of flaky tests that have a similar structure and so coming up with, and applying, a common pattern for signaling a background goroutine to exit gracefully could be useful.
I think ignoring the error is fine in this case. There is no reason for it to occur except when the test has already completed. -
sounds good - I suppose the only concern would be that forcibly aborting the might be an issue (e.g. the server's still working on things and we fail to clean up or race with it). but y'al lknow k8s far better than i do so maybe there's reasons to have confidence that it's a nonissue.
If the operation completes on the server side, then the other registered earlier will delete the PV.
last run was green let's see this is stable now, thanks for the help


Question: for _, field := range fieldsFail { for _, action := range actions { options := &api.ListOptions{FieldSelector: field.AsSelector(), ResourceVersion: "1"} <ins> ctx = context.WithValue(context.WithValue(ctx, "field", field), "action", action) </ins> watcher, err := t.storage.(rest.Watcher).Watch(ctx, options) if err != nil { t.Errorf("unexpected error: %v", err)

https://k8s- Failed: TestWatch {etcd} Previous issues for this test:
https://k8s- Failed: TestWatch {etcd}
https://k8s- Failed: TestWatch {etcd}
https://k8s- Failed: TestWatch {etcd}
https://k8s- Failed: TestWatch {etcd}
https://k8s- Failed: TestWatch {etcd}
/cc
are there any known issues on the clientv3 prior to 3.0.8? Just wondering if we need a rev bump.
I can never repro this locally, and the leaks have been fixed.
https://k8s- Failed: TestWatch {etcd}
So it's failing in different spots, but replicasets is the most common occurrence.
xref:
ok, we have a bead on the issue. It will not affect the released bits, but it does affect the UTs.
PR in flight.
What's the default logging level of unit test?
If we looked at the test, they are sort of wrong themselves. For example, It's doing another action on the same key, but it's watching at version "1" again. From my understanding, if we have action "create, delete", I want to watch two actions one by one. But it's watching at "1", now we just watch "create", then another same "create".
Even worse, it could be hiding problems. For example, if there is some event that happened in previous suites, then it will happen to next suites without differentiating and checking the actual values from ResultChan(). In such case, a wrong test might pass.
https://k8s- Failed: TestWatch {etcd}
https://k8s- Failed: TestWatch {etcd}
- if was supposed to help, it didn't - it is still failing.
has some ideas... and I'm digging too.
https://k8s- Failed: TestWatch {etcd}
Here's my understanding of the scenario: What's happening? A lot of get requests are sent due to prev-value. Note that they are quorum get. Those get request are cancelled due to the wrong logic in tests. What's happening inside etcd is that etcd server is still processing them and trying to send them until grpc fails. These get requests could block further events from being processed.

It gets even worse under heavy loaded environment. What's the problem in the unit test itself? Their logic are wrong themselves. See my previous comments. We need to fix the tests. What's on etcd? I need to stress test this behavior. Then I will coordinate with etcd team to see potentially what could be improved/fixed.
There is just a single etcd instance - so whether this is quorum or not doesn't matter here. Actually - it is a valid usecase that user cancel the request (if I'm not missing anything subtle here). ^
Quorum get affects processing events. The problem is not the cancelling. See my previous comments on the test itself. If we expect to test cancelling requests, then we should have another set of tests for that, and have different checking workflow.
I think the tests watching a "1" expect a new etcd instance each time. Otherwise they would be horribly broken.
https://k8s- Failed: TestWatch {etcd}
I'm still digging into "how" this happens... we fixed a bunch of issues. It's clear the watch is timing out.. sometimes. Every single test fires up it's own client/server pair, each of which is also properly cleaned... now.
share your repro steps?
when digging through latest output across the tests e.g. I'm always seeing a triplet of: But tests always pass, still digging...
This is the problem caused by watching "1". It keeps getting previous events. Meanwhile, watching "1" has some other latency issue. I had done some benchmark that does watch from 1 and latency could spike up to 5 second. I need more time to do more testing on this. But the tests themselves are problematic. We should fix them first. If it doesn't fix the problem, we will try to reproduce it then.
I was assuming your previous comment says that you have changed/fixed the tests and you are still reproducing the problem?
Why is watching from "1" problematic? There shouldn't be any previous events, since we start etcd fresh for every test. , Hongchao Deng wrote:
The tests are problematic. Inside each test there are many test suites not getting a fresh etcd.

Watching "1" will force etcd to query internal database for past events, potentially causing some latency issue on compacted env. With all these said, it's all theories. We still need to do more logging and testing around it to find out real cause.
https://k8s- Failed: TestWatch {etcd}
https://k8s- Failed: TestWatch {etcd}
https://k8s- Failed: TestWatch {etcd}
https://k8s- Failed: TestWatch {etcd}
https://k8s- Failed: TestWatch {etcd}
https://k8s- Failed: TestWatch {etcd}
https://k8s- Failed: TestWatch {etcd}
https://k8s- Failed: TestWatch {etcd}
https://k8s- Failed: TestWatch {etcd}
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: TestWatch {etcd}
https://k8s- Failed: TestWatch {etcd}
https://k8s- Failed: TestWatch {etcd}
for searchability
I see what is saying now re: rewatch @ 1. The tests areas that are failing are in loops that are constantly re-issuing a watch at 1, even though b/c it's in a loop emitting it should not be 1, tests work fine when simply removing entirely. So I'll put together a PR that ups the timeout + removed the ListOption. Although I do think the negative tests should probably be re-written to interleave with the positive tests, to eliminate the time suck.
Thanks. I am very much looking forward to that PR. ;)
Do you have an ETA for the PR to help us figure out if this should be moved to flaky as a stop-gap?
we're still debugging, what doesn't help is that these failures only seem to occur on GCE and no one care repro them outside of that environment.
Frustrating. Are you able to reliable reproduce using the "run until failure" test option?

Merged
https://k8s- Failed: TestWatch {etcd}
https://k8s- Failed: TestWatch {etcd}
https://k8s- Failed: TestWatch {etcd}
Still occurring xref:
Yes, just happened 3x in a row here:
Do you think we can create a new etcd instance every time? As mentioned in
won't fix the problem. I do have another workaround for this until etcd fix comes in.
is on


Question: // awsRegions is a set of recognized regions var awsRegions sets.String <del> // RecognizeRegion is called for each AWS region we know about. </del> <ins> // recognizeRegion is called for each AWS region we know about. </ins> // It currently registers a credential provider for that region. // There are two paths to discovering a region: //  * we hard-code some well-known regions //  * if a region is discovered from instance metadata, we add that <del> func RecognizeRegion(region string) { </del> <ins> func recognizeRegion(region string) { </ins> awsRegionsMutex.Lock() defer awsRegionsMutex.Unlock()

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: From /kind cleanup What happened: What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig cloud-provider /assign


Question: func (d *dummyStorage) Count(_ string) (int64, error) { return 0, fmt.Errorf("unimplemented") } <ins> func (d *dummyStorage) injectError(err error) { d.Lock() defer d.Unlock() d.err = err } </ins> func TestGetListCacheBypass(t *testing.T) { backingStorage := &dummyStorage{}

As Jordan pointed out in: watchcache treats RV=0 and RV="" the same way, even though it's not what we document: This should be fixed. /kind bug /sig api-machinery /priority important-soon
/assign
/triage accepted


Question: err := p.Start() require.NoError(t, err) <del> e, err := newEndpointImpl(socket, "mock", make(map[string]pluginapi.Device), func(n string, a, u, r []pluginapi.Device) {}) </del> <ins> e, err := newEndpointImpl(socket, resourceName, make(map[string]pluginapi.Device), callback) </ins> require.NoError(t, err) return p, e

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: For now, UT in doesn't work because callback function will never be called, is rewritten in this function : Another thing, i'm wondering why we rewrite device health in function which in this file: What you expected to happen: UT assertion should be executed. How to reproduce it (as minimally and precisely as possible): Add some logs in callback function of , it will show nothing. Anything else we need to know?: I will send a PR to fix this. /cc WDYT? /sig node
/assign
GitHub didn't allow me to assign the following users: RenaudWasTaken. Note that only can be assigned. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Good catch


Question: func (pc *metricsCollector) collect() []DataItem { var dataItems []DataItem for _, metric := range pc.Metrics { <del> dataItem := collectHistogram(metric, pc.labels) </del> <ins> dataItem := collectHistogramVec(metric, pc.labels) </ins> if dataItem != nil { dataItems = append(dataItems, *dataItem) }

The new scheduler performance test was introduced back in 1.18, back then, the metric it supported are all . However, along with , some metrics are updated to be , and hence it's no longer appropriate to return the first Histogram: However, it's very common (for instance Preemption) that the metrics may hold multiple histograms: Additionally, metrics and have been removed, we should update them to : /sig scheduling
/triage accepted /assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale


Question: "//pkg/api/validation:go_default_library", "//pkg/apis/policy:go_default_library", "//pkg/client/clientset_generated/internalclientset/typed/policy/internalversion:go_default_library", <ins> "//pkg/client/retry:go_default_library", </ins> "//pkg/kubelet/client:go_default_library", "//pkg/labels:go_default_library", "//pkg/registry/cachesize:go_default_library",

What happened: When I try to test after implementing the 1st approach according to , I found pod disruption budgets cannot handle concurrent requests. The error message is: It is an Optimistic Lock error message. It means PDB can't handle concurrent requests. How to reproduce it (as minimally and precisely as possible): 1) Create a deployment with 6 replicas and schedule them to the same node, say node A. You can leave only one node scheduling enabled, and then create the deployment. 2) Create the PDB that match the deployment's label 3) Uncordon another node, say node B 4) Drain node A using concurrent POST eviction requests. Ref: I'm not sure if this behavior is OK for PDB. I think I can make work by waiting for a while if current pod label(s) match previous pod label(s).
Oh, that's interesting. I think the problem is that this function doesn't retry if UpdateStatus() fails due to version conflict. I think the solution is to modify the caller, i.e. this function so that instead of just it instead does something like This will loop re-doing checkAndDecrement() and getPodDisruptionBudgets() until checkAndDecrement() succeeds, or fails with a non-conflict error. Does that make sense? cc/ for sanity check
Or use PATCH?
Yes, PATCH should work also. I haven't written code that uses PATCH, so I can't advise exactly the code to write here (there are examples by grepping for patch, but I'm not sure exactly what is the right way to do it from inside of API server code, which is where we are here). can you do a PR to implement this, either using the approach I described above or using PATCH? I think this is important to get into 1.5.
I think PATCH may NOT be safe here. For example: 1) current pdb 2) creating eviction A: get 3) creating eviction B: get 4) creating eviction A: PATCH pdb 5) creating eviction B: PATCH pdb 6) pdb's . It's wrong. Retry loop make more sense to me.
Good point! You're right. Retry loop should be OK.
I can help to create a PR using retry loop.

Could you please triage this for 1.5 and apply blocker or non-blocker labels as appropriate along with priority.
Done.
who is actually working on this issue? Could one of you set yourselves as an assignee for this issue? Thanks! cc:
retry logic with PR needs reviews/lgtm


Question: return } action := watch.Added <del> if res.Node.ModifiedIndex != res.Node.CreatedIndex { action = watch.Modified } </del> w.emit(watch.Event{ Type:   action, Object: obj,

Currently, Watch request with resourceVersion=0 results in: returning a single "watch event" for every object existing "now" starts watching from that point This is pretty misleading (e.g. it's different that what we have in etcd, where there is no first step from above). In the v2 api, we should probably eliminate the first stop (or even disallow watch requests with resourceVersion = 0).
/cc
Issues go stale after 30d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
/remove-lifecycle rotten /lifecycle frozen
What do you mean by v2 API? Are we ever going to have a v2 of the machiery and semantics?
In 2015 I thought it's not ridiculous. I'm not sure I believe in it now. Moreover, with we will actually be taking benefits of semantic like this, so we should close this.


Question: if o.Resource != (schema.GroupVersionResource{}) || o.ResourceName != "" { return fmt.Errorf("NonResourceURL and ResourceName can not specified together") } <ins> if !isKnownNonResourceVerb(o.Verb) { fmt.Fprintf(o.ErrOut, "Warning: verb '%s' is not a known verbn", o.Verb) } </ins> } else if !o.Resource.Empty() && !o.AllNamespaces && o.DiscoveryClient != nil { if namespaced, err := isNamespaced(o.Resource, o.DiscoveryClient); err == nil && !namespaced { if len(o.Resource.Group) == 0 {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When using kubectl auth can-i <verb<resourcesthe "verb" isn't check and returns yes on any resource regardless of what the verb is. What you expected to happen: I expected the cli to register an error like when you don't have enough arguments. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): client version GitVersion: 1.19.2 server version: GitVersion: 1.19.0 Cloud provider or hardware configuration: Mac OS X OS (e.g: ): n/a Kernel (e.g. ): Darwin Install tools: brew Network plugin and version (if this is a network-related bug): n/a Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cli
This is working as intended. The authorization layer can answer questions about arbitrary verbs, and the response to can-i mirrors the response you would get from a subjectaccessreview permission check. Getting a "yes" response to arbitrary verbs likely means you are a superuser and would be authorized for any operation.
Wouldn't a superuser only have access to the Kubernetes verbs that exist and not the arbitrary verbs that it is allowed to query that is defined here () This seems to go against the selfsubjectaccessreview which will return a set of actions that a user can perform. Arbitrary verbs seem like they shouldn't be allowed from that definition.
No, the cluster-admin superuser role allows verbs which allows any verb


Question: podsToSync = append(podsToSync, pod) } } <del> glog.V(2).Infof("SyncLoop (SYNC): %d pods", kubeletutil.FormatPodNames(podsToSync)) </del> <ins> if len(podsToSync) == 0 { break } glog.V(4).Infof("SyncLoop (SYNC): %d pods; %s", len(podsToSync), kubeletutil.FormatPodNames(podsToSync)) </ins> kl.HandlePodSyncs(podsToSync) case update := <-kl.livenessManager.Updates(): // We only care about failures (signalling container death) here.

It appears in a recent commit (probably ) this log message was : It's printed as type and is logged every second, which is cluttering up the logs quite a bit. Is this necessary?
Sorry for the spam. I also saw this in the log and wanted to fix it. PR uploaded.
That PR works! Thanks :)


Question: // RegisterNodes registers the given list of nodes, it keeps retrying for `retryCount` times. func (s *NodeController) RegisterNodes(nodes *api.NodeList, retryCount int, retryInterval time.Duration) error { registered := util.NewStringSet() <ins> nodes = s.canonicalizeName(nodes) </ins> for i := 0; i < retryCount; i++ { for _, node := range nodes.Items { if registered.Has(node.Name) {

NodeController is unable to register the value of , because it contains mixed case from the OS (in this case OSX). I'm not positive that all Linux systems are guaranteed to lowercase hostname -f, but it's likely that automated processes may end up collecting hostnames from sources that preserve case and passing that to the kubelet or master. I believe we should lowercase names detected from hostname -f and the values the node controller users - it's a minor thing, but internally we can guarantee that we treat node names as insensitive and in their canonical interior form.
cc
I'll take a look tmr.


Question: podSubnet: "" serviceSubnet: "" nodeName: "" <ins> privilegedPods: false </ins> token: "" unifiedControlPlaneImage: "" `),

Dear All: I'm not sure if I'm right. When i use whith .I found controller-manager logs Someone can help me? Thanks a lot.
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Please give more information about your set up cc
Sorry for report later. My openstack didn't support metadata service,but support config-drive service. More info:
Thanks Based on this, I think it's an openstack cloudprovider problem more than a kubeadm problem...
Thank you for help me. I will change the label.
/area cloudprovider
/remove-kind support
Should be fixed with you can specify a search order for metadata and config drive /close
Specifying a search order doesn't really solve the problem, especially if the metadata api isn't available which is what ZhenyangZhao said . I'm getting the same issue and since the config drive is my only option at the moment.
Like for this to be /open
can you please provide logs/versions for your problem in your environment? Thanks, Dims
Let me know if you need anything else Kubernetes version: 1.8.4 Docker version: 17.09.0-ce Linux: CentOS 7.4 Looking at the docker container log for the api-server (I don't have problems with the controller-manager):
and share the following:
Hi! Thank you for filing this issue. We appreciate your interest in Kubernetes and kubeadm. In order to make the kubernetes/kubernetes issue tracker more usable, we're routing issues to different repositories with different focuses. I think this issue belongs in the kubeadm repository. Please move this issue to the in order for the kubeadm team to be able to track this efficiently. If you disagree with my assessment of moving this issue, please speak up and/or reopen the issue. Finally, please try to reproduce the bug with the latest Kubernetes version, it might as well already have been fixed :) Thank you and have a good day!


Question: for _, label := range labelsFail { for _, action := range actions { options := &api.ListOptions{LabelSelector: label.AsSelector(), ResourceVersion: "1"} <ins> ctx = context.WithValue(context.WithValue(ctx, "label", label), "action", action) </ins> watcher, err := t.storage.(rest.Watcher).Watch(ctx, options) if err != nil { t.Errorf("unexpected error: %v", err)

https://k8s- Failed: TestWatch {etcd} Previous issues for this test:
https://k8s- Failed: TestWatch {etcd}
https://k8s- Failed: TestWatch {etcd}
https://k8s- Failed: TestWatch {etcd}
https://k8s- Failed: TestWatch {etcd}
https://k8s- Failed: TestWatch {etcd}
/cc
are there any known issues on the clientv3 prior to 3.0.8? Just wondering if we need a rev bump.
I can never repro this locally, and the leaks have been fixed.
https://k8s- Failed: TestWatch {etcd}
So it's failing in different spots, but replicasets is the most common occurrence.
xref:
ok, we have a bead on the issue. It will not affect the released bits, but it does affect the UTs.
PR in flight.
What's the default logging level of unit test?
If we looked at the test, they are sort of wrong themselves. For example, It's doing another action on the same key, but it's watching at version "1" again. From my understanding, if we have action "create, delete", I want to watch two actions one by one. But it's watching at "1", now we just watch "create", then another same "create".
Even worse, it could be hiding problems. For example, if there is some event that happened in previous suites, then it will happen to next suites without differentiating and checking the actual values from ResultChan(). In such case, a wrong test might pass.
https://k8s- Failed: TestWatch {etcd}
https://k8s- Failed: TestWatch {etcd}
- if was supposed to help, it didn't - it is still failing.
has some ideas... and I'm digging too.
https://k8s- Failed: TestWatch {etcd}
Here's my understanding of the scenario: What's happening? A lot of get requests are sent due to prev-value. Note that they are quorum get. Those get request are cancelled due to the wrong logic in tests. What's happening inside etcd is that etcd server is still processing them and trying to send them until grpc fails. These get requests could block further events from being processed.

It gets even worse under heavy loaded environment. What's the problem in the unit test itself? Their logic are wrong themselves. See my previous comments. We need to fix the tests. What's on etcd? I need to stress test this behavior. Then I will coordinate with etcd team to see potentially what could be improved/fixed.
There is just a single etcd instance - so whether this is quorum or not doesn't matter here. Actually - it is a valid usecase that user cancel the request (if I'm not missing anything subtle here). ^
Quorum get affects processing events. The problem is not the cancelling. See my previous comments on the test itself. If we expect to test cancelling requests, then we should have another set of tests for that, and have different checking workflow.
I think the tests watching a "1" expect a new etcd instance each time. Otherwise they would be horribly broken.
https://k8s- Failed: TestWatch {etcd}
I'm still digging into "how" this happens... we fixed a bunch of issues. It's clear the watch is timing out.. sometimes. Every single test fires up it's own client/server pair, each of which is also properly cleaned... now.
share your repro steps?
when digging through latest output across the tests e.g. I'm always seeing a triplet of: But tests always pass, still digging...
This is the problem caused by watching "1". It keeps getting previous events. Meanwhile, watching "1" has some other latency issue. I had done some benchmark that does watch from 1 and latency could spike up to 5 second. I need more time to do more testing on this. But the tests themselves are problematic. We should fix them first. If it doesn't fix the problem, we will try to reproduce it then.
I was assuming your previous comment says that you have changed/fixed the tests and you are still reproducing the problem?
Why is watching from "1" problematic? There shouldn't be any previous events, since we start etcd fresh for every test. , Hongchao Deng wrote:
The tests are problematic. Inside each test there are many test suites not getting a fresh etcd.

Watching "1" will force etcd to query internal database for past events, potentially causing some latency issue on compacted env. With all these said, it's all theories. We still need to do more logging and testing around it to find out real cause.
https://k8s- Failed: TestWatch {etcd}
https://k8s- Failed: TestWatch {etcd}
https://k8s- Failed: TestWatch {etcd}
https://k8s- Failed: TestWatch {etcd}
https://k8s- Failed: TestWatch {etcd}
https://k8s- Failed: TestWatch {etcd}
https://k8s- Failed: TestWatch {etcd}
https://k8s- Failed: TestWatch {etcd}
https://k8s- Failed: TestWatch {etcd}
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: TestWatch {etcd}
https://k8s- Failed: TestWatch {etcd}
https://k8s- Failed: TestWatch {etcd}
for searchability
I see what is saying now re: rewatch @ 1. The tests areas that are failing are in loops that are constantly re-issuing a watch at 1, even though b/c it's in a loop emitting it should not be 1, tests work fine when simply removing entirely. So I'll put together a PR that ups the timeout + removed the ListOption. Although I do think the negative tests should probably be re-written to interleave with the positive tests, to eliminate the time suck.
Thanks. I am very much looking forward to that PR. ;)
Do you have an ETA for the PR to help us figure out if this should be moved to flaky as a stop-gap?
we're still debugging, what doesn't help is that these failures only seem to occur on GCE and no one care repro them outside of that environment.
Frustrating. Are you able to reliable reproduce using the "run until failure" test option?

Merged
https://k8s- Failed: TestWatch {etcd}
https://k8s- Failed: TestWatch {etcd}
https://k8s- Failed: TestWatch {etcd}
Still occurring xref:
Yes, just happened 3x in a row here:
Do you think we can create a new etcd instance every time? As mentioned in
won't fix the problem. I do have another workaround for this until etcd fix comes in.
is on


Question: csipbv1 "github.com/container-storage-interface/spec/lib/go/csi" "k8s.io/apimachinery/pkg/api/resource" <ins> "k8s.io/kubernetes/pkg/volume" </ins> "k8s.io/kubernetes/pkg/volume/csi/fake" )

What happened: kubelet crashed due to nil pointer dereference inside calcAndStoreStats function in Jul 03 06:50:42 kubelet[4440]: E0703 06:50:42. 4440 ] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference) Jul 03 06:50:42 kubelet[4440]: goroutine 3224 [running]: Jul 03 06:50:42 kubelet[4440]: (0x3992120, 0x773bdd0) Jul 03 06:50:42 kubelet[4440]: +0x7b Jul 03 06:50:42 kubelet[4440]: (0x0, 0x0, 0x0) Jul 03 06:50:42 kubelet[4440]: +0x82 Jul 03 06:50:42 kubelet[4440]: panic(0x3992120, 0x773bdd0) Jul 03 06:50:42 kubelet[4440]: +0x1b5 Jul 03 06:50:42 kubelet[4440]: (volumeStatCalculator).parsePodVolumeStats(0xc00178ef00, 0xc000c96cc0, 0xe, 0xc0014b8880, 0x0, 0xc000c96cc0, 0xe, 0x0, 0x0, 0x0, .) Jul 03 06:50:42 kubelet[4440]: +0x73 Jul 03 06:50:42 kubelet[4440]: (volumeStatCalculator).calcAndStoreStats(0xc00178ef00) Jul 03 06:50:42 kubelet[4440]: +0x753 Jul 03 06:50:42 kubelet[4440]: (volumeStatCalculator).StartOnce.func1.

1() Jul 03 06:50:42 kubelet[4440]: +0x2a Jul 03 06:50:42 kubelet[4440]: (0xc001c02d10) Jul 03 06:50:42 kubelet[4440]: +0x54 Jul 03 06:50:42 kubelet[4440]: (0xc001c02d10, 0xdf8475800, 0x3ff0000000000000, 0xc000f1dd01, 0xc0012c37a0) Jul 03 06:50:42 kubelet[4440]: +0xf8 Jul 03 06:50:42 kubelet[4440]: created by (volumeStatCalculator).StartOnce.func1 Jul 03 06:50:42 kubelet[4440]: +0x9d I think this happens because GetMetrics function can return nil for both metric(), and error, but calcAndStoreStats expects error to be not nil when metric is nil() What you expected to happen: Skip calculating stats for volumes that do not support volume stats. How to reproduce it (as minimally and precisely as possible): Create a volume using a csi driver that does not support volume stats. Anything else we need to know?: Environment: Kubernetes version (use ): Server version 15.0 Cloud provider or hardware configuration: OS (e.g: ): Ubuntu 16.04 LTS Kernel (e.g. ): 4.4.0-141-generic Install tools:kubeadm Network plugin and version (if this is a network-related bug): Others: Tested using synology-csi driver()
Sounds like you have a PR ready. Mind letting us know the PR number ? Thanks
I do not have a PR yet, but happy to make one. I'll add the PR number once I created it.
I created a PR and linked to this issue.
Didnt this solve this issue ?
No, it does not fix the problem that metric itself can be nil. Also, my pull request fixes the problem that csi_metrics returns nil value for error and the caller expects non nill value when csi driver does not support metrics.


Question: cd "${KUBE_ROOT}" # verify the providerless build # https://github.com/kubernetes/enhancements/blob/master/keps/sig-cloud-provider/1179-building-without-in-tree-providers/README.md <del> hack/verify-typecheck.sh --skip-test --tags=providerless --ignore-dirs=test </del> <ins> hack/verify-typecheck.sh --skip-test --tags=providerless --ignore-dirs=test/e2e </ins> # verify using go list if _out="$(go list -mod=readonly -tags "providerless" -e -json  k8s.io/kubernetes/cmd/kubelet/...

In I tried to reorganize the kube-proxy build a bit, and ran into the problem that verify-typecheck failed because was no longer buildable on darwin/amd64 and darwin/arm64. This was surprising because we don't build kube-proxy on OS X. It seems like verify-typecheck ought to be taking / from into account when deciding what to check. There's no reason to be checking the OS X type compatibility of binaries that we don't build on OS X.
/sig network
/remove-sig network /sig testing Agree, we need to plumb this through. We should also consider plumbing through static vs non-static and cgo enable/disable while we're at it. The goal is to be a low-fidelity cross-build to catch most code errors that would break the full without actually having to run the computationally expensive cross build, but there's some mismatch ...
/triage accepted


Question: "path/filepath" "time" <ins> apierrors "k8s.io/kubernetes/pkg/api/errors" </ins> legacyv1 "k8s.io/kubernetes/pkg/api/v1" rbacv1alpha1 "k8s.io/kubernetes/pkg/apis/rbac/v1alpha1" "k8s.io/kubernetes/pkg/runtime/schema"

Three tests now create RoleBindings to give permissions to test service accounts: NodeProblemDetector Ingress PreStop Creating RBAC Rolebindings currently requires the user issuing the create call to have the required permissions. If RBAC is not enabled, and the cluster hasn't been configured with any RBAC bootstrapping, that won't be possible. We should make this test code conditional on whether RBAC is in use.
c.f.
Quick fix is to tolerate forbidden errors there and assume e2e setups using methods other than RBAC have already granted permissions to the service accounts
I'll put together a PR to tolerate the 403 in the test helper.
I think this can be closed now that the e2e RBAC setup is best effort and failures there don't affect the test if another authorizer is in use
I agree. My original thought was that we'd make the RBAC test stuff only happen on RBAC-enabled setups. But, since we're going to be pushing setups towards using RBAC, we might as well have those pieces stick around in the common code (as best-effort).


Question: <del> The original file is https://github.com/kubernetes-csi/external-provisioner/blob/<version>/deploy/kubernetes/rbac.yaml </del> <ins> The original file is https://github.com/kubernetes-csi/external-provisioner/blob/VERSION/deploy/kubernetes/rbac.yaml </ins>

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: In below README files, URLs are written. But these URL include version name(master, release-1.0, etc...) which are enclosed in <and not shown in md files. What you expected to happen: URL will be shown correctly.
/sig storage
/remove-kind bug /kind documentation
can I take it up, as there is no activity for some days?
I already posted PR, thanks.


Question: - kube-apiserver - kube-controller-manager - kube-scheduler <ins> - monit </ins> - nginx - kube-client-tools - logrotate

We don't currently automatically restart etcd. We should.
Can we run etcd as a Pod without resource limit, so that we could benefit directly from 1) RestartPolicy: Always 2) resource monitoring and logging
When I bring up an e2e cluster (i.e. go run --up), without even running any of the e2e tests against it, it appears that monit is repeatedly killing and restarting etcd: davidopp more [UTC Feb 19 07:39:16] info : monit: generated unique Monit id and stored to '/var/lib/monit/id' [UTC Feb 19 07:39:16] info : Starting monit daemon [UTC Feb 19 07:39:16] error : State file '/var/lib/monit/state': Unable to read magic [UTC Feb 19 07:39:16] info : 'systeme2e-test-davidopp-master.c.striped-torus-760.internal' Monit started [UTC Feb 19 07:39:18] info : monit daemon with pid [5620] killed [UTC Feb 19 07:39:18] info : 'systeme2e-test-davidopp-master.c.striped-torus-760.internal' Monit stopped [UTC Feb 19 07:39:18] info : Starting monit daemon [UTC Feb 19 07:39:18] info : 'system_e2e-test-davidopp-master.c.striped-torus-760.internal' Monit started [UTC Feb 19 07:39:18] error : 'etcd' failed, cannot open a connection to INET[127.0.0.1:4001/v2/keys/] via TCP [UTC Feb 19 07:39:18] info : 'etcd' trying to restart [UTC Feb 19 07:39:18] info : 'etcd' stop: /etc/init.d/etcd [UTC Feb 19 07:39:18] info : 'etcd' start: /etc/init.d/etcd [UTC Feb 19 07:41:18] error : 'etcd' failed, cannot open a connection to INET[127.0.0.

1:4001/v2/keys/] via TCP [UTC Feb 19 07:41:18] info : 'etcd' trying to restart [UTC Feb 19 07:41:18] info : 'etcd' stop: /etc/init.d/etcd [UTC Feb 19 07:41:18] info : 'etcd' start: /etc/init.d/etcd [UTC Feb 19 07:43:18] error : 'etcd' failed, cannot open a connection to INET[127.0.0.1:4001/v2/keys/] via TCP [UTC Feb 19 07:43:18] info : 'etcd' trying to restart [UTC Feb 19 07:43:18] info : 'etcd' stop: /etc/init.d/etcd [UTC Feb 19 07:43:18] info : 'etcd' start: /etc/init.d/etcd [UTC Feb 19 07:45:18] error : 'etcd' failed, cannot open a connection to INET[127.0.0.1:4001/v2/keys/] via TCP [UTC Feb 19 07:45:18] info : 'etcd' trying to restart [UTC Feb 19 07:45:18] info : 'etcd' stop: /etc/init.d/etcd [UTC Feb 19 07:45:18] info : 'etcd' start: /etc/init.d/etcd [.] I examined the etcd logs and there is nothing suspicious there. I suspect this may be the cause of some of the e2e test flakiness.
I think this particular problem is caused by firewall settings.
actually didn't fix the problem. Here's the [UTC Feb 19 23:22:25] info : monit: generated unique Monit id and stored to '/var/lib/monit/id' [UTC Feb 19 23:22:25] info : Starting monit daemon [UTC Feb 19 23:22:25] error : State file '/var/lib/monit/state': Unable to read magic [UTC Feb 19 23:22:25] info : 'systeme2e-test-davidopp-master.c.striped-torus-760.

internal' Monit started [UTC Feb 19 23:22:26] info : monit daemon with pid [5481] killed [UTC Feb 19 23:22:26] info : 'systeme2e-test-davidopp-master.c.striped-torus-760.internal' Monit stopped [UTC Feb 19 23:22:27] info : Starting monit daemon [UTC Feb 19 23:22:27] info : 'system_e2e-test-davidopp-master.c.striped-torus-760.internal' Monit started [UTC Feb 19 23:22:32] error : 'etcd' failed, cannot open a connection to INET[kubernetes-master:4001/v2/keys/] via TCP [UTC Feb 19 23:22:32] info : 'etcd' trying to restart [UTC Feb 19 23:22:32] info : 'etcd' stop: /etc/init.d/etcd [UTC Feb 19 23:22:32] info : 'etcd' start: /etc/init.d/etcd [UTC Feb 19 23:24:37] error : 'etcd' failed, cannot open a connection to INET[kubernetes-master:4001/v2/keys/] via TCP [UTC Feb 19 23:24:37] info : 'etcd' trying to restart [UTC Feb 19 23:24:37] info : 'etcd' stop: /etc/init.d/etcd [UTC Feb 19 23:24:37] info : 'etcd' start: /etc/init.d/etcd [UTC Feb 19 23:26:43] error : 'etcd' failed, cannot open a connection to INET[kubernetes-master:4001/v2/keys/] via TCP [UTC Feb 19 23:26:43] info : 'etcd' trying to restart [UTC Feb 19 23:26:43] info : 'etcd' stop: /etc/init.d/etcd [UTC Feb 19 23:26:43] info : 'etcd' start: /etc/init.d/etcd [UTC Feb 19 23:28:48] error : 'etcd' failed,

cannot open a connection to INET[kubernetes-master:4001/v2/keys/] via TCP [UTC Feb 19 23:28:48] info : 'etcd' trying to restart [UTC Feb 19 23:28:48] info : 'etcd' stop: /etc/init.d/etcd [UTC Feb 19 23:28:48] info : 'etcd' start: /etc/init.d/etcd [UTC Feb 19 23:30:53] error : 'etcd' failed, cannot open a connection to INET[kubernetes-master:4001/v2/keys/] via TCP [UTC Feb 19 23:30:53] info : 'etcd' trying to restart [UTC Feb 19 23:30:53] info : 'etcd' stop: /etc/init.d/etcd [UTC Feb 19 23:30:53] info : 'etcd' start: /etc/init.d/etcd
The reason the fix didn't work is that it hard-codes the hostname as kubernetes-master but the hostname may actually be anything. For example, in e2e test the master hostname is e2e-test-${USER}-master
This is fixed by changing etcd to startup with -bind-addr 0.0.0.0:4001 I'm sending a PR now.
(And thanks to for figuring out the problem was that etcd wasn't listening on the localhost address.)


Question: "k8s.io/apimachinery/pkg/api/testing/codec.go", "k8s.io/apimachinery/pkg/api/testing/fuzzer.go", "k8s.io/apimachinery/pkg/api/testing/roundtrip.go", <ins> "k8s.io/apimachinery/pkg/api/testing/valuefuzz.go", </ins> ], tags = ["automanaged"], deps = [

The apimachinery roundtripping test does not ensure that deep copies work. We need to provide a deep copy test that makes sure we don't end up with shallow copies.
I looked into gofuzz some days ago and noticed that it does not support value-only fuzzing that preserves object structure (= does not change pointer values). So we need something clever, maybe reflection based here. Any other idea?


Question: // Node2 // Image: gcr.io/250:latest 250MB <del> // Score: 100 * (250M/2 - 23M)/(1000M - 23M) = 100 </del> <ins> // Score: 100 * (250M/2 - 23M)/(1000M * 2 - 23M) = 5 </ins> pod:          &v1.Pod{Spec: test40250}, nodes:        []*v1.Node{makeImageNode("machine1", node403002000), makeImageNode("machine2", node25010)}, <del> expectedList: []framework.NodeScore{{Name: "machine1", Score: 0}, {Name: "machine2", Score: 10}}, </del> <ins> expectedList: []framework.NodeScore{{Name: "machine1", Score: 0}, {Name: "machine2", Score: 5}}, </ins> name:         "two images spread on two nodes, prefer the larger image one", }, {

<!-- Please only use this template for submitting enhancement requests --What would you like to be : redefine the formula, the score should be capped per images instead of pod. Why is this needed: For the pod that going to be scheduled, calculate the score based on the below steps, Scaled score base on the image size and it's spread over the cluster, image[i].nodes: number of nodes that already have image ! Capped score based on the assumption 90% image size on dockerhub falls into the range [23M ~ 1000M] lowerbound: 23M upperbound:1000M ! Final scores returned for scheduling, maxscore: 100 ! The second step is not well guarded, the assumption is based on the image size while the pod may hold multiple images, it is not surprise to see that total image size of the pods is larger than 1000M, while each of might be only around 300M. In this below case, following the above formula, NODE1 and NODE2 willl get the same scores(100), since both of them are capped by the max score (100), the ideal result is score of NODE2 should be larger than NODE1 (all requested images are ready on NODE2) and is preferred, the larger the sample1 here the more scores should be expected for NODE2. !
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig scheduler Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scheduling
cc
Thanks a lot for summarizing the problem. I agree that the formula needs improvement. ~I think doing the capping for each image solves the original problem and what said in There are two options: only the total from to for each container. We have to ask ourselves how much signal each formula gives and depending on what. Think about different scenarios: a pod with a lot of very small containers, a pod with a few big containers, etc.
/assign
Here is my thoughts, step1: calculate image level priority, for each image (should be same as exsting func ) step2: cap the each image's score with thresholds, minThreshold and maxThreshold.

(should be same as existing func except that it's per image instead of pod) step3: score normalization per image. step4: calculate pod level priority, and return the final score, range (0~100) step1 and step2 primarily update the exiting code to make it is image level priorities instead of pod level. Based on the above steps, consider the below two cases, a pod with four images, case1: All images are ready on node2 node1 (0, 80, 90, 60) - final score is: (80+90+60) / 4 = 57 node2 (90, 80, 90, 60) - final score is: (90+80+90+60) / 4 = 80 √ case2: each node has three images ready but image1 is larger than image2. node1 (0, 80, 70, 60) - final score is: (80+70+60) / 4 = 52 node2 (90, 0, 70, 60) - final score is: (90 +70+60) / 4 = 55 √ thoughts? cc
Thanks can you update the the examples with the score that we would have gotten with the current function?
For reference, see this analysis for a different plugin
Also it's worth to see the difference with just capping per pod, instead of per container.
can you also describe the image sizes you used? the example in the original post assumed three images, while your analysis is assuming four.
agree with and the explanation in looks good but just to be sure it would be nice to see the math worked out with image sizes, and compared to the current formula
Sorry for the confusion, let me switch back to the original three images, update the image size a little bit to make it easier to understand, and also fit to the case of and . Here is the pod, ! and three nodes in the cluster, ! Let's see the difference between two approaches, capping per pod, step1: ! NODE1: scores = 30 2/3 + 1500 2/3 = 1020 NODE2: scores = 30 2/3 + 600 2/3 + 1500 2/3 = 1420 step2: !

NODE1: 1020 1000 (maxThreshold), cappedscore = 1000 NODE2: 1420 1000 (maxThreshold), cappedscore = 1000 step3: ! upperbound = 100 / lowerbound = 23 / maxscore = 100 NOTE1: score = 100 (1000 - 23) / (1000 - 23) = 100 NODE2: score = 100 (1000 - 23) / (1000 - 23) = 100 potentially, NODE1 will be chose for scheduling. capping per images, step1: calculate image level priority, NODE1: image1: score = 30 2/3 = 20 image2: score = 0 image3: score = 1500 2/3 = 1000 NODE2: image1: score = 30 2/3 = 20 image2: score = 600 2/3 = 400 image3: score = 1500 2/3 = 1000 step2/3: cap each image's score with thresholds and calculate the image score, ! NODE1: image1: score = 100 (23 - 23) / ( 1000 - 23) = 0 image2: score = 0 image3: score = 100 (1000 - 23) / (1000 - 23) = 100 NODE2: image1: score = 100 (23 - 23) / ( 1000 - 23) = 0 image2: score = 100 (400 - 23) / (1000 - 23) = 35 image3: score = 100 * (1000 - 23) / (1000 - 23) = 100 step4: final score NODE1: finalscore = 100 / 3 = 33 NODE2: final_score = 135 / 3 = 45 NODE2 will be chosen for scheduling.
Computing the score per image and normalizing by the number of images make sense to me at first glance, but I would like to see more examples.

It would be much easier to digest the analysis if it was summarized in tabular form comparing three scoring functions: (1) current one, (2) one in and (3) multiple big images, multiple small images (total less than the threshold) and a mix. Also, compare cases where the spread is high vs low.
Yes, we don't need to see the individual steps, but a good set of scenarios with final scores. I didn't mean just the old function. We know it's very ineffective. What I meant is capping per pod using the strategy you used in
Did some comparison here, spread has been included based on the layout of each node, it's not easy for me to pull this factor out to focus on this factor only. I will design more testcases when the patch is ready for review which I hope could be more clear. column 1: pod with three images column 2: three nodes' images layout column 3 / 4 / 5: score of each node for different approaches. pod cap and normalizing per image
[10, 20, 30] & small [0,0,0] [800, 1200, 1500] & big [33, 59, 67] [30, 600, 1500] & mix [33, 45, 0]
Normalizing by the number of images gives each image the same "weight". Another variation of the third option is to try and multiply each image score with its size divided by the total size of all images, can you add that as another option?
I don't understand that suggestion. The score of an image is already proportional to its size. I would say the difference between the two suggestions is: Capping per pod cares about total downloaded size Capping per image gives roughly the same importance to each image. I think capping per pod makes more sense. However, there is some information we are losing in both suggestions, as seen in the first case. Even if a lot of small images need to be downloaded, we are giving a 0 score to all nodes. I think a cap from makes more sense, as suggested in the initial PR.
It helps giving more importance to larger images.
If the statistics are accurate with regards to image size distribution, then having many small images all scoring zero isn't common.

For the more common cases (sizes between 32-1000), it seems that both approaches will likely give similar results, so I am fine with both approaches.
That would make the score proportional to the square of the size. Just to play safe (ie, not make the score too strong in a way that could affect spreading or resource allocation) I suggest we go with capping per pod. And then we can re-evaluate next release.
Speaking of statistics. They might have changed since this plugin was first developed. Is there a reliable source?
True for uncapped images, but everything we are doing is just heuristics, and the suggestion is just for the sake of exploring options, not advocating a specific one. Playing it safe sounds good to me.
This approach needs the smallest change and the minimum impact.
+1, this is not common case we can see that a pod with lots of small images (all < 23M), but a corner case that might be improved. No perfect solution so far, but either one is better than the original one IMHO.
/cc , Do you have any comments so far?
btw, has been updated to the second approach (capping and normalization per image).
I would go with this approach as well, it preserves the behavior for multiple small images, and improve it for multiple big ones.
seems like a simple enough compromise for now, and running through some of the examples with that formula it does get more of the specificity you're looking for with large images without normalizing all smaller images right to 0 (so we keep some of the detail there too). I would like to see some updated statistics on the thresholds, and while 90% of the images might fall between them I'm wondering if that's a totally normal distribution. suggestion in got me (tangentially) thinking if we could use thresholds that are more relevant to the pod in question, rather than all of Dockerhub. Because as it is now hypothetical pods (and clusters) which have all small images don't get much benefit from ImageLocality. But I recognize that could be intentional since download resources for small images is ultimately negligible.
I also believe that scoring of small images by zero is intentional, but seems like making lots of small images in pod accountable for the scheduling is slightly better.
It looks like we have consensus. Can you update the PR?


Question: "io" "strings" "testing" <ins> "time" </ins> "k8s.io/kubernetes/pkg/probe" )

I was trying a scenario where liveness/readiness probe command run inside the container for long time('sleep 10 && exit 10') with shorter timeoutSecod value say 2 Sec. As expected container restarted because of timeout but when i check the events in pod describe it stated only unhealthy and in events actual reason for probe failure is not printed "liveness probe failed:<'no reason printed'". Without clear logs or pod events description it is very difficult to debug and understand what is the failure. Below is the example manifest file to reproduce the issue. apiVersion: v1 kind: Pod metadata: labels: test: liveness name: liveness-exec spec: containers: name: liveness image: args: /bin/sh -c touch /tmp/healthy; sleep 30; rm -rf /tmp/healthy; sleep 600 readinessProbe: exec: command: [/bin/sh, -c , 'sleep 10 && exit10'] initialDelaySeconds: 5 periodSeconds: 5 timeoutSeconds: 2 livenessProbe: exec: command: [/bin/sh, -c ,'sleep 5 && exit10'] initialDelaySeconds: 5 periodSeconds: 5 timeoutSeconds: 2 It would be better if actual reason or something for probe failure printed in pod describe events incase of liveness/readiness probe gets timeout. Example like below in pod describe events: Warning Unhealthy 3s (x2 over 8s) kubelet Liveness probe failed: "probe failed due to timeout " Warning Unhealthy 3s (x2 over 8s) kubelet Readiness probe failed: "probe failed due to timeout " ! It would be better if actual reason or something for probe failure printed in pod describe events incase of liveness/readiness probe gets timeout. Example like below in pod describe events: Warning Unhealthy 3s (x2 over 8s) kubelet Liveness probe failed: "probe failed due to timeout " Warning Unhealthy 3s (x2 over 8s) kubelet Readiness probe failed: "probe failed due to timeout " Below is the example manifest file to reproduce the issue.

apiVersion: v1 kind: Pod metadata: labels: test: liveness name: liveness-exec spec: containers: name: liveness image: args: /bin/sh -c touch /tmp/healthy; sleep 30; rm -rf /tmp/healthy; sleep 600 readinessProbe: exec: command: [/bin/sh, -c , 'sleep 10 && exit10'] initialDelaySeconds: 5 periodSeconds: 5 timeoutSeconds: 2 livenessProbe: exec: command: [/bin/sh, -c ,'sleep 5 && exit10'] initialDelaySeconds: 5 periodSeconds: 5 timeoutSeconds: 2 Create a pod using above manifest file and do pod describe (kubectl describe pod <pod-name ) and check for the events. ! No response kubectl version Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2021-08-31T06:59:53Z", GoVersion:"go1.16.4", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2021-08-31T06:52:44Z", GoVersion:"go1.16.4", Compiler:"gc", Platform:"linux/amd64"} openstack <details</details<details</detailscontainerd No response
I tried to fix one similar issue in
/sig node
/triage accepted
can I work on this issue? Thanks very much. /assign
We've experienced this issue with clusters on v1.20. The for that version mention the following change: Since v1.20 is seemingly when exec probe timeouts started being enforced, would it be possible to have this fix for this issue backported to all minor versions since then? Thanks!
bump on that question
What do you think of? Thanks very much.


Question: // delete immediately, or no graceful deletion supported out := e.NewFunc() if err := e.Storage.Delete(ctx, key, out); err != nil { <ins> // Please refer to the place where we set ignoreNotFound for the reason // why we ignore the NotFound error . if storage.IsNotFound(err) && ignoreNotFound && lastExisting != nil { // The lastExisting object may not be the last state of the object // before its deletion, but it's the best approximation. return e.finalizeDelete(lastExisting, true) } </ins> return nil, etcderr.InterpretDeleteError(err, e.QualifiedResource, name) } return e.finalizeDelete(out, true)

cc
PTAL
The root cause is that the node controller and kubelet delete the pods using their names (they should use UID instead): (this is not an exhaustive list) In test-, we create and delete the same pod multiple times, so it's possible that the node controller or the kubelet sends a deletion request to the api server, with the intention of deleting the Nth occurrence of a pod, while it's actually deleting the N+1th occurrence of the pod. To reproduce the flake, I introduced a random delay between 500~1500 ms in the node controller's execution path (), then using a script to create/delete a pod repeatedly, and the flake will occur after several rounds.
Why is it not okay to delete by name?
namespace+name pair is not an unique identifier across time.
the client library and the API server don't support deletion by UID, I think we should support.
Thanks!! My thoughts: does node controller care about these pods? I'm pretty sure the original author wasn't expecting that. a temporary fix, is it hard to just use new names for every incarnation of these pods?
Deleting by UID is probably a bigger can of worms than we want to open right now, although it will eventually be important. Thinking out loud here, but maybe it'd be possible to add a resource version to the delete command. I.e., "delete only if it hasn't changed since rv=xxxxx". But I still think it's probably ideal to turn off the node controller for this test.
I'm also thinking if we can centralize all the pod deletion logic into kubelet, but it seems some of the logic need to stay in the node controller, e.g, , because only the node controller knows if the node exists. deletion probably can be done by the kubelet.
Looks like node controller is getting an update of the pod where DeleteionTimestamp is set, which seems like a bug if --grace-period is set to 0. My understanding is that if grace period is zero, the pod should get immediately deleted. That pod will never make it to a kubelet because it hasn't been scheduled yet. Thus which kubelet would delete it? I'm pretty sure none of those deletions can be handled in the kubelet.

Btw, when running hack/test- (), the node controller generates many messages about the failed deletions: or , these are because kubelet wins the race and has deleted the pods. These don't cause test- to fail.
per offline discussion with Mike, I think he's right. I'll check the api server's code.
That's my expectation, too (that grace period being zero should bypass all graceful deletion code and just immediately delete the pod).
When the grace period is zero, api server update the storage(), and then delete the pod from the storage (). The check of if grace period is 0 is after the update(). It seems it should not do the update if grace periods is zero.
Do you agree that the above is a bug?
Hm, is the problem a missing or incorrect BeforeDelete handler?
I think we are dealing with two problems here: grace-period=0, the registry shouldn't do an update. client library should provide a Delete() method that takes an unique identifier, the "name+namespace" pair is ambiguous across time.
Thinking about this more, I wonder if the update is intentional. If so, we need some coordination so that both apiserver and kublet don't try to delete.
Grace period 0 is intentional, but may not be required for correct behavior. If the user signals delete immediately, we record that intent so it shows up in watch and can be a signal kubelet to terminate immediately (although we don't actually do that today, so we could walk back the behavior). We definitely need a precondition delete on uid.
There will never be a single owner of pod lifecycle because pods are bound to nodes, and so all components have to deal with races to delete.
I chose to err on the side of being explicit about user intent with the update to zero out termination. But technically explicit deletion can be the same signal.
A delete operation will also trigger a watch event, so kubelet will be signaled even if we skip the update right? [edit] Oh I see you answered right before I asked.
It should be, yes. In the interest of fixing this test in the short term, I think the easiest thing to do might be to turn off the node controller for this section of the test. (Is it needed by anything in the test?)

But we should file at least two issues: "kubectl delete can race with graceful deletion"-- Things have to handle races, but there shouldn't be a race where you delete something and your delete fails because your own delete caused another component to issue a delete for the same thing... As far as I can see from the code, this could happen if apiserver context-switched at the wrong times (even if the more common thing of an old delete getting a new object is the issue at hand). controllers should delete via UID (or namespace/name + RV) -- I wouldn't be surprised if there's already an issue for this.
+1 to resourceVersion and uid preconditions on delete. Yes, maybe we shouldn't update the grace period before deletion in the case that the grace period is 0. Why can't we use unique pod names in the test? Turning off node controller seems like a bad idea. Index by uid is .
I don't see any flag option to turn of node controller. Pod's name is specified in the config file, is there any clean way to change the pod's name each time is called? Anyway, I'll try to make a PR to skip update if grace period is 0. That should alleviate this flake.
cc/ in case you are running into this issue.
As for how to change the name: we could use sed to replace the name. Or we could change the namespace.
This should be fixed by . I filed for deletion via UID or namespace/name+resourceVersion.
This happened again:
This error "] pods "redis-proxy" not found" here seems related to the error in .
I think the issue in is the same. The last few lines of logs: The symptom seems to align with my hypothesis in suggests the nodecontroller has successfully deleted a pod (probably "redis-master"), so kubectl fails with . I'll add more logs in nodecontroller to make sure this is the problem. In fact, we knew this race condition when investigating , but we discovered other race condition that are more likely to happen so we didn't solve this one.
Sent to show what pod nodecontroller has deleted.
Anywhere where we delete lists of pods (and other resources), including those filtered by label selector, we should ignore not-found errors. There are such loops in the controller reapers for cascading deletion. cc
makes safely ignore the NotFound error.

forces to issue a "GET" before submitting "DELETE". The "GET" will effectively check if the user is trying to delete a resource that is never created, so if later the API server sends a NotFound error in response to the "DELETE" request, kubectl can safely ignore it.


Question: ```console $ kubectl create -f docs/admin/limitrange/limits.yaml --namespace=limit-example <del> limitranges/mylimits </del> <ins> limitrange "mylimits" created </ins> ``` Let's describe the limits that we have imposed in our namespace.

Since resource QoS is implemented, limitrange also supports request constraint, we can add an example for that.
cc


Question: } } } <ins> func TestParseEnv(t *testing.T) { tests := []struct { envArray  []string expected  []api.EnvVar expectErr bool test      string }{ { envArray: []string{ "THIS_ENV=isOK", "HAS_COMMAS=foo,bar", "HAS_EQUALS=jJnro54iUu75xNy==", }, expected: []api.EnvVar{ { Name:  "THIS_ENV", Value: "isOK", }, { Name:  "HAS_COMMAS", Value: "foo,bar", }, { Name:  "HAS_EQUALS", Value: "jJnro54iUu75xNy==", }, }, expectErr: false, test:      "test case 1", }, { envArray: []string{ "WITH_OUT_EQUALS", }, expected:  []api.EnvVar{}, expectErr: true, test:      "test case 2", }, { envArray: []string{ "WITH_OUT_VALUES=", }, expected:  []api.EnvVar{}, expectErr: true, test:      "test case 3", }, { envArray: []string{ "=WITH_OUT_NAME", }, expected:  []api.EnvVar{}, expectErr: true, test:      "test case 4", }, } for _, test := range tests { envs, err := parseEnvs(test.envArray) if !test.expectErr && err != nil { t.Errorf("unexpected error: %v (%s)", err, test.test) } if test.expectErr && err != nil { continue } if !reflect.DeepEqual(envs, test.expected) { t.Errorf("nexpected:n%#vnsaw:n%#v (%s)", test.expected, envs, test.test) } } } </ins>

I'm running application with several env variables. One of them has a key ending with ==. Another variable has a comma separated list of email addresses. While perfectly valid on *nix and windows, we can't pass these values into Kubernetes run command. It fails with "error: invalid env:" Example command: The parsing in should be more robust
I volunteer myself to fix this.
Maybe they can be passed from file? I mean s'rsly.. you need to lits 20 variables by hand?? I've specified them in yaml file already, but kubectl-run have no way to specify it.. without this, deployment is totally impossible when you need to run migrations first!