Question: SRCS = nnp ARCH ?= amd64 TARGET ?= $(CURDIR) <ins> GOARM ?= 7 </ins> GOLANG_VERSION ?= latest SRC_DIR = $(notdir $(shell pwd)) export

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Building the image binaries requires the GOARM variable to be set. Because it is not defined in all Makefiles, cannot the image binaries for the images in question. What you expected to happen: should succeed. How to reproduce it (as minimally and precisely as possible): git clone cd kubernetes/test/images/entrypoint-tester make Anything else we need to know?: Environment: N/A Kubernetes version (use ): N/A Cloud provider or hardware configuration: N/A OS (e.g: ): N/A Kernel (e.g. ): N/A Install tools: N/A Network plugin and version (if this is a network-related bug): N/A Others: N/A
/sig testing
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or .

/lifecycle stale
/lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten


Question: if len(fake.Actions()) == 0 { return } <del> for _, action := range fake.Actions() { </del> <ins> for _, action := range filterInformerActions(fake.Actions()) { </ins> t.Logf("unexpected action: %#v", action) } t.Errorf("expected deployment controller to not take action") } <ins> func filterInformerActions(actions []core.Action) []core.Action { ret := []core.Action{} for _, action := range actions { if len(action.GetNamespace()) == 0 && (action.Matches("list", "pods") || action.Matches("list", "deployments") || action.Matches("list", "replicasets") || action.Matches("watch", "pods") || action.Matches("watch", "deployments") || action.Matches("watch", "replicasets")) { continue } ret = append(ret, action) } return ret } </ins>

https://k8s- Failed: TestDeploymentController_dontSyncDeploymentsWithEmptyPodSelector {deployment}
did you see this failure by any chance while working on
No. I didn't see that one.
https://k8s- Failed: TestDeploymentController_dontSyncDeploymentsWithEmptyPodSelector {deployment}
Found it. Opened
https://k8s- Failed: TestDeploymentController_dontSyncDeploymentsWithEmptyPodSelector {deployment}
https://k8s- Failed: TestDeploymentController_dontSyncDeploymentsWithEmptyPodSelector {deployment}
https://k8s- Failed: TestDeploymentController_dontSyncDeploymentsWithEmptyPodSelector {deployment}
Running this test 8 times in a loop on my workstation, I was able to get a reliable failure signal.
This is still failing. Please take a look into


Question: # Dynamically set the master size by the number of nodes, these are guesses # TODO: gather some data if [[ -z ${MASTER_SIZE} ]]; then <del> if (( ${NUM_NODES} < 50 )); then MASTER_SIZE="t2.micro" elif (( ${NUM_NODES} < 150 )); then MASTER_SIZE="t2.small" </del> <ins> if (( ${NUM_NODES} < 150 )); then MASTER_SIZE="m3.medium" </ins> else <del> MASTER_SIZE="t2.medium" </del> <ins> MASTER_SIZE="m3.large" </ins> fi fi

I propose that we default to for the master node (and maybe for 150 nodes?). We've seen problems with running out of memory on the master with the , and we've seen pauses that look like CPU exhaustion on the t2 class in general. Also the master is the hardest instance to resize. It is doable to reconfigure the ASG for the nodes (or add a new one, but the master is tricky). what do you think? cc


Question: resourceName:     "domain1.com/resource1", resourceQuantity: *resource.NewQuantity(int64(2), resource.DecimalSI), devs:             checkpoint.DevicesPerNUMA{0: []string{"dev1", "dev2"}}, <ins> topology:         true, </ins> } res2 := TestResource{ resourceName:     "domain2.com/resource2", resourceQuantity: *resource.NewQuantity(int64(1), resource.DecimalSI), devs:             checkpoint.DevicesPerNUMA{0: []string{"dev3", "dev4"}}, <ins> topology:         false, </ins> } testResources := make([]TestResource, 2) testResources = append(testResources, res1)

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: Which test(s) are failing: Since when has it been failing: Testgrid link: Reason for failure: Anything else we need to know: Can't find this in Triage Example spyglass links: /sig scheduling /cc /priority important-soon
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I don't quite think this test should be owned by SIG-Scheduling. Most of the failures happen in the Nvidia GPU setup phase. BTW: although irrelevant, last time we see a failure of this test is .
Searching keyword "UnexpectedAdmissionError" gets the following occurrences:
/sig node cc in case you're aware of any recent changes that may have caused this failure.
From the I can see a bunch of: Checking the kubelet log, I can see the gpu capacity has been changing And before the capacity changed to 0, we can see the kubelet was attempting to allocate 1 GPU to the test pod, but some NPE was thrown, and kubelet stopped/restarted.
The nil pointer is from here: , which is a new change introduced by
The current device plugin (the one defined ) is using an old version of device plugin api (see ), which does not have field under . So calling will end up with NPE.
Thanks , the analysis makes great sense to me. Could you please work on a fix to address this? (Probably updating )
Hi , I am still looking at the code, because the code does have the logic to handle the case where device plugin does not return affinity: Also, even if the Topology is not defined, I am not sure will throw NPE () (please correct me if I am wrong here). I will need to look more closely at the code to find out where the NPE is from. Updating the might take long time because we will need some testing to make sure it doesn't break anything.

Currently I am thinking of identifying the source of NPE, and make sure can handle that case.
/cc
I submitted a fix. My fault, it was a real case when pointer to Topology could be a nil. But still fails: I don't observe SIGSEGV now in , but 3 problems there: 1113 09:26:19. 1336 ] failed to run Kubelet: cannot create certificate signing request: is forbidden: User "kubelet" cannot create resource "certificatesigningrequests" in API group "" at the cluster scope of nvidia-driver-installer stops and restarts after a while. 's () NoopManager is using
can you show full error msg of "NVIDIA GPU metrics will not be available: %s"?
I1113 15:43:26. 1338 ] Found device with vendorID "0x10de" W1113 15:43:26. 1338 ] NVIDIA GPU metrics will not be available: Could not initialize NVML: could not load NVML library I1113 15:43:26. 1338 ] Machine: {Timestamp:2020-11-13 15:43:26. +0000 UTC m=+5. NumCores:2 NumPhysicalCores:1 NumSockets:1 CpuFrequency: MemoryCapacity: MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:2048 NumPages:0}] MachineID: SystemUUID:-9824-3a64-f41f- BootID:-f891-4c0b-a34d- Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:39 Capacity: Type:vfs Inodes: HasInodes:true} {Device:/dev/sda8 DeviceMajor:8 DeviceMinor:8 Capacity: Type:vfs Inodes:4096 HasInodes:true} {Device:/dev/sda1 DeviceMajor:8 DeviceMinor:1 Capacity: Type:vfs Inodes: HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:19 Capacity: Type:vfs Inodes: HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:21 Capacity: Type:vfs Inodes: HasInodes:true}

{Device:/sys/fs/cgroup DeviceMajor:0 DeviceMinor:22 Capacity: Type:vfs Inodes: HasInodes:true} {Device:/etc/machine-id DeviceMajor:0 DeviceMinor:21 Capacity: Type:vfs Inodes: HasInodes:true} {Device:/mnt/disks DeviceMajor:0 DeviceMinor:38 Capacity: Type:vfs Inodes: HasInodes:true} {Device:overlayfs0-43 DeviceMajor:0 DeviceMinor:43 Capacity: Type:vfs Inodes: HasInodes:true} {Device:/var/lib/cloud DeviceMajor:0 DeviceMinor:45 Capacity: Type:vfs Inodes: HasInodes:true} {Device:/dev/root DeviceMajor:253 DeviceMinor:0 Capacity: Type:vfs Inodes: HasInodes:true}] DiskMap:map[253:0:{Name:dm-0 Major:253 Minor:0 Size: Scheduler:none} 8:0:{Name:sda Major:8 Minor:0 Size: Scheduler:mq-deadline} 9:0:{Name:md0 Major:9 Minor:0 Size:0 Scheduler:none}] NetworkDevices:[{Name:eth0 MacAddress:42:01:0a:28:00:03 Speed:-1 Mtu:1460}] Topology:[{Id:0 Memory: HugePages:[{PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 1] Caches:[{Size: Type:Data Level:1} {Size: Type:Instruction Level:1} {Size: Type:Unified Level:2}] SocketID:0}] Caches:[{Size: Type:Unified Level:3}]}] CloudProvider:GCE InstanceType:n1-standard-2 InstanceID:} I1113 15:43:26. 1338 ] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. I1113 15:43:26. 1338 ] Version: {KernelVersion:5.4.49+ ContainerOsVersion:Container-Optimized OS from Google DockerVersion:Unknown DockerAPIVersion:Unknown CadvisorVersion: CadvisorRevision:} I guess kubelet can't load in cAdvisor the NVML library.
I have looked into this issue actually.

This is caused by a recent change on cadvisor: We used to return a real nvidiaManager but now only a Noop one if the NVML lib is not there. And currently the NVML lib is loaded by daemonset, which will not there when the kubelet starts. This PR should fix it: But btw, this bug will not fail the test because it will only cause missing GPU usage metrics. The plan is to cherry-pick the fix above to cadvisor 0.37, and then update the vendor for kubernetes


Question: cache utilcache.Cache } <del> func init() { metrics.Register() } </del> // Implements storage.Interface. func (h *etcdHelper) Versioner() storage.Versioner { return h.versioner

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: endpoint of different k8s components expose Prometheus metrics that are not for them. For example, exposes metrics: or : What you expected to happen: Prometheus metrics only for to appear on kubelet's endpoint. How to reproduce it (as minimally and precisely as possible): Access the endpoint of all Kubernetes components Anything else we need to know?: This problem comes from usage of functions in different packages that automatically register Prometheus metrics. Just importing that package or some package that depends on it can cause those metrics to appear in unexpected places. In the list bellow you can find the different packages that have init functions and affected binaries: [ ] () cloud-controller-manager hyperkube kube-apiserver kube-controller-manager kube-proxy kube-scheduler [ ] () cloud-controller-manager hyperkube kube-apiserver kube-controller-manager kube-proxy kube-scheduler [ ] cloud-controller-manager hyperkube kube-apiserver kube-controller-manager [ ] cloud-controller-manager hyperkube kube-apiserver kube-controller-manager kubelet [ ] cloud-controller-manager hyperkube kube-apiserver kube-controller-manager kube-proxy kube-scheduler [ ] cloud-controller-manager hyperkube kube-apiserver kube-controller-manager kube-proxy kube-scheduler kubectl kubelet [ ] cloud-controller-manager hyperkube kube-apiserver kube-controller-manager kube-proxy kube-scheduler kubelet [ ] hyperkube kube-apiserver kube-controller-manager [ ] hyperkube kube-apiserver [ ] cloud-controller-manager hyperkube kube-apiserver kube-controller-manager kubelet [ ] cloud-controller-manager hyperkube kube-apiserver kube-controller-manager kube-proxy kube-scheduler kubelet Environment: Kubernetes version (use ):Cloud provider or hardware configuration: N/A OS (e.g. from /etc/os-release): N/A Kernel (e.g. ): N/A Install tools: N/A Others: N/A /sig instrumentation
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: processed bool } <del> // ClaimToClaimKey return namespace/name string for pvc func ClaimToClaimKey(claim *v1.PersistentVolumeClaim) string { return fmt.Sprintf("%s/%s", claim.Namespace, claim.Name) } </del> // UpdatePVSize updates just pv size after cloudprovider resizing is successful func UpdatePVSize( pv *v1.PersistentVolume,

I migrated an existing EBS volume from in-tree plugin to external aws-ebs-csi-driver (). Resizing PVC twice in 6 hour window causes kube-controller-manager to crash. Exception message: Kube-controller-manager shouldn't crash in such case. If external aws-ebs-csi-driver fails to provision the pvc it should just log an warning or error and continue operating as usual running cluster version 1.22.9 using in-tree aws ebs driver. pvc: Install aws-ebs-csi-driver following featureGate CSIMigrationAWS on all cluster components (with draining nodes before enabling featureGate on kubelets - according to warning ) we have ebs volume that has been migrated to aws-ebs-csi-driver. pvc by increasing .spec.resources.request.storage to 2Gi. PVC, PV and EBS volume are correctly resized. once again not later than 6 hours after first resize. Increase .spec.resources.request.storage to 3Gi. Now kube-controller-manager throws an exception and crashes: EBS CSI Driver logs: I tried also enabling alpha featureGate InTreePluginAWSUnregister but the issue still occurs. <details</details<detailsAWS </details<details</details<details</details<detailsdocker </details<detailsaws-ebs-csi-driver version 1.10.0 </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cloud-provider
/sig storage some situation where the sync loop passes a nil parameter to
/assign
SetClaimResizer may return nil pvc and non-nil error, ClaimToClaimKey naively assumes non-nil as described above, will create PR
But the posted bug is I think 1.22 specific and should never happen on master or 1.24 and hence this fix against master is kind of okay and should fix the issue but at the same time code in is still kinda broken in 1.22 because it returns if patching fails.


Question: if err != nil { return nil, err } // if plugin doesnot support volume status, return. if !volumeStatsSet { <del> return nil, nil </del> <ins> return nil, volume.NewNotSupportedErrorWithDriverName( string(mc.csiClientGetter.driverName)) </ins> } // Get Volumestatus metrics, err := csiClient.NodeGetVolumeStats(ctx, mc.volumeID, mc.targetPath)

What happened: kubelet crashed due to nil pointer dereference inside calcAndStoreStats function in Jul 03 06:50:42 kubelet[4440]: E0703 06:50:42. 4440 ] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference) Jul 03 06:50:42 kubelet[4440]: goroutine 3224 [running]: Jul 03 06:50:42 kubelet[4440]: (0x3992120, 0x773bdd0) Jul 03 06:50:42 kubelet[4440]: +0x7b Jul 03 06:50:42 kubelet[4440]: (0x0, 0x0, 0x0) Jul 03 06:50:42 kubelet[4440]: +0x82 Jul 03 06:50:42 kubelet[4440]: panic(0x3992120, 0x773bdd0) Jul 03 06:50:42 kubelet[4440]: +0x1b5 Jul 03 06:50:42 kubelet[4440]: (volumeStatCalculator).parsePodVolumeStats(0xc00178ef00, 0xc000c96cc0, 0xe, 0xc0014b8880, 0x0, 0xc000c96cc0, 0xe, 0x0, 0x0, 0x0, .) Jul 03 06:50:42 kubelet[4440]: +0x73 Jul 03 06:50:42 kubelet[4440]: (volumeStatCalculator).calcAndStoreStats(0xc00178ef00) Jul 03 06:50:42 kubelet[4440]: +0x753 Jul 03 06:50:42 kubelet[4440]: (volumeStatCalculator).StartOnce.func1.

1() Jul 03 06:50:42 kubelet[4440]: +0x2a Jul 03 06:50:42 kubelet[4440]: (0xc001c02d10) Jul 03 06:50:42 kubelet[4440]: +0x54 Jul 03 06:50:42 kubelet[4440]: (0xc001c02d10, 0xdf8475800, 0x3ff0000000000000, 0xc000f1dd01, 0xc0012c37a0) Jul 03 06:50:42 kubelet[4440]: +0xf8 Jul 03 06:50:42 kubelet[4440]: created by (volumeStatCalculator).StartOnce.func1 Jul 03 06:50:42 kubelet[4440]: +0x9d I think this happens because GetMetrics function can return nil for both metric(), and error, but calcAndStoreStats expects error to be not nil when metric is nil() What you expected to happen: Skip calculating stats for volumes that do not support volume stats. How to reproduce it (as minimally and precisely as possible): Create a volume using a csi driver that does not support volume stats. Anything else we need to know?: Environment: Kubernetes version (use ): Server version 15.0 Cloud provider or hardware configuration: OS (e.g: ): Ubuntu 16.04 LTS Kernel (e.g. ): 4.4.0-141-generic Install tools:kubeadm Network plugin and version (if this is a network-related bug): Others: Tested using synology-csi driver()
Sounds like you have a PR ready. Mind letting us know the PR number ? Thanks
I do not have a PR yet, but happy to make one. I'll add the PR number once I created it.
I created a PR and linked to this issue.
Didnt this solve this issue ?
No, it does not fix the problem that metric itself can be nil. Also, my pull request fixes the problem that csi_metrics returns nil value for error and the caller expects non nill value when csi driver does not support metrics.


Question: podInfoList = append(podInfoList, pInfo.(*framework.PodInfo)) } } <ins> queue.lock.Unlock() </ins> if !reflect.DeepEqual(test.expected, podInfoList) { t.Errorf("Unexpected PodInfo list. Expected: %v, got: %v",

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: pull-kubernetes-bazel-test Which test(s) are flaking: pkg/scheduler/internal/queue Testgrid link: Reason for failure: Anything else we need to know: /sig scheduling /assign
I just sent a pr for it according to the error message, but I'm not very sure if it will make things all right.
/assign


Question: ClaimSize            string ExpectedSize         string PvCheck              func(claim *v1.PersistentVolumeClaim) <del> VolumeMode           *v1.PersistentVolumeMode </del> <ins> VolumeMode           v1.PersistentVolumeMode </ins> AllowVolumeExpansion bool }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: I created a block EBS volume using dynamic provisioning. Then I resized the volume from 10GB to 20 GB. Both PV and block volume reflects the change, but the PVC's capacity is still 10GB. This could be caused by in-tree resize put the PVC condition under state, but no one is acting on the condition since it is block volume: /cc /sig storage What you expected to happen: PVC's capacity should be updated to 20GB How to reproduce it (as minimally and precisely as possible): Easy to reproduce using following specs: StorageClass: Claim: Pod: Anything else we need to know?: This will affect whether CSI migration should handle migration scenario for block volume resizing. Environment: Kubernetes version (use ):1.13 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: kops Network plugin and version (if this is a network-related bug): Others:
/sig aws
/assign


Question: fsGroupPolicyEnabled := utilfeature.DefaultFeatureGate.Enabled(features.ConfigurableFSGroupPolicy) <del> klog.Warningf("Setting volume ownership for %s and fsGroup set. If the volume has a lot of files then setting volume ownership could be slow, see https://github.com/kubernetes/kubernetes/issues/69699", mounter.GetPath()) </del> <ins> timer := time.AfterFunc(30*time.Second, func() { klog.Warningf("Setting volume ownership for %s and fsGroup set. If the volume has a lot of files then setting volume ownership could be slow, see https://github.com/kubernetes/kubernetes/issues/69699", mounter.GetPath()) }) defer timer.Stop() </ins> // This code exists for legacy purposes, so as old behaviour is entirely preserved when feature gate is disabled // TODO: remove this when ConfigurableFSGroupPolicy turns GA.

What happened: there are many kubelet warnings in the systemd logs I initialized the cluster with kubeadm init (no modifications) What you expected to happen: Not to have that many kubelet warnings in the system logs Anything else we need to know?: Environment: Kubernetes version (use kubectl version):Cloud provider or hardware configuration:OS (e.g: cat /etc/os-release):Kernel (e.g. uname -a):Install tools:
I have the same issue
should add a option or an annotationo to resolve this /sig node /area kubelet
Same issue kops, kubernetes 1.18.3, aws cloudprovider
Same issue on v1.17.4 installed with rancher
Same issue on v1.18.0 ~
Same issue on 1.17.6
Happens on v1.18.3 as well. This issue only occurs when container has some fsGroup set in its securityContext AND volume being mounted is either secret or a configmap. I guess they are being re-mounted every 70s or so. It's okay when the volume being mounted is a standard PV - log only appears once and doesn't get repeated.
I'm seeing the same issue. I'm willing to fix it.
I've digged into this a little bit further. I believe this is just a annoying warning message. AFAICS, it does not show any issue... I was reading the kubelet, its volume manager and the volume plugins source files. It seems that's the expected behavior. The desiredStateOfWorldPopulator sync loop always call the . The finds the volume plugin and check if the plugin requires mount. Secret plugin, wich is the plugin in discussion, always returns . Thus, the of the volume manager remounts the volume every time. I think the problem is in the which prints the warning message. It's more and I believe it does not take these plugins into account. The secret plugin is as it is now since . Is that make sense to you?
/assign


Question: func (m *azureDiskMounter) GetAttributes() volume.Attributes { readOnly := false volumeSource, err := getVolumeSource(m.spec) <del> if err != nil && volumeSource.ReadOnly != nil { </del> <ins> if err != nil { glog.Infof("azureDisk - mounter failed to get volume source for spec %s %v", m.spec.Name(), err) } else if volumeSource.ReadOnly != nil { </ins> readOnly = *volumeSource.ReadOnly } return volume.Attributes{

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: I deployed a spec using an AzureDisk that triggered an "invalid memory address or nil pointer dereference". The deployment was initializing a pod and mounting an AzureDisk volume after downloading the container. Here are the kubelet logs on the node during the time the error was thrown: The line that throw the exception was , which accesses when the likely returned nil (in this ). What you expected to happen: An error message related to failed disk mounting is returned instead of a nil pointer deference. How to reproduce it (as minimally and precisely as possible): Not very easy to reproduce, but we were in the process of moving pods between nodes running 1.5 -1.8. We suspect there may have been unexpected interactions that caused both and to be missing when the AzureDisk is fetched. Our pod spec uses AzureDisk volumes, not PersistentVolumes. Anything else we need to know?: I'd expect to not access if an error was propagated up. We worked around this issue by deleting the deployment and re-applying our spec, which did the trick, probably since the application of the config took place in an entirely 1.8 setting rather than a hybrid cluster (1.5 and 1.8 worker nodes). Environment: Kubernetes version (use ): kubelet 1.8.1 & master running 1.8.1 Cloud provider or hardware configuration: Azure OS (e.g. from /etc/os-release): CoreOS 1548.4.0 Kernel (e.g. ): Linux master 4.13.9-coreos SMP Thu Oct 26 03:35:38 UTC 2017 x86_64 Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40GHz GenuineIntel GNU/Linux Install tools: Others:
Hello Issue reported should be from while should be is nil, equals Maybe should we print this error into log ? Thanks very much !
Make sense.
/sig azure
thanks for the reporting, this issue is duplicate of which is fixed by and the fix is also in v1.8 now, while it's not in v1.8.1, line 48 in v1.8.1 is


Question: roundedVal: int64(1000), }, { <del> name:        "round overflowed quantity to int64", resource:    resource.MustParse("73786976299133170k"), </del> <ins> name:       "round decimal to GiB", resource:   resource.MustParse("1.2Gi"), roundedVal: int64(2), }, { name:       "round big value to GiB", resource:   resource.MustParse("8191Pi"), roundedVal: int64(8588886016), }, { name:        "round quantity to GiB that would lead to an int64 overflow", resource:    resource.MustParse("8192Pi"), </ins> roundedVal:  int64(0), expectError: true, },

After upgrading our clusters to Kubernetes 1.19, the provisioning of PersistentVolumes started failing with the following Events: The value matches the actual storage request of 3.2Gi. It doesn't seem special in any way, not too big, not too small, and it always worked until 1.19. We look into the issue with and we believe the PR introduced a change that broke the PersistentVolume provisioning even when it doesn’t overflow int64. Amounts of resource in Kubernetes are represented with the . In the PersistentVolume drivers, there are some functions that manipulate storage request in order to allocate a disk with the right size. Before the mentioned PR, those quantities were retrieved by calling which relies on to return the value as an int64. This worked probably fine in most of the cases, but it could happen that the represented value would overflow an int64, and in this case the result of would be wrong. The function’s description states that it’s the caller’s responsibility to check this: This is likely what the PR wanted to handle. The change replaced all calls with which has a different signature. From where it’s used now, you would think that the second return value signals an overflow: : This is probably where the error we’re seeing is generated. I believe that the signature of was misinterpreted. From my understanding of its description, returning doesn’t mean we’re overflowing . It only means that internally, the quantity is represented as a decimal amount: Internally, depending on their values, quantities are stored/represented in : for DecimalExponent (e.g. ) or DecimalSI (e.g. ) for BinarySI that don’t have a decimal part but a unit (simplified) (e.g. ) for all the rest With the PR, Kubernetes now fails to provision volumes that have a storage request represented as . Many of the storage requests are represented this way and still perfectly valid in our opinion. I would have expected the provisioning to continue to work as those requests seemed valid and were always accepted by our underlying cloud-provider in earlier version of Kubernetes.

Create a StatefulSet with a PersistentVolumeClaim that uses vSphere storage: Kubernetes version (use ): 1.19.7 Cloud provider or hardware configuration: vSphere, but I think it impacts other cloud-provider that were changed in the PR: /sig storage /kind bug /cc Possibly related to
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I think the only reason we do not use the quantity library for scaling and why we tried to roll our own rounding code is because when scaling stuff in AsScaledInt64(3) or ScaledValue(3), the library assumes decimal SI (M, 1000), not binary SI units (Mi, 1024): You are right, our self-rolled rounding code doesn't handle values with millibyte or higher precision like 3.2Gi. Our code uses AsInt64 and then performs its own scaling. If AsInt64 returns false, the int64 returned is not in bytes (it must be in millibytes or higher precision), so our rounding code won't work, so it returns that " is too great, overflows int64". In 1.18, we used Value() to round to bytes but we didn't check it against ScaleValue as the signature suggested. So the PR in question fixed cases where Value() overflowed but inadvertently broke your case. The correct way to round, which the resource library's rounding does, is to drop the useless precision and round up. , but as I said it doesn't seem to handle binary SI quantities correctly. For decimal SI quantities, we should rely on the resource library's AsScaledInt64 and be done with it. (We could also use Value() and ScaledValue as the signature suggests but calling AsScaledInt64 directly seems simpler). Like this: For binary SI quantities, we need to do something fancier involving CanonicalizeBytes I think, I'm not sure yet.
I guess for binary SI we can go back to using Value but actually check the overflow this time.

Hi Thanks for the quick feedback on the problem's description already. Indeed, the Scaled() functions don't handle BinarySI and that's why I quickly discarded the idea of using them. Since this is about storage, I suppose most people are using this format. I just finished wrapping up a PR that I hope address both the issue we are facing, but also what was meant to be fixed in the PR your reviewed back then. It doesn't make any difference between the 3 formats however.
Yeah I commented within seconds of your PR. I glanced at it , will review more soon, but it makes sense to me to just use VAlue and check ofr overflow for both decimal SI and binary SI rather than using Scaled() for one and not the other :+1:
I think it means both. i.e - when incoming value is decimal or if value is so big that type can't store it as it switches to decimal. So while my PR fixed later case, it broke the former. :/


Question: break } } <del> framework.ExpectEqual(locatedWardle, true, "Unable to find v1alpha1.wardle.example.com in APIServiceList") </del> <ins> if !locatedWardle { framework.Failf("Unable to find v1alpha1.wardle.example.com in APIServiceList") } </ins> // kubectl delete flunder test-flunder err = dynamicClient.Delete(context.TODO(), flunderName, metav1.DeleteOptions{})

The tests are full of assertions which compare a boolean against true or false, often without any additional explanation. When those assertions fail, the error message is useless for understanding what went wrong, often just saying: Including an explanation is better, but then we still have the distracting noise about the comparison: All of those assertions should be replace with . Such locations can be found by grepping: UPDATE (2024-08-26): in cases where such an if check is not possible (typically in combination with or ), should be used instead of . They emit only the explanation given to them, without including the noise above. The Ginkgo linter (enabled in golangci-lint) points that out. Some progress has been made in the meantime: v1.23.0: 250 v1.24.0-beta.0: 177 However, we were down to 175 at some point, so the number is going back up. Example: Some other places compare values. Those should be replaced with a comparison assertion. Example: It makes sense to split this work into individual PRs, one per OWNERS file under test/e2e. Here's a list of directories: [ ] test/e2e/apimachinery [x] test/e2e/apps: [x] test/e2e/auth [x] test/e2e/autoscaling: [x] test/e2e/cloud/gcp [x] test/e2e/cloud [x] test/e2e/common/network [x] test/e2e/common/node: [x] test/e2e/common/storage: [x] test/e2e/framework: [x] test/e2e/framework/providers/vsphere [x] test/e2e/framework/volume [x] test/e2e/instrumentation/logging [x] test/e2e/instrumentation: [x] test/e2e/kubectl [x] test/e2e/lifecycle [x] test/e2e/node: [x] test/e2e/scheduling: [x] test/e2e/storage: (prior PR: ) [x] test/e2e/testing-manifests/storage-csi [x] test/e2e/upgrades/apps: [x] test/e2e/windows: , When creating a PR, use as title. Use one commit per PR, squash when addressing feedback.

Unless noted otherwise, will work on PRs. Help may be welcome. /sig testing /help Better output for developers when tests fail.
This request has been marked as needing help from a contributor. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : ### What would you like to be ? Example: Some other places compare values. Those should be replaced with a comparison assertion. Example: /sig testing /help </details
/good-first-issue
This request has been marked as suitable for new contributors. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /good-first-issue </details
/triage accepted
I would like to take this up as my first issue
/assign
I would like to take up this issue as my first issue
Hello I am already working on this issue , can you please take some other issue ?
Okay sure ,how do I un assign myself from this issue?
I don't think you are assigned to this issue.
Oh okay
Hey as discussed in pr I am assuming that you are only working for tests under area.

also finished the work on , see
and only one PR is to be made for the correction under test/e2e/storage/, so how can we distribute the task and also make only single PR. Or can we make multiple PR's
I have a TODO list to the issue description. I'll try to keep that up-to-date with what has been done and who is working on it.
you can continue with in
what is to be changed in this case !
can you inform on which directory you are working.
Currently working on apimachinery
Nothing, just leave it as-is.
But that isn't under - before starting somewhere else, please coordinate with
test/e2e/instrumentation/logging don't have any cases like "Expect.Equal([^,], *(true|false)", so I think it should be removed from tasks to do.
I am thinking that the output should be if !ok { framework.Failf(uexec.ExitError) } in this case, am I thinking reasonable here.
isn't valid Go. The first parameter must be a format string. The code checks whether err is a : Therefore the change should be:
Why don't we just add extra explanation into calls like for easy investigation? I might miss some point.
You have to consider the complete output when that case fails. Gomega will write print the actual value and the expected value, then in addition you get the same information in the custom message. Printing just the custom failure message is shorter and more readable, in particular for a comparison of booleans. For a comparison of integers I see two solutions: ExpectEqual with a fixed message that just explains what is being compared. An if check with Failf where the message includes the actual value. The later seems more useful when the expected value is zero because that value doesn't need to be printed when its obvious.
I am getting difficulty in figuring out what should be output here, what should the appropriate output here. Also what steps you took in knowing so, it would help to improve my understanding.
The "ok" value is the result of checking for a specific key in a map. What a developer therefore wants to know when that check fails is: what was the key what's in the map Both can be included in:
can I work for the section :- test/e2e/scheduling ?

And there was only one changes needed for test/e2e/node which I did but my PR was closed as it only contains trivial edits. That was my first PR, so can you please guide me further?
/assign I'll try to coordinate the work, but won't create PRs myself. I ticked of line items which had no remaining work. feel free to start working on .
I replied there and also suggested an improvement. For those following this issue: try to include more information than before in the error message when the assertion fails. Then it is more obvious to the reviewer that the change is useful without having to read this issue. Examples:
Another tip: don't just reference this issue as explanation for the PR. Reviewers are busy, they won't necessarily invest the time and follow the link to understand what the PR is about. Instead, take the first sentence from this issue and adapt it as needed to the specific changes in the code.
can I start working on test/e2e/common/storage ?
Okay.
for test/e2e/network/netpol , there are already message associated with all of the cases, for example :- So should I just change all of them to ? And same for all the cases in test/e2e/network except these two cases : framework.ExpectEqual(netutils.I sIPv4String(internalIPs[0]) != netutils.IsIPv4String(internalIPs[1]), true) framework.ExpectEqual(netutils.IsIPv4String(p.Status.PodIPs[0].IP) != netutils.IsIPv4String(p.Status.PodIPs[1].IP), true )
Yes.
Okay then I will be working on and
It might make sense to combine them in a single PR, depending on how many changes there are - just use common sense.
IMO : this will never fail because whether err will be nil or it will exist. So, we don't need to change this code. What I had done in this case was :
Note that that is redundant (= not wrong, just not needed). I later clarified that in
/reopen
Reopened this issue. <detailsIn response to : /reopen </details
Hi! I've created a PR to address the tests in e2e/apps It is my first PR so hopefully, everything is ok. Thanks a lot!
Hi can I take test/e2e/common/node?
Yes, please do.

Have we considered / helpers that can skip the comparison output? - is noisy and makes it harder to skim tests, especially given that it's a pretty commonly required pattern.
No, we haven't. Do you find it noisy because it's two lines instead of one? I don't have a problem with that. It seems to be a common approach in unit tests and I find it easy to read, partly because it matches how production code would be written.
It becomes three, plus extra indention - but more importantly stands out in a series of expectations. This draws extra scrutiny to those lines. many unit tests using testify would use to express this because of that.
Isn't that a good thing? is bad without any or no useful extra explanation. The if check makes it obvious that a good failure message is needed. Does print less useless boilerplate code than Gomega? We could use that with Ginkgo (), but then we still need to ensure that the optional explanation is useful.
xref
Hi can I take test/e2e/auth?
for test/e2e/auth, for example : Should I just change it to ?
feel free to work on test/e2e/auth. This one particular example is a bit different: here we are comparing string values, and for those it makes sense to print the actual value because the failure message then adds information. I would leave that one as it is.
OK. test/e2e/auth doesn't need replacing. I : <img width="731" alt="Screen Shot 2022-05-30 at 9 12 59 PM" src="https://user-"There are a lot of positions that need to be modified in . I'll take test/e2e/apps.
That looks like a good candidate. Please go for it. When you need to make up new failure messages, try to think about what information may be useful to have and add that. Then the resulting PR is clearly better than the status quo.
Hi can you please assign me a directory. I want to give it a try. Thanks,
Hi I already posted a PR for e2e/apps some days ago in this issue: If it is fine I'd love to proceed with that. Just to avoid us repeating work.
sorry for not noticing the overlap sooner. Let's go with the older PR.

you can take over test/e2e/storage from There is an incomplete PR for it in You can check out that branch, fix it and submit a new PR, or start from scratch.
it looks like is not active anymore. I've remove him from the todo list and you instead for test/e2e/common/storage.
OK. Thank you. Is there an incomplete PR for .
I in and it returns 0. What should I do?
needs modifying. <img width="1440" alt="Screen Shot 2022-05-31 at 2 40 29 PM" src="https://user-"Can I work on it?
Indeed, did test/e2e/common/storage in You can work on test/e2e/windows.
OK. I'm working on it currently.
Sure, I will look into it. Thanks,
: do you want to continue with test/e2e/apimachinery?
Yes. There's an older PR:
Hi is there anything I could take over?
Hey I am not sure how updated the list is, but I am available to help. Is there any part I could start or continue?
sorry for the late reply. I had (and still have) some big changes for the E2E tests going and had put this on the backseat for a while. If you are interested, how about doing test/e2e/apimachinery and test/e2e/autoscaling? Some work was done on the latter, but there are still several calls (again?).
Sure, I can take that!
Hey I'd also like to contribute on this issue. Is there anything I could take over?
feel free to take my part:
Perfect, then I will take part
Hi :smile: , I would like to contribute to this issue, there is anything that I could do?
I was reluctant to start new work on this because I had one big PR pending that touched a lot of test code (PR ), but that has been merged. you are taking test/e2e/autoscaling, right? test/e2e/apimachinery? you can do test/e2e/apps. That leave test/e2e_node in case someone wants to continue.
yes e2e/autoscaling. I had some trouble to get the tests going initially, but seems fine now!
Hi, Can I do something to contribute to this issue?

you could work on test/e2e_node.
Ok, I will start work on test/e2e_node thanks!
Hi , I would like to contribute. Is there anything I could take over?
thanks for your interest, but right now everything seems to be covered.
Hi Thanks. I am searching for "good-first-issue", feel free to assign one to me. I am exploring api-machinery, but happy to contribute in other sig as well.
maybe you can take a look at if you have some time :)
Hi , There is anything that I could do to contribute with this issue?
Hi , I was looking and found the test/e2e/apimachinery without doing it and only then did I see that it was already assigned if this is a problem just refuse my pr and if you have something else to do let me know.
The remaining work will probably finished in , so there isn't anything else to do. If someone wants to investigate writing a linter for this, then this would be a cool science project - not easy, though. I would start with looking at for this. It has the advantage that it is supported indirectly in golangci-lint ().
Hello I would like to try to do a liter for it. It will be a cool project whit a lot of opportunities to learn. There will be a separate issue for the linter or we continue the discussion here?
Let's continue here.
I ended up creating a separate issue for linter ideas: let's continue there.
Ok!
New tests have been again with this anti-pattern since the last time that I checked. It's clear that without a linter, we'll not get rid of it. So let's focus on that first.
is this issue still open
Yes, this is still open. Current status as of v1.28.0-rc.0: Two calls are in , the rest are in . We have such a linter now, with ginkgolinter in golangci-lint and (soon) the ability to prevent usage of , so it makes sense to continue with removing those calls. do you want to prepare one PR for all remaining calls? I think one PR is enough, no need to split it up.
also has it, which was intentional. I'm undecided whether that should be replaced.

Several of those calls are being touched in , let's wait for that PR to be merged before doing any further work.
no longer finds anything. We might be done... :tada: But not quite. What we have left now is checking a boolean via or , which has the same problem. In some cases, there is an extra message (which is still noisy, see ), in other cases there isn't even that (which makes the failure message fairly useless). We can and should replace some of those calls with the pattern.
is almost resolved. My work-in-progress branch forbids the usage of in favor of the new or plain Go. In some cases, moving the Go code that returns true/false into a matcher might be the best solution because that allows dumping the full actual value that was being checked. Here are the remaining issues: <details</detailsDoes anybody want to work on those?
I would love to!
: please go ahead. Assign PRs to me when you create them. Let's start with a few of those calls initially, do a round of reviews, then continue. /assign
You will have to start with my branch, current master doesn't have gomega with thew new API yet.
I'm starting new to kubernetes. Is there anything easy that I could do ?
Hey Is there anything left I can help you with in this issue ?
Thanks for your interest. There's not much to be done at the moment, though.
Should this be closed?
Oh wow! Looks like we are indeed done. Thanks everyone. /close
Closing this issue. <detailsIn response to : Oh wow! Looks like we are indeed done. Thanks everyone. </details


Question: func TestAccept(t *testing.T) { _, s := framework.RunAMaster(t) <del> defer s.Close() </del> <ins> // TODO: Uncomment when fix #19254 // defer s.Close() </ins> resp, err := http.Get(s.URL + "/api/") if err != nil {

http://pr- is another failure.
Also seen in http://pr-
Again. Not assigned to anyone, so I am sending to for routing
Again: Is there anyone working on this? I will spend some time looking into this.
I think that the name of this issue is misleading - the issue seems to be unrelated to etcd_watcher Basically, if you look into this part of log: So it seems that, the WaitGroup that is problematic is in httptest.Server. So my feeling is that we are simply talking to a server that is already being close and this is ~roughly duplicate of We should just find where exactly the problem is and try to work-around it.
I think this is exactly what we had before. I will send a PR with the fix today.


Question: func (l *listener) Close() error { klog.V(5).Infof("forward listener for %s: closing", l.addr) l.cancel() <ins> l.mutex.Lock() defer l.mutex.Unlock() for _, c := range l.connections { if c != nil { c.stream.Close() } } </ins> return nil } func (l *listener) Accept() (net.Conn, error) { <del> select { case <-l.ctx.Done(): return nil, errors.New("listening was stopped") case c := <-l.connections: klog.V(5).Infof("forward listener for %s: got a new connection #%d", l.addr, c.counter) return c, nil </del> <ins> tryAccept := time.NewTicker(connectionPollInterval) defer tryAccept.Stop() for { select { case <-l.ctx.Done(): return nil, errors.New("listening was stopped") case <-tryAccept.C: conn := func() net.Conn { l.mutex.Lock() defer l.mutex.Unlock() for _, c := range l.connections { if c != nil && !c.accepted { klog.V(5).Infof("forward listener for %s: got a new connection #%d", l.addr, c.counter) c.accepted = true return c } } return nil }() if conn != nil { return conn, nil } } </ins> } } type connection struct { <del> stream  *stream addr    Addr counter int32 closed  *int32 mutex   sync.Mutex </del> <ins> l                *listener stream           *stream addr             Addr counter          int mutex            sync.Mutex accepted, closed bool </ins> } var _ net.Conn = &connection{}

ci-kubernetes-csi-canary-test-on-kubernetes-master, [It] [sig-storage] CSI mock volume CSI Volume Snapshots [Feature:VolumeSnapshotDataSource] volumesnapshotcontent and pvc in Bound state with deletion timestamp set should not get deleted while snapshot finalizer exists [It] [sig-storage] CSI mock volume CSI Volume Snapshots secrets [Feature:VolumeSnapshotDataSource] volume snapshot create/delete with secrets 25-08-2022? No response Looking at "volume snapshot create/delete with secrets" test, it finishes OK, but the "Cleaning up resources" freezes for ~30 minutes: After these 30 minutes, the test timeouts and kills API server and etcd a and everything fails miserably. Stack trace from "Cleaning up resources", it's stuck at shutting down gRPC server: /sig storage
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I reproduced this issue in somewhat hacked from external-provisioner (or any other kubernetes-csi repo). The script compiles Kubernetes master from a fresh checkout, runs it in KIND, installs hostpath CSI driver with canary images and runs e2e tests for the CSI driver like this: "hostpath CSI driver with canary images" is the interesting part - when I run the test with the latest tagged images, it succeeds, when using canaries it freezes at the cleanup. I found out that using the latest tagged images for everything except is enough to freeze the cleanup. And further I was able to check that node-driver-registrar at commit works, while master does not. There are just 2 commits in between: new release-tools and update to v1.25 libraries. I suspect it's the new libraries. I still don't understand how node-driver-registrar and its libraries could affect gRPC between the hostpath CSI driver and binary.
does this ring a bell? I though node-driver-registrar does not do anything with proxied "mock" CSI calls between and the CSI driver.

In addition, why only these two tests fail reliably, while the rest works?
For the record, this is diff in k/k that adds canary node-driver-registrar to This diff is enough to make the tests fail:
This does not ring a bell at all. You are right that csi-node-driver-registrar should not be involved in the gRPC communication between and the driver. Does it communicate with the driver after the initial registration? If yes, then that will lead into connections that need to be proxied, which might prevent the shutdown. Just a wild guess. But even then it's not clear how grpc.Server.Stop can get stuck. According to its documentation, "It immediately closes all open connections and listeners.". Hmm, could the proxy code get stuck when its listener is closed?
There is in that gets stuck, I did not check what is in that group.
Well, of course it fails :sweat_smile: There is no such canary image. Hmm, but then why do other tests pass? They all use .
There is some magic in your script that replaces with , the registrar runs and registers the driver in the end.
Yep:
One difference is that the driver-registrary canary no longer detects the loss of connection to the mock proxy. Previously it did: It could be that the proxy code relies on the remote side to close their side of the connection before considering everything shut down. That is necessary because the remote side might still send some data that needs to be received. If the remote side keeps the connection open, everything freezes. One can argue that it shouldn't block forever, but the bigger question is why "Lost connection" is not detected anymore with the more recent gRPC.
The "exit on connection loss" depends on our dialer being called again by gRPC:
/assign No idea yet, but I'll keep looking...
I can reproduce the same failure to shut down also in other tests. Must be a timing issue that it only occurred for the same two tests in the CI - not a very satisfying explanation, unfortunately. I have something that works for me locally: closing connections when shutting down the listener -see


Question: continue } killedContainers[containerID] = empty{} <ins> if podChanged { // Also kill associated pod infra container if the pod changed. if podInfraContainer, found, _ := dockerContainers.FindPodContainer(podFullName, uid, dockertools.PodInfraContainerName); found { if err := kl.killContainer(podInfraContainer); err != nil { glog.V(1).Infof("Failed to kill pod infra container %q: %v", podInfraContainer.ID, err) continue } } } </ins> } // Check RestartPolicy for container

cAdvisor API is not accessible from the node. It is accessible from inside the cadvisor container though. This is the reason for the cadvisor e2e test to fail and the recent heapster errors reported by users. The cadvisor version itself hasn't changed recently though.
cc
There were some changes recently around port forwarding ( that may be the cause.
From the node: From the cadvisor container:
reported the same issue today. Restarting the container seems to fix it. So, its something to do with the initial setup.
Could this be due to Docker 1.5? On Feb 23, 2015 6:12 PM, "Rohit Jnagal" wrote:
test machine is running Docker 1.4.1 It looks like cadvisor container is sharing its networking with the pause container as usual, but the pause container got restarted, so the networking setup isn't working anymore. Restarting cadvisor connects it to the newer instance of the pause container and it starts working again. However docker inspect shows that cadvisor is connected to network of a different pause container: And that container died soon after it started Kubelet killed the pause container. I think this is kubelet's doing. It tried to restart both pause and cadvisor container. It kills and restarts cadvisor container which connects to existing pause container. Then, it goes ahead and kills/restarts the pause container, leaving the new cadvisor container hanging. Maybe this is related to infra pods issue reported by earlier, or maybe just an ordering issue in kubelet.
Thanks for the detailed debugging :) with your report I think I found the issue. In we check if the container has changed and print the message above. After that we the POD container. We should be doing this before the container restart as you pointed out. I'm assigning this to myself and I will fix tomorrow. Will also rename the bug and re-purpose it.
cool, I was just staring at that code. All yours :+1:
Thanks for debugging the issue. This is a regression we introduced lately.
And when the test, we all assumed it was the test failing. Sorry, :)
I fell for that until I realized that the basic networking can also fail :)


Question: // Update implements Manager. func (f *managedFieldsUpdater) Update(liveObj, newObj runtime.Object, managed Managed, manager string) (runtime.Object, Managed, error) { self := "current-operation" <del> formerSet := managed.Fields()[manager] </del> object, managed, err := f.fieldManager.Update(liveObj, newObj, managed, self) if err != nil { return object, managed, err

I created an object with , and later I updated the same object with another . After that I looked at the object's managedFields, and saw nothing about the update. Here is what I got after the second apply: I expected managedFields to get updated by the second . No response <details</details<detailsnone </details<details</details<detailskubeadm </details<detailsdocker </details<details</details
/cc
I also saw the timestamp not updated when I updated using client-go and typed client stubs, Apply method, patch type .
/assign /triage accepted
/cc
The is not updated because you are not changing the structure of the object (adding or removing fields). According to , the time is updated if and only if the fieldSet is changed. It does not take into account the value of fields. One option would be to not compare the fieldSets but directly compare the liveObject & newObject for differences and update the field accordingly. Is it the desired behavior to update the time of when only the value is updated?
Yes! Look at the first paragraph of .
is that documentation correct? Is this what we intended?
I don't see any mention to the timestamp in that first paragraph. We've clarified recently the exact semantic of the timestamp, as defined here: I think you've identified the wrong code if you want to go ahead and fix it, go ahead, I'll happily review!


Question: # This is defined in root Makefile, but some build contexts do not refer to them KUBE_BASE_IMAGE_REGISTRY?=registry.k8s.io BASE_IMAGE_VERSION?=bookworm-v1.0.0 <del> BASEIMAGE?=${KUBE_BASE_IMAGE_REGISTRY}/build-image/debian-base-${ARCH}:${BASE_IMAGE_VERSION} # Keep debian releases (e.g. debian 11 == bullseye) consistent # between BASE_IMAGE_VERSION and DISTROLESS_IMAGE images DISTROLESS_IMAGE?=base-debian11 RUNNERIMAGE?=gcr.io/distroless/${DISTROLESS_IMAGE}:latest </del> <ins> RUNNERIMAGE?=${KUBE_BASE_IMAGE_REGISTRY}/build-image/debian-base-${ARCH}:${BASE_IMAGE_VERSION} </ins> TEMP_DIR:=$(shell mktemp -d -t conformance-XXXXXX)

N/A 07/18 02:38 PDT No response /sig release
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Slack thread:
/sig cli


Question: return false } <del> // checkIptabless to check expected iptables chain and rules </del> <ins> // checkIptables to check expected iptables chain and rules. The got rules must have same number and order as the // expected rules. </ins> func checkIptables(t *testing.T, ipt *iptablestest.FakeIPTables, epIpt netlinktest.ExpectedIptablesChain) { for epChain, epRules := range epIpt { rules := ipt.GetRules(epChain) <del> for _, epRule := range epRules { if !hasJump(rules, epRule.JumpChain, epRule.MatchSet) { t.Errorf("Didn't find jump from chain %v match set %v to %v", epChain, epRule.MatchSet, epRule.JumpChain) </del> <ins> if len(rules) != len(epRules) { t.Errorf("Expected %d iptables rule in chain %s, got %d", len(epRules), epChain, len(rules)) continue } for i, epRule := range epRules { rule := rules[i] if rule[iptablestest.Jump] != epRule.JumpChain || !strings.Contains(rule[iptablestest.MatchSet], epRule.MatchSet) { t.Errorf("Expected MatchSet=%s JumpChain=%s, got MatchSet=%s JumpChain=%s", epRule.MatchSet, epRule.JumpChain, rule[iptablestest.MatchSet], rule[iptablestest.Jump]) </ins> } } }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --ipvs mode didn't preserve client IP for NodePort Service with protocol SCTP and ExternalTrafficPolicy Local. iptables mode works fine. ipvs mode with TCP and UDP protocol works fine. Client IP for NodePort Service should be preseved when ExternalTrafficPolicy is Local regardless of the mode and the protocol. kube-proxy mode to ipvs a NodePort service with protocol SCTP and ExternalTrafficPolicy Local the NodePort service, the client IP was not preserved This is caused by wrong order of iptables rules that match kubeNodePortLocalSetSCTP and kubeNodePortSetSCTP: Kubernetes version (use ): v1.22.0 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig network /assign
/triage accepted


Question: err:              fmt.Errorf("A weird error"), }, } <del> done := make(chan struct{}) eventBroadcaster := NewBroadcaster() defer eventBroadcaster.StartRecordingToSink( &testEventSink{ </del> <ins> eventCorrelator := NewEventCorrelator(util.RealClock{}) randGen := rand.New(rand.NewSource(time.Now().UnixNano())) for caseName, ent := range table { attempts := 0 sink := &testEventSink{ </ins> OnCreate: func(event *api.Event) (*api.Event, error) { <del> if event.Message == "finished" { close(done) return event, nil } item, ok := table[event.Message] if !ok { t.Errorf("Unexpected event: %#v", event) return event, nil } item.attemptsMade++ if item.attemptsMade < item.timesToSendError { return nil, item.err </del> <ins> attempts++ if attempts < ent.timesToSendError { return nil, ent.err </ins> } return event, nil }, <del> }, ).Stop() clock := &util.FakeClock{time.Now()} recorder := recorderWithFakeClock(api.EventSource{Component: "eventTest"}, eventBroadcaster, clock) for caseName := range table { clock.Step(1 * time.Second) recorder.Event(ref, api.EventTypeNormal, "Reason", caseName) runtime.Gosched() } recorder.Event(ref, api.EventTypeNormal, "Reason", "finished") <-done for caseName, item := range table { if e, a := item.attemptsWanted, item.attemptsMade; e != a { t.Errorf("case %v: wanted %v, got %v attempts", caseName, e, a) </del> <ins> } ev := &api.Event{} recordToSink(sink, ev, eventCorrelator, randGen) if attempts != ent.attemptsWanted { t.Errorf("case %v: wanted %d, got %d attempts", caseName, ent.attemptsWanted, attempts) </ins> } } }

Have not reproduced locally even after a few tens of runs.
which version of Go are you using locally? This probably only repros on 1.5.1.
This was from Jenkins runs.
This is caused because an attempts counter is being read/written to without synchronization. I think this is caused by the event creation order not being guaranteed. I am looking a little deeper to track this down.
Got a fix with 130k+ runs and no failures. Will send out PR shortly.


Question: glog.Errorf("Could not create volume mounter for pod %s: %v", pod.UID, err) return nil, err } <del> if mounter == nil { return nil, errUnsupportedVolumeType } </del> // some volumes require attachment before mounter's setup. // The plugin can be nil, but non-nil errors are legitimate errors.

in there is a generic error variable defined that is used when a failure occurs and no other error is surfaced, it is a catch-all when something goes wrong. Think this should be reworded to something that is more reflective of the true issue and not so confusing for users. This error mostly comes when a builder can not be created and it is typically not true that the volume type is unsupported, but rather some other underlying issue, such as missing secrets or endpoints or etc... propose something like: or
are you working on this?
- yes, I would like to submit the PR for this

fyi When using PV/PVC for Persistent Volume volume mounts, if something goes wrong in the builder, a generic error is surfaced in the 'kubectl describe' with the following event error description *unsupported volume type": This only occurs when using the PV/PVC abstraction and not directly using a volume plugin. These type of errors are common when using volume plugins for mounts and would be useful for users to have an easy way to identify the errors rather than searching the logs for root cause. I think there are two issues: Why is the original error surfaced from the volume plugins (, , etc...) getting eaten when they return back to the I can see in the logs that 2 volume types are processed for any pod - the first is the actual underlying type (i.e. ) and this is where the actual real error is surfaced, then immediately after this the 2nd type is processed which is the general persistent claim type (i.e. ) and this when the first type error is eaten A simple work around would be to simply add a recorder event in the when there is a builder failure. This ensures the error is recorded and displayed as an event for the describer and will handle all like errors from other volume plugin types. full modifed function for reference from This will produce a better user experience when trying to see what went wrong - in this example, I am doing glusterfs PV/PVC but am missing end points which you can clearly see now in the events of the describe: rbd example: (Missing or Couldn't get secret)
I haven't had a chance to look at your proposal in detail yet, but I agree this is one of the most confusing things new users hit
another one is when gluster endpoints are missing, we get 'unsupported volume type' with your work extrapolating the permissions for NFS to automatically set supplemental groups, is there also a way to indicate more properly it's a permissions issue for it not mounting?
- the above discussion highlights the missing endpoints as the first example, it also discusses a similar rbd issue (missing secret file) but the point is, any of those type of volume plugin type of errors would be surfaced by adding the event in the
I like it! Is there a PR?


Question: buf.Reset() } } <ins> func TestPrintPersistentVolumeClaim(t *testing.T) { myScn := "my-scn" tests := []struct { pvc    api.PersistentVolumeClaim expect string }{ { // Test name, num of containers, restarts, container ready status api.PersistentVolumeClaim{ ObjectMeta: metav1.ObjectMeta{ Name: "test1", }, Spec: api.PersistentVolumeClaimSpec{ VolumeName: "my-volume", }, Status: api.PersistentVolumeClaimStatus{ Phase:       api.ClaimBound, AccessModes: []api.PersistentVolumeAccessMode{api.ReadOnlyMany}, Capacity: map[api.ResourceName]resource.Quantity{ api.ResourceStorage: resource.MustParse("4Gi"), }, }, }, "test1tBoundtmy-volumet4GitROXtt<unknown>n", }, { // Test name, num of containers, restarts, container ready status api.PersistentVolumeClaim{ ObjectMeta: metav1.ObjectMeta{ Name: "test2", }, Spec: api.PersistentVolumeClaimSpec{}, Status: api.PersistentVolumeClaimStatus{ Phase:       api.ClaimLost, AccessModes: []api.PersistentVolumeAccessMode{api.ReadOnlyMany}, Capacity: map[api.ResourceName]resource.Quantity{ api.ResourceStorage: resource.MustParse("4Gi"), }, }, }, "test2tLostttttt<unknown>n", }, { // Test name, num of containers, restarts, container ready status api.PersistentVolumeClaim{ ObjectMeta: metav1.ObjectMeta{ Name: "test3", }, Spec: api.PersistentVolumeClaimSpec{ VolumeName: "my-volume", }, Status: api.PersistentVolumeClaimStatus{ Phase:       api.ClaimPending, AccessModes: []api.PersistentVolumeAccessMode{api.ReadWriteMany}, Capacity: map[api.ResourceName]resource.Quantity{ api.ResourceStorage: resource.MustParse("10Gi"), }, }, }, "test3tPendingtmy-volumet10GitRWXtt<unknown>n", }, { // Test name, num of containers, restarts, container ready status api.PersistentVolumeClaim{ ObjectMeta: metav1.ObjectMeta{ Name: "test4", }, Spec: api.PersistentVolumeClaimSpec{ VolumeName:       "my-volume", StorageClassName: &myScn, }, Status: api.PersistentVolumeClaimStatus{ Phase:       api.ClaimPending, AccessModes: []api.PersistentVolumeAccessMode{api.ReadWriteOnce}, Capacity: map[api.ResourceName]resource.Quantity{ api.ResourceStorage: resource.MustParse("10Gi"), }, }, }, "test4tPendingtmy-volumet10GitRWOtmy-scnt<unknown>n", }, } buf := bytes.NewBuffer([]byte{}) for _, test := range tests { table, err := printers.NewTablePrinter().With(AddHandlers).PrintTable(&test.pvc, printers.PrintOptions{}) if err != nil { t.Fatal(err) } if err := printers.PrintTable(table, buf, printers.PrintOptions{NoHeaders: true}); err != nil { t.Fatal(err) } if buf.String() != test.expect { fmt.Println(buf.String()) fmt.Println(test.expect) t.Fatalf("Expected: %s, but got: %s", test.expect, buf.String()) } buf.Reset() } } </ins>

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: bug /kind bug What happened: returns the following: VOLUME label is missing. What you expected to happen: Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"8+", GitVersion:"v1.8.0-alpha.0.3928+-dirty", GitCommit:"", GitTreeState:"dirty", BuildDate:"2017-08-18T01:09:23Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"} /sig cli
/assign
GitHub didn't allow me to assign the following users: zjj2wry. Note that only can be assigned. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
thank for you report this


Question: specs := []*goopenapispec.Swagger{} for _, v := range crd.Spec.Versions { // Defaults are not pruned here, but before being served. <del> s, err := builder.BuildSwagger(crd, v.Name, builder.Options{V2: false, StripValueValidation: true, StripNullable: true, AllowNonStructural: true}) </del> <ins> s, err := builder.BuildSwagger(crd, v.Name, builder.Options{V2: false, StripValueValidation: true, StripNullable: true, AllowNonStructural: false}) </ins> if err != nil { return nil, err }

With in a CRD we get: This is because gnostic's OpenAPI proto types have no nullable support. Luckily CRDs keep working because in the upper error case we don't stop serving, just set the SSA schema to nil.
Thanks for reporting!
I think it should be "This is because gnostic's OpenAPI v2 proto types have no nullable support.". The problem is that we're not supposed to pass-in v3.
Source:
OK, We'll write a test and fix this, thanks Stefan!
/reopen This still happens in slightly different shape: This comes from :
That's a fairly different bug but yes, thanks!
Is this related to the stating that use of "anyOf" wouldn't be allowed after 1.16? We're seeing issue on OpenFaaS after going to 1.17.5 and 1.18. Usually upgrading K8s versions hasn't caused breaking changes and we're trying to figure out where to start with this one.
No. That post was about structural schemas. v1 CRDs are limited to structural schemas. v1beta1 CRDs can continue to create non-structural schemas until v1beta1 is no longer served (in 1.22) Also, is not completely disallowed, specific restrictions inside an clause (like ) are what are not allowed ("logical constraints using oneOf, allOf, anyOf, not are allowed").
extracted the newly found issue to to track separately with isolated reproducer /close
Closing this issue. <detailsIn response to : extracted the newly found issue to to track separately Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: These guides are maintained by CoreOS and deploy Kubernetes the "CoreOS Way" with full TLS, the DNS add-on, and more. These guides pass Kubernetes conformance testing and we encourage you to [test this yourself](https://coreos.com/kubernetes/docs/latest/conformance-tests.html). <ins> [**AWS Multi-Node**](https://coreos.com/kubernetes/docs/latest/kubernetes-on-aws.html) Guide and CLI tool for setting up a multi-node cluster on AWS. CloudFormation is used to set up a master and multiple workers in auto-scaling groups. <hr/> </ins> [**Vagrant Multi-Node**](https://coreos.com/kubernetes/docs/latest/kubernetes-on-vagrant.html) Guide to setting up a multi-node cluster on Vagrant. The deployer can independently configure the number of etcd nodes, master nodes, and worker nodes to bring up a fully HA control plane.

Our docs say this: I think though that example is out of date (?). We should make sure to point it to the right place.
cc
where should this point?
The official CoreOS guides are documented/linked here: The decision made a few months ago was to leave the previous community content, but should we revisit that?
Side note, a link to our AWS tool/guide probably needs to be there as well
What file contains that sentence?


Question: : ${NUM_MINIONS:=5} ;; <ins> # kubernetes-upgrade-gce-1.0-1.1 # # This suite: # # 1. launches a cluster at release/latest-1.0, # 2. upgrades the master to ci/latest-1.1 # 3. runs release/latest-1.0 e2es, # 4. upgrades the rest of the cluster, # 5. runs release/latest-1.0 e2es again, then # 6. runs ci/latest-1.1 e2es and tears down the cluster. kubernetes-upgrade-1.0-1.1-gce-step1-deploy) : ${E2E_CLUSTER_NAME:="gce-upgrade-1-0"} : ${E2E_NETWORK:="gce-upgrade-1-0"} : ${JENKINS_PUBLISHED_VERSION:="release/latest-1.0"} : ${PROJECT:="k8s-jkns-gce-upgrade"} : ${E2E_UP:="true"} : ${E2E_TEST:="false"} : ${E2E_DOWN:="false"} : ${KUBE_GCE_INSTANCE_PREFIX:="e2e-upgrade-1-0"} : ${ENABLE_EXPERIMENTAL_API:=true} : ${NUM_MINIONS:=5} ;; kubernetes-upgrade-1.0-1.1-gce-step2-upgrade-master) : ${E2E_CLUSTER_NAME:="gce-upgrade-1-0"} : ${E2E_NETWORK:="gce-upgrade-1-0"} : ${E2E_OPT:="--check_version_skew=false"} # We have to get tars at ci/latest (JENKINS_PUBLISHED_VERSION default) to # get the latest upgrade logic. : ${JENKINS_FORCE_GET_TARS:=y} : ${PROJECT:="k8s-jkns-gce-upgrade"} : ${E2E_UP:="false"} : ${E2E_TEST:="true"} : ${E2E_DOWN:="false"} : ${GINKGO_TEST_ARGS:="--ginkgo.focus=Skipped.*Clustersupgrade.*upgrade-master --upgrade-target=ci/latest-1.1"} : ${KUBE_GCE_INSTANCE_PREFIX:="e2e-upgrade-1-0"} : ${ENABLE_EXPERIMENTAL_API:=true} : ${NUM_MINIONS:=5} ;; kubernetes-upgrade-1.0-1.1-gce-step3-e2e-old) : ${E2E_CLUSTER_NAME:="gce-upgrade-1-0"} : ${E2E_NETWORK:="gce-upgrade-1-0"} : ${E2E_OPT:="--check_version_skew=false"} : ${JENKINS_FORCE_GET_TARS:=y} # Run old e2es : ${JENKINS_PUBLISHED_VERSION:="release/latest-1.0"} : ${PROJECT:="k8s-jkns-gce-upgrade"} : ${E2E_UP:="false"} : ${E2E_TEST:="true"} : ${E2E_DOWN:="false"} : ${GINKGO_TEST_ARGS:="--ginkgo.skip=$(join_regex_allow_empty  ${GCE_DEFAULT_SKIP_TESTS[@]:+${GCE_DEFAULT_SKIP_TESTS[@]}}  ${GCE_FLAKY_TESTS[@]:+${GCE_FLAKY_TESTS[@]}}  )"} : ${KUBE_GCE_INSTANCE_PREFIX:="e2e-upgrade-1-0"} : ${ENABLE_EXPERIMENTAL_API:=true} : ${NUM_MINIONS:=5} ;; kubernetes-upgrade-1.0-1.1-gce-step4-upgrade-cluster) : ${E2E_CLUSTER_NAME:="gce-upgrade-1-0"} : ${E2E_NETWORK:="gce-upgrade-1-0"} : ${E2E_OPT:="--check_version_skew=false"} # We have to get tars at ci/latest (JENKINS_PUBLISHED_VERSION default) to # get the latest upgrade logic. : ${JENKINS_FORCE_GET_TARS:=y} : ${PROJECT:="k8s-jkns-gce-upgrade"} : ${E2E_UP:="false"} : ${E2E_TEST:="true"} : ${E2E_DOWN:="false"} : ${GINKGO_TEST_ARGS:="--ginkgo.focus=Skipped.*Clustersupgrade.*upgrade-cluster --upgrade-target=ci/latest-1.1"} : ${KUBE_GCE_INSTANCE_PREFIX:="e2e-upgrade-1-0"} : ${ENABLE_EXPERIMENTAL_API:=true} : ${NUM_MINIONS:=5} ;; kubernetes-upgrade-1.0-1.1-gce-step5-e2e-old) : ${E2E_CLUSTER_NAME:="gce-upgrade-1-0"} : ${E2E_NETWORK:="gce-upgrade-1-0"} : ${E2E_OPT:="--check_version_skew=false"} : ${JENKINS_FORCE_GET_TARS:=y} # Run old e2es : ${JENKINS_PUBLISHED_VERSION:="release/latest-1.0"} : ${PROJECT:="k8s-jkns-gce-upgrade"} : ${E2E_UP:="false"} : ${E2E_TEST:="true"} : ${E2E_DOWN:="false"} : ${GINKGO_TEST_ARGS:="--ginkgo.skip=$(join_regex_allow_empty  ${GCE_DEFAULT_SKIP_TESTS[@]:+${GCE_DEFAULT_SKIP_TESTS[@]}}  ${GCE_FLAKY_TESTS[@]:+${GCE_FLAKY_TESTS[@]}}  )"} : ${KUBE_GCE_INSTANCE_PREFIX:="e2e-upgrade-1-0"} : ${ENABLE_EXPERIMENTAL_API:=true} : ${NUM_MINIONS:=5} ;; kubernetes-upgrade-1.0-1.1-gce-step6-e2e-new) : ${E2E_CLUSTER_NAME:="gce-upgrade-1-0"} : ${E2E_NETWORK:="gce-upgrade-1-0"} : ${JENKINS_FORCE_GET_TARS:=y} : ${JENKINS_PUBLISHED_VERSION:="ci/latest-1.1"} : ${PROJECT:="k8s-jkns-gce-upgrade"} : ${E2E_UP:="false"} : ${E2E_TEST:="true"} : ${E2E_DOWN:="true"} : ${GINKGO_TEST_ARGS:="--ginkgo.skip=$(join_regex_allow_empty  ${GCE_DEFAULT_SKIP_TESTS[@]:+${GCE_DEFAULT_SKIP_TESTS[@]}}  ${GCE_FLAKY_TESTS[@]:+${GCE_FLAKY_TESTS[@]}}  ${GCE_SLOW_TESTS[@]:+${GCE_SLOW_TESTS[@]}}  )"} : ${KUBE_GCE_INSTANCE_PREFIX:="e2e-upgrade-1-0"} : ${ENABLE_EXPERIMENTAL_API:=true} : ${NUM_MINIONS:=5} ;; </ins> # Run Kubemark test on a fake 100 node cluster to have a comparison # to the real results from scalability suite kubernetes-kubemark-gce)

Which jobs are failing: Since when has it been failing: Testgrid link: Reason for failure: Anything else we need to know: /milestone v1.18 /priority critical-urgent /cc /cc /sig release
-- Yep, this is the one of the ones I was alluding to in the RT meeting. It's failing because of a restructure of the rpm builder: Will try to work on this later in the week. Worst case, early next week. In the meantime, you can treat as not critical. Slack ref: /assign /remove-priority critical-urgent /priority important-soon /area release-eng
Those labels are not set on the issue: <detailsIn response to : -- Yep, this is the one of the ones I was alluding to in the RT meeting. In the meantime, you can treat as not critical. /remove-priority critical-urgent /priority important-soon /area release-eng Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This should be fixed once merges.


Question: resourceName:     "domain1.com/resource1", resourceQuantity: *resource.NewQuantity(int64(2), resource.DecimalSI), devs:             checkpoint.DevicesPerNUMA{0: []string{"dev1", "dev2"}}, <ins> topology:         false, </ins> } as := require.New(t) podsStub := activePodsStub{

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: Which test(s) are failing: Since when has it been failing: Testgrid link: Reason for failure: Anything else we need to know: Can't find this in Triage Example spyglass links: /sig scheduling /cc /priority important-soon
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I don't quite think this test should be owned by SIG-Scheduling. Most of the failures happen in the Nvidia GPU setup phase. BTW: although irrelevant, last time we see a failure of this test is .
Searching keyword "UnexpectedAdmissionError" gets the following occurrences:
/sig node cc in case you're aware of any recent changes that may have caused this failure.
From the I can see a bunch of: Checking the kubelet log, I can see the gpu capacity has been changing And before the capacity changed to 0, we can see the kubelet was attempting to allocate 1 GPU to the test pod, but some NPE was thrown, and kubelet stopped/restarted.
The nil pointer is from here: , which is a new change introduced by
The current device plugin (the one defined ) is using an old version of device plugin api (see ), which does not have field under . So calling will end up with NPE.
Thanks , the analysis makes great sense to me. Could you please work on a fix to address this? (Probably updating )
Hi , I am still looking at the code, because the code does have the logic to handle the case where device plugin does not return affinity: Also, even if the Topology is not defined, I am not sure will throw NPE () (please correct me if I am wrong here). I will need to look more closely at the code to find out where the NPE is from. Updating the might take long time because we will need some testing to make sure it doesn't break anything.

Currently I am thinking of identifying the source of NPE, and make sure can handle that case.
/cc
I submitted a fix. My fault, it was a real case when pointer to Topology could be a nil. But still fails: I don't observe SIGSEGV now in , but 3 problems there: 1113 09:26:19. 1336 ] failed to run Kubelet: cannot create certificate signing request: is forbidden: User "kubelet" cannot create resource "certificatesigningrequests" in API group "" at the cluster scope of nvidia-driver-installer stops and restarts after a while. 's () NoopManager is using
can you show full error msg of "NVIDIA GPU metrics will not be available: %s"?
I1113 15:43:26. 1338 ] Found device with vendorID "0x10de" W1113 15:43:26. 1338 ] NVIDIA GPU metrics will not be available: Could not initialize NVML: could not load NVML library I1113 15:43:26. 1338 ] Machine: {Timestamp:2020-11-13 15:43:26. +0000 UTC m=+5. NumCores:2 NumPhysicalCores:1 NumSockets:1 CpuFrequency: MemoryCapacity: MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:2048 NumPages:0}] MachineID: SystemUUID:-9824-3a64-f41f- BootID:-f891-4c0b-a34d- Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:39 Capacity: Type:vfs Inodes: HasInodes:true} {Device:/dev/sda8 DeviceMajor:8 DeviceMinor:8 Capacity: Type:vfs Inodes:4096 HasInodes:true} {Device:/dev/sda1 DeviceMajor:8 DeviceMinor:1 Capacity: Type:vfs Inodes: HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:19 Capacity: Type:vfs Inodes: HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:21 Capacity: Type:vfs Inodes: HasInodes:true}

{Device:/sys/fs/cgroup DeviceMajor:0 DeviceMinor:22 Capacity: Type:vfs Inodes: HasInodes:true} {Device:/etc/machine-id DeviceMajor:0 DeviceMinor:21 Capacity: Type:vfs Inodes: HasInodes:true} {Device:/mnt/disks DeviceMajor:0 DeviceMinor:38 Capacity: Type:vfs Inodes: HasInodes:true} {Device:overlayfs0-43 DeviceMajor:0 DeviceMinor:43 Capacity: Type:vfs Inodes: HasInodes:true} {Device:/var/lib/cloud DeviceMajor:0 DeviceMinor:45 Capacity: Type:vfs Inodes: HasInodes:true} {Device:/dev/root DeviceMajor:253 DeviceMinor:0 Capacity: Type:vfs Inodes: HasInodes:true}] DiskMap:map[253:0:{Name:dm-0 Major:253 Minor:0 Size: Scheduler:none} 8:0:{Name:sda Major:8 Minor:0 Size: Scheduler:mq-deadline} 9:0:{Name:md0 Major:9 Minor:0 Size:0 Scheduler:none}] NetworkDevices:[{Name:eth0 MacAddress:42:01:0a:28:00:03 Speed:-1 Mtu:1460}] Topology:[{Id:0 Memory: HugePages:[{PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 1] Caches:[{Size: Type:Data Level:1} {Size: Type:Instruction Level:1} {Size: Type:Unified Level:2}] SocketID:0}] Caches:[{Size: Type:Unified Level:3}]}] CloudProvider:GCE InstanceType:n1-standard-2 InstanceID:} I1113 15:43:26. 1338 ] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. I1113 15:43:26. 1338 ] Version: {KernelVersion:5.4.49+ ContainerOsVersion:Container-Optimized OS from Google DockerVersion:Unknown DockerAPIVersion:Unknown CadvisorVersion: CadvisorRevision:} I guess kubelet can't load in cAdvisor the NVML library.
I have looked into this issue actually.

This is caused by a recent change on cadvisor: We used to return a real nvidiaManager but now only a Noop one if the NVML lib is not there. And currently the NVML lib is loaded by daemonset, which will not there when the kubelet starts. This PR should fix it: But btw, this bug will not fail the test because it will only cause missing GPU usage metrics. The plan is to cherry-pick the fix above to cadvisor 0.37, and then update the vendor for kubernetes


Question: } } <ins> // TestPodFailedSchedulingMultipleTimesDoesNotBlockNewerPod tests // that a pod determined as unschedulable multiple times doesn't block any newer pod. // This behavior ensures that an unschedulable pod does not block head of the queue when there // are frequent events that move pods to the active queue. func TestPodFailedSchedulingMultipleTimesDoesNotBlockNewerPod(t *testing.T) { q := NewPriorityQueue(nil) // Add an unschedulable pod to a priority queue. // This makes a situation that the pod was tried to schedule // and had been determined unschedulable so far. unschedulablePod := v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name:      "test-pod-unscheduled", Namespace: "ns1", UID:       "tp001", }, Spec: v1.PodSpec{ Priority: &highPriority, }, Status: v1.PodStatus{ NominatedNodeName: "node1", }, } // Update pod condition to unschedulable. podutil.UpdatePodCondition(&unschedulablePod.Status, &v1.PodCondition{ Type:          v1.PodScheduled, Status:        v1.ConditionFalse, Reason:        v1.PodReasonUnschedulable, Message:       "fake scheduling failure", LastProbeTime: metav1.Now(), }) // Put in the unschedulable queue q.AddUnschedulableIfNotPresent(&unschedulablePod) // Clear its backoff to simulate backoff its expiration q.clearPodBackoff(&unschedulablePod) // Move all unschedulable pods to the active queue. q.MoveAllToActiveQueue() // Simulate a pod being popped by the scheduler, // At this time, unschedulable pod should be popped. p1, err := q.Pop() if err != nil { t.Errorf("Error while popping the head of the queue: %v", err) } if p1 != &unschedulablePod { t.Errorf("Expected that test-pod-unscheduled was popped, got %v", p1.Name) } // Assume newer pod was added just after unschedulable pod // being popped and before being pushed back to the queue. newerPod := v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name:              "test-newer-pod", Namespace:         "ns1", UID:               "tp002", CreationTimestamp: metav1.Now(), }, Spec: v1.PodSpec{ Priority: &highPriority, }, Status: v1.PodStatus{ NominatedNodeName: "node1", }, } q.Add(&newerPod) // And then unschedulablePod was determined as unschedulable AGAIN. podutil.UpdatePodCondition(&unschedulablePod.Status, &v1.PodCondition{ Type:          v1.PodScheduled, Status:        v1.ConditionFalse, Reason:        v1.PodReasonUnschedulable, Message:       "fake scheduling failure", LastProbeTime: metav1.Now(), }) // And then, put unschedulable pod to the unschedulable queue q.AddUnschedulableIfNotPresent(&unschedulablePod) // Clear its backoff to simulate its backoff expiration q.clearPodBackoff(&unschedulablePod) // Move all unschedulable pods to the active queue. q.MoveAllToActiveQueue() // At this time, newerPod should be popped // because it is the oldest tried pod. p2, err2 := q.Pop() if err2 != nil { t.Errorf("Error while popping the head of the queue: %v", err2) } if p2 != &newerPod { t.Errorf("Expected that test-newer-pod was popped, got %v", p2.Name) } } </ins> // TestHighPriorityBackoff tests that a high priority pod does not block // other pods if it is unschedulable func TestHighProirotyBackoff(t *testing.T) {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: In large clusters, events that trigger rescheduling of the unschedulable pods are frequent. For example, in a cluster with 1000 nodes, node updates, which are often just heartbeats, may happen at an average rate of 100/s. These events trigger rescheduling of unschedulable pods. The scheduler moves unschedulable pods to its active scheduling queue when such events are observed. There are two particularly problematic scenarios: are high-priority unschedulable pods and preemption does not help make them schedulable. For example, they may have node affinity that cannot be satisfied, or they may have pod affinity to other pods that do not exist. Such high priority pods may go to the head of the active scheduling queue and block everything behind them. are many pending pods with the same priority, but a number of them are unschedulable. When these unschedulable pods are moved to the active queue, they are expected to go behind other pods in the queue, but since our queue sorts pods by priority and these pods have the same priority, their order is unspecified. The Pop() algorithm of the heap swaps the head of the queue with the tail, effectively causing the heap to act similar to a LIFO. So, pods which are tried recently, go to the head with a high probability and are retried. This starves pods behind them in the queue. Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug /sig scheduling /assign
For case 1: How about add an field to record how many times scheduler have tried to schedule it? The bigger it is, the latter we should schedule it. Then we could consider to add it as a factor for sorting pods.
Yes, we basically need a . Another approach is to . We will pursue both of these approaches.
Well differentiating optionsounds better
Agree. On the one hand we need to reduce the frequency of unnecessary rescheduling, on the other hand we need to optimize the sorting algorithm.

And the would help to reduce the frequency. But I think "differentiate between node heartbeats vs. actual node updates" is a better approach.
Yes its a better approach indeed but what if we use the recursion there
We should have all three changes to address all known scenarios efficiently. The three changes are: back off mechanism fair sorting of pods in the scheduling queue smarter processing of node updates Node updates are not the only trigger for the scheduler to retry unschedulable pods. The scheduler also retries when running pods terminate, among a couple of other events. So, without having backoff mechanism a high priority unschedulable pod could still block head of the queue in clusters with high churn, for example, in clusters where many pods terminate frequently. Back off mechanism by itself addresses the issue in all scenarios, but the other two changes make the scheduling queue smarter and more efficient.
The three prs above are quite comprehensive : ).
All of the above items are done already.


Question: // This object has an invalid selector, it does not match the pod continue } <del> // If a PDB with a nil or empty selector creeps in, it should match nothing, not everything. if selector.Empty() || !selector.Matches(labels.Set(pod.Labels)) { </del> <ins> if !selector.Matches(labels.Set(pod.Labels)) { </ins> continue }

with an empty selector selects all the pods as expected. But, during the /, the pdb is not being honored correctly i.e. all the pods are getting evicted. This issue is only observed with the empty selector and works as expected if a valid label selector is provided that selects all the pods in a namespace. When an empty selector is provided in the 's spec, the created needs to be honored by not evicting all the pods during the api call Create a simple busybox deployment with 3 replicas on a local cluster (or any cluster)Create a pdb object with an selectorCheck the status of the created pdb object that has the observed to be 3 (which is expected)Call the api to see the pdb not being honored and all the pods being evictedDelete the above pdb object and create a new one with the label selectorObserve the expected status of the pdb object same as the old one with the set to 3Call the api to observe the pdb being honored as expected No response <details</details<detailslocal cluster - N/A </details<details</details<details</details<detailsCRI-O </details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig apps
/assign


Question: {% endif -%} {% set kubelet_auth = "--anonymous-auth=false --authorization-mode=Webhook --client-ca-file=" + pillar.get('ca_cert_bundle_path', '/var/lib/kubelet/ca.crt') %} <ins> {% set pki=" --cert-dir=/var/lib/kubelet/pki" -%} </ins> # test_args has to be kept at the end, so they'll overwrite any prior configuration <del> DAEMON_ARGS="{{daemon_args}} {{api_servers}} {{debugging_handlers}} {{hostname_override}} {{cloud_provider}} {{cloud_config}} {{config}} {{manifest_url}} --allow-privileged={{pillar['allow_privileged']}} {{log_level}} {{cluster_dns}} {{cluster_domain}} {{docker_root}} {{kubelet_root}}  {{non_masquerade_cidr}} {{cgroup_root}} {{system_container}} {{pod_cidr}} {{ master_kubelet_args }} {{cpu_cfs_quota}} {{network_plugin}} {{kubelet_port}} {{ hairpin_mode }} {{enable_custom_metrics}} {{runtime_container}} {{kubelet_container}} {{node_labels}} {{eviction_hard}} {{kubelet_auth}} {{feature_gates}} {{test_args}}" </del> <ins> DAEMON_ARGS="{{daemon_args}} {{api_servers}} {{debugging_handlers}} {{hostname_override}} {{cloud_provider}} {{cloud_config}} {{config}} {{manifest_url}} --allow-privileged={{pillar['allow_privileged']}} {{log_level}} {{cluster_dns}} {{cluster_domain}} {{docker_root}} {{kubelet_root}}  {{non_masquerade_cidr}} {{cgroup_root}} {{system_container}} {{pod_cidr}} {{ master_kubelet_args }} {{cpu_cfs_quota}} {{network_plugin}} {{kubelet_port}} {{ hairpin_mode }} {{enable_custom_metrics}} {{runtime_container}} {{kubelet_container}} {{node_labels}} {{eviction_hard}} {{kubelet_auth}} {{pki}} {{feature_gates}} {{test_args}}" </ins>

https://k8s- Judging by the timestamp, it's this batch of merge caused the failure: NONE
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
looks like it's your pr causing the failure ^^^
cc
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
It's still failing, See build log: https://k8s-, kubelet log: error in kubelet log:


Question: name: "simple scale up stabilization", key:  "", recommendations: []timestampedRecommendation{ <del> {4, time.Now().Add(-2 * time.Minute)}, {5, time.Now().Add(-1 * time.Minute)}}, </del> <ins> {4, now.Add(-2 * time.Minute)}, {5, now.Add(-1 * time.Minute)}}, </ins> currentReplicas:              1, prenormalizedDesiredReplicas: 7, expectedStabilizedReplicas:   4, expectedRecommendations: []timestampedRecommendation{ <del> {4, time.Now()}, {5, time.Now()}, {7, time.Now()}, </del> <ins> {4, now}, {5, now}, {7, now}, </ins> }, scaleUpStabilizationWindowSeconds: 60 * 5, },

When configuring new behaviors section in HPA I two scenarios that I think are incorrect. First one: stabilizationWindowSeconds set up to 5 minutes for both scale up and scale down min replica count 2 Application consumes big amount of resources during warm-up 0 minute: current replica count: 2 1 minute: desired replica count: 3 - no scale up stabilization window worked 2 minute: desired replica count: 1 (!) - scale up to 3 (max number from previous recommendations during stabilization window period) ! Second one Stabilization window for scale up 0 Stabilization window for scale down 5 minutes min replica count 2 behavior set to scale up new pod in 2 minutes scaleUp: policies: type: Pods value: 1 periodSeconds: 120 What happens: 0 minute: 2 replicas 1 minute : performance spike scale up to 3 (desired value 4) 2 minute : no spike no scale up 3 minute: no spike scale up to 4(!) - supposedly desired value 1 and supposedly worked this way because of scale down window set to 5 minutes picking max value from that 5 minutes window that was 4 replicas Expected behaviors 1 Scenario no scale up for 5 minutes 2 Scenario correct measurement and no scale up when there is no need for it Environment: Kubernetes version (use ): client 1.19.3 Server 1.19.0 Cloud provider or hardware configuration: Azure aks
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig autoscaling
/assign josephburnett
The HPA controller keeps a flat history of recommendations for stabilization. ! When downscale stabilizing the maximum recommendation is selected. Purple is pre-stabilized recommendation. Green is final recommendation. Blue is max recommendation within window. ! When upscale stabilization the minimum recommendation is selected. ! However when both up and down scale stabilization is configured, the interpretation of the history changes depending on the direction of movement.

Red is the undesired, actuated recommendation. ! What we want is to keep the stabilized recommendation within the envelope of the minimum and maximum over configured stabilization windows. We should only move when the envelope forces a move. ! We first stabilize up and down, and check if the current value fits within it. If it does (or the range is "inverted"), we do nothing. If it does not, we emit the "nearest possible" value, which will be lower or upper bound, depending on where the current size is in relation to the range. (thanks and for the paragraph above)