Question: assert.Equal(t, 1, len(filteredDisks)) assert.Equal(t, newDiskName, *filteredDisks[0].Name) } <ins> func TestIsInstanceNotFoundError(t *testing.T) { testCases := []struct { errMsg         string expectedResult bool }{ { errMsg:         "", expectedResult: false, }, { errMsg:         "other error", expectedResult: false, }, { errMsg:         "not an active Virtual Machine scale set vm", expectedResult: false, }, { errMsg:         `compute.VirtualMachineScaleSetVMsClient#Update: Failure sending request: StatusCode=400 -- Original Error: Code="InvalidParameter" Message="The provided instanceId 1181 is not an active Virtual Machine Scale Set VM instanceId." Target="instanceIds"`, expectedResult: true, }, } for i, test := range testCases { result := isInstanceNotFoundError(fmt.Errorf(test.errMsg)) assert.Equal(t, test.expectedResult, result, "TestCase[%d]", i, result) } } </ins>

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Same symtoms as The detach is not successful until controller-manager is restarted controller-manager logs: What you expected to happen: The detach operation should know that a node is not active or its been deleted and should assume the disk is detached. How to reproduce it (as minimally and precisely as possible): This seems quite hard to reproduce, but we do run into it quite a few times where the pod is stuck in , because the controller-manager is unable to detach disks from deleted nodes. Anything else we need to know?: Environment: Kubernetes version (use ): v1.15.11 Cloud provider or hardware configuration: azure (self-managed) OS (e.g: ): Ubuntu 18.04.3 LTS Kernel (e.g. ): 5.0.0-1027-azure Install tools: kubeadm Network plugin and version (if this is a network-related bug): Others: /sig cloud-provider /area provider/azure paging
/sig azure
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig azure Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cloud-provider
/area provider/azure
hi have you done sth manually on the node? if node is deleted finally, the disk would be in unattached state finally, then disk attach to another node would succeed finally.
as discussed offline, the issue is probably due to vmss instance compute name change(incorrect format: , correct format: ), user should not change vmss instance compute name, otherwise it would break disk attach/detach, since compute name is used in vmss node name passing, code is here:
Hi We don't change the instance compute name - however what i do see in the azure metadata service is in the form - What you're referring to seems to be the computer name, and we do use the defaults as provided by azure. See The attach/detach operations works fairly well most of the time, and I think we're running into some edge case here.

One thing i might have missed out is the fact that there seems to have been an issue with the VM at the time. So the VM was present, but azure platform would return 400 bad request for a detach operation which probably led to this .. But when the VM was eventually deleted by the autoscaler, the re-attach was never attempted by the reconciler it would seem. and the last message about that disk was
should be a valid compute name according to , do you know why we define the format like this?
The function is used to get the name from IMDS, and its format is .
From the above errors, seems the request has been sent to VMSS but failed because it's not active. Do you know the VM state during that time?
I am not sure of the state at the time, but i believe the VM was in some kind of failed state. The VM has since been deleted, but we had errors on the scaleset at the time. The events on the VMSS look like this. The VM was eventually deleted by the autoscaler, but the disk was never re-attached to a different node, and the pod remained in for about 2 days until controller-manager was restarted.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Apparently we hit this issue today as a result of replacing the nodepool. We 2 new nodepools and deleted the old one, and now we have 718 of 1672 pods stuck with this error in one of our clusters. The volumes are failing to be attached to new pods, the VMSS for old nodepool is removed. In AKS portal disks are still reported to be attached to VM instances in the VMSS that was deleted.

hundreds of pods are stuck in Init state won't likely go forward. /remove-lifecycle rotten
/remove-lifecycle rotten
I have seen similar issue when handling one customer's case, it's probably caused by broken vmss cluster, pls file a ticket to VMSS team to fix the issue.
I happened to find this issue in one customer case, this issue is caused by obsolete cache in kube-controller-manager, should skip attach/detach retry when it's 400, below are the details, following two instances were deleted 3min ago before kube-controller-manager trying to attach/detach: workaround restart kube-controller-manager should work
/assign


Question: new message, the check MUST have the update message. The event is deleted and MUST NOT show up when listing all events. release: v1.19 <del> file: test/e2e/framework/events/events.go </del> <ins> file: test/e2e/apimachinery/events.go </ins> - testname: Garbage Collector, delete deployment,  propagation policy background codename: '[sig-api-machinery] Garbage collector should delete RS created by deployment when not orphaning [Conformance]'

[x] Original approval issue: [x] Two weeks soak end date: 6 July 2020 This issue is created to allow edit by /wip /hold According to this APIsnoop query, there are still an endpoint which is untested. a set of 3 Events with a static label within a Namespace. that all 3 Events with the label in the Namespaces are created. the set of Namespaced Events with a label via DeleteCollection. that all Events with the label have been deleted. Discover useragents: List endpoints hit by the test: Display endpoint coverage change: Two other endpoints tested here have tests in progress to conformance already. If a test with these calls gets merged, test coverage will go up by 1 points This test is also created with the goal of conformance promotion.
/sig testing /sig architecture /area conformance
During Conformance Meeting: Is there a way to write this test so it is resource agnostic? If we are just changing the resource... can be parameterize the resource-type
/assign
GitHub didn't allow me to assign the following users: heyste. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen PR Merge and soaking
Reopened this issue. <detailsIn response to : /reopen PR Merge and soaking Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: cmd.Flags().StringP("output", "o", "", "Output format. Must be one of yaml|json") cmd.Flags().StringVarP(&options.Selector, "selector", "l", "", "Selector (label query) to filter on, supports '=', '==', and '!='.") <ins> cmd.Flags().BoolVar(&options.All, "all", false, "select all resources in the namespace of the specified resource types") </ins> usage := "that contains the last-applied-configuration annotations" cmdutil.AddFilenameOptionFlags(cmd, &options.FilenameOptions, usage)

It actually doesn't support --all.
/assign
ohh, thanks, will make a pr to address that right now..


Question: "k8s.io/apimachinery/pkg/runtime/schema" "k8s.io/apimachinery/pkg/types" "k8s.io/apimachinery/pkg/util/intstr" <ins> "k8s.io/apimachinery/pkg/util/rand" </ins> "k8s.io/apimachinery/pkg/util/wait" utilfeature "k8s.io/apiserver/pkg/util/feature" "k8s.io/client-go/informers"

The controllers can create a second EndpointSlice after an immediate sync if the first EndpointSlice they created is not in the cache yet. Here's what can happen: gets event that triggers sync for Service syncs and creates EndpointSlice gets event that triggers sync for same Service syncs and creates EndpointSlice (the slice created in 2 has not made it to the cache yet) A single EndpointSlice to be created. Run integration tests hundreds of times that involve 2 consecutive events that will trigger a sync. Example: /assign /sig network
/triage accepted
/cc
When this bug is fixed, we'll need to rollback
just amend it in the PR that fix the bug, so we merge all together


Question: name := volume.GenerateVolumeName(c.options.ClusterName, c.options.PVName, 63) // GCE PD name can have up to 63 characters capacity := c.options.PVC.Spec.Resources.Requests[v1.ResourceName(v1.ResourceStorage)] <del> requestBytes := capacity.Value() // GCE works with gigabytes, convert to GiB with rounding up requestGB := volume.RoundUpSize(requestBytes, 1024*1024*1024) </del> <ins> // GCE PDs are allocated in chunks of GBs (not GiBs) requestGB := volume.RoundUpToGB(capacity) </ins> // Apply Parameters (case-insensitive). We leave validation of // the values to the cloud provider.

GCE PDs are allocated in chunks of GBs (not GiB) but function incorrectly creates volume in chunks of GiB - cc /sig storage
/kind bug
/assign
/priority important-soon
Hey I submit a pr for this issue. Your suggestions are very important. Thanks.


Question: [interactive tutorial]: https://kubernetes.io/docs/tutorials/kubernetes-basics [kubernetes.io]: https://kubernetes.io [Scalable Microservices with Kubernetes]: https://www.udacity.com/course/scalable-microservices-with-kubernetes--ud615 <del> [troubleshooting guide]: https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ </del>  No newline at end of file <ins> [troubleshooting guide]: https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/ ## Community Meetings The [Calendar](https://www.kubernetes.dev/resources/calendar/) has the list of all the meetings in Kubernetes community in a single location. ## Adopters The [User Case Studies](https://kubernetes.io/case-studies/) website has real-world use cases of organizations across industries that are deploying/migrating to Kubernetes. ## Governance Kubernetes project is governed by a framework of principles, values, policies and processes to help our community and constituents towards our shared goals. The [Kubernetes Community](https://github.com/kubernetes/community/blob/master/governance.md) is the launching point for learning about how we organize ourselves. The [Kubernetes Steering community repo](https://github.com/kubernetes/steering) is used by the Kubernetes Steering Committee, which oversees governance of the Kubernetes project. ## Roadmap The [Kubernetes Enhancements repo](https://github.com/kubernetes/enhancements) provides information about Kubernetes releases, as well as feature tracking and backlogs. </ins>  No newline at end of file

See Most of the things that the tool is complaining about, we have information just not in the form the tool accepts. So let's try to see if we can point to existing resources with a placeholder file or something? ! For example the Roadmap may be a pointer to kubernetes/enhancements repository, governance may be a pointer to relevant docs in community and/or steering repository etc.
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc
the background is that the CNCF has launched a tool clomonitor to help projects do better in adopting best practices and establish a baseline for where information is stored so it is easy for any end users of projects to find things easily. the code for is at so that is open sourced as well. Hope this helps
Right, please also note that we are still in an alpha state, so happy to hear the feedback through the issues here -
Thanks for the nudge
/assign
/remove-kind bug /kind documentation


Question: if err != nil { return false, err } <del> // TODO: move the following podNumber check to podFitsResourcesInternal when Kubelet allows podNumber check (See #20263). allocatable := info.Status.Allocatable allowedPodNumber := allocatable.Pods().Value() if int64(len(nodeInfo.Pods()))+1 > allowedPodNumber { return false, newInsufficientResourceError(podCountResourceName, 1, int64(len(nodeInfo.Pods())), allowedPodNumber) } </del> return podFitsResourcesInternal(pod, nodeName, nodeInfo, info) }

Was there some reason only the checking to the scheduler, not Kubelet?
IIRC it was written at the time when we stated adding checking in Kubelet. In other words - no reason.


Question: func (scaleTest *HPAScaleTest) run(name, kind string, rc *common.ResourceConsumer, f *framework.Framework) { rc = common.NewDynamicResourceConsumer(name, kind, int(scaleTest.initPods), int(scaleTest.totalInitialCPUUsage), 0, 0, scaleTest.perPodCPURequest, 200, f) defer rc.CleanUp() <del> common.CreateCPUHorizontalPodAutoscaler(rc, scaleTest.targetCPUUtilizationPercent, scaleTest.minPods, scaleTest.maxPods) </del> <ins> hpa := common.CreateCPUHorizontalPodAutoscaler(rc, scaleTest.targetCPUUtilizationPercent, scaleTest.minPods, scaleTest.maxPods) defer common.DeleteHorizontalPodAutoscaler(rc, hpa.Name) </ins> rc.WaitForReplicas(int(scaleTest.firstScale)) if scaleTest.firstScaleStasis > 0 { rc.EnsureDesiredReplicas(int(scaleTest.firstScale), scaleTest.firstScaleStasis)

Currently the time it takes for HPA and CA to scale up after load increase is not measured at all, and this can lead to regressions that will be missed. This issue is about adding an e2e test that will prevent such regressions.
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list .
Filip, can you reopen? I cannot do this myself, and this issue is clearly not closed yet.
Thanks Wojtek!


Question: for _, c := range status.Conditions { if kubetypes.PodConditionByKubelet(c.Type) { _, oc := podutil.GetPodCondition(oldCopy, c.Type) <del> if oc == nil || oc.Status != c.Status { </del> <ins> if oc == nil || oc.Status != c.Status || oc.Message != c.Message || oc.Reason != c.Reason { </ins> return false } }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: When the conditionType specified in Readiness Gates changes from non-existent to False, the reason of ready condition keep . What you expected to happen: when set the condition status to False , the message of ready condition can changed. How to reproduce it (as minimally and precisely as possible): a deployment with Readiness Gate without set the condition specified in Readiness the condition Status specified in Readiness to False Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug /sig node /area kubelet


Question: func (PersistentVolumeUpgradeTest) Name() string { return "persistent-volume-upgrade" } const ( <del> pvTestFile string = "/mnt/pv_upgrade_test" </del> <ins> pvTestFile string = "/mnt/volume1/pv_upgrade_test" </ins> pvTestData string = "keep it pv" pvWriteCmd string = "echo "" + pvTestData + "" > " + pvTestFile pvReadCmd  string = "cat " + pvTestFile

https://k8s- This is upgrading from the latest 1.6 branch to master.
If you're not the right person to look into this, please find someone who can.
PR changed the MakePod() call to CreatePod() instead, which I don't think is correct because TestContainerOutput() will already create the pod. what was the reason for changing it?
Hmmm although it looks like the upgrade test was failing even before then, so maybe that's just a red herring.
Actually the test started failing consistently with the same pv_util signature after 3/31, which coincides with the change to call CreatePod(). So I think yes, that is the reason for the latest failures. I will work on a change to move it back to MakePod(). However, the upgrade test in general has been failing for a while. But the signatures have been different. Here's the signature before 3/31: https://k8s-
we updated MakePod to allow passing multiple claims to pod spec. As part of review comment I addressed this change. in the previous code, pod spec has MountPath for volume set to : "/mnt", in the new code we set the mount path to "/mnt/volume1"


Question: echo 'No need to refresh sudo credentials' else echo 'Updating sudo credentials' <del> sudo --validate || exit 1 </del> <ins> sudo -v || exit 1 </ins> fi # Do not use any network plugin by default. User could override the flags with

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: when run , I founded failed. What you expected to happen: Test passed How to reproduce it (as minimally and precisely as possible): Sudo version 1.8.6p7 Anything else we need to know?: Environment: Kubernetes version (use ): 1.17 Cloud provider or hardware configuration: OS (e.g: ):centos Kernel (e.g. ):Linux elk-141 4.9.6- SMP Wed Mar 1 18:39:23 CST 2017 x8664 x8664 x8664 GNU/Linu Install tools: Network plugin and version (if this is a network-related bug): Others:
/cc
/sig node


Question: } } <ins> func getHugePageResourceList(pageSize, value string) corev1.ResourceList { res := corev1.ResourceList{} if pageSize != "" && value != "" { res[corev1.ResourceName(corev1.ResourceHugePagesPrefix+pageSize)] = resource.MustParse(value) } return res } // mergeResourceLists will merge resoure lists. When two lists have the same resourece, the value from // the last list will be present in the result func mergeResourceLists(resourceLists ...corev1.ResourceList) corev1.ResourceList { result := corev1.ResourceList{} for _, rl := range resourceLists { for resource, quantity := range rl { result[resource] = quantity } } return result } </ins> func TestDescribeNode(t *testing.T) { holderIdentity := "holder" <ins> nodeCapacity := mergeResourceLists( getHugePageResourceList("2Mi", "4Gi"), getResourceList("8", "24Gi"), getHugePageResourceList("1Gi", "0"), ) nodeAllocatable := mergeResourceLists( getHugePageResourceList("2Mi", "2Gi"), getResourceList("4", "12Gi"), getHugePageResourceList("1Gi", "0"), ) </ins> fake := fake.NewSimpleClientset( &corev1.Node{

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: scheduler doesn't assign pod because 'not enough resource: hugepages-1Gi',but the node info show it has enough resource. describe node What you expected to happen: the pod was in normal status like pending and running How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: It was change default hugepage size to 1Gi and hugepages=36 numastat was show there has enough hugepage also proc/meminfo was show there has enough resource other info Environment: Kubernetes version (use ):Cloud provider or hardware configuration:OS (e.g: ):Kernel (e.g. ):Install tools:Network plugin and version (if this is a network-related bug):Others:
/sig node
cc do you have thoughts here? I see huge pages and think of you :)
Hmm. Are you sure that there are no other pods using huge page memory ? To me, it looks like the node has (huge page @ 1GiB) memory, you are requsting , but is already used. Can you verify that by listing the pods on the node, and check if some of them are using huge page memory already?
Also, the OS will report that no huge pages are in use as long as no processes have started using the hugepages. Eg. when a pod requesting with is assigned to a node, k8s will then mark the s with huge page memory it as "used". The cgroup controller (if hugetlb cgroup is mounted) will then set the usage limit to . Before the pod actually starts using the huge page memory, the OS will report that no huge page memory is used. When a process inside the pod maps with huge page memory via eg. , the OS will see that the memory is in used. If the process tries to map more than the requested amount, the map will fail (given that the hugetlb cgroup is mounted). To verify if the hugetlb cgroup is mounted, run .

If mounted, it will retrun something like this:
I’m so sorry that there actually has a other pod using hugepagei only see node info available hugepage. so i close the issuethanks a lot
No problem I think this issue is a symptom that something should be done to make it easier to understand. I will look into it.


Question: watchCacheSizes[schema.GroupResource{Resource: "nodes"}] = maxInt(5*clusterSize, 1000) watchCacheSizes[schema.GroupResource{Resource: "pods"}] = maxInt(50*clusterSize, 1000) watchCacheSizes[schema.GroupResource{Resource: "services"}] = maxInt(5*clusterSize, 1000) <ins> watchCacheSizes[schema.GroupResource{Resource: "events"}] = 0 </ins> watchCacheSizes[schema.GroupResource{Resource: "apiservices", Group: "apiregistration.k8s.io"}] = maxInt(5*clusterSize, 1000) return watchCacheSizes }

Currently we signal undecorated storage by simply affecting the undecoratedStorage to the decorator. This is a blocker for kubernetes/enhancements as we need to enable watch cache for events (which currently hardcoded and not configurable) cc /assign /sig api-machinery


Question: ProvisioningCleanupFailed = "ProvisioningCleanupFailed" ProvisioningSucceeded     = "ProvisioningSucceeded" WaitForFirstConsumer      = "WaitForFirstConsumer" <ins> WaitForPodScheduled       = "WaitForPodScheduled" </ins> ExternalExpanding         = "ExternalExpanding" )

What happened: With a StorageClass with , it's not clear why a PVC is not bound when there is no available PV. On Pod, there is a clear event what's wrong: On the PVC side we get only this event: Note that the message says "waiting for first consumer'. But the pod already exists. The reason why it's not bound yet is that the StorageClass has no dynamic provisioning configured () and there are no free PVs to satisfy the claim. What you expected to happen: An event "1 node(s) didn't find available persistent volumes to bind" on the PVC.
/sig scheduling /sig storage
I tried to fix this in VolumeBinding predicate, The predicate is called for each pod - node combination, so it would generate event for each node. Which is too much and it's even wrong when there is a node that has an available PV. We need to evaluate all nodes and then, if the pod is still unschedulable, distribute events to all unbound PVCs of that pod. Scheduler does not offer any way how to track unbound PVCs at that stage - whole volume logic is hidden in VolumeBinding predicate. One option would be to extend  struct to contain also list of objects where to send events (except the pod itself): Scheduler would then send all failed scheduling events it sends to a pod to all the , without any interpretation. is used to manage duplicates easily - each object (its UID) can be in the map only once. Simple array would work too, but then it needs extra work in removing duplicates. Not sure how easy it is to detect duplicate instances. is extensible to allow any object type. predicate would put all unbound PVCs there. does not need any special handling of each PVC. Message "2 node(s) didn't find available persistent volumes to bind" sounds just right in case a pod uses multiple PVCs, each of them might be bind-able on different nodes (node A may satisfy PVC1, but not PVC2, while node B may satisfy PVC2 only). what do you think about such design?
We used to have extension point, which I believe is the right moment for recording the event for unbounds PVCs. But recently it's been refactored so doesn't fit. Do you think this case can be a good example of using as an informational extension point?

Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale wdyt?
I have some other concerns. What if the pod is deleted? Should the scheduler send events that PVC is not in volume binding? The PV controller must be updated to stop emit event in this scenario. It might be simpler to solve on the PV controller side. In this function, we can check if the PVC is being used by a pod. If it is, we can send event like , then people can go to check the progress of pod "foo/bar". If the pod "foo/bar" is deleted, the PV controller can emit "waiting for first consumer ..." event again. What do you think? cc
Sounds better than it's now. My only issue may be performance - PV controller has to check if a pod exists, i.e. list all pods in the same namespace as PVC being reconciled. I am probably too worried.
current attach detach controller has a custom indexer to index pods by PVC keys to avoid iterating all pods every time to find pods which reference given PVC. And there are other places we need to check if a PVC is used by a pod or not from pod informer, e.g. pvcprotectioncontroller pvcontroller we can try to share this custom indexer among these volume controllers in the future. This will improve the overall performance of the IMO. What do you think
Yes, using the indexer looks like a good idea.
/remove-sig scheduling I also agree that solving in PVC side seems like a better idea. Thanks
/assign unbound claims are reconciled in the PV controller every 15 seconds (default), we may have bad worst-case performance (e.g. too many pods in the same namespace). we can add an indexer in the PV controller locally to solve this issue first (also optimize ), then find a way to in manager to improve the overall performance.


Question: buf := bytes.NewBuffer(yamlBytes) reader := utilyaml.NewYAMLReader(bufio.NewReader(buf)) for { <del> typeMetaInfo := runtime.TypeMeta{} </del> // Read one YAML document at a time, until io.EOF is returned b, err := reader.Read() if err == io.EOF {

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Add an external MetaFactory that allows for serializing multiple YAML (or JSON) files in kubeadm. The logic of marshalling YAML should be factored out from to . Why is this needed: This will allow processing component configs containing multiple documents per file. Additionally, an out-of-tree library would allow sharing this functionality with other components.
I will try work on this as carry-on from /sig cluster-lifecycle /cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: SubPathExpr: "$(ANNOTATION)/$(POD_NAME)", }, { <del> Name:      "workdir2", </del> <ins> Name:      "workdir1", </ins> MountPath: "/volume_mount", }, },

/kind cleanup /sig storage /sig node /sig testing Description: Change e2e tests to use EmptyDir instead of HostPath as pre-requisite to promoting VolumeSubpathEnvExpansion to Conformance. Related Issues: Conformance Tests:
/cc
/assign


Question: return attacher, nil } <ins> // newVolumeUnmounterFromPlugins attempts to find a plugin by name and then // create an Unmounter. // Returns a valid Unmounter or an error. </ins> func (kl *Kubelet) newVolumeUnmounterFromPlugins(kind string, name string, podUID types.UID) (volume.Unmounter, error) { plugName := strings.UnescapeQualifiedNameForDisk(kind) plugin, err := kl.volumePluginMgr.FindPluginByName(plugName)

in there is a generic error variable defined that is used when a failure occurs and no other error is surfaced, it is a catch-all when something goes wrong. Think this should be reworded to something that is more reflective of the true issue and not so confusing for users. This error mostly comes when a builder can not be created and it is typically not true that the volume type is unsupported, but rather some other underlying issue, such as missing secrets or endpoints or etc... propose something like: or
are you working on this?
- yes, I would like to submit the PR for this

fyi When using PV/PVC for Persistent Volume volume mounts, if something goes wrong in the builder, a generic error is surfaced in the 'kubectl describe' with the following event error description *unsupported volume type": This only occurs when using the PV/PVC abstraction and not directly using a volume plugin. These type of errors are common when using volume plugins for mounts and would be useful for users to have an easy way to identify the errors rather than searching the logs for root cause. I think there are two issues: Why is the original error surfaced from the volume plugins (, , etc...) getting eaten when they return back to the I can see in the logs that 2 volume types are processed for any pod - the first is the actual underlying type (i.e. ) and this is where the actual real error is surfaced, then immediately after this the 2nd type is processed which is the general persistent claim type (i.e. ) and this when the first type error is eaten A simple work around would be to simply add a recorder event in the when there is a builder failure. This ensures the error is recorded and displayed as an event for the describer and will handle all like errors from other volume plugin types. full modifed function for reference from This will produce a better user experience when trying to see what went wrong - in this example, I am doing glusterfs PV/PVC but am missing end points which you can clearly see now in the events of the describe: rbd example: (Missing or Couldn't get secret)
I haven't had a chance to look at your proposal in detail yet, but I agree this is one of the most confusing things new users hit
another one is when gluster endpoints are missing, we get 'unsupported volume type' with your work extrapolating the permissions for NFS to automatically set supplemental groups, is there also a way to indicate more properly it's a permissions issue for it not mounting?
- the above discussion highlights the missing endpoints as the first example, it also discusses a similar rbd issue (missing secret file) but the point is, any of those type of volume plugin type of errors would be surfaced by adding the event in the
I like it! Is there a PR?


Question: srcs = [ "convert_test.go", "goopenapi_test.go", <ins> "unfold_test.go", </ins> "validation_test.go", ], embed = [":go_default_library"],

What happened: API server panics handling requests for a CRD with OpenAPI validation with What you expected to happen: CRD is applied successfully. How to reproduce it (as minimally and precisely as possible): kind create cluster --image kindest/node:v1.16.1 then kubectl apply: Partial logs from API server: Anything else we need to know?: Environment: Kind Kubernetes version (use ):OS (e.g: ):
/sig api-machinery
/cc
where It seems that s.AnyOf was referencing some invalid address.
yes, which is very strange, given s was dereferenced earlier in the same method: is investigating
reproduced and has a fix
I had the same issue with prometheus (kube 1.16.1). Deploying is fine but when trying to delete, it crashes the api server. For those who can't upgrade to 1.16.2+ (fixed version) there is a workaround : using etcd to delete the resources that crash the api server : here I grep on coreos because I know it is prometheus that crashes the api server Remove every item in the list that causes the crash :
Note this is also resolved in 1.15.5+ and 1.16.2+, upgrading to 1.17.x is not required to resolve this


Question: LastHeartbeatTime:  metav1.Date(2015, 1, 1, 12, 0, 0, 0, time.UTC), LastTransitionTime: metav1.Time{Time: metav1.Date(2015, 1, 1, 12, 0, 0, 0, time.UTC).Add(time.Hour)}, }, <ins> { Type:               v1.NodeMemoryPressure, Status:             v1.ConditionUnknown, Reason:             "NodeStatusNeverUpdated", Message:            "Kubelet never posted node status.", LastHeartbeatTime:  metav1.Date(2012, 1, 1, 0, 0, 0, 0, time.UTC), // should default to node creation time if condition was never updated LastTransitionTime: metav1.Time{Time: metav1.Date(2015, 1, 1, 12, 0, 0, 0, time.UTC).Add(time.Hour)}, }, { Type:               v1.NodeDiskPressure, Status:             v1.ConditionUnknown, Reason:             "NodeStatusNeverUpdated", Message:            "Kubelet never posted node status.", LastHeartbeatTime:  metav1.Date(2012, 1, 1, 0, 0, 0, 0, time.UTC), // should default to node creation time if condition was never updated LastTransitionTime: metav1.Time{Time: metav1.Date(2015, 1, 1, 12, 0, 0, 0, time.UTC).Add(time.Hour)}, }, </ins> }, Capacity: v1.ResourceList{ v1.ResourceName(v1.ResourceCPU):    resource.MustParse("10"),

I just found a cluster like this cc/
do you know if anyone else is currently working on a fix for this? I've already started looking into a fix, would you be willing to review my PR soon?
I don't know of anyone who is working on it.
(I see your PR, and I'll review it.)


Question: "net" "net/http" "sync" <del> "sync/atomic" </del> "time" v1 "k8s.io/api/core/v1"

ci-kubernetes-csi-canary-test-on-kubernetes-master, [It] [sig-storage] CSI mock volume CSI Volume Snapshots [Feature:VolumeSnapshotDataSource] volumesnapshotcontent and pvc in Bound state with deletion timestamp set should not get deleted while snapshot finalizer exists [It] [sig-storage] CSI mock volume CSI Volume Snapshots secrets [Feature:VolumeSnapshotDataSource] volume snapshot create/delete with secrets 25-08-2022? No response Looking at "volume snapshot create/delete with secrets" test, it finishes OK, but the "Cleaning up resources" freezes for ~30 minutes: After these 30 minutes, the test timeouts and kills API server and etcd a and everything fails miserably. Stack trace from "Cleaning up resources", it's stuck at shutting down gRPC server: /sig storage
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I reproduced this issue in somewhat hacked from external-provisioner (or any other kubernetes-csi repo). The script compiles Kubernetes master from a fresh checkout, runs it in KIND, installs hostpath CSI driver with canary images and runs e2e tests for the CSI driver like this: "hostpath CSI driver with canary images" is the interesting part - when I run the test with the latest tagged images, it succeeds, when using canaries it freezes at the cleanup. I found out that using the latest tagged images for everything except is enough to freeze the cleanup. And further I was able to check that node-driver-registrar at commit works, while master does not. There are just 2 commits in between: new release-tools and update to v1.25 libraries. I suspect it's the new libraries. I still don't understand how node-driver-registrar and its libraries could affect gRPC between the hostpath CSI driver and binary.
does this ring a bell? I though node-driver-registrar does not do anything with proxied "mock" CSI calls between and the CSI driver.

In addition, why only these two tests fail reliably, while the rest works?
For the record, this is diff in k/k that adds canary node-driver-registrar to This diff is enough to make the tests fail:
This does not ring a bell at all. You are right that csi-node-driver-registrar should not be involved in the gRPC communication between and the driver. Does it communicate with the driver after the initial registration? If yes, then that will lead into connections that need to be proxied, which might prevent the shutdown. Just a wild guess. But even then it's not clear how grpc.Server.Stop can get stuck. According to its documentation, "It immediately closes all open connections and listeners.". Hmm, could the proxy code get stuck when its listener is closed?
There is in that gets stuck, I did not check what is in that group.
Well, of course it fails :sweat_smile: There is no such canary image. Hmm, but then why do other tests pass? They all use .
There is some magic in your script that replaces with , the registrar runs and registers the driver in the end.
Yep:
One difference is that the driver-registrary canary no longer detects the loss of connection to the mock proxy. Previously it did: It could be that the proxy code relies on the remote side to close their side of the connection before considering everything shut down. That is necessary because the remote side might still send some data that needs to be received. If the remote side keeps the connection open, everything freezes. One can argue that it shouldn't block forever, but the bigger question is why "Lost connection" is not detected anymore with the more recent gRPC.
The "exit on connection loss" depends on our dialer being called again by gRPC:
/assign No idea yet, but I'll keep looking...
I can reproduce the same failure to shut down also in other tests. Must be a timing issue that it only occurred for the same two tests in the CI - not a very satisfying explanation, unfortunately. I have something that works for me locally: closing connections when shutting down the listener -see


Question: StatefulSetLister:              c.statefulSetLister, NodeLister:                     &nodeLister{c.nodeLister}, PDBLister:                      c.pdbLister, <del> NodeInfo:                       &predicates.CachedNodeInfo{NodeLister: c.nodeLister}, </del> <ins> NodeInfo:                       c.schedulerCache, </ins> PVInfo:                         &predicates.CachedPersistentVolumeInfo{PersistentVolumeLister: c.pVLister}, PVCInfo:                        &predicates.CachedPersistentVolumeClaimInfo{PersistentVolumeClaimLister: c.pVCLister}, StorageClassInfo:               &predicates.CachedStorageClassInfo{StorageClassLister: c.storageClassLister},

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: I recently saw an issue in which the scheduler hit this log line many times: After some digging, it turns out that PodAffinityChecker has a and . The points at the scheduler's cache instance, while the points at a light wrapper around a . In the scheduler, we use the cache to provide a globally consistent view of the cluster, even if that view is slightly out of date. Data read from the core may not have been processed by the scheduler cache yet. Letting data "skip ahead" like this can cause stale info to get persisted in the scheduler cache, and other subtle bugs. While the original problem is not likely caused by this issue, I think it needs to be cleaned up to avoid potential bugs down the line. What you expected to happen: Predicates should only read state from the scheduler's cache in order to maintain consistency. How to reproduce it (as minimally and precisely as possible): This was observed in a cluster with high rate of pod creation and node creation/deletion. Anything else we need to know?: Environment: Kubernetes version (use ): 1.12.5 Cloud provider or hardware configuration: GKE with preemptible nodes OS (e.g: ): Kernel (e.g. ): Install tools: Others:
/sig scheduling
We do have this kind of issues, and here is another one:
I'm not sure the urgency of this fix, but if its not super urgent, I would love to pick it up, if there's an eng more experienced with the scheduler who could give me some guidance :)
Hi I dug into the source code, and find out is used by the scheduler and kubelet. Maybe we could change the internal struct of , but I do not know why it appears in kubelet which seems irrelevant to the scheduler.
is just a wrapper of corelisters.NodeLister. kubelet can get node info by it.
I think the point is that the kubernetes scheduler was designed to support single instance at the beginning. Even with the addition of multiple scheduler support, the cache between each scheduler is isolated. So at this stage, we can accept this without a shared state solution.

After all, the node information is not actively modified by the kubernetes scheduler.
So is it necessary to ensure the consistency by using node lister instead of cache in the scheduler, or we do not need to fix this issue which seems like to caused by inconsistent cache.
We've recently hit this issue with the scheduler. It appears that our scheduler somehow gets out of sync with the nodes, and start scheduling pods on tainted nodes. The node's kubelet then (rightfully) refuse to admit the pod. Looking at the scheduler logs, they are flooded with the following : Similar scenario, this is a high churn cluster with thousands of pods scheduled per minute - and nodes being autoscaled. Can we try to get a fix ?
I'm glad to pick this up if we decide to fix this issue.
It would be awesome if you could fix this issue for us, we are currently seeing 100's of pods incorrectly being scheduled onto nodes with taints.
The code cited for logging "Node not found, %v" is now in :
/assign
Thanks for the interest on fixing this bug. Sorry I haven't had time to engage on this issue. Feel free to send PRs my way. I'm not 100% certain that this will fix the problem of pods getting scheduled to tainted nodes. How is the taint being applied to the node? Even if the scheduler is internally consistent, there could be a time between node creation and the taint being applied where we send pods to that node.


Question: // UpdateLease resets the TTL on a master IP in storage func (s *storageLeases) UpdateLease(ip string) error { <del> return s.storage.GuaranteedUpdate(apirequest.NewDefaultContext(), s.baseKey+"/"+ip, &api.Endpoints{}, true, nil, func(input kruntime.Object, respMeta storage.ResponseMeta) (kruntime.Object, *uint64, error) { </del> <ins> key := path.Join(s.baseKey, ip) return s.storage.GuaranteedUpdate(apirequest.NewDefaultContext(), key, &api.Endpoints{}, true, nil, func(input kruntime.Object, respMeta storage.ResponseMeta) (kruntime.Object, *uint64, error) { </ins> // just make sure we've got the right IP set, and then refresh the TTL existing := input.(*api.Endpoints) existing.Subsets = []api.EndpointSubset{

/kind bug The lease reconciler creates the kubernetes endpoint with a prefix of kube-apiserver-endpoint, however that is not the prefix that the rest of kube uses. The end result is that the kubernetes service doesn't have an endpoint, and kubectl won't list the kubernetes endpoint. The lease reconciler also creates lease files with a really long lease time instead of the default 15 seconds.
/cc
cc xref
kube-apiserver-endpoint is used for the lease itself, from what I remember. I don't think that's an issue by itself.
Oh but it does look like the endpointRegistry for the lease reconciler is misconfigured to use that
i'll look into it
what is the name that should be used?
services/endpoints afaik -
Yeah, making that change locally fixed it for me
But I'm not seeing the masterleases/apiServerIPInfo data in etcd, or I'm issuing the wrong command to find it
I had that same problem. The lease timeout is set way high, but for some reason that resulted in the lease files being created for only a very short time. I'm not sure why. Playing with the ttl amount got the lease files to stay around....forever. I've got a PR I'm about to push to fix both issues.
FYI has a PR for this now
cc


Question: ARCH?=amd64 HYPERKUBE_BIN?=_output/dockerized/bin/linux/$(ARCH)/hyperkube <del> BASEIMAGE=gcr.io/google-containers/debian-hyperkube-base-$(ARCH):0.2 </del> <ins> BASEIMAGE=gcr.io/google-containers/debian-hyperkube-base-$(ARCH):0.3 </ins> TEMP_DIR:=$(shell mktemp -d -t hyperkubeXXXXXX) all: build

/kind bug What happened: When a cluster is provisioned using a containerised kubelet using the following images: Pods with PVCs are not created because kubelet cannot mount the disks. If you look at events, you can see that it's missing some binaries: This executable is . This problem can be resolved by mounting paths from the host into the rkt container, specifically and . Is this expected? mentioned in , that mounting operations are moving out from the main process into containers. Is this part of that effort? This caught me quite unexpected, and I didn't find any docs or changelog entries which indicated additional host mounts were necessary. If it's deliberate, I can close this until we perhaps document the breaking change. Is there a good place where we can document this? If it's not, I'll leave this open as a tracking issue. What you expected to happen: The hyperkube image to work without lots of host mounts. How to reproduce it (as minimally and precisely as possible): Run kubelet with rkt in one of the above listed images (see CoreOS' ). Environment: Kubernetes version (use ): Cloud provider or hardware configuration**: OpenStack OS (e.g. from /etc/os-release):Kernel (e.g. ):Install tools: tectonic-installer
/sig storage /sig node
Seems the base hyperkube image was changed: done in /cc
The pkg/util/mount/mount_linux source defaults to ext4 if a fstype is not selected. It makes sense to restore the e2fsprogs package within the hyperkube image.
cc cc cc This seems like something that should be in the 1.8 milestone. Can someone who owns the hyperkube image comment on that?
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/node sig/storage: Issue will be escalated to these SIGs if needed. priority/important-soon: Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. kind/bug: Fixes a bug discovered during the current release. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details
Containerization of mount is not going to happen until 1.9 at the earliest.

So if hyperkube is broken, it should be fixed before ship.
"modprobe" also went missing in 1.8 hyperkube images... Both kubelet and kube-proxy need it.
Are you noticing any buggy behaviour on those components without it? Would you mind creating an issue and/or PR to restore that dep
Done: And done: :)


Question: {Name: volumeName + "-0", MountPath: volumePathPrefix + "-0"}, {Name: volumeName + "-1", MountPath: volumePathPrefix + "-1"}, }, <ins> selinuxOptions, </ins> ) <del> ginkgo.By(fmt.Sprintf("Waiting for the pod (%v/%v) to be running", f.Namespace.Name, podName)) </del> <ins> ginkgo.By(fmt.Sprintf("Waiting for the pod (%s/%s) to be running", f.Namespace.Name, podName)) </ins> err := e2epod.WaitForPodNameRunningInNamespace(ctx, f.ClientSet, podName, f.Namespace.Name) <del> framework.ExpectNoError(err, "Failed to await for the pod to be running: (%v/%v)", f.Namespace.Name, podName) </del> <ins> framework.ExpectNoError(err, "Failed to await for the pod to be running: (%s/%s)", f.Namespace.Name, podName) </ins> for i := range 2 { volumePath := fmt.Sprintf("%s-%d", volumePathPrefix, i)

Related to and SIG storage meeting notes: From me: It is related to image volume plugin and selinux. Is it a known limitation of all the volume plugins when using selinux? I have a different opinion. If two pods don't share the same mountpoint (hostpath), they can run the same node when the selinux is enabled. the current implementation of cri-o is sharing the same hostpath when two pods use the same image volume. More details about my idea: From Sharing each volume between namespaces is kind of natural. Sharing PVs not possible but sharing image quite common. From Although for most volume plugins it is true that, they can't be mounted with 2 different selinux context on same node, we want to understand why is this not possible for Imagevolume source. For a traditional storage persistent storage plugins - this limitation is underlying limitation of storage backend/Linux kernel etc. For ImageVolume source, it seems like a simple solution which unpacks image to two different hostpath would work and not run into this limitation. can you specify, why this limitation exists for ImageVolume plugins? From we choose to keep it simple and follow the lead of other volume types. supporting multiple selinux types would mean making more complicated the way we mount the image into the container AFAIR. But agreed we can chat about this elsewhere, in the meantime for the purposes of this PR: this is consistent with behavior of other volume plugins and is expected behavior. Let's discuss the SELinux behavior for image volumes in this issue. cc /sig node /sig storage
In my opinion, from an end-user perspective, the image volume may act like a read-only with a pre-populated content via image. Two pods from different namespaces can share the same image and run on the same node.
Pasted from According to , the containerd doesn't have the same limitation.
Given that OCI/Image Volume Source support is in alpha, I'd prefer that we consider discussing this as part of the original feature. We can discuss this here but I don't know if we need a separate KEP for this yet.

Yes, so far, in our conversations at the containerd community meetings we decided to start this feature out using a single pod scope for the pod scoped image volume host path because in containerd each runtime handler may have a different snapshotter (possibly on a different fs), or at least that's the plan. Additionally, the pods may have different os versions on windows indicating a different platform can be selected on the same node and this sort of versioning may extend beyond to other os providers. We could ask Kirtana to confirm the versioning issue and status of platform version selection over the platform index manifest, if the snapshotter reason isn't enough to at least start with per pod scope. If this is insufficient support we can certainly discuss. Cheers, Mike
No need a separate KEP. We can update the original KEP once this issue is resolved.
I'm working on a technical solution (just a PoC) in , not sure how feasible that is as time of writing, though.
yeah I agree I don't think we need a different feature for this, especially because it's mostly an implementation detail in the runtime. would you open this as an issue in kubernetes/kubernetes instead of the enhancements repo, or even better in the cri-o repo?
/transfer kubernetes
I moved this issue into kubernetes repo.
/triage accepted /priority important-soon
thanks!
Copid from kep: As merged, do we need to change the KEPs to reflect the new behavior?
no, nothing needs to be changed from a Kubernetes any more :+1:
/remove-sig storage


Question: // since this case has no score plugin, we'll only try to find one node in Filter stage wantEvaluatedNodes: ptr.To[int32](1), }, <ins> { name: "test prefilter plugin returned an invalid node", registerPlugins: []tf.RegisterPluginFunc{ tf.RegisterQueueSortPlugin(queuesort.Name, queuesort.New), tf.RegisterPreFilterPlugin( "FakePreFilter", tf.NewFakePreFilterPlugin("FakePreFilter", &framework.PreFilterResult{ NodeNames: sets.New("invalid-node"), }, nil), ), tf.RegisterFilterPlugin("TrueFilter", tf.NewTrueFilterPlugin), tf.RegisterBindPlugin(defaultbinder.Name, defaultbinder.New), }, nodes:     []string{"1", "2"}, pod:       st.MakePod().Name("test-prefilter").UID("test-prefilter").Obj(), wantNodes: nil, wErr: &framework.FitError{ Pod:         st.MakePod().Name("test-prefilter").UID("test-prefilter").Obj(), NumAllNodes: 2, Diagnosis: framework.Diagnosis{ NodeToStatusMap: framework.NodeToStatusMap{ "1": framework.NewStatus(framework.UnschedulableAndUnresolvable, "node is filtered out by the prefilter result"), "2": framework.NewStatus(framework.UnschedulableAndUnresolvable, "node is filtered out by the prefilter result"), }, }, }, }, </ins> } for _, test := range tests { t.Run(test.name, func(t *testing.T) {

On Kubernetes v1.30.0 (and v1.30.1), can crash with the following panic if a pod is defined in a certain way: The crash happens because the is in certain cases. should not crash, on v1.29.4 it doesn't happen. On v1.29.4 the is just printing these error logs, but doesn't crash: Create a pod like this: The important part is that the affinity doesn't match a real/valid node. Once this pod is being processed by it will crash and continue to do so until the pod is deleted. This issue was triggered during the rotation of control-plane nodes. In our setup we update the control plane by scaling from 1 to 2 instances. Once both a ready, the old is being terminated via EC2 API. At this stage the kube-controller-manager sometimes_ manages to create a replacement daemonset pod once the old node is being deleted. This results in a pod targeting a no longer existing node via as illustrated in the example above. For some reason, when the is crashing the doesn't delete the extra/invalid daemonset pod. Not sure if this is another issue or it has always happened in our setup, but only v1.30 makes crash which causes an actual issue. <details</details<detailsWe run a custom Kubernetes setup on AWS EC2 (not EKS). </details<details</details<detailsCustom tooling: </details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scheduling
cc /priority important-soon
can you take a look? I believe all the latest patch releases are affected too, because of
cc
/priority critical-urgent
This crash happens when preFilter plugins filtered out all nodes ... If preFilter filtered out some nodes, here len(nodes) will be a subset of allNodes. Why are we using this subset at first? I think should works fine?

(seems logical to me)
I think so. In general, we just want to try a different set of nodes. For the case of Daemonsets, it doesn't really matter, as we will just test one node.
/assign Successfully catched this with a unit test. I'll file a small pr with fix and a newly UT.
I am having the same issue with 1.28.10 on a clean new cluster created with kops 1.28.5: The pod that it's trying to create is pending with the following messages: (just for the sake of being indexed for those who will search for the same error to find this github issue)
Fix is on the way: Just waiting for reviews from approver. After this pr got merged, I'll do cherry-picks for v1.28(1.27?)~1.30
cherry-picks:
Thanks for fixing this so fast. How do I pull this new code? I see that your PR was merged, but I don't see a new release to pull? looking for the 1.30
This change will be included in the upcoming patch releases, scheduled for 2024-06-11. You can check out the for more details.
I'm seeing the issue after upgrade v1.26.11-v1.27.14 Should I search for a broken pod? Or what else can be causing this?
Yes, a broken pod could be causing this. You can also try downgrading to v1.27.13
Phew, manually changing the scheduler version to 1.27.13 fixed the issue and showed the broken pod in logs. Thanks!


Question: floatingPool := getStringFromServiceAnnotation(apiService, ServiceAnnotationLoadBalancerFloatingNetworkId, lbaas.opts.FloatingNetworkId) glog.V(4).Infof("EnsureLoadBalancer using floatingPool: %v", floatingPool) <ins> var internalAnnotation bool internal := getStringFromServiceAnnotation(apiService, ServiceAnnotationLoadBalancerInternal, "true") switch internal { case "true": glog.V(4).Infof("Ensure an internal loadbalancer service.") internalAnnotation = true case "false": if len(floatingPool) != 0 { glog.V(4).Infof("Ensure an external loadbalancer service.") internalAnnotation = false } else { return nil, fmt.Errorf("floating-network-id or loadbalancer.openstack.org/floating-network-id should be specified when service.beta.kubernetes.io/openstack-internal-load-balancer is false") } default: return nil, fmt.Errorf("unknow service.beta.kubernetes.io/openstack-internal-load-balancer annotation: %v, specify "true" or "false".", internal) } </ins> // Check for TCP protocol on each port // TODO: Convert all error messages to use an event recorder for _, port := range ports {

Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: Currently openstack cloud provider just support for specifying internal LoadBalancerIP. I think that is unreasonable, actually user just use a floating ip to access a LoadBalancer service. So it is necessary to spport for specifying external LoadBalancerIP. What you expected to happen: Support for specifying external LoadBalancerIP on openstack Anything else we need to know?: Currently LoadBalancerIP can be specified in service.Spec.LoadBalancerIP, we can add annotation to distinguish between internal and external.
/sig openstack /assign I'd like to work for it.


Question: "runtime_conformance_test.go", "security_context_test.go", "summary_test.go", <ins> "system_node_critical_test.go", </ins> "volume_manager_test.go", ], embed = [":go_default_library"],

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: Eviction manager evicts the below static and critical pod when the node observed MemoryPressure. The k8s server version is v1.13.2, and the pod priority is enabled. As far as I can see, the eviction manager does not evict such pods. What you expected to happen: Static pods with are not deleted as well as those with a critical pod annotation. How to reproduce it (as minimally and precisely as possible): pod with system-node-critical priority using by kubelet option memory load to the target node until eviction manager observes memory pressure. critical and static pod is evicted. Anything else we need to know?: When I set critical-pod annotation (and enables ) insted of pod priority, the eviction manager DOES NOT evict the static and critical pod. Environment: Kubernetes version (use ): v1.13.2 Cloud provider or hardware configuration: baremetal
/sig node
from enable priority is , more than I can not understand why it has happened
Right. It seems the expected behavior and the actual behavior are different. will remove next k8s v1.14 (it's written in the document), but I'd like to wait for it until the critical priority can prevent eviction.
I see this as well. Start from the bottom and read up. It ranks ibm-master-proxy-static which is a static pod that has priority system-node-critical. It actually is always ranked first as well.
It can be replicated by causing disk pressure (using ) And then viewing the eviction manager logs. `journalctl -u kubelet | grep eviction There's also the piece that when the static pod is evicted kubelet never ends up starting it back up again.
Enabling ExperimentalCriticalPodAnnotation=true makes it not fail and detect properly
Setting on the static pod also works. It appears that a static pod doesn't get set based on . If this is expected then it should be documented. Although it seems like the priority class name should be honored for static pods.
I've confirmed that the problem occurs when a static pod has set to but does not set accordingly.

doesn't see the priority as being set and thus thinks the pod isn't critical which ultimately causes the eviction. I would have expected to translate to .
The code fix is available in master branch right now, which means it will be auto included in the upcoming 1.14 release. For lower releases, you need to wait for a while before cherrypick PRs get approved and merged.
Cherrypick PRs have been created and waiting for approval. 1.13 - 1.12 - 1.11 -
Your patch does not work for me. The case in that you completely replaces static pods with the mirror pods. But static pods contains important annotations such as . For static pods is , for mirror - . As a result, eviction manager supposes that pod is not static.
I doubt it. A mirror pod should be annotated with : Otherwise the PR won't pass the e2e test.
When I get static pod through I see . But when I try to debug eviction manager, I see that pod has annotation
Thanks for the info. Then we are sort of in a dilemma - on one hand annotation is needed to check if it's a static pod, on the other hand we also need the corresponding mirror pod to get the priority value. I will provide a supplemental fix later.
can we please hold on those cherry-picks until the supplemental fix is also available?
sure.
Here is a detailed explanation on why previous PR passed the e2e test: As , the critical static pod failed on IsStaticPod() check (because of mirror pod is tagged as " api") And it's reasonable that it's being marked as "to be deleted": However, as , there is some weird log showing: It's because the pod we passed onto () is a mirror pod, and it's deleted (or probably a dummy delete) by container runtime without any error. To take an analogy, it's sort of like a "controllee" (mirror pod) gets deleted, but "controller" (static pod) is still alive, so it causes data inconsistency, and the above error msg popped up. And in user's perspective, the static pod is "not" evicted, then it "looks" like previous PR works.
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available .

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Status of cherry pick PRs: 1.13 - - merged 1.12 - - merged 1.11 - - merged
/close As all PRs have been cherrypicked.
Closing this issue. <detailsIn response to : /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
When will this be released for 1.11.x? We are getting this same issue with and believe it's related: The api log:
Maybe in v1.11.10 (but I didn't see it in v1.11.10-beta.0 yet)
Thanks, I assume this is pretty high priority if it's causing to be evicted?
Yes. Prior to get it in 1.11.x, you can work it around by manually setting the priority value (not only setting priority name) to the control plane pods (apiserver, controller-manager and scheduler).
I think the priority value is correct:
Did you get that from ? If so, that's not sufficient. You should work around by hardcoding it into your static pod yaml file.
I did get it from , will try hardcoding. Thanks!
Were you successful ? I'm running into this same problem (unfortunately on a 1.10 cluster, which doesn't have this patch) and received the error: Looks like it comes from
I see this as well but the actual disk is not even full.


Question: // considered acceptable. Once per hour is fine for now, as long as it // doesn't loose too much logs. maxAllowedRestartsPerHour = 1.0 <ins> // lastPodIngestionSlack is the amount of time to wait for the last pod's // logs to be ingested by the logging agent. lastPodIngestionSlack = 5 * time.Minute </ins> ) var _ = framework.KubeDescribe("Cluster level logging implemented by Stackdriver [Feature:StackdriverLogging] [Soak]", func() {

Pods should be run in parallel with checking the logs from the backend to fit into the timeout.


Question: memoryLimit := memoryCapacity.Value() matchResourceMetrics := gomega.And(gstruct.MatchKeys(gstruct.IgnoreMissing, gstruct.Keys{ <del> "scrape_error": gstruct.Ignore(), </del> <ins> "scrape_error":          gstruct.Ignore(), "resource_scrape_error": gstruct.Ignore(), </ins> "node_cpu_usage_seconds_total": gstruct.MatchAllElements(nodeID, gstruct.Elements{ "": boundedSample(1, 1e6), }),

And others: /kind failing-test <!-- If this is a flake, please add: /kind flake --/sig node
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: } } } <ins> func TestSetMultiResourcesImageLocal(t *testing.T) { f, tf, codec, ns := cmdtesting.NewAPIFactory() tf.Client = &fake.RESTClient{ GroupVersion:         legacyscheme.Registry.GroupOrDie(api.GroupName).GroupVersion, NegotiatedSerializer: ns, Client: fake.CreateHTTPClient(func(req *http.Request) (*http.Response, error) { t.Fatalf("unexpected request: %s %#vn%#v", req.Method, req.URL, req) return nil, nil }), } tf.Namespace = "test" tf.ClientConfig = &restclient.Config{ContentConfig: restclient.ContentConfig{GroupVersion: &legacyscheme.Registry.GroupOrDie(api.GroupName).GroupVersion}} buf := bytes.NewBuffer([]byte{}) cmd := NewCmdImage(f, buf, buf) cmd.SetOutput(buf) cmd.Flags().Set("output", "name") cmd.Flags().Set("local", "true") mapper, typer := f.Object() tf.Printer = &printers.NamePrinter{Decoders: []runtime.Decoder{codec}, Typer: typer, Mapper: mapper} opts := ImageOptions{FilenameOptions: resource.FilenameOptions{ Filenames: []string{"../../../../test/fixtures/pkg/kubectl/cmd/set/multi-resource-yaml.yaml"}}, Out:   buf, Local: true} err := opts.Complete(f, cmd, []string{"*=thingy"}) if err == nil { err = opts.Validate() } if err == nil { err = opts.Run() } if err != nil { t.Fatalf("unexpected error: %v", err) } expectedOut := "replicationcontrollers/first-rcnreplicationcontrollers/second-rcn" if buf.String() != expectedOut { t.Errorf("expected out:n%snbut got:n%s", expectedOut, buf.String()) } } </ins>

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: my yaml: exec kubectl set resources command,just update first object: What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration**: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig cli
/kind bug
/assign


Question: import ( "context" v1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/util/uuid"

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --ci-crio-cgroupv1-node-e2e-conformance ContainerLogPath It was failing last week, then I disabled the SELinux while we investigated the issue Once this PR merges, I will enable the SELinux again for CRIO jobs. Once the SELinux is turned on and all the required components in the OS (e.g. crio, runc) are compiled with SELinux support we observed following errors in audit logs, Which points towards incorrect SELinux labeling.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node


Question: "my-cluster": {Server: "https://192.168.0.1:3434"}, }, Contexts: map[string]*clientcmdapi.Context{ <del> "minikube":  {AuthInfo: "minikube", Cluster: "minikube"}, "my-cluser": {AuthInfo: "mu-cluster", Cluster: "my-cluster"}, </del> <ins> "minikube":   {AuthInfo: "minikube", Cluster: "minikube"}, "my-cluster": {AuthInfo: "mu-cluster", Cluster: "my-cluster"}, </ins> }, CurrentContext: "minikube", AuthInfos: map[string]*clientcmdapi.AuthInfo{ <del> "minikube":  {Token: "minikube-token"}, "my-cluser": {Token: "minikube-token"}, </del> <ins> "minikube":   {Token: "minikube-token"}, "mu-cluster": {Token: "minikube-token"}, </ins> }, } <del> test := viewClusterTest{ description: "Testing for kubectl config view --minify=true", config:      conf, flags:       []string{"--minify=true"}, expected: `apiVersion: v1 </del> <ins> testCases := []struct { description string config      clientcmdapi.Config flags       []string expected    string }{ { description: "Testing for kubectl config view --minify=true", config:      conf, flags:       []string{"--minify=true"}, expected: `apiVersion: v1 </ins> clusters: - cluster: server: https://192.168.99.100:8443

/kind bug What happened: Current context was set to cluster1. Ran and got back information about cluster1 instead. What you expected to happen: I expected the output to be the cluster2 information How to reproduce it (as minimally and precisely as possible): Have two contexts in your kube config, A and B Run to set A as current context Run Anything else we need to know?: I'd like to help contribute, so if this is an easy fix, someone point me in the right direction and I'll try to get it done. Environment: Kubernetes version (use ): 1.10.2 OS (e.g. from /etc/os-release): MacOS 10.11.6 Install tools: kubectl was installed via brew
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign Filed


Question: <ins> # Copyright 2019 The Kubernetes Authors. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # #     http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. FROM BASEIMAGE USER ContainerAdministrator ENV chocolateyUseWindowsCompression false RUN powershell -Command  iex ((new-object net.webclient).DownloadString('https://chocolatey.org/install.ps1'));  choco feature disable --name showDownloadProgress;  powershell -Command choco install bind-toolsonly --version 9.10.3 -y RUN powershell -Command " wget -uri 'https://github.com/coredns/coredns/releases/download/v1.5.0/coredns_1.5.0_windows_amd64.tgz' -OutFile 'C:coredns.tgz'; tar -xzvf 'C:coredns.tgz'; Remove-Item 'C:coredns.tgz'" ADD hostname /bin/hostname.exe </ins>

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: N/A Which test(s) are failing: should support configurable pod Since when has it been failing: N/A Testgrid link: N/A Reason for failure: Since ` was used instead. Anything else we need to know: /sig testing /sig windows


Question: func (f *KubeletFlags) AddFlags(fs *pflag.FlagSet) { f.ContainerRuntimeOptions.AddFlags(fs) <del> fs.Var(&f.KubeConfig, "kubeconfig", "Path to a kubeconfig file, specifying how to connect to the API server.") // TODO(#41161:v1.10.0): Remove the default kubeconfig path and --require-kubeconfig. fs.BoolVar(&f.RequireKubeConfig, "require-kubeconfig", f.RequireKubeConfig, "This flag is no longer necessary. It has been deprecated and will be removed in a future version.") fs.MarkDeprecated("require-kubeconfig", "You no longer need to use --require-kubeconfig. This will be removed in a future version. Providing --kubeconfig enables API server mode, omitting --kubeconfig enables standalone mode unless --require-kubeconfig=true is also set. In the latter case, the legacy default kubeconfig path will be used until --require-kubeconfig is removed.") </del> <ins> fs.StringVar(&f.KubeConfig, "kubeconfig", f.KubeConfig, "Path to a kubeconfig file, specifying how to connect to the API server. Providing --kubeconfig enables API server mode, omitting --kubeconfig enables standalone mode.") </ins> fs.MarkDeprecated("experimental-bootstrap-kubeconfig", "Use --bootstrap-kubeconfig") fs.StringVar(&f.BootstrapKubeconfig, "experimental-bootstrap-kubeconfig", f.BootstrapKubeconfig, "deprecated: use --bootstrap-kubeconfig") <ins> // TODO: when pull-kubernetes-e2e-kops-aws does not need this parameter, delete requireKubeConfig. var requireKubeConfig bool fs.BoolVar(&requireKubeConfig, "require-kubeconfig", requireKubeConfig, "This flag is no longer necessary. It has been deprecated and will be removed in a future version.") fs.MarkDeprecated("require-kubeconfig", "You no longer need to use --require-kubeconfig. This will be removed in a future version. Providing --kubeconfig enables API server mode, omitting --kubeconfig enables standalone mode unless --require-kubeconfig=true is also set. In the latter case, the legacy default kubeconfig path will be used until --require-kubeconfig is removed.") </ins> fs.StringVar(&f.BootstrapKubeconfig, "bootstrap-kubeconfig", f.BootstrapKubeconfig, "Path to a kubeconfig file that will be used to get client certificate for kubelet. "+ "If the file specified by --kubeconfig does not exist, the bootstrap kubeconfig is used to request a client certificate from the API server. "+ "On success, a kubeconfig file referencing the generated client certificate and key is written to the path specified by --kubeconfig. "+

This will be the end of the deprecation period for --require-kubeconfig.
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list .
ref
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale


Question: makeServiceMap(fp, makeTestService(svcPortName.Namespace, svcPortName.Name, func(svc *v1.Service) { svc.Spec.ClusterIP = svcIP <ins> svc.Spec.ExternalIPs = []string{extIP} svc.Spec.Type = "LoadBalancer" </ins> svc.Spec.Ports = []v1.ServicePort{{ Name:     svcPortName.Port, Port:     int32(svcPort), Protocol: v1.ProtocolUDP, NodePort: int32(nodePort), }} <ins> svc.Status.LoadBalancer.Ingress = []v1.LoadBalancerIngress{{ IP: lbIngressIP, }} </ins> }), )

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --UDP traffic to a Pod behind a LoadBalancer Service exposing a UDP listener service is blackholed when the replacement Pod is scheduled on the same node as the "old" Pod. A new Pod that gets provisioned (replacing the previous one) will receive traffic once it's ready. a UDP Listener Pod (single pod) and using nodeSelector ensure it runs on a specific node the Pod behind a LoadBalancer Service continuous stream of UDP packets from outside the cluster to the External IP of the Service that traffic is correctly received the Pod and observe as a new one is created, traffic should now be blackholed and not reaching the "new" Pod After the new Pod is recreated, we noticed the following entries on the node: notice that the <SVC EXT IPappears here both in the original and the NATed , this is wrong and I suspect happens because the conntrack for the old Pod were flushed before the iptables rules for the new Pod were created (?) and in the interim new packets have reached the node and resulted in these incorrect conntrack entries. To "fix" this, we manually deleted the above conntrack entries, and this allowed new correct one to be established based on the now existing iptables nat rule: with this new conntrack entries, the traffic started flowing as expected to the new Pod and notice that the NATed now correctly shows the Pod IP (172.30.3.219) instead of the Service External IP. While spelunking through the GitHub issues and PRs, I found this issue which was addressed in , it does mention that: the service has endpoints available. Since the fix in this PR refers to NodePort, perhaps the same also applies to External IP (i.e. LoadBalancer Service?) or there's something else at play here? AFAICT, this fix is included in v1.20.7 that we are running. Kubernetes version (use ):Cloud provider or hardware configuration: GCP/GKE OS (e.g: ):Kernel (e.g.

): Install tools: Terraform (with google provider) Network plugin and version (if this is a network-related bug): Others:
/sig network
According to GCP support (since we hit this on a GKE cluster) there's already a fix for this bug and it in v1.21, however I'm unable to find to link it here (hope they can provide the link shortly).
GCP support referenced as the fix for this issue, however this was already merged and released in v1.20.4 but we are running v1.20.7, so should have the fix already, so the status is unclear at this point.
thanks for the detailed report, at first sight we do clean the external ips is it possible to set kube-proxy in a log level of 2 and capture the logs from the node to correlate with the time it happends, and understand why those entries are not being deleted?
Hey we can try that, however since this is a GKE environment I'd have to check how/if we can change kube-proxy log level. My understanding of the issue is that it's not that the conntrack entries are flushed, is that they are flushed before the nat iptables rules are created, causing faulty conntrack entries to be created in the interim (since the udp packets are incoming all the time).
I'm having some miscommunication about the availability of fix in with GCP/GKE support and was hoping you could confirm that is included in v1.20.5 (as per the release notes there? ) and per the cherry-pick PR GCP Support claim that this change is incorporated in v1.21 and we should wait for v1.21 to be available in GKE.
I see the commit in 1.20.5 , this is the cherry-pick The conntrack entries are always deleted after the conntrack entries are created, there is some code that detects that a service goes stale and based on that it deletes the entries.

I think that current code detects this case, at least I know it works when the service goes from 0 to 1 endpoint, that is why the logs will be useful, we can see exactly what happens ...
Thanks for confirming I just checked and seem to be already running in log level 2: I looked back at the logs when this issue was identified and I see the following entries: This two messages and only these two repeat every time we tried to rollout the deployment. is indeed the and is the ClusterIP of the service: but every time we tried, we ended up with these stale conntrack entries I mentioned in the OP and only after deleting them would the udp packet start flowing into the "new" pod.
oh wait, by external IP you mean ?
/triage accepted /assign
here you go, having good reports on bugs make them easy to spot ;)
indeed! I apologize if I wasn't clear indeed I was referring to the LoadBalancer Service External IP. Thanks for fix!!! But now I'm a bit confused, since I understood that LoadBalancer Service External IP == LoadBalancer Ingress IP, but I see these are two separate attributes? so for example: These are the same IP, but internally they are separate attributes?
hmm I don't know how kubectl prints the information, but you can check the fields with and you could see that ExternalIPs and LoadBalancer.IngressIP are different fields, actually one is in the Status and other in the Spec
interesting, so in the output, I can only see the , there's actually no mention of here: That's why I always assumed that this is referring to the same thing/attribute/field.


Question: The Flocker Agents should be manually installed on *all* Kubernetes nodes. These agents are responsible for (de)attachment and (un)mounting and are therefore services that should be run with appropriate privileges on these hosts. <del> In order for the plugin to connect to Flocker (via REST API), several environment variables must be specified on *all* Kubernetes nodes. This may be specified in an init script for the node's Kubelet service, for example. </del> <ins> In order for the plugin to connect to Flocker (via REST API), several environment variables must be specified on *all* Kubernetes nodes. This may be specified in an init script for the node's Kubelet service, for example, you could store the below environment variables in a file called `/etc/flocker/env` and place `EnvironmentFile=/etc/flocker/env` into `/etc/systemd/system/kubelet.service` or wherever the `kubelet.service` file lives. The environment variables that need to be set are: </ins> - `FLOCKER_CONTROL_SERVICE_HOST` should refer to the hostname of the Control Service - `FLOCKER_CONTROL_SERVICE_PORT` should refer to the port of the Control Service (the API service defaults to 4523 but this must still be specified)

We found that some users coming to the document were getting a little confused. It should include more detail about where to install Flocker services, how to set environment variables, that you can create volumes now with Docker CLI and give more detail in example including how to use it with a spec that includes a replication controller.


Question: VolumeBindingMode: bindingMode, } } <ins> // GetUniqueDriverName returns unique driver name that can be used parallelly in tests func GetUniqueDriverName(driver TestDriver) string { return fmt.Sprintf("%s-%s", driver.GetDriverInfo().Name, driver.GetDriverInfo().Framework.UniqueName) } </ins>

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: Probably caused by <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
https://k8s- /assign
I understand that moved crd creation logic to e2e test's k8s cluster deployment code. On the other hand, I tested it with manually deployed k8s cluster. So, in my case, I guess that in addition to enable and feature gates enabled, I need to create crd manually before tests, like below: I could make two of three SkipAttach tests pass in my environment by above way. For "non-attachable volume does not need VolumeAttachment" case, I'm still looking into it. (It doesn't seem to skip creating volumeAttachment, properly)
I found a typo in 's argument. While hostpath driver's connects driver name and unique name without "-", it connects them with "-". This makes "non-attachable" test fail, due to the mismatch between volume name in crd and provisioner name, as a result, creating volumeAttachment isn't skipped. I will create a PR to fix it. For above bug, is unrelated. It just changed the way in how to test in manually deployed k8s cluster as I mentioned in above comment. So, there should be no problem for the PR, at least for the CI testing if it works well in CI environments.
I can verify that the CRDs are not getting installed on the alpha cluster. Investigating why.
/reopen since I see the Alpha tests failing
/milestone v1.13 /priority important-soon /kind failing-tests
/kind failing-test
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I don't think this is fully resolved. There are still failures on https://k8s- /reopen
Reopened this issue. <detailsIn response to : I don't think this is fully resolved. There are still failures on https://k8s- Instructions for interacting with me using PR comments are available .

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
can you help take a look at these failures? It's testing CSI filesystem volume mode. I think it is unrelated to the crd installation issue that Saad fixed.
I tested with AllAlpha flag and manually created csi crds and confirmed that filesystem volume mode test passed in my environment and also other tests passed there. If I delete csi crds, other tests also fail, so I agree that the issue that Saad fixed is not related. However, above also means that this issue can't be reproduced in my environment.
I1109 14:56:01.141] Nov 9 14:56:01.131: INFO: At 2018-11-09 14:50:54 +0000 UTC - event for security-context--e42e-11e8-9b95-: {attachdetach-controller } FailedAttachVolume: AttachVolume.Attach failed for volume "pvc--e42e-11e8-880b-" : node "bootstrap-e2e-minion-group-3wtg" has no NodeID annotation
Hostpath also seems to be failing because of driver registration issues? Not sure why other alpha tests are passing if that's the case
This test is the only test that uses  to create pod with CSI driver volume and expects the result to be success. One of the pod parameters specified there might affect this failure.
Those 2 failing tests are newly introduced in and started to fail from the beginning. Is that PR blocking 1.13? If not, can we revert and fix later? I have recently set up optional presubmit tests for alpha features at , which will pick up those 2 tests. Currently this failure is blocking
apart from the 2 failing tests, I see which addressed with is also flaky. These failures are now blocking PRs as mentioned above. As we enter code slush today, its very critical to have the PRs get in sooner than later. Can we plz revert the feature and the tests until they are fully resolved and ready to go. Thanks
I prefer disabling the specific tests instead of reverting the entire pr. We will lose important signal for csi
That works for me.

Can you cc me on the PR that disables those tests? When you re-enable the tests, remember to try them out on . :)
Let me consider if we can just skip these test for CSI, instead of reverting the entire patch.
you mean just skipping them in Alpha cluster right?
I was thinking about skipping the below test with the combination of below drivers: [Test] [Testpattern: Dynamic PV (filesystem volmode)] volumeMode[Feature:BlockVolume] should create sc, pod, pv, and pvc, read/write to the pv, and delete all created resources [Driver] csi-hostpath Won't it enough? Or, do we need to skip it only with alpha cluster?
If we just disable the tests alone and not revert the feature, we are just masking the issue. Ideally if we are not able to fix the tests and reenable them the feature should be reverted from 1.13 as well. Are you also actively working on fixing the tests and this skip is just to unblock PRs in the meantime? if so can we timebox by when we can have the tests fixed and re-enabled / the feature reverted? My fear is we might forget the fact that its disabled since there's no active reporting on it.
Actually, there is no feature in either of culprit commits. It tests or intended to improve the way it is tested. Therefore, it must be a test issue or another issue which is found by the test , not an issue introduced by the commits themselves. In the code, there is a clear comment that the test is skipped due to this issue with the reference to this issue#, so this won't be forgot. If you still worry about this issue is not tracked, we might be able to keep this issue open.
IIUC the test that were failing (and are being reverted) were for the new CSI festures going to Alpha in 1.13, right? so even these 2 culprit commits do not contain feature work, it is testing some new code additions to 1.13 and now those will be untested until we re-enable these tests. Is there an ETA to re-enable them, preferably sometime early next week? Yes +1 to keeping this issue open to track the skipped tests.
The two failing tests themselves are not for new alpha features that are in v1.13.

Instead, they are just to test csi driver works well as it works in existing in-tree driver. (And failed because they might be affected by some other alpha features or some of specific configurations in gce.) In addition, ~50 tests for csi driver, so it would be better not to remove all of the tests by reverting the PR, just for the 2 failing tests. I think that just reverting the patch won't help improving quality of k8s. As for ETA to solve the issue, I found difficulty in debuging in my local environment. It seems that the two tests fail in gce environment w/ AllAlpha=true, however, as I've already mentioned, at least csi-hostpath tests in my local environment w/ AllAlpha=true doesn't fail.(And I don't have test environment for gce.) Of course, I will help solving the issue next week. However, I will need someone's help for testing combinations of alpha features in gce environment to narrow down which flag really affects this issue. At least I would like to know the results: w/ MountPropagation=true,CSIPersistentVolume=true,BlockVolume=true,CSIBlockVolume=true all alpha flags except CSIDriverRegistry and CSINodeInfo
/reopen Thanks for the detailed explanation. I think the revert is fine in this case then. I am keeping this issue open however to track and fix the failures in these 2 tests can any of you help answer w.r.t alpha features on gce environment. thanks
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I can help debug when I'm back on Tuesday. In the meantime, I would suggest looking through the failing test logs to try to triage what's going on
Here's one failing run:
I looked into the csi-hostpath failure log. As seen in the for , where the pod which has a csi-hostpath volume is being attached is scheduled, mount volume fails with these errors (1). Related code to this log is (2), so the driver doesn't seem to be registered to . As for the registration code for csiDriver, it is called as (3).

Therefore, will be logged if the driver is triggered for registration. As seen (4), registration is only triggered in , which is different from the one the pod is scheduled. In addition, driver-registrar seems to be deployed only on the node where registration is triggered (5). Is it right behaviour, while registrar is deployed as daemonset? I guess that just this fact seems to cause an issue to make csi-hostpath driver work properly. (Actually, I only tested with 1 node environment, so this might only happen in multiple nodes environment.) <details- (1) (2) Result of grep to check whether registrar is deployed to each node [ for bootstrap-e2e-minion-group-50r7][ for bootstrap-e2e-minion-group-k781][ for bootstrap-e2e-minion-group-z5cg] </details
The hostpath plugin should be deployed as a DaemonSet, so it's strange that it's only getting registered on only one node.
/assign looking into this as well
could you link the PR for skipping [external-gcepd] tests into this issue when you do that as well
skipping gcePD-external is here:
Since gce-cos-alphafeatures is now passing, do we need to keep this issue open? Should it morph into a non-blocking issue for longterm work on a better test?
We have root caused 2 of the 3 failures, which were problems in the test and test conifg. Fixes will be out soon. Still triaging the 3rd failure, which could be a real bug in alpha clusters
I've root caused csi-hostpath failure issue. It just failed to set the node properly in test code side. I will make a PR to fix it.
Or should it be done in your PR too, to avoid modifying twice in the separate PR? Actually, the fix are as below: (Failing to set node before pod creation)
Created for csi-hostpath fix, to prepare for the case that we need a separate PR. We can discuss more on this there, if needed.
/unassign found possible root cause for GCE issue: CSI NodeID annotation is not present on the Node object, could be caused by failed retries on updating Node object.

(apparently when alpha features are open we shouldn't be looking at the Node object anyway except we seem to be, so that might also be a bug). has the most context for the fix /assign
The symptom of the remaining test failure is the test pod fails to create because it fails to attach with the error: . The test succeeds the first time it's run in a cluster, and always fails after that. Two issues caused this error message. the CSI external-attacher tries to fetch the object to get the NodeID, it does so through an informer. The first time the test is run, objects are created for the very first time, and the informer returns the correct object. In subsequent test runs, s from the first run is kept, and for some reason the informer fails to detect the object and returns empty. Making the API call directly works, I'll patch the external-provisioner with this to mitigate the issue, and this is sufficient to get the test to pass. The informer bug will need to be eventually fixed. fails to post the NodeID annotation sometimes, likely due to node status patch not being retried correctly. This is crucial for clusters without feature flag set, and this is the default for 1.13. Will add the appropriate retry mechanism.
Opened to track the informer issue
Are and potential PR for the only 2 fixes being actively worked on for this issue?
Only is needed. We have a workaround for
will you have a followup PR to enable the CSI alpha tests back? or is that already in flight?
also reenables the tests
OK with merged we should see the CSI alpha tests re-enable. to keep an eye out for CI signal
The tests got reenabled 5 days ago, and alpha tests are passing. Closing this issue. Thanks everyone for your rapid turnaround! /close
Closing this issue. <detailsIn response to : The tests got reenabled 5 days ago, and alpha tests are passing. Closing this issue. Thanks everyone for your rapid turnaround! Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The tests are actually running (and passing) in the normal suite now, now that the raw block feature moved to beta.


Question: podname1 := "pod1" podname2 := "pod2" <del> createPodOrFail(f, ns, podname1, jig.Labels, containerPorts1) </del> <ins> createPodOrFail(f, ns, podname1, jig.Labels, containerPorts1, "netexec", "--http-port", strconv.Itoa(port1)) </ins> names[podname1] = true validateEndpointsPortsOrFail(cs, ns, serviceName, portsByPodName{podname1: {port1}}) <del> createPodOrFail(f, ns, podname2, jig.Labels, containerPorts2) </del> <ins> createPodOrFail(f, ns, podname2, jig.Labels, containerPorts2, "netexec", "--http-port", strconv.Itoa(port2)) </ins> names[podname2] = true validateEndpointsPortsOrFail(cs, ns, serviceName, portsByPodName{podname1: {port1}, podname2: {port2}}) <ins> ginkgo.By("Checking if the Service forwards traffic to pods") execPod := e2epod.CreateExecPodOrFail(cs, ns, "execpod", nil) err = jig.CheckServiceReachability(svc, execPod) framework.ExpectNoError(err) </ins> e2epod.DeletePodOrFail(cs, ns, podname1) delete(names, podname1) validateEndpointsPortsOrFail(cs, ns, serviceName, portsByPodName{podname2: {port2}})

The test only validates that the API objects are created E2e tests must verify the behaviour, i.e. that the Service and Endpoints actually work sending traffic. The file is not well organized and there is a lot of duplicate code, one possibility is to use current Services helper has a function to test the reachability that iterates over all service ports However, something that needs to be addressed is that the pods that are backing the service doesn't have any process listening on those ports
/help wanted /sig network /priority important-soon /triage accepted
I think that there are more tests that doesn't verify the reachability , those should be modified too /help-wanted
/help-wanted
/assign
/triage accepted
/assign /unassign I will take on this since it was more difficult than I expected, and is a Conformance test.. I assigned you to , that is similar to this, but creating a new test from scratch