Question: }) It("should create and update matching ingresses in underlying clusters", func() { <del> ingress := createIngressOrFail(f.FederationClientset_1_4, f.Namespace.Name) </del> <ins> ingress := createIngressOrFail(f.FederationClientset_1_4, ns) </ins> defer func() { // Cleanup <del> By(fmt.Sprintf("Deleting ingress %q in namespace %q", ingress.Name, f.Namespace.Name)) err := f.FederationClientset_1_4.Ingresses(f.Namespace.Name).Delete(ingress.Name, &api.DeleteOptions{}) framework.ExpectNoError(err, "Error deleting ingress %q in namespace %q", ingress.Name, f.Namespace.Name) </del> <ins> By(fmt.Sprintf("Deleting ingress %q in namespace %q", ingress.Name, ns)) err := f.FederationClientset_1_4.Ingresses(ns).Delete(ingress.Name, &api.DeleteOptions{}) framework.ExpectNoError(err, "Error deleting ingress %q in namespace %q", ingress.Name, ns) </ins> }() // wait for ingress shards being created <del> waitForIngressShardsOrFail(f.Namespace.Name, ingress, clusters) ingress = updateIngressOrFail(f.FederationClientset_1_4, f.Namespace.Name) waitForIngressShardsUpdatedOrFail(f.Namespace.Name, ingress, clusters) </del> <ins> waitForIngressShardsOrFail(ns, ingress, clusters) ingress = updateIngressOrFail(f.FederationClientset_1_4, ns) waitForIngressShardsUpdatedOrFail(ns, ingress, clusters) </ins> }) var _ = Describe("Ingress connectivity and DNS", func() {

The addition of the federated namespace adminssion controllers broke this test. It's an easy fix. e2e test needs to create the required namespace before trying to create things in it. FYI. has volunteered to fix this. https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e

suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: []

Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
is fejta@ the right owner?
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated

ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated

Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in

underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e suite}
https://k8s- Failed: [] Federated ingresses [Feature:Federation] Federated Ingresses should create and update matching ingresses in underlying clusters {Kubernetes e2e


Question: for _, test := range testCases { host := "1.2.3.4" container := api.Container{ <del> Ports: []api.ContainerPort{{Name: "found", HostPort: 93}}, </del> <ins> Ports: []api.ContainerPort{{Name: "found", ContainerPort: 93}}, </ins> LivenessProbe: &api.Probe{ Handler: api.Handler{ TCPSocket: test.probe,

When using probe, define the port by number, it works. But if the port is defined by name like below, it doesn't work. it seems should return other than .


Question: } framework.ExpectNoError(framework.RunRC(controllerRcConfig)) <del> // Make sure endpoints are propagated. // TODO(piosz): replace sleep with endpoints watch. time.Sleep(10 * time.Second) </del> <ins> // Wait for endpoints to propagate for the controller service. framework.ExpectNoError(framework.WaitForServiceEndpointsNum( c, ns, controllerName, 1, startServiceInterval, startServiceTimeout)) </ins> }

https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite} Happened on a presubmit run in . Previous issues for this test:
failed in https://k8s-
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite} Happened on a presubmit run in .
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed:

https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.

Any updates on this issue If you could you change the milestone to 1.5 and add a label for the responsible team that would be a great help as we triage flakes for the 1.5 release. Thanks!
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
There is a problem with fetching images on GKE. It seems that pod is unable to fetch image for more than 2 minutes. It is strange, as AFAIK the image should be pre-fetched by image-puller:
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
, all issues must be labeled either release blocker or non release blocking by end of day 18 November 2016 PST. If you are not the correct owner please identify the correct owner and add them to this thread for assignment. Thanks for your help. cc:
WRT https://k8s- There was a problem w/removing namespace. This is a duplicate of and is not related to HPA.

https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
Could you please help triage this issue as blocking or non-blocking for 1.5 (or help find someone who can). If there is a real issue here, we need to find out ASAP, or else we risk the 1.5 release date.
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
WRT https://k8s- build: kubelet logs for pod rc-light-qyowb : Kubelet is not starting pod for 2 minutes, reason unknown. The same problem is in issue Can you take a look or assign someone from kubelet team?
I suspect that the logging level is set to in GKE tests and so most of the useful logs are not available. Let me figure out a means to bump the logging level to further debug this issue. you might still have to triage the non v1.4 release failures to address concern .
how can we update the environment variables for GKE test clusters? For OSS, we up the kubelet logging for test clusters
I still don't agree with test environmental changes, which is one of the reasons we've never made this a priority. A GKE test cluster looks the same as a GKE prod cluster. How would you debug this in production? Always ask that, and you come back to test == prod.

cc
In my opinion it is release blocker: sometimes pods are not started
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
The pod startup failure you posted in is for release 1.4 branch. Why do you think it is a release blocker? We need to triage other failures on HEAD and/or 1.5 branch which could be release blockers. If the intent is to keep prod and test in sync, then let's up the logging level in prod as well. When issues happen in prod, we might need those missing log lines anyways. I suspect that we might have to have GCI rotate logs more aggressively as a side effect.
Why is the argument to increase the log level rather than making sure that all logs necessary for debugging production issues are at level 2 or above? Levels 3/4 shouldn't be necessary to figure out why a key control loop isn't behaving correctly.
What you suggest is a good fix for the future. We need to reexamine logging levels across kubelet. However, that might not happen in the next few days. How do we make progress now?
Would isolating logging level changes to specific test suites be acceptable? That way, most of our tests would match production setup.
That's what I'd suggested on the other (internal) code review.
Do we have consensus on how to get past this as a release-blocker?
We are waiting on to give a go ahead from the release perspective. The kubelet failures that and I have been discussing about have not been on v1.5 AFAIK.
Sorry, I didn't notice that this problem is on 1.4 branch. If so, the issue is not release-blocker. So, I'm removing the label.
Ack.

Marking as non-release-blocker
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}
Is it appropriate to move this to the next milestone? (and remove the non-release-blocker tag as well)
PR should be cherry-picked to 1.5 This is not a release-blocker as it affects e2e tests only.
https://k8s- Failed: [] [HPA] Horizontal pod autoscaling (scale resource: CPU) [] ReplicationController light Should scale from 1 pod to 2 pods {Kubernetes e2e suite}


Question: // Stateless functions <ins> const ( pvcKeyIndex string = "pvc-key-index" ) // indexByPVCKey returns PVC keys for given pod func indexByPVCKey(obj interface{}) ([]string, error) { pod, ok := obj.(*v1.Pod) if !ok { return []string{}, nil } if util.IsPodTerminated(pod, pod.Status) { return []string{}, nil } keys := []string{} for _, podVolume := range pod.Spec.Volumes { if pvcSource := podVolume.VolumeSource.PersistentVolumeClaim; pvcSource != nil { keys = append(keys, fmt.Sprintf("%s/%s", pod.Namespace, pvcSource.ClaimName)) } } return keys, nil } </ins> func getClaimStatusForLogging(claim *v1.PersistentVolumeClaim) string { bound := metav1.HasAnnotation(claim.ObjectMeta, pvutil.AnnBindCompleted) boundByController := metav1.HasAnnotation(claim.ObjectMeta, pvutil.AnnBoundByController)

What happened: With a StorageClass with , it's not clear why a PVC is not bound when there is no available PV. On Pod, there is a clear event what's wrong: On the PVC side we get only this event: Note that the message says "waiting for first consumer'. But the pod already exists. The reason why it's not bound yet is that the StorageClass has no dynamic provisioning configured () and there are no free PVs to satisfy the claim. What you expected to happen: An event "1 node(s) didn't find available persistent volumes to bind" on the PVC.
/sig scheduling /sig storage
I tried to fix this in VolumeBinding predicate, The predicate is called for each pod - node combination, so it would generate event for each node. Which is too much and it's even wrong when there is a node that has an available PV. We need to evaluate all nodes and then, if the pod is still unschedulable, distribute events to all unbound PVCs of that pod. Scheduler does not offer any way how to track unbound PVCs at that stage - whole volume logic is hidden in VolumeBinding predicate. One option would be to extend  struct to contain also list of objects where to send events (except the pod itself): Scheduler would then send all failed scheduling events it sends to a pod to all the , without any interpretation. is used to manage duplicates easily - each object (its UID) can be in the map only once. Simple array would work too, but then it needs extra work in removing duplicates. Not sure how easy it is to detect duplicate instances. is extensible to allow any object type. predicate would put all unbound PVCs there. does not need any special handling of each PVC. Message "2 node(s) didn't find available persistent volumes to bind" sounds just right in case a pod uses multiple PVCs, each of them might be bind-able on different nodes (node A may satisfy PVC1, but not PVC2, while node B may satisfy PVC2 only). what do you think about such design?
We used to have extension point, which I believe is the right moment for recording the event for unbounds PVCs. But recently it's been refactored so doesn't fit. Do you think this case can be a good example of using as an informational extension point?

Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale wdyt?
I have some other concerns. What if the pod is deleted? Should the scheduler send events that PVC is not in volume binding? The PV controller must be updated to stop emit event in this scenario. It might be simpler to solve on the PV controller side. In this function, we can check if the PVC is being used by a pod. If it is, we can send event like , then people can go to check the progress of pod "foo/bar". If the pod "foo/bar" is deleted, the PV controller can emit "waiting for first consumer ..." event again. What do you think? cc
Sounds better than it's now. My only issue may be performance - PV controller has to check if a pod exists, i.e. list all pods in the same namespace as PVC being reconciled. I am probably too worried.
current attach detach controller has a custom indexer to index pods by PVC keys to avoid iterating all pods every time to find pods which reference given PVC. And there are other places we need to check if a PVC is used by a pod or not from pod informer, e.g. pvcprotectioncontroller pvcontroller we can try to share this custom indexer among these volume controllers in the future. This will improve the overall performance of the IMO. What do you think
Yes, using the indexer looks like a good idea.
/remove-sig scheduling I also agree that solving in PVC side seems like a better idea. Thanks
/assign unbound claims are reconciled in the PV controller every 15 seconds (default), we may have bad worst-case performance (e.g. too many pods in the same namespace). we can add an indexer in the PV controller locally to solve this issue first (also optimize ), then find a way to in manager to improve the overall performance.


Question: responses: 200: body: <del> application/json: example: !include examples/controller-list.json </del> <ins> example: !include examples/controller-list.json </ins> post: description: Create a new controller. currentState is ignored if present. body: <del> json/application: schema: !include doc/controller-schema.json example: !include examples/controller.json </del> <ins> schema: !include doc/controller-schema.json example: !include examples/controller.json </ins> /{controllerId}: get:

Starting a large data set (32gb allocated, 4gb resident) in the api server causes about 7.5gb allocations of . The tree for that shows: This appears to be due to openapi aggregation. If there is an easy fix in 1.7 it might be worth it. Otherwise this should be addressed for 1.8.
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
This is all API server startup - for small clusters it's about 20-60% of startup time
Most of those are that is the library we used to convert our spec to proto. The key is this line: Do you have any idea how we can get a better space time/complexity here?
I have a PR for that now.
Fixed in
Even without aggregation, starting and serving OpenAPI docs was a huge chunk of startup time. The linked change makes it disappear.
This should not be related to aggregation at all (as far as I can tell from that tree). Thanks for the fix
Thanks, it's in gnostic master now.


Question: } } <del> if s.CloudProvider == kubeletconfigv1alpha1.AutoDetectCloudProvider { glog.Warning("--cloud-provider=auto-detect is deprecated. The desired cloud provider should be set explicitly") } </del> if kubeDeps.Cloud == nil { <del> if !cloudprovider.IsExternal(s.CloudProvider) && s.CloudProvider != kubeletconfigv1alpha1.AutoDetectCloudProvider { </del> <ins> if !cloudprovider.IsExternal(s.CloudProvider) { </ins> cloud, err := cloudprovider.InitCloudProvider(s.CloudProvider, s.CloudConfigFile) if err != nil { return err

/kind feature What happened: Continuing the discussion from , we should deprecate (or remove since its alpha) the option to set . See comment and for more details. related issues: related kubernetes-dev thread: !topic/kubernetes-dev/UAxwa2inbTA EDIT: Deprecation plan: [x] v1.8 - deprecate auto-detect cloud provider feature in kubelet [ ] v1.9 - set no cloud provider as the default in kubelet but still allow opt in for auto-detect [ ] v1.10 - completely remove the option to use
/sig cluster-lifecycle
/area cloudprovider
cc
/assign wlan0
cc - Wanted to ask you if you had any thoughts for/against this issue?
/assign andrewsykim As discussed in our meeting I'll be following up with sig-node to see if we can deprecate this as it's going against what we want for external cloud providers
FYI
Would have been nice to keep this issue open to track progress in the next releases
Indeed, should be kept open. Next task is now targeting v1.9
Thanks I edited the issue description to include the deprecation plan going forward. Open to changes on the deprecation plan if we think it's too aggressive.
Great, I think this is a good timeline. We need to make sure we communicate it very well though can you add this to the draft release notes in kubernetes/features please?
Looks like you beat me to it , gave it a review.
hehe, yes, it at the same time as everything else ;)
Can you create a PR for this soon to get this early into v1.9 please?
[MILESTONENOTIFIER] Milestone Issue Current <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Escalate to the issue owners; move out of the milestone after 1 attempt. : Adding tests, refactoring, fixing old bugs. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
i took the liberty to move this along (filed ). Hope that's ok.
Thanks appreciate it. I think there were some discussions in the mailing list thread about v1.9 being too soon.

I think it largely depends on how many users actually take advantage of the auto-detect feature. It's not many from what I've seen but we should follow up and be sure.
Any recent discussion about that? I haven't seen any activity recently but I might just be subject to email overflow. I hope we can get this into v1.9 cc
I didn't follow any discussion related to this topic, but thought this is right move to decouple cloudprovider components from the core system. If I remembered correctly, only cloudproviders for GCE, AWS and Azura depend on the feature, which can be easily override. Any concerns for v1.9?
yep, GCE/AWS/Azure -
Please look at and give your LGTM
Done. Just better document in 1.9 release notes. LGTM
Thank you!
Reopening for the removal of the option in v1.10
please move this to 1.10 milestone


Question: import ( "fmt" "path" <ins> "time" </ins> . "github.com/onsi/ginkgo" "k8s.io/api/core/v1"

<!-- Please only use this template for submitting enhancement requests --What would you like to be : This is a place holder to track Conformance Coverage on the Topic. This issue should remain open until all related work is accomplished in the k/k repo This issue will contain analysis of the coverage. Existing tests and additional tests requested with links to those issues/PR's Why is this needed: Conformance coverage for CSI, etc. /area conformance /assign /cc
/sig testing


Question: } if nc.runTaintManager { <ins> nc.taintManager = scheduler.NewNoExecuteTaintManager(kubeClient) </ins> nodeInformer.Informer().AddEventHandler(cache.ResourceEventHandlerFuncs{ AddFunc: util.CreateAddNodeHandler(func(node *v1.Node) error { nc.taintManager.NodeUpdated(nil, node)

Just saw this while running kube master locally: Yet to investigate more closely and check which PR could have introduced this. cc
/sig node
any way to repro?
I think I can fix this error.
what is the bug?
I think when call nc.taintManager has not been inited
duplicate of , fixed by
I found does not have related fix.
It fixes the issue systemically by not starting the informers until all controllers have finished setting up
Ok by more carefully read, I see.
fixed by


Question: deps = [ "//cmd/kube-apiserver/app/options:go_default_library", "//staging/src/k8s.io/api/core/v1:go_default_library", <ins> "//staging/src/k8s.io/apiextensions-apiserver/pkg/apis/apiextensions/v1:go_default_library", "//staging/src/k8s.io/apiextensions-apiserver/pkg/client/clientset/clientset/typed/apiextensions/v1:go_default_library", "//staging/src/k8s.io/apiextensions-apiserver/pkg/controller/finalizer:go_default_library", </ins> "//staging/src/k8s.io/apimachinery/pkg/api/equality:go_default_library", <ins> "//staging/src/k8s.io/apimachinery/pkg/api/errors:go_default_library", </ins> "//staging/src/k8s.io/apimachinery/pkg/api/meta:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/apis/meta/v1:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/apis/meta/v1/unstructured:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/runtime/schema:go_default_library", <ins> "//staging/src/k8s.io/apimachinery/pkg/types:go_default_library", </ins> "//staging/src/k8s.io/apimachinery/pkg/util/diff:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/util/sets:go_default_library", <ins> "//staging/src/k8s.io/apimachinery/pkg/util/wait:go_default_library", </ins> "//staging/src/k8s.io/apimachinery/pkg/watch:go_default_library", "//staging/src/k8s.io/client-go/dynamic:go_default_library", <ins> "//staging/src/k8s.io/kube-aggregator/pkg/apis/apiregistration/v1:go_default_library", "//staging/src/k8s.io/kube-aggregator/pkg/client/clientset_generated/clientset/typed/apiregistration/v1:go_default_library", </ins> "//test/integration/framework:go_default_library", "//vendor/github.com/coreos/etcd/clientv3:go_default_library", ],

What happened: Created a CRD: Created another CRD defining an overlapping type with CustomResourceDefinition: Verify both exist: Deleted the one overlapping with the in-tree CRD type: Verify only the overlapping CRD remains: and is stuck deleting: What you expected to happen: Creation and deletion of the overlapping type CRD would have no effect on the in-tree data. Anything else we need to know?: The only types in the kube-apiserver (currently) that store data under a group prefix are: APIService () CustomResourceDefinition () Custom resources () When a CRD is deleted, all instances of the type it defines are first removed. That means that defining and deleting a CustomResourceDefinition that defines an overlapping APIService or CustomResourceDefinition type will remove existing APIService or CustomResourceDefinition data. We can special-case these two resources in the CRD deletion code, but we need a more robust solution before CRD v1. /cc /sig api-machinery /priority important-soon /area custom-resources
/sub
Is anyone working on this? I'll like to take a stab at this.
Not currently, no. Great. I'd suggest discussion a potential solution here before diving into code. The two main aspects are: -time - Should we forbid creating CRDs that overlap with built-in types? -time - How do we detect a CRD overlapping with a built-in type and skip the etcd cleanup for it
I think we should forbid creating CRDs that overlap (option 1).
That can help, but is not sufficient. Imagine: n forbids creating overlapping CRDs on version n creates MyType, which is allowed because that is not a built-in type upgrades to version n+1 which adds MyType as a built-in type deletes CRD What should happen?
For that, we may need delete-time detection of overlapping types as well (option 2), which may make option 1 redundant. Do you have any advice for detecting overlapping CRDs?
for now, just delete-time detection in


Question: rm -fr pki mkdir -p pki openssl genrsa -out pki/ca.key 2048 <del> openssl req -new -x509 -days 3650 -key pki/ca.key -subj "/CN=pod-security-webhook-ca-$(date +%s)" -out pki/ca.crt </del> <ins> openssl req -sha256 -new -x509 -days 3650 -key pki/ca.key -subj "/CN=pod-security-webhook-ca-$(shell date +%s)" -out pki/ca.crt </ins> <del> openssl req -newkey rsa:2048 -nodes -keyout pki/tls.key -subj "/CN=webhook.pod-security-webhook.svc" -out pki/tls.csr </del> <ins> openssl req -sha256 -newkey rsa:2048 -nodes -keyout pki/tls.key -subj "/CN=webhook.pod-security-webhook.svc" -out pki/tls.csr </ins> echo "subjectAltName=DNS:webhook.pod-security-webhook.svc" > pki/extensions.txt echo "extendedKeyUsage=serverAuth" >> pki/extensions.txt <del> openssl x509 -req -extfile pki/extensions.txt -days 730 -in pki/tls.csr -CA pki/ca.crt -CAkey pki/ca.key -CAcreateserial -out pki/tls.crt </del> <ins> openssl x509 -sha256 -req -extfile pki/extensions.txt -days 730 -in pki/tls.csr -CA pki/ca.crt -CAkey pki/ca.key -CAcreateserial -out pki/tls.crt </ins> # Publishes the PodSecurity webhook Docker image to the configured registry. push:

The existing validating webhook PKI configurations fail on Kubernetes v1.24 due to Pod security webhook information example: Example API server error: Additionally: Date is not being interpolated in CA cert CN : "pod-security-webhook-ca-" CA Cert example information:
/transfer kubernetes /sig auth /kind bug
/help
This request has been marked as needing help from a contributor. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage accepted
Fixed in
Thank you


Question: go k.ListenAndServe(kubeCfg, kubeDeps.TLSOptions, kubeDeps.Auth, kubeDeps.TracerProvider) } if kubeCfg.ReadOnlyPort > 0 { <del> go k.ListenAndServeReadOnly(netutils.ParseIPSloppy(kubeCfg.Address), uint(kubeCfg.ReadOnlyPort)) </del> <ins> go k.ListenAndServeReadOnly(netutils.ParseIPSloppy(kubeCfg.Address), uint(kubeCfg.ReadOnlyPort), kubeDeps.TracerProvider) </ins> } go k.ListenAndServePodResources() }

Follow-up to Discussed in As part of kubelet tracing beta, we should add tracing support to the kubelet's read-only endpoint, which currently does not have tracing. It should use WithPublicEndpoint to prevent callers from influencing sampling decisions. Part of
/sig node /sig instrumentation
/triage accepted /assign
can i help you with this issue?
I don't think we'll be able to start with this until merges
Ok, I'm waiting to see if I can help you. Thank you
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale is merged now, so this can be worked on.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
Could I help with this issue?
Sure. See We pass a NoOp Tracer provider to disable tracing for the endpoint. Instead, we need to pass the tracerprovider, but make sure that the eventual uses withPublicEndpoint. Looking closer, i'm not sure its currently possible to use WithPublicEndpoint with the otelrestful package... doesn't contain that option (unlike otelhttp: ), so we may need to add that upstream first.

Another thing to keep in mind is that updating otel libraries is currently blocked by (see my attempt in )
I gave the whole thing another try. At least locally, the dependencies seem now to be able to be updated. :raccoon: Could you maybe have a look at it :)
He's on leave for the time being.
oh, can you provide some feedback or point me to the right people?
I'm back now. Feel free to assign me to PRs related to this, and I'll review
Welcome back :) Could you re-run the CI ? Its just the dependencies. The PR itself shouldn't be that much afterwards.
otel is now up-to-date enough. here:
Hi, I would like to ask about one trouble that caused by WithPublicEndpoint. Aggregated APIServer that I customized uses kubernetes's RecommendedConfig. Due to the fact that WithPublicEndpoint is enabled, it can't be tracked by the full trace for its span-link model. And I think child-parent span model is more appropriate. So, could you please give me some advice? Looking forward to your reply.
thanks. We know about this, and are tracking it in


Question: }) // This test verifies we don't allow scheduling of pods in a way that sum of <del> // limits of pods is greater than machines capacity. </del> <ins> // resource requests of pods is greater than machines capacity. </ins> // It assumes that cluster add-on pods stay stable and cannot be run in parallel // with any other test that touches Nodes or Pods. // It is so because we need to have precise control on what's running in the cluster.

What happened: I am questioning validity of the test. Comments under of say: The test scenario described as it is sounds reasonable. The only nitpick I have is the very first sentence saying "". Pod's resource limits are ignored in predicates. They take effect in the scoring phase when priorities are computed. Also, priorities are computed only when there are at least two candidate nodes. So after revisiting the step number 5 it's impossible to make sure the additional pod does or does not get scheduled when setting the resource limits. There are either no candidate nodes or one. In which case the limits are ignored and the test passes in the first case and fails in the second. In case there are at least two candidate nodes, one of them will always get picked. In which case the additional pod gets scheduled (no matter what the resource limits field says). In the framework: resource limits are accessed through method [1] is used to perform pre-scoring [2] in the scoring phase score is computed through method [3] The test is alternating between passing and failing state. It passes when predicates give empty list of candidate nodes and failing when the list of candidate nodes is non-empty. Thus, making the pod resource limits ineffective. [1] [2] [3] What you expected to happen: Have another node schedule the additional pod validating that all saturated nodes (with scheduled) have lower score (0 in case of plugin). Or, invalidate the test by removing it. Can anyone check the test validity and confirm/disprove my statement about ineffectiveness of the resource limits? How to reproduce it (as minimally and precisely as possible): Make sure there is at least one node with non-zero free cpu resource before running the step 5. Anything else we need to know?: Environment: Kubernetes version (use ): master HEAD Cloud provider or hardware configuration: N/A OS (e.g: ): N/A Kernel (e.g. ): N/A Install tools: N/A Network plugin and version (if this is a network-related bug): N/A Others: N/A
cc
I have noticed you refactored the test in Can you still recall the original intent of the test? I can't grasp how the limits are actually taken into account.
I agree, I think the test is wrong, the scheduler doesn't make fit decisions on resource limits. In the test, should have set. do you mind fixing the test and update the test comment please?
On it /assign


Question: } type FakeEtcdClient struct { <ins> condWatchCompleted *sync.Cond condLock           sync.Mutex </ins> Data        map[string]EtcdResponseWithError DeletedKeys []string Err         error

Detected by
Create a Pod as follows: Name: jenkins Using image: jenkins In a new Kubenetes namespace named website-frontend kubectl create ns website-frontend apiVersion: v1 kind: Pod metadata: name: jenkins namespace: website-frontend spec: containers: name: jenkins image: jenkins kubectl create -f


Question: <ins> // +build windows /* Copyright 2021 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package app import ( "fmt" "os/user" "golang.org/x/sys/windows" ) func isAdmin() (bool, error) { // Get current user u, err := user.Current() if err != nil { return false, fmt.Errorf("Error retrieving current user: %s", err) } // Get IDs of group user is a member of ids, err := u.GroupIds() if err != nil { return false, fmt.Errorf("Error retrieving group ids: %s", err) } // Check for existence of BUILTINADMINISTRATORS group id for i := range ids { // BUILTINADMINISTRATORS if "S-1-5-32-544" == ids[i] { return true, nil } } return false, nil } func checkPermissions() error { //https://github.com/golang/go/issues/28804#issuecomment-505326268 var sid *windows.SID var userIsAdmin bool // https://docs.microsoft.com/en-us/windows/desktop/api/securitybaseapi/nf-securitybaseapi-checktokenmembership err := windows.AllocateAndInitializeSid( &windows.SECURITY_NT_AUTHORITY, 2, windows.SECURITY_BUILTIN_DOMAIN_RID, windows.DOMAIN_ALIAS_RID_ADMINS, 0, 0, 0, 0, 0, 0, &sid) if err != nil { return fmt.Errorf("Error while checking for elevated permissions: %s", err) } //We must free the sid to prevent security token leaks defer windows.FreeSid(sid) token := windows.Token(0) userIsAdmin, err = isAdmin() if err != nil { return fmt.Errorf("Error while checking admin group membership: %s", err) } member, err := token.IsMember(sid) if err != nil { return fmt.Errorf("Error while checking for elevated permissions: %s", err) } if !member { return fmt.Errorf("kubelet needs to run with administrator permissions. Run as admin is: %t, User in admin group: %t", member, userIsAdmin) } return nil } </ins>

What happened: follow on to Need to comment and avoid checking UID on windows. requires some research into what we should be checking so that we can add appropriate TODO. What you expected to happen: Lets also check this at the call site, Anything else we need to know?: Environment: Kubernetes version (use ): master OS (e.g: ): windows
/assign
/assign
/good-first-issue
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig windows
Hi are you working on this issue already? If not I'd like to tackle this issue :)
Hi yeah I am already working on this :)
Started branch for this here
thanks
Hey guys is there anything I can do here?
If you want to test perks patch that would be great ?
That sounds great Where do I get started?
Hey ! is there anything I can help with? and how?
I dont think there is anything left to do anything more that you think we need to do to get this merged?
Tested this with containerd as well, I think its good to test
/triage accepted


Question: defaultVersion, kind, err := api.RESTMapper.VersionAndKindForResource(a.GetResource()) if err != nil { <del> return admission.NewForbidden(a, err) </del> <ins> return errors.NewInternalError(err) </ins> } mapping, err := api.RESTMapper.RESTMapping(kind, defaultVersion) if err != nil { <del> return admission.NewForbidden(a, err) </del> <ins> return errors.NewInternalError(err) </ins> } if mapping.Scope.Name() != meta.RESTScopeNameNamespace { return nil

Currently the admission controller returns a forbidden (403) error if the namespace doesn't exist. We should return a 404 error instead.


Question: "k8s.io/kubernetes/pkg/scheduler/profile" st "k8s.io/kubernetes/pkg/scheduler/testing" tf "k8s.io/kubernetes/pkg/scheduler/testing/framework" <del> testingclock "k8s.io/utils/clock/testing" "k8s.io/utils/ptr" </del> <ins> utiltesting "k8s.io/kubernetes/test/utils/ktesting" </ins> ) func TestSchedulerCreation(t *testing.T) {

/kind failing-test /kind flake First seen in an unrelated PR, but also occurs in periodic jobs. /sig scheduling
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign This feature is reverted once, and I fixed the flake, I'm cherry-pick that back again.


Question: package azure_dd import ( <ins> "fmt" </ins> "path" "strconv" libstrings "strings"

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: In a customer , customer is unable to use azure disk in StatefulSet since /dev/sd changed after detach/attach disk. Currently we are using to reference the data disk which could be changed in some condition, see code: What you expected to happen: How to reproduce it (as minimally and precisely as possible): 5 StatefulSet with 1 replica and 1 azure disk mount each 1 StatefulSet and recreate the same StatefulSet would be azure disks could not be usable since /dev/sd changed Update: It seems that on azure china, this issue could repro very easily. Anything else we need to know?: I have checked the official doc: Environment: Kubernetes version (use ): from v1.6.0 to v1.10.0 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: /sig azure /assign I am working on a fix for this issue, would use instead of
We changed how we find disks on nodes quite some time back. What server version is this?
yes, I have already checked the code, the point is we are still using (code is ) to "mount -bind" k8s path, while could be changed when VM is attach/detaching data disks, see And I have also checked related AWS, GCE code, they are using other than ,And I have also checked related AWS, GCE code, they are using other than , see This issue could be stably repro on Azure China, while I could not repro that on other region up to now. I have already got a fix for this, still in testing now.
also add for awareness, I am working on a fix to use other than .
I would not use "/dev/disk/by-id" instead utilize the special UDEV rules we have created for our VMs -- They have the advantage that they create symlinks for all of the disks attached to the VM and the LUN assigned to a disk remains the same. The sysmlinks you find under /dev/disk/azure. For a master-node you will see a layout like the following.

In scsi1 you find the additonal (data) disks A newly assigned disk is goign to appear as lun1,lun2,... Since we are not able to guaranty the order in which disks get assigned to a VM I would go and base the code on this already created layout
thanks for the check. So do you mean we use , instead? actually in Azure VM with UDEV rules created, they have no difference, they link to the same /dev/sd* (see below layout). While I also got some customer cases(e.g. coreos, see ), they don't have such UDEV rules created, which means does not exist, I would say would be more common, every linux distro have that path. From debugging perspective, the path is more intuitive.
BTW, I have a azure disk debugging doc: which uses , it's really intuitive for debugging perpurse.
it is right that at the end we point to /dev/sdi like in the sample above the advantage we have with the lun0 is that it acts as an abstraction layer where we can be sure that an application is always utilizing the disk with lun0, independent what actuall device name gets assigned to it. So at the moment it is ../../../sdi and after a reboot it could become ../../../sdk. But thanks to these UDEV rules /dev/disk/azure/scsi1/lun0 will always point to the right device behind it. I also would favour the usage of the lun over the disk-id if from a maintanance/support point of view as we can directly correlate what lun belongs to what disk. Otherwise I have at first fo to the VM directly to find out what disk might have an issue Regarding the centos issue, if we have the waagent installed on it then those new UDEV rules should be available as well. So it wonders me that we do not have them working well. At least I had not seen any customer SR who had reported an issue with those rules specifically on CentOS
It's not centos, it's coreos issue, see If we use , then we would heavily depends on UDEV rules.

I agree that one depends on these special UDEV rules in this case , but on each Azure VM the agent is to be installed anyway, so that is background why I suggested to use the LUNs
I think proposal is the cleanest and safest. However we will have to tell all Azure users that they have to have azure UDEV rules configured in nodes, even if WAAGENT is not installed.
I did hit the issue, using an ACS based Kubernetes cluster running 1.7.7. We are already in the process of moving over to AKS based clusters, so was simply hoping that there would be some escalated effort to bring 1.8.7 into AKS given that the azure-disk provisioner is the default StorageClass.Not sure if you are looking for any more debug info at this time, but I still have the affected pod active and should be able to get you some more information, so just let me know. that's great, I would like to take a look to make sure it's that issue, could you follow this to collect the data.
Sadly the node became NotReady, and the cluster redeployed the pod to a separate node, which has rectified the issue. We have been seeing the issue with far frequency, so I will make sure to gather the data next time it pops up.
That said, the reason I was confident it was this issue previously is that when I exec'd into the shell, I got the "Input / Output error" upon trying to "ls" the mount point. Furthermore, attempting to re-mount that point gave an error to the extent of "Special device /dev/sde does not exist"
Could you recall the VM had restarted before you hit that disk issue? Or just after attach/detach disk, that issue come up?


Question: klog.Errorf("Failed to create govmomi client. err: %+v", err) return err } <ins> setVCenterInfoMetric(connection) </ins> return nil }

<!-- Please only use this template for submitting enhancement requests --What would you like to be : We need reporting of version to be shown in the metrics. Why is this needed: The goal behind this is to add some info about the underlying infrastructure for the kubernetes cluster. Some customers were experiencing issues during the cluster upgrades, but nothing in the upgrades themselves or kubernetes code for provider seemingly could cause those. The issue is storage related, and happening on a variety of releases with a significant delay from any other underlying infrastructure change, to which the kubernetes cluster has no relation and a way to know about. So knowing used hostnames, and versions associated with it over time would be extremely beneficial for debugging in similar circumstances. Some of the use cases we face: 1) A is enabled in a customer environment, and so there occurs a periodic automated vMotion for VMs backing kuberentes . We'd like to know on which host the actual kubernetes was located at any moment, and if any vCenter version transitions happened over time, new hosts /removed/renamed.
/sig cloud-provider /wg provider-vsphere
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig cloud-provider /wg provider-vsphere Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/area provider/vmware


Question: expectDelete: false, }, { <del> description:  "terminated pod with emptyDir", node:         node, expected:     cordonedNode, pods:         []corev1.Pod{emptydirTerminatedPod}, rcs:          []corev1.ReplicationController{rc}, args:         []string{"node"}, expectFatal:  false, expectDelete: true, </del> <ins> description:           "terminated pod with emptyDir", node:                  node, expected:              cordonedNode, pods:                  []corev1.Pod{emptydirTerminatedPod}, rcs:                   []corev1.ReplicationController{rc}, args:                  []string{"node"}, expectFatal:           false, expectDelete:          true, expectOutputToContain: "node/node drained", </ins> }, { <del> description:  "pod with EmptyDir and --delete-emptydir-data", node:         node, expected:     cordonedNode, pods:         []corev1.Pod{emptydirPod}, args:         []string{"node", "--force", "--delete-emptydir-data=true"}, expectFatal:  false, expectDelete: true, </del> <ins> description:           "pod with EmptyDir and --delete-emptydir-data", node:                  node, expected:              cordonedNode, pods:                  []corev1.Pod{emptydirPod}, args:                  []string{"node", "--force", "--delete-emptydir-data=true"}, expectFatal:           false, expectDelete:          true, expectOutputToContain: "node/node drained", </ins> }, { <del> description:  "empty node", node:         node, expected:     cordonedNode, pods:         []corev1.Pod{}, rcs:          []corev1.ReplicationController{rc}, args:         []string{"node"}, expectFatal:  false, expectDelete: false, </del> <ins> description:           "empty node", node:                  node, expected:              cordonedNode, pods:                  []corev1.Pod{}, rcs:                   []corev1.ReplicationController{rc}, args:                  []string{"node"}, expectFatal:           false, expectDelete:          false, expectOutputToContain: "node/node drained", </ins> }, { description:                "fail to list pods",

When draining a node, using kubectl drain [NODE], the output resembles this: Expected output should indicate that the node was drained, for example: kubectl drain [NODE] No response <details</details<detailsCan reproduce locally using and printing the output </details<details</details<details</details<details</details<details</details
/sig cli
/triage accepted


Question: return fmt.Sprintf("%dm", minutes) } else if hours := int(d.Hours()); hours < 24 { return fmt.Sprintf("%dh", hours) <del> } else if hours < 24*364 { </del> <ins> } else if hours < 24*365 { </ins> return fmt.Sprintf("%dd", hours/24) } return fmt.Sprintf("%dy", int(d.Hours()/24/365))

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT Kubernetes version (use ): kubectl version Client Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.5", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-03-22T00:23:49Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.5", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-03-22T00:17:51Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"} What happened: kubectl get namespace $NS NAME STATUS AGE $NS Active 0y Show 0y, a kubectl get namespace $NS-o=yaml apiVersion: v1 kind: Namespace metadata: creationTimestamp: 2016-04-18T07:37:18Z shows that its almost a year old now. What you expected to happen: Show 1y or 12m, 0y is a bit confusing. How to reproduce it (as minimally and precisely as possible): Anything else we need to know:
let me take a look at it
A tiny issue in the printers, fixing it now


Question: const nodeWithImpairedVolumes = "NodeWithImpairedVolumes" const ( <del> // These constants help to identify if a node is a master or a minion labelKeyNodeRole    = "kubernetes.io/role" nodeMasterRole      = "master" nodeMinionRole      = "node" labelKeyNodeMaster  = "node-role.kubernetes.io/master" labelKeyNodeCompute = "node-role.kubernetes.io/compute" labelKeyNodeMinion  = "node-role.kubernetes.io/node" ) const ( </del> // volumeAttachmentConsecutiveErrorLimit is the number of consecutive errors we will ignore when waiting for a volume to attach/detach volumeAttachmentStatusConsecutiveErrorLimit = 10 // most attach/detach operations on AWS finish within 1-4 seconds

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: We've seen scalability issues with the current aws-ebs volume provisioner impl, which is querying possible zones from cloud provider directly (). The improvement proposed in did make api calls more efficient, but also raises people's concern about whether it will create regression or not, or is there any more reliable ways to improve. To summarize, there are 2 major concerns: zone info in Node label trustworthy? My answer is yes, since currently the zone labels (beta version only) are maintained by node controller, the information is retrieved from cloud provider directly and is always updated during node reconciliation, it can be trusted. Even if kubelet can be misconfigured by wrong zone information, node controller is going to override them. See: Once we have the GA version of zone labels pushed out, we should update node controller to maintain both zone labels. It's fair enough to assume that if user wants kubelet to report zone/region, the information should be correct. the master/minion labels trustworthy? My answer is "semantic wide they are equivalent to checking node labels from cloud provider" - just as people are not required to set master/minion labels in v1.Node, well-known labels in cloud provider is not guaranteed to be set as well. Therefore besides mainstream cluster provisioners, which maintains these sets of labels in both Kubernetes and cloud provider, the function's behavior will remain unchanged: the original impl skips nodes that are explicitly labeled with well-known master labels, same here. Please suggest. /sig aws /sig cloud-provider /kind bug /priority important-soon /cc
for item 2, an alternative is to first call a List() of all v1.Nodes, get all providerID, and make a DescribeInstance call by passing all these IDs, which might still be more efficient than running the filter, and still very heavy-weighted with large scale. Current logic can be applied to the results though. Need input from aws side.
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available .

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
One of the quick optimizations we can do is, to remove those API calls if pod placement has already been decided because of delayed binding. I think, using delayed binding works well in practice and should eliminate calls being made.
Another thing is - since EBS volume plugin is being targeted for CSI migration, we should be super careful approving any changes into in-tree plugins which is not available in EBS CSI plugin. cc
Thanks ! yes I do agree that for topo aware storage class, when we create volume, the node is already bound, so we just get the zone information from the selected node, and this is a good to have short-term improvement. In long term, as cloud providers are going to be out of tree, we have more freedom to introduce informer to the aws library (gce is already doing that), and get zone information from cache. Since the path we are going is to make kubernetes components node role neutral, it's fair enough to say that we don't have to filter out master only zone in volume provisioner, and user should leverage topology aware storage class along with labels, taints, tolerations to achieve that. what do you think?
I believe in long term we want to default all clusters to use policy. That is the direction sig-storage wants to take(IIRC ), so IMO it may not be worth fixing general immediate binding use case for in-tree EBS driver where cloudprovider itself is deprecated (and any proposed change is non-trivial).
if the direction sig-storage is going is to always use , I agree with you that fixing a deprecated path of a deprecated EBS driver would be not worthwhile. Meanwhile, since in-tree EBS driver will still stay for a few more versions, I'm proposing for optimizing code path. Could you please provide inputs? thanks! cc
Related to
Hello! I'm the bug triage lead for the 1.16 release cycle and considering this issue is tagged for 1.16, I'd like to check its status. The code freeze is starting on August 29th EOD PST (about 5 days from now) and once the code freeze starts, only release-blocking changes will be considered to be merged. Is this issue still planned to be fixed for the 1.16 release?

I have addressed this issue via , will close this issue /close
Closing this issue. <detailsIn response to : I have addressed this issue via , will close this issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: MatchExpressions: []metav1.LabelSelectorRequirement{}, }, }, <ins> { name:          "no resource and selector", args:          args{args: []string{}}, wantResources: []string{}, wantErr:       false, wantSelector:  nil, }, </ins> } for _, tt := range tests { gotResources, gotSelector, err := getResourcesAndSelector(tt.args.args)

Is this a BUG REPORT or FEATURE REQUEST? (choose one):BUG REPORT Kubernetes version (use ):Client Version: version.Info{Major:"1", Minor:"6+", GitVersion:"v1.6.0-alpha.0.2610+-dirty", GitCommit:"", GitTreeState:"dirty", BuildDate:"2016-12-29T06:28:44Z", GoVersion:"go1.7.1", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"6+", GitVersion:"v1.6.0-alpha.0.2610+-dirty", GitCommit:"", GitTreeState:"dirty", BuildDate:"2016-12-29T06:28:44Z", GoVersion:"go1.7.1", Compiler:"gc", Platform:"linux/amd64"} What happened:
Could someone from please help with triage. Thanks!
There is already a PR on the way.


Question: }, expectedTrace: []*spanExpectation{ { <del> name: "KubernetesAPI", </del> <ins> name: "PATCH /api/v1/nodes/{:name}", </ins> attributes: map[string]func(*commonv1.AnyValue) bool{ "http.user_agent": func(v *commonv1.AnyValue) bool { return strings.HasPrefix(v.GetStringValue(), "tracing.test")

When enable apiserver tracing, the webhook request spans only show spanName as /, mostly because use http request operations as default span name, which is hard to understand. ! Following the open-telemetry standard of spanName, it's recommended to use as spanName. ! SpanName of webhook request should be , like ! APIServerTracing feature gates of apiserver a pod tracing dashboard No response <details18 </details<detailsNone </details<details</details<details</details<details</details<details</details
I make a fix for this.
/sig api-machinery instrumentation
Do you know if this is fixed in later versions of opentelemetry?
/triage accepted /assign
Looks like might be related
not yet, I think we can fix this in otel upstream, but it's hard to backport to otel , which is the version of otel used in Kubernetes
Good point. It think fixing it here is a good idea in that case.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
could I take on this issue?
yes. Feel free to work on this.
! I test locally by 1.29 kubernetes and start a sample webhook, looks like the client span has already conforms to semantic conventions , which should be in the form of "HTTP {METHODNAME}"
! For apiserver as a server, now the spanName is still KubernetesAPI, which I think we could enhance it into "HTTP get_account/{accountId}" format
After digging for a while, I found one solution to implement this, I draft this idea in , could you help review about whether this is a correct way to implement it?


Question: It("should spread the pods of a replication controller across zones", func() { SpreadRCOrFail(f, int32((2*zoneCount)+1), image) }) <ins> It("should schedule pods in the same zones as statically provisioned PVs", func() { PodsUseStaticPVsOrFail(f, (2*zoneCount)+1, image) }) </ins> }) // Check that the pods comprising a service get spread evenly across available zones

For statically created PVs, the persistent volume label admission controller adds the zone label to the PV object. If the admission controller is not enabled or has some bugs, then an e2e test would be helpful for catching these errors. /assign
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list


Question: "//vendor/k8s.io/client-go/discovery:go_default_library", "//vendor/k8s.io/client-go/dynamic:go_default_library", "//vendor/k8s.io/client-go/informers:go_default_library", <ins> "//vendor/k8s.io/client-go/kubernetes:go_default_library", </ins> "//vendor/k8s.io/client-go/kubernetes/fake:go_default_library", "//vendor/k8s.io/client-go/rest:go_default_library", "//vendor/k8s.io/client-go/util/workqueue:go_default_library",

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: I am running etcd-operator in kubernetes, and noticed that when I deleted the etcdcluster (Custom resource definition) the pods & services did not get cleaned up. I looked on another kube environment I had running, and on that env everything worked as expected. I checked logs, and could see on the env that was working I saw the following messages in Controller manager logs: I didn't see these in the env that wasn't working, so I started looking at the garbage collector on there. I noticed that there hadn't been any log message from garbage collecor since 31st Jan, when this was logged: Around this time we experienced some environmental network issues, which caused lots of errors around that time, such as: It looks like these issues may have caused the garbage collector to get into some sort of hung state? When I restarted controller manager everything started working as expected. What you expected to happen: Garbage collector should have kept running, and working as expected. How to reproduce it (as minimally and precisely as possible): Not sure on the exact sequence that caused it, but suspect it was related to a time period when errors were occurring. Anything else we need to know?: We have set, and had previously hit this garbage collector issue - However, we are running with Kube 1.8.6 with the fix for that issue cherry-picked (commit ) Environment: Kubernetes version (use ):Cloud provider or hardware configuration: IBM Cloud OS (e.g. from /etc/os-release): Ubuntu "16.04.3 LTS (Xenial Xerus)" Kernel (e.g. ): 0-78-generic -Ubuntu SMP Thu Apr 27 15:29:09 UTC 2017 x8664 x8664 x86_64 GNU/Linux Install tools: Others: I will attach logs from Controller manager.
/sig api-machinery
/assign /assign
cc
I saw garbage collector stopped working in 1.9.2 too. Most controllers (such as node controllers or deployment controllers) stopped reporting logs in controller- when this happened. Manually restarting controller-manager fixed the problem. I think somehow some controllers got stuck (could it be because of some bugs in watch cache?) and it's not specific to garbage collector.

When the issue happened, still reported healthy for controller-manager.
tracks this
This is happening in production for us too.
did this happen to you to? Or is it that only the garbage collector disappeared from the logs?
For me the other controllers still have logs; it was just the GC that crashed.
does your cluster also use custom resources backed by Aggregated API servers?
Yes, although this happened in one cluster where we don't have any resources yet (the type exists). I'm not sure what that means
I still see messages from and and after the garbage collector stops working - so looks like some controllers are still working. Is there anything specific you were looking for?
I think the cause of this garbage collector hanging here is this: function periodically calls to the comment above GetDeletableResources, , an error in discovery causes the discovery client to no longer discover resources in the cluster, but instead of failing and returning an error, it simply logs the error as and returns an empty list of resources Sync function, upon recieving an empty resource list from discovery, detects that the resources have changed, and calls , which calls with as the argument as shown in the log as , which sets the list of monitors to an empty list because it thinks there are no resources to monitor. the Sync function calls controller.WaitForCacheSync, which calls , which will continually retry the function until it returns true, but it will always return false because is 0. So I think this was caused by which enabled number 2.


Question: storagev1 "k8s.io/api/storage/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/labels" <ins> "k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/runtime/schema" </ins> types "k8s.io/apimachinery/pkg/types" utilerrors "k8s.io/apimachinery/pkg/util/errors" "k8s.io/apimachinery/pkg/util/uuid"

[x] test approval issue : [x] test pr : [x] Flake [x] two weeks soak end date : 31 Oct 2023 [X] test promotion pr : [X] remove endpoint from According to following APIsnoop query, there are three PersistentVolume and three PersistentVolumeClaim endpoints that are untested. Using a number of existing e2e test practices a new has been created to provide future Conformance coverage for the 6 endpoints. The e2e logs for this test are listed below. This query shows the following PV and PVC endpoints are hit within a short period of running this e2e test. If a test with these calls gets merged, test coverage will go up by 6 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted /cc
/reopen For tracking the rest of the steps to Promotion
Reopened this issue. <detailsIn response to : /reopen For tracking the rest of the steps to Promotion Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: API Server using the current Aggregator [Conformance]' description: Ensure that the sample-apiserver code from 1.17 and compiled against 1.17 will work on the current Aggregator/API-Server. <del> release: v1.17, v1.21 </del> <ins> release: v1.17, v1.21, v1.27 </ins> file: test/e2e/apimachinery/aggregator.go - testname: Custom Resource Definition Conversion Webhook, convert mixed version list codename: '[sig-api-machinery] CustomResourceConversionWebhook [Privileged:ClusterAdmin]

[X] Test approval issue: [X] Test PR: [x] Two weeks soak end date: [X] Test promotion PR: According to this APIsnoop query, there are still some remaining APIService endpoints which are untested. Using a number of existing e2e test practices the has to cover the outstanding APIService endpoints. There has been some around how the current Conformance test is using RBAC, which is optional and not part of Conformance. Those calls will now only happen if is true. The e2e logs for this test are listed below.

This query shows the following APIService endpoints are hit within a short period of running this e2e test example endpoint | useragent ---------------------------------------------+
createApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] deleteApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] deleteApiregistrationV1CollectionAPIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] listApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] patchApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] patchApiregistrationV1APIServiceStatus | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] readApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] readApiregistrationV1APIServiceStatus | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] replaceApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] replaceApiregistrationV1APIServiceStatus | Should be able to support the 1.17 Sample API Server using the current Aggregator [Conformance] (10 rows) If a test with these calls gets merged, test coverage will go up by 4 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted


Question: return err } refCount, err := getDevicePrefixRefCount(c.mounter, prefix) if err == nil && refCount == 0 { <del> // this portal/iqn are no longer referenced, log out // extract portal and iqn from device path </del> <ins> // This portal/iqn/iface is no longer referenced, log out. // Extract the portal and iqn from device path. </ins> portal, iqn, err := extractPortalAndIqn(device) if err != nil { return err } <del> glog.Infof("iscsi: log out target %s iqn %s", portal, iqn) out, err := c.plugin.execCommand("iscsiadm", []string{"-m", "node", "-p", portal, "-T", iqn, "--logout"}) if err != nil { glog.Errorf("iscsi: failed to detach disk Error: %s", string(out)) </del> <ins> // Extract the iface from the mountPath and use it to log out. If the iface // is not found, maintain the previous behavior to facilitate kubelet upgrade. // Logout may fail as no session may exist for the portal/IQN on the specified interface. iface, found := extractIface(mntPath) if found { glog.Infof("iscsi: log out target %s iqn %s iface %s", portal, iqn, iface) out, err := c.plugin.execCommand("iscsiadm", []string{"-m", "node", "-p", portal, "-T", iqn, "-I", iface, "--logout"}) if err != nil { glog.Errorf("iscsi: failed to detach disk Error: %s", string(out)) } } else { glog.Infof("iscsi: log out target %s iqn %s", portal, iqn) out, err := c.plugin.execCommand("iscsiadm", []string{"-m", "node", "-p", portal, "-T", iqn, "--logout"}) if err != nil { glog.Errorf("iscsi: failed to detach disk Error: %s", string(out)) } </ins> } } }

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): No What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): iscsi session logout Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Environment: Cloud provider or hardware configuration: Libvirt qemu VM, 4 CPU, 8Gig RAM OS (e.g. from /etc/os-release): Ubuntu 16.04.1 LTS (Xenial Xerus) Kernel (e.g. ): Linux localhost 4.4.0-62-generic -Ubuntu SMP Wed Jan 18 14:10:15 UTC 2017 x8664 x8664 x86_64 GNU/Linux Install tools: kubeadm Others: What happened: If you have multiple pods on the same node using PVCs that are backed by iSCSI LUNs on the same external system over a single iSCSI session (same IQN but different LUN numbers), then deleting one of the pods causes the iSCSI session to terminate for all pods on that node that are using it. For example, here and are nginx pods each connected to a different PVC, which are backed by LUNs residing on the same NetApp storage system. These share a single iSCSI session, differing only by LUN number. Once pod2 is deleted, all the connected LUNs are no longer available.

[6:0:0:1] disk NETAPP LUN C-Mode 8300 /dev/sdc What you expected to happen: I expect that in-use iSCSI sessions are not terminated How to reproduce it (as minimally and precisely as possible): 2 PVs and PVCs that are backed by a single storage system that aggrates iSCSI connections into a single session. 2 pods that consume those PVCs one of the pods Anything else we need to know:
Could someone from please take a look? Thanks!
looking...
It looks like the problem is in line 86 of , in the function. The check evaluates whether a given mount path starts with the prefix (protocol + target portal + target IQN) for the LUN; this will always evaluate to false in most set-ups, since each path will look something like . Changing to appears to result in the correct behavior.
The problem is that the mount device shown in /proc/mounts are canonicalized, and thus when searching device prefix, iscsi volume failed to find a match. If we can use mount option, this the device will show up in /proc/mounts. But this is not a reliable fix since there are probable mpio paths too.
If I'm reading the code correctly, the function doesn't look at the actual device that's backing the mount point; it looks at the mount path (i.e., the second token in /proc/mounts, rather than the first). Thus, the change I proposed (HasPrefix -Contains) appears to work with some very basic testing. That said, I've not done anything with multipathing, so there may be something I'm missing; in addition, I suppose there's a chance that Contains may trigger false positives.
I see your point, yes that sounds right to me.
I think in this case, mpio device should work too (since the mount path constructed in the same way). I see you have yet another iscsi issue open, I am good with rescan. Can you create a pr to fix them?
I should be able create a PR for this issue; the LUN rescan PR will probably get handled by someone else on my team. Is the fix itself sufficient, or does it require tests?
thanks, the rescan fix doesn't have to unit test. The mount path fix needs also a unit test fix here

BUG REPORT Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-01-12T04:57:25Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-01-12T04:52:34Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"} Environment: Cloud provider or hardware configuration: Bare metal OS (e.g. from /etc/os-release): CentOS 7; reproduced in Ubuntu 14.0.4.2. Kernel (e.g. ): Linux centos7 3.10.0-514.6.1.el7.x8664 SMP Wed Jan 18 13:06:36 UTC 2017 x8664 x8664 x8664 GNU/Linux; kernel 3.16.0-77-generic What happened: On a host where an iSCSI LUN from a given target is mounted in a pod, a user who creates a new LUN on the same target and attempts to create a pod that mounts the new LUN will fail to create the pod. Specifically, depending on the operating system, the pod will hang in the ContainerCreating stage with either a "Could not attach disk: Timeout after 10s" error (CentOS 7) or an error about establishing a session when one already exists (Ubuntu 14). What you expected to happen: The pod should successfully mount the LUN. How to reproduce it (as minimally and precisely as possible): Given a working iSCSI target and a single node Kubernetes cluster: a LUN on the iSCSI target a pod that uses the LUN from step 1 (either natively or through a PV/PVC) a new LUN on the iSCSI target (note that the target portal and IQN must be identical). a pod that uses the LUN from step 3. The pod created in step 4 will fail to start.

Anything else we need to know: As far as I can tell, this is due to a bug with the Kubernetes iSCSI implementation. The AttachDisk() function in first checks if the LUN exists in /dev/disk/by-path/; if it doesn't, it attempts to log in to the target and recheck for the LUN's existence (lines 123-138 in the 1.5 branch). In this case, though, the session already exists, as it was created with the pod in step 2. In Ubuntu, attempting to log in a second time triggers an error, which gets reported in the pod description. In CentOS 7, this gets treated as a NOP; however, the new LUN will never become visible, and thus, the pod will fail to start. This can be solved manually by logging into the host and running . It should be possible to fix it in Kubernetes by adding a check in AttachDisk() to see if a session already exists for the target portal and IQN and, if so, doing a rescan of the session.
Could someone from please help with triage? Thanks!


Question: } func (m *Master) init(cloud cloudprovider.Interface, podInfoGetter client.PodInfoGetter) { <del> m.random = rand.New(rand.NewSource(int64(time.Now().Nanosecond()))) </del> podCache := NewPodCache(podInfoGetter, m.podRegistry, time.Second*30) go podCache.Loop() <del> s := scheduler.NewRandomFitScheduler(m.podRegistry, m.random) </del> <ins> random := rand.New(rand.NewSource(int64(time.Now().Nanosecond()))) s := scheduler.NewRandomFitScheduler(m.podRegistry, random) </ins> m.storage = map[string]apiserver.RESTStorage{ "pods": registry.MakePodRegistryStorage(m.podRegistry, podInfoGetter, s, m.minionRegistry, cloud, podCache), "replicationControllers": registry.NewControllerRegistryStorage(m.controllerRegistry, m.podRegistry), "services":               registry.MakeServiceRegistryStorage(m.serviceRegistry, cloud, m.minionRegistry), "minions":                registry.MakeMinionRegistryStorage(m.minionRegistry), } } // Run begins serving the Kubernetes API. It never returns.

We got CI failures across all CSI sidecars: ...... I see the following in all the failed jobs: Kubernetes e2e suite: [It] External Storage [Driver: ] [Testpattern: Dynamic PV (default fs)] provisioning should provision correct filesystem size when restoring snapshot to larger size pvc [Feature:VolumeSnapshotDataSource] expandless 46s { failed [FAILED] Filesystem resize failed when restoring from snapshot to PVC with larger size. Restored fs size: bytes is not larger than origin fs size: bytes. HINT: Your driver needs to check the volume in NodeStageVolume and resize fs if needed. HINT: For an example patch see: Expected <int: to be <int: In [It] at: @ 10/04/23 11:50:44.626 } The tests should pass. The failure shows up in the test-grid. No response_ <details</details<details</details<details</details<details</details<details</details<details</details
/sig storage
/triage accepted
CSI hostpath driver does not support expanding volume when creating PVC from VolumeSnapshot, so I expected the test should be skipped before getting to the point where it failed on L530.
These tests are creating a snapshot of a PVC and then creating another PVC from the snapshot with 1Gi more than original size and expect the restored size to be larger than original size. Sounds like something specific to hostpath.
After the creation of a snapshot from an existing PVC. A new PVC is resultant from this snapshot with a size that exceeds the original PVC by 1Gi. The expected result is that the restored PVC size should be larger than the original PVC. it is noted that this issue may be related to the hostpath storage type, suggesting that there might be a specific problem or unexpected behavior associated with hostpath storage may be possible.
I reproduced it in the test environment and found the same result, the size before and after the expansion is the same, the e2e detection logic seems to be useless for csi-driver-host-path, and it is reasonable to say that it should not be able to run the previous k8s version e2e as well. seems like it was changed 8 months ago. any ideas?

Can you update the deployment folder to match the new versions we are supporting now (1.26, 1.27, 1.28)? Make sure FSResizeFromSourceNotSupported is set to true for every version. See this PR:
okay~
This is not fixed yet. /reopen
Reopened this issue. <detailsIn response to : This is not fixed yet. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Tests are still failing on sidecar repos. From the build logs: I don't see the new parameter in test- in v1.11.0: It's in the latest code: We need to cut a new csi-driver-host-path release and update image in in csi-release-tools.
I can help with updating the csi-release-tools after the new version is released~
csi-driver-host-path 1.12 is released: Can you help update the csi-release-tools? Thanks.
ptal~
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Still fails: Need to update images in test-infra:
Fixed now.


Question: nodes:    []string{"machine1", "machine2"}, rcs:      []*v1.ReplicationController{{Spec: v1.ReplicationControllerSpec{Selector: map[string]string{"foo": "bar"}}}}, services: []*v1.Service{{Spec: v1.ServiceSpec{Selector: map[string]string{"bar": "foo"}}}}, <del> // Taken together Service and Replication Controller should match all Pods, hence result should be equal to one above. expectedList: []schedulerapi.HostPriority{{Host: "machine1", Score: 0}, {Host: "machine2", Score: 5}}, name:         "disjoined service and replication controller should be treated equally", </del> <ins> // Taken together Service and Replication Controller should match no pods. expectedList: []schedulerapi.HostPriority{{Host: "machine1", Score: 10}, {Host: "machine2", Score: 10}}, name:         "disjoined service and replication controller matches no pods", </ins> }, { pod: &v1.Pod{ObjectMeta: metav1.ObjectMeta{Labels: map[string]string{"foo": "bar", "bar": "foo"}, OwnerReferences: controllerRef("ReplicaSet", "name", "abc123")}},

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: /kind bug What you expected to happen: Problem 1: Predicates filter nodes. Existing pods on those nodes will be not be counted when calculating max pods per (zone or node) resulting in imbalanced cluster. Eg: removes nodes which cannot . If any of the pods are already scheduled on this node, they are not considered when counting in . Problem 2: When there are 2 selectors(service and replication controller), it is sufficient to match any one selector for distribution. This creates imbalance ]. Pods from previous deploys matches and are counted when distributing pods across zones/nodes (Even though they do not match ) . These pods will be deleted. After the deploy completes, the cluster is imbalanced - by zone and/or pods per node. How to reproduce it (as minimally and precisely as possible): Problem 1: Get pods to get scheduled on nodes with very high cpu utilization [Existing cpu utilization + 1 new pod will result in almost equal to 0]. GeneralPredicate will drop this node after scheduling the first pod. This will lead to over utilization of an already loaded zone. If there are enough nodes like this one, it will create a pile on effect resulting in most pods scheduled in an already loaded zone. Problem 2: Have 2 selectors and deploy 30 pods across 3 zones. Twice. Anything else we need to know?: Environment: Kubernetes version (use ): /sig scheduling <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
re Problem 1: for per node, we only need to compare current nodes as they will share the same maxPods. for per zone, we need to account other nodes, e.g. because of resource predicates. re problem 2: a little about your case :(
Problem 1: : It does not matter if the denominator is all nodes or current nodes. One with higher utilization will still end up with a lower score. : It is critical that the denominator is all nodes. Else we are ending up with highly imbalanced cluster. Problem 2: Here is an example list of selectors.

All pods matching any one service selector are counted in So in our case, old pods are counted during scheduling. But they get killed during the deployment. If the maxSurge and maxUnavailable is high enough, you will end up with an imbalanced zone after deployment.
Gentle ping.
/assign bsalamat
Re: Problem 1 - You can also reproduce this by using pod anti-affinity. Zone score is calculated only based on nodes filtered by predicates. IMO zone score should be calculated using all nodes.
I will try to separate this into two PRs. Any other suggestions welcome.
I am working Problem 1. Looks like I have to open a new issue.