Question: } if !nodeExist { <del> glog.Warningf( "Node %q does not exist. DisksAreAttached will assume vmdk %v are not attached to it.", vSphereInstance, volPaths) return attached, nil </del> <ins> glog.Errorf("DisksAreAttached failed to determine whether disks %v are still attached: node %q does not exist", volPaths, vSphereInstance) return attached, fmt.Errorf("DisksAreAttached failed to determine whether disks %v are still attached: node %q does not exist", volPaths, vSphereInstance) </ins> } // Get VM device list

When a vm is reported as no longer present in cloud provider and is deleted by node controller, there are no attempts to detach respective volumes. For example, if a VM is powered off or paused, and pods are migrated to other nodes. In the case of vSphere, the VM cannot be started again because the VM still holds mount points to volumes that are now mounted to other VMs. In order to re-join this node again, you will have to manually detach these volumes from the powered off vm before starting it.
Once the node is powered off, the controller-manager doesn't send a detachVolume request to vSphere cloud provider. The controller-manager couldn't send the detachVolume request because it checks until the volume is unmounted on the node for a particular timeout. You can see the code over here - Since the volume is never unmounted, the controller manager skips the detachVolume request and issues the attachVolume() on another successfully. I think the controller-manager needs send a detachVolume() request even if the volume is mounted when the node is powered OFF. This is serious problem in our case (vSphere) as volume is now attached on another node and the node which is powered OFF cannot be powered ON again, as the node tries to acquire a lock on the volume to mount it on this node during the power ON. The acquire lock fails as it is already attached/mounted on other node. Also there are a few workarounds for this. When the Node is powered OFF, the cloud provider can detach the volume in NodeExists() - This method is called in DisksAreAttached() method. However i don't think this is a viable solution. : Can you provide your inputs on this. CCing
CC If a node goes offline (for any reason including being powered off) the attach/detach controller will eventually detach the volume even if the node never reports it as safely unmounted. Once a node becomes inaccessible, the node controller waits 5 min 40 seconds, before deleting all the pods that were scheduled to that node. When a pod referencing an attachable volume on that node is deleted, the attach/detach controller tries to detach the volume, but if the node has not reported the volume as safely unmounted (which would be the case if the node was powered off), the attach/detach controller waits 6 minutes before triggering detach anyway.

Once the volume is detached, any attempts to attach it to some other node will finally succeed. This means that it can take up to 15 minutes for this situation to rectify itself. Can you verify what amount of time you waited?
So do you mean that the volume can be attached to another node only after the volume is detached on the node which is powered OFF. In my case, i have waited for significant amount of time. I am sure it is more than 15 mins. And I see the volume succesfully attached to the new node but still not detached from the powered OFF node. In this case I cannot restart the node as explained above. Only if I delete the attached nodes manually I can power ON the node again.
If the storage system supports multi node attach for the access mode, then the attach to the new node should succeed immediately. If the storage systems supports only single node attach for the access mode then the attach calls to the new node fail fail until it is detached from the first node (this is the case with GCE PDs). If you are never seeing the volume detach from the original (powered off) node (even after 15 minutes), that sounds like a bug. AFAIK, this case is working fine in the GCE PD case ( tested this recently). If you have a consistent repro, to help debug, could you grab and share your controller-manager logs showing the original attach, subsequent attach, and lack of detach from the first node.
Sure. I will attach the logs. Between in our case we only support single node attach for a volume.
Between I am testing it on v1.4.7. Is this a change which is recently made to kubernetes code base to address this issue and not present in v1.4.7?
Please verify the logs from controller-manager. I see there is detach volume request being issued. I almost waited for 45 mins, but I didn't send any request attempting to detach volume as in - As in , since it failed to update node status, it couldn't make any further request for detachVolume(). Also, from the logs I see it is the kubernetes control plane which decides whether the storage plane is multi node attach or single node attach by attempting to send an attachVolume request on the new node.

And its not the cloud provider which says it to kubernetes control plane what type of storage plane it is. Correct me if I am wrong on this. : Can you please verify this.
That log is level 5. If your cluster is set up with log level 4, you will not see that log. When a node is down, the following events will happen with code release 4 or before. Once a node becomes inaccessible, the node controller waits for about 5 mins, and then deletes all the pods that were scheduled to that node. Once pods are deleted, Volume manager's reconciler will try to detach the volumes that belong to the pods. However, it will fail to issue to request first because it thinks the volumes are still mounted to the nodes according to the node status. (node status can be updated by the kubelet but since the node is down, the status is not updated). the reconciler fails to issue the detach request for a periods time, it times out (currently it is set to 6 minutes) and will issue detach request anyway and it should find that the volume is already detached since node is already terminated. So it will probably take more than 10 mins for reconciler marks the volume is detached to the node. We recently have a PR that change the above behavior during node termination because this PR adds a periodic check that whether the volumes are still attached. When node is terminated, the reconciler will very quickly (in a few seconds) find out the node is terminated and volume is detached and then mark this change in the actual state. {"log":"W1223 00:27:26. 1 ] Node "node2" does not exist. DisksAreAttached will assume vmdk [[vsanDatastore1] kubevols/redis-master] are not attached to it.n","stream":"stderr","time":"2016-12-23T00:27:26.429562004Z"} {"log":"I1223 00:27:26.

1 ] VolumesAreAttached: check volume "[vsanDatastore1] kubevols/redis-master" (specName: "redis-master-data") is no longer attachedn","stream":"stderr","time":"2016-12-23T00:27:26.44262004Z"} Please let me know if you have any questions about this part. Can you please verify this.
: I am using v1.4.7. Can you let me know if this holds true for v1.4.7? I don't see any behavior as mentioned by in the logs. Also you can see there is no detach call made on the node. As per this - if fails to update the node status after various attempts and hence detach call doesn't pass through. Also, from the logs I see it is the kubernetes control plane which decides whether the storage plane is multi node attach or single node attach by attempting to send an attachVolume request on the new node. And its not the cloud provider which says it to kubernetes control plane what type of storage plane it is. Also can you comment on the above question?
+1 on this. Same goes for k8s v1.5.2. Detach never happens when powering off a node with PVs attached. Disks has to be manually detached before being able to turn on the node again.
The reason detach never happens is as follows, is terminated controller periodically check whether volumes are still attached by sending requests to cloud provider. The function DisksAreAttached will check whether the node still exist. If not, it will assume the volume is no longer attached. If node does exist, check the information returned by the cloud provider to see whether volume is still attached. In this step, I believe it returns as volume is detached volume is already detached. So explicit detach will not be issued since step 2 and 3 already marks it as detached. I will check the function to see if there is potential problem in checking volume detached or not.
Cheers then I understand. I will also have a look.
is a workaround but ideally k8s should call detach for every attach that was issued, if the disk is already detached it should be a no-op. The ideal behavior should be that for each attach there will be a detach (which could be redundant).

The assumption in the current model seems to be that as long as the volume provider does not consider the disk attached the detach does not need to be issued which ignores the configuration for the VM. If the VM still considers the disk attached it should be reconfigured as well. The pod in this case is fine after being rescheduled to a new node but the VM is left with an invalid configuration.
Ok! The workaround seems legit for the time being. Thanks!


Question: } local i for i in $(seq 1 $times); do local out if out=$(curl -fs $url 2>/dev/null); then <del> kube::log::status ${prefix}${out} </del> <ins> kube::log::status "On try ${i}, ${prefix}: ${out}" </ins> return 0 fi <del> sleep $wait </del> <ins> sleep ${wait} </ins> done <del> kube::log::error "Timed out waiting for ${url}" </del> <ins> kube::log::error "Timed out waiting for ${prefix} to answer at ${url}; tried ${times} waiting ${wait} between each" </ins> return 1 }

Example:
Is kubelet healthz coupled with the docker daemon being accessible? ?
That particular error is coming from cAdvisor and should not be causing the test to block. The Kubelet does wait up to 5min for Docker to come up though. Do we use a real Docker in these tests?
Shippable is running on different os dist? ContainerOsVersion:Ubuntu precise?
Yes today kubelet healthz coupled with docker daemon. We even periodically healthcheck docker and kubelet, and restart them even there are issues.
Another example:
bumping priority because we have to get these flaky tests fixed.
This has been marked P0 for 2 months but hasn't had any updates. Is this still happening on Shippable? If so, can we just increase a timeout in the test?
This is ridiculous, I'll fix it myself.


Question: mustFail: true, err:      errors.New("grpc error"), }, <ins> { name:                       "test empty nodeId", mustTimeout:                true, expectedNodeID:             "", expectedMaxVolumePerNode:   16, expectedAccessibleTopology: map[string]string{"com.example.csi-topology/zone": "zone1"}, }, </ins> } for _, tc := range testCases {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Kubelet dies with a panic exception if CSI plugin returns NodeId == "" and nil error. Exception log see at the end of the message What you expected to happen: Kubelet should just raise an error to the plugin and ignore the registration. How to reproduce it (as minimally and precisely as possible): Create a CSI plugin with only Node that would implement GetNodeInfo as follows: Register it as a DaemonSet and see kublet crush Anything else we need to know?: Environment: Kubernetes version (use ): 1.14.10-gke.17 Cloud provider or hardware configuration: GKE OS (e.g: ): Container-Optimized OS (cos) Kernel (e.g. ): - Install tools: Network plugin and version (if this is a network-related bug): Others: Related exception Related line of code: - no check whether error is not nil before calling Error() on it.
/sig storage
/sig node is this the version of the kubelet too? 14 is out of support. is the issue reproducible with a 1.15+ kubelet?
/assign
I checked the code - same defect still exists in current codebase.
thanks /priority important-soon
Wow, that was fast. Thanks!


Question: errorSchedule = "obvious error schedule" // schedule is hourly on the hour onTheHour = "0 * * * ?" <del> A = batchv1.AllowConcurrent f = batchv1.ForbidConcurrent R = batchv1.ReplaceConcurrent T = true F = false </del> ) // returns a cronJob with some fields filled in.

in TestControllerV2SyncCronJob, there are many listing the properties that don't matter (like nil, false). and many constants F and T, which is unnecessary. As per 's request, We shouldn't be listing the properties that don't matter (like nil, false). And get rid of constants F and T check No response <details</details<detailsNone </details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign I would like to work on the optimization
It's not an optimization. It's just about readability.
/remove-kind bug /kind cleanup
/sig apps


Question: URL:    url, } } <ins> func TestInformerSyncHealthChecker(t *testing.T) { t.Run("test that check returns nil when all informers are started", func(t *testing.T) { healthChecker := NewInformerSyncHealthz(cacheSyncWaiterStub{ startedByInformerType: map[reflect.Type]bool{ reflect.TypeOf(corev1.Pod{}): true, }, }) err := healthChecker.Check(nil) if err != nil { t.Errorf("Got %v, expected no error", err) } }) t.Run("test that check returns err when there is not started informer", func(t *testing.T) { healthChecker := NewInformerSyncHealthz(cacheSyncWaiterStub{ startedByInformerType: map[reflect.Type]bool{ reflect.TypeOf(corev1.Pod{}):     true, reflect.TypeOf(corev1.Service{}): false, reflect.TypeOf(corev1.Node{}):    true, }, }) err := healthChecker.Check(nil) if err == nil { t.Errorf("expected error, got: %v", err) } }) } type cacheSyncWaiterStub struct { startedByInformerType map[reflect.Type]bool } // WaitForCacheSync is a stub implementation of the corresponding func // that simply returns the value passed during stub initialization. func (s cacheSyncWaiterStub) WaitForCacheSync(_ <-chan struct{}) map[reflect.Type]bool { return s.startedByInformerType } </ins>

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: for extension API server panics with: Apparently introduces this panic with . What you expected to happen: to do not panic. How to reproduce it (as minimally and precisely as possible): extension API server with - for example the endpoint that the apiserver panics with Anything else we need to know?: Environment: Kubernetes version (use ): / Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/kind regression
/sig api-machinery
/assign
cc
Oops - sorry for that. It seems it is basically this comment: that I blindly applied without e2e retesting (the problem seem to be lack of tests, which is a bit surprising because bugs in my previous implementations were caught by tests). ^^ BTW - this has been cherrypicked to previous releases: 16: 17: 18: so it's actually even more critical...
I can file the automated cherry-pick PRs once the PR against the master branch is merged. :)
Reopening - since we need to cherrypick it still.
I now opened the cherry-picks: 16 - 17 - 18 -
Fix and picks are merged


Question: handler = genericapifilters.WithAuthorization(handler, authz, legacyscheme.Codecs) handler = genericapifilters.WithAuthentication(handler, authn, failedHandler, nil) handler = genericapifilters.WithRequestInfo(handler, requestInfoResolver) <ins> handler = genericapifilters.WithCacheControl(handler) </ins> handler = genericfilters.WithPanicRecovery(handler) return handler

The kube-apiserver is protected, so it's responses should not be cached. The only known exception is for openapi data which is not sensitive, which is large, and which has etags. /assign
/sig api-machinery


Question: CSIVolumePublished CSIVolumePhaseType = "published" ) <ins> var ( deprecatedVolumeProviders = map[string]string{ "kubernetes.io/cinder":  "The Cinder volume provider is deprecated and will be removed in a future release", "kubernetes.io/scaleio": "The ScaleIO volume provider is deprecated and will be removed in a future release", } ) </ins> // VolumeOptions contains option information about a volume. type VolumeOptions struct { // The attributes below are required by volume.Provisioner

The plugin and/or the code behind the plugin does not seem to be maintained, is it time to deprecate it?
/sig storage
cc ?
/area code-organization
LGTM


Question: return "--kubelet-flags=--kernel-memcg-notification=true " + args } <del> // updateOSSpecificKubeletFlags updates the Kubelet args with OS specific // settings. func updateOSSpecificKubeletFlags(args, host, workspace string) (string, error) { output, err := SSH(host, "cat", "/etc/os-release") </del> <ins> // osSpecificActions takes OS specific actions required for the node tests func osSpecificActions(args, host, workspace string) (string, error) { output, err := getOSDistribution(host) </ins> if err != nil { return "", fmt.Errorf("issue detecting node's OS via node's /etc/os-release. Err: %v, Output:n%s", err, output) } switch { <del> case strings.Contains(output, "ID=gci"), strings.Contains(output, "ID=cos"): </del> <ins> case strings.Contains(output, "fedora"), strings.Contains(output, "rhcos"), strings.Contains(output, "centos"), strings.Contains(output, "rhel"): return args, setKubeletSELinuxLabels(host, workspace) case strings.Contains(output, "gci"), strings.Contains(output, "cos"): </ins> args = prependMemcgNotificationFlag(args) return prependCOSMounterFlag(args, host, workspace) <del> case strings.Contains(output, "ID=ubuntu"): </del> <ins> case strings.Contains(output, "ubuntu"): </ins> return prependMemcgNotificationFlag(args), nil } return args, nil } <ins> // setKubeletSELinuxLabels set the appropriate SELinux labels for the // kubelet on Fedora CoreOS distribution func setKubeletSELinuxLabels(host, workspace string) error { cmd := getSSHCommand(" && ", fmt.Sprintf("/usr/bin/chcon -u system_u -r object_r -t bin_t %s", filepath.Join(workspace, "kubelet")), fmt.Sprintf("/usr/bin/chcon -u system_u -r object_r -t bin_t %s", filepath.Join(workspace, "e2e_node.test")), fmt.Sprintf("/usr/bin/chcon -u system_u -r object_r -t bin_t %s", filepath.Join(workspace, "ginkgo")), fmt.Sprintf("/usr/bin/chcon -u system_u -r object_r -t bin_t %s", filepath.Join(workspace, "mounter")), fmt.Sprintf("/usr/bin/chcon -R -u system_u -r object_r -t bin_t %s", filepath.Join(workspace, "cni", "bin/")), ) output, err := SSH(host, "sh", "-c", cmd) if err != nil { return fmt.Errorf("Unable to apply SELinux labels. Err: %v, Output:n%s", err, output) } return nil } func getOSDistribution(host string) (string, error) { output, err := SSH(host, "cat", "/etc/os-release") if err != nil { return "", fmt.Errorf("issue detecting node's OS via node's /etc/os-release. Err: %v, Output:n%s", err, output) } var re = regexp.MustCompile(`(?m)^ID="?(w+)"?`) subMatch := re.FindStringSubmatch(output) if len(subMatch) > 0 { return subMatch[1], nil } return "", fmt.Errorf("Unable to parse os-release for the host, %s", host) } </ins> // RunTest runs test on the node. func (n *NodeE2ERemote) RunTest(host, workspace, results, imageDesc, junitFilePrefix, testArgs, ginkgoArgs, systemSpecName, extraEnvs string, timeout time.Duration) (string, error) { // Install the cni plugins and add a basic CNI configuration.

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: pull-kubernetes-node-crio-e2e Which test(s) are failing: N/A Since when has it been failing: N/A Testgrid link: Reason for failure: Anything else we need to know:
/sig node
/assign


Question: ) const ( <del> rwMask = os.FileMode(0660) roMask = os.FileMode(0440) </del> <ins> rwMask   = os.FileMode(0660) roMask   = os.FileMode(0440) execMask = os.FileMode(0110) </ins> ) // SetVolumeOwnership modifies the given volume to be owned by

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: I am using Persistent Local Volumes. The mountpoints for it are set up with 0700 permissions. When running a pod with a user other than root, and with fsGroup to allow access to the volume, the permissions are wrong and the pod cannot use the volume. The permissions are changed from 0700 to 2760. What you expected to happen: fsGroup makes the volume usable for the group by setting the permissions to 2770. How to reproduce it (as minimally and precisely as possible): You need a cluster with local volumes whose permissions are set to 0700. pv-permission- As you can see, the permissions are 2760. Since the execute bit is missing for the group, the directory cannot be used. Anything else we need to know?: Environment: Kubernetes version (use ): v1.11.2 Cloud provider or hardware configuration: libvirt/QEMU OS (e.g. from /etc/os-release): CentOS Linux release 7.5.1804 (Core) Kernel (e.g. ): 3.10.0-862.11.6.el7.x8664 SMP Tue Aug 14 21:49:04 UTC 2018 x8664 x8664 x8664 GNU/Linux Install tools: kubeadm Others:
/sig storage
The fix is probably to add execute bits here
do you see any problem with setting group execute for fsgroup?
I think that execute should be fine.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten


Question: {ObjectMeta: metav1.ObjectMeta{Name: "nodeB", Labels: label2}}, {ObjectMeta: metav1.ObjectMeta{Name: "nodeC", Labels: label3}}, }, <ins> expectedAntiAffinity: topologyToMatchedTermCount{}, expectedAffinity:     topologyToMatchedTermCount{}, </ins> }, { name: "preFilterState anti-affinity terms are updated correctly after adding and removing a pod", pendingPod: &v1.Pod{ ObjectMeta: metav1.ObjectMeta{Name: "pending", Labels: selector1}, <ins> Spec: v1.PodSpec{ Affinity: &v1.Affinity{ PodAntiAffinity: antiAffinityFooBar, }, }, </ins> }, existingPods: []*v1.Pod{ {ObjectMeta: metav1.ObjectMeta{Name: "p1", Labels: selector1},

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Maybe it should be What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig Scheduling
I feel it is a valid error. But need confirmation from sig-scheduling maintainers. ... is it okay if I work on this?
ok
Yeah, this is a bug in preemption
/assign


Question: } if vmSizeList == nil { <del> result, err := az.VirtualMachineSizesClient.List(context.TODO(), az.Location) </del> <ins> ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second) defer cancel() result, err := az.VirtualMachineSizesClient.List(ctx, az.Location) </ins> if err != nil || result.Value == nil { klog.Errorf("failed to list vm sizes in GetVolumeLimits, plugin.host: %s, location: %s", plugin.host.GetHostName(), az.Location) return volumeLimits, nil

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: A storageAccount API call blocks kubelet from becoming Ready when there isn't any outbound IP, 1. eventually (after 10mins), the kubelet registers again... there is a call to the ARM API done by which blocks for 10mins, before letting kubelet complete with its starting logic and making the node as details: What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): v1.11.5 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: /sig azure /assign


Question: "k8s.io/apimachinery/pkg/api/meta" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" <ins> metav1beta1 "k8s.io/apimachinery/pkg/apis/meta/v1beta1" </ins> "k8s.io/apimachinery/pkg/runtime" <ins> "k8s.io/apimachinery/pkg/util/diff" </ins> "k8s.io/kubernetes/pkg/kubectl/scheme" ) <ins> func toUnstructuredOrDie(data []byte) *unstructured.Unstructured { unstrBody := map[string]interface{}{} err := json.Unmarshal(data, &unstrBody) if err != nil { panic(err) } return &unstructured.Unstructured{Object: unstrBody} } </ins> func encodeOrDie(obj runtime.Object) []byte { data, err := runtime.Encode(scheme.Codecs.LegacyCodec(corev1.SchemeGroupVersion), obj) if err != nil {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: seems to have regressed in v1.13.0 and no longer sorts by creation timestamp. What you expected to happen: should sort by creation timestamp. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: N/A Environment: Kubernetes version (use ): <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
/sig cli
/assign
bisected to () /assign
opened to fix
/assign


Question: } // write bogus file <del> if err := os.WriteFile(configFileName, []byte(`apiVersion: apiserver.config.k8s.io`), os.FileMode(0644)); err != nil { </del> <ins> if err := atomicWriteFile(configFileName, []byte(`apiVersion: apiserver.config.k8s.io`), os.FileMode(0644)); err != nil { </ins> t.Fatal(err) }

ci-kubernetes-integration-master --- FAIL: TestMultiWebhookAuthzConfig (9.23s) (03-20) I cannot search for previous flake. updated recently /sig auth /cc
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Updated the description, the actual cause of the flake is an unexpected change in the file reload timestamp metric
interestingly, doesn't show any failures of this test for this reason (there are a few failures on a PR that broke a bunch of integration tests for a different reason)
Fix open in , switched the file write to an atomic write
No flake after 03-29. Thanks.


Question: nodeCount := 3 zero := int64(0) gracePeriod := int64(1) <ins> heartbeatInternal := time.Second * 2 </ins> testPod := &v1.Pod{ ObjectMeta: metav1.ObjectMeta{Name: "testpod1", DeletionGracePeriodSeconds: &zero}, Spec: v1.PodSpec{

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: Which test(s) are failing: TaintBasedEvictions Since when has it been failing: probably since it was in this PR? Testgrid link: Reason for failure: Anything else we need to know: /sig scheduling /kind flake /priority important-soon /milestone v1.17
You must be a member of the GitHub team to set the milestone. If you believe you should be able to issue the /milestone command, please contact your and have them propose you as an additional delegate for this responsibility. <detailsIn response to : <!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs -- TaintBasedEvictions probably since it was in this PR? Anything else we need to know: /sig scheduling /kind flake /priority important-soon /milestone v1.17 </details
cc.
Looking into this. We moved it to integration because it was flaking in e2e too
/assign damemi
Not sure yet, but I do see this in the logs of the failing tests: which, if I understand this, would remove all the taints and mark nodes as reachable (which would explain why the test is failing waiting for node taints: ). But I need to figure out why all the nodes are becoming unavailable
is that a full fix for this flake or just partial? I still saw it failed yesterday so just checking... :)
I think I also run into this on 13th Oct: Full logs:
Now gets merged, which might fix the flake. Let's keep an eye on this.
reported it still flaked: Can you continue looking into this?
I'll keep trying, from my previous debugging it seems that in some cases the test cluster is entering full disruption mode and removing all of the taints from the nodes. I know we specifically made changes to the test (using multiple nodes) to ensure it wouldn't become fully disrupted, do you have any idea what could still be causing that?
There is some flag controlling how the dispution thing is managed internally, maybe you can check how a node lifecycle manager is constructed.

The test still flakes after we merge , we still need to investigate the cause ,
/assign
Another flake:
These flakes are consistently (always) timeouts waiting on node taints:
I'm working on this. Will send out a PR soon.
/reopen We see another flake since got merged: However, don't panic :) The original "Entering master disruption mode" cause has been fixed - you won't see that message in the above log. This time it's a smaller issue. I will explain it in another PR.
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen The job is still flaky
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/close But for another test :)
Closing this issue. <detailsIn response to : /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: <ins> ## Pull Request Guidelines 1. Please read our [contributor guidelines](https://github.com/kubernetes/kubernetes/blob/master/CONTRIBUTING.md). 1. See our [developer guide](https://github.com/kubernetes/kubernetes/blob/master/docs/devel/development.md). 1. Follow the instructions for [labeling and writing a release note for this PR](https://github.com/kubernetes/kubernetes/blob/master/docs/devel/pull-requests.md#release-notes) in the block below. ```release-note * Use the release-note-* labels to set the release note state * Clear this block to use the PR title as the release note -OR- * Enter your extended release note here ``` [![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/.github/PULL_REQUEST_TEMPLATE.md?pixel)]() </ins>

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Using example for from documentation failed with: What you expected to happen: Example from documentation works. How to reproduce it (as minimally and precisely as possible): How should configuration look like for ? Environment: Kubernetes version (use ):OS (e.g: ):Kernel (e.g. ):
/sig scheduling
/triage accepted oh, I missed a dash. It should be: Would you mind sending a PR to kubernetes/website?


Question: name = "go_default_library", srcs = [ "density.go", <ins> "dns.go", </ins> "framework.go", "gmsa.go", "hybrid_network.go",

We need to add Windows-specific DNS tests to cover the following fields: Test cases needed for : Default, ClusterFirst, None Test cases needed for Test cases needed for Test cases needed for These need to run on Windows containers because the domain resolution policy is different, so different results need to be checked. Windows assumes that any name with a in it is a FQDN, not a PQDN. There is an to cover simple PQDN resolution, but they don't cover the Pod DNS configurations above. /sig windows /milestone v1.14
FYI I'm working on this. I plan to start out by porting over to Windows.
I just moved it into the in progress column. THanks!
What test cases are needed for , , and ? My thoughts were that covers and . I'm adding test cases for Default and ClusterFirst. Does cover the test cases needed for and or do I need more? Or should I just be porting over the entire file?
For the first one - yes. The main differences will be checking the output of instead of , and maybe using instead of but the flow is the same Checking second one now
Agree on second one. I think that will cover it. It may make sense to simplify it a bit because the common methods used there are all based on running bash + dig. If it works with busybox, that may be feasible. The description at matches those test cases
If at all possible, could we use native Windows cmdlets such as instead of relying on Linux cmdlets that have been ported to Windows like ? Am not 100% confident if they are going to behave consistently.
yeah, I think what David's suggesting makes sense if the test cases are forked for Windows.
Opened PR for enabling custom DNS on Azure CNI:
/milestone clear This will need to merge after 1.14 as we're waiting on CNI updates for these tests to pass. I this status note to the Windows kep
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity.

Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten


Question: setInitCommand(input.pod, fmt.Sprintf("ln -s /bin/sh %s", input.subPathDir)) // Pod should fail <del> testPodFailSubpath(input.f, input.pod) </del> <ins> testPodFailSubpath(input.f, input.pod, false) </ins> }) It("should fail if non-existent subpath is outside the volume [Slow]", func() {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: The read only sig-storage tests validate that Pods can come up when readOnly is set. But it doesn't actually validate that the container cannot write to the volume. What you expected to happen: Tests should validate that the container cannot write to readonly volumes. <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
/help
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
can this issue be closed after your last PR or is there more to work on? If yes I'd like to know what's left because I'm looking for something to work on
I think we should actually reinstate the test case I removed and enhance it to handle the two scenarios I mentioned. It would also be good to see if we have other readOnly test cases that could be enhanced as well
Thanks ! I have a few questions: 1) how do I run the tests in locally? and 2) how can I assert writing to volume has failed? Thank you in advance!!


Question: // Images used for ConformanceContainer are not added into NodeImageWhiteList, because this test is // testing image pulling, these images don't need to be prepulled. The ImagePullPolicy // is v1.PullAlways, so it won't be blocked by framework image white list check. <del> imagePullTest := func(image string, hasSecret bool, expectedPhase v1.PodPhase, expectedPullStatus bool) { </del> <ins> imagePullTest := func(image string, hasSecret bool, expectedPhase v1.PodPhase, expectedPullStatus bool, windowsImage bool) { command := []string{"/bin/sh", "-c", "while true; do sleep 1; done"} if windowsImage { // -t: Ping the specified host until stopped. command = []string{"ping", "-t", "localhost"} } </ins> container := ConformanceContainer{ PodClient: f.PodClient(), Container: v1.Container{ Name:            "image-pull-test", Image:           image, <del> Command:         []string{"/bin/sh", "-c", "while true; do sleep 1; done"}, </del> <ins> Command:         command, </ins> ImagePullPolicy: v1.PullAlways, }, RestartPolicy: v1.RestartPolicyNever,

There is now a test tag from I remember a tag being approved in SIG Architecture and SIG Testing, but I don't remember a tag. If this was approved I think we need to broadcast it more widely as many existing test setups including current conformance testing appear to now need to skip / exclude this tag. cc /area conformance /sig architecture /sig testing
/assign
bug may not be the appropriate type, but seemed like the closest issue template to start from initially.
TL;DR nope, that was not something that I'm aware of.
cc
The email thread is in
the in that thread says: That's not the case here.
we should file a PR marked 1.14 and to yank that tag, just so we don't forget. Of course if the decision changes in the email thread, we can revisit the PR. ok?
makes sense to me
Catching up. Quick comment before I read the threads: Documented kinds of tests are currently here: WindowsOnly had been hypothesized, but not yet to that list.
Also, are there already tests tagged as SIG Windows?
On conformance tests: Windows isn't part of conformance yet, so no conformance test should have this tag. And no conformance test should contain any Skip directives. They should die in that case:
I looked at and don't understand why the Windows image test wasn't written as a separate test. To avoid either refactoring or copy/pasting? It looks like a pretty obscure approach -- auto-skipping individual test cases that contain a particular tag?
Pretty much. But it is no longer the case, since all those "should pull image from X" test cases have been refactored. (look at to see how it looks now)
Yes, that's better, thanks. To answer my own question regarding , here's an example: which also invokes inside the test. So we're agreed that we don't need yet?
now removes all instances of the [WindowsOnly] tag.

<!-- Please only use this template for submitting enhancement requests --What would you like to be : /sig testing /sig windows In the the PR we're adding a test case for pulling a Windows image from dockerhub (. However, there are no known public Windows images on , so the only option would be to upload it ourselves. Our proposal is to have it uploaded to the using Windows images). The proposed image manifest list to be uploaded is built from [1], which is currently hosted on dockerhub [2] and is currently used for testing Kubernetes with Windows nodes [3]. The manifest list contains the images: e2eteam/busybox:1.29-1803 (for Windows Server 1803) e2eteam/busybox:1.29-1809 (for Windows Server 2019) [1] [2] [3] Why is this needed: The Windows images will be used for the Windows equivalent test case for the following test:
/assign
/area conformance
It is my contention that we really only want one prod repo and that should be folded into the main repo. The authenticated repo should be for testing authenticated pulls and nothing else. To that end, I propose that we start with this. Does the image need to be named or tagged or something to indicate OS? How about * ? Is that too deep? Does it need Windows in the path or should we put it in the tag? Or can we have a multi-os manifest list? Also, is busybox the right image for testing, or can we do something even more minimal (to address the "what f there's a busybox CVE" question)?
Yes, (all images to date have used ), so I'd prefer that route over yet another name or tag. I'd also prefer something other than busybox if possible. I originally created because there were so many images in and it was hard to tell which of them were production images, which were test images, and which had been completely abandoned. Also, only a small number of Googlers could push to this repo since it contained all the production images. To address the first issue, I decided that all e2e test images would be sourced from the directory. To address the second issue, I created a new project with a wider ACL (though still currently limited to Googlers).

All that said, given the automation we're building for the main GCR repo, it'd probably be fine to fold these images back in.
The image promoter means that a single repo should be BETTER managed than individual repos - we will have a PR trail of who put images there and why.
We should probably broaden the discussion to non-busybox Windows test images too. (related: ). Do we have a lower bar for test images? If so, having a separate repo (or making it clear that the images are test only) does have clear benefits.
Do we want a lower bar? I don't think we do.
Nobody wants a lower bar until they want to add/update the image quickly for a new test, or fix a test failure to unblock the release. If we do effectively hold the same bar today, this shouldn't be a problem.
i agree. also we don't need schema v2. multi-os, multi-arch image here. just (e.g.) a nanoserver image that satisfies a Windows 1803 and Windows 1809 host. a nanoserver image will obviously imply Windows only. for a test called "should be able to pull image from [WindowsOnly]": all we need is to upload this image to [x] and i guess we need to decide on if we want this test to be / passing for Windows Containers GA.
And indeed, a manifest list with nanoserver images could be an alternative (considering that the ) (we did try to find a command which works on both OSes, and ultimately found curl, but not all Linux images contain curl, check PR ).
an official nanoserver image is a more secure download. do we need to build a nanoserver manifest list or are there official images that we can use? i would like to bring a topic here related to pause. why are all the pause images for Windows that i found 1.1 GB or more? we definitely need the absolute minimal pause container uploaded in a location where users can download it and use that location in the guide. AFAIK, 100 MB is as low as we can go for a native Windows container, but that's still much better... i brought this topic in the other PR. we don't need a command that works on both OSes.

if is the command to use on Windows containers only, so be it.
And also, my image? It might be out-of-date.
we need to start planning how Windows containers can used be easily on Windows nodes. the pause image should be pre-built, small and easy to pull by deployment tools, such as kops, kubeadm. that is much better, but still a bit too much? can we go down to ~100MB? this image size was reported by a user in a k/k issue recently: my assumption is that the user was building the image himself.
i am ok to have that conversation, but please file a ticket and let's discuss that in sig-windows
everyone, thank you for your help and responses. trying to bring back this conversation to the original request. what would you like us to do in order for the few folks from google that have permissions to upload the windows images to be used for testing.
I am happy to help publish this via the alias, which I believe is the correct long-term path. We need an image - preferably not one as prone to CVEs as busybox historically is. Hence a suggestion for "pause" or something equivalently useless and tiny (if it is tiny). Then, we need a path. First, we'll be setting precedent for test images, so I need input on that. I propose as the root. Objections from test folks? Second, do we want to squirrel this away as a Windwows-specific binary, or should we not name it in any particular way, so that we can pull the same image for all tests via a multi-os, multi-arch manifest list (which can be done later). E.g. vs How about * ? Is that too deep? Does it need Windows in the path or should we put it in the tag? Michael Michael <:
So, I'll sumarize the previous discussions. The proposed images are: The image currently named image tags:. based on: respectively. image is currently used in Windows E2E tests and should already exist on the nodes, so download should be minimal. The image. a manifest list needs to be created containing the tags: The image is used to tag the image, which is used by kubelet, so it should already exist on the nodes. smaller in size (~350MB).

the test should create pods with the command . I'm fine with either one, although I think people will tend towards the nanoserver image, and in fairness, I think it will be the easiest one to maintain long-term. AFAIK, Microsoft updates their container images periodically, so, we should do the same for the image. If not, the test using the image will then have to pull the entire image (350 MB is ok, IMO). E.g. vs Well, a manifest list can contain both Linux and Windows images, So, if the busybox image is picked, the Linux busybox image can be included in the manifest as well. Ideally, most images work on both OSes (with some exceptions, like the nanoserver or alpine images), so I don't think there's a need to put windows in the path. And if a new root path is being for new test images, I would like to ask that images mustn't container slashes in their names (e.g.:). Docker doesn't support images with slashes in their names (see ). Even if all the images on eventually support Windows, it would still be a good idea to be able to use dockerhub for testing purposes.
We need an image - preferably not one as prone to CVEs as busybox WHOAH. That's a 2GB image. For testing. I am in favor of this just based on the size (I was thinking something like kuberneets' pause, which is 240 kB. Over 1000 times smaller. We have different comprehensions of what "nano" means. 350 MB is not OK. It's acceptable if we have no other choice, but honestly we should create our own that is as small as humanly possible. Then, we need a path. First, we'll be setting precedent for test images, so I need input on that. I propose as the root. Objections from test folks? I don't have enough context on e2e future, so I'd like some input from someone who does. Second, do we want to squirrel this away as a Windwows-specific binary, or should we not name it in any particular way, so that we can pull the same image for all tests via a multi-os, multi-arch manifest list (which can be done later). E.g.

vs I agree (except for busybox, which is a known vector for contaigion). Having a multi-os, multi-arch "serve_hostname" or something equally dumb seems ideal. Docker HUB doesn't. Docker itself does, and GCR does. If you're using dockerhub for testing, you have to change "" to "e2eteam/whatever", right? So what does it matter if the GCR path is longer? That said, I would be OK to have a single dir if that is what test-team wants. It just seems likely to get messy eventually.
Yes, dockerhub, I forgot the "hub" :) And yes, that's what we're doing and it works for the most part. But, if you take a look here [1], basically, the root is , which, as you know, is not accepted by dockerhub. So, the proposal was to not have "image names" with slashes in them (e.g.: in this case). [1]
I see. I think it would be OK to ask that images don't use subdirs, but the e2eRegistry variable could. e.g. // Docker hub e2eRegistry = "e2eteam" // GCR e2eRegistry = "" ResourceController = Config{e2eRegistry, "resource-consumer-controller", "1.0"} That seems workable, though it doesn't answer the question of the actual path in GCR :) Claudiu Belu <:
That seems workable, though it doesn't answer the question of the Yep, exactly right. :)
what are the next steps on this ticket to get the images to
I am still waiting for which image we are going to use, and for any objections to "" as a root. Michael Michael <:
Well, I think we are in agreement that we should use the nanoserver images. As for the root repo, I have no objections.
can you please explicitly give all the images to upload in an authoritative list? i am hoping we can upload all of these by monday EOD.
Sure. As we've discussed previously, the images are:
no manifest list image that supports both 1809 and 1803? are we OK for only having 1809 for that "pull from GCR" test?

Yeah, I meant those images should be included in a single manifest list (something like or something), like we've previously discussed. As for the second question, AFAIK, the GCE Windows CI is using Windows Server 1803 at the moment, so it would be nice to have the new test running on that too.
see above for the two images we want to push. we also need to repush this image resource-consumer/controller:1.0 to avoid the "/" as resource-consumer-controller cc:
i pulled these images on Windows: size of each is roughly ~350MB. and then saved them to TAR and uploaded them here: to go lower than that we might have hack custom images that trim, for instance a bunch of DLLs that we don't need (for e.g. a minimal pause image), however such a custom image will just require yet another unique download if the users are already basing their workload containers on official images. also for such hacks we need someone to verify if the nanoserver LICENSE allows that. i'm pretty sure it doesn't.
no hacks please. we need to upload the images as they officially come from microsoft. :)
/milestone v1.14 We're entering and this looks relevant to the 1.14 release. Please if I am incorrect
The slash thing is kinda irrelevant to me, it's just a limitation of Docker Hub as pointed out
i understand that. but can you make forward progress in testing that autoscale ticket without this work?
i think you have the two images to upload. let us know when this is complete and thank you so much!
The tars shouldn't be necessary. The nanoserver images include 2 layers that are type and should not be pushed. here's an example manifest from the bazel repo. you could probably use the existing rulesdocker to push nanoserver with no changes if you're building and pushing images with bazel
we can't follow this process due to Windows licensing. i will hide your post so that it is not followed by accident.
just to clarify - you can reference those in manifest lists, and docker push / pull them without actually pushing the layer contents to a manifest v2 complaint registry. That's the right way to do it.

That separates out user-owned files in the upper layers from those in the base layers that are always served from a Microsoft server due to licensing restrictions
Oh ok then. Keeping in mind the fact that the nanoserver image is a manifest itself (for amd64 and arm) and that you cannot reference another manifest list in your manifest list, then the the actual images which should be to the manifest list are: 1803 doesn't have multiarch.
the post is still quoted in your response. best to edit that as well? ok, looks like the licensing of these image is slightly more complicated than expected... i hope a tutorial is not required when pushing to GCR. :)
What is the status of this now? I am THOROUGHLY confused
me too according to here and here and as far as i understood, we only have to push the JSON files that hold layer references, but not the actual layers because this violates the licensing of the nanoserver images.
I'll post explicit steps using , etc.
a PowerShell window as an administrator Docker-CE 18.09.2 experimental mode for the Docker CLI Here's the copy & paste steps to pull the existing images, push them to the new registry, and create a manifest. Change $destinationRepo to be something like or your private repo name. The pushes will be nearly instantaneous because they're manifest-only pushes. All layer data is skipped. Now, you can inspect the manifest and it will show multiple images with different os.version And last, push it
Here's the example repo I made on When things are done this way, the original Windows image is still served from Microsoft as required by the license. If I were to add unique layers on top of this image - that content would be pushed to the registry.
Pushed to my staging, will try to get the promoter running ASAP
hmm, it looks like our image promoter is not quite ready for multi-platform yet. Let me try pushing to
yeah, you might be bumping into
Someone test that pull?
does that mean no Windows images can live on GCR? Am I misreading it?
I believe GCR supports it, but some of our tooling (which uses google/containerregistry) may not. OTOH I think tool is just using directly, so it might be unaffected.
!
Awesome, so that seems to work for now.

I guess the next step is to FROM that in your own containers.
or just use it in your tests. Do we close this for now, then?
SGTM, leaving the close call to
This looks right. I did a pull on a 1809 machine and the hashes match. is this enough to finish that test case, or are more images needed? We definitely need to get the images into a CI for 1.15. See /
this should be enough in terms of the test case.
^ FYI, there are now Windows test images at GCR. thanks !
Awesomeness, I've made a PR adding the testcase using this new image. This issue will close automatically once that merges. Test works for me. One question remains though. Do we have to have a Windows equivalent for the , so it fails on Windows. If we have to add a Windows equivalent for that too, then the nanoserver manifest list should also be pushed to the registry as well. Anyways. thanks for the help, everyone!
are you asking if can push the nanoserver manifest to the authenticated-image-pulling registry as well? Tim?
Yep. It is a conformance test, so it would be nice to be able to cover it as well.
any chance you push the nanoserver manifest to the authenticated-image-pulling registry as well? thanks for all your help! once you do the above, we can lose this issue
Try ""
thanks it works! !
please, close the issue once you check this.
thanks Tim! you rock!
can you please close this issue if we are done?
The issue is supposed to close when the PR merges. Closing anyways.


Question: item.fakeNodeHandler.Existing[0].Status = item.newNodeStatus item.fakeNodeHandler.Existing[1].Status = item.secondNodeNewStatus } <ins> if len(item.fakeNodeHandler.Existing[0].Labels) == 0 && len(item.fakeNodeHandler.Existing[1].Labels) == 0 { item.fakeNodeHandler.Existing[0].Labels = labels item.fakeNodeHandler.Existing[1].Labels = labels } </ins> if err := syncNodeStore(nodeController, item.fakeNodeHandler); err != nil { t.Errorf("unexpected error: %v", err) }

<!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"8+", GitVersion:"v1.8.0-alpha.1.51+-dirty", GitCommit:"", GitTreeState:"dirty", BuildDate:"2017-06-20T10:51:10Z", GoVersion:"go1.8", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"8+", GitVersion:"v1.8.0-alpha.1.51+-dirty", GitCommit:"", GitTreeState:"dirty", BuildDate:"2017-06-20T11:29:14Z", GoVersion:"go1.8", Compiler:"gc", Platform:"linux/amd64"} What happened: After adding label to a node, node controller panics. What you expected to happen: Everything should still work. How to reproduce it (as minimally and precisely as possible): Run and wait for the node controller to panic: Anything else we need to know: I've checked the code and it looks like the node controller expects the label to be present during node registration and it can not be later. Even if the label can not be latter it shouldn't panic.
/sig node
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
/assign let me try this :).


Question: v1 "k8s.io/api/core/v1" discoveryv1 "k8s.io/api/discovery/v1" <ins> apierrors "k8s.io/apimachinery/pkg/api/errors" </ins> "k8s.io/apimachinery/pkg/api/resource" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/labels"

Creating a destroying services test is a required test case in , but it's missing in the upstream e2e test. Since creating a destroying services test should work in both windows and linux cluster, it's worthy adding it to create a service and corresponding pod delete the service re-create the service load balancing rules for pods are recovered e2e tests for sig-network would have coverage for recover load balancing rules for pods after service is deleted and re-created. Reproducer is above No response <detailsmain </details<detailsany </details<details</details<details</details<details</details<details</details
/sig network testing
/triage accepted /assign


Question: return err } // being unable to set the version does not prevent the object from being extracted <del> versioner.UpdateObject(obj, rev) </del> <ins> if err := versioner.UpdateObject(obj, rev); err != nil { klog.Errorf("failed to update object version: %v", err) } </ins> if matched, err := pred.Matches(obj); err == nil && matched { v.Set(reflect.Append(v, reflect.ValueOf(obj).Elem())) }

Remove panic in , please return error instead following .
/sig api-machinery /good-first-issue
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /sig api-machinery /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
Please also add some log if returns error
/remove-kind bug /kind cleanup


Question: // Check RestartPolicy for container recentContainers, err := dockertools.GetRecentDockerContainersWithNameAndUUID(kl.dockerClient, podFullName, uid, container.Name) if err != nil { <del> glog.Errorf("Error listing recent containers with name and uid:%s--%s--%s", podFullName, uid, container.Name) </del> <ins> glog.Errorf("Error listing recent containers:%s", dockerContainerName) </ins> // TODO(dawnchen): error handling here? } if len(recentContainers) > 0 && pod.Spec.RestartPolicy.Always == nil { if pod.Spec.RestartPolicy.Never != nil { <del> glog.V(3).Infof("Already ran container with name %s--%s--%s, do nothing", podFullName, uid, container.Name) </del> <ins> glog.V(3).Infof("Already ran container with name %s, do nothing", dockerContainerName) </ins> continue } if pod.Spec.RestartPolicy.OnFailure != nil { // Check the exit code of last run if recentContainers[0].State.ExitCode == 0 { <del> glog.V(3).Infof("Already successfully ran container with name %s--%s--%s, do nothing", podFullName, uid, container.Name) </del> <ins> glog.V(3).Infof("Already successfully ran container with name %s, do nothing", dockerContainerName) </ins> continue } } } <del> glog.V(3).Infof("Container with name %s--%s--%s doesn't exist, creating %#v", podFullName, uid, container.Name, container) </del> <ins> glog.V(3).Infof("Container with name %s doesn't exist, creating %#v", dockerContainerName) </ins> ref, err := containerRef(pod, &container) if err != nil { glog.Errorf("Couldn't make a ref to pod %v, container %v: '%v'", pod.Name, container.Name, err)

-- is the old old format for names.
Just to be sure, is "pod [podFullName] container [container.Name]" the new preferred format? I also see some places comma separating that form.
The new form should either be a list of fields, a strict, or the result of BuildDockerName(...) On Jan 15, 2015 5:37 AM, "commonlisp" wrote:
Closed with


Question: return newRS, allOldRSs, nil } <ins> const ( // limit revision history length to 100 element (~2000 chars) maxRevHistoryLengthInChars = 2000 ) </ins> // Returns a replica set that matches the intent of the given deployment. Returns nil if the new replica set doesn't exist yet. // 1. Get existing new RS (the RS that the given deployment targets, whose pod template is the same as deployment's). // 2. If there's existing new RS, update its revision number if it's smaller than (maxOldRevision + 1), where maxOldRevision is the max revision number among all old RSes.

What happened: We created a Deployment and continuously updated an annotation on it periodically for a week. Eventually, we noticed errors in the controller-manager logs due to the annotation growing too large. Example annotations: controller-manager error: What you expected to happen: that do not change the revision number should not be to the revision history. revision history should be capped at a fixed number of elements, it should not be possible to have unbounded growth for Deployments that are frequently updated. How to reproduce it (as minimally and precisely as possible): a Deployment modify an annotation on it and note that the annotation gets the same number appended every time. Environment: Kubernetes version (use ):
/sig scalability Not 100% sure where this one fits, I couldn't determine which group owns the Deployment/ReplicaSet controllers, but this is a scalability problem
/sig apps
Agree with these, and if the limit is hit, the beginning of the history can be removed.
/assign


Question: for objRef, request := range requests { out := o.addPrefixIfNeeded(objRef, o.Out) if err := o.ConsumeRequestFn(request, out); err != nil { <del> return err </del> <ins> if !o.IgnoreLogErrors { return err } fmt.Fprintf(o.Out, "error: %vn", err) </ins> } }

What happened: Running on a pod with two containers, of which one has restarted but the other hasn't results in a undesired behavior. I get an inconsistent outcome, sometimes just a report of , other times I get the other containers logs followed by the error. What you expected to happen: I expect to get to see the full response of all containers, even though one container without any --previous logs throwed an error. How to reproduce it (as minimally and precisely as possible): The output can look like this... Anything else we need to know?: The flag by itself ("If watching / following pod logs, allow for any errors that occur to be non-fatal") doesn't influence this, but a workaround is to use to be able to see the full output. Kubernetes version (use ): Cloud provider or hardware configuration: GKE OS (e.g: ): Ubuntu 20.04 Kernel (e.g. ): 5.4.0-58-generic
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cli
do you have an example yaml file to create these containers?
yikes I feared I'd be asked this question :) Well, I provided a minimal example in the original post.
Thank you very much. I'll take a look.
Hmm... I can replicate this issue with the example you provided. Very explanatory instructions! I'll investigate tomorrow.
I used the example to reproduce this issue. Created pod through above. The state of pod is Running. I guess the reason is that none of the containers in pod have Terminated state.Only when one of containers is Terminated. The param --previous will return the result normally.
looking at these lines... ...I discovered that adding the option significantly increases the frequency that the "bash" container's logs are printed (the numbers are how many times was printed out of 100 runs):
Can i work on this issue?
/assign
Go ahead I'd love for you too. Hopefully my notes help!


Question: sets.NewString("lucretius"), sets.NewString("epicurus"), }, <ins> { name: "pending pods are ignored", pods: []*v1.Pod{ { ObjectMeta: metav1.ObjectMeta{ Name: "unscheduled", }, Status: v1.PodStatus{ Phase: v1.PodPending, }, }, }, metrics:             metricsclient.PodMetricsInfo{}, resource:            v1.ResourceCPU, expectReadyPodCount: 0, expectIgnoredPods:   sets.NewString("unscheduled"), expectMissingPods:   sets.NewString(), }, </ins> } for _, tc := range tests { <del> readyPodCount, ignoredPods, missingPods := groupPods(tc.pods, tc.metrics, tc.resource, defaultTestingCpuInitializationPeriod, defaultTestingDelayOfInitialReadinessStatus) if readyPodCount != tc.expectReadyPodCount { t.Errorf("%s got readyPodCount %d, expected %d", tc.name, readyPodCount, tc.expectReadyPodCount) } if !ignoredPods.Equal(tc.expectIgnoredPods) { t.Errorf("%s got unreadyPods %v, expected %v", tc.name, ignoredPods, tc.expectIgnoredPods) } if !missingPods.Equal(tc.expectMissingPods) { t.Errorf("%s got missingPods %v, expected %v", tc.name, missingPods, tc.expectMissingPods) } </del> <ins> t.Run(tc.name, func(t *testing.T) { readyPodCount, ignoredPods, missingPods := groupPods(tc.pods, tc.metrics, tc.resource, defaultTestingCpuInitializationPeriod, defaultTestingDelayOfInitialReadinessStatus) if readyPodCount != tc.expectReadyPodCount { t.Errorf("%s got readyPodCount %d, expected %d", tc.name, readyPodCount, tc.expectReadyPodCount) } if !ignoredPods.Equal(tc.expectIgnoredPods) { t.Errorf("%s got unreadyPods %v, expected %v", tc.name, ignoredPods, tc.expectIgnoredPods) } if !missingPods.Equal(tc.expectMissingPods) { t.Errorf("%s got missingPods %v, expected %v", tc.name, missingPods, tc.expectMissingPods) } }) </ins> } }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When the HPA scales up beyond the capacity of the cluster, many unscheduled pods are created. When load is removed, the usage ratio drops and the HPA tries to scale down. The unscheduled pods do not produce metrics, so the HPA fills in the 100% usage and re-evaluates the usage ratio. Because of the many fake 100% metrics, the new usage ratio is above the target value and the direction of scale is reversed. In this case, the HPA decides not to make any change to the current scale. What you expected to happen: I would expect the HPA to not get stuck when load is removed. How to reproduce it (as minimally and precisely as possible): and put it under load in a cluster without cluster autoscaling. the Deployment to increase the CPU request beyond the capability of the cluster (e.g. 500m). This should produce many Pending pods. load. The Pending pods will keep the HPA scaled indefinitely. Anything else we need to know?: Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-06-06T01:44:30Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"12+", GitVersion:"v1.12.9-gke.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-06-06T21:30:53Z", GoVersion:"go1.10.8b4", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig autoscaling
This might Previous I suspected that this was the same issues as (HPA positive feedback loop).

However it turns out to be different, that Pending pods cause HPA to get stuck.
Original reason for using 100% utilization for missing metrics when scaling down:


Question: fs.DurationVar(&o.NodeMonitorPeriod.Duration, "node-monitor-period", o.NodeMonitorPeriod.Duration, fmt.Sprintf("The period for syncing NodeStatus in %s.", names.CloudNodeLifecycleController)) fs.StringVar(&o.ClusterName, "cluster-name", o.ClusterName, "The instance prefix for the cluster.") <del> fs.StringVar(&o.ClusterCIDR, "cluster-cidr", o.ClusterCIDR, "CIDR Range for Pods in cluster. Requires --allocate-node-cidrs to be true") fs.BoolVar(&o.AllocateNodeCIDRs, "allocate-node-cidrs", false, "Should CIDRs for Pods be allocated and set on the cloud provider.") </del> <ins> fs.StringVar(&o.ClusterCIDR, "cluster-cidr", o.ClusterCIDR, "CIDR Range for Pods in cluster. Only used when --allocate-node-cidrs=true; if false, this option will be ignored.") fs.BoolVar(&o.AllocateNodeCIDRs, "allocate-node-cidrs", false, "Should CIDRs for Pods be allocated and set on the cloud provider. Requires --cluster-cidr.") </ins> fs.StringVar(&o.CIDRAllocatorType, "cidr-allocator-type", "RangeAllocator", "Type of CIDR allocator to use") fs.BoolVar(&o.ConfigureCloudRoutes, "configure-cloud-routes", true, "Should CIDRs allocated by allocate-node-cidrs be configured on the cloud provider.")

In on --cluster-cidr flag is written that it requires --allocate-node-cidrs to be true. Bu actual, we can set non empty --cluster-cidr and disable node cidr allocation (set --allocate-node-cidrs to false). I expect to see in documentation of --allocate-node-cidrs flag note that for this node pod cidr allocation functionality is required to set the --cluster-cidr flag. Also in documentation on --cluster-cidr flag I expect removing mention about that it requires --allocate-node-cidrs flag to be true. Run controller manager with 2 flags: --allocate-node-cidrs=false --cluster-cidr=172.20.0.0/16 We are using this functionality to change the expand cluster CIDR subnet. As I see, we can temporary disable nodeIPAM controller work by setting --allocate-node-cidrs=false. I found a with the code that ignores cluster cidr parsing if allocate-node-cidrs flag is false. kubernetes version 1.23.17 bare metal No response No response No response No response
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig docs
if you disable the value of is not used anyhwere, is it? I do not understand exactly what do you mean here, can you elaborate in your use case?
/assign
We have a cluster which was initialized with and . Our cluster grew to 65 nodes. In kube-controlle-manager logs we observed errors like: This error appears because the nodeIPAM controller is enabled and it tries to allocate a podCIDR subnet with /24 mask for each node. have 64 subnets - so for 65'th node there is now free subnet. We want to expand cluster CIDR subnet. For this action, while maintenance, we want to disable kube-controller-manager's nodeIPAM controller. Actually we can do this by changing only one flag in kube-controller-manager . The flag can remain unchanged.

Yes it is. The main idea of the issue is to change the affirmation in . Would be more transparent if flag will require flag to be set. Bot not vice versa. From the docs It is not clear that you can disable allocation of node CIDR and leave the flag cluster-cidr. I try to thing back side: If the flags and are 100% connected and there are used only by nodeIPAM, why do we need this separate flags. As I understand: configures the CIDR parameter for nodeIPAM controller. enables the controller. So if I am right in previous affirmations, the flag actually does not require to be true. NodeIPAM controller requires and with a subnet set.
/cc as we talked about this today
The Kubernetes project currently lacks enough contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/remove-lifecycle rotten
The Kubernetes project currently lacks enough contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at .

/lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/remove-lifecycle rotten
/assign


Question: Eventually(terminateContainer.Present, ContainerStatusRetryTimeout, ContainerStatusPollInterval).Should(BeFalse()) } }) <ins> }) Context("on terminated container", func() { </ins> rootUser := int64(0) nonRootUser := int64(10000) <del> for _, testCase := range []struct { name      string container v1.Container phase     v1.PodPhase message   gomegatypes.GomegaMatcher }{ { name: "if TerminationMessagePath is set [NodeConformance]", container: v1.Container{ Image:                  framework.BusyBoxImage, Command:                []string{"/bin/sh", "-c"}, Args:                   []string{"/bin/echo -n DONE > /dev/termination-log"}, TerminationMessagePath: "/dev/termination-log", SecurityContext: &v1.SecurityContext{ RunAsUser: &rootUser, }, }, phase:   v1.PodSucceeded, message: Equal("DONE"), }, { name: "if TerminationMessagePath is set as non-root user and at a non-default path [NodeConformance]", container: v1.Container{ Image:                  framework.BusyBoxImage, Command:                []string{"/bin/sh", "-c"}, Args:                   []string{"/bin/echo -n DONE > /dev/termination-custom-log"}, TerminationMessagePath: "/dev/termination-custom-log", SecurityContext: &v1.SecurityContext{ RunAsUser: &nonRootUser, }, }, phase:   v1.PodSucceeded, message: Equal("DONE"), }, { name: "from log output if TerminationMessagePolicy FallbackToLogOnError is set [NodeConformance]", container: v1.Container{ Image:                    framework.BusyBoxImage, Command:                  []string{"/bin/sh", "-c"}, Args:                     []string{"/bin/echo -n DONE; /bin/false"}, TerminationMessagePath:   "/dev/termination-log", TerminationMessagePolicy: v1.TerminationMessageFallbackToLogsOnError, }, phase:   v1.PodFailed, message: Equal("DONE"), }, { name: "as empty when pod succeeds and TerminationMessagePolicy FallbackToLogOnError is set [NodeConformance]", container: v1.Container{ Image:                    framework.BusyBoxImage, Command:                  []string{"/bin/sh", "-c"}, Args:                     []string{"/bin/echo DONE; /bin/true"}, TerminationMessagePath:   "/dev/termination-log", TerminationMessagePolicy: v1.TerminationMessageFallbackToLogsOnError, }, phase:   v1.PodSucceeded, message: Equal(""), }, { name: "from file when pod succeeds and TerminationMessagePolicy FallbackToLogOnError is set [NodeConformance]", container: v1.Container{ Image:                    framework.BusyBoxImage, Command:                  []string{"/bin/sh", "-c"}, Args:                     []string{"/bin/echo -n OK > /dev/termination-log; /bin/echo DONE; /bin/true"}, TerminationMessagePath:   "/dev/termination-log", TerminationMessagePolicy: v1.TerminationMessageFallbackToLogsOnError, }, phase:   v1.PodSucceeded, message: Equal("OK"), }, } { testCase := testCase It(fmt.Sprintf("should report termination message %s [LinuxOnly]", testCase.name), func() { // Cannot mount files in Windows Containers. testCase.container.Name = "termination-message-container" c := ConformanceContainer{ PodClient:     f.PodClient(), Container:     testCase.container, RestartPolicy: v1.RestartPolicyNever, } By("create the container") c.Create() defer c.Delete() </del> <ins> // Create and then terminate the container under defined PodPhase to verify if termination message matches the expected output. Lastly delete the created container. matchTerminationMessage := func(container v1.Container, expectedPhase v1.PodPhase, expectedMsg gomegatypes.GomegaMatcher) { container.Name = "termination-message-container" c := ConformanceContainer{ PodClient:     f.PodClient(), Container:     container, RestartPolicy: v1.RestartPolicyNever, } </ins> <del> By(fmt.Sprintf("wait for the container to reach %s", testCase.phase)) Eventually(c.GetPhase, ContainerStatusRetryTimeout, ContainerStatusPollInterval).Should(Equal(testCase.phase)) </del> <ins> By("create the container") c.Create() defer c.Delete() </ins> <del> By("get the container status") status, err := c.GetStatus() Expect(err).NotTo(HaveOccurred()) </del> <ins> By(fmt.Sprintf("wait for the container to reach %s", expectedPhase)) Eventually(c.GetPhase, ContainerStatusRetryTimeout, ContainerStatusPollInterval).Should(Equal(expectedPhase)) </ins> <del> By("the container should be terminated") Expect(GetContainerState(status.State)).To(Equal(ContainerStateTerminated)) </del> <ins> By("get the container status") status, err := c.GetStatus() Expect(err).NotTo(HaveOccurred()) </ins> <del> By("the termination message should be set") Expect(status.State.Terminated.Message).Should(testCase.message) </del> <ins> By("the container should be terminated") Expect(GetContainerState(status.State)).To(Equal(ContainerStateTerminated)) </ins> <del> By("delete the container") Expect(c.Delete()).To(Succeed()) }) </del> <ins> By("the termination message should be set") framework.Logf("Expected: %v to match Container's Termination Message: %v --", expectedMsg, status.State.Terminated.Message) Expect(status.State.Terminated.Message).Should(expectedMsg) By("delete the container") Expect(c.Delete()).To(Succeed()) </ins> } <ins> It("should report termination message [LinuxOnly] if TerminationMessagePath is set [NodeConformance]", func() { // Cannot mount files in Windows Containers. container := v1.Container{ Image:                  framework.BusyBoxImage, Command:                []string{"/bin/sh", "-c"}, Args:                   []string{"/bin/echo -n DONE > /dev/termination-log"}, TerminationMessagePath: "/dev/termination-log", SecurityContext: &v1.SecurityContext{ RunAsUser: &rootUser, }, } matchTerminationMessage(container, v1.PodSucceeded, Equal("DONE")) }) It("should report termination message [LinuxOnly] if TerminationMessagePath is set as non-root user and at a non-default path [NodeConformance]", func() { // Cannot mount files in Windows Containers. container := v1.Container{ Image:                  framework.BusyBoxImage, Command:                []string{"/bin/sh", "-c"}, Args:                   []string{"/bin/echo -n DONE > /dev/termination-custom-log"}, TerminationMessagePath: "/dev/termination-custom-log", SecurityContext: &v1.SecurityContext{ RunAsUser: &nonRootUser, }, } matchTerminationMessage(container, v1.PodSucceeded, Equal("DONE")) }) It("should report termination message [LinuxOnly] from log output if TerminationMessagePolicy FallbackToLogsOnError is set [NodeConformance]", func() { // Cannot mount files in Windows Containers. container := v1.Container{ Image:                    framework.BusyBoxImage, Command:                  []string{"/bin/sh", "-c"}, Args:                     []string{"/bin/echo -n DONE; /bin/false"}, TerminationMessagePath:   "/dev/termination-log", TerminationMessagePolicy: v1.TerminationMessageFallbackToLogsOnError, } matchTerminationMessage(container, v1.PodFailed, Equal("DONE")) }) It("should report termination message [LinuxOnly] as empty when pod succeeds and TerminationMessagePolicy FallbackToLogsOnError is set [NodeConformance]", func() { // Cannot mount files in Windows Containers. container := v1.Container{ Image:                    framework.BusyBoxImage, Command:                  []string{"/bin/sh", "-c"}, Args:                     []string{"/bin/echo DONE; /bin/true"}, TerminationMessagePath:   "/dev/termination-log", TerminationMessagePolicy: v1.TerminationMessageFallbackToLogsOnError, } matchTerminationMessage(container, v1.PodSucceeded, Equal("")) }) It("should report termination message [LinuxOnly] from file when pod succeeds and TerminationMessagePolicy FallbackToLogsOnError is set [NodeConformance]", func() { // Cannot mount files in Windows Containers. container := v1.Container{ Image:                    framework.BusyBoxImage, Command:                  []string{"/bin/sh", "-c"}, Args:                     []string{"/bin/echo -n OK > /dev/termination-log; /bin/echo DONE; /bin/true"}, TerminationMessagePath:   "/dev/termination-log", TerminationMessagePolicy: v1.TerminationMessageFallbackToLogsOnError, } matchTerminationMessage(container, v1.PodSucceeded, Equal("OK")) }) </ins> }) Context("when running a container with a new image", func() { <del> // The service account only has pull permission auth := ` </del> <ins> // Images used for ConformanceContainer are not added into NodeImageWhiteList, because this test is // testing image pulling, these images don't need to be prepulled. The ImagePullPolicy // is v1.PullAlways, so it won't be blocked by framework image white list check. imagePullTest := func(image string, hasSecret bool, expectedPhase v1.PodPhase, expectedPullStatus bool) { container := ConformanceContainer{ PodClient: f.PodClient(), Container: v1.Container{ Name:            "image-pull-test", Image:           image, Command:         []string{"/bin/sh", "-c", "while true; do sleep 1; done"}, ImagePullPolicy: v1.PullAlways, }, RestartPolicy: v1.RestartPolicyNever, } if hasSecret { // The service account only has pull permission auth := ` </ins> { "auths": { "https://gcr.io": {

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Table-based e2e tests are not getting updated to leading to the following error- ( mentioned in ) framework.ConformanceIt() requires the first parameter to be and are associated with at runtime based on different cases/test-scenarios. What you expected to happen: Either disapprove the use of table-based e2e tests or suggestion on an alternative approach to add such tests to Conformance. How to reproduce it (as minimally and precisely as possible): framework.ConformanceIt() to . /area conformance cc
/sig testing
/cc
Approaches to work around this: wrap the whole table based test in a ConformanceIt block refactor to a more boilerplate approach that has a ConformanceIt block for each table entry update the conformance test walker to detect table based tests that use a well known field?
Thanks I am evaluating these approaches especially on "conformance test walker" to retrieve well-known fields like and variables used inside .
Does this close if closes or are there other tests that need to transition?
This issue would be No other e2e test needs the transition from table-based test structure. Though "only one" remaining test should be restructured as an individual test.
/close Closing as per other fruitful approach and efforts would be taken to refactor table-based tests into individual tests wherever needed.
Closing this issue. <detailsIn response to : /close Closing as per other fruitful approach and efforts would be taken to refactor table-based tests into individual tests wherever needed. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: var ready bool if c.State.Running == nil { ready = false <del> } else if result, ok := m.readinessManager.Get(kubecontainer.ParseContainerID(c.ContainerID)); ok { ready = result == results.Success </del> <ins> } else if result, ok := m.readinessManager.Get(kubecontainer.ParseContainerID(c.ContainerID)); ok && result == results.Success { ready = true </ins> } else { // The check whether there is a probe which hasn't run yet. w, exists := m.getWorker(podUID, c.Name, readiness)

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Seems to be impacting a few KIND jobs on the master branch the most, but also assorted other jobs see the triage link above Reported by Sample failure: /sig node <!-- Please see the deflaking doc () for more guidance! --
xref probing tests
/assign ref /triage accepted
Thanks Elana, still trying today... Does anyone know if we can use the kubelet logs in e2e tests?
Not very portably in cluster e2e tests at least. Maybe in node_e2e instead.
Have you considered that the issue is not with the test but with the underlying code in kubelet like detailed in this issue Like detailed in the issue I can reliably reproduce (on k8s 1.18) the startup probe passing within a few seconds, but the container is only marked as ready after about a minute.
there is already a fix in the 1.21 release for that (part of a bigger PR): And actually this test was to verify that part of the code, but unfortunately sometimes it doesn't respond as quick as it should (but never on my PC, only in ci/cd tests)...
Some recent flakes:
one of the most common flakes in kind jobs currently failures like: Mar 25 17:03:42.508: Pod became ready in 42.007286524s, more than 5s after startupProbe succeeded. It means that the delay readiness probes were not initiated immediately after startup finished.
Maybe we could speed up the review on
Thank you I will take a look. I'm making a pass through flakes tonight trying to identify if anything is on kind's end
I don't think it's kind... since running on my PC with kind it works 100%, I think the issue is either the resources... or contention in the queue as mentioned to in the comment
/milestone v1.22
commented on PR with the suggested fix
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.

</details
Yeah...
Thanks for your great work in keeping things in check
FYI
I have (finally) understood what's happening, and I can even explain why the deflake made things worse... Please have a look at the sleep before the worker starts to process probes: So, you have to remember that we have one worker per pod and per probe, and since we have these probe settings for the test: We have (PeriodSeconds=10) starting sometimes immediately (if the Kubelet has started since more than 10s) or after a quick sleep of rand10s. On the contrary, will likely sleep most of the time for rand60s. This causes a delay in the processing of notifications: If we are lucky, this delay is less than 5s and the test passes: Now you can understand why the deflaked settings were actually worse:
The reason I didn't spot these was that I usually start my cluster once for all and iterate on the tests... so the kubelet had plenty of time to start and I never ended in that sleep.
For the solution, I suggest we either remove the sleep completely, or we find something that would make all 3 probe workers start at the same time to avoid sync issues... For the record, I was able to spot this issue thanks to the following log error (because the channel was overflowing): What do you think ?
I have found another issue in that code: If has already ran (and failed), we go to L245 and don't probe via and therefore wait for the next tick, making the test fail.
This seems consistent with what I found when I was adding e2es for the new probe-level termination grace period and why I did my timing math the way I did.
/reopen I don't think this is fixed, I just hit it again and gubernator is still showing plenty of flakes:
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
how to check the exact commit used for each test?

For instance, I am not sure (from the name) that should include my latest fix...
In your PR the fix wasn't there:
you may send cherry-pick to 1.21 for which is using kubelet 1.21, not 1.22.
is using kubelet. Cherry-pick to 1.20 may be needed as well.
I have found this one that failed even with the fix: corresponding to:
I don't know what could explain this atm... but it seems we still have delay propagating container statuses.
In the job artifacts there are multiple places with this info including the clone records/logs, and In testgrid it's a row.
yeah afaict since the tests ran on my PR after your fix was merged into master, it showed the flake was still present :( Prow will always use the latest master when running CI.
Sorry for doubting what you said... I was probably in deny and hoped this annoyance was fixed once for all :(
Plan: we'll see if Clayton's pod lifecycle refactor and/or the jitter refactor for probes affects this.
Clayton's PR jitter refactor
since isn't (likely) coming to 1.22 we could cut off 90% of flakes by increasing to 15s delay:
it seems it fails even after merging so let's continue our discussion on slack
this is not a regression, our suggestion is not block release on it. feel free to remove the milestone
/milestone v1.23
In Kubelet maybe, but in the flakiness of the e2e tests it was. Unfortunately we can't easily explore the current state :/
In the release blocking jobs the frequency is not horrible now, so unblocking is probably fine, it is not as bad as it was. Outside of the release cycle this is still the flakiest test in some jobs, see e.g. where it's the only flake source we have on record currently.
if we end up needing it, we have a timing/period adjustment PR we can take that should help with the flakiness with


Question: // IPTablesMasqueradeBit must be specified or defaulted. return nil, fmt.Errorf("Unable to read IPTablesMasqueradeBit from config") } <del> proxierIPTables, err := iptables.NewProxier(iptInterface, utilsysctl.New(), execer, config.IPTablesSyncPeriod.Duration, config.IPTablesMinSyncPeriod.Duration, config.MasqueradeAll, int(*config.IPTablesMasqueradeBit), config.ClusterCIDR, hostname, getNodeIP(client, hostname)) </del> <ins> proxierIPTables, err := iptables.NewProxier( iptInterface, utilsysctl.New(), execer, config.IPTablesSyncPeriod.Duration, config.IPTablesMinSyncPeriod.Duration, config.MasqueradeAll, int(*config.IPTablesMasqueradeBit), config.ClusterCIDR, hostname, getNodeIP(client, hostname), recorder, ) </ins> if err != nil { glog.Fatalf("Unable to create proxier: %v", err) }

When kube-proxy fails to open a nodeport, it gives up setting up the rest of the iptables rules for the service. Leaving the service not usable. Cannot even use it with clusterIP. Kube-proxy only logs this error in . No useful information can be retrieved with kubectl. Need to record a event at least.
Ideally would the changes be removed and then reported back? Is there a preferred log or place to return the error?
the event maybe a better solution; as other port maybe success.
Need to surface the problem somewhere. An event seems okay
, there's and in ; should we reuse it or create new recorders for ?
Just reuse it. After all, it is within the same component. Are you going to submit a PR?
sure, my pleasure.
Great! Thanks! Let me know when the PR is ready.


Question: if (pattern.VolMode == v1.PersistentVolumeBlock && t.runTestBlock != nil) || (pattern.VolMode == v1.PersistentVolumeFilesystem && t.runTestFile != nil) { ginkgo.It(t.testItStmt, func() { <del> if pattern.VolMode == v1.PersistentVolumeBlock && driver.GetDriverInfo().InTreePluginName == "kubernetes.io/local-volume" { // TODO: https://github.com/kubernetes/kubernetes/issues/74552 framework.Skipf("Local volume volume plugin does not support block volume reconstruction (#74552)") } </del> init() defer cleanup()

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: Which test(s) are failing: [sig-storage] In-tree Volumes [Driver: local][LocalVolumeType: block] [Testpattern: Pre-provisioned PV (block volmode)] disruptive[Disruptive] Should test that pv used in a pod that is force deleted while the kubelet is down cleans up when the kubelet returns. Since when has it been failing: 10/8/19 Testgrid link: https://k8s- Reason for failure: Introduced by but root cause is probably Anything else we need to know:
/assign
/milestone 1.17 /priority important-soon this is consistently failing in release-master-informing job could you please take a look?
The provided milestone is not valid for this repository. Milestones in this repository: [, , , , , , , , ] Use to clear the milestone. <detailsIn response to : /milestone 1.17 /priority important-soon Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/milestone v1.17