Question: // RecordNodeStatusChange records a event related to a node status change. (Common to lifecycle and ipam) func RecordNodeStatusChange(recorder record.EventRecorder, node *v1.Node, newStatus string) { ref := &v1.ObjectReference{ <del> Kind:      "Node", Name:      node.Name, UID:       node.UID, Namespace: "", </del> <ins> APIVersion: "v1", Kind:       "Node", Name:       node.Name, UID:        node.UID, Namespace:  "", </ins> } klog.V(2).Infof("Recording status change %s event message for node %s", newStatus, node.Name) // TODO: This requires a transaction, either both node status is updated

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: I'm attempting to use an watcher to better re-act to cluster modifications. I've received the below event: What you expected to happen: I would expect the to include attribute. Not sure if it's all node events or just this particular one. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): 1.16 Cloud provider or hardware configuration: bare metal, rancher's rke OS (e.g: ): centos host nodes Kernel (e.g. ): 3.18, 4.4 Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig api-machinery
Just got another from kubelet..
/remove-sig api-machinery /sig node
recording code is here:


Question: "k8s.io/kubernetes/cmd/kube-apiserver/app" "k8s.io/kubernetes/cmd/kube-apiserver/app/options" <del> "k8s.io/kubernetes/cmd/kubeadm/app/util/pkiutil" </del> testutil "k8s.io/kubernetes/test/utils" )

found this by running tests on this PR: it's problematic for a couple of reasons: prevents kubeadm from moving out (eventually) there are no API guarantees in the given code/structs/functions in pkiutils the logic in cmd/kube- must be reworked to generate cert/key pair without using kubeadm. originally in pending fix PR
/sig api-machinery cluster-lifecycle architecture /area code-organization
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /sig api-machinery sig-cluster-lifecycle architecture /area code-organization </details
/assign
cc
Please make sure to add an import restriction to prevent a regression in the future.
/triage accepted


Question: // endpointSliceTracker or the managed-by value of the EndpointSlice has changed // from or to this controller. func (c *Controller) onEndpointSliceUpdate(prevObj, obj interface{}) { <del> prevEndpointSlice := obj.(*discovery.EndpointSlice) </del> <ins> prevEndpointSlice := prevObj.(*discovery.EndpointSlice) </ins> endpointSlice := obj.(*discovery.EndpointSlice) if endpointSlice == nil || prevEndpointSlice == nil { utilruntime.HandleError(fmt.Errorf("Invalid EndpointSlice provided to onEndpointSliceUpdate()"))

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Mistake use obj as prevObj prevEndpointSlice := obj.(discovery.EndpointSlice) What you expected to happen: prevEndpointSlice := prevObj.(discovery.EndpointSlice) How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: cc
/sig network
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/remove-triage unresolved PR for fixing it Thanks


Question: } func TestEnsureInternalBackendServiceGroups(t *testing.T) { <ins> t.Parallel() </ins> for desc, tc := range map[string]struct { mockModifier func(*cloud.MockGCE) }{

Seeing several of the tests in ./pkg/cloudprovider/providers/gce flake 1 in 5-10 runs when run locally (I am logged in to gcloud). Is it possible one of the APIs is not correctly shimmed and is fetching real data?
you were the last folks in here - why would this and other internal_tests in here be flaking?
Happens in both master and 1.11 release branches
/cc cheftako


Question: if apierrors.IsNotFound(err) { c.triggerTimeTracker.DeleteService(namespace, name) c.reconciler.deleteService(namespace, name) <ins> c.endpointSliceTracker.DeleteService(namespace, name) </ins> // The service has been deleted, return nil so that it won't be retried. return nil }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: The endpointSliceTracker creates a set of resource versions for each service, the resource versions in the set could be deleted when endpointslices are deleted, but the set and its key in the map is never deleted, leading to memory leak. What you expected to happen: The set of resource versions should be delete from endpointSliceTracker when the service is deleted. Environment: Kubernetes version (use ): 1.18.5
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc


Question: Expect(len(nodes.Items)).NotTo(BeZero(), "No available nodes for scheduling") scName = fmt.Sprintf("%v-%v", testSCPrefix, f.Namespace.Name) // Choose the first node <del> node0 = &config.nodes.Items[0] </del> <ins> node0 = &nodes.Items[0] </ins> config = &localTestConfig{ ns:     f.Namespace.Name,

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: What you expected to happen: Test should pass
/sig storage
can you take a look?
cc
This part looks a bit suspicious, config var at this moment is not initialized properly, it is just a pointer to nil. var ( config localTestConfig node0 v1.Node scName string ) Panic here is because I think config points nowhere node0 = [0] Here is only where config gets instantiated. config = &localTestConfig{ ns: f.Namespace.Name, client: f.ClientSet, nodes: nodes, node0: node0, scName: scName, } Am I on the right track here?
Yeah, looks like should not use
+1 we should just use nodes.Items[0] wondering how this passed ci ...
It didn't :-) See
hmm, ok since u spotted the bug, do you want to put a PR for it?
if you can do it now, then please go ahead. otherwise I will do it tomorrow morning since it is already late on East Coast.
I've created , PTAL


Question: b.errs = append(b.errs, fmt.Errorf("no internal mapper provided")) return b } <del> if b.mapper != nil { </del> <ins> if b.mapper != nil && b.mapper != b.internal { </ins> b.errs = append(b.errs, fmt.Errorf("another mapper was already selected, cannot use internal types")) return b }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: I run and I got Running with shows What you expected to happen: How to reproduce it (as minimally and precisely as possible): With Kubernetes master branch and in different terminal on the same machine Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: VM via libvirtd OS (e.g. from /etc/os-release): Fedora 27 with golang-1.10.1-1.fc29.x86_64 from rawhide Kernel (e.g. ):Install tools: ?? Others:
/sig cli
/assign
/assign


Question: } } <del> func TestGCEDiskConflicts(t *testing.T) { volState := v1.PodSpec{ Volumes: []v1.Volume{ { VolumeSource: v1.VolumeSource{ GCEPersistentDisk: &v1.GCEPersistentDiskVolumeSource{ PDName: "foo", }, }, }, }, } volState2 := v1.PodSpec{ Volumes: []v1.Volume{ { VolumeSource: v1.VolumeSource{ GCEPersistentDisk: &v1.GCEPersistentDiskVolumeSource{ PDName: "bar", }, }, }, }, } tests := []struct { pod      *v1.Pod nodeInfo *schedulernodeinfo.NodeInfo isOk     bool name     string }{ {&v1.Pod{}, schedulernodeinfo.NewNodeInfo(), true, "nothing"}, {&v1.Pod{}, schedulernodeinfo.NewNodeInfo(&v1.Pod{Spec: volState}), true, "one state"}, {&v1.Pod{Spec: volState}, schedulernodeinfo.NewNodeInfo(&v1.Pod{Spec: volState}), false, "same state"}, {&v1.Pod{Spec: volState2}, schedulernodeinfo.NewNodeInfo(&v1.Pod{Spec: volState}), true, "different state"}, } expectedFailureReasons := []PredicateFailureReason{ErrDiskConflict} for _, test := range tests { t.Run(test.name, func(t *testing.T) { ok, reasons, err := NoDiskConflict(test.pod, nil, test.nodeInfo) if err != nil { t.Errorf("unexpected error: %v", err) } if !ok && !reflect.DeepEqual(reasons, expectedFailureReasons) { t.Errorf("unexpected failure reasons: %v, want: %v", reasons, expectedFailureReasons) } if test.isOk && !ok { t.Errorf("expected ok, got none.  %v %s", test.pod, test.nodeInfo) } if !test.isOk && ok { t.Errorf("expected no ok, got one.  %v %s", test.pod, test.nodeInfo) } }) } } func TestAWSDiskConflicts(t *testing.T) { volState := v1.PodSpec{ Volumes: []v1.Volume{ { VolumeSource: v1.VolumeSource{ AWSElasticBlockStore: &v1.AWSElasticBlockStoreVolumeSource{ VolumeID: "foo", }, }, }, }, } volState2 := v1.PodSpec{ Volumes: []v1.Volume{ { VolumeSource: v1.VolumeSource{ AWSElasticBlockStore: &v1.AWSElasticBlockStoreVolumeSource{ VolumeID: "bar", }, }, }, }, } tests := []struct { pod      *v1.Pod nodeInfo *schedulernodeinfo.NodeInfo isOk     bool name     string }{ {&v1.Pod{}, schedulernodeinfo.NewNodeInfo(), true, "nothing"}, {&v1.Pod{}, schedulernodeinfo.NewNodeInfo(&v1.Pod{Spec: volState}), true, "one state"}, {&v1.Pod{Spec: volState}, schedulernodeinfo.NewNodeInfo(&v1.Pod{Spec: volState}), false, "same state"}, {&v1.Pod{Spec: volState2}, schedulernodeinfo.NewNodeInfo(&v1.Pod{Spec: volState}), true, "different state"}, } expectedFailureReasons := []PredicateFailureReason{ErrDiskConflict} for _, test := range tests { t.Run(test.name, func(t *testing.T) { ok, reasons, err := NoDiskConflict(test.pod, nil, test.nodeInfo) if err != nil { t.Errorf("unexpected error: %v", err) } if !ok && !reflect.DeepEqual(reasons, expectedFailureReasons) { t.Errorf("unexpected failure reasons: %v, want: %v", reasons, expectedFailureReasons) } if test.isOk && !ok { t.Errorf("expected ok, got none.  %v %s", test.pod, test.nodeInfo) } if !test.isOk && ok { t.Errorf("expected no ok, got one.  %v %s", test.pod, test.nodeInfo) } }) } } func TestRBDDiskConflicts(t *testing.T) { volState := v1.PodSpec{ Volumes: []v1.Volume{ { VolumeSource: v1.VolumeSource{ RBD: &v1.RBDVolumeSource{ CephMonitors: []string{"a", "b"}, RBDPool:      "foo", RBDImage:     "bar", FSType:       "ext4", }, }, }, }, } volState2 := v1.PodSpec{ Volumes: []v1.Volume{ { VolumeSource: v1.VolumeSource{ RBD: &v1.RBDVolumeSource{ CephMonitors: []string{"c", "d"}, RBDPool:      "foo", RBDImage:     "bar", FSType:       "ext4", }, }, }, }, } tests := []struct { pod      *v1.Pod nodeInfo *schedulernodeinfo.NodeInfo isOk     bool name     string }{ {&v1.Pod{}, schedulernodeinfo.NewNodeInfo(), true, "nothing"}, {&v1.Pod{}, schedulernodeinfo.NewNodeInfo(&v1.Pod{Spec: volState}), true, "one state"}, {&v1.Pod{Spec: volState}, schedulernodeinfo.NewNodeInfo(&v1.Pod{Spec: volState}), false, "same state"}, {&v1.Pod{Spec: volState2}, schedulernodeinfo.NewNodeInfo(&v1.Pod{Spec: volState}), true, "different state"}, } expectedFailureReasons := []PredicateFailureReason{ErrDiskConflict} for _, test := range tests { t.Run(test.name, func(t *testing.T) { ok, reasons, err := NoDiskConflict(test.pod, nil, test.nodeInfo) if err != nil { t.Errorf("unexpected error: %v", err) } if !ok && !reflect.DeepEqual(reasons, expectedFailureReasons) { t.Errorf("unexpected failure reasons: %v, want: %v", reasons, expectedFailureReasons) } if test.isOk && !ok { t.Errorf("expected ok, got none.  %v %s", test.pod, test.nodeInfo) } if !test.isOk && ok { t.Errorf("expected no ok, got one.  %v %s", test.pod, test.nodeInfo) } }) } } func TestISCSIDiskConflicts(t *testing.T) { volState := v1.PodSpec{ Volumes: []v1.Volume{ { VolumeSource: v1.VolumeSource{ ISCSI: &v1.ISCSIVolumeSource{ TargetPortal: "127.0.0.1:3260", IQN:          "iqn.2016-12.server:storage.target01", FSType:       "ext4", Lun:          0, }, }, }, }, } volState2 := v1.PodSpec{ Volumes: []v1.Volume{ { VolumeSource: v1.VolumeSource{ ISCSI: &v1.ISCSIVolumeSource{ TargetPortal: "127.0.0.1:3260", IQN:          "iqn.2017-12.server:storage.target01", FSType:       "ext4", Lun:          0, }, }, }, }, } tests := []struct { pod      *v1.Pod nodeInfo *schedulernodeinfo.NodeInfo isOk     bool name     string }{ {&v1.Pod{}, schedulernodeinfo.NewNodeInfo(), true, "nothing"}, {&v1.Pod{}, schedulernodeinfo.NewNodeInfo(&v1.Pod{Spec: volState}), true, "one state"}, {&v1.Pod{Spec: volState}, schedulernodeinfo.NewNodeInfo(&v1.Pod{Spec: volState}), false, "same state"}, {&v1.Pod{Spec: volState2}, schedulernodeinfo.NewNodeInfo(&v1.Pod{Spec: volState}), true, "different state"}, } expectedFailureReasons := []PredicateFailureReason{ErrDiskConflict} for _, test := range tests { t.Run(test.name, func(t *testing.T) { ok, reasons, err := NoDiskConflict(test.pod, nil, test.nodeInfo) if err != nil { t.Errorf("unexpected error: %v", err) } if !ok && !reflect.DeepEqual(reasons, expectedFailureReasons) { t.Errorf("unexpected failure reasons: %v, want: %v", reasons, expectedFailureReasons) } if test.isOk && !ok { t.Errorf("expected ok, got none.  %v %s", test.pod, test.nodeInfo) } if !test.isOk && ok { t.Errorf("expected no ok, got one.  %v %s", test.pod, test.nodeInfo) } }) } } </del> // TODO: Add test case for RequiredDuringSchedulingRequiredDuringExecution after it's implemented. func TestPodFitsSelector(t *testing.T) { tests := []struct {

/sig scheduling /priority important-soon See as an example. Part of /assign
GitHub didn't allow me to assign the following users: angao. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /sig scheduling /priority important-soon Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
thx, I will do it soon.


Question: if err := f.WaitForPodRunning(pod.Name); err != nil { framework.Failf("Pod did not start running: %v", err) } <ins> defer func() { logs, err := framework.GetPodLogs(f.Client, f.Namespace.Name, pod.Name, "portforwardtester") if err != nil { framework.Logf("Error getting pod log: %v", err) } else { framework.Logf("Pod log:n%s", logs) } }() </ins> By("Running 'kubectl port-forward'") cmd := runPortForward(f.Namespace.Name, pod.Name, 80)

https://k8s- Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
Two hunks (2x 10 "x"s) are not sent back. extends the logging. At least it will tell us whether the sender gets an error or not.
https://k8s- Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
https://k8s- Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
No more insights here. Pod log doesn't show anything special.
https://k8s- Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
https://k8s- Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite} Happened on a presubmit run in .
https://k8s- Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
https://k8s- Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite} Happened on a presubmit run in .
https://k8s- Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite} Happened on a presubmit run in .
I analysed this flake instance: https://k8s- It looks like the pfpod container terminates, the kubelet realizes that and wants to remove the network interface from the net ns using CNI via kubenet.

This fails two times (somehow TeardownNetwork is called multiple times, succeeds twice, but fails after that; compare about multiple TeardownNetwork calls). These failures interrupt the kubelet's tries to kill the pod. It succeeds 30 seconds later during reconciliation. But at that point the e2e test timed out already waiting for the pod to terminate.
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
This is the real problem, not another flake: This happens roughly once per week, pretty stable.
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
Builds: kubernetes-e2e-gke-staging-parallel Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
https://k8s- Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite} Happened on a presubmit run in .
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite} Happened on a presubmit run in .
[FLAKE-PING] This flaky-test issue would love to have more attention.

Builds: kubernetes-e2e-gce-etcd3 kubernetes-e2e-gci-gce-kubenet kubernetes-e2e-gci-gce-release-1.4 kubernetes-e2e-gci-gke-multizone kubernetes-e2e-gci-gke-prod-smoke kubernetes-e2e-gci-gke-staging-parallel kubernetes-e2e-gci-gke kubernetes-e2e-gke-staging-parallel kubernetes-e2e-gke Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
Builds: kubernetes-e2e-gci-gke-staging-parallel kubernetes-e2e-gke-staging-parallel Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
Builds: kubernetes-e2e-aws kubernetes-e2e-gci-gke-prod-smoke kubernetes-e2e-gke-staging-parallel kubernetes-e2e-gke Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite} Happened on a presubmit run in .
https://k8s- Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite} Happened on a presubmit run in .
[FLAKE-PING] This flaky-test issue would love to have more attention.

https://k8s- Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite} Happened on a presubmit run in .
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
Builds: kubernetes-e2e-gci-gke-release-1.4 kubernetes-e2e-gci-gke-staging-parallel kubernetes-e2e-gke Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite} Happened on a presubmit run in .
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
any updates on this issue
No, no further insights. Feel free to invest some time here with some new ideas.

[FLAKE-PING] This flaky-test issue would love to have more attention.
Builds: kubernetes-e2e-gce-multizone kubernetes-e2e-gci-gce-kubenet kubernetes-e2e-gci-gce-proto kubernetes-e2e-gci-gke-staging-parallel Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
Builds: kubernetes-e2e-gci-gce-proto kubernetes-e2e-gci-gke-release-1.4 kubernetes-e2e-gci-gke-staging-parallel kubernetes-e2e-gke-staging-parallel Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
Builds: kubernetes-e2e-aws kubernetes-e2e-gci-gce-kubenet Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.

This needs to be triaged as a release-blocker or not for 1.5
https://k8s- Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
Builds: ci-kubernetes-e2e-aws ci-kubernetes-e2e-gci-gce-kubenet Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
or Can you help us triage this as blocking or non-blocking for 1.5. It was originally marked as non-blocking for the 1.4 release.
The initial triage done by sttts@ is still valid even though we changed something in TeardownNetwork logic in 1.5 release. I don't think this should be release blocker with the same reason provided by sttts@ for 1.4 release. But instead of asking timestclair@ to take further look, I am assigning this to cc/
I marked this as release-blocker just for triage here. Thanks!
Thanks can you take a look and retriage if needed.
We have two flakes here: -- that's what and is about, i.e. Teardown logic. happened 3 times in last 8 days, much less the days before. IMO this is not a blocker. -- this seems to happen multiple times a day when a lot of e2e tests are running. This is the actual network flake. I will try to look into one of those to understand whether there is anything non-network related we can fix.
Here is a debug PR running the flaking test 100 times: None failed until now. Moreover, 200 runs locally do not fail.
Another observation: the number of returned is always a multiple of 10. The test container sleep for 100ms after each 10 . So, this might be a hint that there is some kind of race closing the connection.

Builds: ci-kubernetes-e2e-aws ci-kubernetes-e2e-gci-gke-multizone ci-kubernetes-e2e-gci-gke-prod-parallel ci-kubernetes-e2e-gci-gke-staging-parallel ci-kubernetes-e2e-gci-gke Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
Builds: ci-kubernetes-e2e-aws ci-kubernetes-e2e-gci-gce-release-1.4 ci-kubernetes-e2e-gke-staging-parallel Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
Builds: ci-kubernetes-e2e-aws ci-kubernetes-e2e-gce-release-1.5 ci-kubernetes-e2e-gci-gce-proto ci-kubernetes-e2e-gci-gce Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
Builds: ci-kubernetes-e2e-gce-etcd3 ci-kubernetes-e2e-gci-gce-etcd3 ci-kubernetes-e2e-gci-gce-kubenet Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
https://k8s- Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
Looks like this got a lot worse on 12/3 - https://k8s- Any chance this is related to
yes, it is most probably related to . In fact, was meant to make connection closing much less aggressive on the server side. It looks like our half-open connection support using socat is far from stable. I have in flight which is blocked by broken socat. Maybe that will help eventually to fix this flake.

Builds: ci-kubernetes-e2e-gce-etcd3 ci-kubernetes-e2e-gci-gce ci-kubernetes-soak-gke-test Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
Bumping to P0 - this is happening pretty much all the time now. yes, it is most probably related to . - can we revert that PR then? Or have a fix really soon, this is the biggest trigger of failures now.
we marked both tests as last night: This shouldn't happen anymore after rebasing PRs.
Here is a first attempt for a real fix:
https://k8s- Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
We will re-enable this test via This is also supposed to include a fix. Revert if the issue reappears.
https://k8s- Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
Is it appropriate to move this to the next milestone? (and remove the non-release-blocker tag as well)
Builds: ci-kubernetes-e2e-gci-gce-release-1.4 ci-kubernetes-e2e-gci-gce-release-1.5 Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
Builds: ci-kubernetes-e2e-gci-gke-release-1.4 ci-kubernetes-e2e-gci-gke-staging-parallel Failed: [] Port forwarding [] With a server that expects a client request should support a client that connects, sends data, and disconnects [Conformance] {Kubernetes e2e suite}
Closing this issue as the corresponding test has been removed from cc:


Question: <ins> // +build !windows /* Copyright 2021 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package app import ( "fmt" "os" ) func checkPermissions() error { if uid := os.Getuid(); uid != 0 { return fmt.Errorf("kubelet needs to run as uid `0`. It is being run as %d", uid) } // TODO: Check if kubelet is running in the `initial` user namespace. // http://man7.org/linux/man-pages/man7/user_namespaces.7.html return nil } </ins>

What happened: follow on to Need to comment and avoid checking UID on windows. requires some research into what we should be checking so that we can add appropriate TODO. What you expected to happen: Lets also check this at the call site, Anything else we need to know?: Environment: Kubernetes version (use ): master OS (e.g: ): windows
/assign
/assign
/good-first-issue
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig windows
Hi are you working on this issue already? If not I'd like to tackle this issue :)
Hi yeah I am already working on this :)
Started branch for this here
thanks
Hey guys is there anything I can do here?
If you want to test perks patch that would be great ?
That sounds great Where do I get started?
Hey ! is there anything I can help with? and how?
I dont think there is anything left to do anything more that you think we need to do to get this merged?
Tested this with containerd as well, I think its good to test
/triage accepted


Question: libcontainerStats := &libcontainer.Stats{ CgroupStats: cgroupStats, } <del> stats := newContainerStats(libcontainerStats) </del> <ins> withPerCPU := !ignoreMetrics.Has(container.PerCpuUsageMetrics) stats := newContainerStats(libcontainerStats, withPerCPU) </ins> // If we know the pid then get network stats from /proc/<pid>/net/dev if pid == 0 {

/kind feature This is the once-per-release cut of a new cAdvisor release, which corresponds to the next kubernetes release. cAdvisor v0.29 corresponds to kubernetes v1.10. /sig node /priority important-soon
self-reminder to add per-cpu stats to ignore metrics when vendoring. reminder to add a release notice that one of the container labels fields is being removed.
/kind bug turns out most of the changes are bug-fixes
please mark this issue as approved-for-milestone, since it certainly seems to be, and it's now code slush. Thanks!


Question: res = append(res, obj) } <ins> // Sort res to get deterministic order. sort.Slice(res, func(i, j int) bool { acc1, _ := meta.Accessor(res[i]) acc2, _ := meta.Accessor(res[j]) if acc1.GetNamespace() != acc2.GetNamespace() { return acc1.GetNamespace() < acc2.GetNamespace() } return acc1.GetName() < acc2.GetName() }) </ins> return res, nil }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: I used to fake a clientset to test my code. I found that it became very slow to init the clientset with a big number of objects when I was trying to simulate a large scale cluster, and the time seemed exponential increasing. For example: Initializing a clientset with: 5000 objects took 300ms objects took 2.58s objects took 11.82s objects took 54.64s. It's because the fake clientset used a slice to store each kind of objects, so it's quite slow to init the clientset with massive objects because it checked existence of an object by traversing all objects and comparing with them before adding one, which leads to O(n^2) time complexity. What you expected to happen: The fake clientset is very useful for development, it should have a reasonable time complexity to allow developers to use it for large scale scenario. How to reproduce it (as minimally and precisely as possible): Run this code: You would see the time spent on each operation, like: Anything else we need to know?: Environment: Kubernetes version (use ): 1.15.0-1.18.0 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I had PR that uses map to store objects, which can reduce the init time to a reasonable value (tens of ms for objects). /assign


Question: return allErrs } <del> func validateAudiences(audiences []string, audienceMatchPolicy api.AudienceMatchPolicyType, fldPath, audienceMatchPolicyFldPath *field.Path) field.ErrorList { </del> <ins> func validateAudiences(audiences []string, audienceMatchPolicy api.AudienceMatchPolicyType, fldPath, audienceMatchPolicyFldPath *field.Path, structuredAuthnFeatureEnabled bool) field.ErrorList { </ins> var allErrs field.ErrorList if len(audiences) == 0 {

For example, should not let you set discovery URL when the feature gate is not enabled. I believe this only impacts library users, as direct usage in KAS requires the file based config, which is feature gated. /assign enj aramase /milestone v1.32


Question: sendPod          *v1.Pod algo             algorithm.ScheduleAlgorithm expectErrorPod   *v1.Pod <ins> expectForgetPod  *v1.Pod </ins> expectAssumedPod *v1.Pod expectError      error expectBind       *v1.Binding

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Pod got stuck in Pending state forever when failed to bind to host. What you expected to happen: For it to be rescheduled on another host How to reproduce it (as minimally and precisely as possible): Not always reproducible. But simply created an RC replicas=2 on a 3 host setup, and one of the Pods got stuck in pending state for a while. Following error messages were found in the scheduler logs: Describe pod output: RC yml: Anything else we need to know?: Environment: Kubernetes version (use ): 1.7.1 Cloud provider or hardware configuration**: OS (e.g. from /etc/os-release): RHEL7, Docker native 1.12.6 Kernel (e.g. ): Install tools: Others:
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig scheduling
Can you provide the details steps for this issue? And the output of ?
The log just showed the node was notReady or unreachable. Seems the kubelet node is not quite stable that the connections with the master always get lost.
Another pod of the same RC (replicas=2) was started successfully on one of the nodes. I'd expect the failed one to at least be attempted to start again, may be on the node where its peer is running (as there are no anti affinity rules defined on it). But it was stuck in pending state forever (and there was no other reference for the failed pod besides the original log I've attached). I was able to create new RCs of the same kind on the same set of nodes successfully after the Pod scheduling failure, so it looks like some nodes were available for allocation. Perhaps there are some other logs I can look at /provide? no definite steps.

We just extensive validation tests for kubernetes were a bunch of RCs/Deployments/Services/Ingress controllers get created. And starting k8s 1.7.x started observing this failure. Once we see it again, I'll fetch the node status and update the bug.
From the log, it seemed that failed to bind your pod to the node and also failed to forget the pod, which led to unable to reschedule the pod to other nodes. Would you please append when you start and reproduce this issue. Also the kubelet log of that erroneous node.
^^ could you start the schedule with the above param next time for validation test run?
/cc
I ran our tests again, and hit this issue with several pods: There were a few more. Since none of them is assigned to a node. I included logs for all kubelet's. I also included scheduler log with : :
Seeing this as well. hypothesis: broke the error handling when fails. Before that commit, when failed it would log an error and pass the pod to the error handler (), which is in charge of retrying scheduling the pod. After that commit, when fails it skips the error handling and scheduling is never retried. I'm doing some experiments with patching that and I think there's more to this bug than just that (for example why is failing in the first place? is the scheduler cache corrupted? why?), but that's what I've got so far.
/cc
Can you post the description of per namespace?() Are you enabled ? And what's the config of component?
there are no specific parameters passed on the scheduler start besides address and cloudconfig
I posted a patch that i believe fixes this issue at -- for us it's resolved the issue so far, but I'm not very reliably able to reproduce so it's a bit hard to check. could you apply and see if your validation tests pass?
thanks!! We will be able to validate it either today or early next week
Are you able to validate the fix provided by through ? Just saw this issue, and I think it is a serious regression in 1.7 release and we should patch it. cc/ the patch manager for 1.7 patch release.
our QA have been busy with the current release, and didn't have a chance to validate the fix yet. Hopefully this week.

Patching sounds like a great idea. It is a regression indeed as we haven't hit this issue in the previous versions of k8s with the same set of validation tests.
Thanks everyone who helped investigate this!
just ran validation test against the patched branch - no pods stuck in Pending any more, so the fix did the job! thank you
I have this problem with k8s v1.7.0 . How can I fix it, upgrading to v1.7.X? Thank you!
I am seeing this in 1.9. What steps I need to take to solve this problem?
I am seeing this periodically on Azure VMs
any chance you could open a new issue linked to this one with the requisite troubleshooting info (versions, etc.) and label it with ` Thanks!


Question: f.NamespacePodSecurityEnforceLevel = admissionapi.LevelPrivileged ginkgo.Context("Downward API tests for hugepages", func() { <del> ginkgo.BeforeEach(func() { e2eskipper.SkipUnlessFeatureGateEnabled(kubefeatures.DownwardAPIHugePages) }) </del> ginkgo.It("should provide container's limits.hugepages-<pagesize> and requests.hugepages-<pagesize> as env vars", func() { podName := "downward-api-" + string(uuid.NewUUID()) env := []v1.EnvVar{

/kind failing-test <!-- If this is a flake, please add: /kind flake --/sig node
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Same failure in
I think this is related to the change @ could you please help take a look?
We already discussed this in The change from correctly points out that the test isn't working as expected. It needs to be fixed. /unassign
/assign
After , the tests are still failing. Could you help check it?
The skip function is used in another case as well. We may remove that like .
will fix this.
/close the test grid is green now.
Closing this issue. <detailsIn response to : /close the test grid is green now. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: if latency := tracker.ValidatingWebhookTracker.GetLatency(); latency != 0 { annotations[validatingWebhookLatencyKey] = latency.String() } <ins> if latency := tracker.DecodeTracker.GetLatency(); latency != 0 { annotations[decodeLatencyKey] = latency.String() } </ins> return annotations }

It is possible to issue a list request which exceeds the duration set by the request-timeout apiserver flag. The request-timeout flag does not appear to be enforced when the apiserver is decoding the response from etcd. Specifically, listing a large number of CRDs from etcd can result in high request latency from using json encoding compared to protobuf encoding. While overall etcd latency is low, the overall listing from storage latency may be over the 1 minute timeout, particularly if the apiserver needs to fetch multiple pages from etcd. We expected the list requests to be subject to the duration specified by the request-timeout flag when this latency is coming from the listing from storage phase of the request. Create 4000 CRs and list all the objects from etcd in 2 pages of 2000 objects. If the time to list 1 page + decode takes around 50s, listing 2 pages will result in the listing from storage phase of the request taking around 1 minute and 40s before the request fails and the client is returned a 500 response code. Logs for apiserver Trace message: Audit log entry: Related issue for supporting protobuf for CRDs: <details</details<detailsAWS </details<details</details<details</details<details</details<details</details
/sig api-machinery CC:
Interesting.. - do you know where exactly we stuck? In the call to storage: we pass the context that has timeout set, so that shouldn't hang. OTOH, for the processing part: (a) I don't know if transformer honors the context - it probably depends on the implementation (b) but the decoding (appendListitem) certainly isn't honoring the context So (b) might definitely be a problem, even independently from from the transformer So I guess what I'm saying is that if listing from etcd finishes fast but the decoding is what takes most of time, then what you described can certainly be a problem. But is that the thing you're observing?
Yes, that is our understanding. Specifically, that the decoding which occurs in appendListItem doesn't respect the context: As a separate follow-up, it would be useful to include decoding time, or at least the overall listing from storage time, as a separate latency annotation, given the audit log entry is only capturing this latency as part of
+1.

We need more granular latency tracker especially on the latency spent on decoding. Right now, it only captures etcd round trip and transformer latency.
I would be supportive for both and would be happy to review it if someone opens up a PR for it.
/triage accepted
/assign
/reopen I think we need to reopen it to document how we resolved the unit test failures brought on by adding it
Reopened this issue. <detailsIn response to : /reopen I think we need to reopen it to document how we resolved the unit test failures brought on by adding it Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: } func tooManyRequests(req *http.Request, w http.ResponseWriter) { <del> // "Too Many Requests" response is returned before logger is setup for the request. // So we need to explicitly log it here. defer httplog.NewLogged(req, &w).Log() </del> // Return a 429 status indicating "Too Many Requests" w.Header().Set("Retry-After", retryAfter) http.Error(w, "Too many requests, please try again later.", errors.StatusTooManyRequests)

Example call stack from : We should fix this and add an integration test that sets the limit low and exercises apiserver when it is getting too many requests.
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <bre.g., for API Machinery<br(2) specifying the label manually: <bre.g., for sig/scalability<br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
Should be fixed by the above PR. Closing it.


Question: if err := cc.syncFunc(cKey.(string)); err != nil { cc.queue.AddRateLimited(cKey) <del> utilruntime.HandleError(fmt.Errorf("Sync %v failed with : %v", cKey, err)) </del> <ins> if _, ignorable := err.(ignorableError); !ignorable { utilruntime.HandleError(fmt.Errorf("Sync %v failed with : %v", cKey, err)) } else { glog.V(4).Infof("Sync %v failed with : %v", cKey, err) } </ins> return true }

Each node waiting for bootstrapping creates 1 line per 2-3s in the logs when the requesting client doesn't have access. It should not be logging at v(0), these are all v(4) level. We should be reporting these appropriately. I don't want a 5k node cluster showing this.
I'd like to fix it


Question: } name := env[:pos] value := env[pos+1:] <del> if len(name) == 0 || len(validation.IsCIdentifier(name)) != 0 || len(value) == 0 { </del> <ins> if len(name) == 0 || len(validation.IsCIdentifier(name)) != 0 { </ins> return nil, fmt.Errorf("invalid env: %v", env) } envVar := v1.EnvVar{Name: name, Value: value}

It should be possible to pass empty environment variables to pods run by env (to clear an env var set inside the image, for example). is valid syntax for "set env var FOO to empty string".


Question: } handler.ValidateRequest(t, "/api/"+testapi.Version()+"/minions", "GET", nil) <del> if e, a := len(item.minions), ce.Len(); e != a { t.Errorf("Expected %v, got %v", e, a) </del> <ins> if a := ce.Len(); item.expectedCount != a { t.Errorf("Expected %v, got %v", item.expectedCount, a) </ins> } } }

With latest master it can happen that pods get scheduled on minions that are not up and get stuck in state. Also the command now also lists the down minions.
Master now returns all minions, with minion status set. Scheduler hasn't been updated yet. I'll put a fix for this. kubectl should return only running minions, and take a '-a' option to list all minions now. This also needs to be fixed.
, Deyuan Deng wrote:
Yeah, the main concern is that minion status is not as clear as Pod, e.g. running minion may not be eligible to schedule a pod. I'm still trying to figure out the best way to convey the message to client.
Re-open as now list all minions, including down ones. We should probabaly first filter out unhealthy ones in kubectl, before we can show any status.
What do you mean by "filter out"? We should be caching minion status in the apiserver. As proposed, we should show all minions that exist, including unresponsive ones.
I mean do not show down minions for now for consistency (users do not expect down minion to be listed). I agree with what proposed, but I'm trying to figure out how / what we convey the status message to user. NodePhase + NodeConditions are essential for k8s components, but are we going to expose all such data to user?
Why wouldn't users need to see down minions? I would think as an admin i would want to see phase + conditions
nvm My original concern was that phase and conditions are meant to be used by k8s, so we may need some translation to make it easier to understand for admin. Also, they are not fully populated, so I thought we may delay showing status for now. I certainly have no problem showing down minions and adding status. I've created a PR


Question: args        []string expectFatal bool expectTaint bool <ins> selector    bool </ins> }{ // success cases {

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): No What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): taint, selector Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-03-28T16:36:33Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-04-03T20:33:27Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"} Environment: Cloud provider or hardware configuration: GKE OS (e.g. from /etc/os-release): MacOS client, linux server Kernel (e.g. ): Install tools:gcloud Others: What happened: I tried to kubectl taint some nodes using a label selector: kubectl taint nodes -l dedicated=foo dedicated=foo:PreferNoSchedule and got the error: error: at least one resource name must be specified since 'all' parameter is not set What you expected to happen: Those nodes would be tainted with dedicated=foo:PreferNoSchedule. Trying to use --all along with the selector is also not supported.

How to reproduce it (as minimally and precisely as possible): kubectl taint nodes -l <any label selectordedicated=foo:PreferNoSchedule Anything else we need to know:
can you post the output of kubectl taint nodes -l myLabel=X dedicated=foo:PreferNoSchedule (((And first of course Make sure that myLabel is on the cluster ....)))
Would you like me to explicitly use the label "myLabel"? Otherwise that's the command and output I posted in "What happened" above. (I had set a label dedicated=foo ahead of time on two nodes.)
Ah ok so the taint k/v is the exact same as the label k/v, that makes the example tricky to parse.... but I agree .... this should work .. I can try to reproduce this....
Yup, the same k/v. (I wanted both labels and taints to determine this set of nodes, and it seemed like it would be confusing to have different k/v's for each. I kinda thought this would be the common case...)
Ah , yeah i see it to ... Even though clearly, the selector exists.
Is the taints and tolerations expert on our team. ravi can you look at ? this logic might be shadowing the label selector, such that its impossible to use labels to select the nodes you're trying to taint.
cc. m i definitely think this is a bug ... ? Looks like the taints maybe are shadowing the label selectors above in
/assign
GitHub didn't allow me to assign the following users: ravisantoshgudimetla. Note that only can be assigned.. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. I understand the commands that are listed . </details
/assign
It seems the line is the culprit. Here we are checking for resources if they are 2, the resources allowed are node,nodes,no so if it needs it has to be 1 and this check is already available So, I guess its safe to remove that. I tested that functionality after removing this and it fixed the issue. I will submit a PR for this unless someone has objections.


Question: volumeManager kubeletvolume.VolumeManager // Cloud provider interface. <del> cloud cloudprovider.Interface </del> <ins> cloud                   cloudprovider.Interface autoDetectCloudProvider bool </ins> // Reference to this node. nodeRef *api.ObjectReference

It appears that the kubelet will detect and use the cloud provider as detected by cadvisor thus overriding the cloud provider passed in at runtime: This will break our private cloud configuration where we do not use our cloud provider's load balancing and instead use our own load balancing tier. In my opinion, the cloud provider specified at kubelet runtime should be respected for the node. Am I reading this correctly? Any thoughts on how to handle this properly? Thank you!
If you specify any cloud provider through flags, it will be used by kubelet (see ). It's only when you don't specify any cloud provider that kubelet will detect the cloud provider from the machine info, as described by the comment.
that's exactly my situation, I want to run in our private cloud environment without any of the cloud integration. If I specify cloud-provider="" at runtime it will get overridden.
Ah I see. I guess specifying an empty string is the same as not specifying any cloud provider. This auto-detection feature was introduced by /cc the author and reviewer to see if this is an intended behavior:
Legitimate concern. I'm posting a fix. Do you consider this to be a regression or are you just getting started with k8s.
I am sorry, This was not the original intention. We can keep the auto detection with a flag, or make a flag to ask kubelet not to do it. Unless there is a way to see if the variable was assigned to that I missed. I can post a fix tomorrow if it won't exit by then.
Thank you. We're running on 1.2.4 and won't be running on 1.3 for a few months yet anyways and was just testing to look for future issues when I found this.
I assigned this issue to you since , and I can't assign this to anyway :-)
Thanks for testing our 1.3 release candidate, and reporting this regression to us. We are including the fix to 1.3.


Question: "//staging/src/k8s.io/apimachinery/pkg/util/uuid:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/util/wait:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/watch:go_default_library", <ins> "//staging/src/k8s.io/client-go/dynamic:go_default_library", </ins> "//staging/src/k8s.io/client-go/kubernetes:go_default_library", "//staging/src/k8s.io/client-go/tools/cache:go_default_library", "//staging/src/k8s.io/client-go/tools/watch:go_default_library",

This Issue replace Watch events may be missed and the test may still pass. We need to ensure that all expected watch events are seen. Watch events are not collected and checked at the end of the test. In order to solve the problem, we must collect all watch events as they come in and perform a check at the end of the test to ensure that the order is correct and they contain no errors.
/sig testing /area conformance /assign /cc
This Issue replace
AI: Must allow for other events to occur through the expected watch events stream expected watch events: must allow the following to be valid:
[x] Set up a watch for resource through a static label to the resource with DynamicClient [x] Manage the watch if it dies [x] Feed the watch to the function [x] Read all the watch events from the at the end of the test, ensuring that the events occur question: What does doing this mean to tests which check data? (e.g: - waits for values, sourced from watch) [x] Retry the test, upon failure [x] Manage the resources on retry (delete)
More or less, yes. If we need to read results mid-test, I think that will be OK, rather than waiting for the end. The main point is to wrap the watch such that it gets restarted if it fails. That wrapper will need to sit between the raw watch channel and the test: raw watch -channel -wrapper (restarts watch if closes/fails prematurely) -channel consumed in test I'm a little surprised there isn't already a sort of "buffered watch" object that does this already ? You'll also need a stop channel for the go routine started by the wrapper, so that at the end of the test you can stop that go routine and watch.
Also, if we have the "protected watch" wrapper, not sure we need to retry the test on failure. AFAIU, the only reason we wouldn't see the watch events we expected would be because of etcd compaction, which would not be expected to happen in e2e clusters, and even if it did it would not happen to event histories that are brand new (i.e., it only compacts old history). Again would like to confirm.
/reopen While we did push through this first iteration, this ticket is for ongoing discussion of watch tooling and refactoring of the approach.
Reopened this issue.

<detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen While we did push through this first iteration, this ticket is for ongoing discussion of watch tooling and refactoring of the approach.
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

[x] Original approval issue: - Reverted due to flakes [x] Two weeks soak end date: 20 November 2020 This issue is created to allow edit by /wip /hold According to this APIsnoop query, there are still some remaining Pod endpoints which are untested. a Pod with a static label the Pod with a new Label and updated data the Pod to ensure it's patched the Pod's status the Pod's status Ready condition to False the Pod's status to check if Ready is False all Pods in all Namespaces find the Pod(1) ensure that the Pod is found and is patched Namespaced Pod(1) via a Collection with a LabelSelector the Pod to check that it's deleted Discover useragents: List endpoints hit by the test: Display endpoint coverage change: If a test with these calls gets merged, test coverage will go up by 4 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/reopen Test PR Merge Promotion PR to be created next
Reopened this issue. <detailsIn response to : /reopen Test PR Merge Promotion PR to be created next Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
reverted by due to flaky test
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
/reopen PR on test grid for 2 weeks before promotion to conformance
Reopened this issue. <detailsIn response to : /reopen PR on test grid for 2 weeks before promotion to conformance Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage accepted


Question: # Recreate instance echo "== Recreating instance ${instance}. ==" >&2 <del> recreate=$(gcloud compute instance-groups managed recreate-instances "${group}"  </del> <ins> local recreate_rc local recreate=$(gcloud compute instance-groups managed recreate-instances "${group}"  </ins> --project="${PROJECT}"  --zone="${ZONE}"  --instances="${instance}" 2>&1) && recreate_rc=$? || recreate_rc=$?

This test is currently failing in sig-release master-upgrade suite, This could potentially become release blocking if not addressed sooner. can you please triage this failure? /kind bug /priority failing-test /priority important-soon /sig testing cc /assign
hmm, looks like cluster up etc works but some of the test case(s) are hanging and taking way too much time: ... I0417 18:59:21.467] (Will retry.)
cc ^^ assume that's somewhere inside the upgrade framework
/sig cluster-lifecycle
/assign /unassign Unlikely I will have the time or expertise to patch these, discussed out of band with and these appear to be something in the upgrade or cluster-up code, not kubetest etc..
/milestone v1.11
Debugged one of the jobs. The node upgrade is stalling out. There's logic in the node upgrade to wait for the new GCE vm to register with the k8s API. It does this by checking the node status's external ID and seeing if it matches the instance ID from the GCE vm. That logic is here: It looks like the kubelet is now registering the GCE VM's resource path and not its ID as the node's externalID. from https://k8s- Can I get input on this from ? Is this an intended change? If so, what is a good way to detect that the new node has registered with kubernetes?
cc ^^ :-)
ExternalID was apparently deprecated before kubernetes V1 API, but was never removed properly. PR was merged recently to remove the use of ExternalID Can we use the instance ID instead?
No. Instance ID is returning information like "k8s-jkns-e2e-gce-slow/us-central1-f/bootstrap-e2e-minion-group-0s1q". During a node upgrade, this ID is shared between the old and new VM. We need a way to distinguish them to make sure the new VM has come up, registered with the API server, and is healthy. Without a distinguishing ID, we could be reading the health status of the old node before we deleted it.
Ah, didn't read the issue carefully. Thanks for explaining that. Kubelet also reports machine ID/ system UUID. Are they unique in the cloud environment?

Can you point to where I can find that in the API and save me having to track it down? On Tue, Apr 17, 2018, 17:24 Yu-Ju Hong <:
or
Sorry, I should have been more specific. Where do those come from? Because I can't find that anywhere in
I think those are from . I don't think they are exposed in the gcloud. If we do use them, I think the upgrade script will have to simply check if the machine ID has changed, hmm...
Oh I see. OK I'll see if I can work with that.
It doesn't work. The machine ID is the same before and after the instance recreate. Even confirmed by looking at directly. seems to have the same value as machine ID.
Urghh...recreating an instance actually keeps the machineID. I don't have a good solution. One idea is to check if the bootID changed.
status/approved-for-milestone
/status approved-for-milestone
[MILESTONENOTIFIER] Milestone Issue: Up-to-date for process <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
/reopen I still see the Upgrade test failing in the jobs below.
I still see UpgradeTest and Timeout failing in the jobs below. Are you actively investigating these or should I re-triage?
These failures seem similar to my comment in I'm would appreciate help in looping in the node team on why that is happening.
/cc can either of you please help investigate the node side of this issue to unblock This is currently on of the few tests failing our sig-release upgrade blocking suite. If this is not fixed we will have to delay beta for 1.11 which is slated for 5/15.
/cc
cc
I updated the issue at , which might explain some of the failures here.
I see the tests passing now. Thanks and for all your help.


Question: // create pod which using hostport on the specified node according to the nodeSelector func createHostPortPodOnNode(f *framework.Framework, podName, ns, hostIP string, port int32, protocol v1.Protocol, nodeSelector map[string]string, expectScheduled bool) { <ins> hostIP = framework.TranslateIPv4ToIPv6(hostIP) </ins> createPausePod(f, pausePodConfig{ Name: podName, Ports: []v1.ContainerPort{

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: https://k8s-(IPv6),%20master%20(dev) Which test(s) are failing: [sig-scheduling] SchedulerPredicates [Serial] validates that there is no conflict between pods with same hostPort but different hostIP and protocol [Conformance] Since when has it been failing: Testgrid link: https://k8s-(IPv6),%20master%20(dev) Reason for failure: The tests checks for 0.0.0.0 that is IPv4 only Anything else we need to know: Probably I should check with SIG-Testing how can we add an ipv6 job and update the KEP accordingly EDIT Probably I should check with SIG-Testing how can we add an ipv6 job as part of the k/k presubmits, and update the KEP accordingly
/assign /cc
/sig scheduling

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Introduce a mechanism that allows specifying the IP protocol version used in the e2e testing. Why is this needed: This is a followup on , my understanding is that in order to discriminate between IPv4 or IPv6 we have to use the test labels and this adds complexity to the testing automation and doesn't allow to send this information to the context of the tests. Having a mechanism to specify the IP protocol will simplify the testing for IPv4 and IPv6 and will allow reusing most of the code.
/sig testing /cc /assign Does this make sense? If affirmative, what should be the best way to add this functionality? Checking the code I think that or can provide this, don't know if checking some of the cluster config parameters, ...
the testing-commons group (sig-testing) maintains the framework, but ideally the decision on how to proceed in terms of adding ipv6 detection should come from sig-network. possibly having this as agenda for a sig-network meeting is a good start, then discussing it in a testing-commons meeting. /sig network the linked issue is a big one, so i don't have all the context here. but IMO existing tests should just work for ipv6 without the requirement for a new "IPV6" tag in test names. dual-stack or protocol specific tests should have their descriptions contain the details properly (i.e. what is this test doing) we might want to add an utility function under: (there is a PR to move it BTW ) then such an utility might be useful only to tests that need to detect the protocol. one of the goals is to refactor the code in testcontext / framework, so adding this logic there is less preferred.
/area e2e-test-framework
I think that we can add an option to test_context to specify the cluster IP family the same way that exists one for the cloud provider this is simpler because when setting up the CI jobs you will know beforehand what IP family you want to test, it also will allow reusing all the e2e code for IPv6 and Dual Stack (cc: and by default will be IPv4, so it´s backward compatible.
Being able to override and specify exactly what you want the test to beleive about your cluster is nice, but ideally it should generally work without specifying more flags.

Surely we can detect this?
I was thinking during some time about this, detecting the IP family is very simple having IPv4 or IPv6 only clusters, but with the introduction of dual stack I´m afraid about the supportability and reliability of the autodetecting mechanisms in the long term. If you see the dual stack PRs there are still a lot of unknowns about the cluster behavior. My point is, let´s have something first stable that works and is simple, and once dual stack is implemented, we can replace the flag with a function in the framework . I guess that migrating from a flag variable to a function is straight forward, is just replacing by , what do you think?
in is as part of the context. do you mean adding as a function which will auto-detecting the ip family. less flags for the framework binary is preferred.
once we have this I will open a new issue to start cleaning up and , we no longer need to discriminate based on tags


Question: AllowInvalidTopologySpreadConstraintLabelSelector: false, AllowMutableNodeSelectorAndNodeAffinity:           utilfeature.DefaultFeatureGate.Enabled(features.PodSchedulingReadiness), AllowNamespacedSysctlsForHostNetAndHostIPC:        false, <ins> AllowNonLocalProjectedTokenPath:                   false, </ins> } if oldPodSpec != nil {

Originally posted by in
Hello, I'm new here. Can I try solving this? it seems like a good first issue
sure, context is the change is needed here: We have to tolerate existing pods / pod specs that fail this validation, so we should only do this additional validation on create or if the existing object passes this validation. To do this: an field to , initialize that field to false, and then conditionally override it to if is non-nil and has a projected token volume path that fails (will have to export the function to check this) , if is false, call the validation on the path:
Hi, I'd like to work on this if there is no one is proceeding on. /assign


Question: "time" "github.com/golang/glog" <ins> "golang.org/x/net/http2" </ins> "k8s.io/apimachinery/pkg/api/errors" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime"

While this issue happens without this complicated reproduction, here is a sure-fire way to get it. a long-running kubectl get. In this example, I have a cluster with about 2,000 configmaps each around 100KiB. Once you see these lines Press , then type , and then you'll see... What's happening is that the apiserver is hitting , and then it . As the code says, this should make it obvious to the client (kubectl) that it didn't get the entire body, because no 0-length EOF chunk gets sent. I can confirm from using tcpdump that no such chunk gets sent. However, kubectl doesn't notice this, and instead tries to decode a 0-length Response body, giving the cryptic message. What I expected to happen the very least, a less cryptic error message like "connected closed". , this gets detected by the code and we can internally retry, just as if we got, e.g.,
cc Possibly seen here, too: openshift/origin
/assign mengqiy
Created a PR to detect it. Further improvement may be retrying when detect connection closure
closing this since you filed a PR for it


Question: <ins> // +build integration,!no-etcd /* Copyright 2015 The Kubernetes Authors All rights reserved. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package integration import ( "fmt" "net/http" "net/http/httptest" "testing" "k8s.io/kubernetes/pkg/api" "k8s.io/kubernetes/pkg/api/testapi" "k8s.io/kubernetes/pkg/client/restclient" client "k8s.io/kubernetes/pkg/client/unversioned" "k8s.io/kubernetes/pkg/master" "k8s.io/kubernetes/test/integration/framework" ) func TestPodUpdateActiveDeadlineSeconds(t *testing.T) { var m *master.Master s := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, req *http.Request) { m.Handler.ServeHTTP(w, req) })) defer s.Close() ns := "pod-activedeadline-update" masterConfig := framework.NewIntegrationTestMasterConfig() m, err := master.New(masterConfig) if err != nil { t.Fatalf("Error in bringing up the master: %v", err) } framework.DeleteAllEtcdKeys() client := client.NewOrDie(&restclient.Config{Host: s.URL, ContentConfig: restclient.ContentConfig{GroupVersion: testapi.Default.GroupVersion()}}) var ( iZero = int64(0) i30   = int64(30) i60   = int64(60) iNeg  = int64(-1) ) prototypePod := func() *api.Pod { return &api.Pod{ ObjectMeta: api.ObjectMeta{ Name: "xxx", }, Spec: api.PodSpec{ Containers: []api.Container{ { Name:  "fake-name", Image: "fakeimage", }, }, }, } } cases := []struct { name     string original *int64 update   *int64 valid    bool }{ { name:     "no change, nil", original: nil, update:   nil, valid:    true, }, { name:     "no change, set", original: &i30, update:   &i30, valid:    true, }, { name:     "change to positive from nil", original: nil, update:   &i60, valid:    true, }, { name:     "change to smaller positive", original: &i60, update:   &i30, valid:    true, }, { name:     "change to larger positive", original: &i30, update:   &i60, valid:    false, }, { name:     "change to negative from positive", original: &i30, update:   &iNeg, valid:    false, }, { name:     "change to negative from nil", original: nil, update:   &iNeg, valid:    false, }, // zero is not allowed, must be a positive integer { name:     "change to zero from positive", original: &i30, update:   &iZero, valid:    false, }, { name:     "change to nil from positive", original: &i30, update:   nil, valid:    false, }, } for i, tc := range cases { pod := prototypePod() pod.Spec.ActiveDeadlineSeconds = tc.original pod.ObjectMeta.Name = fmt.Sprintf("activedeadlineseconds-test-%v", i) if _, err := client.Pods(ns).Create(pod); err != nil { t.Errorf("Failed to create pod: %v", err) } pod.Spec.ActiveDeadlineSeconds = tc.update _, err := client.Pods(ns).Update(pod) if tc.valid && err != nil { t.Errorf("%v: failed to update pod: %v", tc.name, err) } else if !tc.valid && err == nil { t.Errorf("%v: unexpected allowed update to pod", tc.name) } deletePodOrErrorf(t, client, ns, pod.Name) } } func TestPodReadOnlyFilesystem(t *testing.T) { var m *master.Master s := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, req *http.Request) { m.Handler.ServeHTTP(w, req) })) defer s.Close() isReadOnly := true ns := "pod-readonly-root" masterConfig := framework.NewIntegrationTestMasterConfig() m, err := master.New(masterConfig) if err != nil { t.Fatalf("Error in bringing up the master: %v", err) } framework.DeleteAllEtcdKeys() client := client.NewOrDie(&restclient.Config{Host: s.URL, ContentConfig: restclient.ContentConfig{GroupVersion: testapi.Default.GroupVersion()}}) pod := &api.Pod{ ObjectMeta: api.ObjectMeta{ Name: "xxx", }, Spec: api.PodSpec{ Containers: []api.Container{ { Name:  "fake-name", Image: "fakeimage", SecurityContext: &api.SecurityContext{ ReadOnlyRootFilesystem: &isReadOnly, }, }, }, }, } if _, err := client.Pods(ns).Create(pod); err != nil { t.Errorf("Failed to create pod: %v", err) } deletePodOrErrorf(t, client, ns, pod.Name) } </ins>

hack/test- passes even with an explicit "t.Fatalf()" in the (see ). Appreciate any insight.
The script essentially runs . Check the build tags at the top of the file. It's not even compiled in.
The script is using the 'no-docker' tag, not 'no-etcd', so I expect the tests to be compiled.
run with those build tags and see if that file in in there, just to be sure.
doesn't have "_test" in the filename.. I'll send a fix.
Thanks the command is handy.


Question: { "SelectorRegexp": "k8s[.]io/kubernetes/test/", "AllowedPrefixes": [ <ins> "k8s.io/kubernetes/test/e2e/framework", </ins> "k8s.io/kubernetes/test/e2e/framework/auth", "k8s.io/kubernetes/test/e2e/framework/ginkgowrapper", <ins> "k8s.io/kubernetes/test/e2e/framework/kubectl", </ins> "k8s.io/kubernetes/test/e2e/framework/log", "k8s.io/kubernetes/test/e2e/framework/metrics", <ins> "k8s.io/kubernetes/test/e2e/framework/network", </ins> "k8s.io/kubernetes/test/e2e/framework/node", "k8s.io/kubernetes/test/e2e/framework/pod", <ins> "k8s.io/kubernetes/test/e2e/framework/rc", </ins> "k8s.io/kubernetes/test/e2e/framework/resource", <ins> "k8s.io/kubernetes/test/e2e/framework/service", </ins> "k8s.io/kubernetes/test/e2e/framework/ssh", "k8s.io/kubernetes/test/e2e/framework/testfiles", "k8s.io/kubernetes/test/e2e/manifest", "k8s.io/kubernetes/test/e2e/perftype", <ins> "k8s.io/kubernetes/test/e2e/storage/utils", "k8s.io/kubernetes/test/e2e/system", </ins> "k8s.io/kubernetes/test/utils", "k8s.io/kubernetes/test/utils/image" ],

What happened: There is to manage import ways under but that doesn't work at our upstream CI. We can enable it by But if doing that, we will face the following error with What you expected to happen: I hope the check is always running to block invalid import ways at upstream CI system. How to reproduce it (as minimally and precisely as possible): As line into arguments.InputDirs of
/sig testing /area e2e-test-framework


Question: // Create a listener for events. events := make([](*api.Event), 0) <del> _, controller := controllerFramework.NewInformer( </del> <ins> _, controller := controllerframework.NewInformer( </ins> &cache.ListWatch{ ListFunc: func() (runtime.Object, error) { return c.Events(ns).List(labels.Everything(), fields.Everything())

Forked from pointed out that all package names should be lower case, no under_scores or mixedCaps. The same rule applies to renaming imported packages. () generated a list of mixedCaps package names in our codebase: I will upload a PR later to rename kubeletTypes to kubetypes, and the rest is up for grabs.
sorry i have not saw this issue as i open a similar one(), do you mind i participate in this issue? i would happy to do some work for this. best regards.
of course, fixes are welcome. Thanks for picking up on this! We'll try to expedite the review and merge process of such PRs to avoid a lot of churns, but chances are that some rebases are needed :)
ok, i will send some fix sonner, and tag you for review. thanks again.
Not done:


Question: "github.com/spf13/cobra" "github.com/golang/glog" <ins> "k8s.io/kubernetes/pkg/api/errors" </ins> "k8s.io/kubernetes/pkg/kubectl" cmdutil "k8s.io/kubernetes/pkg/kubectl/cmd/util" "k8s.io/kubernetes/pkg/kubectl/resource" "k8s.io/kubernetes/pkg/runtime" <ins> "k8s.io/kubernetes/pkg/util/wait" </ins> ) // ReplaceOptions is the start of the data required to perform the operation.  As new fields are added, add them here instead of

When replacing an existing pod, the command fails as it tries to create a pod with a name that is still used by terminating pod. How does the option works? deletes a pod (e.g. mypod) creates a new pod with the same name (mypod) What happens? As the command tries to recreate the pod, it fails as the is in state. Thus it can not be created. The problem is the does not wait for a pod to be deleted. Reproducer: run mypod --image=yapei/hello-openshift --generator=run-pod/v1 get pod mypod -o yaml replace -f --force The last command ends with:
option introduce by
With set to positive value, what should be the default behaviour of the command? When set to , should the command hang for and then create the pod? Or is there a way to create a callback (inside a scheduler?) that creates the pod once the current one is deleted?
You could watch for the resource to be deleted and create after you see the deletion happen
have you ever tried to set , I think this will do a non graceful deletion. should make sure the old resource is deleted before it creates a new resource, if not, that's a bug.
Though the help says the is to be used only with , it returns the error above.
Then I think there are two issues we need to track: 1. option should be ignored or just returns error if users are not using option. 2. should always make sure the old resource has been deleted before create the new resource.


Question: return false, err } nOk := int32(0) <del> notReady := int32(0) </del> <ins> notReady = int32(0) </ins> badPods = []v1.Pod{} desiredPods = len(podList.Items) for _, pod := range podList.Items {

Is this a BUG REPORT or FEATURE REQUEST?: /kind cleanup What happened: This log seems to imply that we wait until less than some number of pods are un-ready, and yet the loop succeeds if there is no improvement. What you expected to happen: Either the loop should terminate earlier (if the condition is acceptable) or fail (if it's unacceptable).
/sig scalability since this probably most affects us
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close


Question: }, }, }, <ins> "spec.template.spec.restartPolicy: Unsupported value": { ObjectMeta: metav1.ObjectMeta{ Name:      "myjob", Namespace: metav1.NamespaceDefault, UID:       types.UID("1a2b3c"), }, Spec: batch.JobSpec{ Selector:       validManualSelector, ManualSelector: newBool(true), Template: api.PodTemplateSpec{ ObjectMeta: metav1.ObjectMeta{ Labels: validManualSelector.MatchLabels, }, Spec: api.PodSpec{ RestartPolicy: "Invalid", DNSPolicy:     api.DNSClusterFirst, Containers:    []api.Container{{Name: "abc", Image: "image", ImagePullPolicy: "IfNotPresent", TerminationMessagePolicy: api.TerminationMessageReadFile}}, }, }, }, }, </ins> } for _, setFeature := range []bool{true, false} {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Deployed an cronJob with an empty and it defaulted to which is an invalid option: Leaving the key out completely has the same result What you expected to happen: The empty key to default to something valid like or . Also document what this default is, my proposal would be to have as default since some tasks can't be reran easily. But it might be nice to force as default to enforce the stateless mindset even for crons :) How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: I guess not but I am open for questions :) Environment: Kubernetes version (use ): 1.19.2
My guess would be /sig api-machinery If it is better practice to wait for the triage instead of guessing the sig, let me know :+1:
/sig api-machinery
/assign
/remove-sig api-machinery /sig apps
/assign
/triage accepted


Question: // TODO: // - check for these errors in unit tests instead It("should fail due to non-existent path", func() { <del> ep := &eventPatterns{ reason:  "FailedMount", pattern: make([]string, 2)} ep.pattern = append(ep.pattern, "MountVolume.NewMounter initialization failed") </del> testVol := &localTestVolume{ node:            config.node0, hostDir:         "/non-existent/location/nowhere",

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: Based on discussion , events can be throttled and should not be used to validate failure conditions in e2e tests. We should remove the event checks from subpath and local pv tests and just wait for pod failure. <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
/assign
could you help look at this for local PV tests?
sure /assign
/area deflake
"Invalid AWS KMS key" also waits for an event

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: fsgroup tests are flaking 1-2 times a day: https://k8s- Some test logs: https://k8s- https://k8s-
could you take a look?
I'll look into this.
I found why the tests failed, see my analysis of logs: I created a PR () trying to fix the issue. I am still investigating kubelet and trying to reproduce the issue locally.
/assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
/lifecycle frozen


Question: stopCh := make(chan struct{}) defer close(stopCh) <del> go informerFactory.Start(stopCh) </del> <ins> informerFactory.Start(stopCh) informerFactory.WaitForCacheSync(stopCh) wait.PollUntil(10*time.Millisecond, func() (bool, error) { return controller.queue.Len() >= 1, nil }, stopCh) </ins> controller.processNextWorkItem() actions := client.Actions() <del> if len(actions) != 3 { t.Errorf("expected 3 actions") </del> <ins> if len(actions) != 1 { t.Errorf("expected 1 actions") </ins> } <del> if a := actions[0]; !a.Matches("list", "certificatesigningrequests") { t.Errorf("unexpected action: %#v", a) } if a := actions[1]; !a.Matches("watch", "certificatesigningrequests") { t.Errorf("unexpected action: %#v", a) } if a := actions[2]; !a.Matches("update", "certificatesigningrequests") || </del> <ins> if a := actions[0]; !a.Matches("update", "certificatesigningrequests") || </ins> a.GetSubresource() != "approval" { t.Errorf("unexpected action: %#v", a) }

Seen in happy to send a fix but didn't know if this might have been address with your current refactors. Output:
This looks new. I'd happily review a fix. Thanks.
Dupe of


Question: } id := f.queue[0] f.queue = f.queue[1:] <ins> depth := len(f.queue) </ins> if f.initialPopulationCount > 0 { f.initialPopulationCount-- }

I am using client-go/tools/cache package NewInformer, find the StreamWatcher and seeing extremely high memory allocations. Attached is a memory profile: From this we have 2.3 GB of memory allocated by the StreamWatcher. To get to this state I had a script to create pod and delete pod over and over again, and every 10 seconds I run it again. I left this running 30 minutes and saw this memory usage. watch code as follows, cache.NewInformer return controller and storewriter is chan queue, receiving event and distribute to queue
/sig api-machinery
Thanks for taking a look
/help
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc
i find that infomer handler be stuck will lead to streamWatcher memory high allocationsdon't hang informer handler and you will resolve that question.
/assign /triage accepted
can you check with a cpu profile if some handler is blocking?
cpu profile is normalas followscode callback handler func (w *Watcher) AddEvent(obj interface{}) { data := w.genSyncData(obj, action.SyncDataActionAdd) if data == nil { return } w.writer.Sync(data) } writer is a buffer chan with length and write chan timeout for one secondprobably is will be stuck when produce lots of eventsfinallyI store events to fifo queue when it produce lots of eventsand consume fifo queue event to writer chanthe issue is solved
So, the problem is that your handlers are slower processing the objects from the delta fifo queue, than the rate at which the events are obtained from the apiserver, this is because of this objects are processed by the handlers locking the deltaFifo queue, hence the streamwatcher accumulates the object in memory until the handlers process the object and Pop() release the lock. The code comments inform about this problem // Add() and Get(), won't be blocked for too long. the only solution I see is logging some errors warning about this (processing withouth locking doesn't seem an option, right?)

I've tried to use queue depth, but that is not an option, since the contention happens before events get into the queue and it nevers goes high. I've tried adding trace for slow handlers operations, however, those maybe harmless if the incoming rate is still lower than the draining rate, unless we decide that we want to trace process handlers that take more than X seconds Is adding another intermediate queue between the streamwatecher and the deltaFifo an option? that will free the streamwatchers ...
Yeah. I don't see it as a high priority, but that could be helpful. Following the standard hander to workqueue pattern, it isn't a practical problem. /priority backlog
i don't hang informer hander by fifo queue, i find that memory will come down when lots of create and delete podsbut i running code onlinethe streamwatcher still exist high memory usage as follows pprof Please take a look at it
i am also facing same issue any guidance for the watcher memory increase in high pod churn


Question: <ins> #!/usr/bin/env bash # Copyright 2021 The Kubernetes Authors. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # #     http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. set -o errexit set -o nounset set -o pipefail # This will canonicalize the path KUBE_ROOT=$(cd "$(dirname "${BASH_SOURCE[0]}")"/../.. && pwd -P) cd "${KUBE_ROOT}" # build ginkgo and e2e.test # NOTE: we do *not* use `make WHAT=...` because we do *not* want to be running # make generated_files when diffing things (see: hack/verify-conformance-yaml.sh) # other update/verify already handle the generated files hack/make-rules/build.sh vendor/github.com/onsi/ginkgo/ginkgo test/e2e/e2e.test # dump spec ./_output/bin/ginkgo --dryRun=true --focus='[Conformance]' ./_output/bin/e2e.test -- --spec-dump "${KUBE_ROOT}/_output/specsummaries.json" > /dev/null </ins>

The currently provided command for generating requires bazel, the command should be updated to one which does not require bazel Currently: Suggested: /sig testing /sig architecture /area conformance
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/assign /remove-lifecycle rotten
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle rotten
/remove-lifecycle rotten /help
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /remove-lifecycle rotten /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.

</details
We're going to need to update this to verify under as well instead of under the bazel "unit tests", ref: I'll work on this.
done in
This is wonderful to see. Thank you


Question: return fmt.Errorf("waiting for service to be down timed out") } <del> func CleanupServiceGCEResources(loadBalancerName string) { </del> <ins> func CleanupServiceResources(loadBalancerName, zone string) { if TestContext.Provider == "gce" || TestContext.Provider == "gke" { CleanupServiceGCEResources(loadBalancerName, zone) } // TODO: we need to add this function with other cloud providers, if there is a need. } func CleanupServiceGCEResources(loadBalancerName, zone string) { </ins> if pollErr := wait.Poll(5*time.Second, LoadBalancerCleanupTimeout, func() (bool, error) { <del> if err := CleanupGCEResources(loadBalancerName); err != nil { </del> <ins> if err := CleanupGCEResources(loadBalancerName, zone); err != nil { </ins> Logf("Still waiting for glbc to cleanup: %v", err) return false, nil }

In federation tests, if we observe the build logs. we see cleaning up LoadBalancer from cloud provider seems not working. It has 2 effects: delays the tests by waiting to delete but always returns error. so we waste like 5 minutes. maybe we are leaking the LoadBalancer in the project.
we can see these errors in every build. but just for reference here are some build logs https://k8s-
cc
All the errors in logs are . It doesn't necessarily indicate we are leaking resource, just now we must have clusterID initialized before calling EnsureLoadBalancerDeleted(). Would be better to not call this function directly in e2e test :)
Federation cleanup should use the same implementation as non-federated e2e test:
function looks like only specific to GCE. in future we would run the e2e tests in AWS too. it would be better to use generic functions if any.
using is working for us. here is the test logs https://k8s-
EnsureLoadBalancerResourcesDeleted() actually cleans up nothing, it only lists forwarding rules and probably logging :)


Question: func filterInvalidPods(pods []*v1.Pod, source string, recorder record.EventRecorder) (filtered []*v1.Pod) { names := sets.String{} for i, pod := range pods { <del> var errlist field.ErrorList // TODO: remove the conversion when validation is performed on versioned objects. internalPod := &api.Pod{} if err := k8s_api_v1.Convert_v1_Pod_To_api_Pod(pod, internalPod, nil); err != nil { glog.Warningf("Pod[%d] (%s) from %s failed to convert to v1, ignoring: %v", i+1, format.Pod(pod), source, err) recorder.Eventf(pod, v1.EventTypeWarning, "FailedConversion", "Error converting pod %s from %s, ignoring: %v", format.Pod(pod), source, err) </del> <ins> // Pods from each source are assumed to have passed validation individually. // This function only checks if there is any naming conflict. name := kubecontainer.GetPodFullName(pod) if names.Has(name) { glog.Warningf("Pod[%d] (%s) from %s failed validation due to duplicate pod name %q, ignoring", i+1, format.Pod(pod), source, pod.Name) recorder.Eventf(pod, v1.EventTypeWarning, events.FailedValidation, "Error validating pod %s from %s due to duplicate pod name %q, ignoring", format.Pod(pod), source, pod.Name) </ins> continue <del> } if errs := validation.ValidatePod(internalPod); len(errs) != 0 { errlist = append(errlist, errs...) // If validation fails, don't trust it any further - // even Name could be bad. </del> } else { <del> name := kubecontainer.GetPodFullName(pod) if names.Has(name) { // TODO: when validation becomes versioned, this gets a bit // more complicated. errlist = append(errlist, field.Duplicate(field.NewPath("metadata", "name"), pod.Name)) } else { names.Insert(name) } } if len(errlist) > 0 { err := errlist.ToAggregate() glog.Warningf("Pod[%d] (%s) from %s failed validation, ignoring: %v", i+1, format.Pod(pod), source, err) recorder.Eventf(pod, v1.EventTypeWarning, events.FailedValidation, "Error validating pod %s from %s, ignoring: %v", format.Pod(pod), source, err) continue </del> <ins> names.Insert(name) </ins> } filtered = append(filtered, pod) } return

To read a pod manifest file from disk (or from the http source), kubelet execute the following code to Decode a on-disk v1 pod manifest to an internal pod object (the defaults are applied in this process). Validate the internal pod. Convert the internal pod back to a v1 pod. The downstream pod cache (which gets v1 pods from different sources) performs similar operations: Convert the v1 pod to an internal pod. Validate the internal pod. One of two conversion/validation should be removed, and some of them should be simplified.
/cc for suggestions on defaulting and conversion.
If no one is working on this, I will take it.