Question: } } <del> // TODO(#111514): refactor by spliting into dedicated test functions func TestJobStrategy(t *testing.T) { ctx := genericapirequest.NewDefaultContext() if !Strategy.NamespaceScoped() { t.Errorf("Job must be namespace scoped") } if Strategy.AllowCreateOnUpdate() { t.Errorf("Job should not allow create on update") } validSelector := &metav1.LabelSelector{ MatchLabels: map[string]string{"a": "b"}, } validPodTemplateSpec := api.PodTemplateSpec{ ObjectMeta: metav1.ObjectMeta{ Labels: validSelector.MatchLabels, }, Spec: api.PodSpec{ RestartPolicy: api.RestartPolicyOnFailure, DNSPolicy:     api.DNSClusterFirst, Containers:    []api.Container{{Name: "abc", Image: "image", ImagePullPolicy: "IfNotPresent", TerminationMessagePolicy: api.TerminationMessageReadFile}}, }, } job := &batch.Job{ ObjectMeta: metav1.ObjectMeta{ Name:      "myjob", Namespace: metav1.NamespaceDefault, Annotations: map[string]string{ "foo": "bar", }, ResourceVersion: "0", }, Spec: batch.JobSpec{ Selector:       validSelector, Template:       validPodTemplateSpec, ManualSelector: pointer.BoolPtr(true), Completions:    pointer.Int32Ptr(2), // Set gated values. Suspend:                 pointer.BoolPtr(true), TTLSecondsAfterFinished: pointer.Int32Ptr(0), CompletionMode:          completionModePtr(batch.IndexedCompletion), }, Status: batch.JobStatus{ Active: 11, }, } Strategy.PrepareForCreate(ctx, job) if job.Status.Active != 0 { t.Errorf("Job does not allow setting status on create") } if job.Generation != 1 { t.Errorf("expected Generation=1, got %d", job.Generation) } errs := Strategy.Validate(ctx, job) if len(errs) != 0 { t.Errorf("Unexpected error validating %v", errs) } if job.Spec.CompletionMode == nil { t.Errorf("Job should allow setting .spec.completionMode") } wantAnnotations := map[string]string{ "foo":                        "bar", batchv1.JobTrackingFinalizer: "", } if diff := cmp.Diff(wantAnnotations, job.Annotations); diff != "" { t.Errorf("Job has annotations (-want,+got):n%s", diff) } parallelism := int32(10) // ensure we do not change generation for non-spec updates updatedLabelJob := job.DeepCopy() updatedLabelJob.Labels = map[string]string{"a": "true"} Strategy.PrepareForUpdate(ctx, updatedLabelJob, job) if updatedLabelJob.Generation != 1 { t.Errorf("expected Generation=1, got %d", updatedLabelJob.Generation) } errs = Strategy.ValidateUpdate(ctx, updatedLabelJob, job) if len(errs) != 0 { t.Errorf("Unexpected update validation error") } updatedJob := &batch.Job{ ObjectMeta: metav1.ObjectMeta{ Name:            "bar", ResourceVersion: "4", // remove one annotation and try to enforce the job tracking finalizer. Annotations: map[string]string{batchv1.JobTrackingFinalizer: ""}, }, Spec: batch.JobSpec{ Parallelism: &parallelism, Completions: pointer.Int32Ptr(2), // Update gated features. TTLSecondsAfterFinished: pointer.Int32Ptr(1), CompletionMode:          completionModePtr(batch.IndexedCompletion), // No change because field is immutable. }, Status: batch.JobStatus{ Active: 11, }, } // Ensure we do not change status job.Status.Active = 10 Strategy.PrepareForUpdate(ctx, updatedJob, job) if updatedJob.Status.Active != 10 { t.Errorf("PrepareForUpdate should have preserved prior version status") } if updatedJob.Generation != 2 { t.Errorf("expected Generation=2, got %d", updatedJob.Generation) } wantAnnotations = map[string]string{ batchv1.JobTrackingFinalizer: "", } if diff := cmp.Diff(wantAnnotations, updatedJob.Annotations); diff != "" { t.Errorf("Job has annotations (-want,+got):n%s", diff) } errs = Strategy.ValidateUpdate(ctx, updatedJob, job) if len(errs) == 0 { t.Errorf("Expected a validation error") } // Test updating suspend false->true and nil-> true when the feature gate is // disabled. We don't care about other combinations. job.Spec.Suspend, updatedJob.Spec.Suspend = pointer.BoolPtr(false), pointer.BoolPtr(true) Strategy.PrepareForUpdate(ctx, updatedJob, job) job.Spec.Suspend, updatedJob.Spec.Suspend = nil, pointer.BoolPtr(true) Strategy.PrepareForUpdate(ctx, updatedJob, job) </del> <ins> func TestJobStrategy_GarbageCollectionPolicy(t *testing.T) { </ins> // Make sure we correctly implement the interface. // Otherwise a typo could silently change the default. var gcds rest.GarbageCollectionDeleteStrategy = Strategy

The gets very long testing mulitple functions in such as , and . These checks should be performed in dedicated Test functions using table for different scenarios. Target functions and are being as part of The remaining checks should be migrated to these functions. Also, a test function should be to migrate the remaining tests.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig apps
/assign
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
Any progress on this? I'm working on and I noticed that has relatively low coverage so I was going to work on getting the coverage up.
I created that is somewhat related to this.
no progess. go ahead please
/assign


Question: } func (s *sharedIndexInformer) HasSynced() bool { <ins> s.startedLock.Lock() defer s.startedLock.Unlock() if s.controller == nil { return false } </ins> return s.controller.HasSynced() }

Observed on but I don't think it's the pr. Similar to but I don't think I can dupe the stack traces.


Question: if !reflect.DeepEqual(n.podCIDRs, podCIDRs) { klog.ErrorS(nil, "Using NodeCIDR LocalDetector mode, current PodCIDRs are different than previous PodCIDRs, restarting", "node", klog.KObj(node), "newPodCIDRs", podCIDRs, "oldPODCIDRs", n.podCIDRs) <del> panic("Current Node PodCIDRs are different than previous PodCIDRs, restarting") </del> <ins> klog.FlushAndExit(klog.ExitFlushTimeout, 1) </ins> } }

There are some cases when kube-proxy has to exit and restart, in order to do that we panic, but this has the problem that the log output shows a literal "panic" message and engineers, support, customers and automation gets confused, thinking that is a problem when is the expected behavior. As suggested by in , we can replace by /sig network /help /kind cleanup
This request has been marked as needing help from a contributor. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : There are some cases when kube-proxy has to exit and restart, in order to do that we panic, but this has the problem that the log output shows a literal "panic" message and engineers, support, customers and automation gets confused, thinking that is a problem when is the expected behavior. /help /kind cleanup </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign I would like to raise a PR to fix this.


Question: // ServeHTTP logs the action that occurred and always returns the associated status code func (f *fakeActionHandler) ServeHTTP(response http.ResponseWriter, request *http.Request) { <del> f.lock.Lock() defer f.lock.Unlock() </del> <ins> func() { f.lock.Lock() defer f.lock.Unlock() f.actions = append(f.actions, fakeAction{method: request.Method, path: request.URL.Path, query: request.URL.RawQuery}) fakeResponse, ok := f.response[request.Method+request.URL.Path] if !ok { fakeResponse.statusCode = 200 fakeResponse.content = []byte("{"kind": "List"}") } response.Header().Set("Content-Type", "application/json") response.WriteHeader(fakeResponse.statusCode) response.Write(fakeResponse.content) }() </ins> <del> f.actions = append(f.actions, fakeAction{method: request.Method, path: request.URL.Path, query: request.URL.RawQuery}) fakeResponse, ok := f.response[request.Method+request.URL.Path] if !ok { fakeResponse.statusCode = 200 fakeResponse.content = []byte("{"kind": "List"}") </del> <ins> // This is to allow the fakeActionHandler to simulate a watch being opened if strings.Contains(request.URL.RawQuery, "watch=true") { hijacker, ok := response.(http.Hijacker) if !ok { return } connection, _, err := hijacker.Hijack() if err != nil { return } defer connection.Close() time.Sleep(30 * time.Second) </ins> } <del> response.Header().Set("Content-Type", "application/json") response.WriteHeader(fakeResponse.statusCode) response.Write(fakeResponse.content) </del> } // testServerAndClientConfig returns a server that listens and a config that can reference it

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: I am running etcd-operator in kubernetes, and noticed that when I deleted the etcdcluster (Custom resource definition) the pods & services did not get cleaned up. I looked on another kube environment I had running, and on that env everything worked as expected. I checked logs, and could see on the env that was working I saw the following messages in Controller manager logs: I didn't see these in the env that wasn't working, so I started looking at the garbage collector on there. I noticed that there hadn't been any log message from garbage collecor since 31st Jan, when this was logged: Around this time we experienced some environmental network issues, which caused lots of errors around that time, such as: It looks like these issues may have caused the garbage collector to get into some sort of hung state? When I restarted controller manager everything started working as expected. What you expected to happen: Garbage collector should have kept running, and working as expected. How to reproduce it (as minimally and precisely as possible): Not sure on the exact sequence that caused it, but suspect it was related to a time period when errors were occurring. Anything else we need to know?: We have set, and had previously hit this garbage collector issue - However, we are running with Kube 1.8.6 with the fix for that issue cherry-picked (commit ) Environment: Kubernetes version (use ):Cloud provider or hardware configuration: IBM Cloud OS (e.g. from /etc/os-release): Ubuntu "16.04.3 LTS (Xenial Xerus)" Kernel (e.g. ): 0-78-generic -Ubuntu SMP Thu Apr 27 15:29:09 UTC 2017 x8664 x8664 x86_64 GNU/Linux Install tools: Others: I will attach logs from Controller manager.
/sig api-machinery
/assign /assign
cc
I saw garbage collector stopped working in 1.9.2 too. Most controllers (such as node controllers or deployment controllers) stopped reporting logs in controller- when this happened. Manually restarting controller-manager fixed the problem. I think somehow some controllers got stuck (could it be because of some bugs in watch cache?) and it's not specific to garbage collector.

When the issue happened, still reported healthy for controller-manager.
tracks this
This is happening in production for us too.
did this happen to you to? Or is it that only the garbage collector disappeared from the logs?
For me the other controllers still have logs; it was just the GC that crashed.
does your cluster also use custom resources backed by Aggregated API servers?
Yes, although this happened in one cluster where we don't have any resources yet (the type exists). I'm not sure what that means
I still see messages from and and after the garbage collector stops working - so looks like some controllers are still working. Is there anything specific you were looking for?
I think the cause of this garbage collector hanging here is this: function periodically calls to the comment above GetDeletableResources, , an error in discovery causes the discovery client to no longer discover resources in the cluster, but instead of failing and returning an error, it simply logs the error as and returns an empty list of resources Sync function, upon recieving an empty resource list from discovery, detects that the resources have changed, and calls , which calls with as the argument as shown in the log as , which sets the list of monitors to an empty list because it thinks there are no resources to monitor. the Sync function calls controller.WaitForCacheSync, which calls , which will continually retry the function until it returns true, but it will always return false because is 0. So I think this was caused by which enabled number 2.


Question: "crypto/md5" "encoding/hex" "fmt" <ins> "strings" </ins> "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: I created a static pod for etcd as following with vsphere as cloud provider: But the hostname is using upper name, and when kubelet start the etcd static pod, it will report error as follows: I think that the vsphere should not return upper case hostname, but only lower case hostname. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/area cloudprovider
Here is a quick
/sig node


Question: } By("restarting all of the nodes") <del> err = restartNodes(framework.TestContext.Provider, framework.RestartPerNodeTimeout) </del> <ins> err = restartNodes(f, nodeNamesBefore) </ins> Expect(err).NotTo(HaveOccurred()) By("ensuring all nodes are ready after the restart")

This test wants to restart all nodes and make sure that kubernetes survives the process. That's a neat test, but it does the actual restart in a poor way. It does a MIG rolling-update to its own image. If a rolling-update already exists in some sort of error state then all future rolling-updates in that project on an instance group with the same name will fail until someone manually cancels the borked update. We can restart in other less-confusing ways. I suggest one of or . Both seem reasonable to me. cc
ref:
ref/


Question: // wait for all pods to be deleted ginkgo.By("waiting for all pods to be deleted") <del> err = wait.PollImmediate(podRetryPeriod, podRetryTimeout, checkPodListQuantity(f, "type=Testing", 0)) </del> <ins> err = wait.PollImmediate(podRetryPeriod, f.Timeouts.PodDelete, checkPodListQuantity(f, "type=Testing", 0)) </ins> framework.ExpectNoError(err, "found a pod(s)") })

pull-kubernetes-e2e-kind-ipv6 Pods should delete a collection of pods According to testgrid, March 18th Timeout in No response /sig node
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Looking at The test is here - It creates 3 pods which go into Failed phase and then expects them to within . Looking at failure in the build-log, there were three pods created, , , . did not get deleted from the API-server within the 1 minute timeout causing the test to fail. From the build-log, the test starts checking for all the pods to be deleted at: So the test expected all the pods to be deleted at 1 minute + 18:35:03 = 18:36:03 Looking at the logs of which landed on : DELETE was sent at landed on DELETE was sent at which is ~20 sec after when the test with the 1 minute timeout expected which caused the test to fail.
I suggest we bump the timeout a bit since everything seems to be working, it's just taking a bit longer than the timeout is currently set to. This also may be affected by recent which slightly increases how long it takes for pod to be marked as terminal by kubelet.


Question: reasons: []PredicateFailureReason{&PredicateFailureError{nodeports.Name, nodeports.ErrReason}}, name:    "hostport conflict", }, <ins> { pod: &v1.Pod{ Spec: v1.PodSpec{ Tolerations: []v1.Toleration{ {Key: "foo"}, {Key: "bar"}, }, }, }, nodeInfo: schedulerframework.NewNodeInfo(), node: &v1.Node{ ObjectMeta: metav1.ObjectMeta{Name: "machine1"}, Spec: v1.NodeSpec{ Taints: []v1.Taint{ {Key: "foo", Effect: v1.TaintEffectNoSchedule}, {Key: "bar", Effect: v1.TaintEffectNoExecute}, }, }, Status: v1.NodeStatus{Capacity: makeResources(10, 20, 32, 0, 0, 0).Capacity, Allocatable: makeAllocatableResources(10, 20, 32, 0, 0, 0)}, }, name: "taint/toleration match", }, { pod:      &v1.Pod{}, nodeInfo: schedulerframework.NewNodeInfo(), node: &v1.Node{ ObjectMeta: metav1.ObjectMeta{Name: "machine1"}, Spec: v1.NodeSpec{ Taints: []v1.Taint{ {Key: "foo", Effect: v1.TaintEffectNoSchedule}, }, }, Status: v1.NodeStatus{Capacity: makeResources(10, 20, 32, 0, 0, 0).Capacity, Allocatable: makeAllocatableResources(10, 20, 32, 0, 0, 0)}, }, name: "NoSchedule taint/toleration not match", }, { pod:      &v1.Pod{}, nodeInfo: schedulerframework.NewNodeInfo(), node: &v1.Node{ ObjectMeta: metav1.ObjectMeta{Name: "machine1"}, Spec: v1.NodeSpec{ Taints: []v1.Taint{ {Key: "bar", Effect: v1.TaintEffectNoExecute}, }, }, Status: v1.NodeStatus{Capacity: makeResources(10, 20, 32, 0, 0, 0).Capacity, Allocatable: makeAllocatableResources(10, 20, 32, 0, 0, 0)}, }, reasons: []PredicateFailureReason{&PredicateFailureError{tainttoleration.Name, tainttoleration.ErrReasonNotMatch}}, name:    "NoExecute taint/toleration not match", }, { pod:      &v1.Pod{}, nodeInfo: schedulerframework.NewNodeInfo(), node: &v1.Node{ ObjectMeta: metav1.ObjectMeta{Name: "machine1"}, Spec: v1.NodeSpec{ Taints: []v1.Taint{ {Key: "baz", Effect: v1.TaintEffectPreferNoSchedule}, }, }, Status: v1.NodeStatus{Capacity: makeResources(10, 20, 32, 0, 0, 0).Capacity, Allocatable: makeAllocatableResources(10, 20, 32, 0, 0, 0)}, }, name: "PreferNoSchedule taint/toleration not match", }, { pod: &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Annotations: map[string]string{ types.ConfigSourceAnnotationKey: types.FileSource, }, }, }, nodeInfo: schedulerframework.NewNodeInfo(), node: &v1.Node{ ObjectMeta: metav1.ObjectMeta{Name: "machine1"}, Spec: v1.NodeSpec{ Taints: []v1.Taint{ {Key: "foo", Effect: v1.TaintEffectNoSchedule}, {Key: "bar", Effect: v1.TaintEffectNoExecute}, }, }, Status: v1.NodeStatus{Capacity: makeResources(10, 20, 32, 0, 0, 0).Capacity, Allocatable: makeAllocatableResources(10, 20, 32, 0, 0, 0)}, }, name: "static pods ignore taints", }, </ins> } for _, test := range resourceTests { t.Run(test.name, func(t *testing.T) {

The method doesn't check taints and tolerations. This is problematic in 2 scenarios: If the users bypass scheduler by setting a value in If the node restarts. The node would try to restart scheduled pods once it comes back up, even if it's not ready (has a not-ready taint). Kubelet to check taints and tolerations. a taint in a node. a Pod with set to the Node above, without a matching toleration. /sig node /sig scheduling
this reminds me of your kubelet sync PR ...
I don't think this is a syncing issue. The logic doesn't check taints at all.
scenario 1 is pretty bad, i will let the #sig-node slack know about this.
It doesn't sound like a syncing issue, but we seem to have multiple issues with pod admission/management on restart currently
I'm aware of that issue. This should be unrelated.
I would like to address this. /assign
I think it would be great to proceed after merging ref:
Please add me as reviewer
/triage accepted
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale PR in review:
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.


Question: } func TestWatchDeleteEventObjectHaveLatestRV(t *testing.T) { <del> ctx, store, client := testSetup(t) </del> <ins> ctx, store, _ := testSetup(t) </ins> key, storedObj := storagetesting.TestPropagateStore(ctx, t, store, &example.Pod{ObjectMeta: metav1.ObjectMeta{Name: "foo"}}) <del> w, err := store.Watch(ctx, key, storage.ListOptions{ResourceVersion: storedObj.ResourceVersion, Predicate: storage.Everything}) </del> <ins> watchCtx, _ := context.WithTimeout(ctx, wait.ForeverTestTimeout) w, err := store.Watch(watchCtx, key, storage.ListOptions{ResourceVersion: storedObj.ResourceVersion, Predicate: storage.Everything}) </ins> if err != nil { t.Fatalf("Watch failed: %v", err) } <del> rv, err := storage.APIObjectVersioner{}.ObjectResourceVersion(storedObj) if err != nil { t.Fatalf("failed to parse resourceVersion on stored object: %v", err) } etcdW := client.Watch(ctx, key, clientv3.WithRev(int64(rv))) if err := store.Delete(ctx, key, &example.Pod{}, &storage.Preconditions{}, storage.ValidateAllObjectFunc, nil); err != nil { </del> <ins> deletedObj := &example.Pod{} if err := store.Delete(ctx, key, deletedObj, &storage.Preconditions{}, storage.ValidateAllObjectFunc, nil); err != nil { </ins> t.Fatalf("Delete failed: %v", err) } <del> var e watch.Event watchCtx, _ := context.WithTimeout(ctx, wait.ForeverTestTimeout) select { case e = <-w.ResultChan(): case <-watchCtx.Done(): t.Fatalf("timed out waiting for watch event") } deletedRV, err := deletedRevision(watchCtx, etcdW) if err != nil { t.Fatalf("did not see delete event in raw watch: %v", err) } watchedDeleteObj := e.Object.(*example.Pod) watchedDeleteRev, err := store.versioner.ParseResourceVersion(watchedDeleteObj.ResourceVersion) if err != nil { t.Fatalf("ParseWatchResourceVersion failed: %v", err) } if int64(watchedDeleteRev) != deletedRV { t.Errorf("Object from delete event have version: %v, should be the same as etcd delete's mod rev: %d", watchedDeleteRev, deletedRV) </del> <ins> // Verify that ResourceVersion has changed on deletion. if storedObj.ResourceVersion == deletedObj.ResourceVersion { t.Fatalf("ResourceVersion didn't changed on deletion: %s", deletedObj.ResourceVersion) </ins> } <del> } func deletedRevision(ctx context.Context, watch <-chan clientv3.WatchResponse) (int64, error) { for { select { case <-ctx.Done(): return 0, ctx.Err() case wres := <-watch: for _, evt := range wres.Events { if evt.Type == mvccpb.DELETE && evt.Kv != nil { return evt.Kv.ModRevision, nil } } </del> <ins> select { case event := <-w.ResultChan(): watchedDeleteObj := event.Object.(*example.Pod) if e, a := deletedObj.ResourceVersion, watchedDeleteObj.ResourceVersion; e != a { t.Errorf("Unexpected resource version: %v, expected %v", a, e) </ins> } } }

While working on generalizing tests to somehow better define spec for storage interface (and share them across all implementation of storage.interface - in particular etcd3 and watchcache), I found an unexpected inconsistency in Delete operation in etcd3. What is happening is that it effectively returns the state of the object from just before the deletion: However, this is inconsistent with how our watch is working, which is returning the state of the object from just before the deletion, but it sets the RV to the moment of deletion (which Delete call isn't doing). This sounds like a bug to me, but given this is a long-standing behavior we may need to take additional care when fixing this. /kind bug /sig api-machinery /priority important-soon /cc
I opened as a fix for it.
/triage accepted


Question: // older handlers http.HandleFunc("/hostName", hostNameHandler) http.HandleFunc("/shutdown", shutdownHandler) <ins> } func startHTTPSServer(httpsPort int, certFile, privKeyFile string) { log.Fatal(http.ListenAndServeTLS(fmt.Sprintf(":%d", httpPort), certFile, privKeyFile, nil)) } func startHTTPServer(httpPort int) { </ins> log.Fatal(http.ListenAndServe(fmt.Sprintf(":%d", httpPort), nil)) }

<!-- Please only use this template for submitting enhancement requests --What would you like to be : The image used in E2E tests is used for testing a wide variety of OS-agnostic features. One of these is , which runs an HTTP server with various endpoints that can be used to validate things like request/response. It would be useful to allow for 's functionality to also support HTTPS so all of that logic can be tested for TLS-based traffic. Currently, can be run like: I'm proposing something along the lines of: The default certificate/private key being basically identical to what provides. Why is this needed: One use case is where the intent is to add HTTPS functionality to lifecycle hooks (postStart/preStop). There is currently no easy way to test this logic because the container which receives the HTTPS request (in this case, ) does not support HTTPS traffic.
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig testing


Question: utilsysctl.VmPanicOnOOM:       utilsysctl.VmPanicOnOOMInvokeOOMKiller, utilsysctl.KernelPanic:        utilsysctl.KernelPanicRebootTimeout, utilsysctl.KernelPanicOnOops:  utilsysctl.KernelPanicOnOopsAlways, <ins> utilsysctl.RootMaxKeys:        utilsysctl.RootMaxKeysSetting, utilsysctl.RootMaxBytes:       utilsysctl.RootMaxBytesSetting, </ins> } sysctl := utilsysctl.New()

Forked from To make debian-based node compatible with docker 1.11.X, I updated both /proc/sys/kernel/keys/rootmaxkey and /proc/sys/kernel/keys/rootmaxbytes. Also other kubernetes components do the similar thing. For example, kube_proxy: We should clean up those configurations into one common place.
cc/
Is this looking for an owner? It seems the ask is to generalize these types of settings in a module.
Yeah. We update quite a few kernel settings in kubelet itself. We should move the ones mentioned in this issue to kubelet if it is applicable to other distros too. want to own this?
Sure, I'll start poking.
Test results so far for a few distributions' default settings Operating SystemValue --- --- Debian Jessie 8.7.5200 --- --- --- Ubuntu 14. --- --- --- Ubuntu 16. --- --- --- CentOS --- --- --- Fedora --- --- --- Redhat Atomic --- --- --- CoreOS stable1000000 --- --- undefined --- undefined


Question: // Pre-fetch all images tests depend on so that we don't fail in an actual test func PrePullAllImages() error { for _, image := range ImageRegistry { <del> output, err := exec.Command("docker", "pull", image).CombinedOutput() </del> <ins> var ( err    error output []byte ) for i := maxImagePullRetries; i > 0; i++ { if output, err = exec.Command("docker", "pull", image).CombinedOutput(); err == nil { break } time.Sleep(imagePullRetryDelay) } </ins> if err != nil { glog.Warningf("Could not pre-pull image %s %v output:  %s", image, err, output) return err

Another image pull related failure: Looking at the docker logs, it looks like docker is getting a 404 pulling some of the images
The error message looks normal. Kubelet always inspects to check whether the image is present before pulling image. If the image is not present, docker will record 404 error in log. However, the wired thing is that, if we've already prepull images, the image should be present. :)
The logs are from May 2nd, when we didn't have the image prepulling. Closing the issue.


Question: // Matches the analytics blurb, with or without the munge headers. analyticsRE = regexp.MustCompile(`[n]*` + analyticsExp + `[n]?` + <del> `|` + `[n]*` + beginMungeExp + `[^<]*` + endMungeExp + `[n]*`) </del> <ins> `|` + `[n]*` + beginMungeExp + `[^<]*` + endMungeExp) </ins> ) // This adds the analytics link to every .md file.

If you try mimicking the release flow for a release on right now, it's busted: If I generate a dummy commit out of that failure, then run to see what gets generated, it's attempting to actually add the unversioned warnings back in: I was trying to write up a PR to enact something for , but this blocks the release piece for that. Which is fine, we have some time.
cc
Running gendocs should add the unversioned warnings, unless the doc has <!-- TAG IS_VERSIONED --. Do you know if mark-new-version that tag before failing?
Ah, I think I see what might be happening. This is the post- version of Notice the newline fun.
Sorry which newline are you talking about? :)
My PR fixes that
Ah, yes, cool. It would help if you flagged functional changes along more loudly. :)
Yea, sorry about that. Have updated the PR description to highlight each fix.


Question: if pod.DeletionTimestamp != nil || pod.Status.Phase == v1.PodFailed { continue } <ins> // Pending pods are ignored. if pod.Status.Phase == v1.PodPending { ignoredPods.Insert(pod.Name) continue } // Pods missing metrics. </ins> metric, found := metrics[pod.Name] if !found { missingPods.Insert(pod.Name) continue } <ins> // Unready pods are ignored. </ins> if resource == v1.ResourceCPU { var ignorePod bool _, condition := podutil.GetPodCondition(&pod.Status, v1.PodReady)

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When the HPA scales up beyond the capacity of the cluster, many unscheduled pods are created. When load is removed, the usage ratio drops and the HPA tries to scale down. The unscheduled pods do not produce metrics, so the HPA fills in the 100% usage and re-evaluates the usage ratio. Because of the many fake 100% metrics, the new usage ratio is above the target value and the direction of scale is reversed. In this case, the HPA decides not to make any change to the current scale. What you expected to happen: I would expect the HPA to not get stuck when load is removed. How to reproduce it (as minimally and precisely as possible): and put it under load in a cluster without cluster autoscaling. the Deployment to increase the CPU request beyond the capability of the cluster (e.g. 500m). This should produce many Pending pods. load. The Pending pods will keep the HPA scaled indefinitely. Anything else we need to know?: Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-06-06T01:44:30Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"12+", GitVersion:"v1.12.9-gke.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-06-06T21:30:53Z", GoVersion:"go1.10.8b4", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig autoscaling
This might Previous I suspected that this was the same issues as (HPA positive feedback loop).

However it turns out to be different, that Pending pods cause HPA to get stuck.
Original reason for using 100% utilization for missing metrics when scaling down:


Question: current := 0 same := 0 <del> defer func() { By("Cleaning up the replication controller") err := DeleteRC(c, ns, name) Expect(err).NotTo(HaveOccurred()) }() </del> By(fmt.Sprintf("Creating replication controller %s", name)) _, err := c.ReplicationControllers(ns).Create(&api.ReplicationController{ ObjectMeta: api.ObjectMeta{

Currently the density tests are ok if all the pods come up eventually. We should verify there are no failure/error events while the pods are coming up. This could indicate issues dealing with the number of pods and should be investigated.
has been merged, and there's already an issue filed for followup actions, so I think we can close this.


Question: CROSS_BUILD_COPY qemu-QEMUARCH-static /usr/bin/ RUN apk add --no-cache bind-tools dnsmasq <ins> ADD https://github.com/coredns/coredns/releases/download/v1.5.0/coredns_1.5.0_linux_BASEARCH.tgz /coredns.tgz RUN tar -xzvf /coredns.tgz && rm -f /coredns.tgz </ins>

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: N/A Which test(s) are failing: should support configurable pod Since when has it been failing: N/A Testgrid link: N/A Reason for failure: Since ` was used instead. Anything else we need to know: /sig testing /sig windows


Question: } func (plugin *execNetworkPlugin) validate() error { <del> if syscall.Access(plugin.getExecutable(), X_OK) != nil { </del> <ins> if !isExecutable(plugin.getExecutable()) { </ins> errStr := fmt.Sprintf("Invalid exec plugin. Executable '%s' does not have correct permissions.", plugin.execName) return errors.New(errStr) }

/priority critical-urgent /priority failing-test /kind bug /status approved-for-milestone This test has been failing since 2017-11-22 in which is on the , and prevents us from cutting [v1.9.0-beta.1] (kubernetes/sig-release). Is there work ongoing to bring this job back to green? https://k8s- first bad: https://k8s- this is also failing on (non-release-blocking) triage cluster suspect commit range:
[MILESTONENOTIFIER] Milestone Issue Needs Attention Action required: During code freeze, issues in the milestone should be in progress. If this issue is not being actively worked on, please remove it from the milestone. If it is being worked on, please add the label so it can be tracked with other in-flight issues. Note: This issue is marked as , and must be updated every 1 day during code freeze. Example update: <details<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
/remove-area platform/gke bad copy-pasta
This is a new test that was for a 1.9 alpha feature but actually exposed a bug with a different 1.8 alpha feature. Some discussion in . Since this seems to be a bug introduced in 1.8, I'm not sure if it needs to block 1.9. cc


Question: framework.ExpectNoError(err, "Service didn't start within time out period. %v", pollErr) } }) <ins> ginkgo.It("A set of valid responses are returned for both pod and service Proxy", func() { ns := f.Namespace.Name msg := "foo" testSvcName := "e2e-proxy-test-service" testSvcLabels := map[string]string{"e2e-test": "proxy-endpoints"} framework.Logf("Creating pod...") _, err := f.ClientSet.CoreV1().Pods(ns).Create(context.TODO(), &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name: "agnhost", Labels: map[string]string{ "e2e-test": "proxy-endpoints"}, }, Spec: v1.PodSpec{ Containers: []v1.Container{{ Image:   imageutils.GetE2EImage(imageutils.Agnhost), Name:    "agnhost", Command: []string{"/agnhost", "porter", "--json-response"}, Env: []v1.EnvVar{{ Name:  "SERVE_PORT_80", Value: msg, }}, }}, RestartPolicy: v1.RestartPolicyNever, }}, metav1.CreateOptions{}) framework.ExpectNoError(err, "failed to create pod") err = wait.PollImmediate(podRetryPeriod, podRetryTimeout, checkPodStatus(f, "e2e-test=proxy-endpoints")) framework.ExpectNoError(err, "Pod didn't start within time out period") framework.Logf("Creating service...") _, err = f.ClientSet.CoreV1().Services(ns).Create(context.TODO(), &v1.Service{ ObjectMeta: metav1.ObjectMeta{ Name:      testSvcName, Namespace: ns, Labels:    testSvcLabels, }, Spec: v1.ServiceSpec{ Ports: []v1.ServicePort{{ Port:       80, TargetPort: intstr.FromInt(80), Protocol:   v1.ProtocolTCP, }}, Selector: map[string]string{ "e2e-test": "proxy-endpoints", }, }}, metav1.CreateOptions{}) framework.ExpectNoError(err, "Failed to create the service") transportCfg, err := f.ClientConfig().TransportConfig() framework.ExpectNoError(err, "Error creating transportCfg") restTransport, err := transport.New(transportCfg) framework.ExpectNoError(err, "Error creating restTransport") client := &http.Client{ CheckRedirect: func(req *http.Request, via []*http.Request) error { return http.ErrUseLastResponse }, Transport: restTransport, } // All methods for Pod Proxy return 200 // The response body returns 'foo' with the received http method httpVerbs := []string{"DELETE", "OPTIONS", "PATCH", "POST", "PUT"} for _, httpVerb := range httpVerbs { urlString := f.ClientConfig().Host + "/api/v1/namespaces/" + ns + "/pods/agnhost/proxy?method=" + httpVerb framework.Logf("Starting http.Client for %s", urlString) pollErr := wait.PollImmediate(requestRetryPeriod, requestRetryTimeout, validateProxyVerbRequest(client, urlString, httpVerb, msg)) framework.ExpectNoError(pollErr, "Pod didn't start within time out period. %v", pollErr) } // All methods for Service Proxy return 200 // The response body returns 'foo' with the received http method for _, httpVerb := range httpVerbs { urlString := f.ClientConfig().Host + "/api/v1/namespaces/" + ns + "/services/" + testSvcName + "/proxy?method=" + httpVerb framework.Logf("Starting http.Client for %s", urlString) pollErr := wait.PollImmediate(requestRetryPeriod, requestRetryTimeout, validateProxyVerbRequest(client, urlString, httpVerb, msg)) framework.ExpectNoError(pollErr, "Service didn't start within time out period. %v", pollErr) } // Test that each method returns 301 for both pod and service endpoints redirectVerbs := []string{"GET", "HEAD"} for _, redirectVerb := range redirectVerbs { urlString := f.ClientConfig().Host + "/api/v1/namespaces/" + ns + "/pods/agnhost/proxy?method=" + redirectVerb validateRedirectRequest(client, redirectVerb, urlString) urlString = f.ClientConfig().Host + "/api/v1/namespaces/" + ns + "/services/" + testSvcName + "/proxy?method=" + redirectVerb validateRedirectRequest(client, redirectVerb, urlString) } }) </ins> }) }) <ins> func validateRedirectRequest(client *http.Client, redirectVerb string, urlString string) { framework.Logf("Starting http.Client for %s", urlString) request, err := http.NewRequest(redirectVerb, urlString, nil) framework.ExpectNoError(err, "processing request") resp, err := client.Do(request) framework.ExpectNoError(err, "processing response") defer resp.Body.Close() framework.Logf("http.Client request:%s StatusCode:%d", redirectVerb, resp.StatusCode) framework.ExpectEqual(resp.StatusCode, 301, "The resp.StatusCode returned: %d", resp.StatusCode) } </ins> func checkPodStatus(f *framework.Framework, label string) func() (bool, error) { return func() (bool, error) { var err error

needs to be resolved before a number of e2e tests can be completed. 23 Oct 2021 [x] two weeks soak end date : 5 Nov 2021 [x] test promotion pr : kubernetes/kubernetes According to this APIsnoop query, there are still some remaining PodProxyOptions endpoints which are untested. As the these endpoints are redirected to their related endpoints. Create pod and confirm that the pod is running so that the tested endpoints can be found. Create a http.Client that checks for a redirect so that status code can be checked Loop through all http verbs, testing that the pod proxy endpoint returns the required 301 status code. Discover useragents: List endpoints hit by the test: Display endpoint coverage change: If a test with these calls gets merged, test coverage will go up by 7 points This test is also created with the goal of conformance promotion. After the merger of the test was updated to cover both the Pod and Service Proxy endpoint. Therefore If a test with these calls gets merged, test coverage will go up by 12 points connectCoreV1DeleteNamespacedPodProxy connectCoreV1DeleteNamespacedServiceProxy connectCoreV1OptionsNamespacedPodProxy connectCoreV1OptionsNamespacedServiceProxy connectCoreV1PatchNamespacedPodProxy connectCoreV1PatchNamespacedServiceProxy connectCoreV1PostNamespacedPodProxy connectCoreV1PostNamespacedServiceProxy connectCoreV1PutNamespacedPodProxy connectCoreV1PutNamespacedServiceProxy connectCoreV1HeadNamespacedPodProxy onnectCoreV1HeadNamespacedServiceProxy /sig testing /sig architecture /area conformance
/assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale To be revisited in 1.21
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at .

/lifecycle stale
/remove-lifecycle stale Focus for 1.22
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale Focus for 1.23
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale Focus for 1.23 Will move forward once merged
/open Waiting for 2 week flake free before promotion
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage accepted


Question: metrics.FrameworkExtensionPointDuration.WithLabelValues(metrics.Score, status.Code().String(), f.profileName).Observe(metrics.SinceInSeconds(startTime)) }() allNodePluginScores := make([]framework.NodePluginScores, len(nodes)) <del> numPlugins := len(f.scorePlugins) - state.SkipScorePlugins.Len() </del> <ins> numPlugins := len(f.scorePlugins) </ins> plugins := make([]framework.ScorePlugin, 0, numPlugins) pluginToNodeScores := make(map[string]framework.NodeScoreList, numPlugins) for _, pl := range f.scorePlugins {

When initializing kube-scheduler, We provided a custom config file for KubeSchedulerConfiguration and disabled the score plugin. This resulted in a panic in kube-scheduler at kube-scheduler up and running Using the below KubeSchedulerConfiguration file to enable kube-scheduler. apiVersion: kind: KubeSchedulerConfiguration clientConnection: kubeconfig: profiles: plugins: score: disabled: name: '*' extenders: urlPrefix: https://127.0.0.1:/scheduler filterVerb: "predicates" weight: 1 httpTimeout: 60s enableHTTPS: true nodeCacheCapable: false ignorable: false While trying to debug the issue, I found a change done recently for preScore calculation which might be causing the issue When we disable only the Score plugin then scheduler configuration will result in default preScorePlugins with 3 enabled [NodeResourcesFit, InterPodAffinity, TaintToleration] And due to this change, the preScore for InterPodAffinity results in skip status for some case (still not able to figured out what exactly it is), and this might result in a panic for kube-scheduler. The numplugin variable turns out to be negative. 28.3 N/A No response kubeadm No response No response
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scheduling
Let me take a look.
/assign
maybe related to
Past me argued this in the PR
scheduler-plugins' integration test also hits this: To end-users: if the fix () or its cherrypicks are not around in your k8s cluster. The workaround is that when disabling Score plugins, ensure to disable their PreScore counterparts as well.


Question: func (o *createClusterOptions) complete(cmd *cobra.Command) error { args := cmd.Flags().Args() if len(args) != 1 { <del> cmd.Help() return fmt.Errorf("Unexpected args: %v", args) </del> <ins> return helpErrorf(cmd, "Unexpected args: %v", args) </ins> } o.name = args[0]

The '-context' does not seem to add any value
add priority and team please.
Blocked on
/sig cli


Question: // String is the method to format the flag's value, part of the flag.Value interface. func (c *cidrs) String() string { <del> return strings.Join(c.ipn.StringSlice(), ",") </del> <ins> s := c.ipn.StringSlice() sort.Strings(s) return strings.Join(s, ",") </ins> } // Set supports a value of CSV or the flag repeated multiple times

/kind bug What happened: When building kubernetes-1.7.0 packages for openSUSE, results differ for each build in and other man pages. coming from that uses and probably sets do not have any specific order, so it would need to be sorted on output. What you expected to happen: Every build should give identical results. See https://reproducible- for why this matters. How to reproduce it (as minimally and precisely as possible): See Anything else we need to know?: Environment: Kubernetes version: 1.7.0 OS (e.g. from /etc/os-release): openSUSE Tumbleweed
/sig release
somehow this bug seems to have been forgotten - can you help getting the right people to have a look?
- can you please confirm that PR works for you?
I tested it and it looks good.
/assign I enjoy reading my highlights to find the issue has already been addressed. :)
Sorry this wasn't addressed sooner. It does seem like it fell into a black hole, and pinging was the right way to resolve that. :)
/area build-release
related to /reopen
Reopening this issue. <detailsIn response to : related to Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: if len(v) > 0 { out, err := b.exec.Run("iscsiadm", "-m", "node", "-p", tp, "-T", b.Iqn, "-I", b.Iface, "-o", "update", "-n", k, "-v", v) if err != nil { <del> return fmt.Errorf("iscsi: failed to update node session key %q with value %q error: %v", k, v, string(out)) </del> <ins> return fmt.Errorf("iscsi: failed to update node session key %q error: %v", k, string(out)) </ins> } } }

This issue was reported in the Description Kubernetes can be configured to use iSCSI volumes. When using CHAP authentication, CHAP secrets are stored using the Secrets API, such as in this . When a pod is configured to use iSCSI and the AttachDisk method is called, this will call the code in Figure 1. Figure 19.1: iSCSI secret handling These two functions both iterate over a slice of strings that are keys that reference secrets in a map. These are then used to generate iscsiadm commands. As shown, if there are errors in executing these commands, errors are returned with both the key and secret values in the error string. These errors will eventually be logged using klog: Someone with access to these logs would be able to view the sensitive secrets and could potentially gain access to iSCSI volumes. Exploit Scenario Alice runs a cluster, and wishes to use iSCSI for data storage. Eve has access sufficient to collect the logs, and uses this access to connect to iSCSI storage devices as a privileged user. Recommendation Short term, as in TOB-K8S-001: Bearer tokens revealed in logs, do not log sensitive credentials at any logging level, as they may accidentally leak into inappropriate environments, such as production. Long term, implement policies that enforce code review to ensure that sensitive data is not exposed in logs, or implement logging filters that check for sensitive data and remove it prior to reification within logs. In either case, ensure that sensitive data cannot be stored in logs. Anything else we need to know?: See for current status of all issues created from these findings. The vendor gave this issue an ID of ATR-K8S-003 and it was finding 22 of the report. The vendor considers this issue Medium Severity. To view the original finding, begin on page 61 of the Environment: Kubernetes version: 1.13.4
/sig storage /area security /kind bug /remove-kind feature


Question: }) } } <ins> func TestJobBackoffForOnFailure(t *testing.T) { jobConditionFailed := batch.JobFailed testCases := map[string]struct { // job setup parallelism  int32 completions  int32 backoffLimit int32 // pod setup jobKeyForget  bool restartCounts []int32 // expectations expectedActive          int32 expectedSucceeded       int32 expectedFailed          int32 expectedCondition       *batch.JobConditionType expectedConditionReason string }{ "too many job failures - single pod": { 1, 5, 2, true, []int32{2}, 0, 0, 1, &jobConditionFailed, "BackoffLimitExceeded", }, "too many job failures - multiple pods": { 2, 5, 2, true, []int32{1, 1}, 0, 0, 2, &jobConditionFailed, "BackoffLimitExceeded", }, "not enough failures": { 2, 5, 3, true, []int32{1, 1}, 2, 0, 0, nil, "", }, } for name, tc := range testCases { t.Run(name, func(t *testing.T) { // job manager setup clientset := clientset.NewForConfigOrDie(&restclient.Config{Host: "", ContentConfig: restclient.ContentConfig{GroupVersion: &legacyscheme.Registry.GroupOrDie(v1.GroupName).GroupVersion}}) manager, sharedInformerFactory := newJobControllerFromClient(clientset, controller.NoResyncPeriodFunc) fakePodControl := controller.FakePodControl{} manager.podControl = &fakePodControl manager.podStoreSynced = alwaysReady manager.jobStoreSynced = alwaysReady var actual *batch.Job manager.updateHandler = func(job *batch.Job) error { actual = job return nil } // job & pods setup job := newJob(tc.parallelism, tc.completions, tc.backoffLimit) job.Spec.Template.Spec.RestartPolicy = v1.RestartPolicyOnFailure sharedInformerFactory.Batch().V1().Jobs().Informer().GetIndexer().Add(job) podIndexer := sharedInformerFactory.Core().V1().Pods().Informer().GetIndexer() for i, pod := range newPodList(int32(len(tc.restartCounts)), v1.PodRunning, job) { pod.Status.ContainerStatuses = []v1.ContainerStatus{{RestartCount: tc.restartCounts[i]}} podIndexer.Add(&pod) } // run forget, err := manager.syncJob(getKey(job, t)) if err != nil { t.Errorf("unexpected error syncing job.  Got %#v", err) } if forget != tc.jobKeyForget { t.Errorf("unexpected forget value. Expected %v, saw %vn", tc.jobKeyForget, forget) } // validate status if actual.Status.Active != tc.expectedActive { t.Errorf("unexpected number of active pods.  Expected %d, saw %dn", tc.expectedActive, actual.Status.Active) } if actual.Status.Succeeded != tc.expectedSucceeded { t.Errorf("unexpected number of succeeded pods.  Expected %d, saw %dn", tc.expectedSucceeded, actual.Status.Succeeded) } if actual.Status.Failed != tc.expectedFailed { t.Errorf("unexpected number of failed pods.  Expected %d, saw %dn", tc.expectedFailed, actual.Status.Failed) } // validate conditions if tc.expectedCondition != nil && !getCondition(actual, *tc.expectedCondition, tc.expectedConditionReason) { t.Errorf("expected completion condition.  Got %#v", actual.Status.Conditions) } }) } } </ins>

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: When creating a job with and , the pod with the configuration included below continued to restart and was never marked as failed: What you expected to happen: We expected only 2 attempted pod restarts before the job was marked as failed. The pod kept restarting and was never set to failed, remained active. How to reproduce it (as minimally and precisely as possible): Create the above job and observe the number of pod restarts. Anything else we need to know?: The flag works as expected when . Environment: Kubernetes version (use ): 1.8.0 using minikube
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I think this is because for , kubelet will try to restart failed pod in place, and the pod status will be even the container is failed to run, so jobcontroller get a illusion as the pod is still running. Here's my naive thoughts: for short-term to avoid this, add a , for long-term we need consider how to recognize and count this kind of Pod as failed pod. cc
Just ran into the issue as well. It might be useful to amend the documentation to warn about the fact that won't be effective if is chosen as restart policy. At least, it wasn't obvious to me and I didn't catch any relevant note in the docs. Happy to do a quick PR if that's a desirable short-term action. For Pods failing as reported in this issue, what's the difference in behavior between a restart policy and a hypothetical scenario where is respected by ? Anything other than failing Pods being replaced by newly created ones in the former case vs. existing Pods being restarted in the latter?
For OnFailure, the failed Pod will restart on the same kubelet host, so emptyDir/hostPath is still there, for Never the failed Pod got delete and a replacement will create on another(may be the same) kubelet host, if the Pod use emptyDir/hostPath to store some tmp data, the tmp data may be lost.

For Pod don't use emptyDir/hostPath, I think there's no much difference for usage( will involve controller, apiserver, scheduler to do some work as it needs to create a replacement, backoffLimit policy will avoid bad jobSpec & container cause to create too many replacement consume control plane bandwidth).
Might it be a reasonable idea to move the parameter into the Pod? I'd think it could prove valuable to have an upper bound on the restarts on Pods themselves independently of Jobs, and it may solve the "how does the Cronjob controller determine that the limit has been reached" problem more elegantly if the Pod's state automatically transitions to failed once the limit is hit. WDYT?
can anyone tell me if I set the Never and the back off to 3 will it try a restart 3 times? this is very confusing
yes, it will. In fact, setting it to is the only way right now to have a back off limit be applied and effective.
I should mention that my statement is based on Kubernetes 1.8. I don't know if any fixes possibly made it into the upcoming 1.9. Then again, if there were, this issue should have been updated or at least referenced.
this is not working for a cronjob job spec (i am running 1.8.6) in the picture i still get new pods when i kill the previous on (5 created 3 should have been created and the job should have failed) any idea how to get this to actually work? !
what's your spec's ? I don't see it in your manifest.
every minute just for testing
Aren't you risking then that restarting jobs from a failed schedule and those from a succeeding schedule might be overlapping? How does it look like when you set the interval to, say, 10 minutes?
same result with 10 minutes interval
I ran a simplified version of your manifest against my Kubernetes 1.8.2 cluster with the command field replaced by so that every run fails, the schedule set to every 5 minutes, and the back-off limit reduced to 2. Here's my manifest: Initially, I get a series of three failing pods: Then it takes pretty much 5 minutes for the next batch to start up and fail: Interestingly, I only get two new containers instead of three.

The next batch though comes with three again: and so does the final batch I tested: Apart from the supposed off-by-one error, things seem to work for me.
Maybe you can double-check using my manifest?
In the we've stated it should work for both restart policies, I've addressed the problem in It counts pod's restartCount, if overall (iow. all containers in all pods) number exceeds backoff limit the job will fail.
I think the backoffLimit doesn't work at all. My kubernetes version is 1.10.3
See .
NAME READY STATUS RESTARTS AGE job-backofflimit-2-9n74x 0/1 Error 0 34s job-backofflimit-2-bt796 0/1 Error 0 1m job-backofflimit-2-fgktc 0/1 Error 0 49s job-backofflimit-2-gfhlx 0/1 Error 0 1m job-backofflimit-2-gzd54 0/1 ContainerCreating 0 8s job-backofflimit-2-ljlpx 0/1 Error 0 1m job-backofflimit-2-p5fxm 0/1 Error 0 1m job-backofflimit-2-w6s5d 0/1 Error 0 21s apiVersion: batch/v1 kind: Job metadata: name: job-backofflimit-2 spec: backoffLimit: 2 template: spec: containers: name: job-with-python2 image: restartPolicy: Never v1.10.4
9 has the same problem. My cluster is 1.9.6, and i run into the same issue


Question: } // WaitForServiceEndpointsNum waits until the amount of endpoints that implement service to expectNum. <ins> // Some components use EndpointSlices other Endpoints, we must verify that both objects meet the requirements. </ins> func WaitForServiceEndpointsNum(ctx context.Context, c clientset.Interface, namespace, serviceName string, expectNum int, interval, timeout time.Duration) error { return wait.PollWithContext(ctx, interval, timeout, func(ctx context.Context) (bool, error) { Logf("Waiting for amount of service:%s endpoints to be %d", serviceName, expectNum) <del> list, err := c.CoreV1().Endpoints(namespace).List(ctx, metav1.ListOptions{}) </del> <ins> endpoint, err := c.CoreV1().Endpoints(namespace).Get(ctx, serviceName, metav1.GetOptions{}) </ins> if err != nil { <del> return false, err </del> <ins> Logf("Unexpected error trying to get Endpoints for %s : %v", serviceName, err) return false, nil </ins> } <del> for _, e := range list.Items { if e.Name == serviceName && countEndpointsNum(&e) == expectNum { return true, nil } </del> <ins> if countEndpointsNum(endpoint) != expectNum { Logf("Unexpected number of Endpoints, got %d, expected %d", countEndpointsNum(endpoint), expectNum) return false, nil </ins> } <del> return false, nil </del> <ins> esList, err := c.DiscoveryV1().EndpointSlices(namespace).List(ctx, metav1.ListOptions{LabelSelector: fmt.Sprintf("%s=%s", discoveryv1.LabelServiceName, serviceName)}) if err != nil { Logf("Unexpected error trying to get EndpointSlices for %s : %v", serviceName, err) return false, nil } if len(esList.Items) == 0 { Logf("Waiting for at least 1 EndpointSlice to exist") return false, nil } if countEndpointsSlicesNum(esList) != expectNum { Logf("Unexpected number of Endpoints on Slices, got %d, expected %d", countEndpointsSlicesNum(esList), expectNum) return false, nil } return true, nil </ins> }) }

Kubernetes e2e suite: [It] [sig-api-machinery] AdmissionWebhook [Privileged:ClusterAdmin] should reject everything except leases The job was run on Aug 5th. I had a PR fixing typos in comments of kubelet source code. It is not related to api-machinery ipv6 but the test failed. The re-test passed. No response No response It seems the test is recently . /sig api-machinery /sig testing
It is failing considerably , it seems it fails to connect to the webhook that matches the time the test was I don't fully understand the test but it seems that is interacting with the cluster, I don't have clear if the deny policy affects everything and this is causing the race with the pod that implements the webhook /priority critical-urgent /release blocker /cc
/assign
cc
Debugged with Joe offline, paste for reference click open stderr on the failure to get the timeline
still failing after merge of
/triage accepted
It's suspicious that of all the admission webhook tests that start up a webhook pod, this is the one that flakes. I'll look more closely.
Looks like the webhook in this test may be configured to deny requests except leases globally in the cluster once enabled., which might create a race where, if the webhook is registered first , the controller setting up the webhook pod is denied requests by the webhook. I'll scope the webhook matching to the test namespace per offline suggestion.
Please let me know if I can be of help debugging this.Have free cycle and would like to contribute.
/assign
I can still see references to the expression that gives error Example run: Could you help confirm that the tests are completely removed? Thank you. cc
I think this is another test that is impacting the rest of the tests. Probably because the tests are running in parrallel. We might want to scope things down. Currently working on a fix.
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.

</details
this is a new bug, please review all the test that were and verify they are scoped to the custom created namespaces by the test framework, and if not, please tag them as cc:
/cc
I assume this could be closed with merged? /cc
Looks like the exact flak as reported in this issue has been resolved. /close
Closing this issue. <detailsIn response to : Looks like the exact flak as reported in this issue has been resolved. /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: <del> #!/bin/bash # Copyright 2014 Google Inc. All rights reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # #     http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. function detect-master () { echo "Running locally" } </del>

I've been pulling my hair out trying get kubectl talk to my local cluster. It seems that kubectl merges the local file with values pulled down from the cloud. I've done the following: The I just set is silently overridden. Questions: is this documented anywhere? should kubectl merge in my GCE cluster when I set my cloud-provider to something else? should kubectl override my previously set current-context?
workaround is to use flag to override the config file value.
but not a good UX long term
Someone else hit this as well: . Better fix is to have the local-up script set a context, which I will try to add today. As for the overwriting behavior, it's documented in , but based on what you describe, the intended load order is not working; local .kubeconfig should overwrite global. cc
That sounds infuriating. I'll take a look this morning.


Question: } var err error <ins> var output string </ins> podName := g.kubeControllerManager g.waitForControllerManagerReadyOnce.Do(func() { if readyErr := e2epod.WaitForPodsReady(g.client, metav1.NamespaceSystem, podName, 0); readyErr != nil {

! /kind failing-test <!-- If this is a flake, please add: /kind flake --/sig instrumentation /assign
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The flake happened in before.
This is fixed. !
/reopen This is still failing. Sample run:
Reopened this issue. <detailsIn response to : /reopen Sample run: Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/priority critical-urgent
The error message in makes me think the apiserver is not available during that test run: . Can someone link to the frequency graphs to see if the problem is widespread?
got it:
I don't think I was right above. I think the "the server is currently unable to handle the request" is an error coming from the proxy. The proxy was around the time this issue was filed (), so it seems likely to be related to it. can you take a look?
This is a known issue, and is working on it . Now that is being blocked by the debugging handler problem, could we merge to mitigate this problem first? cc
I would prefer to reach a decision about enabling the debug handlers, but if that's not possible, then merging PR as mitigation might make sense.
Think I hit this also.
It appears to still be failing based on
Why rebase? Any new test run will be based on merge with current k/k master and thus contain
<img width="1111" alt="Screen Shot 2021-05-28 at 13 04 52" src="https://user-" looks like it has become more stable now.
/close
Closing this issue. <detailsIn response to : /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: { Name:  "etcd", Image: etcdImage, <ins> Command: []string{ "/usr/local/bin/etcd", }, </ins> }, } d := &apps.Deployment{

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: I see only amd64 architecture etcd is used for the which will create issues for the other architectures like ppc64le, arm etc. We need to replace this etcd image with the k8s maintained fat manifest image present in
/sig testing


Question: } nodeIP := detectNodeIP(client, hostname, config.BindAddress) <del> protocol := utiliptables.ProtocolIPv4 if utilsnet.IsIPv6(nodeIP) { klog.V(0).Infof("kube-proxy node IP is an IPv6 address (%s), assume IPv6 operation", nodeIP.String()) protocol = utiliptables.ProtocolIPv6 } else { klog.V(0).Infof("kube-proxy node IP is an IPv4 address (%s), assume IPv4 operation", nodeIP.String()) } iptInterface = utiliptables.New(execer, protocol) </del> <ins> klog.Infof("Detected node IP %s", nodeIP.String()) </ins> // Create event recorder eventBroadcaster := record.NewBroadcaster()

(update: likely this is benign, but it seems really weird when your reading kube proxy logs) in healthy, IPTABLES jobs, the various ip6tables-restore commands are happy....MEANWHILE in ipvs jobs (healthy, or non healthy, we see that some of the iptables dont exist which makes sense... why is ipvs kube proxy tyrying to do iptables-restore operations on the table ? why is it trying to do iptables stuff at all ? It seems like we get error messages when running ipv6. not sure why though? in particular i found these errors when looking at the test failures for You wouldnt see error messages for ip6tables-restore when running ipvs tests. This may be my own ignorance however, i.e. maybe this is normal. I guess every job has this (wether or not the job passes or fails) ? Here is an example from a passing test.... Similar logs from a failing tests that i originally saw this message ... has this failure, it is reproduced via the corresponding ci job ... the logs in the kube-proxy for one of the nodes look like this: ill add more links as i collect more data. Kubernetes version (use ): prow (main branch as of feb 12 2021)
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig network
So, this happens in HEALTHY jobs to, so i guess its not a driver for an actual bug.... but its still weird to me :) ... updating original issue...
any idea on why those ipv6 errors are printed in ipvs code ?
The proxier is configured in dual-stack mode in the CI Since we enabled the dual-stack gate recently, this is the expected behaviour
I don't get similar printouts myself so I can't say anything for sure. By first check the old re-occuring (and very annoying) "KUBE-MARK-DROP" problem.

This chain is usually created by "kubeadm" but so far kubeadm is unaware of dual-stack and has only created this chain for ipv4 (in iptables). Then it needs to be created for IPv6 before kube-proxy is started; In my test cluster I always do this in a prep-stage before starting K8s.
The ci nodes doesn't have the kernel ip6tables nat modules loaded, hence they fail all the operation related
hmmm ok so is the issue here that we need to load that module in the CI jobs?
signing here because I want to at least find why ip6tables restore is called on kube-proxy ipvs things :D
ipvs does uses iptables too I think that kube-proxy should implement something similar to what implemented in kubelet The nat table for IPv6 was in 3.7 and it seems that it may be not present in some systems. /cc
/assign
I think it's mostly just about whether IPv6 is enabled, not about the kernel version.
IIRC this specific error in the CI is about the nat module only Anyway, you fixed it :clap: /unassign /assign


Question: apipod "k8s.io/kubernetes/pkg/api/v1/pod" "k8s.io/kubernetes/pkg/features" "k8s.io/kubernetes/pkg/scheduler/framework" <ins> "k8s.io/kubernetes/pkg/scheduler/framework/parallelize" </ins> "k8s.io/kubernetes/pkg/scheduler/metrics" "k8s.io/kubernetes/pkg/scheduler/util" )

When a pod requires preempting more than one pod, this happens sequentially. It should be done in parallel. /sig scheduling /sig scalability Preemption happens in the scheduling cycle, meaning that no other scheduling attempts happen while preemption is ongoing, hurting throughput. An alternative is to the deletes in a separate routine, like we do for binding.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
cc
I'm a newbie, may I have a try at this proposal?
Yes, this is a good candidate for new contributors. However, I would like to hear from SIG scalability whether there are any concerns about this.
Parallelizing the loops looks like the easier option to implement, but deleting in a separate routine will likely offer the better throughput for the average case.
Parallelizing loops seems fine from SIG Scalability perspective - we already do something similar for example in KCM when creating pods or in scheduler each binding is done by separate goroutine. One question though, is there a limit of pods that we want to preempt here? 100 parallel calls will be probably fine, but 1000 wouldn't be most likely.
We only preempt pods from a single node, so that's a natural threshold. And we have a parallelizing mini-library that has 16 workers by default.
Hi, do you still want to work on this one?
YesIt's still under discussion.
So we can use the parallelizing mini-library to do this?
You can use this parallelizer we defined in scheduler framework directly.
feel free to take it by writing on a comment.
/assign


Question: }, expectedNeedsUpdate: true, }, <ins> { testName: "If TargetGroup is different 1", updateFn: func() { oldSvc = &v1.Service{ ObjectMeta: metav1.ObjectMeta{ Name:      "tcp-service", Namespace: "default", }, Spec: v1.ServiceSpec{ Ports: []v1.ServicePort{{ Port:       80, Protocol:   v1.ProtocolTCP, TargetPort: intstr.Parse("20"), }}, Type: v1.ServiceTypeLoadBalancer, }, } newSvc = oldSvc.DeepCopy() newSvc.Spec.Ports[0].TargetPort = intstr.Parse("21") }, expectedNeedsUpdate: true, }, { testName: "If TargetGroup is different 2", updateFn: func() { oldSvc = &v1.Service{ ObjectMeta: metav1.ObjectMeta{ Name:      "tcp-service", Namespace: "default", }, Spec: v1.ServiceSpec{ Ports: []v1.ServicePort{{ Port:       80, Protocol:   v1.ProtocolTCP, TargetPort: intstr.Parse("22"), }}, Type: v1.ServiceTypeLoadBalancer, }, } newSvc = oldSvc.DeepCopy() newSvc.Spec.Ports[0].TargetPort = intstr.Parse("dns") }, expectedNeedsUpdate: true, }, </ins> } controller, _, _ := newController()

Hello , I have saw the commit , can we check the for enqueue to serivce cotnroller work queue? basecause of, In many case for Public Cloud, We always abstract a set for health checks , such as TargetGroup in AWS, instanceGroup in aliyun. We can map to such a collection of instance. If the service controller to check the targetgroup changes, the traffic pass-through pod is easier to implement. In most cases, the instance that in same group has an associated port for external services, and service controller set a target port if which not set in service yaml file. On the other handWe only lose some other performance of service controller. and Implementing targetgroup mapping to targetport depends on the cloud provider. What do you think? /sig cloud-provider /sig network
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.


Question: (disk.ManagedDisk != nil && diskURI != "" && strings.EqualFold(*disk.ManagedDisk.ID, diskURI)) { // found the disk klog.V(2).Infof("azureDisk - detach disk: name %q uri %q", diskName, diskURI) <del> if strings.EqualFold(ss.cloud.Environment.Name, "AZURESTACKCLOUD") { </del> <ins> if strings.EqualFold(ss.cloud.Environment.Name, AzureStackCloudName) { </ins> disks = append(disks[:i], disks[i+1:]...) } else { disks[i].ToBeDetached = to.BoolPtr(true)

<!-- Please only use this template for submitting enhancement requests --What would you like to be : PR() already proves that old api-version on Azure stack works on global Azure too, we could support azure stack in azure cloud provider lib, and then make sure following components works in Azure stack: CSI drivers: azure disk, azure file CCM Since it's mainly api-version change, we could also add e2e tests against global Azure too. Why is this needed: /assign /priority important-soon /sig cloud-provider /area provider/azure
there is blocking issue here(also need azure stack supports latest data model struct): just making the change in interfaces and not go-sdk but facing this error, since we don't have a switch inside go-sdk, we only do switch in azure cloud provider lib,
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale


Question: package e2e_node import ( <ins> . "github.com/onsi/ginkgo" </ins> "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" <ins> "k8s.io/apimachinery/pkg/util/uuid" </ins> "k8s.io/kubernetes/pkg/kubelet" kubecontainer "k8s.io/kubernetes/pkg/kubelet/container" "k8s.io/kubernetes/test/e2e/framework" <del> . "github.com/onsi/ginkgo" </del> ) const ( <del> logString = "This is the expected log content of this node e2e test" logPodName    = "logger-pod" logContName   = "logger-container" checkPodName  = "checker-pod" checkContName = "checker-container" </del> <ins> logString        = "This is the expected log content of this node e2e test" logContainerName = "logger" </ins> ) var _ = framework.KubeDescribe("ContainerLogPath [NodeConformance]", func() { f := framework.NewDefaultFramework("kubelet-container-log-path") <ins> var podClient *framework.PodClient </ins> Describe("Pod with a container", func() { Context("printed log to stdout", func() { <del> BeforeEach(func() { if framework.TestContext.ContainerRuntime == "docker" { // Container Log Path support requires JSON logging driver. // It does not work when Docker daemon is logging to journald. d, err := getDockerLoggingDriver() framework.ExpectNoError(err) if d != "json-file" { framework.Skipf("Skipping because Docker daemon is using a logging driver other than "json-file": %s", d) } // Even if JSON logging is in use, this test fails if SELinux support // is enabled, since the isolation provided by the SELinux policy // prevents processes running inside Docker containers (under SELinux // type svirt_lxc_net_t) from accessing the log files which are owned // by Docker (and labeled with the container_var_lib_t type.) // // Therefore, let's also skip this test when running with SELinux // support enabled. e, err := isDockerSELinuxSupportEnabled() framework.ExpectNoError(err) if e { framework.Skipf("Skipping because Docker daemon is running with SELinux support enabled") } } }) It("should print log to correct log path", func() { podClient := f.PodClient() ns := f.Namespace.Name logDirVolumeName := "log-dir-vol" logDir := kubelet.ContainerLogsDir logPod := &v1.Pod{ </del> <ins> makeLogPod := func(podName, log string) *v1.Pod { return &v1.Pod{ </ins> ObjectMeta: metav1.ObjectMeta{ <del> Name: logPodName, </del> <ins> Name: podName, </ins> }, Spec: v1.PodSpec{ // this pod is expected to exit successfully

The should be updated to check CRI log paths.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
May I try to help with this?
GitHub didn't allow me to assign the following users: fatsheep9146. Note that only and repo collaborators can be assigned. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: t.Errorf("expected error: %q", errStr) } } <ins> func TestDeleteRule(t *testing.T) { fcmd := fakeexec.FakeCmd{ CombinedOutputScript: []fakeexec.FakeAction{ // Exists func() ([]byte, []byte, error) { return []byte(`Bridge table: filter Bridge chain: OUTPUT, entries: 4, policy: ACCEPT -j TEST `), nil, nil }, // Fail to delete func() ([]byte, []byte, error) { return nil, nil, &fakeexec.FakeExitError{Status: 2} }, // Does not Exists. func() ([]byte, []byte, error) { return []byte(`Bridge table: filter Bridge chain: TEST, entries: 0, policy: ACCEPT`), nil, nil }, }, } fexec := fakeexec.FakeExec{ CommandScript: []fakeexec.FakeCommandAction{ func(cmd string, args ...string) exec.Cmd { return fakeexec.InitFakeCmd(&fcmd, cmd, args...) }, func(cmd string, args ...string) exec.Cmd { return fakeexec.InitFakeCmd(&fcmd, cmd, args...) }, func(cmd string, args ...string) exec.Cmd { return fakeexec.InitFakeCmd(&fcmd, cmd, args...) }, }, } runner := New(&fexec) err := runner.DeleteRule(TableFilter, ChainOutput, "-j", "TEST") errStr := "Failed to delete rule: exit 2, output: " if err == nil || err.Error() != errStr { t.Errorf("expected error: %q", errStr) } err = runner.DeleteRule(TableFilter, ChainOutput, "-j", "TEST") if err != nil { t.Errorf("expected err = nil") } } </ins>

<!-- Please only use this template for submitting enhancement requests --What would you like to be : The ebtables utility in is missing 3 useful features: method in its Interface table chain Why is this needed: The missing features are useful to allow for: rule deletion as there is a way to add (ensure) a rule but not removing it currently. manipulating the broute table and brouting chain The first feature would also align ebtables with iptables, as a DeleteRule method is available there.
/sig network
/sig network
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
Is there some feature/bugfix you're working on in kubernetes that needs this?
Yes, in kubernetes/dns (node-cache) actually. That PR is working, but it's only in my fork of kubernetes/dns at the moment as it depends on this. It solves this: Adding this code to ebtables would help me avoid re-inventing the wheel there.
... not sure if you've seen that ... but I have created a PR to solve this issue already:
Yes, I saw the PR first. So, kubernetes/kubernetes isn't really intended to be vendored into other repos and kubernetes/dns is being bad here. It's dubious to add methods to that kubernetes itself isn't using because someone else might just end up removing them later when they notice that they're not being used... OTOH the PR needs access to the internals of the ebtables code so you couldn't just add it to kubernetes/dns directly without forking all of ...
Thanks for your comments. I saw iptables used that way in node-cache and I thought it would be good for consistency to go down the same route with ebtables. To be honest I am not very keen on forking the entire package, for obvious reasons, but I can understand your reservations about this change.
/assign
Hi Did you manage to discuss this?
yes, sorry, didn't get a chance to get back to this on Friday. (I /lgtm'ed the PR) /remove-triage unresolved
Thanks!


Question: } type BindPlugin struct { <ins> name                  string </ins> numBindCalled         int <del> PluginName            string </del> bindStatus            *framework.Status client                clientset.Interface pluginInvokeEventChan chan pluginInvokeEvent

See - currently there's just one test that triggers Unreserve through a pre-bind failure.
/sig scheduling
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale
/good-first-issue
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign I shall work on it.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale
/assign
i saw commented the only test code, do you have any idea why this going on?
well, I found some functions missing, maybe because of the code refactor.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale


Question: nodes:        []string{"machine1", "machine2"}, services:     []*v1.Service{{Spec: v1.ServiceSpec{Selector: map[string]string{"bar": "foo"}}}}, sss:          []*apps.StatefulSet{{Spec: apps.StatefulSetSpec{Selector: &metav1.LabelSelector{MatchLabels: map[string]string{"foo": "bar"}}}}}, <del> expectedList: []schedulerapi.HostPriority{{Host: "machine1", Score: 0}, {Host: "machine2", Score: 5}}, name:         "disjoined service and replica set should be treated equally", </del> <ins> expectedList: []schedulerapi.HostPriority{{Host: "machine1", Score: 10}, {Host: "machine2", Score: 10}}, name:         "disjoined service and stateful set matches no pods", </ins> }, { pod: &v1.Pod{ObjectMeta: metav1.ObjectMeta{Labels: labels1, OwnerReferences: controllerRef("ReplicationController", "name", "abc123")}},

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: /kind bug What you expected to happen: Problem 1: Predicates filter nodes. Existing pods on those nodes will be not be counted when calculating max pods per (zone or node) resulting in imbalanced cluster. Eg: removes nodes which cannot . If any of the pods are already scheduled on this node, they are not considered when counting in . Problem 2: When there are 2 selectors(service and replication controller), it is sufficient to match any one selector for distribution. This creates imbalance ]. Pods from previous deploys matches and are counted when distributing pods across zones/nodes (Even though they do not match ) . These pods will be deleted. After the deploy completes, the cluster is imbalanced - by zone and/or pods per node. How to reproduce it (as minimally and precisely as possible): Problem 1: Get pods to get scheduled on nodes with very high cpu utilization [Existing cpu utilization + 1 new pod will result in almost equal to 0]. GeneralPredicate will drop this node after scheduling the first pod. This will lead to over utilization of an already loaded zone. If there are enough nodes like this one, it will create a pile on effect resulting in most pods scheduled in an already loaded zone. Problem 2: Have 2 selectors and deploy 30 pods across 3 zones. Twice. Anything else we need to know?: Environment: Kubernetes version (use ): /sig scheduling <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
re Problem 1: for per node, we only need to compare current nodes as they will share the same maxPods. for per zone, we need to account other nodes, e.g. because of resource predicates. re problem 2: a little about your case :(
Problem 1: : It does not matter if the denominator is all nodes or current nodes. One with higher utilization will still end up with a lower score. : It is critical that the denominator is all nodes. Else we are ending up with highly imbalanced cluster. Problem 2: Here is an example list of selectors.

All pods matching any one service selector are counted in So in our case, old pods are counted during scheduling. But they get killed during the deployment. If the maxSurge and maxUnavailable is high enough, you will end up with an imbalanced zone after deployment.
Gentle ping.
/assign bsalamat
Re: Problem 1 - You can also reproduce this by using pod anti-affinity. Zone score is calculated only based on nodes filtered by predicates. IMO zone score should be calculated using all nodes.
I will try to separate this into two PRs. Any other suggestions welcome.
I am working Problem 1. Looks like I have to open a new issue.


Question: } } <del> // TestValidateAllowPrivilegeEscalation will test that when the podSecurityPolicy // AllowPrivilegeEscalation is false we cannot set a container's securityContext // to allowPrivilegeEscalation, but when it is true we can. func TestValidateAllowPrivilegeEscalation(t *testing.T) { pod := defaultPod() pe := true pod.Spec.Containers[0].SecurityContext.AllowPrivilegeEscalation = &pe // create a PSP that does not allow privilege escalation psp := defaultPSP() psp.Spec.AllowPrivilegeEscalation = false provider, err := NewSimpleProvider(psp, "namespace", NewSimpleStrategyFactory()) if err != nil { t.Errorf("error creating provider: %v", err.Error()) } // expect a denial for this PSP and test the error message to ensure it's related to allowPrivilegeEscalation errs := provider.ValidateContainer(pod, &pod.Spec.Containers[0], field.NewPath("")) if len(errs) != 1 { t.Errorf("expected exactly 1 error but got %v", errs) } else { if !strings.Contains(errs.ToAggregate().Error(), "Allowing privilege escalation for containers is not allowed") { t.Errorf("did not find the expected error, received: %v", errs) } } // now add allowPrivilegeEscalation to the podSecurityPolicy psp.Spec.AllowPrivilegeEscalation = true errs = provider.ValidateContainer(pod, &pod.Spec.Containers[0], field.NewPath("")) if len(errs) != 0 { t.Errorf("directly allowing privilege escalation expected no errors but got %v", errs) } } // TestValidateDefaultAllowPrivilegeEscalation will test that when the podSecurityPolicy // DefaultAllowPrivilegeEscalation is false we cannot set a container's // securityContext to allowPrivilegeEscalation but when it is true we can. func TestValidateDefaultAllowPrivilegeEscalation(t *testing.T) { pod := defaultPod() pe := true pod.Spec.Containers[0].SecurityContext.AllowPrivilegeEscalation = &pe // create a PSP that does not allow privilege escalation psp := defaultPSP() dpe := false psp.Spec.DefaultAllowPrivilegeEscalation = &dpe psp.Spec.AllowPrivilegeEscalation = false provider, err := NewSimpleProvider(psp, "namespace", NewSimpleStrategyFactory()) if err != nil { t.Errorf("error creating provider: %v", err.Error()) } // expect a denial for this PSP and test the error message to ensure it's related to allowPrivilegeEscalation errs := provider.ValidateContainer(pod, &pod.Spec.Containers[0], field.NewPath("")) if len(errs) != 1 { t.Errorf("expected exactly 1 error but got %v", errs) } else { if !strings.Contains(errs.ToAggregate().Error(), "Allowing privilege escalation for containers is not allowed") { t.Errorf("did not find the expected error, received: %v", errs) } } // now add DefaultAllowPrivilegeEscalation to the podSecurityPolicy dpe = true psp.Spec.DefaultAllowPrivilegeEscalation = &dpe psp.Spec.AllowPrivilegeEscalation = false // expect a denial for this PSP because we did not allowPrivilege Escalation via the PodSecurityPolicy // and test the error message to ensure it's related to allowPrivilegeEscalation errs = provider.ValidateContainer(pod, &pod.Spec.Containers[0], field.NewPath("")) if len(errs) != 1 { t.Errorf("expected exactly 1 error but got %v", errs) } else { if !strings.Contains(errs.ToAggregate().Error(), "Allowing privilege escalation for containers is not allowed") { t.Errorf("did not find the expected error, received: %v", errs) } } // Now set AllowPrivilegeEscalation psp.Spec.AllowPrivilegeEscalation = true errs = provider.ValidateContainer(pod, &pod.Spec.Containers[0], field.NewPath("")) if len(errs) != 0 { t.Errorf("directly allowing privilege escalation expected no errors but got %v", errs) } // Now set the psp spec to false and reset AllowPrivilegeEscalation psp.Spec.AllowPrivilegeEscalation = false pod.Spec.Containers[0].SecurityContext.AllowPrivilegeEscalation = nil errs = provider.ValidateContainer(pod, &pod.Spec.Containers[0], field.NewPath("")) if len(errs) != 1 { t.Errorf("expected exactly 1 error but got %v", errs) } else { if !strings.Contains(errs.ToAggregate().Error(), "Allowing privilege escalation for containers is not allowed") { t.Errorf("did not find the expected error, received: %v", errs) </del> <ins> func TestAllowPrivilegeEscalation(t *testing.T) { ptr := func(b bool) *bool { return &b } tests := []struct { pspAPE    bool  // PSP AllowPrivilegeEscalation pspDAPE   *bool // PSP DefaultAllowPrivilegeEscalation podAPE    *bool // Pod AllowPrivilegeEscalation expectErr bool expectAPE *bool // Expected value of pod APE (if no error) }{ // Test all valid combinations of PSP AllowPrivilegeEscalation, // DefaultAllowPrivilegeEscalation, and Pod AllowPrivilegeEscalation. {true, nil, nil, false, nil}, {true, nil, ptr(false), false, ptr(false)}, {true, nil, ptr(true), false, ptr(true)}, {true, ptr(false), nil, false, ptr(false)}, {true, ptr(false), ptr(false), false, ptr(false)}, {true, ptr(false), ptr(true), false, ptr(true)}, {true, ptr(true), nil, false, ptr(true)}, {true, ptr(true), ptr(false), false, ptr(false)}, {true, ptr(true), ptr(true), false, ptr(true)}, {false, nil, nil, false, ptr(false)}, {false, nil, ptr(false), false, ptr(false)}, {false, nil, ptr(true), true, nil}, {false, ptr(false), nil, false, ptr(false)}, {false, ptr(false), ptr(false), false, ptr(false)}, {false, ptr(false), ptr(true), true, nil}, // Invalid cases: pspAPE=false, pspDAPE=true } fmtPtr := func(b *bool) string { if b == nil { return "nil" </ins> } <del> } // Now unset both AllowPrivilegeEscalation psp.Spec.AllowPrivilegeEscalation = true pod.Spec.Containers[0].SecurityContext.AllowPrivilegeEscalation = nil errs = provider.ValidateContainer(pod, &pod.Spec.Containers[0], field.NewPath("")) if len(errs) != 0 { t.Errorf("resetting allowing privilege escalation expected no errors but got %v", errs) </del> <ins> return strconv.FormatBool(*b) } for _, test := range tests { t.Run(fmt.Sprintf("pspAPE:%t_pspDAPE:%s_podAPE:%s", test.pspAPE, fmtPtr(test.pspDAPE), fmtPtr(test.podAPE)), func(t *testing.T) { pod := defaultPod() pod.Spec.Containers[0].SecurityContext.AllowPrivilegeEscalation = test.podAPE psp := defaultPSP() psp.Spec.AllowPrivilegeEscalation = test.pspAPE psp.Spec.DefaultAllowPrivilegeEscalation = test.pspDAPE provider, err := NewSimpleProvider(psp, "namespace", NewSimpleStrategyFactory()) require.NoError(t, err) err = provider.DefaultContainerSecurityContext(pod, &pod.Spec.Containers[0]) require.NoError(t, err) errs := provider.ValidateContainer(pod, &pod.Spec.Containers[0], field.NewPath("")) if test.expectErr { assert.NotEmpty(t, errs, "expected validation error") } else { assert.Empty(t, errs, "expected no validation errors") ape := pod.Spec.Containers[0].SecurityContext.AllowPrivilegeEscalation assert.Equal(t, test.expectAPE, ape, "expected pod AllowPrivilegeEscalation") } }) </ins> } }

/kind bug What happened: In , there are tests for {Default,}AllowPrivilegeEscalation fields, but those in actually test the AllowPrivilegeEscalation as this is set in the test instead of testing its suggested Default* form, thus being mostly redundant to tests in . On top of that, part of the tests in test a scenario which is not allowed, that being are wrong due to but since this is assigned directly, it was never validated and thus no errors were found. What you expected to happen: There are no redundant or incorrect tests in . Environment: Kubernetes version: master
/sig auth
Resolved by


Question: go install ./cmd/... # Use eval to preserve embedded quoted strings. <del> eval "goflags=(${KUBE_GOFLAGS:-})" </del> <ins> eval "goflags=(${GOFLAGS:-})" </ins> # Filter out arguments that start with "-" and move them to goflags. targets=()

We should just use GOFLAGS
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
/sig contributor-experience
/assign