Question: return false } <del> // GetAllocatableDevices returns information about all the devices known to the manager </del> <ins> // GetAllocatableDevices returns information about all the healthy devices known to the manager </ins> func (m *ManagerImpl) GetAllocatableDevices() ResourceDeviceInstances { m.mutex.Lock() <del> resp := m.allDevices.Clone() m.mutex.Unlock() klog.V(4).InfoS("Known devices", "numDevices", len(resp)) </del> <ins> defer m.mutex.Unlock() resp := m.allDevices.Filter(m.healthyDevices) klog.V(4).InfoS("GetAllocatableDevices", "known", len(m.allDevices), "allocatable", len(resp)) </ins> return resp }

When calling GetAllocatableDevices, the API returns all the registered devices, regardless of their health status. Thing is, the - and in general doesn't make much sense to report as allocatable Unhealthy devices, as is also documented in the (see last paragraph) The API should not report unhealthy device. This means the reported device set can potentially change across multiple invocations of the API, as . a kube = 1.21 environment, enable the feature gate any device plugin - including sample device plugin sure 1+ devices are reported unhealthy the podresources GetAllocatableResources API, observe the unhealthy device is reported in the return value. N/A Kubernetes version (use ): 1.21 and beyond Cloud provider or hardware configuration: not relevant OS (e.g: ): any Kernel (e.g. ): not relevant Install tools: not relevant Network plugin and version (if this is a network-related bug): not relevant Others: not relevant
/sig node
/triage accepted


Question: // pod deletion doesn't affect. Pods are deployed on the same node or different nodes depending on requiresSameNode. // Read/write check are done across pod, by check reading both what pod{n-1} and pod{n} wrote from pod{n}. func TestConcurrentAccessToSingleVolume(f *framework.Framework, cs clientset.Interface, ns string, <del> node e2epod.NodeSelection, pvc *v1.PersistentVolumeClaim, numPods int, requiresSameNode bool) { </del> <ins> node e2epod.NodeSelection, pvc *v1.PersistentVolumeClaim, numPods int, requiresSameNode bool, readOnly bool) { </ins> var pods []*v1.Pod

What happened: Running two pods that use the same iSCSI volume, one of them fails with: What you expected to happen: Both pods should be running Anything else we need to know?: It's caused by iscsi volume plugin persiting iSCSI settings into the mount directory itself (e.g. ): volume plugin creates the global mount directory. volume plugin creates there. volume plugin mounts the volume there. This shadows the file, but it's fine, as long as no other pod uses the same volume on the same node. creates a second pod that uses the volume. volume plugin tries to create in the global volume directory. Since the volume is mounted, it will either writes the file to the volume (!) or fails, if the volume is read-only.
/sig storage /assign /priority impotant-soon
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig storage /assign /priority impotant-soon Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/priority important-soon
Our e2e did not catch this, because they always create only one pod when testing read-only volumes.


Question: scalefake "k8s.io/client-go/scale/fake" core "k8s.io/client-go/testing" "k8s.io/client-go/tools/cache" <ins> "k8s.io/client-go/tools/record" </ins> "k8s.io/client-go/util/workqueue" "k8s.io/klog/v2" _ "k8s.io/kubernetes/pkg/apis/core/install"

When using PDBs with or (percentage), it is not supported to reference pods via a PDB selector, that do not have a supported controller owner reference: Deployment, ReplicationController, ReplicaSet, and StatefulSet controllers or a resource implementing subresource This is also documented Nevertheless, there are still people who create PDBs a pods without controllers that ignore this. For example PDBs that set , want to block eviction, and node drain (it works for them but the PDB status is invalid). Unfortunately there is no feedback for the user to receive when this happens. The KCM logs is the only place that informs the user when unmanaged pods are found. Which I think is insufficient. There should be some way to inform users about this. Since it cannot be know on admission we probably have to do it when reconciling the PDB and unmanaged pods are found. Options: an event when this is detected - probably needs to be throttled in some way a condition in a PDB status that would be set and admins could monitor these to detect misbehaving PDBs create a PDB create unmanaged pod observe incorrect status because this configuration is unsupported No response <detailsNA </details<detailsNA </details<detailsNA </details<details</details<details</details<details</details
/sig apps
/triage accepted This was discussed during today's sig-apps, we've agreed to add a warning event right next to the log statement from the original description
I'll take this on.
/assign
/reopen I did some more testing for the unsupported controllers and found out that these are covered by the event, so I posted a followup to reflect that The new event should cover only the naked pods
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: i += 2 } <del> // filenames have `/go/src/k8s.io` prefix which dont exist locally for i, v := range frames { frames[i].File = strings.Replace(v.File, "/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes", *k8sPath, 1) </del> <ins> // filenames are in one of two special GOPATHs depending on if they were // built dockerized or with the host go // we want to trim this prefix to produce portable relative paths k8sSRC := *k8sPath + "/_output/local/go/src/k8s.io/kubernetes/" for i := range frames { trimmedFile := strings.TrimPrefix(frames[i].File, k8sSRC) trimmedFile = strings.TrimPrefix(trimmedFile, "/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/") frames[i].File = trimmedFile </ins> } for _, curFrame := range frames {

The currently provided command for generating requires bazel, the command should be updated to one which does not require bazel Currently: Suggested: /sig testing /sig architecture /area conformance
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/assign /remove-lifecycle rotten
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle rotten
/remove-lifecycle rotten /help
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /remove-lifecycle rotten /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.

</details
We're going to need to update this to verify under as well instead of under the bazel "unit tests", ref: I'll work on this.
done in
This is wonderful to see. Thank you


Question: // Update the deployment template - the new replicaset should stay the same framework.Logf("Updating paused deployment %q", deploymentName) <del> newTGPS := int64(40) </del> <ins> newTGPS := int64(0) </ins> deployment, err = framework.UpdateDeploymentWithRetries(c, ns, d.Name, func(update *extensions.Deployment) { update.Spec.Template.Spec.TerminationGracePeriodSeconds = &newTGPS })

https://k8s- Failed: [] Deployment paused deployment should be ignored by the controller {Kubernetes e2e suite} Previous issues for this test:
https://k8s- Failed: [] Deployment paused deployment should be ignored by the controller {Kubernetes e2e suite}
dup
https://k8s- Failed: [] Deployment paused deployment should be ignored by the controller {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment paused deployment should be ignored by the controller {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment paused deployment should be ignored by the controller {Kubernetes e2e suite}
All of these failures are related to the cleanup phase of the test - pods are terminating but never actually removed. We should probably use a zero grace period for pods.
https://k8s- Failed: [] Deployment paused deployment should be ignored by the controller {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.


Question: } return item.refCount } <del> assert.Equal(t, refs("ns1", "s1"), 1) assert.Equal(t, refs("ns1", "s10"), 1) assert.Equal(t, refs("ns1", "s2"), 1) assert.Equal(t, refs("ns1", "s3"), 3) assert.Equal(t, refs("ns1", "s30"), 2) assert.Equal(t, refs("ns1", "s4"), 2) assert.Equal(t, refs("ns1", "s5"), 4) assert.Equal(t, refs("ns1", "s50"), 2) assert.Equal(t, refs("ns1", "s6"), 0) assert.Equal(t, refs("ns1", "s60"), 0) assert.Equal(t, refs("ns1", "s7"), 0) </del> <ins> assert.Equal(t, 1, refs("ns1", "s1")) assert.Equal(t, 1, refs("ns1", "s10")) assert.Equal(t, 1, refs("ns1", "s2")) assert.Equal(t, 3, refs("ns1", "s3")) assert.Equal(t, 2, refs("ns1", "s30")) assert.Equal(t, 2, refs("ns1", "s4")) assert.Equal(t, 4, refs("ns1", "s5")) assert.Equal(t, 2, refs("ns1", "s50")) assert.Equal(t, 0, refs("ns1", "s6")) assert.Equal(t, 0, refs("ns1", "s60")) assert.Equal(t, 1, refs("ns1", "s7")) assert.Equal(t, 1, refs("ns1", "s70")) </ins> } func TestCachingConfigMapManager(t *testing.T) {

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Occasionally (more frequently if the node is heavily loaded) the node fails to mount a secret or configmap for a pod. Error recovery fails and the pod must be replaced with a new pod to have any hope of getting the secret or configmap volume mounted. The problem appears to be that if a pod is updated, its secrets are removed from the secretStore's secrets map. See: Normally the effects of the bug are not felt since the usual order of operations masks the bug: 1) The pod is created and registered the first time with the secrets manager 2) The volume mounter gets the secret 3) The pod is updated and registered again with the secrets manager, which deletes the secrets (the bug) If the race is lost, though, then the bug symptoms are manifest: 1) The pod is created and registered the first time with the secrets manager 2) The pod is updated and registered again with the secrets manager, which deletes the secrets (the bug) 3) The volume mounter tries to get the secret and fails with the message What you expected to happen: On update, the secrets manager should only delete secrets if they are not present in the latest pod definition (rather than deleting all prior secrets). Whether the pod update happens before or after the volume mounter attempts to get the secret, it should succeed. How to reproduce it (as minimally and precisely as possible): Because it is a race condition, it's difficult to reproduce. We've most often seen the bug on systems that were under load. 1) Create a pod that has a secrets or configmap volume 2) See if the pod fails to mount the volume. Anything else we need to know?: If the above analysis is correct, this bug has been present since 1.6 for secret volumes, and 1.7 for configmap volumes. Environment: We've seen the bug in software based upon the 1.6 release.
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list .

The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
The PR addresses this issue. /sig node
This may be a false alarm. I've never actually reproduced it in Kubernetes proper, only in OpenShift. Since our smoking gun was anything but, I'm going to close this until I manage to find an actual problem in Kubernetes.
To close the loop on this for any that find it via future searches: It turns out that this was a false alarm. The above analysis was flawed and this turns out not to be a bug (or maybe a very minor one if we consider it a bug to log a an error on a (semantic) non-error). The error messages like were not an indicator of a failure to mount a secret volume, but instead a failure to update a secret volume during pod teardown. Here's how the error message is triggered: pod is created volume manager gets the secret and mounts the volume the lifetime of the pod, the volume manager periodically re-checks the secret to make sure that it's still in a good state. pod goes away volume is unmounted If one of the refreshes from step 3 happens after step 4 but before step 5, the log message will appear.


Question: "testing" "github.com/GoogleCloudPlatform/kubernetes/pkg/api" <ins> "github.com/GoogleCloudPlatform/kubernetes/pkg/util" </ins> ) <ins> func TestFindPort(t *testing.T) { manifest := api.ContainerManifest{ Containers: []api.Container{ { Ports: []api.Port{ { Name:          "foo", ContainerPort: 8080, HostPort:      9090, }, { Name:          "bar", ContainerPort: 8000, HostPort:      9000, }, }, }, }, } port, err := findPort(&manifest, util.IntOrString{Kind: util.IntstrString, StrVal: "foo"}) expectNoError(t, err) if port != 8080 { t.Errorf("Expected 8080, Got %d", port) } port, err = findPort(&manifest, util.IntOrString{Kind: util.IntstrString, StrVal: "bar"}) expectNoError(t, err) if port != 8000 { t.Errorf("Expected 8000, Got %d", port) } port, err = findPort(&manifest, util.IntOrString{Kind: util.IntstrInt, IntVal: 8000}) if port != 8000 { t.Errorf("Expected 8000, Got %d", port) } port, err = findPort(&manifest, util.IntOrString{Kind: util.IntstrInt, IntVal: 7000}) if port != 7000 { t.Errorf("Expected 7000, Got %d", port) } port, err = findPort(&manifest, util.IntOrString{Kind: util.IntstrString, StrVal: "baz"}) if err == nil { t.Error("unexpected non-error") } port, err = findPort(&manifest, util.IntOrString{Kind: util.IntstrString, StrVal: ""}) expectNoError(t, err) if port != 8080 { t.Errorf("Expected 8080, Got %d", port) } port, err = findPort(&manifest, util.IntOrString{}) expectNoError(t, err) if port != 8080 { t.Errorf("Expected 8080, Got %d", port) } } </ins> func TestSyncEndpointsEmpty(t *testing.T) { serviceRegistry := MockServiceRegistry{} podRegistry := MockPodRegistry{}

What would you like to be : We'd like to have the Prometheus stack () running in our scalability tests. Why is this needed: There are couple of reasons: will simplify writing SLI assertions/tests that are based on metrics not necessarily exported on master. Some examples here are the new In-Cluster Network Programming Latency SLI (exported in kube-proxy) or soon to be implemented RTT SLI where the metric will be exported in regular pods (probes). With Prometheus stack, collecting and aggregating such metrics is very simple. -prometheus provides many useful metrics about the state of the cluster, such metrics might be invaluable when debugging scalability test failures. mimics what customers do in their cluster, making the test setup more realistic.
/sig scalability
/assign
The GCE 5K Node Performance test with enabled Prometheus completed successfully. We can enable Prometheus in all tests.
Looks like kubemark-500 has some issue, test fails often with Kubemark-5000 also failed once with Will disable Prometheus in kubemark, will work on fixing it when I'm back from my vacation.
For the first kind of the failure that kubemark-500 suffered from, i.e. the root cause is Looks like the token for the "prometheus-operator" service account was not ready, as we can see in the logs it was created after the POST to create the servicemonitor was called.
It looks like the second failure (kubemark-5000) was caused by the prometheus-k8s pods not scheduled by the kube-scheduler. This is strange but may be related to other issues we had. Hard to investigate what really happened there, will leave it for now. Will re-enable prometheus tomorrow and will see if it happens again.
Closing, as this is basically done in a minimal version. Other improvements will be tracked in separate issues. /close
Closing this issue. <detailsIn response to : Closing, as this is basically done in a minimal version. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: if err := wrapped.SetUpAt(dir, fsGroup); err != nil { return err } <ins> if err := volumeutil.MakeNestedMountpoints(b.volName, dir, b.pod); err != nil { return err } </ins> optional := b.source.Optional != nil && *b.source.Optional secret, err := b.getSecret(b.pod.Namespace, b.source.SecretName)

This bug affects any pod that has a container with a nested volume (any type) beneath an API data volume, such as a secret, configMap, downwardAPI or projected volume. In the description and reproduction details below I use a secret volume, but any of the four will demonstrate this behavior. Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: When creating a pod with nested secrets volumes (e.g. one volume mounted at and one mounted at ), the container fails to start with an error like: What you expected to happen: The container should start with both and mounted. How to reproduce it (as minimally and precisely as possible): out master, any version since a file with 2 secrets and a pod that makes volumes for the 2 secrets, with one of them mounted within the other: the 2 secrets and the pod. Assuming you named the above file : Anything else we need to know?: This may work for other runtimes if they create mountpoints outside of the sandbox. But it's definitely broken with Docker as the runtime.
This should be fixed by /sig storage /sig node /assign


Question: for nodeName, cpu := range nodeToAllocatableMap { requestedCPU := cpu * 7 / 10 fillerPods = append(fillerPods, createPausePod(f, pausePodConfig{ <del> Name: "filler-pod-" + nodeName, </del> <ins> Name: "filler-pod-" + string(uuid.NewUUID()), </ins> Resources: &v1.ResourceRequirements{ Limits: v1.ResourceList{ v1.ResourceCPU: *resource.NewMilliQuantity(requestedCPU, "DecimalSI"),

/kind bug cc What happened: Running one scheduling predicates conformance test on a 1.7.10 cluster where node names are FQDN, e.g. kube-node-, fails. What you expected to happen: The tests should pass. How to reproduce it (as minimally and precisely as possible): Follow or bootstrap a Kubernetes cluster where nodes names are FQDN, e.g. my-node- Anything else we need to know?: Also affects master but not 1.8.x. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: GCP OS (e.g. from /etc/os-release):Kernel (e.g. ):Install tools: Others: none
Cherry-pick for 1.7.x Again, it doesn't seem to affect 1.8.x.


Question: resources := []schema.GroupVersionResource{ batch.Resource("cronjobs").WithVersion("v1beta1"), networking.Resource("ingresses").WithVersion("v1beta1"), <del> // TODO #83513 csinodes override can be removed in 1.18 apisstorage.Resource("csinodes").WithVersion("v1beta1"), </del> apisstorage.Resource("csidrivers").WithVersion("v1beta1"), }

<!-- Please only use this template for submitting enhancement requests --What would you like to be : storage/v1.CSINode is being introduced in 1.17 via The storage version should switch to v1 in 1.18. Why is this needed: We need to wait until 1.18 to handle the HA upgrade case where a controller may be 1 version higher than the API server. /assign
These files will have to be updated:


Question: if err != nil { glog.Errorf("Failed to execute iptables-restore: %v", err) // Revert new local ports. <del> for k, v := range newLocalPorts { glog.Errorf("Closing local port %s", k.String()) v.Close() } </del> <ins> revertPorts(replacementPortsMap, proxier.portsMap) </ins> return } // Close old local ports and save new ones. for k, v := range proxier.portsMap { <del> if newLocalPorts[k] == nil { </del> <ins> if replacementPortsMap[k] == nil { </ins> v.Close() } } <del> proxier.portsMap = newLocalPorts </del> <ins> proxier.portsMap = replacementPortsMap </ins> // Clean up the older SNAT rule which was directly in POSTROUTING. // TODO(thockin): Remove this for v1.3 or v1.4.

The symptom is the nodeport service is accessible thru hostip:hostport. But the port on host is not occupied. This is exposed by e2e on kubenet. I noticed in the kube- Kube-proxy did open the port. But later, kube-proxy closed the ports upon iptables-restore failure. It never reopen the ports. Hence causing the e2e failure. The code section in question: Not sure why failed. Error messge is not helpful since line 139 is ref:
Also of interest: did it exit with 1? This should be clear from the iptables commands it executed before the commit, which are hopefully in the logs (can you post them). If it's something like a non-exist chain, we should fix that. we retry on exit 1? Handwaving without looking at code: I'd expect something to get requeued, and to reinvoke syncProxyRules once again, in a time frame << test timeout.
Yeah, what was line 139 in the restore output? , Prashanth B wrote:
is No retry though. It is going to catch up in the next sync. So I think no retry for is needed.
FYI Jenkins job:
That iptables restore is 139 lines long, so it's unhappy about the last line? can you hunt ths? , Minhan Xia wrote:
The next sync is whenver the next service/endpoint changes right? Errors during the sync should cause an immediate retry of sync, preferably with backoff. If a service or endpoint never changes we should still recover. Yes we should hunt down the bad line, but it's not the last one right? COMMIT will fail if eg: Even though it's obviously not commit but the tcp/upd thing.
iptables resyncs every 30s, not ideal but in this case I don't expect that to fix the error I meant that we need to root cause it :) , Prashanth B wrote:
But in this case, iptables-restore is going to complain . Not line 2
Hmm
I manually restored the 139 lines above it worked fine - no errors. Even more curious. , Prashanth B wrote:
I also tried to execute the same sequence of iptables-restore. It worked. No errors.

Was this a one-time thing? Were there any other changes in your git repo? , Minhan Xia wrote:
This happened in kubenet suite. Only difference is . I looked at a few runs where e2e failed. This happened 3-4 times over the whole test, but not consecutively. In all-passed runs I checked, this never happened (Maybe I am lucky).
Of , Minhan Xia wrote:


Question: plugin := plugins[0] fakeMounter := iscsiDiskMounter{ iscsiDisk: &iscsiDisk{ <ins> Iface:  "192.168.1.10:pv0001", </ins> plugin: plugin.(*iscsiPlugin)}, exec: fakeExec, } <del> newIface := "192.168.1.10:pv0001" cloneIface(fakeMounter, newIface) </del> <ins> cloneIface(fakeMounter) </ins> if cmdCount != 4 { t.Errorf("expected 4 CombinedOutput() calls, got %d", cmdCount) }

What happened: When creating a block volume, the iSCSI storage plugin stores volume-related data in different directories: As a result, some data is not cleaned up once the volume is detached: How to reproduce it (as minimally and precisely as possible): These are the objects I used to reproduce this issue: pod- /sig storage CC
/assign


Question: if servicePort > 0 && !util.IsDNSLabel(name) { return fmt.Errorf("Service creation requested, but an invalid name for a service was provided (%s). Service names must be valid DNS labels.", name) } <ins> ports, err := portsFromString(portSpec) if err != nil { return err } </ins> controller := &api.ReplicationController{ JSONBase: api.JSONBase{ ID: name,

Kubernetes will create replication controller and pods in case of port spec error, e.g. will create 'myNginx'. A following correct spec will fail:
/cc


Question: sc := &schedulerConfigurator{ // policyfile and algorithm are intentionally undefined. } <del> _, error := sc.Create() if error == nil { </del> <ins> _, err := sc.Create() if err == nil { </ins> t.Fatalf("Expected error message when creating with incomplete configurator.") } }

There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals


Question: container, err = strconv.Atoi(pieces[0]) if err != nil { glog.Errorf("Container port is not integer: %s %v", pieces[0], err) <del> continue </del> <ins> return nil, err </ins> } } else { host, err = strconv.Atoi(pieces[0]) if err != nil { glog.Errorf("Host port is not integer: %s %v", pieces[0], err) <del> continue </del> <ins> return nil, err </ins> } container, err = strconv.Atoi(pieces[1]) if err != nil { glog.Errorf("Container port is not integer: %s %v", pieces[1], err) <del> continue </del> <ins> return nil, err </ins> } } if container < 1 { glog.Errorf("Container port is not valid: %d", container) <del> continue </del> <ins> return nil, err </ins> } result = append(result, api.Port{ContainerPort: container, HostPort: host}) } <del> return result </del> <ins> return result, nil </ins> } // RunController creates a new replication controller named 'name' which creates 'replicas' pods running 'image'.

Kubernetes will create replication controller and pods in case of port spec error, e.g. will create 'myNginx'. A following correct spec will fail:
/cc


Question: } err := o.deleteOrEvictPodsSimple() <del> // TODO: update IsTooManyRequests() when the TooManyRequests(429) error returned from the API server has a non-empty Reason field for i := 1; i <= maxPatchRetry && apierrors.IsTooManyRequests(err); i++ { if i > triesBeforeBackOff { currBackOffPeriod := time.Duration(math.Exp2(float64(i-triesBeforeBackOff))) * backOffPeriod fmt.Fprintf(o.errOut, "Retry in %vn", currBackOffPeriod) o.backOff.Sleep(currBackOffPeriod) } fmt.Fprintf(o.errOut, "Retryingn") err = o.deleteOrEvictPodsSimple() } </del> if err == nil { cmdutil.PrintSuccess(o.mapper, false, o.out, "node", o.nodeInfo.Name, false, "drained") }

In writing some documentation for I noticed some issues with the implementation. I think the simplest user experience is that the command doesn't return until all the user pods are drained off the machine. In particular this means wait until all graceful termination is finished infinitely retry evictions that are rejected due to insufficient PodDisruptionBudget The current implementation does not do this, in particular wait for graceful termination appears to be at 10 sec number of pods, which may be much shorter than the max graceful termination we only retry evictions number of times My suggestion is to change the behavior so that we compute how long to wait for graceful termination, by taking max(grace periods of all pods that we want to delete) (or the value of --grace-period command line argument instead, if that is specified), and wait exactly that amount of time plus a small buffer we retry evictions infinitely WDYT? I think we should do this in 1.5 because the current behavior is very non-obvious.
The problem here is how long should we wait, if the user doesn't specify a timeout. The default timeout used in drain is following how long the reapers have in , , and . Maybe this is not long enough. I can make the changes. Do we still need exponential back off?
To summarize discussion with there are two possible approaches (pseudocode below) 1) Goroutine approach: For each pod P that meets the , do the following: 2) Non-goroutine approach: where "not timed out yet" means you haven't finished waiting max(--grace period argument to kubectl drain, max graceful termination spec across all pods on the node)
Could you please triage this for 1.5 and apply blocker or non-blocker labels as appropriate along with priority.
The PR for this issue is waiting to be merged.
marking this as a release blocker due to being marked as such. Please update if needed cc:


Question: } } <ins> func TestMissingFromIndex(t *testing.T) { api.ForTesting_ReferencesAllowBlankSelfLinks = true o := testclient.NewObjects(api.Scheme, api.Scheme) if err := testclient.AddObjectsFromPath("../../examples/persistent-volumes/claims/claim-01.yaml", o, api.Scheme); err != nil { t.Fatal(err) } if err := testclient.AddObjectsFromPath("../../examples/persistent-volumes/volumes/local-01.yaml", o, api.Scheme); err != nil { t.Fatal(err) } client := &testclient.Fake{ReactFn: testclient.ObjectReaction(o, latest.RESTMapper)} pv, err := client.PersistentVolumes().Get("any") if err != nil { t.Error("Unexpected error getting PV from client: %v", err) } claim, error := client.PersistentVolumeClaims("ns").Get("any") if error != nil { t.Errorf("Unexpected error getting PVC from client: %v", err) } volumeIndex := NewPersistentVolumeOrderedIndex() mockClient := &mockBinderClient{ volume: pv, claim:  claim, } // the default value of the PV is Pending. // if has previously been processed by the binder, it's status in etcd would be Available. // Only Pending volumes were being indexed and made ready for claims. pv.Status.Phase = api.VolumeAvailable // adds the volume to the index, making the volume available syncVolume(volumeIndex, mockClient, pv) if pv.Status.Phase != api.VolumeAvailable { t.Errorf("Expected phase %s but got %s", api.VolumeBound, pv.Status.Phase) } // an initial sync for a claim will bind it to an unbound volume, triggers state change err = syncClaim(volumeIndex, mockClient, claim) if err != nil { t.Fatalf("Expected Clam to be bound, instead got an error: %+vn", err) } // state change causes another syncClaim to update statuses syncClaim(volumeIndex, mockClient, claim) // claim updated volume's status, causing an update and syncVolume call syncVolume(volumeIndex, mockClient, pv) if pv.Spec.ClaimRef == nil { t.Errorf("Expected ClaimRef but got nil for pv.Status.ClaimRef: %+vn", pv) } if pv.Status.Phase != api.VolumeBound { t.Errorf("Expected phase %s but got %s", api.VolumeBound, pv.Status.Phase) } if claim.Status.Phase != api.ClaimBound { t.Errorf("Expected phase %s but got %s", api.ClaimBound, claim.Status.Phase) } if len(claim.Status.AccessModes) != len(pv.Spec.AccessModes) { t.Errorf("Expected phase %s but got %s", api.ClaimBound, claim.Status.Phase) } if claim.Status.AccessModes[0] != pv.Spec.AccessModes[0] { t.Errorf("Expected access mode %s but got %s", claim.Status.AccessModes[0], pv.Spec.AccessModes[0]) } // pretend the user deleted their claim mockClient.claim = nil syncVolume(volumeIndex, mockClient, pv) if pv.Status.Phase != api.VolumeReleased { t.Errorf("Expected phase %s but got %s", api.VolumeReleased, pv.Status.Phase) } } </ins> type mockBinderClient struct { volume *api.PersistentVolume claim  *api.PersistentVolumeClaim

In case of cluster crash and status recovery, the persistentVolumeClaimBinder would not re-add available volumes to its internal index. Users would not be able to claim those volumes.
This is really bad. I think we should just nix Pending phase and always consider every Available volume for indexing - map lookups are cheap.


Question: // NewServiceConfig creates a new ServiceConfig. func NewServiceConfig(ctx context.Context, serviceInformer v1informers.ServiceInformer, resyncPeriod time.Duration) *ServiceConfig { result := &ServiceConfig{ <del> listerSynced: serviceInformer.Informer().HasSynced, logger:       klog.FromContext(ctx), </del> <ins> logger: klog.FromContext(ctx), </ins> } <del> _, _ = serviceInformer.Informer().AddEventHandlerWithResyncPeriod( </del> <ins> handlerRegistration, _ := serviceInformer.Informer().AddEventHandlerWithResyncPeriod( </ins> cache.ResourceEventHandlerFuncs{ AddFunc:    result.handleAddService, UpdateFunc: result.handleUpdateService,

AKS had a customer report repeated issues in their clusters where: -proxy would redeploy (e.g. due to AKS deploying a new kube-proxy image with CVE fixes) c-ares DNS client would send repeated DNS queries to the kube-dns service VIP from the same src IP address, creating a UDP conntrack entry to the svc VIP and keeping it alive. For example, with kube-dns svc VIP 192.168.0.10, conntrack showed: queries to the svc VIP were blackholed until the customer manually ran a command like: Issue occurred only in clusters with many services and endpoints (~10k services and ~6k endpoints). However, the customer has seen this issue repeatedly for months, about 1-5 times per week across their clusters. kube-proxy code to delete stale UDP conntrack entries should have deleted the conntrack entry to kube-dns svc VIP automatically in the first sync after service and endpointslice cache is initialized. Unfortunately, the customer said they were unable to reproduce this issue in other environments outside of their production clusters. AKS engineers were also unable to repro this issue. Update (2024-08-03): I'm now able to reproduce the issue using a DNS client that reuses the same src IP / port and clears DNAT conntrack entries between queries, in a k8s 1.29.7 cluster with 2,000 services. The bug is triggered reliably in this setup with The test client code, scripts, and steps to reproduce are in this repository: The customer shared kube-proxy logs (at verbosity "3") from when the issue occurred. Unfortunately, I don't have permission to share the full logs publicly, but I think what I'm seeing in the logs gives a clue about what's happening. 05:00:02. nodes, endpointslices, and services all synced: -dns endpointslices processed after , but before iptables sync: iptables sync at 05:00:02. shows 4658 services, which is much less than the number of services in the cluster. (We can't see if kube-dns is one of these services, since kube-proxy at doesn't log which services were processed.)

at 05:00:02. and the next sync at 05:00:05., we see thousands of lines that look like services and endpointslices updates: sync at 05:00:05. shows 9605 services: I believe this could explain why kube-proxy does not clean up the stale UDP conntrack entry: kube-proxy detects that the endpointslice and service caches are synced using respectively. Comments on the say that this indicates that the informer cache is up-to-date, but warns "Note that this doesn't tell you if an individual handler is synced!! For that, please call HasSynced on the handle returned by AddEventHandler." This comment, along with the output of kube-proxy logs, makes me suspect that kube-proxy is initiating a sync before it has received and processed all the services and endpoints in the informer cache. If so, this could explain why the "delete conntrack" codepath is skipped. kube-proxy deletes stale conntrack entries for UDP services that are updated from and are in . If in the first sync is missing kube-dns, but includes kube-dns, then kube-proxy would . Subsequent syncs would see at least one endpoint for kube-dns, so would continue to skip deletion. If this theory is correct, then I wonder if kube-proxy could change the initialization check to ensure that all pre-sync events from the informer cache are delivered before the first iptables sync, maybe like this: Should kube-proxy wait until all pre-sync events are delivered before setting ? kube-proxy version is v1.28.5 Note that the kube-proxy image that AKS built for 1.28.5 includes the backported patch to fix this race condition in iptables partial sync: Azure Kubernetes Service Linux (I don't have the exact version but could get it if it's relevant) N/A N/A N/A
Possibly this issue would be addressed by
/sig network
you can workaround this in the meantime with an init container flushing all udp entries `
Just to clarify, an init container on the kube-proxy pod?

If I'm reading the code correctly, when kube-dns is missing from , then the iptables restore on the first sync won't install DNAT rules for kube-dns If so, I think even with an init container flushing conntrack, there's still a period between iptables restore and the next iptables sync where a DNS client could create a conntrack entry to the svc VIP.
I see, I don't have time to test it, but then you need to do two things up the kube-proxy iptables rules to avoid to have stale entries to services with endpoints that does not exist the existing UDP entries directed to Services a strawman approach to add to the existing kube-proxy manifest
joined us on the call today, and confirmed that we should move forward with the patch he proposed: /triage accepted /assign
I'm now able to reproduce the bug reliably using the test client and steps here: After applying the patch to wait for , the issue no longer occurs.


Question: return allErrs } <del> // AccumulateUniquePorts runs an extraction function on each Port of each Container, </del> <ins> // AccumulateUniqueHostPorts extracts each HostPort of each Container, </ins> // accumulating the results and returning an error if any ports conflict. <del> func AccumulateUniquePorts(containers []api.Container, accumulator map[int]bool, extract func(*api.ContainerPort) int) errs.ValidationErrorList { </del> <ins> func AccumulateUniqueHostPorts(containers []api.Container, accumulator *util.StringSet) errs.ValidationErrorList { </ins> allErrs := errs.ValidationErrorList{} for ci, ctr := range containers { cErrs := errs.ValidationErrorList{} for pi := range ctr.Ports { <del> port := extract(&ctr.Ports[pi]) </del> <ins> port := ctr.Ports[pi].HostPort </ins> if port == 0 { continue } <del> if accumulator[port] { cErrs = append(cErrs, errs.NewFieldDuplicate("port", port)) </del> <ins> str := fmt.Sprintf("%d/%s", port, ctr.Ports[pi].Protocol) if accumulator.Has(str) { cErrs = append(cErrs, errs.NewFieldDuplicate("port", str)) </ins> } else { <del> accumulator[port] = true </del> <ins> accumulator.Insert(str) </ins> } } allErrs = append(allErrs, cErrs.PrefixIndex(ci)...)

Pods which use to bind both TCP and UDP sockets to the same port number are rejected with the error: An example pod follows.
Looks like we don't check protocol
The issue was discussed before in , which was closed due to it being low priority.
Hmm, that's 2 pods and the scheduler, this is 2 containers in the same pod and validation at create time. Maybe we should just be more clear about where we stand with hostPort.
Posting here what I posted in the other bug: This can't be worked around when running NFS from a pod. Linux does not allow the NFS port (2049) to be changed, and the Portmap port (111) also cannot be changed. Additionally, you can't separate the UDP from the TCP ports.
Why does NFS need host ports? , Russell Harmon wrote:
Fix is in , Tim Hockin wrote:


Question: }, } <ins> // Flag errors exit with code 1, however according to the diff // command it means changes were found. // Thus, it should return status code greater than 1. cmd.SetFlagErrorFunc(func(command *cobra.Command, err error) error { cmdutil.CheckDiffErr(cmdutil.UsageErrorf(cmd, err.Error())) return nil }) </ins> usage := "contains the configuration to diff" cmd.Flags().StringVarP(&options.Selector, "selector", "l", options.Selector, "Selector (label query) to filter on, supports '=', '==', and '!='.(e.g. -l key1=value1,key2=value2)") cmdutil.AddFilenameOptionFlags(cmd, &options.FilenameOptions, usage)

returns 1 as an exit code: ( keep last exit code in Fish Shell - behaves the same in bash) Based on the help message for diff subcommand: I would expect to have anything returned. In Bash: I found it, troubleshooting misbehavior of a deployment process on my CI server with older kubectl version. That inconsistency might lead to unexpected behavior. Kubernetes version (use ):
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig sig-cli
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig sig-cli Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cli
/assign
For unsupported flag, it is a general case which handled by cobra lib. By default, we set the exit code as 1 when something wrong with the kubectl command. subCommand never gets a chance to overwrite the exit code in this case.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale
/cc Round up in your KEP?
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
It still seems to be a problem. /remove-lifecycle stale
Thanks for fixing that!


Question: # Make /tmp hermetic build --sandbox_tmpfs_path=/tmp <ins> # Ensure that Bazel never runs as root, which can cause unit tests to fail. # This flag requires Bazel 0.5.0+ build --sandbox_fake_username </ins>

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug /sig api-machinery What happened: pull-kubernetes-bazel job is failing See https://k8s- What you expected to happen: test should pass How to reproduce it (as minimally and precisely as possible): bazel test //thirdparty/forked/etcd221/pkg/fileutil:godefault_test
/priority failing-test /area test-infra
/assign
Something weird going on with the prow environment, because this test passes locally with bazel 0.5.2.
Since it is late I will preemptively declare MISSION ACCOMPLISHED from my aircraft carrier and 0.4.5 downgrade workaround (kubernetes/test-infra). I'm sure once we figure things out we'll want to revert this job back to 0.5.2 but in the mean time this should unblock things and the queue should resume merging over the weekend. Expectation is this passes: https://k8s-
I don't know if it's a different issue, but bazel is still failing/flaking all over the place: https://k8s-
/close


Question: "strings" "testing" <ins> corev1 "k8s.io/api/core/v1" </ins> "k8s.io/apimachinery/pkg/util/sets" "k8s.io/apiserver/pkg/endpoints/metrics" "k8s.io/component-base/metrics/legacyregistry"

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: for extension API server panics with: Apparently introduces this panic with . What you expected to happen: to do not panic. How to reproduce it (as minimally and precisely as possible): extension API server with - for example the endpoint that the apiserver panics with Anything else we need to know?: Environment: Kubernetes version (use ): / Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/kind regression
/sig api-machinery
/assign
cc
Oops - sorry for that. It seems it is basically this comment: that I blindly applied without e2e retesting (the problem seem to be lack of tests, which is a bit surprising because bugs in my previous implementations were caught by tests). ^^ BTW - this has been cherrypicked to previous releases: 16: 17: 18: so it's actually even more critical...
I can file the automated cherry-pick PRs once the PR against the master branch is merged. :)
Reopening - since we need to cherrypick it still.
I now opened the cherry-picks: 16 - 17 - 18 -
Fix and picks are merged


Question: record_command run_nodes_tests fi <ins> ######################## # Authentication ######################## record_command run_exec_credentials_tests </ins> ######################## # authorization.k8s.io #

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: With the exec credentials plugin configured in my kubeconf, and a stale/expired token present in the same file issuing e.g. triggers the authentication flow as handled by the plugin - once the flow is returned to kubectl (by printing the ExecCredential object) the new token is stored to the kubeconfig but not immediately used to resolve the request, leading to: Immediately issuing again works as kubectl now seem to use the credentials (token) stored following the first request. What you expected to happen: That kubectl would read the new token as fetched by the exec credential plugin and use it already for the first command issued. How to reproduce it (as minimally and precisely as possible): the kubernetes API server with the OIDC authenticator enabled. kubectl to use an exec credential plugin printing the token received out of band. that the token was indeed written to the kubeconf file. that the first kubectl command failed. that the next identical command succeeds. Anything else we need to know?: This works as expected when there is no token previously stored in the kubeconf - the behavior is only triggered when there is a token stored but it expired, or for whatever other reason not accepted by the API server. Environment: Kubernetes version (use ): 1.14, also tried with 1.17 Cloud provider or hardware configuration: OS (e.g: ): Mac OS for kubectl client Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig auth /sig cli
could you provide more details around how is configured. For example, it is unclear to me if you are using the built in oidc plugin or a custom exec plugin. I think you are using a custom exec plugin but its also trying to write tokens to the file (which it should not do)? /triage needs-information
Thanks for looking into it . You are indeed correct in your assumption that I'm using a which stores the token in the after retrieval. I'd love to learn why this should not be done, but the problem remains also without persisting the token after retrieval.

Basically the steps to reproduce could be reduced to these: containing an expired token. same configuring an exec plugin which retrieves a valid token out of band and echoes it. Expected behavior: The valid token retrieved is used for the subsequent authentication. Possibly expected behavior: The exec plugin flow not started (or exited with a warning) due to an expired token being present in the . Current behavior seems to acknowledge the invalid token and hence tries to retrieve a new, valid one - which it then discards as soon as it arrives.
Seems we can reduce it even further by removing the token altogether from the file and instead provide it directly. So the only step needed to reproduce is essentially this: configuring an exec plugin which retrieves a valid token out of band and echoes it as part of an . Now any will trigger the exec plugin to retrieve a valid token, but will still not use it for subsequent authentication.
This one is caused by config merging with clientcmd overrides: and exec.Auth not being in this switch:
Interesting, thanks Will add it to the switch along with some tests and if everything works out well I'll submit a PR next week.
Having spent some time on this in the debugger, it seems somewhat more intricate than I first thought. Turns out the exec plugin is executed whether the presented is invalid or valid - the exec action is actually called before the first call is even dispatched. This due to the exec credential plugin possibly being needed to setup the client certificate requested for the transport. Once the exec plugin's is called it assign the authenticators function to : ..which is then called when setting up the transport, thus triggering the exec plugin regardless of a token being present or not: Since the exec plugin has no way of knowing whether a token or a client certificate will be returned by the exec action, this kind of makes sense. It however defeats letting any present to override the exec plugin, which clearly seems to be the intent: Possible options here that I can think of: (and fix) behavior of letting override the exec plugin. This looks doable by adding a check for token auth early in the 's method body: But might be there are consequences to this I'm not aware of. the behavior of letting override the exec plugin being called.

third, more complex option, would be to keep the token override but dispatch to the exec credential plugin should the response from the first call with the token come back with a 401. This was what I originally believed was the intent as my first tests always included an invalid token. While this behavior seems nice I'm more and more thinking it may violate the principle of least surprise. Either you present a token and use that for auth or you let the exec plugin fetch the credentials for you. What do you guys think?
As a colleague just made me aware of, certain tools like kubetail seem to have kubectl call the exec plugin so many times that eventually the threshold of 1000 is reached and warn logs (exec:201) start spewing - even though no certificate (and hence no rotation) should be involved. This has me more convinced that the first option is the right one, skipping the client TLS ceremonies should a token be present for authentication.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: os.Remove(deviceMountPath) return err } <ins> glog.V(4).Infof("formatting spec %v devicePath %v deviceMountPath %v fs %v with options %+v", spec.Name(), devicePath, deviceMountPath, volumeSource.FSType, options) </ins> } return nil }

Eg: Suspect it's because of:
If this hasn't been seen in 3 months, okay to close?
Yes we fixed it with


Question: // if you just want remove specific provider, call func RemovePredicateKeyFromAlgoProvider() factory.RemovePredicateKeyFromAlgorithmProviderMap(predicates.CheckNodeConditionPred) <ins> // Fit is determined based on whether a node has Unschedulable spec factory.RegisterMandatoryFitPredicate(predicates.CheckNodeUnschedulablePred, predicates.CheckNodeUnschedulablePredicate) </ins> // Fit is determined based on whether a pod can tolerate all of the node's taints factory.RegisterMandatoryFitPredicate(predicates.PodToleratesNodeTaintsPred, predicates.PodToleratesNodeTaints) <del> // Insert Key "PodToleratesNodeTaints" To All Algorithm Provider </del> <ins> // Insert Key "PodToleratesNodeTaints" and "CheckNodeUnschedulable" To All Algorithm Provider </ins> // The key will insert to all providers which in algorithmProviderMap[] // if you just want insert to specific provider, call func InsertPredicateKeyToAlgoProvider() factory.InsertPredicateKeyToAlgorithmProviderMap(predicates.PodToleratesNodeTaintsPred) <ins> factory.InsertPredicateKeyToAlgorithmProviderMap(predicates.CheckNodeUnschedulablePred) </ins> glog.Warningf("TaintNodesByCondition is enabled, PodToleratesNodeTaints predicate is mandatory") }

DaemonSet should not respect node's field. It's because cluster operators often want system components to be running and ready before workloads run on a given node. OTOH, DaemonSet does respect NoSchedule taints. This behavior is broken by , because DaemonSet share some predicates code with default scheduler.
/assign
Added this to 1.10 milestone, because it's a regression introduced in 1.10.
can you (or a SIG lead) add status/approved-for-milestone please? I'll add the priority. /priority important-soon
how is this use case satisfied by the daemonset controller creating pods and depending on the scheduler to assign them to nodes? having the scheduler ignore the unschedulable field seems nonsensical
I'd like get this as limitation for alpha feature; when upgrade to beta, I'd like to use taint to replace un-scheduable, so DaemonSet pod can tolerate it on-demand.
Escalating to critical-urgent, since we are now in Code Freeze. Please update daily with status, thanks! /priority critical-urgent /remove-priority important-soon
In short term, scheduler can check of pods and ignore bit of nodes for DS pods. Long term, when TaintNodesByCondition is stable, we should replace bit by taint and have DS controller add toleration for the taint when creating pods.
That doesn't seem proper. Making pods behave differently depending on the type of their creating object is very counterintuitive. That seems more reasonable. I'm less certain about auto- tolerations
- indeed there's some work currently to move from to a taint based behaviour, tracked by


Question: results[i][index], err = priorityConfigs[i].Map(pod, meta, nodeInfo) if err != nil { appendError(err) <del> return </del> <ins> results[i][index].Host = nodes[index].Name </ins> } } }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: A failed Prioritize Map Function caused Scheduler break down. What you expected to happen: Pod remains Pending and describe pod will show errors of failed functions. How to reproduce it (as minimally and precisely as possible): It happens when some prioritize's map functions return with error. Anything else we need to know?: Environment: Kubernetes version (use ): v1.9.6 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
As shown above, if priorityConfigs[i].Map(pod, meta, nodeInfo) returns with error, there will be no chance for results[i + 1] to initialize its value, thus causing panic when calling config.Reduce(pod, meta, nodeNameToInfo, results[i+1])
/sig scheduling
I opened a PR to initialize results with real host name instead of empty string.


Question: expected: []*v1.Node{nodes[6], nodes[5], nodes[2], nodes[0]}, }, { <del> name: "Remove non-existing node", operations: []operation{ addNode(0), addNode(1), updateSnapshot(), }, expected: []*v1.Node{nodes[1], nodes[0]}, }, { </del> name: "Update some nodes", operations: []operation{ addNode(0), addNode(1), addNode(5), updateSnapshot(), updateNode(1),

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: If we delete node first, then remove pods on this node, the scheduler will crash in function What you expected to happen: Not crashed How to reproduce it (as minimally and precisely as possible): some pods on node A delete node A delete pods on node A Anything else we need to know?: Environment: Kubernetes version (use ): 1.16-1.18, even master Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig scheduling
can you show the core dump please of this crash?
/assign
I patched the unit test shown in into the , and branches. None crashed. I'll await for the crash dump.
Here is the fresh dump.
and which version was this?
This is master branch. And I've tested it from 1.16-1.18, it can be reproduced easily.
Thanks a lot. I was able to finally reproduce in unit tests. The scenario is: a node with at least 2 pods. node. pod1. In unit test, this is first an update to set the termination timestamp. No issues up to this point. pod2. Again, first an update to set the termination timestamp. Here it crashes. I'm working on a fix.
It looks like this is not an issue in 1.17 and below, because we preserved a placeholder node where the operations occur. do you have stacktraces for 1.17 and 1.16?
Yes, it seems only be found at 1.18. :smile:


Question: sst.WaitForRunningAndReady(*ss.Spec.Replicas, ss) }) <ins> // This can't be Conformance yet because it depends on a default // StorageClass and a dynamic provisioner. It("should perform rolling updates and roll backs of template modifications with PVCs", func() { By("Creating a new StatefulSet with PVCs") *(ss.Spec.Replicas) = 3 rollbackTest(c, ns, ss) }) </ins> /* Release : v1.9 Testname: StatefulSet, Rolling Update

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: It would be good to test rolling updates with RWO PVCs that cannot be attached to multiple nodes at the same time.
There have been reports of this scenario


Question: fp.exec = &fexec svcIP := "10.20.30.41" <ins> extIP := "1.1.1.1" lbIngressIP := "2.2.2.2" </ins> svcPort := 80 nodePort := 31201 svcPortName := proxy.ServicePortName{

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --UDP traffic to a Pod behind a LoadBalancer Service exposing a UDP listener service is blackholed when the replacement Pod is scheduled on the same node as the "old" Pod. A new Pod that gets provisioned (replacing the previous one) will receive traffic once it's ready. a UDP Listener Pod (single pod) and using nodeSelector ensure it runs on a specific node the Pod behind a LoadBalancer Service continuous stream of UDP packets from outside the cluster to the External IP of the Service that traffic is correctly received the Pod and observe as a new one is created, traffic should now be blackholed and not reaching the "new" Pod After the new Pod is recreated, we noticed the following entries on the node: notice that the <SVC EXT IPappears here both in the original and the NATed , this is wrong and I suspect happens because the conntrack for the old Pod were flushed before the iptables rules for the new Pod were created (?) and in the interim new packets have reached the node and resulted in these incorrect conntrack entries. To "fix" this, we manually deleted the above conntrack entries, and this allowed new correct one to be established based on the now existing iptables nat rule: with this new conntrack entries, the traffic started flowing as expected to the new Pod and notice that the NATed now correctly shows the Pod IP (172.30.3.219) instead of the Service External IP. While spelunking through the GitHub issues and PRs, I found this issue which was addressed in , it does mention that: the service has endpoints available. Since the fix in this PR refers to NodePort, perhaps the same also applies to External IP (i.e. LoadBalancer Service?) or there's something else at play here? AFAICT, this fix is included in v1.20.7 that we are running. Kubernetes version (use ):Cloud provider or hardware configuration: GCP/GKE OS (e.g: ):Kernel (e.g.

): Install tools: Terraform (with google provider) Network plugin and version (if this is a network-related bug): Others:
/sig network
According to GCP support (since we hit this on a GKE cluster) there's already a fix for this bug and it in v1.21, however I'm unable to find to link it here (hope they can provide the link shortly).
GCP support referenced as the fix for this issue, however this was already merged and released in v1.20.4 but we are running v1.20.7, so should have the fix already, so the status is unclear at this point.
thanks for the detailed report, at first sight we do clean the external ips is it possible to set kube-proxy in a log level of 2 and capture the logs from the node to correlate with the time it happends, and understand why those entries are not being deleted?
Hey we can try that, however since this is a GKE environment I'd have to check how/if we can change kube-proxy log level. My understanding of the issue is that it's not that the conntrack entries are flushed, is that they are flushed before the nat iptables rules are created, causing faulty conntrack entries to be created in the interim (since the udp packets are incoming all the time).
I'm having some miscommunication about the availability of fix in with GCP/GKE support and was hoping you could confirm that is included in v1.20.5 (as per the release notes there? ) and per the cherry-pick PR GCP Support claim that this change is incorporated in v1.21 and we should wait for v1.21 to be available in GKE.
I see the commit in 1.20.5 , this is the cherry-pick The conntrack entries are always deleted after the conntrack entries are created, there is some code that detects that a service goes stale and based on that it deletes the entries.

I think that current code detects this case, at least I know it works when the service goes from 0 to 1 endpoint, that is why the logs will be useful, we can see exactly what happens ...
Thanks for confirming I just checked and seem to be already running in log level 2: I looked back at the logs when this issue was identified and I see the following entries: This two messages and only these two repeat every time we tried to rollout the deployment. is indeed the and is the ClusterIP of the service: but every time we tried, we ended up with these stale conntrack entries I mentioned in the OP and only after deleting them would the udp packet start flowing into the "new" pod.
oh wait, by external IP you mean ?
/triage accepted /assign
here you go, having good reports on bugs make them easy to spot ;)
indeed! I apologize if I wasn't clear indeed I was referring to the LoadBalancer Service External IP. Thanks for fix!!! But now I'm a bit confused, since I understood that LoadBalancer Service External IP == LoadBalancer Ingress IP, but I see these are two separate attributes? so for example: These are the same IP, but internally they are separate attributes?
hmm I don't know how kubectl prints the information, but you can check the fields with and you could see that ExternalIPs and LoadBalancer.IngressIP are different fields, actually one is in the Status and other in the Spec
interesting, so in the output, I can only see the , there's actually no mention of here: That's why I always assumed that this is referring to the same thing/attribute/field.


Question: "k8s.io/client-go/discovery" "k8s.io/client-go/dynamic" restclient "k8s.io/client-go/rest" <ins> "k8s.io/client-go/tools/cache" </ins> "k8s.io/kubernetes/pkg/api" "k8s.io/kubernetes/pkg/api/v1" "k8s.io/kubernetes/pkg/client/clientset_generated/clientset"

https://k8s- Failed: TestBlockingOwnerRefDoesBlock {garbagecollector}
cc:
https://k8s- Failed: TestBlockingOwnerRefDoesBlock {garbagecollector}
I don't have a good theory yet. It's occurring , so I'll fix the more urgent issues first.
If its that low, its not a blocker. Moving to 1.6.1
Could this be a race condition? Perhaps needs some retry logic? Note that the creation never reported an error that would be fatal to the test:
I checked the log, it appeared when GC was processing the foreground deletion of the owner RC, GC didn't find any dependents of RC. This probably is the same race we expected in no.1. To fix the test, the test can wait some seconds before sending the deletion request, so that the garbage collector will have time to update its dependency graph.


Question: // Create a pod in one node to get evicted ginkgo.By("creating a client pod that is going to be evicted for the service " + serviceName) evictedPod := e2epod.NewAgnhostPod(namespace, "evicted-pod", nil, nil, nil) <del> evictedPod.Spec.Containers[0].Command = []string{"/bin/sh", "-c", "sleep 10; fallocate -l 10M file; sleep 10000"} </del> <ins> evictedPod.Spec.Containers[0].Command = []string{"/bin/sh", "-c", "sleep 10; dd if=/dev/zero of=file bs=1M count=10; sleep 10000"} </ins> evictedPod.Spec.Containers[0].Name = "evicted-pod" evictedPod.Spec.Containers[0].Resources = v1.ResourceRequirements{ Limits: v1.ResourceList{"ephemeral-storage": resource.MustParse("5Mi")},

in this test the rootfs is limited to 5M and the pod is evicted by creating a 10M file with; But this fails if the fs doesn't support the fallocate syscall. The K8s e2e must not itself require support for features that K8s does not require. I am unsure why my overlay fs in containers doesn't support fallocate(2) while the fs on the node (ext4) does. I have queried on the slack. I build my Linux kernel myself but I can't find anything related. If the command is changed to; the test works on an fs that doesn't support fallocate(2). It may seem more "primitive" but it doesn't require a feature that K8s doesn't. <detailsNone </detailsNo response No response
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig testing
Interesting /Triage accepted


Question: // the job stabilized and won't be synced until modification or full // resync happens, we don't want to wait for the latter so we force // sync modifying it <del> job.Spec.Parallelism = &completions job, err = updateJob(f.ClientSet, f.Namespace.Name, job) </del> <ins> _, err = framework.UpdateJobWithRetries(f.ClientSet, f.Namespace.Name, job.Name, func(update *batch.Job) { update.Spec.Parallelism = &completions }) </ins> Expect(err).NotTo(HaveOccurred()) err = waitForJobFail(f.ClientSet, f.Namespace.Name, job.Name, jobTimeout) }

https://k8s- Failed: [] Job should fail a job {Kubernetes e2e suite} Happened on a presubmit run in . Previous issues for this test:
https://k8s- Failed: [] Job should fail a job {Kubernetes e2e suite}
https://k8s- Failed: [] Job should fail a job {Kubernetes e2e suite} Happened on a presubmit run in .
https://k8s- Failed: [] Job should fail a job {Kubernetes e2e suite}
https://k8s- Failed: [] Job should fail a job {Kubernetes e2e suite}
https://k8s- Failed: [] Job should fail a job {Kubernetes e2e suite}
https://k8s- Failed: [] Job should fail a job {Kubernetes e2e suite}
https://k8s- Failed: [] Job should fail a job {Kubernetes e2e suite}
https://k8s- Failed: [] Job should fail a job {Kubernetes e2e suite}
https://k8s- Failed: [] Job should fail a job {Kubernetes e2e suite}
https://k8s- Failed: [] Job should fail a job {Kubernetes e2e suite}
https://k8s- Failed: [] Job should fail a job {Kubernetes e2e suite}
https://k8s- Failed: [] Job should fail a job {Kubernetes e2e suite}
https://k8s- Failed: [] Job should fail a job {Kubernetes e2e suite}


Question: if selfVersion.LTE(version) { r.isDeprecated = true } <ins> if ShouldShowHidden() { klog.Warningf("Hidden metrics have been manually overridden, showing this very deprecated metric.") return } </ins> if selfVersion.LT(version) { klog.Warningf("This metric has been deprecated for more than one release, hiding.") r.isHidden = true

The metrics framework requires the ability to optionally display hidden (deprecated) metrics. A boolean needs to be piped through the registry (possibly at initialization or possibly set later) which then needs to be passed to the Create function so that we can determine whether we need to actually create the metric. /sig instrumentation


Question: } } <del> func makeDevice(devOnNUMA checkpoint.DevicesPerNUMA) map[string]pluginapi.Device { </del> <ins> func makeDevice(devOnNUMA checkpoint.DevicesPerNUMA, topology bool) map[string]pluginapi.Device { </ins> res := make(map[string]pluginapi.Device) <ins> var topologyInfo *pluginapi.TopologyInfo </ins> for node, devs := range devOnNUMA { <ins> if topology { topologyInfo = &pluginapi.TopologyInfo{Nodes: []*pluginapi.NUMANode{{ID: node}}} } else { topologyInfo = nil } </ins> for idx := range devs { <del> res[devs[idx]] = pluginapi.Device{ID: devs[idx], Topology: &pluginapi.TopologyInfo{Nodes: []*pluginapi.NUMANode{{ID: node}}}} </del> <ins> res[devs[idx]] = pluginapi.Device{ID: devs[idx], Topology: topologyInfo} </ins> } } return res

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: Which test(s) are failing: Since when has it been failing: Testgrid link: Reason for failure: Anything else we need to know: Can't find this in Triage Example spyglass links: /sig scheduling /cc /priority important-soon
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I don't quite think this test should be owned by SIG-Scheduling. Most of the failures happen in the Nvidia GPU setup phase. BTW: although irrelevant, last time we see a failure of this test is .
Searching keyword "UnexpectedAdmissionError" gets the following occurrences:
/sig node cc in case you're aware of any recent changes that may have caused this failure.
From the I can see a bunch of: Checking the kubelet log, I can see the gpu capacity has been changing And before the capacity changed to 0, we can see the kubelet was attempting to allocate 1 GPU to the test pod, but some NPE was thrown, and kubelet stopped/restarted.
The nil pointer is from here: , which is a new change introduced by
The current device plugin (the one defined ) is using an old version of device plugin api (see ), which does not have field under . So calling will end up with NPE.
Thanks , the analysis makes great sense to me. Could you please work on a fix to address this? (Probably updating )
Hi , I am still looking at the code, because the code does have the logic to handle the case where device plugin does not return affinity: Also, even if the Topology is not defined, I am not sure will throw NPE () (please correct me if I am wrong here). I will need to look more closely at the code to find out where the NPE is from. Updating the might take long time because we will need some testing to make sure it doesn't break anything.

Currently I am thinking of identifying the source of NPE, and make sure can handle that case.
/cc
I submitted a fix. My fault, it was a real case when pointer to Topology could be a nil. But still fails: I don't observe SIGSEGV now in , but 3 problems there: 1113 09:26:19. 1336 ] failed to run Kubelet: cannot create certificate signing request: is forbidden: User "kubelet" cannot create resource "certificatesigningrequests" in API group "" at the cluster scope of nvidia-driver-installer stops and restarts after a while. 's () NoopManager is using
can you show full error msg of "NVIDIA GPU metrics will not be available: %s"?
I1113 15:43:26. 1338 ] Found device with vendorID "0x10de" W1113 15:43:26. 1338 ] NVIDIA GPU metrics will not be available: Could not initialize NVML: could not load NVML library I1113 15:43:26. 1338 ] Machine: {Timestamp:2020-11-13 15:43:26. +0000 UTC m=+5. NumCores:2 NumPhysicalCores:1 NumSockets:1 CpuFrequency: MemoryCapacity: MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:2048 NumPages:0}] MachineID: SystemUUID:-9824-3a64-f41f- BootID:-f891-4c0b-a34d- Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:39 Capacity: Type:vfs Inodes: HasInodes:true} {Device:/dev/sda8 DeviceMajor:8 DeviceMinor:8 Capacity: Type:vfs Inodes:4096 HasInodes:true} {Device:/dev/sda1 DeviceMajor:8 DeviceMinor:1 Capacity: Type:vfs Inodes: HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:19 Capacity: Type:vfs Inodes: HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:21 Capacity: Type:vfs Inodes: HasInodes:true}

{Device:/sys/fs/cgroup DeviceMajor:0 DeviceMinor:22 Capacity: Type:vfs Inodes: HasInodes:true} {Device:/etc/machine-id DeviceMajor:0 DeviceMinor:21 Capacity: Type:vfs Inodes: HasInodes:true} {Device:/mnt/disks DeviceMajor:0 DeviceMinor:38 Capacity: Type:vfs Inodes: HasInodes:true} {Device:overlayfs0-43 DeviceMajor:0 DeviceMinor:43 Capacity: Type:vfs Inodes: HasInodes:true} {Device:/var/lib/cloud DeviceMajor:0 DeviceMinor:45 Capacity: Type:vfs Inodes: HasInodes:true} {Device:/dev/root DeviceMajor:253 DeviceMinor:0 Capacity: Type:vfs Inodes: HasInodes:true}] DiskMap:map[253:0:{Name:dm-0 Major:253 Minor:0 Size: Scheduler:none} 8:0:{Name:sda Major:8 Minor:0 Size: Scheduler:mq-deadline} 9:0:{Name:md0 Major:9 Minor:0 Size:0 Scheduler:none}] NetworkDevices:[{Name:eth0 MacAddress:42:01:0a:28:00:03 Speed:-1 Mtu:1460}] Topology:[{Id:0 Memory: HugePages:[{PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 1] Caches:[{Size: Type:Data Level:1} {Size: Type:Instruction Level:1} {Size: Type:Unified Level:2}] SocketID:0}] Caches:[{Size: Type:Unified Level:3}]}] CloudProvider:GCE InstanceType:n1-standard-2 InstanceID:} I1113 15:43:26. 1338 ] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. I1113 15:43:26. 1338 ] Version: {KernelVersion:5.4.49+ ContainerOsVersion:Container-Optimized OS from Google DockerVersion:Unknown DockerAPIVersion:Unknown CadvisorVersion: CadvisorRevision:} I guess kubelet can't load in cAdvisor the NVML library.
I have looked into this issue actually.

This is caused by a recent change on cadvisor: We used to return a real nvidiaManager but now only a Noop one if the NVML lib is not there. And currently the NVML lib is loaded by daemonset, which will not there when the kubelet starts. This PR should fix it: But btw, this bug will not fail the test because it will only cause missing GPU usage metrics. The plan is to cherry-pick the fix above to cadvisor 0.37, and then update the vendor for kubernetes


Question: return "", err } <del> if len(volumeSource.TargetWWNs) != 0 { </del> <ins> // API server validates these parameters beforehand but attach/detach // controller creates volumespec without validation. They may be nil // or zero length. We should check again to avoid unexpected conditions. if len(volumeSource.TargetWWNs) != 0 && volumeSource.Lun != nil { </ins> // TargetWWNs are the FibreChannel target worldwide names <del> return fmt.Sprintf("%v", volumeSource.TargetWWNs), nil </del> <ins> return fmt.Sprintf("%v:%v", volumeSource.TargetWWNs, *volumeSource.Lun), nil </ins> } else if len(volumeSource.WWIDs) != 0 { // WWIDs are the FibreChannel World Wide Identifiers return fmt.Sprintf("%v", volumeSource.WWIDs), nil

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: FC plugin returns TargetWWNs at GetVolumeName(), but TargetWWNs are not unique identifier if user has multiple volumes from same target wwn. As a result, after introducing attach/detach functions, FC plugin always fails to attach volumes if a pod requests 2+ luns. The GetVolumeName() is called from GetUniqueVolumeNameFromSpec() during Pod creation. FC plugin should return unique identifier in the GetVolumeName(). If user uses WWID to specify FC volume, this problem doesn't happen because the WWID is unique identifier. What you expected to happen: All FC volumes are attached to pod. How to reproduce it (as minimally and precisely as possible): Create two PVs and PVCs which uses same target wwn. Create Pod with these two PVCs Anything else we need to know?: Environment: Kubernetes version (use ):Install tools: local-up- /cc
/sig storage
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals


Question: } else { writeLine(proxier.filterChains, utiliptables.MakeChainLine(kubeServicesChain)) } <ins> if chain, ok := existingFilterChains[kubeForwardChain]; ok { writeLine(proxier.filterChains, chain) } else { writeLine(proxier.filterChains, utiliptables.MakeChainLine(kubeForwardChain)) } </ins> if chain, ok := existingNATChains[kubeServicesChain]; ok { writeLine(proxier.natChains, chain) } else {

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): 1.5.1 Environment: Cloud provider or hardware configuration: 3x Ubuntu 16.04 nodes running in VMware vSphere OS (e.g. from /etc/os-release): Ubuntu 16.04 Kernel (e.g. ): 4.4.0-57-generic Install tools: kubeadm Others: CNI: Canal (flannel + calico) What happened: I originally opened to make Canal CNI work when the node's FORWARD chain default policy is set to DROP. However, I realized that the DROP policy was the root cause for why my other CNI tests also were failing. When either weave and flannel CNI is used and kube-proxy has --masquerade-all enabled, pod to pod and pod to service communication fails. Specifically, kubernetes-dashboard and tiller-deploy fail to reach the kube-apiserver and enter a crash loop. Also, the kube-dns pod also crashes. <details<summarykubectl get pods --all-namespaces -o wide</summary</details<details<summarykubectl logs --namespace=kube-system kubernetes-dashboard--5vzdq</summary</details<details<summarykubectl describe --namespace=kube-system pod kubernetes-dashboard--5vzdq</summary</details<details<summarykubectl describe --namespace=kube-system pod kube-dns--gz34l</summary</details<details<summarysudo iptables -t filter -v -L FORWARD --line-numbers (197 currently has the dashboard pod scheduled. Note the 2 dropped packets.

)</summary</details</brWhat you expected to happen: Either it should be better documented that the default FORWARD chain policy should be ACCEPT or rules should be to prevent packets from hitting the default policy. As a note, Ubuntu 16.04 defaults the FORWARD chain to DROP. I assume the is why the packets are going through the FORWARD chain in the first place but without , the dashboard still can not access the kube-apiserver. One option that I suggested in is to have kube-proxy add an iptables FORWARD rule to ACCEPT packets that have been marked by KUBE-MARK-MASQ. How to reproduce it (as minimally and precisely as possible): Create a cluster with: Weave or Flannel CNI enabled on all nodes kube-proxy set to kubernetes-dashboard enabled. kubernetes-dashboard will crash often. Anything else do we need to know: is closed because I thought the issue was specific to Canal and instead opened However, this looks to be a larger issue that kubernetes itself should address.
Weave issue:
I think this is going to hit a lot of people as Docker 1.13 is now setting the FORWARD chain to drop by default if ip_forward is set. See
See also
overlap with ?
I think a solution for this issue or (which is closer to what I feel like a proper solution is) should be considered for inclusion for K8s 1.8 since it will officially support docker 1.13. The problem as I see it is that kube-proxy sets up rules to forward traffic for services (specifically I have seen the problem with NodePorts) yet does not ensure or have any requirement (to my knowledge) that the forwarded traffic be allowed when the traffic is forwarded to a host other than the one receiving the traffic. I observed this when using K8s v1.7.5 installed with kubeadm v1.7.5 and Calico v2.5. It can be remedied with the addition of but hopefully there is a better answer than doing that manually and hopefully something more specific to what is needed. I saw the comment on the Docker 1.13 validation issue linking to the configuration script used in the GCE docker images which is why the e2es are not having any issues.
If we can be more precise about which traffic we require to ALLOW, I'm OK with adding rules to open that.

I've been doing some testing with the first two options and chatted with and about these potential options for adding some forwarding rules. They all are a certainly lot more specific than the previous blanket all ALLOW. any traffic sourced or destined for the clusterCIDR. traffic that has the KUBE-MARK-MASQ set and conntrack rules to accept the follow on traffic. a third option of adding a more specific mark that is only set on specific traffic that it is known will need to be forwarded and match on that (this option would probably need the conntrack rules like the above option too) do any of those options appeal to you or do you see any problems with them?
If we add some more rules, we shouldn't put them in the top-level chain since it makes them hard to manage/clean up. Best to add them to a chain that kube-proxy owns that is jumped to from the top-level chain.
doesn't necessarily know the ClusterCIDR. I could see your proposal making sense in conjunction with .
Very good point I should have mentioned that the options I presented are assuming clusterCIDR is set. If clusterCIDR is not defined then the first option is not possible (except for the wide open option that these rules are attempting to avoid). I guess on options 2 & 3 if it was not set then a conntrack rule could be without the clusterCIDR. I would appreciate feedback on the desired behavior if clusterCIDR is not set too.
The PR that was merged solved the issue for pods inside clusterCIDR. But it doesn't handle hostNetwork pods that use the node IP, like when you want to expose the API server as a load-balanced service externally. I'm not sure why the “kubernetes forwarding conntrack pod source rule” has to be limited to the pod CIDR, since --ctstate RELATED,ESTABLISHED should already ensure it does not allow traffic that wasn't already allowed by another rule.
I'd argue that any iptables/firewalling for a pod using isn't the responsibility of Kubernetes, There's no reasonable way for k8s to actually know what interfaces a host-network container binds to, what ports it listens on, or what expectations it has. The cluster operator should probably be responsible for that networking case.
I meant explicitly regarding services where those pods are the endpoints of a service, so Kubernetes does know the address and the port.

Due to limitation of state RELATED,ESTABLISHED in FORWARD rule i.e. current implementation, the unidirectional UDP traffic apps won't work. Because packets (except first) aren't ESTABILISHED from iptables's perspective (no reply from endpoint, no handshake). Shouldn't this be something like and not
To add to comment, bidirectional UDP apps are brittle as well. Dropping or failing to reply to the first packet will prevent the app from ever receiving a second. More discussion on


Question: import ( "fmt" "net" <ins> "runtime" </ins> "strconv" "strings"

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: When running , and passing the flag which was introduced in , the following error occurs: What you expected to happen: The proxy should work as it used to, supporting this mode. How to reproduce it (as minimally and precisely as possible): ./kube- --proxy-mode=kernelspace Anything else we need to know?: This was introduced after the validation was refactored in which regressed validation for . IPVS validation (and thus support) was later , but we need another mode for Windows. Environment: Kubernetes version (use ): v1.10.0-alpha.0.684+ Cloud provider or hardware configuration: n/a OS (e.g. from ): Windows Server Core 1709 Kernel (e.g. ): Install tools: Others:
/sig windows /sig network
Tentatively adding milestone so we can track getting consensus on release-blocking status.
/priority critical-urgent
[MILESTONENOTIFIER] Milestone Issue Needs Approval Action required: This issue must have the label applied by a SIG maintainer. <details<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details