Question: fs.BoolVar(&o.CleanupAndExit, "cleanup-iptables", o.CleanupAndExit, "If true cleanup iptables and ipvs rules and exit.") fs.MarkDeprecated("cleanup-iptables", "This flag is replaced by --cleanup.") fs.BoolVar(&o.CleanupAndExit, "cleanup", o.CleanupAndExit, "If true cleanup iptables and ipvs rules and exit.") <del> fs.BoolVar(&o.CleanupIPVS, "cleanup-ipvs", o.CleanupIPVS, "If true make kube-proxy cleanup ipvs rules before running.  Default is true") </del> <ins> fs.BoolVar(&o.CleanupIPVS, "cleanup-ipvs", o.CleanupIPVS, "If true and --cleanup is specified, kube-proxy will also flush IPVS rules, in addition to normal cleanup.") </ins> // All flags below here are deprecated and will eventually be removed.

What would you like to be : Remove any logic in kube-proxy that tries to automatically clean up proxy rules based on the proxy mode. Only attempt to clean up if is set. Why is this needed: This is a continuation of a discussion from There's a lot of complexity that comes with switching the proxy mode on kube-proxy because it tries to delete proxy rules created from other proxy modes. This has a lot of side-effects and we should be encouraging users to reboot thier nodes along with switching proxy modes. One example of this can be found
/sig network
I can take this on - does this issue have/need a KEP?
I think not but will defer to
I think a micro-KEP is appropriate. Specifically to talk about compatibility with regards to flags and to talk about docs (not sure which, but we have to write this down :)
also to talk about --cleanup (I think it should probably clean up its own mode, not the others -- or maybe I am wrong, please think about the right thing here :)
/assign (Yay, I can do this now!)
/priority important-soon

What happened: When restarting kube-proxy in iptables mode, there will be several seconds of timeouts because it flushes the KUBE-SERVICES nat chain and several others. What you expected to happen: Restarting kube-proxy should not impact traffic in any way. How to reproduce it (as minimally and precisely as possible): an HTTP server behind a ClusterIP service an http client in a loop making requests to the HTTP server -15 the kube-proxy process on the client node timeout for a few seconds until kube-proxy re-syncs the iptables rules Anything else we need to know?: Running with makes it pretty clear what's happening: it says it's cleaning up inactive rules, but these are crucial to the iptables proxier. You can trivially reproduce by just running: Environment: Kubernetes version (use ): 1.10.11 Cloud provider or hardware configuration: amazon OS (e.g: ): custom Kernel (e.g. ): 4.14.77-hs623.el6.x86_64 Install tools: custom Others: iptables 1.6.2 We're running kube-proxy 1.10.11, but I've confirmed this issue with kube-proxy 1.13.4 too
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Possibly a bug in the proxy. It flushes iptabes if the proxier is able to use IPVS which seems like the wrong behavior to me if there are shared tables/chains between the IPVS and iptables proxier. One workaround I can think of right now is to unload the ipvs kernel module to skip this check.
/assign
oh, that's an interesting workaround, I'll give that a shot, thanks!
np, I'll try to work on the bug fix in the meantime!
/triage unresolved
note that unloading the ipvs module is not actually enough of a workaround because kube-proxy will load the kernel module if it sees that it is available. If I unload the ipvs module and then rename to something else in then it does the right thing. Dug slightly further: the way it probes for modules is by actually running modprobe.

So the simplest hack that allows us to keep ipvs loaded for other things on the system is to put a modprobe script on its PATH that just always exits 1 for the ip_vs modules. (We can stop mounting the modules dir into the kube-proxy container in our kubernetes clusters, but we also run kube-proxy on non-kubernetes nodes via the init system, so the PATH hack works well enough there). This should hold us over fine until we get to a version with your fix in it.
Good to know, thanks for sharing!
/assign
Quick update on this issue from today's SIG Network call: we're going to try to get rid of the automatic proxy clean up altogether for v1.14.1 since this is considered a bug. is working on the KEP & implementation.


Question: valid := instanceType.getEBSMountDevices() chosen := mountDevice("") for _, mountDevice := range valid { <del> _, found := self.deviceMappings[mountDevice] </del> <ins> _, found := deviceMappings[mountDevice] </ins> if !found { chosen = mountDevice break

Environment: CoreOS 815.0.0, k8s 1.1.0 () It happens to me quite often that a volume marked as "available" on AWS remains in a "detached" state according to the kubelet: Deleting the pod and recreating it eventually succeeds.
I have some fixes in progress that should fix this
/cc
I can reliably reproduce similar issue, I am not sure it's the same: create a AWS EBS, let's say it has ID Create a pod that uses this EBS: Create the pod. It succeeds eventually: Delete the pod Create the pod again. It will be forever IMO, it's caused by not removing the detached volume from internal map of attached volumes . Subsequent will find it there and it won't even try to attach it again. Patch is available in referenced PR.
can you please confirm that your bugs is the same as I can reproduce? Or do you see different bug, only with similar symptoms?
It is
Was this fixed by ?
I think you referenced the wrong issue / PR, didn't you? As far as I know this is not fixed yet.
Yes, I meant to link . mentioned there that it might fix this issue.
Oh right! It made it to v1.2.0-alpha.4
did not fix this issue (but fixed the DeviceMapping cache invalidation)
I think, what is happening is that you run into an error or timeout while waiting for the disk operation. That leaves the devicecache yet again in an inconsistent state. I have a fix here that just discards the cache on any error. Additionally, it is totally possible that there is concurrent operations which lead to an error and bork the cache. The fix: Unfortunately, this is in our tests still not 100% rock solid. Especially in timeout cases. I'm missing the capability to properly test this to engage in a bigger refactoring of this whole issue. A workaround to actually fixing it properly would be to add a TTL to the cache, so that it eventually starts working again.
Just for information for people who might use the GitHub search, as of Kubernetes 1.1.7 the errors logged by the look like: Deleting the pod or restarting doesn't help, the instance keeps failing attaching the EBS with the same errors if the pod get rescheduled on the same host.

thanks for the fix!
Oh... the same host. I bet I messed up that case... I'm going to move into 1.2 so I take another look at this!
What's the status on this?
I just don't think this is fixable with the volume cache we have. I proposed a PR to remove the cache, and replace it with a map of active attachment operations in progress. A little less efficient, but a lot more correct (presuming passes testing).
How to check what K8 version has this fix?
Is there any progress on this issue? I'm running K8s version v1.5.2, and I'm having similar issues. awsElasticBlockStore volumes are virtually unusable due to one of three errors I'm getting: My volume claim is ReadWriteOnce, so this next error confuses me: And finally: [SchedulerPredicates failed due to persistentvolumeclaims "mysql" not found, which is unexpected., SchedulerPredicates failed due to persistentvolumeclaims "mysql" not found, which is unexpected., SchedulerPredicates failed due to persistentvolumeclaims "mysql" not found, which is unexpected.] Unable to mount volumes for pod "staging-connect-mysql-<id(<id)": timeout expired waiting for volumes to attach/mount for pod "default"/"<id". list of unattached/unmounted volumes=[mysql] Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod "default"/"<id". list of unattached/unmounted volumes=[mysql]
Also having the same issue as ^ Should we be using PV+VPC instead of claiming the EBS volume directly from the deployment spec?


Question: healthy := false switch { <del> case lastUpdated.IsZero(): </del> <ins> case oldestPendingQueued.IsZero(): </ins> // The proxy is healthy while it's starting up <del> // TODO: this makes it useless as a readinessProbe. Consider changing // to only become healthy after the proxy is fully synced. </del> <ins> // or the proxy is fully synced. </ins> healthy = true <del> case lastUpdated.After(lastQueued): // We've processed all updates healthy = true case currentTime.Sub(lastQueued) < h.hs.healthTimeout: </del> <ins> case currentTime.Sub(oldestPendingQueued) < h.hs.healthTimeout: </ins> // There's an unprocessed update queued, but it's not late yet healthy = true }

We upgraded iptables on one of our hosts, and kube-proxy began logging errors about being unable to acquire the . However, we didn't notice this, because kube-proxy's health check () was still passing (and that's generally what we look at as our sign that something is amiss.) As a result, kube-proxy ran for days without being able to update iptables rules, eventually causing an outage for one of our services. We expected that kube-proxy's health check would fail if it was not able to interface with iptables. Our system was running Debian stretch, running kube-proxy in iptables mode with systemd, with iptables version 1.6.0. Given that starting configuration, the repro case is: iptables to 1.6.2. that kube-proxy begins emitting logs saying that it cannot update iptables because another process has the . that kube-proxy's health check endpoint () does not return an error. Example health check output: <details</details<details</details<detailsNone </details<details</details<detailsThe hard way </details<detailsDocker version 19.03.8, build </details<detailsCalico v3.18.4 </details
/sig network
you upgraded iptables just one node ? what is the version of iptables on other nodes ?
does it means that is not able to never get the lock or is just that sometimes is able to get it?
Yeah we upgraded on just one node at first. Like my original bug report says, we started at 1.6.0, so all the other nodes had 1.6.0. it never gets the lock. You have to restart kube-proxy so it can go through the iptables version detection routine again and change its behavior to the 1.6.2 logic branch. That's fine, it's whatever, but the health check doesn't fail even though kube-proxy can't do its job, which is a bug in my opinion. Also yes, we checked to see if some other process had the lock. It didn't. It's something to do with changes in how iptables actually works after the upgrade to 1.6.2, specifically. I believe this launchpad issue is relevant:
the logic of the healthcheck should fail if is not able to process the updates, right?

I this to the description as well, but here is an example response during this failure mode: And here are what the logs look like: Running gives no output.
This is interesting the and values are the same. This is the healthz logic, it will be unhealthy if the difference between both is greater than the health timeout. is it possible to share the entire kube-proxy log ? I want to discard something
to save me some time, could you please tell me what sort of messages you are looking for? I would rather go search splunk for them than to set up a repro case again.
just the first 1000 lines since kube-proxy starts should be enough
The JSON is kind of meaningless these days; if the proxier thinks it's healthy, it will just write out the current time as both and . It's only when it thinks its unhealthy that it writes out the real time. See I'm not sure how this could happen...
I'm afraid I'm not at liberty to give you 1000 lines of logs without some more targeted context. Company policy and whatnot. What I will tell you is I put the proxy up to and reproduced the issue and pulled those logs, and if you tell me the kind of messages you are looking for, I can pull the (redacted) context around them. What I can give you now, even without being directed, is the more verbose version of what I posted before, i.e., what happens around the time that the error gets logged: <detailsFollowed by the whole list of iptables rules. Then </detailsI can also give you the startup arguments: <details</detailsAnd finally, I can give you these lines from the iptables code, the health check getting registered, and a couple other startup items: <details</detailsAny more than this, though, and I'll have to ask you to tell me something to grep for.
First thing to check is that the healthcheck actually works EVER. Given the same timestamp (we should fix that?) it thinks it is healthy.
/assign
/triage accepted
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
A situation: when there is a service or endpoint is constantly updated (which will update the ): so there is a high probability that the difference between the and the is less than the , so this will be considered healthy If you reduce ( iptables-sync-period * 2, the defaults value of flag is 30s), you may see response code is 503, but in fact, it is not easy to estimate how much it should be reduced. so As for case 3, I think there are something that can be optimized.
Oh! That's wrong. We don't care about the time of the most-recently queued update, we care about the time of the oldest queued update that was after the last . So rather than having and updating it every time gets called, we should have a variable something like , and can set it to 0, and should only set it to if it was previously 0. In fact, we don't even need . Kube-proxy would be healthy if or .
Thanks I see, I'd fix it, can I take this one?
sure /assign


Question: <ins> /* Copyright 2022 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package openapi import ( "context" "encoding/json" "testing" "time" "github.com/stretchr/testify/assert" apiextensionsv1 "k8s.io/apiextensions-apiserver/pkg/apis/apiextensions/v1" "k8s.io/apiextensions-apiserver/pkg/client/clientset/clientset" "k8s.io/apiextensions-apiserver/test/integration/fixtures" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" genericfeatures "k8s.io/apiserver/pkg/features" utilfeature "k8s.io/apiserver/pkg/util/feature" "k8s.io/client-go/dynamic" kubernetes "k8s.io/client-go/kubernetes" featuregatetesting "k8s.io/component-base/featuregate/testing" "k8s.io/kube-openapi/pkg/spec3" apiservertesting "k8s.io/kubernetes/cmd/kube-apiserver/app/testing" "k8s.io/kubernetes/test/integration/framework" ) const scaleSchemaName = "io.k8s.api.autoscaling.v1.Scale" const statusSchemaName = "io.k8s.apimachinery.pkg.apis.meta.v1.Status" const objectMetaSchemaName = "io.k8s.apimachinery.pkg.apis.meta.v1.ObjectMeta" func TestOpenAPIV3MultipleCRDsSameGV(t *testing.T) { defer featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, genericfeatures.OpenAPIV3, true)() server, err := apiservertesting.StartTestServer(t, apiservertesting.NewDefaultTestServerOptions(), nil, framework.SharedEtcd()) if err != nil { t.Fatal(err) } defer server.TearDownFn() config := server.ClientConfig apiExtensionClient, err := clientset.NewForConfig(config) if err != nil { t.Fatal(err) } dynamicClient, err := dynamic.NewForConfig(config) if err != nil { t.Fatal(err) } clientset, err := kubernetes.NewForConfig(config) if err != nil { t.Fatal(err) } fooWithSubresourceCRD := &apiextensionsv1.CustomResourceDefinition{ ObjectMeta: metav1.ObjectMeta{ Name: "foosubs.cr.bar.com", }, Spec: apiextensionsv1.CustomResourceDefinitionSpec{ Group: "cr.bar.com", Scope: apiextensionsv1.NamespaceScoped, Names: apiextensionsv1.CustomResourceDefinitionNames{ Plural: "foosubs", Kind:   "FooSub", }, Versions: []apiextensionsv1.CustomResourceDefinitionVersion{ { Name:    "v1", Served:  true, Storage: true, Schema: &apiextensionsv1.CustomResourceValidation{ OpenAPIV3Schema: &apiextensionsv1.JSONSchemaProps{ Type: "object", Properties: map[string]apiextensionsv1.JSONSchemaProps{ "spec": { Type: "object", Properties: map[string]apiextensionsv1.JSONSchemaProps{ "replicas": { Type: "integer", }, }, }, "status": { Type: "object", Properties: map[string]apiextensionsv1.JSONSchemaProps{ "replicas": { Type: "integer", }, }, }, }, }, }, Subresources: &apiextensionsv1.CustomResourceSubresources{ Status: &apiextensionsv1.CustomResourceSubresourceStatus{}, Scale: &apiextensionsv1.CustomResourceSubresourceScale{ SpecReplicasPath:   ".spec.replicas", StatusReplicasPath: ".status.replicas", }, }, }, }, }, } bazWithSubresourceCRD := &apiextensionsv1.CustomResourceDefinition{ ObjectMeta: metav1.ObjectMeta{ Name: "bazsubs.cr.bar.com", }, Spec: apiextensionsv1.CustomResourceDefinitionSpec{ Group: "cr.bar.com", Scope: apiextensionsv1.NamespaceScoped, Names: apiextensionsv1.CustomResourceDefinitionNames{ Plural: "bazsubs", Kind:   "BazSub", }, Versions: []apiextensionsv1.CustomResourceDefinitionVersion{ { Name:    "v1", Served:  true, Storage: true, Schema: &apiextensionsv1.CustomResourceValidation{ OpenAPIV3Schema: &apiextensionsv1.JSONSchemaProps{ Type: "object", Properties: map[string]apiextensionsv1.JSONSchemaProps{ "spec": { Type: "object", Properties: map[string]apiextensionsv1.JSONSchemaProps{ "replicas": { Type: "integer", }, }, }, "status": { Type: "object", Properties: map[string]apiextensionsv1.JSONSchemaProps{ "replicas": { Type: "integer", }, }, }, }, }, }, Subresources: &apiextensionsv1.CustomResourceSubresources{ Status: &apiextensionsv1.CustomResourceSubresourceStatus{}, Scale: &apiextensionsv1.CustomResourceSubresourceScale{ SpecReplicasPath:   ".spec.replicas", StatusReplicasPath: ".status.replicas", }, }, }, }, }, } _, err = fixtures.CreateNewV1CustomResourceDefinition(fooWithSubresourceCRD, apiExtensionClient, dynamicClient) if err != nil { t.Fatal(err) } _, err = fixtures.CreateNewV1CustomResourceDefinition(bazWithSubresourceCRD, apiExtensionClient, dynamicClient) if err != nil { t.Fatal(err) } // It takes a second for CRD specs to propagate to the aggregator time.Sleep(4 * time.Second) jsonData, err := clientset.RESTClient().Get().AbsPath("/openapi/v3/apis/cr.bar.com/v1").Do(context.TODO()).Raw() if err != nil { t.Fatal(err) } openAPISpec := spec3.OpenAPI{} err = json.Unmarshal(jsonData, &openAPISpec) if err != nil { t.Fatal(err) } // Ensure both CRD schemas are present in the OpenAPI // Scale, status, and metadata dependencies must also be present assert.Contains(t, openAPISpec.Components.Schemas, scaleSchemaName) assert.Contains(t, openAPISpec.Components.Schemas, statusSchemaName) assert.Contains(t, openAPISpec.Components.Schemas, objectMetaSchemaName) assert.Contains(t, openAPISpec.Components.Schemas, "com.bar.cr.v1.BazSub") assert.Contains(t, openAPISpec.Components.Schemas, "com.bar.cr.v1.FooSub") } </ins>

Related to Seeing the following errors in kube-apiserver logs when CRDs are applied: No errors in logs Apply any CRD manifest No response <details</details<details</details<details</details<detailsRancher RKE2 </details<detailscontainerd 1.6.4 </details<detailscanal </details
/sig api-machinery
/assign
Did this cause any specific trouble to your cluster I'm curious what the impact/severity of this is.
Everything seems to work fine, but there's a bunch of log spew every time a CRD manifest is loaded. In K3s we mix all the control-plane component logs together in a single pair of stdout/stderr streams so it's quite noisy, compared to previous releases. The fact that these come out at WARN level with a scary prefix makes them look pretty serious, and I expect we will get a fair bit of users asking about it once we release.
/triage accepted


Question: // HasStatusCause returns true if the provided error has a details cause // with the provided type name. <ins> // It supports wrapped errors and returns false when the error is nil. </ins> func HasStatusCause(err error, name metav1.CauseType) bool { _, ok := StatusCause(err, name) return ok } // StatusCause returns the named cause from the provided error if it exists and <del> // the error is of the type APIStatus. Otherwise it returns false. </del> <ins> // the error unwraps to the type APIStatus. Otherwise it returns false. </ins> func StatusCause(err error, name metav1.CauseType) (metav1.StatusCause, bool) { <del> apierr, ok := err.(APIStatus) if !ok || apierr == nil || apierr.Status().Details == nil { return metav1.StatusCause{}, false } for _, cause := range apierr.Status().Details.Causes { if cause.Type == name { return cause, true </del> <ins> status, ok := err.(APIStatus) if (ok || errors.As(err, &status)) && status.Status().Details != nil { for _, cause := range status.Status().Details.Causes { if cause.Type == name { return cause, true } </ins> } } return metav1.StatusCause{}, false

I have a controller that creates objects. It uses to notice when create fails due to a deleting namespace. This check was working well until I moved it away from each call. In some cases, the error is wrapped using causing the function to always return false. I expected to work on both wrapped and unwrapped errors, like the functions do. This short program shows the issue: I am able to work around it by first calling . This type assertion should be replaced with a call to : /sig api-machinery v1.22.5, but I also see the affected code in master. <details</detailsn/a n/a k3s n/a n/a
/cc /triage accepted
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/remove-lifecycle rotten


Question: // Periodically refresh the RESTMapper with new discovery information and sync // the garbage collector. <del> go garbageCollector.Sync(gcClientset.Discovery(), 30*time.Second, ctx.Stop) </del> <ins> go garbageCollector.Sync(discoveryClient, 30*time.Second, ctx.Stop) </ins> return garbagecollector.NewDebugHandler(garbageCollector), true, nil }

Ran kube-controller-manager with and observed this in the log every 30 seconds: This is also happening in 1.20, 1.19, and 1.18 The GC and quota clients look up discovery information every 30 seconds. They have a QPS of 20 requests per second and a burst of 30 requests. Default configuration of kube-controller-manager runs without constant throttling warnings. /sig api-machinery
/assign /triage accepted


Question: // restart may think these old pods are consuming CPU and we // will get an OutOfCpu error. ginkgo.By("verifying restartNever pods succeed and restartAlways pods stay running") <del> for start := time.Now(); time.Since(start) < startTimeout; time.Sleep(10 * time.Second) { postRestartRunningPods := waitForPods(f, numAllPods, time.Minute) if len(postRestartRunningPods) < numAllPods { framework.Failf("less pods are running after node restart, got %d but expected %d", len(postRestartRunningPods), numAllPods) } </del> <ins> postRestartRunningPods := waitForPods(f, numAllPods, startTimeout) if len(postRestartRunningPods) < numAllPods { framework.Failf("less pods are running after node restart, got %d but expected %d", len(postRestartRunningPods), numAllPods) </ins> } }) })

node-kubelet-serial Kubelet should correctly account for terminated pods after restart Since test was created, flakes 10-20% of the time. Not entirely clear why, I haven't looked. No response /assign /sig node
Introduced by , maybe we should use to replace See the log below, a pod becomes Succeed may exhaust tens of seconds
/triage accepted
Yeah I only tested this locally and not in CI, it seems that in CI the pods take much longer. Timings just need to be tweaked.
/reopen still a number of flakes since the attached PR merged, I don't think that fixed it.
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
from this link, I see all suites passed, which flakes do you mean?
I see this failure in
so this problem should have been fixed by
From gubernator: In all cases, it's timing out. I'll try upping the startTimeout.
/reopen still flaking
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Luckily the kubelet log is clear as to what's going on. From (flake ) The test should work fine if we drop the CPU request a bit, let me tweak it to make up for the possible extra 100m scheduled. I'm not sure what's causing that (aren't these tests serial? what the heck is getting scheduled) but we should be resilient to it. Also, needs to be fixed. This test will no longer detect the regression it was designed to show after that change. I will revert it and also fix the resource requirements and that should fix this test.
/milestone v1.23 need to fix this before test freeze


Question: // Storing endpoint of newly registered DRA Plugin into the map, where plugin name will be the key // all other DRA components will be able to get the actual socket of DRA plugins by its name. draPlugins.Set(pluginName, &Plugin{ <ins> conn:                    nil, </ins> endpoint:                endpoint, highestSupportedVersion: highestSupportedVersion, })

See comment for details:
/sig node
/area kubelet
/assign
The Kubernetes project currently lacks enough contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
/triage accepted


Question: case core.ResourceQuotaScopeBestEffort, core.ResourceQuotaScopeNotBestEffort, core.ResourceQuotaScopeTerminating, core.ResourceQuotaScopeNotTerminating, core.ResourceQuotaScopeCrossNamespacePodAffinity: if req.Operator != core.ScopeSelectorOpExists { allErrs = append(allErrs, field.Invalid(fldPath.Child("operator"), req.Operator, <del> "must be 'Exist' when scope is any of ResourceQuotaScopeTerminating, ResourceQuotaScopeNotTerminating, ResourceQuotaScopeBestEffort, ResourceQuotaScopeNotBestEffort or ResourceQuotaScopeCrossNamespacePodAffinity")) </del> <ins> "must be 'Exists' when scope is any of ResourceQuotaScopeTerminating, ResourceQuotaScopeNotTerminating, ResourceQuotaScopeBestEffort, ResourceQuotaScopeNotBestEffort or ResourceQuotaScopeCrossNamespacePodAffinity")) </ins> } }

On the document, the operator should be 'Exists' ! But in the Error message, it becomes 'Exist' ! the error message should be create a yaml and apply it It's my first time, I am not sure should we change to ? <details</details<detailsK3S </details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig apps
/assign
/assign


Question: #TODO(mikedanese): allow static pods to start before creating a client #flags+=" --bootstrap-kubeconfig=/var/lib/kubelet/bootstrap-kubeconfig" #flags+=" --kubeconfig=/var/lib/kubelet/kubeconfig" <ins> flags+=" --register-with-taints=node-role.kubernetes.io/master=:NoSchedule" </ins> flags+=" --kubeconfig=/var/lib/kubelet/bootstrap-kubeconfig" flags+=" --register-schedulable=false" fi

What happened: Created a daemonset ala but missing the master toleration. The created pods are running on every node, including the master. What you expected to happen: Lacking the master toleration, I would expect there to be no pod from the daemonset running on the master. Realized the master Node is missing the master taint. How to reproduce it (as minimally and precisely as possible): Created a daemonset on GCE with no tolerations. (You may need to set the priorityClassName: system-cluster-critical) Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: GCE OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
There are no sig labels on this issue. Please add a sig label by either: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. See the . The in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: // InstanceShutdownByProviderID returns true if the instance is in safe state to detach volumes func (az *Cloud) InstanceShutdownByProviderID(ctx context.Context, providerID string) (bool, error) { <del> return false, cloudprovider.NotImplemented </del> <ins> nodeName, err := az.vmSet.GetNodeNameByProviderID(providerID) if err != nil { return false, err } provisioningState, err := az.vmSet.GetProvisioningStateByNodeName(string(nodeName)) if err != nil { return false, err } return strings.ToLower(provisioningState) == "stopped" || strings.ToLower(provisioningState) == "deallocated", nil </ins> } // getComputeMetadata gets compute information from instance metadata.

Implement proposal for Azure We should implement interface in Azure cloud provider. returns true if the instance is in stopped state to detach volumes. But if instance is deleted we should not return error. Related features: Related proposal: Related PRs: /kind feature /sig azure /cc
- can you assign me on this one ? I already have a PR in flight.
That's great! /assign just loop in for this RP review, thanks.
GitHub didn't allow me to assign the following users: yastij. Note that only and repo collaborators can be assigned. <detailsIn response to : That's great! /assign just loop in for this RP review, thanks. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
o, you are not a k8s member yet, anyway, you could send out the PR for this feature, related to this issue, thanks.
Thanks, looking forward to your PR.


Question: } ctx, cancel := watchtools.ContextWithOptionalTimeout(context.Background(), timeout) defer cancel() <del> _, err := watchtools.UntilWithSync(ctx, lw, &v1.ServiceAccount{}, nil, serviceAccountHasSecrets) </del> <ins> _, err := watchtools.UntilWithSync(ctx, lw, &v1.ServiceAccount{}, nil, func(event watch.Event) (bool, error) { switch event.Type { case watch.Deleted: return false, apierrors.NewNotFound(schema.GroupResource{Resource: "serviceaccounts"}, serviceAccountName) case watch.Added, watch.Modified: return true, nil } return false, nil }) </ins> return err } <del> // serviceAccountHasSecrets returns true if the service account has at least one secret, // false if it does not, or an error. func serviceAccountHasSecrets(event watch.Event) (bool, error) { switch event.Type { case watch.Deleted: return false, apierrors.NewNotFound(schema.GroupResource{Resource: "serviceaccounts"}, "") } switch t := event.Object.(type) { case *v1.ServiceAccount: return len(t.Secrets) > 0, nil } return false, nil } </del> // WaitForDefaultServiceAccountInNamespace waits for the default service account to be provisioned // the default service account is what is associated with pods when they do not specify a service account // as a result, pods are not able to be provisioned in a namespace until the service account is provisioned

Originally posted by in
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig testing /cc
/hold I have a subtly different fix in :)
Oh wait, this is an issue. Yay, thanks :). Fixed in


Question: mutex     sync.Mutex server *grpc.Server <ins> wg     sync.WaitGroup </ins> // activePods is a method for listing active pods on the node // so the amount of pluginResources requested by existing pods

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: I'm running into a test failure during "make test". To get more output and to avoid issue , I am using: What you expected to happen: The test should pass. How to reproduce it (as minimally and precisely as possible): It happens for me only when running all tests, but then it happens reliably. When limiting the test run with , it works. The failing calls are not surprising, because got unlinked after binding to it. Those might come from 's : it removes everything in the socket dir, which happens to be . For comparison, here's a successful run: Anything else we need to know?: I'm wondering whether it is guaranteed that different tests in the same .go file run serialized. Does anyone know for sure? I'm asking because all tests in use the same socket directory and thus would conflict with each other if they were to run in parallel. Note that the and calls in the failed test are for different pids. I've not checked whether those are truly different processes (as in fork+exec) or worker threads (clone). Environment: Kubernetes version (use ): master branch, commit OS (e.g. from /etc/os-release): Debian Stretch (9.0) Kernel (e.g. ): 4.9.65-3+deb9u2
After fixing the /tmp dir handling with PR I can no longer reproduce this issue. It might be worth keeping this issue open nevertheless, because the issue has not been explained completely and there might be some remaining problem lurking somewhere.
I think you originally wrote this test. Perhaps you can comment here and comment on PR ?
/cc This test has been re-written multiple times, I'll take a look ASAP :)
/sig node
Will take a look ASAP.
I experienced the same. What's interesting is I can get it to pass when running the devicemanager suite, but only about half the time.
/area hw-accelerators
Thanks a lot for reporting the issue! I think your analysis is right. Looks like the problem is because the devicemanager unit tests are interfering with each other. I will make a change to have each unit test in the file to use a separate test tmp dir.
I'm not exactly sure why, though.

Do they really run in parallel? According to my understanding of testing.Parallel, one has to explicitly request parallel execution, and the tests don't do that. When they run sequentially, then it isn't obvious why the tests interfere and thus using per-test socket dirs would be just a band-aid. Please have a look at PR , it has such a change.
Ah sorry I didn't realize you were already working on a fix. I think go test by default run tests in parallel, so we need to make sure each unit test is self contained. That also explains why you hit this problem more easily when running more tests. I think we perhaps need to modify setup() to make sure each newly created manager uses a separate test dir. Maybe you can take a look at my WIP change and see whether you can merge some of its changes into PR ?
I suggest you take a look at the documentation: On Fri, 9 Feb 2018, 22:34 Jiaying Zhang, <:
On Sat, 2018-02-10 at 05:31 +0000, Vikas Choudhary (vikasc) wrote: True, but which unit tests run concurrently? To answer that, I re-read the "testing" documentation and ran a quick experiment. That confirmed that individual tests inside the same test binary run sequentially. A test function has to explicitly call t.Parallel() before the next test starts, and that test blocks unless it also calls t.Parallel(). In other words, using the same /tmp/deviceplugin between all test functions should have been okay. I suspect it goes wrong sometimes because one test function starts a goroutine, does not wait for its completion, and that goroutine then interferes with the next test function when using the same dir, depending on timing. Running other test binaries in parallel and thus increasing system load probably also matters, together with overall number of CPUs. I think merging PR makes sense because a dynamic test directory is more flexible and enables running the tests in parallel (which could be a separate PR or be done as part of PR ), but I also think that it would be good if someone with a better understanding of the code that gets tested here could look into this race condition. [(quite) a bit later...] I chased down the spurious unlinkat() to UnixStream.Close for the socket, which unlinks the socket path [1].

By wrapping the listen socket such that a custom Close logs the call and a stack trace, I now have the smoking gun. [1] With some extra logging one can see that the that was created for TestNewManagerImplStart gets closed and unlinked while TestDevicePluginReRegistration already runs. The goroutine which runs too long is the from was called, but didn't wait for the goroutine to actually stop: That now begs the question: what is the right way to stop a grpc server? Below is the change I made to to track when it closes the socket:
On Tue, 2018-02-13 at 15:57 +0100, Patrick Ohly wrote: Server.Stop is the right way, it just doesn't fully stop the server. I consider this an issue in gRPC, therefore I have created https://github .com/grpc/grpc-go/issues/1861
Is it possible that defer statements are executed at the same time another test might start? EDIT: NVM looks like that's not the case
On Tue, 2018-02-13 at 07:39 -0800, Renaud Gaubert wrote: No, defer statements are guaranteed to run before a function returns. I think they can even change the return code and/or catch a panic.
thanks so much for your detailed debugging analysis and details about the tools you used (e.g. using filtered strace). I'm fairly new to golang and I've been messing around with the k8s source locally trying to get a feel for the project and work towards contributing, and I was debugging this test on my own but flailing against how to instrument the running tests. very much appreciated and nice work!


Question: var output string output, err = remote.SSH(name, "sh", "-c", <del> "'systemctl list-units  --type=service  --state=running | grep -e docker -e containerd'") </del> <ins> "'systemctl list-units  --type=service  --state=running | grep -e docker -e containerd -e crio'") </ins> if err != nil { <del> err = fmt.Errorf("instance %s not running docker/containerd daemon - Command failed: %s", name, output) </del> <ins> err = fmt.Errorf("instance %s not running docker/containerd/crio daemon - Command failed: %s", name, output) </ins> continue } if !strings.Contains(output, "docker.service") && <del> !strings.Contains(output, "containerd.service") { err = fmt.Errorf("instance %s not running docker/containerd daemon: %s", name, output) </del> <ins> !strings.Contains(output, "containerd.service") && !strings.Contains(output, "crio.service") { err = fmt.Errorf("instance %s not running docker/containerd/crio daemon: %s", name, output) </ins> continue } instanceRunning = true

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: /test pull-kubernetes-node-crio1-18-e2e Which test(s) are failing: pull-kubernetes-node-crio1-18-e2e Reason for failure: Just before node e2e tests are started, we verify that the runtime (containerd or docker) is properly running or not on the target system. This check doesn't include crio, so the targeted node is not reported ready if it's using crio as the runtime.
/sig testing
/sig node /remove-sig testing This check is owned by SIG node
The code has also not allowed this sort of dependency (beyond some legacy dockershim specific bits), but I guess it's somewhat more understandable to check in .


Question: } func TestComponentPod(t *testing.T) { <ins> // priority value for system-node-critical class priority := int32(2000001000) </ins> var tests = []struct { name     string expected v1.Pod

etcd / kube-apiserver stops before most containers, despite using . On a single node server, with multus (and likely other CNI talking to the API), this spam the logs and make the shutdown slower should stop one group at a time On a single node cluster, have a slow to shutdown pod (sleep inf), look at the logs to see in which order they are killed Reading the kep , the summary says so I would expect all pods with priority between 0 and to be stopped first, then coredns (), then the rest. This is definitely not what's happening: <details</detailsN/A <details</detailskubeadm containerd multus
/cc /sig node
Each will match all Pods whose is less than or equal to it. This may suit your want
/kind support
sorry for the delay I missed the notification ... this is the opposite of what the documentation says :( Maybe i'm misunderstanding the feature, I want shutdown ordering, and it's not working for me. All my pods have priority set to either 0, or . I want all the pods with priority 0 to shutdown first, once they are all shutdown or grace period is elapsed, shutdown the ones with priority , wait for them to shutdown or grace period, and finally take care of the ones with priority . Here some failing tests: test1: First killed pod was etcd (verified priority is ) ... test2: first pod killed was kube-scheduler test3 (same as test2): first pod killed was kube-controller-manager test4: first pod killed was etcd test5 (config in reverse order as test1): etcd was killed 3rd but still before pods with priority 0
I see what you mean, but this shouldn't be happening. It should only shutdown based on the priority of the group from lowest to highest. I would like to know the priority of all Pods on the node you shut down, and the configuration of , to try to reproduce it on my side
I'm reproducing it on an empty cluster, but just noticed coredns is killed correctly, it's just all containers from last group end up in first group
For some reasons the system-node-critical / pods go with the priority 0 pods
I think the pods quit too quickly so it looks like they were deleted together, maybe you should block them until the last moment.

I think I found a lead, it's all the static pods that are killed too soon, not kubeproxy () they have priorityClassName set, but not priority, so maybe priority is nil when getting of a static pod
I see that in numeric priorities to manifests in cluster/gce/manifests/ with the comment in the PR , are priority mandatory for static pods ? should we make kubeadm add them, or fix that in kube somewhere ?
This is the result from the API, I bet kubelet has different path to get pod manifests for cluster pods and static pods, as kubelet can work without the API
If I manually add to etcd manifest it's killed in the last group with kubeproxy
/remove-kind support
/triage accepted
[For a single-node cluster], I wonder if we should actually set a higher priority than for all or a subset of control plane Pods, because even with this change we could still shut down the API server quickly and before a node-critical component has completed its own shutdown.
I agree with this idea, because of the dependencies between the components of node-critical.


Question: } strOut := strings.TrimSuffix(string(out), "n") <del> glog.V(5).Infof("IsLikelyNotMountPoint findmnt output: %v", strOut) </del> <ins> glog.V(5).Infof("IsLikelyNotMountPoint findmnt output for path %s: %v", file, strOut) </ins> if strOut == file { return false, nil }

Currently, there many error messages in that do not contain any context. This means that these messages are basically meaningless in real systems where many pods may be on a node. Examples: Every error message in this file should identify the path that the message is in relation to.
Can you handle this, since you most recently touched in ?


Question: fakeCloud            *fake_cloud.FakeCloud matchRE              string expectedRequestCount int <del> expectedCreated      []string </del> <ins> expectedNameCreated  []string expectedExtIDCreated []string </ins> expectedDeleted      []string }{ {

I see the following in the kube-controller-manager log: - This is happening to me on HEAD and is the reason why pod status is always showing unknown even when they are scheduled and running.
This is with the vagrant cloud provider. It looks like the delta between the new and old minion is the Node.Spec.ExternalID is being set on the incoming minion but is not set in the originally created minion.
I am not sure how this is not affecting other cloud providers. It looks to me that NodeController populated the node ExternalID here: It looks like the Kubelet provided value is missing the ExternalID
Actually, I am now confused on what the NodeController may or may not be doing here that is causing the status to not get updated.
It looks like the client is passing an ExternalID on the incoming request, but the original value that is in etcd does not have an ExternalID set. Not sure what changed that would have caused this issue to appear.
I think this causes problems: The memory is re-used in a for-loop by Go so referring to a pointer value in a map here will cause problems I think.
Ok, the referenced line is the problem. Sending a PR.
Well, I go back on that now. A simple way to reproduce this is to delete all the minions in your cluster, and verify the node controller is able to add them back. For me, this is causing the node controller to add them back in an unknown status, and then all updates to the node fail. The update fails because the node controller is not setting an ExternalID on the original object.
I am looking into this now. cc/ too
Related issue opened while debugging this:
Debugged through here: I can via manual fmt see that the object has an ExternalID set prior to the call Create. When I look at etcd, I end up with this persisted: Note: it is missing an ExternalID somehow ;-)
Ok, I think I see the issue Nodes returned from here: Do not have an ExternalID Which can make Nodes created from here: End up without an External ID. Having two different nodeClient.Create() paths in this code is confusing. Also we are doing bad things in maps holding pointers to memory allocated in for-loop. We need to fix that ;-)
Yeah, this is bug we should fix. Thanks for looking into this.

If you already have a potential fix, please feel free to grab the issue from me.
I think I have a PR that should fix. Testing that now. Will cc you on the PR.
Thanks for fixing - it did end up affecting AWS as well (AWS isn't in a fully working state just yet, particularly with node discovery).


Question: set_num_migs if [[ ${KUBE_USE_EXISTING_MASTER:-} == "true" ]]; then <ins> parse-master-env </ins> create-nodes create-autoscaler else

When we add nodes to an existing cluster, we need to reuse the existing configuration. In particular, we need to extract the kubelet cert & key, otherwise we get a cert error connecting to the master.
WIP fix in


Question: return profiles.HandlesSchedulerName(pod.Spec.SchedulerName) } <del> // skipPodUpdate checks whether the specified pod update should be ignored. // This function will return true if //   - The pod has already been assumed, AND //   - The pod has only its ResourceVersion, Spec.NodeName, Annotations, //     ManagedFields, Finalizers and/or Conditions updated. func (sched *Scheduler) skipPodUpdate(pod *v1.Pod) bool { // Non-assumed pods should never be skipped. isAssumed, err := sched.SchedulerCache.IsAssumedPod(pod) if err != nil { utilruntime.HandleError(fmt.Errorf("failed to check whether pod %s/%s is assumed: %v", pod.Namespace, pod.Name, err)) return false } if !isAssumed { return false } // Gets the assumed pod from the cache. assumedPod, err := sched.SchedulerCache.GetPod(pod) if err != nil { utilruntime.HandleError(fmt.Errorf("failed to get assumed pod %s/%s from cache: %v", pod.Namespace, pod.Name, err)) return false } // Compares the assumed pod in the cache with the pod update. If they are // equal (with certain fields excluded), this pod update will be skipped. f := func(pod *v1.Pod) *v1.Pod { p := pod.DeepCopy() // ResourceVersion must be excluded because each object update will // have a new resource version. p.ResourceVersion = "" // Spec.NodeName must be excluded because the pod assumed in the cache // is expected to have a node assigned while the pod update may nor may // not have this field set. p.Spec.NodeName = "" // Annotations must be excluded for the reasons described in // https://github.com/kubernetes/kubernetes/issues/52914. p.Annotations = nil // Same as above, when annotations are modified with ServerSideApply, // ManagedFields may also change and must be excluded p.ManagedFields = nil // The following might be changed by external controllers, but they don't // affect scheduling decisions. p.Finalizers = nil p.Status.Conditions = nil return p } assumedPodCopy, podCopy := f(assumedPod), f(pod) if !reflect.DeepEqual(assumedPodCopy, podCopy) { return false } klog.V(3).InfoS("Pod update ignored because changes won't affect scheduling", "pod", klog.KObj(pod)) return true } </del> // addAllEventHandlers is a helper function used in tests and in Scheduler // to add event handlers for various informers. func addAllEventHandlers(

Currently the scheduler may add an assumed pod back to the queue if the update didn't include specific fields (see ). There doesn't seem to be a reason why we can't simply ignore all updates of assumed pods. Assumed pods are waiting to be bound, if binding succeeded then the pod is scheduled. If binding fails, the pod's latest version will be put back into the unschedulable queue anyways. Performance: we save on unnecessary scheduling attempts. This is a followup on the discussion we had in /sig scheduling /triage accepted /help
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : #### What would you like to be : /triage accepted /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
What we basically need to do is: 1) remove function 2) in the places where we call , we should simply call
/assign


Question: import ( "bytes" <del> "context" </del> "encoding/json" "fmt" "io/ioutil"

The upper test heavily flakes in master:
/cc
Hey I think this might have something to do with your changes. Something is definitely a bit off. It probably is a faulty test...
duplicate of
Looks like an occasional failure, and I don't think this is caused by my code change because I run tests for a few times, and sometime it is passed.... Still I can help to locate the root cause.
I revert my code change, test still fails from time to time...
I think i've found out the root cause, I'll pull a request soon, . cc

Which jobs are failing: pull-kubernetes-bazel-test Which test(s) are failing: //pkg/kubelet/dockershim/network/cni:godefaulttest Since when has it been failing: Unsure, it doesn't appear in triage, but I see it regularly in PRs. Testgrid link: This package is the top flake currently Reason for failure: Anything else we need to know: /sig node /sig network
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/assign
The test results have the following three phenomena. b c


Question: RestartPolicy: v1.RestartPolicyNever, Containers: []v1.Container{ { <del> Name:  fmt.Sprintf("%s-container", podName), </del> <ins> Name:  "client", </ins> Image: imageutils.GetE2EImage(imageutils.BusyBox), Args: []string{ "/bin/sh",

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: In tests, always fails when the tests fail, with the error: . This is the message indicating that APIServer responds 400 code. What you expected to happen: Container logs should be collected correctly if tests fail. How to reproduce it (as minimally and precisely as possible): a NetworkPolicy test to make it fail and run it, or run NetworkPolicy tests with a CNI that doesn't implement NetworkPolicy. the tests fail, all attempts to get container logs will fail like the following. Anything else we need to know?: Environment: Kubernetes version (use ): 16.3 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig testing /assign
/remove-sig testing /sig network
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/remove-triage unresolved


Question: return fmt.Sprintf("%s:%d/%s", s.IP, s.Port, s.Protocol) } <ins> type realServerKey struct { Address net.IP Port    uint16 } func (r *realServerKey) String() string { return net.JoinHostPort(r.Address.String(), strconv.Itoa(int(r.Port))) } </ins> //NewFake creates a fake ipvs implementation - a cache store. func NewFake() *FakeIPVS { return &FakeIPVS{

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: IPVS real server can't be right deleted in FakeIPVS for ipvs testing. What you expected to happen: When delete ipvs real server, we can't give the weight information, so we can't use deep equal in .
/sig network /area ipvs
/kind bug


Question: IPVS runs on a host and acts as a load balancer in front of a cluster of real servers. IPVS can direct requests for TCP and UDP-based services to the real servers, and make services of real servers appear as virtual services on a single IP address. <ins> ## IPVS vs. IPTABLES IPVS mode was introduced in Kubernetes v1.8 and goes beta in v1.9. IPTABLES mode was added in v1.1 and become the default operating mode since v1.2. Both IPVS and IPTABLES are based on `netfilter`. Differences between IPVS mode and IPTABLES mode are as follows: - [IPVS](#ipvs) - [What is IPVS](#what-is-ipvs) - [IPVS vs. IPTABLES](#ipvs-vs-iptables) - [When ipvs falls back to iptables](#when-ipvs-falls-back-to-iptables) - [Run kube-proxy in ipvs mode](#run-kube-proxy-in-ipvs-mode) - [Prerequisite](#prerequisite) - [Local UP Cluster](#local-up-cluster) - [GCE Cluster](#gce-cluster) - [Cluster Created by Kubeadm](#cluster-created-by-kubeadm) - [Debug](#debug) - [Check IPVS proxy rules](#check-ipvs-proxy-rules) - [Why kube-proxy can't start IPVS mode](#why-kube-proxy-cant-start-ipvs-mode) 2. IPVS provides better scalability and performance for large clusters. 3. IPVS supports more sophisticated load balancing algorithms than iptables (least load, least connections, locality, weighted, etc.). 4. IPVS supports server health checking and connection retries, etc. ### When ipvs falls back to iptables IPVS proxier will employ iptables in doing packet filtering, SNAT and supporting NodePort type service. Specifically, ipvs proxier will fall back on iptables in the following 4 scenarios. **1. kube-proxy starts with --masquerade-all=true** If kube-proxy starts with `--masquerade-all=true`, ipvs proxier will masquerade all traffic accessing service Cluster IP, which behaves the same as what iptables proxier. Suppose there is a service with Cluster IP `10.244.5.1` and port `8080`, then the iptables installed by ipvs proxier should be like what is shown below. ```shell # iptables -t nat -nL Chain PREROUTING (policy ACCEPT) target     prot opt source               destination KUBE-SERVICES  all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes service portals */ Chain OUTPUT (policy ACCEPT) target     prot opt source               destination KUBE-SERVICES  all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes service portals */ Chain POSTROUTING (policy ACCEPT) target     prot opt source               destination KUBE-POSTROUTING  all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes postrouting rules */ Chain KUBE-POSTROUTING (1 references) target     prot opt source               destination MASQUERADE  all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes service traffic requiring SNAT */ mark match 0x4000/0x4000 Chain KUBE-MARK-DROP (0 references) target     prot opt source               destination MARK       all  --  0.0.0.0/0            0.0.0.0/0            MARK or 0x8000 Chain KUBE-MARK-MASQ (6 references) target     prot opt source               destination MARK       all  --  0.0.0.0/0            0.0.0.0/0            MARK or 0x4000 Chain KUBE-SERVICES (2 references) target     prot opt source               destination KUBE-MARK-MASQ  tcp  -- 0.0.0.0/0        10.244.5.1            /* default/foo:http cluster IP */ tcp dpt:8080 ``` **2. Specify cluster CIDR in kube-proxy startup** If kube-proxy starts with `--cluster-cidr=<cidr>`, ipvs proxier will masquerade off-cluster traffic accessing service Cluster IP, which behaves the same as what iptables proxier. Suppose kube-proxy is provided with the cluster cidr `10.244.16.0/24`, and service Cluster IP is `10.244.5.1` and port is `8080`, then the iptables installed by ipvs proxier should be like what is shown below. ```shell # iptables -t nat -nL Chain PREROUTING (policy ACCEPT) target     prot opt source               destination KUBE-SERVICES  all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes service portals */ Chain OUTPUT (policy ACCEPT) target     prot opt source               destination KUBE-SERVICES  all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes service portals */ Chain POSTROUTING (policy ACCEPT) target     prot opt source               destination KUBE-POSTROUTING  all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes postrouting rules */ Chain KUBE-POSTROUTING (1 references) target     prot opt source               destination MASQUERADE  all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes service traffic requiring SNAT */ mark match 0x4000/0x4000 Chain KUBE-MARK-DROP (0 references) target     prot opt source               destination MARK       all  --  0.0.0.0/0            0.0.0.0/0            MARK or 0x8000 Chain KUBE-MARK-MASQ (6 references) target     prot opt source               destination MARK       all  --  0.0.0.0/0            0.0.0.0/0            MARK or 0x4000 Chain KUBE-SERVICES (2 references) target     prot opt source               destination KUBE-MARK-MASQ  tcp  -- !10.244.16.0/24        10.244.5.1            /* default/foo:http cluster IP */ tcp dpt:8080 ``` **3. Load Balancer Source Ranges is specified for LB type service** When service's `LoadBalancerStatus.ingress.IP` is not empty and service's `LoadBalancerSourceRanges` is specified, ipvs proxier will install iptables which looks like what is shown below. Suppose service's `LoadBalancerStatus.ingress.IP` is `10.96.1.2` and service's `LoadBalancerSourceRanges` is `10.120.2.0/24`. ```shell # iptables -t nat -nL Chain PREROUTING (policy ACCEPT) target     prot opt source               destination KUBE-SERVICES  all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes service portals */ Chain OUTPUT (policy ACCEPT) target     prot opt source               destination KUBE-SERVICES  all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes service portals */ Chain POSTROUTING (policy ACCEPT) target     prot opt source               destination KUBE-POSTROUTING  all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes postrouting rules */ Chain KUBE-POSTROUTING (1 references) target     prot opt source               destination MASQUERADE  all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes service traffic requiring SNAT */ mark match 0x4000/0x4000 Chain KUBE-MARK-DROP (0 references) target     prot opt source               destination MARK       all  --  0.0.0.0/0            0.0.0.0/0            MARK or 0x8000 Chain KUBE-MARK-MASQ (6 references) target     prot opt source               destination MARK       all  --  0.0.0.0/0            0.0.0.0/0            MARK or 0x4000 Chain KUBE-SERVICES (2 references) target     prot opt source       destination ACCEPT  tcp  -- 10.120.2.0/24    10.96.1.2       /* default/foo:http loadbalancer IP */ tcp dpt:8080 DROP    tcp  -- 0.0.0.0/0        10.96.1.2       /* default/foo:http loadbalancer IP */ tcp dpt:8080 ``` **4. Support NodePort type service** For supporting NodePort type service, ipvs will recruit the existing implementation in iptables proxier. For example, ```shell # kubectl describe svc nginx-service Name:			nginx-service ... Type:			NodePort IP:			    10.101.28.148 Port:			http	3080/TCP NodePort:		http	31604/TCP Endpoints:		172.17.0.2:80 Session Affinity:	None # iptables -t nat -nL [root@100-106-179-225 ~]# iptables -t nat -nL Chain PREROUTING (policy ACCEPT) target     prot opt source               destination KUBE-SERVICES  all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes service portals */ Chain OUTPUT (policy ACCEPT) target     prot opt source               destination KUBE-SERVICES  all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes service portals */ Chain KUBE-SERVICES (2 references) target     prot opt source               destination KUBE-MARK-MASQ  tcp  -- !172.16.0.0/16        10.101.28.148        /* default/nginx-service:http cluster IP */ tcp dpt:3080 KUBE-SVC-6IM33IEVEEV7U3GP  tcp  --  0.0.0.0/0            10.101.28.148        /* default/nginx-service:http cluster IP */ tcp dpt:3080 KUBE-NODEPORTS  all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes service nodeports; NOTE: this must be the last rule in this chain */ ADDRTYPE match dst-type LOCAL Chain KUBE-NODEPORTS (1 references) target     prot opt source               destination KUBE-MARK-MASQ  tcp  --  0.0.0.0/0            0.0.0.0/0            /* default/nginx-service:http */ tcp dpt:31604 KUBE-SVC-6IM33IEVEEV7U3GP  tcp  --  0.0.0.0/0            0.0.0.0/0            /* default/nginx-service:http */ tcp dpt:31604 Chain KUBE-SVC-6IM33IEVEEV7U3GP (2 references) target     prot opt source               destination KUBE-SEP-Q3UCPZ54E6Q2R4UT  all  --  0.0.0.0/0            0.0.0.0/0            /* default/nginx-service:http */ Chain KUBE-SEP-Q3UCPZ54E6Q2R4UT (1 references) target     prot opt source               destination KUBE-MARK-MASQ  all  --  172.17.0.2           0.0.0.0/0            /* default/nginx-service:http */ DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0            /* default/nginx-service:http */ tcp to:172.17.0.2:80 ``` </ins> ## Run kube-proxy in ipvs mode <del> Currently, local-up scripts and kubeadm support switching IPVS proxy mode via exporting environment variables or specifying flags. </del> <ins> Currently, local-up scripts, GCE scripts and kubeadm support switching IPVS proxy mode via exporting environment variables or specifying flags. ### Prerequisite Ensure the following kernel modules required by IPVS-based kube-proxy have been compiled into the node kernel (use `lsmod` to check): ```shell ip_vs ip_vs_rr ip_vs_wrr ip_vs_sh nf_conntrack_ipv4 ``` Packages such as `ipset` should also be installed on the node before using IPVS mode. Kube-proxy will fall back to IPTABLES mode if those requirements are not met. </ins> ### Local UP Cluster Kube-proxy will run in iptables mode by default in a [local-up cluster](https://github.com/kubernetes/community/blob/master/contributors/devel/running-locally.md). <del> Users should export the env `KUBE_PROXY_MODE=ipvs` to specify the ipvs mode before deploying the cluster if want to run kube-proxy in ipvs mode. </del> <ins> To use IPVS mode, users should export the env `KUBE_PROXY_MODE=ipvs` to specify the ipvs mode before [starting the cluster](https://github.com/kubernetes/community/blob/master/contributors/devel/running-locally.md#starting-the-cluster): ```shell #before running `hack/local-up-cluster.sh` export KUBE_PROXY_MODE=ipvs ``` ### GCE Cluster Similar to local-up cluster, kube-proxy in [clusters running on GCE](https://kubernetes.io/docs/getting-started-guides/gce/) run in iptables mode by default. Users need to  export the env `KUBE_PROXY_MODE=ipvs` before [starting a cluster](https://kubernetes.io/docs/getting-started-guides/gce/#starting-a-cluster): ```shell #before running one of the commmands chosen to start a cluster: # curl -sS https://get.k8s.io | bash # wget -q -O - https://get.k8s.io | bash # cluster/kube-up.sh export KUBE_PROXY_MODE=ipvs ``` </ins> ### Cluster Created by Kubeadm Kube-proxy will run in iptables mode by default in a cluster deployed by [kubeadm](https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/). If you are using kubeadm with a [configuration file](https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-init/#config-file), you can specify the ipvs mode adding `SupportIPVSProxyMode: true` below the `kubeProxy` field. <del> Then the configuration file is similar to: </del> ```json kind: MasterConfiguration

/kind documentation What happened: IPVS needs to be documented more. Specifically, the existing documentation needs to be updated to properly describe how to bring up a cluster with ipvs in the kubeadm, local and gce case. Furthermore, there needs to be another piece of documentation that describes the main differences between iptables and ipvs and also dives into implementation differences between the two (ex. ipset). It would also be really nice if there could be some screenshots detailing the difference between iptables and ipvs rules (i.e bring up a cluster, create a service, dump iptables and ipvs rules, compare) All this documentation should be to pkg/proxy/ipvs /assign /area ipvs /sig network


Question: // TestWatchFromZero tests that // - watch from 0 should sync up and grab the object added before <ins> // - watch from 0 is able to return events for objects whose previous version has been compacted </ins> // - watch from non-0 should just watch changes after given version func TestWatchFromZero(t *testing.T) { ctx, store, cluster := testSetup(t) defer cluster.Terminate(t) <del> key, storedObj := testPropogateStore(ctx, t, store, &api.Pod{ObjectMeta: api.ObjectMeta{Name: "foo"}}) </del> <ins> key, storedObj := testPropogateStore(ctx, t, store, &api.Pod{ObjectMeta: api.ObjectMeta{Name: "foo", Namespace: "ns"}}) </ins> w, err := store.Watch(ctx, key, "0", storage.Everything) if err != nil { t.Fatalf("Watch failed: %v", err) } testCheckResult(t, 0, watch.Added, w, storedObj) <ins> w.Stop() // Update out := &api.Pod{} err = store.GuaranteedUpdate(ctx, key, out, true, nil, storage.SimpleUpdate( func(runtime.Object) (runtime.Object, error) { return &api.Pod{ObjectMeta: api.ObjectMeta{Name: "foo", Namespace: "ns", Annotations: map[string]string{"a": "1"}}}, nil })) if err != nil { t.Fatalf("GuaranteedUpdate failed: %v", err) } // Make sure when we watch from 0 we receive an ADDED event w, err = store.Watch(ctx, key, "0", storage.Everything) if err != nil { t.Fatalf("Watch failed: %v", err) } testCheckResult(t, 1, watch.Added, w, out) w.Stop() // Update again out = &api.Pod{} err = store.GuaranteedUpdate(ctx, key, out, true, nil, storage.SimpleUpdate( func(runtime.Object) (runtime.Object, error) { return &api.Pod{ObjectMeta: api.ObjectMeta{Name: "foo", Namespace: "ns"}}, nil })) if err != nil { t.Fatalf("GuaranteedUpdate failed: %v", err) } // Compact previous versions revToCompact, err := strconv.Atoi(out.ResourceVersion) if err != nil { t.Fatalf("Error converting %q to an int: %v", storedObj.ResourceVersion, err) } _, err = cluster.RandClient().Compact(ctx, int64(revToCompact), clientv3.WithCompactPhysical()) if err != nil { t.Fatalf("Error compacting: %v", err) } // Make sure we can still watch from 0 and receive an ADDED event w, err = store.Watch(ctx, key, "0", storage.Everything) if err != nil { t.Fatalf("Watch failed: %v", err) } testCheckResult(t, 2, watch.Added, w, out) </ins> } // TestWatchFromNoneZero tests that

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): No What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): etcd watcher Is this a BUG REPORT or FEATURE REQUEST? (choose one): Bug report <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): HEAD / commit Environment: Cloud provider or hardware configuration: local VM OS (e.g. from /etc/os-release): Fedora 24 Kernel (e.g. ): Linux localhost.localdomain 4.7.7-200.fc24.x8664 SMP Sat Oct 8 00:21:59 UTC 2016 x8664 x8664 x8664 GNU/Linux Install tools: N/A Others: N/A What happened: If you do a watch from resource version "0" using (the etcd v2 client code), it initially performs a get of the key in question and returns Added or Modified events, based on a comparison of each node's createdIndex and modifiedIndex. This works all the time. If you do a watch from resource version "0" using (the etcd v3 client code), it attempts to get for each kv returned, in an attempt to provide both the previous and current versions of each kv/object. This works as long as has not been compacted. If compaction has occurred, the initial will fail, and it's not possible to watch from "0"' for this particular key.

What you expected to happen: Either: v3 watcher should not change the behavior from the v2 watcher; namely, it should not attempt to get the previous revision of each object and instead just return the current version as a event agree that it's ok to change the behavior, and in the event that the database has been compacted and the previous revision of an object cannot be retrieved, that object is sent as a event instead of . How to reproduce it (as minimally and precisely as possible): I'm working on a unit test and fix. Anything else do we need to know: cc
I will take a look and get back to you tomorrow.
If you have a pr ready, just assign to me or hongchao.
I'll have one tonight hopefully
Although it depends on if we pick option 1 or 2
The behavior of etcd2 is not technically correct. Watch from 0 and will give different results if watch from 0 always return "Added" even if 0 and should be the same. But we do not really have defined what does "0" mean in the interface. Also for etcd3, we cannot guarantee that watch can always return the previous value. Option 1 seems to be easier to achieve. If we do option2, then we will have a "Modified" event without having previous value. I am not sure if that is acceptable or not.
Just give an update here: I talked with on this a bit. We agreed that the behavior should be the same as etcd2 watcher. Of course the API seems non-standard right now. We need to rewrite it in better API once this gets more stable.
Actually, it looks like the event type is a red herring.

The etcd2 client has always set Added/Modified based on the equality/inequality of the node's createdindex/modifiedindex: the watch cache always sets the event type to Added when watching from resourceVersion=0 etcd3 (when watching from 0) is still doing a Get() of the previous revision of every KeyValue: The extra get of the previous version is unnecessary if ModRevision == ModCreation, and can fail if the Get returns an ErrCompacted (it also seems like a potential performance problem to fan out N Get() calls during a single watch establishment
potential fix in
I pretty dislike the fact that: on 0 will first do a sync and then start to watch are forced to return prev value for all modification events watch on 0 should just start watch from the latest revision. recovery the initial state (thus the sync or the initial get) should be a separate operation. But, yea, I guess we need to preserve this at least for now.
- yes, I think there is general agreement for it, though we can't change that in v1 API. There is more-than-1-year old issue for that:
For this case, it won't fail. It returns empty result.


Question: "fmt" "net/http" "strconv" <ins> "sync" </ins> "testing" "time"

The standard event recorder, when recording an event, calls the underlying Broadcaster in a goroutine, so that even recording is non-blocking. Things like leader election emit events over the course of their lifecycle via this broadcaster. Consider the following scenario: GR1: spawn leader election ----GR2 GR2: Leader election: emit event "Lease Acquired" ----spawn GR3 () GR1: Shutdown leader election ----blocks until GR2 is done GR1: Shutdown broadcaster --() GR3: ----send on closed channel () Practically, we've observed this behavior popping up occaisionally in tests in controller-runtime:
/sig api-machinery /kind bug
<details<summaryrelevant race detector output/stack traces</summary</details
were you planning on sending a fix, or need our help to send one to fix it?
I can probably try to figure out a fix if you want, but the whole model for how event broadcasters currently work is inverse to how Go suggests you manage channels (only writers should ever close channels, not readers). I'll need to do a bit of playing around with things.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
cc
I think this is fixed by . Please reopen if it's not the case.
/triage accepted
Would be possible to cherry-pick this fix in 1.20?


Question: }, wantErr: "already registered", }, <ins> { name: "Override MultiPoint plugins weights and avoid a plugin being loaded multiple times", plugins: &config.Plugins{ MultiPoint: config.PluginSet{ Enabled: []config.Plugin{ {Name: testPlugin}, {Name: scorePlugin1}, }, }, Score: config.PluginSet{ Enabled: []config.Plugin{ {Name: scorePlugin1, Weight: 5}, {Name: scorePlugin2, Weight: 5}, {Name: testPlugin, Weight: 3}, }, }, }, wantPlugins: &config.Plugins{ QueueSort:  config.PluginSet{Enabled: []config.Plugin{{Name: testPlugin}}}, PreFilter:  config.PluginSet{Enabled: []config.Plugin{{Name: testPlugin}}}, Filter:     config.PluginSet{Enabled: []config.Plugin{{Name: testPlugin}}}, PostFilter: config.PluginSet{Enabled: []config.Plugin{{Name: testPlugin}}}, PreScore: config.PluginSet{Enabled: []config.Plugin{ {Name: testPlugin}, {Name: scorePlugin1}, }}, Score: config.PluginSet{Enabled: []config.Plugin{ {Name: scorePlugin1, Weight: 5}, {Name: testPlugin, Weight: 3}, {Name: scorePlugin2, Weight: 5}, }}, Reserve:  config.PluginSet{Enabled: []config.Plugin{{Name: testPlugin}}}, Permit:   config.PluginSet{Enabled: []config.Plugin{{Name: testPlugin}}}, PreBind:  config.PluginSet{Enabled: []config.Plugin{{Name: testPlugin}}}, Bind:     config.PluginSet{Enabled: []config.Plugin{{Name: testPlugin}}}, PostBind: config.PluginSet{Enabled: []config.Plugin{{Name: testPlugin}}}, }, }, </ins> } for _, tc := range tests {

Scheduler's default plugin returns an invalid score that is not in the range of [0, 100], resulting in no Pods being scheduled: After investigation, I found that there was a bug in the code that built the framework when the scheduler started. When using the plugins declared in the regular extension point to override the plugins declared in the MultiPoint, the traversed variable will be modified in the following loop: This may cause a same scoring plugin (e.g. ) to be loaded multiple times: Further, this results in a node being processed multiple times by a plugin's (e.g. 's) method, resulting in an invalid score that is not in the range of [0, 100]: If the scheduler's MultiPoint feature is used (either through the default plugins or through manual configuration), and the configuration (e.g. the scoring weight) of a certain plugin is overridden in a regular extension point, the plugin will only be loaded once in this extension point, and no illegal scores will be returned. An example KubeSchedulerConfiguration: No response K8s v1.24.15 or higher None <details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scheduling
Can I work on this?
this is being worked on. see
I see, Is there any other task that you are aware of and I can work on? Thanks.


Question: type hostpathCSIDriver struct { driverInfo       storageframework.DriverInfo manifests        []string <del> cleanupHandle    framework.CleanupActionHandle </del> volumeAttributes []map[string]string }

Tracking bug for /sig storage /important-longterm /assign
/sig storage /triage accepted
/priority important-longterm


Question: the each of service proxy endpoints in the cluster using a http post(protocol=udp)  and the request MUST be successful. Container will execute curl command to reach the service port within specified max retry limit and MUST result in reporting unique <del> hostnames. </del> <ins> hostnames. This test is marked LinuxOnly it breaks when using Overlay networking with Windows. </ins> release: v1.9 file: test/e2e/common/network/networking.go - testname: Proxy, validate Proxy responses

ci-kubernetes-e2e-capz-master-containerd-windows should function for node-pod communication: http [NodeConformance] should function for node-pod communication: udp [NodeConformance] PR merged on April 4th 2022 Discussed at sig-windows triage meeting as well as regular meeting: This test is passing for flannel overlay: and aks-engine: We plan temporary disable this on the CAPZ test suite while we diagnose what is missing. /sig windows /sig networking /cc
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : ### Which jobs are failing? should function for node-pod communication: udp [NodeConformance] ### Since when has it been failing? /sig networking Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig network Did some looking into this and the tests is not deploying a hostprocess pod for Windows: Following the test flow Creates test pods: Then the hostnetwork is set to true but the windows Hostprocess fields are not set:
This pod being tested has which seems to be causing issues with calico, if I remove that then it works (as the other tests that do pod-pod demonstrate). Once I switched this test to use a hostprocess this test works for on the same host when using Windows overlay. Overlay has this limitation that we have seen before: fail (according to the , it is not mandatory for kubelet to be able to access pods on a different node).
My thoughts are we should disable these two tests for 1.24 because they are not testing what they should be and it requires some tough changes to the framework to make it work as is. These tests also doesn't make sense for . A example of the changes need to make this partially work on l2bridge but not overlay:


Question: # KUBE_USE_EXISTING_MASTER is used to add minions to an existing master if [[ "${KUBE_USE_EXISTING_MASTER:-}" == "true" ]]; then <del> # Detect existing master </del> detect-master <ins> parse-master-env </ins> # Start minions start-minions

When we add nodes to an existing cluster, we need to reuse the existing configuration. In particular, we need to extract the kubelet cert & key, otherwise we get a cert error connecting to the master.
WIP fix in


Question: By("deleting host0Pod") framework.ExpectNoError(podClient.Delete(host0Pod.Name, api.NewDeleteOptions(0)), "Failed to delete host0Pod") } <ins> By("Test completed successfully, waiting for PD to safely detach") waitForPDDetach(diskName, host0Name) </ins> }) It("should schedule a pod w/two RW PDs both mounted to one container, write to PD, verify contents, delete pod, recreate pod, verify contents, and repeat in rapid succession [Slow]", func() {

PD E2E tests have very aggressive clean up routine that does not wait for volumes to safely detached. After the test completes, the test cleanup routine unilaterally detaches the PD (instead of waiting for k8s to do it cleanly). Although this does not cause the test to fail, it results in errors in the kernel logs: We should modify these tests to wait for k8s to safely detach the disk.


Question: if len(plist.FrontendVIPs) == 0 || plist.FrontendVIPs[0] != vip { continue } <ins> } else if len(plist.FrontendVIPs) != 0 { continue </ins> } LogJson(plist, "Found existing Hns loadbalancer policy resource", 1) return &loadBalancerInfo{

What happened: I created a service where the was the same as the , and having done so, then the was not accessible externally on the Windows node where the pod was running (it was, however, accessible on the Linux master). When I changed either the or the so that they were different then the service was accessible on the Windows node. What you expected to happen: I expect services to work when the and the are the same value. How to reproduce it (as minimally and precisely as possible): Here's the file that I used: Then, in a web browser on my desktop machine, I visited the on both the Linux master's IP address, and then in a different tab, using the again, the IP address of the Windows worker node where that pod was scheduled. It only worked using the IP address of the Linux master. The tab using the Windows worker node's IP address did not load. Then, I changed either the or the so that they were different values, and when they were different, it worked using the of both the Linux master's IP address and the IP address of the Windows worker node where that pod was scheduled. Anything else we need to know?: On-premise install of k8s The Windows nodes all have the following feature gates enabled: WinOverlay WindowsGMSA Linux master: OS (e.g. ): Windows worker: If you need any more information then please let me know.
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Can you take a look?


Question: By("Running pods to generate events while waiting for some of them to be ingested") wait.PollUntil(eventCreationInterval, func() (bool, error) { podName := "synthlogger" <del> createLoggingPod(f, podName, "", 1, 1*time.Second) </del> <ins> startNewLoggingPod(f, podName, "", 1, 1*time.Second) </ins> defer f.PodClient().Delete(podName, &meta_v1.DeleteOptions{}) err = framework.WaitForPodSuccessInNamespace(f.ClientSet, podName, f.Namespace.Name) if err != nil {

Pods should be run in parallel with checking the logs from the backend to fit into the timeout.


Question: runtimeService:    runtimeService, imageService:      imageService, logMetricsService: logMetricsService, <ins> cpuUsageCache:     make(map[string]*runtimeapi.CpuUsage), </ins> } }

Related: On Windows, kubelet reports only the cumulative for cpu usage. On Linux, kubelet reports both and the instantaneous . Although In the future, most components/monitoring pipelines will switch to using the cumulative metrics, a few features still depend on the instantaneous metrics today. Examples are: , HPA/VPA. Kubelet should report this metrics on Windows to support the existing features. /cc who explained to me what features depend on the cpu metrics. /cc /cc
- is this something that's clear enough in the CRI layer that you can implement it, or do we need to fix that first?
/assign
do you have any notes or a doc you can point to that better describes what features depend on which metrics? It would be good for us to share and see if anything other than is broken
The horizontal and vertical pod autoscalers also depend on the
Documentation on metrics API and metrics server (i.e., the aggregator): The metrics-server pod keeps spewing errors like this: Running : I haven't verified whether pod autoscaling works in my cluster, but I assume they won't since metrics server seems to discard the metrics from the windows pods. I believe the implementation is in kubelet (for the docker integration), so this should not need CRI changes.
I think this is generally an issue for non-Docker CRIs (not necessarily Windows-exclusive). The kubelet source code suggests that these metrics will be gathered from the CRI (if the CRI is not Docker). The struct that kubelet expects returned from the ContainerStats RPC does not include usageNanoCores. Does this mean that, until this is resolved, custom CRIs are unable to satisfactorily report metrics to the 'metrics-server'?
- are you able to work on this?
that is a good point. For CRI runtimes, the kubelet should calculate the average over 10s to produce this metric.
Good point. We designed CRI around core metrics API, but some CRI runtimes have integrated with cadvisor directly to fill the gap while we wait for core metrics API. I think it's reasonable for kubelet to do this on behalf of all CRI runtimes. I spoke to offline, and this would probably require no more than 2 days of work. If sig-windows (e.g., can pick this up, that'd be great.

has agreed to review the changes (thanks!) If not, we should clearly document that this is not supported in Windows GA for 1.14.
thanks for the update. we will wait to hear back from
Currently, is got from docker's stats API on Windows. From docker stats API , only is available, hence is not set on Windows: Do you know any way to get instantaneous for Windows containers?
I have no idea. any idea?
I think this is broader than just a Windows issue. Officially supported tools such as the kubernetes metrics-server expect the field, but custom CRIs that conform to the interfaces defined in the package are not given the liberty to return this value.
suggested previously to cache the stats temporarily and compute the stats using the diff. David, could you chime in on how this could be done? Yes, this was mentioned in the previous comments too. There are plans to transition those components off this field, but it's WIP. We are looking for a workaround in kubelet for now.
my biggest worry is that if we say this impacts the horizonal POD autoscaler, how is that passing in our end2end and conformance tests?
Do you test HPA? It's not part of conformance, and I doubt this is included in windows testing.
As to how this could be done, in and we could just add a in the struct, which we assign the previous CpuUsage metric for each container. Then, when we need to calculate the rate, we pass the previous CpuUsage along with the current one to , and calculate a rate based on the timestamp and values.
ListPodCPUAndMemoryStats() we could just add a map[string]*runtimeapi.CpuUsage in the criStatsProvider struct I've done some tinkering on this myself and I originally thought that it would be as simple as that. It gets much more complicated when you have to work in between the existing cAdvisor-based logic. I assume the intent is to cut it out entirely eventually, but the information passed through the CRI doesn't even come close to being so comprehensive. So far I've found that this approach fixes the problem for the "/stats/summary" api, but not necessarily for the other apis. In particular, "/stats/container" seems to be completely based on cAdvisor, being unimpacted by the changes we made to those two functions.

Some monitoring tools and metrics scrapers rely on the "/stats/container" api for their metrics, which means it is insufficient to update one api without doing the other.
We don't need to fill in everything in the summary API, just "usageNanoCores". The "/stats/summary" api is the one used by the metrics server, which is why we care about it in this case. The "/stats/container" isn't something we need to implement to support this.
we a couple of tickets to validate if HPA for Windows containers works. My guess is that it works for memory and not for CPU metrics because of this issue. If this is complicated to fix, i wonder if waiting for the CRI containerD changes in windows would be the best option for supporting this.
containerd and cri-o bypass this issue by leveraging cadvisor as a temporary solution. I don't think that'd work for windows.
Sorry the for late response. HPA is actually working for Windows containers. e.g. in my cluster (v1.12.5), cpu-based HPA is working as expected: Which version of kubernetes are you referring for HPA issues?
are you running heapster, or the metrics server to serve HPA? Heapster uses the cumulative metric, but the metrics-server uses the 10 average rate metric.
metrics-server is used in my cluster.
See .
/milestone v1.14


Question: ) # tell the user which we've selected and lint all scripts <ins> # The shellcheck errors are printed to stdout by default, hence they need to be redirected # to stderr in order to be well parsed for Junit representation by juLog function </ins> res=0 if ${HAVE_SHELLCHECK}; then echo "Using host shellcheck ${SHELLCHECK_VERSION} binary." <del> shellcheck "${SHELLCHECK_OPTIONS[@]}" "${all_shell_scripts[@]}" || res=$? </del> <ins> shellcheck "${SHELLCHECK_OPTIONS[@]}" "${all_shell_scripts[@]}" >&2 || res=$? </ins> else echo "Using shellcheck ${SHELLCHECK_VERSION} docker image." "${DOCKER}" run  --rm -v "${KUBE_ROOT}:${KUBE_ROOT}" -w "${KUBE_ROOT}"  "${SHELLCHECK_IMAGE}"  <del> shellcheck "${SHELLCHECK_OPTIONS[@]}" "${all_shell_scripts[@]}" || res=$? </del> <ins> shellcheck "${SHELLCHECK_OPTIONS[@]}" "${all_shell_scripts[@]}" >&2 || res=$? </ins> fi # print a message based on the result

junit output: full output with actual error (expand build- and grep around) we want the actual error to be included in the junit so it will be displayed at the top, right now the error at the top is confusing and useless. /sig testing
/assign


Question: <del> #!/bin/bash if [[ "$(grep -c "COMMIT_BLOCKED" $1)" -gt 0 ]]; then echo "FAILED: Unresolved errors - aborting the commit." echo "The message of your attempted commit was:" cat $1 exit 1 fi exit 0 </del>

Multiple, for example: Since See above The test tolerations are now to strict: No response /sig node
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
cc
I think it's from if you check the cri-o versions the succeessful one was and the failing was
Working on a fix in
cc
/reopen need to bump in kube as well to fully get the fix
Reopened this issue. <detailsIn response to : /reopen </details