Question: Accelerators utilfeature.Feature = "Accelerators" // owner: @jiayingz <del> // alpha: v1.8 </del> <ins> // beta: v1.10 </ins> // // Enables support for Device Plugins <del> // Only Nvidia GPUs are tested as of v1.8. </del> DevicePlugins utilfeature.Feature = "DevicePlugins" // owner: @gmarek

/kind feature What happened: The DevicePlugins feature was introduced in 1.8 Kubernetes release as an alpha feature. Since then, we have seen good development work on stabling the feature as well as early adopters on using the framework to support device resources in Kubernetes. This issue is to track the work requirements of entering this feature into beta. FYI, here is list of requirements from the OSS Kubernetes feature stage doc: """ Beta level: Object Versioning: API version name contains beta (e.g. v2beta3) Availability: in official Kubernetes releases, and enabled by default Audience: users interested in providing feedback on features Completeness: all API operations, CLI commands, and UI support should be implemented; end-to-end tests complete; the API has had a thorough API review and is thought to be complete, though use during beta may frequently turn up API issues not thought of during review Upgradeability: the object schema and semantics may change in a later software release; when this happens, an upgrade path will be documented; in some cases, objects will be automatically converted to the new version; in other cases, a manual upgrade may be necessary; a manual upgrade may require downtime for anything relying on the new feature, and may require manual conversion of objects to the new version; when manual conversion is necessary, the project will provide documentation on the process Cluster Reliability: since the feature has e2e tests, enabling the feature via a flag should not create new bugs in unrelated features; because the feature is new, it may have minor bugs Support: the project commits to complete the feature, in some form, in a subsequent Stable version; typically this will happen within 3 months, but sometimes longer; releases should simultaneously support two consecutive versions (e.g. v1beta1 and v1beta2; or v1beta2 and v1) for at least one minor release cycle (typically 3 months) so that users have enough time to upgrade and migrate objects Recommended Use Cases: in short-lived testing clusters; in production clusters as part of a short-lived evaluation of the feature in order to provide feedback """ I think we met most of the requirements here, with the commit that if we need to make incompatible API change in the future, we will document the upgrade path and if necessary, may support multiple API versions in a couple of releases.

Things that I think would be nice to have to enter beta (entering beta may not necessary depends on their completeness) Regarding to the thorough API review requirement, I think we had good discussions during the design proposal but there have been some recent discussions on whether we should use registration vs probe model for the general Kubelet plugin communication. I think we should have a general discussion on this topic with larger audience like people working on CSI plugins. I will create a FR issue to track this discussion soon. We would like to see multiple device plugin implementations. We know some people have started working on device plugin implementations for Solarflare, FPGA, and Infiniband, and probably want to keep closer track on their progresses. We would like to target this FR to 1.10. Please let us know your thoughts or add things that you think are critical to finish. Thanks! What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig node
I'm a fan of probe model. I do feel it unfriendly to plugin developers to let them care about the re-registration after kubelet's restart. And have received complaint from my colleague.
Hi, for cluster reliability, I think we need more test cases, including scale and soak tests, to simulate more production scenarios.
can we consolidate this with ?
Sure. Will do this with Vish after KubeCon.
Sorry to bring up this topic a bit late (feature freeze is the 22nd of Jan). Here are the features we (NVIDIA) consider important or blocker before entering beta: [BLOCKER] Per container allocation Add labels during device plugin registration (e.g: gpu-family: tesla) Add annotations to a container for beta and add an option to CRI for GA to support OCI hooks Things that are not explicitly related to the device plugin: We're also wondering if we should tie up
Thanks a lot for your input, For per container allocation, I think we should be able to implement it in 1.10. I think we should be able to support ER quota in 1.10 as well.

I think allowing device plugins to export device properties is an important feature, especially when we introduce ResourceClass, but I don't think it is a beta blocker. For "Add annotations to a container for beta and add an option to CRI for GA to support OCI hooks", this seems outside the scope of device plugin. There are different ways to implement them (if they have to be implemented) on Kubernetes. We haven't seen strong use cases indicating that they have to go through device plugin.
Sorry, I meant from the device plugin. i.e: the device plugin's returns container annotations for beta Product wise this would allow us to have the nvidia-runtime be supported by CRI-O (which allows hooks through container annotations) as well as docker + nvidia as the default runtime. BTW That would allow a convergence towards a single device plugin in beta. As for GA this would be a different discussion but it would be part of converging towards using any runtime and have them support the nvidia-hooks.
Not saying it's a beta blocker but it would be useful to have labels on homogeneous nodes about what devices are present. Especially: GPU memory GPU compute capability ... It's also not a big change, it just requires consensus
As a heads up, I am going to send a PR soon to change DevicePlugins feature to beta. After introducing the feature in 1.8, we have seen good development work on improving the feature and received many valuable feedbacks from early adopters on how we may extend the framework to support various kinds of resources on Kubernetes. To us, entering beta means: a) We think we have done enough early experiments during alpha phase to get the confidence that this a useful feature that we are willing to carry on to beta and to GA. b) We think the current DevicePlugins is ready to be tried on production. It has good e2e and e2enode test coverage. Some early adopters have tried it on production and didn't report any major breakage. Very likely we will keep seeing new feature requests coming in after beta, as more people try to use DevicePlugins for their specific use cases. Some of these features may even require incompatible API changes. When this happens, the required complexity to support multiple device plugin API versions on Kubelet side should be manageable and mostly local to the devicemanager component.

It should also be manageable for a device plugin to support multiple API versions. As an example, in , I extended gke gpu device plugin to support both v1alpha and v1beta1 versions. For reference, here is list of changes we made on device plugin during 1.10: Support resource quota on extended resources (Issue ). Owner: Added PreStartContainer grpc function to allow device plugins to perform per container start operations. Owner: Add annotations to the device plugin API. Owner: Track unhealthy devices in devicemanager, and have node status capacity to reflect raw capacity but allocatable to reflect healthy device capacity. Owner: Stub device plugin based e2enode test Owner: There are a few pending PRs that would be nice to get into 1.10 but we don't consider as beta blocker: Determine and implement a general Kubelet plugin communication establish model and use it in device plugin. Owner: Uses common checkpointing mechanism in device plugin (PR Issue ) Owner: Device plugin stress test. Owner: Please let us know if anyone has objections on entering the feature to beta in 1.10. Thanks!
Hi, for stress test, here is a PR :)
/cc
Should we reopen this tracing issue?
I think this issue can be closed with We can open separate issues to track future device plugin changes.


Question: // PodMetrics sets resource usage metrics of a pod. type PodMetrics struct { <del> metav1.TypeMeta   `json:",inline"` </del> <ins> metav1.TypeMeta `json:",inline"` // Standard object's metadata. // More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata // +optional </ins> metav1.ObjectMeta `json:"metadata,omitempty" protobuf:"bytes,1,opt,name=metadata"` // The following fields define time interval from which metrics were

no explanation for metadata filed of NodeMetrics/PodMetrics, code: should be like $ kubectl explain NodeMetrics KIND: NodeMetrics VERSION: DESCRIPTION: NodeMetrics sets resource usage metrics of a node. FIELDS: apiVersion <stringAPIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: kind <stringKind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: metadata <Objecttimestamp <string-required- The following fields define time interval from which metrics were collected from the interval [Timestamp-Window, Timestamp]. usage <map[string]string-required- The memory usage is the memory working set. window <string-required- and $ kubectl explain PodMetrics KIND: PodMetrics VERSION: DESCRIPTION: PodMetrics sets resource usage metrics of a pod. FIELDS: apiVersion <stringAPIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: containers <[]Object-required- Metrics for all containers are collected within the same time window. kind <stringKind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: metadata <Objecttimestamp <string-required- The following fields define time interval from which metrics were collected from the interval [Timestamp-Window, Timestamp]. window <string-required- able to see the explanation for metadata filed of NodeMetrics/PodMetrics Kubernetes Version: v1.22.0-rc.0+
/sig instrumentation
/kind documentation /assign /triage accepted


Question: return } <del> fs.StringVar(&o.ServiceAccountKeyFile, "service-account-private-key-file", o.ServiceAccountKeyFile, "Filename containing a PEM-encoded private RSA or ECDSA key used to sign service account tokens.") </del> <ins> fs.StringVar(&o.ServiceAccountKeyFile, "service-account-private-key-file", o.ServiceAccountKeyFile, "Enables legacy secret-based tokens when set. Filename containing a PEM-encoded private RSA or ECDSA key used to sign service account tokens.") </ins> fs.Int32Var(&o.ConcurrentSATokenSyncs, "concurrent-serviceaccount-token-syncs", o.ConcurrentSATokenSyncs, "The number of service account token objects that are allowed to sync concurrently. Larger number = more responsive token generation, but more CPU (and network) load") fs.StringVar(&o.RootCAFile, "root-ca-file", o.RootCAFile, "If set, this root certificate authority will be included in service account's token secret. This must be a valid PEM-encoded CA bundle.") }

The kube-controller-manager issues a warning during startup when is specified without providing a . This warning is misleading because the legacy service account token controller, which relies on the , is no longer required for most modern Kubernetes clusters unless explicitly needed for generating long-lived tokens. Additionally, the documentation does not clearly state the purpose of this flag or its impact on legacy token mechanisms. I expected that the warning related to the would either not appear or be updated to reflect its relevance only to the legacy token mechanism. Furthermore, I expected the documentation to clearly explain that this flag is only necessary for enabling the legacy service account token controller and to provide updated guidance on generating long-lived tokens. kube-controller-manager with the --use-service-account-credentials flag but without the --service-account-private-key-file flag. the warning message that is issued during initialization. the current documentation for kube-controller-manager flags and the process for generating long-lived service account tokens. for context <details</details<detailsBare metal </details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/sig auth


Question: }, }, }, <ins> "job does not allow setting status on create": { job: batch.Job{ ObjectMeta: getValidObjectMeta(0), Spec: batch.JobSpec{ Selector: validSelector, Template: validPodTemplateSpec, }, Status: batch.JobStatus{ Active: 1, }, }, wantJob: batch.Job{ ObjectMeta: getValidObjectMetaWithAnnotations(1, map[string]string{batchv1.JobTrackingFinalizer: ""}), Spec: batch.JobSpec{ Selector: validSelector, Template: validPodTemplateSpec, }, }, }, </ins> } for name, tc := range cases {

The gets very long testing mulitple functions in such as , and . These checks should be performed in dedicated Test functions using table for different scenarios. Target functions and are being as part of The remaining checks should be migrated to these functions. Also, a test function should be to migrate the remaining tests.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig apps
/assign
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
Any progress on this? I'm working on and I noticed that has relatively low coverage so I was going to work on getting the coverage up.
I created that is somewhat related to this.
no progess. go ahead please
/assign


Question: // registering all of these functions. Then, if you want to be able to understand // v1beta1 objects, you just import that package for its side effects. AddConversionFuncs( <del> // EnvVar's Name is depricated in favor of Key. </del> <ins> // EnvVar's Key is deprecated in favor of Name. </ins> func(in *EnvVar, out *v1beta1.EnvVar) error { out.Value = in.Value out.Key = in.Name

I'm trying to make sure I understand how Kubernetes will maintain compatibility between current and future api versions. While digging through the api and conversion packages I noticed the comment at states that EnvVar.Key will be deprecated soon. When I look in the conversion functions at it seems like EnvVar.Key is preferred over name. Should I assume both Key and Name are going away?
Name is staying, Key is going away. If you take another look at the file, you'll see that when we copy from internal to v1beta1, we set both Key and Name (to maintain compatibility with clients using Key). When we copy from v1beta1 to internal, we look at Key only if Name == "", that is, Name takes precedence if it's set. ...Unfortunately, the comment there is backwards, which is my fault, I'm sure.
Sorry about the confusion!
At some point we talked about just changing env to be a map[string]string and accepting it as such in JSON/YAML. Any feelings here? I think it might make a better API for humans , Daniel Smith wrote:
I'm OK with that and am willing to write the code to make it happen if everyone else is, too.
Yeah, that comment in was throwing me for a loop. I think the rest is making sense to me now so feel free to close this whenever the comment is updated. On a related note, the comments on sound promising to me. Right now I'm wrapping my head around what upgrading Kubernetes infrastructure is going to be like for admins (essentially, how hard will it be to perform rolling updates when APIs change between versions).
Yeah, while we don't have an official policy about version lifecycle/deprecation timeframe, it is on our roadmap to produce one. Ideally, churn will be low and the number of supported older versions high enough that updating one's config files is an infrequent activity. Additionally, given the way we're doing this, it would be easy for us to provide a tool that automatically updates config files to the latest version.
About moving to for this -- I'm torn on these things. It does look better but it can also make the serialization of the API more confusing when written out -- especially for the novice user.

Specifically, the way things are now it is very clear that when you serialize this as YAML that everything of the left side of an assignment is a system defined string while everything on the right side is something that the user provides. I admit, for things like environment variables things are pretty self explanatory. But it breaks the pattern. I'd be afraid we'd try and apply the same pattern (user values on the left) in other places where the semantics aren't as clear.
that's why I invoked your name - I think you feel most strongly of all of us, and given that you get veto power. I yield the floor to the gentleman from Seattle. , Joe Beda wrote:
Yeah -- let's leave it as-is for now. I thought of an example where we'd be tempted to follow this pattern and it'd be more confusing. Let's apply the same thing to . We have a name on and we could easily change to be a . But because ports don't have the same historical semantics new users might be confused if the port name is magic or not. Basically I think that this: Is less confusing than this: Even though the second one is shorter, to the novice user it looks like and are both magic system keywords when actually is just a user provided token.
I agree that the second is a step back for ports. For env it's a lot more obvious, but it is harder to hold the line once you have crossed it once. , Joe Beda wrote:
I agree that Joe's example is not a thing we should do, and also agree that it's much clearer with env vars. I think there's probably room in the middle of this slippery slope.


Question: ret=1 fi <del> if ! _out="$(diff -Naupr -x "BUILD" -x "AUTHORS*" -x "CONTRIBUTORS*" vendor "${_kubetmp}/vendor")"; then </del> <ins> if ! _out="$(diff -Naupr -x "BUILD" -x "zz_generated.openapi.go" -x "AUTHORS*" -x "CONTRIBUTORS*" vendor "${_kubetmp}/vendor")"; then </ins> echo "Your vendored results are different:" >&2 echo "${_out}" >&2 echo "Godeps Verify failed." >&2

Which jobs are failing: Which test(s) are failing: Since when has it been failing: 2019-03-07, in master, in 1.14 Testgrid link: /sig testing /priority critical-urgent /milestone v1.14 cc
/milestone v1.14
/remove-sig testing /sig api-machinery Related: , ,
cc It looks like this started failing right after this merge: (commit )
fix in
/reopen For CI Signal to verify that things look green once we got more job runs
Reopened this issue. <detailsIn response to : /reopen For CI Signal to verify that things look green once we got more job runs Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
All green :tada: /close
Closing this issue. <detailsIn response to : All green :tada: /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: // Teardown cleans up any remaining resources. func (t *HPAUpgradeTest) Teardown(f *framework.Framework) { // rely on the namespace deletion to clean up everything <ins> common.DeleteHorizontalPodAutoscaler(t.rc, t.hpa.Name) </ins> t.rc.CleanUp() }

Currently the time it takes for HPA and CA to scale up after load increase is not measured at all, and this can lead to regressions that will be missed. This issue is about adding an e2e test that will prevent such regressions.
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list .
Filip, can you reopen? I cannot do this myself, and this issue is clearly not closed yet.
Thanks Wojtek!


Question: l := klog.Level(1) l.Set("1") // hijack the klog output <del> tmpWriteBuffer := bytes.NewBuffer(nil) </del> <ins> tmpWriteBuffer := new(buffer) </ins> klog.SetOutput(tmpWriteBuffer) klog.LogToStderr(false)

There seems to be a race condition in the following unit test The race happens between read at and write at No race condition CI logs where race was detected: <details</details<details</details<details</details<details</details<details</details<details</details
/sig node /cc
/remove-kind bug /kind flake (potential flake, did not use the flake template because CI is green for this test)
A related pr merged
/assign
Thanks for checking again. fyi
Maybe we need something like this:
are u working on resolve this issue? i wonder why doesn't face a race condition
I think he's still working on it. The race condition doesn't happen there because doesn't start another go routine. In the kubelet's case however, the write part of the race condition happens in another go routine that is spawned: write:
/triage accepted /priority important-soon
seen in this spiked in the last couple days - is this related to the klog bump ()? cc /priority critical-urgent /milestone v1.24
increasing priority since it is impacting test runs, and marking as blocker until root caused and we understand whether it is prod-impacting
/assign ^ I'll take this one up on monday - still feel free to open a PR if you have WIP investigation/research
please see if you have comments/inputs on the already open one.
I don't see how the klog bump could have made it worse. Perhaps the change around the flush daemon changed some timing conditions, but that's a rather wild guess. These tests have been faulty all along and need to be fixed. What klog can do is support unit tests like this better. I've opened two issues: short term: long term as it depends on contextual logging:
I think it's safe to remove the release-blocker here. The root cause is largely as explained here: (appears to be test-only)


Question: Description: The ValidatingAdmissionPolicy should type check a CRD. */ <del> framework.ConformanceIt("should type check a CRD", func(ctx context.Context) { </del> <ins> framework.It("should type check a CRD", func(ctx context.Context) { </ins> crd := crontabExampleCRD() crd.Spec.Group = "stable." + f.UniqueName crd.Name = crd.Spec.Names.Plural + "." + crd.Spec.Group

<img width="1015" alt="image" src="" /kind failing-test <!-- If this is a flake, please add: /kind flake --/sig api-machinery
/priority critical-urgent Looking at the last green and first red shows the following: xref : /assign
hrmm, looks super solid in
yea something screwy going on with the kubeadm jobs :(
my main concern what that this was a test ..
kinder or kind jobs have been mostly solid across the board for a few years. if something is failing in kind / kinder then likely it would also fail in resource constraint scenarios. sorry but my 2c are that this is worth investigating by owning sigs
i don't know what else could be the cause here. kubeadm is crd agnostic.
completely agree this needs resolution, I'm just confused since is a really similar setup and looks completely solid The test waits until the status is set, so even a resource-constrained env should just wait a little longer ... the failures I see are bailing out of that loop after only one wait (105ms), so something is turning that field non-nil without populating the expected results I'm also pretty confused
the diff between the two jobs (kinder/kind) reveals that the kind job has much higher resource limits/requests (for CPU) full diff: <details</details
i'm not familiar with this area of the code. it does appear to be waiting for the TypeChecking struct to be populated, but not for the Warnings slice to be populated. logically (without trying to find the code): if TypeChecking is expected to be populated with Warnings during a Get call after the Create call, we are catching a flake bug. if the Warnings in the TypeChecking are populated async of the TypeChecking then we are catching a test flake and the test must be updated to wait for the Warnings too.
well, the code is here: it appears that the only way for this to happen is if the Check() call returns nil warnings under "some conditions". the problem is not reproducible in the unit test here if the interval is set to a lower value <= 100ms and tested with the tool. but that's with a fake client.

this extra err check seems redundant: it might be worth temporary adding a print of the crd and policy before this line: then allow this test to flake again and inspect the results.
/milestone v1.30
Thanks for raising the issue and thanks for the investigation! I don't know exactly why the solid test in kind would be flaky in kubeadm by a quick scan but agree it needs attention.(The tests look less flaky though..) /assign the type checking code owner. Would you mind adding a printout as suggested above and see if we could get more info? Thanks!
note, both kind and kinder (from the compared jobs) use kubeadm.
and i wonder if this is an informer cache issue.
Same guess as failed for missing event for service patch since 03/06(similar time). [Not sure; just guess]
shows that the test failed for a different reason. failure in ci-kubernetes-e2e-ec2-alpha-enabled-default: from 03/04 to 03/06.
/remove-kind failing-test /kind flake
the API field doc indicates a non-nil value means type checking is complete: It doesn't look like the type checker has any way to signal that it was lacking some information needed to complete type checking, and will record an empty set of warnings if it is: are there any scenarios where a laggy REST mapper or informer can make the controller think it completed type checking but be missing some info that makes it produce an empty list of warnings? I think our options are: the test to wait for non-empty warnings in the status (not just non-nil typeChecking) the controller to not write an empty typeChecking stanza if it fails to type check in some way option 2 seems better for observers, but I don't know how invasive that is
option 2 feels like we are creating a hole? (folks will see warnings sometimes and sometimes they don't?)
Option 2 might cause confusion especially when deleting existing warnings from typechecking field.

(We will remove the typechecking warning if observed the fix in schema)
I'm not suggesting deleting existing warnings, but it appears we are currently reporting an empty typeChecking stanza when we have not actually finished type checking... that doesn't match the API docs and is likely confusing to API clients (it's confusing our own e2e test we wrote based on the documented behavior)
Ignoring errors while finishing the type checking is intended behavior defined by the original KEP. Can we go back to change the tests to do a retry loop until the CRD appears in the RESTMapper? The root problem of laggy RESTMapper seems out of the scope of this test case.
if is fixing the test then it's masking a state not guaranteed by the field comment: i guess already explained that above.
The type checking is completed with this field set to non-nil. The problem is that the type checking finishes with error, and it cause some of the referred types to be ignored.
should this error be surfaced to the user and does that need an e2e test?
No. The KEP defines that errors will be ignored. However, there is a next step to make the type checker reacts on CRD changes, which handles the case when the CRD schema is late for the type checking.
i think users might be confused by this behavior (the same way people on this thread are) even if it's documented in a KEP, so a field comment update feels worth it, at minimum.
The current type checking is informational and only works on ValidatingAdmissionPolicy. If the CRD is created after type checking or CRD created but not showing in schema watcher yet, the error would be ignored. And it will not retry as long as ValidatingAdmissionPolicy remains unchanged. There is a PR for adding retry with schema resolver raised in which will solve the issue entirely. In current case, the guess would be schema watcher is slow so it didn't have crd info yet when performing typechecking. The temporary fix by in will resolve the test flaky by adding manually retry. The field comment of makes sense to me since it indicates the completion of the type checking but not handling the case where the CRD has not been created or created but not showing in schema watcher yet. FYI The says I agree it might not fit in conformance test. Not sure about the conformance test criteria.
cc
/triage accepted


Question: JumpChain: "ACCEPT", MatchSet: kubeHealthCheckNodePortSet, }}, string(kubeServicesChain): {{ <ins> JumpChain: "RETURN", SourceAddress: "127.0.0.0/8", }, { </ins> JumpChain: string(kubeMarkMasqChain), MatchSet: kubeClusterIPSet, }, { JumpChain: string(kubeNodePortChain), MatchSet: "",

Loopback addresses (127.0.0.0/8, ::1/128) can't be used by services in proxy-mode=ipvs it makes sense to dismiss those early, add RETURN rule to KUBE-NODE-PORT,KUBE-SERVICES No response <details</details25 <details</detailsnone <details</details<details</details<details</details<details</details
/sig network
As mentioned in , this is not a bug since the nodeport and ephemeral ranges overlap.
/kind cleanup /kind remove-bug
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /kind cleanup /kind remove-bug </details
/remove-kind bug
The title of this issue should be changed to something like; Which would indicate that this is not a bug fix but an improvement. Also remove the invalid use-case in the description and add references to comments in like I think to dismiss nodePort connects to localhost early for proxy-mode=ipvs is an improvement, and in the majority of systems traffic to localhost is more common than traffic to nodePorts so this change will also improve overall performance. This is not a fix for overlapping ephemeral and nodePort ranges! So if the title and description is not updated the correct triage is to close this issue as it violates the non overlapping rule for ephemeral/nodeport ranges, please see
/area ipvs
Ok, i will change issue title and descripiton
/triage accepted


Question: func (p *Plugin) InspectFeatureGates(featureGates featuregate.FeatureGate) { p.expansionRecoveryEnabled = featureGates.Enabled(features.RecoverVolumeExpansionFailure) p.dynamicResourceAllocationEnabled = featureGates.Enabled(features.DynamicResourceAllocation) <del> p.kubeletCSRAdmissionValidationDisabled = featureGates.Enabled(features.DisableKubeletCSRAdmissionValidation) </del> <ins> p.allowInsecureKubeletCertificateSigningRequests = featureGates.Enabled(features.AllowInsecureKubeletCertificateSigningRequests) </ins> } // SetExternalKubeInformerFactory registers an informer factory into Plugin

This comes from We've a feature gate, . However, we try to name feature gates after the positive thing they add, even if it's on by default, or even if we're setting if false because the positive thing is being dropped (rare, but it happens). Typically we don't start feature gate names with , as it makes the docs jar: Example quote 1: To temporarily revert to the legacy behavior, enable the feature gate. vs Example quote 2: You can enable the feature gate if you run code that relies on the previous behavior. You should plan to migrate. and because disabling things isn't really a feature. Some examples of gates you can or once could disable by turning them off_ include: - - - So, I recommend a rename of the gate before the v1.31 release. /sig auth architecture
FYI
Example gate names: (off by default) (off by default) (on by default) (on by default) (on by default) from (off by default, deprecated)
/milestone v1.31
You must be a member of the GitHub team to set the milestone. If you believe you should be able to issue the /milestone command, please contact your Milestone Maintainers Team and have them propose you as an additional delegate for this responsibility. <detailsIn response to : /milestone v1.31 Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Another possibility: , deprecated, off by default
Hello there! I am new to kubernetes and I want to contribute in it. Is it something that I can contribute to?
This issue is not a good issue for a first-time contributor to work on.
I'm a fan of , deprecated, off by default
hoping one of you are doing the PR? :) /milestone v1.31


Question: before running <del> `kube init --config <path_to_configuration_file>` </del> <ins> `kubeadm init --config <path_to_configuration_file>` </ins> to specify the ipvs mode before deploying the cluster.

says: It likely means not , but
/sig network /good-first-issue Thanks! TODO: Change to
This request has been marked as suitable for new contributors. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /sig network /good-first-issue Thanks! Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/unassign
/triage accepted


Question: // of StartRecordingToSink. This lets you also process events in a custom way (e.g. in tests). // NOTE: events received on your eventHandler should be copied before being used. // TODO: figure out if this can be removed. <del> StartEventWatcher(eventHandler func(event runtime.Object)) func() </del> <ins> StartEventWatcher(eventHandler func(event runtime.Object)) (func(), error) </ins> // StartStructuredLogging starts sending events received from this EventBroadcaster to the structured // logging function. The return value can be ignored or used to stop recording, if desired.

Follow-up cleanup work for: Most of the information about this issue can be found in , where the work merged with a to change the function signature of to return a func() as well as an error. This issue will help to track that. All TODOs introduced in are resolved. N/A N/A <detailsN/A </details<detailsN/A </details<details</details<details</details<details</details<details</details
/sig api-machinery
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /priority low Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc /triage accepted
/priority important-longterm
/assign
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale


Question: - piosz - jsafrane - jbeda <ins> vendor-reviewers: - thockin vendor-approvers: - thockin </ins>

/assign /cc Please ensure that every sub directory of / contains an owners file. The following sub directories do not define them: The result of this is that any PR which changes anything in api, Godeps, staging, vendor, etc will always get assigned to one of you for both the review and approval.
I recall someone also mentioning that certain files or dirs may not need approvers files. If that would help, we can add that functionality (i've already started the work)
staging and api should be easy to find owners for. may have ideas for translations. The big one is Godeps, third_party, vendor. Those are just grunt work. And figuring out criteria for approving is hard as they are a lot of code and it is unclear that things are tested well or automatically. (I'm thinking things like libraries for accessing cloud APIs). Having a dedicated team/process there might help.
I'll open some proposals for staging and API.
Godep in particular needs to be watched for things like licenses, which I have mostly been doing.
Thanks Clayton for the staging and api additions! How about if we pick two of you at random and shove them in as owners for each of the remaining directories?
What's left: translations: the kubectl specific translations is being handled by sig-cli, but the rest of the files like README and test have been handled and approved mostly by thirdparty: this is a tricky mix of stuff like forked code.. mostly handled by the likes of or for approvals. Godep/vendor: These could probably be solved by having a process to follow, and then delegating to a team in OWNERSALIASES for both folders.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Still relevant. /remove-lifecycle stale


Question: } if s.BootstrapToken != nil { <del> fs.BoolVar(&s.BootstrapToken.Allow, "experimental-bootstrap-token-auth", s.BootstrapToken.Allow, ""+ </del> <ins> fs.BoolVar(&s.BootstrapToken.Enable, "experimental-bootstrap-token-auth", s.BootstrapToken.Enable, ""+ "Deprecated (use --enable-bootstrap-token-auth).") fs.MarkDeprecated("experimental-bootstrap-token-auth", "use --enable-bootstrap-token-auth instead.") fs.BoolVar(&s.BootstrapToken.Enable, "enable-bootstrap-token-auth", s.BootstrapToken.Enable, ""+ </ins> "Enable to allow secrets of type 'bootstrap.kubernetes.io/token' in the 'kube-system' "+ "namespace to be used for TLS bootstrapping authentication.") }

/kind feature As part of , the flag on the API server should be renamed to something like . We'll leave deprecated (should it be a no-op or not?) for v1.8. Or we should consider adding an explicit flag to the API server in the same way we have  flag on controller-manager. Right now, it enables/disables most authn plugins implicitely based on other flags (often pointing to various files on disk; if you specified a path to that file, you also want to enable this authn module) What way do you prefer?
/assign
Rough plan: (1.8) Add (1.8) Make it an error to specify both and (so we don't have to decide on precedence) (1.8) Add a warning if is used. (1.9) Drop
is: Mark deprecated. cc Approve?
/reopen Re-opening because there's a little bit left to do in 1.9 (ready to go in ).
[MILESTONENOTIFIER] Milestone Labels Incomplete Action required: This issue requires label changes. If the required changes are not made within 6 days, the issue will be moved out of the v1.8 milestone. priority: Must specify exactly one of [, , ]. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details


Question: // Shouldn't happen; log so we know if it is glog.Warningf("Found multiple attachments for volume %q: %v", d.awsID, info) } <del> var attachment *ec2.VolumeAttachment </del> attachmentStatus := "" for _, a := range info.Attachments { if attachmentStatus != "" {

In a cloud provider environment it's possible to use up the API quota grinding the cluster to a halt: In a multi tenant environment, one user could consume the quota for an entire cluster. Some cloud providers allow throttling of API calls by type but Kube does not differentiate on the call types, so DOS still possible.
Adding some colors. Cloud providers like AWS/GCE/Azure have rate limits on API calls. GCE and Azure have fixed API calls per second, while AWS doesn’t offer fixed quota. Once API call quota are exhausted, no more API calls are allowed. Consequences include: If the error code path are not well tested, unexpected behavior will happen. A malicious user may launch DoS by triggering frequent API calls. The entire cluster would be impacted AWS cloudprovider has implemented API throttling at global level as well as on per call type level most recently. However, at least on AWS, quota for call types are different: query type API calls have higher quota while modification type API calls have lower quota. There should be strategies to differentiate these calls. Quota for Cloudprovider API calls should be supported on per account/project basis, so DoS can be averted. (Per This needs significant enhancements in Attach/Detach logic, volume plugin gets only pv.Spec, which does not include information about namespace of the PVC bound to it nor the user that initiated the operation. And only volume plugin (on top of cloud provider) can throttle API calls.) Throttling at per call type level should be supported.
I understand there is a problem, but I am not convinced on any particular solution. Can we get a more detailed enumeration on the code paths that invoke cloud provider backed interfaces, and where we would want some bucket to control rate for those calls? I don't think the existing object should bleed concepts from the underlying cloud provider, and I don't think its setup to support rate based quota models at this time.
/cc
I went through storage parts of the cloud provider and on AWS it looks like this: Deleter (in persistent volume controller) calls ec2.DeleteVolume() once per delete attempt Provisioner (in persistent volume controller) calls ec2.DescribeInstances() with filter "instance-state-name" == "running" to get all zones that contain a node. Once per provisioning attempt.

calls ec2.CreateVolume() once per provisioning attempt calls ec2.CreateTags() - up to 60 times per provisioning call (until it succeeds). Typically one attempt needed. calls ec2.DescribeVolumes() once per provisioning attempt When provisioning or deletion of a volume fails, the controller re-tries every 15 seconds until it succeeds. Typically it succeeds on the first attempt. PersistentVolume admission plugin: ec2.DescribeVolumes() once per new PV instance created (to fill its labels) Attach (in attach/detach controller) ec2.DescribeInstances() with filterprivate-dns-name" = nodeName once per attach attempt (to get list of already attached volumes) ec2.AttachVolume() once per attempt ec2.DescribeVolumes() with filter volumeId = volume every 10 seconds until it's attached Detach (in attach/detach controller) ec2.DescribeInstances() with filterprivate-dns-name" = nodeName once per attempt ec2.DetachVolume once per attempt ec2.DescribeVolumes() with filter volumeId = volume every 10 seconds until it's detached please confirm how often the controller re-tries on failure. Is it ? With many volumes being attached/detached at the same time, each volume polls its own DescribeVolumes() every 10 seconds while waiting for the operation to finish. We could save a lot of calls if we had one DescribeVolumes() call for all "interesting" volumes and distribute the results to proper places. In addition, some exponential backoff would be helpful, AWS is pretty slow when attaching/detaching volumes. Only storage was investigated!!!
The same issue also applies to ELB on AWS. Service controller invokes which calls:
It should be noted exponential backoff is disabled in - , I am not sure if it was intentional. Also, we should consider case where k8s cluster is in, inconsistent state. For example - if volume being deleted is still mounted to a pod, there would be a flood of API requests for deleting volume in pvcontroller (without any exponential backoff)
You're right, PV controller does not use retry logic of GoRoutineMap and retries provisioning and deletion on its own and currently without exponential backoff. The reason is that PV controller can react to deletion of object, e.g. when user removes a PVC then the controller does not need to retry provisioning a volume for it.

PV controller retries failed operation every 15 seconds by default. Some exponential backoff could be if 15 seconds are too short.
If Attach or Detach operation fails (e.g. due to throttling), there already is exponential backoff, starting with 0.5 seconds and doubling with each failed attempt, capped at 2 minutes. Summary: there already is exponential backoff for Attach/DetachVolume with we will have the same exponential backoff for Create/DeleteVolume the only fixed is DescribeVolumes() when we wait for a volume to get really attached/detached. I am going to fix it.


Question: fi fi cat > "${config_path}" <<EOF <ins> version = 2 # Kubernetes requires the cri plugin. required_plugins = ["io.containerd.grpc.v1.cri"] </ins> # Kubernetes doesn't use containerd restart manager. <del> disabled_plugins = ["restart"] </del> <ins> disabled_plugins = ["io.containerd.internal.v1.restart"] </ins> oom_score = -999 [debug] level = "${CONTAINERD_LOG_LEVEL:-"info"}" <del> [plugins.cri] </del> <ins> [plugins."io.containerd.grpc.v1.cri"] </ins> stream_server_address = "127.0.0.1" max_container_log_line_size = ${CONTAINERD_MAX_CONTAINER_LOG_LINE:-262144} <del> [plugins.cri.cni] </del> <ins> [plugins."io.containerd.grpc.v1.cri".cni] </ins> bin_dir = "${KUBE_HOME}/bin" conf_dir = "/etc/cni/net.d" conf_template = "${cni_template_path}" <del> [plugins.cri.registry.mirrors."docker.io"] </del> <ins> [plugins."io.containerd.grpc.v1.cri".containerd] default_runtime_name = "runc" [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc] runtime_type = "io.containerd.runc.v2" [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"] </ins> endpoint = ["https://mirror.gcr.io","https://registry-1.docker.io"] EOF

Both jobs are failing with the same error message. pull-kubernetes-e2e-gce-ubuntu-containerd gce-ubuntu-master-containerd Tests are not run as clusters fail to come up. gce-ubuntu-master-containerd: Cluster creations failures started 2021/04/09 05:09 AM PST. pull-kubernetes-e2e-gce-ubuntu-containerd: Cluster creation failures started around 2021/04/08 3:15 AM PST Cluster creation script fails
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig testing
I0410 20:45:23. ] "Receiving a new pod" pod="kube-system/etcd-server-e2e---master" I0410 20:45:23. ] "Receiving a new pod" pod="kube-system/kube-apiserver-e2e---master" I0410 20:45:23. ] "Receiving a new pod" pod="kube-system/kube-controller-manager-e2e---master" E0410 20:45:23. ] "Version from runtime service failed" err="rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService" E0410 20:45:23. ] "Get runtime version failed" err="get remote runtime typed version failed: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService" E0410 20:45:23. ] "Failed to run kubelet" err="failed to run Kubelet: failed to create kubelet: get remote runtime typed version failed: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService" kubelet.service: Main process exited, code=exited, status=1/FAILURE kubelet.service: Failed with result 'exit-code'.
Looks like infra-commit is related PR: /cc
fix is going in here


Question: func (f *Fit) EventsToRegister() []framework.ClusterEvent { return []framework.ClusterEvent{ {Resource: framework.Pod, ActionType: framework.Delete}, <del> {Resource: framework.Node, ActionType: framework.Add | framework.UpdateNodeAllocatable}, </del> <ins> {Resource: framework.Node, ActionType: framework.Add | framework.Update}, </ins> } }

We have an e2e test that involves cluster autoscaler; the test creates two pods back to back which forces CA to scale up the cluster to create two nodes also back to back one for each pod. The two nodes are not created at the same time, there is some delay between them. The problem is that the second pod never gets scheduled. We see in the logs a single scheduling attempt for the second pod that attempted to schedule it on the first node and failed because it didn't fit the node's remaining cpu cores. We don't see any attempts after that for the pod and the test times out waiting for that pod to schedule. What changed in the scheduler: Pre 1.22, on most update/add/delete events to pods/nodes/pvc/storageClass resources, we used to move all unschedulable pods back to the queue. Also every we did force move all unschedulable pods back to the queue indiscriminately. This behavior is not efficient, wasted scheduling cycles and since we re-queue based on last scheduling attempt timestamp, it missed up badly with the best effort FIFO behavior we intend to have for pod scheduling.Since 1.22 a couple of things changed such that unschedulable pods are not placed back into the queue on those events indiscriminately; on each such event, we iterate over unschedulable pods and we place them back into the queue if they pass two checks: the first one happens only on node update/add events, it is carried by the function ; it basically runs the kubelet admission logic (node resource, affinity, taints/tolerations and host port checks) the second check runs on all events (including the node ones); this one is a little more complicated and fine grained: for each unschedulable pod we remember which plugins last failed to fit the pod; at the same time plugins register which events could make an unschedulable pod schedulable (e.g. nodeResources filter plugin registers pod delete and node add/update events). A pod is put back into the queue only if the event matches any of the ones registered by the plugins that last failed the pod.

In 1.24 we increased the 60s to (our north star to completely remove this) What went wrong: My theory is that we incorrectly determined that the pod was not eligible to be put back into the queue on events related to the new node; and that the 5min timeout was too long to force moving it back to the queue (I am guessing the whole test timed out), while the 60s was basically masking the problem for 1.22 and 1.23. So what checks are incorrectly failing to move the pod? Here is what happened: Before adding the new node, the pod failed the nodeResource plugin, which only registers for New node was , scheduler receives an add event. The pod was not moved back to the queue because it failed the check on NotReady taint. When the new node became ready, the scheduler receives a node update event, but since it is a taint update event, it didn't match the events that the nodeResources plugin registered for. How to fix it: Since we have the ; pods that previously failed because of NodeResources, NodePorts, TaintsAndTolerations and NodeAffinity (the predicates we already ran in ) should always be placed back into the queue; basically those plugins should register for all node update events. /sig scheduling /cc The second pod to schedule. See above. No response 24 GKE <details</details<details</details<details</details<details</details
/triage accepted


Question: // This is a snapshot of expected options parsed by args. expected := &ServerRunOptions{ ServiceNodePortRange:   kubeoptions.DefaultServiceNodePortRange, <ins> ServiceClusterIPRange:  kubeoptions.DefaultServiceIPCIDR, </ins> MasterCount:            5, EndpointReconcilerType: string(reconcilers.MasterCountReconcilerType), AllowPrivileged:        false,

Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: There is currently no easy way to know what Kubernetes is using as the default CIDR for service cluster IPs. What you expected to happen: should be able to tell users that "10.0.0.0/24" is used as the default. They'd better check if their deployment environment has a conflict with this assumption. They need to customize this parameter to avoid such overlaps if any. How to reproduce it (as minimally and precisely as possible): Run , from the help, the default value for --service-cluster-ip-range is not mentioned. Anything else we need to know?: No. Environment: Kubernetes version (use ): Any Cloud provider or hardware configuration**: Any OS (e.g. from /etc/os-release): Any Kernel (e.g. ): Any Install tools: N/A Others: N/A
/sig api-machinery
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale


Question: vols, vErrs := ValidateVolumes(spec.Volumes, fldPath.Child("volumes")) allErrs = append(allErrs, vErrs...) <del> allErrs = append(allErrs, validateContainers(spec.Containers, vols, fldPath.Child("containers"))...) </del> <ins> allErrs = append(allErrs, validateContainers(spec.Containers, false, vols, fldPath.Child("containers"))...) </ins> allErrs = append(allErrs, validateInitContainers(spec.InitContainers, spec.Containers, vols, fldPath.Child("initContainers"))...) allErrs = append(allErrs, validateRestartPolicy(&spec.RestartPolicy, fldPath.Child("restartPolicy"))...) allErrs = append(allErrs, validateDNSPolicy(&spec.DNSPolicy, fldPath.Child("dnsPolicy"))...)

I found that kubelet will not allocate hostports specified from initContainers. And PodFitsHostPorts in kube-scheduler does not take initContainer into consideration. Also, kube-apiserver think such pod is illegal, but I think it OK. We should make the behavior of spec.HostPorts in initContainer clear
/sig node
Exactly. I think this makes sense. Since initContainers are expected to be completed, we should not expose extra port mappings during creating pod sandbox, which may bring in security risks.
Sounds good to me.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: "k8s.io/apimachinery/pkg/util/sets" "k8s.io/apimachinery/pkg/util/wait" "k8s.io/apiserver/pkg/authentication/authenticator" <ins> "k8s.io/apiserver/pkg/authentication/group" </ins> "k8s.io/apiserver/pkg/authentication/request/bearertoken" "k8s.io/apiserver/pkg/authentication/request/union" serviceaccountapiserver "k8s.io/apiserver/pkg/authentication/serviceaccount"

While moving APF to beta (see comments on ), we discovered that the integration tests commonly present users which are not members of any groups. We had thought that the system enforced membership in either the authenticated or unauthenticated group. If this is supposed to be a system invariant (as we thought), we'd like to ask SIG Auth: Please fix the integration tests Please add e.g. logging so that if real, operational clusters hit this there's at least a log message and people will file bugs. /sig auth
Can I suggest also, if this is a real invariant, that it be documented in some place(s) likely to be read?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale /priority important-longterm /triage accepted
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/lifecycle frozen
fixes the authenticator configured in one of the integration test API server startup helpers which resolved this issue for . Remaining tests that still need fixing that are encountering unexpected users look to be ones set up to specifically test auth paths:
open to fix remaining integration test users


Question: limitRange := validLimitRangeNoDefaults() limitRanges := []corev1.LimitRange{limitRange} <del> var count int64 </del> mockClient := &fake.Clientset{} <del> unhold := make(chan struct{}) </del> <ins> var ( testCount  int64 test1Count int64 ) </ins> mockClient.AddReactor("list", "limitranges", func(action core.Action) (bool, runtime.Object, error) { <del> atomic.AddInt64(&count, 1) </del> <ins> switch action.GetNamespace() { case "test": atomic.AddInt64(&testCount, 1) case "test1": atomic.AddInt64(&test1Count, 1) default: t.Error("unexpected namespace") } </ins> limitRangeList := &corev1.LimitRangeList{ ListMeta: metav1.ListMeta{

pull-kubernetes-unit /cc /sig scalability
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
In local tests, he always passes, so I'll see if the problem is caused by the environment
If the lru cache exceeds 30 seconds, the cache will be invalidated, which will lead to unit test errors, and the problem will run normally again after retesting.
After I manually reduce an unhold <- struct{}{}, the same problem occurs, so it seems that the problem is caused by the cache being invalidated for more than 30 seconds. If the test doesn't complete in 30 seconds, then this is probably a machine performance issue
stress will hang after about a minute in my local test.
/cc
The issue is being fixed


Question: <ins> /* Copyright 2024 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package inuserns import libcontaineruserns "github.com/opencontainers/runc/libcontainer/userns" // RunningInUserNS detects whether the current process is running in a user namespace. func RunningInUserNS() bool { return libcontaineruserns.RunningInUserNS() } </ins>

is no longer compilable on macOS in the master branch. It was compilable with Kubernetes v1.29.0 and v1.30.0-alpha.0. It should be compilable. No response <detailsmaster () (This is just the latest commit, not the commit that introduced the regression) </details<detailsN/A </details<detailsmacOS 14.2.1 (Intel) </details<detailsN/A </details<detailsN/A </details<detailsN/A </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig testing
Regression in:


Question: { // De-authorize the load balancer security group from the instances security group <del> err = c.updateInstanceSecurityGroupsForLoadBalancer(lb, nil) </del> <ins> err = c.updateInstanceSecurityGroupsForLoadBalancer(lb, nil, service.Annotations) </ins> if err != nil { klog.Errorf("Error deregistering load balancer from instance security groups: %q", err) return err

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When multiple securityGroup are specified for AWS ELB type service, a random one will be chosen to grant inbound access to worker node. What you expected to happen: Always first LoadBalancer securityGroup be chosen to access worker nodes. How to reproduce it (as minimally and precisely as possible): use or on aws elb. Anything else we need to know?: Environment: Kubernetes version (use ): <=1.18 Cloud provider or hardware configuration: aws OS (e.g: ): linux Kernel (e.g. ): linux Install tools: kops/eks Network plugin and version (if this is a network-related bug): Others:
/sig cloud-provider These SIGs are my best guesses for this issue. Please comment if I am incorrect about one. I am a bot run by vllry.

What happened: We were running a service with LoadBalancer type with AWS cloud provider and after deleting this service, the cleanup process on AWS removed a wrong ingress input rule which provoked that all traffic between nodes was unable to reach out the destination. The service we were running had a configuration similar to: After applying this manfiest on the cluster, the cloud provider created the associated load balancer with two security groups: sg- (due to extra security group annotation) sg- (a new SG created by the AWS cloud provider with the rules defined on the ) Cloud provider also created a new input rule on the nodes security group which allows all traffic from all ports from (which is the sg created for the newly created load balancer in our example). One particularity of our setup is that the security group we as was the nodes security group. The input rules of this security group after the creation of this service was similar to: Everything worked fine with this setup until we deleted this service and the cloud provider started the deletion process of the LoadBalancer on AWS. We've been debugging the deletion code and we found that the first thing it does is cleaning up the security group. To do it, it first removes the input rule from the nodes security group which allows traffic from the load balancer security group (sg-, on this example) and then, as it's not referenced, it removes the security group. The expected output of this process, would be that nodes security groups would be: But, in our case, it ended up as: which dropped all the traffic between nodes (as it didn't allow any traffic between them). The cleanup process didn't properly remove the input rule from the node security group (the one which allows all traffic from source sg-) and it also did not remove the sg- security group. In our case, as we were using the nodes security group on the extra security groups, this bug provoked a network partition between nodes, but we think that if is used, the cleanup process may be broken and have unexpected side-effects like those we observed. We've been debugging and we found that the process which removes the input rule from the nodes security groups starts here: The first thing this function does is this: : When extra security groups are used, will contain the security group created for the LoadBalancer and all the extra security groups.

We also found that AWS returns this SecurityGroups list in distinct orders after changing the LoadBalancer configuration. That code is always assuming that latest security group on the list is the , which may not be true when extra security groups are used. On the cases where the load balancer security group is not returned on the latest position, the cleanup process does not remove the security group and the input rule from the nodes security group is not removed as expected. We've seen that master branch still contains this code and we think this may be hapenning on newer versions: ), so we think that newer versions may be affected by this bug (we didn't try it). In our case, we saw several times this log during the removal process: Additionally, the code tried to remove the security group (but being unable to do so), after wrongly revoking the rule we have talked about before. What you expected to happen: The cleanup process should properly detect the load balancer security group, remove the input rule from nodes security group and remove the load balancer security group, regardless of the order in which AWS returns the security group list on . How to reproduce it (as minimally and precisely as possible): We did not find any consistent process to create the scenario, as it happens randomly (we are not sure if due to AWS or any kind of race condition on AWS cloud provider). Setup: Create a Service with running controller manager with AWS cloud provider with tha annotation containing one or more security groups (as the one the section) Wait until load balancer is created Run Check if does not have the newly created security group for the balancer in the latest position. Repeat until the balancer does not have the balancer security group on the latest position Trigger the bug: Result:The log will appear multiple times on controller-manager logs Load balancer associated security group is not removed Nodes security group input rule allowing all traffic from load balancer security group is not removed If the latest security group on the list was the nodes security group, the input rule allowing all traffic on all ports from all the other nodes is removed and no traffic is allowed between nodes on the cluster.

Environment: Kubernetes version (use ): version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.10", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-05-01T04:05:01Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: aws OS (e.g: ):Kernel (e.g. ): Linux ip-10-2-13-67 4.9.0-7-amd64 SMP Debian 4.9.110-3+deb9u2 (2018-08-13) x8664 GNU/Linux Install tools: kops
/sig cloud-provider
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/area provider/aws /lifecycle frozen
Working on a repro for this issue (on 1.11 first, then I'll try on master) and I've not quite observed the same behavior yet. Depending on the order of the security groups on the load balancer, I only see the ingress rule for the new LB security group to the node security group if the new security group comes last. Otherwise, if the NODE security group comes last, the ingress rule is not , and I see in the logs. Even without a full repro yet, I find it troubling that behavior is dependent on the order of security groups attached to the load balancer. One question I have related to your setup, can you describe the reason why you're adding the node security group as an ? I'd like to understand your use case a little better.
Hi Nicholas. I have a working patch for this specific issue: This bug was the subject of a talk for a local meetup I did recently.

I also published a personal blog post talking about it: There you can see more details about how it can be reproduced easily. If you think the patch is valid, I'll try to add a few test cases for it and then propose the change upstream. Thanks!
It has been a while since we got this error and I'm not sure if we saw this error. What I experienced is that the order is changed by Kubernetes or by some call to AWS. It took me not more than 3 attempts to get this behavior. There's something I can do to help you reproduce the error? Do you need more details of our setup? This was a misconfiguration on our side. When we noticed that this setup was not correct we removed this security group from , which removed the mentioned security group rule and took down the Kubernetes cluster (no communication between nodes was allowed).


Question: <ins> apiVersion: v1beta3 kind: Pod metadata: labels: example: logging-sidecar name: logging-sidecar-example spec: containers: - name: synthetic-logger image: ubuntu:14.04 command: ["bash", "-c", "i="0"; while true; do echo "`hostname`: $i " >> /var/log/synthetic-count.log; date --rfc-3339 ns >> /var/log/synthetic-dates.log; sleep 4; i=$[$i+1]; done"] volumeMounts: - name: log-storage mountPath: /var/log - name: sidecar-log-collector image: gcr.io/google_containers/fluentd-sidecar-gcp:1.0 env: - name: FILES_TO_COLLECT value: "/mnt/log/synthetic-count.log /mnt/log/synthetic-dates.log" volumeMounts: - name: log-storage readOnly: true mountPath: /mnt/log volumes: - name: log-storage emptyDir: {} </ins>

We should provide a dockerfile for a pre-configured Fluentd image that can be passed file paths to collect logs from. We'll likely want one for exporting to elasticsearch and one for exporting to Google Cloud Logging, as we have today for the per-node stdout/stderr collectors. How to collect log files from within a container is a question that comes up occasionally (including ), and it provides a strong example of the usefulness of sidecar containers. cc
A controller that keeps one instance of a pod alive on each node would be useful for this. Shall I go ahead and write such a controller?
Would it be? This is intended to run as a sidecar container in a pod with the user's actual application. I think a pod-per-node controller is orthogonal (and is being discussed in and ). I'll create the actual Dockerfile, and a quick example of how to use it.
Sorry, my rush to catch up I think I mis-understood this thread.


Question: glog.ErrorDepth(2, err) } <ins> type rudimentaryErrorBackoff struct { minPeriod time.Duration // immutable // TODO(lavalamp): use the clock for testability. Need to move that // package for that to be accessible here. lastErrorTimeLock sync.Mutex lastErrorTime     time.Time } // OnError will block if it is called more often than the embedded period time. // This will prevent overly tight hot error loops. func (r *rudimentaryErrorBackoff) OnError(error) { r.lastErrorTimeLock.Lock() defer r.lastErrorTimeLock.Unlock() d := time.Since(r.lastErrorTime) if d < r.minPeriod { time.Sleep(r.minPeriod - d) } r.lastErrorTime = time.Now() } </ins> // GetCaller returns the caller of the function that calls it. func GetCaller() string { var pc [1]uintptr

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): TPR. GC. OwnerReference. REST mapping. Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT. <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): master. v1.5.1 Environment: Cloud provider or hardware configuration: Repro in "hack/local-up-". But I have seen it in many different env. OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: What happened: The controller manager is unresponsive and overloaded with error message: This error just keeps bloating up without any backoff. This seems that REST mapping didn't recognize the kind. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Generic way: Create a TPR. Create a TPR object "A" Create pods that has OwnerReference pointing to "A". Easy way: Use kubectl create -f kubectl create -f example/example-etcd- Anything else do we need to know:
well, the garbage collector still uses a static RESTMapper(), and in this case, TPRs aren't recognized at all.
Not recognizing is fine and expected. But controller manager keeps on printing out error message without backing off. Can you confirm this is unintended? Or we are missing anything here?
Right. I should have mentioned it: the controller manager is unresponsive and overloaded with error message. Updated in title and top comment.

there are two things that need fixing here: garbage collector should use a dynamic restmapper that picks up types the server knows about, not just compiled-in types a. this means, at minimum, discovering types at start-up b. it also means refreshing at some interval cases where an ownerRef references a type that is not found by the garbage collector's restmapper, it should not panic or hotloop. I'm not really sure what the behavior should be, actually should it treat owners it can't look up as if they were deleted, or ignore ownerRefs for unknown kinds?
If a user runs kubernetes and sees messages like "ignore XXX ownerRef due to unknown kind", he can't really do anything with it. It would be great to recognize the kind, even if there is a delay.
ownerref could have bad data, or it could reference a kind that is no longer part of the cluster. need to define what happens in those cases. when it is referencing a dynamic kind (either a kind contributed by an add-on server or by a thirdpartyresource), yes, it should re-discover on some interval to recognize the kind
I have the same issue when launching an etcd cluster with the coreos etcd-operator. If causes the api server to run out of disk space as it logs infinitely. I have to keep restart the controller container to reclaim disk space. From a security point of view this means a tenant of a cluster can bring the whole cluster down.
Why the target fix is in 1.6 not a patch release for 1.5?
I would not anticipate the garbage collector actually enabling support for TPRs in 1.5, but the bug that causes it to become unresponsive when it sees an ownerRef of an unknown type should be fixed
Changed the milestone to 1.5
I agree. I do not expect the gc to work with TPR either. But it should not "kill" controller manager. And it is the issue we should fix in 1.5. We can make GC work for TPR in 1.6.
For me this is currently a DOS attack
Repro instructions on master in GCE:
so you don't even need to have something with an owner ref pointing at the TPR to trigger this?

I would guess an ownerRef pointing to any arbitrary unknown kind would trigger the same behavior
the etcd operator creates etcd pods with: I suspect is correct.
Ah, the operator triggers it. OK, I see.
etcd operator does not rely on the k8s GC features (it has internal GC). But it does write owner ref. when k8s gc works for tpr, we will remove our temp internal GC. but now the owner ref causes controller manager to spam.
shall we reopen? i do not think this is fixed.
Yeah this just happened to me. Does not seem fixed.
what version are you running? This isn't in a release version yet.
"masks" the problem, not fixes it. We should close this issue when the fix lands.
I just installed with kubeadm two days ago. I believe is 1.5.2. I then tried using etcd-operator and it killed my master with out of disk space writing 300MB/s of log from GC
a fix will be in the 1.5.3 release. It was just merged and hasn't had time to make it into a release. Sure, not arguing that this issue should be closed. I do think is probably good enough for the 1.4, 1.5 branches.
I'll do it after the refactor of GC, , lands. makes gc use the rate limited workqueue, so it will prevents the hotloop. [edit] GC treats unresolveable ownerRefs as though they perpetually exist. That is, an object with an unresolvable ownerRef will not be deleted by GC. It is meant to be a safety net. For example, during a cluster upgrades, if the APIVersion of replication controllers bumps from "v1" to "v2", and assuming all the pods' ownerRefs still have "v1", if GC treats those refs as if the owners were deleted, then all the pods will be accidentally deleted.
Does GC only treat the preferred version as a legitimate ownerRef, or all known versions, or was that just an example?
I guess there is a third option as well, treat unresolveable ownerRefs as though they perpetually exist (which is different from ignoring them).
All known versions.

What I mean in the example is that assuming v1/replicatoncontroller is not supported by the API server anymore.
Actually this is the current case. GC doesn't delete an object with unresolvable ownerRef. I edited my previous comment.
40Gb per day of logs to this mess for us... So we would love a fix too.
is close to get in. It will fix the hot loops. We still need a dynamic restmapper. I'll start a fix after lands.
FWIW, here's the patch I wrote to get back to a more reasonable behavior: (then, rebuild your controller-manager or hyperkube)
Thanks changed gc to use the ratelimited queue, so the hotloop should have been fixed on the master branch. [edit] That said, we still need to make the restmapper dynamic, i'm working on a fix.
My patch was done on the branch, should anyone prefer to avoid upgrading to alpha.0 :)
BTW, is there a backport to 1.5 planned? is flagged so, should a smaller patch be needed, feel free to use this one of course.
Update: I tried to reuse to dynamically translate TPR, but it didn't seem to work because of I'll find a workaround.
what version are you on? said was landed in v1.5.3, so the problems should be masked. Is that true?
I had to overwrite hyperkube in the docker image
Ahhh it has this commit which says: Adjust global log limit to 1ms So probably not a difference a human can detect.
I see. 1 ms is still too spamming.
At least, it make the log unusable without filtering ;) I have a global feeling that Kubernetes log system could be improved but I still need to have a better of that. The global problem to solve is probably the repetition of log, something that's solved by events (or at the way kubectl shows them); so maybe I should move my practice more to events and, when I don't have the answer in the events, file an issue/PR.
the backoff removed the log flooding, right? if so, I think this can be moved to 1.7
c.f.
Thanks You are right, this is not a 1.6 blocker. makes sure the log message is only printed once per un-resolvable ownerReference.

I think the original issue of "spamming error messages" was solved. I created to track the issue of "GC should support non-core API". Closing this issue.


Question: #  CFSSLJSON_BIN: The path of the installed cfssljson binary # function kube::util::ensure-cfssl { <del> echo "Installing cfssl from vendor" GOBIN="${KUBE_OUTPUT_BINPATH:-}" go install k8s.io/kubernetes/vendor/github.com/cloudflare/cfssl/cmd/cfssl GOBIN="${KUBE_OUTPUT_BINPATH:-}" go install k8s.io/kubernetes/vendor/github.com/cloudflare/cfssl/cmd/cfssljson CFSSL_BIN="$(PATH="${KUBE_OUTPUT_BINPATH:-}:${PATH}" command -v cfssl)" CFSSLJSON_BIN="$(PATH="${KUBE_OUTPUT_BINPATH:-}:${PATH}" command -v cfssljson)" if [[ ! -x ${CFSSL_BIN} || ! -x ${CFSSLJSON_BIN} ]]; then echo "Failed to install cfssl." >&2 exit 1 </del> <ins> if command -v cfssl &>/dev/null && command -v cfssljson &>/dev/null; then CFSSL_BIN=$(command -v cfssl) CFSSLJSON_BIN=$(command -v cfssljson) return 0 fi # Create a temp dir for cfssl if no directory was given local cfssldir=${1:-} if [[ -z "${cfssldir}" ]]; then kube::util::ensure-temp-dir cfssldir="${KUBE_TEMP}/cfssl" </ins> fi <ins> mkdir -p "${cfssldir}" pushd "${cfssldir}" > /dev/null echo "Unable to successfully run 'cfssl' from $PATH; downloading instead..." kernel=$(uname -s) case "${kernel}" in Linux) curl --retry 10 -L -o cfssl https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 curl --retry 10 -L -o cfssljson https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 ;; Darwin) curl --retry 10 -L -o cfssl https://pkg.cfssl.org/R1.2/cfssl_darwin-amd64 curl --retry 10 -L -o cfssljson https://pkg.cfssl.org/R1.2/cfssljson_darwin-amd64 ;; *) echo "Unknown, unsupported platform: ${kernel}." >&2 echo "Supported platforms: Linux, Darwin." >&2 exit 2 esac chmod +x cfssl || true chmod +x cfssljson || true CFSSL_BIN="${cfssldir}/cfssl" CFSSLJSON_BIN="${cfssldir}/cfssljson" if [[ ! -x ${CFSSL_BIN} || ! -x ${CFSSLJSON_BIN} ]]; then echo "Failed to download 'cfssl'. Please install cfssl and cfssljson and verify they are in $PATH." echo "Hint: export PATH=$PATH:$GOPATH/bin; go get -u github.com/cloudflare/cfssl/cmd/..." exit 1 fi popd > /dev/null </ins> } # kube::util::ensure_dockerized

<!-- This form is for bug reports ! If you're looking for help check and the . If the matter is security related, please disclose it privately via --What happened: Unable to spin up cluster using the kubetest, with 'ci/latest' image. The CI jobs are failing, because the cluster is not provisioned properly. I have tried locally, i am still getting the same error. Could be releated to this PR /cc
is your kubernetes checkout in ? the one in that CI image is a bit odd actually. cc
ohhh this is not good, kubetest --extract will of course not extract over your $GOPATH, but will then use the cluster/* scripts :man_facepalming:
i use kubetest directly, it extracts the image given in the --extract arg.
Yes
This is the full o/p from my local run.
PR sent in
I am also getting this error. My k8s is at and command that i am using is:
This should be fixed now by a revert in , sorry about that. We'll follow up with something better soon.


Question: return false, errors.Errorf("%s is not a known certificate", name) } <del> return handler.readwriter.Exists(), nil </del> <ins> return handler.readwriter.Exists() </ins> } // GetCAExpirationInfo returns CA expiration info.

When the command is run with superuser privileges, it prints the requested information. Print a warning that files could not be read due to insufficient permission, or that information was missing (e.g., when the /etc/kubernetes directory is not initialized). When the directory is readable, it works as expected: where OS and kubeadm version detail was: Issue the command as a normal user when the directory /etc/kubernetes/pki exists but not readable. No response <details</details<detailsBare metal </details<details</details<detailsapt-get install kubeadm (source repo: ) </details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cluster-lifecycle


Question: }) } } <ins> func Test_isAffinityViolatingNUMAAllocations(t *testing.T) { testsCases := []struct { description         string machineState        map[int]*state.NUMANodeState topologyHint        *topologymanager.TopologyHint isViolationExpected bool }{ { description: "violating NUMA allocations because given affinity asks for NUMA ID 1 which is on different cells group", machineState: map[int]*state.NUMANodeState{ 0: { NumberOfAssignments: 1, Cells:               []int{0, 1}, }, 1: { NumberOfAssignments: 1, Cells:               []int{0, 1}, }, 2: { NumberOfAssignments: 1, Cells:               []int{2}, }, 3: { NumberOfAssignments: 0, Cells:               []int{3}, }, }, topologyHint: &topologymanager.TopologyHint{ NUMANodeAffinity: newNUMAAffinity(1, 2), }, isViolationExpected: true, }, { description: "violating NUMA allocations because given affinity with multiple nodes asks for NUMA ID 1 which is used for a single NUMA node memory allocation", machineState: map[int]*state.NUMANodeState{ 0: { NumberOfAssignments: 0, Cells:               []int{0, 1}, }, 1: { NumberOfAssignments: 1, Cells:               []int{1}, }, }, topologyHint: &topologymanager.TopologyHint{ NUMANodeAffinity: newNUMAAffinity(0, 1), }, isViolationExpected: true, }, { description: "valid affinity, no prior assignments", machineState: map[int]*state.NUMANodeState{ 0: { NumberOfAssignments: 0, Cells:               []int{0}, }, 1: { NumberOfAssignments: 0, Cells:               []int{1}, }, }, topologyHint: &topologymanager.TopologyHint{ NUMANodeAffinity: newNUMAAffinity(0, 1), }, isViolationExpected: false, }, } for _, tc := range testsCases { t.Run(tc.description, func(t *testing.T) { if isAffinityViolatingNUMAAllocations(tc.machineState, tc.topologyHint.NUMANodeAffinity) != tc.isViolationExpected { t.Errorf("isAffinityViolatingNUMAAllocations with affinity %v expected to return %t, got %t", tc.topologyHint.NUMANodeAffinity.GetBits(), tc.isViolationExpected, !tc.isViolationExpected) } }) } } </ins>

Dual socket server with 96 threads total (2242), ~192G of RAM, cpu & memory manager static policy, topologyManagerPolicy best-effort, 10Gi of RAM reserved on NUMA node 0, 1 core (2 threads) reserved on NUMA node 0 kubeadm config: If I try to allocate 2 Guaranteed pod with 85Gi of RAM each, one pod is admitted, and the second one fails with even if it would fit using memory of both NUMA nodes. As I'm using deployments a new pod is recreated right away and you end up with tons of failed pod in state. Error in the logs is: Either the pod becomes pending, or memory from both NUMA nodes is used Have some memory reserved on NUMA 0, launch 2 identical pods with memory limits really close to the max so 1 pod fit on NUMA 1 but the second doesn't fit on NUMA 0 (the sum fits on the server) Here the If I launch 1 pod with 46 CPUs / 170Gi RAM, I get CPUs from 1 NUMA nodes, and memory from both NUMA nodes <details</detailsIf I launch a pod with 46 CPUs / 85Gi RAM, it's properly placed on NUMA 1 <details</detailsIf I try to launch 2 pods with 46 CPUs / 85Gi RAM, one fails with If I try to launch 2 pods with 46 CPUs / 80Gi RAM, everything works 26.8 NONE / bare-metal Alma 8.8 base + rpm-ostree kubeadm containerd 1.7.5 No response
/sig node
/cc Hi! Just to clarify, the memory manager can't keep the pod pending, this is a scheduler feature and builtin kube scheduler (= any of its plugin) is not NUMA-aware atm. So in this case memory manager should extend the pod to both NUMA zones, and failing to do so is likely the issue
Hello Is it possible that you are hitting the exact scenario describe at the KEP: ? Please check the first two steps.
I was going to say I'm hitting the expected unexpected error, but I'm not sure the KEP doc 100% match the current memoryManager behavior (at least when using ).

test 1: the "reject pod2 from the KEP" actually works: start pod 60 cpus / 120Gi RAM wait start pod 20 cpus / 20Gi RAM <details</detailstest 2 (start a 1 NUMA node container then one than spans 2 NUMA node) also works: start pod 20 cpus / 20Gi RAM wait start pod 60 cpus / 120Gi RAM <details</detailsWhat doesn't work is when you get too close to the limits it seems
Interesting those pods are guaranteed I presume? And what about the total node memory capacity? When you check the node status do you have enough memory left to fulfill the second pod request?
click on details you will have the content of yes they are guaranteed else they would not show up in memorymanagerstate it's less then the total node capacity else pod would be pending
/assign
assigned to me for bookkeeping reason but is doing the real work here
Hello so far I didn't manage to reproduce the issue you're describing. In my case I manage to ran both pods, or end up with the second one . However, I managed to find something similar (different affinity error) but with topology manager policy and not as you described which I'm going to open a separate issue for. Could you please share the journal logs while you're reproducing the issue again? Basically on the node where the pod is about to land, share the output
One more thing please share output of the node before and after you reproduce the issue
/triage accepted /priority important-longterm
off until Tuesday, I'll try to reproduce with 1.28.2 then and share the logs next week
NP. I won't be here next week either, so no rush
Hello did you manage to reproduce the issue?
yes, reproduced on my dev machine in 1.28.2, tell me what else you need / if you want me to rebuild kubelet with some debug that's also possible <details</details
Hi I did mange to reproduce the issue, but following the code it seems exactly like I mentioned in my previous I'll try to elaborate: When the first pod gets admitted it placed on NUMA 0 (in my case).

When the second pod gets admitted, there is not enough memory on each NUMA individually (NUMA 0 or NUMA1) to satisfy the memory request, so memory manager tries to span the memory allocation through NUMA 0 and NUMA 1. But it failed short in this line: because NUMA node 0 is now used for single-NUMA memory allocation which align with the KEP . In such case, the pod cannot move to because is done at the scheduler level, while from the scheduler's POV there is enough memory on the node to satisfy the request. Now, about your comment here: wait start pod 20 cpus / 20Gi RAM Make sense because both NUMA 0 and NUMA 1 are already serves mulit-NUMA allocation, since the first pod spans across the two NUMAs, the second pod can squeezed into one of them with no problems. wait start pod 60 cpus / 120Gi RAM This case is more interesting and I didn't manage to reproduce it - I always got which is align with the KEP. Could you please increase Kubelet logs verbosity, reproduce test 2 and attached the following information: logs file before 1 manifest file after pod1 2 manifest file after pod2 Thanks
Hi this is a different server with a bit less CPUs, but test 2 still works for me for verbosity I've used v=99 instead of v=2 Here part of my KubeletConfiguration The logs: And the commands: <details</details
I think I pinpointed a bit more the issue For my test2 pod the cpu manager generated a hint with both NUMA nodes. For test2 pod, if CPU request span both NUMA nodes, then the memory allocation will work whether it requires to use 1 or 2 NUMA nodes If CPU request only requires 1 NUMA node, then it'll fail if memory allocation need to extend to 2 NUMA node.
Hello , After carefully going through the code and the KEP I think that the behavior you're expecting: "Either the pod becomes pending, or memory from both NUMA nodes is used" is not aligned with the architecture.

Let's break it down: "The pod becomes pending" - From the default scheduler POV, the node have enough resources (because the scheduler is not NUMA aware) so the fact that the pod scheduled on this node is align with the expected behavior. Sidenote: if you want to enable NUMA aware scheduling you can take a look at "memory from both NUMA nodes is used" - Because NUMA 0 is already serves as single NUMA memory allocation, it cannot serves as multi NUMA memory allocation as well, otherwise the memory pinning is not guaranteed since linux kernel doesn't support multi-NUMA allocation (pinning). What it can do is grouping NUMA 0 with NUMA 1 into a single group but again this is against the original design: Please check and look for "No spare capacity, reject Pod 7" (last example) And you were right, when the hint generated by cpu manager, spans for both NUMAs the pod gets admitted, and that's exactly where actual bug is - the pod should never get admitted. IOW you were suppose get the error: So the fix here should be preventing the pod admission even when the CPU manager hint spans across both NUMAs. Now, in your case, if you still want and need both pods to get admitted you can change the from to because as I mentioned, the memory pinning in your scenario is not guaranteed anyway.
Hello I still think the bug is that memoryManager doesn't respect topologyManagerPolicy: best-effort. For now I'm exploring using a simple NRI plugin to set the to the nodes of
, The best-effort means that even if the affinity is not preferred, the pod would get admit anyway. But here we're talking about respecting the NUMA node grouping done by memory manager. This is an architectural design we should refer to the original creator memory manager or at least to topology maintainers folks. do you know who can answer on such questions?
some additional info about the limitation i mentioned above


Question: resultCh  chan watch.Event } <del> func newDecoratedWatcher(w watch.Interface, decorator func(runtime.Object)) *decoratedWatcher { ctx, cancel := context.WithCancel(context.Background()) </del> <ins> func newDecoratedWatcher(ctx context.Context, w watch.Interface, decorator func(runtime.Object)) *decoratedWatcher { ctx, cancel := context.WithCancel(ctx) </ins> d := &decoratedWatcher{ w:         w, decorator: decorator,

Using the Fabric8 client, which defaults to using WebSockets for Watches, I established a Watch on Service resources. After the timeout (implicit or explicit) rather than the WebSocket seeing a close control frame or even an EOF it continued to be open and fulfill ping/pong requests - but no Service events are sent. The WebSocket would close. Create a WebSocket based Watch with the ListOptions timeoutSeconds set on the Service resource along with a steady production of Service events, such as create/delete of a dummy Service. Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2020-10-27T05:15:27Z", GoVersion:"go1.15.0", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2021-01-13T13:20:00Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"} The behavior has been seen in both local minikube and OpenShift distributions based upon Kubernetes 1.20. OS (e.g: ): Fedora 33
/sig api-machinery
Is this the same issue as ?
It could be related, but in that case the client is calling close. In this case we expect the server to initiate the close.
A couple of things were discussed when we triaged this issue: a) we need to involve the Fabric8 team (~java client?) b) We don't always default to Websocket, in fact we discussed discontinuing it c) But the code is still there, so it's allowed for now and we should support it. Not sure what to do next
a) no, it sounds like an apiserver problem, it should be reproducible with a go client in a test. b) If anything, we should standardize on websockets. Brower based clients depend on them. And when/if we get rid of SPDY, websockets are the replacement. This may be relevant to interests.

I didn't see an option to use websockets with the go client the impl is at Is there another code path or go client that will use a websocket for Watches?
Technically I didn't say our go client, just a go client. I just had in mind using the raw websockets library. must be testing the websocket watches somehow?
I adapted to It will simply keep running - with no events after the timeout. If you switch to configmap, it will terminate.
that's a terrific reproducer, thanks!
Yeah, we should absolutely be able to stick that in a unit / integration test somewhere, thanks!
opened in configmap watch works correctly, service watch hits timeout without the server closing the connection
I suspect the issue is in newDecoratedWatcher. Services implement a decorator and configmaps don't.
yup, the request context wasn't propagated to the decorated watcher
Service decorator was in 1.20 in Fix with unit and integration test in Will pick to 1.21 and 1.20 once merged
/triage accepted