Question: } assert.Equal(t, expectedWindowsConfig, containerConfig.Windows) } <ins> func TestCalculateCPUMaximum(t *testing.T) { tests := []struct { name     string cpuLimit resource.Quantity cpuCount int64 want     int64 }{ { name:     "max range when same amount", cpuLimit: resource.MustParse("1"), cpuCount: 1, want:     10000, }, { name:     "percentage calculation is working as intended", cpuLimit: resource.MustParse("94"), cpuCount: 96, want:     9791, }, { name:     "half range when half amount", cpuLimit: resource.MustParse("1"), cpuCount: 2, want:     5000, }, { name:     "max range when more requested than available", cpuLimit: resource.MustParse("2"), cpuCount: 1, want:     10000, }, { name:     "min range when less than minimum", cpuLimit: resource.MustParse("1m"), cpuCount: 100, want:     1, }, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { assert.Equal(t, tt.want, calculateCPUMaximum(&tt.cpuLimit, tt.cpuCount)) }) } } </ins>

When submitting a windows pod using resource limits on a node with more than one processor group, the specified count is incorrect due to the usage of which does not take processor groups into account. For example: node with 96 cores cpu limit 94 returns This results in a CPU maximum of since the range is from [1, ]: Our initial goal for this was to limit a process to not use all available CPU and therefore render the node uncontrollable. A CPU maximum of instead of for that specific case. For the calculation, it should use the CPU count for all available processor groups instead of just the current one. Launch a node with a lot of CPUs, for example and schedule the following pod on it: Then SSH into the node and inspect the running container using . The value will be set to . (which just uses ) is able to correctly find out the number of CPUs available to a system. Using for calculation or incorporating that code might solve this issue. <details</details<detailsAzure in-tree </details<details</details<details</details<detailscontainerd://1.6.10 </details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig windows
There is already some code which uses GetActiveProcessorCount:
/assign
/triage accepted


Question: // we confirm unreachability. Logf("Failed to execute %q: %v, stdout: %q, stderr: %q", cmd, err, stdout, stderr) } else { <ins> Logf("maxTries: %d, in try: %d, stdout: %v, stderr: %v", tries, i, stdout, stderr) </ins> var output map[string][]string if err := json.Unmarshal([]byte(stdout), &output); err != nil { Logf("WARNING: Failed to unmarshal curl response. Cmd %v run in %v, output: %s, err: %v",

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug /priority failing-test /sig network What happened: From https://k8s-, seems to be broken right after they were by Example error message: and Anything else we need to know?: Is it as expected?
Oct 11 16:37:49.106: test session affinity, cost time: 1m31.742990485s This is a mistake. Sorry for my debug lines sneak in. I was just wondering why this test case doesn't run in upstream CI, so I this debug line. Now, I see because this test case only runs in . Oct 11 16:39:42.406: Expect endpoints: map[], got: map[netserver-1:{}] This check failed unexpectedly, let me try to find out why. I am working on it and will let you know soon when I have some updates.
Thanks. These two tests don't run on pull jobs as they are marked as , though https://k8s-, https://k8s- (and other CIs that include ) would have them running.
Do you mean when I raise a PR, these test cases will not be triggered? My question is how can I make sure my follow-up PR can fix this bug? Where can I access the log message when I raise a new PR?
Can I see the updated test results via directly visiting https://k8s-, https://k8s-
Yep, that is the current behavior. A hacky way to make it run on PR --- add a temporary commit to remove the word from test name so some pull jobs ( I think?) will run it. And you could view logs by clicking the button.
Nope, this CI only includes changes that are merged. It is basically a cronjob that runs tests against head.


Question: }, gvk: example2v1.SchemeGroupVersion.WithKind("ReplicaSet"), expectedObj: &example2v1.ReplicaSet{ <ins> TypeMeta: metav1.TypeMeta{ APIVersion: "example2.apiserver.k8s.io/v1", Kind:       "ReplicaSet", }, </ins> ObjectMeta: metav1.ObjectMeta{ Name: "rs1", Labels: map[string]string{

The admission webhook review requests seem not to include a GVK in the raw object. It's included elsewhere in the review request, so it's generally not an issue, except that the unstructured codec requires GVKs to be present (it's not a technical requirement AFAICT, just a check done in the codec logic). This means that trying to use or on an Unstructured fails. A workaround in the mean time is to use , but it would be nice if this just worked properly (fixes to the review request logic and the unstructured codec logic. <details</summary</details<details</summary</detailsserving CA and certs (at and ) webhook with CA bundle `kubectl create -f that the output object doesn't have GVK in the field Environment: Kubernetes version (use ): master-ish () /sig api-machinery
when encoded to the wire, apiVersion and kind must always be included. do you have a gist handy of what the webhook receives for a built-in type (like a pod) and a for a custom resource?
Looks like custom objects (held in unstructured objects) would include GVK, but in-tree types would not. It might be as simple as explicitly setting the GVK in ConvertToGVK: and we need to update the webhook test to ensure objects in requests always have an apiVersion/kind
fixed in
/milestone v1.14 We're entering and this looks relevant to the 1.14 release. Please if I am incorrect


Question: existingPods                  []*v1.Pod pod                           *v1.Pod initTokens                    int <ins> enablePreFilter               bool </ins> unresolvable                  bool preemptedPodIndexes           map[int]struct{} enablePodDisruptionConditions bool

In k8s 1.27, scheduler introduced a new "short circuit" semantics in PreFilter to bypass a Pod quickly. A change introduced here is it relies on that plugin to be stored in , which makes sense as what it was explained . However, the info, although is populated upon each plugin's PreFilter invocation, is not injected into when some PreFilter returns unschedulable: So think about this workflow: pluginA returns Skip in its PreFilter pluginB return Unschedulable in its PreFilter note: pluginA is not included in due to aforementioned bug a Pod is claimed as unschedulable (failed by pluginB) during PostFilter, it's supposed to bypass pluginA's PreEnqueueExtension's AddPod/RemovePod, but it doesn't calling a "skipped" Plugin's AddPod/RemovePod during preemption, sometimes will abort the whole preemption as the logic tries to read info from cycleState, and some plugin's "skip" version doesn't fill cycleState properly, e.g., InterPodAffinity (skip immediately w/o write preFilterStateKey). Short circuiting using Skip is fine, but should injected properly at all cases. Described in the workflow above. It's firstly captured in an out-of-tree PostFilter plugin: , but can also happen in in-tree plugins. No response v1.27 <details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/kind regression /sig scheduling
cc
Among in-tree plugins, 3 plugins employ non-nil and may return Skip in their PreFilter() implementation w/o writing to cycleState, while return error immediately if is missing in their Add/RemovePod() functions.

So technically, as long as a scheduling flow hit the following criteria, it would hit the bug (preemption doesn't function): plugin 1/2/3 return Skip in PreFilter arbitrary PreFilter plugin after the "skipped" plugin returns Unschedulable in its PreFilter
/cc
/cc
Cherry-pick PRs: 1.28: 1.27:


Question: // ControlPlaneComponent holds settings common to control plane component of the cluster type ControlPlaneComponent struct { // ExtraArgs is an extra set of flags to pass to the control plane component. <ins> // A key in this map is the flag name as it appears on the // command line except without leading dash(es). </ins> // TODO: This is temporary and ideally we would like to switch all components to // use ComponentConfig + ConfigMaps. // +optional

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --I was reading the doc about NodeRegistration, with an interest in specifying some . I noticed that the doc (e.g., ) does not tell me exactly how to identify each arg: exactly as on the command line (including leading dash(es)), as on the command line but without leading dash(es), or camelCase. I expected the doc to enable me to write some with confidence. Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cluster-lifecycle
FWIW the answer is flag names as on the command name but without any leading dashes. This also applies to controllerManager.extraArgs, scheduler.extraArgs, etc. which hopefully should all cover this similarly (if not now, after fixing ...)

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --I was reading about how to use kubeadm and noticed a typo in the API documentation. speaks of the "node subnet" where I think "pod subnet" is intended. Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cluster-lifecycle
/area kubeadm


Question: devicePluginUpdateTimeout := 1 * time.Minute ginkgo.It("will report a Healthy and then Unhealthy single device in the pod status", func(ctx context.Context) { <ins> e2eskipper.SkipUnlessFeatureGateEnabled(features.ResourceHealthStatus) </ins> // randomizing so tests can run in parallel resourceName := fmt.Sprintf("test.device/%s", f.UniqueName) devices := []kubeletdevicepluginv1beta1.Device{{ID: "testdevice", Health: kubeletdevicepluginv1beta1.Healthy}}

In multi test grid: /kind failing-test /sig node
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc
/cc This is caused by that those test CI didn't enable ResourceHealthStatus. A fix in is to skip it if the FG is disabled.


Question: expectedEvictPods:   false, description:         "Node created recently, with no status.", }, <ins> // Node created recently without FailureDomain labels which is added back later, with no status (happens only at cluster startup). { fakeNodeHandler: &testutil.FakeNodeHandler{ Existing: []*v1.Node{ { ObjectMeta: metav1.ObjectMeta{ Name:              "node0", CreationTimestamp: fakeNow, }, }, { ObjectMeta: metav1.ObjectMeta{ Name:              "node1", CreationTimestamp: metav1.Date(2012, 1, 1, 0, 0, 0, 0, time.UTC), }, Status: v1.NodeStatus{ Conditions: []v1.NodeCondition{ { Type:               v1.NodeReady, Status:             v1.ConditionTrue, LastHeartbeatTime:  metav1.Date(2015, 1, 1, 12, 0, 0, 0, time.UTC), LastTransitionTime: metav1.Date(2015, 1, 1, 12, 0, 0, 0, time.UTC), }, }, }, }, }, Clientset: fake.NewSimpleClientset(&v1.PodList{Items: []v1.Pod{*testutil.NewPod("pod0", "node0")}}), }, daemonSets:          nil, timeToPass:          0, newNodeStatus:       v1.NodeStatus{}, secondNodeNewStatus: healthyNodeNewStatus, expectedEvictPods:   false, description:         "Node created recently without FailureDomain labels which is added back later, with no status (happens only at cluster startup).", }, </ins> // Node created long time ago, and kubelet posted NotReady for a short period of time. { fakeNodeHandler: &testutil.FakeNodeHandler{

<!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"8+", GitVersion:"v1.8.0-alpha.1.51+-dirty", GitCommit:"", GitTreeState:"dirty", BuildDate:"2017-06-20T10:51:10Z", GoVersion:"go1.8", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"8+", GitVersion:"v1.8.0-alpha.1.51+-dirty", GitCommit:"", GitTreeState:"dirty", BuildDate:"2017-06-20T11:29:14Z", GoVersion:"go1.8", Compiler:"gc", Platform:"linux/amd64"} What happened: After adding label to a node, node controller panics. What you expected to happen: Everything should still work. How to reproduce it (as minimally and precisely as possible): Run and wait for the node controller to panic: Anything else we need to know: I've checked the code and it looks like the node controller expects the label to be present during node registration and it can not be later. Even if the label can not be latter it shouldn't panic.
/sig node
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
/assign let me try this :).


Question: } func TestEnsureLoadBalancerHealthCheck(t *testing.T) { tests := []struct { name                string annotations         map[string]string

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Using a pod like the nginx-ing-cntlr with an AWS Network Load Balancer (NLB), and then draining the node its running on to force a migration to another node pool, does not reliably deregister the drained nodes from the NLB Target group. This causes the nodeport used by the NLB to remain open, and the drained node ends up continuing to funnel traffic for other pods cause of being used. This leads to connection timeouts for clients, and downtime on migrations. Switching to a Classic ELB, instead of an NLB, with connection draining enabled does work, and the instances that are drained are practically removed immediately from the instances targeted by the ELB, as expected. There seems to be some sort of issue with NLB's as this is not happening, or happening rarely (only saw it once in many tests but couldn't recreate it). The node + LB set seems to be working as I see in the controller-mgr logs that the nodes are updated as I went from 11 nodes to 8, after attempting to drain + migrate a nodepool of 3 nodes with nginx-ing-cntlr all running on them: However, this update does not seem to be translating into deregistering instances from the NLB. And i've waited minutes & hours in case timeouts were playing a hand, but that didn't seem to help. What you expected to happen: On node draining, with being used, that the NLB would deregister the instances as expected. How to reproduce it (as minimally and precisely as possible): Launch a Deployment + NLB service on a node and drain it to force it over to another node pool. Watch the instances in the AWS NLB Target group never get out of "healthy" status or be drained + removed. Anything else we need to know?: Environment: AWS EKS Kubernetes version (use ): 1.12 & 1.13 Cloud provider or hardware configuration: AWS OS (e.g: ): AWS Linux 2 Kernel (e.g.

): 4.14.123-111.109.amzn2.x8664 Install tools: Network plugin and version (if this is a network-related bug): core-dns v1.2.6 Others:
/sig aws
Related:
/sig cloud-provider
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: }, JobController: &JobControllerOptions{ &jobconfig.JobControllerConfiguration{ <del> ConcurrentJobSyncs: 5, </del> <ins> ConcurrentJobSyncs: 10, </ins> }, }, CronJobController: &CronJobControllerOptions{

What would you like to be : The ability to set for the kube-controller-manager at runtime. It looks like the option exists in the code, but a mechanism is not available to set it via kops. If I understand the code correctly, a flag needs to be in order to be able to set the value using kops. Why is this needed: We want to shift a lot of our batch processing jobs from running directly on our CI servers to running as jobs on Kubernetes. Unfortunately if too many jobs are launched on kubernetes, they begin to back up. We've noticed up to a 10 minute delay between when the logs indicate that a job has finished and when the kubernetes API actually reflects the same on our small ~30 node cluster.
/sig scalability
I think scalability is the best fit since this is impacting our ability to fully utilize and scale how we use our kubernetes cluster.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
/sig api-machinery
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with .

/close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Heyo -- we're running into a similar issue as this with our Kubernetes cluster. Is there anything I can do to help get landed?
You can't reopen an issue/PR unless you authored it or you are a collaborator. <detailsIn response to : Heyo -- we're running into a similar issue as this with our Kubernetes cluster. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign /remove-sig api-machinery /sig apps
/triage accepted /remove-lifecycle rotten
can you please describe your use case in a more detail? how How this affects you, and how this flag would help to achieve your goals?
-- I left a comment describing my use case in the PR:


Question: disks := []compute.DataDisk{} if vm.StorageProfile != nil && vm.StorageProfile.DataDisks != nil { <del> disks = *vm.StorageProfile.DataDisks </del> <ins> disks = make([]compute.DataDisk, len(*vm.StorageProfile.DataDisks)) copy(disks, *vm.StorageProfile.DataDisks) </ins> } if isManagedDisk { disks = append(disks,

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: In Azure disk attach/detach function, it uses slice() to store the disk list: And after changed, the original would also be changed, so original azure data disk list is corrupted after attach/detach disk example code as below: What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /sig azure /assign


Question: func NewForConfig(c *$.Config|raw$) (*Clientset, error) { configShallowCopy := *c <ins> if configShallowCopy.UserAgent == "" { configShallowCopy.UserAgent = $.DefaultKubernetesUserAgent|raw$() } </ins> // share the transport between all clients httpClient, err := $.RESTHTTPClientFor|raw$(&configShallowCopy) if err != nil {

It looks like kubelet started using generic user agent like "Go-http-client/2.0" and makes it harder to attribute the request to kubelet. In the past we have used "kubelet" user-agent which was providing some value. Sample logs: Kubelet's requests are using customised user-agent Run kubelet Check kube-apiserver's logs: No response master tested on gce <details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
The default user agent name is generated like below. I am using v1.23.4 and the user agent can show kubelet as the user agent.(not on gce)
/cc We should identify those and set the user-agent correctly , is right, it is very useful field ... but I see ~kubectl~ kubelet user-agents in the log linked
Interesting... in I see both kubelet user agent and "Go-http-client/2.0": Moreover, it looks like "kubelet" user agent is used only for events-related calls and other (like leases) are using "Go-http-client/2.0".
interesting , the user agent is to the request by wrapping the roundtripper, that is when the client is generated the clientset creation is here Speculating, I'm wondering if the updateTransport is mutating the transport and not adding the corresponding wrapper in ?
/assign I think that this was my fault, let me check it in more detail
I think that this was the problem


Question: } // update our local observations based on the amount reported to have been reclaimed. // note: this is optimistic, other things could have been still consuming the pressured resource in the interim. <del> signal := resourceToSignal[resourceToReclaim] value, ok := observations[signal] if !ok { glog.Errorf("eviction manager: unable to find value associated with signal %v", signal) continue </del> <ins> for _, signal := range resourceClaimToSignal[resourceToReclaim] { value, ok := observations[signal] if !ok { glog.Errorf("eviction manager: unable to find value associated with signal %v", signal) continue } value.available.Add(*reclaimed) </ins> } <del> value.available.Add(*reclaimed) </del> // evaluate all current thresholds to see if with adjusted observations, we think we have met min reclaim goals if len(thresholdsMet(m.thresholdsMet, observations, true)) == 0 { return true

It is possible to have multiple signals that point to the same type of resource, e.g., both SignalNodeFsAvailable and SignalAllocatableNodeFsAvailable refer to the same resource NodeFs. Change the map from map[v1.ResourceName]evictionapi.Signal to map[v1.ResourceName][]evictionapi.Signal
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/kind cleanup /priority important/soon taking a guess at labels to satisfy the milestone munger
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/node sig/storage: Issue will be escalated to these SIGs if needed. priority/important-soon: Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. kind/cleanup: Adding tests, refactoring, fixing old bugs. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details
Looks like the PR for this is just awaiting merge


Question: framework.ExpectEqual(*(ss.Spec.Replicas), int32(4), "statefulset should have 4 replicas") }) <del> ginkgo.It("should list, patch and delete a collection of StatefulSets", func() { </del> <ins> /* Release: v1.22 Testname: StatefulSet, list, patch and delete a collection of StatefulSets Description: When a StatefulSet is created it MUST succeed. It MUST succeed when listing StatefulSets via a label selector. It MUST succeed when patching a StatefulSet. It MUST succeed when deleting the StatefulSet via deleteCollection. */ framework.ConformanceIt("should list, patch and delete a collection of StatefulSets", func() { </ins> ssPatchReplicas := int32(2) ssPatchImage := imageutils.GetE2EImage(imageutils.Pause)

[X] test pr : kuberenetes/kubernetes 17 June 2021 [x] two weeks soak end date : 1 July 2021 [X] test promotion pr : kubernetes/kubernetes According to this APIsnoop query, there are still some remaining StatafulSet endpoints that are untested. with this query, you can filter untested endpoints by their category and eligibility for conformance. e.g below shows a query to find all conformance eligible untested, stable, core endpoints Create a StatefulSet Patch the StatefulSet List all StatefulSet in the namespace Delete the collection if StatefulSets in the namespace Verify that all StatefulSet was deleted Due to the complexity of setting up the resources for APIService we have used the current e2e test as a template. It has been extended in a for review.

example endpoint | useragent ---------------------------------------------|
createAppsV1NamespacedStatefulSet | (linux/amd64) kubernetes/$Format -- [sig-apps] StatefulSet Basic StatefulSet functionality [StatefulSetBasic] should list, patch and delete a collection of StatefulSets deleteAppsV1CollectionNamespacedStatefulSet | (linux/amd64) kubernetes/$Format -- [sig-apps] StatefulSet Basic StatefulSet functionality [StatefulSetBasic] should list, patch and delete a collection of StatefulSets listAppsV1NamespacedStatefulSet | (linux/amd64) kubernetes/$Format -- [sig-apps] StatefulSet Basic StatefulSet functionality [StatefulSetBasic] should list, patch and delete a collection of StatefulSets listAppsV1StatefulSetForAllNamespaces | (linux/amd64) kubernetes/$Format -- [sig-apps] StatefulSet Basic StatefulSet functionality [StatefulSetBasic] should list, patch and delete a collection of StatefulSets patchAppsV1NamespacedStatefulSet | (linux/amd64) kubernetes/$Format -- [sig-apps] StatefulSet Basic StatefulSet functionality [StatefulSetBasic] should list, patch and delete a collection of StatefulSets readAppsV1NamespacedStatefulSet | (linux/amd64) kubernetes/$Format -- [sig-apps] StatefulSet Basic StatefulSet functionality [StatefulSetBasic] should list, patch and delete a collection of StatefulSets replaceAppsV1NamespacedStatefulSetStatus | kube-controller-manager/v1.20.4 (linux/amd64) kubernetes//system:serviceaccount:kube-system:statefulset-controller (7 rows) If a test with these calls gets merged, test coverage will go up by 3 points This test is also created with the goal of conformance promotion. /sig apps /sig testing /sig architecture /area conformance
/triage accepted
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: Suffix("containerLogs", ns, podStatus.Name, containerName). Do(). Raw() <del> fmt.Sprintf("pod logs:%vn", string(logs)) By(fmt.Sprintf("pod logs:%vn", string(logs))) if strings.Contains(string(logs), "Internal Error") { By(fmt.Sprintf("Failed to get logs from node %q pod %q container %q: %v", podStatus.Spec.NodeName, podStatus.Name, containerName, string(logs))) </del> <ins> if err == nil && strings.Contains(string(logs), "Internal Error") { err = fmt.Errorf("Fetched log contains "Internal Error": %q.", string(logs)) } if err != nil { By(fmt.Sprintf("Warning: Failed to get logs from node %q pod %q container %q. %v", podStatus.Spec.NodeName, podStatus.Name, containerName, err)) </ins> time.Sleep(5 * time.Second) continue } <ins> By(fmt.Sprintf("Succesfully fetched pod logs:%vn", string(logs))) </ins> break }

This happened on 5/6/2015 at 07:49 am on our continuous integration system, against upstream/master head at that time (i.e. last commit ). Needs to be repro'd and debugged. Docker Containers should be able to override the image's default command and arguments Failed to create pod: Pod "client-containers--f401-11e4-9118-" is forbidden: Post http://127.0.0.1:8080/api/v1beta3/namespaces: EOF
This test is still flaky. Containers should be able to override the image's default commmand (docker entrypoint) (Failed 2 times in the last 30 runs. Stability: 93 % Containers should be able to override the image's default arguments (docker cmd) (Failed 1 times in the last 30 runs. Stability: 96 %)
I will take a look at this now
Sample log from failure:
The indicates two things 1) that the test only tried once to fetch the logs, and 2) that either the call returned an empty log or an error (which it looks like the test does not check for). See


Question: EnableLogsSupport:      true, MasterServiceNamespace: api.NamespaceDefault, ClusterName:            "kubernetes", <ins> SyncPodStatus:          true, </ins> RuntimeConfig: make(util.ConfigurationMap), KubeletConfig: client.KubeletConfig{

Split off as a smaller more specific work item. Apiserver should have a flag that makes it not attempt to GET from kubelets. This would be used in conjunction with . After we have converted the top N Getting-started-guides to the new style, we can rip out the kubelet client from the apiserver altogether.
This is tightly related to so I'll take it.


Question: // pod and confirm it doesn't show up as an endpoint. func (config *NetworkingTestConfig) DialFromContainer(protocol, dialCommand, containerIP, targetIP string, containerHTTPPort, targetPort, maxTries, minTries int, expectedResponses sets.String) { ipPort := net.JoinHostPort(containerIP, strconv.Itoa(containerHTTPPort)) <del> // The current versions of curl included in CentOS and RHEL distros // misinterpret square brackets around IPv6 as globbing, so use the -g // argument to disable globbing to handle the IPv6 case. cmd := fmt.Sprintf("curl -g -q -s 'http://%s/dial?request=%s&protocol=%s&host=%s&port=%d&tries=1'", ipPort, dialCommand, protocol, targetIP, targetPort) </del> <ins> cmd := makeCURLDialCommand(ipPort, dialCommand, protocol, targetIP, targetPort) </ins> responses := sets.NewString() for i := 0; i < maxTries; i++ { <del> stdout, stderr, err := config.f.ExecShellInPodWithFullOutput(config.TestContainerPod.Name, cmd) </del> <ins> resp, err := config.GetResponseFromContainer(protocol, dialCommand, containerIP, targetIP, containerHTTPPort, targetPort) </ins> if err != nil { <del> // A failure to kubectl exec counts as a try, not a hard fail. // Also note that we will keep failing for maxTries in tests where // we confirm unreachability. framework.Logf("Failed to execute %q: %v, stdout: %q, stderr %q", cmd, err, stdout, stderr) } else { var output map[string][]string if err := json.Unmarshal([]byte(stdout), &output); err != nil { framework.Logf("WARNING: Failed to unmarshal curl response. Cmd %v run in %v, output: %s, err: %v", cmd, config.HostTestContainerPod.Name, stdout, err) continue } for _, response := range output["responses"] { trimmed := strings.TrimSpace(response) if trimmed != "" { responses.Insert(trimmed) } </del> <ins> continue } for _, response := range resp.Responses { trimmed := strings.TrimSpace(response) if trimmed != "" { responses.Insert(trimmed) </ins> } } framework.Logf("Waiting for responses: %v", expectedResponses.Difference(responses))

Example failure: Stacktrace: The problem seem to be related to the fact that we're running the test on private cluster (nodes doesn't have external IP address).
/assign
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
Given that there are more issues with the NetLB tests than just this particular one, I think we should revert my PR that re-enabled them. ! Wojtek just sent a PR to do that -
But let's leave this issue open - we want to fix and re-enable it again.
The revert merged, but we didn't managed to get it in before the yesterday's run started. Today's run should be green.
Yeah - but that doesn't mean we should close it later - we should fix and reenable this test. The issue should be opened until that test is reenabled.
That's clear, I didn't say anywhere that we should close it
Ok, the tests have been disabled and they don't fail the correctness tests anymore. When it comes to fixing this particular test. I took a cursory glance and I think it is fixable. We should change the code to not expect public IPs on the nodes. And then when poking over http we should do it not from the test but proxy via some container. There is already a code written for that, see /help wanted
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
/good-first-issue
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
/remove-triage unresolved
I would like to take this /assign
- thanks!

contains brief description on how to fix that
so are we going to access all the nodes via a test container?
Not sure if I fully understand your question, so if you meant something different let me know: to make the test work in large clusters (5k nodes), we can't access all - we may want to do tricks we do in other tests like access min(cluster size, some constant (e.g. 10 or 100) if you meant "via a test container" - yes, I think it's fine to just migrate the test to always use it
Reopening - since we need to reenable this test.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Does this issue need an owner? I got a little lost wading through all of the issues/PRs attached to this issue. If I wanted to take up the reins, how do I trigger the 5k node test on a PR or is it automatically triggered?
The test is disabled. I think the crucial part is done now - which is: I can't remember what else left to be done. Unfortunately you can't do that from oss. We only have post-submits exposes as these are too expensive. We can try running it internally to check if it now works...
We reenabled this test and it seem to work. So closing this one. Thanks for all the help on it!


Question: "time" v1 "k8s.io/api/core/v1" <ins> "k8s.io/apimachinery/pkg/util/sets" utilfeature "k8s.io/apiserver/pkg/util/feature" </ins> "k8s.io/client-go/tools/cache" "k8s.io/klog/v2" apipod "k8s.io/kubernetes/pkg/api/v1/pod" <ins> "k8s.io/kubernetes/pkg/features" </ins> "k8s.io/utils/clock" "k8s.io/utils/ptr" )

Today the function () only account for the regular containers finish time, presumably assuming that init containers have completed before before. However, with the sidecar (restartable init) containers, sidecar containers will always finish later than regular containers. And those needs to be accounted for the calculation. /kind bug The sidecar finish time will be accounted when calculating the job's finish time. Sidecar container with prolonged termination logic on a job will result in incorrect finish time reporting. /sig apps /priority backlog KEP:
/assign
/triage accepted


Question: manifest_image_folder=`echo "$${registry_prefix}${IMAGE}" | sed "s|/|_|g" | sed "s/:/-/"`;  for arch in $(ALL_ARCH.windows);  do  for osversion in ${ALL_OSVERSIONS.windows}; do  <del> docker manifest annotate --os windows --arch $${arch} ${IMAGE}:${TAG} ${IMAGE}-windows-$${arch}-$${osversion}:${TAG};  </del> <ins> docker manifest annotate --os windows --arch $${arch} ${IMAGE}:${TAG} ${IMAGE}:${TAG}-windows-$${arch}-$${osversion};  </ins> BASEIMAGE=${BASE.windows}:$${osversion};  full_version=`docker manifest inspect ${BASE.windows}:$${osversion} | grep "os.version" | head -n 1 | awk '{print $$2}'` || true;  <del> sed -i -r "s/("os":"windows")/0,"os.version":$${full_version}/" "${HOME}/.docker/manifests/$${manifest_image_folder}-${TAG}/$${manifest_image_folder}-windows-$${arch}-$${osversion}-${TAG}";  </del> <ins> sed -i -r "s/("os":"windows")/0,"os.version":$${full_version}/" "${HOME}/.docker/manifests/$${manifest_image_folder}-${TAG}/$${manifest_image_folder}-${TAG}-windows-$${arch}-$${osversion}";  </ins> done;  done docker manifest push --purge ${IMAGE}:${TAG}

We could either: where appropriate - least amount of code change but cumbersome OS version in the tag instead of the repository /cc
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig windows
/kind bug


Question: By("Stop the NFS server") stopNfsServer(nfsServerPod) <del> By("Delete the pod mounted to the NFS volume") framework.ExpectNoError(framework.DeletePodWithWait(f, c, pod), "Failed to delete pod ", pod.Name) </del> <ins> By("Delete the pod mounted to the NFS volume -- expect failure") err := framework.DeletePodWithWait(f, c, pod) Expect(err).To(HaveOccurred()) </ins> // pod object is now stale, but is intentionally not nil By("Check if pod's host has been cleaned up -- expect not")

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: See One /sig storage
/assign
[Slow] test suite was run. Glusterfs io stream test failed due to ns not being deleted, which was due to 1 pod remaining. The glusterfs-server pod was deleted. It appears that the glusterfs-client pod was not deleted even though returned no error. In cases where this test succeeds (which is most of the time) returns which is seen in the log as INFO: Ignore "not found" error above. Pod "gluster-io-client" successfully deleted. Sometimes when the plugin io streaming tests pass you'll see logged for the client pod. However, sometimes a client pod is deleted but the above log entry is not triggered. Eg in you'll see the nfs-client pod is deleted (the ns was able to be deleted) yet the above log msg stating to ignore the "not found" error was not generated. (scroll down to the line "I0724 20:27:20.364] Jul 24 20:27:00.559: INFO: Waiting for pod nfs-io-client in namespace 'e2e-tests-volume-io-pmdks' status to be 'terminated due to deadline exceeded'(found phase: "Running", readiness: true) (29.883170178s elapsed) I0724 20:27:20.365] [AfterEach] [sig-storage] NFS"). My guess is that in this case the plugin's client pod did not terminate in 30s -- WHY NOT THE FULL 5m? -- but did terminate in time for the ns to be deleted. And sometimes the client pod cannot be deleted in 30s (again WHY NOT THE FULL 5m?) and is still present when the ns is deleted. Note: if we cannot detect that the client pod was not actually deleted and then we delete the server pod, there is no chance that the ns can be deleted. Once the server pod is gone the client pod will not delete.
It looks like the client pod failed to terminate:
30s may not always be enough time for the gluster client pod to terminate?

also, why is it so common to see ~29 seconds for the client pods (nfs, gluster) to supposedly terminate. The wait should be for up to 5m yet I never see = 30s wait times... yet, we know that sometimes the gluster client pod is not terminated. Seems like maybe a bug in this area?
can you take a look? My findings so far are this test fails about 20% of the time. The gluster client pod's Status is set to Failed and therefore the pod won't be deleted. There is e2e code that waits for the pod to be deleted but escapes if the pod has "Failed" and if the passed-in reason matches the pod's Status.Reason. In the gluster pod test failures the pod's Status.Reason == "". This causes the test to incorrectly assume that the pod was deleted when it wasn't. This problem manifests itself when the namespace is deleted and that step fails. It seems like the Failed pod's Status.Reason should not be blank. cc
It looks like gluster unmount failed:
But in the log I looked at today (https://k8s-) I see the pod's Status=Running. Then the pod is deleted and the status remains Running for ~30s then changes to Failed. Even though the pod is Failed its Reason remains "". So 1) I wonder why the pod is marked as Running if its mount failed, and 2) once the pod is Failed why is Reason still blank?
It was the unmount that failed, not the mount.
ah, ok thanks, I missed that. Still need to know why Pod.Status.Reason is blank...
Below are some records from a related to the glusterfs-client pod, gluster-io-cliente2e-tests-volume-io-6h140(-778e-11e7-afad-)_ and the "e2e-tests-volume-io-6h140" ns:
I think the issue is that the test is deleting the gluster endpoints before deleting the pod.
Nm, the test does seem to be cleaning up in the correct order: client pod, then server pod and endpoints. So the issue seems to be that during Pod termination, it goes to Failed state.

Before, I thought it was because the unmount failed, but based on the timestamps of the logs, it seems like it fails before we start unmounting.
Just spoke to and he said that it's normal for a Pod to go to Failed state during the termination sequence. In between Failed state and Pod object deletion is when we unmount the volumes. So I think the issue here is that the WaitForPodTerminated function needs to wait only for the Pod object to be deleted, and not return early if Pod state is Failed.
do you have time to look at this? The test is still flaking
yes I can take a look. Do you know if there has been a pr started to address this issue? BTW, I will be gone most of tomorrow but will start tomorrow night.


Question: github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc h1:U9qPSI2PIWSS1VwoXQT9A3Wy9MM3WgvqSxFWenqJduM= github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= <del> github.com/emicklei/go-restful/v3 v3.8.0/go.mod h1:6n3XBCmQQb25CM2LCACGz8ukIrRry+4bhvbpWn3mrbc= </del> <ins> github.com/emicklei/go-restful/v3 v3.11.0 h1:rAQeMHw1c7zTmncogyy8VvRZwtkmkZ4FxERmMY4rD+g= github.com/emicklei/go-restful/v3 v3.11.0/go.mod h1:6n3XBCmQQb25CM2LCACGz8ukIrRry+4bhvbpWn3mrbc= </ins> github.com/fxamacker/cbor/v2 v2.7.0-beta h1:m5bO941uTVpSms26QjzEJxUZaC3S/h1pSJexBCu4wAA= github.com/fxamacker/cbor/v2 v2.7.0-beta/go.mod h1:pxXPTn3joSm21Gbwsv0w9OSA2y1HFR9qXEeXQVeNoDQ= github.com/go-logr/logr v1.4.1 h1:pKouT5E8xu9zeFC39JXRDukb6JFQPXM5p5I91188VAQ=

We're trying to upgrade to () and found the issue on executing (). The problem is that on () missed dependency that not allow to install () No error To reproduce the same what happening on , just execute this command on code-generator folder: I think it's happens after this commit <details</details<details</details<details</details<details</details<details</details<details</details
/cc
/assign
/sig api-machinery
I was able to reproduce, but just because I had an older version of kube-openapi in my GOPATH. Using the prefix in the command fixed it. I think line needs to be
Yeah, with works for me too.
Either that or we pin to a more recent commit, for example . For some reason if you ommit the version it is using a from Feb 27.
thanks for investigating. Question, if we use would this not cause problems in the future if we try to build a release branch from the past? Maybe this is not a big deal. OTOH if we hardcode a tag, it may create maintenance costs. I'm wondering if there is a better way, say to version and use the corresponding tag, say ?
Using tags would be a way to have more reproducible builds, they still need to be tagged from time to time tho. That is still better than using or micro managing commit revisions.
Thanks and for the research. Is either of you thinking on submitting a fix for this? Thanks!
I can do it.
Thanks! /assign mbobrovskyi
Yes, please don't use Generally instead we'd control the version of a tool in the hack/tools module and install it from there (see e.g.
This allows us to also gain for the package while not polluting the main module with dev-only dependencies.
Not sure if we need to make another Because it's not just a dev-only dependency. I propose to add it to of code-generator.
This is about semversioning
/triage accepted


Question: } encoder := f.JSONEncoder() <ins> decoder := f.Decoder(false) </ins> count := 0 err = r.Visit(func(info *resource.Info, err error) error {

kubectl create, delete, etc. can already take a List object. kubectl apply can take the list object when it does creation, but it gives an (ugly) error when it's configuring the objects. Then... expected: Actual:
what's you ?
0-alpha.5
Sounds like a bug. Are you interested in taking a look
sure I will take a look during this weekend.
I think the root cause is , the actually is , in the origin , is not , but after the , this object is nil. Would you like to double check this?
I'll take a look.
if you have make sure about this, please let me know. Thanks :)
Is there any ways we could get this into 1.2? It's actually making our config generation quite painful to have to make a separate file for every object. It looks like got pretty far? would you like to make a PR that we can review?
I don't see how. It could have made 1.2 if a PR had been sent within a few days of the issue being filed.
I unassigned all issues assigned to you. Please only assign issues that are actively being worked on, so that others can know which issues are available to be picked up.
My apologies. If I'm not mistaken, this issue was filed after the code freeze for 1.2. I didn't realize that it was time critical or that it could make the release at that time. However, I should have asked. Sorry for the inconvenience. Thank you. I still plan to address those issues for the next release, as discussed, but will assign each one to myself as I work on it.
I created which fixes this. I'm really hoping we can get this into 1.2 since it's a pretty small, isolated fix but a pretty big bug that is complicating our config generation system. Let me know what else I can do to help.
Thanks for the PR. We're short on reviewer bandwidth at the moment. I assigned it to Clayton for review or delegation.


Question: return NewNamedReflector(naming.GetNameFromCallsite(internalPackages...), lw, expectedType, store, resyncPeriod) } <del> // reflectorDisambiguator is used to disambiguate started reflectors. // initialized to an unstable value to ensure meaning isn't attributed to the suffix. var reflectorDisambiguator = int64(time.Now().UnixNano() % 12345) </del> // NewNamedReflector same as NewReflector, but with a specified name for logging func NewNamedReflector(name string, lw ListerWatcher, expectedType interface{}, store Store, resyncPeriod time.Duration) *Reflector { <del> reflectorSuffix := atomic.AddInt64(&reflectorDisambiguator, 1) </del> r := &Reflector{ <del> name: name, // we need this to be unique per process (some names are still the same) but obvious who it belongs to metrics:       newReflectorMetrics(makeValidPrometheusMetricLabel(fmt.Sprintf("reflector_"+name+"_%d", reflectorSuffix))), </del> <ins> name:          name, </ins> listerWatcher: lw, store:         store, expectedType:  reflect.TypeOf(expectedType),

What happened: After upgrading to kubernetes 1.12.5 we observe failing nodes, that are caused by kubelet eating all over the memory after some time. ! I use image to run kubelet on 102 clusters and since a week we see some nodes leaking memory, caused by kubelet. I investigated some of these kubelets with strace and pprof. With 3s of running strace I saw = 50 openat() calls to the same file from the same threadid (pid) from kubelet: If I do pprof kubelet it shows clientgo metrics and compress is taking over most of the compute time. Memory profile png: ! The reflector metrics seem to be the problem: What you expected to happen: I expect that kubelet does not need so much memory How to reproduce it (as minimally and precisely as possible): I don't know Anything else we need to know?: One of the affected clusters has only 120 Pods and 3 Pods are in CrashLoopBackOff state, one pod since 6 days on an affected node which was investigated. Environment: Kubernetes version (use ): v1.12.5 Cloud provider or hardware configuration: AWS OS (e.g. from /etc/os-release):Kernel (e.g. ): Linux ip-172-31-10- 4.14.63-coreos SMP Wed Aug 15 22:26:16 UTC 2018 x86_64 Intel(R) Xeon(R) CPU E5-2686 v4 @ 2.30GHz GenuineIntel GNU/Linux Install tools: custom
/sig node
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I think it might be a side effect of , but I couldn't find any numbers in there. The comments mention an offline discussion, was this brought up?
What happens (I only investigated it very briefly) is that kubelet creates a bunch of reflectors for every pod — secret mounts, configmap mounts, token mounts etc. Each reflector registers a number of histograms, gauges, etc. and even when the reflector is stopped and removed the metrics are never removed.

After a few days on a busy cluster you get millions of metrics bloating up kubelet and everything else using the reflectors.
This was broken since v1.12.0, so probably unrelated to .
correct that PR is only in v1.14.0-alpha.2 v1.14.0-alpha.1 and master
For everyone that finds this issue and needs a patch to disable the reflector metrics:
Do you have same metrics after disabling ReflectorMetricsProvider?
no, it drops the reflector metrics
is this indirectly caused by switching to watch-based managers?
Yeah - switching to watch resulted in more extensive using of a reflector. It seems that those metrics are directly coming from reflector (I wasn't aware of those). So it's highly probably that this change cause it. I think that we don't really need those metrics, so if we could switch them off in Kubelets, that should solve the problem.
is sent out for review to fix that. We should probably cherrypick it to previous releases if we agree that's the right way to solve it.
Actually, I realized that I don't fully understand the problem. It should be a problem in default Kubernetes (what we build from this repo), because those metrics are not switched on for kubelet: metrics factory is defaulted to noop the prometheus metrics need to be explicitly linked and initialized (as they are in apiserver e.g.: )
Wouldn't it be better to rewrite the reflector metrics so they're aggregated in a better way rather than relying on people not accidentally enabling them? Or at least put a huge warning on top saying "this will leak memory like crazy" just so the same situation doesn't repeat in other code using client-go?
- that's a separate issue whether the metrics are reasonable or not (and node sig-node related). One thing that I would like to confirm: do you customize k8s yourself? I would like to ensure that this issue you is in your setup and not in upstream k8s.
Our production configuration is this one: Kubelet config: We run with an image called , but it's just a mirror of . So upstream Kubernetes.
I try to figure out an issue why these metrics are even there and I found issues from the past showing leaks already in older versions: Sounds for, that we need a postmortem to make this not happening again.

I don't find any issue in kubernetes nor in client-go, that could reasonable explain why these were introduced in the first place. I guess I just did not found it, and I hope someone can highlight where the decision came from.
I am running Kubernetes v1.12.3 and I don't see reflector metrics being used by kubelet ( shows nothing). FWIW, the cluster is created using kops. Any idea, how are reflector metrics enabled?
and myself will also look into this as well and see what we can find.
Thanks
The issue only exists if hyperkube is used when running kubelet, as hyperkube has imported the package for apiserver and controller-manager. If kubelet binary is used, then there're no reflector metrics (this is same with your findings). Since the metrics is not exposed by kubelet cmd, what do you think of disabling it for kubelet subcommand in hyperkube? I can't reproduce the issue (with hyperkube used) on my own cluster: ! Since both apiserver and controller-manager have also imported the package, have you also observed same issues for them?
+1
Yes, we observe the same thing for apiserver and controller-manager as well.
I would also suggest not using imports for side effects because this usually creates hard to debug and non-obvious problems like this.
So you mean disable the reflector metrics totally?
No, I suggest changing the code so that the end users (kubelet, apiserver, etc) would have to explicitly enable the metrics by calling a function instead of having this happen as a side effect of an import statement.
LGTM. WDYT? We should probably do this for both kubelet and control plane services (apiserver and controller-manager).
I'm fine with that.
+1
+1
Why wouldn't we want to just actually disable this metric wholesale? Enabling this flag would basically be the same thing as saying enable a memory leak, no?
/assign
Han points out that, since the metric has a random suffix each time, it's not very useful for monitoring anyway.
/sig api-machinery


Question: svc.Spec.ExternalTrafficPolicy = v1.ServiceExternalTrafficPolicyTypeCluster execAffinityTestForLBService(f, cs, svc, true) }) <ins> It("should implement service.kubernetes.io/service-proxy-name", func() { // this test uses framework.NodeSSHHosts that does not work if a Node only reports LegacyHostIP framework.SkipUnlessProviderIs(framework.ProvidersWithSSH...) // this test does not work if the Node does not support SSH Key framework.SkipUnlessSSHKeyPresent() ns := f.Namespace.Name numPods, servicePort := 3, defaultServeHostnameServicePort serviceProxyNameLabels := map[string]string{"service.kubernetes.io/service-proxy-name": "foo-bar"} // We will create 2 services to test creating services in both states and also dynamic updates // svcDisabled: Created with the label, will always be disabled. We create this early and //              test again late to make sure it never becomes available. // svcToggled: Created without the label then the label is toggled verifying reachability at each step. By("creating service-disabled in namespace " + ns) svcDisabled := getServeHostnameService("service-disabled") svcDisabled.ObjectMeta.Labels = serviceProxyNameLabels _, svcDisabledIP, err := framework.StartServeHostnameService(cs, internalClientset, svcDisabled, ns, numPods) Expect(err).NotTo(HaveOccurred()) By("creating service in namespace " + ns) svcToggled := getServeHostnameService("service") podToggledNames, svcToggledIP, err := framework.StartServeHostnameService(cs, internalClientset, svcToggled, ns, numPods) Expect(err).NotTo(HaveOccurred()) jig := framework.NewServiceTestJig(cs, svcToggled.ObjectMeta.Name) hosts, err := framework.NodeSSHHosts(cs) Expect(err).NotTo(HaveOccurred()) if len(hosts) == 0 { framework.Failf("No ssh-able nodes") } host := hosts[0] By("verifying service is up") framework.ExpectNoError(framework.VerifyServeHostnameServiceUp(cs, ns, host, podToggledNames, svcToggledIP, servicePort)) By("verifying service-disabled is not up") framework.ExpectNoError(framework.VerifyServeHostnameServiceDown(cs, host, svcDisabledIP, servicePort)) By("adding service-proxy-name label") jig.UpdateServiceOrFail(ns, svcToggled.ObjectMeta.Name, func(svc *v1.Service) { svc.ObjectMeta.Labels = serviceProxyNameLabels }) By("verifying service is not up") framework.ExpectNoError(framework.VerifyServeHostnameServiceDown(cs, host, svcToggledIP, servicePort)) By("removing service-proxy-name annotation") jig.UpdateServiceOrFail(ns, svcToggled.ObjectMeta.Name, func(svc *v1.Service) { svc.ObjectMeta.Labels = nil }) By("verifying service is up") framework.ExpectNoError(framework.VerifyServeHostnameServiceUp(cs, ns, host, podToggledNames, svcToggledIP, servicePort)) By("verifying service-disabled is still not up") framework.ExpectNoError(framework.VerifyServeHostnameServiceDown(cs, host, svcDisabledIP, servicePort)) }) </ins> }) // TODO: Get rid of [DisabledForLargeClusters] tag when issue #56138 is fixed.

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Implement the feature described here: Why is this needed: In a cluster that has a service mesh a lot of the work being done by kube-proxy is redundant and wasted. Specifically, services that are only reached via other services in the mesh will never use the service abstraction implemented by kube-proxy in iptables (or ipvs). By informing the kube-proxy of this, we can lighten the work it is doing and the burden on its proxy backend. <!-- DO NOT EDIT BELOW THIS LINE --/kind feature
/sig network


Question: image: e2e-node-containervm-v20160321-image project: kubernetes-node-e2e-images gci-family: <del> image_regex: gci-dev-54-8743-3-0 </del> <ins> image_regex: gci-dev-55-8820-0-0 </ins> project: google-containers metadata: "user-data<test/e2e_node/jenkins/gci-init.yaml"

The upgraded version of container-vm (gci) has significant performance regressions with localhost networking when compared to its previous versions. is working on making gci kernel fixes to resolve this issue. This issue is meant to track the GCI fix from release 1.4 perspective. The fix for this issue will result in upgrading the gci version pinned to release 1.4 branch. ETA: <TBDStatus: Yellow
can you update the ETA for fixing this issue?
cc for release tracking purposes.
is this a release blocker (why?) or are we aiming for a minor release?
It is a blocker because it causes about 70% drop in throughput which is a significant regression from a k8s on GCP user perspective.
The kernel fix should get merged in GCI kernel today, and hopefully we will be able to release a GCI version tomorrow that includes the fix.
Hmm, I somehow fail to see why a kubernetes release needs to be blocked on a bug fix going into one os the kubernetes release is being deployed on top of, but that's just my opinion
I get your point of view. But, k8s on GCE cannot have serious user-facing issues. The distro is just an implementation detail, but an important one. If we include first-class support for additional distros like ubuntu, debian, etc. in the future, we will have to make the very same priority calls. At the end of the day, the release needs to work for most users in most environments. , Prashanth B wrote:
I think gci related issues shouldn't blocker OSS k8s 1.4 release. I also agreed that k8s on GCE cannot have serious user-facing issues, but that can be resolved by reverting the changes of making GCE node on GCI default. I am not suggesting that we do that now, but I do suggest we have the summary of the entire issue (not only this particular one) on GCI on node, and deeply understand the risk, then make the final decision. On another hand, we are still planning to enable GCI on node default between 1.4 and 1.5 time framework. cc/


Question: `, }, { <ins> name: "nil container metrics", summary: &statsapi.Summary{ Pods: []statsapi.PodStats{ { PodRef: statsapi.PodReference{ Name:      "pod_a", Namespace: "namespace_a", }, Containers: []statsapi.ContainerStats{ { Name:      "container_a", StartTime: metav1.NewTime(staticTimestamp.Add(-30 * time.Second)), CPU: &statsapi.CPUStats{ Time:                 testTime, UsageCoreNanoSeconds: nil, }, Memory: &statsapi.MemoryStats{ Time:            testTime, WorkingSetBytes: nil, }, }, }, }, { PodRef: statsapi.PodReference{ Name:      "pod_b", Namespace: "namespace_b", }, Containers: []statsapi.ContainerStats{ { Name:      "container_a", StartTime: metav1.NewTime(staticTimestamp.Add(-10 * time.Minute)), CPU: &statsapi.CPUStats{ Time:                 testTime, UsageCoreNanoSeconds: uint64Ptr(10000000000), }, Memory: &statsapi.MemoryStats{ Time:            testTime, WorkingSetBytes: uint64Ptr(1000), }, }, }, }, }, }, summaryErr: nil, expectedMetrics: ` # HELP container_cpu_usage_seconds_total [ALPHA] Cumulative cpu time consumed by the container in core-seconds # TYPE container_cpu_usage_seconds_total counter container_cpu_usage_seconds_total{container="container_a",namespace="namespace_b",pod="pod_b"} 10 1624396278302 # HELP container_memory_working_set_bytes [ALPHA] Current working set of the container in bytes # TYPE container_memory_working_set_bytes gauge container_memory_working_set_bytes{container="container_a",namespace="namespace_b",pod="pod_b"} 1000 1624396278302 # HELP container_start_time_seconds [ALPHA] Start time of the container since unix epoch in seconds # TYPE container_start_time_seconds gauge container_start_time_seconds{container="container_a",namespace="namespace_a",pod="pod_a"} 1.6243962483020916e+09 1624396248302 container_start_time_seconds{container="container_a",namespace="namespace_b",pod="pod_b"} 1.6243956783020916e+09 1624395678302 # HELP scrape_error [ALPHA] 1 if there was an error while getting container metrics, 0 otherwise # TYPE scrape_error gauge scrape_error 0 `, }, { </ins> name: "arbitrary pod metrics", summary: &statsapi.Summary{ Pods: []statsapi.PodStats{

kubelet panic when accessing metrics/resource endpoint kubelet log kubelet works fine access metrics/resource endpoint of kubelet No response <details</details<details</details<details</details<details</details<detailsINFO[2022-07-14 05:36:21.474173892Z] Starting CRI-O, version: 1.23.0, git: (dirty) Version: 1.23.0 GitCommit: GitTreeState: dirty BuildDate: 1980-01-01T00:00:00Z GoVersion: go1.17.4 Compiler: gc Platform: linux/amd64 Linkmode: static BuildTags: apparmor, excludegraphdriverdevicemapper, seccomp, selinux SeccompEnabled: true AppArmorEnabled: true </details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/sig node


Question: CROSS_BUILD_COPY qemu-QEMUARCH-static /usr/bin/ <ins> # WARNING: Please note that the script below removes the security packages from arm64 and ppc64el images # as they do not exist anymore in the debian repositories for jessie. So we do not recommend using this # image for any production use and limit use of this image to just test scenarios. COPY fixup-apt-list.sh / RUN ["/fixup-apt-list.sh"] </ins> RUN apt-get -q update &&  apt-get install -y dnsutils &&  apt-get clean

[X] "goproxy" to "" [X] "jessie-dnsutils" to "" [X] "redis" to "" [X] "serve-hostname" to ""
/assign /assign
we can use to iterate
/sig testing
tip for uploading is here -
should be published to now. I'm rerunning for the other 3 images.
I think the and images were published successfully this time. :
can you please try ?
should now be published too thanks to PR.
related to


Question: "github.com/coreos/go-etcd/etcd" ) <ins> type testRESTStrategy struct { runtime.ObjectTyper api.NameGenerator namespaceScoped     bool allowCreateOnUpdate bool } func (t *testRESTStrategy) NamespaceScoped() bool     { return t.namespaceScoped } func (t *testRESTStrategy) AllowCreateOnUpdate() bool { return t.allowCreateOnUpdate } func (t *testRESTStrategy) ResetBeforeCreate(obj runtime.Object) {} func (t *testRESTStrategy) Validate(obj runtime.Object) errors.ValidationErrorList { return nil } func (t *testRESTStrategy) ValidateUpdate(obj, old runtime.Object) errors.ValidationErrorList { return nil } func hasCreated(t *testing.T, pod *api.Pod) func(runtime.Object) bool { return func(obj runtime.Object) bool { actualPod := obj.(*api.Pod) if !api.Semantic.DeepDerivative(pod.Status, actualPod.Status) { t.Errorf("not a deep derivative %#v", actualPod) return false } return api.HasObjectMetaSystemFieldValues(&actualPod.ObjectMeta) } } </ins> func NewTestGenericEtcdRegistry(t *testing.T) (*tools.FakeEtcdClient, *Etcd) { f := tools.NewFakeEtcdClient(t) f.TestIndex = true h := tools.EtcdHelper{f, testapi.Codec(), tools.RuntimeVersionAdapter{testapi.MetadataAccessor()}} <ins> strategy := &testRESTStrategy{api.Scheme, api.SimpleNameGenerator, true, false} </ins> return f, &Etcd{ <del> NewFunc:      func() runtime.Object { return &api.Pod{} }, NewListFunc:  func() runtime.Object { return &api.PodList{} }, EndpointName: "pods", KeyRootFunc:  func(ctx api.Context) string { return "/registry/pods" }, </del> <ins> NewFunc:        func() runtime.Object { return &api.Pod{} }, NewListFunc:    func() runtime.Object { return &api.PodList{} }, EndpointName:   "pods", CreateStrategy: strategy, UpdateStrategy: strategy, KeyRootFunc:    func(ctx api.Context) string { return "/registry/pods" }, </ins> KeyFunc: func(ctx api.Context, id string) (string, error) { return path.Join("/registry/pods", id), nil }, <del> Helper: h, </del> <ins> ObjectNameFunc: func(obj runtime.Object) (string, error) { return obj.(*api.Pod).Name, nil }, Helper:         h, </ins> } }

The use of AtomicUpdate by genericetcd doesn't handle conflicting writes by rejecting that to a user. It should. Eventually, we may want to hide write conflicts to /pods when status has changed, and to spec when /pods/status changes, but that requires us to have two version counters and that's not worth the effort. Fix incoming.
Also we need a unit/integration test for this.


Question: $CLOUDCFG delete services/redismaster $CLOUDCFG delete pods/redis-master-2 <del> POD_LIST_2=$($CLOUDCFG -json list pods | jq ".items[].id") </del> <ins> POD_LIST_2=$($CLOUDCFG '-template={{range.Items}}{{.ID}} {{end}}' list pods) </ins> echo "Pods running after shutdown: ${POD_LIST_2}" exit 0

We have an e2e test that runs the guestbook example. We should also have one that runs through the other example.


Question: return clusterName } <del> func (az *Cloud) getLoadBalancerRuleName(service *v1.Service, protocol v1.Protocol, port int32, subnetName *string) string { </del> <ins> func (az *Cloud) getLoadBalancerRuleName(service *v1.Service, protocol v1.Protocol, port int32) string { </ins> prefix := az.getRulePrefix(service) <del> if subnetName == nil { return fmt.Sprintf("%s-%s-%d", prefix, protocol, port) </del> <ins> ruleName := fmt.Sprintf("%s-%s-%d", prefix, protocol, port) subnet := subnet(service) if subnet == nil { return ruleName </ins> } <del> return fmt.Sprintf("%s-%s-%s-%d", prefix, *subnetName, protocol, port) </del> <ins> // Load balancer rule name must be less or equal to 80 characters, so excluding the hyphen two segments cannot exceed 79 subnetSegment := *subnet if len(ruleName)+len(subnetSegment)+1 > loadBalancerRuleNameMaxLength { subnetSegment = subnetSegment[:loadBalancerRuleNameMaxLength-len(ruleName)-1] } return fmt.Sprintf("%s-%s-%s-%d", prefix, subnetSegment, protocol, port) </ins> } func (az *Cloud) getSecurityRuleName(service *v1.Service, port v1.ServicePort, sourceAddrPrefix string) string {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When the specified internal LB uses a subnet with longer name, the LB configuration will fail with following error: What you expected to happen: When constructing the frontend IP configuration name and the load balancing rules name, the subnet name should be truncated if too long. How to reproduce it (as minimally and precisely as possible): Use a subnet that has long name (70 charactors) and create ILB service. Anything else we need to know?: Environment: Kubernetes version (use ): 1.15 Cloud provider or hardware configuration: azure OS (e.g: ): Ubuntu 16.04 Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig cloud-provider /area provider/azure /assign


Question: pods = append(pods, pod) } <del> events, _ := d.Events(namespace).Search(node) </del> <ins> var events *api.EventList if ref, err := api.GetReference(node); err != nil { glog.Errorf("Unable to construct reference to '%#v': %v", node, err) } else { // TODO: We haven't decided the namespace for Node object yet. ref.UID = types.UID(ref.Name) events, _ = d.Events("").Search(ref) } </ins> return describeNode(node, pods, events) }

now shows NodePhase, NodeCondition, etc. information from NodeStatus. It should also show related events like what we did for pods.


Question: }) } } <ins> func TestBuildOpenAPIModelsForApply(t *testing.T) { // This is a list of validation that we expect to work. tests := []apiextensionsv1.CustomResourceValidation{ { OpenAPIV3Schema: &apiextensionsv1.JSONSchemaProps{ Type:       "object", Properties: map[string]apiextensionsv1.JSONSchemaProps{"num": {Type: "integer", Description: "v1beta1 num field"}}, }, }, { OpenAPIV3Schema: &apiextensionsv1.JSONSchemaProps{ Type:         "", XIntOrString: true, }, }, { OpenAPIV3Schema: &apiextensionsv1.JSONSchemaProps{ Type: "object", Properties: map[string]apiextensionsv1.JSONSchemaProps{ "oneOf": { OneOf: []apiextensionsv1.JSONSchemaProps{ {Type: "boolean"}, {Type: "string"}, }, }, }, }, }, { OpenAPIV3Schema: &apiextensionsv1.JSONSchemaProps{ Type: "object", Properties: map[string]apiextensionsv1.JSONSchemaProps{ "nullable": { Type:     "integer", Nullable: true, }, }, }, }, } staticSpec, err := getOpenAPISpecFromFile() if err != nil { t.Fatalf("Failed to load openapi spec: %v", err) } crd := apiextensionsv1.CustomResourceDefinition{ ObjectMeta: metav1.ObjectMeta{Name: "example.stable.example.com", UID: types.UID("12345")}, Spec: apiextensionsv1.CustomResourceDefinitionSpec{ Group: "stable.example.com", Names: apiextensionsv1.CustomResourceDefinitionNames{ Plural: "examples", Singular: "example", Kind: "Example", ShortNames: []string{"ex"}, ListKind: "ExampleList", Categories: []string{"all"}, }, Conversion:            &apiextensionsv1.CustomResourceConversion{Strategy: apiextensionsv1.NoneConverter}, Scope:                 apiextensionsv1.ClusterScoped, PreserveUnknownFields: false, Versions: []apiextensionsv1.CustomResourceDefinitionVersion{ { Name: "v1beta1", Served: true, Storage: true, Subresources: &apiextensionsv1.CustomResourceSubresources{Status: &apiextensionsv1.CustomResourceSubresourceStatus{}}, }, }, }, } for _, test := range tests { crd.Spec.Versions[0].Schema = &test if _, err := buildOpenAPIModelsForApply(staticSpec, &crd); err != nil { t.Fatalf("failed to convert to apply model: %v", err) } } } func getOpenAPISpecFromFile() (*spec.Swagger, error) { path := filepath.Join( strings.Repeat(".."+string(filepath.Separator), 6), "api", "openapi-spec", "swagger.json") _, err := os.Stat(path) if err != nil { return nil, err } byteSpec, err := ioutil.ReadFile(path) if err != nil { return nil, err } staticSpec := &spec.Swagger{} err = yaml.Unmarshal(byteSpec, staticSpec) if err != nil { return nil, err } return staticSpec, nil } </ins>

With in a CRD we get: This is because gnostic's OpenAPI proto types have no nullable support. Luckily CRDs keep working because in the upper error case we don't stop serving, just set the SSA schema to nil.
Thanks for reporting!
I think it should be "This is because gnostic's OpenAPI v2 proto types have no nullable support.". The problem is that we're not supposed to pass-in v3.
Source:
OK, We'll write a test and fix this, thanks Stefan!
/reopen This still happens in slightly different shape: This comes from :
That's a fairly different bug but yes, thanks!
Is this related to the stating that use of "anyOf" wouldn't be allowed after 1.16? We're seeing issue on OpenFaaS after going to 1.17.5 and 1.18. Usually upgrading K8s versions hasn't caused breaking changes and we're trying to figure out where to start with this one.
No. That post was about structural schemas. v1 CRDs are limited to structural schemas. v1beta1 CRDs can continue to create non-structural schemas until v1beta1 is no longer served (in 1.22) Also, is not completely disallowed, specific restrictions inside an clause (like ) are what are not allowed ("logical constraints using oneOf, allOf, anyOf, not are allowed").
extracted the newly found issue to to track separately with isolated reproducer /close
Closing this issue. <detailsIn response to : extracted the newly found issue to to track separately Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: ns := fmt.Sprintf("--namespace=%v", f.Namespace.Name) ginkgo.By("client-side validation (kubectl create and apply) allows request with any unknown properties") <del> randomCR := fmt.Sprintf(`{%s,"spec":{"b":[{"c":"d"}]}}`, meta) </del> <ins> randomCR := fmt.Sprintf(`{%s,"spec":{"a":null,"b":[{"c":"d"}]}}`, meta) </ins> if _, err := framework.RunKubectlInput(f.Namespace.Name, randomCR, ns, "create", "-f", "-"); err != nil { framework.Failf("failed to create random CR %s for CRD that allows unknown properties in a nested object: %v", randomCR, err) }

What happened: I have CRD with a spec field of type and annotation . I want to store json data in this field which is actually a Grafana Dashboard json. When I try to create a CR of this type, I am getting validation error from json bytes. . In the json data . What you expected to happen: I expected no error since I set How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): v1.17.0 Cloud provider or hardware configuration: kind OS (e.g: ): Ubuntu 18.04 Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /sig api-machinery /area custom-resources
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : What happened: I have CRD with a spec field of type and annotation . I want to store json data in this field which is actually a Grafana Dashboard json. Kubernetes version (use ): v1.17.0 Cloud provider or hardware configuration: kind OS (e.g: ): Ubuntu 18.04 Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /sig api-machinery /area custom-resouces Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
cc:
, you are able to create the CRD, which is also working for me. Are you able to create the CR for this CRD? Does the following command work for you?
Running the second command:
I think you have not read the issue clearly. The CRD creation is working fine. I am getting validation error when trying to create CR for this CRD.
I edited the previous comment with result for second command.
Thanks ! This is the issue I see. I expected no error since I set .
If you pass --validate=false, does it create successfully?
Yes, if I pass --validate=false, it creates successfully.
Then the issue is in client-side kubectl validation, and the server is behaving as expected
/cc for openapi validation
Yeah, we should probably make anything that has the accept anything.

I thought we already did so for types that are . Maybe this never landed actually, I could look it up.
/assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
That's probably not super hard to solve if someone wants to try.
A good way to start would be to solve all the TODOs in Or possibly, to address this bug specifically: This is the function that validates the "kind" (i.e. structures): When I initially wrote this code, we assumed that these type would never have additionalProperties. We probably need to change this condition since it's not really valid: and this one: We create the kind here: That function would have to be changed to save the "additionalProperties" type (if given) or the ability to have additionalProperties.
Actually, the problem also is that it doesn't know how to handle null. This one is a little tricky.
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
I think the publishing of the "object" type must be supressed for fields. We have similar cases already.

Touching kubectl is not very effective, or at least it takes multiple releases to makes its way through the ecosystem.
/assign


Question: return nil, errors } <ins> // HttpProxyCheck checks if https connection to specific host is going // to be done directly or over proxy. If proxy detected, it will return warning. type HttpProxyCheck struct { Proto string Host  string Port  int } func (hst HttpProxyCheck) Check() (warnings, errors []error) { url := fmt.Sprintf("%s://%s:%d", hst.Proto, hst.Host, hst.Port) req, err := http.NewRequest("GET", url, nil) if err != nil { return nil, []error{err} } proxy, err := http.DefaultTransport.(*http.Transport).Proxy(req) if err != nil { return nil, []error{err} } if proxy != nil { return []error{fmt.Errorf("Connection to %q uses proxy %q. If that is not intended, adjust your proxy settings", url, proxy)}, nil } return nil, nil } </ins> func RunInitMasterChecks(cfg *kubeadmapi.MasterConfiguration) error { // TODO: Some of these ports should come from kubeadm config eventually: checks := []PreFlightCheck{

Environment: VMware CentOS Linux release 7.2.1511 (Core) kernel: 3.10.0-327.36.1.el7.x8664 Installed packages: kubeadm.x8664 1.5.0-0.alpha.0.1534.gcf7301f kubectl.x8664 1.4.0-0 kubelet.x8664 1.4.0-0 kubernetes-cni.x8664 0.3.0.1-0. docker-engine.x8664 1.11.2- docker-engine- 1.12.1- What happened: kubeadm init hangs indefintely after: kubeadm init <master/tokensgenerated token: "." <master/pkicreated keys and certificates in "/etc/kubernetes/pki" <util/kubeconfigcreated "" <util/kubeconfigcreated "" <master/apiclientcreated API client configuration <master/apiclientcreated API client, waiting for the control plane to become readykubeadm init What you expected to happen: Master node initialize Anything else do we need to know: Only the kubeadm and kublet processes are running. When I do docker ps no containers are running. If i type docker images the only image that has been downloaded are: I've also tried: kubeadm init --api-advertise-addresses but with the same outcome (followed the clean-up at ) Im behind a firewall so I'm using proxy settings for docker. I have confirmed I can reach images on docker hub and google cloud registry
will download the required binaries viz etcd. When you are behind firewall kubeadm will hang/wait forever to download the binaries. I dont know whether it is addressed correctly in kubeadm implementation. I am also facing the same issue and waiting for the resolution.
Thx, I suspected it to be some firewall issue. Do you know if there's a way to add httpsproxy & httpproxy settings to kubeadm?
Kubeadm doesn't seem to pick proxy environment variables. I am looking at recompiling with proxy support may work. But I dont know how to do that. Waiting for updates about proxy
This issue same as Lets join there. The solution mentioned there may work.

You can try kubeadm with patch applied from PR : It works at least in my environment behind restrictive proxy.
thanks, I will look into it


Question: When building final release tars, they are first staged into `_output/release-stage` before being tar'd up and put into `_output/release-tars`. <ins> ## Proxy Settings If you are behind a proxy, you need to edit `build/build-image/Dockerfile` and add proxy settings to execute command in that container correctly. example: `ENV http_proxy http://username:password@proxyaddr:proxyport` `ENV https_proxy http://username:password@proxyaddr:proxyport` Besides, to avoid integration test touch the proxy while connecting to local etcd service, you need to set `ENV no_proxy 127.0.0.1` </ins> ## TODOs These are in no particular order

I was tring to build kubernetes following But failed for some times. I looked into the error and found it's because I'm behind a proxy. Then I edited the dockerfile under build/build-image/ folder, some proxy settings into it and solved the problem. Though it may not be a problem for experienced guys. Would it be better for the beginners to describe these detaied settings in the document? Can I add the proxy settings I found into the doc? I'm new to kubernetes and need your opinion.
Yes, please, by all means. Please describe your problem and the solution in the doc and send a PR.


Question: {ConfigMap: &api.ConfigMapProjection{LocalObjectReference: api.LocalObjectReference{Name: "kube-root-ca.crt"}, Items: []api.KeyToPath{{Key: "ca.crt", Path: "ca.crt"}}}}, {DownwardAPI: &api.DownwardAPIProjection{Items: []api.DownwardAPIVolumeFile{{Path: "namespace", FieldRef: &api.ObjectFieldSelector{APIVersion: "v1", FieldPath: "metadata.namespace"}}}}}, }, <ins> DefaultMode: utilpointer.Int32(0644), </ins> }, }, }}

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --my mutating webhook calculates a hash of the , allows reinvocation; no side effects my validating webhook validates the hash of the and fails because the has changed webhook auditing indicates that my webhooks are the only ones processing the upon serializing the spec to JSON for both the mutator and validator, the only change is related to the "kube-api-access-xyz" projected volume, managed by a built-in k8s admission plugin the aforementioned plugin does not appear to mutate the reinvocation context, even through it may mutate a pod (robbing other mutators of the chance to observe the change): it appears that both my hooks see the injected volume, but the validation hook sees the injected SA volume with a "defaultMode" field set, and my mutation hook does NOT see this field set my validation hook should calculate the same hash mutating, validating webhooks are implemented atop controller-runtime v0.8.3, pod objects are being decoded into targets Kubernetes version (use ): 1.21.3 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g.

): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig api-machinery
cc
interesting... defaults are applied when going from versioned → internal server decodes v1 Pod request → applies defaults → converts to internal Pod -in admission control operates on the internal Pod service account admission injects the kube-api-access volume admission is called internal Pod → converts to v1 Pod → serializes → sends to mutating webhook mutating webhook returns patch patch is applied to v1 Pod JSON resulting v1 Pod JSON is decoded → defaults applied (defaultMode is set on the api-token volume here) → converted to internal Pod webhook admission is called internal Pod → converts to v1 Pod → serializes → sends to validating webhook That means if any built-in admission plugin sets up data that would be modified by applying defaulting to the pod, the data sent to mutating webhooks will be missing those applied defaults until the first mutating webhook that returns a patch that changes the object
Possible solutions: defaults are applied to the internal → versioned object before invoking webhooks built-in admission always populates data matching default values for unset fields
As an aside, I'm not sure what you're using the hashing for, but be aware that hashing is fragile in the presence of defaults and new fields over time. A hash you add to a pod spec in one release might not match a hash computed in a different release, or with different admission in place.
/triage accepted /assign /priority backlog


Question: } } <ins> // Finished writing into the file, close it, so we can delete it later. err = file.Close() assert.NoErrorf(t, err, "could not close file.") </ins> time.Sleep(20 * time.Millisecond) // Make the function ReadLogs end. fakeRuntimeService.Lock()

The test is failing on Windows [0]: The idea of the test is to check that kubelet still follows and reads the Container Logs after a Log Rotation occured. However, because the followed file is open, it cannot be renamed, and thus, the container logs cannot be rotated. This issue could be prevented if the file would be opened with the [0] [1] Kubelet should not block container log rotation. Run the test ` is failing on Windows. /sig windows /kind bug /kind failing-test latest, master. This unit test has been failing for a while. N/A Windows nodes </detailsN/A N/A N/A
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/kind failing-test /sig windows


Question: ) // updateReplicationControllerStatus attempts to update the Status.Replicas of the given controller, with a single GET/PUT retry. <del> func updateReplicationControllerStatus(c v1core.ReplicationControllerInterface, rc v1.ReplicationController, newStatus v1.ReplicationControllerStatus) (updateErr error) { </del> <ins> func updateReplicationControllerStatus(c v1core.ReplicationControllerInterface, rc v1.ReplicationController, newStatus v1.ReplicationControllerStatus) (*v1.ReplicationController, error) { </ins> // This is the steady state. It happens when the rc doesn't have any expectations, since // we do a periodic relist every 30s. If the generations differ but the replicas are // the same, a caller might've resized to the same replica count.

Is this a request for help? No. What keywords did you search in Kubernetes issues before filing this one? dangling ReplicaSet Is this a BUG REPORT or FEATURE REQUEST? BUG REPORT Kubernetes version (use ): Environment: Cloud provider or hardware configuration: Azure A1 OS (e.g. from /etc/os-release): NAME="Container Linux by CoreOS" ID=coreos VERSION=1235.9.0 VERSIONID=1235.9.0 BUILDID=2017-02-02-0235 PRETTYNAME="Container Linux by CoreOS 1235.9.0 (Ladybug)" ANSICOLOR="38;5;75" HOMEURL="" BUGREPORTURL="" Kernel (e.g. ): Linux node-3-vm 4.7.3-coreos-r2 SMP Thu Feb 2 02:26:10 UTC 2017 x8664 Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40GHz GenuineIntel GNU/Linux Install tools: own Others: What happened: Created new deployment by altering the image tag of the deployment descriptor and using kubectl apply -f "". Noticed that one container of the previous deployment stayed there. What you expected to happen: All containers from the previous deployment should have been terminated. How to reproduce it (as minimally and precisely as possible): I cannot reliably trigger this, I do several deployments per hour and sometimes this problem manifests itself. Anything else we need to know: Deployment descriptor:
Can you also post the output of kubectl get deployment tviftttapp -o yaml kubectl get rs (dangling rs) -o yaml When this manifests again?
Here are the current deployment, the dangling RS, and the "good" RS. Deployment: Dangling RS: Current RS:
It seems that status.availableReplicas for the new replica set is never correctly updated to 2. Short term if you stop using minReadySeconds (minReadySeconds=2 doesn't offer you much anyway), this problem should stop. Long term, we should fix the status update. cc:
can one of you guys paste logs from the controller manager when this issue occurs? Log level should be ideally set to 2.

I managed to reproduce it:
Will close this one once is cherry-picked into 1.5
4 is out and contains a fix that should force the Deployment to be resynced eventually and have availableReplicas updated. Not optimal yet but eventually consistent. I have queued up some follow-up work to refine the resync.
Does anyone know when version 1.5.4 OR 1.6 will be available using Redhat package downloads (in RPM form) ? I only see version 1.5.2 being the latest there. I have the same problem that is being discussed in this thread and want to upgrade so that my K8S cluster can support Deployments.


Question: if err != nil { klog.V(3).Infof("Unable to check for quota support on %s: %s", dir, err.Error()) } else if hasQuotas { <del> _, err := fsquota.GetQuotaOnDir(ed.mounter, dir) if err != nil { klog.V(4).Infof("Attempt to check quota on dir %s failed: %v", dir, err) // this is not a fatal error so return nil return nil } </del> klog.V(4).Infof("emptydir trying to assign quota %v on %s", mounterSize, dir) if err := fsquota.AssignQuota(ed.mounter, dir, ed.pod.UID, mounterSize); err != nil { klog.V(3).Infof("Set quota on %s failed %s", dir, err.Error())

I have kubernetes 1.24.7. I tried to use on formated volume. It works until first reboot. After reboot, I get messages: I expect that project quota can be assigned even after reboot. Use xfs for /var/lib/containerd with discard,quota,prjquota No response <details</details<detailsbaremetal </details<details</details<details</details<detailsrke2 </details<details</details
/sig node Can this be reproduced? Can this pod be recreated correctly if you force delete the pod?
/triage accepted
It can be reproduced. Pod is created if I remove . I can test force delete.
/assign we are looking into gate to understand if we need to keep it or deprecate it.
If I force delete pod, it does not start again:
/assign Let me check on the latest code with the fix


Question: return "", "", err } <del> diskController = getSCSIController(vmDevices, vs.cfg.Disk.SCSIControllerType) if diskController == nil { </del> <ins> scsiController = getSCSIController(vmDevices, vs.cfg.Disk.SCSIControllerType) if scsiController == nil { </ins> glog.Errorf("cannot find SCSI controller in VM - %v", err) // attempt clean up of scsi controller cleanUpController(newSCSIController, vmDevices, vm, ctx)

vSphere cloud provider volume plugin attach disk limits the number of volumes because of two issues: new SCSI controller is only created when one does not exist. A new controller should be when required and limit not reached. disk will fail when attaching disk to slot 7 on controller. Slot 7 on SCSI controller is reserved and therefore should be skipped. This issue has been brought up in Govmomi library .