Question: framework.Failf("Failed to get statefulset resource: %v", err) } framework.ExpectEqual(*(ss.Spec.Replicas), int32(2)) <ins> ginkgo.By("Patch a scale subresource") scale.ResourceVersion = "" // indicate the scale update should be unconditional scale.Spec.Replicas = 4    // should be 2 after "UpdateScale" operation, now Patch to 4 ssScalePatchPayload, err := json.Marshal(autoscalingv1.Scale{ Spec: autoscalingv1.ScaleSpec{ Replicas: scale.Spec.Replicas, }, }) framework.ExpectNoError(err, "Could not Marshal JSON for patch payload") _, err = c.AppsV1().StatefulSets(ns).Patch(context.TODO(), ssName, types.StrategicMergePatchType, []byte(ssScalePatchPayload), metav1.PatchOptions{}, "scale") framework.ExpectNoError(err, "Failed to patch stateful set: %v", err) ginkgo.By("verifying the statefulset Spec.Replicas was modified") ss, err = c.AppsV1().StatefulSets(ns).Get(context.TODO(), ssName, metav1.GetOptions{}) framework.ExpectNoError(err, "Failed to get statefulset resource: %v", err) framework.ExpectEqual(*(ss.Spec.Replicas), int32(4), "statefulset should have 4 replicas") </ins> }) })

[ ] two weeks soak start date : testgrid-link [ ] two weeks soak end date : [ ] test promotion pr : kubernetes/kubernetes#? According to this APIsnoop query, there are still some remaining StatefulsetScale endpoints that are untested. with this query, you can filter untested endpoints by their category and eligibility for conformance. e.g below shows a query to find all conformance eligible untested, stable, apps endpoints sql-mode select distinct endpoint, right(useragent,73) AS useragent from testing.auditevent -- where useragent ilike '%subresource%' where endpoint ilike '%AppsV1NamespacedStatefulSetScale%' and releasedate::BIGINT round(((EXTRACT(EPOCH FROM NOW()))::numeric)1000,0) - and useragent like 'e2e%' order by endpoint limit 30; example endpoint | useragent -----------------------------------------|--------------------------------------------------------------------------- patchAppsV1NamespacedStatefulSetScale | [StatefulSetBasic] should have a working scale subresource [Conformance] readAppsV1NamespacedStatefulSetScale | [StatefulSetBasic] should have a working scale subresource [Conformance] replaceAppsV1NamespacedStatefulSetScale | [StatefulSetBasic] should have a working scale subresource [Conformance] (3 rows) If a test with these calls gets merged, test coverage will go up by 1 point* This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: "email": "jane@example.com", "exp": %d }`, valid.Unix()), <ins> want: &user.DefaultInfo{ Name: "jane@example.com", }, }, { name: "invalid-email-verified-claim", options: Options{ IssuerURL:     "https://auth.example.com", ClientID:      "my-client", UsernameClaim: "email", now:           func() time.Time { return now }, }, signingKey: loadRSAPrivKey(t, "testdata/rsa_1.pem", jose.RS256), pubKeys: []*jose.JSONWebKey{ loadRSAKey(t, "testdata/rsa_1.pem", jose.RS256), }, // string value for "email_verified" claims: fmt.Sprintf(`{ "iss": "https://auth.example.com", "aud": "my-client", "email": "jane@example.com", "email_verified": "false", "exp": %d }`, valid.Unix()), </ins> wantErr: true, }, {

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: Bug Report /kind bug What happened: This is related to I am trying to integrate Cloud Foundry UAA () with kubernetes using the OIDC flow and the JWT validation is failing on the kubernetes side with an error of emailverified is required. emailverified is not a required claim in the OIDC core spec From a security point of view emailverified is useful when Self Service is enabled which is true of internet identity providers User Provisioning workflow in almost all cases is admin driven and not self service and emailverified becomes an obsolete claim. With things as they stand today, this flow is not conducive to Enterprise Identity Providers. What you expected to happen: Authentication should succeeed How to reproduce it (as minimally and precisely as possible): Pass a JWT which is missing email_verified claim using the OIDC flow into kubernetes Anything else we need to know?: Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-12-16T03:15:38Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"darwin/amd64"} Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
It's not marked optional either, which was the original motivation for taking the safe route and assuming false if it wasn't provided. This would be a feature for us to detect the presence of the claim and act differently if it isn't there. Do any other providers do this? Almost all that I've worked with do support this claim.
Hi , Enterprise providers like ADFS , Okta don't support this as part of the user schema. There is the concept of whether a use is active/enabled which is not the same as email verified always.

If a user is not active, a token should not be minted in the first place. I am trying to understand what additional security does email_verified provided in the context of entire identity providers where there is no self self service use creation. Thanks, Sree
Okta supports , but it looks like ADFS doesn't: We don't know when the provider is self-service or not. Not assuming makes the client logic uniform and less error prone instead of trying to reason about the intent of the provider. There's some discussion when we enabled this validation, including some UAA notes. , specifically
More than 80% of the enterprise customers of Cloud Foundry are using either AD/ADFS . This becomes a breaking change. UAA can add a email_verified claim. However its going to be of little value if the underlying AD doesn't propagate that. My two cents is this check doesn't make sense in an enterprise set up
has agreed to take this (can't assign her because she's not part of the org) /sig auth /kind feature /milestone v1.11 /priority important-longterm /status approved-for-milestone
[MILESTONENOTIFIER] Milestone Issue: Up-to-date for process <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Escalate to the issue owners; move out of the milestone after 1 attempt. : New functionality. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details


Question: } ds.networkPlugin = plug glog.Infof("Docker cri networking managed by %v", plug.Name()) <ins> // NOTE: cgroup driver is only detectable in docker 1.11+ var cgroupDriver string dockerInfo, err := ds.client.Info() if err != nil { glog.Errorf("failed to execute Info() call to the Docker client: %v", err) glog.Warningf("Using fallback default of cgroupfs as cgroup driver") } else { cgroupDriver = dockerInfo.CgroupDriver if len(kubeCgroupDriver) != 0 && kubeCgroupDriver != cgroupDriver { return nil, fmt.Errorf("misconfiguration: kubelet cgroup driver: %q is different from docker cgroup driver: %q", kubeCgroupDriver, cgroupDriver) } glog.Infof("Setting cgroupDriver to %s", cgroupDriver) } ds.cgroupDriver = cgroupDriver </ins> return ds, nil }

Kubernetes version (use ): v1.5.0-alpha.1.2189+-dirty docker: 1.12.3 Environment: fedora24 docker-engine-1.12.3-1.fc24.x86_64 What happened: if docker Cgroup Driver=cgroupfs, and set --cgroup-driver=systemd in kubelet, then enable --experimental-cgroups-per-qos=true; The container cgroup some is not correct. How to reproduce it (as minimally and precisely as possible): Anything else do we need to know: 1) docker Cgroup Driver=cgroupfs and --cgroup-driver=cgroupfs in kubelet works well 2) docker Cgroup Driver=systemd and --cgroup-driver=systemd in kubelet works well
help check, thanks.
In Kubernetes 1.6, we can work to detect this case and return an error. IMHO its a configuration error to have docker running with a cgroupfs driver paired with the kubelet running the systemd cgroup driver. /cc
We can also document the compatibility matrix for now.


Question: // load SNI certs namedTLSCerts := make([]dynamiccertificates.SNICertKeyContentProvider, 0, len(s.SNICertKeys)) for _, nck := range s.SNICertKeys { <del> tlsCert, err := dynamiccertificates.NewStaticSNICertKeyContentFromFiles(nck.CertFile, nck.KeyFile, nck.Names...) </del> <ins> tlsCert, err := dynamiccertificates.NewDynamicSNIContentFromFiles("sni-serving-cert", nck.CertFile, nck.KeyFile, nck.Names...) </ins> namedTLSCerts = append(namedTLSCerts, tlsCert) if err != nil { return fmt.Errorf("failed to load SNI cert and key: %v", err)

When we want to rotate certificates for aggregated API servers, we have to restart (kill) pods in order for API server to pick the new certificates. It would be very useful to implement some sort of self-awareness for the aggregated API servers, so they can pick-up new certificates when they're updated, without user interaction. If we have certificate files in pods, that are provided by some volumeSource, that can be persistent storage or a Secret, when file gets changed on the volume, the kubelet updates file in the pod usually in matters of few seconds. However, aggregated API servers are not self-aware of such change and users have to manually restart the pods. This brings additional complexity, especially if you want to automate the process. The first problem is that you need to know what pods you need to restart. But the potentially biggest problem is that you need additional permissions to do so. That would include RBAC permissions for listing pods, listing deployments and for deleting pods. That can make a security issue in some setups. By making aggregated API servers sort of self-aware, this could be avoided. Create a health check somewhere in , that utilizes a file watcher to watch for certificates changes. When certificate is changed, fail a health check. Failing health check will restart the pod without user interaction. This could be the most easiest and simplest solution to the problem. Such feature should/could be optional and guarded by a flag, let's name it . Some delay could be required between restarting pods. We probably don't want to kill all pods in the same time. This is up to be discussed. I would like to work on this issue, so feel free to assign it to me. /kind feature /sig api-machinery /area apiserver /cc
this could also be done by a purpose-built container that watched the same input files. as you point out, you likely would not want to restart all apiservers at once, which makes it unclear whether you'd want to build in the restart logic to the individual servers
That coordination would certainly work and be a valuable tool for someone to build to help manage rolling generic processes if we never get a pod restart. However, being able to have a "usually won't all go down together" option built-in is easy to understand and will resolve the problem until then.

It isn't a stretch to say that if your process isn't fulfilling the current intent of the pod, it isn't healthy. This gives us a thing between dynamic config reloading and nothing. A container that can coordinate based on a service would be really cool, but I think that'd be a separate thing.
/cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: eventsRule(), }, }) <del> addControllerRole(&controllerRoles, &controllerRoleBindings, rbacv1.ClusterRole{ ObjectMeta: metav1.ObjectMeta{Name: saRolePrefix + "disruption-controller"}, Rules: []rbacv1.PolicyRule{ rbacv1helpers.NewRule("get", "list", "watch").Groups(extensionsGroup, appsGroup).Resources("deployments").RuleOrDie(), rbacv1helpers.NewRule("get", "list", "watch").Groups(appsGroup, extensionsGroup).Resources("replicasets").RuleOrDie(), rbacv1helpers.NewRule("get", "list", "watch").Groups(legacyGroup).Resources("replicationcontrollers").RuleOrDie(), rbacv1helpers.NewRule("get", "list", "watch").Groups(policyGroup).Resources("poddisruptionbudgets").RuleOrDie(), rbacv1helpers.NewRule("get", "list", "watch").Groups(appsGroup).Resources("statefulsets").RuleOrDie(), rbacv1helpers.NewRule("update").Groups(policyGroup).Resources("poddisruptionbudgets/status").RuleOrDie(), rbacv1helpers.NewRule("get").Groups("*").Resources("*/scale").RuleOrDie(), eventsRule(), }, }) </del> <ins> addControllerRole(&controllerRoles, &controllerRoleBindings, func() rbacv1.ClusterRole { role := rbacv1.ClusterRole{ ObjectMeta: metav1.ObjectMeta{Name: saRolePrefix + "disruption-controller"}, Rules: []rbacv1.PolicyRule{ rbacv1helpers.NewRule("get", "list", "watch").Groups(extensionsGroup, appsGroup).Resources("deployments").RuleOrDie(), rbacv1helpers.NewRule("get", "list", "watch").Groups(appsGroup, extensionsGroup).Resources("replicasets").RuleOrDie(), rbacv1helpers.NewRule("get", "list", "watch").Groups(legacyGroup).Resources("replicationcontrollers").RuleOrDie(), rbacv1helpers.NewRule("get", "list", "watch").Groups(policyGroup).Resources("poddisruptionbudgets").RuleOrDie(), rbacv1helpers.NewRule("get", "list", "watch").Groups(appsGroup).Resources("statefulsets").RuleOrDie(), rbacv1helpers.NewRule("update").Groups(policyGroup).Resources("poddisruptionbudgets/status").RuleOrDie(), rbacv1helpers.NewRule("get").Groups("*").Resources("*/scale").RuleOrDie(), eventsRule(), }, } if utilfeature.DefaultFeatureGate.Enabled(features.PodDisruptionConditions) { role.Rules = append(role.Rules, rbacv1helpers.NewRule("patch").Groups(legacyGroup).Resources("pods/status").RuleOrDie()) } return role }()) </ins> addControllerRole(&controllerRoles, &controllerRoleBindings, rbacv1.ClusterRole{ ObjectMeta: metav1.ObjectMeta{Name: saRolePrefix + "endpoint-controller"}, Rules: []rbacv1.PolicyRule{

The of a stale pod disruption condition remains after two minutes as the disruption controller fails to change it to to disable the condition. The of a stale pod disruption condition should be changed to after two minutes. with Alpha feature gate enabled is without associated delete operation (it may happen if preemption delete call fails, but it can be simulated by API) 2min as after this time the condition is recognized as stale by disruption controller, as the pod is not terminating The patch of disruption controller fails with the following message in the logs: <details25.x <!-- --</details<detailsReproducible on KIND. </details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig apps /assign


Question: <ins> /* Copyright 2022 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ // This is an empty file to avoid "build constraints exclude all Go files" error while running unit tests on non-amd64 && non-linux platforms. package cleanup </ins>

Recent failed job : https://prow.ppc64le- Expected tests to be run fine without any issues Run following command on any non-amd64/linux platform Could be the side effect of PR <details</details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig testing


Question:
mountPropagation

mountPropagation determines how mounts are propagated from the host to container and the other way around. When not set, MountPropagationHostToContainer is used. This field is alpha in 1.8 and can be reworked or removed in a future release.

mountPropagation determines how mounts are propagated from the host to container and the other way around. When not set, MountPropagationHostToContainer is used. This field is beta in 1.10.

false

v1.MountPropagationMode


Question: return obj3 } <ins> func TestSetDefaultReplicationController(t *testing.T) { tests := []struct { rc             *current.ReplicationController expectLabels   bool expectSelector bool }{ { rc: &current.ReplicationController{ DesiredState: current.ReplicationControllerState{ PodTemplate: current.PodTemplate{ Labels: map[string]string{ "foo": "bar", }, }, }, }, expectLabels:   true, expectSelector: true, }, { rc: &current.ReplicationController{ Labels: map[string]string{ "bar": "foo", }, DesiredState: current.ReplicationControllerState{ PodTemplate: current.PodTemplate{ Labels: map[string]string{ "foo": "bar", }, }, }, }, expectLabels:   false, expectSelector: true, }, { rc: &current.ReplicationController{ Labels: map[string]string{ "bar": "foo", }, DesiredState: current.ReplicationControllerState{ ReplicaSelector: map[string]string{ "some": "other", }, PodTemplate: current.PodTemplate{ Labels: map[string]string{ "foo": "bar", }, }, }, }, expectLabels:   false, expectSelector: false, }, { rc: &current.ReplicationController{ DesiredState: current.ReplicationControllerState{ ReplicaSelector: map[string]string{ "some": "other", }, PodTemplate: current.PodTemplate{ Labels: map[string]string{ "foo": "bar", }, }, }, }, expectLabels:   true, expectSelector: false, }, } for _, test := range tests { rc := test.rc obj2 := roundTrip(t, runtime.Object(rc)) rc2, ok := obj2.(*current.ReplicationController) if !ok { t.Errorf("unexpected object: %v", rc2) t.FailNow() } if test.expectSelector != reflect.DeepEqual(rc2.DesiredState.ReplicaSelector, rc2.DesiredState.PodTemplate.Labels) { if test.expectSelector { t.Errorf("expected: %v, got: %v", rc2.DesiredState.PodTemplate.Labels, rc2.DesiredState.ReplicaSelector) } else { t.Errorf("unexpected equality: %v", rc2.DesiredState.PodTemplate.Labels) } } if test.expectLabels != reflect.DeepEqual(rc2.Labels, rc2.DesiredState.PodTemplate.Labels) { if test.expectLabels { t.Errorf("expected: %v, got: %v", rc2.DesiredState.PodTemplate.Labels, rc2.Labels) } else { t.Errorf("unexpected equality: %v", rc2.DesiredState.PodTemplate.Labels) } } } } </ins> func TestSetDefaultService(t *testing.T) { svc := &current.Service{} obj2 := roundTrip(t, runtime.Object(svc))

It doesn't make sense to define pod template with a set of labels that is strict subset of replicaSelector (or at least I can't find an use-case). For that reason I think that the labels in pod template part of replication controller should be optional and user should care about them only if he wants to define a superset of what replicaSelector already defines.
I've heard the same comments from a client. I think that it'd be more intuitive if we always specify pod labels in the template (so that we can copy&paste pod definition), and replicaSelector is optional (by default pod labels).
It would be reasonable for the selector to be a subset. I'm fine with intelligent defaulting but we need to be careful not to conflate the 2 concepts, which I think would increase the amount of confusion.
Setting RC selector from pod template labels SGTM. Any concerns It wouldn't have to work with separate templates, though it would be desirable if it did. It would be nice if we could add preconditions based on other objects (resource version of pod template in this case) to etcd transactions.
I think we agreed in another issue that defaulting the selector from the pod template labels is reasonable. No argument from me. One problem with pod templates is that we lose the strict relationship between selector and template, so if you change the template the RC needs to behave intelligently (I refuse to create more pods because the template selector doesn't match mine, etc)
We should do this, and we should default replicas to 1. We could also consider defaulting replication controller labels from pod template labels. That way, essentially the only thing that would need to be specified for a singleton RC would be the pod template.
I don't think we can default replicas to 1, since we can't tell the difference between an explicit zero and a default zero, and I think there are lots of reasonable cases where you want to set replicas == 0 on create. (esp. in the context of socket activation)
Responded on the PR: We can distinguish default from 0 if we make it a pointer.


Question: import ( "context" "fmt" <ins> "path" </ins> "time" "github.com/Azure/azure-sdk-for-go/services/compute/mgmt/2019-03-01/compute"

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Pods with PVs could not be moved from one node to another. According to the controller-manager logs, K8s tried to detach to Azure disks from the same node. One of the detach operation waited for this lock (). While it waited for the lock, the controller-manager pod was restarted (for whatever reason, not related to this bug, lets assume an operator restarted the pod). After the restart, K8s only tried to attach the disk to the new node but is not trying to detach it first. What you expected to happen: After a restart of the controller-manager, it should continue unfinished detach operations How to reproduce it (as minimally and precisely as possible): 2 Pods with PVs running on one node Move both pods to another node restart the controller-manager shortly after the move Anything else we need to know?: Environment: Kubernetes version (use ): 1.15.1 Cloud provider or hardware configuration: Azure OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig azure
/sig cloud-provider
/area provider/azure that's expected behavior, k8s controller won't continue detach volume if controller manager restarted, user needs to detach that disk volume manually.
Could you please elaborate why this behaviour is expected. I would assume that the controller manager tries to reach the desired state after a restart? Then it would need to detect that the volume is still attached to the wrong node and should try to detach it once more, before it continuing to attach the (still attached) volume? This leads to many ... Why is it required to resolve this situation by manual interaction?
I can't imagine that "manual intervention" is "desired behaviour" (yes, you wrote "expected", but that doesn't matter, if you make no proposal, how to improve this). If this isn't fixed, we can't use Azure, period. We can't have clusters that need to be babysitted / you cannot expect people running workloads on Azure clusters and babysit them and manually fixing what should/can fix itself.

Level-triggered behaviour and resilience is what made Kubernetes so successful... please bring that also to Azure's KCM/CCM.
do you know why controller manager is restarted, it's not a common scenario, we cannot guarantee disk attach/detach functionality works if controller manager restarted at that time.
related to
I think there can be many reasons why the controller-manager is restarted during running detach or attach operations e. g. the hosting machine turn into error state, if you run the controller-manager in k8s -vertical pod autoscaling,... etc. I understand you can't guarantee that attach/detach works when the controller-manager is restarted, but I don't understand why the controller-manager can't recover the situation autonomously after a restart. After restart it tries to attach the volume of the rescheduled pod and this does not work because the volume is still attached to another machine. This is indicated by a error from the arm and the controller-manager could react to it properly, right?
This is an upstream issue, and there is a discussion in , let's check whether we can fix it in cloud provider driver in that thread. Thanks.
I will try to fix this issue in azure disk driver, thanks all for the discussion. details: /assign
Thank you very much It means a lot to us, that there are no manual operations. I wonder how you guys deal with it in AKS, since Azure uses the same controllers, right? Anyway, it helps tremendously, that the clusters are self-healing.
AKS use same kube-controller-manager, while it won't be restarted easily.
OK, but how? I understand that AKS only supports 100 nodes. So, I guess you have some upper limit and just hope, nobody stresses it, because if this happens, you it either restarts or you need to scale it up and then it gets restarted as well. Also, regular maintenance, low-level as well as Kubernetes updates, all must impact it. Considering the probably thousands over thousands of clusters (I don't know, but I expect you run many), any such problem, even if unlikely by itself, must be in your interest to fix, simply because the number of clusters increases the likelihood that it happens, I would think. Anyway, thank you for making this change
Yes, that's reasonable, I will try to fix it as my top priority.


Question: windows.JobObjectBasicLimitInformation, uintptr(unsafe.Pointer(&limitInfo)), uint32(unsafe.Sizeof(limitInfo))); err != nil { <del> return windows.InvalidHandle, errors.Wrap(err, "windows.SetInformationJobObject failed") </del> <ins> return windows.InvalidHandle, fmt.Errorf("windows.SetInformationJobObject failed: %w", err) </ins> } return job, nil }

Why is this needed? The functionality that the package provides is already present in golang's builtin errors (using in ) and the github repo is now read-only. /kind cleanup /priority backlog /good-first-issue Notes to people that want to collaborate Avoid creating a single PR for the entire repo. Create multiple PRs, each targeting a specific SIG. Do not submit PRs for
/area code-organization
/kind cleanup
this is only partially true, e.g. %w does not provide stack traces
/assign
I see. Should we close this issue then?
I think it depends on how it's being used. If the traces are not the goal then we can do a simple substitution. It's probably best to drop read only dependencies ASAP but it's worth noting that if we depended on the full functionality in any way we'll need a more complex replacement (KIND did/does, not sure about k/k)
k/k git log suggests that some cleanup has already been done in the past (maybe in other commits too): This issue triggered 3 PRs and 2 of them are merged. there does not seem to be calls to so most cases it looks it's just wrapping errors.
for cmd/kubeadm in kubernetes/kubernetes we decided to not drop usage of pkg/errors even if it's a deprecated project. prior ticket and comments are here
for cmd/kubeadm in kubernetes/kubernetes we decided to not drop usage of pkg/errors even if it's a deprecated project. prior ticket and comments are here
I see. Added a note.
is it still open ?
should be replaced with the standard library for the following reasons: archived library with no maintenance or security updates the go community used pkg/errors to introduce eventually into the standard library hence the approach that was selected is more idiomatic stack traces are hardly used in error wrapping and are not a reason not to refactor
On place left in Except the k/k repo, as is archived, some other repos may need an update.
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage accepted
The label cannot be applied. Only GitHub organization members can add the label. <detailsIn response to : /triage accepted Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
is it necessary to remove "" from every location? I noticed it in many places. Also, do I need to create separate PRs for each file where I remove it?
/assign
From kuberentes/kubernetes repo view, only two locations of kubectl code should be updated. (Do not submit PRs for ) see For other repos, we should follow up until the dependencies' repo is updated. I will track them in
/assign
/assign
to remove dependency you can use the standard Go error package or creating your own handling package, this could contain custom functions for generating and handling errors.
let me work on this issue now please /assign
We are in code freeze. I suggest waiting for a couple of weeks.
Hello, is this still up?
Let's get this going
/assign
Just opened a PR for this issue
/assign
Update the remaining todo items.
/sig architecture umbrella / code org
/assign
Hey Are you still working on this one? If not- would you mind if I take a stab at it? Thanks!
Is this issue really still open? When I do I only get hits for cmd/kubeadm. Am I missing something?
/close
Closing this issue. <detailsIn response to : /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: sort.Sort(l) } <del> // HigherPriorityPod return true when priority of the first pod is higher than // the second one. It takes arguments of the type "interface{}" to be used with // SortableList, but expects those arguments to be *v1.Pod. func HigherPriorityPod(pod1, pod2 interface{}) bool { return GetPodPriority(pod1.(*v1.Pod)) > GetPodPriority(pod2.(*v1.Pod)) </del> <ins> // MoreImportantPod return true when priority of the first pod is higher than // the second one. If two pods' priorities are equal, compare their StartTime. // It takes arguments of the type "interface{}" to be used with SortableList, // but expects those arguments to be *v1.Pod. func MoreImportantPod(pod1, pod2 interface{}) bool { p1 := GetPodPriority(pod1.(*v1.Pod)) p2 := GetPodPriority(pod2.(*v1.Pod)) if p1 != p2 { return p1 > p2 } return GetPodStartTime(pod1.(*v1.Pod)).Before(GetPodStartTime(pod2.(*v1.Pod))) </ins> }

<!-- Please only use this template for submitting enhancement requests --What would you like to be : When selecting victims on node in (), take pod's into account. When two pod's Priority is equal, preempt pod whose is later first. Why is this needed: If a pod's has been running for a longer time, it might finish earlier, or it might have done more things. It might make more sense to preempt them later, e.g. a user submitted a job A at time 9:00 am, then he submitted a job B at time 10:00 am, he might expect that scheduler preempts pods for job B first. <!-- DO NOT EDIT BELOW THIS LINE --/kind feature
/sig scheduling /assign cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/close
Closing this issue. <detailsIn response to : /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
PR implements this feature. We are currently in code freeze for 1.14. The feature will be in 1.15.
Thanks for it. It is similar with a little difference, the PR aims to select a node, and this issue aims to pick a pod whose StartTime is latter after the node has been selected.
I just sent a PR for it. PTAL :)


Question: return fmt.Errorf("nil %q node object", kl.nodeName) } <del> kl.updatePodCIDR(node.Spec.PodCIDR) </del> <ins> if node.Spec.PodCIDR != "" { kl.updatePodCIDR(node.Spec.PodCIDR) } </ins> kl.setNodeStatus(node) // Patch the current status on the API server

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: In my test setup I allocate POD CIDRs manually, bridge is configured before starts. Important settings: on example node: on master: Then I see confusing entries in logs (): Fortunately everything keeps working, thanks to this condition: , i.e. it is not propagated to CNI plugin. It appears that this very line tries to set the CIDR to whatever the master returns during node status update: What you expected to happen: / use the pod CIDR given at command line and never mess with it Actually I'm a bit confused if manual per-node pod CIDR assignment is supported at all, and if I'm doing it correctly. I'm aware it's unusual thing to do in production environment, yet for testing it seems perfectly legitimate. Environment: Kubernetes version (use ): v1.10.0 Cloud provider or hardware configuration: bare metal OS (e.g. from /etc/os-release): Ubuntu 17.10 Kernel (e.g. ): Install tools:
/sig network
Kubelet need to make a check of CIDR before
/assign


Question: {"foo": "bar"}, {"baz": "biz"}, }}, <ins> {"---nfoo: barn--- # with Commentnbaz: biz", 100, false, false, []generic{ {"foo": "bar"}, {"baz": "biz"}, }}, </ins> {"foo: barn---n", 100, false, false, []generic{ {"foo": "bar"}, }},

Attempting to create three namespaces (foo, bar, baz). Notice the first YAML document separator has the comment after it on the same line. Output: Exit code is 0 and only and were created. It should either create all three namespaces or else it should fail with an error telling me my YAML is invalid. See Above. Same thing happens if I put random characters after the separator. For example: No error message is printed, exit code is 0, and the next resource is just silently ignored Kubernetes version (use ):Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/cc
/assign
/sig api-machinery likely something that does not tolerate? you can try reproducing that with a unit test.
Thanks for the hint! I actually had a look at it and the Yaml document's termination logic seems to happen here: We currently assume that three dashes only terminate the document if followed by spaces (or by new line immediately), otherwise we keep scanning the document consider it as a single doc (including three dashes and any characters that follow). Ref: Doing that will make the yaml unmarshalling ignore everything that is followed by these three dashes (as it does not support unmarshalling multiple docs) To prove it: Will output: According to the Yaml Spec it's OK to have any character after these three dashes, but it will be considered part of the new doc already (as three dashes indicate seperation of Yaml directives from document's content) see One thing to note here is we ignore the use of three dots ("...") as a separator of one document's from another's Yaml directives. So for example if we had this kind of Yaml we would consider to be part of the first doc, which is wrong assumption according to the spec. To keep it simple though, I will sign a PR (including tests) later that will handle the three dashes and the followed characters to be a termination indicator and possible taken under consideration for the following (second) doc.
Thanks for having a look.
Let us know when you send a PR, thank you!

/triage accepted
I would love to have your opinions on the following: There are 3 fix stages to this bug as I see it: 1) We should split to different Yaml documents when three dashes appears, whether they followed by comments (or any other content), and not only spaces, so we would handle the strange behavior described in the bug description. 2) We should consider three dashes as part of the next document (and not only as a terminator - so for example in we won't be dropping the existence of This one is breaking because when three dashes appear at the end of the document i.e we would consider this file as containing 2 docs (the second is empty). 3) We need to take 3 dots under consideration as described above. This would also be breaking change obviously (and in many more ways in contrast to the 2nd change) and in my opinion needs to be filed as an entirely different issue Each one of these changes get us closer to an alignment state with the Yaml spec. What do you suggest to include for this PR, stage1, or stage 2 as well?
I'm in the process of . To keep it simple, the approach I settled on, at least so far, is to allow (and ignore) white space or comments, but any other content will result in an error. As you pointed out there is a lot more to the YAML syntax. However, it can be quite a bit more work trying to support all the possible things that are syntactically correct but not something that would be used in kubernetes, like . I also chose not to address because my impression was that this is mostly used for "streamed" YAML where there is no EOF. I actually didn't even know that existed before digging into the spec. I'm not saying it would be bad to support that, just that I don't think anyone uses it for their manifests so the "ROI" for making that change would be low.
my vote would be to not introduce breaking changes, because this code path is used by literally everyone with manifest files. note that the same code is used for JSON multi-doc splitting too. i think ideally we should not be splitting at any occurrence of . is the specification explicit about the being at position 0 of a new line?

if so the accurate way would be: split at add back a to item add everything after to item (excluding the delimiter ) but that might be breaking already. i think the current behavior is to drop empty docs and the start and end. applies to: resulting in only one doc. this is quite common in manifests. probably we shouldn't manage 3 dots because i haven't seen requests for it. also depends on what the spec says and if the split should happen at and not any . special casing only white space and comments after a makes sense for the time being, IMO.
I agree, breaking changes are dangerous here and should be done gradually (although you right we already apply empty docs as in this case: so taking things like as described above would not be actually breaking. Anyways, I think we'll act cautiously as a result of this change's sensitivity, and go with approach to this here by special casing whitespaces and comments. Thanks for the involvement.
PR is ready. ^


Question: glog.V(2).Infof("SyncLoop (PLEG): %q, event: %#v", format.Pod(pod), e) handler.HandlePodSyncs([]*api.Pod{pod}) } if e.Type == pleg.ContainerDied { <del> if podStatus, err := kl.podCache.Get(e.ID); err == nil { if containerID, ok := e.Data.(string); ok { kl.containerDeletor.deleteContainersInPod(containerID, podStatus) } </del> <ins> if containerID, ok := e.Data.(string); ok { kl.cleanUpContainersInPod(e.ID, containerID) </ins> } } case <-syncCh:

Kubelet evicts pods whenever there is memory or disk pressure. It marks the pods as to evict them. To help with reducing disk pressure, kubelet should delete all containers belonging to pods right away. This will result in loss of logs. In the future, we can copy out logs from dead containers before deleting them. Logs can be rotated separately from dead containers. IMO, this feature is required for disk usage based evictions to work reliably in kubelet.


Question: } } <ins> crdSpec := &spec.Swagger{} </ins> for _, s := range crdSpecs { <del> mergeSpec(&specToReturn, s) </del> <ins> // merge specs without checking conflicts, since the naming controller prevents // conflicts between user-defined CRDs mergeSpec(crdSpec, s) </ins> } <del> return &specToReturn </del> <ins> // The static spec has the highest priority. Resolve conflicts to prevent user-defined // CRDs potentially overlapping the built-in apiextensions API if err := aggregator.MergeSpecsIgnorePathConflict(&specToReturn, crdSpec); err != nil { return nil, err } return &specToReturn, nil </ins> } // mergeSpec copies paths and definitions from source to dest, mutating dest, but not source.

What happened: Created a CRD with a group/version/kind that defines a CustomResourceDefinition type overlapping with the in-tree kubernetes type (WARNING: do not create this CRD on a cluster you care about, see ) Note that two definitions for the group/version/kind appear in openapi: What you expected to happen: published openapi only includes the in-tree type, which has precedence /cc /area custom-resources /priority important-soon /sig api-machinery
/cc
haven't looked into the code yet. I'd expect the naming controller / apiextensions server to prevent conflicting CR GVK, but they are unaware of builtin API groups
CRD handling is isolated to the apiextensions server... it does not have information about all other built-in and aggregated APIs
mentioned something he was working on to make the aggregation layer honor APIService priorities (so that later definitions would not be included if something had already defined a GVK) edit: possibly
a naive idea is to reserve k8s groups in a hard code way that sounds good the CRD in defines a but the kubectl results are about . Did I miss something?
oops... fixed. I started with an API type under but the issue only surfaces for specific APIs (due to the generated definition name colliding for some types)
that wouldn't address duplicate GVKs from aggregated APIs
Three kinds of openapi paths/definitions that we want to prevent CRDs from overwriting: CRD API the spec aggregation in apiextensions-apiserver allows overwriting. I will send a fix native APIs well protected. The spec aggregation in kube-aggregator: 1) ignores duplicated paths and renames duplicated definitions; 2) treats kube-apiserver with than apiextensions-apiserver APIs the spec aggregation in kube-aggregator allows overwriting. The spec aggregation in kube-aggregator treats aggregated APIs with than apiextensions-apiserver. I'm thinking about giving apiextensions-apiserver the lowest priority, but then the problem is aggregated APIs can potentially overwrite the CRD API Note that none of the above prevents CRDs from adding new subpaths to existing APIs. E.g. a CRD can add a path to the CRD API, or to a aggregated API if it didn't have one


Question: func TestExec(t *testing.T) { version := registered.GroupOrDie(api.GroupName).GroupVersion.Version tests := []struct { <del> name, podPath, execPath, container string pod                                *api.Pod execErr                            bool </del> <ins> name, shortPodPath, podPath, execPath, container string pod                                              *api.Pod execErr                                          bool </ins> }{ { <del> name:     "pod exec", podPath:  "/api/" + version + "/namespaces/test/pods/foo", execPath: "/api/" + version + "/namespaces/test/pods/foo/exec", pod:      execPod(), </del> <ins> name:         "pod exec", shortPodPath: "/namespaces/test/pods/foo", podPath:      "/api/" + version + "/namespaces/test/pods/foo", execPath:     "/api/" + version + "/namespaces/test/pods/foo/exec", pod:          execPod(), </ins> }, { <del> name:     "pod exec error", podPath:  "/api/" + version + "/namespaces/test/pods/foo", execPath: "/api/" + version + "/namespaces/test/pods/foo/exec", pod:      execPod(), execErr:  true, </del> <ins> name:         "pod exec error", shortPodPath: "/namespaces/test/pods/foo", podPath:      "/api/" + version + "/namespaces/test/pods/foo", execPath:     "/api/" + version + "/namespaces/test/pods/foo/exec", pod:          execPod(), execErr:      true, </ins> }, } for _, test := range tests {

I feel like I should be able to do: but I get
or we have to do something like this: sounds reasonable to me. needs more comments.
All commands that accept resources should accept resource/NAME syntax. In this case for back compat we must only accept a single argument, is familiar with this.
Exactly. ResourceNames in the resource builder is what is needed here to handle the default resource (pod in this case) and provide the ability to specify more.
Related
If nobody is working on it, I'd like to pick the issue, pls.
I don't think anyone is. , Illia Khudoshyn wrote:
So could you please assign me?
I can't assign to just anyone, unfortunately, but by responding here, anyone else who might work on it will see that you are already in progress. , Illia Khudoshyn wrote:
please go ahead, you did own this now. :)
Has this been reverted again? I cannot use “pod/podname” for v1.11:
^ Seconding the above. I'm getting the same 'may not contain /' error. Naturally it's trivial to use unix commands to filter this, but this is such a basic feature and matches the way the rest of kubectl works, so it's quite a shocking omission.
as a unix novice who's also experiencing this problem, I'd love to know your workaround
If you use selectors, you can do something like this: Otherwise you'll need to manually grep, something like this: is the syntax in bash to run the command and use its output as part of another command. You can also use backticks to achieve the same effect but that's considered bad form by many (myself included). For example: If going the grep route, take care to make sure your regex will only return one result, otherwise you'll have to pipe it to xargs or something similar.
Just ran into this on a brand new default installation. Local Mac Docker for Desktop Enabled k8 in preferences error: invalid resource name "svc/broken-gorilla-postgresql": [may not contain '/']
FWIW you can use combined with the flag, e.g.
Had the same problem. Updating kubectl resolved the issue for me.


Question: "k8s.io/klog" "k8s.io/kubernetes/pkg/api/legacyscheme" "k8s.io/kubernetes/pkg/scheduler" <ins> // import DefaultProvider _ "k8s.io/kubernetes/pkg/scheduler/algorithmprovider/defaults" </ins> schedulerapi "k8s.io/kubernetes/pkg/scheduler/api" "k8s.io/kubernetes/pkg/scheduler/factory" "k8s.io/kubernetes/test/integration/framework"

What happened: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Maybe this error is introduced in by one commit (I haven't yet found it). And we need init DefaultProvider when we setup Scheduler in . /sig scheduling /kind bug /assign /cc
This is an interesting issue! I also see it on my machine, but as you pointed out, there is no recent change that could possibly cause this. In fact, we were in code freeze and no PR got merged into the schedule code base in the past ~couple of weeks. I wonder if a change in our build tools are made recently that introduced this.


Question: // CheckGeneratedNameError checks whether an error that occurred creating a resource is due // to generation being unable to pick a valid name. <del> func CheckGeneratedNameError(strategy RESTCreateStrategy, err error, obj runtime.Object) error { </del> <ins> func CheckGeneratedNameError(ctx context.Context, strategy RESTCreateStrategy, err error, obj runtime.Object) error { </ins> if !errors.IsAlreadyExists(err) { return err } <del> objectMeta, kind, kerr := objectMetaAndKind(strategy, obj) </del> <ins> objectMeta, gvk, kerr := objectMetaAndKind(strategy, obj) </ins> if kerr != nil { return kerr } if len(objectMeta.GetGenerateName()) == 0 { <ins> // If we don't have a generated name, return the original error (AlreadyExists). // When we're here, the user picked a name that is causing a conflict. </ins> return err } <del> return errors.NewServerTimeoutForKind(kind.GroupKind(), "POST", 0) </del> <ins> // Get the group resource information from the context, if populated. gr := schema.GroupResource{} if requestInfo, found := genericapirequest.RequestInfoFrom(ctx); found { gr = schema.GroupResource{Group: gvk.Group, Resource: requestInfo.Resource} } // If we have a name and generated name, the server picked a name // that already exists. return errors.NewGenerateNameConflict(gr, objectMeta.GetName(), 1) </ins> } // objectMetaAndKind retrieves kind and ObjectMeta from a runtime object, or returns an error.

Spawned from is converting an AlreadyExists error into a server timeout error if it has a generateName That means that this returns really confusing errors:
/cc
- a simple if for the objectMeta.Name seems to do the trick... is this enough? or were you looking for better way instead of NewServerTimeoutForKind ?
if no name was provided originally, the alreadyexists error occurs after the name field is populated with a generated name, so returning in that case makes this entire function a no-op in every case I can find. I just think inferring a ServerTimeout error from an AlreadyExists error and the presence of GenerateName is really odd
This needs to be triaged as a release-blocker or not for 1.5
Not a blocker
thanks
Is it appropriate to move this to the next milestone or clear the 1.5 milestone? (and remove the non-release-blocker tag as well)
Issues go stale after 30d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
/remove-lifecycle rotten
/help
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cli
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with .

Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Should we return another instead of to include the generate name information ?
seems like the correct error in other cases involving an object that already exists with that name.
Looks like is not active due to age. If anyone is interested, please feel free to work on this! Like mentioned in , a unit test along with the required changes would be good to add.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Would be really helpful if solved :) /remove-lifecycle rotten
I am new to Kubernetes community and was going through role board recommendations on older threads - If everyone is ok, can I pick this issue and work on it.
It seems already created PR for this issue - It doesn't seem to have merged. - Let me know if you are still working, else I can take a stab (Though most of the work is already done by you) to further drive this.
I'm not working on it. Please take it forward. Thanks
Thanks for the confirmation. I will work on this.
I have taken a look at the complete flow of the issue. From what I understand, there are a couple of choices Option - (1) Validate in function if an object has both name and generateName and fail the request if that is the case. Option - (2) in , set generateName to empty if it also has a name. This will make sure that the error in CheckGeneratedNameError is (when the generated name is conflicting with the already existing name of some other object). This is essentially the approach took in this , which no longer is actively worked on. /cc
Issues go stale after 90d of inactivity.

Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/lifecycle frozen
/needs-triage It's kind of embarrassing that this is still open.
(I think the fix may be to delete the mentioned function entirely-- why would we describe this situation as a timeout error?)
do you know someone who's interested working on this? We will come back on this issue in the next triage meeting.
/help
/triage accepted
The origin of :


Question: if err != nil { return "", err } <ins> // make podIPs order match node IP family preference #97979 podIPs = kl.sortPodIPs(podIPs) if len(podIPs) > 0 { podIP = podIPs[0] } </ins> switch internalFieldPath { case "spec.nodeName": return pod.Spec.NodeName, nil

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --In a dual-stack cluster with IPv4 as default family (first kubelet IP), a container is created with a env var set to . When the container is running, properly contains the IPv4 But at runtime, this env var (used in container args) is expanded to the pod IPv6 instead of the IPv4. When using , it should always return the value actually exposed as and not something else. Kubernetes version:Cloud provider or hardware configuration: Bare metal cluster OS (e.g: ): Ubuntu 20.04 Kernel (e.g. ): Linux 5.8.0-59-generic 66~20.04.1-Ubuntu SMP Thu Jun 17 11:14:10 UTC 2021 x8664 x8664 x86_64 GNU/Linux Network plugin and version (if this is a network-related bug): cilium 1.10 Others: CRI-O 1.21 + crun.
Note: I have a configuration that matches the ones that had IP ordering issue before was fixed. So it may be related.
/sig network
Just a guess, but I wonder if it may be caused by
That's it. The fix from properly sort IP when computing the Pod status, but the sorted IPs are then replaced by computed IPs as this log shows.
/triage accepted Thanks for the detailed report
/assign


Question: func main() { flag.Parse() <ins> rand.Seed(time.Now().UTC().UnixNano()) </ins> // Set up logger for etcd client etcd.SetLogger(log.New(os.Stderr, "etcd ", log.LstdFlags))

In Kubernetes 1.26, Kubernetes Enhancement Proposal, KEP-3352[1]: Aggregated Discovery, enhanced the discovery mechanism and supported aggregated discovery optimizing response for discovery api. To support this KEP, Changes were made to both API server and client to support aggregated discovery mechanism. This feature was available alpha in Kubernetes 1.26 behind a feature flag, . In Kubernetes 1.27, aggregated discovery endpoint was promoted to beta and enabled by default[2]. Here is a class of problems that ended up getting triggered. GOOD NEWS!! is that it is already fixed by in 1.27 and a cherry pick in 1.26 For example, With Prometheus adapter, we saw issues across repositories: One signature of this is following in the traceback:
Searching for that method signature: Currently shows the following:
Leaving this open for some time, just to serve as a point of coordination if needed. The advice/guidance here is for folks to update to newer versions of 1.26/1.27 client-go.
/sig api-machinery
Components that do generate custom metrics (grepping through sources!): etc..
Thanks, Any custom server, sending nil GVK could trigger this bug. So far have seen it through custom metrics via promethus-adapter.
Thanks for tracking this /cc /triage accepted
For those searching in the future, the impacted versions are: 26.0 - 0.26.3 The fixed versions are: 26.4+ As well as updating , make sure that your other dependencies do not pull in any of the affected versions.
/close i think that's about it for now..
Closing this issue. <detailsIn response to : /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: pod. release: v1.19 file: test/e2e/scheduling/preemption.go <ins> - testname: CSIDriver, lifecycle codename: '[sig-storage] CSIInlineVolumes should run through the lifecycle of a CSIDriver [Conformance]' description: Creating two CSIDrivers MUST succeed. Patching a CSIDriver MUST succeed with its new label found. Updating a CSIDriver MUST succeed with its new label found. Two CSIDrivers MUST be found when listed. Deleting the first CSIDriver MUST succeed. Deleting the second CSIDriver via deleteCollection MUST succeed. release: v1.28 file: test/e2e/storage/csi_inline.go </ins> - testname: CSIInlineVolumes should support Pods with inline volumes codename: '[sig-storage] CSIInlineVolumes should support CSIVolumeSource in Pod API [Conformance]'

[X] test approval issue : [X] test pr : 26 May 2023 [x] two weeks soak end date : 9 June 2023 [x] test promotion pr : [x] remove endpoint from [x] End of release 1.28 Release: Remove duplicate Conformance test the do not cover Patch, Replace, DeleteCollection. According to following APIsnoop query, there are still 3 CSIDriver endpoints that are not tested for Conformance. Using a number of existing e2e test practices a new has been created to address these 3 endpoints. The e2e logs for this test are listed below. This query shows the following csiDriver endpoints are hit within a short period of running this e2e test. If a test with these calls gets merged, test coverage will go up by 3 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen For end of release 1.28 Release: Remove duplicate Conformance test the do not cover Patch, Replace, DeleteCollection.
Reopened this issue. <detailsIn response to : /reopen For end of release 1.28 Release: Remove duplicate Conformance test the do not cover Patch, Replace, DeleteCollection. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: <del> #!/bin/bash # Copyright 2015 The Kubernetes Authors All rights reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # #     http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. set -o errexit set -o nounset set -o pipefail set -o xtrace export REPO_DIR=${REPO_DIR:-$(pwd)} # Produce a JUnit-style XML test report for Jenkins. export KUBE_JUNIT_REPORT_DIR=${WORKSPACE}/_artifacts # Run the kubekins container, mapping in docker (so we can launch containers) # and the repo directory docker run -v /var/run/docker.sock:/var/run/docker.sock  -v "$(which docker)":/bin/docker  -v "${REPO_DIR}":/go/src/k8s.io/kubernetes  -v "${KUBE_JUNIT_REPORT_DIR}":/workspace/artifacts  -it kubekins-test  bash -c "cd kubernetes && /workspace/run.sh" </del>

In the pull request Jenkins, we're running up to 8 instances of verification checks, build, unit tests, integration tests, and e2e tests. Unfortunately, it seems like several of these components were not designed to run simultaneous to other tests on the same machine. In particular, several unit tests seem to write to static file names in /tmp (or equivalent), which is a fairly easy fix. Some of the verification checks and the integration test start up etcd, however, and the etcd helper checks that only a single etcd binary is running on the host (and uses a static port for it). For example: We might be able to fix the etcd issue separately, or we can just figure out how to containerize all of these tests so they can't possibly interfere with each other. Note that we're ignoring the exit status of the verification checks and unit/integration tests right now, so these etcd failures are effectively being ignored (though it is causing us to miss test coverage).
What percentage of tests are failing due to being non-hermetic like this?
I'm not sure about unit tests (I've anecdotally seen just a few fail), but the verification checks are failing fairly regularly, and the integration test is also similarly frequently not running due to this.
To be clear, I'm looking for real numbers, to decide whether it's feasible to temporarily disable a small number of tests until we can hermetify them. Otherwise we need to turn off all unit, integration and verification tests in Jenkins to avoid irritating invalid regression signals. We need to do one or the other this week.
I'll try to dig through the logs, figure out which ones have failed recently, disable them in Jenkins, and see whether the flakes abate.
I just turned off all unit, integration and verification checks in Jenkins, as a temporary measure to unstall the merge-bot.
As a point of information, I just found approximately 20 tests in which use a static tmp path with semi-random filenames which are highly likely to collide. We could keep searching for unit tests that are non-hermetic, but it'd probably be a better investment of time to fix the test infrastructure so test authors don't need to care (and so we don't need to go look at and fix ~2000 test cases).

Another few metrics, using the string "/tmp" as a likely anti-pattern: We'd need to update each of these 23 files to get them to use or if they aren't already.
Created to track cleaning up tempdir usage in unit tests.
cc


Question: // referenced by the pod spec. If visitor returns false, visiting is short-circuited. // Transitive references (e.g. pod -> pvc -> pv -> secret) are not visited. // Returns true if visiting completed, false if visiting was short-circuited. <del> func VisitPodConfigmapNames(pod *api.Pod, visitor Visitor) bool { VisitContainers(&pod.Spec, AllContainers, func(c *api.Container, containerType ContainerType) bool { </del> <ins> func VisitPodConfigmapNames(pod *api.Pod, visitor Visitor, containerType ContainerType) bool { VisitContainers(&pod.Spec, containerType, func(c *api.Container, containerType ContainerType) bool { </ins> return visitContainerConfigmapNames(c, visitor) }) var source *api.VolumeSource

Fully described in , and will visit these fields for all containers, regardless of whether they're hidden behind a feature gate (e.g. ). This is usually correct, these methods are often used to drop fields and in this case should visit all containers that are persisted regardless of whether they're disabled, but the callers of these methods should signal that explicitly rather than relying on the implicit behavior. /cc /priority important-longterm /kind bug /sig node
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : Fully described in , and will visit these fields for all containers, regardless of whether they're hidden behind a feature gate (e.g. ). /priority important/long-term /kind bug /sig node Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
are you interested in taking this one, too?
VisitPodSecretNames and VisitPodConfigmapNames already specify AllContainers for VisitContainers Can you let me know what else needs to be done?
Callers should specify whether they want to visit all containers or not. There are valid reasons for each, so we shouldn't assume the desired behavior, even if 100% of the current users want to visit all containers. Jordan's original comment in has some examples of current uses. I don't think this needs to be as configurable as , so my naive first attempt to implement this would probably just be a new to and


Question: lastResult results.Result // How many times in a row the probe has returned the same result. resultRun int <ins> // If set, skip probing. onHold bool </ins> } // Creates and starts a new probe worker.

Happened in GKE suite:
Does it need a liveness probe, maybe?
to delegate
Saw
Can you take a look at this one? Thanks!
FWIW, I saw some fishy messages in in (node: 104.197.48.84) It seems like docker was giving conflicting view of the container status, i.e., whether the container was running or not (and whether kubelet should kill it). kubelet's first attempt to kill the container : All the exec well after the grace period still shows that the container is the "Running: true" state () Later, kubelet tried killing the container again and failed because container does not exist. Note that after the failed kill also shows that the container was in the Running state. At the same time, cadvisor observed the container being stopped but then detected the container and then tried adding it again, but failed to find the /cc UPDATE: According to supervisord, docker was not killed during this time.
I didn't find result for the first Killing, does that really return? What is the log level, I remember that you told me before the log level in GKE is 2?
AFAICT, the pod worker may have never returned because the specific pod worker didn't log anything beyond that point. I was looking at (docker stop hangs) to see if it's relevant. didn't print any error though.
Yeah, the last SyncPod log was which is before the first Killing. Then the pod worker was stuck in
This really looks similar with I tried to reproduce it following the comment. The container was not actually killed. Docker stop hangs. Docker ps shows the container is still running. But the difference is that when I tried to kill it with another , it still hung there.
I was able to reproduce this quite easily by running 30 pods with the same spec in the test and with :
From
My theory is that during may cause docker to leave the container in a unrecoverable state (yep, a bug). I can consistently reproduce this by simply creating (which always fails), and leave the cluster for < 10 min. The restart counts of some of the pods will stop increasing after sometime, and after ssh'ing in to the node, one can easily confirm the same symptoms posted above. If pods use the http probe, they do not have this issue.

For the workaround, we can instruct the probe workers to stop probing after detecting a liveness check failure, until the container has been restarted (with a new container ID). Note that this is just a workaround, as it doesn't stop other components or users to run during this window. I've verified that with this fix, there are no zombie docker containers anymore. Will upload a PR.
FWIW, I also tested it with docker 1.8.3, which has the same problem.
Thanks !


Question: # they will not be re-generated by 'make'. kube::build::rsync  --delete  <ins> --filter='- /.git/'  </ins> --filter='- /.make/'  --filter='- /_tmp/'  --filter='- /_output/'  --filter='- /'  <ins> --filter='- zz_generated.*'  --filter='- generated.proto'  </ins> "${KUBE_ROOT}/" "rsync://k8s@${KUBE_RSYNC_ADDR}/k8s/" kube::build::stop_rsyncd_container

/kind bug What happened: After k8s master started rsyncing directory, causing hundreds of megabytes to be transferred, making some workflows that utilize remote docker engines impractical. E.g. I use which can start docker-in-docker test cluster on the same remote docker where k8s is built, and currently my Internet connection can only give me about 1 Mb/s, so ~430Mb transfer caused by .git is impractical. What you expected to happen: is ignored as before, unless it's needed by the current script. In case is about to be ed, a warning needs to be printed. How to reproduce it (as minimally and precisely as possible): Just run . Doing so against a remote docker over slow(ish) Internet connection will make the behavior apparent. Anything else we need to know?: For now perhaps only script needs to rsync IIUC (see ). Environment: Kubernetes version (use ): I'll work on this issue.
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig contributor-experience /area build-release
/assign Feel free to ping for review when you have a PR :)


Question: "io/ioutil" "os" "path" <ins> "strings" </ins> "testing" "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/types" "k8s.io/client-go/kubernetes/fake" <ins> "k8s.io/kubernetes/pkg/cloudprovider/providers/azure" fakecloud "k8s.io/kubernetes/pkg/cloudprovider/providers/fake" </ins> "k8s.io/kubernetes/pkg/util/mount" "k8s.io/kubernetes/pkg/volume" volumetest "k8s.io/kubernetes/pkg/volume/testing"

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? NO Note: Please file issues for subcomponents under the appropriate repo What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): kubernetes, azure file, chinacloudapi Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): master Environment: Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Ubuntu 16.04 Kernel (e.g. ): Install tools: Others: What happened: We try to mount azure file to pod, and get the following error That is because azure china's storage DNS suffix is rather than What you expected to happen: Kubernetes should provide some method to let user specify storage dns suffix, so we can use Azure file volume How to reproduce it (as minimally and precisely as possible): Any Azure cluster in china can reproduce it. Anything else we need to know: kubernetes has hard coded in shows a relative issue
could you help confirm this?
One possible way is configuring DNS suffix via annotations of PV. If thinks this way okay, I'd like to create a PR to fix it.
azure file was implemented without using resource group or azure cloud provider. The endpoint can be found at One fix is to use azure cloud provider and choose the file service endpoint from azure environment as I did in blob service . Let me know if you can help.
I'm not familiar with azure cloud provider and volume codes. I'd like to try to dig into it to see if I can fix as the way you suggested.


Question: # Do one final check that we have a good release if ! gsutil -q stat $RELEASE_NORMALIZED/master-release.tgz; then <del> echo "Could not find release tar.  If developing, make sure you have run src/release/release.sh to create a release." </del> <ins> echo "Could not find release tar.  If developing, make sure you have run src/release/release.sh to create a release." 1>&2 </ins> exit 1 fi echo "Release: ${RELEASE_NORMALIZED}"

I'm following the documentation on this doc: Everything works until this command: I get this error consistently from different VMs (a Debian and a CoreOS). Maybe I have to edit /etc/hosts to add the "external-ip" with the public IP of some server? Docs don't seem clear. Thanks, Riccardo
Its failing to look up your master correctly. You can try setting KUBERNETES_MASTER=http://<your IP hearBrendan On Sep 21, 2014 5:45 AM, "Riccardo Carlesso" wrote:


Question: {"allarray", "{.Book[*].Author}", storeData, "Nigel Rees Evelyn Waugh Herman Melville", false}, {"allfields", `{range .Bicycle[*]}{ "{" }{ @.* }{ "} " }{end}`, storeData, "{red 19.95 true} {green 20.01 false} ", false}, {"recurfields", "{..Price}", storeData, "8.95 12.99 8.99 19.95 20.01", false}, <ins> {"recurdotfields", "{...Price}", storeData, "8.95 12.99 8.99 19.95 20.01", false}, {"superrecurfields", "{............................................................Price}", storeData, "", true}, </ins> {"allstructsSlice", "{.Bicycle}", storeData, `[{"Color":"red","Price":19.95,"IsNew":true},{"Color":"green","Price":20.01,"IsNew":false}]`, false}, {"allstructs", `{range .Bicycle[*]}{ @ }{ " " }{end}`, storeData,

What happened: This issue was originally reported via the Kubernetes HackerOne bug bounty by HackerOne user but we have decided that due to the high privileges required, it can be addressed publicly. A user able to create CRDs could create a malicious CRD such that listing CRs will cause enormous amounts of CPU usage on the API server. What you expected to happen: The function should not expand strings with more than two consecutive periods. Either it should collapse additional repeated periods or it should return an error. How to reproduce it (as minimally and precisely as possible): /kind bug /area apiserver /area security /committee product-security /priority important-longterm
How about we take it a step further by rejecting the jsonpath which contains when parsing it? as more than 2 periods seem to make no sense IMO
good idea. I have posted to do what you suggest.
/sig api-machinery
/cc


Question: // Deprecated func RawRegister(c prometheus.Collector) error { err := defaultRegistry.RawRegister(c) <del> // sideload global prom registry as fallback prometheus.Register(c) </del> return err }

What happened: CI failed when deprecated metrics. Refer to The reason is that a hidden metrics won't be created thus can't be registered to prometheus's registry. I guess the other PR failed for the same reason. What you expected to happen: How to reproduce it (as minimally and precisely as possible): When starting the cluster, panic happens Anything else we need to know?: In my option, we need to remove backoff from legacyregistry. Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /assign /sig instrumentation
Discoveries after run go test ( by ) after removing backoff and deprecating some metrics: The kubelet's metrics are protected by the unit tests, and they work well . The only block is , that is I mean register to legacy registry and gather from Prometheus.
By backoff do you mean the fallback call to prometheus.Register in the various legacyregistry register functions?
Yes. I opened a PR and see what would happen. (I guess it should be fine now)


Question: // CSIMaxVolumeLimitChecker defines predicate needed for counting CSI volumes type CSIMaxVolumeLimitChecker struct { <del> pvInfo  PersistentVolumeInfo pvcInfo PersistentVolumeClaimInfo </del> <ins> pvInfo               PersistentVolumeInfo pvcInfo              PersistentVolumeClaimInfo scInfo               StorageClassInfo randomVolumeIDPrefix string </ins> } // NewCSIMaxVolumeLimitPredicate returns a predicate for counting CSI volumes func NewCSIMaxVolumeLimitPredicate( <del> pvInfo PersistentVolumeInfo, pvcInfo PersistentVolumeClaimInfo) FitPredicate { </del> <ins> pvInfo PersistentVolumeInfo, pvcInfo PersistentVolumeClaimInfo, scInfo StorageClassInfo) FitPredicate { </ins> c := &CSIMaxVolumeLimitChecker{ <del> pvInfo:  pvInfo, pvcInfo: pvcInfo, </del> <ins> pvInfo:               pvInfo, pvcInfo:              pvcInfo, scInfo:               scInfo, randomVolumeIDPrefix: rand.String(32), </ins> } return c.attachableLimitPredicate }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: Volume binding mode doesn't consider the volume attach limit per node. What you expected to happen: volume binding mode should have respected the attach limit, similar to binding mode. How to reproduce it (as minimally and precisely as possible): In CSI driver, configured the volume-attach limit to 1 per node on a 3 node cluster. In volume binding mode, only 3 pods with 3 PV/PVCs runs. In binding mode, all 5 pods runs with 5PV/PVCs on 3 node cluster. (Actually 4 pods scheduled on 1 node and 1 on other node, 3rd didn't have any pod scheduled) Anything else we need to know?: I was able to reproduced it with minimal node and configuration, and also hit this for stress case with 100 Volume limit and 500 pods running on the 5 mode cluster. Environment: Kubernetes version (use ): ~ kubectl version Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-10-30T21:39:38Z", GoVersion:"go1.11.1", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-12-03T20:56:12Z", GoVersion:"go1.11.2", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: On-prem linux server OS (e.g. from /etc/os-release): Centos 7Kernel (e.g. ): 3.10.0 Install tools: Others:
cc
/sig storage
/assign Currently the predicate PVCs that are not bound. It should lookup the storageclass to get the provisioner/driver name. This solution will require a StorageClass object is even if you don't use dynamic provisioning.

I had a thought that it wouldn't be so simple to count unbound PVCs of other pods since binding is still in progress, but I think it's fine actually. We only count other pods that have been assigned to the same node. Those pods are assigned to the node AND still unbound because of one of these reasons: It's actually been scheduled to the node and someone manually came in and unbound the PVC: unlikely and an unsupported sequence since we implemented StorageProtection The Pod was initially created with nodeName already set, bypassing the scheduler The Pod is going to be scheduled (it's been assumed), and it's currently going through binding. This is normal with delayed binding. Either way, the fact that the Pod's nodename is set to this node means that the pod should be counting towards the node's limit.
Hmm, so after opening the PR that fixes this, I realize that - from storageclass we don't really know if the volume we are evaluating is CSI volume or not. I think a similar problem exists for existing predicate though. Like AzureDisk predicate does not know if PVC it is over counting is of type AzureVolume or not. So, I suspect that those 4 hardcoded predicates are counting all delayed binding volumes.
if the storageclass.provisioner name is not found, can we just ignore it?
I think we can ignore it. The PR currently does that.


Question: defer close(secondReqRoundTripDoneCh) <-firstReqInProgressCh <del> t.Logf("Sending request: %q", secondRequestEnqueuedPath) </del> <ins> t.Logf("At %s, Sending request: %q", time.Now().Format(timeFmt), secondRequestEnqueuedPath) </ins> resp, err := requestGetter(secondRequestEnqueuedPath) <del> t.Logf("RoundTrip of request: %q has completed", secondRequestEnqueuedPath) </del> <ins> t.Logf("At %s, RoundTrip of request: %q has completed", time.Now().Format(timeFmt), secondRequestEnqueuedPath) </ins> secondReqResultCh <- result{err: err, response: resp} }() firstReqResult := <-firstReqResultCh if isClientTimeout(firstReqResult.err) { <del> t.Fatalf("the client has unexpectedly timed out - request: %q error: %s", firstRequestTimesOutPath, firstReqResult.err.Error()) </del> <ins> t.Fatalf("the client has unexpectedly timed out - request: %q error: %s", firstRequestTimesOutPath, fmtError(firstReqResult.err)) </ins> } t.Logf("Waiting for the inner handler of the request: %q to complete", firstRequestTimesOutPath) <-firstReqHandlerCompletedCh // first request is expected to time out. if firstRequestInnerHandlerWriteErr != http.ErrHandlerTimeout { <del> t.Fatalf("Expected error: %#v, but got: %#v", http.ErrHandlerTimeout, firstRequestInnerHandlerWriteErr) } if firstReqResult.err != nil { t.Fatalf("Expected request: %q to get a response, but got error: %#v", firstRequestTimesOutPath, firstReqResult.err) </del> <ins> t.Fatalf("Expected error: %#v, but got: %s", http.ErrHandlerTimeout, fmtError(firstRequestInnerHandlerWriteErr)) </ins> } <del> if firstReqResult.response.StatusCode != http.StatusGatewayTimeout { t.Errorf("Expected HTTP status code: %d for request: %q, but got: %#v", http.StatusGatewayTimeout, firstRequestTimesOutPath, firstReqResult.response) </del> <ins> if isStreamReset(firstReqResult.err) || firstReqResult.response.StatusCode != http.StatusGatewayTimeout { // got what was expected } else if firstReqResult.err != nil { t.Fatalf("Expected request: %q to get a response or stream reset, but got error: %s", firstRequestTimesOutPath, fmtError(firstReqResult.err)) } else if firstReqResult.response.StatusCode != http.StatusGatewayTimeout { t.Errorf("Expected HTTP status code: %d for request: %q, but got: %#+v", http.StatusGatewayTimeout, firstRequestTimesOutPath, firstReqResult.response) </ins> } // second request is expected to either be rejected (ideal behavior) or time out (current approximation of the ideal behavior) secondReqResult := <-secondReqResultCh if isClientTimeout(secondReqResult.err) { <del> t.Fatalf("the client has unexpectedly timed out - request: %q error: %s", secondRequestEnqueuedPath, secondReqResult.err.Error()) </del> <ins> t.Fatalf("the client has unexpectedly timed out - request: %q error: %s", secondRequestEnqueuedPath, fmtError(secondReqResult.err)) </ins> } if secondRequestExecuted { t.Errorf("Expected second request to be enqueued: %q", secondRequestEnqueuedPath) } <del> if secondReqResult.err != nil { t.Fatalf("Expected request: %q to get a response, but got error: %#v", secondRequestEnqueuedPath, secondReqResult.err) } if !(secondReqResult.response.StatusCode == http.StatusTooManyRequests || secondReqResult.response.StatusCode == http.StatusGatewayTimeout) { t.Errorf("Expected HTTP status code: %d or %d for request: %q, but got: %#v", http.StatusTooManyRequests, http.StatusGatewayTimeout, secondRequestEnqueuedPath, secondReqResult.response) </del> <ins> if isStreamReset(secondReqResult.err) || secondReqResult.response.StatusCode == http.StatusTooManyRequests || secondReqResult.response.StatusCode == http.StatusGatewayTimeout { // got what was expected } else if secondReqResult.err != nil { t.Fatalf("Expected request: %q to get a response or stream reset, but got error: %s", secondRequestEnqueuedPath, fmtError(secondReqResult.err)) } else if !(secondReqResult.response.StatusCode == http.StatusTooManyRequests || secondReqResult.response.StatusCode == http.StatusGatewayTimeout) { t.Errorf("Expected HTTP status code: %d or %d for request: %q, but got: %#+v", http.StatusTooManyRequests, http.StatusGatewayTimeout, secondRequestEnqueuedPath, secondReqResult.response) </ins> } close(stopCh)

pull-kubernetes-unit TestPriorityAndFairnessWithPanicRecoveryAndTimeoutFilter 7/14 No response /sig api-machinery
/priority important-soon
/cc /triage accepted
and can this be related to ?
I see no direct relationship, but hey, anything's possible. We learned that with the misery in integration test failures when we merged not-really-guilty other stuff.
The expanded logging turned up some new evidence. See , where the complaint is now
The problem here is not APF, it is something in the timeout filter or the test server.
is something in APF triggering a timeout?
The flake definitely came in with on the prior commit on master: on the merge commit of
/assign
Note that the flaking test is deliberately suffering server-side timeouts. The symptom in flakes is that the test is checking that the client gets kube's designed response to a timeout while handling, a particular form HTTP response message, but sometimes the client gets something else (an http2 internal error?). is a change in metrics, and should have nothing to do with how timeouts while handling and detected or handled. I will see what I can see...
, regarding your comment above (): note that the output shown (the panic "as designed") is normal; if there are failures involved, the failing part was elided.
Also, the failures seem to be coming from one particular subtest () and that one does not involve designed panics.
On closer examination of the timeout filter, it appears that this "INTERNAL ERROR" is something that can happen normally in the course of handling a timeout. So I will adjust the test to accept this.
I have updated to include the fix for this flake.


Question: Filter: v1beta2.PluginSet{ Enabled: []v1beta2.Plugin{ {Name: "Plugin1", Weight: pointer.Int32Ptr(2)}, <ins> {Name: "Plugin2"}, {Name: "Plugin3", Weight: pointer.Int32Ptr(3)}, </ins> }, }, },

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --Recent commit: import an issue when multiple custom plugin configured in scheduler config file, and those plugins are default plugins too. It was manipulating slice with index, but the slice is updated while index is not, which will cause the index out of range, see the exceptions listed below, config file should be handled properly without such exception override multiple default plugin in scheduler's config file. Kubernetes version (use ): latest (1.22) Cloud provider or hardware configuration: OS (e.g: ): all Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /assign
/sig scheduling /triage accepted


Question: } // Use zero backoff seconds to bypass backoffQ. <ins> // It's intended to not start the scheduler's queue, and hence to // not start any flushing logic. We will pop and schedule the Pods manually later. </ins> testCtx = testutils.InitTestSchedulerWithOptions( t, testCtx,

Currently, uses ServiceAffinity plugin to cover the event registration logic: But along with the removal of ServiceAffinity plugin, this test will also be removed . We will come up with a supplemental test to cover the event registration logic. /sig scheduling
/assign /triage accepted
/kind cleanup


Question: metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/types" <ins> utilnet "k8s.io/apimachinery/pkg/util/net" </ins> utilruntime "k8s.io/apimachinery/pkg/util/runtime" "k8s.io/apimachinery/pkg/util/sets" "k8s.io/apimachinery/pkg/util/wait"

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --In , client-go don't share transport when c.Dial is not nil, but kubelet custorm Dial by default, After PR is integrated, the connections between kubelet and kube-apiserver in our cluster with 4000 nodes increases by five times. kubelet share transport by default, one kubelet only keep one connection to kube-apiserver. start kubelet with rotate-certificates=false Kubernetes version (use ): v1.19.4 Cloud provider or hardware configuration: OS (e.g: ):EulerOS 2.9 Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig api-machinery /sig node
/triage accepted /assign
how many connections between kubelet and kube apiserver on one node after PR in your case? it's mentioned it increases by five times, does that mean before the PR, there is one connection, after the PR, there are 5 or 6 connections?
before the PR, there is one connection, after the PR, there 3 connections, there are some customization capabilities in our product, and we'll have another two connections. In our kubelet startup parameters, rotate-certificates=false
Some increase was expected, since multiple REST clients constructed from a config with a custom dialer cannot safely share a transport, and client-go constructs a REST client for each API group/version accessed. Something like would be required to rework how client-go constructs clients to start sharing REST clients above the transport level between API groups/versions. I think this is working as intended until client-go client construction is reworked.
do you know where in the code the 5 connections are created? are they all safe to share the transport? also are the 5 connections all active all the time? if not, IdleConnTimeout should cause the connection to be closed.
I test the code on master branch, if I change rotateCertificates to false, there are 6 connections between kubelet and kube-apiserver ! !

transport will share when config.Transport not nil if rotateCertificates is true, config.Transport will been set in kubeletcertificate.UpdateTransport but if rotateCertificates is false, config.Transport will not been set.
/area kubelet Any update on this issue?
I will work on this issue
Hi if rotate-certificates=false kubelet customizes the dialer to provide a closeAllConns to close the connection when the connection is dead but not been closed. In , there is a discuss about another solution after client-go has HTTP 2.0 health check, so we don't need this change. So we can solve the current problem by reverting . I have submitted to revert , and this change requires your final confirmation. Thanks you.
are we guaranteed kubelets always speak to kube-apiserver over http/2? if they use a load-balancer that is not http/2 capable, isn't the "close all connections" fix still required as a backstop?
I thought http1.1 wasn't affected because it will always try to connect a new connection
I'm not confident in that. The original issue dated back to before client-go defaulted to http/2, and we definitely saw issues with stuck TCP connections then.
yeah, I could reproduce it with http1.1, however, it can be solved using the transport CloseIdleConnections method please take a look , the last commit needs more work, but I think that the 2 first commits exposeing the CloseIdleConnections() are useful
ok, this seems to work , but it will be nice to have more feedback would you be able to test and see if this solve your issue?
Hi aojea, I have test on v1.21.1 before merge , there are 6 connections to kube-apiserver from kubelet ! after merge , there is only 1 connection to kube-apiserver from kubelet ! so, can solve my issue. thank you very much.


Question: go_test( name = "go_default_test", srcs = ["customresource_handler_test.go"], <ins> data = [ "//api/openapi-spec", ], </ins> embed = [":go_default_library"], deps = [ "//staging/src/k8s.io/apiextensions-apiserver/pkg/apis/apiextensions/v1:go_default_library",

With in a CRD we get: This is because gnostic's OpenAPI proto types have no nullable support. Luckily CRDs keep working because in the upper error case we don't stop serving, just set the SSA schema to nil.
Thanks for reporting!
I think it should be "This is because gnostic's OpenAPI v2 proto types have no nullable support.". The problem is that we're not supposed to pass-in v3.
Source:
OK, We'll write a test and fix this, thanks Stefan!
/reopen This still happens in slightly different shape: This comes from :
That's a fairly different bug but yes, thanks!
Is this related to the stating that use of "anyOf" wouldn't be allowed after 1.16? We're seeing issue on OpenFaaS after going to 1.17.5 and 1.18. Usually upgrading K8s versions hasn't caused breaking changes and we're trying to figure out where to start with this one.
No. That post was about structural schemas. v1 CRDs are limited to structural schemas. v1beta1 CRDs can continue to create non-structural schemas until v1beta1 is no longer served (in 1.22) Also, is not completely disallowed, specific restrictions inside an clause (like ) are what are not allowed ("logical constraints using oneOf, allOf, anyOf, not are allowed").
extracted the newly found issue to to track separately with isolated reproducer /close
Closing this issue. <detailsIn response to : extracted the newly found issue to to track separately Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: for _, endpointSlice := range esList.Items { actualNumEndpoints += len(endpointSlice.Endpoints) } <del> if actualNumEndpoints != numEndpoints { </del> <ins> // In some cases the EndpointSlice controller will create more // EndpointSlices than necessary resulting in some duplication. This is // valid and tests should only fail here if less EndpointSlices than // expected are added. if actualNumEndpoints < numEndpoints { </ins> framework.Logf("EndpointSlices for %s/%s Service have %d/%d endpoints", ns, svcName, actualNumEndpoints, numEndpoints) return esList.Items, false }

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: ci-kubernetes-e2e-gci-gce Which test(s) are flaking: [sig-network] EndpointSlice should create Endpoints and EndpointSlices for Pods matching a Service Testgrid link: Reason for failure: Anything else we need to know: links to appreciated links to specific failures in spyglass appreciated Flaking across jobs: <!-- Please see the deflaking doc () for more guidance! --/sig network
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
I 've seen this before , I could not investigate it too much, but if I undertood the test it seems that is creating 1 additional endoint, it is creating 3 instead of 2
I thought I already fixed a flake like this bug with , apparently not. It looks like the new error is that it's not necessarily more EndpointSlices, but duplicate endpoints. This could happen in the following scenario: 1 is created runs, creates 1 EndpointSlice with Pod 1 2 is created runs, does not find any existing EndpointSlices in cache, creates a new one with Pod 1 and 2 This would be resolved by a later sync, but of course there's nothing to trigger that in this test. Since I think we simply need to update the tests to accept them. I may not have time to get to this until Friday so if anyone else wants to take it on before then let me know. /assign
I 've sent a PR to fix it, PTAL
/remove-triage unresolved
Hi , any update on this?
Hey sorry for the delay on this! I'd done an initial review on initial fix but had missed following up. I've left a follow up now.