Question: key.podName == EmptyUniquePodName || previousOp.key.podName == key.podName <ins> podNameExactMatch := previousOp.key.podName == key.podName </ins> nodeNameMatch := previousOp.key.nodeName == EmptyNodeName || key.nodeName == EmptyNodeName || previousOp.key.nodeName == key.nodeName <ins> nodeNameExactMatch := previousOp.key.nodeName == key.nodeName </ins> if volumeNameMatch && podNameMatch && nodeNameMatch { <del> return true, previousOpIndex </del> <ins> // nonExactMatch pending first if previousOp.operationPending { return true, previousOpIndex } // nonExactMatch with no pending, set opIndex to the first nonExactMatch // exactMatch can override opIndex to expected if opIndex == -1 || (podNameExactMatch && nodeNameExactMatch) { opIndex = previousOpIndex } </ins> } } <ins> return opIndex != -1, opIndex </ins> <del> return false, -1 </del> } func (grm *nestedPendingOperations) getOperation(key operationKey) (uint, error) {

on my scenescreate same cronjob use same sfs(the nfs storage on Huawei Cloud) type storage by pvc to do same training work. At the same time, a node will schedule many pods using the same pvc, and there will be many mount and umount tasks about this pvc executed at the same time. And we find all pods on a node at container creating state, because of waiting pvc to mount. check log, we can't find any we can't find any operationExecutor.MountVolume started or failed log like "operationExecutor.MountVolume started for" and also we check volumemanagertotalvolumes metrics, volume is add to dsw. analyze code, if isExpectedError, kubelet not log any thing and operationExecutor will generate operation, add call pendingOperations Run, pendingOperations Run will check operation slice, to find whether has same operation exists, and if operation exist, will check operation's operationPending status, if equal true, means same operation now is excuting, and return AlreadyExists. operationKey composed of volumeName, podName, nodeName so MountVolume operationKey {volumeName, EmptyUniquePodName,EmptyNodeName}, use EmptyUniquePodName to avoid executing mount/map from multiple pods referencing the same volume in parallel soUnmountVolume operationKey {volumeName, podUID,EmptyNodeName}, because All volume plugins can execute unmount/unmap for multiple pods referencing the same volume in parallel now see isOperationExists func, if previousOp.podName equals to EmptyUniquePodName or current operation podName equals to EmptyUniquePodName, will return podNameMatch, mount operation can find umount operation or umount operation can find mount operation because of isOperationExists, this mean if a volume now is excuting mount, a umount can't add to operations, and if a umount now is excuting, a mount operation can't add to operations, but multiple pods referencing the same volume can add multiple umount operation into operation, may lead operation leak.

t0 two pod do umount volume operation t1 pod1 umount failed, update pending to false t2 pod3 add, operation excuter add mount operation, will use index 0, override umount operation t3 mount operation failed , update pending to false t4 pod2 umount operation add, will find index 0, add override opertation, now ,we have two same umount operation, now we have two goroutines to do pod2 umount operations t5 first goroutine pod2 umount failed, will update index 0 operation, update pending to false t6 second goroutine pod2 umount success, will delete index 0 operation, finally lead a umount opertion in cache, and this operation will lead all mount opeartion return AlreadyExistsError, and not cause any csi call. and i think there exists some other scene can trigger this leak. and also, when operation success, deleteOperation, will use tail element to replace, and may increase the probalility of occurrence. because isOperationExists will always pick first match one. and also override mechanism may lead exponentialBackOff invalid for example mount and umount take turns to excute. pv can mount , pod can running reproduce step: create same cronjob that use same pvc(use csi sfs) and csi driver restart periodic No response_ <details</details23 <detailsHuaweiCloud </details<details</details<details</details<details</details<details</detailsCSI
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage
/sig node
+1 meet the same problem
I don't know much about volumeManager. It seems that there is a problem with the override mechanism. Can mount and unmount of the same volume be executed at the same time? If it can, mount should not override any unmount operation. Otherwise, the mount operation should override all matched operations, not just the first one.

also don't know much about design history background, but comments in volume manager seems to conflict with each other among UmountDevice and UnmountVolume. need sameone who familar with it to explain Otherwise, the mount operation should override all matched operations, not just the first one. yes, must discuss rules for the existence of an operation
cc
Suppose there are the following cases: volume's "volumemount" should block the next same "volumemount" or "umountdevice" and all related pods's "volumeumount" exp1: , then could not be , then could be volume's "volumeumount" should block the next same "volumeumount" and it's "volumemount" or "umountdevice", but it should not block it's related other pods's "volumeumount" exp2: , then could not be , then could not be , then could not be , then could be volume's "umountdevice" should block the next same "umountdevice" or "volumemount" and all related pods's "volumeumount" exp3: , then could not be , then could be So we should improve the isOperationExists to satify case 2, that is, the intermediate state should not occur. Maybe since this pr: , this logic remains the same. /cc
/remove-sig node for sig storage to triage
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale


Question: import ( "fmt" "reflect" <ins> "sort" </ins> "sync" jsonpatch "github.com/evanphx/json-patch"

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: I used to fake a clientset to test my code. I found that it became very slow to init the clientset with a big number of objects when I was trying to simulate a large scale cluster, and the time seemed exponential increasing. For example: Initializing a clientset with: 5000 objects took 300ms objects took 2.58s objects took 11.82s objects took 54.64s. It's because the fake clientset used a slice to store each kind of objects, so it's quite slow to init the clientset with massive objects because it checked existence of an object by traversing all objects and comparing with them before adding one, which leads to O(n^2) time complexity. What you expected to happen: The fake clientset is very useful for development, it should have a reasonable time complexity to allow developers to use it for large scale scenario. How to reproduce it (as minimally and precisely as possible): Run this code: You would see the time spent on each operation, like: Anything else we need to know?: Environment: Kubernetes version (use ): 1.15.0-1.18.0 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I had PR that uses map to store objects, which can reduce the init time to a reasonable value (tens of ms for objects). /assign


Question: } func (loader *fsLoader) Load() (*kubeletconfig.KubeletConfiguration, error) { <del> errfmt := fmt.Sprintf("failed to load Kubelet config files from %q, error: ", loader.configDir) + "%v" </del> // require the config be in a file called "kubelet" <del> path := filepath.Join(loader.configDir, "kubelet") </del> <ins> path := filepath.Join(loader.configDir, kubeletFile) </ins> data, err := loader.fs.ReadFile(path) if err != nil { return nil, fmt.Errorf("failed to read init config file %q, error: %v", path, err)

From in The most important things I want to see through node e2e tests: 1) the kubelet is started from last-known-good-config file 2) kubelet can download the new config properly to the node; and restart itself from the new config file 3) kubelet can mark the config bad properly after several round of crashing caused the bad config 4) Kubelet can roll back the last-known-good-config and restart itself from that good one 5) kubelet can properly reports its configuration status Note: (2) is covered by existing e2enode tests (1, 4, 5) were covered by the , all of which I plan to turn into automated tests after this PR merges - but I'm trying not to increase the size of this PR any further ~(3) we need to figure out how to test this. I don't want to rely on a "valid" config that causes a crash loop - because the very existence of such a config should be viewed as a bug. I also don't want to include a "please crash loop for our tests" field in the config, because that's an ugly API. The best solution is not clear here.~ We decided to remove crash-loop detection, see . Filing this issue to track this work: into e2e_node tests [x] remote config [x] local (from file) config [x] ~test the crash-loop detection~ removed, see above /cc
[MILESTONENOTIFIER] Milestone Removed Important: This issue was missing labels required for the v1.8 milestone for more than 7 days: kind: Must specify exactly one of [, , ]. priority: Must specify exactly one of [, , ]. Removing it from the milestone. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
The test cases in the OP have been covered by our e2e_node tests for a while.


Question: <ins> /* Copyright 2014 Google Inc. All rights reserved. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package util import ( "hash/adler32" "testing" ) type wheel struct { radius uint32 } type unicycle struct { primaryWheel   *wheel licencePlateID string } func TestDeepObjectPointer(t *testing.T) { // Arrange wheel1 := wheel{radius: 17} wheel2 := wheel{radius: 22} wheel3 := wheel{radius: 17} myUni1 := unicycle{licencePlateID: "blah", primaryWheel: &wheel1} myUni2 := unicycle{licencePlateID: "blah", primaryWheel: &wheel2} myUni3 := unicycle{licencePlateID: "blah", primaryWheel: &wheel3} hasher1 := adler32.New() hasher2 := adler32.New() hasher3 := adler32.New() // Act DeepHashObject(hasher1, myUni1) hash1 := hasher1.Sum32() DeepHashObject(hasher2, myUni2) hash2 := hasher2.Sum32() DeepHashObject(hasher3, myUni3) hash3 := hasher3.Sum32() // Assert if hash1 == hash2 { t.Errorf("hash1 (%d) and hash2(%d) must be different because they have different values for wheel size", hash1, hash2) } if hash1 != hash3 { t.Errorf("hash1 (%d) and hash3(%d) must be the same because although they point to different objects, they have the same values for wheel size", hash1, hash3) } } </ins>

Is this to follow pointers?
Context: The follows pointers when printing. Unlike JSON marshaling/unmarshaling it also handles private fields.

I am experiencing some weird issues when enabling HTTP liveness probes for some containers. I configure the following for all HTTP-serving containers (this is part of a template used by a replication controller with a single replica): When this is configured, the following issues start to occur: One pod is already running without problems. When I add another replication controller which creates a new pod (also with the liveness probe set), the first pod (the one that was already running) is restarted. This happens even when the two pods are in different namespaces. But when a pod's HTTP server fails completely (connection to the port is refused), the liveness probe will not trigger and the pod will not be restarted. This behavior occurs consistently when I start with no replication controllers and then add them one by one. When I disable the liveness probes on the affected pods/templates, they are no longer restarted in this manner.
You say you're starting two replication controllers-- can you verify that their label selectors do not intersect?
Yes, each replication controller has a selector over a single label in the form of so the labels do not overlap. Also, this problem occurs even when the apps are in different namespaces and goes away when the liveness probe is removed.
Thanks-- last question-- Do the pods have to be scheduled on the same node for this to occur? Your second point I believe is a problem here: we treat an error during HTTP GET as healthy for some inexplicable reason. I can't really explain your first observation, unless perhaps the pods and containers have the same names, but in different namespaces AND kubelet has a namespace handling bug?
cc
"Unknown" != "Healthy" I think it's an open question about whether we want to treat a GET failure as "Unhealthy" we certainly could... --brendan , Daniel Smith wrote:
Giving this to Saad. I think it's pretty clear that if kubelet can't do an HTTP GET, then the pod isn't healthy-- they're on the same host, HTTP get can't fail for network reasons. Our internal system makes this assumption. (I broke it once!)
tl;dr... When a second replication controller is the hash of the container of the first replication controller changes causing it to be killed and recreated. The hash changes because the LivenessProbe pointer changes. Need to figure out why that happens.

Spent time trying to understand why creating two replication controllers, both with http liveness probes, causes the container from the first replication controller to be killed and then recreated when the second replication controller is created. It turns out, as soon as the second replication controller is created, the hash of the container from the first replication controller changes and thus the container is killed: Let's see how the expected and actual hashes are computed and how they end up differing. The actual hash is pulled from the docker container name (dockerContainers.FindPodContainer): The expected hash is created from the object. Specifically, this code (): Expected hash before second replication controller is : Expected hash after second replication controller is : The difference between the container before and after is: So the hash changes because the LivenessProbe pointer in the object changes. Need to figure out why this happens.
Great detective work-- I think it's clear that we shouldn't be including pointers in our hash, so we need to revisit that and explicitly hash only the things that we care about.
Adding second replication controller changes the pointer of LivenessProbe in container config is another mystery. We need to make sure not randomly update other fields too.
Dug in to this some more: confirmed that the value of the original container's LivenessProbe doesn't actually change when the second replication controller is created--only the pointer to the LivenessProbe object changes. It turns out that the instance of the  object being used to compute the expected hash is not guaranteed to be the same instance for the lifespan of the actual docker container. In fact, every time there is an etcd update (e.g. a new pod is created) a new boundpods object is created and although the values of the existing pod/container container objects don't change, because new instances are created the pointers do change (by design). So the fix for this is to ensure that the expected hash is computed only on container values not pointers. One way to ensure this would be to make sure the container object specifies no pointer fields. However, LivenessProbe and Lifecycle, since they are optional, must be able to be set to . Thus, pointer fields can't simply be converted to value fields. Another approach would be to only hash some subset of the values (maybe the name and image) of the container.

However, containers should be destroyed/recreated (meaning the hash must change) if the actual values of any of the container fields changes (e.g. the cpu, memory, volume mounts, liveness probe path, etc.), not just the container name/image. Thus, pointer fields can't simply be disregarded for the hash computation. Instead the fix must modify the expected hash to be computed on the complete deep values of the entire container object (including the actual LivenessProbe values, instead of the pointer). The easiest way to do this is to marshal the container object to json and compute the hash from the resulting byte array.


Question: } // Currently two modes of proxying are available: 'userspace' (older, stable) or 'iptables' <del> // (experimental). If blank, look at the Node object on the Kubernetes API and respect the // 'net.experimental.kubernetes.io/proxy-mode' annotation if provided.  Otherwise use the // best-available proxy (currently userspace, but may change in future versions).  If the // iptables proxy is selected, regardless of how, but the system's kernel or iptables // versions are insufficient, this always falls back to the userspace proxy. </del> <ins> // (newer, faster). If blank, use the best-available proxy (currently iptables, but may // change in future versions).  If the iptables proxy is selected, regardless of how, but // the system's kernel or iptables versions are insufficient, this always falls back to the // userspace proxy. </ins> type ProxyMode string const (

This annotation dates back almost a year. This is a relatively easy project for someone to contribute.
I'll work on getting a pull request in tomorrow for this. Ive been looking at contributing and think this would be a great place to start
nevermind. Looks like a pull request already exists. Overlooked the mention due to being on mobile. Cheers
We'll file more :)


Question: ContainersToKill:  getKillMapWithInitContainers(basePod, baseStatus, []int{}), }, }, <ins> "Pod sandbox not ready, init container failed, but RestartPolicy == Never; kill pod only": { mutatePodFn: func(pod *v1.Pod) { pod.Spec.RestartPolicy = v1.RestartPolicyNever }, mutateStatusFn: func(status *kubecontainer.PodStatus) { status.SandboxStatuses[0].State = runtimeapi.PodSandboxState_SANDBOX_NOTREADY }, actions: podActions{ KillPod:           true, CreateSandbox:     false, SandboxID:         baseStatus.SandboxStatuses[0].Id, Attempt:           uint32(1), ContainersToStart: []int{}, ContainersToKill:  getKillMapWithInitContainers(basePod, baseStatus, []int{}), }, }, "Pod sandbox not ready, and RestartPolicy == Never, but no visible init containers;  create a new pod sandbox": { mutatePodFn: func(pod *v1.Pod) { pod.Spec.RestartPolicy = v1.RestartPolicyNever }, mutateStatusFn: func(status *kubecontainer.PodStatus) { status.SandboxStatuses[0].State = runtimeapi.PodSandboxState_SANDBOX_NOTREADY status.ContainerStatuses = []*kubecontainer.ContainerStatus{} }, actions: podActions{ KillPod:                  true, CreateSandbox:            true, SandboxID:                baseStatus.SandboxStatuses[0].Id, Attempt:                  uint32(1), NextInitContainerToStart: &basePod.Spec.InitContainers[0], ContainersToStart:        []int{}, ContainersToKill:         getKillMapWithInitContainers(basePod, baseStatus, []int{}), }, }, </ins> } { pod, status := makeBasePodAndStatusWithInitContainers() if test.mutatePodFn != nil {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: If a pod's restart policy is "Never", and kubelet failed to create the PodSandbox (for whatever reason) once, the pod will be stuck at "ContainerCreating" forever. What you expected to happen: Since the user/application containers were never created, kubelet should retry the PodSandbox creation. The fact that the pod remains in a ContainerCreating state while kubelet explicitly does nothing to resolve the situation also confuses the user. How to reproduce it (as minimally and precisely as possible): I don't have an easy way to trigger a podsandbox creation failure off the top of my head, but this should be doable by messing with the CNI binaries/configurations. Anything else we need to know?: Relevant code snippet: Environment: Kubernetes version (use ): Only checked the master branch (1.16), but all supported release branches should have the issue Cloud provider or hardware configuration: not relevant OS (e.g: ): not relevant Kernel (e.g. ): not relevant Install tools: Network plugin and version (if this is a network-related bug): not relevant; though the initial PodSandbox creation failure could be caused by networking issues. Others:
thanks for posting! I'm looking at the code, and it seems like one fix could be checking if this pod has an associated sandbox - if it does, we want to respect the current behavior. If it doesn't, we want to attempt to create the sandbox. I was thinking something similar to: Thoughts? Is there a simplex fix?
Hi! Thank you for filing this - it matches the behaviour we've been seeing. Am I correct that this is a regression from (), and so only affects Kubernetes =1.13 ?
Your solution may not work for the one specific failure I checked, where the PodSandbox container was created, but call to the CNI plugin failed (for some reason; could be caused by a race condition). This left us a not-ready PodSandbox with no IP. Probably have to actually determine whether the user containers were created before. Need to check the code closer though.
Thanks. That's probably correct. I saw this in happening in 1.13
Gotcha - thanks! Taking a look at your pr now :)


Question: } func (c *nodePlugin) admitNode(nodeName string, a admission.Attributes) error { <del> if a.GetName() != nodeName { return admission.NewForbidden(a, fmt.Errorf("cannot modify other nodes")) </del> <ins> requestedName := a.GetName() // On create, get name from new object if unset in admission if len(requestedName) == 0 && a.GetOperation() == admission.Create { node, ok := a.GetObject().(*api.Node) if !ok { return admission.NewForbidden(a, fmt.Errorf("unexpected type %T", a.GetObject())) } requestedName = node.Name } if requestedName != nodeName { return admission.NewForbidden(a, fmt.Errorf("node %s cannot modify node %s", nodeName, requestedName)) </ins> } return nil }

Is this a BUG REPORT or FEATURE REQUEST? (choose one):BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"7+", GitVersion:"v1.7.0-alpha.4.52+", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-05-19T06:57:30Z", GoVersion:"go1.8.1", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"7+", GitVersion:"v1.7.0-alpha.4.52+", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-05-19T06:57:30Z", GoVersion:"go1.8.1", Compiler:"gc", Platform:"linux/amd64"} Environment: Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: What happened: What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know: None


Question: var quotaTestCases = []quotaTestCase{ { <ins> "SupportsQuotaOnQuotaVolume", </ins> "/quota1/a", "", 1024, "Supports", "", "", true, true, 0, 0, 0, 0, 1, 1, 0, 0, 1, 1, 1, }, { <ins> "AssignQuotaFirstTime", </ins> "/quota1/a", "", 1024, "Set", projects1, projid1, true, true, 1, 1, 1, 1, 0, 0, 1, 1, 0, 0, 0, }, { <ins> "AssignQuotaFirstTime", </ins> "/quota1/b", "x", 1024, "Set", projects2, projid2, true, true, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, }, { <ins> "AssignQuotaFirstTime", </ins> "/quota2/b", "x", 1024, "Set", projects3, projid3, true, true, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, }, { <ins> "AssignQuotaSecondTimeWithSameSize", </ins> "/quota1/b", "x", 1024, "Set", projects3, projid3, <ins> true, true, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, }, { "AssignQuotaSecondTimeWithDifferentSize", "/quota2/b", "x", 2048, "Set", projects3, projid3, </ins> true, false, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, }, { <ins> "ClearQuotaFirstTime", </ins> "/quota1/b", "", 1024, "Clear", projects4, projid4, true, true, -1, -1, -1, -1, 0, -1, -1, -1, -1, -1, -1, }, { <ins> "SupportsQuotaOnNonQuotaVolume", </ins> "/noquota/a", "", 1024, "Supports", projects4, projid4, false, false, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, }, { <ins> "ClearQuotaFirstTime", </ins> "/quota1/a", "", 1024, "Clear", projects5, projid5, true, true, -1, -1, -1, -1, 0, -1, -1, -1, -1, -1, -1, }, { <ins> "ClearQuotaSecondTime", </ins> "/quota1/a", "", 1024, "Clear", projects5, projid5, true, false, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, }, { <ins> "ClearQuotaFirstTime", </ins> "/quota2/b", "", 1024, "Clear", "", "", true, true, -1, -1, -1, -1, 0, -1, -1, -1, -1, -1, -1, },

When sizeLimit is specified subsequent invocations will fail with the message: No error should be raised. Create a pod with emptyDir and sizeLimit with FSQuotaMonitoring enabled, MountVolume.SetUp is called again only if the first iteration takes longer then the actual reconciler loop interval (100ms). No response <detailsversion.Info{Major:"1", Minor:"27+", GitVersion:"v1.27.0-beta.0.87+", GitCommit:"", GitTreeState:"clean", BuildDate:"2023-04-10T09:05:56Z", GoVersion:"go1.20.2", Compiler:"gc", Platform:"linux/amd64"} </details<detailsSelf-managed cluster in Azure </details<detailsNAME="Red Hat Enterprise Linux" VERSION="9.1 (Plow)" ID="rhel" IDLIKE="fedora" VERSIONID="9.1" Linux test-master-0 6.3.0-rc3 SMP PREEMPTDYNAMIC Thu Apr 6 20:44:15 UTC 2023 x8664 x8664 x8664 GNU/Linux </details<detailsContainerd 1.6.15 </details<detailsCNI: CANAL 3.24.1 </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
May you please share more information? Which version of Kubernetes you are using? OS version etc.
The problem occurs with the latest master branch, I updated the initial post with versions.
Thank you.


Question: } glog.Infof("ensureExternalLoadBalancer(%s): Deleted forwarding rule.", lbRefStr) } <del> if tpExists && tpNeedsUpdate { // Pass healthchecks to DeleteExternalTargetPoolAndChecks to cleanup health checks after cleaning up the target pool itself. var hcNames []string if hcToDelete != nil { hcNames = append(hcNames, hcToDelete.Name) } if err := gce.DeleteExternalTargetPoolAndChecks(apiService, loadBalancerName, gce.region, clusterID, hcNames...); err != nil { return nil, fmt.Errorf("failed to delete existing target pool for load balancer (%s) update: %v", lbRefStr, err) } glog.Infof("ensureExternalLoadBalancer(%s): Deleted target pool.", lbRefStr) } // Once we've deleted the resources (if necessary), build them back up (or for // the first time if they're new). if tpNeedsUpdate { createInstances := hosts if len(hosts) > maxTargetPoolCreateInstances { createInstances = createInstances[:maxTargetPoolCreateInstances] } // Pass healthchecks to createTargetPool which needs them as health check links in the target pool if err := gce.createTargetPool(apiService, loadBalancerName, serviceName.String(), ipAddressToUse, gce.region, clusterID, createInstances, affinityType, hcToCreate); err != nil { return nil, fmt.Errorf("failed to create target pool for load balancer (%s): %v", lbRefStr, err) } if hcToCreate != nil { glog.Infof("ensureExternalLoadBalancer(%s): Created health checks %v.", lbRefStr, hcToCreate.Name) } if len(hosts) <= maxTargetPoolCreateInstances { glog.Infof("ensureExternalLoadBalancer(%s): Created target pool.", lbRefStr) } else { glog.Infof("ensureExternalLoadBalancer(%s): Created initial target pool (now updating with %d hosts).", lbRefStr, len(hosts)-maxTargetPoolCreateInstances) created := sets.NewString() for _, host := range createInstances { created.Insert(host.makeComparableHostPath()) } if err := gce.updateTargetPool(loadBalancerName, created, hosts); err != nil { return nil, fmt.Errorf("failed to update target pool for load balancer (%s): %v", lbRefStr, err) } glog.Infof("ensureExternalLoadBalancer(%s): Updated target pool (with %d hosts).", lbRefStr, len(hosts)-maxTargetPoolCreateInstances) } </del> <ins> if err := gce.ensureTargetPoolAndHealthCheck(tpExists, tpNeedsRecreation, apiService, loadBalancerName, clusterID, ipAddressToUse, hosts, hcToCreate, hcToDelete); err != nil { return nil, err </ins> } <del> if tpNeedsUpdate || fwdRuleNeedsUpdate { </del> <ins> if tpNeedsRecreation || fwdRuleNeedsUpdate { </ins> glog.Infof("ensureExternalLoadBalancer(%s): Creating forwarding rule, IP %s (tier: %s).", lbRefStr, ipAddressToUse, netTier) if err := createForwardingRule(gce, loadBalancerName, serviceName.String(), gce.region, ipAddressToUse, gce.targetPoolURL(loadBalancerName), ports, netTier); err != nil { return nil, fmt.Errorf("failed to create forwarding rule for load balancer (%s): %v", lbRefStr, err)

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug /sig network /area cloudprovider What happened: The cloud provider interface that should . The GCE provider does this for but not for . Every other in-tree provider implements by fully updating the load balancer including updated backend nodes: , , , And the following out-of-tree providers do as well and Oracle (not oss yet). What you expected to happen: I expect that all cloud providers implement by creating or fully updating the load balancer if it already exists. Related discussion: /cc
/assign Make sense to make implementation in GCE consistent with others for the ease of up-coming service controller enhancement. This sounds like a 1.10 task.
is this task blocking you on another attempt at refactoring the service controller sync logic?
yup


Question: "k8s.io/api/core/v1" apierrors "k8s.io/apimachinery/pkg/api/errors" <ins> apierrs "k8s.io/apimachinery/pkg/api/errors" </ins> metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/fields" utilerrors "k8s.io/apimachinery/pkg/util/errors"

Fix status: [x] fixed in master in [x] fixed in release-1.10 branch in , will be in 1.10.0 [x] fixed in release-1.9 branch in , will be in 1.9.7 [x] fixed in release-1.8 branch in , will be in 1.8.11 [x] fixed in release-1.7 branch in , will be in 1.7.16 In SubPath fixes, PR adds bind mount for subpaths. It uses as the path name. The volumename in this path is the container's mount volume name which is also called OuterVolumeSpecName in . This becomes an issue for volume reconstruction when kubelet restarts. Volume scans the volume directories and recover the InnerVolumeSpecName (which is used to construct pod volume paths). Volume reconstruction has not enough information to recover the OuterVolumeSpecName and figure out which volume it maps to. In case kubelet needs to clean up mounts (it happens when volume spec cannot be reconstructed), it will fail to clean up the subpaths because of lacking the OuterVolumeSpecName. We think it is better to use InnerVolumeSpecName for volume subPath instead of OuterVolumeSpecName so that volume clean up can work. The issue we might have is that some users might already deploy a cluster with subpath might already . If the users perform an in-place upgrade, since new version will use InnerVolumeSpecName which might have conflicts with some other older version's OuterVolumeSpecName among different volumes. This change requires clear documentation and communication with the users.
cc
This impacts reconstruction of PVCs, where the inner and outer names are different.
/assign
While testing the e2es against a cluster without the fix, I noticed that subsequent pods using volumes failed to start. My theory is that the unmount volume failed because the subpath was still mounted, and that blocked the volume manager from processing other pods. But I thought that volume manager is supposed to run the unmount volume operation in a separate thread per volume, so this shouldn't happen. Need to inspect the code more carefully to confirm the theory. could the volume reconstruction refactor have a unmount volume call in the main volume manager thread?
runs in reconciler's goroutine - so if that function is blocked, it will not be able to process any more volumes.

Was the problem that failed to unmount volume but got hung/stuck ?
I haven't confirmed for sure, but my guess is that unmount volume failed because the subpath was still mounted
moving into the milestone for discussion /milestone v1.10
/priority critical-urgent /status in-progress
/status approved-for-milestone per discussion in sig-release meeting today
[MILESTONENOTIFIER] Milestone Issue: Up-to-date for process <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
Re-opening to track fixes in release branches
keeping in the v1.10 milestone until merges
since this is now fixed in 1.10, I'm taking it out of the milestone so that the release team doesn't keep checking it.
Can this issue be closed now?


Question: Properties: map[string]spec.Schema{ "ready": { SchemaProps: spec.SchemaProps{ <del> Description: "ready indicates that this endpoint is prepared to receive traffic, according to whatever system is managing the endpoint. A nil value indicates an unknown state. In most cases consumers should interpret this unknown state as ready. For compatibility reasons, ready should never be "true" for terminating endpoints.", </del> <ins> Description: "ready indicates that this endpoint is prepared to receive traffic, according to whatever system is managing the endpoint. A nil value indicates an unknown state. In most cases consumers should interpret this unknown state as ready. For compatibility reasons, ready should never be "true" for terminating endpoints, except when the normal readiness behavior is being explicitly overridden, for example when the associated Service has set the publishNotReadyAddresses flag.", </ins> Type:        []string{"boolean"}, Format:      "", },

The EndpointSlice API says and it works as expected in normal circumstances I delete the pod and the endpoints conditions are the expected However, if during this time I set we run into an incoherence with the API doc, but it seems that we already documented this edge case in the code I think that at this point is just to amend this information to the API, no? /kind bug /kind documentation
/sig network /assign
I would like to work on this. I just want a bit more clarification on what needs to be done!
I proposed a solution, that is clarify it in the doc ... once we reach an agreement in this issue we implement it ...
+1 to docs. I'd make it a little more vague, like: """ when the normal readiness behavior is being explicitly overridden, for example when the associated Service has set the publishNotReadyAddresses flag. """
so, the thing is to update the docs here with Tim's suggestion and here and do to update the generated code and submit a PR
/assign
I am not able to run command in windows, I am checking it. This error message is coming: *
/triage accepted
Is there a PR for this? Happy to review
I will raise the PR if my above problem gets solved. I am trying to make an update but the error mentioned above is coming.
What platform are you on? If the Makefile were really broken EVERYTHING would be failing.
he's on windows ... you will need a linux/macos. see
Let me know if you want someone to take it over.
I think in that case, you can assign this to someone else.


Question: - gocritic text: "ifElseChain: rewrite if-else to switch statement" <ins> # Only packages listed here opt into the strict "exported symbols must be documented". # # Exclude texts from https://github.com/golangci/golangci-lint/blob/ab3c3cd69e602ff53bb4c3e2c188f0caeb80305d/pkg/config/issues.go#L11-L103 - linters: - golint - revive - stylecheck text: comment on exported (method|function|type|const)|should have( a package)? comment|comment should be of the form|comment on exported (method|function|type|const)|should have( a package)? comment|comment should be of the form|exported (.+) should have comment( (or a comment on this block))? or be unexported|package comment should be of the form "(.+)...|comment on exported (.+) should be of the form "(.+)...|should have a package comment path-except: cmd/kubeadm </ins> # The following issues were deemed "might be worth fixing, needs to be # decided on a case-by-case basis".  This was initially decided by a # majority of the developers who voted in

Looks like the golangci-lint-pr-hints linter was updated recently to put the feature gate comment ahead of the owner field in All the feature gates in are ordered by the following comments: // owner: // kep: // alpha: // beta: // // Describe the feature gate <declare the feature gateThe new linter changes now requires that the comment describing the feature gate should even precede the owner comment line which goes against the structure followed by the entire file and this needs to be fixed Feature gates should be ordered as follows to maintain parity with the rest of // owner: // kep: // alpha: // beta: // // Describe the feature gate <declare the feature gateUpdate/add new feature gate to No response <details</detailsmaster branch of k8s/k8s <details</detailsN/A <details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
!
cc
cc
it was supposed to only apply to cmd/kubeadm.
/sig testing
/assign
Thank you for fixing that.
If I break it, I fix it... :hot_face:


Question: } func (rs *REST) updateNodePorts(oldService, newService *api.Service, nodePortOp *portallocator.PortAllocationOperation) error { <del> oldNodePorts := CollectServiceNodePorts(oldService) </del> <ins> oldNodePortsNumbers := CollectServiceNodePorts(oldService) newNodePorts := []ServiceNodePort{} portAllocated := map[int]bool{} </ins> <del> newNodePorts := []int{} </del> for i := range newService.Spec.Ports { servicePort := &newService.Spec.Ports[i] <del> nodePort := int(servicePort.NodePort) if nodePort != 0 { if !contains(oldNodePorts, nodePort) { err := nodePortOp.Allocate(nodePort) </del> <ins> nodePort := ServiceNodePort{Protocol: servicePort.Protocol, NodePort: servicePort.NodePort} if nodePort.NodePort != 0 { if !containsNumber(oldNodePortsNumbers, int(nodePort.NodePort)) && !portAllocated[int(nodePort.NodePort)] { err := nodePortOp.Allocate(int(nodePort.NodePort)) </ins> if err != nil { <del> el := field.ErrorList{field.Invalid(field.NewPath("spec", "ports").Index(i).Child("nodePort"), nodePort, err.Error())} </del> <ins> el := field.ErrorList{field.Invalid(field.NewPath("spec", "ports").Index(i).Child("nodePort"), nodePort.NodePort, err.Error())} </ins> return errors.NewInvalid(api.Kind("Service"), newService.Name, el) } <ins> portAllocated[int(nodePort.NodePort)] = true </ins> } } else { <del> nodePort, err := nodePortOp.AllocateNext() </del> <ins> nodePortNumber, err := nodePortOp.AllocateNext() </ins> if err != nil { // TODO: what error should be returned here?  It's not a // field-level validation failure (the field is valid), and it's // not really an internal error. return errors.NewInternalError(fmt.Errorf("failed to allocate a nodePort: %v", err)) } <del> servicePort.NodePort = int32(nodePort) </del> <ins> servicePort.NodePort = int32(nodePortNumber) nodePort.NodePort = servicePort.NodePort </ins> } <del> // Detect duplicate node ports; this should have been caught by validation, so we panic if contains(newNodePorts, nodePort) { panic("duplicate node port") </del> <ins> if containsNodePort(newNodePorts, nodePort) { return fmt.Errorf("duplicate nodePort: %v", nodePort) </ins> } newNodePorts = append(newNodePorts, nodePort) } <ins> newNodePortsNumbers := CollectServiceNodePorts(newService) </ins> // The comparison loops are O(N^2), but we don't expect N to be huge // (there's a hard-limit at 2^16, because they're ports; and even 4 ports would be a lot) <del> for _, oldNodePort := range oldNodePorts { if contains(newNodePorts, oldNodePort) { </del> <ins> for _, oldNodePortNumber := range oldNodePortsNumbers { if containsNumber(newNodePortsNumbers, oldNodePortNumber) { </ins> continue } <del> nodePortOp.ReleaseDeferred(oldNodePort) </del> <ins> nodePortOp.ReleaseDeferred(int(oldNodePortNumber)) </ins> } return nil

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --BUG REPORT /kind bug What happened: I tried to submit a Service Replace () to the kube-apiserver and this results the kube-apiserver container crashing and restarting. PUT localhost:8001/api/v1/namespaces/<namespace/services/<servicenameWhat you expected to happen: I expected the server not to crash and to get validation errors if the request was invalid. How to reproduce it (as minimally and precisely as possible): Create a service, get the resourceVersion and clusterIP from the created resource GET localhost:8001/api/v1/namespaces/<namespace/services/<servicenameSubmit a replace command to PUT localhost:8001/api/v1/namespaces/<namespace/services/<servicenamewith body like If the same request is submitted with the nodePort fields in the ports array then the request succeeds. I get that I may be sending an invalid request without a node port (though shouldn't the merge deal with that?) but it shouldn't take down the API server. Anything else we need to know?: Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-05-10T23:29:08Z", GoVersion:"go1.8.1", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-10-11T23:16:41Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: Bare metal cluster OS (e.g. from /etc/os-release): Ubuntu 16.04.3 LTS (Xenial Xerus) Kernel (e.g. ): 4.4.0-98-generic Install tools: Others:
thanks for reporting this! Seems like this is similar to and should be fixed by . can you PTAL that PR to see if it's good to merge?
/sig api-machinery
Duplicate of

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Re-applying a NodePort service that has two ports fails and crashes the API server: What you expected to happen: Do not crash, service is configured correctly. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: on VBox provider in macOS OS (e.g. from /etc/os-release): macOS/Minikube Kernel (e.g. ): Install tools: Others:
Seems that in func updateNodePorts, it ignored protocol when Detecting duplicate nodeport, I would have a fix on it :)
/sig network
FYI: I think this would be fixed in
/cc Maybe this is the time to have another review on and merge it :)
/assign
Fine, That makes sense

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug /sig network /cc /assign What happened: a ClusterIP type service the service. Change it's type to NodePort and add a Port. Note: the new port has same port number but protocol with the existing Port. the updated service. The TCP Port was eaten. What you expected to happen: The updated service has both TCP and UDP ports. How to reproduce it (as minimally and precisely as possible): See the step above. Anything else we need to know?: If I create a NodePort service with the above two ports, it works. I write a test case to verify this case. I got a panic. The root cause is as below. We should take another look at this logic. BTW: changed some related code but it doesn't change the logic here. So I believe this bug is not introduced by . I also tested this on v1.5.1, it has the same issue. Environment: Kubernetes version (use ): HEAD Cloud provider or hardware configuration**: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
I will be OOO the next week. Since this bug is not so urgent, I will fix it ASAP when I'm back. Folks who are interested in this can take it :) BTW, I can join in the discussion from my phone.
I'm not sure this is a bug. In the first configuration yaml, you're defining the ports array twice. I'm the second, you're only defining it once. The ports array should only be defined once.
Did you mean the below yaml by ? Please note that the two ports have same port number but different protocols. This is allowed.
Yes, that's the one I meant. But note you have "ports:" twice.
Ah, thanks for pointing it out! This might be the cause. But I remembered I also tested the case without the second line, it had the same issue. I can't verify this now. Will test it when I'm back to my keyboard.
I try the case without the second line, the issue also exists.

So I believe there is a bug here. Seems like there might be two issues here: a port is lost as is described in this issue. we shouldn't allow to configure two lines in the manifest. i.e. we should add validation for this case.
/remove-priority awaiting-more-evidence
can you please provide (or edit) example YAMLs then to allow for attempts at duplication?
Update: reproducing without the extra line. a ClusterIP type service the service (). Change it's type to NodePort and add a Port which has the same port number but different protocol with the existing Port. the updated service. We can see that the UDP Port was not created. I have dug into the code and will send a patch to fix it.
PR sent. Reviews welcome! Thanks!
There are/were bugs in the patch implementation wrt ports, because they need compound merge keys.
Yeah I had some discussios with in . But we also do have a bug in our service updating codes. See:


Question: logOptions.SinceSeconds = &sec } <del> if len(o.Selector) > 0 && o.Tail == -1 { </del> <ins> if len(o.Selector) > 0 && o.Tail == -1 && !o.TailSpecified { </ins> logOptions.TailLines = &selectorTail } else if o.Tail != -1 { logOptions.TailLines = &o.Tail

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When attempting to get all log lines using only 10 lines of logs are returned What you expected to happen: All log lines should be returned How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: This was introduced in in an attempt to fix another issue where selectors would ignore tail values !=-1. I think the problem here is the logic cannot tell if the was set explicitly by the command line or is simply the initial default value. Environment: Kubernetes version (use ): 1.13.3 Cloud provider or hardware configuration: Kubeadm OS (e.g: ): 18.04.2 LTS (Bionic Beaver) Kernel (e.g. ): Linux ubuntu 4.15.0-45-generic -Ubuntu SMP Tue Jan 29 16:28:13 UTC 2019 x8664 x8664 x86_64 GNU/Linux Install tools: Others:
/sig cli
Maybe is not a bug. See the kubectl help. --tail=-1: Lines of recent log file to display. Defaults to -1 with no selector, showing all log lines otherwise 10, if a selector is provided. I think i mean a selector is provided the --tail should not use -1 ?
Thanks for you reply. There are some background in below tickets, --tail=-1 is one of the option. Please check.
The help indicates that the default is 10 if the selector is used. But if someone explicitly asks for -1 it should still work, at least in my opinion.
Encountering similar behaviour to this myself. Is there any chance whatever fixes get applied can also be applied to the branch? Since that also has strange behaviour around logs with label selectors and setting .

See
This ticket has two issues: First, It's said "TailLines must be greater than or equal to 0", however we can specify -1. kubectl logs -l app=test --tail=-3 error: TailLines must be greater than or equal to 0 kubectl logs -l app=test --tail=-1 : Wed Apr 10 06:51:34 UTC 2019 : Wed Apr 10 06:51:35 UTC 2019 : Wed Apr 10 06:51:36 UTC 2019 : Wed Apr 10 06:51:37 UTC 2019 : Wed Apr 10 06:51:38 UTC 2019 : Wed Apr 10 06:51:39 UTC 2019 : Wed Apr 10 06:51:40 UTC 2019 : Wed Apr 10 06:51:41 UTC 2019 : Wed Apr 10 06:51:42 UTC 2019 : Wed Apr 10 06:51:43 UTC 2019 Second, --tail=-1 is the legacy usage which show all lines, here just return 10 lines. kubectl logs -l app=test --tail=-1 The behavior after fix: Test cases ./kubectl logs -l app=test --tail=-3 ./kubectl logs -l app=test --tail=-1 ./kubectl logs -l app=test --tail=2 ./kubectl logs -l app=test --tail=0
Interesting. I think this also causes to fail: ^-- expected record ^-- some later record (there are only 2 pods running, so is not relevant) ^-- adding makes it work
A bit strange that it finds records a few seconds older as well.. but that's not a big concern, and not relevant for this issue. vs. Although that only happens when the workaround is used: :thinking: P.S.


Question: } return false, nil }) <del> framework.ExpectNoError(err, "failed to see %v event", watch.Modified) </del> <ins> if err != nil { framework.Logf("failed to see %v event: %v", watch.Modified, err) } </ins> ginkgo.By("getting the Pod and ensuring that it's patched") pod, err := f.ClientSet.CoreV1().Pods(testNamespaceName).Get(context.TODO(), testPodName, metav1.GetOptions{})

pull-kubernetes-conformance-kind-ga-only-parallel [sig-node] Pods should run through the lifecycle of Pods and PodStatus [Conformance] After is merged. It relies on events which are not reliable as discussed . No response /sig node
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The test is not failing that much looking at the testgrid. Is it actually an issue?
Not so often, but it does happen. So, I think that it is worth fixing.
still flaking -
flaking conformance tests are concerning and should take priority over new conformance test additions
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale I still see this test failing , also in periodic jobs
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale seems like it's is still failing
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
still flake /remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/assign Namanl2001
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/remove-lifecycle rotten
please unassign if you are not working on it. Seems to be still flaking
It still flakes but not that much. I checked the history: adjusted the timeout from 30s to 1m. changed it not to fail for watching Delete Event, and just print it like below: This time, we see the above adding warning and then failed. Looking into kubelet/container runtime log. The pod is just deleted after 1 min, maybe 1min10s. I'm afraid that we have to adjust the timeout of the delete pod again to fix it. Is a valid pod deletion timeout?

I think most deletion timeout of pod is 5min, however this is too long after pod with .
but why it takes so long kubelet to delete a pod? what are all of those ""Failed to delete cgroup paths" messages? can this be hiding some problem with runc or the kubelet?
Sep 19 13:14:56 kinder-external-etcd-worker-2 kubelet[359]: E0919 13:14:56. 359 ] "Error syncing pod, skipping" err="failed to delete cgroup paths for [kubepods besteffort pod53d19839-496a-4be8-8964-] : unable to destroy cgroup paths for cgroup [kubepods besteffort pod53d19839-496a-4be8-8964-] : Timed out while waiting for systemd to remove kubepods-besteffort-" pod="pods-900/pod-test" podUID=-496a-4be8-8964- Sep 19 13:14:56 kinder-external-etcd-worker-2 kubelet[359]: I0919 13:14:56. 359 ] "Failed to delete cgroup paths" cgroupName=[kubepods besteffort pod53d19839-496a-4be8-8964-] err="unable to destroy cgroup paths for cgroup [kubepods besteffort pod53d19839-496a-4be8-8964-] : Timed out while waiting for systemd to remove kubepods-besteffort-" Sep 19 13:14:56 kinder-external-etcd-worker-2 containerd[220]: time="2023-09-19T13:14:56.925525892Z" level=info msg="StartContainer for "e3b435116d2640d7c8166332e1c4bf6722f97ac1401035cbe443704c8b442efd" returns successfully" Sep 19 13:14:56 kinder-external-etcd-worker-2 kubelet[359]: I0919 13:14:56.

359 ] "Failed to delete cgroup paths" cgroupName=[kubepods besteffort pod53d19839-496a-4be8-8964-] err="unable to destroy cgroup paths for cgroup [kubepods besteffort pod53d19839-496a-4be8-8964-] : Timed out while waiting for systemd to remove kubepods-besteffort-" Sep 19 13:15:01 kinder-external-etcd-worker-2 kubelet[359]: I0919 13:15:01. 359 ] "Cleaned up orphaned pod volumes dir" podUID=-496a-4be8-8964- path="/var/lib/kubelet/pods/-496a-4be8-8964-/volumes" It means timeout. Another failure is not for the same reason.
/joke
I wish I could clean mirrors for a living. It's just something I can see myself doing. <detailsIn response to : /joke Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This is starting to be worrisome
another failure
was another occurrence of this I think. I'm seeing the timeout error for deleting the pod in the build logs: And then the error: It has the same sanbok not found error in the kind-worker logs: Along with the same cgroup error in the logs: So it's probably an instance of the first case mentioned caused by timeout.
Flake in . Delete timeout is 1 minute here. It still need more investigation why this is not deleted in 1 min.(heavy load or logic problems.)
Flake twice in yersterday.


Question: klog.V(4).Infof("Port %s was open before and is still needed", lp.String()) replacementPortsMap[lp] = proxier.portsMap[lp] } else { <del> socket, err := proxier.portMapper.OpenLocalPort(&lp) </del> <ins> socket, err := proxier.portMapper.OpenLocalPort(&lp, isIPv6) </ins> if err != nil { msg := fmt.Sprintf("can't open %s, skipping this externalIP: %v", lp.String(), err)

/kind bug What happened: When creating a node port application (in this case the example guestbook) on a host server that is dual stacked and has AAAA and A DNS entries, the node port on the v6 address is open to kube-proxy and hangs. What you expected to happen: The v6 address would refuse the connection and the client would fall back onto the v4 address which would connect correctly. How to reproduce it (as minimally and precisely as possible): kubeadm init with the above, and join in some workers. Then install the guestbook application Obtain the node port with and telnet to one of the servers on that port from a dual stacked client. The client will connect and refuse to respond to 'GET /' telnet directly to the IPv4 address of the server on the port and all works as expected. Anything else we need to know?: kube-proxy is noted as listening on the port. Environment: Kubernetes version (use ): ubuntu kubectl version Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-07-17T18:53:20Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-07-17T18:43:26Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: Brightbox OS (e.g. from /etc/os-release): Ubuntu 18.04.1 LTS Kernel (e.g. ): Linux srv-jidtj 4.15.0-29-generic -Ubuntu SMP Tue Jul 17 15:39:52 UTC 2018 x8664 x8664 x86_64 GNU/Linux Install tools: kubeadm 1.11 Others:
/sig network
Looks like kube-proxy in ipvs mode listens on the nodeport ports to ensure they're not in use by something else. It just binds and listens, it doesn't serve requests.

It binds on both ipv4 and ipv6 but never expects to receive connections because the iptables/ipvs rules intercept them, but on ipv4 clusters there is nothing stopping ipv6 connections coming in. The comment there does admit that only a bind is necessary to do this, but doing a full listen just made e2e tests easier. /cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen This now causes conformance failure on e2e tests due to the promotion of the Nodeport test in
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Still an issue on 1.16 when running in standard IPv4 mode
As I understand this thread, I have the same problem. I thought that it is a node configuration or some kubernetes flag that I didn't set but it seems it ain't.

I ask already on SO: Environment Kubernetes version (use kubectl version): Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-09-18T14:36:53Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-10-02T16:51:36Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: On premise, on virtual machines using hyper-v OS (e.g. from /etc/os-release): NAME="Ubuntu" VERSION="18.04.3 LTS (Bionic Beaver)" Kernel (e.g. uname -a): Linux kubernetes 5.0.0-29-generic # 31~18.04.1-Ubuntu SMP Thu Sep 12 18:29:21 UTC 2019 x8664 x8664 x86_64 GNU/Linux Install tools: kubeadm version: &version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-09-18T14:34:01Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}
/lifecycle frozen
/assign bowei
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
Looking through the comments, this looks like a bug that could have a targeted fix. /remove-triage unresolved /good-first-issue /help-wanted
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : Looking through the comments, this looks like a bug that could have a targeted fix.

/good-first-issue /help-wanted </details
I will attempt to solve this issue.
any success or progress? If you left this issue I would like to give it a go as a first time contributor. Thanks, Yuval
I'm still working on this as my first issue, and planning to produce a solution by 20th.
I wasn't able to tackle the issue and will not continue trying, feel free to take it.
/assign yuvalsteuer
Still, open -- anyone actively working on this? I would like to attempt if not. :) /assign rkc007
OK, again still open, doesn't look like anyone is working on this. I'd like to try it. The fix that occurs to me is simply to only listen to IPv6 addresses if the cluster is IPv6. If the e2e tests fail in this case (from looking at I don't think they will) then maybe they need to be updated. Does that make sense? What I'm not sure about is if we should do nothing with IPv6 ports or bind (to claim the port) but not listen.
/assign michaelbeaumont


Question: e2emetrics "k8s.io/kubernetes/test/e2e/framework/metrics" imageutils "k8s.io/kubernetes/test/utils/image" admissionapi "k8s.io/pod-security-admission/api" <del> "github.com/onsi/ginkgo" "github.com/onsi/gomega" </del> ) const (

Test: [sig-node] Density [Serial] [Slow] create a sequence of pods latency/resource should be within limit when create 10 pods with 50 background pods /kind flake <!-- If this is a flake, please add: /kind flake --/sig node
/triage accepted /priority important-soon
Although the pod polling exhausted 4 times (interval 2s, total 6s+) to get pod Running, but from the kubelet log, the time from the pod found to Running status reported is almost less than 5s, so wo may reduce the poll interval to 1s, or make this time more relaxed
http://perf- ! This is caused by as . The commit will be reverted, and it make fail yesterday. ! EDITED I mean if this flake points out there are some other performance downgrades in v1.24. Only recently, no version 99% -- -- v1.20 ~2200ms v1.21 ~2200ms v1.22 ~2300ms v1.23 ~2650ms v1.24(before) ~2200ms v1.24(after ) ~6600ms
Editing title as this was also failing on containerd: /milestone v1.24 I think this confirms that the revert above fixed the issue, as the failures stopped after the revert merged. /close
Closing this issue. <detailsIn response to : Editing title as this was also failing on containerd: Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: Expect(scale.Status.Replicas).To(Equal(int32(1))) By("updating a scale subresource") <del> scale.ResourceVersion = "" //unconditionally update to 2 replicas </del> scale.Spec.Replicas = 2 <del> err = c.AppsV1().RESTClient().Put().AbsPath("/apis/apps/v1").Namespace(ns).Resource("statefulsets").Name(ssName).SubResource("scale").Body(scale).Do().Into(scaleResult) </del> <ins> scaleResult, err := c.AppsV1().StatefulSets(ns).UpdateScale(ssName, scale) </ins> if err != nil { framework.Failf("Failed to put scale subresource: %v", err) }

What would you like to be : e2e test suites for: GET /apis/apps/v1/namespaces/{namespace}/statefulsets/{name}/scale PUT /apis/apps/v1/namespaces/{namespace}/statefulsets/{name}/scale Why is this needed: PR has already been merged, but the following tests are still commented out. Therefore, scale sub-resource of statefulsets is not tested and test should be . /sig testing


Question: glusterfs-cluster   10.240.106.152:1,10.240.79.157:1 ``` <ins> We need also create a service for this endpoints, so that the endpoints will be persistented. We will add this service without a selector to tell Kubernetes we want to add its endpoints manually. You can see [glusterfs-service.json](glusterfs-service.json) for details. Use this command to create the service: ```sh $ kubectl create -f examples/glusterfs/glusterfs-service.json ``` </ins> ### Create a POD The following *volume* spec in [glusterfs-pod.json](glusterfs-pod.json) illustrates a sample configuration.

I create a glusterfs endpoint, but after I restart kube-apiserver, the endpoint is missing. Does Kubernetes persistence endpoints?
Do you mean and Endpoints object in the API or something else? , Bin Wang wrote:
Here is what I do: Then use I can see this line: But after I restart kube-apiserver, this endpoint is lost.
Do you have a Service named ? Otherwise it should persist, just the same as any other API object , Bin Wang wrote:
No. This endpoints is just used for persistent volumes, which is used by other pods. I followed . Is it necessary to define a service?
Can one of you guys try to repro?
What does that mean exactly? What kind of cluster are you running? Does anything persist and endure between restarts of the API server? When you restart the API server (why?), are you also restarting etcd or other services?
The cluster just have one master. The pods, secrets, controllers and services are recovered after the restart, but the endpoints are not. About why I restart the API server, well, it may fail (or something else). You cannot expect the api-server runs and never restart. After I restart it, I expect all the things are recovered. In my case, I just restart api-server through /etc/init.d/kube-apiserver (kill the process manually should do the same thing). Etcd is always running and the data in it is not lost.
Any update on this? I cannot use kubernetes in production if this problem is not solved. Or do you have a work around, like define a service with the glusterfs endpoints?
I'm running into this too on GKE so I'm not sure if it's the api-server restarting, but just now after a loadtest and intermittently a couple of other times, after my app tries to come back up it failes to mount from gluster due to the gluster endpoints having mysteriously disappeared.
Do you mean the whole object disappeared or that it exists but is empty?
In the original YAML "IP" should be "ip"
does not show the resource On Mar 27, 2017 8:16 PM, "Tim Hockin" <:
I am trying to repro, but no luck so far.

Just checked.. IP is does come up, and it works well enough as the pods are able to mount the gluster volumes, but every once in a while I find that they're gone. On Mar 27, 2017 8:18 PM, "Tim Hockin" <:
I haven't been able to either, it happens every once in a while. I think maybe they're deleted before I notice and I only notice because a pod is fails and on restart it fails to mount it's volume On Mar 27, 2017 8:21 PM, "Tim Hockin" <:
Just checked a cluster I created an hour ago, and yup all the pods that had been created are running smoothly but the gluster endpoints resource is gone , Jonathan <: — You are receiving this because you commented. Reply to this email directly, view it on GitHub , or mute the thread
Interesting. It appears this is intentional. If you create a service with clusterIP: None and an empty selector, this will not be activated. It is trying to prevent submarine problems, such as creating an Endpoints then later creating a Service and conflicting. , Jonathan Kosgei <:
So the fix is to create a corresponding service for the endpoints?
Yes. Endpoints are usually a derived value, and in hindsight they should probably have been a sub-resource of Service. On Mar 28, 2017 7:20 AM, "Jonathan Kosgei" <:
In case you're wondering (like I was) how to attach a service to the EP the was updated to include an example service for it.
I'm having this issue again in a very different setup, basically I'm using the Nginx Ingress controller and it uses endpoints and these endpoints keep disappearing intermittently breaking all ingress to the cluster. Opened an issue at the nginx-ingress repo Saving the endpoints with a and recreating them when resolution failure occurs fixes it immediately.
this will make kubelet notready if i create a Service named glusterfs-cluster, after reboot endpoint is not lost. but the kubelet will be notready , something wrong? hope response tks.
The correct explanation should be When the cluster starts Endpoint-Controller Will check all Endpoint list Then put Endpoint name in workqueue, worker get value from workqueue, Found Endpoint no corresponding Service,Just delete Endpoint.

yes, the manually created EP resource will not be reconciled by the controller. This is the cause. Usually we do not create EP, this kind of secondary resource directly. It is created for the corresponding service resource.


Question: "strings" "testing" <ins> "github.com/Azure/go-autorest/autorest/to" </ins> "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/types"

If I look at the code of azureFile plugin, it has fsGroup as an argument, but don't see its usage anywhere I also don't see anywhere explicitely stated that azureFile volume supports fsGroup.
/sig storage
For those landing here, a separate path to get the mount point to have the right permissions is to add this annotation to the PV metadata: This makes the plugin mount the volume in the Pod under those UID,GID. Obviously, this implies that one knows what those are for their workload in the container ...
If i create a PV like this, still mounted with the default root used into POD. StorageClass PV
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale cc Let's check whether works for azure file.
/sig azure
is another way to set groupid(gid) While according to this , we also need to support Here is the code about setting fsgroup from securityContext. I will enable this feature and do a test, it should be 3 lines of code. /assign
BTW, this is all for Linux(UserID, GroupID), is there an identical setting for Windows container?
No, setting user for volumes doesn't work on Windows. Only the rw/ro options are implemented.
According to my investigation, though I put the following code in azure file , it does not take effect. The reason is azure file(cifs) mount does not support gid setting after file system mount, by default would be 0(root) So in summary, set fsGroup code in azure file is useless, user could set gid by mountOptions in storage class, here is an Let me know if you hav any concern about this issue Update: Root cause is azure file(cifs) mount does not support gid setting after file system mount, while azure disk supports.
is this true, that cifs does not support guid setting?
Of course cifs support gid setting, see example: I mean cifs does not support second time gid setting, kind of differnet than azure disk.

Anyway, I am working on a PR to support if no gid setting in azure file storage class, while if both sets, only gid from storage class would be used.
while the code is quite different than just add below 3 lines of code, actually below does not work: I will show you the PR tomorrow.


Question: // CheckNodeMemoryPressurePredicate checks if a pod can be scheduled on a node // reporting memory pressure condition. func CheckNodeMemoryPressurePredicate(pod *v1.Pod, meta interface{}, nodeInfo *schedulercache.NodeInfo) (bool, []algorithm.PredicateFailureReason, error) { <del> node := nodeInfo.Node() if node == nil { return false, nil, fmt.Errorf("node not found") } </del> var podBestEffort bool if predicateMeta, ok := meta.(*predicateMetadata); ok { podBestEffort = predicateMeta.podBestEffort

Looking into profiles of scheduler, there shows a bunch of things that we are doing unnecessarily. In particular, instead of computing the same information for a given node multiple times, we should cache it as part of NodeInfo: The information that should be cached and reuse later include: [x] node taints: [x] node conditions: disk/memory/inode pressure
Another quick optimization should be optimizing "selectHost" method: In fact it doesn't need to sort the whole list (which is O(nlgn). All we need to do is to choose the elements that have highest score, and sort only those by node name (to leave the logic that we currently have with round-robic logic of choosing from those).
And we should also limit the amount of places where we call "Parallelize" (because those are relatively expensive). The whole MR-like priorities is a step in this direction too.
OK - I've just sent out a PR fixing the first two things.
can we close this, or are there still work items here?
Some of those are not yet done.
- it's not a high priority for us (for now), so if you want it to be done soon, you probably need to find someone to do it.
- assigned.
Why "Parallelize" is considered as relatively expensive ? (comparing to MR style?)
You do MR when there are certain points that need some synchronizations. Or fully parallelized computation will always be faster.
To clarify, I was referring to the Parallelize function: Using this for many very small functions is pretty expensive - you can take a look into cpu-profiles. It happens that 20% or sth of cpu usage is spent on the operations on the synchronization channel.
Is all of the cacheing part completed? I assume "yes". What is remaining here.... Just an audit of parallelization ? Also, I assume "yes". If so I can audit where the parelellization is getting us hung up and then we can close this one out.
It's hard to tell if caching is done - when adding new priority functions/moving existing ones to MR-framework one pretty often discovers that more things need to be cached. There's my PR , that moves basic spreading priority function to MR-framework, which for some reason makes it worse. I didn't have time to look into that though.

This needs to be done, which may require modifying MR-framework slightly (allowing different type as a Map phase result).
Im not sure what the MR Framework stuff is referring to. Can you provide me a pointer ? Should we close this issue since the original patches for caching were done? Sounds like the work we're referring to now is actually more like a second iteration. if not, what work remains to be done before we can close this ticket? maybe we can make a checklist.
"MR-style" parallelization, that was mentioned before. Yeah, I guess we can close this - this is more like an ongoing process thing, when everyone who adds anything to the scheduler needs to keep it in mind.


Question: } func TestLoadBalancer(t *testing.T) { <del> cfg, ok := configFromEnv() </del> <ins> cfg, ok := configFromEnvWithPasswd() </ins> if !ok { t.Skip("No config found in environment") }

This issue was reported in the Description When configuring components of infrastructure, environment variables allow a trivial method of gathering settings. However, not all settings should be derived from these variables. For example, the file used the CFSSLCAPKPASSWORD environment variable, where a plain-text password should be found within the variable. Figure 8.1: A string password being recovered from the CFSSLCAPKPASSWORD environment variable. If this variable is configured, an attacker could potentially gain access to its stored value through environment logging, or further exploitation of the endpoint. The assessment team found seemingly sensitive environment variables in at least the following locations: Exploit Scenario Alice configures her environment with the CFSSLCAPKPASSWORD environment variable. Eve gains access to Alice’s environment and determines that the CFSSLCAPK_PASSWORD environment variable is set. Because this variable contains the private key password, Eve is able to recover the private key and use it trivially, leading to further exploitation of Alice’s environment. Recommendation Short term, ensure highly sensitive information is not collected directly from environment variables for long periods of time. Long term, consider using Kubernetes secrets for all areas of the system. This will allow users to have one unified interface and location for all secrets, and avoid accidentally exposing secrets to other users within a host system. Anything else we need to know?: See for current status of all issues created from these findings. The vendor gave this issue an ID of TOB-K8S-005 and it was finding 9 of the report. The vendor considers this issue Medium Severity. To view the original finding, begin on page 35 of the Environment: Kubernetes version: 1.13.4
/sig auth /sig cloud-provider /area security /kind bug /remove-kind feature
Also secrets
Bug triage for 1.17 here with a gentle reminder that code freeze for this release is on November 14. Is this issue still intended for 1.17?
is it ok to close this issue also cc:
no, still remains
cc
seen this yet?
Thanks for the attention, we are using it as a fallback option mostly for test and dev env, we are working to remove them.
AFAIK is the only use for this testing. Would it be appropriate to move this code into ?
Submitted a PR to move password retrieval to


Question: } func TestCheckGeneratedNameError(t *testing.T) { <ins> ctx := genericapirequest.WithRequestInfo(genericapirequest.NewContext(), &genericapirequest.RequestInfo{ Resource: "foos", }) </ins> testStrategy, _ := newStrategy("10.0.0.0/16", false) expect := errors.NewNotFound(api.Resource("foos"), "bar") <del> if err := rest.CheckGeneratedNameError(testStrategy, expect, &api.Service{}); err != expect { </del> <ins> if err := rest.CheckGeneratedNameError(ctx, testStrategy, expect, &api.Service{}); err != expect { </ins> t.Errorf("NotFoundError should be ignored: %v", err) } expect = errors.NewAlreadyExists(api.Resource("foos"), "bar") <del> if err := rest.CheckGeneratedNameError(testStrategy, expect, &api.Service{}); err != expect { </del> <ins> if err := rest.CheckGeneratedNameError(ctx, testStrategy, expect, &api.Service{}); err != expect { </ins> t.Errorf("AlreadyExists should be returned when no GenerateName field: %v", err) } expect = errors.NewAlreadyExists(api.Resource("foos"), "bar") <del> if err := rest.CheckGeneratedNameError(testStrategy, expect, &api.Service{ObjectMeta: metav1.ObjectMeta{GenerateName: "foo"}}); err == nil || !errors.IsServerTimeout(err) { </del> <ins> if err := rest.CheckGeneratedNameError(ctx, testStrategy, expect, &api.Service{ObjectMeta: metav1.ObjectMeta{GenerateName: "foo"}}); err == nil || !errors.IsAlreadyExists(err) { </ins> t.Errorf("expected try again later error: %v", err) } }

Spawned from is converting an AlreadyExists error into a server timeout error if it has a generateName That means that this returns really confusing errors:
/cc
- a simple if for the objectMeta.Name seems to do the trick... is this enough? or were you looking for better way instead of NewServerTimeoutForKind ?
if no name was provided originally, the alreadyexists error occurs after the name field is populated with a generated name, so returning in that case makes this entire function a no-op in every case I can find. I just think inferring a ServerTimeout error from an AlreadyExists error and the presence of GenerateName is really odd
This needs to be triaged as a release-blocker or not for 1.5
Not a blocker
thanks
Is it appropriate to move this to the next milestone or clear the 1.5 milestone? (and remove the non-release-blocker tag as well)
Issues go stale after 30d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
/remove-lifecycle rotten
/help
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cli
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with .

Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Should we return another instead of to include the generate name information ?
seems like the correct error in other cases involving an object that already exists with that name.
Looks like is not active due to age. If anyone is interested, please feel free to work on this! Like mentioned in , a unit test along with the required changes would be good to add.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Would be really helpful if solved :) /remove-lifecycle rotten
I am new to Kubernetes community and was going through role board recommendations on older threads - If everyone is ok, can I pick this issue and work on it.
It seems already created PR for this issue - It doesn't seem to have merged. - Let me know if you are still working, else I can take a stab (Though most of the work is already done by you) to further drive this.
I'm not working on it. Please take it forward. Thanks
Thanks for the confirmation. I will work on this.
I have taken a look at the complete flow of the issue. From what I understand, there are a couple of choices Option - (1) Validate in function if an object has both name and generateName and fail the request if that is the case. Option - (2) in , set generateName to empty if it also has a name. This will make sure that the error in CheckGeneratedNameError is (when the generated name is conflicting with the already existing name of some other object). This is essentially the approach took in this , which no longer is actively worked on. /cc
Issues go stale after 90d of inactivity.

Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/lifecycle frozen
/needs-triage It's kind of embarrassing that this is still open.
(I think the fix may be to delete the mentioned function entirely-- why would we describe this situation as a timeout error?)
do you know someone who's interested working on this? We will come back on this issue in the next triage meeting.
/help
/triage accepted
The origin of :


Question: } var extraPods *v1.PodList <ins> var volumeAttachments *storagev1.VolumeAttachmentList var pvs *v1.PersistentVolumeList var nodes *v1.NodeList </ins> func CreateTestClient() *fake.Clientset { fakeClient := &fake.Clientset{}

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Background: The attach/detach controller create objects for CSI Volumes to indicate that a volume needs to be attached to a node. It deletes the object to indicate that the volume should be detached. Problem: If the Attach/Detach controller crashes, and in the meantime the pod using that volume is deleted, when the AD controller restarts, it won't know that the volume was previously attached and should be cleaned up. What you expected to happen: Proposal: We should have the AD controller periodically list all objects and for any that are not accounted for, update its internal actual/desired state caches so that the volume is gracefully detached. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: /sig storage Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
could I take it?
/assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available .

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/lifecycle frozen /reopen
Reopened this issue. <detailsIn response to : /lifecycle frozen /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
should address this


Question: }) } <ins> // MaxInFlight limits the number of in-flight requests to buffer size of the passed in channel. func MaxInFlightLimit(c chan bool, longRunningRequestRE *regexp.Regexp, handler http.Handler) http.Handler { if c == nil { return handler } return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { if longRunningRequestRE.MatchString(r.URL.Path) { // Skip tracking long running events. handler.ServeHTTP(w, r) return } select { case c <- true: defer func() { <-c }() handler.ServeHTTP(w, r) default: tooManyRequests(w) } }) } </ins> // RateLimit uses rl to rate limit accepting requests to 'handler'. func RateLimit(rl util.RateLimiter, handler http.Handler) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, req *http.Request) {

Related with issue , the API server should have a cap on the number of outstanding operations/requests it's handling, and return an error otherwise. A simple semaphore is an idea for this.
+1 !
The good news/bad news on this is that there is a rate limiter in the API server (), but the defaults for it seem too high: 10 qps average, 200 burst (). We could start by lowering the burst level to something smaller, like 30. It may be interesting to also differentiate between GET and PUT/POST requests.
I think that we need limits on the number of outstanding requests, not the number of QPS.
-An max outstanding would be useful. But it may force us to have separate buckets for different types of requests, i.e. a small number of long running requests causing API server to reject short requests. -Heard that you might be looking into adding support for flow control (or something like it?). We may need it soon to help avoid bugs like runaway node status updates (see issue ).
I had not - the things we were looking at were the bastion changes to apiserver which will break logs and proxying into their own rest resources so they can be granularly controlled. Being able to timebox and classify requests differently would be good. ----- Original Message -----
- Ah, np, I probably misheard.
What are our plans for handling this client side? Do we want clients to handle "you're being throttled" error themselves, or do we plan to handle it in the library? I'm asking because currently kubelet does not have a 'short' retry loop, hence it tries to report NodeStatus every X (2) seconds, whether it succeed or not. Neither we have a sidechannel for heartbeat messages. This would mean, that if API server will be under heavy load, and with not-negligible probability NodeStatus reporting will be throttled, we may incorrectly mark a Node as unreachable. Currently it's not a huge problem, because X << Y, where Y is the time we wait before we decide a Node is unreachable. But because of the effort somewhere close to it'll probably get worse.
The client library should have options for retry on client.Config which include the ability to disable the behavior. Automatic retry is probably an "opt-out" behavior.

----- Original Message -----
says there are people working on rate limiting client connections to API server.
I have in the past adjusted the rate limiter to support the creation of large clusters. I also the logic. If you adjust the rate limiter settings, please make sure it is still possible to create large clusters.
I'm closing this as fixed, given the original subject. If we want to expand client-side support for retry, please open a dedicated issue. (Also note that QPS based throttling was to the client recently too)